Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
f24d50eed4 | ||
 clawbot
|
719691d1e8 | ||
|
clawbot
|
aede434f6a | ||
|
clawbot
|
6850cb0e9f | ||
|
clawbot
|
d7c62f2854 | ||
|
clawbot
|
c66277b111 | ||
|
sneak
|
f6435b1a73 | ||
|
clawbot
|
2349b2a081 | ||
|
clawbot
|
42e06bf298 |
No files matched your search
+11
-2
@@ -136,10 +136,19 @@ RUN mkdir /etc/sysusers.d \
|
||||
# ca-certificates, nix-bin and runit, from Ubuntu's archive. apt fetches
|
||||
# them over plain HTTP and checks each one against the archive's signed
|
||||
# package lists, so no CA certificates are needed.
|
||||
RUN apt-get update \
|
||||
#
|
||||
# nix-bin depends on xz-utils, whose install links man pages with
|
||||
# update-alternatives. The image's dpkg config drops man pages, so each
|
||||
# link warns. z-man, read after /etc/dpkg/dpkg.cfg.d/excludes, re-includes
|
||||
# the man1 pages for this install alone; the same step deletes them again,
|
||||
# so no man page or package is added.
|
||||
RUN echo 'path-include=/usr/share/man/man1/*' > /etc/dpkg/dpkg.cfg.d/z-man \
|
||||
&& apt-get update \
|
||||
&& DEBIAN_FRONTEND=noninteractive apt-get install --yes --no-install-recommends \
|
||||
ca-certificates nix-bin runit \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
&& rm -rf /var/lib/apt/lists/* /etc/dpkg/dpkg.cfg.d/z-man \
|
||||
&& find /usr/share/man/man1 ! -type d -delete \
|
||||
&& find /etc/alternatives -type l -xtype l -delete
|
||||
|
||||
# Nix run by root expects a group of build users, which nix-bin does not
|
||||
# create; with the setting empty, root's builds run without them.
|
||||
|
||||
+56
-44
@@ -12,8 +12,8 @@ configured only by environment variables, that provides:
|
||||
- R1: rate limits per minute, per hour and per day
|
||||
- R2: netblocks that bypass rate limiting
|
||||
- R3: alerts on attacks through webhook, Slack or ntfy
|
||||
- R4: alerts on anomalies: requests or bytes per minute or hour, for given IPs
|
||||
or netblocks, crossing a threshold
|
||||
- R4: alerts on anomalies: requests made or bytes transferred per minute or
|
||||
hour, for given IPs or netblocks, crossing a threshold
|
||||
- R5: temporary blocks for abusers, permanent bans for repeat offenders
|
||||
- R6: one or more RBLs or IP reputation APIs
|
||||
- R7: AS number lookup
|
||||
@@ -24,14 +24,19 @@ configured only by environment variables, that provides:
|
||||
|
||||
## Verdict
|
||||
|
||||
Nothing existing is suitable. No surveyed tool meets R8 (scaling limits by AS
|
||||
number or country) or the bytes half of R4 at all, and the tools that come
|
||||
closest on the rest fail R10: they need several containers, YAML or a web UI.
|
||||
The recommendation is to build a small single-binary sidecar; `SPEC.md` in this
|
||||
directory begins that spec. It reuses the mature parts of the surveyed field as
|
||||
libraries and data sources (Coraza and the OWASP Core Rule Set for attack
|
||||
detection, DNSBLs, AbuseIPDB, optionally a CrowdSec decision feed) instead of
|
||||
reimplementing them.
|
||||
Nothing existing is suitable. No surveyed tool meets R8, which asks for limits
|
||||
scaled by AS number or country. None meets R4's alerts on bytes transferred. The
|
||||
tools that come closest on the rest fail R10: they need several containers, YAML
|
||||
or a web UI. The recommendation is to build a small single-binary sidecar;
|
||||
`SPEC.md` in this directory begins that spec.
|
||||
|
||||
The sidecar reuses the mature parts of the surveyed field instead of
|
||||
reimplementing them. It takes these as libraries and data sources:
|
||||
|
||||
- Coraza and the OWASP Core Rule Set, for attack detection
|
||||
- DNSBLs
|
||||
- AbuseIPDB
|
||||
- optionally, a CrowdSec decision feed
|
||||
|
||||
Closest existing options, and why each still falls short:
|
||||
|
||||
@@ -45,13 +50,16 @@ Closest existing options, and why each still falls short:
|
||||
external blacklist URLs, optional CrowdSec), R7 partly (AS number used for
|
||||
blacklist and whitelist decisions), R9 (ModSecurity with the Core Rule
|
||||
Set, or Coraza plugin), env-var settings.
|
||||
- Fails: R8 (AS number and country can only allow or deny, never scale a
|
||||
limit), R4 (no volume or byte threshold alerts in the free edition;
|
||||
reporting is a paid feature), R5 escalation, and R10 in spirit: since 1.6
|
||||
it needs a `bunkerweb` container plus a `bw-scheduler` container and a
|
||||
database, or the all-in-one image that bundles nginx, scheduler, UI and
|
||||
Redis in one container. It is designed to be the front door for many
|
||||
sites, not a per-app sidecar. AGPL-3.0.
|
||||
- Fails:
|
||||
- R8: AS number and country can only allow or deny, never scale a limit.
|
||||
- R4: no alerts on request-count or byte thresholds in the free edition;
|
||||
reporting is a paid feature.
|
||||
- R5: no escalation for repeat offenders.
|
||||
- R10: since 1.6 it is not one plain sidecar. It needs a `bunkerweb`
|
||||
container, a `bw-scheduler` container and a database, or the
|
||||
all-in-one image that bundles nginx, the scheduler, the UI and Redis.
|
||||
- It is designed as the proxy in front of many sites, not as a sidecar for
|
||||
one app. AGPL-3.0.
|
||||
- Unverified: whether several rates (minute, hour, day) can be stacked on
|
||||
the same URL pattern; the documented design is one rate per URL pattern.
|
||||
- CrowdSec is the strongest detection and ban engine, but it is not a proxy.
|
||||
@@ -66,16 +74,20 @@ Closest existing options, and why each still falls short:
|
||||
listed AS numbers is possible, but each is a hand-written YAML scenario,
|
||||
it reacts after the fact by banning, and it is not an inline limiter that
|
||||
answers 429.
|
||||
- Fails: R10 (needs the security engine container with persistent state, log
|
||||
acquisition from traefik, a bouncer such as the traefik plugin, and YAML
|
||||
for acquisition, profiles, scenarios and notifications), bytes half of R4.
|
||||
- Fails:
|
||||
- R10, because it needs:
|
||||
- the security engine container with persistent state
|
||||
- log acquisition from traefik
|
||||
- a bouncer such as the traefik plugin
|
||||
- YAML for acquisition, profiles, scenarios and notifications
|
||||
- R4's alerts on bytes.
|
||||
- CrowdSec plus traefik's own `rateLimit` middleware is the best combination
|
||||
with no new code. It gives inline limiting (one window per middleware, keyed
|
||||
by IP, with `sourceCriterion` exclusions), bans with escalation, alerts and
|
||||
reputation. It still fails R8, the bytes half of R4, R1's three stacked
|
||||
windows only by chaining three middlewares per router, and R10 entirely: the
|
||||
configuration lives in traefik labels and CrowdSec YAML on each host, not in
|
||||
one sidecar's environment.
|
||||
reputation lookups. It still fails R8, R4's alerts on bytes, and R10 entirely:
|
||||
the configuration lives in traefik labels and CrowdSec YAML on each host, not
|
||||
in one sidecar's environment. It meets R1's three windows only by chaining
|
||||
three middlewares per router.
|
||||
|
||||
## Candidate by candidate
|
||||
|
||||
@@ -87,10 +99,10 @@ Closest existing options, and why each still falls short:
|
||||
cached, streamed decision list (recommended), standalone against the
|
||||
central API, or AppSec only. Supports captcha remediation.
|
||||
- Covers: R5, R6, R9, R3 and R7 through the engine (see Verdict).
|
||||
- Misses: the plugin itself does no rate limiting; R8; R4 bytes.
|
||||
- Misses: the plugin itself does no rate limiting; R8; R4's alerts on bytes.
|
||||
- Configuration: traefik static config to load the plugin, dynamic config or
|
||||
labels for the middleware, CrowdSec YAML for everything else.
|
||||
- Sidecar fit: no. It lives inside traefik, plus a separate engine
|
||||
- Sidecar fit: no. It runs inside traefik and needs a separate engine
|
||||
container.
|
||||
- Maturity: widely used (about 900 stars, listed in the traefik plugin
|
||||
catalog, documented by CrowdSec itself), actively maintained. Traefik
|
||||
@@ -115,7 +127,7 @@ Closest existing options, and why each still falls short:
|
||||
- What it is: nginx-derived (Tengine) reverse proxy with a proprietary
|
||||
semantic detection engine and a web console. About 22,000 stars.
|
||||
- Covers: R9 well, basic per-IP rate limiting with block duration, bot
|
||||
challenge, authentication gate.
|
||||
challenge, authentication in front of the app.
|
||||
- Misses: R8, R7, R4; R1 has one window per rule. Country blocking, alert
|
||||
messages, syslog forwarding and the better reputation database are paid
|
||||
(Lite about 10 USD per month, Pro about 100 USD per month); the free
|
||||
@@ -128,7 +140,7 @@ Closest existing options, and why each still falls short:
|
||||
- Coraza (`corazawaf/coraza`) and Coraza-based proxies
|
||||
- What it is: a Go library that implements the ModSecurity rule language and
|
||||
runs the OWASP Core Rule Set. OWASP project, actively maintained, the
|
||||
successor path now that ModSecurity is in maintenance only.
|
||||
successor to ModSecurity, which is now in maintenance only.
|
||||
- Packagings: `coraza-caddy` (Caddy module), `coraza-spoa` (HAProxy),
|
||||
`coraza-proxy-wasm` (Envoy), a traefik WASM plugin, and
|
||||
`coreruleset/coraza-crs-docker` (Caddy plus Coraza plus the Core Rule Set,
|
||||
@@ -142,7 +154,7 @@ Closest existing options, and why each still falls short:
|
||||
- ModSecurity Core Rule Set containers (`owasp/modsecurity-crs`)
|
||||
- What it is: official images of Apache or nginx with ModSecurity and the
|
||||
Core Rule Set, as a reverse proxy configured by env vars (`BACKEND`,
|
||||
paranoia level, anomaly thresholds, many more). Current images carry
|
||||
paranoia level, anomaly thresholds, many more). Current images contain
|
||||
ModSecurity 2.9.x on Apache and Core Rule Set 4.x.
|
||||
- Covers: R9, and R10 (single container, env vars, one backend).
|
||||
- Misses: R1 to R8.
|
||||
@@ -152,8 +164,8 @@ Closest existing options, and why each still falls short:
|
||||
- Anubis (`TecharoHQ/anubis`), 1.27 current
|
||||
- What it is: a single-binary reverse proxy that makes browsers solve a
|
||||
proof-of-work challenge before passing them to `TARGET`. Aimed at
|
||||
scrapers, which is likely a large share of unwanted traffic on a public
|
||||
code forge.
|
||||
scrapers, which are likely a large share of the unwanted traffic on a
|
||||
public code forge.
|
||||
- Covers: R10 well (one container, env vars for listener, target,
|
||||
difficulty, cookies). Policy rules can match on path, user agent, headers,
|
||||
IP ranges, and with the vendor's hosted data service also AS number and
|
||||
@@ -180,13 +192,13 @@ Closest existing options, and why each still falls short:
|
||||
temporary ban, no escalation, state lost on traefik restart) and R2
|
||||
(allowlist). Nothing else.
|
||||
- `juitde/traefik-plugin-fail2ban`, fail2ban connector plugins, GeoBlock
|
||||
plugins (country allow or deny through an external lookup API): same
|
||||
shape, each one small feature, configured in traefik, state in memory.
|
||||
plugins (country allow or deny through an external lookup API): each adds
|
||||
one small feature, is configured in traefik and keeps its state in memory.
|
||||
- Sidecar fit: no; they live inside traefik.
|
||||
- Traefik built-in middlewares
|
||||
- `rateLimit` (one average-and-burst window per middleware, optional Redis
|
||||
in traefik 3.x), `inFlightReq`, `ipAllowList`. No bans, alerts, reputation
|
||||
or AS number awareness.
|
||||
lookups or AS number lookups.
|
||||
- caddy-waf (`fabriziosalmi/caddy-waf`), 0.4.x
|
||||
- What it is: a Caddy module with regex rules and anomaly scoring, per-IP
|
||||
and per-path rate limiting with one configurable window, IP and DNS
|
||||
@@ -198,12 +210,12 @@ Closest existing options, and why each still falls short:
|
||||
- open-appsec (Check Point)
|
||||
- Machine-learning WAF agent attached to nginx, Kong, Envoy or similar, with
|
||||
a declarative policy file or the vendor's cloud console. Covers R9 only;
|
||||
rate limiting and richer features are in paid tiers. Not a sidecar in the
|
||||
required sense.
|
||||
rate limiting and richer features are in paid tiers. It attaches to an
|
||||
existing proxy, so it is not the sidecar R10 asks for.
|
||||
- iocaine
|
||||
- Serves generated garbage pages to clients the fronting proxy classifies as
|
||||
scrapers. Not a limiter, WAF or ban tool; out of scope except as a
|
||||
curiosity for scraper traffic.
|
||||
scrapers. Not a limiter, WAF or ban tool; out of scope, and only of
|
||||
passing interest for scraper traffic.
|
||||
- Pangolin
|
||||
- A tunnelled access platform that bundles traefik and optionally CrowdSec.
|
||||
Replaces the ingress rather than adding a sidecar; out of scope.
|
||||
@@ -214,17 +226,17 @@ Closest existing options, and why each still falls short:
|
||||
Everyone else offers one window per rule or none.
|
||||
- R2 bypass netblocks: BunkerWeb, CrowdSec, traefik, fail2ban plugins.
|
||||
- R3 alerts: CrowdSec (best), BunkerWeb (denied requests only), SafeLine (paid).
|
||||
- R4 volume and byte anomalies: none. CrowdSec can approximate request-count
|
||||
anomalies with a custom scenario; nothing handles bytes.
|
||||
- R5 temporary then permanent: CrowdSec only. BunkerWeb has fixed or permanent,
|
||||
not both by history.
|
||||
- R4 alerts on request counts and bytes: none. CrowdSec can approximate alerts
|
||||
on request counts with a custom scenario; no surveyed tool alerts on bytes.
|
||||
- R5 temporary then permanent: CrowdSec only. BunkerWeb's ban is either one
|
||||
fixed length or permanent, and does not change with a client's history.
|
||||
- R6 reputation: CrowdSec, BunkerWeb, go-away, caddy-waf (static lists).
|
||||
- R7 AS number lookup: CrowdSec (enrichment), BunkerWeb and caddy-waf (allow or
|
||||
deny), Anubis (hosted service).
|
||||
- R8 biased thresholds: none.
|
||||
- R9 attack detection: Coraza or ModSecurity with the Core Rule Set (open,
|
||||
standard), CrowdSec AppSec, SafeLine (closed engine), BunkerWeb (wraps the
|
||||
first).
|
||||
standard), CrowdSec AppSec, SafeLine (closed engine), BunkerWeb (wraps Coraza
|
||||
or ModSecurity).
|
||||
- R10 env-var sidecar: `owasp/modsecurity-crs`, `coraza-crs-docker`, Anubis.
|
||||
None of these covers anything beyond its one job.
|
||||
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
.PHONY: bootstrap setup test lint fmt fmt-check tidy check docker hooks build \
|
||||
run example-app
|
||||
.PHONY: bootstrap setup test lint fmt fmt-check tidy check docker publish \
|
||||
hooks build run example-app
|
||||
|
||||
# Makefile targets are thin shims; the implementations live in script/
|
||||
# per the scripts-to-rule-them-all pattern (see the Entrypoints section
|
||||
# of README.md). tidy writes go.mod and go.sum as `go mod tidy` does,
|
||||
# which test checks. build and run are for working on the code by hand;
|
||||
# example-app checks the image with an app built on it.
|
||||
# example-app checks the image with an app built on it; publish builds
|
||||
# the image on the Docker host and pushes it to the registry.
|
||||
|
||||
bootstrap:
|
||||
@script/bootstrap
|
||||
@@ -34,6 +35,9 @@ check:
|
||||
docker:
|
||||
@script/docker
|
||||
|
||||
publish:
|
||||
@script/publish
|
||||
|
||||
hooks:
|
||||
@script/install-precommit
|
||||
|
||||
|
||||
@@ -25,11 +25,11 @@ without an answer in time, `rproxy` passes the request to the app.
|
||||
It protects against:
|
||||
|
||||
- Scrapers that fetch every page of a site, all day. Each client gets a limit on
|
||||
requests and on bytes per minute, per hour and per day. The defaults leave
|
||||
room for a person reading a site for hours. They catch a scraper that keeps up
|
||||
one request every 2 seconds. A client over a limit is refused and banned: for
|
||||
a while at first, for three times as long at each repeat, and permanently when
|
||||
it keeps at it.
|
||||
requests and on body bytes per minute, per hour and per day. The defaults
|
||||
leave room for a person reading a site for hours. They catch a scraper that
|
||||
keeps up one request every 2 seconds. A client that breaks a limit is banned:
|
||||
for a while at first, for three times as long at each repeat, and permanently
|
||||
when it keeps at it.
|
||||
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
|
||||
default rule file bans such a client at its first probe, and its next request
|
||||
during the ban makes the ban permanent. Your own rules, or a list of trap
|
||||
@@ -68,8 +68,9 @@ Not built, and not planned:
|
||||
|
||||
### The image
|
||||
|
||||
No image is published. Build it from a clone. `make docker` runs the tests and
|
||||
the linter, then builds the image, tagged `smallwebwaf`.
|
||||
The image is published at `git.eeqj.de/sneak/smallwebwaf`. You can also build it
|
||||
from a clone. `make docker` runs the tests and the linter, then builds the
|
||||
image, tagged `smallwebwaf`.
|
||||
|
||||
```sh
|
||||
git clone https://git.eeqj.de/sneak/smallwebwaf.git
|
||||
@@ -172,7 +173,7 @@ does.
|
||||
the request have `"action":"forward"`.
|
||||
- To see a refusal without banning yourself, run `discriminator` with
|
||||
`SWW_MODE=observe` and ask for `/.env`. The request reaches the app.
|
||||
`discriminator`'s log lines have `"would_action":"banned"` and
|
||||
`discriminator`'s log lines have `"would_action":"temporarily_banned"` and
|
||||
`"rule_ids":["env-file"]`. A ban you get in `enforce` mode is lifted as
|
||||
"Clients and bans" below describes.
|
||||
- With `SWW_METRICS_TOKEN` set, this answers with the metrics, from a host on
|
||||
@@ -209,6 +210,10 @@ account, a key, a token or an alert destination stays off until you give one.
|
||||
percentage is a whole number from 0 to 100. A path prefix starts with `/`.
|
||||
- `off` switches a limit, a threshold or a timeout off where the table says a
|
||||
setting takes it.
|
||||
- The byte limits and the byte anomaly thresholds count only body bytes: the
|
||||
app's response bodies, the visitors' request bodies, or both, as
|
||||
`SWW_BYTES_COUNT` says. Headers are never counted. What the visitor sent over
|
||||
a WebSocket counts as request bytes, and what the app sent as response bytes.
|
||||
- An invalid setting stops the process at start with one log line naming the
|
||||
variable and what is wrong with the value. The settings in effect are logged
|
||||
on one line at start, with tokens, keys and webhook paths shown as `********`.
|
||||
@@ -216,7 +221,7 @@ account, a key, a token or an alert destination stays off until you give one.
|
||||
Every setting the code reads, with its default:
|
||||
|
||||
| Variable | Default | What it does |
|
||||
| ---------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| ---------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `discriminator`. |
|
||||
| `SWRP_WAF_BACKEND` | required | `discriminator`'s URL, such as `http://discriminator:8080`. |
|
||||
| `SWRP_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. |
|
||||
@@ -225,8 +230,8 @@ Every setting the code reads, with its default:
|
||||
| `SWRP_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `discriminator`'s answer to a question and to a report, in milliseconds. |
|
||||
| `SWRP_DEFAULT_DISPOSITION` | `allow` | What `rproxy` does with a request when `discriminator` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. |
|
||||
| `SWRP_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. |
|
||||
| `SWRP_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`; a larger one is answered `502`, or cut off once it has started. |
|
||||
| `SWRP_MAX_REQUEST_TIME_MS` | `60000` | How long the visitor may take to send its request line and headers, and then the whole request to reach the app, in milliseconds or `off`. |
|
||||
| `SWRP_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
|
||||
| `SWRP_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
|
||||
| `SWRP_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
|
||||
| `SWW_LISTEN_ADDR` | `:8080` | Where `discriminator` listens; leave the host part empty. |
|
||||
| `SWW_INSTANCE_NAME` | the container's host name | The name in every line `discriminator` logs, and in every alert and metric, such as `fsn1/discriminator`. |
|
||||
@@ -235,35 +240,36 @@ Every setting the code reads, with its default:
|
||||
| `SWW_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. |
|
||||
| `SWW_MAX_BANS` | `5000` | The most bans `discriminator` keeps of its own; bans an admin made do not count. |
|
||||
| `SWW_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. |
|
||||
| `SWW_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients the request and byte limits neither count nor refuse. |
|
||||
| `SWW_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients are exempt from rate limiting and the byte limits. |
|
||||
| `SWW_DENY_NETS` | empty | Netblocks whose clients are refused. |
|
||||
| `SWW_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; one more bans it. |
|
||||
| `SWW_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; one more bans it. |
|
||||
| `SWW_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; one more bans it. |
|
||||
| `SWW_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes whose requests the request and byte limits neither count nor refuse, such as `/assets/`. |
|
||||
| `SWW_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes a client's requests may carry in a minute, a size or `off`; the request that takes a client past any byte limit bans it. |
|
||||
| `SWW_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes a client's requests may carry in an hour, a size or `off`. |
|
||||
| `SWW_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes a client's requests may carry in a day, a size or `off`. |
|
||||
| `SWW_BYTES_COUNT` | `both` | Which body bytes the byte limits count: `response`, `request` or `both`. |
|
||||
| `SWW_ERROR_BURST_THRESHOLD` | `30` | The most refusals by a rule file, a trap path or the Core Rule Set a client may get in a minute, a count or `off`; one more bans it. |
|
||||
| `SWW_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; the next request is refused and bans the client. |
|
||||
| `SWW_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; the next request is refused and bans the client. |
|
||||
| `SWW_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; the next request is refused and bans the client. |
|
||||
| `SWW_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. |
|
||||
| `SWW_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes counted for a client in a minute, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||
| `SWW_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes counted for a client in an hour, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||
| `SWW_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes counted for a client in a day, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||
| `SWW_BYTES_COUNT` | `both` | What the byte limits and the byte anomaly thresholds count: `response`, the app's response bodies; `request`, the visitors' request bodies; or `both`. What the visitor sent over a WebSocket counts as request bytes, and what the app sent as response bytes. |
|
||||
| `SWW_ERROR_BURST_THRESHOLD` | `30` | The most refusals a client may get in a minute from a rule file, a trap path or the Core Rule Set, a count or `off`. The next one bans the client. |
|
||||
| `SWW_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. |
|
||||
| `SWW_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. |
|
||||
| `SWW_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A limit broken again within this time after the last such ban ended gets a ban three times as long. |
|
||||
| `SWW_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A client that breaks a limit again within this time after its last such ban ended gets a ban three times as long. |
|
||||
| `SWW_MAX_BAN_DURATION` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. |
|
||||
| `SWW_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. |
|
||||
| `SWW_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
|
||||
| `SWW_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs`, a web service, `file`, or `off`. |
|
||||
| `SWW_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
|
||||
| `SWW_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
||||
| `SWW_LOOKUP_TIMEOUT` | `800ms` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take. |
|
||||
| `SWW_ADD_LOOKUP_HEADERS` | `false` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
||||
| `SWW_DENIED_COUNTRIES` | empty | Countries whose clients are refused, such as `cn,ru`. |
|
||||
| `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
||||
| `SWW_ASN_LIMIT_PERCENT` | empty | AS numbers whose clients get a percentage of every request and byte limit, such as `AS14061:50,AS16276:25`. |
|
||||
| `SWW_COUNTRY_LIMIT_PERCENT` | empty | The same by country, such as `cn:25`. |
|
||||
| `SWW_ASN_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `SWW_ASN_LIMIT_PERCENT`. |
|
||||
| `SWW_COUNTRY_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `SWW_COUNTRY_LIMIT_PERCENT`. |
|
||||
| `SWW_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
||||
| `SWW_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. |
|
||||
| `SWW_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. |
|
||||
| `SWW_ROUTEWATCH_URL` | unset | The URL of a routewatch instance, which gives each client's `asn=` and `prefix=` tags. Unset, nothing asks routewatch. |
|
||||
| `SWW_ASN_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. |
|
||||
| `SWW_COUNTRY_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. |
|
||||
| `SWW_UNKNOWN_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. |
|
||||
| `SWW_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; where it and `SWW_ASN_LIMIT_PERCENT` both give a percentage, the lower applies. |
|
||||
| `SWW_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_TAG_LIMIT_PERCENT` (the lowest wins). |
|
||||
| `SWW_RULES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. |
|
||||
| `SWW_RULES_ENABLED` | `true` | `false` reads no rule file. |
|
||||
| `SWW_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. |
|
||||
@@ -274,6 +280,7 @@ Every setting the code reads, with its default:
|
||||
| `SWW_WAF_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
|
||||
| `SWW_WAF_BODY_LIMIT` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. |
|
||||
| `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
|
||||
| `SWW_APPS` | unset | The apps each hostname serves, such as `git.example.com:gitea;cloud.example.com:gitea`; only those apps' fixes apply there. Unset, every app's fixes apply to every hostname. |
|
||||
| `SWW_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
|
||||
| `SWW_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. |
|
||||
| `SWW_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. |
|
||||
@@ -305,32 +312,32 @@ Every setting the code reads, with its default:
|
||||
| `SWW_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. |
|
||||
| `SWW_ALERT_NTFY_URL` | unset | An ntfy topic each alert is published to, as its full URL, such as `https://ntfy.sh/my-alerts`. |
|
||||
| `SWW_ALERT_NTFY_TOKEN` | unset | An ntfy access token sent with each alert. |
|
||||
| `SWW_ALERT_EVENTS` | `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. |
|
||||
| `SWW_ALERT_EVENTS` | `ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. |
|
||||
| `SWW_ALERT_COOLDOWN` | `15m` | How long after an alert its repeats are held back and counted, a duration or `off`. |
|
||||
| `SWW_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. |
|
||||
| `SWW_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Alert when one client makes more requests than this in a minute, a count. |
|
||||
| `SWW_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Alert when one client makes more requests than this in an hour, a count. |
|
||||
| `SWW_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Alert when one client's requests carry more bytes than this in a minute, a size. |
|
||||
| `SWW_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Alert when one client's requests carry more bytes than this in an hour, a size. |
|
||||
| `SWW_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one client in a minute, a size. |
|
||||
| `SWW_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one client in an hour, a size. |
|
||||
| `SWW_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Alert when the netblock around a client makes more requests than this in a minute, a count. |
|
||||
| `SWW_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Alert when the netblock around a client makes more requests than this in an hour, a count. |
|
||||
| `SWW_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Alert when the netblock around a client carries more bytes than this in a minute, a size. |
|
||||
| `SWW_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Alert when the netblock around a client carries more bytes than this in an hour, a size. |
|
||||
| `SWW_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the netblock around a client in a minute, a size. |
|
||||
| `SWW_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the netblock around a client in an hour, a size. |
|
||||
| `SWW_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_ANOMALY_NET_` thresholds count, from 0 to 32. |
|
||||
| `SWW_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_ANOMALY_NET_` thresholds count, from 0 to 128. |
|
||||
| `SWW_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Alert when one AS number makes more requests than this in a minute, a count. |
|
||||
| `SWW_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Alert when one AS number makes more requests than this in an hour, a count. |
|
||||
| `SWW_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Alert when one AS number carries more bytes than this in a minute, a size. |
|
||||
| `SWW_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Alert when one AS number carries more bytes than this in an hour, a size. |
|
||||
| `SWW_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one AS number in a minute, a size. |
|
||||
| `SWW_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one AS number in an hour, a size. |
|
||||
| `SWW_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Alert when the whole service gets more requests than this in a minute, a count. |
|
||||
| `SWW_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Alert when the whole service gets more requests than this in an hour, a count. |
|
||||
| `SWW_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Alert when the whole service carries more bytes than this in a minute, a size. |
|
||||
| `SWW_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Alert when the whole service carries more bytes than this in an hour, a size. |
|
||||
| `SWW_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the whole service in a minute, a size. |
|
||||
| `SWW_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the whole service in an hour, a size. |
|
||||
| `SWW_WATCH_NETS` | empty | Named netblocks the `SWW_WATCH_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. |
|
||||
| `SWW_WATCH_REQUESTS_PER_MINUTE` | `off` | Alert when a watched netblock makes more requests than this in a minute, a count. |
|
||||
| `SWW_WATCH_REQUESTS_PER_HOUR` | `off` | Alert when a watched netblock makes more requests than this in an hour, a count. |
|
||||
| `SWW_WATCH_BYTES_PER_MINUTE` | `off` | Alert when a watched netblock carries more bytes than this in a minute, a size. |
|
||||
| `SWW_WATCH_BYTES_PER_HOUR` | `off` | Alert when a watched netblock carries more bytes than this in an hour, a size. |
|
||||
| `SWW_WATCH_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for a watched netblock in a minute, a size. |
|
||||
| `SWW_WATCH_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for a watched netblock in an hour, a size. |
|
||||
|
||||
[`docs/settings.md`](docs/settings.md) gives the values each setting takes in
|
||||
full, and the limits that are fixed rather than settings.
|
||||
@@ -413,8 +420,7 @@ services:
|
||||
SWW_RATE_LIMIT_PER_HOUR: "600"
|
||||
SWW_RATE_LIMIT_PER_DAY: "3000"
|
||||
# Hosting networks and one country get a share of those limits.
|
||||
SWW_ASN_LIMIT_PERCENT: AS14061:50,AS16276:25
|
||||
SWW_COUNTRY_LIMIT_PERCENT: cn:25
|
||||
SWW_TAG_LIMIT_PERCENT: asn=14061:50,asn=16276:25,country=CN:25
|
||||
# Paths only scanners ask for; one request bans the client.
|
||||
SWW_TRAP_PATHS: /wp-login.php,/xmlrpc.php
|
||||
# Answer a refused client with 429 rather than 403.
|
||||
@@ -464,7 +470,7 @@ services:
|
||||
# Alerts to an ntfy topic, with its token read from a file.
|
||||
SWW_ALERT_NTFY_URL: https://ntfy.example/smallwebwaf
|
||||
SWW_ALERT_NTFY_TOKEN_FILE: /etc/sww/tokens/ntfy
|
||||
SWW_ALERT_EVENTS: ban,permanent_ban,waf_block,source_failure,file_error
|
||||
SWW_ALERT_EVENTS: ban,permanent_ban,core_rule_set_match,source_failure,file_error
|
||||
volumes:
|
||||
- discriminator-state:/var/lib/sww
|
||||
- /srv/ipinfo:/var/lib/ipinfo:ro
|
||||
@@ -506,31 +512,35 @@ of the request reaches the app.
|
||||
|
||||
`discriminator` takes the checks:
|
||||
|
||||
3. The static lists, `SWW_ALLOW_NETS` and `SWW_DENY_NETS`.
|
||||
4. The bans.
|
||||
5. The lookup of the client's AS number and country.
|
||||
6. The country lists.
|
||||
7. The reputation sources.
|
||||
8. The client's limit percentage, from its AS number, its country or a
|
||||
reputation hit.
|
||||
9. The rate limits.
|
||||
10. The trap paths.
|
||||
11. The rule files.
|
||||
12. The header limits, and then the Core Rule Set.
|
||||
13. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||
3. The tag lists.
|
||||
4. The static lists, `SWW_ALLOW_NETS` and `SWW_DENY_NETS`.
|
||||
5. The bans.
|
||||
6. The lookup of the client's AS number and country.
|
||||
7. The country list.
|
||||
8. The reputation sources.
|
||||
9. The tag refusals of `SWW_TAG_DENY`.
|
||||
10. The client's limit percentage, from its tags, its AS number, its country or
|
||||
a reputation hit.
|
||||
11. The rate limits.
|
||||
12. The trap paths.
|
||||
13. The rule files.
|
||||
14. The header limits, and then the Core Rule Set.
|
||||
15. The tag refusals again, so that `exploitsignature` can refuse too.
|
||||
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||
prefix.
|
||||
|
||||
`rproxy` takes the rest, with `discriminator` counting at the end:
|
||||
|
||||
14. The request is passed to the app, streaming.
|
||||
15. After the answer, `rproxy` sends `discriminator` a report with the status
|
||||
and the bytes each way. For an allowed request, `discriminator` counts the
|
||||
bytes toward the byte limits and adds the request to the client's history.
|
||||
17. The request is passed to the app, streaming.
|
||||
18. After the answer, `rproxy` sends `discriminator` a report with the status
|
||||
and the body bytes each way. For an allowed request, `discriminator` counts
|
||||
the bytes `SWW_BYTES_COUNT` names toward the byte limits, bans the client if
|
||||
they take it past one, and adds the request to the client's history.
|
||||
|
||||
A refused visitor is answered with `SWW_BAN_RESPONSE`. A `block` rule or a Core
|
||||
Rule Set match answers `403` and bans nobody, since the Core Rule Set has false
|
||||
positives; such refusals count toward `SWW_ERROR_BURST_THRESHOLD` instead. A
|
||||
request over the header limits of step 12 is answered `431`, and bans nobody
|
||||
request over the header limits of step 14 is answered `431`, and bans nobody
|
||||
either. `rproxy` itself answers `405` to a method it does not pass on, and the
|
||||
status for a size or time limit, such as `413` for a body too large.
|
||||
|
||||
@@ -549,17 +559,38 @@ status for a size or time limit, such as `413` for a body too large.
|
||||
bans. An admin adds, keeps or lifts a ban by editing `bans.json` or through
|
||||
the ban endpoints (see [`docs/admin-endpoints.md`](docs/admin-endpoints.md)).
|
||||
|
||||
### Request tags
|
||||
|
||||
Each request gets tags from what `discriminator` knows of its client, written in
|
||||
the `tags` field of its log line:
|
||||
|
||||
- `asn=<number>` and `country=<code>`, from the lookup;
|
||||
- one tag per list of `SWW_TAG_LISTS` that holds the client, such as `torexit`
|
||||
or `vpn=mullvad`;
|
||||
- `crowdsec`, `dnsbl=<zone>` and `abuseipdb`, when that reputation source lists
|
||||
the client;
|
||||
- `exploitsignature`, when a rule file or the Core Rule Set matched the request;
|
||||
- `asn=<number>` and `prefix=<netblock>`, from a routewatch instance, while
|
||||
`SWW_ROUTEWATCH_URL` is set.
|
||||
|
||||
`SWW_TAG_DENY` refuses requests by tag, and `SWW_TAG_LIMIT_PERCENT` lowers a
|
||||
client's limits by tag. See "Request tags" in [`SPEC.md`](SPEC.md).
|
||||
|
||||
### Attack detection
|
||||
|
||||
- The Core Rule Set reads the method, the URL with its query string, and the
|
||||
headers of every request, and no body unless `SWW_WAF_BODY_LIMIT` is set. In
|
||||
front of a code forge, bodies carry what people write and publish, which the
|
||||
rules take for attacks.
|
||||
- `smallwebwaf` changes the Core Rule Set in eight ways so that it lets ordinary
|
||||
- `smallwebwaf` changes the Core Rule Set in six ways so that it lets ordinary
|
||||
requests to a code forge through, such as git over HTTP and sign-in through
|
||||
Git Credential Manager. On gitea's own paths, its searches, file names and
|
||||
branch names get through too. No setting undoes the changes; "Attack
|
||||
detection" in [`SPEC.md`](SPEC.md) lists them.
|
||||
Git Credential Manager. No setting undoes the changes; "Attack detection" in
|
||||
[`SPEC.md`](SPEC.md) lists them.
|
||||
- Each supported app also has its own fixes, which let its ordinary requests
|
||||
through. gitea's let its searches, file names and branch names through.
|
||||
`SWW_APPS` names the apps each hostname serves. A hostname gets only its own
|
||||
apps' fixes, and a hostname it does not list gets none. Unset, every app's
|
||||
fixes apply to every hostname.
|
||||
- `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` switches single rules off for an app's own
|
||||
fields or paths.
|
||||
- Rule files are plain text, one rule to a line, edited by hand and read again
|
||||
@@ -606,9 +637,10 @@ is set aside as `<name>.bad`, and never stops the running process (see
|
||||
- `internal/rproxy`: the reverse proxy in front of one app.
|
||||
- `internal/discriminator`: `discriminator`'s server, which runs the checks and
|
||||
answers its own endpoints.
|
||||
- `internal/ratelimit`, `internal/bans`, `internal/rules`, `internal/waf`,
|
||||
`internal/lookup`, `internal/reputation`, `internal/anomaly` and
|
||||
`internal/alerts`: one check or source each.
|
||||
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
|
||||
`internal/coreruleset`, `internal/lookup`, `internal/reputation`,
|
||||
`internal/routewatch`, `internal/anomaly` and `internal/alerts`: one check or
|
||||
source each.
|
||||
- `internal/state`, `internal/requestlog`, `internal/remotelog` and
|
||||
`internal/metrics`: the state files, the log lines, the syslog sender and the
|
||||
metrics.
|
||||
@@ -665,6 +697,15 @@ on the host.
|
||||
`make check`.
|
||||
- `script/docker`: builds the image, whose build runs the tests and the linter
|
||||
first. `make docker`.
|
||||
- `script/publish`: runs `script/docker`, whose build happens on the Docker host
|
||||
named by `DOCKER_HOST` (`ssh://root@fsn1app1` unless set). It then pushes the
|
||||
image to `PUBLISH_IMAGE` (`git.eeqj.de/sneak/smallwebwaf` unless set). It
|
||||
refuses a working tree with uncommitted changes. `make publish`. It pushes
|
||||
four tags:
|
||||
- the UTC date as `YYYY-MM-DD`;
|
||||
- the short commit hash;
|
||||
- the branch name;
|
||||
- `latest`.
|
||||
- `script/cibuild`: what CI runs: `script/bootstrap`, `script/check`, then the
|
||||
image build.
|
||||
- `script/precommit`: run by the git pre-commit hook; runs `script/check`.
|
||||
|
||||
@@ -22,8 +22,8 @@ to them.
|
||||
key or a token stays off until the operator supplies it.
|
||||
- It keeps a site answering while the part that decides is down, slow or
|
||||
restarting (see "Without an answer from `discriminator`").
|
||||
- It limits each client's requests and bytes per minute, hour and day. It bounds
|
||||
the size and duration of every request and answer. It detects attacks.
|
||||
- It limits each client's requests and body bytes per minute, hour and day. It
|
||||
bounds the size and duration of every request and answer. It detects attacks.
|
||||
- It bans abusers: briefly at first, for seven days on a clear sign of attack,
|
||||
and permanently when they keep at it.
|
||||
- It looks up each client's AS number and country. It can refuse whole countries
|
||||
@@ -81,12 +81,14 @@ to them.
|
||||
of bans is full.
|
||||
- `internal/rules`: reads the rule files at start and again when they
|
||||
change, and tells which rules a request matches.
|
||||
- `internal/waf`: the Core Rule Set, with the eight changes listed under
|
||||
"Attack detection", as Coraza directives.
|
||||
- `internal/coreruleset`: the Core Rule Set, with the six changes and the
|
||||
apps' fixes listed under "Attack detection", as Coraza directives.
|
||||
- `internal/lookup`: looks clients up through GeoJS or in the lookup
|
||||
database.
|
||||
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
|
||||
AbuseIPDB in the background, and keeps their answers.
|
||||
- `internal/routewatch`: asks a routewatch instance about clients in the
|
||||
background, and keeps its answers.
|
||||
- `internal/anomaly`: the anomaly counters and their alerts.
|
||||
- `internal/alerts`: holds back repeats and alerts past the hourly limit,
|
||||
and sends the rest to the webhook, Slack and ntfy, each from a queue of
|
||||
@@ -119,12 +121,14 @@ to them.
|
||||
|
||||
## What happens to a request
|
||||
|
||||
`rproxy` runs steps 1, 2 and 14, and sends the report of step 15.
|
||||
`discriminator` runs steps 3 to 13, and does the counting of step 15. The first
|
||||
step that produces a final answer ends the checks. Step 15 runs for each request
|
||||
`rproxy` runs steps 1, 2 and 17, and sends the report of step 18.
|
||||
`discriminator` runs steps 3 to 16, and does the counting of step 18. The first
|
||||
step that produces a final answer ends the checks. Step 18 runs for each request
|
||||
`discriminator` answers with `allow`, `deny` or `close`. `rproxy`'s size and
|
||||
time limits apply to the whole exchange. Each step names the section that holds
|
||||
its rules.
|
||||
its rules. A step that names a heuristic in parentheses runs that heuristic. The
|
||||
request log's `heuristic` field gives that name. `refusal_limit` and
|
||||
`byte_limits` give theirs in `after_answer_heuristic` instead.
|
||||
|
||||
1. `GET /_sww/healthz` is answered at once by `rproxy`, before any check, so a
|
||||
health checker never needs an exemption and is never refused.
|
||||
@@ -141,55 +145,70 @@ its rules.
|
||||
- An IPv6 client is its group: the netblock of `SWW_IPV6_GROUP_PREFIX`
|
||||
around its address, a /64 by default, since one abuser usually controls a
|
||||
whole /64. A client is therefore one IPv4 address or one IPv6 group.
|
||||
3. The static lists. A client in `SWW_ALLOW_NETS` skips steps 4 to 12, the byte
|
||||
limits and the error burst. A client in `SWW_DENY_NETS` is refused; a client
|
||||
in both lists is let through.
|
||||
4. Bans. A client under a ban is refused (see "Bans").
|
||||
5. The lookup, unless `SWW_LOOKUP_SOURCE` is `off` (see "Lookup of AS number and
|
||||
3. The tag lists (`tag_lists`; see "Request tags"). Each list that holds the
|
||||
client adds its tag. Next, `routewatch_tags` adds the client's `asn=` and
|
||||
`prefix=` tags while `SWW_ROUTEWATCH_URL` is set. A client in
|
||||
`SWW_ALLOW_NETS` gets no tag.
|
||||
4. The static lists. A client in `SWW_ALLOW_NETS` skips steps 5 to 15, the byte
|
||||
limits and the error burst. A client in `SWW_DENY_NETS` is refused
|
||||
(`deny_nets`); a client in both lists is let through.
|
||||
5. Bans. A client under a ban is refused (`active_ban`; see "Bans").
|
||||
6. The lookup, unless `SWW_LOOKUP_SOURCE` is `off` (see "Lookup of AS number and
|
||||
country"). The request goes on at once. It waits for the first GeoJS answer,
|
||||
up to `SWW_LOOKUP_TIMEOUT`, only when one of these needs it:
|
||||
- a country list;
|
||||
- the country list;
|
||||
- a biased threshold;
|
||||
- `SWW_TAG_DENY` or `SWW_TAG_LIMIT_PERCENT` naming an `asn=` or `country=`
|
||||
tag;
|
||||
- `SWW_ADD_LOOKUP_HEADERS`.
|
||||
|
||||
A request that ends before the answer comes is logged without it.
|
||||
|
||||
6. The country lists (see "Country lists").
|
||||
7. The reputation sources (see "Reputation"). A client on a fetched blocklist
|
||||
gets `SWW_BLOCKLIST_ACTION`. A client on the CrowdSec decision list is
|
||||
banned. A client listed by a DNS blocklist, or scored at least
|
||||
`SWW_ABUSEIPDB_MIN_SCORE` by AbuseIPDB, gets `SWW_REPUTATION_ACTION`.
|
||||
8. The client's limit percentage (see "Biased thresholds").
|
||||
9. Rate limiting (see "Request rate limits, per client"). A request over a limit
|
||||
times the percentage is refused and bans the client.
|
||||
10. The trap paths (see "Attack detection"). A request for one is refused and
|
||||
bans the client.
|
||||
11. The rule files (see "Rule files"). Matching stops at the first rule that
|
||||
refuses.
|
||||
12. The header limits (see "Answering in time"), and then the Core Rule Set (see
|
||||
"Attack detection"). A Core Rule Set match is refused with `403` while
|
||||
`SWW_WAF_MODE` is `block`, and only logged and alerted while it is `detect`.
|
||||
13. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||
7. The country list (`countries`; see "Country list").
|
||||
8. The reputation sources (see "Reputation"), in this order:
|
||||
- A client on a fetched blocklist gets `SWW_BLOCKLIST_ACTION`
|
||||
(`blocklists`).
|
||||
- A client on the CrowdSec decision list is banned (`crowdsec`).
|
||||
- A client listed by a DNS blocklist gets `SWW_REPUTATION_ACTION` (`dnsbl`).
|
||||
- A client scored at least `SWW_ABUSEIPDB_MIN_SCORE` by AbuseIPDB gets
|
||||
`SWW_REPUTATION_ACTION` (`abuseipdb`).
|
||||
9. The tag refusals (`tag_deny`; see "Request tags"). A request carrying a tag
|
||||
in `SWW_TAG_DENY` is refused.
|
||||
10. The client's limit percentage (see "Biased thresholds").
|
||||
11. Rate limiting (`rate_limits`; see "Request rate limits, per client"). A
|
||||
request over a limit times the percentage is refused and bans the client.
|
||||
12. The trap paths (`trap_paths`; see "Attack detection"). A request for one is
|
||||
refused and bans the client.
|
||||
13. The rule files (`rule_files`; see "Rule files"). Matching stops at the first
|
||||
rule that refuses.
|
||||
14. The header limits (`header_limits`; see "Answering in time"), and then the
|
||||
Core Rule Set (`core_rule_set`; see "Attack detection"). A Core Rule Set
|
||||
match is refused with `403` while `SWW_WAF_MODE` is `block`, and only logged
|
||||
and alerted while it is `detect`.
|
||||
15. The tag refusals again (`tag_deny`), so that `exploitsignature` can refuse
|
||||
too.
|
||||
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||
prefix.
|
||||
14. `rproxy` passes the request to `SWRP_UPSTREAM_URL`, streaming (see "Passing
|
||||
17. `rproxy` passes the request to `SWRP_UPSTREAM_URL`, streaming (see "Passing
|
||||
a request to the app"). With `SWW_ADD_LOOKUP_HEADERS`, `X-Client-ASN` and
|
||||
`X-Client-Country` are added for the app's own logs; a header whose value is
|
||||
unknown is left out. Whatever that setting says, any such header the client
|
||||
sent is removed.
|
||||
15. After the visitor's answer has ended, `rproxy` sends the report (see "The
|
||||
18. After the visitor's answer has ended, `rproxy` sends the report (see "The
|
||||
report"). `discriminator` then counts:
|
||||
- For a request it allowed, the response bytes, the request bytes or both
|
||||
are added to the byte counters, as `SWW_BYTES_COUNT` says. For a
|
||||
WebSocket, what it carried each way counts once it closes. A client whose
|
||||
total passes a byte limit times the percentage has broken that limit and
|
||||
is banned. The response in progress is not cut off.
|
||||
- For a request it allowed, the body bytes of the response, of the request
|
||||
or of both are added to the byte counters, as `SWW_BYTES_COUNT` says. For
|
||||
a WebSocket, what it carried each way counts once it closes. A client
|
||||
whose total passes a byte limit times the percentage has broken that limit
|
||||
and is banned (`byte_limits`). The request that broke it has already been
|
||||
passed on whole; nothing is cut off.
|
||||
- For a request it allowed, the client's history and the anomaly counters
|
||||
are updated, and the alert thresholds are evaluated.
|
||||
|
||||
For a request it refused, `discriminator` adds nothing to the byte limits.
|
||||
It updates the history and the anomaly counters at its decision. It counts
|
||||
the refusals the error burst counts at the decision too (see "Attack
|
||||
detection").
|
||||
the refusals the error burst counts at the decision too (`refusal_limit`;
|
||||
see "Attack detection").
|
||||
|
||||
In `SWW_MODE=observe`, every refusal above is logged as what would have
|
||||
happened, nothing is refused, and `discriminator` makes no ban of its own. A
|
||||
@@ -245,7 +264,8 @@ to a ban, each by its own rule:
|
||||
clear sign of attack bans it permanently at once.
|
||||
- A broken limit, one of:
|
||||
- a request over a rate limit;
|
||||
- a response that takes the client's byte total past a byte limit;
|
||||
- the bytes counted for a request and its response taking the client's total
|
||||
past a byte limit;
|
||||
- more than `SWW_ERROR_BURST_THRESHOLD` refusals within a minute.
|
||||
|
||||
The bans for a broken limit:
|
||||
@@ -279,7 +299,8 @@ no offence:
|
||||
|
||||
- `SWW_DENY_NETS`;
|
||||
- a blocklist or reputation hit whose action is `deny`;
|
||||
- the country lists.
|
||||
- the country list;
|
||||
- `SWW_TAG_DENY`.
|
||||
|
||||
Every ban carries notes with what an admin needs to decide whether to lift it,
|
||||
drawn from what `discriminator` already knows. No lookup is made for the notes
|
||||
@@ -300,6 +321,13 @@ ban is dropped alike. A scanner whose ban was dropped that way is refused and
|
||||
banned again by its next probe. A ban whose cause is `admin` is never dropped
|
||||
and does not count toward `SWW_MAX_BANS`, so such bans can never fill the table.
|
||||
|
||||
A request refused under a ban that runs out is logged with the action
|
||||
`temporarily_banned`. Under a permanent ban the action is `permanently_banned`.
|
||||
A request refused for a clear sign of attack or a CrowdSec decision takes the
|
||||
action for the ban it made. A request that makes a ban permanent is logged as
|
||||
`permanently_banned`. A request over a rate limit is logged as
|
||||
`client_is_rate_limited`, whichever ban it makes.
|
||||
|
||||
## Settings
|
||||
|
||||
`docs/settings.md` lists every setting with its default and the values it takes,
|
||||
@@ -386,7 +414,7 @@ Conventions:
|
||||
|
||||
### Static lists
|
||||
|
||||
- `SWW_ALLOW_NETS` skips the checks step 3 names, for monitoring or the owner's
|
||||
- `SWW_ALLOW_NETS` skips the checks step 4 names, for monitoring or the owner's
|
||||
own networks. `SWW_RATE_LIMIT_EXEMPT_NETS` bypasses the request and byte
|
||||
limits only; the error burst, attack detection and bans still apply.
|
||||
- Many people can share one address, as in an office, on a mobile carrier or
|
||||
@@ -433,10 +461,15 @@ the same way.
|
||||
|
||||
### Byte limits, per client
|
||||
|
||||
By default there is no minute limit, so a client can download in a burst at full
|
||||
speed. The hour limit defaults to 50 GiB and the day limit to 100 GiB. They
|
||||
allow several downloads of a few GiB each. A response's bytes are counted when
|
||||
it ends (see step 15).
|
||||
The byte limits count body bytes: the app's response bodies, the visitors'
|
||||
request bodies, or both, as `SWW_BYTES_COUNT` says. Headers are never counted.
|
||||
What the visitor sent over a WebSocket counts as request bytes, and what the app
|
||||
sent as response bytes. The bytes counted for a request and its response are
|
||||
added to the client's total once the response has ended (see step 18). So the
|
||||
request that takes a client past a limit is passed on whole, and the ban refuses
|
||||
the client's requests from then on. By default there is no minute limit, so a
|
||||
client can download in a burst at full speed. The hour limit defaults to 50 GiB
|
||||
and the day limit to 100 GiB. They allow several downloads of a few GiB each.
|
||||
|
||||
### Lookup of AS number and country
|
||||
|
||||
@@ -465,12 +498,12 @@ service that keeps its visitors' addresses on its own host.
|
||||
dropped from that table. The answer reaches the client's history and ban notes
|
||||
when it comes.
|
||||
- `SWW_LOOKUP_TIMEOUT` is how long a visitor's request waits for its client's
|
||||
first answer, when a setting needs it (see step 5). It also bounds each
|
||||
first answer, when a setting needs it (see step 6). It also bounds each
|
||||
request to GeoJS. A client whose answer has not come in time counts as unknown
|
||||
until it comes, and its later requests do not wait. Its address is asked about
|
||||
again in the background.
|
||||
- A client the lookup cannot place has an unknown country and AS number, which
|
||||
the country lists and the biased thresholds treat as described there. Nothing
|
||||
the country list and the biased thresholds treat as described there. Nothing
|
||||
else treats it differently.
|
||||
- GeoJS publishes no rate limit, but its terms forbid "an excessive amount of
|
||||
API requests", judged by GeoJS alone, and let it block a caller. While GeoJS
|
||||
@@ -490,12 +523,12 @@ service that keeps its visitors' addresses on its own host.
|
||||
`file_error` alert and ignored. The database already loaded stays in use. A
|
||||
client whose address the file does not hold counts as unknown.
|
||||
|
||||
### Country lists
|
||||
### Country list
|
||||
|
||||
Both are empty by default: the defaults judge a client by what it does, not by
|
||||
where it comes from.
|
||||
`SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` is empty by default: the defaults judge a
|
||||
client by what it does, not by where it comes from. A `country=` tag in
|
||||
`SWW_TAG_DENY` refuses one country (see "Request tags").
|
||||
|
||||
- `SWW_DENIED_COUNTRIES` refuses every request from a listed country.
|
||||
- `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` lets through only clients placed in a
|
||||
listed country. Every request from any other country is refused. So is every
|
||||
request from a client the lookup cannot place, such as internal monitoring or
|
||||
@@ -504,45 +537,102 @@ where it comes from.
|
||||
client in whenever GeoJS stops answering. The cost: while GeoJS fails, this
|
||||
list refuses every new client. No other failure of GeoJS, a reputation source
|
||||
or an alert destination comes this close to taking the service down.
|
||||
- Both may be set. `SWW_DENIED_COUNTRIES` then adds nothing, since the exclusive
|
||||
list already refuses every other country. A code on both lists stops the start
|
||||
- A code in this list whose `country=` tag is in `SWW_TAG_DENY` stops the start
|
||||
with a message naming it.
|
||||
- A refused request is answered with `SWW_BAN_RESPONSE`, as soon as the client's
|
||||
address has been looked up. It is logged with the action `country_denied`, and
|
||||
counted in the metrics.
|
||||
|
||||
### Request tags
|
||||
|
||||
Each request gets tags from what `discriminator` knows of its client and what
|
||||
the checks found. The operator refuses or slows clients by tag. Tags appear in
|
||||
the log line as `tags`, sorted, with empty lists left out (see
|
||||
`docs/request-log.md`).
|
||||
|
||||
Each tag and where it comes from:
|
||||
|
||||
- `asn=<number>` and `country=<code>`, such as `asn=64496` and `country=DE`,
|
||||
from the lookup. A client the lookup cannot place has neither. A request
|
||||
refused by `deny_nets` or `active_ban` has neither, because those checks come
|
||||
before the lookup.
|
||||
- One tag per list of `SWW_TAG_LISTS` that holds the client (`tag_lists`), the
|
||||
tag being the list's name, such as `torexit` or `vpn=mullvad`. These lists are
|
||||
fetched and kept like a blocklist. `discriminator` ships none of its own.
|
||||
- `crowdsec`, `dnsbl=<zone>` and `abuseipdb`, when that reputation source lists
|
||||
the client, whatever the source's action. A zone with a key is named with the
|
||||
key masked, as the `reputation` field gives it.
|
||||
- `exploitsignature`, when a rule file or the Core Rule Set matched the request.
|
||||
For the Core Rule Set a match is a score at or over
|
||||
`SWW_WAF_ANOMALY_THRESHOLD`.
|
||||
- `asn=<number>` and `prefix=<netblock>` from a routewatch instance
|
||||
(`routewatch_tags`), while `SWW_ROUTEWATCH_URL` is set:
|
||||
- The tag `prefix=` holds the netblock the client's address is announced in,
|
||||
such as `prefix=192.0.2.0/24`.
|
||||
- An `asn=` tag has the same spelling as the lookup's. When routewatch and
|
||||
the lookup give the same AS number, a request has one `asn=` tag. When
|
||||
they differ, it has both.
|
||||
- They are added before `deny_nets`, so the log line of a request it refused
|
||||
has them.
|
||||
- `discriminator` asks routewatch in the background and never waits for it.
|
||||
An answer is kept for 1 hour, in memory only.
|
||||
- After a query fails, routewatch is not asked for 1 minute.
|
||||
- A request without these tags gives the reason in the log field
|
||||
`routewatch_tags_missing` (see
|
||||
[`docs/request-log.md`](docs/request-log.md)).
|
||||
|
||||
A client in `SWW_ALLOW_NETS` skips the checks and gets no tag. Every tag is in
|
||||
lower case, except a country code, which is in capitals. `SWW_TAG_DENY` and
|
||||
`SWW_TAG_LIMIT_PERCENT` accept tags in either case.
|
||||
|
||||
Two settings act on tags:
|
||||
|
||||
- `SWW_TAG_DENY` lists tags whose requests are refused with the action
|
||||
`client_denied` (`tag_deny`). The check runs twice: just before the rate
|
||||
limits, and again after the Core Rule Set, so that `exploitsignature` can
|
||||
refuse too.
|
||||
- `SWW_TAG_LIMIT_PERCENT` lists tags with a percentage from 1 to 99. Their
|
||||
clients get that percentage of every request and byte limit. The lowest
|
||||
percentage among a client's tags applies. `exploitsignature` cannot be listed,
|
||||
since a request gets it only after the rate limits have counted it.
|
||||
|
||||
The tag settings are empty by default, for the same reason as the country list.
|
||||
The lookup waits for GeoJS when either names an `asn=` or `country=` tag. A
|
||||
setting that names one stops the start while `SWW_LOOKUP_SOURCE` is `off`. The
|
||||
exception is an `asn=` tag while `SWW_ROUTEWATCH_URL` is set, since routewatch
|
||||
gives that tag.
|
||||
|
||||
### Biased thresholds
|
||||
|
||||
The biased thresholds lower the limits of clients in listed AS numbers and
|
||||
countries. The lists are empty by default, for the same reason as the country
|
||||
lists. The request log and the metrics show which AS numbers and countries a
|
||||
service's abuse comes from.
|
||||
The biased thresholds lower the limits of clients by AS number, country and tag.
|
||||
The lists are empty by default, for the same reason as the country list. The
|
||||
request log and the metrics show which AS numbers and countries a service's
|
||||
abuse comes from.
|
||||
|
||||
- Each list gives AS numbers or countries a percentage, such as `AS14061:50`.
|
||||
Their clients get that percentage of every request and byte limit
|
||||
(`SWW_ASN_LIMIT_PERCENT`, `SWW_COUNTRY_LIMIT_PERCENT`). A lowered limit is
|
||||
rounded down. `0` is a zero allowance: the client's first request breaks a
|
||||
limit and bans it.
|
||||
- `SWW_TAG_LIMIT_PERCENT` gives a tag a percentage, such as `asn=14061:50` (see
|
||||
"Request tags"). A lowered limit is rounded down.
|
||||
- `SWW_UNKNOWN_LIMIT_PERCENT` at `0` is a zero allowance: the client's first
|
||||
request breaks a limit and bans it.
|
||||
- `SWW_ASN_BYTES_PERCENT` and `SWW_COUNTRY_BYTES_PERCENT` set the byte limits'
|
||||
percentage alone, when it should differ from the request limits'. For an AS
|
||||
number they list, they replace its percentage from `SWW_ASN_LIMIT_PERCENT` and
|
||||
`SWW_ASN_LIMIT_PERCENT_URL`; for a country, the one from
|
||||
`SWW_COUNTRY_LIMIT_PERCENT`.
|
||||
number they list, they replace its `asn=` tag's percentage and its entry in
|
||||
`SWW_ASN_LIMIT_PERCENT_URL`; for a country, its `country=` tag's percentage.
|
||||
- `SWW_ASN_LIMIT_PERCENT_URL` names a text file of `AS:percent` lines, so one
|
||||
abuse-source list can be shared by every `discriminator` in the fleet. It is
|
||||
fetched and kept like a blocklist (see "Reputation"). Where it and
|
||||
`SWW_ASN_LIMIT_PERCENT` both give a percentage, the lower applies.
|
||||
fetched and kept like a blocklist (see "Reputation"). Each entry counts like
|
||||
an `asn=` tag in `SWW_TAG_LIMIT_PERCENT`. Where both give a percentage, the
|
||||
lower applies.
|
||||
- A client's limit percentage is the lowest of those that apply to it: its AS
|
||||
number's, its country's, or a reputation hit's `limit:<percent>`. A client
|
||||
without a country gets `SWW_UNKNOWN_LIMIT_PERCENT` in place of a country's.
|
||||
The percentage is 100 if none applies. Clients given a lower percentage are
|
||||
banned after fewer requests, first for a while and then, for repeated abuse,
|
||||
permanently. The log line of each counted request, and the notes of each ban,
|
||||
say which setting lowered the limit.
|
||||
number's, its country's, its tags', or a reputation hit's `limit:<percent>`. A
|
||||
client without a country gets `SWW_UNKNOWN_LIMIT_PERCENT` in place of a
|
||||
country's. The percentage is 100 if none applies. Clients given a lower
|
||||
percentage are banned after fewer requests, first for a while and then, for
|
||||
repeated abuse, permanently. The log line of each counted request, and the
|
||||
notes of each ban, say which setting lowered the limit.
|
||||
- No budget is shared by a whole country or AS number. One abuser could use it
|
||||
up and lock out everyone else there: a denial of service nobody chose, and the
|
||||
one this tool exists to prevent. Refusing a whole country is left to the
|
||||
operator's choice, through the country lists.
|
||||
operator's choice, through `SWW_TAG_DENY` or the country list.
|
||||
|
||||
### Attack detection
|
||||
|
||||
@@ -565,7 +655,7 @@ files, the limits and the bans still apply to the client that sends it.
|
||||
match, and scores are still logged. A request with more than 1000 query
|
||||
parameters or body fields adds 5 to its score, since Coraza reads only the
|
||||
first 1000. For a match, in `block` and `detect` mode alike, the log line
|
||||
gives the rule ids and the score, and a `waf_block` alert is sent.
|
||||
gives the rule ids and the score, and a `core_rule_set_match` alert is sent.
|
||||
- `SWW_WAF_DISABLED_RULES` switches off rule ids when an app trips a false
|
||||
positive. The default switches off six rules:
|
||||
- 920340, 920420 and 920640 refuse a request for the type of its body, when
|
||||
@@ -584,7 +674,7 @@ files, the limits and the bans still apply to the client that sends it.
|
||||
- `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` names a file of exclusions, written as
|
||||
Coraza directives. An exclusion switches one rule off for some fields or paths
|
||||
only, where `SWW_WAF_DISABLED_RULES` switches it off everywhere. The file
|
||||
holds an app's own exclusions; the eighth change below holds gitea's.
|
||||
holds an app's own exclusions; gitea's fixes below hold gitea's.
|
||||
- `discriminator` loads the file after `smallwebwaf`'s own setup rules and
|
||||
before the Core Rule Set's rules. The Core Rule Set loads its own
|
||||
`REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf` at the same point.
|
||||
@@ -630,7 +720,7 @@ files, the limits and the bans still apply to the client that sends it.
|
||||
scanner from an ordinary client. In front of a code forge, registry and git
|
||||
clients get `404` and `401` by design, hundreds of times a minute.
|
||||
|
||||
`smallwebwaf` changes the Core Rule Set 4.25.0 in eight ways, since in front of
|
||||
`smallwebwaf` changes the Core Rule Set 4.25.0 in six ways, since in front of
|
||||
gitea it would otherwise refuse ordinary requests. The changes hold in front of
|
||||
every app, and no setting undoes them:
|
||||
|
||||
@@ -649,8 +739,52 @@ every app, and no setting undoes them:
|
||||
and tea, which gitea registers for OAuth sign-in, ask to be sent back to
|
||||
`http://127.0.0.1` on the user's own machine. The server never fetches that
|
||||
address.
|
||||
4. Ten query parameters are not checked against the Core Rule Set's lists of
|
||||
system files (930120), shell paths (932160) and command names (932260):
|
||||
4. `Referer` is not checked for a Unix command given without arguments (932340),
|
||||
for Java starting a process (944110) or for a Java class name (944130).
|
||||
`Referer` names the page a request comes from, on everything that page loads
|
||||
and on every link followed from it. The Core Rule Set takes some page
|
||||
addresses for attacks:
|
||||
- a search for `env`;
|
||||
- OpenJDK's `Runtime.java`, whose path holds both `runtime` and `java.`;
|
||||
- a file named `java.lang.String.md`.
|
||||
|
||||
Every request from such a page would be refused. `Referer` keeps the other
|
||||
rules, except after some gitea searches (gitea's fixes). An app that passes
|
||||
`Referer` to a shell must check it itself.
|
||||
|
||||
5. Responses are not inspected. A raw file from a repository, such as a shell
|
||||
script, looks to the response rules like source code leaking from the server.
|
||||
6. The `from` parameter of a container registry's upload path,
|
||||
`/v2/<image>/blobs/uploads/`, is not checked for command names (932260). When
|
||||
a push reuses a layer, `from` names the image the layer comes from. An image
|
||||
such as `owner/python3-app` would have that push refused.
|
||||
|
||||
Each supported app also has its own fixes, which let its ordinary requests
|
||||
through. An app's fixes hold for the hostnames `SWW_APPS` lists the app for:
|
||||
|
||||
- `SWW_APPS` holds entries separated by semicolons. Each is a hostname, a colon,
|
||||
then one or more apps separated by commas, for example
|
||||
`git.example.com:gitea;code.example.com:gitea`.
|
||||
- The request's `Host` is matched by exact name, without its port and without
|
||||
regard to case. There are no wildcards.
|
||||
- A hostname not listed gets no app's fixes, so every rule checks it in full.
|
||||
Set empty, `SWW_APPS` lists no hostname.
|
||||
- Unset, every app's fixes hold for every hostname. An app's fixes then weaken
|
||||
the check of any other app that serves the same paths or fields.
|
||||
- Any of these mistakes stops the start:
|
||||
- an unknown app;
|
||||
- a hostname that is not a DNS name, such as `*.example.com`;
|
||||
- a hostname listed twice;
|
||||
- an app listed twice for one hostname;
|
||||
- an entry without a colon.
|
||||
|
||||
`gitea` is the only app supported. Each app's fixes are in one file, such as
|
||||
`internal/coreruleset/gitea.conf`, and one entry of `apps` in
|
||||
`internal/coreruleset/coreruleset.go`. gitea's fixes are:
|
||||
|
||||
1. Ten query parameters are not checked against the Core Rule Set's lists of
|
||||
system files (930120), shell paths (932160) and command names (932260), on
|
||||
any path:
|
||||
- `path`, `files`, `skip-to` and `sub_path`;
|
||||
- `ref`, `sha`, `branch` and `workflow`;
|
||||
- `artifactName` and `redirect_to`.
|
||||
@@ -663,44 +797,21 @@ every app, and no setting undoes them:
|
||||
catch reaches the app, such as the command `|cat /etc/passwd`. Path
|
||||
traversal, SQL and script injection, and PHP, Java and Node.js code are
|
||||
still refused there. Other parameters keep all three rules, except on the
|
||||
gitea paths the eighth change names. An app that uses one of these
|
||||
parameters as a file on the server, or passes it to a shell, must check the
|
||||
value itself. A rule file can refuse it instead.
|
||||
gitea paths the third fix names.
|
||||
|
||||
5. Two cookies are left out, and `Referer` keeps fewer rules:
|
||||
- The Core Rule Set reads the request without the `gitea_flash` and
|
||||
2. The Core Rule Set reads the request without the `gitea_flash` and
|
||||
`redirect_to` cookies. gitea puts a message naming what was just done into
|
||||
`gitea_flash`. It puts into `redirect_to` the page to return to after
|
||||
signing in. The Core Rule Set takes file names in them, such as
|
||||
`package.json`, for attacks. A browser sends such a cookie with every
|
||||
request until gitea replaces it. gitea cannot replace it while those
|
||||
requests are refused, so one refusal would keep the browser out of the
|
||||
whole site. gitea shows the message with any script removed and returns
|
||||
only to its own pages. Every other cookie is read in full. An app that
|
||||
reads a cookie of either name must check it itself.
|
||||
- `Referer` is not checked for a Unix command given without arguments
|
||||
(932340), for Java starting a process (944110) or for a Java class name
|
||||
(944130). `Referer` names the page a request comes from, on everything
|
||||
that page loads and on every link followed from it. The Core Rule Set
|
||||
takes some page addresses for attacks:
|
||||
- a search for `env`;
|
||||
- OpenJDK's `Runtime.java`, whose path holds both `runtime` and `java.`;
|
||||
- a file named `java.lang.String.md`.
|
||||
|
||||
Every request from such a page would be refused. `Referer` keeps the
|
||||
other rules, except after some gitea searches (the eighth change). An
|
||||
app that passes `Referer` to a shell must check it itself.
|
||||
|
||||
6. Responses are not inspected. A raw file from a repository, such as a shell
|
||||
script, looks to the response rules like source code leaking from the server.
|
||||
7. The `from` parameter of a container registry's upload path,
|
||||
`/v2/<image>/blobs/uploads/`, is not checked for command names (932260). When
|
||||
a push reuses a layer, `from` names the image the layer comes from. An image
|
||||
such as `owner/python3-app` would have that push refused.
|
||||
8. On gitea's paths, some fields skip the rules that refused ordinary gitea
|
||||
`gitea_flash`. It puts into `redirect_to` the page to return to after signing
|
||||
in. The Core Rule Set takes file names in them, such as `package.json`, for
|
||||
attacks. A browser sends such a cookie with every request until gitea
|
||||
replaces it. gitea cannot replace it while those requests are refused, so one
|
||||
refusal would keep the browser out of the whole site. gitea shows the message
|
||||
with any script removed and returns only to its own pages. Every other cookie
|
||||
is read in full.
|
||||
3. On gitea's paths, some fields skip the rules that refused ordinary gitea
|
||||
requests there. They are the fields where gitea sends a search, a file name
|
||||
or a branch name. `internal/waf/gitea.conf` lists each rule and the request
|
||||
it refused. The fields are:
|
||||
or a branch name. `internal/coreruleset/gitea.conf` lists each rule and the
|
||||
request it refused. The fields are:
|
||||
- `q` on gitea's searches of code, of issues and pull requests, and of
|
||||
commits. A code search holds code, such as `<script>`,
|
||||
`SELECT * FROM users WHERE` or `../`.
|
||||
@@ -715,8 +826,7 @@ every app, and no setting undoes them:
|
||||
- packages.
|
||||
|
||||
A user's or an organisation's page is any path of one segment, so on
|
||||
every such path, on any app, `q` skips these three rules, for example on
|
||||
`/search`.
|
||||
every such path `q` skips these three rules, for example on `/search`.
|
||||
|
||||
Some of these searches also skip the rule for Java class names (944130),
|
||||
which refused `org.apache.commons.logging`.
|
||||
@@ -754,11 +864,14 @@ every app, and no setting undoes them:
|
||||
`actions/upload-artifact@v3` (920450).
|
||||
|
||||
Elsewhere these fields keep every rule. The paths are those of gitea at the
|
||||
site root. An app other than gitea that serves one of them must check these
|
||||
fields itself.
|
||||
site root.
|
||||
|
||||
The parameter and cookie names in these changes are matched without regard to
|
||||
case, as Coraza matches them.
|
||||
On a hostname that gets gitea's fixes, an app other than gitea must check these
|
||||
fields and cookies itself. Listing each hostname's own apps in `SWW_APPS` keeps
|
||||
the full check for the others.
|
||||
|
||||
The parameter names in these changes and fixes are matched without regard to
|
||||
case, and the cookie names exactly.
|
||||
|
||||
### Ban lengths, response and scope
|
||||
|
||||
@@ -897,9 +1010,10 @@ left unset sends no alert. An alert needs a destination only the operator can
|
||||
supply, and what counts as unusual depends on each service's normal traffic,
|
||||
which the metrics show. They reveal a widely distributed scraper, thousands of
|
||||
addresses at low rates each, which the per-client limits do not see. A low
|
||||
`SWW_ASN_LIMIT_PERCENT` for its AS number is the response.
|
||||
percentage for its `asn=` tag in `SWW_TAG_LIMIT_PERCENT` is the response.
|
||||
|
||||
- Requests and bytes are counted per minute and per hour, for each of:
|
||||
- Requests and body bytes, as `SWW_BYTES_COUNT` counts them, are counted per
|
||||
minute and per hour, for each of:
|
||||
- a client;
|
||||
- the surrounding netblock;
|
||||
- an AS number;
|
||||
@@ -1175,7 +1289,7 @@ the metrics.
|
||||
|
||||
### The answer
|
||||
|
||||
`discriminator` runs steps 3 to 13 of "What happens to a request", in that
|
||||
`discriminator` runs steps 3 to 16 of "What happens to a request", in that
|
||||
order. Every answer but the last below carries `X-Sww-Disposition`.
|
||||
|
||||
- `X-Sww-Disposition: allow`, status `200`, no body. `rproxy` passes the request
|
||||
@@ -1209,18 +1323,18 @@ short.
|
||||
### Answering in time
|
||||
|
||||
`rproxy` stops waiting for `discriminator` after `SWRP_TIMEOUT_MS`, by default 2
|
||||
seconds, and `SWRP_DEFAULT_DISPOSITION` then decides. So at the defaults,
|
||||
`discriminator` answers a request well within that time, whatever the request
|
||||
and whatever its outside sources do.
|
||||
seconds, and `SWRP_DEFAULT_DISPOSITION` then decides. The longest a question
|
||||
waits on an outside source is `SWW_LOOKUP_TIMEOUT`, by default 800 milliseconds,
|
||||
well within that time.
|
||||
|
||||
GeoJS is the only outside source a question can wait on. A question waits only
|
||||
when step 5 says so, and for at most `SWW_LOOKUP_TIMEOUT`, by default 800
|
||||
when step 6 says so, and for at most `SWW_LOOKUP_TIMEOUT`, by default 800
|
||||
milliseconds. A client without an answer by then counts as unknown. No question
|
||||
waits on these, which run in the background:
|
||||
|
||||
- the DNS blocklists and AbuseIPDB;
|
||||
- fetching the blocklists, the CrowdSec decision list and the file
|
||||
`SWW_ASN_LIMIT_PERCENT_URL` names.
|
||||
- fetching the blocklists, the tag lists, the CrowdSec decision list and the
|
||||
file `SWW_ASN_LIMIT_PERCENT_URL` names.
|
||||
|
||||
In `enforce` mode, a question does not wait for GeoJS when it is a request:
|
||||
|
||||
@@ -1255,9 +1369,9 @@ over either one:
|
||||
`Transfer-Encoding` do not count. `X-Request-ID` and `X-Forwarded-Proto` do.
|
||||
|
||||
Real clients never reach these limits. The answer is `deny` with `431`, and the
|
||||
log line's action is `too_large`. The refusal makes no ban and is no offence.
|
||||
The rate limits count the request as usual, so a client that keeps sending such
|
||||
requests breaks a limit.
|
||||
log line's action is `request_header_too_large`. The refusal makes no ban and is
|
||||
no offence. The rate limits count the request as usual, so a client that keeps
|
||||
sending such requests breaks a limit.
|
||||
|
||||
The limits apply on every path and in every `SWW_WAF_MODE`. A client in
|
||||
`SWW_ALLOW_NETS` skips them, as it skips the Core Rule Set. In `observe` mode
|
||||
@@ -1287,11 +1401,11 @@ these fields, named as the request log's fields of the same meaning:
|
||||
- `request_bytes` and `response_bytes`: the body bytes each way, with what a
|
||||
WebSocket carried each way, each a whole number from `0` to `1125899906842624`
|
||||
(1 PiB);
|
||||
- `duration_total`: milliseconds from the request's headers being read to the
|
||||
- `duration_total_ms`: milliseconds from the request's headers being read to the
|
||||
end of the answer, from `0` to `1000000000000` (about 31 years);
|
||||
- `duration_upstream_total`: milliseconds from the hand-over to the app to the
|
||||
end of the answer, in the same range, and `0` for a request not passed to the
|
||||
app.
|
||||
- `duration_upstream_total_ms`: milliseconds from the hand-over to the app to
|
||||
the end of the answer, in the same range, and `0` for a request not passed to
|
||||
the app.
|
||||
|
||||
`discriminator` answers `400` and writes a process warning when the body:
|
||||
|
||||
@@ -1307,9 +1421,9 @@ answer.
|
||||
For a request it holds, `discriminator` writes the request's last log line (see
|
||||
"The log"). For a request it allowed, it first:
|
||||
|
||||
- counts the bytes as `SWW_BYTES_COUNT` says, toward the client's byte limits at
|
||||
the percentages it gave that request;
|
||||
- bans a client that passes a byte limit;
|
||||
- counts the body bytes `SWW_BYTES_COUNT` names, toward the client's byte limits
|
||||
at the percentages it gave that request;
|
||||
- bans a client whose total passes a byte limit;
|
||||
- adds the request with its status to the client's history, the anomaly counters
|
||||
and the metrics.
|
||||
|
||||
|
||||
+4
-4
@@ -9,9 +9,9 @@ back or dropped.
|
||||
- `ban`: a ban `discriminator` made.
|
||||
- `permanent_ban`: a permanent ban it made, or an existing ban made permanent by
|
||||
a request.
|
||||
- `waf_block`: the Core Rule Set scored a request at or over the threshold. The
|
||||
request was refused unless the alert's `detail` gives `mode`: `detect` or
|
||||
`observe`.
|
||||
- `core_rule_set_match`: the Core Rule Set scored a request at or over the
|
||||
threshold. The request was refused unless the alert's `detail` gives `mode`:
|
||||
`detect` or `observe`.
|
||||
- `anomaly`: a count over an anomaly threshold.
|
||||
- `reputation_hit`: a request from a client on a blocklist, the CrowdSec
|
||||
decision list or a DNS blocklist, or with an AbuseIPDB score that is a hit.
|
||||
@@ -130,7 +130,7 @@ The priority and tag by event:
|
||||
|
||||
- `ban`: priority `default`, tag `no_entry`.
|
||||
- `permanent_ban`: priority `high`, tag `no_entry`.
|
||||
- `waf_block`: priority `default`, tag `shield`.
|
||||
- `core_rule_set_match`: priority `default`, tag `shield`.
|
||||
- `anomaly`: priority `high`, tag `chart_with_upwards_trend`.
|
||||
- `reputation_hit`: priority `low`, tag `label`.
|
||||
- `source_failure` and `file_error`: priority `high`, tag `warning`.
|
||||
|
||||
+9
-5
@@ -5,6 +5,9 @@ What `GET /_sww/metrics` answers with, at `discriminator`'s own address.
|
||||
keeps the `instance` label as `exported_instance` unless the scrape sets
|
||||
`honor_labels: true`. `rproxy` has no metrics.
|
||||
|
||||
Every `sww_*request_bytes_total` and `sww_*response_bytes_total` series counts
|
||||
body bytes, never headers, whatever `SWW_BYTES_COUNT` says.
|
||||
|
||||
## Traffic
|
||||
|
||||
- `sww_requests_total`, `sww_request_bytes_total` and
|
||||
@@ -25,7 +28,8 @@ keeps the `instance` label as `exported_instance` unless the scrape sets
|
||||
## Attack detection
|
||||
|
||||
- `sww_rule_matches_total`, by `rule_id` and `action`, and `sww_rules_loaded`;
|
||||
- `sww_waf_matches_total`, by `mode` and `rule_id`, for the rules that fired.
|
||||
- `sww_core_rule_set_matches_total`, by `mode` and `rule_id`, for the rules that
|
||||
fired.
|
||||
|
||||
## Countries and AS numbers
|
||||
|
||||
@@ -40,8 +44,8 @@ For the `SWW_METRICS_TOP_N` busiest of each, the rest as `other`:
|
||||
## Lookups
|
||||
|
||||
- `sww_geojs_requests_total` and `sww_geojs_failures_total`;
|
||||
- `sww_geojs_unanswered_total`: the requests that needed an answer and went on
|
||||
without it;
|
||||
- `sww_requests_without_geojs_answer_total`: the requests that needed an answer
|
||||
and went on without it;
|
||||
- with the file source, `sww_lookup_database_last_read_timestamp_seconds` and
|
||||
`sww_lookup_database_read_failures_total`.
|
||||
|
||||
@@ -67,8 +71,8 @@ By `file`:
|
||||
## Remote log, alerts and the process
|
||||
|
||||
- while a syslog server is set, `sww_remote_log_lines_sent_total`,
|
||||
`sww_remote_log_lines_dropped_total` and `sww_remote_log_buffer_depth`;
|
||||
- by `destination`, `sww_alerts_sent_total`, `sww_alerts_failed_total`,
|
||||
`sww_remote_log_lines_dropped_total` and `sww_remote_log_lines_waiting`;
|
||||
- by `destination`, `sww_alerts_sent_total`, `sww_alert_send_failures_total`,
|
||||
`sww_alerts_suppressed_total` and `sww_alerts_dropped_total`;
|
||||
- Go's own `go_` metrics and the process's `process_` metrics.
|
||||
|
||||
|
||||
+77
-34
@@ -26,7 +26,7 @@ These questions write no line:
|
||||
A `report` line:
|
||||
|
||||
```
|
||||
{"type":"request","discriminator_line":"report","time":"2026-10-03T12:00:00.123Z","instance":"fsn1/discriminator","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"waf_score":0,"duration_total":3.217,"duration_checks":0.041,"duration_waf":0.029,"duration_upstream_total":3.104}
|
||||
{"type":"request","discriminator_line":"report","time":"2026-10-03T12:00:00.123Z","instance":"fsn1/discriminator","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"core_rule_set_score":0,"duration_total_ms":3.217,"duration_checks_ms":0.041,"duration_core_rule_set_ms":0.029,"duration_upstream_total_ms":3.104}
|
||||
```
|
||||
|
||||
### Every line
|
||||
@@ -37,7 +37,7 @@ Every one of these lines has:
|
||||
- the standard web log fields;
|
||||
- `request_id`, `peer_ip` and `client_group`;
|
||||
- `asn`, `as_name` and `country`;
|
||||
- `action` and `duration_total`.
|
||||
- `action` and `duration_total_ms`.
|
||||
|
||||
### Standard web log fields
|
||||
|
||||
@@ -96,49 +96,92 @@ Every one of these lines has:
|
||||
|
||||
- `action`: what was done:
|
||||
- `forward`: answered `allow`, so that `rproxy` passes it to the app;
|
||||
- `denied`: refused for `SWW_DENY_NETS`, or for a reputation source with the
|
||||
action `deny`;
|
||||
- `banned`: refused under a ban, or for a `ban` rule, a trap path or a
|
||||
CrowdSec decision, each of which bans the client;
|
||||
- `country_denied`, `rate_limited`, `rule_blocked` and `waf_blocked`:
|
||||
refused by that check;
|
||||
- `too_large`: over a header limit;
|
||||
- `admin`: answered at one of `discriminator`'s own endpoints, or `404` for
|
||||
a path under `/_sww/`.
|
||||
- `client_denied`: refused for `SWW_DENY_NETS`, for a reputation source with
|
||||
the action `deny`, or for a tag in `SWW_TAG_DENY`;
|
||||
- `temporarily_banned`: refused under a ban that runs out, or for a `ban`
|
||||
rule, a trap path or a CrowdSec decision that made such a ban;
|
||||
- `permanently_banned`: refused under a ban that never runs out, including a
|
||||
ban this request made or made permanent;
|
||||
- `country_denied`, `client_is_rate_limited`, `blocked_by_block_rule` and
|
||||
`blocked_by_core_rule_set`: refused by that check;
|
||||
- `request_header_too_large`: over a header limit;
|
||||
- `own_endpoint`: answered at one of `discriminator`'s own endpoints, or
|
||||
`404` for a path under `/_sww/`.
|
||||
- `would_action`: in `observe` mode, what `enforce` mode would have done with a
|
||||
request it would have refused. `action` then names what was done, normally
|
||||
`forward`.
|
||||
- `limit_percent` and `limit_percent_setting`: the client's percentage of the
|
||||
rate limits and the setting that lowered it, when one did. `bytes_percent` and
|
||||
`bytes_percent_setting` do the same for the byte limits.
|
||||
- `heuristic`: the heuristic that refused the request, in `observe` mode too. It
|
||||
is left out when none did.
|
||||
- `heuristic_figures`: an object of the figures that heuristic acted on. Each
|
||||
heuristic and its figures:
|
||||
- `deny_nets`: `netblock`, the entry of `SWW_DENY_NETS` the client is in;
|
||||
- `active_ban`: `netblock`, the ban's netblock, and `ban_expires`;
|
||||
- `countries`: `country`, and `setting`, the country list that refused it;
|
||||
- `tag_deny`: `tag`, the tag of `SWW_TAG_DENY` the request carries;
|
||||
- `blocklists`: `lists`, the blocklist URLs that list the client;
|
||||
- `crowdsec`: `scenario`, the decision's scenario, and `expires`, when the
|
||||
decision ends;
|
||||
- `dnsbl`: `zones`, the DNS blocklist zones that list the client, their keys
|
||||
masked;
|
||||
- `abuseipdb`: `score`, and `min_score`, `SWW_ABUSEIPDB_MIN_SCORE`;
|
||||
- `rate_limits`: `window`, `count` and `limit`;
|
||||
- `trap_paths`: `path`, the trap path asked for;
|
||||
- `rule_files`: `rule_id` and `rule_action`, of the rule that refused it;
|
||||
- `header_limits`: `fields`, `fields_limit`, `cookies` and `cookies_limit`;
|
||||
- `core_rule_set`: `score`, and `threshold`, `SWW_WAF_ANOMALY_THRESHOLD`;
|
||||
- `refusal_limit`: `window`, `count` and `limit`, of the error burst;
|
||||
- `byte_limits`: `window`, `count` and `limit`, the count in bytes.
|
||||
- `after_answer_heuristic`: the heuristic that banned the client after
|
||||
`discriminator` answered the request, in `observe` mode too: `refusal_limit`
|
||||
or `byte_limits`.
|
||||
- `after_answer_heuristic_figures`: an object of the figures that heuristic
|
||||
acted on, as listed above.
|
||||
|
||||
`byte_limits` bans when the report comes, so only the `report` line names
|
||||
it.
|
||||
|
||||
- `rate_limit_percent` and `rate_limit_percent_setting`: the client's percentage
|
||||
of the rate limits and the setting that lowered it, when one did.
|
||||
`bytes_limit_percent` and `bytes_limit_percent_setting` do the same for the
|
||||
byte limits.
|
||||
- `counts`: the client's totals as rate limiting counts them, this request
|
||||
included. It is left out when rate limiting did not count the request. It
|
||||
holds:
|
||||
- `minute`, `hour` and `day`: requests;
|
||||
- `minute_bytes`, `hour_bytes` and `day_bytes`: bytes.
|
||||
- `minute_bytes`, `hour_bytes` and `day_bytes`: the body bytes
|
||||
`SWW_BYTES_COUNT` names, in that window.
|
||||
|
||||
A count can have a fraction (see "Counting method" in
|
||||
[`SPEC.md`](../SPEC.md)). The bytes of an allowed request are counted when
|
||||
its report comes. So on the `decision` line, the bytes are without this
|
||||
request's.
|
||||
[`SPEC.md`](../SPEC.md)). The bytes counted for an allowed request and its
|
||||
response are added when its report comes. So on the `decision` line, the
|
||||
three byte totals leave them out.
|
||||
|
||||
- `limit_hit`: the limit broken:
|
||||
- `minute`, `hour` or `day` for a rate limit;
|
||||
- `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit;
|
||||
- `error_burst`.
|
||||
- `minute_refusals` for the error burst (`SWW_ERROR_BURST_THRESHOLD`).
|
||||
|
||||
A request whose bytes broke a limit was passed on whole, so its `action`
|
||||
stays `forward`. Only its `report` line gives that limit.
|
||||
When the bytes counted for a request and its response break a limit, the
|
||||
request has been passed on whole, so its `action` stays `forward`. Only its
|
||||
`report` line gives that limit.
|
||||
|
||||
- `rule_ids`: the rule file rules that matched, in order, up to the one that
|
||||
refused.
|
||||
- `waf_score` and `waf_rule_ids`: the Core Rule Set's score and the rules that
|
||||
matched, for a request it inspected.
|
||||
- `core_rule_set_score` and `core_rule_set_rule_ids`: the Core Rule Set's score
|
||||
and the rules that matched, for a request it inspected.
|
||||
- `reputation`: the sources the client is on, in this order:
|
||||
- blocklist URLs;
|
||||
- the CrowdSec decision list's URL;
|
||||
- DNS blocklist zones;
|
||||
- `abuseipdb`.
|
||||
- `tags`: the request's tags, sorted, left out when it has none. Each tag and
|
||||
where it comes from is under "Request tags" in [`SPEC.md`](../SPEC.md).
|
||||
- `routewatch_tags_missing`: why the request got no tags from routewatch,
|
||||
present only while `SWW_ROUTEWATCH_URL` is set:
|
||||
- `no_answer_yet`: routewatch has not answered about this client yet;
|
||||
- `request_failed`: a query to routewatch failed within the last minute, and
|
||||
the process log says why;
|
||||
- `no_route`: routewatch knows no live route for the address.
|
||||
- `offence`: `limit`, for a request that broke a limit.
|
||||
- `ban_expires`: when the client's ban ends, for a request that made one or was
|
||||
refused under one: a time, or `permanent`.
|
||||
@@ -147,14 +190,14 @@ Every one of these lines has:
|
||||
|
||||
All in milliseconds:
|
||||
|
||||
- `duration_total`:
|
||||
- `duration_total_ms`:
|
||||
- on the `decision` line, from `discriminator` having read the question's
|
||||
body to the line;
|
||||
- on the `report` line, from `rproxy` having read the request's headers to
|
||||
the end of the answer.
|
||||
- `duration_checks`: to the end of the checks;
|
||||
- `duration_waf`: the part of that the Core Rule Set took;
|
||||
- `duration_upstream_total`: on the `report` line of an allowed request, from
|
||||
- `duration_checks_ms`: to the end of the checks;
|
||||
- `duration_core_rule_set_ms`: the part of that the Core Rule Set took;
|
||||
- `duration_upstream_total_ms`: on the `report` line of an allowed request, from
|
||||
the hand-over to the app to the end of the answer.
|
||||
|
||||
### Requests to its own endpoints
|
||||
@@ -167,8 +210,8 @@ these meanings where they differ:
|
||||
- `scheme`: `http`.
|
||||
- `time`: when the request's headers had been read.
|
||||
- `request_bytes`: the bytes read of a ban to add, and otherwise `0`.
|
||||
- `action`: `admin`.
|
||||
- `duration_total`: from the request's headers being read to the line.
|
||||
- `action`: `own_endpoint`.
|
||||
- `duration_total_ms`: from the request's headers being read to the line.
|
||||
|
||||
## `rproxy`'s line
|
||||
|
||||
@@ -186,10 +229,10 @@ these meanings where they differ:
|
||||
a WebSocket carried from the visitor.
|
||||
- `response_bytes`: the bytes of the answer's body, with what a WebSocket
|
||||
carried from the app.
|
||||
- `duration_request`: milliseconds from the request's headers being read until
|
||||
the app had been sent the whole request, or the answer started if that was
|
||||
sooner. It includes the wait for `discriminator`.
|
||||
- `duration_response`: milliseconds from then until the end of the answer.
|
||||
- `duration_request_ms`: milliseconds from the request's headers being read
|
||||
until the app had been sent the whole request, or the answer started if that
|
||||
was sooner. It includes the wait for `discriminator`.
|
||||
- `duration_response_ms`: milliseconds from then until the end of the answer.
|
||||
- `disposition`: what was done with the request: `allow`, `deny` or `close`. It
|
||||
is `discriminator`'s answer, or the value of `SWRP_DEFAULT_DISPOSITION`. It is
|
||||
left out when:
|
||||
@@ -203,5 +246,5 @@ these meanings where they differ:
|
||||
For example:
|
||||
|
||||
```
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","client_ip":"203.0.113.9","host":"app.example","method":"GET","path":"/","query":"","status":200,"request_bytes":0,"response_bytes":5120,"duration_request":1.204,"duration_response":3.104,"disposition":"allow"}
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","client_ip":"203.0.113.9","host":"app.example","method":"GET","path":"/","query":"","status":200,"request_bytes":0,"response_bytes":5120,"duration_request_ms":1.204,"duration_response_ms":3.104,"disposition":"allow"}
|
||||
```
|
||||
+103
-26
@@ -90,14 +90,16 @@ between the two (see "The image's command" below).
|
||||
## Static lists
|
||||
|
||||
- `SWW_ALLOW_NETS` (default empty): netblocks you trust, such as your monitoring
|
||||
(see step 3 in [`SPEC.md`](../SPEC.md)).
|
||||
(see step 4 in [`SPEC.md`](../SPEC.md)).
|
||||
- `SWW_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients are
|
||||
exempt from rate limiting and the byte limits.
|
||||
- `SWW_DENY_NETS` (default empty): netblocks whose clients are refused (see step
|
||||
3 in [`SPEC.md`](../SPEC.md)).
|
||||
4 in [`SPEC.md`](../SPEC.md)).
|
||||
|
||||
## Request rate limits, per client
|
||||
|
||||
The request past one of these limits is refused, and the client is banned.
|
||||
|
||||
- `SWW_RATE_LIMIT_PER_MINUTE` (default `120`, 2 a second): the most requests a
|
||||
client may make in a minute. A count or `off`. The default leaves room for a
|
||||
person opening two pages at once with an empty cache, up to 60 files each.
|
||||
@@ -120,19 +122,32 @@ between the two (see "The image's command" below).
|
||||
|
||||
## Byte limits, per client
|
||||
|
||||
These count body bytes, never headers. What the visitor sent over a WebSocket
|
||||
counts as request bytes, and what the app sent as response bytes. The bytes
|
||||
counted for a request and its response are added to the client's total once the
|
||||
response has ended. The request that takes a client's total past a limit is not
|
||||
cut off; the client is banned.
|
||||
|
||||
- `SWW_BYTES_LIMIT_PER_MINUTE` (default `off`), `SWW_BYTES_LIMIT_PER_HOUR`
|
||||
(default `50G`) and `SWW_BYTES_LIMIT_PER_DAY` (default `100G`): the most bytes
|
||||
counted for a client in each window. A size or `off`.
|
||||
- `SWW_BYTES_COUNT` (default `both`): which body bytes the byte limits count:
|
||||
`response`, `request` or `both`.
|
||||
counted for a client in each window, a size or `off`. `SWW_BYTES_COUNT` says
|
||||
which bytes count: response bodies, request bodies or both. The request that
|
||||
passes the limit is not cut off, and the client is banned when its response
|
||||
ends.
|
||||
- `SWW_BYTES_COUNT` (default `both`): what the byte limits and the byte anomaly
|
||||
thresholds count: `response`, the app's response bodies; `request`, the
|
||||
visitors' request bodies; or `both`. What the visitor sent over a WebSocket
|
||||
counts as request bytes, and what the app sent as response bytes.
|
||||
|
||||
## Lookup of AS number and country
|
||||
|
||||
- `SWW_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `off` stops
|
||||
the start while a setting that needs a lookup is set:
|
||||
- the two country lists;
|
||||
- `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES`;
|
||||
- `SWW_ADD_LOOKUP_HEADERS`;
|
||||
- the four biased threshold lists and `SWW_ASN_LIMIT_PERCENT_URL`;
|
||||
- `SWW_TAG_DENY` or `SWW_TAG_LIMIT_PERCENT` naming a `country=` tag, or an
|
||||
`asn=` tag while `SWW_ROUTEWATCH_URL` is unset;
|
||||
- the two byte percentage lists and `SWW_ASN_LIMIT_PERCENT_URL`;
|
||||
- `SWW_UNKNOWN_LIMIT_PERCENT` below `100`;
|
||||
- an anomaly threshold per AS number.
|
||||
- `SWW_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its
|
||||
@@ -144,28 +159,70 @@ between the two (see "The image's command" below).
|
||||
the client's AS number in `X-Client-ASN` and its country in
|
||||
`X-Client-Country`.
|
||||
|
||||
## Country lists
|
||||
## Country list
|
||||
|
||||
- `SWW_DENIED_COUNTRIES` (default empty): countries whose clients are refused,
|
||||
such as `cn,ru,kp`.
|
||||
- `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` (default empty): when set, the only
|
||||
countries whose clients get through, such as `us,de`. A country on both lists
|
||||
stops the start.
|
||||
countries whose clients get through, such as `us,de`. A country whose
|
||||
`country=` tag is in `SWW_TAG_DENY` stops the start.
|
||||
|
||||
## Request tags
|
||||
|
||||
Every request gets tags from what `discriminator` knows of its client. See
|
||||
"Request tags" in [`SPEC.md`](../SPEC.md).
|
||||
|
||||
- `SWW_TAG_LISTS` (default empty): address lists that each add a tag, as
|
||||
`name=URL` pairs, such as `torexit=https://example.com/tor.txt`. A name is the
|
||||
tag, and may hold `=`, such as `vpn=mullvad`. Each list is an `http` or
|
||||
`https` URL fetched like a blocklist. A name or URL listed twice is refused.
|
||||
So is a URL that is also in `SWW_BLOCKLIST_URLS`, `SWW_ASN_LIMIT_PERCENT_URL`,
|
||||
or the CrowdSec decision list.
|
||||
- `SWW_TAG_DENY` (default empty): tags whose requests are refused, such as
|
||||
`torexit,country=CN`. A tag listed twice, an `asn=` tag without a number, or a
|
||||
`country=` tag without a country, is refused.
|
||||
- `SWW_TAG_LIMIT_PERCENT` (default empty): tags that lower a client's limits, as
|
||||
tag and percentage pairs, such as `vpn:50,asn=14061:25`. A percentage is from
|
||||
`1` to `99`, of every rate limit and byte limit. The lowest percentage among a
|
||||
client's tags applies. A tag listed twice is refused. So is
|
||||
`exploitsignature`: a request gets that tag after the rate limits have counted
|
||||
it.
|
||||
|
||||
- `SWW_ROUTEWATCH_URL` (default unset): the `http` or `https` URL of a
|
||||
routewatch instance. [routewatch](https://git.eeqj.de/sneak/routewatch) is a
|
||||
service that gives an address's announced netblock and origin AS.
|
||||
`discriminator` asks it about each client, in the background, and adds the
|
||||
tags `asn=<number>` and `prefix=<netblock>` from its answer. While it is
|
||||
unset, nothing asks routewatch and no request gets a `prefix=` tag.
|
||||
|
||||
`SWW_TAG_DENY` and `SWW_TAG_LIMIT_PERCENT` accept tags in either case. A tag
|
||||
that no request can get stops the start. A request can get only these:
|
||||
|
||||
- a name in `SWW_TAG_LISTS`;
|
||||
- an `asn=` or `country=` tag;
|
||||
- a `prefix=` tag with a netblock, while `SWW_ROUTEWATCH_URL` is set. It matches
|
||||
only that exact announced netblock, such as `prefix=192.0.2.0/24`;
|
||||
- a `dnsbl=` tag of a zone in `SWW_DNSBL_ZONES`, in lower case, with a key
|
||||
masked as the `reputation` field shows it;
|
||||
- `abuseipdb`, while `SWW_ABUSEIPDB_KEY` is set;
|
||||
- `exploitsignature`, in `SWW_TAG_DENY` only, while `SWW_RULES_ENABLED` is
|
||||
`true`, or while neither `SWW_WAF_MODE` nor `SWW_WAF_ANOMALY_THRESHOLD` is
|
||||
`off`.
|
||||
|
||||
`crowdsec` stops the start in both settings, because a CrowdSec decision refuses
|
||||
the request first.
|
||||
|
||||
## Biased thresholds
|
||||
|
||||
- `SWW_ASN_LIMIT_PERCENT` (default empty): a list of AS numbers, each with `:`
|
||||
and a percentage, such as `AS14061:50,AS16276:50,AS45102:25`. An AS number
|
||||
listed twice is refused.
|
||||
- `SWW_COUNTRY_LIMIT_PERCENT` (default empty): the same by country, such as
|
||||
`cn:25,ru:50`.
|
||||
- `SWW_ASN_BYTES_PERCENT` and `SWW_COUNTRY_BYTES_PERCENT` (default empty): the
|
||||
same, for the byte limits alone.
|
||||
- `SWW_ASN_BYTES_PERCENT` and `SWW_COUNTRY_BYTES_PERCENT` (default empty): a
|
||||
percentage of the byte limits alone, for the AS numbers or countries listed,
|
||||
such as `AS14061:50` or `cn:25`. It takes the place of the percentage
|
||||
`SWW_TAG_LIMIT_PERCENT` gives their `asn=` or `country=` tag. An AS number or
|
||||
country listed twice is refused.
|
||||
- `SWW_UNKNOWN_LIMIT_PERCENT` (default `100`): the percentage for a client the
|
||||
lookup cannot place.
|
||||
- `SWW_ASN_LIMIT_PERCENT_URL` (default unset): the `http` or `https` URL of a
|
||||
text file with one AS number and percentage on each line, such as
|
||||
`AS64496:50`. A URL that is also in `SWW_BLOCKLIST_URLS` is refused.
|
||||
`AS64496:50`. Each entry counts like an `asn=` tag in `SWW_TAG_LIMIT_PERCENT`.
|
||||
A URL that is also in `SWW_BLOCKLIST_URLS` is refused.
|
||||
|
||||
## Attack detection
|
||||
|
||||
@@ -198,8 +255,27 @@ between the two (see "The image's command" below).
|
||||
a range. So does a directive that removes or changes rules as the file loads,
|
||||
such as `SecRuleRemoveById`, since none of the Core Rule Set's rules is loaded
|
||||
yet. It has no `_FILE` form.
|
||||
- `SWW_ERROR_BURST_THRESHOLD` (default `30`): the most refusals of one client in
|
||||
a minute (see "Attack detection" in `SPEC.md`). A count or `off`.
|
||||
- `SWW_APPS` (default unset): the apps each hostname serves. A request gets the
|
||||
fixes of its hostname's apps only (see "Attack detection" in
|
||||
[`SPEC.md`](../SPEC.md)).
|
||||
- Entries are separated by semicolons. Each is a hostname, a colon, then one
|
||||
or more apps separated by commas, for example
|
||||
`git.example.com:gitea;code.example.com:gitea`.
|
||||
- `gitea` is the only app supported.
|
||||
- The request's `Host` is matched by exact name, without its port and
|
||||
without regard to case. There are no wildcards.
|
||||
- A hostname not listed gets no app's fixes, so every rule checks it in
|
||||
full. Set empty, it lists no hostname.
|
||||
- Unset, every app's fixes apply to every hostname.
|
||||
- Any of these mistakes stops the start:
|
||||
- an unknown app;
|
||||
- a hostname that is not a DNS name, such as `*.example.com`;
|
||||
- a hostname listed twice;
|
||||
- an app listed twice for one hostname;
|
||||
- an entry without a colon.
|
||||
- `SWW_ERROR_BURST_THRESHOLD` (default `30`): the most refusals a client may get
|
||||
in a minute from a rule file, a trap path or the Core Rule Set, a count or
|
||||
`off`. The next one bans the client (see "Attack detection" in `SPEC.md`).
|
||||
|
||||
## Ban lengths, response and scope
|
||||
|
||||
@@ -266,7 +342,7 @@ between the two (see "The image's command" below).
|
||||
- `SWW_ALERT_NTFY_TOKEN` (default unset): an ntfy access token. A control
|
||||
character in it is refused.
|
||||
- `SWW_ALERT_EVENTS` (default
|
||||
`ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`):
|
||||
`ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit,source_failure,file_error`):
|
||||
the event types alerts are sent for, from those in [`alerts.md`](alerts.md)
|
||||
but `summary`. Any other name is refused.
|
||||
- `SWW_ALERT_COOLDOWN` (default `15m`): how long after an alert is sent its
|
||||
@@ -276,9 +352,10 @@ between the two (see "The image's command" below).
|
||||
|
||||
## Anomaly thresholds
|
||||
|
||||
Each scope has four thresholds: the most requests and the most bytes counted in
|
||||
a minute and in an hour. Requests are counts, bytes are sizes. Each is `off` by
|
||||
default. The four thresholds per client are:
|
||||
Each scope has four thresholds: the most requests and the most body bytes
|
||||
counted in a minute and in an hour. The bytes are those `SWW_BYTES_COUNT` names.
|
||||
Requests are counts, bytes are sizes. Each is `off` by default. The four
|
||||
thresholds per client are:
|
||||
|
||||
- `SWW_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`;
|
||||
- `SWW_ANOMALY_CLIENT_REQUESTS_PER_HOUR`;
|
||||
|
||||
+4
-3
@@ -61,10 +61,11 @@ since it was first seen:
|
||||
- the first and last time seen;
|
||||
- the AS number, AS name and country last looked up, and when;
|
||||
- total requests, and how many were forwarded and how many refused;
|
||||
- bytes in each direction;
|
||||
- body bytes of its requests, and of the responses it was sent;
|
||||
- responses by status class;
|
||||
- offences by kind: `limit`, `attack`, `rule_blocked` and `waf_blocked`. A file
|
||||
may also hold `token_refused`, which is kept, and which nothing adds to.
|
||||
- offences by kind: `limit`, `attack`, `blocked_by_block_rule` and
|
||||
`blocked_by_core_rule_set`. A file may also hold `token_refused`, which is
|
||||
kept, and which nothing adds to.
|
||||
|
||||
## `lookups.json`
|
||||
|
||||
|
||||
+12
-10
@@ -44,10 +44,10 @@ const (
|
||||
// EventAnomaly is a count of requests or bytes over an anomaly
|
||||
// threshold.
|
||||
EventAnomaly = "anomaly"
|
||||
// EventWAFBlock is a request the Core Rule Set scored at or over
|
||||
// EventCoreRuleSetMatch is a request the Core Rule Set scored at or over
|
||||
// SWW_WAF_ANOMALY_THRESHOLD, refused in block mode, let through in
|
||||
// detect mode.
|
||||
EventWAFBlock = "waf_block"
|
||||
EventCoreRuleSetMatch = "core_rule_set_match"
|
||||
// EventReputationHit is a request whose client a blocklist, the
|
||||
// CrowdSec decision list or a DNSBL zone lists, or whose AbuseIPDB score
|
||||
// is a hit.
|
||||
@@ -72,7 +72,7 @@ const (
|
||||
// default.
|
||||
func Events() []string {
|
||||
return []string{
|
||||
EventBan, EventPermanentBan, EventWAFBlock, EventAnomaly,
|
||||
EventBan, EventPermanentBan, EventCoreRuleSetMatch, EventAnomaly,
|
||||
EventReputationHit, EventSourceFailure, EventFileError,
|
||||
}
|
||||
}
|
||||
@@ -220,7 +220,7 @@ type State struct {
|
||||
// requests to it failed, and how many alerts were dropped from its full
|
||||
// queue or given up as it refused them.
|
||||
type Counts struct {
|
||||
Sent, Failed, Dropped int64
|
||||
Sent, SendFailures, Dropped int64
|
||||
}
|
||||
|
||||
// Queue takes the alerts raised, holds back those it must, and sends the
|
||||
@@ -264,7 +264,7 @@ type destination struct {
|
||||
// waiting are the alerts waiting to be sent, oldest first.
|
||||
waiting []*Alert
|
||||
|
||||
sent, failed, dropped atomic.Int64
|
||||
sent, sendFailures, dropped atomic.Int64
|
||||
}
|
||||
|
||||
// cooldownKey is what makes an alert a repeat of another: the same event
|
||||
@@ -426,7 +426,9 @@ func (q *Queue) Counts(name string) Counts {
|
||||
for _, d := range q.destinations {
|
||||
if d.name == name {
|
||||
return Counts{
|
||||
Sent: d.sent.Load(), Failed: d.failed.Load(), Dropped: d.dropped.Load(),
|
||||
Sent: d.sent.Load(),
|
||||
SendFailures: d.sendFailures.Load(),
|
||||
Dropped: d.dropped.Load(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -675,7 +677,7 @@ func (d *destination) run(ctx context.Context) {
|
||||
retryAt = time.Time{}
|
||||
case errors.Is(err, errRefused):
|
||||
d.remove(alert)
|
||||
d.failed.Add(1)
|
||||
d.sendFailures.Add(1)
|
||||
d.dropped.Add(1)
|
||||
|
||||
retryDelay = 0
|
||||
@@ -684,7 +686,7 @@ func (d *destination) run(ctx context.Context) {
|
||||
d.processLog.Warn("gave up an alert "+d.setting+" refused",
|
||||
"event", alert.Event, "error", err.Error())
|
||||
case ctx.Err() == nil: // not cut off as sww stops
|
||||
d.failed.Add(1)
|
||||
d.sendFailures.Add(1)
|
||||
|
||||
retryDelay = min(max(retryDelayFactor*retryDelay, firstRetryDelay),
|
||||
maxRetryDelay)
|
||||
@@ -871,7 +873,7 @@ func ntfyPriority(event string) string {
|
||||
return "high"
|
||||
case EventReputationHit:
|
||||
return "low"
|
||||
default: // ban, waf_block and summary
|
||||
default: // ban, core_rule_set_match and summary
|
||||
return "default"
|
||||
}
|
||||
}
|
||||
@@ -882,7 +884,7 @@ func ntfyTag(event string) string {
|
||||
switch event {
|
||||
case EventBan, EventPermanentBan:
|
||||
return "no_entry"
|
||||
case EventWAFBlock:
|
||||
case EventCoreRuleSetMatch:
|
||||
return "shield"
|
||||
case EventAnomaly:
|
||||
return "chart_with_upwards_trend"
|
||||
|
||||
@@ -897,7 +897,7 @@ func TestDestinationThatDoesNotAnswerHoldsUpNeitherOther(t *testing.T) {
|
||||
synctest.Wait()
|
||||
|
||||
wantDestinationCounts(t, q, alerts.DestinationSlack,
|
||||
alerts.Counts{Sent: 3, Failed: 1})
|
||||
alerts.Counts{Sent: 3, SendFailures: 1})
|
||||
|
||||
logged := log.String()
|
||||
if !strings.Contains(logged,
|
||||
@@ -934,7 +934,7 @@ func TestDropsAreCountedForEachDestination(t *testing.T) {
|
||||
for destination, want := range map[string]alerts.Counts{
|
||||
alerts.DestinationWebhook: {Sent: all},
|
||||
alerts.DestinationSlack: {Dropped: 1},
|
||||
alerts.DestinationNtfy: {Failed: all, Dropped: all},
|
||||
alerts.DestinationNtfy: {SendFailures: all, Dropped: all},
|
||||
} {
|
||||
wantDestinationCounts(t, q, destination, want)
|
||||
}
|
||||
@@ -1223,12 +1223,12 @@ func wantEvents(t *testing.T, webhook *standIn, want ...string) {
|
||||
// requests to it that it counts as failed, the alerts it counts as held
|
||||
// back, and those it counts as dropped for the webhook.
|
||||
func wantCounts(
|
||||
t *testing.T, q *alerts.Queue, sent, failed, suppressed, dropped int64,
|
||||
t *testing.T, q *alerts.Queue, sent, sendFailures, suppressed, dropped int64,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
wantDestinationCounts(t, q, alerts.DestinationWebhook,
|
||||
alerts.Counts{Sent: sent, Failed: failed, Dropped: dropped})
|
||||
alerts.Counts{Sent: sent, SendFailures: sendFailures, Dropped: dropped})
|
||||
|
||||
if q.Suppressed() != suppressed {
|
||||
t.Errorf("%d alerts held back, want %d", q.Suppressed(), suppressed)
|
||||
@@ -1281,12 +1281,12 @@ func anAlertForEachEvent() []eventMessage {
|
||||
},
|
||||
{
|
||||
alerts.Alert{
|
||||
Event: alerts.EventWAFBlock, Client: netip.MustParseAddr("192.0.2.1"),
|
||||
Event: alerts.EventCoreRuleSetMatch, Client: netip.MustParseAddr("192.0.2.1"),
|
||||
Netblock: netip.MustParsePrefix("192.0.2.1/32"),
|
||||
Reason: "refused by the Core Rule Set",
|
||||
Host: "app.example",
|
||||
},
|
||||
"fsn1app1/blog: waf_block", "default shield",
|
||||
"fsn1app1/blog: core_rule_set_match", "default shield",
|
||||
"refused by the Core Rule Set\nhost: app.example\nclient: 192.0.2.1\n" +
|
||||
"netblock: 192.0.2.1/32",
|
||||
},
|
||||
|
||||
@@ -37,13 +37,13 @@ const (
|
||||
ScopeASN = "asn"
|
||||
// ScopeTotal is the whole service.
|
||||
ScopeTotal = "total"
|
||||
// ScopeWatch is a named netblock of SWW_WATCH_NETS.
|
||||
ScopeWatch = "watch"
|
||||
// ScopeWatchedNet is a named netblock of SWW_WATCH_NETS.
|
||||
ScopeWatchedNet = "watched_net"
|
||||
)
|
||||
|
||||
// Scopes returns every scope.
|
||||
func Scopes() []string {
|
||||
return []string{ScopeClient, ScopeNet, ScopeASN, ScopeTotal, ScopeWatch}
|
||||
return []string{ScopeClient, ScopeNet, ScopeASN, ScopeTotal, ScopeWatchedNet}
|
||||
}
|
||||
|
||||
// The windows a counter counts in, as the alerts name them.
|
||||
@@ -257,7 +257,7 @@ func (c *Counters) scopesOf(r Request) []scope {
|
||||
for _, named := range p.NamedNetblocks {
|
||||
if named.Netblock.Contains(client) {
|
||||
all = append(all, scope{
|
||||
key{scope: ScopeWatch, netblock: named.Netblock, name: named.Name}, p.Watch,
|
||||
key{scope: ScopeWatchedNet, netblock: named.Netblock, name: named.Name}, p.Watch,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -391,7 +391,7 @@ func alertFor(r Request, k key, o overThreshold) alerts.Alert {
|
||||
detail["asn"] = k.asn
|
||||
case ScopeTotal:
|
||||
counted = "the whole service"
|
||||
default: // watch
|
||||
default: // watched_net
|
||||
counted = "the named netblock " + k.name + ", " + k.netblock.String()
|
||||
detail["name"] = k.name
|
||||
}
|
||||
|
||||
+59
-14
@@ -13,7 +13,7 @@ func TestBanWithoutACauseIsAnAdmins(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
netblock := netip.MustParsePrefix("203.0.113.0/24")
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{{Netblock: netblock, Start: midnight()}})
|
||||
|
||||
if got := ledger.Bans(netblock)[0].Cause; got != bans.CauseAdmin {
|
||||
@@ -24,9 +24,9 @@ func TestBanWithoutACauseIsAnAdmins(t *testing.T) {
|
||||
func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
rules := defaultRules()
|
||||
rules.MaxBans = 1
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBans = 1
|
||||
ledger := bans.New(params)
|
||||
adminsOnly := netip.MustParsePrefix("198.51.100.0/24")
|
||||
both := netip.MustParsePrefix("203.0.113.1/32")
|
||||
second := netip.MustParsePrefix("203.0.113.2/32")
|
||||
@@ -64,9 +64,9 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
|
||||
func TestEarlierBansCountABanDroppedAfterAnAdminsBan(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
rules := defaultRules()
|
||||
rules.MaxBans = 2
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBans = 2
|
||||
ledger := bans.New(params)
|
||||
netblock := netip.MustParsePrefix("203.0.113.1/32")
|
||||
other := netip.MustParsePrefix("203.0.113.2/32")
|
||||
|
||||
@@ -93,7 +93,7 @@ func TestEarlierBansCountABanDroppedAfterAnAdminsBan(t *testing.T) {
|
||||
func TestEarlierBansCountABanAnAdminKeepsOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.1/32")
|
||||
|
||||
// The netblock breaks a limit twice, the second time after its first
|
||||
@@ -117,7 +117,7 @@ func TestEarlierBansCountABanAnAdminKeepsOnce(t *testing.T) {
|
||||
func TestReasonOfTheBansSwwMakes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
|
||||
limit, _ := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.1/32"), midnight(),
|
||||
bans.Notes{Kind: "requests", Limit: 1000, Window: "minute"})
|
||||
@@ -150,7 +150,7 @@ func TestLiftedBanForALimitRefusesNothingAndMakesNoBanLonger(t *testing.T) {
|
||||
Lifted: midnight().Add(10 * time.Minute),
|
||||
}
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{lifted})
|
||||
|
||||
// While it would still last, it refuses nothing, and a limit broken
|
||||
@@ -181,7 +181,7 @@ func TestLiftedBanForAnAttackRefusesNothingAndMakesNoBanLonger(t *testing.T) {
|
||||
|
||||
// A permanent ban for a clear sign of attack, lifted.
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{{
|
||||
Netblock: netblock,
|
||||
Start: midnight(),
|
||||
@@ -211,7 +211,7 @@ func TestLiftedBanForAnAttackRefusesNothingAndMakesNoBanLonger(t *testing.T) {
|
||||
func TestLoadEditCountsTheBansAnAdminMade(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
made, _ := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.1/32"), midnight(),
|
||||
bans.Notes{})
|
||||
atStart := bans.Ban{
|
||||
@@ -246,7 +246,7 @@ func TestAdminsBanIsMadeWhileAnotherLasts(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
netblock := netip.MustParsePrefix("203.0.113.0/24")
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
|
||||
// An hour's ban for a broken limit.
|
||||
ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
||||
@@ -292,7 +292,7 @@ func TestLiftLiftsEveryActiveBanCoveringTheClient(t *testing.T) {
|
||||
wide := netip.MustParsePrefix("203.0.113.0/24")
|
||||
other := netip.MustParsePrefix("203.0.113.10/32")
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{
|
||||
// Ended an hour ago.
|
||||
{
|
||||
@@ -341,3 +341,48 @@ func TestLiftLiftsEveryActiveBanCoveringTheClient(t *testing.T) {
|
||||
|
||||
wantChanged(t, ledger, false)
|
||||
}
|
||||
|
||||
func TestPermanentBanOnAWiderNetblockIsReturnedOverATemporaryBan(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
client := netip.MustParseAddr("203.0.113.9")
|
||||
ledger := bans.New(defaultParams())
|
||||
|
||||
// The temporary ban is made first, so that its netblock is looked at first.
|
||||
ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"), midnight(), bans.Notes{})
|
||||
ledger.BanForAdmin(netip.MustParsePrefix("203.0.113.0/24"), midnight(), time.Time{},
|
||||
"for good")
|
||||
|
||||
wantPermanentBan(t, ledger, client, midnight().Add(time.Minute))
|
||||
}
|
||||
|
||||
func TestOlderPermanentBanIsReturnedOverANewerTemporaryBan(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
netblock := netip.MustParsePrefix("203.0.113.0/24")
|
||||
ledger := bans.New(defaultParams())
|
||||
|
||||
ledger.BanForAdmin(netblock, midnight(), time.Time{}, "for good")
|
||||
ledger.BanForAdmin(netblock, midnight().Add(time.Minute), midnight().Add(time.Hour),
|
||||
"for an hour")
|
||||
|
||||
wantPermanentBan(t, ledger, netblock.Addr(), midnight().Add(2*time.Minute))
|
||||
}
|
||||
|
||||
// wantPermanentBan checks that Find and Check return a permanent ban for
|
||||
// client at now.
|
||||
func wantPermanentBan(
|
||||
t *testing.T, ledger *bans.Ledger, client netip.Addr, now time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
for name, look := range map[string]func(netip.Addr, time.Time) (bans.Ban, bool, bool){
|
||||
"Find": ledger.Find,
|
||||
"Check": ledger.Check,
|
||||
} {
|
||||
ban, banned, _ := look(client, now)
|
||||
if !banned || !ban.Permanent() {
|
||||
t.Errorf("%s returned %+v and banned %v, want a permanent ban", name, ban, banned)
|
||||
}
|
||||
}
|
||||
}
|
||||
+41
-19
@@ -40,8 +40,8 @@ const repeatFactor = 3
|
||||
// maxTextBytes is how much of each text in a ban's notes is kept.
|
||||
const maxTextBytes = 256
|
||||
|
||||
// Rules are how long a ban lasts, and how many bans are held.
|
||||
type Rules struct {
|
||||
// Params are how long a ban lasts, and how many bans are held.
|
||||
type Params struct {
|
||||
// LimitBanDuration is how long a first ban for a broken limit lasts.
|
||||
LimitBanDuration time.Duration
|
||||
// LimitBanRepeatWindow is how soon after the end of the netblock's
|
||||
@@ -180,7 +180,7 @@ type Request struct {
|
||||
|
||||
// Ledger holds the bans. It is safe for concurrent use.
|
||||
type Ledger struct {
|
||||
rules Rules
|
||||
params Params
|
||||
// changed receives a value when a ban is made, unless one is waiting
|
||||
// already.
|
||||
changed chan struct{}
|
||||
@@ -190,7 +190,7 @@ type Ledger struct {
|
||||
// Find make each netblock they find the most recently seen.
|
||||
netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
|
||||
// held is how many bans netblocks holds whose cause is not CauseAdmin,
|
||||
// at most rules.MaxBans.
|
||||
// at most params.MaxBans.
|
||||
held int
|
||||
// made is how many bans have been made since the start, by cause: by
|
||||
// the ledger, and by an admin, through BanForAdmin or in an edit of
|
||||
@@ -205,7 +205,7 @@ type Ledger struct {
|
||||
}
|
||||
|
||||
// New returns a Ledger with no ban yet.
|
||||
func New(rules Rules) *Ledger {
|
||||
func New(params Params) *Ledger {
|
||||
// The ledger drops bans itself, and never those whose cause is
|
||||
// CauseAdmin, however many there are, so the LRU has no limit of its
|
||||
// own: it keeps the netblocks in the order they were last seen.
|
||||
@@ -215,7 +215,7 @@ func New(rules Rules) *Ledger {
|
||||
}
|
||||
|
||||
return &Ledger{
|
||||
rules: rules,
|
||||
params: params,
|
||||
changed: make(chan struct{}, 1),
|
||||
netblocks: netblocks,
|
||||
made: map[string]int{},
|
||||
@@ -274,16 +274,27 @@ func (l *Ledger) Find(client netip.Addr, now time.Time) (Ban, bool, bool) {
|
||||
|
||||
// activeBan returns the ban in bans, a netblock's bans oldest first, that
|
||||
// is active at now, or nil when none is. If several are, it returns the
|
||||
// one that started last. Every ban is looked at, since a ban an admin adds
|
||||
// to bans.json can start before the netblock's others and outlast them.
|
||||
// newest permanent one, or the newest when none is permanent. Every ban is
|
||||
// looked at, since a ban an admin adds to bans.json can start before the
|
||||
// netblock's others and outlast them.
|
||||
func activeBan(bans []Ban, now time.Time) *Ban {
|
||||
var newest *Ban
|
||||
|
||||
for i := len(bans) - 1; i >= 0; i-- {
|
||||
if bans[i].ActiveAt(now) {
|
||||
if !bans[i].ActiveAt(now) {
|
||||
continue
|
||||
}
|
||||
|
||||
if bans[i].Permanent() {
|
||||
return &bans[i]
|
||||
}
|
||||
|
||||
if newest == nil {
|
||||
newest = &bans[i]
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
return newest
|
||||
}
|
||||
|
||||
// BanForLimit bans netblock at now for a broken limit, with notes, and
|
||||
@@ -743,13 +754,16 @@ func (l *Ledger) markChanged() {
|
||||
}
|
||||
|
||||
// active returns the ban active at now on a netblock client is in, or
|
||||
// nil.
|
||||
// nil. If several netblocks have one, it returns a permanent ban when
|
||||
// there is one.
|
||||
func (l *Ledger) active(client netip.Addr, now time.Time) *Ban {
|
||||
lengths := l.v6Lengths
|
||||
if client.Is4() {
|
||||
lengths = l.v4Lengths
|
||||
}
|
||||
|
||||
var first *Ban
|
||||
|
||||
for _, length := range lengths {
|
||||
bans, found := l.netblocks.Get(netip.PrefixFrom(client, length).Masked())
|
||||
if !found {
|
||||
@@ -757,12 +771,20 @@ func (l *Ledger) active(client netip.Addr, now time.Time) *Ban {
|
||||
}
|
||||
|
||||
ban := activeBan(*bans, now)
|
||||
if ban != nil {
|
||||
if ban == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
if ban.Permanent() {
|
||||
return ban
|
||||
}
|
||||
|
||||
if first == nil {
|
||||
first = ban
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
return first
|
||||
}
|
||||
|
||||
// covering returns the bans of each netblock held that client is in,
|
||||
@@ -790,7 +812,7 @@ func (l *Ledger) covering(client netip.Addr) []*[]Ban {
|
||||
// unless ban's cause is CauseAdmin, which does not count toward MaxBans.
|
||||
func (l *Ledger) add(ban Ban) {
|
||||
counted := ban.Cause != CauseAdmin
|
||||
if counted && l.held == l.rules.MaxBans {
|
||||
if counted && l.held == l.params.MaxBans {
|
||||
l.dropOne()
|
||||
}
|
||||
|
||||
@@ -825,7 +847,7 @@ func (l *Ledger) add(ban Ban) {
|
||||
// new ban longer. A ban an admin adds to bans.json can start after
|
||||
// another and end before it, so that one is looked for among them all.
|
||||
func (l *Ledger) limitExpiry(held []Ban, now time.Time) time.Time {
|
||||
length := l.rules.LimitBanDuration
|
||||
length := l.params.LimitBanDuration
|
||||
|
||||
var last *Ban
|
||||
|
||||
@@ -836,18 +858,18 @@ func (l *Ledger) limitExpiry(held []Ban, now time.Time) time.Time {
|
||||
}
|
||||
}
|
||||
|
||||
if last != nil && now.Sub(last.Expires) <= l.rules.LimitBanRepeatWindow {
|
||||
if last != nil && now.Sub(last.Expires) <= l.params.LimitBanRepeatWindow {
|
||||
lastLength := last.Expires.Sub(last.Start)
|
||||
// This is repeatFactor * lastLength > MaxBanDuration, written so
|
||||
// that it cannot overflow.
|
||||
if lastLength > l.rules.MaxBanDuration/repeatFactor {
|
||||
if lastLength > l.params.MaxBanDuration/repeatFactor {
|
||||
return time.Time{}
|
||||
}
|
||||
|
||||
length = repeatFactor * lastLength
|
||||
}
|
||||
|
||||
if length > l.rules.MaxBanDuration {
|
||||
if length > l.params.MaxBanDuration {
|
||||
return time.Time{}
|
||||
}
|
||||
|
||||
@@ -865,7 +887,7 @@ func (l *Ledger) attackExpiry(held []Ban, now time.Time) time.Time {
|
||||
}
|
||||
}
|
||||
|
||||
return now.Add(l.rules.AttackBanDuration)
|
||||
return now.Add(l.params.AttackBanDuration)
|
||||
}
|
||||
|
||||
// dropOne drops the earliest ban whose cause is not CauseAdmin of the
|
||||
|
||||
+27
-27
@@ -15,7 +15,7 @@ const day = 24 * time.Hour
|
||||
func TestRepeatsTripleUntilPermanent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
now := midnight()
|
||||
|
||||
@@ -62,7 +62,7 @@ func TestRepeatWindowRunsOut(t *testing.T) {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
||||
@@ -80,9 +80,9 @@ func TestRepeatWindowRunsOut(t *testing.T) {
|
||||
func TestFirstBanLongerThanTheMaximumIsPermanent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
rules := defaultRules()
|
||||
rules.LimitBanDuration = rules.MaxBanDuration + time.Hour
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.LimitBanDuration = params.MaxBanDuration + time.Hour
|
||||
ledger := bans.New(params)
|
||||
|
||||
ban, _ := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"), midnight(),
|
||||
bans.Notes{})
|
||||
@@ -97,9 +97,9 @@ func TestLongestBanSetFarOffDoesNotOverflow(t *testing.T) {
|
||||
// With bans of up to 100,000 days, the 14th ban in a row, of 3^13
|
||||
// hours, is within the maximum, and three times as long would not fit
|
||||
// in a time.Duration. The 15th is permanent.
|
||||
rules := defaultRules()
|
||||
rules.MaxBanDuration = 100000 * day
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBanDuration = 100000 * day
|
||||
ledger := bans.New(params)
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
now := midnight()
|
||||
|
||||
@@ -121,7 +121,7 @@ func TestLongestBanSetFarOffDoesNotOverflow(t *testing.T) {
|
||||
func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
first, made := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
||||
@@ -146,7 +146,7 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
|
||||
func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
ban, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{Requests: 5})
|
||||
|
||||
@@ -178,7 +178,7 @@ func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
|
||||
func TestFindCountsNothing(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
ban, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{Requests: 5})
|
||||
|
||||
@@ -200,9 +200,9 @@ func TestFindCountsNothing(t *testing.T) {
|
||||
func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
rules := defaultRules()
|
||||
rules.MaxBans = 3
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBans = 3
|
||||
ledger := bans.New(params)
|
||||
a := netip.MustParsePrefix("203.0.113.1/32")
|
||||
b := netip.MustParsePrefix("203.0.113.2/32")
|
||||
c := netip.MustParsePrefix("203.0.113.3/32")
|
||||
@@ -239,9 +239,9 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
|
||||
|
||||
// With room for one ban, the netblock's ended ban goes to make room for
|
||||
// its new one, whose notes still count it.
|
||||
rules := defaultRules()
|
||||
rules.MaxBans = 1
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBans = 1
|
||||
ledger := bans.New(params)
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
||||
@@ -258,7 +258,7 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
|
||||
func TestRequestDuringAnAttackBanMakesItPermanent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
notes := bans.Notes{RuleID: "env-file", Target: "path"}
|
||||
|
||||
@@ -304,7 +304,7 @@ func TestRequestDuringAnAttackBanMakesItPermanent(t *testing.T) {
|
||||
func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
// A ban for a broken limit before does not count.
|
||||
@@ -336,7 +336,7 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
|
||||
func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
||||
wantChanged(t, ledger, true)
|
||||
@@ -380,7 +380,7 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
|
||||
func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
now := midnight()
|
||||
|
||||
@@ -414,7 +414,7 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
|
||||
func TestAttackBanDoesNotLengthenTheNextBanForALimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
// Three times the seven days would be permanent; a limit broken as the
|
||||
@@ -436,7 +436,7 @@ func TestAttackBanDoesNotLengthenTheNextBanForALimit(t *testing.T) {
|
||||
func TestRequestTextsAreCutTo256Bytes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
long := strings.Repeat("a", 300)
|
||||
request := bans.Request{
|
||||
@@ -458,7 +458,7 @@ func TestRequestTextsAreCutTo256Bytes(t *testing.T) {
|
||||
func TestLookupFillsTheNotesOfTheNetblocksBansWithoutOne(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
other := netip.MustParsePrefix("198.51.100.7/32")
|
||||
|
||||
@@ -494,9 +494,9 @@ func TestLookupFillsTheNotesOfTheNetblocksBansWithoutOne(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// defaultRules are the rules at the settings' defaults.
|
||||
func defaultRules() bans.Rules {
|
||||
return bans.Rules{
|
||||
// defaultParams are the params at the settings' defaults.
|
||||
func defaultParams() bans.Params {
|
||||
return bans.Params{
|
||||
LimitBanDuration: time.Hour,
|
||||
LimitBanRepeatWindow: day,
|
||||
MaxBanDuration: 7 * day,
|
||||
|
||||
@@ -15,7 +15,7 @@ const scenario = "crowdsecurity/ssh-bf"
|
||||
func TestCrowdSecBanLastsUntilTheDecisionEnds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
expires := midnight().Add(4 * time.Hour)
|
||||
|
||||
@@ -49,7 +49,7 @@ func TestCrowdSecBanLastsUntilTheDecisionEnds(t *testing.T) {
|
||||
func TestCrowdSecBanIsNeverMadePermanent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
expires := midnight().Add(4 * time.Hour)
|
||||
ledger.BanForCrowdSec(netblock, midnight(), expires, scenario, bans.Notes{})
|
||||
@@ -72,7 +72,7 @@ func TestCrowdSecBanIsNeverMadePermanent(t *testing.T) {
|
||||
func TestCrowdSecBanIsCountedAndDoesNotLengthenTheNextBanForALimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
// Three times the three days would be permanent; a limit broken as the
|
||||
|
||||
@@ -14,7 +14,7 @@ import (
|
||||
func TestChangedAfterABanIsMade(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
wantChanged(t, ledger, false)
|
||||
@@ -38,7 +38,7 @@ func TestChangedAfterABanIsMade(t *testing.T) {
|
||||
func TestSnapshotListsEveryBanByNetblock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
v6 := netip.MustParsePrefix("2001:db8::/64")
|
||||
high := netip.MustParsePrefix("203.0.113.10/32")
|
||||
low := netip.MustParsePrefix("203.0.113.9/32")
|
||||
@@ -67,14 +67,14 @@ func TestSnapshotListsEveryBanByNetblock(t *testing.T) {
|
||||
func TestLoadedBansCarryOn(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
before := bans.New(defaultRules())
|
||||
before := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
ban, _ := before.BanForLimit(netblock, midnight(), bans.Notes{Limit: 1})
|
||||
|
||||
// Loaded into a new ledger, as across a restart, the ban still refuses
|
||||
// while it lasts, and once it has ended a broken limit bans for three
|
||||
// times as long, with the loaded ban counted among the earlier ones.
|
||||
after := bans.New(defaultRules())
|
||||
after := bans.New(defaultParams())
|
||||
after.Load(before.Snapshot())
|
||||
|
||||
_, banned, _ := after.Check(netblock.Addr(), ban.Expires.Add(-time.Second))
|
||||
@@ -96,7 +96,7 @@ func TestLoadedBanRefusesEveryClientInItsNetblock(t *testing.T) {
|
||||
// Two entries as an admin might write them, with addresses not masked
|
||||
// to their lengths, the IPv6 one shorter than the /64 an IPv6 client's
|
||||
// ban covers, beside a ban the ledger makes on one IPv4 address.
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{
|
||||
{Netblock: netip.MustParsePrefix("203.0.113.9/24"), Start: midnight()},
|
||||
{Netblock: netip.MustParsePrefix("2001:db8::1/48"), Start: midnight()},
|
||||
@@ -145,7 +145,7 @@ func TestPermanentBanStartedBeforeAnEndedOneRefuses(t *testing.T) {
|
||||
Expires: midnight().Add(time.Hour),
|
||||
}
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{permanent, ended})
|
||||
|
||||
now := midnight().Add(2 * time.Hour)
|
||||
@@ -189,7 +189,7 @@ func TestNextBanWorkedOutFromTheBanThatEndedLast(t *testing.T) {
|
||||
Expires: midnight().Add(2 * time.Hour),
|
||||
}
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{nineHours, admins})
|
||||
|
||||
// Once both have ended, a limit broken within the repeat window bans
|
||||
@@ -219,9 +219,9 @@ func TestLoadKeepsAtMostMaxBansDroppingTheEarliest(t *testing.T) {
|
||||
Cause: bans.CauseLimit,
|
||||
}
|
||||
|
||||
rules := defaultRules()
|
||||
rules.MaxBans = 1
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBans = 1
|
||||
ledger := bans.New(params)
|
||||
ledger.Load([]bans.Ban{later, earlier})
|
||||
|
||||
held := ledger.Snapshot()
|
||||
@@ -235,9 +235,9 @@ func TestLoadReplacesTheBansHeld(t *testing.T) {
|
||||
|
||||
// Room for three bans, so that the second load, were it added to the
|
||||
// two bans held, would drop none of them to make room.
|
||||
rules := defaultRules()
|
||||
rules.MaxBans = 3
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBans = 3
|
||||
ledger := bans.New(params)
|
||||
kept := bans.Ban{
|
||||
Netblock: netip.MustParsePrefix("2001:db8::/64"),
|
||||
Start: midnight(),
|
||||
@@ -285,7 +285,7 @@ func TestLoadCutsTheTextsTo256Bytes(t *testing.T) {
|
||||
}},
|
||||
}
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{ban})
|
||||
|
||||
cut := long[:256]
|
||||
|
||||
+538
-59
@@ -12,6 +12,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"math"
|
||||
"net"
|
||||
"net/http"
|
||||
@@ -28,6 +29,7 @@ import (
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
||||
)
|
||||
|
||||
@@ -93,29 +95,33 @@ type Config struct {
|
||||
LookupDBPath string
|
||||
LookupTimeout time.Duration
|
||||
AddLookupHeaders bool
|
||||
// DeniedCountries are the countries whose clients are refused
|
||||
// (SWW_DENIED_COUNTRIES). ExclusivelyAllowedCountries, when not
|
||||
// empty, are the only countries whose clients are let through
|
||||
// (SWW_EXCLUSIVELY_ALLOWED_COUNTRIES). Both hold two-letter codes in
|
||||
// capitals, as GeoJS gives them.
|
||||
DeniedCountries []string
|
||||
// ExclusivelyAllowedCountries, when not empty, are the only countries
|
||||
// whose clients are let through (SWW_EXCLUSIVELY_ALLOWED_COUNTRIES),
|
||||
// two-letter codes in capitals, as GeoJS gives them.
|
||||
ExclusivelyAllowedCountries []string
|
||||
// The biased thresholds. ASNLimitPercent and CountryLimitPercent give
|
||||
// the clients of the AS numbers and the countries they list that
|
||||
// percentage of every rate limit and byte limit
|
||||
// (SWW_ASN_LIMIT_PERCENT and SWW_COUNTRY_LIMIT_PERCENT).
|
||||
// ASNBytesPercent and CountryBytesPercent give those they list a
|
||||
// percentage of the byte limits in place of that one
|
||||
// TagLists are the address lists that each add a tag to the requests of
|
||||
// the clients they hold (SWW_TAG_LISTS). TagDeny are the tags whose
|
||||
// requests are refused (SWW_TAG_DENY). TagLimitPercent gives the clients
|
||||
// with a tag that percentage of every rate limit and byte limit, from 1
|
||||
// to 99 (SWW_TAG_LIMIT_PERCENT). All hold tags as requests carry them: in
|
||||
// lower case, a country code in capitals.
|
||||
TagLists []TagList
|
||||
TagDeny []string
|
||||
TagLimitPercent map[string]int64
|
||||
// RouteWatchURL is where the routewatch instance is that adds the asn=
|
||||
// and prefix= tags, "" for none (SWW_ROUTEWATCH_URL).
|
||||
RouteWatchURL string
|
||||
// The biased thresholds. ASNBytesPercent and CountryBytesPercent give
|
||||
// those they list a percentage of the byte limits in place of the one
|
||||
// TagLimitPercent gives their asn= or country= tag
|
||||
// (SWW_ASN_BYTES_PERCENT and SWW_COUNTRY_BYTES_PERCENT). Each holds
|
||||
// percentages from 0 to 100, by AS number, written as AS64496, or by
|
||||
// country, a two-letter code in capitals, as the lookup gives them.
|
||||
// UnknownLimitPercent is the percentage of every limit a client without
|
||||
// a country gets (SWW_UNKNOWN_LIMIT_PERCENT). ASNLimitPercentURL is
|
||||
// where a file of AS:percent lines is fetched from, whose percentages
|
||||
// count as those of ASNLimitPercent do (SWW_ASN_LIMIT_PERCENT_URL), ""
|
||||
// while it is unset.
|
||||
ASNLimitPercent map[string]int64
|
||||
CountryLimitPercent map[string]int64
|
||||
// count like asn= tags in TagLimitPercent (SWW_ASN_LIMIT_PERCENT_URL),
|
||||
// "" while it is unset.
|
||||
ASNBytesPercent map[string]int64
|
||||
CountryBytesPercent map[string]int64
|
||||
UnknownLimitPercent int64
|
||||
@@ -164,8 +170,8 @@ type Config struct {
|
||||
// BanResponse is the status a refused client is answered with, 403
|
||||
// or 429, or 0 to close the connection without an answer
|
||||
// (SWW_BAN_RESPONSE). It answers a banned client, a request that
|
||||
// breaks a rate limit or matches a ban rule, SWW_DENY_NETS and the
|
||||
// country lists.
|
||||
// breaks a rate limit or matches a ban rule, SWW_DENY_NETS,
|
||||
// SWW_EXCLUSIVELY_ALLOWED_COUNTRIES and SWW_TAG_DENY.
|
||||
BanResponse int
|
||||
// LimitBanDuration is the ban for a first broken rate limit
|
||||
// (SWW_LIMIT_BAN_DURATION). A limit broken again within
|
||||
@@ -230,6 +236,11 @@ type Config struct {
|
||||
WAFExemptPaths []string
|
||||
WAFBodyLimit int64
|
||||
CoreRuleSetExclusionsFile string
|
||||
// Apps are the names of the apps each hostname serves, by hostname in
|
||||
// lowercase (SWW_APPS): a request gets only its hostname's apps' fixes
|
||||
// to the Core Rule Set. Apps is nil while SWW_APPS is unset, and every
|
||||
// request gets every app's fixes.
|
||||
Apps map[string][]string
|
||||
// TrapPaths are the paths a request for which is a clear sign of
|
||||
// attack (SWW_TRAP_PATHS), each starting with / and without a ?.
|
||||
TrapPaths []string
|
||||
@@ -331,7 +342,7 @@ const (
|
||||
maxParanoiaLevel = 4
|
||||
// firstSetupRuleID to lastSetupRuleID are the ids the Core Rule Set
|
||||
// keeps for the rules that set it up, which sww's own rules
|
||||
// have too (see internal/waf). Switching one off would undo a change
|
||||
// have too (see internal/coreruleset). Switching one off would undo a change
|
||||
// that no setting undoes.
|
||||
firstSetupRuleID = 900000
|
||||
lastSetupRuleID = 900999
|
||||
@@ -367,7 +378,7 @@ var (
|
||||
errHeaderTakenOut = errors.New(
|
||||
"is taken out of every request by Go's HTTP server, so it can never " +
|
||||
"be logged")
|
||||
errOnBothLists = errors.New("is in SWW_DENIED_COUNTRIES too")
|
||||
errOnBothLists = errors.New("is in SWW_TAG_DENY too")
|
||||
errNotLookupSource = errors.New("is not geojs, file or off")
|
||||
errNeedsLookups = errors.New("it needs each client looked up")
|
||||
errNeedsDBPath = errors.New("it names the file to look clients up in")
|
||||
@@ -397,18 +408,26 @@ var (
|
||||
errNotPathPrefix = errors.New(
|
||||
"is not a path prefix starting with /, such as /assets/")
|
||||
errNotTrapPath = errors.New(
|
||||
"is not a path starting with / and without a ?, such as /wp-login.php")
|
||||
"is not a path starting with / and without a ?, such as /wp-login.php",
|
||||
)
|
||||
errNotAppsEntry = errors.New(
|
||||
"is not a hostname, : and a list of apps, such as git.example.com:gitea",
|
||||
)
|
||||
errNotHostname = errors.New("is not a DNS name such as git.example.com")
|
||||
errNotApp = errors.New("is not an app sww has fixes for; the apps are")
|
||||
errNotWAFMode = errors.New("is not off, detect or block")
|
||||
errNotParanoiaLevel = errors.New("is not a paranoia level, from 1 to 4")
|
||||
errNotRuleID = errors.New(
|
||||
"is not the id of a Core Rule Set rule, a whole number such as 942100")
|
||||
"is not the id of a Core Rule Set rule, a whole number such as 942100",
|
||||
)
|
||||
errSetupRuleID = errors.New(
|
||||
"is from 900000 to 900999, the ids of the rules that set the Core Rule Set " +
|
||||
"up and of sww's own, which cannot be switched off")
|
||||
errNotBoolean = errors.New("is not true or false")
|
||||
errNotLogRemoteURL = errors.New(
|
||||
"is not syslog+udp, syslog+tcp or syslog+tls with a host and a port, " +
|
||||
"and nothing more, such as syslog+tls://logs.example:6514")
|
||||
"and nothing more, such as syslog+tls://logs.example:6514",
|
||||
)
|
||||
errNoCertificate = errors.New("holds no PEM certificate")
|
||||
errNotFacility = errors.New("is not a syslog facility such as local0 or daemon")
|
||||
errNotAppName = errors.New(
|
||||
@@ -421,16 +440,21 @@ var (
|
||||
"is not a header name followed by : and the header's value, " +
|
||||
"such as Authorization:Bearer <token>")
|
||||
errControlCharacter = errors.New(
|
||||
"holds a control character, such as the carriage return of a Windows line end")
|
||||
"holds a control character, such as the carriage return of a Windows line end",
|
||||
)
|
||||
errNotAlertEvent = errors.New(
|
||||
"is not ban, permanent_ban, waf_block, anomaly, reputation_hit, " +
|
||||
"is not ban, permanent_ban, core_rule_set_match, anomaly, reputation_hit, " +
|
||||
"source_failure or file_error")
|
||||
errNotNumberOrOff = errors.New("is not a whole number above zero, such as 60, or off")
|
||||
errNotNumberOrOff = errors.New(
|
||||
"is not a whole number above zero, such as 60, or off",
|
||||
)
|
||||
errNotUTF8 = errors.New("is not valid UTF-8")
|
||||
errNotASN = errors.New("is not an AS number such as AS64496")
|
||||
errNotPercentItem = errors.New(
|
||||
"is not a code, : and a percentage, such as AS64496:50 or cn:25")
|
||||
errNotPercent = errors.New("is not a percentage, a whole number from 0 to 100")
|
||||
errNotPercent = errors.New(
|
||||
"is not a percentage, a whole number from 0 to 100",
|
||||
)
|
||||
errListedTwice = errors.New("is listed twice")
|
||||
errNotListURL = errors.New(
|
||||
"is not an http or https URL without a user or a fragment, " +
|
||||
@@ -439,18 +463,49 @@ var (
|
||||
errNotAnHourOrMore = errors.New("is not a duration of 1h or more, such as 24h")
|
||||
errNotAction = errors.New(
|
||||
"is not deny, limit:<percent> such as limit:25, or log")
|
||||
errNotZone = errors.New("is not a DNS zone such as dnsbl.dronebl.org")
|
||||
errZoneTooLong = errors.New("is longer than 189 characters, too long for the " +
|
||||
"names IPv6 clients are asked about by")
|
||||
errNotResolver = errors.New("is not an IP address with an optional port, " +
|
||||
"such as 192.0.2.53 or [2001:db8::53]:5353")
|
||||
errNotZone = errors.New(
|
||||
"is not a DNS zone such as dnsbl.dronebl.org",
|
||||
)
|
||||
errZoneTooLong = errors.New(
|
||||
"is longer than 189 characters, too long for the " +
|
||||
"names IPv6 clients are asked about by",
|
||||
)
|
||||
errNotResolver = errors.New(
|
||||
"is not an IP address with an optional port, " +
|
||||
"such as 192.0.2.53 or [2001:db8::53]:5353",
|
||||
)
|
||||
errNotLAPIURL = errors.New(
|
||||
"is not an http or https URL without a user or a fragment, " +
|
||||
"such as http://172.17.0.1:8080")
|
||||
errNotRouteWatchURL = errors.New(
|
||||
"is not an http or https URL without a user or a fragment, " +
|
||||
"such as https://routewatch.example")
|
||||
errNeedsLAPIKey = errors.New("the engine answers no request without it")
|
||||
errLAPIKeyUnused = errors.New("it is sent only to the engine at that URL")
|
||||
errAnotherList = errors.New(
|
||||
"is in SWW_BLOCKLIST_URLS or is SWW_ASN_LIMIT_PERCENT_URL too")
|
||||
errFetchedOtherwise = errors.New("is in SWW_BLOCKLIST_URLS, is " +
|
||||
"SWW_ASN_LIMIT_PERCENT_URL or is the CrowdSec decision list too")
|
||||
errNotTagList = errors.New(
|
||||
"is not a name, = and an http or https URL, such as " +
|
||||
"torexit=https://example.com/tor.txt")
|
||||
errNotTagPercentItem = errors.New(
|
||||
"is not a tag, : and a percentage, such as vpn:50 or asn=14061:25")
|
||||
errNotTagPercent = errors.New("is not a percentage from 1 to 99")
|
||||
errNotASNTag = errors.New("is not asn= and a number, such as asn=14061")
|
||||
errNotPrefixTag = errors.New(
|
||||
"is not prefix= and a netblock, such as prefix=192.0.2.0/24")
|
||||
errExploitTag = errors.New(
|
||||
"cannot lower limits: the rate limits count a request before its " +
|
||||
"rules are matched")
|
||||
errCrowdSecTag = errors.New(
|
||||
"cannot be used: a CrowdSec decision refuses the request first")
|
||||
errNoRequestGetsTag = errors.New(
|
||||
"is a tag no request can get: it is not a name in SWW_TAG_LISTS, an " +
|
||||
"asn= or country= tag, a dnsbl= tag of a zone in SWW_DNSBL_ZONES, " +
|
||||
"a prefix= tag with SWW_ROUTEWATCH_URL set, abuseipdb with " +
|
||||
"SWW_ABUSEIPDB_KEY set, or exploitsignature with rule files or the " +
|
||||
"Core Rule Set on")
|
||||
)
|
||||
|
||||
// FromEnvironment reads the settings with lookupEnv, normally
|
||||
@@ -460,7 +515,9 @@ var (
|
||||
// setting that is set but invalid is an error that names it.
|
||||
//
|
||||
//nolint:funlen // one line for each setting, a list that grows with them
|
||||
func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
||||
func FromEnvironment(
|
||||
lookupEnv func(string) (string, bool),
|
||||
) (*Config, error) {
|
||||
env := &environment{lookupEnv: lookupEnv}
|
||||
cfg := &Config{
|
||||
ListenAddr: env.address("SWW_LISTEN_ADDR"),
|
||||
@@ -483,11 +540,12 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
||||
LookupDBPath: env.value("SWW_LOOKUP_DB_PATH", ""),
|
||||
LookupTimeout: env.durationNotOff("SWW_LOOKUP_TIMEOUT", "800ms"),
|
||||
AddLookupHeaders: env.boolean("SWW_ADD_LOOKUP_HEADERS", "false"),
|
||||
DeniedCountries: env.countries("SWW_DENIED_COUNTRIES", ""),
|
||||
ExclusivelyAllowedCountries: env.countries(
|
||||
"SWW_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
|
||||
ASNLimitPercent: env.percents("SWW_ASN_LIMIT_PERCENT", ParseASN),
|
||||
CountryLimitPercent: env.percents("SWW_COUNTRY_LIMIT_PERCENT", parseCountry),
|
||||
TagLists: env.tagLists("SWW_TAG_LISTS"),
|
||||
TagDeny: env.tags("SWW_TAG_DENY"),
|
||||
TagLimitPercent: env.tagPercents("SWW_TAG_LIMIT_PERCENT"),
|
||||
RouteWatchURL: env.routeWatchURL("SWW_ROUTEWATCH_URL"),
|
||||
ASNBytesPercent: env.percents("SWW_ASN_BYTES_PERCENT", ParseASN),
|
||||
CountryBytesPercent: env.percents("SWW_COUNTRY_BYTES_PERCENT", parseCountry),
|
||||
UnknownLimitPercent: env.percent("SWW_UNKNOWN_LIMIT_PERCENT", "100"),
|
||||
@@ -530,6 +588,7 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
||||
WAFBodyLimit: env.wafBodyLimit("SWW_WAF_BODY_LIMIT", off),
|
||||
CoreRuleSetExclusionsFile: env.filePath(
|
||||
"SWW_CORE_RULE_SET_EXCLUSIONS_FILE"),
|
||||
Apps: env.apps("SWW_APPS"),
|
||||
TrapPaths: env.trapPaths("SWW_TRAP_PATHS"),
|
||||
ErrorBurstThreshold: env.count("SWW_ERROR_BURST_THRESHOLD", "30"),
|
||||
LogRemoteURL: env.logRemoteURL("SWW_LOG_REMOTE_URL"),
|
||||
@@ -567,6 +626,8 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
||||
env.checkCountriesAndLookups(cfg)
|
||||
env.checkASNLimitPercentURL(cfg)
|
||||
env.checkCrowdSec(cfg)
|
||||
env.checkTagLists(cfg)
|
||||
env.checkTags(cfg)
|
||||
|
||||
if env.err != nil {
|
||||
return nil, env.err
|
||||
@@ -658,7 +719,10 @@ func (e *environment) lookup(name string) (string, bool) {
|
||||
}
|
||||
|
||||
if set {
|
||||
e.check(name, fmt.Errorf("is set, and so is %s; %w", fileName, errSetTwice))
|
||||
e.check(
|
||||
name,
|
||||
fmt.Errorf("is set, and so is %s; %w", fileName, errSetTwice),
|
||||
)
|
||||
|
||||
return value, set
|
||||
}
|
||||
@@ -780,6 +844,25 @@ func (e *environment) pathPrefixes(name, defaultValue string) []string {
|
||||
return prefixes
|
||||
}
|
||||
|
||||
// apps reads the setting that is the apps each hostname serves. It is nil
|
||||
// while the setting is unset, and logged as null, unlike a setting set
|
||||
// empty.
|
||||
func (e *environment) apps(name string) map[string][]string {
|
||||
value, set := e.lookup(name)
|
||||
if !set {
|
||||
e.settings = append(e.settings, slog.Any(name, nil))
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
e.settings = append(e.settings, slog.String(name, value))
|
||||
|
||||
apps, err := parseApps(value)
|
||||
e.check(name, err)
|
||||
|
||||
return apps
|
||||
}
|
||||
|
||||
// trapPaths reads the setting that is the list of trap paths. It is empty
|
||||
// by default.
|
||||
func (e *environment) trapPaths(name string) []string {
|
||||
@@ -841,6 +924,31 @@ func (e *environment) percents(
|
||||
return percents
|
||||
}
|
||||
|
||||
// tagLists reads SWW_TAG_LISTS, which is empty by default.
|
||||
func (e *environment) tagLists(name string) []TagList {
|
||||
lists, err := parseTagLists(e.value(name, ""))
|
||||
e.check(name, err)
|
||||
|
||||
return lists
|
||||
}
|
||||
|
||||
// tags reads a setting that is a list of tags. It is empty by default.
|
||||
func (e *environment) tags(name string) []string {
|
||||
tags, err := parseTags(e.value(name, ""))
|
||||
e.check(name, err)
|
||||
|
||||
return tags
|
||||
}
|
||||
|
||||
// tagPercents reads a setting that is a list of tags, each with a
|
||||
// percentage. It is empty by default.
|
||||
func (e *environment) tagPercents(name string) map[string]int64 {
|
||||
percents, err := parseTagPercents(e.value(name, ""))
|
||||
e.check(name, err)
|
||||
|
||||
return percents
|
||||
}
|
||||
|
||||
// percent reads a setting that is a percentage, from 0 to 100.
|
||||
func (e *environment) percent(name, defaultValue string) int64 {
|
||||
percent, err := ParsePercent(e.value(name, defaultValue))
|
||||
@@ -916,7 +1024,10 @@ func (e *environment) zones(name string) []string {
|
||||
logged[i] = MaskZoneKey(zone)
|
||||
}
|
||||
|
||||
e.settings = append(e.settings, slog.String(name, strings.Join(logged, ",")))
|
||||
e.settings = append(
|
||||
e.settings,
|
||||
slog.String(name, strings.Join(logged, ",")),
|
||||
)
|
||||
|
||||
return zones
|
||||
}
|
||||
@@ -950,6 +1061,22 @@ func (e *environment) crowdSecDecisionsURL(name string) string {
|
||||
return lapi.JoinPath("v1", "decisions").String()
|
||||
}
|
||||
|
||||
// routeWatchURL reads the setting that is the URL of a routewatch instance,
|
||||
// "" while it is unset or empty.
|
||||
func (e *environment) routeWatchURL(name string) string {
|
||||
value := e.value(name, "")
|
||||
if value == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
instance, err := url.Parse(value)
|
||||
if err != nil || !isHTTPURL(instance) {
|
||||
e.check(name, fmt.Errorf("%q %w", value, errNotRouteWatchURL))
|
||||
}
|
||||
|
||||
return value
|
||||
}
|
||||
|
||||
// lookupSource reads the setting that is where clients are looked up:
|
||||
// geojs, file, or off.
|
||||
func (e *environment) lookupSource(name, defaultValue string) string {
|
||||
@@ -975,15 +1102,17 @@ func (e *environment) checkLookupDBPath(cfg *Config) {
|
||||
}
|
||||
}
|
||||
|
||||
// checkCountriesAndLookups refuses a country on both country lists, and,
|
||||
// checkCountriesAndLookups refuses a country in
|
||||
// SWW_EXCLUSIVELY_ALLOWED_COUNTRIES whose tag is in SWW_TAG_DENY, and,
|
||||
// while SWW_LOOKUP_SOURCE is off, each setting that needs clients looked
|
||||
// up: the country lists, SWW_ADD_LOOKUP_HEADERS, the biased thresholds,
|
||||
// up: the country list, SWW_ADD_LOOKUP_HEADERS, the tag settings that name
|
||||
// an asn= or country= tag, the biased thresholds,
|
||||
// SWW_ASN_LIMIT_PERCENT_URL among them, of which
|
||||
// SWW_UNKNOWN_LIMIT_PERCENT needs them only below 100, where it lowers a
|
||||
// limit, and the anomaly thresholds per AS number.
|
||||
func (e *environment) checkCountriesAndLookups(cfg *Config) {
|
||||
for _, country := range cfg.ExclusivelyAllowedCountries {
|
||||
if slices.Contains(cfg.DeniedCountries, country) {
|
||||
if slices.Contains(cfg.TagDeny, CountryTagPrefix+country) {
|
||||
e.check("SWW_EXCLUSIVELY_ALLOWED_COUNTRIES",
|
||||
fmt.Errorf("%q %w", country, errOnBothLists))
|
||||
}
|
||||
@@ -993,15 +1122,21 @@ func (e *environment) checkCountriesAndLookups(cfg *Config) {
|
||||
return
|
||||
}
|
||||
|
||||
// routewatch gives asn= tags too.
|
||||
onlyLookupGives := func(tag string) bool {
|
||||
return isLookupTag(tag) &&
|
||||
(cfg.RouteWatchURL == "" || !strings.HasPrefix(tag, ASNTagPrefix))
|
||||
}
|
||||
|
||||
for _, setting := range []struct {
|
||||
name string
|
||||
set bool
|
||||
}{
|
||||
{"SWW_DENIED_COUNTRIES", len(cfg.DeniedCountries) > 0},
|
||||
{"SWW_EXCLUSIVELY_ALLOWED_COUNTRIES", len(cfg.ExclusivelyAllowedCountries) > 0},
|
||||
{"SWW_ADD_LOOKUP_HEADERS", cfg.AddLookupHeaders},
|
||||
{"SWW_ASN_LIMIT_PERCENT", len(cfg.ASNLimitPercent) > 0},
|
||||
{"SWW_COUNTRY_LIMIT_PERCENT", len(cfg.CountryLimitPercent) > 0},
|
||||
{"SWW_TAG_DENY", slices.ContainsFunc(cfg.TagDeny, onlyLookupGives)},
|
||||
{"SWW_TAG_LIMIT_PERCENT", slices.ContainsFunc(
|
||||
slices.Collect(maps.Keys(cfg.TagLimitPercent)), onlyLookupGives)},
|
||||
{"SWW_ASN_BYTES_PERCENT", len(cfg.ASNBytesPercent) > 0},
|
||||
{"SWW_COUNTRY_BYTES_PERCENT", len(cfg.CountryBytesPercent) > 0},
|
||||
{"SWW_UNKNOWN_LIMIT_PERCENT", cfg.UnknownLimitPercent < 100},
|
||||
@@ -1047,6 +1182,94 @@ func (e *environment) checkCrowdSec(cfg *Config) {
|
||||
}
|
||||
}
|
||||
|
||||
// checkTagLists refuses a list of SWW_TAG_LISTS whose URL is fetched as
|
||||
// another list too: the file at a URL is read as one kind of list.
|
||||
func (e *environment) checkTagLists(cfg *Config) {
|
||||
for _, list := range cfg.TagLists {
|
||||
if slices.Contains(cfg.BlocklistURLs, list.URL) ||
|
||||
list.URL == cfg.ASNLimitPercentURL || list.URL == cfg.CrowdSecDecisionsURL {
|
||||
e.check("SWW_TAG_LISTS", fmt.Errorf("%q %w", list.URL, errFetchedOtherwise))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// checkTags refuses a tag of SWW_TAG_DENY or SWW_TAG_LIMIT_PERCENT that would
|
||||
// never apply: crowdsec, whose request is refused before tags are acted on, and
|
||||
// any tag these settings cannot give a request.
|
||||
func (e *environment) checkTags(cfg *Config) {
|
||||
for _, tag := range cfg.TagDeny {
|
||||
e.check("SWW_TAG_DENY", checkTag(cfg, tag))
|
||||
}
|
||||
|
||||
for tag := range cfg.TagLimitPercent {
|
||||
e.check("SWW_TAG_LIMIT_PERCENT", checkTag(cfg, tag))
|
||||
}
|
||||
}
|
||||
|
||||
// checkTag returns an error when tag would never apply, as checkTags says.
|
||||
func checkTag(cfg *Config, tag string) error {
|
||||
if tag == CrowdSecTag {
|
||||
return fmt.Errorf("%q %w", tag, errCrowdSecTag)
|
||||
}
|
||||
|
||||
if !requestCanGetTag(cfg, tag) {
|
||||
return fmt.Errorf("%q %w", MaskZoneKey(tag), errNoRequestGetsTag)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// requestCanGetTag reports whether anything cfg sets up can give a request tag.
|
||||
func requestCanGetTag(cfg *Config, tag string) bool {
|
||||
isListName := func(list TagList) bool { return list.Name == tag }
|
||||
if isLookupTag(tag) || slices.ContainsFunc(cfg.TagLists, isListName) {
|
||||
return true
|
||||
}
|
||||
|
||||
if tag == ExploitSignatureTag {
|
||||
coreRuleSetOn := cfg.WAFMode != WAFModeOff && cfg.WAFAnomalyThreshold != 0
|
||||
|
||||
return cfg.RulesEnabled || coreRuleSetOn
|
||||
}
|
||||
|
||||
if tag == AbuseIPDBTag {
|
||||
return cfg.AbuseIPDBKey != ""
|
||||
}
|
||||
|
||||
if strings.HasPrefix(tag, PrefixTagPrefix) {
|
||||
return cfg.RouteWatchURL != ""
|
||||
}
|
||||
|
||||
zone, isZoneTag := strings.CutPrefix(tag, DNSBLTagPrefix)
|
||||
|
||||
return isZoneTag && slices.ContainsFunc(cfg.DNSBLZones, func(each string) bool {
|
||||
return strings.ToLower(MaskZoneKey(each)) == zone
|
||||
})
|
||||
}
|
||||
|
||||
// TagsNeedLookup reports whether SWW_TAG_DENY or SWW_TAG_LIMIT_PERCENT
|
||||
// names an asn= or country= tag.
|
||||
func (c *Config) TagsNeedLookup() bool {
|
||||
return slices.ContainsFunc(c.TagDeny, isLookupTag) || tagLimitNeedsLookups(c)
|
||||
}
|
||||
|
||||
// tagLimitNeedsLookups reports whether SWW_TAG_LIMIT_PERCENT names an asn=
|
||||
// or country= tag.
|
||||
func tagLimitNeedsLookups(cfg *Config) bool {
|
||||
for tag := range cfg.TagLimitPercent {
|
||||
if isLookupTag(tag) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// isLookupTag reports whether tag is an asn= or country= tag.
|
||||
func isLookupTag(tag string) bool {
|
||||
return strings.HasPrefix(tag, ASNTagPrefix) || strings.HasPrefix(tag, CountryTagPrefix)
|
||||
}
|
||||
|
||||
// headerNames reads a setting that is a list of header names, and
|
||||
// returns them in lower case.
|
||||
func (e *environment) headerNames(name, defaultValue string) []string {
|
||||
@@ -1058,7 +1281,9 @@ func (e *environment) headerNames(name, defaultValue string) []string {
|
||||
|
||||
// durationNotOff reads a setting that is a duration and, unlike a
|
||||
// timeout, cannot be off.
|
||||
func (e *environment) durationNotOff(name, defaultValue string) time.Duration {
|
||||
func (e *environment) durationNotOff(
|
||||
name, defaultValue string,
|
||||
) time.Duration {
|
||||
duration, err := ParseDurationNotOff(e.value(name, defaultValue))
|
||||
e.check(name, err)
|
||||
|
||||
@@ -1257,7 +1482,10 @@ func (e *environment) instanceName() string {
|
||||
// appName reads the setting that is the APP-NAME of the records the log
|
||||
// lines are sent in, by default the instance name. Its value is checked
|
||||
// when it is set, and, while lines are sent, when it is the instance name.
|
||||
func (e *environment) appName(name, instanceName string, sending bool) string {
|
||||
func (e *environment) appName(
|
||||
name, instanceName string,
|
||||
sending bool,
|
||||
) string {
|
||||
value, set := e.lookup(name)
|
||||
if !set {
|
||||
value = instanceName
|
||||
@@ -1280,7 +1508,10 @@ func (e *environment) appName(name, instanceName string, sending bool) string {
|
||||
// checkInstanceNameForNtfy refuses an instance name that holds a control
|
||||
// character while ntfySet, SWW_ALERT_NTFY_URL being set: ntfy is sent
|
||||
// the instance name in a header, which cannot hold one.
|
||||
func (e *environment) checkInstanceNameForNtfy(instanceName string, ntfySet bool) {
|
||||
func (e *environment) checkInstanceNameForNtfy(
|
||||
instanceName string,
|
||||
ntfySet bool,
|
||||
) {
|
||||
if ntfySet && strings.ContainsFunc(instanceName, unicode.IsControl) {
|
||||
e.check("SWW_INSTANCE_NAME", fmt.Errorf(
|
||||
"%q %w, and is sent to ntfy in a header while SWW_ALERT_NTFY_URL is set",
|
||||
@@ -1604,7 +1835,10 @@ func parseNamedNetblocks(value string) ([]anomaly.NamedNetblock, error) {
|
||||
return nil, fmt.Errorf("%q %w", name, errListedTwice)
|
||||
}
|
||||
|
||||
named = append(named, anomaly.NamedNetblock{Name: name, Netblock: netblock})
|
||||
named = append(
|
||||
named,
|
||||
anomaly.NamedNetblock{Name: name, Netblock: netblock},
|
||||
)
|
||||
}
|
||||
|
||||
return named, nil
|
||||
@@ -1646,6 +1880,65 @@ func parseTrapPaths(value string) ([]string, error) {
|
||||
return paths, nil
|
||||
}
|
||||
|
||||
// parseApps reads the apps each hostname serves: entries separated by
|
||||
// semicolons, each a hostname, a colon, and a comma-separated list of the
|
||||
// names coreruleset.AppNames returns, such as
|
||||
// git.example.com:gitea;cloud.example.com:nextcloud,wordpress. Each
|
||||
// hostname is a DNS name, without a wildcard. It returns them by hostname
|
||||
// in lowercase, since a hostname is matched without regard to case. An
|
||||
// empty value lists no hostname.
|
||||
func parseApps(value string) (map[string][]string, error) {
|
||||
apps := map[string][]string{}
|
||||
|
||||
if strings.TrimSpace(value) == "" {
|
||||
return apps, nil
|
||||
}
|
||||
|
||||
for entry := range strings.SplitSeq(value, ";") {
|
||||
hostname, list, found := strings.Cut(entry, ":")
|
||||
if !found {
|
||||
return nil, fmt.Errorf("%q %w", entry, errNotAppsEntry)
|
||||
}
|
||||
|
||||
hostname = strings.ToLower(strings.TrimSpace(hostname))
|
||||
if hostname == "" {
|
||||
return nil, fmt.Errorf("%q %w", entry, errNotAppsEntry)
|
||||
}
|
||||
|
||||
if !isZone(hostname) {
|
||||
return nil, fmt.Errorf("%q %w", hostname, errNotHostname)
|
||||
}
|
||||
|
||||
if _, listed := apps[hostname]; listed {
|
||||
return nil, fmt.Errorf("%q %w", hostname, errListedTwice)
|
||||
}
|
||||
|
||||
names, err := parseList(list)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if len(names) == 0 {
|
||||
return nil, fmt.Errorf("%q %w", entry, errNotAppsEntry)
|
||||
}
|
||||
|
||||
for i, name := range names {
|
||||
if !slices.Contains(coreruleset.AppNames(), name) {
|
||||
return nil, fmt.Errorf("%q %w %s", name, errNotApp,
|
||||
strings.Join(coreruleset.AppNames(), ", "))
|
||||
}
|
||||
|
||||
if slices.Contains(names[:i], name) {
|
||||
return nil, fmt.Errorf("%q %w", name, errListedTwice)
|
||||
}
|
||||
}
|
||||
|
||||
apps[hostname] = names
|
||||
}
|
||||
|
||||
return apps, nil
|
||||
}
|
||||
|
||||
// parseRuleIDs reads a comma-separated list of the ids of Core Rule Set
|
||||
// rules, each a whole number above zero and outside firstSetupRuleID to
|
||||
// lastSetupRuleID.
|
||||
@@ -1752,6 +2045,172 @@ func ParseASN(value string) (string, error) {
|
||||
return "AS" + strconv.FormatUint(number, 10), nil
|
||||
}
|
||||
|
||||
// The tags a request can get that the settings can name, as discriminator
|
||||
// writes them. The prefixes are followed by a value.
|
||||
const (
|
||||
ASNTagPrefix = "asn="
|
||||
CountryTagPrefix = "country="
|
||||
DNSBLTagPrefix = "dnsbl="
|
||||
PrefixTagPrefix = "prefix="
|
||||
AbuseIPDBTag = "abuseipdb"
|
||||
CrowdSecTag = "crowdsec"
|
||||
// ExploitSignatureTag is the tag of a request a rule file or the Core
|
||||
// Rule Set matched.
|
||||
ExploitSignatureTag = "exploitsignature"
|
||||
)
|
||||
|
||||
// TagList is an address list of SWW_TAG_LISTS: the tag it adds to the
|
||||
// requests of the clients it holds, and the URL it is fetched from.
|
||||
type TagList struct {
|
||||
Name string
|
||||
URL string
|
||||
}
|
||||
|
||||
// parseTagLists reads a comma-separated list of name=URL pairs, such as
|
||||
// torexit=https://example.com/tor.txt, and returns each list with its name
|
||||
// in lower case. A name can hold a =, as vpn=mullvad does, and ends at the
|
||||
// = before http. A name or a URL listed twice is an error.
|
||||
func parseTagLists(value string) ([]TagList, error) {
|
||||
items, err := parseList(value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
lists := make([]TagList, 0, len(items))
|
||||
|
||||
for _, item := range items {
|
||||
end := strings.Index(item, "=http")
|
||||
if end <= 0 || !isListURL(item[end+1:]) {
|
||||
return nil, fmt.Errorf("%q %w", item, errNotTagList)
|
||||
}
|
||||
|
||||
list := TagList{Name: strings.ToLower(item[:end]), URL: item[end+1:]}
|
||||
|
||||
if slices.ContainsFunc(lists, func(earlier TagList) bool {
|
||||
return earlier.Name == list.Name || earlier.URL == list.URL
|
||||
}) {
|
||||
return nil, fmt.Errorf("%q %w", item, errListedTwice)
|
||||
}
|
||||
|
||||
lists = append(lists, list)
|
||||
}
|
||||
|
||||
return lists, nil
|
||||
}
|
||||
|
||||
// parseTags reads a comma-separated list of tags, which parseTag reads. A
|
||||
// tag listed twice is an error, which shows the tag as MaskZoneKey does.
|
||||
func parseTags(value string) ([]string, error) {
|
||||
items, err := parseList(value)
|
||||
if err != nil {
|
||||
// parseList's error, for an empty item, shows the whole value, keys
|
||||
// included.
|
||||
return nil, errEmptyItem
|
||||
}
|
||||
|
||||
tags := make([]string, 0, len(items))
|
||||
|
||||
for _, item := range items {
|
||||
tag, err := parseTag(item)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if slices.Contains(tags, tag) {
|
||||
return nil, fmt.Errorf("%q %w", MaskZoneKey(item), errListedTwice)
|
||||
}
|
||||
|
||||
tags = append(tags, tag)
|
||||
}
|
||||
|
||||
return tags, nil
|
||||
}
|
||||
|
||||
// parseTag reads a tag in either case, with spaces around it, and returns it
|
||||
// as a request carries it: in lower case, but for a country code, which is in
|
||||
// capitals, an asn= tag's number without leading zeros, and a prefix= tag's
|
||||
// netblock as routewatch writes it. An asn=, country= or prefix= tag must name
|
||||
// a number, a country or a netblock, so that a mistyped one stops the start
|
||||
// instead of never matching.
|
||||
func parseTag(value string) (string, error) {
|
||||
tag := strings.ToLower(strings.TrimSpace(value))
|
||||
|
||||
if digits, isASN := strings.CutPrefix(tag, ASNTagPrefix); isASN {
|
||||
number, err := strconv.ParseUint(digits, 10, 32)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%q %w", value, errNotASNTag)
|
||||
}
|
||||
|
||||
return ASNTagPrefix + strconv.FormatUint(number, 10), nil
|
||||
}
|
||||
|
||||
if code, isCountry := strings.CutPrefix(tag, CountryTagPrefix); isCountry {
|
||||
country, err := parseCountry(code)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return CountryTagPrefix + country, nil
|
||||
}
|
||||
|
||||
if text, isPrefix := strings.CutPrefix(tag, PrefixTagPrefix); isPrefix {
|
||||
netblock, err := netip.ParsePrefix(text)
|
||||
if err != nil || netblock != netblock.Masked() {
|
||||
return "", fmt.Errorf("%q %w", value, errNotPrefixTag)
|
||||
}
|
||||
|
||||
return PrefixTagPrefix + netblock.String(), nil
|
||||
}
|
||||
|
||||
return tag, nil
|
||||
}
|
||||
|
||||
// parseTagPercents reads a comma-separated list of tags, each with : and a
|
||||
// percentage from 1 to 99, such as vpn:50 or asn=14061:25. A tag listed
|
||||
// twice is an error, and so is ExploitSignatureTag, which a request gets
|
||||
// only after the rate limits have counted it.
|
||||
func parseTagPercents(value string) (map[string]int64, error) {
|
||||
items, err := parseList(value)
|
||||
if err != nil {
|
||||
// parseList's error, for an empty item, shows the whole value, keys
|
||||
// included.
|
||||
return nil, errEmptyItem
|
||||
}
|
||||
|
||||
percents := make(map[string]int64, len(items))
|
||||
|
||||
for _, item := range items {
|
||||
colon := strings.LastIndex(item, ":")
|
||||
if colon <= 0 {
|
||||
return nil, fmt.Errorf("%q %w", MaskZoneKey(item), errNotTagPercentItem)
|
||||
}
|
||||
|
||||
tag, err := parseTag(item[:colon])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
percent, err := strconv.ParseInt(item[colon+1:], 10, 64)
|
||||
if err != nil || percent < 1 || percent > 99 {
|
||||
return nil, fmt.Errorf("%q %w", MaskZoneKey(item[colon+1:]), errNotTagPercent)
|
||||
}
|
||||
|
||||
if tag == ExploitSignatureTag {
|
||||
return nil, fmt.Errorf("%q %w", tag, errExploitTag)
|
||||
}
|
||||
|
||||
if _, listed := percents[tag]; listed {
|
||||
// tag, since item[:colon] may end in a space, which MaskZoneKey
|
||||
// would not mask.
|
||||
return nil, fmt.Errorf("%q %w", MaskZoneKey(tag), errListedTwice)
|
||||
}
|
||||
|
||||
percents[tag] = percent
|
||||
}
|
||||
|
||||
return percents, nil
|
||||
}
|
||||
|
||||
// parsePercents reads a comma-separated list of items, each an AS number
|
||||
// or a country, which parseCode reads, then : and a percentage, such as
|
||||
// AS64496:50 or cn:25, and returns each one's percentage. An empty value
|
||||
@@ -1843,8 +2302,11 @@ func parseHeaderNames(value string) ([]string, error) {
|
||||
header := strings.ToLower(item)
|
||||
switch header {
|
||||
case "host":
|
||||
return nil, fmt.Errorf("%q %w; the request's host is the field host",
|
||||
item, errHeaderTakenOut)
|
||||
return nil, fmt.Errorf(
|
||||
"%q %w; the request's host is the field host",
|
||||
item,
|
||||
errHeaderTakenOut,
|
||||
)
|
||||
case "transfer-encoding":
|
||||
return nil, fmt.Errorf("%q %w", item, errHeaderTakenOut)
|
||||
}
|
||||
@@ -1881,9 +2343,11 @@ func parseUpstreamURL(value string) (*url.URL, error) {
|
||||
}
|
||||
|
||||
onlySchemeAndHost := (upstream.Scheme == "http" || upstream.Scheme == "https") &&
|
||||
upstream.Hostname() != "" && upstream.User == nil && upstream.Opaque == "" &&
|
||||
upstream.Hostname() != "" && upstream.User == nil &&
|
||||
upstream.Opaque == "" &&
|
||||
(upstream.Path == "" || upstream.Path == "/") &&
|
||||
upstream.RawQuery == "" && upstream.Fragment == ""
|
||||
upstream.RawQuery == "" &&
|
||||
upstream.Fragment == ""
|
||||
if !onlySchemeAndHost {
|
||||
return nil, fmt.Errorf("%q %w", value, errNotUpstreamURL)
|
||||
}
|
||||
@@ -1907,7 +2371,11 @@ func parseLogRemoteURL(value string) (*url.URL, error) {
|
||||
return nil, fmt.Errorf("%q %w", value, errNotLogRemoteURL)
|
||||
}
|
||||
|
||||
schemes := []string{remotelog.SchemeUDP, remotelog.SchemeTCP, remotelog.SchemeTLS}
|
||||
schemes := []string{
|
||||
remotelog.SchemeUDP,
|
||||
remotelog.SchemeTCP,
|
||||
remotelog.SchemeTLS,
|
||||
}
|
||||
port, err := strconv.ParseUint(remote.Port(), 10, 16)
|
||||
|
||||
onlySchemeHostAndPort := slices.Contains(schemes, remote.Scheme) &&
|
||||
@@ -1970,7 +2438,9 @@ func isHTTPURL(u *url.URL) bool {
|
||||
|
||||
return (u.Scheme == "http" || u.Scheme == "https") && u.Hostname() != "" &&
|
||||
(u.Port() == "" || (err == nil && port != 0)) &&
|
||||
u.User == nil && u.Opaque == "" && u.Fragment == ""
|
||||
u.User == nil &&
|
||||
u.Opaque == "" &&
|
||||
u.Fragment == ""
|
||||
}
|
||||
|
||||
// isListURL reports whether value is a URL a list can be fetched from, as
|
||||
@@ -2037,9 +2507,12 @@ func parseZones(value string) ([]string, error) {
|
||||
|
||||
for i, zone := range zones {
|
||||
shown := MaskZoneKey(zone)
|
||||
listedBefore := slices.ContainsFunc(zones[:i], func(earlier string) bool {
|
||||
listedBefore := slices.ContainsFunc(
|
||||
zones[:i],
|
||||
func(earlier string) bool {
|
||||
return strings.EqualFold(MaskZoneKey(earlier), shown)
|
||||
})
|
||||
},
|
||||
)
|
||||
|
||||
switch {
|
||||
case len(zone) > maxZoneLength:
|
||||
@@ -2072,7 +2545,8 @@ func MaskZoneKey(zone string) string {
|
||||
return masked + "." + rest
|
||||
}
|
||||
|
||||
// isZone reports whether each label of zone is as parseZones takes it.
|
||||
// isZone reports whether each label of zone is as parseZones takes it. It
|
||||
// also checks the hostnames of SWW_APPS, which are DNS names too.
|
||||
func isZone(zone string) bool {
|
||||
for label := range strings.SplitSeq(zone, ".") {
|
||||
badChar := strings.ContainsFunc(label, func(char rune) bool {
|
||||
@@ -2126,8 +2600,13 @@ func parseWebhookHeaders(value string) (http.Header, string, error) {
|
||||
name, headerValue, found := strings.Cut(item, ":")
|
||||
name = strings.TrimSpace(name)
|
||||
|
||||
if !found || !IsHeaderName(name) || strings.ContainsAny(headerValue, "\r\n\x00") {
|
||||
return nil, "", fmt.Errorf("item %d %w", i+1, errNotWebhookHeader)
|
||||
if !found || !IsHeaderName(name) ||
|
||||
strings.ContainsAny(headerValue, "\r\n\x00") {
|
||||
return nil, "", fmt.Errorf(
|
||||
"item %d %w",
|
||||
i+1,
|
||||
errNotWebhookHeader,
|
||||
)
|
||||
}
|
||||
|
||||
headers.Add(name, strings.TrimSpace(headerValue))
|
||||
|
||||
+399
-54
@@ -42,10 +42,10 @@ const (
|
||||
lookupDBPath = "SWW_LOOKUP_DB_PATH"
|
||||
lookupTimeout = "SWW_LOOKUP_TIMEOUT"
|
||||
addLookupHeaders = "SWW_ADD_LOOKUP_HEADERS"
|
||||
deniedCountries = "SWW_DENIED_COUNTRIES"
|
||||
allowedCountries = "SWW_EXCLUSIVELY_ALLOWED_COUNTRIES"
|
||||
asnLimitPercent = "SWW_ASN_LIMIT_PERCENT"
|
||||
countryLimitPercent = "SWW_COUNTRY_LIMIT_PERCENT"
|
||||
tagLists = "SWW_TAG_LISTS"
|
||||
tagDeny = "SWW_TAG_DENY"
|
||||
tagLimitPercent = "SWW_TAG_LIMIT_PERCENT"
|
||||
asnBytesPercent = "SWW_ASN_BYTES_PERCENT"
|
||||
countryBytesPercent = "SWW_COUNTRY_BYTES_PERCENT"
|
||||
unknownLimitPercent = "SWW_UNKNOWN_LIMIT_PERCENT"
|
||||
@@ -61,6 +61,7 @@ const (
|
||||
reputationAction = "SWW_REPUTATION_ACTION"
|
||||
reputationCacheTTL = "SWW_REPUTATION_CACHE_TTL"
|
||||
reputationTimeout = "SWW_REPUTATION_TIMEOUT"
|
||||
routeWatchURL = "SWW_ROUTEWATCH_URL"
|
||||
crowdSecURL = "SWW_CROWDSEC_LAPI_URL"
|
||||
crowdSecKey = "SWW_CROWDSEC_LAPI_KEY"
|
||||
banResponse = "SWW_BAN_RESPONSE"
|
||||
@@ -88,6 +89,7 @@ const (
|
||||
wafExemptPaths = "SWW_WAF_EXEMPT_PATHS"
|
||||
wafBodyLimit = "SWW_WAF_BODY_LIMIT"
|
||||
exclusionsFile = "SWW_CORE_RULE_SET_EXCLUSIONS_FILE"
|
||||
apps = "SWW_APPS"
|
||||
trapPaths = "SWW_TRAP_PATHS"
|
||||
errorBurstThreshold = "SWW_ERROR_BURST_THRESHOLD"
|
||||
logRemoteURL = "SWW_LOG_REMOTE_URL"
|
||||
@@ -159,7 +161,7 @@ func loggedAnomalyDefaults() map[string]string {
|
||||
// defaultAlertEvents is the default of SWW_ALERT_EVENTS, and
|
||||
// defaultAlertCooldown that of SWW_ALERT_COOLDOWN.
|
||||
const (
|
||||
defaultAlertEvents = "ban,permanent_ban,waf_block,anomaly,reputation_hit," +
|
||||
defaultAlertEvents = "ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit," +
|
||||
"source_failure,file_error"
|
||||
defaultAlertCooldown = "15m"
|
||||
)
|
||||
@@ -208,6 +210,9 @@ const off = "off"
|
||||
// enabled is true, as a setting's value.
|
||||
const enabled = "true"
|
||||
|
||||
// disabled is false, as a setting's value.
|
||||
const disabled = "false"
|
||||
|
||||
// defaultLookupSource is the default of SWW_LOOKUP_SOURCE, and
|
||||
// fileSource the source that is the lookup database.
|
||||
const (
|
||||
@@ -274,9 +279,13 @@ func TestDefaults(t *testing.T) {
|
||||
wantNetblocks(t, cfg.AllowNets)
|
||||
wantNetblocks(t, cfg.RateLimitExemptNets)
|
||||
wantNetblocks(t, cfg.DenyNets)
|
||||
wantCountries(t, deniedCountries, cfg.DeniedCountries)
|
||||
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries)
|
||||
|
||||
if len(cfg.TagLists) != 0 || len(cfg.TagDeny) != 0 || len(cfg.TagLimitPercent) != 0 {
|
||||
t.Errorf("tag settings %v, %v and %v by default, want all empty",
|
||||
cfg.TagLists, cfg.TagDeny, cfg.TagLimitPercent)
|
||||
}
|
||||
|
||||
hostname, err := os.Hostname()
|
||||
if err != nil || hostname == "" || cfg.InstanceName != hostname {
|
||||
t.Errorf("%s is %q, want the host's name %q (%v)", instanceName,
|
||||
@@ -327,7 +336,6 @@ func TestValuesAsSet(t *testing.T) {
|
||||
rateLimitPerMinute: "60",
|
||||
rateLimitPerHour: "600",
|
||||
rateLimitPerDay: "6000",
|
||||
deniedCountries: "cn, RU,kp,Xk",
|
||||
allowedCountries: "de",
|
||||
banResponse: "429",
|
||||
limitBanDuration: "15m",
|
||||
@@ -342,7 +350,7 @@ func TestValuesAsSet(t *testing.T) {
|
||||
metricsToken: token,
|
||||
metricsTopN: "10",
|
||||
rulesDir: "/srv/waf-rules",
|
||||
rulesEnabled: "false",
|
||||
rulesEnabled: disabled,
|
||||
})
|
||||
|
||||
wantSettings(t, cfg, config.Config{
|
||||
@@ -370,7 +378,6 @@ func TestValuesAsSet(t *testing.T) {
|
||||
wantNetblocks(t, cfg.AllowNets, "192.0.2.7/32")
|
||||
wantNetblocks(t, cfg.RateLimitExemptNets, "2001:db8::/48", "10.9.8.7/32")
|
||||
wantNetblocks(t, cfg.DenyNets, "198.51.100.0/24")
|
||||
wantCountries(t, deniedCountries, cfg.DeniedCountries, "CN", "RU", "KP", "XK")
|
||||
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
|
||||
}
|
||||
|
||||
@@ -576,6 +583,92 @@ func TestCoreRuleSetSettings(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestApps(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
env environment
|
||||
want map[string][]string
|
||||
}{
|
||||
{environment{}, nil},
|
||||
{environment{apps: ""}, map[string][]string{}},
|
||||
{
|
||||
environment{apps: "Git.Example.com:gitea; cloud.example.com : gitea "},
|
||||
map[string][]string{
|
||||
"git.example.com": {"gitea"}, "cloud.example.com": {"gitea"},
|
||||
},
|
||||
},
|
||||
} {
|
||||
cfg := fromEnvironment(t, tc.env)
|
||||
if !reflect.DeepEqual(cfg.Apps, tc.want) {
|
||||
t.Errorf("%v gave %#v, want %#v", tc.env, cfg.Apps, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Unset and set empty, SWW_APPS has opposite effects, so it logs differently.
|
||||
func TestUnsetAndEmptyAppsAreLoggedDifferently(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
env environment
|
||||
want string
|
||||
}{
|
||||
{environment{}, `"` + apps + `":null`},
|
||||
{environment{apps: ""}, `"` + apps + `":""`},
|
||||
} {
|
||||
var out bytes.Buffer
|
||||
|
||||
slog.New(slog.NewJSONHandler(&out, nil)).
|
||||
Info("starting", "settings", fromEnvironment(t, tc.env))
|
||||
|
||||
if !strings.Contains(out.String(), tc.want) {
|
||||
t.Errorf("%v logged %s, want %s in it", tc.env, out.String(), tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestInvalidAppsStopsTheStart(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
notEntry = " is not a hostname, : and a list of apps, " +
|
||||
"such as git.example.com:gitea"
|
||||
notHostname = " is not a DNS name such as git.example.com"
|
||||
)
|
||||
|
||||
for _, tc := range []struct{ value, want string }{
|
||||
{"git.example.com:gitea;blog.example.com", `"blog.example.com"` + notEntry},
|
||||
{"git.example.com:gitea;", `""` + notEntry},
|
||||
{":gitea", `":gitea"` + notEntry},
|
||||
{"git.example.com:", `"git.example.com:"` + notEntry},
|
||||
{"*.example.com:gitea", `"*.example.com"` + notHostname},
|
||||
{"git.example.com/:gitea", `"git.example.com/"` + notHostname},
|
||||
{"git example.com:gitea", `"git example.com"` + notHostname},
|
||||
{
|
||||
"git.example.com:gitea,nextcloud",
|
||||
`"nextcloud" is not an app sww has fixes for; the apps are gitea`,
|
||||
},
|
||||
{
|
||||
"git.example.com:gitea;GIT.example.com:gitea",
|
||||
`"git.example.com" is listed twice`,
|
||||
},
|
||||
{"git.example.com:gitea,gitea", `"gitea" is listed twice`},
|
||||
{"git.example.com:gitea,", `"gitea," has an empty item in its list`},
|
||||
} {
|
||||
t.Run(tc.value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(environment{apps: tc.value}.lookupEnv)
|
||||
|
||||
want := apps + ": " + tc.want
|
||||
if err == nil || err.Error() != want {
|
||||
t.Errorf("error %v, want %s", err, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestWAFBodyLimitOf64K(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -1097,15 +1190,15 @@ func TestInstanceNameNotUTF8StopsTheStart(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
|
||||
func TestCodeInAllowedCountriesAndTagDenyStopsTheStart(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(environment{
|
||||
deniedCountries: "cn,ru",
|
||||
tagDeny: "country=cn,country=ru",
|
||||
allowedCountries: "de,RU",
|
||||
}.lookupEnv)
|
||||
if err == nil {
|
||||
t.Fatal("ru on both country lists was accepted")
|
||||
t.Fatal("ru in SWW_EXCLUSIVELY_ALLOWED_COUNTRIES and SWW_TAG_DENY was accepted")
|
||||
}
|
||||
|
||||
if !strings.HasPrefix(err.Error(), allowedCountries+": ") ||
|
||||
@@ -1176,12 +1269,11 @@ func TestSettingNeedingLookupsStopsTheStartWhileTheyAreOff(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for name, value := range map[string]string{
|
||||
deniedCountries: "kp",
|
||||
allowedCountries: "de",
|
||||
addLookupHeaders: enabled,
|
||||
tagDeny: "country=KP",
|
||||
tagLimitPercent: "asn=64496:50",
|
||||
asnLimitPercentURL: asnURL,
|
||||
asnLimitPercent: "AS64496:50",
|
||||
countryLimitPercent: "cn:25",
|
||||
asnBytesPercent: "AS64496:50",
|
||||
countryBytesPercent: "cn:25",
|
||||
unknownLimitPercent: "99",
|
||||
@@ -1204,12 +1296,13 @@ func TestSettingNeedingLookupsStopsTheStartWhileTheyAreOff(t *testing.T) {
|
||||
}
|
||||
|
||||
// Set empty, the lists need nothing looked up, and nor does
|
||||
// SWW_UNKNOWN_LIMIT_PERCENT at 100, which lowers no limit, an anomaly
|
||||
// threshold per AS number that is off, any other anomaly threshold, or
|
||||
// a blocklist.
|
||||
// SWW_UNKNOWN_LIMIT_PERCENT at 100, which lowers no limit, a tag the
|
||||
// lookup does not give, an anomaly threshold per AS number that is off,
|
||||
// any other anomaly threshold, or a blocklist.
|
||||
env := environment{
|
||||
lookupSource: off, deniedCountries: "", allowedCountries: "",
|
||||
asnLimitPercent: "", countryLimitPercent: "", asnBytesPercent: "",
|
||||
lookupSource: off, allowedCountries: "", asnBytesPercent: "",
|
||||
tagLists: "torexit=https://example.com/tor.txt,vpn=https://example.com/vpn.txt",
|
||||
tagDeny: torExitTag, tagLimitPercent: "vpn:50",
|
||||
countryBytesPercent: "", unknownLimitPercent: "100", asnLimitPercentURL: "",
|
||||
blocklistURLs: dropURL,
|
||||
}
|
||||
@@ -1355,34 +1448,251 @@ func TestInvalidAnomalySettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// torExitTag, zenTag and exploitTag are tags these tests name.
|
||||
const (
|
||||
torExitTag = "torexit"
|
||||
zenTag = "dnsbl=zen.example"
|
||||
exploitTag = "exploitsignature"
|
||||
)
|
||||
|
||||
func TestTagSettingsAsSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const torURL = "https://example.com/tor.txt"
|
||||
|
||||
// Names and tags in either case, an AS number with leading zeros, and a
|
||||
// name holding a =.
|
||||
cfg := fromEnvironment(t, environment{
|
||||
tagLists: "TorExit=" + torURL +
|
||||
", vpn=mullvad=https://example.com/mullvad.txt",
|
||||
dnsblZones: "Zen.example",
|
||||
tagDeny: "TorExit, Country=kp, asn=014061",
|
||||
tagLimitPercent: "vpn=mullvad :50, asn=014061:25, COUNTRY=cn:1, dnsbl=zen.example:99",
|
||||
})
|
||||
|
||||
wantLists := []config.TagList{
|
||||
{Name: torExitTag, URL: torURL},
|
||||
{Name: "vpn=mullvad", URL: "https://example.com/mullvad.txt"},
|
||||
}
|
||||
if !slices.Equal(cfg.TagLists, wantLists) {
|
||||
t.Errorf("%s gave %+v, want %+v", tagLists, cfg.TagLists, wantLists)
|
||||
}
|
||||
|
||||
wantDeny := []string{torExitTag, "country=KP", "asn=14061"}
|
||||
if !slices.Equal(cfg.TagDeny, wantDeny) {
|
||||
t.Errorf("%s gave %v, want %v", tagDeny, cfg.TagDeny, wantDeny)
|
||||
}
|
||||
|
||||
wantPercents := map[string]int64{
|
||||
"vpn=mullvad": 50, "asn=14061": 25, "country=CN": 1, zenTag: 99,
|
||||
}
|
||||
if !maps.Equal(cfg.TagLimitPercent, wantPercents) {
|
||||
t.Errorf("%s gave %v, want %v", tagLimitPercent, cfg.TagLimitPercent, wantPercents)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRouteWatchURLAndPrefixTags(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const rw = "https://routewatch.example"
|
||||
|
||||
if cfg := fromEnvironment(t, environment{}); cfg.RouteWatchURL != "" {
|
||||
t.Errorf("default %s is %q, want unset", routeWatchURL, cfg.RouteWatchURL)
|
||||
}
|
||||
|
||||
cfg := fromEnvironment(t, environment{
|
||||
routeWatchURL: rw, tagDeny: "prefix=2001:DB8::/32",
|
||||
tagLimitPercent: "prefix=192.0.2.0/24:50",
|
||||
})
|
||||
wantDeny := []string{"prefix=2001:db8::/32"}
|
||||
|
||||
if cfg.RouteWatchURL != rw || !slices.Equal(cfg.TagDeny, wantDeny) ||
|
||||
cfg.TagLimitPercent["prefix=192.0.2.0/24"] != 50 {
|
||||
t.Errorf("gave %q, %v, %v", cfg.RouteWatchURL, cfg.TagDeny, cfg.TagLimitPercent)
|
||||
}
|
||||
|
||||
// routewatch gives asn= tags too, so an asn= tag needs no lookups while it is set.
|
||||
fromEnvironment(t, environment{
|
||||
lookupSource: off, routeWatchURL: rw, tagDeny: "asn=64496",
|
||||
})
|
||||
|
||||
for _, stopped := range []struct {
|
||||
env environment
|
||||
name string
|
||||
}{
|
||||
{environment{routeWatchURL: "ftp://routewatch.example"}, routeWatchURL},
|
||||
{environment{routeWatchURL: "routewatch.example"}, routeWatchURL},
|
||||
{environment{tagDeny: "prefix=192.0.2.0/24"}, tagDeny},
|
||||
{environment{routeWatchURL: rw, tagDeny: "prefix=nowhere"}, tagDeny},
|
||||
{environment{routeWatchURL: rw, tagDeny: "prefix=192.0.2.5/24"}, tagDeny},
|
||||
{
|
||||
environment{routeWatchURL: rw, tagLimitPercent: "prefix=192.0.2.0/33:50"},
|
||||
tagLimitPercent,
|
||||
},
|
||||
{environment{lookupSource: off, routeWatchURL: rw, tagDeny: "country=DE"}, tagDeny},
|
||||
{environment{lookupSource: off, tagDeny: "asn=64496"}, tagDeny},
|
||||
} {
|
||||
_, err := config.FromEnvironment(stopped.env.lookupEnv)
|
||||
if err == nil || !strings.HasPrefix(err.Error(), stopped.name+": ") {
|
||||
t.Errorf("%v gave the error %v, want one naming %s", stopped.env, err, stopped.name)
|
||||
}
|
||||
}
|
||||
|
||||
badURL := environment{routeWatchURL: "ftp://routewatch.example"}
|
||||
_, err := config.FromEnvironment(badURL.lookupEnv)
|
||||
|
||||
want := routeWatchURL + `: "ftp://routewatch.example" is not an http or https URL ` +
|
||||
"without a user or a fragment, such as https://routewatch.example"
|
||||
if err == nil || err.Error() != want {
|
||||
t.Errorf("error %v, want %s", err, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagThatNoRequestCanGetIsToldApartFromOneItCan(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
list = "torexit=https://example.com/tor.txt"
|
||||
key = "abc123"
|
||||
zone = key + ".xbl.dq.spamhaus.net"
|
||||
masked = "********.xbl.dq.spamhaus.net"
|
||||
)
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
env environment
|
||||
want string // "" is accepted
|
||||
}{
|
||||
{"a list's name", environment{tagLists: list, tagDeny: torExitTag}, ""},
|
||||
{
|
||||
"a mistyped list's name", environment{tagLists: list, tagDeny: "torexti"},
|
||||
`"torexti"`,
|
||||
},
|
||||
{"a masked zone", environment{dnsblZones: zone, tagDeny: "dnsbl=" + masked}, ""},
|
||||
{
|
||||
"a zone with its key", environment{dnsblZones: zone, tagDeny: "dnsbl=" + zone},
|
||||
masked,
|
||||
},
|
||||
{"another zone", environment{dnsblZones: zone, tagDeny: zenTag}, zenTag},
|
||||
{
|
||||
"abuseipdb with a key",
|
||||
environment{abuseIPDBKey: "k", tagLimitPercent: "abuseipdb:50"}, "",
|
||||
},
|
||||
{"crowdsec, though a list has its name", environment{
|
||||
tagLists: "crowdsec=https://example.com/c.txt", tagDeny: "crowdsec",
|
||||
}, `"crowdsec" cannot be used`},
|
||||
{exploitTag, environment{tagDeny: exploitTag}, ""},
|
||||
{
|
||||
exploitTag + " with rule files off",
|
||||
environment{tagDeny: exploitTag, rulesEnabled: disabled}, "",
|
||||
},
|
||||
{
|
||||
exploitTag + " with the Core Rule Set off",
|
||||
environment{tagDeny: exploitTag, wafMode: off}, "",
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(tc.env.lookupEnv)
|
||||
|
||||
switch {
|
||||
case tc.want == "" && err != nil:
|
||||
t.Errorf("error %v, want none", err)
|
||||
case tc.want != "" && (err == nil || !strings.Contains(err.Error(), tc.want)):
|
||||
t.Errorf("error %v, want one holding %s", err, tc.want)
|
||||
case err != nil && strings.Contains(err.Error(), key):
|
||||
t.Errorf("error %v shows the zone's key", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestErrorOfATagSettingLeavesOutAZoneKey(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
tag = "dnsbl=" + spamhaus
|
||||
emptyItem = "has an empty item in its list"
|
||||
twice = `"` + spamhausMasked + `" is listed twice`
|
||||
)
|
||||
|
||||
for _, tc := range []struct{ name, value, want string }{
|
||||
{tagDeny, tag + "," + tag, twice},
|
||||
{tagDeny, tag + ",,", emptyItem},
|
||||
{
|
||||
tagLimitPercent, tag,
|
||||
`"` + spamhausMasked + `" is not a tag, : and a percentage, ` +
|
||||
"such as vpn:50 or asn=14061:25",
|
||||
},
|
||||
{
|
||||
tagLimitPercent, "50:" + tag,
|
||||
`"` + spamhausMasked + `" is not a percentage from 1 to 99`,
|
||||
},
|
||||
{tagLimitPercent, tag + ":50," + tag + " :25", twice},
|
||||
{tagLimitPercent, tag + ":50,,", emptyItem},
|
||||
} {
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
|
||||
|
||||
want := tc.name + ": " + tc.want
|
||||
if err == nil || err.Error() != want {
|
||||
t.Errorf("%s=%s gave the error %v, want %s", tc.name, tc.value, err, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagListURLFetchedAsAnotherListStopsTheStart(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const lapi = "http://172.17.0.1:8080"
|
||||
|
||||
for _, tc := range []struct {
|
||||
name, listURL string
|
||||
env environment
|
||||
}{
|
||||
{"a blocklist", dropURL, environment{blocklistURLs: dropURL}},
|
||||
{"the file of AS:percent lines", asnURL, environment{asnLimitPercentURL: asnURL}},
|
||||
{
|
||||
"the CrowdSec decision list", lapi + "/v1/decisions",
|
||||
environment{"SWW_CROWDSEC_LAPI_URL": lapi, "SWW_CROWDSEC_LAPI_KEY": "key"},
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tc.env[tagLists] = "dup=" + tc.listURL
|
||||
|
||||
want := tagLists + `: "` + tc.listURL + `" is in SWW_BLOCKLIST_URLS, is ` +
|
||||
"SWW_ASN_LIMIT_PERCENT_URL or is the CrowdSec decision list too"
|
||||
|
||||
_, err := config.FromEnvironment(tc.env.lookupEnv)
|
||||
if err == nil || err.Error() != want {
|
||||
t.Errorf("error %v, want %s", err, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestBiasedThresholdsAsSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := fromEnvironment(t, environment{})
|
||||
if len(cfg.ASNLimitPercent) != 0 || len(cfg.CountryLimitPercent) != 0 ||
|
||||
len(cfg.ASNBytesPercent) != 0 || len(cfg.CountryBytesPercent) != 0 ||
|
||||
if len(cfg.ASNBytesPercent) != 0 || len(cfg.CountryBytesPercent) != 0 ||
|
||||
cfg.UnknownLimitPercent != 100 {
|
||||
t.Errorf("biased thresholds %v, %v, %v, %v and %d by default, "+
|
||||
"want four empty lists and 100", cfg.ASNLimitPercent, cfg.CountryLimitPercent,
|
||||
t.Errorf("biased thresholds %v, %v and %d by default, "+
|
||||
"want two empty lists and 100",
|
||||
cfg.ASNBytesPercent, cfg.CountryBytesPercent, cfg.UnknownLimitPercent)
|
||||
}
|
||||
|
||||
// AS numbers and countries in either case, an AS number with leading
|
||||
// zeros, 0 and 100.
|
||||
cfg = fromEnvironment(t, environment{
|
||||
asnLimitPercent: "AS14061:50, as16276:0,AS045102:100",
|
||||
countryLimitPercent: "cn:25,RU:50",
|
||||
asnBytesPercent: "as16276:75",
|
||||
countryBytesPercent: "ru:10",
|
||||
unknownLimitPercent: "0",
|
||||
})
|
||||
|
||||
for name, tc := range map[string]struct{ got, want map[string]int64 }{
|
||||
asnLimitPercent: {
|
||||
cfg.ASNLimitPercent,
|
||||
map[string]int64{"AS14061": 50, "AS16276": 0, "AS45102": 100},
|
||||
},
|
||||
countryLimitPercent: {cfg.CountryLimitPercent, map[string]int64{"CN": 25, "RU": 50}},
|
||||
asnBytesPercent: {cfg.ASNBytesPercent, map[string]int64{"AS16276": 75}},
|
||||
countryBytesPercent: {cfg.CountryBytesPercent, map[string]int64{"RU": 10}},
|
||||
} {
|
||||
@@ -1406,16 +1716,16 @@ func TestInvalidBiasedThresholdStopsTheStartSayingWhatIsWrong(t *testing.T) {
|
||||
)
|
||||
|
||||
for _, tc := range []struct{ name, value, want string }{
|
||||
{asnLimitPercent, "14061:50", `"14061"` + notASN},
|
||||
{asnLimitPercent, "AS4294967296:50", `"AS4294967296"` + notASN},
|
||||
{asnLimitPercent, "AS14061", `"AS14061"` + notItem},
|
||||
{asnLimitPercent, "AS14061:101", `"101"` + notPercent},
|
||||
{asnLimitPercent, "AS14061:50,as14061:25", `"as14061" is listed twice`},
|
||||
{asnBytesPercent, "14061:50", `"14061"` + notASN},
|
||||
{asnBytesPercent, "AS4294967296:50", `"AS4294967296"` + notASN},
|
||||
{asnBytesPercent, "AS14061", `"AS14061"` + notItem},
|
||||
{asnBytesPercent, "AS14061:101", `"101"` + notPercent},
|
||||
{asnBytesPercent, "AS14061:50,as14061:25", `"as14061" is listed twice`},
|
||||
{
|
||||
countryLimitPercent, "nk:25",
|
||||
countryBytesPercent, "nk:25",
|
||||
`"nk" is not a two-letter country code such as de or kp`,
|
||||
},
|
||||
{countryLimitPercent, "cn:25,CN:50", `"CN" is listed twice`},
|
||||
{countryBytesPercent, "cn:25,CN:50", `"CN" is listed twice`},
|
||||
{asnBytesPercent, "AS14061:-1", `"-1"` + notPercent},
|
||||
{countryBytesPercent, "cn:50%", `"50%"` + notPercent},
|
||||
{unknownLimitPercent, "101", `"101"` + notPercent},
|
||||
@@ -2003,23 +2313,37 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
|
||||
{lookupSource, "ipinfo"}, {lookupSource, "GeoJS"}, {lookupSource, ""},
|
||||
{lookupTimeout, off}, {lookupTimeout, "0s"}, {lookupTimeout, "1"},
|
||||
{addLookupHeaders, "yes"},
|
||||
{deniedCountries, "nk"},
|
||||
{deniedCountries, "kp,,ir"},
|
||||
{deniedCountries, "prk"},
|
||||
{deniedCountries, "408"},
|
||||
{deniedCountries, "k"},
|
||||
{deniedCountries, "eu"},
|
||||
{deniedCountries, "un"},
|
||||
{deniedCountries, "su"},
|
||||
{allowedCountries, "nk"},
|
||||
{allowedCountries, "kp,,ir"},
|
||||
{allowedCountries, "prk"},
|
||||
{allowedCountries, "408"},
|
||||
{allowedCountries, "k"},
|
||||
{allowedCountries, "eu"},
|
||||
{allowedCountries, "un"},
|
||||
{allowedCountries, "su"},
|
||||
{allowedCountries, "ac"},
|
||||
{allowedCountries, "uk"},
|
||||
{allowedCountries, "zz"},
|
||||
{allowedCountries, "de,germany"},
|
||||
{asnLimitPercent, "AS14061:50,,AS16276:50"}, {asnLimitPercent, "ASX:50"},
|
||||
{asnLimitPercent, "AS14061:"}, {asnLimitPercent, "AS14061 :50"},
|
||||
{asnLimitPercent, "AS14061:1.5"}, {asnLimitPercent, "AS-1:50"},
|
||||
{countryLimitPercent, "cn"}, {countryLimitPercent, "cn:"},
|
||||
{countryLimitPercent, "cn:25:50"}, {countryLimitPercent, "china:25"},
|
||||
{asnBytesPercent, "AS14061:50,,AS16276:50"}, {asnBytesPercent, "ASX:50"},
|
||||
{asnBytesPercent, "AS14061:"}, {asnBytesPercent, "AS14061 :50"},
|
||||
{asnBytesPercent, "AS14061:1.5"}, {asnBytesPercent, "AS-1:50"},
|
||||
{countryBytesPercent, "cn"}, {countryBytesPercent, "cn:"},
|
||||
{countryBytesPercent, "cn:25:50"}, {countryBytesPercent, "china:25"},
|
||||
{tagDeny, "a,,b"}, {tagDeny, "country=nk"}, {tagDeny, "asn=abc"},
|
||||
{tagDeny, "asn="}, {tagDeny, "torexit,TOREXIT"}, {tagDeny, "torexti"},
|
||||
{tagDeny, zenTag}, {tagDeny, "abuseipdb"}, {tagDeny, "crowdsec"},
|
||||
{tagLimitPercent, "torexti:50"}, {tagLimitPercent, "dnsbl=zen.example:50"},
|
||||
{tagLimitPercent, "abuseipdb:50"}, {tagLimitPercent, "crowdsec:50"},
|
||||
{tagLimitPercent, "vpn :50"}, {tagLimitPercent, "vpn"}, {tagLimitPercent, ":50"},
|
||||
{tagLimitPercent, "vpn:"}, {tagLimitPercent, "vpn:0"}, {tagLimitPercent, "vpn:100"},
|
||||
{tagLimitPercent, "vpn:1.5"}, {tagLimitPercent, "vpn:50,VPN:25"},
|
||||
{tagLimitPercent, "exploitsignature:50"}, {tagLimitPercent, "asn=abc:50"},
|
||||
{tagLists, torExitTag}, {tagLists, "=https://example.com/a.txt"},
|
||||
{tagLists, "a,b=https://example.com/a.txt"},
|
||||
{tagLists, "a=ftp://example.com/a.txt"},
|
||||
{tagLists, "a=https://example.com/a.txt,A=https://example.com/b.txt"},
|
||||
{tagLists, "a=https://example.com/a.txt,b=https://example.com/a.txt"},
|
||||
{asnBytesPercent, "AS14061:101"}, {countryBytesPercent, "su:50"},
|
||||
{unknownLimitPercent, ""}, {unknownLimitPercent, "-1"},
|
||||
{unknownLimitPercent, "50%"},
|
||||
@@ -2031,6 +2355,25 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
|
||||
{logLevel, "INFO"}, {logLevel, "warning"}, {logLevel, "trace"}, {logLevel, ""},
|
||||
{rulesEnabled, "yes"}, {rulesEnabled, "True"},
|
||||
})
|
||||
|
||||
wantExploitTagStopsTheStartWithRulesAndCoreRuleSetOff(t)
|
||||
}
|
||||
|
||||
// wantExploitTagStopsTheStartWithRulesAndCoreRuleSetOff checks that with rule
|
||||
// files off, and either setting turning off the Core Rule Set, the tag
|
||||
// exploitsignature in SWW_TAG_DENY stops the start.
|
||||
func wantExploitTagStopsTheStartWithRulesAndCoreRuleSetOff(t *testing.T) {
|
||||
t.Helper()
|
||||
|
||||
for _, name := range []string{wafMode, wafAnomalyThreshold} {
|
||||
_, err := config.FromEnvironment(environment{
|
||||
tagDeny: exploitTag, rulesEnabled: disabled, name: off,
|
||||
}.lookupEnv)
|
||||
if err == nil || !strings.HasPrefix(err.Error(), tagDeny+": ") {
|
||||
t.Errorf("%s=off with rule files off gave the error %v, want one naming %s",
|
||||
name, err, tagDeny)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestInvalidBanOrStateValueStopsTheStart(t *testing.T) {
|
||||
@@ -2285,11 +2628,11 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
lookupSource: defaultLookupSource,
|
||||
lookupDBPath: "",
|
||||
lookupTimeout: "800ms",
|
||||
addLookupHeaders: "false",
|
||||
deniedCountries: "",
|
||||
addLookupHeaders: disabled,
|
||||
allowedCountries: "",
|
||||
asnLimitPercent: "",
|
||||
countryLimitPercent: "",
|
||||
tagLists: "",
|
||||
tagDeny: "",
|
||||
tagLimitPercent: "",
|
||||
asnBytesPercent: "",
|
||||
countryBytesPercent: "",
|
||||
unknownLimitPercent: "100",
|
||||
@@ -2305,6 +2648,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
reputationAction: "limit:25",
|
||||
reputationCacheTTL: defaultReputationCacheTTL,
|
||||
reputationTimeout: "2s",
|
||||
routeWatchURL: "",
|
||||
crowdSecURL: "",
|
||||
crowdSecKey: "",
|
||||
banResponse: "403",
|
||||
@@ -2332,6 +2676,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
wafExemptPaths: "",
|
||||
wafBodyLimit: off,
|
||||
exclusionsFile: "",
|
||||
apps: "", // logged as null, which decodes as ""
|
||||
trapPaths: "",
|
||||
errorBurstThreshold: "30",
|
||||
logRemoteURL: "",
|
||||
|
||||
@@ -1,21 +1,23 @@
|
||||
// Package waf runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
|
||||
// Package coreruleset runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
|
||||
// method, the URL with its query and the headers of a request, and on its
|
||||
// body while SWW_WAF_BODY_LIMIT is set, with the eight changes sww
|
||||
// makes to it and the exclusions in SWW_CORE_RULE_SET_EXCLUSIONS_FILE, as
|
||||
// "Attack detection" under "Settings" in SPEC.md describes them. It reads no
|
||||
// response.
|
||||
// body while SWW_WAF_BODY_LIMIT is set, with the six changes sww
|
||||
// makes to it, the fixes for the apps SWW_APPS names, and the exclusions in
|
||||
// SWW_CORE_RULE_SET_EXCLUSIONS_FILE, as "Attack detection" under "Settings"
|
||||
// in SPEC.md describes them. It reads no response.
|
||||
//
|
||||
// sww writes only to its state directory, so Coraza is built with
|
||||
// its no_fs_access tag, as the Dockerfile and script/build build it: of a
|
||||
// file in a multipart body, Coraza then counts the bytes instead of
|
||||
// writing them to the system's temporary directory.
|
||||
package waf
|
||||
package coreruleset
|
||||
|
||||
import (
|
||||
_ "embed"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"maps"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"os"
|
||||
@@ -32,8 +34,8 @@ import (
|
||||
|
||||
// directives are the Core Rule Set as sww runs it, with the
|
||||
// paranoia level for %d, bodyDirectives for the first %s while
|
||||
// SWW_WAF_BODY_LIMIT is set, the exclusions file for the second, and
|
||||
// giteaExclusions for the third.
|
||||
// SWW_WAF_BODY_LIMIT is set, the exclusions file for the second, and the
|
||||
// directives of every app in apps for the third.
|
||||
// Each rule sww adds has an id from 900000 to 900999, the ids the
|
||||
// Core Rule Set keeps for the rules that set it up, which
|
||||
// SWW_WAF_DISABLED_RULES and the exclusions file refuse, so that no
|
||||
@@ -81,44 +83,28 @@ SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
|
||||
# before the rules of phase 1 too.
|
||||
%s
|
||||
|
||||
# The eighth change comes before the rules for the same reason.
|
||||
# The apps' fixes come before the rules for the same reason.
|
||||
%s
|
||||
|
||||
# The sixth: only the rules for requests are loaded, and no response is
|
||||
# The fifth: only the rules for requests are loaded, and no response is
|
||||
# inspected.
|
||||
Include @owasp_crs/REQUEST-*.conf
|
||||
|
||||
# The third: redirect_uri is not checked for a URL naming an IP address or
|
||||
# localhost. Coraza matches a parameter name here, and in the fourth,
|
||||
# without regard to case. ARGS holds the fields of a form data or multipart
|
||||
# body Coraza reads as well as the query parameters, so a field of one of
|
||||
# these names is left out too.
|
||||
# localhost. Coraza matches a parameter name here without regard to case.
|
||||
# ARGS holds the fields of a form data or multipart body Coraza reads as
|
||||
# well as the query parameters, so a field of this name is left out too.
|
||||
SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri"
|
||||
SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri"
|
||||
|
||||
# The fourth: the query parameters in which gitea sends names within a
|
||||
# repository or its own records, or a page of its own site, are not
|
||||
# checked against the lists of system files, shell paths and command
|
||||
# names. Coraza takes one rule id per directive.
|
||||
SecRuleUpdateTargetById 930120 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
|
||||
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
|
||||
!ARGS:artifactName|!ARGS:redirect_to"
|
||||
SecRuleUpdateTargetById 932160 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
|
||||
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
|
||||
!ARGS:artifactName|!ARGS:redirect_to"
|
||||
SecRuleUpdateTargetById 932260 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
|
||||
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
|
||||
!ARGS:artifactName|!ARGS:redirect_to"
|
||||
|
||||
# The fifth, for Referer: it is not checked for a Unix command without
|
||||
# arguments, for Java starting a process, or for a Java class name. 944130
|
||||
# refused every page opened from the page of java.lang.String.md. The
|
||||
# cookies are left out in Inspect.
|
||||
# The fourth: Referer is not checked for a Unix command without arguments,
|
||||
# for Java starting a process, or for a Java class name. 944130 refused
|
||||
# every page opened from the page of java.lang.String.md.
|
||||
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
|
||||
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
|
||||
SecRuleUpdateTargetById 944130 "!REQUEST_HEADERS:Referer"
|
||||
|
||||
# The seventh: a container image push that reuses a layer names the image
|
||||
# The sixth: a container image push that reuses a layer names the image
|
||||
# it takes the layer from in the query parameter from, which is not
|
||||
# checked for command names. 932260 refused the push of a layer from
|
||||
# owner/python3-app.
|
||||
@@ -168,18 +154,36 @@ SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
|
||||
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
|
||||
`
|
||||
|
||||
// giteaExclusions are the eighth change: the fields of gitea's own
|
||||
// requests that skip the rules that refused them there, as Coraza
|
||||
// directives.
|
||||
//
|
||||
//go:embed gitea.conf
|
||||
var giteaExclusions string
|
||||
// app is an app whose ordinary requests the Core Rule Set would refuse,
|
||||
// with the fixes that let them pass.
|
||||
type app struct {
|
||||
// directives are Coraza exclusions, loaded before the Core Rule Set's
|
||||
// rules. Their first rule skips the rest unless Inspect sets the
|
||||
// transaction variable app_ and the app's name, and they end with the
|
||||
// marker it skips to.
|
||||
directives string
|
||||
// cookies are the cookies the Core Rule Set reads a request without.
|
||||
cookies []string
|
||||
}
|
||||
|
||||
// cookiesNotRead are the cookies the Core Rule Set reads a request
|
||||
// without, the rest of the fifth change.
|
||||
//go:embed gitea.conf
|
||||
var giteaDirectives string
|
||||
|
||||
// apps are the apps whose fixes sww has, by the name SWW_APPS gives each.
|
||||
//
|
||||
//nolint:gochecknoglobals // a constant cannot be a list
|
||||
var cookiesNotRead = []string{"gitea_flash", "redirect_to"}
|
||||
//nolint:gochecknoglobals // a constant cannot be a map
|
||||
var apps = map[string]app{
|
||||
"gitea": {
|
||||
directives: giteaDirectives,
|
||||
cookies: []string{"gitea_flash", "redirect_to"},
|
||||
},
|
||||
}
|
||||
|
||||
// AppNames returns the names of the apps whose fixes sww has, in
|
||||
// alphabetical order.
|
||||
func AppNames() []string {
|
||||
return slices.Sorted(maps.Keys(apps))
|
||||
}
|
||||
|
||||
// Params are what New needs.
|
||||
type Params struct {
|
||||
@@ -195,6 +199,10 @@ type Params struct {
|
||||
// Core Rule Set's rules (SWW_CORE_RULE_SET_EXCLUSIONS_FILE), "" while
|
||||
// there is none.
|
||||
ExclusionsFile string
|
||||
// Apps are the names of the apps each hostname serves, by hostname in
|
||||
// lowercase (SWW_APPS). A request gets the fixes of its hostname's
|
||||
// apps only, or of every app while Apps is nil.
|
||||
Apps map[string][]string
|
||||
}
|
||||
|
||||
// The ids of sww's own rules, which an exclusions file may not
|
||||
@@ -284,15 +292,17 @@ func checkExclusions(exclusions string) error {
|
||||
// CoreRuleSet is the Core Rule Set, ready to inspect requests. It is safe
|
||||
// for concurrent use.
|
||||
type CoreRuleSet struct {
|
||||
waf coraza.WAF
|
||||
engine coraza.WAF
|
||||
bodyLimit int64
|
||||
apps map[string][]string
|
||||
}
|
||||
|
||||
// New returns the Core Rule Set with the eight changes, at params'
|
||||
// paranoia level, without the rules it switches off, reading request
|
||||
// bodies up to params' limit, and with params' exclusions file loaded
|
||||
// before the Core Rule Set's rules. An exclusions file is an error if it
|
||||
// cannot be read or does not parse, or if checkExclusions refuses it.
|
||||
// New returns the Core Rule Set with the six changes and the apps' fixes,
|
||||
// at params' paranoia level, without the rules it switches off, reading
|
||||
// request bodies up to params' limit, and with params' exclusions file
|
||||
// loaded before the Core Rule Set's rules. An exclusions file is an error
|
||||
// if it cannot be read or does not parse, or if checkExclusions refuses
|
||||
// it.
|
||||
func New(params Params) (*CoreRuleSet, error) {
|
||||
body := ""
|
||||
if params.BodyLimit > 0 {
|
||||
@@ -315,8 +325,13 @@ func New(params Params) (*CoreRuleSet, error) {
|
||||
}
|
||||
}
|
||||
|
||||
var fixes strings.Builder
|
||||
for _, name := range AppNames() {
|
||||
fixes.WriteString(apps[name].directives + "\n")
|
||||
}
|
||||
|
||||
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
|
||||
giteaExclusions)
|
||||
fixes.String())
|
||||
|
||||
if len(params.DisabledRules) > 0 {
|
||||
ids := make([]string, len(params.DisabledRules))
|
||||
@@ -327,14 +342,16 @@ func New(params Params) (*CoreRuleSet, error) {
|
||||
text += "SecRuleRemoveById " + strings.Join(ids, " ") + "\n"
|
||||
}
|
||||
|
||||
waf, err := coraza.NewWAF(coraza.NewWAFConfig().
|
||||
engine, err := coraza.NewWAF(coraza.NewWAFConfig().
|
||||
WithRootFS(coreruleset.FS).
|
||||
WithDirectives(text))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load the Core Rule Set: %w", err)
|
||||
}
|
||||
|
||||
return &CoreRuleSet{waf: waf, bodyLimit: params.BodyLimit}, nil
|
||||
return &CoreRuleSet{
|
||||
engine: engine, bodyLimit: params.BodyLimit, apps: params.Apps,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Result is what the Core Rule Set found in a request.
|
||||
@@ -346,25 +363,37 @@ type Result struct {
|
||||
Score int
|
||||
}
|
||||
|
||||
// Inspect runs the Core Rule Set on r, a request from client: on its
|
||||
// method, its URL with the query, and its headers, the Cookie header
|
||||
// without the cookies in cookiesNotRead, and on body, r's body as the
|
||||
// caller has it, as readBody reads it. It returns what it found, what it
|
||||
// read of body, and the error that ended the reading early, if one did.
|
||||
// Inspect runs the Core Rule Set on r, a request from client, with the
|
||||
// fixes of the apps appsFor names for r's hostname: on its method, its URL
|
||||
// with the query, and its headers, the Cookie header without those apps'
|
||||
// cookies, and on body, r's body as the caller has it, as readBody reads
|
||||
// it. It returns what it found, what it read of body, and the error that
|
||||
// ended the reading early, if one did.
|
||||
func (c *CoreRuleSet) Inspect(
|
||||
r *http.Request, client netip.Addr, body io.Reader,
|
||||
) (Result, []byte, error) {
|
||||
tx := c.waf.NewTransaction()
|
||||
tx := c.engine.NewTransaction()
|
||||
// Closing would remove the files Coraza wrote, and it writes none.
|
||||
defer func() { _ = tx.Close() }()
|
||||
|
||||
// The transaction's variables are set only through Coraza's interface
|
||||
// for plugins.
|
||||
state := tx.(plugintypes.TransactionState) //nolint:forcetypeassert // every one is
|
||||
|
||||
var cookiesNotRead []string
|
||||
|
||||
for _, name := range c.appsFor(r.Host) {
|
||||
state.Variables().TX().Set("app_"+name, []string{"1"})
|
||||
cookiesNotRead = append(cookiesNotRead, apps[name].cookies...)
|
||||
}
|
||||
|
||||
tx.ProcessConnection(client.String(), 0, "", 0)
|
||||
tx.ProcessURI(r.URL.String(), r.Method, r.Proto)
|
||||
|
||||
for name, values := range r.Header {
|
||||
for _, value := range values {
|
||||
if name == "Cookie" {
|
||||
value = withoutCookiesNotRead(value)
|
||||
value = withoutCookies(value, cookiesNotRead)
|
||||
if value == "" {
|
||||
continue // it held those cookies alone
|
||||
}
|
||||
@@ -457,9 +486,26 @@ func score(tx types.Transaction) int {
|
||||
return n
|
||||
}
|
||||
|
||||
// withoutCookiesNotRead returns value, a Cookie header's, without the
|
||||
// cookies in cookiesNotRead.
|
||||
func withoutCookiesNotRead(value string) string {
|
||||
// appsFor returns the names of the apps whose fixes a request for host,
|
||||
// its Host header, gets: every app while SWW_APPS is unset, and otherwise
|
||||
// those SWW_APPS lists for host's name, without the port and in
|
||||
// lowercase.
|
||||
func (c *CoreRuleSet) appsFor(host string) []string {
|
||||
if c.apps == nil {
|
||||
return AppNames()
|
||||
}
|
||||
|
||||
hostname, _, err := net.SplitHostPort(host)
|
||||
if err != nil {
|
||||
hostname = host // it has no port
|
||||
}
|
||||
|
||||
return c.apps[strings.ToLower(hostname)]
|
||||
}
|
||||
|
||||
// withoutCookies returns value, a Cookie header's, without the cookies
|
||||
// named in cookiesNotRead.
|
||||
func withoutCookies(value string, cookiesNotRead []string) string {
|
||||
var kept []string
|
||||
|
||||
for cookie := range strings.SplitSeq(value, ";") {
|
||||
@@ -1,4 +1,4 @@
|
||||
package waf_test
|
||||
package coreruleset_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
@@ -12,7 +12,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
)
|
||||
|
||||
// defaultDisabledRules are the rules SWW_WAF_DISABLED_RULES switches off
|
||||
@@ -23,10 +23,12 @@ var defaultDisabledRules = []int{920340, 920420, 920440, 920640, 930130, 930140}
|
||||
|
||||
// newCoreRuleSet returns the Core Rule Set at paranoia level level, with
|
||||
// the rules in disabled switched off.
|
||||
func newCoreRuleSet(t *testing.T, level int, disabled ...int) *waf.CoreRuleSet {
|
||||
func newCoreRuleSet(t *testing.T, level int, disabled ...int) *coreruleset.CoreRuleSet {
|
||||
t.Helper()
|
||||
|
||||
crs, err := waf.New(waf.Params{ParanoiaLevel: level, DisabledRules: disabled})
|
||||
crs, err := coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: level, DisabledRules: disabled,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("load the Core Rule Set: %v", err)
|
||||
}
|
||||
@@ -49,7 +51,7 @@ func get(target string, headers ...string) request {
|
||||
|
||||
// inspect returns what crs finds in r, sent to git.example by a browser,
|
||||
// whose Host, User-Agent and Accept r.headers may replace.
|
||||
func inspect(t *testing.T, crs *waf.CoreRuleSet, r request) waf.Result {
|
||||
func inspect(t *testing.T, crs *coreruleset.CoreRuleSet, r request) coreruleset.Result {
|
||||
t.Helper()
|
||||
|
||||
result, _ := inspectBody(t, crs, r, "")
|
||||
@@ -60,8 +62,8 @@ func inspect(t *testing.T, crs *waf.CoreRuleSet, r request) waf.Result {
|
||||
// inspectBody is inspect for r with body, which is announced with its
|
||||
// Content-Length unless it is "", and returns what crs read of body too.
|
||||
func inspectBody(
|
||||
t *testing.T, crs *waf.CoreRuleSet, r request, body string,
|
||||
) (waf.Result, string) {
|
||||
t *testing.T, crs *coreruleset.CoreRuleSet, r request, body string,
|
||||
) (coreruleset.Result, string) {
|
||||
t.Helper()
|
||||
|
||||
req := httptest.NewRequestWithContext(t.Context(), r.method,
|
||||
@@ -96,7 +98,9 @@ func inspectBody(
|
||||
}
|
||||
|
||||
// wantResult checks what crs finds in r.
|
||||
func wantResult(t *testing.T, crs *waf.CoreRuleSet, r request, want waf.Result) {
|
||||
func wantResult(
|
||||
t *testing.T, crs *coreruleset.CoreRuleSet, r request, want coreruleset.Result,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
if got := inspect(t, crs, r); !reflect.DeepEqual(got, want) {
|
||||
@@ -106,28 +110,29 @@ func wantResult(t *testing.T, crs *waf.CoreRuleSet, r request, want waf.Result)
|
||||
|
||||
// matched is the result of a request that the rules ids match, each of
|
||||
// them a critical one, which adds 5 to the score.
|
||||
func matched(ids ...int) waf.Result {
|
||||
func matched(ids ...int) coreruleset.Result {
|
||||
const critical = 5
|
||||
|
||||
return waf.Result{RuleIDs: ids, Score: critical * len(ids)}
|
||||
return coreruleset.Result{RuleIDs: ids, Score: critical * len(ids)}
|
||||
}
|
||||
|
||||
// atDefaults returns the Core Rule Set as sww runs it by default.
|
||||
func atDefaults(t *testing.T) *waf.CoreRuleSet {
|
||||
func atDefaults(t *testing.T) *coreruleset.CoreRuleSet {
|
||||
t.Helper()
|
||||
|
||||
return newCoreRuleSet(t, 1, defaultDisabledRules...)
|
||||
}
|
||||
|
||||
// wantChange checks that crs lets through passes, a gitea request one of
|
||||
// the eight changes is for, and still finds result in refused, a request
|
||||
// like it that the change is not for.
|
||||
// the six changes or gitea's fixes are for, and still finds result in
|
||||
// refused, a request like it that they are not for.
|
||||
func wantChange(
|
||||
t *testing.T, crs *waf.CoreRuleSet, passes, refused request, result waf.Result,
|
||||
t *testing.T, crs *coreruleset.CoreRuleSet, passes, refused request,
|
||||
result coreruleset.Result,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
wantResult(t, crs, passes, waf.Result{})
|
||||
wantResult(t, crs, passes, coreruleset.Result{})
|
||||
wantResult(t, crs, refused, result)
|
||||
}
|
||||
|
||||
@@ -162,12 +167,12 @@ func TestExpectAndContentEncodingAreAllowed(t *testing.T) {
|
||||
|
||||
wantResult(t, crs,
|
||||
request{http.MethodPost, push, []string{pushType, length, "Expect: 100-continue"}},
|
||||
waf.Result{})
|
||||
coreruleset.Result{})
|
||||
wantResult(t, crs,
|
||||
request{http.MethodPost, fetch, []string{
|
||||
fetchType, length, "Content-Encoding: gzip",
|
||||
}},
|
||||
waf.Result{})
|
||||
coreruleset.Result{})
|
||||
|
||||
// Every other header on the Core Rule Set's list stays refused.
|
||||
for _, header := range []string{
|
||||
@@ -196,7 +201,7 @@ func TestTransferEncodingIsRead(t *testing.T) {
|
||||
"Content-Type: application/x-git-receive-pack-request",
|
||||
"Transfer-Encoding: chunked",
|
||||
}},
|
||||
waf.Result{})
|
||||
coreruleset.Result{})
|
||||
}
|
||||
|
||||
func TestMoreParametersThanCorazaKeepsIsAMatch(t *testing.T) {
|
||||
@@ -250,7 +255,7 @@ func TestParametersGiteaSendsNamesInSkipTheListsOfFilesPathsAndCommands(t *testi
|
||||
for _, value := range []struct {
|
||||
name string
|
||||
// result is what a parameter that is not one of gitea's gets.
|
||||
result waf.Result
|
||||
result coreruleset.Result
|
||||
}{
|
||||
// A file on the list of system files.
|
||||
{".gitignore", matched(930120)},
|
||||
@@ -271,7 +276,7 @@ func TestParametersGiteaSendsNamesInSkipTheListsOfFilesPathsAndCommands(t *testi
|
||||
wantChange(t, crs, get("/?path=|cat%20/etc/passwd"), get("/?q=|cat%20/etc/passwd"),
|
||||
matched(930120, 932160))
|
||||
wantResult(t, crs, get("/?path=../../etc/passwd"),
|
||||
waf.Result{RuleIDs: []int{930100, 930110}, Score: 20})
|
||||
coreruleset.Result{RuleIDs: []int{930100, 930110}, Score: 20})
|
||||
wantResult(t, crs, get("/?path=1'%20OR%20'1'='1"), matched(942100))
|
||||
}
|
||||
|
||||
@@ -341,7 +346,7 @@ func TestEmptyHeaderIsRead(t *testing.T) {
|
||||
|
||||
// An empty User-Agent is a notice, which adds 2.
|
||||
wantResult(t, atDefaults(t), get("/", "User-Agent: "),
|
||||
waf.Result{RuleIDs: []int{920330}, Score: 2})
|
||||
coreruleset.Result{RuleIDs: []int{920330}, Score: 2})
|
||||
}
|
||||
|
||||
func TestParanoiaLevel(t *testing.T) {
|
||||
@@ -350,7 +355,7 @@ func TestParanoiaLevel(t *testing.T) {
|
||||
// Accept-Charset is refused from paranoia level 2.
|
||||
r := get("/", "Accept-Charset: utf-8")
|
||||
|
||||
wantResult(t, newCoreRuleSet(t, 1), r, waf.Result{})
|
||||
wantResult(t, newCoreRuleSet(t, 1), r, coreruleset.Result{})
|
||||
wantResult(t, newCoreRuleSet(t, 2), r, matched(920451))
|
||||
}
|
||||
|
||||
@@ -362,8 +367,8 @@ func TestEachDisabledRuleIsSwitchedOff(t *testing.T) {
|
||||
r := request{http.MethodTrace, "/", []string{"Host: 192.0.2.1"}}
|
||||
|
||||
wantResult(t, newCoreRuleSet(t, 1), r,
|
||||
waf.Result{RuleIDs: []int{911100, 920350}, Score: 8})
|
||||
wantResult(t, newCoreRuleSet(t, 1, 920350, 911100), r, waf.Result{})
|
||||
coreruleset.Result{RuleIDs: []int{911100, 920350}, Score: 8})
|
||||
wantResult(t, newCoreRuleSet(t, 1, 920350, 911100), r, coreruleset.Result{})
|
||||
}
|
||||
|
||||
func TestImagePushThatReusesALayerIsNotCheckedForCommandNames(t *testing.T) {
|
||||
@@ -381,10 +386,100 @@ func TestImagePushThatReusesALayerIsNotCheckedForCommandNames(t *testing.T) {
|
||||
matched(930120, 932160))
|
||||
}
|
||||
|
||||
// giteaFixed are gitea requests that only gitea's fixes let through, one
|
||||
// for each kind of fix, each with what the Core Rule Set finds in it
|
||||
// without them.
|
||||
//
|
||||
//nolint:gochecknoglobals // a constant cannot be a list
|
||||
var giteaFixed = []struct {
|
||||
request request
|
||||
result coreruleset.Result
|
||||
}{
|
||||
// A search of code, on the path of gitea's search.
|
||||
{get("/explore/code?q=SELECT%20*%20FROM%20users%20WHERE"), matched(942100)},
|
||||
// A file name, in a parameter gitea sends names in, on any path.
|
||||
{get("/?path=.gitignore"), matched(930120)},
|
||||
// A file name, in a cookie of gitea's.
|
||||
{
|
||||
get("/owner/repo", "Cookie: gitea_flash=success%3DFile%2Bpackage.json%2Bdeleted"),
|
||||
matched(930120),
|
||||
},
|
||||
}
|
||||
|
||||
// The hostnames of the tests of SWW_APPS: the one that serves gitea, and
|
||||
// one that serves another app.
|
||||
const (
|
||||
giteaHost = "git.example"
|
||||
otherHost = "blog.example"
|
||||
)
|
||||
|
||||
// withHost returns r sent to host.
|
||||
func withHost(r request, host string) request {
|
||||
return request{r.method, r.target, append([]string{"Host: " + host}, r.headers...)}
|
||||
}
|
||||
|
||||
// servingApps returns the Core Rule Set at the defaults, with SWW_APPS
|
||||
// giving apps.
|
||||
func servingApps(t *testing.T, apps map[string][]string) *coreruleset.CoreRuleSet {
|
||||
t.Helper()
|
||||
|
||||
crs, err := coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, Apps: apps,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("load the Core Rule Set: %v", err)
|
||||
}
|
||||
|
||||
return crs
|
||||
}
|
||||
|
||||
func TestAppsUnsetGivesEveryHostnameEveryAppsFixes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
crs := servingApps(t, nil)
|
||||
|
||||
for _, fixed := range giteaFixed {
|
||||
for _, host := range []string{giteaHost, otherHost} {
|
||||
wantResult(t, crs, withHost(fixed.request, host), coreruleset.Result{})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestListedHostnameGetsItsAppsFixes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
crs := servingApps(t, map[string][]string{giteaHost: {"gitea"}})
|
||||
|
||||
// The hostname is matched without the port and without regard to case.
|
||||
for _, host := range []string{giteaHost, "GIT.Example:8443"} {
|
||||
for _, fixed := range giteaFixed {
|
||||
wantResult(t, crs, withHost(fixed.request, host), coreruleset.Result{})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnlistedHostnameGetsTheFullCheck(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, apps := range []map[string][]string{
|
||||
{giteaHost: {"gitea"}},
|
||||
// SWW_APPS set empty lists no hostname.
|
||||
{},
|
||||
} {
|
||||
crs := servingApps(t, apps)
|
||||
|
||||
for _, host := range []string{otherHost, otherHost + ":8443", "git.example.org"} {
|
||||
for _, fixed := range giteaFixed {
|
||||
wantResult(t, crs, withHost(fixed.request, host), fixed.result)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// withExclusions returns the Core Rule Set at the defaults, with the
|
||||
// exclusions file at path.
|
||||
func withExclusions(path string) (*waf.CoreRuleSet, error) {
|
||||
return waf.New(waf.Params{
|
||||
func withExclusions(path string) (*coreruleset.CoreRuleSet, error) {
|
||||
return coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, ExclusionsFile: path,
|
||||
})
|
||||
}
|
||||
@@ -422,7 +517,7 @@ func TestExclusionsFileIsLoadedBeforeTheRules(t *testing.T) {
|
||||
func TestRuleSwitchedOffAndNamedInTheExclusionsFileIsOff(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
crs, err := waf.New(waf.Params{
|
||||
crs, err := coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: 1, DisabledRules: []int{920500},
|
||||
ExclusionsFile: exclusionsFile(t, backupFileExclusion),
|
||||
})
|
||||
@@ -430,7 +525,7 @@ func TestRuleSwitchedOffAndNamedInTheExclusionsFileIsOff(t *testing.T) {
|
||||
t.Fatalf("load the exclusions: %v", err)
|
||||
}
|
||||
|
||||
wantResult(t, crs, get("/notes.txt~"), waf.Result{})
|
||||
wantResult(t, crs, get("/notes.txt~"), coreruleset.Result{})
|
||||
}
|
||||
|
||||
func TestExclusionsFileMayNameAnyNumberInAComment(t *testing.T) {
|
||||
@@ -566,10 +661,10 @@ const injection = "1' OR '1'='1"
|
||||
|
||||
// readingBodies returns the Core Rule Set as sww runs it by
|
||||
// default, but reading bodies up to bodyLimit.
|
||||
func readingBodies(t *testing.T) *waf.CoreRuleSet {
|
||||
func readingBodies(t *testing.T) *coreruleset.CoreRuleSet {
|
||||
t.Helper()
|
||||
|
||||
crs, err := waf.New(waf.Params{
|
||||
crs, err := coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyLimit: bodyLimit,
|
||||
})
|
||||
if err != nil {
|
||||
@@ -603,7 +698,8 @@ func padded(head, tail string, n int) string {
|
||||
// wantBody checks what crs finds in r with body, and that what it read of
|
||||
// body is read.
|
||||
func wantBody(
|
||||
t *testing.T, crs *waf.CoreRuleSet, r request, body string, want waf.Result,
|
||||
t *testing.T, crs *coreruleset.CoreRuleSet, r request, body string,
|
||||
want coreruleset.Result,
|
||||
read string,
|
||||
) {
|
||||
t.Helper()
|
||||
@@ -627,7 +723,7 @@ func TestBodiesAreReadOnlyWhileBodyLimitIsSet(t *testing.T) {
|
||||
{jsonBody, `{"q":"` + injection + `"}`},
|
||||
{xmlBody, "<q>" + injection + "</q>"},
|
||||
} {
|
||||
wantBody(t, off, post(tc.header), tc.body, waf.Result{}, "")
|
||||
wantBody(t, off, post(tc.header), tc.body, coreruleset.Result{}, "")
|
||||
wantBody(t, on, post(tc.header), tc.body, matched(942100), tc.body)
|
||||
}
|
||||
}
|
||||
@@ -650,7 +746,7 @@ func TestFormDataAndMultipartAreReadUpToTheLimit(t *testing.T) {
|
||||
} {
|
||||
wantBody(t, crs, post(tc.header), tc.attackFirst, matched(942100),
|
||||
tc.attackFirst[:bodyLimit+1])
|
||||
wantBody(t, crs, post(tc.header), tc.attackLast, waf.Result{},
|
||||
wantBody(t, crs, post(tc.header), tc.attackLast, coreruleset.Result{},
|
||||
tc.attackLast[:bodyLimit+1])
|
||||
}
|
||||
|
||||
@@ -660,7 +756,7 @@ func TestFormDataAndMultipartAreReadUpToTheLimit(t *testing.T) {
|
||||
wantBody(t, crs, post(formData), atLimit, matched(930120, 932160), atLimit)
|
||||
|
||||
pastLimit := padded("pad=", "&q=/etc/passwd", bodyLimit+1)
|
||||
wantBody(t, crs, post(formData), pastLimit, waf.Result{}, pastLimit)
|
||||
wantBody(t, crs, post(formData), pastLimit, coreruleset.Result{}, pastLimit)
|
||||
}
|
||||
|
||||
func TestJSONAndXMLAreReadOnlyWhenNoLargerThanTheLimit(t *testing.T) {
|
||||
@@ -676,7 +772,7 @@ func TestJSONAndXMLAreReadOnlyWhenNoLargerThanTheLimit(t *testing.T) {
|
||||
wantBody(t, crs, post(tc.header), fits, matched(942100), fits)
|
||||
|
||||
larger := padded(tc.head, tc.tail, bodyLimit+1)
|
||||
wantBody(t, crs, post(tc.header), larger, waf.Result{}, larger)
|
||||
wantBody(t, crs, post(tc.header), larger, coreruleset.Result{}, larger)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -694,7 +790,7 @@ func TestOtherBodiesAreNotRead(t *testing.T) {
|
||||
"Content-Type: text/plain",
|
||||
"Content-Type: application/x-git-receive-pack-request",
|
||||
} {
|
||||
wantBody(t, crs, post(header), body, waf.Result{}, "")
|
||||
wantBody(t, crs, post(header), body, coreruleset.Result{}, "")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -722,8 +818,8 @@ func TestContentEncodingIsRefusedOnTheKindsOfBodyTheCoreRuleSetReads(t *testing.
|
||||
// It is allowed on a body of any other kind, and on every body while no
|
||||
// body is read.
|
||||
fetch := "Content-Type: application/x-git-upload-pack-request"
|
||||
wantBody(t, crs, post(fetch, gzip), "a", waf.Result{}, "")
|
||||
wantBody(t, atDefaults(t), post(formData, gzip), "a", waf.Result{}, "")
|
||||
wantBody(t, crs, post(fetch, gzip), "a", coreruleset.Result{}, "")
|
||||
wantBody(t, atDefaults(t), post(formData, gzip), "a", coreruleset.Result{}, "")
|
||||
}
|
||||
|
||||
func TestParametersGiteaSendsNamesInAreLeftOutAmongFormFieldsToo(t *testing.T) {
|
||||
@@ -734,18 +830,18 @@ func TestParametersGiteaSendsNamesInAreLeftOutAmongFormFieldsToo(t *testing.T) {
|
||||
|
||||
for _, tc := range []struct {
|
||||
body string
|
||||
want waf.Result
|
||||
want coreruleset.Result
|
||||
}{
|
||||
{"path=.gitignore", waf.Result{}},
|
||||
{"path=.gitignore", coreruleset.Result{}},
|
||||
{"q=.gitignore", matched(930120)},
|
||||
{"redirect_uri=" + local, waf.Result{}},
|
||||
{"redirect_uri=" + local, coreruleset.Result{}},
|
||||
{"next=" + local, matched(931100, 934110)},
|
||||
} {
|
||||
wantBody(t, crs, post(formData), tc.body, tc.want, tc.body)
|
||||
}
|
||||
|
||||
body := field("path", ".gitignore") + end
|
||||
wantBody(t, crs, post(multipart), body, waf.Result{}, body)
|
||||
wantBody(t, crs, post(multipart), body, coreruleset.Result{}, body)
|
||||
|
||||
body = field("q", ".gitignore") + end
|
||||
wantBody(t, crs, post(multipart), body, matched(930120), body)
|
||||
@@ -869,7 +965,7 @@ func TestCorazaWritesNoFile(t *testing.T) {
|
||||
func TestBodyLimitOf1GLoads(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := waf.New(waf.Params{ParanoiaLevel: 1, BodyLimit: 1 << 30})
|
||||
_, err := coreruleset.New(coreruleset.Params{ParanoiaLevel: 1, BodyLimit: 1 << 30})
|
||||
if err != nil {
|
||||
t.Errorf("load the Core Rule Set reading bodies up to 1G: %v", err)
|
||||
}
|
||||
@@ -1,8 +1,51 @@
|
||||
# The eighth change: on gitea's paths, some fields skip the rules that refused
|
||||
# ordinary gitea requests there. They are the fields where gitea sends a search,
|
||||
# a file name or a branch name. Each exclusion runs with the request, before the
|
||||
# rules it switches off, those of phase 1 included, so this file is loaded before
|
||||
# the Core Rule Set's rules. Each comment names a request the rule refused.
|
||||
# gitea's fixes: some fields skip the rules that refused ordinary gitea requests.
|
||||
# They are the fields where gitea sends a search, a file name or a branch name.
|
||||
# Each exclusion runs with the request, before the rules it switches off, those
|
||||
# of phase 1 included, so this file is loaded before the Core Rule Set's rules.
|
||||
# Each comment names a request the rule refused.
|
||||
|
||||
# A request whose hostname SWW_APPS does not list gitea for skips every rule
|
||||
# below. Inspect sets tx.app_gitea on the others.
|
||||
SecRule &TX:app_gitea "@eq 0" \
|
||||
"id:900590,phase:1,pass,nolog,skipAfter:END_GITEA"
|
||||
|
||||
# The query parameters in which gitea sends names within a repository or its
|
||||
# own records, or a page of its own site, are not checked against the lists of
|
||||
# system files, shell paths and command names, on any path:
|
||||
# 930120: .gitignore
|
||||
# 932160: bin/docker-entrypoint
|
||||
# 932260: docker-build
|
||||
SecAction "id:900591,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:files,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:skip-to,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:sub_path,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:ref,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:sha,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:branch,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:workflow,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:artifactName,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:redirect_to,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:files,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:skip-to,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:sub_path,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:ref,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:sha,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:branch,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:workflow,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:artifactName,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:redirect_to,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:files,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:skip-to,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:sub_path,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:ref,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:sha,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:branch,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:workflow,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:artifactName,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:redirect_to"
|
||||
|
||||
# gitea's searches of code, of issues and pull requests, and of commits send the
|
||||
# text searched for in q, which skips these rules there:
|
||||
@@ -155,8 +198,8 @@ SecRule REQUEST_FILENAME \
|
||||
# 942140: information_schema.sql
|
||||
# 944110: java/lang/Runtime.java
|
||||
# 944130: java.lang.String.md
|
||||
# Each request skips the rules that refused it. The fourth change already has
|
||||
# path and redirect_to skip 930120, 932160 and 932260 on every path.
|
||||
# Each request skips the rules that refused it. Rule 900591 already has path and
|
||||
# redirect_to skip 930120, 932160 and 932260 on every path.
|
||||
|
||||
# A file's commit graph, in file.
|
||||
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/graph$" \
|
||||
@@ -281,3 +324,5 @@ SecRule REQUEST_FILENAME \
|
||||
"@rx ^/api/actions_pipeline/_apis/pipelines/workflows/[0-9]+/artifacts/[^/]+/upload$" \
|
||||
"id:900640,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=920450;REQUEST_HEADERS_NAMES:Content-Range"
|
||||
|
||||
SecMarker END_GITEA
|
||||
Vendored
+1
-1
@@ -1,6 +1,6 @@
|
||||
# Ordinary gitea 1.27 requests that the Core Rule Set must let through at the
|
||||
# defaults. Each line is a path and query, and may end with one header after a
|
||||
# space. The tests of internal/waf inspect each as a GET. The Core Rule Set reads
|
||||
# space. The tests of internal/coreruleset inspect each as a GET. The Core Rule Set reads
|
||||
# the query and the headers the same way for any method.
|
||||
|
||||
# Searches with ordinary words and symbols.
|
||||
@@ -0,0 +1,58 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic abuseipdb. It reads SWW_ABUSEIPDB_KEY,
|
||||
// SWW_ABUSEIPDB_MIN_SCORE and SWW_REPUTATION_ACTION.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// abuseIPDB adds the tag abuseipdb and notes AbuseIPDB, as noteHit does, with
|
||||
// the score, when its score of the client is a hit, and refuses the request
|
||||
// when SWW_REPUTATION_ACTION is deny, as dnsbl does for a zone. While
|
||||
// SWW_ABUSEIPDB_KEY is unset it does nothing. A client without a score is
|
||||
// checked in the background, by the request's address, if its history counts
|
||||
// an offence, and the request does not wait for the answer. The score is then
|
||||
// used for each address of the client. Its figures are the score and
|
||||
// SWW_ABUSEIPDB_MIN_SCORE.
|
||||
type abuseIPDB struct{}
|
||||
|
||||
func (abuseIPDB) name() string { return "abuseipdb" }
|
||||
|
||||
func (abuseIPDB) check(ctx context.Context, rq *request, _ time.Time) *verdict {
|
||||
if rq.h.config.AbuseIPDBKey == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
client := rq.h.clientGroup(rq.client)
|
||||
held, _ := rq.h.limiter.Client(client)
|
||||
offender := held.History.Offences != ratelimit.Offences{}
|
||||
|
||||
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
|
||||
if !hit {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.abuseIPDBHit = true
|
||||
rq.addTag(config.AbuseIPDBTag)
|
||||
rq.noteHit(bans.ReputationHit{Source: reputation.AbuseIPDBSource, Score: &score},
|
||||
"scored by AbuseIPDB at or over SWW_ABUSEIPDB_MIN_SCORE")
|
||||
|
||||
if rq.h.config.ReputationAction != deny {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionClientDenied,
|
||||
figures: map[string]any{
|
||||
figureScore: score, "min_score": rq.h.config.AbuseIPDBMinScore,
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -42,7 +42,7 @@ var (
|
||||
|
||||
// answerHealth answers a request for the health endpoint.
|
||||
func (rq *request) answerHealth() {
|
||||
rq.line.Action = requestlog.ActionAdmin
|
||||
rq.line.Action = requestlog.ActionOwnEndpoint
|
||||
// Set here rather than left to Go's server, which would set it only
|
||||
// after the log line has taken the response's headers.
|
||||
rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
||||
@@ -55,7 +55,7 @@ func (rq *request) answerHealth() {
|
||||
// refused with 401. An endpoint whose token is unset answers 404, as any other
|
||||
// path does.
|
||||
func (rq *request) answerAdmin() {
|
||||
rq.line.Action = requestlog.ActionAdmin
|
||||
rq.line.Action = requestlog.ActionOwnEndpoint
|
||||
|
||||
token, answer := rq.endpoint()
|
||||
|
||||
|
||||
@@ -40,7 +40,7 @@ func TestAdminEndpointsAreOffWhileTheTokenIsUnset(t *testing.T) {
|
||||
// An empty token does not match the unset one either.
|
||||
for _, authorization := range []string{adminBearer, bearer, "Bearer ", ""} {
|
||||
for _, e := range adminEndpoints() {
|
||||
s.adminRequest(authorization, e.method, e.path, e.body, http.StatusNotFound)
|
||||
s.ownEndpointRequest(authorization, e.method, e.path, e.body, http.StatusNotFound)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -67,7 +67,7 @@ func TestAdminEndpointsNeedTheAdminToken(t *testing.T) {
|
||||
for _, authorization := range []string{
|
||||
"", bearer, "Bearer " + strings.ToUpper(adminSecret), "Basic " + adminSecret,
|
||||
} {
|
||||
got := s.adminRequest(authorization, e.method, e.path, e.body,
|
||||
got := s.ownEndpointRequest(authorization, e.method, e.path, e.body,
|
||||
http.StatusUnauthorized)
|
||||
if got.header.Get("WWW-Authenticate") != "Bearer" {
|
||||
t.Errorf("%s %s with %q was answered without WWW-Authenticate: Bearer",
|
||||
@@ -121,7 +121,7 @@ func TestBanAddedListedAndLiftedThroughTheEndpoints(t *testing.T) {
|
||||
wantBans(t, s.admin(http.MethodPost, discriminator.BansPath, banOtherClient,
|
||||
http.StatusOK), want)
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
if line.BanExpires != requestlog.FormatTime(expires) {
|
||||
t.Errorf("the ban ends at %s, want %s", line.BanExpires, expires)
|
||||
}
|
||||
@@ -276,7 +276,7 @@ func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) {
|
||||
// limit of two a minute, and bans it.
|
||||
s.get(otherClient, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(otherClient, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(otherClient, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(otherClient, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
// Asked about by its address in IPv6 form too.
|
||||
for _, addr := range []string{otherClient, "::ffff:" + otherClient} {
|
||||
@@ -329,7 +329,7 @@ func TestAdminEndpointsNeedTheTokenInObserveMode(t *testing.T) {
|
||||
})
|
||||
|
||||
for _, e := range adminEndpoints() {
|
||||
s.adminRequest("", e.method, e.path, e.body, http.StatusUnauthorized)
|
||||
s.ownEndpointRequest("", e.method, e.path, e.body, http.StatusUnauthorized)
|
||||
}
|
||||
|
||||
if held := server.Ledger.Snapshot(); len(held) != 0 {
|
||||
@@ -356,16 +356,16 @@ func adminEndpoints() []adminEndpoint {
|
||||
|
||||
// admin sends a request with method for path, with body, to discriminator's own
|
||||
// address, with the admin token, and checks that it is answered with status, its
|
||||
// log line's action admin. It returns the answer.
|
||||
// log line's action own_endpoint. It returns the answer.
|
||||
func (s *sender) admin(method, path, body string, status int) answer {
|
||||
s.t.Helper()
|
||||
|
||||
return s.adminRequest(adminBearer, method, path, body, status)
|
||||
return s.ownEndpointRequest(adminBearer, method, path, body, status)
|
||||
}
|
||||
|
||||
// adminRequest is admin with authorization as the request's Authorization header
|
||||
// unless it is "".
|
||||
func (s *sender) adminRequest(
|
||||
// ownEndpointRequest is admin with authorization as the request's Authorization
|
||||
// header unless it is "".
|
||||
func (s *sender) ownEndpointRequest(
|
||||
authorization, method, path, body string, status int,
|
||||
) answer {
|
||||
s.t.Helper()
|
||||
@@ -380,7 +380,7 @@ func (s *sender) adminRequest(
|
||||
|
||||
line := s.out.endedLines(s.t, s.sent+1)[s.sent]
|
||||
s.sent++
|
||||
wantLine(s.t, line, status, requestlog.ActionAdmin)
|
||||
wantLine(s.t, line, status, requestlog.ActionOwnEndpoint)
|
||||
|
||||
return got
|
||||
}
|
||||
|
||||
@@ -31,14 +31,14 @@ func TestBanForABrokenLimitRaisesABanAlertWithItsNotes(t *testing.T) {
|
||||
start := clk.Now()
|
||||
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
netblock := netip.MustParsePrefix("203.0.113.0/24")
|
||||
ban := server.Ledger.Bans(netblock)[0]
|
||||
|
||||
// A request refused under the ban raises no other alert.
|
||||
clk.advance(time.Minute)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
wantAlerts(t, queue, banAlert(alerts.EventBan, start, client, bans.Ban{
|
||||
Netblock: netblock, Cause: bans.CauseLimit,
|
||||
@@ -62,22 +62,24 @@ func TestAttackBanRaisesABanAlertThenAPermanentBanAlert(t *testing.T) {
|
||||
|
||||
// The probe bans the client for seven days, and its next request makes
|
||||
// the ban permanent. The request after that changes nothing.
|
||||
s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(client, "/.env", http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
attackBan := server.Ledger.Bans(netblock)[0]
|
||||
|
||||
clk.advance(time.Minute)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionPermanentlyBanned)
|
||||
|
||||
permanentBan := server.Ledger.Bans(netblock)[0]
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionPermanentlyBanned)
|
||||
|
||||
// Another client's probe after its first ban has run out without a
|
||||
// request makes a permanent ban at once.
|
||||
s.request(otherClient, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(otherClient, "/.env", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
clk.advance(7 * 24 * time.Hour)
|
||||
s.request(otherClient, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(otherClient, "/.env", http.StatusForbidden,
|
||||
requestlog.ActionPermanentlyBanned)
|
||||
|
||||
otherBans := server.Ledger.Bans(other)
|
||||
|
||||
|
||||
@@ -73,7 +73,7 @@ func TestEachScopeAndWindowOverItsThresholdAlertsOncePerCooldown(t *testing.T) {
|
||||
{anomalyASN, anomaly.ScopeASN, netip.Prefix{}, asnDE, map[string]any{"asn": asnDE}},
|
||||
{anomalyTotal, anomaly.ScopeTotal, netip.Prefix{}, "the whole service", nil},
|
||||
{
|
||||
anomalyWatch, anomaly.ScopeWatch, netip.MustParsePrefix(clientsNet),
|
||||
anomalyWatch, anomaly.ScopeWatchedNet, netip.MustParsePrefix(clientsNet),
|
||||
"the named netblock office, " + clientsNet, map[string]any{"name": "office"},
|
||||
},
|
||||
} {
|
||||
@@ -187,7 +187,7 @@ func TestEveryRequestIsCountedWhateverIsDoneWithIt(t *testing.T) {
|
||||
{allowed, "/", http.StatusOK, requestlog.ActionForward},
|
||||
{exempt, "/", http.StatusOK, requestlog.ActionForward},
|
||||
{client, "/static/app.js", http.StatusOK, requestlog.ActionForward},
|
||||
{denied, "/", http.StatusForbidden, requestlog.ActionDenied},
|
||||
{denied, "/", http.StatusForbidden, requestlog.ActionClientDenied},
|
||||
} {
|
||||
for range 3 {
|
||||
s.request(sent.from, sent.path, sent.status, sent.action)
|
||||
|
||||
@@ -36,8 +36,8 @@ const (
|
||||
// tooLarge is how discriminator refuses a request over the header limits.
|
||||
tooLarge = http.StatusRequestHeaderFieldsTooLarge
|
||||
// silentAbuseIPDBURL is where AbuseIPDB never answers: at the stand-in
|
||||
// that unansweredGeoJSURL names too.
|
||||
silentAbuseIPDBURL = "unanswered://abuseipdb/api/v2/check"
|
||||
// that silentGeoJSURL names too.
|
||||
silentAbuseIPDBURL = "silent://abuseipdb/api/v2/check"
|
||||
// wholeLimit is the percentage of a rate limit that lowers it not at all.
|
||||
wholeLimit = 100
|
||||
// clientsDecision is the CrowdSec decision list holding client alone.
|
||||
@@ -84,13 +84,25 @@ func TestSixteenThousandCookiesAreRefusedWellWithinTheTimeLimit(t *testing.T) {
|
||||
wantDisposition(t, got, tooLarge, discriminator.DispositionDeny)
|
||||
|
||||
line := s.out.requestLine(t)
|
||||
wantLine(t, line, tooLarge, requestlog.ActionTooLarge)
|
||||
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
|
||||
|
||||
if line.DurationWAF != nil {
|
||||
t.Errorf("the Core Rule Set read the request, for %f ms", *line.DurationWAF)
|
||||
if line.DurationCoreRuleSetMS != nil {
|
||||
t.Errorf("the Core Rule Set read the request, for %f ms", *line.DurationCoreRuleSetMS)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeaderLimitRefusalNamesItsHeuristicCountsAndLimits(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := startDiscriminator(t, "", nil)
|
||||
req := s.question(siteA, client, "/")
|
||||
fillHeaders(req, headerFieldLimit+1)
|
||||
do(t, req)
|
||||
|
||||
wantHeuristic(t, s.out.requestLine(t), "header_limits",
|
||||
`{"fields": 101, "fields_limit": 100, "cookies": 0, "cookies_limit": 50}`)
|
||||
}
|
||||
|
||||
func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -145,7 +157,7 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
|
||||
|
||||
if tc.refused {
|
||||
wantDisposition(t, got, tooLarge, discriminator.DispositionDeny)
|
||||
wantLine(t, line, tooLarge, requestlog.ActionTooLarge)
|
||||
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
|
||||
|
||||
return
|
||||
}
|
||||
@@ -153,7 +165,7 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
|
||||
wantDisposition(t, got, http.StatusOK, discriminator.DispositionAllow)
|
||||
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
if line.DurationWAF == nil {
|
||||
if line.DurationCoreRuleSetMS == nil {
|
||||
t.Error("the Core Rule Set did not read a request within the limits")
|
||||
}
|
||||
})
|
||||
@@ -207,9 +219,10 @@ func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
|
||||
wantDisposition(t, got, http.StatusOK, discriminator.DispositionAllow)
|
||||
|
||||
line := s.out.requestLine(t)
|
||||
if line.WouldAction != requestlog.ActionTooLarge || line.DurationWAF != nil {
|
||||
t.Errorf("would_action %q and duration_waf %v, want %q and none",
|
||||
line.WouldAction, line.DurationWAF, requestlog.ActionTooLarge)
|
||||
if line.WouldAction != requestlog.ActionRequestHeaderTooLarge ||
|
||||
line.DurationCoreRuleSetMS != nil {
|
||||
t.Errorf("would_action %q and duration_core_rule_set_ms %v, want %q and none",
|
||||
line.WouldAction, line.DurationCoreRuleSetMS, requestlog.ActionRequestHeaderTooLarge)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -228,7 +241,7 @@ func TestRequestsOverTheHeaderLimitsCountTowardTheRateLimits(t *testing.T) {
|
||||
wantDisposition(t, ask(), http.StatusForbidden, discriminator.DispositionDeny)
|
||||
|
||||
lines := s.out.requestLines(t, 2)
|
||||
wantLine(t, lines[1], http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
wantLine(t, lines[1], http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
}
|
||||
|
||||
func TestTheHeaderLimitsComeAfterTheTrapPathsAndTheRuleFiles(t *testing.T) {
|
||||
@@ -237,8 +250,8 @@ func TestTheHeaderLimitsComeAfterTheTrapPathsAndTheRuleFiles(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name, path, action string
|
||||
}{
|
||||
{"a trap path", "/wp-login.php", requestlog.ActionBanned},
|
||||
{"a block rule", "/blocked", requestlog.ActionRuleBlocked},
|
||||
{"a trap path", "/wp-login.php", requestlog.ActionTemporarilyBanned},
|
||||
{"a block rule", "/blocked", requestlog.ActionBlockedByBlockRule},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -268,10 +281,10 @@ func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) {
|
||||
wantDisposition(t, do(t, req), tooLarge, discriminator.DispositionDeny)
|
||||
|
||||
line := s.out.requestLine(t)
|
||||
wantLine(t, line, tooLarge, requestlog.ActionTooLarge)
|
||||
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
|
||||
|
||||
if line.LimitPercent != nil {
|
||||
t.Errorf("limit percentage %d, want none", *line.LimitPercent)
|
||||
if line.RateLimitPercent != nil {
|
||||
t.Errorf("limit percentage %d, want none", *line.RateLimitPercent)
|
||||
}
|
||||
|
||||
// The client is under no ban, and GeoJS places it in a country.
|
||||
@@ -292,7 +305,8 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(t *testing.T
|
||||
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
||||
|
||||
wantDisposition(t, do(t, req), http.StatusForbidden, discriminator.DispositionDeny)
|
||||
wantLine(t, s.out.requestLine(t), http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
wantLine(t, s.out.requestLine(t), http.StatusForbidden,
|
||||
requestlog.ActionClientIsRateLimited)
|
||||
}
|
||||
|
||||
func TestObserveModeWaitsForGeoJSForARequestOverTheHeaderLimits(t *testing.T) {
|
||||
@@ -323,10 +337,10 @@ func newDiscriminatorInBubble(
|
||||
) (*discriminator.Server, *output) {
|
||||
t.Helper()
|
||||
|
||||
settings := map[string]string{deniedCountries: "KP"}
|
||||
settings := map[string]string{tagDeny: "country=KP"}
|
||||
maps.Copy(settings, env)
|
||||
|
||||
server, out, _ := newDiscriminator(t, unansweredGeoJSURL, time.Now, settings)
|
||||
server, out, _ := newDiscriminator(t, silentGeoJSURL, time.Now, settings)
|
||||
|
||||
return server, out
|
||||
}
|
||||
@@ -400,33 +414,37 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
|
||||
}{
|
||||
{
|
||||
name: "SWW_DENY_NETS", env: map[string]string{denyNets: client},
|
||||
prepare: leaveAsItIs, status: forbidden, action: requestlog.ActionDenied,
|
||||
prepare: leaveAsItIs, status: forbidden, action: requestlog.ActionClientDenied,
|
||||
},
|
||||
{
|
||||
name: "a ban",
|
||||
prepare: banClient, status: forbidden, action: requestlog.ActionBanned,
|
||||
prepare: banClient, status: forbidden, action: requestlog.ActionTemporarilyBanned,
|
||||
},
|
||||
{
|
||||
name: "the rate limits", env: map[string]string{rateLimitPerMinute: "1"},
|
||||
prepare: countClient, status: forbidden, action: requestlog.ActionRateLimited,
|
||||
prepare: countClient, status: forbidden,
|
||||
action: requestlog.ActionClientIsRateLimited,
|
||||
},
|
||||
{
|
||||
name: "the rate limits, with a biased threshold",
|
||||
env: map[string]string{rateLimitPerMinute: "1", unknownLimitPercent: "50"},
|
||||
prepare: countClient, status: forbidden, action: requestlog.ActionRateLimited,
|
||||
prepare: countClient, status: forbidden,
|
||||
action: requestlog.ActionClientIsRateLimited,
|
||||
},
|
||||
{
|
||||
name: "a blocklist", env: map[string]string{blocklistURLs: dropURL},
|
||||
prepare: listClient, status: forbidden, action: requestlog.ActionDenied,
|
||||
prepare: listClient, status: forbidden, action: requestlog.ActionClientDenied,
|
||||
},
|
||||
{
|
||||
name: "the CrowdSec decision list",
|
||||
env: map[string]string{crowdSecURL: lapi, crowdSecKey: bouncerKey},
|
||||
prepare: decideOnClient, status: forbidden, action: requestlog.ActionBanned,
|
||||
prepare: decideOnClient, status: forbidden,
|
||||
action: requestlog.ActionTemporarilyBanned,
|
||||
},
|
||||
{
|
||||
name: "the header limits", cookie: cookies(cookieLimit + 1),
|
||||
prepare: leaveAsItIs, status: tooLarge, action: requestlog.ActionTooLarge,
|
||||
prepare: leaveAsItIs, status: tooLarge,
|
||||
action: requestlog.ActionRequestHeaderTooLarge,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -530,7 +548,7 @@ func TestDiscriminatorAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
|
||||
|
||||
return discriminator.New(params)
|
||||
}, held, time.Now, map[string]string{
|
||||
deniedCountries: "KP", wafMode: block,
|
||||
tagDeny: "country=KP", wafMode: block,
|
||||
dnsblZones: dnsblZone, dnsblResolver: resolver.LocalAddr().String(),
|
||||
abuseIPDBKey: accountKey,
|
||||
blocklistURLs: held + "/drop.txt", asnLimitPercentURL: held + "/asn.txt",
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic active_ban, and the code that makes bans, which several
|
||||
// heuristics share. active_ban reads SWW_MODE; the bans are the ban ledger's.
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"context"
|
||||
"net/netip"
|
||||
"time"
|
||||
|
||||
@@ -18,11 +21,15 @@ func (rq *request) banResponse(action string) *refusal {
|
||||
return &refusal{status: rq.h.config.BanResponse, action: action}
|
||||
}
|
||||
|
||||
// banned reports whether a ban on a netblock the client is in covers the
|
||||
// request at now, and notes for the log line when that ban ends. A
|
||||
// request that makes the ban permanent, or in observe mode would have,
|
||||
// raises the alert for it.
|
||||
func (rq *request) banned(now time.Time) bool {
|
||||
// activeBan refuses a client a ban covers: a ban on a netblock the client is
|
||||
// in, in force at now. It notes for the log line when that ban ends. A request
|
||||
// that makes the ban permanent, or in observe mode would have, raises the alert
|
||||
// for it. Its figures are the ban's netblock and when it ends.
|
||||
type activeBan struct{}
|
||||
|
||||
func (activeBan) name() string { return "active_ban" }
|
||||
|
||||
func (activeBan) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
check := rq.h.ledger.Check
|
||||
if rq.h.config.Observe {
|
||||
check = rq.h.ledger.Find // the ban refuses nothing, and stays as it is
|
||||
@@ -38,85 +45,34 @@ func (rq *request) banned(now time.Time) bool {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return banned
|
||||
if !banned {
|
||||
return nil
|
||||
}
|
||||
|
||||
// limitBroken counts the request for the rate limits at now, notes the
|
||||
// client's counts for the log line, and reports whether the request takes
|
||||
// the client over a rate limit, as its limit percentage lowers it, which
|
||||
// breaks it.
|
||||
func (rq *request) limitBroken(now time.Time) bool {
|
||||
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now,
|
||||
rq.limitPercent.percent)
|
||||
rq.line.Counts = counts
|
||||
|
||||
if over {
|
||||
rq.banForLimit(now, hit, rq.h.config.BanResponse)
|
||||
}
|
||||
|
||||
return over
|
||||
}
|
||||
|
||||
// countBytes counts the request's bytes, as countedBytes gives them, for
|
||||
// the byte limits, once its report has come, and notes the client's byte totals
|
||||
// for the log line; its requests stay there as the rate limits counted them.
|
||||
// Only an allowed request has them counted, and only one the rate limits
|
||||
// counted; in observe mode, not one that enforce mode would have refused. Bytes
|
||||
// that take the client over a byte limit, as its limit percentage for the byte
|
||||
// limits lowers it, break it; the response was passed on whole.
|
||||
func (rq *request) countBytes() {
|
||||
if !rq.counted || rq.line.WouldAction != "" {
|
||||
return
|
||||
}
|
||||
|
||||
now := rq.h.now()
|
||||
|
||||
counts, hit, over := rq.h.limiter.CountBytes(rq.h.clientGroup(rq.client), now,
|
||||
rq.countedBytes(), rq.bytesPercent.percent)
|
||||
rq.line.Counts.MinuteBytes = counts.MinuteBytes
|
||||
rq.line.Counts.HourBytes = counts.HourBytes
|
||||
rq.line.Counts.DayBytes = counts.DayBytes
|
||||
|
||||
if over {
|
||||
rq.banForLimit(now, hit, rq.out.status)
|
||||
return &verdict{
|
||||
action: banAction(ban.Permanent()),
|
||||
figures: map[string]any{
|
||||
figureNetblock: ban.Netblock.String(), "ban_expires": rq.line.BanExpires,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// countRefusal counts the request for the error burst once it has been
|
||||
// answered, if discriminator refused it after a rule file match, a trap path or a
|
||||
// Core Rule Set match, and in observe mode if enforce mode would have: more than
|
||||
// SWW_ERROR_BURST_THRESHOLD such refusals of the client within a minute
|
||||
// break a limit. A client in SWW_ALLOW_NETS, which the checks skip, is
|
||||
// not counted, and nothing is while the threshold is off.
|
||||
func (rq *request) countRefusal() {
|
||||
cfg := rq.h.config
|
||||
if cfg.ErrorBurstThreshold == 0 {
|
||||
return
|
||||
// banAction is the action of a request refused under a ban or for making one:
|
||||
// permanent says the ban never runs out.
|
||||
func banAction(permanent bool) string {
|
||||
if permanent {
|
||||
return requestlog.ActionPermanentlyBanned
|
||||
}
|
||||
|
||||
if !rq.attack && !rq.ruleBlocked && !rq.wafBlocked {
|
||||
return
|
||||
return requestlog.ActionTemporarilyBanned
|
||||
}
|
||||
|
||||
now := rq.h.now()
|
||||
|
||||
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
|
||||
cfg.ErrorBurstThreshold)
|
||||
if !over {
|
||||
return
|
||||
// limitFigures are the figures of a heuristic that broke hit, a limit: its
|
||||
// window, the count and the limit.
|
||||
func limitFigures(hit ratelimit.Hit) map[string]any {
|
||||
return map[string]any{
|
||||
figureWindow: hit.Window, figureCount: hit.Count, figureLimit: hit.Limit,
|
||||
}
|
||||
|
||||
// What the client was sent, or in observe mode would have been.
|
||||
status := rq.out.status
|
||||
|
||||
switch rq.line.WouldAction {
|
||||
case requestlog.ActionRuleBlocked, requestlog.ActionWAFBlocked:
|
||||
status = http.StatusForbidden
|
||||
case requestlog.ActionBanned:
|
||||
status = cfg.BanResponse
|
||||
}
|
||||
|
||||
rq.banForLimit(now, hit, status)
|
||||
}
|
||||
|
||||
// countedBytes returns the request's bytes, once it has ended, as the
|
||||
@@ -150,7 +106,7 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
|
||||
case ratelimit.KindBytes:
|
||||
rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals
|
||||
case ratelimit.KindRefusals:
|
||||
rq.line.LimitHit = requestlog.LimitHitErrorBurst
|
||||
rq.line.LimitHit = requestlog.LimitHitMinuteRefusals
|
||||
default:
|
||||
rq.line.LimitHit = hit.Window
|
||||
}
|
||||
@@ -178,9 +134,9 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
|
||||
|
||||
switch hit.Kind {
|
||||
case ratelimit.KindRequests:
|
||||
notes.LimitPercent, notes.LimitPercentSetting = rq.limitPercent.logged()
|
||||
notes.LimitPercent, notes.LimitPercentSetting = rq.rateLimitPercent.logged()
|
||||
case ratelimit.KindBytes:
|
||||
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesPercent.logged()
|
||||
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
|
||||
}
|
||||
|
||||
if rq.h.config.Observe {
|
||||
@@ -205,11 +161,11 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
|
||||
// attack, which notes name: the ban rule that matched, or the trap path
|
||||
// asked for. It fills in the rest of the notes. In observe mode it makes
|
||||
// no ban, and raises the alert for the ban it would have made, if that
|
||||
// alert would be sent.
|
||||
func (rq *request) banForAttack(now time.Time, notes bans.Notes) {
|
||||
// alert would be sent. It returns the action for that ban.
|
||||
func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
|
||||
netblock := rq.h.netblock(rq.client)
|
||||
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseAttack) {
|
||||
return
|
||||
return banAction(rq.h.ledger.WouldBePermanent(netblock, now, bans.CauseAttack))
|
||||
}
|
||||
|
||||
notes.ASN = rq.line.ASN
|
||||
@@ -225,7 +181,7 @@ func (rq *request) banForAttack(now time.Time, notes bans.Notes) {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return
|
||||
return banAction(ban.Permanent())
|
||||
}
|
||||
|
||||
ban, made := rq.h.ledger.BanForAttack(netblock, now, notes)
|
||||
@@ -234,16 +190,18 @@ func (rq *request) banForAttack(now time.Time, notes bans.Notes) {
|
||||
if made {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return banAction(ban.Permanent())
|
||||
}
|
||||
|
||||
// banForCrowdSec bans the client's netblock at now until decision,
|
||||
// CrowdSec's decision on the client, ends. In observe mode it makes no
|
||||
// ban, and raises the alert for the ban it would have made, if that alert
|
||||
// would be sent.
|
||||
func (rq *request) banForCrowdSec(now time.Time, decision reputation.Decision) {
|
||||
// would be sent. It returns the action for that ban.
|
||||
func (rq *request) banForCrowdSec(now time.Time, decision reputation.Decision) string {
|
||||
netblock := rq.h.netblock(rq.client)
|
||||
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseCrowdSec) {
|
||||
return
|
||||
return banAction(rq.h.ledger.WouldBePermanent(netblock, now, bans.CauseCrowdSec))
|
||||
}
|
||||
|
||||
notes := bans.Notes{
|
||||
@@ -262,7 +220,7 @@ func (rq *request) banForCrowdSec(now time.Time, decision reputation.Decision) {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return
|
||||
return banAction(ban.Permanent())
|
||||
}
|
||||
|
||||
ban, made := rq.h.ledger.BanForCrowdSec(netblock, now, decision.Expires,
|
||||
@@ -272,6 +230,8 @@ func (rq *request) banForCrowdSec(now time.Time, decision reputation.Decision) {
|
||||
if made {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return banAction(ban.Permanent())
|
||||
}
|
||||
|
||||
// wouldAlertBan reports whether the alert for a ban on netblock for cause
|
||||
|
||||
@@ -40,7 +40,7 @@ func TestBrokenLimitBansTheClient(t *testing.T) {
|
||||
// client for an hour, the default.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
if line.LimitHit != minute || line.Offence != requestlog.OffenceLimit ||
|
||||
line.BanExpires != expires {
|
||||
t.Errorf("log line has limit_hit %q, offence %q and ban_expires %q, "+
|
||||
@@ -51,7 +51,7 @@ func TestBrokenLimitBansTheClient(t *testing.T) {
|
||||
// Every request while the ban lasts is refused.
|
||||
clk.advance(time.Hour - time.Second)
|
||||
|
||||
line = s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
line = s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
if line.BanExpires != expires || line.Offence != "" || line.LimitHit != "" {
|
||||
t.Errorf("log line has ban_expires %q, offence %q and limit_hit %q, "+
|
||||
"want %s and neither of the others", line.BanExpires, line.Offence,
|
||||
@@ -78,7 +78,9 @@ func TestBanLengthsFollowTheSettings(t *testing.T) {
|
||||
breakLimit := func() string {
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
return s.get(client, http.StatusForbidden, requestlog.ActionRateLimited).BanExpires
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
return line.BanExpires
|
||||
}
|
||||
wantExpires := func(got string, length time.Duration) {
|
||||
t.Helper()
|
||||
@@ -111,7 +113,7 @@ func TestBanLengthsFollowTheSettings(t *testing.T) {
|
||||
}
|
||||
|
||||
clk.advance(365 * 24 * time.Hour)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionPermanentlyBanned)
|
||||
}
|
||||
|
||||
func TestBanIsNotCountedAndResetsTheCounters(t *testing.T) {
|
||||
@@ -123,10 +125,10 @@ func TestBanIsNotCountedAndResetsTheCounters(t *testing.T) {
|
||||
// client for an hour.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
for range 3 {
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
// Later the same day the client has its whole allowance again: the
|
||||
@@ -135,7 +137,7 @@ func TestBanIsNotCountedAndResetsTheCounters(t *testing.T) {
|
||||
clk.advance(time.Hour)
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
banned := server.Ledger.Bans(netip.MustParsePrefix(client + "/32"))
|
||||
if len(banned) != 2 || banned[0].Notes.Refused != 3 {
|
||||
@@ -190,10 +192,10 @@ func TestBanCoversTheClientsNetblock(t *testing.T) {
|
||||
s, _, _ := startWithClock(t, "", env)
|
||||
|
||||
s.get(tc.breaker, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(tc.breaker, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(tc.breaker, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
for _, sent := range tc.refused {
|
||||
s.get(sent, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(sent, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
for _, sent := range tc.let {
|
||||
@@ -211,14 +213,14 @@ func TestBannedClientIsRefusedBeforeItsCountryIsLookedUp(t *testing.T) {
|
||||
lookupTimeout: "1h",
|
||||
rateLimitPerMinute: "1",
|
||||
banScopeV4Prefix: "24",
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
|
||||
// fromDE's ban covers otherClient, which is refused unasked about.
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
line := s.get(otherClient, http.StatusForbidden, requestlog.ActionBanned)
|
||||
line := s.get(otherClient, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
if line.Country != "" {
|
||||
t.Errorf("log line has country %q, want none", line.Country)
|
||||
}
|
||||
@@ -228,6 +230,22 @@ func TestBannedClientIsRefusedBeforeItsCountryIsLookedUp(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefusalUnderABanNamesItsHeuristicNetblockAndEnd(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, _ := startWithClock(t, "", map[string]string{
|
||||
rateLimitPerMinute: "1", banScopeV4Prefix: "24",
|
||||
})
|
||||
expires := requestlog.FormatTime(clk.Now().Add(time.Hour))
|
||||
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
line := s.get(otherClient, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
wantHeuristic(t, line, "active_ban",
|
||||
`{"netblock": "`+clientsNet+`", "ban_expires": "`+expires+`"}`)
|
||||
}
|
||||
|
||||
func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -250,7 +268,7 @@ func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
|
||||
lookupTimeout: "1h",
|
||||
rateLimitPerMinute: "1",
|
||||
denyNets: denied,
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
}
|
||||
|
||||
if tc.setting != "" {
|
||||
@@ -259,11 +277,11 @@ func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
|
||||
|
||||
s, _, _ := startWithClock(t, geojsURL, env)
|
||||
|
||||
s.get(denied, tc.status, requestlog.ActionDenied)
|
||||
s.get(fromKP, tc.status, requestlog.ActionCountryDenied)
|
||||
s.get(denied, tc.status, requestlog.ActionClientDenied)
|
||||
s.get(fromKP, tc.status, requestlog.ActionClientDenied)
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(fromDE, tc.status, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, tc.status, requestlog.ActionBanned)
|
||||
s.get(fromDE, tc.status, requestlog.ActionClientIsRateLimited)
|
||||
s.get(fromDE, tc.status, requestlog.ActionTemporarilyBanned)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -275,15 +293,15 @@ func TestBanNotes(t *testing.T) {
|
||||
s, clk, server := startWithClock(t, geojsURL, map[string]string{
|
||||
lookupTimeout: "1h",
|
||||
rateLimitPerMinute: "1",
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
start := clk.Now()
|
||||
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
s.request(fromDE, "/repo/commits?page=2",
|
||||
http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
|
||||
http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
netblock := netip.MustParsePrefix(fromDE + "/32")
|
||||
want := bans.Ban{
|
||||
@@ -326,7 +344,7 @@ func TestBanNotes(t *testing.T) {
|
||||
// The next ban counts this one among the earlier.
|
||||
clk.advance(time.Hour)
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
got = ledger.Bans(netblock)
|
||||
if len(got) != 2 || got[1].Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
|
||||
@@ -344,12 +362,12 @@ func TestMaxBansDropsTheBanOfTheNetblockSeenLongestAgo(t *testing.T) {
|
||||
|
||||
// One ban is held, so otherClient's ban drops client's.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
s.get(otherClient, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(otherClient, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(otherClient, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(otherClient, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(otherClient, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
// clock is the time a test sets, by which sww counts requests and
|
||||
@@ -397,7 +415,7 @@ func startWithClock(
|
||||
|
||||
// sender sends requests to rproxy one after another, each on a connection of its
|
||||
// own, and checks each one's answer and log line. discriminatorAddr is where
|
||||
// discriminator's own endpoints are, which adminRequest asks. Every request
|
||||
// discriminator's own endpoints are, which ownEndpointRequest asks. Every request
|
||||
// discriminator gets must come through the sender, since the log lines are
|
||||
// matched to them in order.
|
||||
type sender struct {
|
||||
|
||||
@@ -17,30 +17,31 @@ type percentage struct {
|
||||
}
|
||||
|
||||
// biasedThresholdsSet reports whether a biased threshold can lower a
|
||||
// client's limits: one of its lists is not empty,
|
||||
// SWW_UNKNOWN_LIMIT_PERCENT is below 100, or SWW_ASN_LIMIT_PERCENT_URL
|
||||
// is set. The client's lookup is then needed before its request goes on.
|
||||
// client's limits by its lookup: SWW_ASN_BYTES_PERCENT or
|
||||
// SWW_COUNTRY_BYTES_PERCENT is not empty, SWW_UNKNOWN_LIMIT_PERCENT is
|
||||
// below 100, or SWW_ASN_LIMIT_PERCENT_URL is set. The client's lookup is
|
||||
// then needed before its request goes on.
|
||||
func biasedThresholdsSet(cfg *config.Config) bool {
|
||||
return len(cfg.ASNLimitPercent) > 0 || len(cfg.CountryLimitPercent) > 0 ||
|
||||
len(cfg.ASNBytesPercent) > 0 || len(cfg.CountryBytesPercent) > 0 ||
|
||||
return len(cfg.ASNBytesPercent) > 0 || len(cfg.CountryBytesPercent) > 0 ||
|
||||
cfg.UnknownLimitPercent < whole || cfg.ASNLimitPercentURL != ""
|
||||
}
|
||||
|
||||
// limitPercentages returns the client's limit percentages, for the rate
|
||||
// limits and for the byte limits, by its AS number and country as looked
|
||||
// up, each "" when unknown, and the blocklists, DNSBL zones and AbuseIPDB
|
||||
// that list it. Each is the lowest of those the settings give it, the
|
||||
// first of them in the order below when several are lowest: the
|
||||
// percentage SWW_ASN_LIMIT_PERCENT gives its AS number, the one the file
|
||||
// SWW_ASN_LIMIT_PERCENT_URL names gives it, the one
|
||||
// SWW_COUNTRY_LIMIT_PERCENT gives its country, for a client without a
|
||||
// country, unless its request skipped the wait for GeoJS,
|
||||
// SWW_UNKNOWN_LIMIT_PERCENT, for a client a blocklist lists, the
|
||||
// percentage of SWW_BLOCKLIST_ACTION while it is limit, and for a client
|
||||
// a DNSBL zone's verdict lists, or whose AbuseIPDB score is a hit, the
|
||||
// percentage of SWW_REPUTATION_ACTION while it is limit. For the byte
|
||||
// limits, SWW_ASN_BYTES_PERCENT and SWW_COUNTRY_BYTES_PERCENT take the
|
||||
// place of the first three for an AS number or a country they list.
|
||||
// limits and for the byte limits, by its tags, its AS number and country
|
||||
// as looked up, each "" when unknown, and the blocklists, DNSBL zones and
|
||||
// AbuseIPDB that list it. Each is the lowest of those the settings give
|
||||
// it, the first of them in the order below when several are lowest: the
|
||||
// percentage SWW_TAG_LIMIT_PERCENT gives its asn= tag, the one the file
|
||||
// SWW_ASN_LIMIT_PERCENT_URL names gives its AS number, the one
|
||||
// SWW_TAG_LIMIT_PERCENT gives its country= tag, the lowest it gives its
|
||||
// other tags, for a client without a country, unless its request skipped
|
||||
// the wait for GeoJS, SWW_UNKNOWN_LIMIT_PERCENT, for a client a blocklist
|
||||
// lists, the percentage of SWW_BLOCKLIST_ACTION while it is limit, and
|
||||
// for a client a DNSBL zone's verdict lists, or whose AbuseIPDB score is
|
||||
// a hit, the percentage of SWW_REPUTATION_ACTION while it is limit. For
|
||||
// the byte limits, SWW_ASN_BYTES_PERCENT and SWW_COUNTRY_BYTES_PERCENT
|
||||
// take the place of the first three for an AS number or a country they
|
||||
// list.
|
||||
func (rq *request) limitPercentages() (percentage, percentage) {
|
||||
cfg := rq.h.config
|
||||
asn, country := rq.line.ASN, rq.line.Country
|
||||
@@ -65,10 +66,18 @@ func (rq *request) limitPercentages() (percentage, percentage) {
|
||||
reputationListed = percentage{cfg.ReputationLimitPercent, "SWW_REPUTATION_ACTION"}
|
||||
}
|
||||
|
||||
asnRequests := lowest(given(cfg.ASNLimitPercent, asn, "SWW_ASN_LIMIT_PERCENT"),
|
||||
fetched)
|
||||
countryRequests := given(cfg.CountryLimitPercent, country,
|
||||
"SWW_COUNTRY_LIMIT_PERCENT")
|
||||
const tagSetting = "SWW_TAG_LIMIT_PERCENT"
|
||||
|
||||
asnRequests := lowest(given(cfg.TagLimitPercent, asnTag(asn), tagSetting), fetched)
|
||||
countryRequests := given(cfg.TagLimitPercent, countryTag(country), tagSetting)
|
||||
|
||||
otherTags := percentage{percent: whole}
|
||||
|
||||
for _, tag := range rq.line.Tags {
|
||||
if tag != asnTag(asn) && tag != countryTag(country) {
|
||||
otherTags = lowest(otherTags, given(cfg.TagLimitPercent, tag, tagSetting))
|
||||
}
|
||||
}
|
||||
|
||||
asnBytes, countryBytes := asnRequests, countryRequests
|
||||
if _, listed := cfg.ASNBytesPercent[asn]; listed {
|
||||
@@ -79,12 +88,13 @@ func (rq *request) limitPercentages() (percentage, percentage) {
|
||||
countryBytes = given(cfg.CountryBytesPercent, country, "SWW_COUNTRY_BYTES_PERCENT")
|
||||
}
|
||||
|
||||
return lowest(asnRequests, countryRequests, unknown, blocklisted, reputationListed),
|
||||
lowest(asnBytes, countryBytes, unknown, blocklisted, reputationListed)
|
||||
return lowest(asnRequests, countryRequests, otherTags, unknown, blocklisted,
|
||||
reputationListed),
|
||||
lowest(asnBytes, countryBytes, otherTags, unknown, blocklisted, reputationListed)
|
||||
}
|
||||
|
||||
// given returns the percentage percents, the setting named setting, gives
|
||||
// code, an AS number or a country, or whole when it does not list code.
|
||||
// code, an AS number, a country or a tag, or whole when it does not list code.
|
||||
func given(percents map[string]int64, code, setting string) percentage {
|
||||
percent, listed := percents[code]
|
||||
if !listed {
|
||||
|
||||
@@ -22,8 +22,7 @@ import (
|
||||
|
||||
// The biased thresholds.
|
||||
const (
|
||||
asnLimitPercent = "SWW_ASN_LIMIT_PERCENT"
|
||||
countryLimitPercent = "SWW_COUNTRY_LIMIT_PERCENT"
|
||||
tagLimitPercent = "SWW_TAG_LIMIT_PERCENT"
|
||||
asnBytesPercent = "SWW_ASN_BYTES_PERCENT"
|
||||
countryBytesPercent = "SWW_COUNTRY_BYTES_PERCENT"
|
||||
unknownLimitPercent = "SWW_UNKNOWN_LIMIT_PERCENT"
|
||||
@@ -35,6 +34,11 @@ const (
|
||||
asnDEHalf = asnDE + ":50"
|
||||
asnDEQuarter = asnDE + ":25"
|
||||
countryDEHalf = "de:50"
|
||||
// The same, as SWW_TAG_LIMIT_PERCENT names fromDE's tags.
|
||||
asnTagDEHalf = "asn=64496:50"
|
||||
asnTagDEQuarter = "asn=64496:25"
|
||||
countryTagDEHalf = "country=DE:50"
|
||||
countryTagDEQuarter = "country=DE:25"
|
||||
// noCountry is in an AS of its own, AS64500, and in no country.
|
||||
noCountry = "192.0.2.80"
|
||||
// fourAMinute is the rate limit these tests set: half of it is 2
|
||||
@@ -54,8 +58,8 @@ func TestEachBiasedThresholdLowersTheRateLimits(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
setting, value, from string
|
||||
}{
|
||||
{asnLimitPercent, asnDEHalf, fromDE},
|
||||
{countryLimitPercent, countryDEHalf, fromDE},
|
||||
{tagLimitPercent, asnTagDEHalf, fromDE},
|
||||
{tagLimitPercent, countryTagDEHalf, fromDE},
|
||||
{unknownLimitPercent, "50", unplaced},
|
||||
} {
|
||||
t.Run(tc.setting, func(t *testing.T) {
|
||||
@@ -72,17 +76,19 @@ func TestEachBiasedThresholdLowersTheRateLimits(t *testing.T) {
|
||||
}{
|
||||
{http.StatusOK, requestlog.ActionForward},
|
||||
{http.StatusOK, requestlog.ActionForward},
|
||||
{http.StatusForbidden, requestlog.ActionRateLimited},
|
||||
{http.StatusForbidden, requestlog.ActionClientIsRateLimited},
|
||||
} {
|
||||
line := s.get(tc.from, sent.status, sent.action)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
"50 from "+tc.setting)
|
||||
}
|
||||
|
||||
// fromKP, which no setting lists, has the whole limit.
|
||||
for range 3 {
|
||||
line := s.get(fromKP, http.StatusOK, requestlog.ActionForward)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
none)
|
||||
}
|
||||
})
|
||||
@@ -96,8 +102,8 @@ func TestEachBiasedThresholdLowersTheByteLimits(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
setting, value, from string
|
||||
}{
|
||||
{asnLimitPercent, asnDEHalf, fromDE},
|
||||
{countryLimitPercent, "DE:50", fromDE},
|
||||
{tagLimitPercent, asnTagDEHalf, fromDE},
|
||||
{tagLimitPercent, "Country=de:50", fromDE},
|
||||
{unknownLimitPercent, "50", unplaced},
|
||||
{asnBytesPercent, "as64496:50", fromDE},
|
||||
{countryBytesPercent, countryDEHalf, fromDE},
|
||||
@@ -115,7 +121,8 @@ func TestEachBiasedThresholdLowersTheByteLimits(t *testing.T) {
|
||||
t.Errorf("log line has limit_hit %q, want %s", line.LimitHit, minuteBytes)
|
||||
}
|
||||
|
||||
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting,
|
||||
"50 from "+tc.setting)
|
||||
|
||||
// fromKP, which no setting lists, has the whole limit.
|
||||
@@ -124,7 +131,8 @@ func TestEachBiasedThresholdLowersTheByteLimits(t *testing.T) {
|
||||
t.Errorf("log line for %s has limit_hit %q, want none", fromKP, line.LimitHit)
|
||||
}
|
||||
|
||||
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting, none)
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting, none)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -135,9 +143,9 @@ func TestBytesPercentSettingsTakeThePlaceOfTheOthersForByteLimits(t *testing.T)
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
env map[string]string
|
||||
// limitPercent and bytesPercent are the log line's, as percentText
|
||||
// gives them, and limitHit is its limit_hit.
|
||||
limitPercent, bytesPercent, limitHit string
|
||||
// rateLimitPercent and bytesLimitPercent are the log line's, as
|
||||
// percentText gives them, and limitHit is its limit_hit.
|
||||
rateLimitPercent, bytesLimitPercent, limitHit string
|
||||
}{
|
||||
{
|
||||
"lowering the byte limits alone",
|
||||
@@ -151,13 +159,13 @@ func TestBytesPercentSettingsTakeThePlaceOfTheOthersForByteLimits(t *testing.T)
|
||||
},
|
||||
{
|
||||
"raising the byte limits back",
|
||||
map[string]string{asnLimitPercent: asnDEHalf, asnBytesPercent: asnDE + ":100"},
|
||||
"50 from " + asnLimitPercent, none, "",
|
||||
map[string]string{tagLimitPercent: asnTagDEHalf, asnBytesPercent: asnDE + ":100"},
|
||||
"50 from " + tagLimitPercent, none, "",
|
||||
},
|
||||
{
|
||||
"raising the byte limits back, by country",
|
||||
map[string]string{countryLimitPercent: countryDEHalf, countryBytesPercent: "de:100"},
|
||||
"50 from " + countryLimitPercent, none, "",
|
||||
map[string]string{tagLimitPercent: countryTagDEHalf, countryBytesPercent: "de:100"},
|
||||
"50 from " + tagLimitPercent, none, "",
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -173,10 +181,12 @@ func TestBytesPercentSettingsTakeThePlaceOfTheOthersForByteLimits(t *testing.T)
|
||||
t.Errorf("log line has limit_hit %q, want %q", line.LimitHit, tc.limitHit)
|
||||
}
|
||||
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
tc.limitPercent)
|
||||
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
|
||||
tc.bytesPercent)
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
tc.rateLimitPercent)
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting,
|
||||
tc.bytesLimitPercent)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -184,19 +194,19 @@ func TestBytesPercentSettingsTakeThePlaceOfTheOthersForByteLimits(t *testing.T)
|
||||
func TestZeroPercentIsAZeroAllowance(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithLookups(t, map[string]string{asnLimitPercent: asnDE + ":0"})
|
||||
s, _, _ := startWithLookups(t, map[string]string{unknownLimitPercent: "0"})
|
||||
|
||||
// The first request breaks the limit, and bans the client; the log line
|
||||
// gives the 0.
|
||||
line := s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
if line.fields["limit_percent"] != float64(0) ||
|
||||
line.fields["limit_percent_setting"] != asnLimitPercent {
|
||||
t.Errorf("log line has limit_percent %v from %v, want 0 from %s",
|
||||
line.fields["limit_percent"], line.fields["limit_percent_setting"],
|
||||
asnLimitPercent)
|
||||
line := s.get(unplaced, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
if line.fields["rate_limit_percent"] != float64(0) ||
|
||||
line.fields["rate_limit_percent_setting"] != unknownLimitPercent {
|
||||
t.Errorf("log line has rate_limit_percent %v from %v, want 0 from %s",
|
||||
line.fields["rate_limit_percent"], line.fields["rate_limit_percent_setting"],
|
||||
unknownLimitPercent)
|
||||
}
|
||||
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(unplaced, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestLowestPercentageApplies(t *testing.T) {
|
||||
@@ -206,34 +216,34 @@ func TestLowestPercentageApplies(t *testing.T) {
|
||||
name string
|
||||
env map[string]string
|
||||
from string
|
||||
// want is the log line's limit_percent, as percentText gives it.
|
||||
// want is the log line's rate_limit_percent, as percentText gives it.
|
||||
want string
|
||||
}{
|
||||
{
|
||||
"the country's",
|
||||
map[string]string{asnLimitPercent: asnDEHalf, countryLimitPercent: "de:25"},
|
||||
fromDE, "25 from " + countryLimitPercent,
|
||||
map[string]string{tagLimitPercent: asnTagDEHalf + "," + countryTagDEQuarter},
|
||||
fromDE, "25 from " + tagLimitPercent,
|
||||
},
|
||||
{
|
||||
"the AS number's",
|
||||
map[string]string{asnLimitPercent: asnDEQuarter, countryLimitPercent: countryDEHalf},
|
||||
fromDE, "25 from " + asnLimitPercent,
|
||||
map[string]string{tagLimitPercent: asnTagDEQuarter + "," + countryTagDEHalf},
|
||||
fromDE, "25 from " + tagLimitPercent,
|
||||
},
|
||||
{
|
||||
"the AS number's, the first of two alike",
|
||||
map[string]string{asnLimitPercent: asnDEQuarter, countryLimitPercent: "de:25"},
|
||||
fromDE, "25 from " + asnLimitPercent,
|
||||
map[string]string{tagLimitPercent: asnTagDEQuarter + "," + countryTagDEQuarter},
|
||||
fromDE, "25 from " + tagLimitPercent,
|
||||
},
|
||||
{
|
||||
"that for a client without a country",
|
||||
map[string]string{asnLimitPercent: "AS64500:50", unknownLimitPercent: "25"},
|
||||
map[string]string{tagLimitPercent: "asn=64500:50", unknownLimitPercent: "25"},
|
||||
noCountry, "25 from " + unknownLimitPercent,
|
||||
},
|
||||
{
|
||||
// SWW_UNKNOWN_LIMIT_PERCENT is left at its default, 100.
|
||||
"the AS number's, for a client without a country",
|
||||
map[string]string{asnLimitPercent: "AS64500:25"},
|
||||
noCountry, "25 from " + asnLimitPercent,
|
||||
map[string]string{tagLimitPercent: "asn=64500:25"},
|
||||
noCountry, "25 from " + tagLimitPercent,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -246,8 +256,9 @@ func TestLowestPercentageApplies(t *testing.T) {
|
||||
// A quarter of 4 requests a minute: the second breaks the limit.
|
||||
s.get(tc.from, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line := s.get(tc.from, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
line := s.get(tc.from, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
tc.want)
|
||||
})
|
||||
}
|
||||
@@ -265,7 +276,7 @@ func TestUnknownLimitPercentGivesEveryClientWithoutACountryItsPercentage(t *test
|
||||
for _, from := range []string{unplaced, "10.0.0.8"} {
|
||||
s.get(from, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(from, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(from, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(from, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
}
|
||||
|
||||
// One in a country has the whole limit.
|
||||
@@ -280,7 +291,7 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownLimitPercent(t *testing.T) {
|
||||
// In a synctest bubble, as TestRequestWaitsAsLongAsTheLookupTimeoutSays
|
||||
// says, with a GeoJS that never answers.
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
server, out, _ := newDiscriminator(t, unansweredGeoJSURL, time.Now,
|
||||
server, out, _ := newDiscriminator(t, silentGeoJSURL, time.Now,
|
||||
map[string]string{unknownLimitPercent: "0"})
|
||||
|
||||
// Once the time the request waits for its answer is up, the client
|
||||
@@ -289,8 +300,9 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownLimitPercent(t *testing.T) {
|
||||
serveFromDE(t, server, http.MethodGet, http.NoBody)
|
||||
|
||||
line := out.requestLine(t)
|
||||
wantLine(t, line, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantLine(t, line, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
"0 from "+unknownLimitPercent)
|
||||
})
|
||||
}
|
||||
@@ -304,8 +316,12 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
|
||||
setting, value string
|
||||
waits bool
|
||||
}{
|
||||
{asnLimitPercent, asnDEHalf, true},
|
||||
{countryLimitPercent, countryDEHalf, true},
|
||||
{tagLimitPercent, asnTagDEHalf, true},
|
||||
{tagLimitPercent, countryTagDEHalf, true},
|
||||
{tagDeny, denyKP, true},
|
||||
// A tag the lookup does not give needs no answer.
|
||||
{tagLimitPercent, "vpn:50", false},
|
||||
{tagDeny, "vpn", false},
|
||||
{asnBytesPercent, asnDEHalf, true},
|
||||
{countryBytesPercent, countryDEHalf, true},
|
||||
{unknownLimitPercent, "99", true},
|
||||
@@ -319,8 +335,10 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
|
||||
// In a synctest bubble, as TestRequestWaitsAsLongAsTheLookupTimeoutSays
|
||||
// says, with a GeoJS that never answers.
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
server, out, _ := newDiscriminator(t, unansweredGeoJSURL, time.Now,
|
||||
map[string]string{lookupTimeout: timeout.String(), tc.setting: tc.value})
|
||||
server, out, _ := newDiscriminator(t, silentGeoJSURL, time.Now,
|
||||
map[string]string{
|
||||
lookupTimeout: timeout.String(), tagLists: "vpn=" + vpnList, tc.setting: tc.value,
|
||||
})
|
||||
began := time.Now()
|
||||
|
||||
serveFromDE(t, server, http.MethodPost, strings.NewReader("ab"))
|
||||
@@ -363,15 +381,15 @@ func TestBanForALoweredLimitGivesThePercentageInItsNotesAndItsAlert(t *testing.T
|
||||
{
|
||||
// A quarter of 12 requests a minute is 3: the fourth breaks it.
|
||||
"a rate limit",
|
||||
map[string]string{rateLimitPerMinute: "12", asnLimitPercent: asnDEQuarter},
|
||||
3, http.StatusForbidden, requestlog.ActionRateLimited,
|
||||
"requests per minute over the limit of 3", "25 from " + asnLimitPercent,
|
||||
map[string]string{rateLimitPerMinute: "12", tagLimitPercent: asnTagDEQuarter},
|
||||
3, http.StatusForbidden, requestlog.ActionClientIsRateLimited,
|
||||
"requests per minute over the limit of 3", "25 from " + tagLimitPercent,
|
||||
},
|
||||
{
|
||||
// The byte limits' percentage, not the rate limits'.
|
||||
"a byte limit",
|
||||
map[string]string{
|
||||
bytesLimitPerMinute: twoUploads, asnLimitPercent: asnDEQuarter,
|
||||
bytesLimitPerMinute: twoUploads, tagLimitPercent: asnTagDEQuarter,
|
||||
asnBytesPercent: asnDEHalf,
|
||||
},
|
||||
0, http.StatusOK, requestlog.ActionForward,
|
||||
@@ -487,7 +505,7 @@ func wantPercent(t *testing.T, what string, percent *int64, setting, want string
|
||||
}
|
||||
|
||||
// percentText gives a limit percentage and the setting that gave it as
|
||||
// text, such as "50 from SWW_ASN_LIMIT_PERCENT", or none when both are
|
||||
// text, such as "50 from SWW_TAG_LIMIT_PERCENT", or none when both are
|
||||
// left out.
|
||||
func percentText(percent *int64, setting string) string {
|
||||
switch {
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic blocklists. It reads SWW_BLOCKLIST_ACTION, and the blocklists
|
||||
// through the reputation lists.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// blocklists notes the blocklists that list the client, as noteListed does,
|
||||
// and refuses the request when SWW_BLOCKLIST_ACTION is deny. Being limit, it
|
||||
// lowers the client's limits instead (see limitPercentages), and being log, it
|
||||
// does nothing more. Its figure is the blocklists that list the client.
|
||||
type blocklists struct{}
|
||||
|
||||
func (blocklists) name() string { return "blocklists" }
|
||||
|
||||
func (blocklists) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
listedBy := rq.h.lists.ListedBy(rq.client)
|
||||
rq.blocklisted = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a blocklist")
|
||||
|
||||
if !rq.refusedByBlocklists(listedBy) {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionClientDenied,
|
||||
figures: map[string]any{"lists": listedBy},
|
||||
}
|
||||
}
|
||||
|
||||
// refusedByBlocklists reports whether the heuristic blocklists refuses a
|
||||
// client, given listedBy, the blocklists that list it: it does when one lists
|
||||
// it and SWW_BLOCKLIST_ACTION is deny.
|
||||
func (rq *request) refusedByBlocklists(listedBy []string) bool {
|
||||
return len(listedBy) > 0 && rq.h.config.BlocklistAction == deny
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic byte_limits. It reads SWW_BYTES_COUNT, and the byte limits
|
||||
// and biased thresholds through the limiter and limitPercentages.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// byteLimits counts the request's bytes, as countedBytes gives them, for the
|
||||
// byte limits, once its report has come, and notes the client's byte totals for
|
||||
// the log line; its requests stay there as the rate limits counted them. Only
|
||||
// an allowed request has them counted, and only one the rate limits counted; in
|
||||
// observe mode, not one that enforce mode would have refused. Bytes that take
|
||||
// the client over a byte limit, as its limit percentage for the byte limits
|
||||
// lowers it, break it, which bans the client; the response was passed on whole.
|
||||
// Its figures are the window, the count and the limit.
|
||||
type byteLimits struct{}
|
||||
|
||||
func (byteLimits) name() string { return "byte_limits" }
|
||||
|
||||
func (byteLimits) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
if !rq.counted || rq.line.WouldAction != "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
counts, hit, over := rq.h.limiter.CountBytes(rq.h.clientGroup(rq.client), now,
|
||||
rq.countedBytes(), rq.bytesLimitPercent.percent)
|
||||
rq.line.Counts.MinuteBytes = counts.MinuteBytes
|
||||
rq.line.Counts.HourBytes = counts.HourBytes
|
||||
rq.line.Counts.DayBytes = counts.DayBytes
|
||||
|
||||
if !over {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.banForLimit(now, hit, rq.out.status)
|
||||
|
||||
return &verdict{figures: limitFigures(hit)}
|
||||
}
|
||||
@@ -90,7 +90,7 @@ func TestEachByteLimitBansOnceTheResponseHasEnded(t *testing.T) {
|
||||
line.BanExpires, tc.window, expires)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
wantMetric(t, s.scrape(), `sww_rate_limit_hits_total{`+
|
||||
`instance="`+alertInstance+`",kind="bytes",window="`+tc.window+`"}`, 1)
|
||||
@@ -98,6 +98,21 @@ func TestEachByteLimitBansOnceTheResponseHasEnded(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestByteLimitBanNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{bytesLimitPerMinute: byteLimit})
|
||||
|
||||
line, _ := s.download()
|
||||
wantAfterAnswerHeuristic(t, line, "", "null")
|
||||
|
||||
// Two answers of 70 bytes are over the limit of 99.
|
||||
line, _ = s.download()
|
||||
wantHeuristic(t, line, "", "null")
|
||||
wantAfterAnswerHeuristic(t, line, "byte_limits",
|
||||
`{"window": "minute", "count": 140, "limit": 99}`)
|
||||
}
|
||||
|
||||
func TestResponseOverAByteLimitByItselfIsPassedOnWhole(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -110,7 +125,7 @@ func TestResponseOverAByteLimitByItselfIsPassedOnWhole(t *testing.T) {
|
||||
len(got.body), got.err, line.LimitHit, answerBytes, minuteBytes)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestBytesOfAnAnswerThatBreaksOffAreCounted(t *testing.T) {
|
||||
@@ -132,7 +147,7 @@ func TestBytesOfAnAnswerThatBreaksOffAreCounted(t *testing.T) {
|
||||
answerBytes, minuteBytes, expires)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestWebSocketBytesAreCountedOnceItCloses(t *testing.T) {
|
||||
@@ -161,7 +176,7 @@ func TestWebSocketBytesAreCountedOnceItCloses(t *testing.T) {
|
||||
line.LimitHit, line.Counts.MinuteBytes, minuteBytes, tc.counted)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -203,7 +218,7 @@ func TestBytesCountSaysWhichBytesCount(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -371,7 +386,7 @@ func TestObserveModeLeavesOutTheBytesOfARequestEnforceModeRefuses(t *testing.T)
|
||||
// counted, and the byte limit is not broken. Its line gives the bytes
|
||||
// counted before it.
|
||||
line := s.upload()
|
||||
wantWouldAction(t, line, requestlog.ActionRateLimited)
|
||||
wantWouldAction(t, line, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
if line.LimitHit != minute || line.Counts.MinuteBytes != bodyBytes+answerBytes {
|
||||
t.Errorf("log line has limit_hit %q and minute_bytes %v, want minute and %d",
|
||||
|
||||
@@ -1,26 +1,34 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic core_rule_set. It reads SWW_WAF_MODE, SWW_WAF_EXEMPT_PATHS
|
||||
// and SWW_WAF_ANOMALY_THRESHOLD.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
)
|
||||
|
||||
// checkCoreRuleSet inspects the request with the Core Rule Set, unless
|
||||
// coreRuleSet inspects the request with the Core Rule Set, unless
|
||||
// SWW_WAF_MODE is off or SWW_WAF_EXEMPT_PATHS exempts its path, as
|
||||
// pathExempt decides, and notes the rules it matched and its score in the
|
||||
// log line, and the rules in the metrics. A score at or over
|
||||
// SWW_WAF_ANOMALY_THRESHOLD is a match: it raises the waf_block alert,
|
||||
// and in block mode refuses the request, which is an offence its client's
|
||||
// history counts, and so returns ActionWAFBlocked. It returns "" for a
|
||||
// request it does not refuse.
|
||||
func (rq *request) checkCoreRuleSet() string {
|
||||
// SWW_WAF_ANOMALY_THRESHOLD is a match: it adds the tag exploitsignature,
|
||||
// raises the core_rule_set_match alert, and in block mode refuses the request with 403,
|
||||
// which is an offence its client's history counts. Its figures are the score
|
||||
// and the threshold.
|
||||
type coreRuleSet struct{}
|
||||
|
||||
func (coreRuleSet) name() string { return "core_rule_set" }
|
||||
|
||||
func (coreRuleSet) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
cfg := rq.h.config
|
||||
if cfg.WAFMode == config.WAFModeOff || pathExempt(rq.in, cfg.WAFExemptPaths) {
|
||||
return ""
|
||||
return nil
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
@@ -30,36 +38,40 @@ func (rq *request) checkCoreRuleSet() string {
|
||||
result, read, _ := rq.h.coreRuleSet.Inspect(rq.in, rq.client, rq.in.Body)
|
||||
rq.requestBytes = int64(len(read))
|
||||
|
||||
rq.line.DurationWAF = new(requestlog.Milliseconds(time.Since(start)))
|
||||
rq.line.WAFRuleIDs = result.RuleIDs
|
||||
rq.line.WAFScore = &result.Score
|
||||
rq.line.DurationCoreRuleSetMS = new(requestlog.Milliseconds(time.Since(start)))
|
||||
rq.line.CoreRuleSetRuleIDs = result.RuleIDs
|
||||
rq.line.CoreRuleSetScore = &result.Score
|
||||
|
||||
for _, id := range result.RuleIDs {
|
||||
rq.h.metrics.WAFMatched(cfg.WAFMode, id)
|
||||
rq.h.metrics.CoreRuleSetMatched(cfg.WAFMode, id)
|
||||
}
|
||||
|
||||
threshold := cfg.WAFAnomalyThreshold
|
||||
if threshold == 0 || result.Score < threshold {
|
||||
return ""
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.alertWAFBlock(result)
|
||||
rq.addTag(config.ExploitSignatureTag)
|
||||
rq.alertCoreRuleSetMatch(result)
|
||||
|
||||
if cfg.WAFMode == config.WAFModeDetect {
|
||||
return ""
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.wafBlocked = true
|
||||
rq.blockedByCoreRuleSet = true
|
||||
|
||||
return requestlog.ActionWAFBlocked
|
||||
return &verdict{
|
||||
action: requestlog.ActionBlockedByCoreRuleSet,
|
||||
figures: map[string]any{figureScore: result.Score, "threshold": threshold},
|
||||
}
|
||||
}
|
||||
|
||||
// alertWAFBlock raises the waf_block alert for the request, which the Core
|
||||
// Rule Set scored at result, at or over SWW_WAF_ANOMALY_THRESHOLD. Its
|
||||
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
|
||||
// which the Core Rule Set scored at result, at or over SWW_WAF_ANOMALY_THRESHOLD. Its
|
||||
// detail gives the rule ids, the score, the method and the path with the
|
||||
// query, and, for a request that is not refused for it, the mode: detect,
|
||||
// or observe in observe mode.
|
||||
func (rq *request) alertWAFBlock(result waf.Result) {
|
||||
func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
|
||||
detail := map[string]any{
|
||||
"rule_ids": result.RuleIDs,
|
||||
"score": result.Score,
|
||||
@@ -75,7 +87,7 @@ func (rq *request) alertWAFBlock(result waf.Result) {
|
||||
}
|
||||
|
||||
rq.h.alerts.Raise(alerts.Alert{
|
||||
Event: alerts.EventWAFBlock,
|
||||
Event: alerts.EventCoreRuleSetMatch,
|
||||
Host: rq.site,
|
||||
Client: rq.client,
|
||||
Netblock: rq.h.clientGroup(rq.client),
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/discriminator"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
@@ -21,6 +22,7 @@ const (
|
||||
wafDisabledRules = "SWW_WAF_DISABLED_RULES"
|
||||
wafExemptPaths = "SWW_WAF_EXEMPT_PATHS"
|
||||
wafBodyLimit = "SWW_WAF_BODY_LIMIT"
|
||||
apps = "SWW_APPS"
|
||||
block = "block"
|
||||
detect = "detect"
|
||||
)
|
||||
@@ -40,27 +42,69 @@ const (
|
||||
// SWW_WAF_ANOMALY_THRESHOLD.
|
||||
const sqlInjection = "/?id=1'%20OR%20'1'='1"
|
||||
|
||||
// wantWAF checks the request log line's waf_rule_ids and waf_score, and
|
||||
// that it has duration_waf, or with no score, that it has none of the
|
||||
// three: the Core Rule Set did not inspect the request.
|
||||
func wantWAF(t *testing.T, line logLine, score *int, ruleIDs ...int) {
|
||||
// wantCoreRuleSet checks the request log line's core_rule_set_rule_ids and
|
||||
// core_rule_set_score, and that it has duration_core_rule_set_ms, or with no
|
||||
// score, that it has none of the three: the Core Rule Set did not inspect the
|
||||
// request.
|
||||
func wantCoreRuleSet(t *testing.T, line logLine, score *int, ruleIDs ...int) {
|
||||
t.Helper()
|
||||
|
||||
if !slices.Equal(line.WAFRuleIDs, ruleIDs) {
|
||||
t.Errorf("log line has waf_rule_ids %v, want %v", line.WAFRuleIDs, ruleIDs)
|
||||
if !slices.Equal(line.CoreRuleSetRuleIDs, ruleIDs) {
|
||||
t.Errorf("log line has core_rule_set_rule_ids %v, want %v",
|
||||
line.CoreRuleSetRuleIDs, ruleIDs)
|
||||
}
|
||||
|
||||
switch {
|
||||
case score == nil && (line.WAFScore != nil || line.DurationWAF != nil):
|
||||
t.Errorf("log line has waf_score %v and duration_waf %v, want neither",
|
||||
line.fields["waf_score"], line.fields["duration_waf"])
|
||||
case score != nil && (line.WAFScore == nil || *line.WAFScore != *score):
|
||||
t.Errorf("log line has waf_score %v, want %d", line.fields["waf_score"], *score)
|
||||
case score != nil && line.DurationWAF == nil:
|
||||
t.Error("log line has no duration_waf")
|
||||
case score == nil &&
|
||||
(line.CoreRuleSetScore != nil || line.DurationCoreRuleSetMS != nil):
|
||||
t.Errorf("log line has core_rule_set_score %v and "+
|
||||
"duration_core_rule_set_ms %v, want neither",
|
||||
line.fields["core_rule_set_score"], line.fields["duration_core_rule_set_ms"])
|
||||
case score != nil &&
|
||||
(line.CoreRuleSetScore == nil || *line.CoreRuleSetScore != *score):
|
||||
t.Errorf("log line has core_rule_set_score %v, want %d",
|
||||
line.fields["core_rule_set_score"], *score)
|
||||
case score != nil && line.DurationCoreRuleSetMS == nil:
|
||||
t.Error("log line has no duration_core_rule_set_ms")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoreRuleSetMatchAddsTheExploitSignatureTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
mode string
|
||||
status int
|
||||
action string
|
||||
}{
|
||||
{block, block, http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet},
|
||||
{detect, detect, http.StatusOK, requestlog.ActionForward},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{wafMode: tc.mode})
|
||||
|
||||
wantTags(t, s.request(client, sqlInjection, tc.status, tc.action),
|
||||
"exploitsignature")
|
||||
wantTags(t, s.get(client, http.StatusOK, requestlog.ActionForward))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetectedAttackIsRefusedByTagDenyLast(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
wafMode: detect, tagDeny: "exploitsignature",
|
||||
})
|
||||
|
||||
line := s.request(client, sqlInjection, http.StatusForbidden,
|
||||
requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, "tag_deny", `{"tag": "exploitsignature"}`)
|
||||
}
|
||||
|
||||
func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -97,19 +141,29 @@ func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(t *test
|
||||
mode, action string
|
||||
status int
|
||||
}{
|
||||
{block, requestlog.ActionWAFBlocked, http.StatusForbidden},
|
||||
{block, requestlog.ActionBlockedByCoreRuleSet, http.StatusForbidden},
|
||||
{detect, requestlog.ActionForward, http.StatusOK},
|
||||
} {
|
||||
s, _, _ := startWithClock(t, "", map[string]string{wafMode: tc.mode})
|
||||
|
||||
line := s.requestWithHeader(client, attack.path, attack.header,
|
||||
tc.status, tc.action)
|
||||
wantWAF(t, line, &attack.score, attack.ruleIDs...)
|
||||
wantCoreRuleSet(t, line, &attack.score, attack.ruleIDs...)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoreRuleSetRefusalNamesItsHeuristicScoreAndThreshold(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
|
||||
|
||||
line := s.request(client, sqlInjection, http.StatusForbidden,
|
||||
requestlog.ActionBlockedByCoreRuleSet)
|
||||
wantHeuristic(t, line, "core_rule_set", `{"score": 5, "threshold": 5}`)
|
||||
}
|
||||
|
||||
func TestOrdinaryRequestIsInspectedAndPassed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -117,7 +171,7 @@ func TestOrdinaryRequestIsInspectedAndPassed(t *testing.T) {
|
||||
|
||||
line := s.request(client, "/owner/repo/src/branch/main/README.md?display=source",
|
||||
http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, new(0))
|
||||
wantCoreRuleSet(t, line, new(0))
|
||||
}
|
||||
|
||||
func TestCoreRuleSetIsNotRunWhenOffOrForAnExemptClientPathOrRuleFileRefusal(
|
||||
@@ -137,26 +191,26 @@ func TestCoreRuleSetIsNotRunWhenOffOrForAnExemptClientPathOrRuleFileRefusal(
|
||||
// A client in SWW_ALLOW_NETS, and a path SWW_WAF_EXEMPT_PATHS
|
||||
// exempts, are not inspected.
|
||||
line := s.request(allowed, sqlInjection, http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, nil)
|
||||
wantCoreRuleSet(t, line, nil)
|
||||
line = s.request(client, "/api/v1/repos?id=1'%20OR%20'1'='1", http.StatusOK,
|
||||
requestlog.ActionForward)
|
||||
wantWAF(t, line, nil)
|
||||
wantCoreRuleSet(t, line, nil)
|
||||
|
||||
// The prefix is matched as rate limit exempt paths are: a path that
|
||||
// goes up and out of it is inspected.
|
||||
line = s.request(client, "/api/../?id=1'%20OR%20'1'='1", http.StatusForbidden,
|
||||
requestlog.ActionWAFBlocked)
|
||||
wantWAF(t, line, new(25), 930100, 930110, 942100)
|
||||
requestlog.ActionBlockedByCoreRuleSet)
|
||||
wantCoreRuleSet(t, line, new(25), 930100, 930110, 942100)
|
||||
|
||||
// A request a rule file refuses is not inspected.
|
||||
line = s.request(otherClient, "/blocked?id=1'%20OR%20'1'='1", http.StatusForbidden,
|
||||
requestlog.ActionRuleBlocked)
|
||||
wantWAF(t, line, nil)
|
||||
requestlog.ActionBlockedByBlockRule)
|
||||
wantCoreRuleSet(t, line, nil)
|
||||
|
||||
// With SWW_WAF_MODE off, no request is.
|
||||
s, _, _ = startWithClock(t, "", map[string]string{wafMode: off})
|
||||
line = s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, nil)
|
||||
wantCoreRuleSet(t, line, nil)
|
||||
}
|
||||
|
||||
func TestAnomalyThreshold(t *testing.T) {
|
||||
@@ -170,13 +224,14 @@ func TestAnomalyThreshold(t *testing.T) {
|
||||
})
|
||||
|
||||
line := s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, new(5), 942100)
|
||||
wantCoreRuleSet(t, line, new(5), 942100)
|
||||
}
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
wafMode: block, wafAnomalyThreshold: "5",
|
||||
})
|
||||
s.request(client, sqlInjection, http.StatusForbidden, requestlog.ActionWAFBlocked)
|
||||
s.request(client, sqlInjection, http.StatusForbidden,
|
||||
requestlog.ActionBlockedByCoreRuleSet)
|
||||
}
|
||||
|
||||
func TestDisabledRulesSwitchOffWhatGiteaWouldBeRefused(t *testing.T) {
|
||||
@@ -219,7 +274,7 @@ func TestDisabledRulesSwitchOffWhatGiteaWouldBeRefused(t *testing.T) {
|
||||
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
|
||||
line, _ := s.requestWithBody(request.method, client, request.path,
|
||||
request.header, body, http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, new(0))
|
||||
wantCoreRuleSet(t, line, new(0))
|
||||
|
||||
// A list given replaces the default.
|
||||
s, _, _ = startWithClock(t, "", map[string]string{
|
||||
@@ -227,16 +282,37 @@ func TestDisabledRulesSwitchOffWhatGiteaWouldBeRefused(t *testing.T) {
|
||||
})
|
||||
score := 5 * len(request.ruleIDs)
|
||||
line, _ = s.requestWithBody(request.method, client, request.path,
|
||||
request.header, body, http.StatusForbidden, requestlog.ActionWAFBlocked)
|
||||
wantWAF(t, line, &score, request.ruleIDs...)
|
||||
request.header, body, http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet)
|
||||
wantCoreRuleSet(t, line, &score, request.ruleIDs...)
|
||||
|
||||
// And switches off the rules it lists.
|
||||
line = s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, new(0))
|
||||
wantCoreRuleSet(t, line, new(0))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGiteasFixesApplyOnlyToAHostnameThatServesGitea(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Only gitea's fixes let a file name through in path, a parameter
|
||||
// gitea sends file names in.
|
||||
const fileName = "/?path=.gitignore"
|
||||
|
||||
s := startDiscriminator(t, "", map[string]string{
|
||||
wafMode: block, apps: siteA + ":gitea",
|
||||
})
|
||||
|
||||
wantDisposition(t, do(t, s.question(siteA, client, fileName)),
|
||||
http.StatusOK, discriminator.DispositionAllow)
|
||||
wantDisposition(t, do(t, s.question(siteB, client, fileName)),
|
||||
http.StatusForbidden, discriminator.DispositionDeny)
|
||||
|
||||
lines := s.out.requestLines(t, 2)
|
||||
wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward)
|
||||
wantLine(t, lines[1], http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet)
|
||||
}
|
||||
|
||||
func TestAttackInAFormBodyIsRefusedOnlyWhileBodiesAreRead(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -248,14 +324,14 @@ func TestAttackInAFormBodyIsRefusedOnlyWhileBodiesAreRead(t *testing.T) {
|
||||
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
|
||||
line, _ := s.requestWithBody(http.MethodPost, client, "/", header, body,
|
||||
http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, new(0))
|
||||
wantCoreRuleSet(t, line, new(0))
|
||||
|
||||
s, _, _ = startWithClock(t, "", map[string]string{
|
||||
wafMode: block, wafBodyLimit: sizeLimitSetting,
|
||||
})
|
||||
line, _ = s.requestWithBody(http.MethodPost, client, "/", header, body,
|
||||
http.StatusForbidden, requestlog.ActionWAFBlocked)
|
||||
wantWAF(t, line, new(5), 942100)
|
||||
http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet)
|
||||
wantCoreRuleSet(t, line, new(5), 942100)
|
||||
}
|
||||
|
||||
func TestBodiesReachTheAppAsSentWhileBodiesAreRead(t *testing.T) {
|
||||
@@ -345,32 +421,33 @@ func TestCoreRuleSetRefusalIsAnOffenceAndCountsTowardTheErrorBurst(t *testing.T)
|
||||
})
|
||||
|
||||
for range 2 {
|
||||
s.request(client, sqlInjection, http.StatusForbidden, requestlog.ActionWAFBlocked)
|
||||
s.request(client, sqlInjection, http.StatusForbidden,
|
||||
requestlog.ActionBlockedByCoreRuleSet)
|
||||
}
|
||||
|
||||
// The third refusal in a minute breaks the error burst, and bans the
|
||||
// client.
|
||||
line := s.request(client, sqlInjection, http.StatusForbidden,
|
||||
requestlog.ActionWAFBlocked)
|
||||
if line.LimitHit != requestlog.LimitHitErrorBurst ||
|
||||
requestlog.ActionBlockedByCoreRuleSet)
|
||||
if line.LimitHit != requestlog.LimitHitMinuteRefusals ||
|
||||
line.Offence != requestlog.OffenceLimit {
|
||||
t.Errorf("log line has limit_hit %q and offence %q, want error_burst and limit",
|
||||
t.Errorf("log line has limit_hit %q and offence %q, want minute_refusals and limit",
|
||||
line.LimitHit, line.Offence)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
want := ratelimit.Offences{Limit: 1, WAFBlocked: 3}
|
||||
want := ratelimit.Offences{Limit: 1, BlockedByCoreRuleSet: 3}
|
||||
if offences := historyOf(t, server, client).Offences; offences != want {
|
||||
t.Errorf("history counts the offences %+v, want %+v", offences, want)
|
||||
}
|
||||
|
||||
metrics := s.scrape()
|
||||
wantMetric(t, metrics,
|
||||
`sww_waf_matches_total{instance="app",mode="block",rule_id="942100"}`, 3)
|
||||
`sww_core_rule_set_matches_total{instance="app",mode="block",rule_id="942100"}`, 3)
|
||||
wantMetric(t, metrics,
|
||||
`sww_offences_total{instance="app",kind="waf_blocked"}`, 3)
|
||||
wantMetric(t, metrics, `sww_requests_total{action="waf_blocked",`+
|
||||
`sww_offences_total{instance="app",kind="blocked_by_core_rule_set"}`, 3)
|
||||
wantMetric(t, metrics, `sww_requests_total{action="blocked_by_core_rule_set",`+
|
||||
`instance="app",status_class="4xx"}`, 3)
|
||||
}
|
||||
|
||||
@@ -394,9 +471,9 @@ func TestDetectModeMatchIsNoOffenceAndNotCountedTowardTheErrorBurst(t *testing.T
|
||||
|
||||
metrics := s.scrape()
|
||||
wantMetric(t, metrics,
|
||||
`sww_waf_matches_total{instance="app",mode="detect",rule_id="942100"}`, 3)
|
||||
`sww_core_rule_set_matches_total{instance="app",mode="detect",rule_id="942100"}`, 3)
|
||||
wantNoSeries(t, metrics,
|
||||
`sww_offences_total{instance="app",kind="waf_blocked"}`)
|
||||
`sww_offences_total{instance="app",kind="blocked_by_core_rule_set"}`)
|
||||
}
|
||||
|
||||
func TestObserveModeLogsWhatTheCoreRuleSetWouldDo(t *testing.T) {
|
||||
@@ -405,17 +482,17 @@ func TestObserveModeLogsWhatTheCoreRuleSetWouldDo(t *testing.T) {
|
||||
s, _, server := startWithClock(t, "", map[string]string{wafMode: block, mode: observe})
|
||||
|
||||
line := s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionWAFBlocked)
|
||||
wantWAF(t, line, new(5), 942100)
|
||||
wantWouldAction(t, line, requestlog.ActionBlockedByCoreRuleSet)
|
||||
wantCoreRuleSet(t, line, new(5), 942100)
|
||||
|
||||
// It is an offence as in enforce mode.
|
||||
want := ratelimit.Offences{WAFBlocked: 1}
|
||||
want := ratelimit.Offences{BlockedByCoreRuleSet: 1}
|
||||
if offences := historyOf(t, server, client).Offences; offences != want {
|
||||
t.Errorf("history counts the offences %+v, want %+v", offences, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoreRuleSetMatchRaisesTheWAFBlockAlert(t *testing.T) {
|
||||
func TestCoreRuleSetMatchRaisesTheCoreRuleSetMatchAlert(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
@@ -429,7 +506,7 @@ func TestCoreRuleSetMatchRaisesTheWAFBlockAlert(t *testing.T) {
|
||||
}{
|
||||
{
|
||||
"block", map[string]string{wafMode: block},
|
||||
http.StatusForbidden, requestlog.ActionWAFBlocked, "",
|
||||
http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet, "",
|
||||
},
|
||||
{
|
||||
"detect", map[string]string{wafMode: detect},
|
||||
@@ -464,7 +541,7 @@ func TestCoreRuleSetMatchRaisesTheWAFBlockAlert(t *testing.T) {
|
||||
wantAlerts(t, queue, alerts.Alert{
|
||||
Instance: alertInstance,
|
||||
Time: clk.Now(),
|
||||
Event: alerts.EventWAFBlock,
|
||||
Event: alerts.EventCoreRuleSetMatch,
|
||||
Host: appHost,
|
||||
Client: netip.MustParseAddr(client),
|
||||
Netblock: netip.MustParsePrefix(client + "/32"),
|
||||
|
||||
@@ -1,21 +1,35 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic countries. It reads SWW_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// countryDenied reports whether the country lists refuse the request, by
|
||||
// the client's country as it was looked up. A client without a country,
|
||||
// or whose country cannot be found, is refused only by
|
||||
// SWW_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||
func (rq *request) countryDenied() bool {
|
||||
denied := rq.h.config.DeniedCountries
|
||||
// countries refuses a request when SWW_EXCLUSIVELY_ALLOWED_COUNTRIES is set and
|
||||
// does not hold the client's country as it was looked up. A client without a
|
||||
// country, or whose country cannot be found, is refused. Its figures are the
|
||||
// country and the setting that refused it.
|
||||
type countries struct{}
|
||||
|
||||
func (countries) name() string { return "countries" }
|
||||
|
||||
func (countries) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
allowed := rq.h.config.ExclusivelyAllowedCountries
|
||||
country := rq.line.Country
|
||||
|
||||
if slices.Contains(denied, country) {
|
||||
return true
|
||||
if len(allowed) == 0 || slices.Contains(allowed, country) {
|
||||
return nil
|
||||
}
|
||||
|
||||
return len(allowed) > 0 && !slices.Contains(allowed, country)
|
||||
return &verdict{
|
||||
action: requestlog.ActionCountryDenied,
|
||||
figures: map[string]any{
|
||||
"country": country, "setting": "SWW_EXCLUSIVELY_ALLOWED_COUNTRIES",
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -33,13 +33,13 @@ func TestCountryLists(t *testing.T) {
|
||||
env map[string]string
|
||||
refused []string
|
||||
}{
|
||||
{"denied", map[string]string{deniedCountries: "kp"}, []string{fromKP}},
|
||||
{
|
||||
"exclusively allowed", map[string]string{allowedCountries: "DE"},
|
||||
[]string{fromKP, unplaced},
|
||||
},
|
||||
{
|
||||
"both", map[string]string{deniedCountries: "kp", allowedCountries: "de,fr"},
|
||||
"with a country in SWW_TAG_DENY",
|
||||
map[string]string{tagDeny: denyKP, allowedCountries: "de,fr"},
|
||||
[]string{fromKP, unplaced},
|
||||
},
|
||||
} {
|
||||
@@ -90,6 +90,23 @@ func TestCountryLists(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCountryRefusalNamesItsHeuristicCountryAndSetting(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
geojsURL, _ := startGeoJS(t)
|
||||
s, _, _ := startWithClock(t, geojsURL, map[string]string{
|
||||
lookupTimeout: "1h", allowedCountries: "de",
|
||||
})
|
||||
|
||||
line := s.get(fromKP, http.StatusForbidden, requestlog.ActionCountryDenied)
|
||||
wantHeuristic(t, line, "countries",
|
||||
`{"country": "KP", "setting": "SWW_EXCLUSIVELY_ALLOWED_COUNTRIES"}`)
|
||||
|
||||
line = s.get(unplaced, http.StatusForbidden, requestlog.ActionCountryDenied)
|
||||
wantHeuristic(t, line, "countries",
|
||||
`{"country": "", "setting": "SWW_EXCLUSIVELY_ALLOWED_COUNTRIES"}`)
|
||||
}
|
||||
|
||||
func TestCountryRefusalKeepsARequestFromTheApp(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -102,7 +119,7 @@ func TestCountryRefusalKeepsARequestFromTheApp(t *testing.T) {
|
||||
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
lookupTimeout: "1h",
|
||||
deniedCountries: "kp",
|
||||
allowedCountries: "de",
|
||||
})
|
||||
|
||||
req := newRequest(t, http.MethodPost, addr, "/", strings.NewReader("a body"))
|
||||
@@ -185,7 +202,7 @@ func TestPrivateAddressIsNeverLookedUp(t *testing.T) {
|
||||
env map[string]string
|
||||
}{
|
||||
{"no setting needs the lookup", nil},
|
||||
{"a country list is set", map[string]string{deniedCountries: "kp"}},
|
||||
{"a tag setting needs it", map[string]string{tagDeny: denyKP}},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic crowdsec. It reads SWW_CROWDSEC_LAPI_URL, and the decisions
|
||||
// through the reputation lists.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// crowdSec refuses a request when a decision of the CrowdSec decision list on
|
||||
// the client is in force at now. It then adds the tag crowdsec, notes the
|
||||
// list, as noteHit does, and bans the client until that decision ends. Its
|
||||
// figures are the decision's scenario and when it ends.
|
||||
type crowdSec struct{}
|
||||
|
||||
func (crowdSec) name() string { return "crowdsec" }
|
||||
|
||||
func (crowdSec) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
decision, listed := rq.crowdSecDecisionInForce(now)
|
||||
if !listed {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.addTag(config.CrowdSecTag)
|
||||
rq.noteHit(bans.ReputationHit{Source: rq.h.config.CrowdSecDecisionsURL},
|
||||
"listed by the CrowdSec decision list")
|
||||
|
||||
return &verdict{
|
||||
action: rq.banForCrowdSec(now, decision),
|
||||
figures: map[string]any{
|
||||
"scenario": decision.Scenario,
|
||||
"expires": requestlog.FormatTime(decision.Expires),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// crowdSecDecisionInForce returns the decision of the CrowdSec decision list on
|
||||
// the client that is in force at now, and reports whether there is one, which
|
||||
// the heuristic crowdsec refuses the request for.
|
||||
func (rq *request) crowdSecDecisionInForce(
|
||||
now time.Time,
|
||||
) (reputation.Decision, bool) {
|
||||
return rq.h.lists.CrowdSecDecision(rq.client, now)
|
||||
}
|
||||
@@ -39,7 +39,7 @@ func TestClientTheCrowdSecDecisionListListsIsBannedUntilTheDecisionEnds(t *testi
|
||||
expires := requestlog.FormatTime(fetched.Add(4 * time.Hour))
|
||||
|
||||
// Its first request is refused, and bans it until the decision ends.
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
wantReputation(t, line, decisionsURL)
|
||||
|
||||
if line.BanExpires != expires {
|
||||
@@ -72,7 +72,7 @@ func TestClientTheCrowdSecDecisionListListsIsBannedUntilTheDecisionEnds(t *testi
|
||||
// ban, and once it has ended the client is let through.
|
||||
clk.advance(4*time.Hour - time.Second)
|
||||
|
||||
line = s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
line = s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
wantReputation(t, line)
|
||||
|
||||
if line.BanExpires != expires {
|
||||
@@ -95,6 +95,22 @@ func TestClientTheCrowdSecDecisionListListsIsBannedUntilTheDecisionEnds(t *testi
|
||||
float64(fetched.Unix()))
|
||||
}
|
||||
|
||||
func TestCrowdSecBanNamesItsHeuristicScenarioAndEnd(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server, _ := startWithAlerts(t, map[string]string{
|
||||
crowdSecURL: lapi, crowdSecKey: bouncerKey,
|
||||
})
|
||||
loadDecisions(t, server, clk.Now(), `[{"duration": "4h0m0s", `+
|
||||
`"scenario": "crowdsecurity/ssh-bf", "scope": "Ip", "type": "ban", `+
|
||||
`"value": "`+client+`"}]`)
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
wantHeuristic(t, line, "crowdsec", `{"scenario": "crowdsecurity/ssh-bf", `+
|
||||
`"expires": "`+requestlog.FormatTime(clk.Now().Add(4*time.Hour))+`"}`)
|
||||
wantTags(t, line, "crowdsec")
|
||||
}
|
||||
|
||||
func TestEndedCrowdSecDecisionNoLongerBansThoughTheCopyStillHoldsIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -113,9 +129,9 @@ func TestEndedCrowdSecDecisionNoLongerBansThoughTheCopyStillHoldsIt(t *testing.T
|
||||
// Just before its end, the decision bans a client in the netblock, and
|
||||
// one on an IPv6 address bans the address's group, the /64.
|
||||
clk.advance(time.Minute - time.Nanosecond)
|
||||
s.get("198.51.100.7", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get("2001:db8::9", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get("2001:db8::5", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get("198.51.100.7", http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
s.get("2001:db8::9", http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
s.get("2001:db8::5", http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
// Once it has ended, it bans no other client, and the bans it made end
|
||||
// with it.
|
||||
@@ -148,7 +164,7 @@ func TestObserveModeForwardsAClientTheCrowdSecDecisionListListsAndAlertsTheBan(
|
||||
`"value": "`+client+`"}]`)
|
||||
|
||||
line := s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionBanned)
|
||||
wantWouldAction(t, line, requestlog.ActionTemporarilyBanned)
|
||||
wantReputation(t, line, decisionsURL)
|
||||
|
||||
if held := server.Ledger.Snapshot(); len(held) != 0 {
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic deny_nets. It reads SWW_DENY_NETS.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// denyNets refuses a client in SWW_DENY_NETS. Its figure is the netblock of
|
||||
// the setting the client is in.
|
||||
type denyNets struct{}
|
||||
|
||||
func (denyNets) name() string { return "deny_nets" }
|
||||
|
||||
func (denyNets) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
for _, netblock := range rq.h.config.DenyNets {
|
||||
if netblock.Contains(rq.client) {
|
||||
return &verdict{
|
||||
action: requestlog.ActionClientDenied,
|
||||
figures: map[string]any{figureNetblock: netblock.String()},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package discriminator
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -52,13 +53,6 @@ const ReportPath = "/_sww/report"
|
||||
// QuestionBodyMaxBytes is the most of a visitor's body a question carries.
|
||||
const QuestionBodyMaxBytes = 64 << 10
|
||||
|
||||
// The header limits: the most cookies and header fields a request may have
|
||||
// when the checks reach the Core Rule Set. Real clients never reach them.
|
||||
const (
|
||||
maxCookies = 50
|
||||
maxHeaderFields = 100
|
||||
)
|
||||
|
||||
const (
|
||||
// maxAwaitingReport is the most answered requests held until their
|
||||
// reports come. Past it, the request answered longest ago is dropped.
|
||||
@@ -99,8 +93,8 @@ type Report struct {
|
||||
Status int `json:"status"`
|
||||
RequestBytes int64 `json:"request_bytes"`
|
||||
ResponseBytes int64 `json:"response_bytes"`
|
||||
DurationTotal float64 `json:"duration_total"`
|
||||
DurationUpstreamTotal float64 `json:"duration_upstream_total"`
|
||||
DurationTotalMS float64 `json:"duration_total_ms"`
|
||||
DurationUpstreamTotalMS float64 `json:"duration_upstream_total_ms"`
|
||||
}
|
||||
|
||||
// New returns the server `sww discriminator` runs. It answers
|
||||
@@ -129,6 +123,7 @@ func New(params Params) *Server {
|
||||
Lists: h.lists,
|
||||
DNSBL: h.dnsbl,
|
||||
AbuseIPDB: h.abuseIPDB,
|
||||
RouteWatch: h.routeWatch,
|
||||
Metrics: h.metrics,
|
||||
}
|
||||
}
|
||||
@@ -148,48 +143,20 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
// refusedWhateverTheAnswer reports whether the request is one that does not wait
|
||||
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
|
||||
// its client: it is over the header limits, a blocklist lists its client while
|
||||
// SWW_BLOCKLIST_ACTION is deny, the CrowdSec decision list lists it, or it is
|
||||
// over a whole rate limit. In observe mode, which refuses nothing, none is.
|
||||
// rproxy lets a request through when discriminator answers too late.
|
||||
// its client: the heuristic crowdsec, header_limits, blocklists or rate_limits
|
||||
// will refuse it, as a function in that heuristic's file says. In observe mode,
|
||||
// which refuses nothing, none is. rproxy lets a request through when
|
||||
// discriminator answers too late.
|
||||
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
|
||||
if rq.h.config.Observe {
|
||||
return false
|
||||
}
|
||||
|
||||
lists := rq.h.lists
|
||||
_, decided := lists.CrowdSecDecision(rq.client, now)
|
||||
_, decided := rq.crowdSecDecisionInForce(now)
|
||||
|
||||
return decided || rq.overHeaderLimits() ||
|
||||
(rq.h.config.BlocklistAction == deny && len(lists.ListedBy(rq.client)) > 0) ||
|
||||
(rq.countedByRateLimits() &&
|
||||
rq.h.limiter.OverARateLimit(rq.h.clientGroup(rq.client), now))
|
||||
}
|
||||
|
||||
// overHeaderLimits reports whether discriminator refuses the request for its
|
||||
// headers: it has more than maxHeaderFields fields, each value of a header sent
|
||||
// twice being one, or more than maxCookies cookies, each part of a Cookie header
|
||||
// between semicolons being one unless it is empty or only spaces and tabs. The
|
||||
// Core Rule Set runs each of its rules on each cookie, so that thousands of small
|
||||
// cookies keep it busy for over a second. It trims only spaces and control
|
||||
// characters, of which a header value can hold only tabs, so a part of Unicode
|
||||
// spaces is a cookie to it.
|
||||
func (rq *request) overHeaderLimits() bool {
|
||||
fields, cookies := 0, 0
|
||||
|
||||
for _, values := range rq.in.Header {
|
||||
fields += len(values)
|
||||
}
|
||||
|
||||
for _, value := range rq.in.Header.Values("Cookie") {
|
||||
for part := range strings.SplitSeq(value, ";") {
|
||||
if strings.Trim(part, " \t") != "" {
|
||||
cookies++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return fields > maxHeaderFields || cookies > maxCookies
|
||||
return decided || overHeaderLimits(rq.headerCounts()) ||
|
||||
rq.refusedByBlocklists(rq.h.lists.ListedBy(rq.client)) ||
|
||||
rq.overAWholeRateLimit(now)
|
||||
}
|
||||
|
||||
// answerQuestion runs the checks on the visitor's request a question
|
||||
@@ -216,22 +183,22 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
|
||||
rq.line.DiscriminatorLine = requestlog.DiscriminatorLineDecision
|
||||
refused := rq.check(r.Context())
|
||||
rq.checked = time.Now()
|
||||
underOwnPrefix := strings.HasPrefix(r.URL.Path, adminPrefix)
|
||||
underOwnPrefix := strings.HasPrefix(r.URL.Path, ownEndpointPrefix)
|
||||
|
||||
if refused == nil && !underOwnPrefix {
|
||||
// In observe mode, what enforce mode would have refused counts here.
|
||||
defer rq.countRefusal()
|
||||
defer rq.runAfterAnswer(r.Context())
|
||||
|
||||
rq.allow()
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// The last deferred runs first: countRefusal before addToHistory, so that a
|
||||
// broken error burst is in the client's history.
|
||||
// The last deferred runs first: runAfterAnswer before addToHistory, so that
|
||||
// a broken refusal limit is in the client's history.
|
||||
defer rq.addToHistory()
|
||||
defer rq.countAnomalies()
|
||||
defer rq.countRefusal()
|
||||
defer rq.runAfterAnswer(r.Context())
|
||||
|
||||
if refused != nil {
|
||||
rq.deny(*refused)
|
||||
@@ -241,7 +208,7 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
// The app never sees this prefix. discriminator's own endpoints are at its
|
||||
// own address, so this answers as a path with no endpoint does.
|
||||
rq.line.Action = requestlog.ActionAdmin
|
||||
rq.line.Action = requestlog.ActionOwnEndpoint
|
||||
rq.out.Header().Set(DispositionHeader, DispositionDeny)
|
||||
http.Error(rq.out, http.StatusText(http.StatusNotFound), http.StatusNotFound)
|
||||
}
|
||||
@@ -413,7 +380,7 @@ func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
|
||||
// Remove reports true to one report only, of several that name the id. A
|
||||
// question still being handled is held as nil.
|
||||
if found && held != nil && h.awaitingReport.Remove(report.RequestID) {
|
||||
h.finishWith(held, report)
|
||||
h.finishWith(r.Context(), held, report)
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
@@ -438,7 +405,7 @@ func readReport(w http.ResponseWriter, r *http.Request) (Report, error) {
|
||||
|
||||
for _, name := range []string{
|
||||
"request_id", "status", "request_bytes", "response_bytes",
|
||||
"duration_total", "duration_upstream_total",
|
||||
"duration_total_ms", "duration_upstream_total_ms",
|
||||
} {
|
||||
value, there := fields[name]
|
||||
if !there || string(value) == "null" {
|
||||
@@ -464,11 +431,11 @@ func (report *Report) checkRanges() error {
|
||||
return fmt.Errorf("%w: request_bytes", errReportValue)
|
||||
case report.ResponseBytes < 0 || report.ResponseBytes > maxReportedBytes:
|
||||
return fmt.Errorf("%w: response_bytes", errReportValue)
|
||||
case report.DurationTotal < 0 || report.DurationTotal > maxReportedMilliseconds:
|
||||
return fmt.Errorf("%w: duration_total", errReportValue)
|
||||
case report.DurationUpstreamTotal < 0 ||
|
||||
report.DurationUpstreamTotal > maxReportedMilliseconds:
|
||||
return fmt.Errorf("%w: duration_upstream_total", errReportValue)
|
||||
case report.DurationTotalMS < 0 || report.DurationTotalMS > maxReportedMilliseconds:
|
||||
return fmt.Errorf("%w: duration_total_ms", errReportValue)
|
||||
case report.DurationUpstreamTotalMS < 0 ||
|
||||
report.DurationUpstreamTotalMS > maxReportedMilliseconds:
|
||||
return fmt.Errorf("%w: duration_upstream_total_ms", errReportValue)
|
||||
}
|
||||
|
||||
return nil
|
||||
@@ -476,11 +443,11 @@ func (report *Report) checkRanges() error {
|
||||
|
||||
// finishWith finishes a request with what its report says, and writes the
|
||||
// request's last log line, which holds its decision's fields and the report's.
|
||||
// For an allowed request it first counts the request's bytes, and adds it to its
|
||||
// client's history, the anomaly counters and the metrics, with the status and
|
||||
// bytes the report gives. A denied request's counting was finished at its
|
||||
// decision.
|
||||
func (h *handler) finishWith(held *decision, report Report) {
|
||||
// For an allowed request it first runs the heuristics of onReport, and adds the
|
||||
// request to its client's history, the anomaly counters and the metrics, with
|
||||
// the status and bytes the report gives. A denied request's counting was
|
||||
// finished at its decision. ctx is the report's own context.
|
||||
func (h *handler) finishWith(ctx context.Context, held *decision, report Report) {
|
||||
rq := &request{
|
||||
decision: *held,
|
||||
h: h,
|
||||
@@ -492,18 +459,18 @@ func (h *handler) finishWith(held *decision, report Report) {
|
||||
rq.line.Status = report.Status
|
||||
rq.line.RequestBytes = report.RequestBytes
|
||||
rq.line.ResponseBytes = report.ResponseBytes
|
||||
rq.line.DurationTotal = report.DurationTotal
|
||||
rq.line.DurationTotalMS = report.DurationTotalMS
|
||||
|
||||
if rq.allowed {
|
||||
rq.line.DurationUpstreamTotal = &report.DurationUpstreamTotal
|
||||
rq.line.DurationUpstreamTotalMS = &report.DurationUpstreamTotalMS
|
||||
|
||||
rq.countBytes()
|
||||
rq.runBanningHeuristics(ctx, onReport())
|
||||
rq.countAnomalies()
|
||||
rq.addToHistory()
|
||||
|
||||
rq.h.metrics.RequestReported(&rq.line,
|
||||
time.Duration(report.DurationTotal*float64(time.Millisecond)),
|
||||
time.Duration(report.DurationUpstreamTotal*float64(time.Millisecond)))
|
||||
time.Duration(report.DurationTotalMS*float64(time.Millisecond)),
|
||||
time.Duration(report.DurationUpstreamTotalMS*float64(time.Millisecond)))
|
||||
}
|
||||
|
||||
rq.writeLine()
|
||||
|
||||
@@ -48,8 +48,8 @@ const (
|
||||
reportStatus = "status"
|
||||
reportRequestBytes = "request_bytes"
|
||||
reportResponseBytes = "response_bytes"
|
||||
reportDuration = "duration_total"
|
||||
reportUpstreamDuration = "duration_upstream_total"
|
||||
reportDuration = "duration_total_ms"
|
||||
reportUpstreamDuration = "duration_upstream_total_ms"
|
||||
)
|
||||
|
||||
// Discriminator is a discriminator running for one test, on a clock set to midnight.
|
||||
@@ -414,7 +414,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
|
||||
|
||||
geojsURL, asked, release := startHeldGeoJS(t)
|
||||
s := startDiscriminator(t, geojsURL, map[string]string{
|
||||
lookupTimeout: "1h", deniedCountries: "kp", bytesLimitPerMinute: byteLimit,
|
||||
lookupTimeout: "1h", tagDeny: denyKP, bytesLimitPerMinute: byteLimit,
|
||||
})
|
||||
|
||||
first := s.question(siteA, client, "/")
|
||||
@@ -639,7 +639,7 @@ func TestQuestionDeniedGetsTheBanResponse(t *testing.T) {
|
||||
got := s.ask(client, "/")
|
||||
wantDisposition(t, got, status, discriminator.DispositionDeny)
|
||||
wantBody(t, got, http.StatusText(status)+"\n")
|
||||
wantLine(t, s.out.requestLine(t), status, requestlog.ActionDenied)
|
||||
wantLine(t, s.out.requestLine(t), status, requestlog.ActionClientDenied)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -655,8 +655,8 @@ func TestBanResponseCloseIsAnAnswerOfItsOwn(t *testing.T) {
|
||||
wantDisposition(t, got, http.StatusOK, discriminator.DispositionClose)
|
||||
wantBody(t, got, "")
|
||||
// The visitor is sent nothing.
|
||||
wantLine(t, s.out.requestLine(t), 0, requestlog.ActionDenied)
|
||||
wantMetric(t, s.metrics(), `sww_requests_total{action="denied",instance="`+
|
||||
wantLine(t, s.out.requestLine(t), 0, requestlog.ActionClientDenied)
|
||||
wantMetric(t, s.metrics(), `sww_requests_total{action="client_denied",instance="`+
|
||||
alertInstance+`",status_class="none"}`, 1)
|
||||
}
|
||||
|
||||
@@ -671,12 +671,12 @@ func TestBlockRuleIsDeniedWith403AndABanRuleWithTheBanResponse(t *testing.T) {
|
||||
got := s.ask(client, "/blocked")
|
||||
wantDisposition(t, got, http.StatusForbidden, discriminator.DispositionDeny)
|
||||
wantLine(t, s.out.requestLine(t), http.StatusForbidden,
|
||||
requestlog.ActionRuleBlocked)
|
||||
requestlog.ActionBlockedByBlockRule)
|
||||
|
||||
got = s.ask(client, "/.env")
|
||||
wantDisposition(t, got, http.StatusTooManyRequests, discriminator.DispositionDeny)
|
||||
wantLine(t, s.out.requestLines(t, 2)[1], http.StatusTooManyRequests,
|
||||
requestlog.ActionBanned)
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(t *testing.T) {
|
||||
@@ -693,7 +693,7 @@ func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(t *testing.T) {
|
||||
wantDisposition(t, got, http.StatusNotFound, discriminator.DispositionDeny)
|
||||
}
|
||||
|
||||
wantLine(t, s.out.requestLine(t), http.StatusNotFound, requestlog.ActionAdmin)
|
||||
wantLine(t, s.out.requestLine(t), http.StatusNotFound, requestlog.ActionOwnEndpoint)
|
||||
|
||||
got := s.ask(otherClient, discriminator.MetricsPath)
|
||||
wantDisposition(t, got, http.StatusForbidden, discriminator.DispositionDeny)
|
||||
@@ -709,7 +709,7 @@ func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) {
|
||||
|
||||
line := s.out.requestLine(t)
|
||||
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionDenied)
|
||||
wantWouldAction(t, line, requestlog.ActionClientDenied)
|
||||
}
|
||||
|
||||
func TestObserveModeRefusalAnsweredAllowCountsForTheErrorBurst(t *testing.T) {
|
||||
@@ -725,9 +725,9 @@ func TestObserveModeRefusalAnsweredAllowCountsForTheErrorBurst(t *testing.T) {
|
||||
}
|
||||
|
||||
lines := s.out.requestLines(t, 2)
|
||||
if lines[0].LimitHit != "" || lines[1].LimitHit != requestlog.LimitHitErrorBurst {
|
||||
if lines[0].LimitHit != "" || lines[1].LimitHit != requestlog.LimitHitMinuteRefusals {
|
||||
t.Errorf("log lines have limit_hit %q and %q, want none and %q",
|
||||
lines[0].LimitHit, lines[1].LimitHit, requestlog.LimitHitErrorBurst)
|
||||
lines[0].LimitHit, lines[1].LimitHit, requestlog.LimitHitMinuteRefusals)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -744,9 +744,9 @@ func TestDeniedQuestionCountsForTheErrorBurst(t *testing.T) {
|
||||
}
|
||||
|
||||
lines := s.out.requestLines(t, 2)
|
||||
if lines[0].LimitHit != "" || lines[1].LimitHit != requestlog.LimitHitErrorBurst {
|
||||
if lines[0].LimitHit != "" || lines[1].LimitHit != requestlog.LimitHitMinuteRefusals {
|
||||
t.Errorf("log lines have limit_hit %q and %q, want none and %q",
|
||||
lines[0].LimitHit, lines[1].LimitHit, requestlog.LimitHitErrorBurst)
|
||||
lines[0].LimitHit, lines[1].LimitHit, requestlog.LimitHitMinuteRefusals)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -765,7 +765,7 @@ func TestAllowCarriesTheLookupHeadersWhileTheSettingIsSet(t *testing.T) {
|
||||
|
||||
geojsURL, _ := startGeoJS(t)
|
||||
s := startDiscriminator(t, geojsURL, map[string]string{
|
||||
addLookupHeaders: tc.setting, lookupTimeout: "1h", deniedCountries: "kp",
|
||||
addLookupHeaders: tc.setting, lookupTimeout: "1h", tagDeny: denyKP,
|
||||
})
|
||||
|
||||
got := s.ask(fromDE, "/")
|
||||
@@ -894,14 +894,14 @@ func TestDeniedRequestIsFinishedAtItsDecision(t *testing.T) {
|
||||
t.Errorf("history is %+v, want the refused request alone", history)
|
||||
}
|
||||
|
||||
const series = `sww_requests_total{action="rate_limited",instance="` +
|
||||
const series = `sww_requests_total{action="client_is_rate_limited",instance="` +
|
||||
alertInstance + `",status_class="4xx"}`
|
||||
|
||||
wantMetric(t, s.metrics(), series, 1)
|
||||
|
||||
got := s.report(discriminator.Report{
|
||||
RequestID: denied.Header.Get(requestIDHeader), Status: http.StatusForbidden,
|
||||
ResponseBytes: 1000, DurationTotal: 2,
|
||||
ResponseBytes: 1000, DurationTotalMS: 2,
|
||||
})
|
||||
wantStatus(t, got, http.StatusNoContent)
|
||||
|
||||
@@ -935,7 +935,7 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
|
||||
got := s.report(discriminator.Report{
|
||||
RequestID: id, Status: http.StatusBadGateway,
|
||||
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
|
||||
DurationTotal: 5, DurationUpstreamTotal: 4,
|
||||
DurationTotalMS: 5, DurationUpstreamTotalMS: 4,
|
||||
})
|
||||
wantStatus(t, got, http.StatusNoContent)
|
||||
|
||||
@@ -951,7 +951,8 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
|
||||
unscraped := startDiscriminator(t, "", map[string]string{metricsToken: token})
|
||||
|
||||
got = unscraped.report(discriminator.Report{
|
||||
RequestID: unscraped.allowed(client), Status: http.StatusBadGateway, DurationTotal: 5,
|
||||
RequestID: unscraped.allowed(client), Status: http.StatusBadGateway,
|
||||
DurationTotalMS: 5,
|
||||
})
|
||||
wantStatus(t, got, http.StatusNoContent)
|
||||
wantMetric(t, unscraped.metrics(),
|
||||
@@ -1189,8 +1190,8 @@ func wantReportsFields(t *testing.T, line logLine) {
|
||||
if line.DiscriminatorLine != requestlog.DiscriminatorLineReport ||
|
||||
line.Status != http.StatusCreated ||
|
||||
line.RequestBytes != bodyBytes || line.ResponseBytes != answerBytes ||
|
||||
line.DurationTotal != 12.5 || line.DurationUpstreamTotal == nil ||
|
||||
*line.DurationUpstreamTotal != 10.25 {
|
||||
line.DurationTotalMS != 12.5 || line.DurationUpstreamTotalMS == nil ||
|
||||
*line.DurationUpstreamTotalMS != 10.25 {
|
||||
t.Errorf("the report's line lacks the report's fields: %+v", line.Line)
|
||||
}
|
||||
}
|
||||
@@ -1205,8 +1206,8 @@ func wantSameDecision(t *testing.T, reported, decided logLine) {
|
||||
same.Status = decided.Status
|
||||
same.RequestBytes = decided.RequestBytes
|
||||
same.ResponseBytes = decided.ResponseBytes
|
||||
same.DurationTotal = decided.DurationTotal
|
||||
same.DurationUpstreamTotal = decided.DurationUpstreamTotal
|
||||
same.DurationTotalMS = decided.DurationTotalMS
|
||||
same.DurationUpstreamTotalMS = decided.DurationUpstreamTotalMS
|
||||
same.Counts.MinuteBytes = decided.Counts.MinuteBytes
|
||||
same.Counts.HourBytes = decided.Counts.HourBytes
|
||||
same.Counts.DayBytes = decided.Counts.DayBytes
|
||||
@@ -1224,7 +1225,7 @@ func wantDecisionsFields(t *testing.T, line logLine, id, query, accept string) {
|
||||
|
||||
if line.DiscriminatorLine != requestlog.DiscriminatorLineDecision ||
|
||||
line.Status != http.StatusOK || line.ResponseBytes != 0 ||
|
||||
line.DurationUpstreamTotal != nil || line.DurationChecks == nil ||
|
||||
line.DurationUpstreamTotalMS != nil || line.DurationChecksMS == nil ||
|
||||
line.RequestID != id || line.Query != query ||
|
||||
!line.HasAuthorization || !line.HasCookie ||
|
||||
line.RequestHeaders["accept"] != accept || len(line.RuleIDs) != 1 ||
|
||||
@@ -1260,7 +1261,7 @@ func TestReportWritesTheLastLineWithTheDecisionsAndTheReportsFields(t *testing.T
|
||||
got := s.report(discriminator.Report{
|
||||
RequestID: id, Status: http.StatusCreated,
|
||||
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
|
||||
DurationTotal: 12.5, DurationUpstreamTotal: 10.25,
|
||||
DurationTotalMS: 12.5, DurationUpstreamTotalMS: 10.25,
|
||||
})
|
||||
wantStatus(t, got, http.StatusNoContent)
|
||||
|
||||
@@ -1298,14 +1299,14 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
|
||||
}{
|
||||
{
|
||||
"deny", map[string]string{denyNets: client}, "/", http.StatusForbidden,
|
||||
requestlog.ActionDenied,
|
||||
requestlog.ActionClientDenied,
|
||||
},
|
||||
{
|
||||
"close", map[string]string{denyNets: client, banResponse: banResponseClose}, "/",
|
||||
0, requestlog.ActionDenied,
|
||||
0, requestlog.ActionClientDenied,
|
||||
},
|
||||
{"own prefix", nil, discriminator.HealthPath, http.StatusNotFound,
|
||||
requestlog.ActionAdmin},
|
||||
requestlog.ActionOwnEndpoint},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -1317,7 +1318,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
|
||||
do(t, req)
|
||||
|
||||
got := s.report(discriminator.Report{
|
||||
RequestID: id, Status: tc.status, ResponseBytes: 8, DurationTotal: 3,
|
||||
RequestID: id, Status: tc.status, ResponseBytes: 8, DurationTotalMS: 3,
|
||||
})
|
||||
wantStatus(t, got, http.StatusNoContent)
|
||||
|
||||
@@ -1325,8 +1326,8 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
|
||||
wantLine(t, line, tc.status, tc.action)
|
||||
|
||||
if line.DiscriminatorLine != requestlog.DiscriminatorLineReport ||
|
||||
line.RequestID != id || line.ResponseBytes != 8 || line.DurationTotal != 3 ||
|
||||
line.DurationUpstreamTotal != nil {
|
||||
line.RequestID != id || line.ResponseBytes != 8 || line.DurationTotalMS != 3 ||
|
||||
line.DurationUpstreamTotalMS != nil {
|
||||
t.Errorf("the report's line is %+v", line.Line)
|
||||
}
|
||||
})
|
||||
@@ -1583,9 +1584,10 @@ func TestCoreRuleSetAlertNamesTheSite(t *testing.T) {
|
||||
wantDisposition(t, got, http.StatusForbidden, discriminator.DispositionDeny)
|
||||
|
||||
waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
if len(waiting) != 1 || waiting[0].Event != alerts.EventWAFBlock ||
|
||||
if len(waiting) != 1 || waiting[0].Event != alerts.EventCoreRuleSetMatch ||
|
||||
waiting[0].Host != siteA {
|
||||
t.Errorf("alerts waiting are %+v, want a waf_block naming %s", waiting, siteA)
|
||||
t.Errorf("alerts waiting are %+v, want a core_rule_set_match naming %s",
|
||||
waiting, siteA)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic dnsbl. It reads SWW_REPUTATION_ACTION, and the DNSBL zones'
|
||||
// verdicts through the DNSBL checker.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// dnsbl notes the DNSBL zones whose verdict lists the client, as noteListed
|
||||
// does, adds the tag dnsbl=<zone> for each, the zone's key masked, and refuses
|
||||
// the request when SWW_REPUTATION_ACTION is deny. Being
|
||||
// limit, it lowers the client's limits instead (see limitPercentages), and
|
||||
// being log, it does nothing more. A zone without a verdict on the client is
|
||||
// asked about it in the background, and the request does not wait for the
|
||||
// answer. Its figure is the zones that list the client, their keys masked.
|
||||
type dnsbl struct{}
|
||||
|
||||
func (dnsbl) name() string { return "dnsbl" }
|
||||
|
||||
func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict {
|
||||
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
|
||||
rq.dnsblListed = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a DNSBL zone")
|
||||
|
||||
for _, zone := range listedBy {
|
||||
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
|
||||
}
|
||||
|
||||
if !rq.dnsblListed || rq.h.config.ReputationAction != deny {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionClientDenied,
|
||||
figures: map[string]any{"zones": listedBy},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic header_limits. It reads no setting: its limits are fixed.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// The header limits: the most cookies and header fields a request may have
|
||||
// when the checks reach the Core Rule Set. Real clients never reach them.
|
||||
const (
|
||||
maxCookies = 50
|
||||
maxHeaderFields = 100
|
||||
)
|
||||
|
||||
// headerLimits refuses a request over the header limits, with 431. Its figures
|
||||
// are the request's header fields and cookies, and the limit of each.
|
||||
type headerLimits struct{}
|
||||
|
||||
func (headerLimits) name() string { return "header_limits" }
|
||||
|
||||
func (headerLimits) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
fields, cookies := rq.headerCounts()
|
||||
if !overHeaderLimits(fields, cookies) {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionRequestHeaderTooLarge,
|
||||
figures: map[string]any{
|
||||
"fields": fields, "fields_limit": maxHeaderFields,
|
||||
"cookies": cookies, "cookies_limit": maxCookies,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// overHeaderLimits reports whether a request with these counts of header fields
|
||||
// and cookies, as headerCounts counts them, is over the header limits: it has
|
||||
// more than maxHeaderFields fields or more than maxCookies cookies.
|
||||
func overHeaderLimits(fields, cookies int) bool {
|
||||
return fields > maxHeaderFields || cookies > maxCookies
|
||||
}
|
||||
|
||||
// headerCounts returns the request's header fields, each value of a header sent
|
||||
// twice being one, and its cookies, each part of a Cookie header between
|
||||
// semicolons being one unless it is empty or only spaces and tabs. The Core
|
||||
// Rule Set runs each of its rules on each cookie, so that thousands of small
|
||||
// cookies keep it busy for over a second. It trims only spaces and control
|
||||
// characters, of which a header value can hold only tabs, so a part of Unicode
|
||||
// spaces is a cookie to it.
|
||||
func (rq *request) headerCounts() (int, int) {
|
||||
fields, cookies := 0, 0
|
||||
|
||||
for _, values := range rq.in.Header {
|
||||
fields += len(values)
|
||||
}
|
||||
|
||||
for _, value := range rq.in.Header.Values("Cookie") {
|
||||
for part := range strings.SplitSeq(value, ";") {
|
||||
if strings.Trim(part, " \t") != "" {
|
||||
cookies++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return fields, cookies
|
||||
}
|
||||
@@ -21,7 +21,8 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
|
||||
)
|
||||
|
||||
for range healthChecks {
|
||||
got := s.adminRequest("", http.MethodGet, discriminator.HealthPath, "", http.StatusOK)
|
||||
got := s.ownEndpointRequest("", http.MethodGet, discriminator.HealthPath, "",
|
||||
http.StatusOK)
|
||||
|
||||
if string(got.body) != "ok\n" || got.header.Get("Content-Type") != contentType {
|
||||
t.Errorf("health endpoint answered %q with Content-Type %q, want ok "+
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
package discriminator
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// To add a heuristic, write one file with a type that implements heuristic,
|
||||
// and add one line for it to one of the lists below. Its settings go in
|
||||
// config.Config, and the comment at the top of its file names them.
|
||||
|
||||
// heuristic is one of the checks that decide what discriminator does with a
|
||||
// request.
|
||||
type heuristic interface {
|
||||
// name is the heuristic's name, which the log line gives.
|
||||
name() string
|
||||
// check looks at the request at now, and returns nil to let it through.
|
||||
// ctx is the request's own context.
|
||||
check(ctx context.Context, rq *request, now time.Time) *verdict
|
||||
}
|
||||
|
||||
// verdict is a heuristic refusing a request, or banning its client: the action
|
||||
// the log line names, one of requestlog's Action constants, and the figures the
|
||||
// heuristic acted on, by name, such as the count and the limit.
|
||||
type verdict struct {
|
||||
action string
|
||||
figures map[string]any
|
||||
}
|
||||
|
||||
// The names of the figures that more than one heuristic gives.
|
||||
const (
|
||||
figureWindow = "window"
|
||||
figureCount = "count"
|
||||
figureLimit = "limit"
|
||||
figureScore = "score"
|
||||
figureNetblock = "netblock"
|
||||
)
|
||||
|
||||
// The lists of heuristics, each in the order its heuristics run. A client in
|
||||
// SWW_ALLOW_NETS skips them all.
|
||||
|
||||
// beforeLookup are the heuristics that run before the client's AS number and
|
||||
// country are looked up, so that a client they refuse is not looked up.
|
||||
func beforeLookup() []heuristic {
|
||||
return []heuristic{
|
||||
tagLists{},
|
||||
routeWatchTags{},
|
||||
denyNets{},
|
||||
activeBan{},
|
||||
}
|
||||
}
|
||||
|
||||
// afterLookup are the heuristics that run once the lookup has been made.
|
||||
// tag_deny runs twice: before the rate limits, on the tags known by then, and
|
||||
// last, on the tag rule_files and core_rule_set can add.
|
||||
func afterLookup() []heuristic {
|
||||
return []heuristic{
|
||||
countries{},
|
||||
blocklists{},
|
||||
crowdSec{},
|
||||
dnsbl{},
|
||||
abuseIPDB{},
|
||||
tagDeny{},
|
||||
rateLimits{},
|
||||
trapPaths{},
|
||||
ruleFiles{},
|
||||
headerLimits{},
|
||||
coreRuleSet{},
|
||||
tagDeny{},
|
||||
}
|
||||
}
|
||||
|
||||
// afterAnswer are the heuristics that run on a request once discriminator has
|
||||
// answered it, before its decision line is written. They ban the client, and do
|
||||
// not change what the request got: their verdict's action is not used.
|
||||
func afterAnswer() []heuristic {
|
||||
return []heuristic{
|
||||
refusalLimit{},
|
||||
}
|
||||
}
|
||||
|
||||
// onReport are the heuristics that run on an allowed request when rproxy's
|
||||
// report of it comes, before its report line is written. Like afterAnswer, they
|
||||
// only ban.
|
||||
func onReport() []heuristic {
|
||||
return []heuristic{
|
||||
byteLimits{},
|
||||
}
|
||||
}
|
||||
|
||||
// runAfterAnswer runs the heuristics of afterAnswer on the request, at the time
|
||||
// it is called.
|
||||
func (rq *request) runAfterAnswer(ctx context.Context) {
|
||||
rq.runBanningHeuristics(ctx, afterAnswer())
|
||||
}
|
||||
|
||||
// runHeuristics runs list, beforeLookup or afterLookup, on the request at now,
|
||||
// and returns the first verdict, or nil when none gives one. The log line's
|
||||
// heuristic and heuristic_figures name the heuristic that gave it, and give its
|
||||
// figures. ctx is the request's own context.
|
||||
func (rq *request) runHeuristics(
|
||||
ctx context.Context, list []heuristic, now time.Time,
|
||||
) *verdict {
|
||||
name, v := rq.firstVerdict(ctx, list, now)
|
||||
if v != nil {
|
||||
rq.line.Heuristic = name
|
||||
rq.line.HeuristicFigures = v.figures
|
||||
}
|
||||
|
||||
return v
|
||||
}
|
||||
|
||||
// runBanningHeuristics runs list, afterAnswer or onReport, on the request at
|
||||
// the time it is called. The log line's after_answer_heuristic and
|
||||
// after_answer_heuristic_figures name the heuristic that gave a verdict, and
|
||||
// give its figures, so that heuristic and heuristic_figures still name the one
|
||||
// that refused the request. ctx is the request's own context.
|
||||
func (rq *request) runBanningHeuristics(ctx context.Context, list []heuristic) {
|
||||
name, v := rq.firstVerdict(ctx, list, rq.h.now())
|
||||
if v != nil {
|
||||
rq.line.AfterAnswerHeuristic = name
|
||||
rq.line.AfterAnswerHeuristicFigures = v.figures
|
||||
}
|
||||
}
|
||||
|
||||
// firstVerdict runs list on the request at now, in order, until a heuristic
|
||||
// gives a verdict, and returns that heuristic's name and its verdict, or "" and
|
||||
// nil when none gives one. ctx is the request's own context.
|
||||
func (rq *request) firstVerdict(
|
||||
ctx context.Context, list []heuristic, now time.Time,
|
||||
) (string, *verdict) {
|
||||
for _, each := range list {
|
||||
v := each.check(ctx, rq, now)
|
||||
if v != nil {
|
||||
return each.name(), v
|
||||
}
|
||||
}
|
||||
|
||||
return "", nil
|
||||
}
|
||||
@@ -20,7 +20,7 @@ func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
|
||||
s, clk, server := startWithClock(t, geojsURL, map[string]string{
|
||||
lookupTimeout: "1h",
|
||||
rateLimitPerMinute: "2",
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
start := clk.Now()
|
||||
|
||||
@@ -31,9 +31,9 @@ func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
clk.advance(time.Second)
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
clk.advance(time.Second)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
want := ratelimit.History{
|
||||
FirstSeen: start,
|
||||
@@ -100,8 +100,8 @@ func TestOwnEndpointsAreNotInTheHistory(t *testing.T) {
|
||||
|
||||
s, _, server := startWithClock(t, "", map[string]string{metricsToken: token})
|
||||
|
||||
s.adminRequest("", http.MethodGet, discriminator.HealthPath, "", http.StatusOK)
|
||||
s.adminRequest("", http.MethodGet, discriminator.MetricsPath, "",
|
||||
s.ownEndpointRequest("", http.MethodGet, discriminator.HealthPath, "", http.StatusOK)
|
||||
s.ownEndpointRequest("", http.MethodGet, discriminator.MetricsPath, "",
|
||||
http.StatusUnauthorized)
|
||||
s.scrape()
|
||||
|
||||
|
||||
@@ -4,8 +4,10 @@ import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
)
|
||||
|
||||
@@ -19,7 +21,8 @@ const (
|
||||
)
|
||||
|
||||
// lookUp looks up the client's AS number and country, in the lookup
|
||||
// database or through GeoJS, and notes them for the log line, unless
|
||||
// database or through GeoJS, and notes them for the log line and as the
|
||||
// request's asn= and country= tags, unless
|
||||
// SWW_LOOKUP_SOURCE is off or the client is on a private, loopback or
|
||||
// link-local address, which no lookup can place. The lookup database
|
||||
// answers at once. With GeoJS, while a setting needs the answer, such as a
|
||||
@@ -47,6 +50,34 @@ func (rq *request) lookUp(ctx context.Context, now time.Time) {
|
||||
rq.line.ASN = rq.lookupAnswer.ASN
|
||||
rq.line.ASName = rq.lookupAnswer.ASName
|
||||
rq.line.Country = rq.lookupAnswer.Country
|
||||
|
||||
if tag := asnTag(rq.line.ASN); tag != "" {
|
||||
rq.addTag(tag)
|
||||
}
|
||||
|
||||
if tag := countryTag(rq.line.Country); tag != "" {
|
||||
rq.addTag(tag)
|
||||
}
|
||||
}
|
||||
|
||||
// asnTag returns the tag of asn, an AS number as the lookup gives it, such as
|
||||
// asn=64496 for AS64496, or "" for no AS number.
|
||||
func asnTag(asn string) string {
|
||||
if asn == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
return config.ASNTagPrefix + strings.TrimPrefix(asn, "AS")
|
||||
}
|
||||
|
||||
// countryTag returns the tag of country, a code as the lookup gives it, such
|
||||
// as country=DE for DE, or "" for no country.
|
||||
func countryTag(country string) string {
|
||||
if country == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
return config.CountryTagPrefix + country
|
||||
}
|
||||
|
||||
// addLookup adds answer, an answer about a client from the lookup
|
||||
|
||||
@@ -40,7 +40,7 @@ func TestEveryClientIsLookedUpWithoutWaitingWhileNoSettingNeedsIt(t *testing.T)
|
||||
// comes too. None waits for GeoJS.
|
||||
for _, line := range []logLine{
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited),
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited),
|
||||
s.get(fromKP, http.StatusOK, requestlog.ActionForward),
|
||||
} {
|
||||
got := asnAndCountry{line.ASN, line.ASName, line.Country}
|
||||
@@ -96,7 +96,7 @@ func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) {
|
||||
map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
lookupTimeout: "1h",
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
|
||||
requestFromDE := func() *http.Request {
|
||||
@@ -179,7 +179,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(t *testing.
|
||||
|
||||
for _, line := range []logLine{
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited),
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited),
|
||||
} {
|
||||
if got := (asnAndCountry{line.ASN, line.ASName, line.Country}); got != want {
|
||||
t.Errorf("log line has %+v, want %+v", got, want)
|
||||
@@ -264,7 +264,7 @@ func TestClientsAreLookedUpInTheLookupDatabaseAndGeoJSIsNotAsked(t *testing.T) {
|
||||
want asnAndCountry
|
||||
}{
|
||||
{s.get(fromDE, http.StatusOK, requestlog.ActionForward), de},
|
||||
{s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited), de},
|
||||
{s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited), de},
|
||||
{
|
||||
s.get(fromKP, http.StatusForbidden, requestlog.ActionCountryDenied),
|
||||
asnAndCountry{asnKP, asNameKP, "KP"},
|
||||
@@ -385,7 +385,7 @@ func TestRequestWaitsAsLongAsTheLookupTimeoutSays(t *testing.T) {
|
||||
// is never reached.
|
||||
const timeout = 3 * time.Second
|
||||
|
||||
server, out, _ := newDiscriminator(t, unansweredGeoJSURL, time.Now,
|
||||
server, out, _ := newDiscriminator(t, silentGeoJSURL, time.Now,
|
||||
map[string]string{
|
||||
lookupTimeout: timeout.String(),
|
||||
allowedCountries: "DE",
|
||||
@@ -405,25 +405,25 @@ func TestRequestWaitsAsLongAsTheLookupTimeoutSays(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
// unansweredGeoJSURL is where a GeoJS that never answers is asked: a
|
||||
// silentGeoJSURL is where a GeoJS that never answers is asked: a
|
||||
// request to it waits, without the network, until it is abandoned.
|
||||
// TestMain registers it with Go's default transport, through which GeoJS
|
||||
// is asked.
|
||||
const unansweredGeoJSURL = "unanswered://geojs/v1/ip/geo.json"
|
||||
const silentGeoJSURL = "silent://geojs/v1/ip/geo.json"
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
transport, _ := http.DefaultTransport.(*http.Transport)
|
||||
transport.RegisterProtocol("unanswered", unansweredGeoJS{})
|
||||
transport.RegisterProtocol("silent", silentGeoJS{})
|
||||
transport.RegisterProtocol("abuseipdb", abuseIPDBStandIn{})
|
||||
|
||||
m.Run()
|
||||
}
|
||||
|
||||
// unansweredGeoJS is the GeoJS at unansweredGeoJSURL.
|
||||
type unansweredGeoJS struct{}
|
||||
// silentGeoJS is the GeoJS at silentGeoJSURL.
|
||||
type silentGeoJS struct{}
|
||||
|
||||
// RoundTrip waits until req is abandoned.
|
||||
func (unansweredGeoJS) RoundTrip(req *http.Request) (*http.Response, error) {
|
||||
func (silentGeoJS) RoundTrip(req *http.Request) (*http.Response, error) {
|
||||
<-req.Context().Done()
|
||||
|
||||
return nil, req.Context().Err()
|
||||
|
||||
@@ -32,7 +32,7 @@ func TestMetricsAreOffWhileTheTokenIsUnset(t *testing.T) {
|
||||
|
||||
// An empty token does not match the unset one either.
|
||||
for _, authorization := range []string{bearer, "Bearer ", ""} {
|
||||
s.adminRequest(authorization, http.MethodGet, discriminator.MetricsPath, "",
|
||||
s.ownEndpointRequest(authorization, http.MethodGet, discriminator.MetricsPath, "",
|
||||
http.StatusNotFound)
|
||||
}
|
||||
}
|
||||
@@ -60,7 +60,7 @@ func TestMetricsNeedTheTokenAndOtherPathsAreNotFound(t *testing.T) {
|
||||
{http.MethodGet, "/_sww/bans", bearer, http.StatusNotFound},
|
||||
{http.MethodPost, discriminator.HealthPath, "", http.StatusNotFound},
|
||||
} {
|
||||
got := s.adminRequest(tc.authorization, tc.method, tc.path, "", tc.status)
|
||||
got := s.ownEndpointRequest(tc.authorization, tc.method, tc.path, "", tc.status)
|
||||
|
||||
if tc.status == http.StatusUnauthorized &&
|
||||
got.header.Get("WWW-Authenticate") != "Bearer" {
|
||||
@@ -91,7 +91,7 @@ func TestMetricsCountTheTraffic(t *testing.T) {
|
||||
out.endedLines(t, 2)
|
||||
|
||||
forward := `{action="forward",instance="app",status_class="2xx"}`
|
||||
notFound := `{action="admin",instance="app",status_class="4xx"}`
|
||||
notFound := `{action="own_endpoint",instance="app",status_class="4xx"}`
|
||||
|
||||
// The request for the metrics is itself under way.
|
||||
metrics := scrape(t, server.Addr)
|
||||
@@ -125,15 +125,15 @@ func TestMetricsCountLimitsAndBans(t *testing.T) {
|
||||
})
|
||||
|
||||
// SWW_BAN_RESPONSE=close sends no status at all.
|
||||
s.get(denied, 0, requestlog.ActionDenied)
|
||||
s.get(denied, 0, requestlog.ActionClientDenied)
|
||||
|
||||
// A first broken limit bans for an hour.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, 0, requestlog.ActionRateLimited)
|
||||
s.get(client, 0, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
metrics := s.scrape()
|
||||
wantMetric(t, metrics,
|
||||
`sww_requests_total{action="denied",instance="app",status_class="none"}`, 1)
|
||||
`sww_requests_total{action="client_denied",instance="app",status_class="none"}`, 1)
|
||||
wantMetric(t, metrics, `sww_rate_limit_hits_total{instance="app",`+
|
||||
`kind="requests",window="minute"}`, 1)
|
||||
wantMetric(t, metrics, `sww_offences_total{instance="app",kind="limit"}`, 1)
|
||||
@@ -147,7 +147,7 @@ func TestMetricsCountLimitsAndBans(t *testing.T) {
|
||||
// A limit broken again right after would ban for three hours, longer
|
||||
// than SWW_MAX_BAN_DURATION, so the ban is permanent.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, 0, requestlog.ActionRateLimited)
|
||||
s.get(client, 0, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
metrics = s.scrape()
|
||||
wantMetric(t, metrics, `sww_rate_limit_hits_total{instance="app",`+
|
||||
@@ -198,7 +198,7 @@ func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) {
|
||||
trustedProxies: trustLocalhost,
|
||||
metricsToken: token,
|
||||
metricsTopN: "2",
|
||||
deniedCountries: "kp",
|
||||
allowedCountries: "de,fr",
|
||||
}
|
||||
geojsURL, _ := startGeoJS(t)
|
||||
addr, out, server := startProxyWithClock(t, app.URL, geojsURL, time.Now, env)
|
||||
@@ -324,11 +324,11 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
|
||||
map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
metricsToken: token,
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
|
||||
// GeoJS fails, so the client counts as coming from an unknown country,
|
||||
// which SWW_DENIED_COUNTRIES does not refuse.
|
||||
// GeoJS fails, so the client has no country= tag, and SWW_TAG_DENY does
|
||||
// not refuse it.
|
||||
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
|
||||
req.Header.Set(forwardedFor, fromDE)
|
||||
wantStatus(t, do(t, req), http.StatusOK)
|
||||
@@ -347,7 +347,7 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
|
||||
|
||||
wantMetric(t, metrics, `sww_geojs_requests_total{instance="app"}`, 1)
|
||||
wantMetric(t, metrics, `sww_geojs_failures_total{instance="app"}`, 1)
|
||||
wantMetric(t, metrics, `sww_geojs_unanswered_total{instance="app"}`, 1)
|
||||
wantMetric(t, metrics, `sww_requests_without_geojs_answer_total{instance="app"}`, 1)
|
||||
}
|
||||
|
||||
// keptAnswer returns GeoJS's answer that the client at addr is in
|
||||
@@ -381,8 +381,8 @@ func scrape(t *testing.T, addr string) string {
|
||||
func (s *sender) scrape() string {
|
||||
s.t.Helper()
|
||||
|
||||
return string(s.adminRequest(bearer, http.MethodGet, discriminator.MetricsPath, "",
|
||||
http.StatusOK).body)
|
||||
return string(s.ownEndpointRequest(bearer, http.MethodGet, discriminator.MetricsPath,
|
||||
"", http.StatusOK).body)
|
||||
}
|
||||
|
||||
// metric returns the value of series in metrics, which are in the
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
@@ -14,6 +15,105 @@ import (
|
||||
// observe is the value of SWW_MODE for observe mode.
|
||||
const observe = "observe"
|
||||
|
||||
func TestBanActionSaysWhetherTheBanRunsOut(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const trapPath = "/wp-login.php"
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
cause string
|
||||
// endsIn is how long after the request the ban ends: negative for a
|
||||
// ban that has ended, and zero for a permanent ban.
|
||||
endsIn time.Duration
|
||||
path string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
"temporary ban", bans.CauseAdmin, time.Hour, "/",
|
||||
requestlog.ActionTemporarilyBanned,
|
||||
},
|
||||
{"permanent ban", bans.CauseAdmin, 0, "/", requestlog.ActionPermanentlyBanned},
|
||||
{
|
||||
"request that makes a ban permanent", bans.CauseAttack, time.Hour, "/",
|
||||
requestlog.ActionPermanentlyBanned,
|
||||
},
|
||||
{
|
||||
"trap path after an attack ban ended", bans.CauseAttack, -time.Hour,
|
||||
trapPath, requestlog.ActionPermanentlyBanned,
|
||||
},
|
||||
{
|
||||
"first trap path after another ban ended", bans.CauseAdmin, -time.Hour,
|
||||
trapPath, requestlog.ActionTemporarilyBanned,
|
||||
},
|
||||
} {
|
||||
for _, modeValue := range []string{"enforce", observe} {
|
||||
t.Run(tc.name+" in "+modeValue, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server := startWithClock(t, "", map[string]string{
|
||||
trapPaths: trapPathList, mode: modeValue,
|
||||
})
|
||||
|
||||
ban := bans.Ban{
|
||||
Netblock: netip.MustParsePrefix(client + "/32"),
|
||||
Start: clk.Now().Add(-2 * time.Hour),
|
||||
Cause: tc.cause,
|
||||
}
|
||||
if tc.endsIn != 0 {
|
||||
ban.Expires = clk.Now().Add(tc.endsIn)
|
||||
}
|
||||
|
||||
server.Ledger.Load([]bans.Ban{ban})
|
||||
|
||||
if modeValue == observe {
|
||||
line := s.request(client, tc.path, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, tc.want)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
s.request(client, tc.path, http.StatusForbidden, tc.want)
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestObserveModeNamesAPermanentBanWhoseAlertWouldBeSent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server, queue := startWithAlerts(t, map[string]string{
|
||||
trapPaths: trapPathList, mode: observe,
|
||||
})
|
||||
|
||||
server.Ledger.Load([]bans.Ban{{
|
||||
Netblock: netip.MustParsePrefix(client + "/32"),
|
||||
Start: clk.Now().Add(-2 * time.Hour),
|
||||
Expires: clk.Now().Add(-time.Hour),
|
||||
Cause: bans.CauseAttack,
|
||||
}})
|
||||
|
||||
line := s.request(client, "/wp-login.php", http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionPermanentlyBanned)
|
||||
|
||||
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
if len(waiting) != 1 || waiting[0].Event != alerts.EventPermanentBan {
|
||||
t.Errorf("alerts waiting are %+v, want one %s", waiting, alerts.EventPermanentBan)
|
||||
}
|
||||
}
|
||||
|
||||
func TestObserveModeNamesTheHeuristicThatWouldHaveRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
mode: observe, denyNets: client,
|
||||
})
|
||||
|
||||
line := s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, "deny_nets", `{"netblock": "`+client+`/32"}`)
|
||||
}
|
||||
|
||||
func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -38,7 +138,7 @@ func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
|
||||
lookupTimeout: "1h",
|
||||
rateLimitPerMinute: "1",
|
||||
denyNets: denied,
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
}
|
||||
|
||||
if tc.setting != "" {
|
||||
@@ -57,10 +157,10 @@ func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
for _, sent := range []struct{ from, refusal string }{
|
||||
{denied, requestlog.ActionDenied},
|
||||
{banned, requestlog.ActionBanned},
|
||||
{fromKP, requestlog.ActionCountryDenied},
|
||||
{fromDE, requestlog.ActionRateLimited},
|
||||
{denied, requestlog.ActionClientDenied},
|
||||
{banned, requestlog.ActionTemporarilyBanned},
|
||||
{fromKP, requestlog.ActionClientDenied},
|
||||
{fromDE, requestlog.ActionClientIsRateLimited},
|
||||
} {
|
||||
if !tc.observe {
|
||||
line := s.get(sent.from, http.StatusForbidden, sent.refusal)
|
||||
@@ -98,7 +198,7 @@ func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
|
||||
|
||||
for range 2 {
|
||||
line := s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionRateLimited)
|
||||
wantWouldAction(t, line, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
if line.LimitHit != minute || line.Offence != requestlog.OffenceLimit ||
|
||||
line.BanExpires != "" {
|
||||
@@ -111,7 +211,7 @@ func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
|
||||
// The ban read from bans.json refuses nothing, and so counts no
|
||||
// refusal in its notes, but is kept.
|
||||
line := s.get(otherClient, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionBanned)
|
||||
wantWouldAction(t, line, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
if line.BanExpires != requestlog.FormatTime(kept.Expires) {
|
||||
t.Errorf("log line has ban_expires %q, want %s", line.BanExpires,
|
||||
|
||||
@@ -16,17 +16,18 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
"sneak.berlin/go/smallwebwaf/internal/routewatch"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
)
|
||||
|
||||
// adminPrefix starts the path of each of discriminator's own endpoints. The app never
|
||||
// sees a path under it.
|
||||
const adminPrefix = "/_sww/"
|
||||
// ownEndpointPrefix starts the path of each of discriminator's own endpoints. The app
|
||||
// never sees a path under it.
|
||||
const ownEndpointPrefix = "/_sww/"
|
||||
|
||||
// HealthPath is the health endpoint of discriminator and of rproxy, which the
|
||||
// container's health check asks.
|
||||
@@ -64,7 +65,7 @@ type Params struct {
|
||||
LookupFile *lookup.File
|
||||
// CoreRuleSet is what NewCoreRuleSet returns: what inspects requests
|
||||
// while SWW_WAF_MODE is not off, and nil while it is.
|
||||
CoreRuleSet *waf.CoreRuleSet
|
||||
CoreRuleSet *coreruleset.CoreRuleSet
|
||||
// Now tells the time by which requests are counted for the rate
|
||||
// limits, bans are made and run out, and GeoJS's answers are kept,
|
||||
// normally time.Now in UTC, the time the state files give.
|
||||
@@ -85,7 +86,8 @@ type Params struct {
|
||||
// Server is the server discriminator runs, with the parts whose state the state
|
||||
// files keep, the lookup database, nil unless SWW_LOOKUP_SOURCE is file, the
|
||||
// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts,
|
||||
// AbuseIPDB's scores and checks spent, and the metrics.
|
||||
// AbuseIPDB's scores and checks spent, and the metrics. RouteWatch, nil while
|
||||
// SWW_ROUTEWATCH_URL is unset, keeps its answers in memory only.
|
||||
type Server struct {
|
||||
*http.Server
|
||||
|
||||
@@ -97,6 +99,7 @@ type Server struct {
|
||||
Lists *reputation.Lists
|
||||
DNSBL *reputation.DNSBL
|
||||
AbuseIPDB *reputation.AbuseIPDB
|
||||
RouteWatch *routewatch.Client
|
||||
Metrics *metrics.Metrics
|
||||
}
|
||||
|
||||
@@ -126,7 +129,7 @@ func newHandler(params Params) *handler {
|
||||
BytesPerHour: params.Config.BytesLimitPerHour,
|
||||
BytesPerDay: params.Config.BytesLimitPerDay,
|
||||
}, params.Config.MaxTrackedClients),
|
||||
ledger: bans.New(bans.Rules{
|
||||
ledger: bans.New(bans.Params{
|
||||
LimitBanDuration: params.Config.LimitBanDuration,
|
||||
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
|
||||
MaxBanDuration: params.Config.MaxBanDuration,
|
||||
@@ -152,14 +155,21 @@ func newHandler(params Params) *handler {
|
||||
coreRuleSet: params.CoreRuleSet,
|
||||
alerts: params.Alerts,
|
||||
}
|
||||
if params.Config.RouteWatchURL != "" {
|
||||
h.routeWatch = routewatch.New(routewatch.Params{
|
||||
URL: params.Config.RouteWatchURL, Now: params.Now, ProcessLog: params.ProcessLog,
|
||||
})
|
||||
}
|
||||
|
||||
h.geojs = lookup.New(lookup.Params{
|
||||
URL: params.GeoJSURL,
|
||||
Timeout: params.Config.LookupTimeout,
|
||||
// The country lists, the headers and the biased thresholds act on
|
||||
// the answer before the request goes on.
|
||||
Wait: len(params.Config.DeniedCountries) > 0 ||
|
||||
len(params.Config.ExclusivelyAllowedCountries) > 0 ||
|
||||
params.Config.AddLookupHeaders || biasedThresholdsSet(params.Config),
|
||||
// The country list, the headers, the tag settings that name an asn=
|
||||
// or country= tag and the biased thresholds act on the answer before
|
||||
// the request goes on.
|
||||
Wait: len(params.Config.ExclusivelyAllowedCountries) > 0 ||
|
||||
params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() ||
|
||||
biasedThresholdsSet(params.Config),
|
||||
Answered: h.addLookup,
|
||||
Now: params.Now,
|
||||
ProcessLog: params.ProcessLog,
|
||||
@@ -183,6 +193,7 @@ func newReputation(
|
||||
lists := reputation.New(reputation.Params{
|
||||
BlocklistURLs: cfg.BlocklistURLs, Refresh: cfg.BlocklistRefresh,
|
||||
ASNLimitPercentURL: cfg.ASNLimitPercentURL,
|
||||
TagLists: cfg.TagLists,
|
||||
CrowdSecDecisionsURL: cfg.CrowdSecDecisionsURL, CrowdSecKey: cfg.CrowdSecKey,
|
||||
Now: params.Now, ProcessLog: params.ProcessLog, Alerts: params.Alerts,
|
||||
})
|
||||
@@ -210,21 +221,23 @@ func newReputation(
|
||||
// NewCoreRuleSet returns the Core Rule Set at SWW_WAF_PARANOIA_LEVEL,
|
||||
// without the rules SWW_WAF_DISABLED_RULES switches off, reading bodies
|
||||
// up to SWW_WAF_BODY_LIMIT, with the exclusions in
|
||||
// SWW_CORE_RULE_SET_EXCLUSIONS_FILE, or nil while SWW_WAF_MODE is off. It
|
||||
// SWW_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_APPS,
|
||||
// or nil while SWW_WAF_MODE is off. It
|
||||
// reads at most QuestionBodyMaxBytes less one byte: a question's body of
|
||||
// QuestionBodyMaxBytes may be cut short, and the Core Rule Set does not read a
|
||||
// JSON or XML body over its limit. An exclusions file that waf.New refuses is
|
||||
// JSON or XML body over its limit. An exclusions file that coreruleset.New refuses is
|
||||
// an error in every mode. While the mode is off, a set file is loaded with the
|
||||
// Core Rule Set only to check it.
|
||||
func NewCoreRuleSet(cfg *config.Config) (*waf.CoreRuleSet, error) {
|
||||
func NewCoreRuleSet(cfg *config.Config) (*coreruleset.CoreRuleSet, error) {
|
||||
if cfg.WAFMode == config.WAFModeOff && cfg.CoreRuleSetExclusionsFile == "" {
|
||||
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
|
||||
}
|
||||
|
||||
coreRuleSet, err := waf.New(waf.Params{
|
||||
coreRuleSet, err := coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: cfg.WAFParanoiaLevel, DisabledRules: cfg.WAFDisabledRules,
|
||||
BodyLimit: min(cfg.WAFBodyLimit, QuestionBodyMaxBytes-1),
|
||||
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
|
||||
Apps: cfg.Apps,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("SWW_CORE_RULE_SET_EXCLUSIONS_FILE: %w", err)
|
||||
@@ -253,8 +266,9 @@ type handler struct {
|
||||
lists *reputation.Lists
|
||||
dnsbl *reputation.DNSBL
|
||||
abuseIPDB *reputation.AbuseIPDB
|
||||
routeWatch *routewatch.Client
|
||||
rules *rules.Files
|
||||
coreRuleSet *waf.CoreRuleSet
|
||||
coreRuleSet *coreruleset.CoreRuleSet
|
||||
alerts *alerts.Queue
|
||||
// awaitingReport holds the decision on each answered question by its id
|
||||
// until its report comes, and nil for a question still being handled.
|
||||
|
||||
@@ -69,7 +69,9 @@ const (
|
||||
lookupDBPath = "SWW_LOOKUP_DB_PATH"
|
||||
lookupTimeout = "SWW_LOOKUP_TIMEOUT"
|
||||
addLookupHeaders = "SWW_ADD_LOOKUP_HEADERS"
|
||||
deniedCountries = "SWW_DENIED_COUNTRIES"
|
||||
tagDeny = "SWW_TAG_DENY"
|
||||
// denyKP is the value of SWW_TAG_DENY that refuses fromKP's country.
|
||||
denyKP = "country=KP"
|
||||
allowedCountries = "SWW_EXCLUSIVELY_ALLOWED_COUNTRIES"
|
||||
banResponse = "SWW_BAN_RESPONSE"
|
||||
limitBanDuration = "SWW_LIMIT_BAN_DURATION"
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic rate_limits. It reads SWW_RATE_LIMIT_EXEMPT_NETS,
|
||||
// SWW_RATE_LIMIT_EXEMPT_PATHS and SWW_BAN_RESPONSE, and the rate limits and
|
||||
// biased thresholds through the limiter and limitPercentages.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// rateLimits counts the request for the rate limits, unless
|
||||
// countedByRateLimits exempts it, by the client's limit percentages, which it
|
||||
// notes for the log line with the client's counts. A request that takes the
|
||||
// client over a rate limit, as its limit percentage lowers it, breaks it: it is
|
||||
// refused, and bans the client. A request exempt from the rate limits is exempt
|
||||
// from the byte limits too. Its figures are the window, the count and the
|
||||
// limit.
|
||||
type rateLimits struct{}
|
||||
|
||||
func (rateLimits) name() string { return "rate_limits" }
|
||||
|
||||
func (rateLimits) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
rq.counted = rq.countedByRateLimits()
|
||||
if !rq.counted {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.rateLimitPercent, rq.bytesLimitPercent = rq.limitPercentages()
|
||||
rq.line.RateLimitPercent, rq.line.RateLimitPercentSetting =
|
||||
rq.rateLimitPercent.logged()
|
||||
rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting =
|
||||
rq.bytesLimitPercent.logged()
|
||||
|
||||
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now,
|
||||
rq.rateLimitPercent.percent)
|
||||
rq.line.Counts = counts
|
||||
|
||||
if !over {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.banForLimit(now, hit, rq.h.config.BanResponse)
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionClientIsRateLimited, figures: limitFigures(hit),
|
||||
}
|
||||
}
|
||||
|
||||
// overAWholeRateLimit reports whether the heuristic rate_limits will refuse the
|
||||
// request at now, whatever its client's limit percentages: the rate limits
|
||||
// count it, and one more request from its client takes it over a whole rate
|
||||
// limit. It counts nothing.
|
||||
func (rq *request) overAWholeRateLimit(now time.Time) bool {
|
||||
return rq.countedByRateLimits() &&
|
||||
rq.h.limiter.OverARateLimit(rq.h.clientGroup(rq.client), now)
|
||||
}
|
||||
|
||||
// countedByRateLimits reports whether the rate limits count the request: its
|
||||
// client is not in SWW_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt
|
||||
// under SWW_RATE_LIMIT_EXEMPT_PATHS.
|
||||
func (rq *request) countedByRateLimits() bool {
|
||||
cfg := rq.h.config
|
||||
|
||||
return !isInside(rq.client, cfg.RateLimitExemptNets) &&
|
||||
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
|
||||
}
|
||||
@@ -59,7 +59,7 @@ func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
|
||||
if sent.want == http.StatusOK {
|
||||
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||
} else {
|
||||
wantLine(t, line, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
wantLine(t, line, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
if line.LimitHit != minute {
|
||||
t.Errorf("log line has limit_hit %q, want minute", line.LimitHit)
|
||||
@@ -72,6 +72,17 @@ func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimitRefusalNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{rateLimitPerMinute: "1"})
|
||||
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
wantHeuristic(t, line, "rate_limits", `{"window": "minute", "count": 2, "limit": 1}`)
|
||||
}
|
||||
|
||||
func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -91,7 +102,7 @@ func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) {
|
||||
status int
|
||||
action string
|
||||
}{
|
||||
{rateLimitPerMinute, "1", http.StatusForbidden, requestlog.ActionRateLimited},
|
||||
{rateLimitPerMinute, "1", http.StatusForbidden, requestlog.ActionClientIsRateLimited},
|
||||
{bytesLimitPerMinute, byteLimit, http.StatusOK, requestlog.ActionForward},
|
||||
} {
|
||||
t.Run(tc.setting, func(t *testing.T) {
|
||||
@@ -125,7 +136,7 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
|
||||
rateLimitPerMinute: "1",
|
||||
rateLimitExemptPaths: "/assets/,/favicon.ico",
|
||||
denyNets: denied,
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
|
||||
// The answers are kept before the requests, so that none waits for
|
||||
@@ -149,14 +160,17 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
|
||||
|
||||
// A path outside every prefix is counted: /assets is not under
|
||||
// /assets/, and breaks the limit.
|
||||
s.request(client, "/assets", http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.request(client, "/assets", http.StatusForbidden,
|
||||
requestlog.ActionClientIsRateLimited)
|
||||
|
||||
// A ban, SWW_DENY_NETS and the country lists still refuse a path
|
||||
// under a prefix.
|
||||
s.request(client, "/assets/app.js", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(denied, "/assets/app.js", http.StatusForbidden, requestlog.ActionDenied)
|
||||
// A ban, SWW_DENY_NETS and SWW_TAG_DENY still refuse a path under a
|
||||
// prefix.
|
||||
s.request(client, "/assets/app.js", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
s.request(denied, "/assets/app.js", http.StatusForbidden,
|
||||
requestlog.ActionClientDenied)
|
||||
s.request(fromKP, "/assets/app.js",
|
||||
http.StatusForbidden, requestlog.ActionCountryDenied)
|
||||
http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
}
|
||||
|
||||
func TestRateLimitCountsPathsThatAreNotExempt(t *testing.T) {
|
||||
@@ -200,7 +214,7 @@ func TestRateLimitCountsPathsThatAreNotExempt(t *testing.T) {
|
||||
// Counted, the second request breaks the limit of one request
|
||||
// a minute.
|
||||
s.request(client, sent, http.StatusOK, requestlog.ActionForward)
|
||||
s.request(client, sent, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.request(client, sent, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic refusal_limit. It reads SWW_ERROR_BURST_THRESHOLD and
|
||||
// SWW_BAN_RESPONSE.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// refusalLimit counts the request once it has been answered, if discriminator
|
||||
// refused it after a rule file match, a trap path or a Core Rule Set match, and
|
||||
// in observe mode if enforce mode would have: more than
|
||||
// SWW_ERROR_BURST_THRESHOLD such refusals of the client within a minute break
|
||||
// a limit, which bans the client. A client in SWW_ALLOW_NETS, which the
|
||||
// heuristics skip, is not counted, and nothing is while the threshold is off.
|
||||
// Its figures are the window, the count and the limit.
|
||||
type refusalLimit struct{}
|
||||
|
||||
func (refusalLimit) name() string { return "refusal_limit" }
|
||||
|
||||
func (refusalLimit) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
cfg := rq.h.config
|
||||
if cfg.ErrorBurstThreshold == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
if !rq.attack && !rq.blockedByBlockRule && !rq.blockedByCoreRuleSet {
|
||||
return nil
|
||||
}
|
||||
|
||||
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
|
||||
cfg.ErrorBurstThreshold)
|
||||
if !over {
|
||||
return nil
|
||||
}
|
||||
|
||||
// What the client was sent, or in observe mode would have been.
|
||||
status := rq.out.status
|
||||
|
||||
switch rq.line.WouldAction {
|
||||
case requestlog.ActionBlockedByBlockRule, requestlog.ActionBlockedByCoreRuleSet:
|
||||
status = http.StatusForbidden
|
||||
case requestlog.ActionTemporarilyBanned, requestlog.ActionPermanentlyBanned:
|
||||
status = cfg.BanResponse
|
||||
}
|
||||
|
||||
rq.banForLimit(now, hit, status)
|
||||
|
||||
return &verdict{figures: limitFigures(hit)}
|
||||
}
|
||||
+50
-18
@@ -37,9 +37,10 @@ func (r refused) send(s *sender, from string) logLine {
|
||||
switch r {
|
||||
case blockRule:
|
||||
return s.request(from, blockedPath, http.StatusForbidden,
|
||||
requestlog.ActionRuleBlocked)
|
||||
requestlog.ActionBlockedByBlockRule)
|
||||
case banRule:
|
||||
return s.request(from, probePath, http.StatusForbidden, requestlog.ActionBanned)
|
||||
return s.request(from, probePath, http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
s.t.Fatalf("no request for the refusal %d", r)
|
||||
@@ -71,9 +72,20 @@ func TestErrorBurstBreaksAtOneOverTheThreshold(t *testing.T) {
|
||||
name string
|
||||
// refusals are four, one over the threshold of three.
|
||||
refusals []refused
|
||||
// nextAction is the action of the client's next request: the next
|
||||
// request after a ban for a ban rule makes it permanent.
|
||||
nextAction string
|
||||
}{
|
||||
{"block rule", []refused{blockRule, blockRule, blockRule, blockRule}},
|
||||
{"a ban rule last", []refused{blockRule, blockRule, blockRule, banRule}},
|
||||
{
|
||||
"block rule",
|
||||
[]refused{blockRule, blockRule, blockRule, blockRule},
|
||||
requestlog.ActionTemporarilyBanned,
|
||||
},
|
||||
{
|
||||
"a ban rule last",
|
||||
[]refused{blockRule, blockRule, blockRule, banRule},
|
||||
requestlog.ActionPermanentlyBanned,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -95,17 +107,36 @@ func TestErrorBurstBreaksAtOneOverTheThreshold(t *testing.T) {
|
||||
// The fourth is answered as the others were, breaks the error
|
||||
// burst, and bans the client.
|
||||
line := tc.refusals[3].send(s, client)
|
||||
if line.LimitHit != requestlog.LimitHitErrorBurst ||
|
||||
if line.LimitHit != requestlog.LimitHitMinuteRefusals ||
|
||||
line.Offence != requestlog.OffenceLimit {
|
||||
t.Errorf("log line has limit_hit %q and offence %q, want error_burst "+
|
||||
t.Errorf("log line has limit_hit %q and offence %q, want minute_refusals "+
|
||||
"and limit", line.LimitHit, line.Offence)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, tc.nextAction)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefusalLimitBanNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startForErrorBurst(t, "1", nil)
|
||||
|
||||
const ruleFigures = `{"rule_id": "blocked", "rule_action": "block"}`
|
||||
|
||||
line := blockRule.send(s, client)
|
||||
wantHeuristic(t, line, "rule_files", ruleFigures)
|
||||
wantAfterAnswerHeuristic(t, line, "", "null")
|
||||
|
||||
// The second refusal is the request refusal_limit bans for, and the line
|
||||
// names both.
|
||||
line = blockRule.send(s, client)
|
||||
wantHeuristic(t, line, "rule_files", ruleFigures)
|
||||
wantAfterAnswerHeuristic(t, line, "refusal_limit",
|
||||
`{"window": "minute", "count": 2, "limit": 1}`)
|
||||
}
|
||||
|
||||
func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -150,7 +181,7 @@ func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) {
|
||||
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
|
||||
}
|
||||
|
||||
wantOffences := ratelimit.Offences{Limit: 1, RuleBlocked: 3}
|
||||
wantOffences := ratelimit.Offences{Limit: 1, BlockedByBlockRule: 3}
|
||||
if offences := historyOf(t, server, client).Offences; offences != wantOffences {
|
||||
t.Errorf("history counts the offences %+v, want %+v", offences, wantOffences)
|
||||
}
|
||||
@@ -160,7 +191,7 @@ func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) {
|
||||
`kind="refusals",window="minute"}`, 1)
|
||||
wantMetric(t, metrics, `sww_offences_total{instance="app",kind="limit"}`, 1)
|
||||
wantMetric(t, metrics,
|
||||
`sww_offences_total{instance="app",kind="rule_blocked"}`, 3)
|
||||
`sww_offences_total{instance="app",kind="blocked_by_block_rule"}`, 3)
|
||||
wantMetric(t, metrics, `sww_bans_made_total{cause="limit",instance="app"}`, 1)
|
||||
}
|
||||
|
||||
@@ -172,7 +203,7 @@ func TestErrorBurstIsNotLoweredForAClientWithLowerLimits(t *testing.T) {
|
||||
lookupTimeout: "1h",
|
||||
errorBurstThreshold: "2",
|
||||
rulesDir: writeRules(t, testRules),
|
||||
countryLimitPercent: countryDEHalf,
|
||||
tagLimitPercent: countryTagDEHalf,
|
||||
})
|
||||
|
||||
// Half of the threshold would be one, which the second refusal is over.
|
||||
@@ -246,7 +277,8 @@ func TestErrorBurstOffOrAtItsDefault(t *testing.T) {
|
||||
line = blockRule.send(s, client)
|
||||
}
|
||||
|
||||
if broken := line.LimitHit == requestlog.LimitHitErrorBurst; broken != tc.broken {
|
||||
broken := line.LimitHit == requestlog.LimitHitMinuteRefusals
|
||||
if broken != tc.broken {
|
||||
t.Errorf("the 31st refusal broke the error burst: %t, want %t",
|
||||
broken, tc.broken)
|
||||
}
|
||||
@@ -275,11 +307,11 @@ func TestErrorBurstCountsEachClientTheChecksApplyTo(t *testing.T) {
|
||||
blockRule.send(s, exempt)
|
||||
|
||||
line := blockRule.send(s, exempt)
|
||||
if line.LimitHit != requestlog.LimitHitErrorBurst {
|
||||
t.Errorf("log line has limit_hit %q, want error_burst", line.LimitHit)
|
||||
if line.LimitHit != requestlog.LimitHitMinuteRefusals {
|
||||
t.Errorf("log line has limit_hit %q, want minute_refusals", line.LimitHit)
|
||||
}
|
||||
|
||||
s.get(exempt, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(exempt, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestErrorBurstBanSetsTheRefusalsBackToZero(t *testing.T) {
|
||||
@@ -322,7 +354,7 @@ func TestObserveModeLogsAndAlertsTheErrorBurst(t *testing.T) {
|
||||
for range 2 {
|
||||
line := s.request(otherClient, blockedPath, http.StatusOK,
|
||||
requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionBanned)
|
||||
wantWouldAction(t, line, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
if line.LimitHit != "" {
|
||||
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
|
||||
@@ -334,10 +366,10 @@ func TestObserveModeLogsAndAlertsTheErrorBurst(t *testing.T) {
|
||||
s.request(client, blockedPath, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line := s.request(client, blockedPath, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionRuleBlocked)
|
||||
wantWouldAction(t, line, requestlog.ActionBlockedByBlockRule)
|
||||
|
||||
if line.LimitHit != requestlog.LimitHitErrorBurst || line.BanExpires != "" {
|
||||
t.Errorf("log line has limit_hit %q and ban_expires %q, want error_burst "+
|
||||
if line.LimitHit != requestlog.LimitHitMinuteRefusals || line.BanExpires != "" {
|
||||
t.Errorf("log line has limit_hit %q and ban_expires %q, want minute_refusals "+
|
||||
"and none", line.LimitHit, line.BanExpires)
|
||||
}
|
||||
|
||||
@@ -1,91 +1,17 @@
|
||||
package discriminator
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
// What the reputation heuristics share: blocklists, crowdsec, dnsbl and
|
||||
// abuseipdb.
|
||||
|
||||
import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
)
|
||||
|
||||
// deny is the SWW_BLOCKLIST_ACTION and the SWW_REPUTATION_ACTION that
|
||||
// refuses the requests of a client a source lists.
|
||||
const deny = "deny"
|
||||
|
||||
// blocklistDenied notes the blocklists that list the client, as
|
||||
// noteListed does, and reports whether SWW_BLOCKLIST_ACTION, being deny,
|
||||
// refuses the request. Being limit, it lowers the client's limits instead
|
||||
// (see limitPercentages), and being log, it does nothing more.
|
||||
func (rq *request) blocklistDenied() bool {
|
||||
listedBy := rq.h.lists.ListedBy(rq.client)
|
||||
rq.blocklisted = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a blocklist")
|
||||
|
||||
return rq.blocklisted && rq.h.config.BlocklistAction == deny
|
||||
}
|
||||
|
||||
// crowdSecBanned reports whether a decision of the CrowdSec decision list
|
||||
// on the client is in force at now. If one is, it notes the list, as
|
||||
// noteHit does, and bans the client until that decision ends.
|
||||
func (rq *request) crowdSecBanned(now time.Time) bool {
|
||||
decision, listed := rq.h.lists.CrowdSecDecision(rq.client, now)
|
||||
if !listed {
|
||||
return false
|
||||
}
|
||||
|
||||
rq.noteHit(bans.ReputationHit{Source: rq.h.config.CrowdSecDecisionsURL},
|
||||
"listed by the CrowdSec decision list")
|
||||
rq.banForCrowdSec(now, decision)
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// dnsblDenied notes the DNSBL zones whose verdict lists the client, as
|
||||
// noteListed does, and reports whether SWW_REPUTATION_ACTION, being
|
||||
// deny, refuses the request. Being limit, it lowers the client's limits
|
||||
// instead (see limitPercentages), and being log, it does nothing more. A
|
||||
// zone without a verdict on the client is asked about it in the
|
||||
// background, and the request does not wait for the answer. ctx is the
|
||||
// request's own context.
|
||||
func (rq *request) dnsblDenied(ctx context.Context) bool {
|
||||
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
|
||||
rq.dnsblListed = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a DNSBL zone")
|
||||
|
||||
return rq.dnsblListed && rq.h.config.ReputationAction == deny
|
||||
}
|
||||
|
||||
// abuseIPDBDenied notes AbuseIPDB, as noteHit does, with the score, when
|
||||
// its score of the client is a hit, and reports whether
|
||||
// SWW_REPUTATION_ACTION, being deny, refuses the request, as dnsblDenied
|
||||
// does for a zone. While SWW_ABUSEIPDB_KEY is unset it does nothing. A
|
||||
// client without a score is checked in the background, by the request's
|
||||
// address, if its history counts an offence, and the request does not
|
||||
// wait for the answer. The score is then used for each address of the
|
||||
// client. ctx is the request's own context.
|
||||
func (rq *request) abuseIPDBDenied(ctx context.Context) bool {
|
||||
if rq.h.config.AbuseIPDBKey == "" {
|
||||
return false
|
||||
}
|
||||
|
||||
client := rq.h.clientGroup(rq.client)
|
||||
held, _ := rq.h.limiter.Client(client)
|
||||
offender := held.History.Offences != ratelimit.Offences{}
|
||||
|
||||
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
|
||||
if !hit {
|
||||
return false
|
||||
}
|
||||
|
||||
rq.abuseIPDBHit = true
|
||||
rq.noteHit(bans.ReputationHit{Source: reputation.AbuseIPDBSource, Score: &score},
|
||||
"scored by AbuseIPDB at or over SWW_ABUSEIPDB_MIN_SCORE")
|
||||
|
||||
return rq.h.config.ReputationAction == deny
|
||||
}
|
||||
|
||||
// noteListed notes each of sources, the URLs of the blocklists or the
|
||||
// DNSBL zones, their keys masked, that list the client, as noteHit does,
|
||||
// with reason.
|
||||
|
||||
@@ -49,12 +49,12 @@ const (
|
||||
func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
forward, denied := requestlog.ActionForward, requestlog.ActionDenied
|
||||
forward, denied := requestlog.ActionForward, requestlog.ActionClientDenied
|
||||
|
||||
for _, tc := range []struct {
|
||||
action string
|
||||
// statuses and actions are those of a listed client's three
|
||||
// requests, and percent their limit_percent, as percentText gives it.
|
||||
// requests, and percent their rate_limit_percent, as percentText gives it.
|
||||
statuses []int
|
||||
actions []string
|
||||
percent string
|
||||
@@ -66,7 +66,7 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
|
||||
{
|
||||
// Half of 4 requests a minute: the third breaks the limit.
|
||||
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||
[]string{forward, forward, requestlog.ActionRateLimited},
|
||||
[]string{forward, forward, requestlog.ActionClientIsRateLimited},
|
||||
"50 from " + blocklistAction,
|
||||
},
|
||||
{
|
||||
@@ -90,8 +90,8 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
|
||||
for i := range 3 {
|
||||
line := s.get(from, tc.statuses[i], tc.actions[i])
|
||||
wantReputation(t, line, dropURL)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent,
|
||||
line.LimitPercentSetting, tc.percent)
|
||||
wantPercent(t, "rate_limit_percent", line.RateLimitPercent,
|
||||
line.RateLimitPercentSetting, tc.percent)
|
||||
|
||||
// A request refused for the list is not counted.
|
||||
counted := line.fields["counts"] != nil
|
||||
@@ -106,7 +106,8 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
|
||||
for range 3 {
|
||||
line := s.get(unplaced, http.StatusOK, forward)
|
||||
wantReputation(t, line)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
none)
|
||||
}
|
||||
|
||||
@@ -118,11 +119,61 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlocklistsComeAfterTheCountryListsAndSkipAllowNets(t *testing.T) {
|
||||
func TestReputationRefusalNamesItsHeuristicAndWhatListedTheClient(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
heuristic string
|
||||
env map[string]string
|
||||
load func(*testing.T, *discriminator.Server)
|
||||
figures string
|
||||
}{
|
||||
{
|
||||
"blocklists",
|
||||
map[string]string{blocklistURLs: dropURL, blocklistAction: actionDeny},
|
||||
func(t *testing.T, server *discriminator.Server) {
|
||||
t.Helper()
|
||||
loadLists(t, server, map[string][]string{dropURL: {client}})
|
||||
},
|
||||
`{"lists": ["` + dropURL + `"]}`,
|
||||
},
|
||||
{
|
||||
"dnsbl",
|
||||
map[string]string{
|
||||
dnsblZones: dnsblZone, dnsblResolver: noResolver,
|
||||
reputationAction: actionDeny,
|
||||
},
|
||||
func(_ *testing.T, server *discriminator.Server) {
|
||||
loadVerdicts(server, map[string][]string{client: {dnsblZone}})
|
||||
},
|
||||
`{"zones": ["` + dnsblZone + `"]}`,
|
||||
},
|
||||
{
|
||||
"abuseipdb",
|
||||
map[string]string{abuseIPDBKey: accountKey, reputationAction: actionDeny},
|
||||
func(_ *testing.T, server *discriminator.Server) {
|
||||
loadScores(server, map[string]int64{client: 80})
|
||||
},
|
||||
`{"score": 80, "min_score": 75}`,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.heuristic, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, tc.env)
|
||||
tc.load(t, server)
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, tc.heuristic, tc.figures)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlocklistsComeAfterAllowedCountriesAndSkipAllowNets(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, queue := startWithLookups(t, map[string]string{
|
||||
blocklistURLs: dropURL, deniedCountries: "kp", allowNets: fromDE,
|
||||
blocklistURLs: dropURL, allowedCountries: "de", allowNets: fromDE,
|
||||
})
|
||||
loadLists(t, server, map[string][]string{dropURL: {fromDE, fromKP}})
|
||||
|
||||
@@ -143,7 +194,7 @@ func TestObserveModeForwardsAClientABlocklistDeniesAndAlertsIt(t *testing.T) {
|
||||
loadLists(t, server, map[string][]string{dropURL: {fromDE}})
|
||||
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionDenied)
|
||||
wantWouldAction(t, line, requestlog.ActionClientDenied)
|
||||
wantReputation(t, line, dropURL)
|
||||
|
||||
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
@@ -157,31 +208,33 @@ func TestBlocklistLimitTakesPartInTheLowestPercentageOfEveryLimit(t *testing.T)
|
||||
|
||||
for _, tc := range []struct {
|
||||
action, asnPercent string
|
||||
// want is the upload's limit_percent and bytes_percent, as
|
||||
// want is the upload's rate_limit_percent and bytes_limit_percent, as
|
||||
// percentText gives them, and limitHit its limit_hit.
|
||||
want, limitHit string
|
||||
}{
|
||||
{limitHalf, asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
|
||||
{limitQuarter, asnDEHalf, "25 from " + blocklistAction, minuteBytes},
|
||||
{limitHalf, asnTagDEQuarter, "25 from " + tagLimitPercent, minuteBytes},
|
||||
{limitQuarter, asnTagDEHalf, "25 from " + blocklistAction, minuteBytes},
|
||||
// The AS number's, the first of two alike.
|
||||
{limitQuarter, asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
|
||||
{actionLog, asnDE + ":100", none, ""},
|
||||
{limitQuarter, asnTagDEQuarter, "25 from " + tagLimitPercent, minuteBytes},
|
||||
{actionLog, "asn=64511:25", none, ""},
|
||||
} {
|
||||
t.Run(tc.action+" "+tc.asnPercent, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
bytesLimitPerMinute: twoUploads, blocklistURLs: dropURL,
|
||||
blocklistAction: tc.action, asnLimitPercent: tc.asnPercent,
|
||||
blocklistAction: tc.action, tagLimitPercent: tc.asnPercent,
|
||||
})
|
||||
loadLists(t, server, map[string][]string{dropURL: {fromDE}})
|
||||
|
||||
// The upload's 100 bytes are over 49, a quarter of 199, and 99,
|
||||
// half of it, and within 199.
|
||||
line := s.uploadFrom(fromDE)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
tc.want)
|
||||
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting,
|
||||
tc.want)
|
||||
|
||||
if line.LimitHit != tc.limitHit {
|
||||
@@ -196,30 +249,30 @@ func TestASNLimitPercentFileCountsAsTheSettingDoesTheLowerWinning(t *testing.T)
|
||||
|
||||
const (
|
||||
fromURL = "25 from " + asnLimitPercentURL
|
||||
fromSetting = "25 from " + asnLimitPercent
|
||||
fromSetting = "25 from " + tagLimitPercent
|
||||
)
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
env map[string]string
|
||||
file string
|
||||
// limitPercent and bytesPercent are the upload's, as percentText
|
||||
// gives them.
|
||||
limitPercent, bytesPercent string
|
||||
// rateLimitPercent and bytesLimitPercent are the upload's, as
|
||||
// percentText gives them.
|
||||
rateLimitPercent, bytesLimitPercent string
|
||||
}{
|
||||
{"the file's alone", nil, asnDEQuarter, fromURL, fromURL},
|
||||
{
|
||||
"the file's, lower than the setting's",
|
||||
map[string]string{asnLimitPercent: asnDEHalf}, asnDEQuarter, fromURL, fromURL,
|
||||
map[string]string{tagLimitPercent: asnTagDEHalf}, asnDEQuarter, fromURL, fromURL,
|
||||
},
|
||||
{
|
||||
"the setting's, lower than the file's",
|
||||
map[string]string{asnLimitPercent: asnDEQuarter}, asnDEHalf,
|
||||
map[string]string{tagLimitPercent: asnTagDEQuarter}, asnDEHalf,
|
||||
fromSetting, fromSetting,
|
||||
},
|
||||
{
|
||||
"the setting's, the first of two alike",
|
||||
map[string]string{asnLimitPercent: asnDEQuarter}, asnDEQuarter,
|
||||
map[string]string{tagLimitPercent: asnTagDEQuarter}, asnDEQuarter,
|
||||
fromSetting, fromSetting,
|
||||
},
|
||||
{"none, for an AS number the file does not list", nil, asnKP + ":25", none, none},
|
||||
@@ -238,10 +291,12 @@ func TestASNLimitPercentFileCountsAsTheSettingDoesTheLowerWinning(t *testing.T)
|
||||
loadLists(t, server, map[string][]string{asnURL: {"# by AS number", tc.file}})
|
||||
|
||||
line := s.uploadFrom(fromDE)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
tc.limitPercent)
|
||||
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
|
||||
tc.bytesPercent)
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
tc.rateLimitPercent)
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting,
|
||||
tc.bytesLimitPercent)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -330,12 +385,12 @@ const (
|
||||
func TestEachReputationActionForAClientADNSBLZoneLists(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
forward, denied := requestlog.ActionForward, requestlog.ActionDenied
|
||||
forward, denied := requestlog.ActionForward, requestlog.ActionClientDenied
|
||||
|
||||
for _, tc := range []struct {
|
||||
action string
|
||||
// statuses and actions are those of a listed client's three
|
||||
// requests, and percent their limit_percent, as percentText gives it.
|
||||
// requests, and percent their rate_limit_percent, as percentText gives it.
|
||||
statuses []int
|
||||
actions []string
|
||||
percent string
|
||||
@@ -347,7 +402,7 @@ func TestEachReputationActionForAClientADNSBLZoneLists(t *testing.T) {
|
||||
{
|
||||
// Half of 4 requests a minute: the third breaks the limit.
|
||||
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||
[]string{forward, forward, requestlog.ActionRateLimited},
|
||||
[]string{forward, forward, requestlog.ActionClientIsRateLimited},
|
||||
"50 from " + reputationAction,
|
||||
},
|
||||
{
|
||||
@@ -372,8 +427,8 @@ func TestEachReputationActionForAClientADNSBLZoneLists(t *testing.T) {
|
||||
line := s.get(from, tc.statuses[i], tc.actions[i])
|
||||
// In the order SWW_DNSBL_ZONES names them.
|
||||
wantReputation(t, line, listedBy[from]...)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent,
|
||||
line.LimitPercentSetting, tc.percent)
|
||||
wantPercent(t, "rate_limit_percent", line.RateLimitPercent,
|
||||
line.RateLimitPercentSetting, tc.percent)
|
||||
|
||||
// A request refused for the verdict is not counted.
|
||||
counted := line.fields["counts"] != nil
|
||||
@@ -388,7 +443,8 @@ func TestEachReputationActionForAClientADNSBLZoneLists(t *testing.T) {
|
||||
for range 3 {
|
||||
line := s.get(unplaced, http.StatusOK, forward)
|
||||
wantReputation(t, line)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
none)
|
||||
}
|
||||
|
||||
@@ -418,7 +474,7 @@ func TestDNSBLZonesComeAfterTheBlocklistsAndSkipAllowNets(t *testing.T) {
|
||||
// The blocklist refuses fromKP before its verdict is looked at, and
|
||||
// fromDE, in SWW_ALLOW_NETS, is not checked at all: neither is noted
|
||||
// for the zone, nor alerted, nor asked about.
|
||||
wantReputation(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionDenied),
|
||||
wantReputation(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionClientDenied),
|
||||
dropURL)
|
||||
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward))
|
||||
|
||||
@@ -432,6 +488,46 @@ func TestDNSBLZonesComeAfterTheBlocklistsAndSkipAllowNets(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestReputationSourcesTagTheRequestWhateverTheirAction(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
env map[string]string
|
||||
load func(*discriminator.Server)
|
||||
want []string
|
||||
}{
|
||||
{
|
||||
"dnsbl",
|
||||
map[string]string{
|
||||
dnsblZones: dnsblZone, dnsblResolver: noResolver, reputationAction: actionLog,
|
||||
},
|
||||
func(server *discriminator.Server) {
|
||||
loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}})
|
||||
},
|
||||
[]string{"asn=64496", "country=DE", "dnsbl=" + dnsblZone},
|
||||
},
|
||||
{
|
||||
abuseipdb,
|
||||
map[string]string{abuseIPDBKey: accountKey, reputationAction: actionLog},
|
||||
func(server *discriminator.Server) {
|
||||
loadScores(server, map[string]int64{fromDE: 80})
|
||||
},
|
||||
[]string{abuseipdb, "asn=64496", "country=DE"},
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, tc.env)
|
||||
tc.load(server)
|
||||
|
||||
wantTags(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||
tc.want...)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestObserveModeForwardsAClientADNSBLZoneDeniesAndAlertsIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -442,7 +538,7 @@ func TestObserveModeForwardsAClientADNSBLZoneDeniesAndAlertsIt(t *testing.T) {
|
||||
loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}})
|
||||
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionDenied)
|
||||
wantWouldAction(t, line, requestlog.ActionClientDenied)
|
||||
wantReputation(t, line, dnsblZone)
|
||||
|
||||
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
@@ -456,7 +552,7 @@ func TestReputationLimitTakesPartInTheLowestPercentageOfEveryLimit(t *testing.T)
|
||||
|
||||
for _, tc := range []struct {
|
||||
blocklistAction, reputationAction string
|
||||
// want is the request's limit_percent and bytes_percent, as
|
||||
// want is the request's rate_limit_percent and bytes_limit_percent, as
|
||||
// percentText gives them.
|
||||
want string
|
||||
}{
|
||||
@@ -481,9 +577,11 @@ func TestReputationLimitTakesPartInTheLowestPercentageOfEveryLimit(t *testing.T)
|
||||
// Named by the blocklist, then by the zone.
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
wantReputation(t, line, dropURL, dnsblZone)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
tc.want)
|
||||
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting,
|
||||
tc.want)
|
||||
})
|
||||
}
|
||||
@@ -651,8 +749,8 @@ func TestOnlyAClientThatHasCommittedAnOffenceIsCheckedWithAbuseIPDB(t *testing.T
|
||||
s.get(fromDE, http.StatusOK, forward)
|
||||
s.get(fromDE, http.StatusOK, forward)
|
||||
s.get(fromKP, http.StatusOK, forward)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
wantAbuseIPDBChecks(t, server, 0)
|
||||
|
||||
// Once the ban has ended, fromDE's first request has it checked in the
|
||||
@@ -691,7 +789,7 @@ func TestIPv6ClientCostsOneAbuseIPDBCheckWhicheverOfItsAddressesSends(t *testing
|
||||
s.get(addresses[0], http.StatusOK, forward)
|
||||
}
|
||||
|
||||
s.get(addresses[0], http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(addresses[0], http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
clk.advance(time.Hour)
|
||||
|
||||
// Once the ban has ended, which set its counters back to zero, a
|
||||
@@ -724,20 +822,20 @@ func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(t *test
|
||||
want ratelimit.Offences
|
||||
}{
|
||||
{
|
||||
"a block rule", blockedPath, http.StatusForbidden, requestlog.ActionRuleBlocked,
|
||||
ratelimit.Offences{RuleBlocked: 1},
|
||||
"a block rule", blockedPath, http.StatusForbidden,
|
||||
requestlog.ActionBlockedByBlockRule, ratelimit.Offences{BlockedByBlockRule: 1},
|
||||
},
|
||||
{
|
||||
"the Core Rule Set", sqlInjection, http.StatusForbidden,
|
||||
requestlog.ActionWAFBlocked, ratelimit.Offences{WAFBlocked: 1},
|
||||
requestlog.ActionBlockedByCoreRuleSet, ratelimit.Offences{BlockedByCoreRuleSet: 1},
|
||||
},
|
||||
{
|
||||
"a ban rule", probePath, http.StatusForbidden, requestlog.ActionBanned,
|
||||
"a ban rule", probePath, http.StatusForbidden, requestlog.ActionTemporarilyBanned,
|
||||
ratelimit.Offences{Attack: 1},
|
||||
},
|
||||
{
|
||||
"a trap path", "/xmlrpc.php", http.StatusForbidden, requestlog.ActionBanned,
|
||||
ratelimit.Offences{Attack: 1},
|
||||
"a trap path", "/xmlrpc.php", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned, ratelimit.Offences{Attack: 1},
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -768,12 +866,12 @@ func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(t *test
|
||||
func TestEachReputationActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
forward, denied := requestlog.ActionForward, requestlog.ActionDenied
|
||||
forward, denied := requestlog.ActionForward, requestlog.ActionClientDenied
|
||||
|
||||
for _, tc := range []struct {
|
||||
action string
|
||||
// statuses and actions are those of fromDE's three requests, and
|
||||
// percent their limit_percent, as percentText gives it.
|
||||
// percent their rate_limit_percent, as percentText gives it.
|
||||
statuses []int
|
||||
actions []string
|
||||
percent string
|
||||
@@ -785,7 +883,7 @@ func TestEachReputationActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(t *test
|
||||
{
|
||||
// Half of 4 requests a minute: the third breaks the limit.
|
||||
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||
[]string{forward, forward, requestlog.ActionRateLimited},
|
||||
[]string{forward, forward, requestlog.ActionClientIsRateLimited},
|
||||
"50 from " + reputationAction,
|
||||
},
|
||||
{
|
||||
@@ -806,7 +904,8 @@ func TestEachReputationActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(t *test
|
||||
for i := range 3 {
|
||||
line := s.get(fromDE, tc.statuses[i], tc.actions[i])
|
||||
wantReputation(t, line, abuseipdb)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
tc.percent)
|
||||
|
||||
// A request refused for the score is not counted.
|
||||
@@ -820,7 +919,8 @@ func TestEachReputationActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(t *test
|
||||
for range 3 {
|
||||
line := s.get(fromKP, http.StatusOK, forward)
|
||||
wantReputation(t, line)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
none)
|
||||
}
|
||||
|
||||
@@ -882,7 +982,7 @@ func TestWithoutAnAbuseIPDBKeyNoClientIsCheckedNorAScoreUsed(t *testing.T) {
|
||||
// fromDE's score is not used, and once it has committed an offence it
|
||||
// is not checked either.
|
||||
wantReputation(t, s.get(fromDE, http.StatusOK, forward))
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
clk.advance(time.Hour)
|
||||
wantReputation(t, s.get(fromDE, http.StatusOK, forward))
|
||||
wantAbuseIPDBChecks(t, server, 0)
|
||||
@@ -906,10 +1006,10 @@ func TestBanNotesNameEachReputationSourceThatListedTheClient(t *testing.T) {
|
||||
}{
|
||||
{"for a broken rate limit", func(s *sender, from string) {
|
||||
s.get(from, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(from, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(from, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
}},
|
||||
{"for a clear sign of attack", func(s *sender, from string) {
|
||||
s.request(from, probePath, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(from, probePath, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
|
||||
@@ -48,16 +48,16 @@ type decision struct {
|
||||
// so that not waiting never lowers its client's limits.
|
||||
skippedLookupWait bool
|
||||
// counted is true for a request the rate limits counted, whose bytes
|
||||
// the byte limits count once it has ended. limitPercent and
|
||||
// bytesPercent are then its client's limit percentages for the rate
|
||||
// the byte limits count once it has ended. rateLimitPercent and
|
||||
// bytesLimitPercent are then its client's limit percentages for the rate
|
||||
// limits and for the byte limits.
|
||||
counted bool
|
||||
limitPercent, bytesPercent percentage
|
||||
rateLimitPercent, bytesLimitPercent percentage
|
||||
// attack is true for a request that matched a ban rule or asked for a
|
||||
// trap path, ruleBlocked for one a block rule refused, and wafBlocked for
|
||||
// one the Core Rule Set refused, each an offence its client's history
|
||||
// counts.
|
||||
attack, ruleBlocked, wafBlocked bool
|
||||
// trap path, blockedByBlockRule for one a block rule refused, and
|
||||
// blockedByCoreRuleSet for one the Core Rule Set refused, each an offence
|
||||
// its client's history counts.
|
||||
attack, blockedByBlockRule, blockedByCoreRuleSet bool
|
||||
// reputation is the reputation sources that list the client, for the
|
||||
// notes of a ban the request makes.
|
||||
reputation []bans.ReputationHit
|
||||
@@ -138,6 +138,15 @@ func (h *handler) startRequest(
|
||||
return rq
|
||||
}
|
||||
|
||||
// addTag adds tag to the request's tags, which stay sorted and hold each tag
|
||||
// once.
|
||||
func (rq *request) addTag(tag string) {
|
||||
at, found := slices.BinarySearch(rq.line.Tags, tag)
|
||||
if !found {
|
||||
rq.line.Tags = slices.Insert(rq.line.Tags, at, tag)
|
||||
}
|
||||
}
|
||||
|
||||
// requestHeaders returns the headers of r that names lists, by name in
|
||||
// lower case, each with its values joined by ", ". Authorization, Cookie
|
||||
// and Set-Cookie are never among them, whatever names says.
|
||||
@@ -160,58 +169,49 @@ func requestHeaders(r *http.Request, names []string) map[string]string {
|
||||
}
|
||||
|
||||
// check is the one place where a request can be refused. It returns nil to let
|
||||
// the request through. The checks of checkClient decide: a request they refuse
|
||||
// is answered with SWW_BAN_RESPONSE, 403 for a block rule or the Core Rule Set,
|
||||
// or 431 for the header limits. In observe mode nothing is refused, and the log
|
||||
// line names what enforce mode would have done. ctx is the request's own context.
|
||||
// the request through. The heuristics of checkClient decide: a request they
|
||||
// refuse is answered with SWW_BAN_RESPONSE, 403 for a block rule or the Core
|
||||
// Rule Set, or 431 for the header limits. In observe mode nothing is refused,
|
||||
// and the log line names what enforce mode would have done. ctx is the
|
||||
// request's own context.
|
||||
func (rq *request) check(ctx context.Context) *refusal {
|
||||
action := rq.checkClient(ctx)
|
||||
v := rq.checkClient(ctx)
|
||||
if v == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
action := v.action
|
||||
|
||||
switch {
|
||||
case action == "":
|
||||
return nil
|
||||
case rq.h.config.Observe:
|
||||
rq.line.WouldAction = action
|
||||
|
||||
return nil
|
||||
case action == requestlog.ActionRuleBlocked || action == requestlog.ActionWAFBlocked:
|
||||
case action == requestlog.ActionBlockedByBlockRule ||
|
||||
action == requestlog.ActionBlockedByCoreRuleSet:
|
||||
return &refusal{status: http.StatusForbidden, action: action}
|
||||
case action == requestlog.ActionTooLarge:
|
||||
case action == requestlog.ActionRequestHeaderTooLarge:
|
||||
return &refusal{status: http.StatusRequestHeaderFieldsTooLarge, action: action}
|
||||
default:
|
||||
return rq.banResponse(action)
|
||||
}
|
||||
}
|
||||
|
||||
// checkClient runs the checks on the request's client, and returns the
|
||||
// action of the first that refuses the request, or "" when none does. A
|
||||
// client in SWW_ALLOW_NETS skips them, and is not looked up. For any
|
||||
// other client, SWW_DENY_NETS comes first, then a ban on its netblock,
|
||||
// so that a client either refuses is not looked up, then the lookup of
|
||||
// its AS number and country, then the country lists, then the blocklists,
|
||||
// then the CrowdSec decision list, which bans the client it lists, then
|
||||
// the DNSBL zones' verdicts, and then AbuseIPDB's score; a request any of
|
||||
// them refuses is not counted for the rate limits. Then come the
|
||||
// rate limits, unless the client is in SWW_RATE_LIMIT_EXEMPT_NETS or the
|
||||
// request's path is exempt under SWW_RATE_LIMIT_EXEMPT_PATHS, so that
|
||||
// every other request is counted, each of them by the client's limit
|
||||
// percentages, then SWW_TRAP_PATHS, then the rule files, then the header
|
||||
// limits, and last the Core Rule Set. A request exempt from the rate limits
|
||||
// is exempt from the byte limits too. ctx is the request's own context.
|
||||
func (rq *request) checkClient(ctx context.Context) string {
|
||||
cfg := rq.h.config
|
||||
if isInside(rq.client, cfg.AllowNets) {
|
||||
return ""
|
||||
// checkClient runs the heuristics of beforeLookup on the request, then looks
|
||||
// up the client's AS number and country, then runs those of afterLookup. It
|
||||
// returns the first verdict, or nil when none gives one. A client in
|
||||
// SWW_ALLOW_NETS skips them all, and is not looked up. ctx is the request's
|
||||
// own context.
|
||||
func (rq *request) checkClient(ctx context.Context) *verdict {
|
||||
if isInside(rq.client, rq.h.config.AllowNets) {
|
||||
return nil
|
||||
}
|
||||
|
||||
now := rq.h.now()
|
||||
|
||||
if isInside(rq.client, cfg.DenyNets) {
|
||||
return requestlog.ActionDenied
|
||||
}
|
||||
|
||||
if rq.banned(now) {
|
||||
return requestlog.ActionBanned
|
||||
v := rq.runHeuristics(ctx, beforeLookup(), now)
|
||||
if v != nil {
|
||||
return v
|
||||
}
|
||||
|
||||
rq.lookUp(ctx, now)
|
||||
@@ -220,59 +220,7 @@ func (rq *request) checkClient(ctx context.Context) string {
|
||||
// again: a request counted at the time it arrived could be counted
|
||||
// after a later one of its client's, which the rate limits would take
|
||||
// for the clock set back.
|
||||
now = rq.h.now()
|
||||
|
||||
if rq.countryDenied() {
|
||||
return requestlog.ActionCountryDenied
|
||||
}
|
||||
|
||||
if rq.blocklistDenied() {
|
||||
return requestlog.ActionDenied
|
||||
}
|
||||
|
||||
if rq.crowdSecBanned(now) {
|
||||
return requestlog.ActionBanned
|
||||
}
|
||||
|
||||
if rq.dnsblDenied(ctx) || rq.abuseIPDBDenied(ctx) {
|
||||
return requestlog.ActionDenied
|
||||
}
|
||||
|
||||
rq.counted = rq.countedByRateLimits()
|
||||
if rq.counted {
|
||||
rq.limitPercent, rq.bytesPercent = rq.limitPercentages()
|
||||
rq.line.LimitPercent, rq.line.LimitPercentSetting = rq.limitPercent.logged()
|
||||
rq.line.BytesPercent, rq.line.BytesPercentSetting = rq.bytesPercent.logged()
|
||||
}
|
||||
|
||||
if rq.counted && rq.limitBroken(now) {
|
||||
return requestlog.ActionRateLimited
|
||||
}
|
||||
|
||||
if rq.trapPath(now) {
|
||||
return requestlog.ActionBanned
|
||||
}
|
||||
|
||||
action := rq.checkRules(now)
|
||||
if action != "" {
|
||||
return action
|
||||
}
|
||||
|
||||
if rq.overHeaderLimits() {
|
||||
return requestlog.ActionTooLarge
|
||||
}
|
||||
|
||||
return rq.checkCoreRuleSet()
|
||||
}
|
||||
|
||||
// countedByRateLimits reports whether the rate limits count the request: its
|
||||
// client is not in SWW_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt
|
||||
// under SWW_RATE_LIMIT_EXEMPT_PATHS.
|
||||
func (rq *request) countedByRateLimits() bool {
|
||||
cfg := rq.h.config
|
||||
|
||||
return !isInside(rq.client, cfg.RateLimitExemptNets) &&
|
||||
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
|
||||
return rq.runHeuristics(ctx, afterLookup(), rq.h.now())
|
||||
}
|
||||
|
||||
// pathExempt reports whether r is exempt under prefixes,
|
||||
@@ -314,8 +262,8 @@ func (rq *request) finish() {
|
||||
}
|
||||
|
||||
duration := time.Since(rq.start)
|
||||
line.DurationTotal = requestlog.Milliseconds(duration)
|
||||
line.DurationChecks = timing(rq.start, rq.checked)
|
||||
line.DurationTotalMS = requestlog.Milliseconds(duration)
|
||||
line.DurationChecksMS = timing(rq.start, rq.checked)
|
||||
|
||||
// Counted before the log line is written, so that the metrics count
|
||||
// every request whose line is out. An allowed request is the exception:
|
||||
@@ -363,8 +311,8 @@ func (rq *request) addToHistory() {
|
||||
ResponseBytes: rq.out.bytes,
|
||||
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
|
||||
Attack: rq.attack,
|
||||
RuleBlocked: rq.ruleBlocked,
|
||||
WAFBlocked: rq.wafBlocked,
|
||||
BlockedByBlockRule: rq.blockedByBlockRule,
|
||||
BlockedByCoreRuleSet: rq.blockedByCoreRuleSet,
|
||||
}
|
||||
|
||||
rq.h.limiter.AddToHistory(rq.h.clientGroup(rq.client), rq.h.now(), request)
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package discriminator_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"math"
|
||||
"net/http"
|
||||
@@ -155,9 +156,9 @@ func wantBareLine(t *testing.T, line logLine) {
|
||||
|
||||
// withTimings returns want with the timings of line.
|
||||
func withTimings(line logLine, want requestlog.Line) requestlog.Line {
|
||||
want.DurationTotal = line.DurationTotal
|
||||
want.DurationChecks = line.DurationChecks
|
||||
want.DurationUpstreamTotal = line.DurationUpstreamTotal
|
||||
want.DurationTotalMS = line.DurationTotalMS
|
||||
want.DurationChecksMS = line.DurationChecksMS
|
||||
want.DurationUpstreamTotalMS = line.DurationUpstreamTotalMS
|
||||
|
||||
return want
|
||||
}
|
||||
@@ -225,10 +226,10 @@ func TestTimingsAreInOrder(t *testing.T) {
|
||||
|
||||
// A request passed to the app has every timing; one refused, none of
|
||||
// the app's; the health check, which runs no check, only the total.
|
||||
wantTimings(t, forwarded, "duration_total", "duration_checks",
|
||||
"duration_upstream_total")
|
||||
wantTimings(t, refused, "duration_total", "duration_checks")
|
||||
wantTimings(t, health, "duration_total")
|
||||
wantTimings(t, forwarded, "duration_total_ms", "duration_checks_ms",
|
||||
"duration_upstream_total_ms")
|
||||
wantTimings(t, refused, "duration_total_ms", "duration_checks_ms")
|
||||
wantTimings(t, health, "duration_total_ms")
|
||||
|
||||
if t.Failed() {
|
||||
return
|
||||
@@ -236,9 +237,9 @@ func TestTimingsAreInOrder(t *testing.T) {
|
||||
|
||||
// In whole microseconds, as they are logged, so that the sum below is
|
||||
// exact.
|
||||
total := microseconds(forwarded.DurationTotal)
|
||||
checks := microseconds(*forwarded.DurationChecks)
|
||||
upstream := microseconds(*forwarded.DurationUpstreamTotal)
|
||||
total := microseconds(forwarded.DurationTotalMS)
|
||||
checks := microseconds(*forwarded.DurationChecksMS)
|
||||
upstream := microseconds(*forwarded.DurationUpstreamTotalMS)
|
||||
|
||||
// The checks end before the request is handed to the app.
|
||||
if checks+upstream > total {
|
||||
@@ -246,9 +247,9 @@ func TestTimingsAreInOrder(t *testing.T) {
|
||||
total, checks, upstream)
|
||||
}
|
||||
|
||||
if *refused.DurationChecks > refused.DurationTotal {
|
||||
if *refused.DurationChecksMS > refused.DurationTotalMS {
|
||||
t.Errorf("refused request's checks took %v of %v milliseconds",
|
||||
*refused.DurationChecks, refused.DurationTotal)
|
||||
*refused.DurationChecksMS, refused.DurationTotalMS)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -272,6 +273,48 @@ func wantTimings(t *testing.T, line logLine, want ...string) {
|
||||
}
|
||||
}
|
||||
|
||||
// wantHeuristic checks that line names heuristic as the one that refused the
|
||||
// request, with figures, a JSON object. A heuristic of "" with figures "null" is
|
||||
// a line that names none.
|
||||
func wantHeuristic(t *testing.T, line logLine, heuristic, figures string) {
|
||||
t.Helper()
|
||||
|
||||
if line.Heuristic != heuristic ||
|
||||
!reflect.DeepEqual(line.HeuristicFigures, decodeFigures(t, figures)) {
|
||||
t.Errorf("log line has heuristic %q with heuristic_figures %v, want %q with %s",
|
||||
line.Heuristic, line.HeuristicFigures, heuristic, figures)
|
||||
}
|
||||
}
|
||||
|
||||
// wantAfterAnswerHeuristic checks after_answer_heuristic and
|
||||
// after_answer_heuristic_figures as wantHeuristic checks heuristic and
|
||||
// heuristic_figures.
|
||||
func wantAfterAnswerHeuristic(t *testing.T, line logLine, heuristic, figures string) {
|
||||
t.Helper()
|
||||
|
||||
if line.AfterAnswerHeuristic != heuristic ||
|
||||
!reflect.DeepEqual(line.AfterAnswerHeuristicFigures, decodeFigures(t, figures)) {
|
||||
t.Errorf("log line has after_answer_heuristic %q with "+
|
||||
"after_answer_heuristic_figures %v, want %q with %s",
|
||||
line.AfterAnswerHeuristic, line.AfterAnswerHeuristicFigures, heuristic, figures)
|
||||
}
|
||||
}
|
||||
|
||||
// decodeFigures decodes figures, a JSON object of a heuristic's figures, as
|
||||
// the log line's figures are decoded.
|
||||
func decodeFigures(t *testing.T, figures string) map[string]any {
|
||||
t.Helper()
|
||||
|
||||
var decoded map[string]any
|
||||
|
||||
err := json.Unmarshal([]byte(figures), &decoded)
|
||||
if err != nil {
|
||||
t.Fatalf("decode %s: %v", figures, err)
|
||||
}
|
||||
|
||||
return decoded
|
||||
}
|
||||
|
||||
// microseconds is a timing in whole microseconds.
|
||||
func microseconds(milliseconds float64) int64 {
|
||||
return int64(math.Round(milliseconds * 1000))
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic routewatch_tags. It reads SWW_ROUTEWATCH_URL.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
)
|
||||
|
||||
// routeWatchTags adds to the request the tags asn=<number> and prefix=<netblock>
|
||||
// from what routewatch answered about the client. It refuses nothing, and the
|
||||
// request does not wait for routewatch. A request that gets no tags from it has
|
||||
// the reason in the log line's routewatch_tags_missing. While SWW_ROUTEWATCH_URL
|
||||
// is unset it does nothing. Answers are kept by client and asked for by the
|
||||
// request's address, and a route without an origin AS gives only the prefix= tag.
|
||||
type routeWatchTags struct{}
|
||||
|
||||
func (routeWatchTags) name() string { return "routewatch_tags" }
|
||||
|
||||
func (routeWatchTags) check(ctx context.Context, rq *request, _ time.Time) *verdict {
|
||||
if rq.h.routeWatch == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
route, missing := rq.h.routeWatch.Route(ctx, rq.h.clientGroup(rq.client), rq.client)
|
||||
if missing != "" {
|
||||
rq.line.RouteWatchTagsMissing = string(missing)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.addTag(config.PrefixTagPrefix + route.Netblock.String())
|
||||
|
||||
if route.ASN != 0 {
|
||||
rq.addTag(config.ASNTagPrefix + strconv.FormatUint(uint64(route.ASN), 10))
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,245 @@
|
||||
package discriminator_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/discriminator"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// routeWatchURL is SWW_ROUTEWATCH_URL, and tagsMissing the log line's field
|
||||
// that says why a request got no tags from routewatch.
|
||||
const (
|
||||
routeWatchURL = "SWW_ROUTEWATCH_URL"
|
||||
tagsMissing = "routewatch_tags_missing"
|
||||
fromV6 = "2001:db8:1::5"
|
||||
prefixV6 = "2001:db8::/32"
|
||||
prefixDE = "203.0.113.0/24"
|
||||
prefixKP = "198.51.100.0/24"
|
||||
)
|
||||
|
||||
// standInRoute is what the stand-in for routewatch answers about an address.
|
||||
type standInRoute struct {
|
||||
Netblock string `json:"netblock"`
|
||||
ASN int `json:"asn"`
|
||||
}
|
||||
|
||||
// startRouteWatch starts a stand-in for routewatch's IP info API that knows the
|
||||
// routes given by address, and returns its URL.
|
||||
func startRouteWatch(t *testing.T, routes map[string]standInRoute) string {
|
||||
t.Helper()
|
||||
|
||||
server := httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
address := strings.TrimPrefix(r.URL.Path, "/api/v1/ip/")
|
||||
|
||||
result := map[string]any{"ip": address}
|
||||
if route, known := routes[address]; known {
|
||||
result["netblock"], result["asn"] = route.Netblock, route.ASN
|
||||
}
|
||||
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"status": "ok",
|
||||
"data": map[string]any{"results": []any{result}},
|
||||
})
|
||||
}))
|
||||
t.Cleanup(server.Close)
|
||||
|
||||
return server.URL
|
||||
}
|
||||
|
||||
// waitForRouteWatch waits until routewatch's answer about from, in the client
|
||||
// group of the given length, is kept, so that the next request gets its tags.
|
||||
func waitForRouteWatch(
|
||||
t *testing.T, server *discriminator.Server, from string, bits int,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
addr := netip.MustParseAddr(from)
|
||||
client := netip.PrefixFrom(addr, bits).Masked()
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
|
||||
for {
|
||||
_, missing := server.RouteWatch.Route(context.Background(), client, addr)
|
||||
if missing == "" {
|
||||
return
|
||||
}
|
||||
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("routewatch's answer about %s not kept: %q", from, missing)
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func wantTagsMissing(t *testing.T, line logLine, want string) {
|
||||
t.Helper()
|
||||
|
||||
got, _ := line.fields[tagsMissing].(string)
|
||||
if got != want {
|
||||
t.Errorf("log line has %s %q, want %q", tagsMissing, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRouteWatchTagsAreOnTheLogLineWithTheLookupsTags(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
url := startRouteWatch(t, map[string]standInRoute{fromDE: {prefixDE, 64496}})
|
||||
s, server, _ := startWithLookups(t, map[string]string{routeWatchURL: url})
|
||||
|
||||
waitForRouteWatch(t, server, fromDE, 32)
|
||||
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
wantTags(t, line, "asn=64496", "country=DE", "prefix="+prefixDE)
|
||||
wantTagsMissing(t, line, "")
|
||||
}
|
||||
|
||||
func TestRouteWatchTagsAreOnALineDenyNetsRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
url := startRouteWatch(t, map[string]standInRoute{fromKP: {prefixKP, 64511}})
|
||||
s, _, server := startWithClock(t, "", map[string]string{
|
||||
lookupSource: off, routeWatchURL: url, denyNets: fromKP,
|
||||
})
|
||||
|
||||
waitForRouteWatch(t, server, fromKP, 32)
|
||||
|
||||
line := s.get(fromKP, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, "deny_nets", `{"netblock": "`+fromKP+`/32"}`)
|
||||
wantTags(t, line, "asn=64511", "prefix="+prefixKP)
|
||||
}
|
||||
|
||||
func TestRouteWithoutAnOriginASGivesOnlyThePrefixTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
url := startRouteWatch(t, map[string]standInRoute{fromDE: {prefixDE, 0}})
|
||||
s, _, server := startWithClock(t, "", map[string]string{
|
||||
lookupSource: off, routeWatchURL: url,
|
||||
})
|
||||
|
||||
waitForRouteWatch(t, server, fromDE, 32)
|
||||
|
||||
wantTags(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||
"prefix="+prefixDE)
|
||||
}
|
||||
|
||||
func TestUnsetRouteWatchChangesNothing(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, server := startWithClock(t, "", map[string]string{lookupSource: off})
|
||||
|
||||
if server.RouteWatch != nil {
|
||||
t.Error("RouteWatch is set while SWW_ROUTEWATCH_URL is unset")
|
||||
}
|
||||
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
if _, present := line.fields[tagsMissing]; present {
|
||||
t.Errorf("log line has %s", tagsMissing)
|
||||
}
|
||||
|
||||
if _, present := line.fields["tags"]; present {
|
||||
t.Errorf("log line has tags %v", line.fields["tags"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnreachableRouteWatchLetsTheRequestThroughAndSaysWhy(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// A stand-in that is closed at once, so that the port refuses connections.
|
||||
dead := httptest.NewServer(http.NotFoundHandler())
|
||||
url := dead.URL
|
||||
|
||||
dead.Close()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
lookupSource: off, routeWatchURL: url,
|
||||
})
|
||||
|
||||
wantTagsMissing(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||
"no_answer_yet")
|
||||
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
|
||||
for {
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
missing, _ := line.fields[tagsMissing].(string)
|
||||
if missing == "request_failed" {
|
||||
return
|
||||
}
|
||||
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("still %q, want request_failed", missing)
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddressWithoutARouteHasNoTagsAndTheLineSaysSo(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
lookupSource: off, routeWatchURL: startRouteWatch(t, nil),
|
||||
})
|
||||
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
|
||||
for {
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
missing, _ := line.fields[tagsMissing].(string)
|
||||
if missing == "no_route" {
|
||||
return
|
||||
}
|
||||
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("still %q, want no_route", missing)
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagDenyRefusesOnARouteWatchPrefix(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
url := startRouteWatch(t, map[string]standInRoute{
|
||||
fromDE: {prefixDE, 64496}, fromKP: {prefixKP, 64511},
|
||||
})
|
||||
s, _, server := startWithClock(t, "", map[string]string{
|
||||
lookupSource: off, routeWatchURL: url, tagDeny: "prefix=" + prefixKP,
|
||||
})
|
||||
|
||||
waitForRouteWatch(t, server, fromKP, 32)
|
||||
waitForRouteWatch(t, server, fromDE, 32)
|
||||
|
||||
wantHeuristic(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionClientDenied),
|
||||
"tag_deny", `{"tag": "prefix=`+prefixKP+`"}`)
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
}
|
||||
|
||||
func TestTagLimitPercentWorksWithAnIPv6Prefix(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
url := startRouteWatch(t, map[string]standInRoute{fromV6: {prefixV6, 64496}})
|
||||
s, _, server := startWithClock(t, "", map[string]string{
|
||||
lookupSource: off, routeWatchURL: url, rateLimitPerMinute: "2",
|
||||
tagLimitPercent: "prefix=" + prefixV6 + ":50",
|
||||
})
|
||||
|
||||
waitForRouteWatch(t, server, fromV6, 64)
|
||||
|
||||
// Half of two is one: the second request breaks the limit.
|
||||
line := s.get(fromV6, http.StatusOK, requestlog.ActionForward)
|
||||
wantTags(t, line, "asn=64496", "prefix="+prefixV6)
|
||||
s.get(fromV6, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic rule_files. It reads the rules of the rule files, which
|
||||
// SWW_RULES_DIR and SWW_RULES_ENABLED name.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// ruleFiles checks the request against the rules of the rule files at now,
|
||||
// notes the ids of those it matches in the log line, adds the tag
|
||||
// exploitsignature when it matches any, and refuses the request
|
||||
// when the last rule it matches refuses it: with ActionBlockedByBlockRule for a
|
||||
// block rule, and for a ban rule with the action for the ban it makes. A ban
|
||||
// rule bans the client's netblock for a clear sign of attack, or in observe
|
||||
// mode raises the alert for the ban
|
||||
// it would have made. Either rule's match is noted as an offence, for the
|
||||
// client's history. Its figures are the rule's id and action.
|
||||
type ruleFiles struct{}
|
||||
|
||||
func (ruleFiles) name() string { return "rule_files" }
|
||||
|
||||
func (ruleFiles) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
matched := rq.h.rules.Match(rq.in)
|
||||
|
||||
for _, rule := range matched {
|
||||
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
|
||||
rq.h.metrics.RuleMatched(rule.ID, rule.Action)
|
||||
}
|
||||
|
||||
if len(matched) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.addTag(config.ExploitSignatureTag)
|
||||
|
||||
// Only the last rule matched can refuse the request.
|
||||
last := matched[len(matched)-1]
|
||||
figures := map[string]any{"rule_id": last.ID, "rule_action": last.Action}
|
||||
|
||||
switch last.Action {
|
||||
case rules.ActionBlock:
|
||||
rq.blockedByBlockRule = true
|
||||
|
||||
return &verdict{action: requestlog.ActionBlockedByBlockRule, figures: figures}
|
||||
case rules.ActionBan:
|
||||
rq.attack = true
|
||||
|
||||
return &verdict{
|
||||
action: rq.banForAttack(now, bans.Notes{RuleID: last.ID, Target: last.Target}),
|
||||
figures: figures,
|
||||
}
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
@@ -37,13 +37,14 @@ func TestEachRuleAction(t *testing.T) {
|
||||
// A block rule refuses with 403, whatever SWW_BAN_RESPONSE is, and
|
||||
// bans no one.
|
||||
line = s.request(client, "/blocked", http.StatusForbidden,
|
||||
requestlog.ActionRuleBlocked)
|
||||
requestlog.ActionBlockedByBlockRule)
|
||||
wantRuleIDs(t, line, "noted", "blocked")
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
// A ban rule refuses with SWW_BAN_RESPONSE, and bans the client for
|
||||
// seven days, the default.
|
||||
line = s.request(client, "/.env", http.StatusTooManyRequests, requestlog.ActionBanned)
|
||||
line = s.request(client, "/.env", http.StatusTooManyRequests,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
wantRuleIDs(t, line, "noted", "probe")
|
||||
|
||||
if line.BanExpires != requestlog.FormatTime(start.Add(7*24*time.Hour)) {
|
||||
@@ -82,7 +83,7 @@ func TestEachRuleAction(t *testing.T) {
|
||||
// against the rules, and makes the ban permanent.
|
||||
clk.advance(time.Hour)
|
||||
|
||||
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
|
||||
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionPermanentlyBanned)
|
||||
wantRuleIDs(t, line)
|
||||
|
||||
if line.BanExpires != permanent {
|
||||
@@ -90,7 +91,37 @@ func TestEachRuleAction(t *testing.T) {
|
||||
}
|
||||
|
||||
clk.advance(365 * 24 * time.Hour)
|
||||
s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusTooManyRequests, requestlog.ActionPermanentlyBanned)
|
||||
}
|
||||
|
||||
func TestEachRuleActionAddsTheExploitSignatureTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
rulesDir: writeRules(t, "noted path log ^/noted$\n"+testRules),
|
||||
})
|
||||
|
||||
wantTags(t, s.request(client, "/noted", http.StatusOK, requestlog.ActionForward),
|
||||
"exploitsignature")
|
||||
wantTags(t, s.request(client, "/blocked", http.StatusForbidden,
|
||||
requestlog.ActionBlockedByBlockRule), "exploitsignature")
|
||||
wantTags(t, s.get(client, http.StatusOK, requestlog.ActionForward))
|
||||
}
|
||||
|
||||
func TestRuleRefusalNamesItsHeuristicAndTheRule(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
rulesDir: writeRules(t, "noted path log ^/\n"+testRules),
|
||||
})
|
||||
|
||||
line := s.request(client, "/blocked", http.StatusForbidden,
|
||||
requestlog.ActionBlockedByBlockRule)
|
||||
wantHeuristic(t, line, "rule_files", `{"rule_id": "blocked", "rule_action": "block"}`)
|
||||
|
||||
line = s.request(client, "/.env", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
wantHeuristic(t, line, "rule_files", `{"rule_id": "probe", "rule_action": "ban"}`)
|
||||
}
|
||||
|
||||
func TestNextClearSignOfAttackAfterABanBansPermanently(t *testing.T) {
|
||||
@@ -102,7 +133,8 @@ func TestNextClearSignOfAttackAfterABanBansPermanently(t *testing.T) {
|
||||
})
|
||||
|
||||
// The first probe bans for SWW_ATTACK_BAN_DURATION.
|
||||
line := s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
line := s.request(client, "/.env", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
if line.BanExpires != requestlog.FormatTime(clk.Now().Add(time.Hour)) {
|
||||
t.Errorf("log line has ban_expires %q, want an hour on", line.BanExpires)
|
||||
}
|
||||
@@ -112,7 +144,8 @@ func TestNextClearSignOfAttackAfterABanBansPermanently(t *testing.T) {
|
||||
clk.advance(time.Hour)
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line = s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
line = s.request(client, "/.env", http.StatusForbidden,
|
||||
requestlog.ActionPermanentlyBanned)
|
||||
if line.BanExpires != permanent {
|
||||
t.Errorf("log line has ban_expires %q, want permanent", line.BanExpires)
|
||||
}
|
||||
@@ -140,7 +173,8 @@ func TestRulesComeAfterTheOtherChecks(t *testing.T) {
|
||||
// A probe over the rate limit breaks the limit before any rule sees
|
||||
// it.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
line = s.request(client, "/.env", http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
line = s.request(client, "/.env", http.StatusForbidden,
|
||||
requestlog.ActionClientIsRateLimited)
|
||||
wantRuleIDs(t, line)
|
||||
|
||||
limitBan := server.Ledger.Bans(netip.MustParsePrefix(client + "/32"))
|
||||
@@ -150,7 +184,7 @@ func TestRulesComeAfterTheOtherChecks(t *testing.T) {
|
||||
|
||||
// A client the rate limits do not apply to is still checked.
|
||||
s.get(exempt, http.StatusOK, requestlog.ActionForward)
|
||||
s.request(exempt, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(exempt, "/.env", http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestObserveModeLogsWhatTheRulesWouldDo(t *testing.T) {
|
||||
@@ -162,11 +196,11 @@ func TestObserveModeLogsWhatTheRulesWouldDo(t *testing.T) {
|
||||
})
|
||||
|
||||
line := s.request(client, "/blocked", http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionRuleBlocked)
|
||||
wantWouldAction(t, line, requestlog.ActionBlockedByBlockRule)
|
||||
wantRuleIDs(t, line, "blocked")
|
||||
|
||||
line = s.request(client, "/.env", http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionBanned)
|
||||
wantWouldAction(t, line, requestlog.ActionTemporarilyBanned)
|
||||
wantRuleIDs(t, line, "probe")
|
||||
|
||||
if line.BanExpires != "" {
|
||||
@@ -190,9 +224,10 @@ func TestMetricsCountRuleMatchesAndBansForAnAttack(t *testing.T) {
|
||||
metricsToken: token,
|
||||
})
|
||||
|
||||
s.request(client, "/blocked", http.StatusForbidden, requestlog.ActionRuleBlocked)
|
||||
s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(client, "/blocked", http.StatusForbidden,
|
||||
requestlog.ActionBlockedByBlockRule)
|
||||
s.request(client, "/.env", http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionPermanentlyBanned)
|
||||
|
||||
metrics := s.scrape()
|
||||
wantMetric(t, metrics,
|
||||
@@ -200,10 +235,10 @@ func TestMetricsCountRuleMatchesAndBansForAnAttack(t *testing.T) {
|
||||
wantMetric(t, metrics,
|
||||
`sww_rule_matches_total{action="ban",instance="app",rule_id="probe"}`, 1)
|
||||
wantMetric(t, metrics, `sww_rules_loaded{instance="app"}`, 2)
|
||||
wantMetric(t, metrics, `sww_requests_total{action="rule_blocked",`+
|
||||
wantMetric(t, metrics, `sww_requests_total{action="blocked_by_block_rule",`+
|
||||
`instance="app",status_class="4xx"}`, 1)
|
||||
wantMetric(t, metrics,
|
||||
`sww_offences_total{instance="app",kind="rule_blocked"}`, 1)
|
||||
`sww_offences_total{instance="app",kind="blocked_by_block_rule"}`, 1)
|
||||
wantMetric(t, metrics, `sww_offences_total{instance="app",kind="attack"}`, 1)
|
||||
wantMetric(t, metrics, `sww_bans_made_total{cause="attack",instance="app"}`, 1)
|
||||
wantMetric(t, metrics, `sww_bans_made_total{cause="limit",instance="app"}`, 0)
|
||||
|
||||
@@ -1,64 +0,0 @@
|
||||
package discriminator
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// trapPath reports whether the request asks for a path in
|
||||
// SWW_TRAP_PATHS: its path as a path rule sees it, without the query,
|
||||
// as sent or once percent-decoded, is one of them. Such a request is a
|
||||
// clear sign of attack, as a ban rule's match is: it bans the client's
|
||||
// netblock, or in observe mode raises the alert for the ban it would have
|
||||
// made. The ban's notes give the path as sent.
|
||||
func (rq *request) trapPath(now time.Time) bool {
|
||||
path := rules.Path(rq.in)
|
||||
if !slices.Contains(rq.h.config.TrapPaths, path) &&
|
||||
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
|
||||
return false
|
||||
}
|
||||
|
||||
rq.attack = true
|
||||
rq.banForAttack(now, bans.Notes{TrapPath: path})
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// checkRules checks the request against the rules of the rule files at
|
||||
// now, notes the ids of those it matches in the log line, and returns the
|
||||
// action of the rule that refuses it, ActionRuleBlocked for a block rule
|
||||
// and ActionBanned for a ban rule, or "" when none does. A ban rule bans
|
||||
// the client's netblock for a clear sign of attack, or in observe mode
|
||||
// raises the alert for the ban it would have made. Either rule's match
|
||||
// is noted as an offence, for the client's history.
|
||||
func (rq *request) checkRules(now time.Time) string {
|
||||
matched := rq.h.rules.Match(rq.in)
|
||||
|
||||
for _, rule := range matched {
|
||||
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
|
||||
rq.h.metrics.RuleMatched(rule.ID, rule.Action)
|
||||
}
|
||||
|
||||
if len(matched) == 0 {
|
||||
return ""
|
||||
}
|
||||
|
||||
// Only the last rule matched can refuse the request.
|
||||
switch last := matched[len(matched)-1]; last.Action {
|
||||
case rules.ActionBlock:
|
||||
rq.ruleBlocked = true
|
||||
|
||||
return requestlog.ActionRuleBlocked
|
||||
case rules.ActionBan:
|
||||
rq.attack = true
|
||||
rq.banForAttack(now, bans.Notes{RuleID: last.ID, Target: last.Target})
|
||||
|
||||
return requestlog.ActionBanned
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
@@ -32,13 +32,12 @@ func TestAllowNetsSkipEveryCheck(t *testing.T) {
|
||||
trustedProxies: trustLocalhost,
|
||||
allowNets: "198.51.100.0/24",
|
||||
denyNets: fromKP,
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
rateLimitPerMinute: "1",
|
||||
})
|
||||
|
||||
// Neither SWW_DENY_NETS, the country lists nor the limit of one
|
||||
// request a minute refuses the client, and its country is not looked
|
||||
// up.
|
||||
// Neither SWW_DENY_NETS, SWW_TAG_DENY nor the limit of one request a
|
||||
// minute refuses the client, and its country is not looked up.
|
||||
wantAnswers(t, addr, out, []sentRequest{
|
||||
{fromKP, http.StatusOK, requestlog.ActionForward},
|
||||
{fromKP, http.StatusOK, requestlog.ActionForward},
|
||||
@@ -69,7 +68,7 @@ func TestRequestFromAllowNetsIsNotCounted(t *testing.T) {
|
||||
{listedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{listedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{unlistedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
|
||||
{unlistedAddr, http.StatusForbidden, requestlog.ActionClientIsRateLimited},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -85,7 +84,7 @@ func TestDenyNetsRefuseBeforeTheLookup(t *testing.T) {
|
||||
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
denyNets: "203.0.113.0/24",
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
|
||||
req := newRequest(t, http.MethodPost, addr, "/", strings.NewReader("a body"))
|
||||
@@ -93,7 +92,7 @@ func TestDenyNetsRefuseBeforeTheLookup(t *testing.T) {
|
||||
wantStatus(t, do(t, req), http.StatusForbidden)
|
||||
|
||||
line := out.endedLine(t)
|
||||
wantLine(t, line, http.StatusForbidden, requestlog.ActionDenied)
|
||||
wantLine(t, line, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
|
||||
if len(asked()) != 0 {
|
||||
t.Errorf("GeoJS was asked about %v, want nothing", asked())
|
||||
@@ -104,6 +103,15 @@ func TestDenyNetsRefuseBeforeTheLookup(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestDenyNetsRefusalNamesItsHeuristicAndNetblock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{denyNets: clientsNet})
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, "deny_nets", `{"netblock": "`+clientsNet+`"}`)
|
||||
}
|
||||
|
||||
func TestRequestRefusedByDenyNetsIsNotCounted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -117,10 +125,10 @@ func TestRequestRefusedByDenyNetsIsNotCounted(t *testing.T) {
|
||||
// listedAddr's refused requests are not counted, so the first request
|
||||
// from unlistedAddr is within the limit of one a minute.
|
||||
wantAnswers(t, addr, out, []sentRequest{
|
||||
{listedAddr, http.StatusForbidden, requestlog.ActionDenied},
|
||||
{listedAddr, http.StatusForbidden, requestlog.ActionDenied},
|
||||
{listedAddr, http.StatusForbidden, requestlog.ActionClientDenied},
|
||||
{listedAddr, http.StatusForbidden, requestlog.ActionClientDenied},
|
||||
{unlistedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
|
||||
{unlistedAddr, http.StatusForbidden, requestlog.ActionClientIsRateLimited},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -133,19 +141,19 @@ func TestRateLimitExemptNetsAreNeitherCountedNorRefused(t *testing.T) {
|
||||
trustedProxies: trustLocalhost,
|
||||
lookupTimeout: "1h",
|
||||
rateLimitExemptNets: listedAddr + "," + fromKP,
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
rateLimitPerMinute: "1",
|
||||
})
|
||||
|
||||
// listedAddr's requests are neither refused nor counted, so the first
|
||||
// request from unlistedAddr is within the limit of one a minute. The
|
||||
// country lists still refuse an exempt client.
|
||||
// request from unlistedAddr is within the limit of one a minute.
|
||||
// SWW_TAG_DENY still refuses an exempt client.
|
||||
wantAnswers(t, addr, out, []sentRequest{
|
||||
{listedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{listedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{unlistedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
|
||||
{fromKP, http.StatusForbidden, requestlog.ActionCountryDenied},
|
||||
{unlistedAddr, http.StatusForbidden, requestlog.ActionClientIsRateLimited},
|
||||
{fromKP, http.StatusForbidden, requestlog.ActionClientDenied},
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic tag_deny. It reads SWW_TAG_DENY.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// tagDeny refuses a request that carries a tag in SWW_TAG_DENY. Its figure is
|
||||
// the first such tag, in sorted order.
|
||||
type tagDeny struct{}
|
||||
|
||||
func (tagDeny) name() string { return "tag_deny" }
|
||||
|
||||
func (tagDeny) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
for _, tag := range rq.line.Tags {
|
||||
if slices.Contains(rq.h.config.TagDeny, tag) {
|
||||
return &verdict{
|
||||
action: requestlog.ActionClientDenied,
|
||||
figures: map[string]any{"tag": tag},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
package discriminator_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
func TestTagDenyRefusesALookupTagAndNamesTheTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// The setting takes a country code in either case; the request's tag is
|
||||
// in capitals.
|
||||
s, _, _ := startWithLookups(t, map[string]string{tagDeny: "Country=kp"})
|
||||
|
||||
line := s.get(fromKP, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, "tag_deny", `{"tag": "country=KP"}`)
|
||||
wantTags(t, line, "asn=64511", "country=KP")
|
||||
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
}
|
||||
|
||||
func TestTagDenyRefusesAnASNTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithLookups(t, map[string]string{tagDeny: "asn=64511"})
|
||||
|
||||
wantHeuristic(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionClientDenied),
|
||||
"tag_deny", `{"tag": "asn=64511"}`)
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
}
|
||||
|
||||
func TestTagDenyRefusesAListTagBeforeTheRateLimitsCount(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
tagLists: "torexit=" + torList, tagDeny: "torexit", rateLimitPerMinute: "1",
|
||||
})
|
||||
loadLists(t, server, map[string][]string{torList: {fromDE}})
|
||||
|
||||
// Each is refused for its tag: none is counted to break the limit of one.
|
||||
for range 3 {
|
||||
wantHeuristic(t, s.get(fromDE, http.StatusForbidden, requestlog.ActionClientDenied),
|
||||
"tag_deny", `{"tag": "torexit"}`)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagDenyRefusesAMixedCaseZoneByItsLowerCaseTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const mixedCaseZone = "DNSBL.example"
|
||||
|
||||
// The zone's tag is in lower case, however SWW_DNSBL_ZONES writes it.
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
dnsblZones: mixedCaseZone, dnsblResolver: noResolver,
|
||||
reputationAction: actionLog, tagDeny: "dnsbl=" + dnsblZone,
|
||||
})
|
||||
loadVerdicts(server, map[string][]string{fromDE: {mixedCaseZone}})
|
||||
|
||||
wantHeuristic(t, s.get(fromDE, http.StatusForbidden, requestlog.ActionClientDenied),
|
||||
"tag_deny", `{"tag": "dnsbl=`+dnsblZone+`"}`)
|
||||
}
|
||||
|
||||
func TestTagDenyRefusesAnExploitSignatureOfARuleThatOnlyLogs(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
rulesDir: writeRules(t, "noted path log ^/\n"), tagDeny: "exploitsignature",
|
||||
})
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, "tag_deny", `{"tag": "exploitsignature"}`)
|
||||
wantTags(t, line, "exploitsignature")
|
||||
}
|
||||
|
||||
func TestTagLimitPercentLowersALimitTheLowestWinning(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
rateLimitPerMinute: fourAMinute,
|
||||
tagLists: "vpn=" + vpnList + ",tor=" + torList,
|
||||
tagLimitPercent: "vpn:50,tor:25",
|
||||
})
|
||||
loadLists(t, server, map[string][]string{vpnList: {fromDE}, torList: {fromDE}})
|
||||
|
||||
// A quarter of 4 requests a minute: the second breaks the limit.
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line := s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
"25 from "+tagLimitPercent)
|
||||
}
|
||||
|
||||
func TestTagLimitPercentLowersAByteLimitByAListTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
bytesLimitPerMinute: twoUploads,
|
||||
tagLists: "vpn=" + vpnList,
|
||||
tagLimitPercent: "vpn:50",
|
||||
})
|
||||
loadLists(t, server, map[string][]string{vpnList: {fromDE}})
|
||||
|
||||
// The upload's 100 bytes are over half of 199, 99.
|
||||
line := s.uploadFrom(fromDE)
|
||||
if line.LimitHit != minuteBytes {
|
||||
t.Errorf("log line has limit_hit %q, want %s", line.LimitHit, minuteBytes)
|
||||
}
|
||||
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting,
|
||||
"50 from "+tagLimitPercent)
|
||||
|
||||
// fromKP, which the list does not hold, has the whole limit.
|
||||
line = s.uploadFrom(fromKP)
|
||||
if line.LimitHit != "" {
|
||||
t.Errorf("log line for %s has limit_hit %q, want none", fromKP, line.LimitHit)
|
||||
}
|
||||
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting, none)
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic tag_lists. It reads the address lists of SWW_TAG_LISTS through
|
||||
// the reputation lists.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// tagLists adds to the request one tag for each list of SWW_TAG_LISTS that
|
||||
// holds the client, the tag being the list's name. It refuses nothing, and runs
|
||||
// first so that every refusal's log line carries these tags.
|
||||
type tagLists struct{}
|
||||
|
||||
func (tagLists) name() string { return "tag_lists" }
|
||||
|
||||
func (tagLists) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
for _, name := range rq.h.lists.TagsOf(rq.client) {
|
||||
rq.addTag(name)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
package discriminator_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// tagLists is SWW_TAG_LISTS, and vpnList and torList lists it names.
|
||||
const (
|
||||
tagLists = "SWW_TAG_LISTS"
|
||||
vpnList = "https://lists.example/vpn.txt"
|
||||
torList = "https://lists.example/torexit.txt"
|
||||
)
|
||||
|
||||
// wantTags checks the log line's tags.
|
||||
func wantTags(t *testing.T, line logLine, want ...string) {
|
||||
t.Helper()
|
||||
|
||||
got, _ := line.fields["tags"].([]any)
|
||||
|
||||
tags := make([]string, 0, len(got))
|
||||
for _, tag := range got {
|
||||
text, _ := tag.(string)
|
||||
tags = append(tags, text)
|
||||
}
|
||||
|
||||
if !slices.Equal(tags, want) {
|
||||
t.Errorf("log line has tags %v, want %v", tags, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLookupAndTagListsGiveTheRequestItsTagsSorted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
tagLists: "vpn=mullvad=" + vpnList + ",TorExit=" + torList,
|
||||
})
|
||||
loadLists(t, server, map[string][]string{
|
||||
vpnList: {fromDE}, torList: {fromDE + "/32", fromKP},
|
||||
})
|
||||
|
||||
wantTags(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||
"asn=64496", "country=DE", "torexit", "vpn=mullvad")
|
||||
wantTags(t, s.get(fromKP, http.StatusOK, requestlog.ActionForward),
|
||||
"asn=64511", "country=KP", "torexit")
|
||||
}
|
||||
|
||||
func TestRequestWithoutATagLeavesTheFieldOut(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// With no lookup, a client is not placed and no setting gives it a tag.
|
||||
s, _, _ := startWithClock(t, "", map[string]string{lookupSource: "off"})
|
||||
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
if _, present := line.fields["tags"]; present {
|
||||
t.Errorf("log line has tags %v, want none", line.fields["tags"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagListsTagEvenARequestTheCountryListRefuses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
tagLists: "vpn=" + vpnList, allowedCountries: "de",
|
||||
})
|
||||
loadLists(t, server, map[string][]string{vpnList: {fromKP}})
|
||||
|
||||
wantTags(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionCountryDenied),
|
||||
"asn=64511", "country=KP", "vpn")
|
||||
}
|
||||
|
||||
func TestTagListsTagARequestRefusedBeforeTheLookup(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
tagLists: "vpn=" + vpnList, denyNets: fromKP, rateLimitPerMinute: "1",
|
||||
})
|
||||
loadLists(t, server, map[string][]string{vpnList: {fromDE, fromKP}})
|
||||
|
||||
line := s.get(fromKP, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
if line.Heuristic != "deny_nets" {
|
||||
t.Errorf("log line has heuristic %q, want deny_nets", line.Heuristic)
|
||||
}
|
||||
|
||||
wantTags(t, line, "vpn")
|
||||
|
||||
// The second request breaks the limit of one a minute, which bans the client.
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
line = s.get(fromDE, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
if line.Heuristic != "active_ban" {
|
||||
t.Errorf("log line has heuristic %q, want active_ban", line.Heuristic)
|
||||
}
|
||||
|
||||
wantTags(t, line, "vpn")
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic trap_paths. It reads SWW_TRAP_PATHS.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// trapPaths refuses a request for a path in SWW_TRAP_PATHS: its path as a
|
||||
// path rule sees it, without the query, as sent or once percent-decoded, is one
|
||||
// of them. Such a request is a clear sign of attack, as a ban rule's match is:
|
||||
// it bans the client's netblock, or in observe mode raises the alert for the
|
||||
// ban it would have made. The ban's notes give the path as sent, and so does
|
||||
// its figure.
|
||||
type trapPaths struct{}
|
||||
|
||||
func (trapPaths) name() string { return "trap_paths" }
|
||||
|
||||
func (trapPaths) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
path := rules.Path(rq.in)
|
||||
if !slices.Contains(rq.h.config.TrapPaths, path) &&
|
||||
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.attack = true
|
||||
|
||||
return &verdict{
|
||||
action: rq.banForAttack(now, bans.Notes{TrapPath: path}),
|
||||
figures: map[string]any{"path": path},
|
||||
}
|
||||
}
|
||||
@@ -43,7 +43,7 @@ func TestTrapPathBansAsABanRuleDoes(t *testing.T) {
|
||||
|
||||
// The query is not part of the path.
|
||||
line := s.request(client, "/wp-login.php?redirect_to=x", http.StatusTooManyRequests,
|
||||
requestlog.ActionBanned)
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
wantRuleIDs(t, line)
|
||||
|
||||
if line.BanExpires != requestlog.FormatTime(start.Add(7*24*time.Hour)) {
|
||||
@@ -77,12 +77,22 @@ func TestTrapPathBansAsABanRuleDoes(t *testing.T) {
|
||||
}
|
||||
|
||||
// The next request is refused under the ban, and makes it permanent.
|
||||
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
|
||||
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionPermanentlyBanned)
|
||||
if line.BanExpires != permanent {
|
||||
t.Errorf("log line has ban_expires %q, want permanent", line.BanExpires)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTrapPathRefusalNamesItsHeuristicAndPath(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{trapPaths: trapPathList})
|
||||
|
||||
line := s.request(client, "/wp-login.php?redirect_to=x", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
wantHeuristic(t, line, "trap_paths", `{"path": "/wp-login.php"}`)
|
||||
}
|
||||
|
||||
func TestTrapPathMatchesThePathDecodedOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -90,7 +100,8 @@ func TestTrapPathMatchesThePathDecodedOnce(t *testing.T) {
|
||||
|
||||
// %2577 decodes once to %77, not to w.
|
||||
s.request(otherClient, "/%2577p-login.php", http.StatusOK, requestlog.ActionForward)
|
||||
s.request(client, "/%77p-login.php", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(client, "/%77p-login.php", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
|
||||
// The ban's notes give the path as the client sent it.
|
||||
got := server.Ledger.Bans(netip.MustParsePrefix(client + "/32"))
|
||||
@@ -107,7 +118,8 @@ func TestTrapPathsNeedNoRuleFiles(t *testing.T) {
|
||||
"SWW_RULES_ENABLED": "false",
|
||||
})
|
||||
|
||||
s.request(client, "/xmlrpc.php", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(client, "/xmlrpc.php", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestObserveModeLogsWhatATrapPathWouldDo(t *testing.T) {
|
||||
@@ -119,7 +131,7 @@ func TestObserveModeLogsWhatATrapPathWouldDo(t *testing.T) {
|
||||
})
|
||||
|
||||
line := s.request(client, "/xmlrpc.php", http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionBanned)
|
||||
wantWouldAction(t, line, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
// No ban was made.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
+14
-4
@@ -133,8 +133,11 @@ func (f *File) Watch(ctx context.Context) {
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
f.params.ProcessLog.Error("cannot watch the lookup database for replacements",
|
||||
"error", err.Error())
|
||||
f.params.ProcessLog.Error(
|
||||
"cannot watch the lookup database for replacements",
|
||||
"error",
|
||||
err.Error(),
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
@@ -194,7 +197,10 @@ func (f *File) readAgain() {
|
||||
f.params.Alerts.Raise(alerts.Alert{
|
||||
Event: alerts.EventFileError,
|
||||
Reason: kept,
|
||||
Detail: map[string]any{"file": f.params.Path, "error": err.Error()},
|
||||
Detail: map[string]any{
|
||||
"file": f.params.Path,
|
||||
"error": err.Error(),
|
||||
},
|
||||
})
|
||||
f.params.ProcessLog.Error(kept, "error", err.Error())
|
||||
|
||||
@@ -212,7 +218,11 @@ func (f *File) use(reader *maxminddb.Reader) {
|
||||
f.lastRead = f.params.Now()
|
||||
f.mu.Unlock()
|
||||
|
||||
f.params.ProcessLog.Info("read the lookup database", "file", f.params.Path)
|
||||
f.params.ProcessLog.Info(
|
||||
"read the lookup database",
|
||||
"file",
|
||||
f.params.Path,
|
||||
)
|
||||
}
|
||||
|
||||
// read reads the lookup database at path. The whole file is read into
|
||||
|
||||
@@ -198,7 +198,7 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
|
||||
|
||||
answer, found := g.kept(client)
|
||||
if !found {
|
||||
g.metrics.GeoJSUnanswered.Inc()
|
||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
||||
}
|
||||
|
||||
w, waiting := g.waiting[client]
|
||||
@@ -296,7 +296,7 @@ func (g *GeoJS) answerOrWait(
|
||||
}
|
||||
|
||||
if w == nil {
|
||||
g.metrics.GeoJSUnanswered.Inc()
|
||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
||||
|
||||
return Answer{}, nil // too many clients wait already
|
||||
}
|
||||
@@ -307,7 +307,7 @@ func (g *GeoJS) answerOrWait(
|
||||
}
|
||||
|
||||
if w.late {
|
||||
g.metrics.GeoJSUnanswered.Inc()
|
||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
||||
|
||||
return Answer{}, nil
|
||||
}
|
||||
|
||||
@@ -362,7 +362,7 @@ func TestWithoutWaitTheRequestGoesOnAtOnceAndTheAnswerIsGivenWhenItComes(
|
||||
mu.Unlock()
|
||||
|
||||
wantCountry(t, g, client, germany)
|
||||
wantUnanswered(t, m, 0)
|
||||
wantRequestsWithoutGeoJSAnswer(t, m, 0)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -596,7 +596,7 @@ func TestClientsWithoutAnAnswerAreCounted(t *testing.T) {
|
||||
// GeoJS fails, so the first client goes without an answer, and GeoJS
|
||||
// is left alone for a second, which does not pass in this test.
|
||||
wantCountry(t, g, clients(), "")
|
||||
wantUnanswered(t, m, 1)
|
||||
wantRequestsWithoutGeoJSAnswer(t, m, 1)
|
||||
|
||||
// Meanwhile each new client goes without one at once, while there is
|
||||
// room for it among the 10,000 that may wait.
|
||||
@@ -604,11 +604,11 @@ func TestClientsWithoutAnAnswerAreCounted(t *testing.T) {
|
||||
wantCountry(t, g, clients(), "")
|
||||
}
|
||||
|
||||
wantUnanswered(t, m, 10000)
|
||||
wantRequestsWithoutGeoJSAnswer(t, m, 10000)
|
||||
|
||||
// One more, for which there is no room, goes without one too.
|
||||
wantCountry(t, g, clients(), "")
|
||||
wantUnanswered(t, m, 10001)
|
||||
wantRequestsWithoutGeoJSAnswer(t, m, 10001)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -841,12 +841,12 @@ func wantAsked(t *testing.T, geojs *standIn, i int, want ...string) {
|
||||
}
|
||||
}
|
||||
|
||||
// wantUnanswered checks how many requests m counts as having gone without
|
||||
// an answer from GeoJS.
|
||||
func wantUnanswered(t *testing.T, m *metrics.Metrics, want float64) {
|
||||
// wantRequestsWithoutGeoJSAnswer checks how many requests m counts as having
|
||||
// gone without an answer from GeoJS.
|
||||
func wantRequestsWithoutGeoJSAnswer(t *testing.T, m *metrics.Metrics, want float64) {
|
||||
t.Helper()
|
||||
|
||||
got := testutil.ToFloat64(m.GeoJSUnanswered)
|
||||
got := testutil.ToFloat64(m.RequestsWithoutGeoJSAnswer)
|
||||
if got != want {
|
||||
t.Errorf("%v requests went without an answer, want %v", got, want)
|
||||
}
|
||||
|
||||
@@ -25,7 +25,7 @@ type busiest struct {
|
||||
requests *prometheus.CounterVec
|
||||
requestBytes *prometheus.CounterVec
|
||||
responseBytes *prometheus.CounterVec
|
||||
// refused are, by country, the requests the country lists refused; nil
|
||||
// refused are, by country, the requests the country list refused; nil
|
||||
// by AS number.
|
||||
refused *prometheus.CounterVec
|
||||
|
||||
@@ -42,7 +42,7 @@ type busiest struct {
|
||||
func newCountries(topN int) *busiest {
|
||||
countries := newBusiest(topN, "country", "the client's country")
|
||||
countries.refused = counterVec("sww_country_list_refusals_total",
|
||||
"Requests the country lists refused, by the client's country.",
|
||||
"Requests SWW_EXCLUSIVELY_ALLOWED_COUNTRIES refused, by the client's country.",
|
||||
[]string{"country"})
|
||||
|
||||
return countries
|
||||
@@ -88,7 +88,7 @@ func (b *busiest) Collect(ch chan<- prometheus.Metric) {
|
||||
}
|
||||
|
||||
// vecs returns the metrics: by AS number, those of requests and bytes; by
|
||||
// country, the refusals by the country lists as well.
|
||||
// country, the refusals by the country list as well.
|
||||
func (b *busiest) vecs() []*prometheus.CounterVec {
|
||||
vecs := []*prometheus.CounterVec{b.requests, b.requestBytes, b.responseBytes}
|
||||
if b.refused != nil {
|
||||
|
||||
+18
-18
@@ -36,7 +36,7 @@ type Metrics struct {
|
||||
upstreamDuration prometheus.Histogram
|
||||
rateLimitHits *prometheus.CounterVec
|
||||
offences *prometheus.CounterVec
|
||||
wafMatches *prometheus.CounterVec
|
||||
coreRuleSetMatches *prometheus.CounterVec
|
||||
// ruleMatches are made by AddRules, and reputationHits by
|
||||
// AddReputation.
|
||||
ruleMatches *prometheus.CounterVec
|
||||
@@ -45,12 +45,12 @@ type Metrics struct {
|
||||
asns *busiest
|
||||
|
||||
// GeoJSRequests are the requests to GeoJS, and GeoJSFailures those
|
||||
// that failed. GeoJSUnanswered are the requests that needed their
|
||||
// client's answer, for a setting that acts on it, and went on without
|
||||
// it because GeoJS had not given it in time.
|
||||
// that failed. RequestsWithoutGeoJSAnswer are the requests that needed
|
||||
// their client's answer, for a setting that acts on it, and went on
|
||||
// without it because GeoJS had not given it in time.
|
||||
GeoJSRequests prometheus.Counter
|
||||
GeoJSFailures prometheus.Counter
|
||||
GeoJSUnanswered prometheus.Counter
|
||||
RequestsWithoutGeoJSAnswer prometheus.Counter
|
||||
|
||||
stateFileWrites *prometheus.CounterVec
|
||||
stateFileWriteFailures *prometheus.CounterVec
|
||||
@@ -101,7 +101,7 @@ func New(topN int, instanceName string) *Metrics {
|
||||
[]string{"window", "kind"}),
|
||||
offences: counterVec("sww_offences_total",
|
||||
"Offences, by kind.", []string{"kind"}),
|
||||
wafMatches: counterVec("sww_waf_matches_total",
|
||||
coreRuleSetMatches: counterVec("sww_core_rule_set_matches_total",
|
||||
"Requests that matched a rule of the Core Rule Set, by SWW_WAF_MODE "+
|
||||
"and the rule's id.", []string{"mode", "rule_id"}),
|
||||
countries: newCountries(topN),
|
||||
@@ -114,8 +114,8 @@ func New(topN int, instanceName string) *Metrics {
|
||||
Name: "sww_geojs_failures_total",
|
||||
Help: "Requests to GeoJS that failed.",
|
||||
}),
|
||||
GeoJSUnanswered: prometheus.NewCounter(prometheus.CounterOpts{
|
||||
Name: "sww_geojs_unanswered_total",
|
||||
RequestsWithoutGeoJSAnswer: prometheus.NewCounter(prometheus.CounterOpts{
|
||||
Name: "sww_requests_without_geojs_answer_total",
|
||||
Help: "Requests that needed their client's answer from GeoJS and " +
|
||||
"went on without it, because GeoJS had not given it in time.",
|
||||
}),
|
||||
@@ -139,9 +139,9 @@ func New(topN int, instanceName string) *Metrics {
|
||||
collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}),
|
||||
m.inFlight, m.requests, m.requestBytes, m.responseBytes,
|
||||
m.requestDuration, m.upstreamDuration,
|
||||
m.rateLimitHits, m.offences, m.wafMatches,
|
||||
m.rateLimitHits, m.offences, m.coreRuleSetMatches,
|
||||
m.countries, m.asns,
|
||||
m.GeoJSRequests, m.GeoJSFailures, m.GeoJSUnanswered,
|
||||
m.GeoJSRequests, m.GeoJSFailures, m.RequestsWithoutGeoJSAnswer,
|
||||
m.stateFileWrites, m.stateFileWriteFailures,
|
||||
m.stateFileLastWrite, m.stateFileSize,
|
||||
m.stateFileEditsTakenIn, m.stateFileEditsSetAside,
|
||||
@@ -232,7 +232,7 @@ func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
|
||||
return float64(remote.Dropped())
|
||||
}),
|
||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||
Name: "sww_remote_log_buffer_depth",
|
||||
Name: "sww_remote_log_lines_waiting",
|
||||
Help: "Log lines in the buffer, waiting to be sent.",
|
||||
}, func() float64 {
|
||||
return float64(remote.Depth())
|
||||
@@ -355,11 +355,11 @@ func (m *Metrics) AddAlerts(queue *alerts.Queue) {
|
||||
return float64(queue.Counts(name).Sent)
|
||||
}),
|
||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||
Name: "sww_alerts_failed_total",
|
||||
Name: "sww_alert_send_failures_total",
|
||||
Help: "Requests to the destination that failed.",
|
||||
ConstLabels: destination,
|
||||
}, func() float64 {
|
||||
return float64(queue.Counts(name).Failed)
|
||||
return float64(queue.Counts(name).SendFailures)
|
||||
}),
|
||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||
Name: "sww_alerts_suppressed_total",
|
||||
@@ -426,8 +426,8 @@ func (m *Metrics) Offences(r ratelimit.Request) {
|
||||
for kind, committed := range map[string]bool{
|
||||
"limit": r.BrokeLimit,
|
||||
"attack": r.Attack,
|
||||
"rule_blocked": r.RuleBlocked,
|
||||
"waf_blocked": r.WAFBlocked,
|
||||
"blocked_by_block_rule": r.BlockedByBlockRule,
|
||||
"blocked_by_core_rule_set": r.BlockedByCoreRuleSet,
|
||||
} {
|
||||
if committed {
|
||||
m.offences.WithLabelValues(kind).Inc()
|
||||
@@ -441,10 +441,10 @@ func (m *Metrics) RuleMatched(id, action string) {
|
||||
m.ruleMatches.WithLabelValues(id, action).Inc()
|
||||
}
|
||||
|
||||
// WAFMatched counts a request that matched the Core Rule Set's rule id,
|
||||
// CoreRuleSetMatched counts a request that matched the Core Rule Set's rule id,
|
||||
// with SWW_WAF_MODE at mode.
|
||||
func (m *Metrics) WAFMatched(mode string, id int) {
|
||||
m.wafMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc()
|
||||
func (m *Metrics) CoreRuleSetMatched(mode string, id int) {
|
||||
m.coreRuleSetMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc()
|
||||
}
|
||||
|
||||
// StateFileWritten counts a write of the state file name, of size bytes,
|
||||
|
||||
@@ -122,12 +122,13 @@ type Responses struct {
|
||||
type Offences struct {
|
||||
// Limit is its requests that broke a rate limit, a byte limit or the
|
||||
// error burst, Attack those that were a clear sign of attack, a match
|
||||
// of a ban rule or a request for a trap path, RuleBlocked those a block
|
||||
// rule refused, and WAFBlocked those the Core Rule Set refused.
|
||||
// of a ban rule or a request for a trap path, BlockedByBlockRule those a
|
||||
// block rule refused, and BlockedByCoreRuleSet those the Core Rule Set
|
||||
// refused.
|
||||
Limit int64 `json:"limit"`
|
||||
Attack int64 `json:"attack"`
|
||||
RuleBlocked int64 `json:"rule_blocked"`
|
||||
WAFBlocked int64 `json:"waf_blocked"`
|
||||
BlockedByBlockRule int64 `json:"blocked_by_block_rule"`
|
||||
BlockedByCoreRuleSet int64 `json:"blocked_by_core_rule_set"`
|
||||
// TokenRefused is what a clients.json written by an earlier version gives,
|
||||
// which must still load. Nothing adds to it.
|
||||
TokenRefused int64 `json:"token_refused,omitempty"`
|
||||
@@ -147,12 +148,12 @@ type Request struct {
|
||||
ResponseBytes int64
|
||||
// BrokeLimit is true for a request that broke a rate limit, a byte
|
||||
// limit or the error burst, Attack for one that matched a ban rule or
|
||||
// asked for a trap path, RuleBlocked for one a block rule refused, and
|
||||
// WAFBlocked for one the Core Rule Set refused.
|
||||
// asked for a trap path, BlockedByBlockRule for one a block rule refused,
|
||||
// and BlockedByCoreRuleSet for one the Core Rule Set refused.
|
||||
BrokeLimit bool
|
||||
Attack bool
|
||||
RuleBlocked bool
|
||||
WAFBlocked bool
|
||||
BlockedByBlockRule bool
|
||||
BlockedByCoreRuleSet bool
|
||||
}
|
||||
|
||||
// New returns a Limiter for limits, with no client counted yet, whose
|
||||
@@ -323,12 +324,12 @@ func (l *Limiter) AddToHistory(client netip.Prefix, now time.Time, r Request) {
|
||||
h.Offences.Attack++
|
||||
}
|
||||
|
||||
if r.RuleBlocked {
|
||||
h.Offences.RuleBlocked++
|
||||
if r.BlockedByBlockRule {
|
||||
h.Offences.BlockedByBlockRule++
|
||||
}
|
||||
|
||||
if r.WAFBlocked {
|
||||
h.Offences.WAFBlocked++
|
||||
if r.BlockedByCoreRuleSet {
|
||||
h.Offences.BlockedByCoreRuleSet++
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
// Package reputation fetches the lists the settings name by URL: the
|
||||
// blocklists of SWW_BLOCKLIST_URLS, the file of AS:percent lines
|
||||
// SWW_ASN_LIMIT_PERCENT_URL names, and the decision list of the CrowdSec
|
||||
// engine SWW_CROWDSEC_LAPI_URL names. It keeps the last good copy of
|
||||
// SWW_ASN_LIMIT_PERCENT_URL names, the address lists of SWW_TAG_LISTS, and
|
||||
// the decision list of the CrowdSec engine SWW_CROWDSEC_LAPI_URL names. It
|
||||
// keeps the last good copy of
|
||||
// each, whole, comment lines included, which is used while a fetch fails,
|
||||
// and when each was last tried. It also asks the DNSBL zones of
|
||||
// SWW_DNSBL_ZONES about clients, and keeps their verdicts, and checks
|
||||
@@ -71,6 +72,8 @@ type Params struct {
|
||||
// (SWW_ASN_LIMIT_PERCENT_URL), "" while it is unset.
|
||||
BlocklistURLs []string
|
||||
ASNLimitPercentURL string
|
||||
// TagLists are the address lists that add a tag (SWW_TAG_LISTS).
|
||||
TagLists []config.TagList
|
||||
// CrowdSecDecisionsURL is the CrowdSec decision list, "" while
|
||||
// SWW_CROWDSEC_LAPI_URL is unset, fetched with CrowdSecKey
|
||||
// (SWW_CROWDSEC_LAPI_KEY).
|
||||
@@ -112,11 +115,11 @@ type list struct {
|
||||
failures int
|
||||
}
|
||||
|
||||
// entries are what the lines of a copy say: for a blocklist, the netblocks
|
||||
// it names, with the lengths among them, for the file of AS:percent lines,
|
||||
// the percentage it gives each AS number, and for the CrowdSec decision
|
||||
// list, the decision on each netblock that ends last, with the lengths
|
||||
// among them.
|
||||
// entries are what the lines of a copy say: for a blocklist or a tag list, the
|
||||
// netblocks it names, with the lengths among them, for the file of AS:percent
|
||||
// lines, the percentage it gives each AS number, and for the CrowdSec decision
|
||||
// list, the decision on each netblock that ends last, with the lengths among
|
||||
// them.
|
||||
type entries struct {
|
||||
netblocks map[netip.Prefix]bool
|
||||
lengths []int
|
||||
@@ -152,14 +155,18 @@ func New(params Params) *Lists {
|
||||
}
|
||||
|
||||
// URLs returns the URL of every list: the blocklists' in the order
|
||||
// SWW_BLOCKLIST_URLS names them, then SWW_ASN_LIMIT_PERCENT_URL, then
|
||||
// the CrowdSec decision list's.
|
||||
// SWW_BLOCKLIST_URLS names them, then SWW_ASN_LIMIT_PERCENT_URL, then the
|
||||
// tag lists', then the CrowdSec decision list's.
|
||||
func (l *Lists) URLs() []string {
|
||||
urls := slices.Clone(l.params.BlocklistURLs)
|
||||
if l.params.ASNLimitPercentURL != "" {
|
||||
urls = append(urls, l.params.ASNLimitPercentURL)
|
||||
}
|
||||
|
||||
for _, list := range l.params.TagLists {
|
||||
urls = append(urls, list.URL)
|
||||
}
|
||||
|
||||
if l.params.CrowdSecDecisionsURL != "" {
|
||||
urls = append(urls, l.params.CrowdSecDecisionsURL)
|
||||
}
|
||||
@@ -184,6 +191,22 @@ func (l *Lists) ListedBy(addr netip.Addr) []string {
|
||||
return listedBy
|
||||
}
|
||||
|
||||
// TagsOf returns the names of the tag lists whose copy holds addr.
|
||||
func (l *Lists) TagsOf(addr netip.Addr) []string {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
var names []string
|
||||
|
||||
for _, list := range l.params.TagLists {
|
||||
if l.lists[list.URL].entries.contain(addr) {
|
||||
names = append(names, list.Name)
|
||||
}
|
||||
}
|
||||
|
||||
return names
|
||||
}
|
||||
|
||||
// ASNLimitPercent returns the percentage the copy of the file of
|
||||
// AS:percent lines gives asn, and whether it lists asn.
|
||||
func (l *Lists) ASNLimitPercent(asn string) (int64, bool) {
|
||||
@@ -481,10 +504,10 @@ func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) {
|
||||
}
|
||||
|
||||
// parse reads the lines of the list at listURL, fetched at fetched: those
|
||||
// of a blocklist, of the file of AS:percent lines, or of the CrowdSec
|
||||
// decision list. In the first two, anything after a ; or a # on a line is
|
||||
// left out, and so is a line left blank. Any other line that does not read
|
||||
// is an error naming it by its number.
|
||||
// of a blocklist or a tag list, of the file of AS:percent lines, or of the
|
||||
// CrowdSec decision list. In all but the decision list, anything after a ; or
|
||||
// a # on a line is left out, and so is a line left blank. Any other line that
|
||||
// does not read is an error naming it by its number.
|
||||
func (l *Lists) parse(
|
||||
listURL string, lines []string, fetched time.Time,
|
||||
) (entries, error) {
|
||||
@@ -498,8 +521,8 @@ func (l *Lists) parse(
|
||||
}
|
||||
}
|
||||
|
||||
// parseNetblocks reads a blocklist's lines, each an address or a netblock
|
||||
// as the settings take them.
|
||||
// parseNetblocks reads the lines of a blocklist or a tag list, each an address
|
||||
// or a netblock as the settings take them.
|
||||
func parseNetblocks(lines []string) (entries, error) {
|
||||
found := entries{netblocks: map[netip.Prefix]bool{}}
|
||||
|
||||
@@ -551,7 +574,7 @@ func parseNetblock(text string) (netip.Prefix, bool) {
|
||||
}
|
||||
|
||||
// parsePercents reads the lines of the file of AS:percent lines, each an
|
||||
// AS number, : and a percentage, as SWW_ASN_LIMIT_PERCENT takes them. An
|
||||
// AS number, : and a percentage, such as AS64496:50. An
|
||||
// AS number listed more than once gets the lowest of its percentages.
|
||||
func parsePercents(lines []string) (entries, error) {
|
||||
found := entries{percents: map[string]int64{}}
|
||||
|
||||
@@ -19,32 +19,45 @@ const (
|
||||
// ActionForward is a request discriminator allowed, which rproxy passes to
|
||||
// the app.
|
||||
ActionForward = "forward"
|
||||
// ActionTooLarge is a request over the header limits.
|
||||
ActionTooLarge = "too_large"
|
||||
// ActionRateLimited is a request refused because it took its client
|
||||
// over a rate limit, which bans the client.
|
||||
ActionRateLimited = "rate_limited"
|
||||
// ActionBanned is a request refused because a ban covers its client,
|
||||
// or because it matched a ban rule, asked for a trap path or the
|
||||
// CrowdSec decision list lists its client, each of which bans the
|
||||
// client.
|
||||
ActionBanned = "banned"
|
||||
// ActionRuleBlocked is a request refused because it matched a block
|
||||
// rule.
|
||||
ActionRuleBlocked = "rule_blocked"
|
||||
// ActionWAFBlocked is a request refused because the Core Rule Set
|
||||
// scored it at or over SWW_WAF_ANOMALY_THRESHOLD.
|
||||
ActionWAFBlocked = "waf_blocked"
|
||||
// ActionDenied is a request refused because its client is in
|
||||
|
||||
// ActionRequestHeaderTooLarge is a request over the header limits.
|
||||
ActionRequestHeaderTooLarge = "request_header_too_large"
|
||||
|
||||
// ActionClientIsRateLimited is a request refused because it took its
|
||||
// client over a rate limit, which bans the client.
|
||||
ActionClientIsRateLimited = "client_is_rate_limited"
|
||||
|
||||
// ActionTemporarilyBanned is a request refused because a ban that runs
|
||||
// out covers its client, or because it matched a ban rule, asked for a
|
||||
// trap path or the CrowdSec decision list lists its client, and so made
|
||||
// such a ban.
|
||||
ActionTemporarilyBanned = "temporarily_banned"
|
||||
|
||||
// ActionPermanentlyBanned is a request refused because a ban that never
|
||||
// runs out covers its client, or because the request made such a ban or
|
||||
// made its client's ban permanent.
|
||||
ActionPermanentlyBanned = "permanently_banned"
|
||||
|
||||
// ActionBlockedByBlockRule is a request refused because it matched a
|
||||
// block rule.
|
||||
ActionBlockedByBlockRule = "blocked_by_block_rule"
|
||||
|
||||
// ActionBlockedByCoreRuleSet is a request refused because the Core Rule
|
||||
// Set scored it at or over SWW_WAF_ANOMALY_THRESHOLD.
|
||||
ActionBlockedByCoreRuleSet = "blocked_by_core_rule_set"
|
||||
|
||||
// ActionClientDenied is a request refused because its client is in
|
||||
// SWW_DENY_NETS, in a blocklist while SWW_BLOCKLIST_ACTION is deny,
|
||||
// or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while
|
||||
// SWW_REPUTATION_ACTION is deny.
|
||||
ActionDenied = "denied"
|
||||
// ActionCountryDenied is a request refused for its client's country.
|
||||
// SWW_REPUTATION_ACTION is deny, or that carries a tag in SWW_TAG_DENY.
|
||||
ActionClientDenied = "client_denied"
|
||||
|
||||
// ActionCountryDenied is a request refused because its client's country
|
||||
// is not in SWW_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||
ActionCountryDenied = "country_denied"
|
||||
// ActionAdmin is a request discriminator answered at one of its own
|
||||
// ActionOwnEndpoint is a request discriminator answered at one of its own
|
||||
// endpoints, or a question about a path under /_sww/.
|
||||
ActionAdmin = "admin"
|
||||
ActionOwnEndpoint = "own_endpoint"
|
||||
)
|
||||
|
||||
// OffenceLimit is the offence a request line names for a request that
|
||||
@@ -52,9 +65,9 @@ const (
|
||||
// limit.
|
||||
const OffenceLimit = "limit"
|
||||
|
||||
// LimitHitErrorBurst is the limit_hit a request line names for a request
|
||||
// LimitHitMinuteRefusals is the limit_hit a request line names for a request
|
||||
// that broke the error burst.
|
||||
const LimitHitErrorBurst = "error_burst"
|
||||
const LimitHitMinuteRefusals = "minute_refusals"
|
||||
|
||||
// The discriminator_line a request line names, which says which of
|
||||
// discriminator's two lines about a question it is.
|
||||
@@ -126,20 +139,31 @@ type Line struct {
|
||||
// The decision.
|
||||
Action string `json:"action"`
|
||||
// WouldAction is, in observe mode, the action enforce mode would have
|
||||
// taken with a request it would have refused: ActionDenied,
|
||||
// ActionBanned, ActionCountryDenied, ActionRateLimited,
|
||||
// ActionRuleBlocked, ActionWAFBlocked, or, in discriminator, ActionTooLarge
|
||||
// for the header limits.
|
||||
// taken with a request it would have refused: ActionClientDenied,
|
||||
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
|
||||
// ActionClientIsRateLimited, ActionBlockedByBlockRule,
|
||||
// ActionBlockedByCoreRuleSet, or, in discriminator,
|
||||
// ActionRequestHeaderTooLarge for the header limits.
|
||||
WouldAction string `json:"would_action,omitempty"`
|
||||
// LimitPercent and LimitPercentSetting are, for a request the rate
|
||||
// limits counted whose client a biased threshold gives a percentage of
|
||||
// the rate limits below 100, that percentage and the setting that gave
|
||||
// it. BytesPercent and BytesPercentSetting are the same for the byte
|
||||
// limits.
|
||||
LimitPercent *int64 `json:"limit_percent,omitempty"`
|
||||
LimitPercentSetting string `json:"limit_percent_setting,omitempty"`
|
||||
BytesPercent *int64 `json:"bytes_percent,omitempty"`
|
||||
BytesPercentSetting string `json:"bytes_percent_setting,omitempty"`
|
||||
// Heuristic is the name of the heuristic that refused the request, in
|
||||
// observe mode too, and HeuristicFigures are the figures it acted on, by
|
||||
// name.
|
||||
Heuristic string `json:"heuristic,omitempty"`
|
||||
HeuristicFigures map[string]any `json:"heuristic_figures,omitempty"`
|
||||
// AfterAnswerHeuristic and AfterAnswerHeuristicFigures are the same for
|
||||
// the heuristic that banned the client after discriminator answered the
|
||||
// request.
|
||||
AfterAnswerHeuristic string `json:"after_answer_heuristic,omitempty"`
|
||||
AfterAnswerHeuristicFigures map[string]any `json:"after_answer_heuristic_figures,omitempty"` //nolint:lll // a struct tag cannot be split
|
||||
// RateLimitPercent and RateLimitPercentSetting are, for a request the
|
||||
// rate limits counted whose client a biased threshold gives a percentage
|
||||
// of the rate limits below 100, that percentage and the setting that gave
|
||||
// it. BytesLimitPercent and BytesLimitPercentSetting are the same for the
|
||||
// byte limits.
|
||||
RateLimitPercent *int64 `json:"rate_limit_percent,omitempty"`
|
||||
RateLimitPercentSetting string `json:"rate_limit_percent_setting,omitempty"`
|
||||
BytesLimitPercent *int64 `json:"bytes_limit_percent,omitempty"`
|
||||
BytesLimitPercentSetting string `json:"bytes_limit_percent_setting,omitempty"`
|
||||
// Counts are, for a request the rate limits counted, the client's
|
||||
// requests as they counted them with this one, and its bytes as the
|
||||
// byte limits counted them, with this request's once it has ended if
|
||||
@@ -147,36 +171,42 @@ type Line struct {
|
||||
Counts ratelimit.Counts `json:"counts,omitzero"`
|
||||
// RuleIDs are the ids of the rule file rules the request matched.
|
||||
RuleIDs []string `json:"rule_ids,omitempty"`
|
||||
// WAFRuleIDs are the ids of the Core Rule Set's rules the request
|
||||
// matched, and WAFScore its anomaly score, nil for a request the Core
|
||||
// Rule Set did not inspect.
|
||||
WAFRuleIDs []int `json:"waf_rule_ids,omitempty"`
|
||||
WAFScore *int `json:"waf_score,omitempty"`
|
||||
// CoreRuleSetRuleIDs are the ids of the Core Rule Set's rules the request
|
||||
// matched, and CoreRuleSetScore its anomaly score, nil for a request the
|
||||
// Core Rule Set did not inspect.
|
||||
CoreRuleSetRuleIDs []int `json:"core_rule_set_rule_ids,omitempty"`
|
||||
CoreRuleSetScore *int `json:"core_rule_set_score,omitempty"`
|
||||
// LimitHit is the window whose limit the request went over, named as
|
||||
// Counts names its count: minute, hour or day for a rate limit, and
|
||||
// minute_bytes, hour_bytes or day_bytes for a byte limit; or
|
||||
// LimitHitErrorBurst for the error burst.
|
||||
// LimitHitMinuteRefusals for the error burst.
|
||||
LimitHit string `json:"limit_hit,omitempty"`
|
||||
// Reputation are the URLs of the blocklists that list the client, then
|
||||
// that of the CrowdSec decision list when it does, then the DNSBL zones
|
||||
// whose verdict lists it, their keys masked, then abuseipdb when its
|
||||
// score is a hit.
|
||||
Reputation []string `json:"reputation,omitempty"`
|
||||
// Tags are the request's tags, sorted.
|
||||
Tags []string `json:"tags,omitempty"`
|
||||
// RouteWatchTagsMissing, while SWW_ROUTEWATCH_URL is set, says why the
|
||||
// request got no tags from routewatch: no_answer_yet, request_failed or
|
||||
// no_route.
|
||||
RouteWatchTagsMissing string `json:"routewatch_tags_missing,omitempty"`
|
||||
// Offence is the offence the request was held as, OffenceLimit.
|
||||
Offence string `json:"offence,omitempty"`
|
||||
// BanExpires is when the ban the request made, or was refused under,
|
||||
// ends: a time, or "permanent".
|
||||
BanExpires string `json:"ban_expires,omitempty"`
|
||||
|
||||
// The timings, in milliseconds. DurationChecks is the time until the
|
||||
// checks were done, and DurationWAF the part of it the Core Rule Set
|
||||
// took. DurationUpstreamTotal is what the report of an allowed request gives,
|
||||
// The timings. DurationChecksMS is the time until the checks were done, and
|
||||
// DurationCoreRuleSetMS the part of it the Core Rule Set took.
|
||||
// DurationUpstreamTotalMS is what the report of an allowed request gives,
|
||||
// from the hand-over to the app until the end of its answer. Each but
|
||||
// DurationTotal is nil for a request that did not get that far.
|
||||
DurationTotal float64 `json:"duration_total"`
|
||||
DurationChecks *float64 `json:"duration_checks,omitempty"`
|
||||
DurationWAF *float64 `json:"duration_waf,omitempty"`
|
||||
DurationUpstreamTotal *float64 `json:"duration_upstream_total,omitempty"`
|
||||
// DurationTotalMS is nil for a request that did not get that far.
|
||||
DurationTotalMS float64 `json:"duration_total_ms"`
|
||||
DurationChecksMS *float64 `json:"duration_checks_ms,omitempty"`
|
||||
DurationCoreRuleSetMS *float64 `json:"duration_core_rule_set_ms,omitempty"`
|
||||
DurationUpstreamTotalMS *float64 `json:"duration_upstream_total_ms,omitempty"`
|
||||
}
|
||||
|
||||
// RProxyLine is rproxy's short line in the request log about one request. The
|
||||
@@ -196,11 +226,11 @@ type RProxyLine struct {
|
||||
Status int `json:"status"`
|
||||
RequestBytes int64 `json:"request_bytes"`
|
||||
ResponseBytes int64 `json:"response_bytes"`
|
||||
// DurationRequest runs in milliseconds from the request's headers being read
|
||||
// until the app has been sent the whole request, or the answer has started
|
||||
// if that is sooner. DurationResponse runs from then until the answer ends.
|
||||
DurationRequest float64 `json:"duration_request"`
|
||||
DurationResponse float64 `json:"duration_response"`
|
||||
// DurationRequestMS runs from the request's headers being read until the app
|
||||
// has been sent the whole request, or the answer has started if that is
|
||||
// sooner. DurationResponseMS runs from then until the answer ends.
|
||||
DurationRequestMS float64 `json:"duration_request_ms"`
|
||||
DurationResponseMS float64 `json:"duration_response_ms"`
|
||||
// Disposition is the valid disposition discriminator answered or, while
|
||||
// DefaultDisposition is true, the value of SWRP_DEFAULT_DISPOSITION. It is ""
|
||||
// when neither gave one.
|
||||
@@ -245,7 +275,11 @@ func FormatTime(t time.Time) string {
|
||||
|
||||
// Milliseconds is d in milliseconds, to the microsecond.
|
||||
func Milliseconds(d time.Duration) float64 {
|
||||
return float64(d.Microseconds()) / float64(time.Millisecond/time.Microsecond)
|
||||
return float64(
|
||||
d.Microseconds(),
|
||||
) / float64(
|
||||
time.Millisecond/time.Microsecond,
|
||||
)
|
||||
}
|
||||
|
||||
// NewProcessLogger returns the logger for the process's own messages:
|
||||
@@ -253,17 +287,25 @@ func Milliseconds(d time.Duration) float64 {
|
||||
// as a request line's, and instanceName, SWW_INSTANCE_NAME, as instance.
|
||||
// It writes only the messages at level, SWW_LOG_LEVEL, or more severe;
|
||||
// the request lines Write writes are never held back.
|
||||
func NewProcessLogger(w io.Writer, instanceName string, level slog.Level) *slog.Logger {
|
||||
func NewProcessLogger(
|
||||
w io.Writer,
|
||||
instanceName string,
|
||||
level slog.Level,
|
||||
) *slog.Logger {
|
||||
handler := slog.NewJSONHandler(w, &slog.HandlerOptions{
|
||||
Level: level,
|
||||
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
|
||||
if attr.Key == slog.TimeKey && len(groups) == 0 {
|
||||
return slog.String(slog.TimeKey, FormatTime(attr.Value.Time()))
|
||||
return slog.String(
|
||||
slog.TimeKey,
|
||||
FormatTime(attr.Value.Time()),
|
||||
)
|
||||
}
|
||||
|
||||
return attr
|
||||
},
|
||||
})
|
||||
|
||||
return slog.New(handler).With("type", "process", "instance", instanceName)
|
||||
return slog.New(handler).
|
||||
With("type", "process", "instance", instanceName)
|
||||
}
|
||||
@@ -22,7 +22,7 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
|
||||
ClientIP: "203.0.113.9",
|
||||
Status: 200,
|
||||
Action: requestlog.ActionForward,
|
||||
DurationTotal: requestlog.Milliseconds(1500 * time.Microsecond),
|
||||
DurationTotalMS: requestlog.Milliseconds(1500 * time.Microsecond),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("write: %v", err)
|
||||
@@ -43,7 +43,7 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
|
||||
want := map[string]any{
|
||||
"type": "request", "time": "2026-10-03T12:00:00.000Z",
|
||||
"client_ip": "203.0.113.9", "status": 200.0, "action": "forward",
|
||||
"duration_total": 1.5,
|
||||
"duration_total_ms": 1.5,
|
||||
}
|
||||
for name, value := range want {
|
||||
if fields[name] != value {
|
||||
@@ -55,9 +55,9 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
|
||||
"forwarded_for", "content_type", "content_length", "request_headers",
|
||||
"has_authorization", "has_cookie", "websocket", "response_content_type",
|
||||
"upstream_status", "cache_control", "location", "aborted", "counts",
|
||||
"limit_hit", "offence", "ban_expires", "duration_checks",
|
||||
"limit_hit", "offence", "ban_expires", "duration_checks_ms",
|
||||
"duration_upstream_connect", "duration_upstream_first_byte",
|
||||
"duration_upstream_total",
|
||||
"duration_upstream_total_ms",
|
||||
}
|
||||
for _, name := range unset {
|
||||
_, present := fields[name]
|
||||
@@ -82,7 +82,7 @@ func TestWriteRProxyLineWritesOneJSONLineWithOnlyItsFields(t *testing.T) {
|
||||
|
||||
const want = `{"type":"request","time":"","request_id":"an-id","client_ip":"",` +
|
||||
`"host":"","method":"","path":"","query":"","status":0,"request_bytes":0,` +
|
||||
`"response_bytes":0,"duration_request":0,"duration_response":0,` +
|
||||
`"response_bytes":0,"duration_request_ms":0,"duration_response_ms":0,` +
|
||||
`"disposition":"allow"}` + "\n"
|
||||
if out.String() != want {
|
||||
t.Errorf("wrote %s, want %s", out.String(), want)
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
// Package routewatch asks a routewatch instance about client addresses, in the
|
||||
// background, and keeps its answers in memory. A routewatch instance knows, for
|
||||
// an address, the netblock it is announced in and the AS that originates it.
|
||||
package routewatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||
)
|
||||
|
||||
const (
|
||||
// answerKept is how long an answer is used after it came.
|
||||
answerKept = time.Hour
|
||||
// failureKept is how long routewatch is not asked again after a query
|
||||
// fails, so that a dead routewatch is not asked on every request.
|
||||
failureKept = time.Minute
|
||||
// queryTimeout is how long one query may take. routewatch answers only
|
||||
// after a reverse DNS lookup it allows 2 seconds, so this is longer.
|
||||
queryTimeout = 5 * time.Second
|
||||
// maxAnswers is how many answers are kept. Past it, the one that came
|
||||
// longest ago is dropped.
|
||||
maxAnswers = 100000
|
||||
// maxQueries is how many queries may be under way at once. Past it, a
|
||||
// client is not asked about until its next request, so that a swarm of
|
||||
// new addresses cannot fill the memory.
|
||||
maxQueries = 100
|
||||
// maxAnswerBytes is the most of an answer that is read.
|
||||
maxAnswerBytes = 64 << 10
|
||||
)
|
||||
|
||||
// Missing is why Route gives no route. The request log's field
|
||||
// routewatch_tags_missing gives these values.
|
||||
type Missing string
|
||||
|
||||
const (
|
||||
// NoAnswerYet is a client routewatch has not answered about yet.
|
||||
NoAnswerYet Missing = "no_answer_yet"
|
||||
// RequestFailed is no answer because a recent query failed.
|
||||
RequestFailed Missing = "request_failed"
|
||||
// NoRoute is routewatch knowing no live route for the address.
|
||||
NoRoute Missing = "no_route"
|
||||
)
|
||||
|
||||
var (
|
||||
errStatus = errors.New("routewatch answered")
|
||||
errNoResult = errors.New("the answer holds no result")
|
||||
errNotNetblock = errors.New("the answer's netblock is not a netblock")
|
||||
)
|
||||
|
||||
// Route is what routewatch knows of an address: the netblock it is announced
|
||||
// in, and the AS that originates it.
|
||||
type Route struct {
|
||||
Netblock netip.Prefix
|
||||
ASN uint32
|
||||
}
|
||||
|
||||
// Params are what New needs.
|
||||
type Params struct {
|
||||
// URL is where routewatch is (SWW_ROUTEWATCH_URL).
|
||||
URL string
|
||||
// Now tells the time, normally time.Now in UTC.
|
||||
Now func() time.Time
|
||||
// ProcessLog receives the query that fails, and why.
|
||||
ProcessLog *slog.Logger
|
||||
}
|
||||
|
||||
// Client asks a routewatch instance about clients, in the background, and keeps
|
||||
// its answers. It is safe for concurrent use.
|
||||
type Client struct {
|
||||
params Params
|
||||
httpClient *http.Client
|
||||
|
||||
mu sync.Mutex
|
||||
// answers are by client. Each is added as it comes and never moved up,
|
||||
// so that the one that came longest ago is the first dropped.
|
||||
answers *simplelru.LRU[netip.Prefix, answer]
|
||||
// asking are the clients whose query is under way.
|
||||
asking map[netip.Prefix]bool
|
||||
// retryAt is when routewatch may be asked again after a query failed.
|
||||
retryAt time.Time
|
||||
}
|
||||
|
||||
// answer is one kept answer: the route, or why there is none, and when it came.
|
||||
type answer struct {
|
||||
route Route
|
||||
missing Missing
|
||||
came time.Time
|
||||
}
|
||||
|
||||
// New returns a Client with no answer yet.
|
||||
func New(params Params) *Client {
|
||||
answers, err := simplelru.NewLRU[netip.Prefix, answer](maxAnswers, nil)
|
||||
if err != nil {
|
||||
panic(err) // NewLRU fails only for a size below one
|
||||
}
|
||||
|
||||
return &Client{
|
||||
params: params,
|
||||
httpClient: &http.Client{},
|
||||
answers: answers,
|
||||
asking: map[netip.Prefix]bool{},
|
||||
}
|
||||
}
|
||||
|
||||
// Route returns what routewatch answered about client, an IPv4 address or an
|
||||
// IPv6 group, or why it has no route. It never waits for a query. A client
|
||||
// without an answer is asked about in the background, by addr, the address its
|
||||
// request came from, unless its query is under way, a query failed less than
|
||||
// failureKept ago, or maxQueries are under way. ctx is the context of the
|
||||
// client's request, and a query goes on after the request ends.
|
||||
func (c *Client) Route(
|
||||
ctx context.Context, client netip.Prefix, addr netip.Addr,
|
||||
) (Route, Missing) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
now := c.params.Now()
|
||||
|
||||
kept, found := c.answers.Peek(client)
|
||||
if found && now.Sub(kept.came) < answerKept {
|
||||
return kept.route, kept.missing
|
||||
}
|
||||
|
||||
if now.Before(c.retryAt) {
|
||||
return Route{}, RequestFailed
|
||||
}
|
||||
|
||||
if !c.asking[client] && len(c.asking) < maxQueries {
|
||||
c.asking[client] = true
|
||||
|
||||
go c.ask(context.WithoutCancel(ctx), client, addr)
|
||||
}
|
||||
|
||||
return Route{}, NoAnswerYet
|
||||
}
|
||||
|
||||
// ask asks routewatch about addr, keeps the answer as client's, and notes the
|
||||
// query as no longer under way. A query that fails gives no answer, and
|
||||
// routewatch is not asked again for failureKept. Only the first failure of
|
||||
// those under way is logged.
|
||||
func (c *Client) ask(ctx context.Context, client netip.Prefix, addr netip.Addr) {
|
||||
route, missing, err := c.query(ctx, addr)
|
||||
now := c.params.Now()
|
||||
|
||||
c.mu.Lock()
|
||||
|
||||
delete(c.asking, client)
|
||||
|
||||
firstFailure := err != nil && !now.Before(c.retryAt)
|
||||
|
||||
if err == nil {
|
||||
c.answers.Add(client, answer{route: route, missing: missing, came: now})
|
||||
} else {
|
||||
c.retryAt = now.Add(failureKept)
|
||||
}
|
||||
|
||||
c.mu.Unlock()
|
||||
|
||||
if firstFailure {
|
||||
c.params.ProcessLog.Warn("asking routewatch failed", "error", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
// query asks routewatch's IP info API about addr. An answer other than 200, one
|
||||
// that cannot be read, and none within queryTimeout, fail. A result with no
|
||||
// netblock is an address without a live route.
|
||||
func (c *Client) query(ctx context.Context, addr netip.Addr) (Route, Missing, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, queryTimeout)
|
||||
defer cancel()
|
||||
|
||||
endpoint, err := url.JoinPath(c.params.URL, "api/v1/ip", addr.String())
|
||||
if err != nil {
|
||||
return Route{}, "", fmt.Errorf("make the URL: %w", err)
|
||||
}
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, http.NoBody)
|
||||
if err != nil {
|
||||
return Route{}, "", fmt.Errorf("make the request: %w", err)
|
||||
}
|
||||
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
res, err := c.httpClient.Do(req)
|
||||
if err != nil {
|
||||
// Do's error names the URL, which holds the client's address, which
|
||||
// is not to be logged: only what went wrong is kept.
|
||||
return Route{}, "", fmt.Errorf("ask routewatch: %w", errors.Unwrap(err))
|
||||
}
|
||||
|
||||
defer func() {
|
||||
_ = res.Body.Close()
|
||||
}()
|
||||
|
||||
if res.StatusCode != http.StatusOK {
|
||||
return Route{}, "", fmt.Errorf("%w %s", errStatus, res.Status)
|
||||
}
|
||||
|
||||
return readAnswer(io.LimitReader(res.Body, maxAnswerBytes))
|
||||
}
|
||||
|
||||
// readAnswer reads routewatch's answer about one address.
|
||||
func readAnswer(body io.Reader) (Route, Missing, error) {
|
||||
var decoded struct {
|
||||
Data struct {
|
||||
Results []struct {
|
||||
Netblock string `json:"netblock"`
|
||||
ASN uint32 `json:"asn"`
|
||||
} `json:"results"`
|
||||
} `json:"data"`
|
||||
}
|
||||
|
||||
err := json.NewDecoder(body).Decode(&decoded)
|
||||
if err != nil {
|
||||
return Route{}, "", fmt.Errorf("read the answer: %w", err)
|
||||
}
|
||||
|
||||
if len(decoded.Data.Results) == 0 {
|
||||
return Route{}, "", errNoResult
|
||||
}
|
||||
|
||||
result := decoded.Data.Results[0]
|
||||
if result.Netblock == "" {
|
||||
return Route{}, NoRoute, nil
|
||||
}
|
||||
|
||||
netblock, err := netip.ParsePrefix(result.Netblock)
|
||||
if err != nil {
|
||||
return Route{}, "", fmt.Errorf("%w: %q", errNotNetblock, result.Netblock)
|
||||
}
|
||||
|
||||
return Route{Netblock: netblock.Masked(), ASN: result.ASN}, "", nil
|
||||
}
|
||||
@@ -0,0 +1,333 @@
|
||||
package routewatch_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/routewatch"
|
||||
)
|
||||
|
||||
const (
|
||||
clientAddr = "192.0.2.1"
|
||||
routedAnswer = `{"status":"ok","data":{"results":[` +
|
||||
`{"ip":"192.0.2.1","netblock":"192.0.2.0/24","asn":64496}]}}`
|
||||
noRouteAnswer = `{"status":"ok","data":{"results":[{"ip":"192.0.2.1"}]}}`
|
||||
// slowAnswer is longer than routewatch allows a reverse DNS lookup.
|
||||
slowAnswer = 2500 * time.Millisecond
|
||||
// waitLimit is longer than the Client allows a query.
|
||||
waitLimit = 10 * time.Second
|
||||
)
|
||||
|
||||
// clock is a time the tests move by hand.
|
||||
type clock struct {
|
||||
mu sync.Mutex
|
||||
now time.Time
|
||||
}
|
||||
|
||||
func (c *clock) Now() time.Time {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
return c.now
|
||||
}
|
||||
|
||||
func (c *clock) advance(by time.Duration) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
c.now = c.now.Add(by)
|
||||
}
|
||||
|
||||
// fixture is a Client asking a stand-in for routewatch. queries counts what the
|
||||
// stand-in was asked, paths holds the last path it was asked for, and log holds
|
||||
// the process log.
|
||||
type fixture struct {
|
||||
t *testing.T
|
||||
server *httptest.Server
|
||||
client *routewatch.Client
|
||||
clk *clock
|
||||
queries *atomic.Int32
|
||||
path *atomic.Value
|
||||
log *syncBuffer
|
||||
}
|
||||
|
||||
// syncBuffer is a buffer the background queries can log to while a test reads it.
|
||||
type syncBuffer struct {
|
||||
mu sync.Mutex
|
||||
buf bytes.Buffer
|
||||
}
|
||||
|
||||
func (b *syncBuffer) Write(p []byte) (int, error) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
|
||||
return b.buf.Write(p)
|
||||
}
|
||||
|
||||
func (b *syncBuffer) String() string {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
|
||||
return b.buf.String()
|
||||
}
|
||||
|
||||
func newFixture(t *testing.T, standIn http.HandlerFunc) *fixture {
|
||||
t.Helper()
|
||||
|
||||
f := &fixture{
|
||||
t: t,
|
||||
clk: &clock{now: time.Date(2026, 11, 9, 0, 0, 0, 0, time.UTC)},
|
||||
queries: &atomic.Int32{},
|
||||
path: &atomic.Value{},
|
||||
log: &syncBuffer{},
|
||||
}
|
||||
|
||||
f.server = httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
f.queries.Add(1)
|
||||
f.path.Store(r.URL.Path)
|
||||
standIn(w, r)
|
||||
}))
|
||||
t.Cleanup(f.server.Close)
|
||||
|
||||
f.client = routewatch.New(routewatch.Params{
|
||||
URL: f.server.URL + "/", Now: f.clk.Now,
|
||||
ProcessLog: slog.New(slog.NewTextHandler(f.log, nil)),
|
||||
})
|
||||
|
||||
return f
|
||||
}
|
||||
|
||||
func answering(body string) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(body))
|
||||
}
|
||||
}
|
||||
|
||||
func (f *fixture) route() (routewatch.Route, routewatch.Missing) {
|
||||
addr := netip.MustParseAddr(clientAddr)
|
||||
client := netip.PrefixFrom(addr, addr.BitLen())
|
||||
|
||||
return f.client.Route(context.Background(), client, addr)
|
||||
}
|
||||
|
||||
// waitFor asks until the answer is why, or, while why is "", until there is a
|
||||
// route, and returns the route.
|
||||
func (f *fixture) waitFor(why routewatch.Missing) routewatch.Route {
|
||||
f.t.Helper()
|
||||
|
||||
deadline := time.Now().Add(waitLimit)
|
||||
|
||||
for {
|
||||
route, missing := f.route()
|
||||
if missing == why {
|
||||
return route
|
||||
}
|
||||
|
||||
if time.Now().After(deadline) {
|
||||
f.t.Fatalf("still %q after %s, want %q", missing, waitLimit, why)
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// wantFailureLoggedWithoutTheClient waits for the process log to name a failed
|
||||
// query, and checks that it does not name the client.
|
||||
func (f *fixture) wantFailureLoggedWithoutTheClient() {
|
||||
f.t.Helper()
|
||||
|
||||
// The failure is noted before it is logged.
|
||||
deadline := time.Now().Add(waitLimit)
|
||||
for !strings.Contains(f.log.String(), "asking routewatch failed") {
|
||||
if time.Now().After(deadline) {
|
||||
f.t.Fatalf("process log %q names no failure", f.log.String())
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
|
||||
if strings.Contains(f.log.String(), clientAddr) {
|
||||
f.t.Errorf("process log %q names the client", f.log.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRouteGivesRoutewatchsAnswerAndAsksByAddress(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, answering(routedAnswer))
|
||||
|
||||
if _, missing := f.route(); missing != routewatch.NoAnswerYet {
|
||||
t.Errorf("before the answer: %q, want %q", missing, routewatch.NoAnswerYet)
|
||||
}
|
||||
|
||||
route := f.waitFor("")
|
||||
|
||||
want := routewatch.Route{Netblock: netip.MustParsePrefix("192.0.2.0/24"), ASN: 64496}
|
||||
if route != want {
|
||||
t.Errorf("route %+v, want %+v", route, want)
|
||||
}
|
||||
|
||||
if got := f.path.Load(); got != "/api/v1/ip/"+clientAddr {
|
||||
t.Errorf("asked for path %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnswerAfterASlowReverseDNSLookupIsKept(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
time.Sleep(slowAnswer)
|
||||
|
||||
_, _ = w.Write([]byte(routedAnswer))
|
||||
})
|
||||
|
||||
f.route()
|
||||
|
||||
route := f.waitFor("")
|
||||
|
||||
want := routewatch.Route{Netblock: netip.MustParsePrefix("192.0.2.0/24"), ASN: 64496}
|
||||
if route != want {
|
||||
t.Errorf("route %+v, want %+v", route, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddressWithoutARouteIsKeptAsNoRoute(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, answering(noRouteAnswer))
|
||||
|
||||
f.route()
|
||||
f.waitFor(routewatch.NoRoute)
|
||||
|
||||
if got := f.queries.Load(); got != 1 {
|
||||
t.Errorf("%d queries, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnswerRoutewatchCannotBeTakenFromFails(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := map[string]http.HandlerFunc{
|
||||
"500": func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
},
|
||||
"not JSON": answering("no"),
|
||||
"no results": answering(`{"status":"ok","data":{"results":[]}}`),
|
||||
"bad netblock": answering(
|
||||
`{"status":"ok","data":{"results":[{"netblock":"nowhere","asn":1}]}}`),
|
||||
}
|
||||
for name, standIn := range tests {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, standIn)
|
||||
|
||||
f.route()
|
||||
f.waitFor(routewatch.RequestFailed)
|
||||
f.wantFailureLoggedWithoutTheClient()
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefusedConnectionFailsTheQueryAndTheLogOmitsTheClient(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, answering(routedAnswer))
|
||||
|
||||
f.server.Close()
|
||||
|
||||
f.route()
|
||||
f.waitFor(routewatch.RequestFailed)
|
||||
f.wantFailureLoggedWithoutTheClient()
|
||||
}
|
||||
|
||||
func TestFailureIsKeptForAMinuteSoRoutewatchIsNotAskedPerRequest(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
})
|
||||
|
||||
f.route()
|
||||
f.waitFor(routewatch.RequestFailed)
|
||||
f.route()
|
||||
|
||||
if got := f.queries.Load(); got != 1 {
|
||||
t.Errorf("%d queries while the failure is kept, want 1", got)
|
||||
}
|
||||
|
||||
f.clk.advance(time.Minute)
|
||||
f.route()
|
||||
|
||||
deadline := time.Now().Add(waitLimit)
|
||||
for f.queries.Load() != 2 {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("%d queries after the failure expired, want 2", f.queries.Load())
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHangingRoutewatchNeverDelaysAndFailsTheQuery(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, func(_ http.ResponseWriter, r *http.Request) {
|
||||
<-r.Context().Done()
|
||||
})
|
||||
|
||||
start := time.Now()
|
||||
|
||||
f.route()
|
||||
|
||||
if elapsed := time.Since(start); elapsed > time.Second {
|
||||
t.Errorf("Route took %s while routewatch hangs", elapsed)
|
||||
}
|
||||
|
||||
f.waitFor(routewatch.RequestFailed)
|
||||
f.wantFailureLoggedWithoutTheClient()
|
||||
}
|
||||
|
||||
func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
release := make(chan struct{})
|
||||
f := newFixture(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
<-release
|
||||
|
||||
_, _ = w.Write([]byte(routedAnswer))
|
||||
})
|
||||
|
||||
for range 50 {
|
||||
f.route()
|
||||
}
|
||||
|
||||
close(release)
|
||||
f.waitFor("")
|
||||
|
||||
for range 50 {
|
||||
f.route()
|
||||
}
|
||||
|
||||
if got := f.queries.Load(); got != 1 {
|
||||
t.Errorf("%d queries for repeated requests, want 1", got)
|
||||
}
|
||||
|
||||
f.clk.advance(time.Hour)
|
||||
f.waitFor(routewatch.NoAnswerYet)
|
||||
f.waitFor("")
|
||||
|
||||
if got := f.queries.Load(); got != 2 {
|
||||
t.Errorf("%d queries after the answer expired, want 2", got)
|
||||
}
|
||||
}
|
||||
@@ -371,8 +371,8 @@ func (rq *request) finish(ctx context.Context) {
|
||||
Status: rq.out.status,
|
||||
RequestBytes: requestBytes,
|
||||
ResponseBytes: responseBytes,
|
||||
DurationRequest: requestlog.Milliseconds(requestEnd.Sub(rq.start)),
|
||||
DurationResponse: requestlog.Milliseconds(end.Sub(requestEnd)),
|
||||
DurationRequestMS: requestlog.Milliseconds(requestEnd.Sub(rq.start)),
|
||||
DurationResponseMS: requestlog.Milliseconds(end.Sub(requestEnd)),
|
||||
Disposition: rq.disposition,
|
||||
DefaultDisposition: rq.byDefault,
|
||||
})
|
||||
@@ -389,10 +389,10 @@ func (rq *request) finish(ctx context.Context) {
|
||||
Status: min(rq.out.status, reportStatusMax),
|
||||
RequestBytes: requestBytes,
|
||||
ResponseBytes: responseBytes,
|
||||
DurationTotal: requestlog.Milliseconds(end.Sub(rq.start)),
|
||||
DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)),
|
||||
}
|
||||
if !rq.upstreamStart.IsZero() {
|
||||
report.DurationUpstreamTotal = requestlog.Milliseconds(end.Sub(rq.upstreamStart))
|
||||
report.DurationUpstreamTotalMS = requestlog.Milliseconds(end.Sub(rq.upstreamStart))
|
||||
}
|
||||
|
||||
rq.h.sendReport(ctx, report)
|
||||
|
||||
@@ -45,9 +45,9 @@ const (
|
||||
idleTimeout = 120 * time.Second
|
||||
)
|
||||
|
||||
// adminPrefix starts the path of every request for sww itself, which
|
||||
// ownEndpointPrefix starts the path of every request for sww itself, which
|
||||
// never reaches the app.
|
||||
const adminPrefix = "/_sww/"
|
||||
const ownEndpointPrefix = "/_sww/"
|
||||
|
||||
// allowedMethods lists the methods rproxy passes on, as an Allow header gives
|
||||
// them. It answers any other method 405 itself.
|
||||
@@ -211,8 +211,8 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
|
||||
// passToApp passes the request on to the app, unless a limit or its path keeps
|
||||
// it from the app. discriminator answers a path under adminPrefix itself, so such a
|
||||
// request gets here only when SWRP_DEFAULT_DISPOSITION allowed it. ctx is the
|
||||
// it from the app. discriminator answers a path under ownEndpointPrefix itself, so
|
||||
// such a request gets here only when SWRP_DEFAULT_DISPOSITION allowed it. ctx is the
|
||||
// request's own context.
|
||||
func (rq *request) passToApp(ctx context.Context) {
|
||||
maxBytes := rq.h.config.MaxRequestSize
|
||||
@@ -220,7 +220,7 @@ func (rq *request) passToApp(ctx context.Context) {
|
||||
switch {
|
||||
case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes):
|
||||
rq.answer(http.StatusRequestEntityTooLarge)
|
||||
case strings.HasPrefix(rq.in.URL.Path, adminPrefix):
|
||||
case strings.HasPrefix(rq.in.URL.Path, ownEndpointPrefix):
|
||||
rq.answer(http.StatusNotFound)
|
||||
default:
|
||||
rq.forward(ctx)
|
||||
|
||||
@@ -727,7 +727,7 @@ func TestDenyGivesTheVisitorDiscriminatorsAnswer(t *testing.T) {
|
||||
report := standIn.report(t)
|
||||
if report.Status != http.StatusForbidden ||
|
||||
report.ResponseBytes != int64(len(deniedBody)) ||
|
||||
report.DurationUpstreamTotal != 0 {
|
||||
report.DurationUpstreamTotalMS != 0 {
|
||||
t.Errorf("report %+v, of a request denied", report)
|
||||
}
|
||||
|
||||
@@ -871,9 +871,9 @@ func TestALineForARequestClosedWithoutAnAnswerHasNoNegativeTime(t *testing.T) {
|
||||
|
||||
line := out.requestLine(t)
|
||||
|
||||
duration, ok := line["duration_response"].(float64)
|
||||
duration, ok := line["duration_response_ms"].(float64)
|
||||
if !ok || duration < 0 {
|
||||
t.Errorf("line %v, want a duration_response that is not negative", line)
|
||||
t.Errorf("line %v, want a duration_response_ms that is not negative", line)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1428,8 +1428,8 @@ func TestReportGivesWhatHappened(t *testing.T) {
|
||||
if report.RequestID == "" || report.Status != http.StatusOK ||
|
||||
report.RequestBytes != int64(len(body)) ||
|
||||
report.ResponseBytes != int64(len(greeting)) ||
|
||||
report.DurationTotal <= 0 || report.DurationUpstreamTotal <= 0 ||
|
||||
report.DurationUpstreamTotal > report.DurationTotal {
|
||||
report.DurationTotalMS <= 0 || report.DurationUpstreamTotalMS <= 0 ||
|
||||
report.DurationUpstreamTotalMS > report.DurationTotalMS {
|
||||
t.Errorf("report %+v, of %d bytes up and %d back", report, len(body), len(greeting))
|
||||
}
|
||||
|
||||
@@ -1748,7 +1748,7 @@ func TestLogLineGivesTheRequestAsSent(t *testing.T) {
|
||||
t.Errorf("default_disposition is %v, want it left out", line["default_disposition"])
|
||||
}
|
||||
|
||||
for _, name := range []string{"duration_request", "duration_response"} {
|
||||
for _, name := range []string{"duration_request_ms", "duration_response_ms"} {
|
||||
duration, ok := line[name].(float64)
|
||||
if !ok || duration <= 0 {
|
||||
t.Errorf("%s is %v, want a time", name, line[name])
|
||||
|
||||
@@ -64,7 +64,7 @@ var (
|
||||
errCause = errors.New("is not limit, attack, admin or crowdsec")
|
||||
errExpires = errors.New("is not after its start")
|
||||
errDestination = errors.New("is not webhook, slack or ntfy")
|
||||
errScope = errors.New("is not client, net, asn, total or watch")
|
||||
errScope = errors.New("is not client, net, asn, total or watched_net")
|
||||
errWaitingList = errors.New(`waiting is a list, but now lists the alerts by ` +
|
||||
`destination: put the list under "webhook", as "waiting": {"webhook": [...]}, ` +
|
||||
`or remove the file`)
|
||||
@@ -858,8 +858,8 @@ func (f *alertsFile) check([]byte) error {
|
||||
}
|
||||
|
||||
// checkAnomalyCounters refuses an anomaly counter whose scope is not
|
||||
// client, net, asn, total or watch, most likely misspelt, and one without
|
||||
// a field it needs, as missingFromCounter tells.
|
||||
// client, net, asn, total or watched_net, most likely misspelt, and one
|
||||
// without a field it needs, as missingFromCounter tells.
|
||||
func checkAnomalyCounters(counters []anomaly.Counter) error {
|
||||
for i, counter := range counters {
|
||||
if !slices.Contains(anomaly.Scopes(), counter.Scope) {
|
||||
@@ -879,9 +879,9 @@ func checkAnomalyCounters(counters []anomaly.Counter) error {
|
||||
// missingFromCounter returns the first field counter, an anomaly counter,
|
||||
// needs and has not, or "" when it has them all: what tells it from the
|
||||
// others in its scope, without which it would never be counted again, the
|
||||
// netblock of a client, net or watch counter, the AS number of an asn one
|
||||
// and the name of a watch one; and the start of a window in which it has
|
||||
// requests or bytes, without which they would be dropped.
|
||||
// netblock of a client, net or watched_net counter, the AS number of an
|
||||
// asn one and the name of a watched_net one; and the start of a window in
|
||||
// which it has requests or bytes, without which they would be dropped.
|
||||
func missingFromCounter(counter anomaly.Counter) string {
|
||||
scope := counter.Scope
|
||||
|
||||
@@ -891,7 +891,7 @@ func missingFromCounter(counter anomaly.Counter) string {
|
||||
return "netblock"
|
||||
case scope == anomaly.ScopeASN && counter.ASN == "":
|
||||
return "asn"
|
||||
case scope == anomaly.ScopeWatch && counter.Name == "":
|
||||
case scope == anomaly.ScopeWatchedNet && counter.Name == "":
|
||||
return "name"
|
||||
case countsWithoutStart(counter.Minute):
|
||||
return "minute.start"
|
||||
|
||||
@@ -208,7 +208,7 @@ const filledAlertsJSON = `{
|
||||
}
|
||||
},
|
||||
{
|
||||
"scope": "watch",
|
||||
"scope": "watched_net",
|
||||
"netblock": "203.0.113.0/24",
|
||||
"name": "office",
|
||||
"hour": {
|
||||
@@ -534,7 +534,7 @@ func TestFileThatDoesNotParseStopsTheStart(t *testing.T) {
|
||||
`{"version": 1, "anomaly_counters": [{"scope": "total"}, ` +
|
||||
`{"scope": "nett", "netblock": "203.0.113.0/24"}]}`,
|
||||
`: anomaly_counters entry 2's scope "nett" is not client, net, asn, total ` +
|
||||
`or watch`,
|
||||
`or watched_net`,
|
||||
},
|
||||
{
|
||||
"a copy of a list with a line that does not read", reputationJSON,
|
||||
@@ -831,12 +831,12 @@ func TestAlertsJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
|
||||
{
|
||||
"an anomaly counter of a named netblock without its name",
|
||||
`{"version": 1, "anomaly_counters": [` +
|
||||
`{"scope": "watch", "netblock": "203.0.113.0/24"}]}`,
|
||||
`{"scope": "watched_net", "netblock": "203.0.113.0/24"}]}`,
|
||||
`: anomaly_counters entry 1 has no "name"`,
|
||||
},
|
||||
{
|
||||
"an anomaly counter of a named netblock without its netblock",
|
||||
`{"version": 1, "anomaly_counters": [{"scope": "watch", "name": "office"}]}`,
|
||||
`{"version": 1, "anomaly_counters": [{"scope": "watched_net", "name": "office"}]}`,
|
||||
`: anomaly_counters entry 1 has no "netblock"`,
|
||||
},
|
||||
{
|
||||
@@ -1729,7 +1729,7 @@ func newParams(dir string) state.Params {
|
||||
Dir: dir,
|
||||
WriteDelay: time.Hour,
|
||||
CounterInterval: time.Hour,
|
||||
Ledger: bans.New(bans.Rules{
|
||||
Ledger: bans.New(bans.Params{
|
||||
LimitBanDuration: time.Hour,
|
||||
LimitBanRepeatWindow: 24 * time.Hour,
|
||||
MaxBanDuration: 7 * 24 * time.Hour,
|
||||
|
||||
+2
-2
@@ -22,6 +22,7 @@ import (
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/discriminator"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
||||
@@ -29,7 +30,6 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
"sneak.berlin/go/smallwebwaf/internal/state"
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
)
|
||||
|
||||
// shutdownTimeout is how long requests in progress may take to finish
|
||||
@@ -197,7 +197,7 @@ func RunDiscriminator(ctx context.Context, params Params) int {
|
||||
func newServer(
|
||||
cfg *config.Config, stdout io.Writer, processLog *slog.Logger,
|
||||
now func() time.Time, ruleFiles *rules.Files, alertQueue *alerts.Queue,
|
||||
crs *waf.CoreRuleSet,
|
||||
crs *coreruleset.CoreRuleSet,
|
||||
) (*discriminator.Server, error) {
|
||||
var lookupFile *lookup.File
|
||||
|
||||
|
||||
+15
-10
@@ -373,13 +373,13 @@ func TestStateKeptAcrossRestarts(t *testing.T) {
|
||||
out := runUntilStopped(t, env, func(url string) {
|
||||
wantRefused(t, url)
|
||||
})
|
||||
out.line(t, "action", "rate_limited")
|
||||
out.line(t, "action", "client_is_rate_limited")
|
||||
|
||||
// After another, the ban still refuses it.
|
||||
out = runUntilStopped(t, env, func(url string) {
|
||||
wantRefused(t, url)
|
||||
})
|
||||
out.line(t, "action", "banned")
|
||||
out.line(t, "action", "temporarily_banned")
|
||||
}
|
||||
|
||||
func TestBanRefusesItsNetblockAfterARestartWithAnotherScope(t *testing.T) {
|
||||
@@ -526,7 +526,7 @@ func TestRuleFileAddedWhileRunningTakesEffect(t *testing.T) {
|
||||
time.Sleep(pollInterval)
|
||||
}
|
||||
})
|
||||
out.line(t, "action", "rule_blocked")
|
||||
out.line(t, "action", "blocked_by_block_rule")
|
||||
}
|
||||
|
||||
func TestRuleFileThatDoesNotParseStopsTheStart(t *testing.T) {
|
||||
@@ -618,7 +618,7 @@ func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) {
|
||||
instanceName: instance,
|
||||
lookupSource: "file",
|
||||
lookupDBPath: path,
|
||||
"SWW_DENIED_COUNTRIES": "kp",
|
||||
"SWW_TAG_DENY": "country=KP",
|
||||
// The requests sent until a replacement takes effect, and those
|
||||
// for the metrics, must not break a rate limit, whose ban would
|
||||
// refuse them too.
|
||||
@@ -740,7 +740,7 @@ func TestBlocklistTriesAndCopiesKeptInReputationJSONAcrossRestarts(t *testing.T)
|
||||
time.Sleep(pollInterval)
|
||||
}
|
||||
})
|
||||
wantDeniedByList(t, out.line(t, "action", "denied"), dropURL)
|
||||
wantDeniedByList(t, out.line(t, "action", "client_denied"), dropURL)
|
||||
|
||||
if fetches := torFetches.Load(); fetches != 1 {
|
||||
t.Errorf("tor.txt fetched %d times, want once, before the restart", fetches)
|
||||
@@ -843,7 +843,8 @@ func wantDeniedByList(t *testing.T, line map[string]any, listURL string) {
|
||||
t.Helper()
|
||||
|
||||
reputation, _ := line["reputation"].([]any)
|
||||
if line["action"] != "denied" || len(reputation) != 1 || reputation[0] != listURL {
|
||||
if line["action"] != "client_denied" || len(reputation) != 1 ||
|
||||
reputation[0] != listURL {
|
||||
t.Errorf("request log line %v, want one denied for %s", line, listURL)
|
||||
}
|
||||
}
|
||||
@@ -961,7 +962,7 @@ func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) {
|
||||
metrics := metricsText(t, url+"_sww/metrics", token)
|
||||
for _, series := range []string{
|
||||
`sww_remote_log_lines_sent_total{instance="fsn1app1/blog"} 0`,
|
||||
`sww_remote_log_buffer_depth{instance="fsn1app1/blog"} 1`,
|
||||
`sww_remote_log_lines_waiting{instance="fsn1app1/blog"} 1`,
|
||||
} {
|
||||
if !strings.Contains(metrics, "\n"+series+"\n") {
|
||||
t.Errorf("no %q in the metrics:\n%s", series, metrics)
|
||||
@@ -1067,8 +1068,12 @@ func TestBanIsAlertedAndAnAlertNotSentIsKeptAcrossARestart(t *testing.T) {
|
||||
metrics := metricsWith(t, url+"_sww/metrics", token,
|
||||
"\nsww_alerts_sent_total"+ofWebhook+" 1\n")
|
||||
|
||||
for _, series := range []string{"failed", "suppressed", "dropped"} {
|
||||
zero := "\nsww_alerts_" + series + "_total" + ofWebhook + " 0\n"
|
||||
for _, series := range []string{
|
||||
"sww_alert_send_failures_total",
|
||||
"sww_alerts_suppressed_total",
|
||||
"sww_alerts_dropped_total",
|
||||
} {
|
||||
zero := "\n" + series + ofWebhook + " 0\n"
|
||||
if !strings.Contains(metrics, zero) {
|
||||
t.Errorf("no %q in the metrics:\n%s", zero, metrics)
|
||||
}
|
||||
@@ -1280,7 +1285,7 @@ func wantStartingLine(t *testing.T, line map[string]any, dir string) {
|
||||
"SWW_RATE_LIMIT_PER_HOUR": "1500",
|
||||
rateLimitPerDay: "6000",
|
||||
"SWW_RATE_LIMIT_EXEMPT_PATHS": "",
|
||||
"SWW_DENIED_COUNTRIES": "",
|
||||
"SWW_TAG_DENY": "",
|
||||
"SWW_EXCLUSIVELY_ALLOWED_COUNTRIES": "",
|
||||
"SWW_BAN_RESPONSE": "403",
|
||||
"SWW_LIMIT_BAN_DURATION": "1h",
|
||||
|
||||
+5
-4
@@ -187,15 +187,16 @@ main() {
|
||||
echo "example-app: rproxy passes a request to the app and its answer back"
|
||||
|
||||
refused || fail "a second request in a minute was not refused"
|
||||
wait_for "discriminator logged no ban" logged "$DISCRIMINATOR" '"action":"rate_limited"'
|
||||
wait_for "discriminator logged no ban" \
|
||||
logged "$DISCRIMINATOR" '"action":"client_is_rate_limited"'
|
||||
echo "example-app: a second request in a minute bans the client"
|
||||
|
||||
refused_from 203.0.113.9 /.env || fail "a probe for /.env was not refused"
|
||||
wait_for "discriminator logged no ban for the probe" \
|
||||
logged "$DISCRIMINATOR" '"action":"banned"' '"rule_ids":["env-file"]'
|
||||
logged "$DISCRIMINATOR" '"action":"temporarily_banned"' '"rule_ids":["env-file"]'
|
||||
refused_from 203.0.113.9 / || fail "the client of the probe was let through"
|
||||
wait_for "the client's next request did not make its ban permanent" \
|
||||
logged "$DISCRIMINATOR" '"ban_expires":"permanent"'
|
||||
logged "$DISCRIMINATOR" '"action":"permanently_banned"' '"ban_expires":"permanent"'
|
||||
echo "example-app: a probe for /.env bans the client, its next request for good"
|
||||
|
||||
docker exec "$DISCRIMINATOR" sv stop sww >/dev/null ||
|
||||
@@ -232,7 +233,7 @@ main() {
|
||||
start_app
|
||||
refused || fail "the new discriminator container let the banned client through"
|
||||
wait_for "discriminator logged no request refused under the ban" \
|
||||
logged "$DISCRIMINATOR" '"action":"banned"'
|
||||
logged "$DISCRIMINATOR" '"action":"temporarily_banned"'
|
||||
echo "example-app: a new discriminator container on the same volume keeps the ban"
|
||||
|
||||
docker rm --force "$DISCRIMINATOR" >/dev/null
|
||||
|
||||
Executable
+54
@@ -0,0 +1,54 @@
|
||||
#!/bin/sh
|
||||
# script/publish: build the image on the Docker host and push it to the
|
||||
# registry under four tags: the UTC date (YYYY-MM-DD), the short commit
|
||||
# hash, the branch name and latest. The build is script/docker's, so the
|
||||
# tests and the linter run first. The host and the registry name are
|
||||
# environment variables with defaults, for running it against another.
|
||||
#
|
||||
# The working tree must be clean: the hash tag names a commit, and an
|
||||
# image built from uncommitted changes would carry a hash that does not
|
||||
# describe it.
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
||||
|
||||
: "${DOCKER_HOST:=ssh://root@fsn1app1}"
|
||||
: "${PUBLISH_IMAGE:=git.eeqj.de/sneak/smallwebwaf}"
|
||||
export DOCKER_HOST
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
|
||||
# Own line: a failing command substitution inside an argument does
|
||||
# not trip `set -e`.
|
||||
changes="$(git status --porcelain)"
|
||||
if [ -n "$changes" ]; then
|
||||
echo "script/publish: the working tree is not clean; commit or stash first" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
date="$(date -u +%Y-%m-%d)"
|
||||
hash="$(git rev-parse --short HEAD)"
|
||||
branch="$(git rev-parse --abbrev-ref HEAD)"
|
||||
if [ "$branch" = "HEAD" ]; then
|
||||
echo "script/publish: detached HEAD; check out a branch first" >&2
|
||||
exit 1
|
||||
fi
|
||||
# A tag is [A-Za-z0-9_.-]; a branch name such as feature/x is not.
|
||||
branch="$(printf '%s' "$branch" | tr -c 'A-Za-z0-9_.-' '-')"
|
||||
|
||||
local_tag="$("$SCRIPT_DIR/projectname")"
|
||||
"$SCRIPT_DIR/docker"
|
||||
|
||||
for tag in "$date" "$hash" "$branch" latest; do
|
||||
docker tag "$local_tag" "$PUBLISH_IMAGE:$tag"
|
||||
done
|
||||
for tag in "$date" "$hash" "$branch" latest; do
|
||||
docker push "$PUBLISH_IMAGE:$tag"
|
||||
done
|
||||
|
||||
echo "pushed $PUBLISH_IMAGE with tags: $date $hash $branch latest"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
Reference in new issue
Block a user