Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
92ed102595 | ||
 clawbot
|
aede434f6a | ||
|
clawbot
|
6850cb0e9f | ||
|
clawbot
|
d7c62f2854 | ||
|
clawbot
|
c66277b111 | ||
|
sneak
|
f6435b1a73 | ||
|
clawbot
|
2349b2a081 | ||
|
clawbot
|
42e06bf298 | ||
|
clawbot
|
beb503b233 | ||
|
clawbot
|
2e2f6f7175 |
No files matched your search
+13
-4
@@ -136,10 +136,19 @@ RUN mkdir /etc/sysusers.d \
|
||||
# ca-certificates, nix-bin and runit, from Ubuntu's archive. apt fetches
|
||||
# them over plain HTTP and checks each one against the archive's signed
|
||||
# package lists, so no CA certificates are needed.
|
||||
RUN apt-get update \
|
||||
#
|
||||
# nix-bin depends on xz-utils, whose install links man pages with
|
||||
# update-alternatives. The image's dpkg config drops man pages, so each
|
||||
# link warns. z-man, read after /etc/dpkg/dpkg.cfg.d/excludes, re-includes
|
||||
# the man1 pages for this install alone; the same step deletes them again,
|
||||
# so no man page or package is added.
|
||||
RUN echo 'path-include=/usr/share/man/man1/*' > /etc/dpkg/dpkg.cfg.d/z-man \
|
||||
&& apt-get update \
|
||||
&& DEBIAN_FRONTEND=noninteractive apt-get install --yes --no-install-recommends \
|
||||
ca-certificates nix-bin runit \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
&& rm -rf /var/lib/apt/lists/* /etc/dpkg/dpkg.cfg.d/z-man \
|
||||
&& find /usr/share/man/man1 ! -type d -delete \
|
||||
&& find /etc/alternatives -type l -xtype l -delete
|
||||
|
||||
# Nix run by root expects a group of build users, which nix-bin does not
|
||||
# create; with the setting empty, root's builds run without them.
|
||||
@@ -175,12 +184,12 @@ RUN groupadd --system --gid 65532 sww \
|
||||
--gid sww --no-create-home --shell /usr/sbin/nologin \
|
||||
sww
|
||||
|
||||
# The state files' directory, SWWAF_STATE_DIR by default, where a volume
|
||||
# The state files' directory, SWW_STATE_DIR by default, where a volume
|
||||
# is mounted to keep them across deploys. The run script gives it to the
|
||||
# sww user at each start of discriminator.
|
||||
RUN mkdir /var/lib/sww
|
||||
|
||||
# The default rule file, in SWWAF_RULES_DIR by default, where a Dockerfile
|
||||
# The default rule file, in SWW_RULES_DIR by default, where a Dockerfile
|
||||
# built FROM this image can copy rule files of its own beside it.
|
||||
COPY share/rules.d/00-default.rules /etc/sww/rules.d/00-default.rules
|
||||
|
||||
|
||||
+56
-44
@@ -12,8 +12,8 @@ configured only by environment variables, that provides:
|
||||
- R1: rate limits per minute, per hour and per day
|
||||
- R2: netblocks that bypass rate limiting
|
||||
- R3: alerts on attacks through webhook, Slack or ntfy
|
||||
- R4: alerts on anomalies: requests or bytes per minute or hour, for given IPs
|
||||
or netblocks, crossing a threshold
|
||||
- R4: alerts on anomalies: requests made or bytes transferred per minute or
|
||||
hour, for given IPs or netblocks, crossing a threshold
|
||||
- R5: temporary blocks for abusers, permanent bans for repeat offenders
|
||||
- R6: one or more RBLs or IP reputation APIs
|
||||
- R7: AS number lookup
|
||||
@@ -24,14 +24,19 @@ configured only by environment variables, that provides:
|
||||
|
||||
## Verdict
|
||||
|
||||
Nothing existing is suitable. No surveyed tool meets R8 (scaling limits by AS
|
||||
number or country) or the bytes half of R4 at all, and the tools that come
|
||||
closest on the rest fail R10: they need several containers, YAML or a web UI.
|
||||
The recommendation is to build a small single-binary sidecar; `SPEC.md` in this
|
||||
directory begins that spec. It reuses the mature parts of the surveyed field as
|
||||
libraries and data sources (Coraza and the OWASP Core Rule Set for attack
|
||||
detection, DNSBLs, AbuseIPDB, optionally a CrowdSec decision feed) instead of
|
||||
reimplementing them.
|
||||
Nothing existing is suitable. No surveyed tool meets R8, which asks for limits
|
||||
scaled by AS number or country. None meets R4's alerts on bytes transferred. The
|
||||
tools that come closest on the rest fail R10: they need several containers, YAML
|
||||
or a web UI. The recommendation is to build a small single-binary sidecar;
|
||||
`SPEC.md` in this directory begins that spec.
|
||||
|
||||
The sidecar reuses the mature parts of the surveyed field instead of
|
||||
reimplementing them. It takes these as libraries and data sources:
|
||||
|
||||
- Coraza and the OWASP Core Rule Set, for attack detection
|
||||
- DNSBLs
|
||||
- AbuseIPDB
|
||||
- optionally, a CrowdSec decision feed
|
||||
|
||||
Closest existing options, and why each still falls short:
|
||||
|
||||
@@ -45,13 +50,16 @@ Closest existing options, and why each still falls short:
|
||||
external blacklist URLs, optional CrowdSec), R7 partly (AS number used for
|
||||
blacklist and whitelist decisions), R9 (ModSecurity with the Core Rule
|
||||
Set, or Coraza plugin), env-var settings.
|
||||
- Fails: R8 (AS number and country can only allow or deny, never scale a
|
||||
limit), R4 (no volume or byte threshold alerts in the free edition;
|
||||
reporting is a paid feature), R5 escalation, and R10 in spirit: since 1.6
|
||||
it needs a `bunkerweb` container plus a `bw-scheduler` container and a
|
||||
database, or the all-in-one image that bundles nginx, scheduler, UI and
|
||||
Redis in one container. It is designed to be the front door for many
|
||||
sites, not a per-app sidecar. AGPL-3.0.
|
||||
- Fails:
|
||||
- R8: AS number and country can only allow or deny, never scale a limit.
|
||||
- R4: no alerts on request-count or byte thresholds in the free edition;
|
||||
reporting is a paid feature.
|
||||
- R5: no escalation for repeat offenders.
|
||||
- R10: since 1.6 it is not one plain sidecar. It needs a `bunkerweb`
|
||||
container, a `bw-scheduler` container and a database, or the
|
||||
all-in-one image that bundles nginx, the scheduler, the UI and Redis.
|
||||
- It is designed as the proxy in front of many sites, not as a sidecar for
|
||||
one app. AGPL-3.0.
|
||||
- Unverified: whether several rates (minute, hour, day) can be stacked on
|
||||
the same URL pattern; the documented design is one rate per URL pattern.
|
||||
- CrowdSec is the strongest detection and ban engine, but it is not a proxy.
|
||||
@@ -66,16 +74,20 @@ Closest existing options, and why each still falls short:
|
||||
listed AS numbers is possible, but each is a hand-written YAML scenario,
|
||||
it reacts after the fact by banning, and it is not an inline limiter that
|
||||
answers 429.
|
||||
- Fails: R10 (needs the security engine container with persistent state, log
|
||||
acquisition from traefik, a bouncer such as the traefik plugin, and YAML
|
||||
for acquisition, profiles, scenarios and notifications), bytes half of R4.
|
||||
- Fails:
|
||||
- R10, because it needs:
|
||||
- the security engine container with persistent state
|
||||
- log acquisition from traefik
|
||||
- a bouncer such as the traefik plugin
|
||||
- YAML for acquisition, profiles, scenarios and notifications
|
||||
- R4's alerts on bytes.
|
||||
- CrowdSec plus traefik's own `rateLimit` middleware is the best combination
|
||||
with no new code. It gives inline limiting (one window per middleware, keyed
|
||||
by IP, with `sourceCriterion` exclusions), bans with escalation, alerts and
|
||||
reputation. It still fails R8, the bytes half of R4, R1's three stacked
|
||||
windows only by chaining three middlewares per router, and R10 entirely: the
|
||||
configuration lives in traefik labels and CrowdSec YAML on each host, not in
|
||||
one sidecar's environment.
|
||||
reputation lookups. It still fails R8, R4's alerts on bytes, and R10 entirely:
|
||||
the configuration lives in traefik labels and CrowdSec YAML on each host, not
|
||||
in one sidecar's environment. It meets R1's three windows only by chaining
|
||||
three middlewares per router.
|
||||
|
||||
## Candidate by candidate
|
||||
|
||||
@@ -87,10 +99,10 @@ Closest existing options, and why each still falls short:
|
||||
cached, streamed decision list (recommended), standalone against the
|
||||
central API, or AppSec only. Supports captcha remediation.
|
||||
- Covers: R5, R6, R9, R3 and R7 through the engine (see Verdict).
|
||||
- Misses: the plugin itself does no rate limiting; R8; R4 bytes.
|
||||
- Misses: the plugin itself does no rate limiting; R8; R4's alerts on bytes.
|
||||
- Configuration: traefik static config to load the plugin, dynamic config or
|
||||
labels for the middleware, CrowdSec YAML for everything else.
|
||||
- Sidecar fit: no. It lives inside traefik, plus a separate engine
|
||||
- Sidecar fit: no. It runs inside traefik and needs a separate engine
|
||||
container.
|
||||
- Maturity: widely used (about 900 stars, listed in the traefik plugin
|
||||
catalog, documented by CrowdSec itself), actively maintained. Traefik
|
||||
@@ -115,7 +127,7 @@ Closest existing options, and why each still falls short:
|
||||
- What it is: nginx-derived (Tengine) reverse proxy with a proprietary
|
||||
semantic detection engine and a web console. About 22,000 stars.
|
||||
- Covers: R9 well, basic per-IP rate limiting with block duration, bot
|
||||
challenge, authentication gate.
|
||||
challenge, authentication in front of the app.
|
||||
- Misses: R8, R7, R4; R1 has one window per rule. Country blocking, alert
|
||||
messages, syslog forwarding and the better reputation database are paid
|
||||
(Lite about 10 USD per month, Pro about 100 USD per month); the free
|
||||
@@ -128,7 +140,7 @@ Closest existing options, and why each still falls short:
|
||||
- Coraza (`corazawaf/coraza`) and Coraza-based proxies
|
||||
- What it is: a Go library that implements the ModSecurity rule language and
|
||||
runs the OWASP Core Rule Set. OWASP project, actively maintained, the
|
||||
successor path now that ModSecurity is in maintenance only.
|
||||
successor to ModSecurity, which is now in maintenance only.
|
||||
- Packagings: `coraza-caddy` (Caddy module), `coraza-spoa` (HAProxy),
|
||||
`coraza-proxy-wasm` (Envoy), a traefik WASM plugin, and
|
||||
`coreruleset/coraza-crs-docker` (Caddy plus Coraza plus the Core Rule Set,
|
||||
@@ -142,7 +154,7 @@ Closest existing options, and why each still falls short:
|
||||
- ModSecurity Core Rule Set containers (`owasp/modsecurity-crs`)
|
||||
- What it is: official images of Apache or nginx with ModSecurity and the
|
||||
Core Rule Set, as a reverse proxy configured by env vars (`BACKEND`,
|
||||
paranoia level, anomaly thresholds, many more). Current images carry
|
||||
paranoia level, anomaly thresholds, many more). Current images contain
|
||||
ModSecurity 2.9.x on Apache and Core Rule Set 4.x.
|
||||
- Covers: R9, and R10 (single container, env vars, one backend).
|
||||
- Misses: R1 to R8.
|
||||
@@ -152,8 +164,8 @@ Closest existing options, and why each still falls short:
|
||||
- Anubis (`TecharoHQ/anubis`), 1.27 current
|
||||
- What it is: a single-binary reverse proxy that makes browsers solve a
|
||||
proof-of-work challenge before passing them to `TARGET`. Aimed at
|
||||
scrapers, which is likely a large share of unwanted traffic on a public
|
||||
code forge.
|
||||
scrapers, which are likely a large share of the unwanted traffic on a
|
||||
public code forge.
|
||||
- Covers: R10 well (one container, env vars for listener, target,
|
||||
difficulty, cookies). Policy rules can match on path, user agent, headers,
|
||||
IP ranges, and with the vendor's hosted data service also AS number and
|
||||
@@ -180,13 +192,13 @@ Closest existing options, and why each still falls short:
|
||||
temporary ban, no escalation, state lost on traefik restart) and R2
|
||||
(allowlist). Nothing else.
|
||||
- `juitde/traefik-plugin-fail2ban`, fail2ban connector plugins, GeoBlock
|
||||
plugins (country allow or deny through an external lookup API): same
|
||||
shape, each one small feature, configured in traefik, state in memory.
|
||||
plugins (country allow or deny through an external lookup API): each adds
|
||||
one small feature, is configured in traefik and keeps its state in memory.
|
||||
- Sidecar fit: no; they live inside traefik.
|
||||
- Traefik built-in middlewares
|
||||
- `rateLimit` (one average-and-burst window per middleware, optional Redis
|
||||
in traefik 3.x), `inFlightReq`, `ipAllowList`. No bans, alerts, reputation
|
||||
or AS number awareness.
|
||||
lookups or AS number lookups.
|
||||
- caddy-waf (`fabriziosalmi/caddy-waf`), 0.4.x
|
||||
- What it is: a Caddy module with regex rules and anomaly scoring, per-IP
|
||||
and per-path rate limiting with one configurable window, IP and DNS
|
||||
@@ -198,12 +210,12 @@ Closest existing options, and why each still falls short:
|
||||
- open-appsec (Check Point)
|
||||
- Machine-learning WAF agent attached to nginx, Kong, Envoy or similar, with
|
||||
a declarative policy file or the vendor's cloud console. Covers R9 only;
|
||||
rate limiting and richer features are in paid tiers. Not a sidecar in the
|
||||
required sense.
|
||||
rate limiting and richer features are in paid tiers. It attaches to an
|
||||
existing proxy, so it is not the sidecar R10 asks for.
|
||||
- iocaine
|
||||
- Serves generated garbage pages to clients the fronting proxy classifies as
|
||||
scrapers. Not a limiter, WAF or ban tool; out of scope except as a
|
||||
curiosity for scraper traffic.
|
||||
scrapers. Not a limiter, WAF or ban tool; out of scope, and only of
|
||||
passing interest for scraper traffic.
|
||||
- Pangolin
|
||||
- A tunnelled access platform that bundles traefik and optionally CrowdSec.
|
||||
Replaces the ingress rather than adding a sidecar; out of scope.
|
||||
@@ -214,17 +226,17 @@ Closest existing options, and why each still falls short:
|
||||
Everyone else offers one window per rule or none.
|
||||
- R2 bypass netblocks: BunkerWeb, CrowdSec, traefik, fail2ban plugins.
|
||||
- R3 alerts: CrowdSec (best), BunkerWeb (denied requests only), SafeLine (paid).
|
||||
- R4 volume and byte anomalies: none. CrowdSec can approximate request-count
|
||||
anomalies with a custom scenario; nothing handles bytes.
|
||||
- R5 temporary then permanent: CrowdSec only. BunkerWeb has fixed or permanent,
|
||||
not both by history.
|
||||
- R4 alerts on request counts and bytes: none. CrowdSec can approximate alerts
|
||||
on request counts with a custom scenario; no surveyed tool alerts on bytes.
|
||||
- R5 temporary then permanent: CrowdSec only. BunkerWeb's ban is either one
|
||||
fixed length or permanent, and does not change with a client's history.
|
||||
- R6 reputation: CrowdSec, BunkerWeb, go-away, caddy-waf (static lists).
|
||||
- R7 AS number lookup: CrowdSec (enrichment), BunkerWeb and caddy-waf (allow or
|
||||
deny), Anubis (hosted service).
|
||||
- R8 biased thresholds: none.
|
||||
- R9 attack detection: Coraza or ModSecurity with the Core Rule Set (open,
|
||||
standard), CrowdSec AppSec, SafeLine (closed engine), BunkerWeb (wraps the
|
||||
first).
|
||||
standard), CrowdSec AppSec, SafeLine (closed engine), BunkerWeb (wraps Coraza
|
||||
or ModSecurity).
|
||||
- R10 env-var sidecar: `owasp/modsecurity-crs`, `coraza-crs-docker`, Anubis.
|
||||
None of these covers anything beyond its one job.
|
||||
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
.PHONY: bootstrap setup test lint fmt fmt-check tidy check docker hooks build \
|
||||
run example-app
|
||||
.PHONY: bootstrap setup test lint fmt fmt-check tidy check docker publish \
|
||||
hooks build run example-app
|
||||
|
||||
# Makefile targets are thin shims; the implementations live in script/
|
||||
# per the scripts-to-rule-them-all pattern (see the Entrypoints section
|
||||
# of README.md). tidy writes go.mod and go.sum as `go mod tidy` does,
|
||||
# which test checks. build and run are for working on the code by hand;
|
||||
# example-app checks the image with an app built on it.
|
||||
# example-app checks the image with an app built on it; publish builds
|
||||
# the image on the Docker host and pushes it to the registry.
|
||||
|
||||
bootstrap:
|
||||
@script/bootstrap
|
||||
@@ -34,6 +35,9 @@ check:
|
||||
docker:
|
||||
@script/docker
|
||||
|
||||
publish:
|
||||
@script/publish
|
||||
|
||||
hooks:
|
||||
@script/install-precommit
|
||||
|
||||
|
||||
@@ -7,8 +7,8 @@ requests of scrapers, scanners and other abusers before they reach an app.
|
||||
## What it does
|
||||
|
||||
A web application firewall reads each request on its way to a web app and
|
||||
refuses the ones that look like abuse. `smallwebwaf` is one binary, shipped in
|
||||
one image, with two commands that do this together:
|
||||
refuses the ones that look like abuse. `smallwebwaf` is one program, `sww`,
|
||||
shipped in one image, with two commands that do this together:
|
||||
|
||||
- `sww rproxy` is a small reverse proxy in front of one app. traefik routes to
|
||||
it. It asks `discriminator` about each request, passes the allowed ones to the
|
||||
@@ -25,11 +25,11 @@ without an answer in time, `rproxy` passes the request to the app.
|
||||
It protects against:
|
||||
|
||||
- Scrapers that fetch every page of a site, all day. Each client gets a limit on
|
||||
requests and on bytes per minute, per hour and per day. The defaults leave
|
||||
room for a person reading a site for hours. They catch a scraper that keeps up
|
||||
one request every 2 seconds. A client over a limit is refused and banned: for
|
||||
a while at first, for three times as long at each repeat, and permanently when
|
||||
it keeps at it.
|
||||
requests and on body bytes per minute, per hour and per day. The defaults
|
||||
leave room for a person reading a site for hours. They catch a scraper that
|
||||
keeps up one request every 2 seconds. A client that breaks a limit is banned:
|
||||
for a while at first, for three times as long at each repeat, and permanently
|
||||
when it keeps at it.
|
||||
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
|
||||
default rule file bans such a client at its first probe, and its next request
|
||||
during the ban makes the ban permanent. Your own rules, or a list of trap
|
||||
@@ -68,8 +68,9 @@ Not built, and not planned:
|
||||
|
||||
### The image
|
||||
|
||||
No image is published. Build it from a clone. `make docker` runs the tests and
|
||||
the linter, then builds the image, tagged `smallwebwaf`.
|
||||
The image is published at `git.eeqj.de/sneak/smallwebwaf`. You can also build it
|
||||
from a clone. `make docker` runs the tests and the linter, then builds the
|
||||
image, tagged `smallwebwaf`.
|
||||
|
||||
```sh
|
||||
git clone https://git.eeqj.de/sneak/smallwebwaf.git
|
||||
@@ -171,11 +172,11 @@ does.
|
||||
`https://<site>/_sww/healthz` answers `ok`. `discriminator`'s log lines about
|
||||
the request have `"action":"forward"`.
|
||||
- To see a refusal without banning yourself, run `discriminator` with
|
||||
`SWWAF_MODE=observe` and ask for `/.env`. The request reaches the app.
|
||||
`discriminator`'s log lines have `"would_action":"banned"` and
|
||||
`SWW_MODE=observe` and ask for `/.env`. The request reaches the app.
|
||||
`discriminator`'s log lines have `"would_action":"temporarily_banned"` and
|
||||
`"rule_ids":["env-file"]`. A ban you get in `enforce` mode is lifted as
|
||||
"Clients and bans" below describes.
|
||||
- With `SWWAF_METRICS_TOKEN` set, this answers with the metrics, from a host on
|
||||
- With `SWW_METRICS_TOKEN` set, this answers with the metrics, from a host on
|
||||
`discriminator`'s network:
|
||||
`curl -H "Authorization: Bearer $TOKEN" http://discriminator:8080/_sww/metrics`.
|
||||
- `make example-app` builds the image and an app on it, runs them, and checks
|
||||
@@ -185,16 +186,17 @@ does.
|
||||
|
||||
Every setting is an environment variable, read once at start; changing one means
|
||||
restarting the container. `rproxy` reads only the `SWRP_` settings, and
|
||||
`discriminator` only the `SWWAF_` ones. Every setting but `SWRP_WAF_BACKEND` has
|
||||
a default, chosen for a service on the open internet. A setting that needs an
|
||||
`discriminator` only the `SWW_` ones. Every setting but `SWRP_WAF_BACKEND` has a
|
||||
default, chosen for a service on the open internet. A setting that needs an
|
||||
account, a key, a token or an alert destination stays off until you give one.
|
||||
|
||||
- Any `SWWAF_` setting can be given as a file instead, for secrets and long
|
||||
lists: `SWWAF_ADMIN_TOKEN_FILE` names the file that holds `SWWAF_ADMIN_TOKEN`.
|
||||
The file's contents are the value, less one newline at the end. Setting both
|
||||
forms stops the start. `SWWAF_LOG_REMOTE_TLS_CA_FILE` already names a file, so
|
||||
it has no `_FILE` form, and no `SWRP_` setting has one, since `rproxy` never
|
||||
touches disk.
|
||||
- Any `SWW_` setting can be given as a file instead, for secrets and long lists:
|
||||
`SWW_ADMIN_TOKEN_FILE` names the file that holds `SWW_ADMIN_TOKEN`. The file's
|
||||
contents are the value, less one newline at the end. Setting both forms stops
|
||||
the start. `SWW_LOG_REMOTE_TLS_CA_FILE` and
|
||||
`SWW_CORE_RULE_SET_EXCLUSIONS_FILE` already name a file, so they have no
|
||||
`_FILE` form. `SWW_COMMAND` cannot be given as a file. No `SWRP_` setting has
|
||||
one, since `rproxy` never touches disk.
|
||||
- A duration is Go's syntax, with `d` for days: `90s`, `15m`, `24h`, `7d`.
|
||||
- A size is a number of bytes with an optional `K`, `M` or `G`, in powers of
|
||||
1024: `512K`, `100M`, `5G`.
|
||||
@@ -208,127 +210,133 @@ account, a key, a token or an alert destination stays off until you give one.
|
||||
percentage is a whole number from 0 to 100. A path prefix starts with `/`.
|
||||
- `off` switches a limit, a threshold or a timeout off where the table says a
|
||||
setting takes it.
|
||||
- The byte limits and the byte anomaly thresholds count only body bytes: the
|
||||
app's response bodies, the visitors' request bodies, or both, as
|
||||
`SWW_BYTES_COUNT` says. Headers are never counted. What the visitor sent over
|
||||
a WebSocket counts as request bytes, and what the app sent as response bytes.
|
||||
- An invalid setting stops the process at start with one log line naming the
|
||||
variable and what is wrong with the value. The settings in effect are logged
|
||||
on one line at start, with tokens, keys and webhook paths shown as `********`.
|
||||
|
||||
Every setting the code reads, with its default:
|
||||
|
||||
| Variable | Default | What it does |
|
||||
| ------------------------------------------ | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `discriminator`. |
|
||||
| `SWRP_WAF_BACKEND` | required | `discriminator`'s URL, such as `http://discriminator:8080`. |
|
||||
| `SWRP_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. |
|
||||
| `SWRP_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. |
|
||||
| `SWRP_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. |
|
||||
| `SWRP_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `discriminator`'s answer to a question and to a report, in milliseconds. |
|
||||
| `SWRP_DEFAULT_DISPOSITION` | `allow` | What `rproxy` does with a request when `discriminator` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. |
|
||||
| `SWRP_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. |
|
||||
| `SWRP_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`; a larger one is answered `502`, or cut off once it has started. |
|
||||
| `SWRP_MAX_REQUEST_TIME_MS` | `60000` | How long the visitor may take to send its request line and headers, and then the whole request to reach the app, in milliseconds or `off`. |
|
||||
| `SWRP_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
|
||||
| `SWWAF_LISTEN_ADDR` | `:8080` | Where `discriminator` listens; leave the host part empty. |
|
||||
| `SWWAF_INSTANCE_NAME` | the container's host name | The name in every line `discriminator` logs, and in every alert and metric, such as `fsn1/discriminator`. |
|
||||
| `SWWAF_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
|
||||
| `SWWAF_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. |
|
||||
| `SWWAF_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. |
|
||||
| `SWWAF_MAX_BANS` | `5000` | The most bans `discriminator` keeps of its own; bans an admin made do not count. |
|
||||
| `SWWAF_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. |
|
||||
| `SWWAF_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients the request and byte limits neither count nor refuse. |
|
||||
| `SWWAF_DENY_NETS` | empty | Netblocks whose clients are refused. |
|
||||
| `SWWAF_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; one more bans it. |
|
||||
| `SWWAF_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; one more bans it. |
|
||||
| `SWWAF_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; one more bans it. |
|
||||
| `SWWAF_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes whose requests the request and byte limits neither count nor refuse, such as `/assets/`. |
|
||||
| `SWWAF_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes a client's requests may carry in a minute, a size or `off`; the request that takes a client past any byte limit bans it. |
|
||||
| `SWWAF_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes a client's requests may carry in an hour, a size or `off`. |
|
||||
| `SWWAF_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes a client's requests may carry in a day, a size or `off`. |
|
||||
| `SWWAF_BYTES_COUNT` | `both` | Which body bytes the byte limits count: `response`, `request` or `both`. |
|
||||
| `SWWAF_ERROR_BURST_THRESHOLD` | `30` | The most refusals by a rule file, a trap path or the Core Rule Set a client may get in a minute, a count or `off`; one more bans it. |
|
||||
| `SWWAF_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. |
|
||||
| `SWWAF_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. |
|
||||
| `SWWAF_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A limit broken again within this time after the last such ban ended gets a ban three times as long. |
|
||||
| `SWWAF_MAX_BAN_DURATION` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. |
|
||||
| `SWWAF_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. |
|
||||
| `SWWAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
|
||||
| `SWWAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs`, a web service, `file`, or `off`. |
|
||||
| `SWWAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
||||
| `SWWAF_LOOKUP_TIMEOUT` | `800ms` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take. |
|
||||
| `SWWAF_ADD_LOOKUP_HEADERS` | `false` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
||||
| `SWWAF_DENIED_COUNTRIES` | empty | Countries whose clients are refused, such as `cn,ru`. |
|
||||
| `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
||||
| `SWWAF_ASN_LIMIT_PERCENT` | empty | AS numbers whose clients get a percentage of every request and byte limit, such as `AS14061:50,AS16276:25`. |
|
||||
| `SWWAF_COUNTRY_LIMIT_PERCENT` | empty | The same by country, such as `cn:25`. |
|
||||
| `SWWAF_ASN_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `SWWAF_ASN_LIMIT_PERCENT`. |
|
||||
| `SWWAF_COUNTRY_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `SWWAF_COUNTRY_LIMIT_PERCENT`. |
|
||||
| `SWWAF_UNKNOWN_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. |
|
||||
| `SWWAF_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; where it and `SWWAF_ASN_LIMIT_PERCENT` both give a percentage, the lower applies. |
|
||||
| `SWWAF_RULES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. |
|
||||
| `SWWAF_RULES_ENABLED` | `true` | `false` reads no rule file. |
|
||||
| `SWWAF_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. |
|
||||
| `SWWAF_WAF_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. |
|
||||
| `SWWAF_WAF_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. |
|
||||
| `SWWAF_WAF_ANOMALY_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. |
|
||||
| `SWWAF_WAF_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. |
|
||||
| `SWWAF_WAF_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
|
||||
| `SWWAF_WAF_BODY_LIMIT` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. |
|
||||
| `SWWAF_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
|
||||
| `SWWAF_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. |
|
||||
| `SWWAF_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. |
|
||||
| `SWWAF_DNSBL_ZONES` | empty | DNS blocklists asked about each new client in the background, such as `dnsbl.dronebl.org`. |
|
||||
| `SWWAF_DNSBL_RESOLVER` | unset | The resolver the zones are asked through, an IP address with an optional port; unset, the host's resolver is used. |
|
||||
| `SWWAF_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account; set, a client that has committed an offence is checked in the background. |
|
||||
| `SWWAF_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. |
|
||||
| `SWWAF_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. |
|
||||
| `SWWAF_CROWDSEC_LAPI_URL` | unset | The local API of a CrowdSec engine you run, such as `http://172.17.0.1:8080`; its decision list is fetched every minute, and a listed client is banned. |
|
||||
| `SWWAF_CROWDSEC_LAPI_KEY` | unset | The key the engine gave for `smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it; set with the URL, never alone. |
|
||||
| `SWWAF_REPUTATION_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:<percent>` or `log`. |
|
||||
| `SWWAF_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. |
|
||||
| `SWWAF_REPUTATION_TIMEOUT` | `2s` | How long a DNS blocklist query or an AbuseIPDB check may take. |
|
||||
| `SWWAF_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. |
|
||||
| `SWWAF_STATE_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. |
|
||||
| `SWWAF_STATE_COUNTER_INTERVAL` | `15m` | How often every state file is written. |
|
||||
| `SWWAF_LOG_LEVEL` | `info` | The least severe of `discriminator`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. |
|
||||
| `SWWAF_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. |
|
||||
| `SWWAF_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `discriminator` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. |
|
||||
| `SWWAF_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. |
|
||||
| `SWWAF_LOG_REMOTE_BUFFER` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. |
|
||||
| `SWWAF_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. |
|
||||
| `SWWAF_LOG_REMOTE_APP_NAME` | `SWWAF_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. |
|
||||
| `SWWAF_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. |
|
||||
| `SWWAF_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. |
|
||||
| `SWWAF_METRICS_TOP_N` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. |
|
||||
| `SWWAF_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. |
|
||||
| `SWWAF_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer <token>`. |
|
||||
| `SWWAF_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. |
|
||||
| `SWWAF_ALERT_NTFY_URL` | unset | An ntfy topic each alert is published to, as its full URL, such as `https://ntfy.sh/my-alerts`. |
|
||||
| `SWWAF_ALERT_NTFY_TOKEN` | unset | An ntfy access token sent with each alert. |
|
||||
| `SWWAF_ALERT_EVENTS` | `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. |
|
||||
| `SWWAF_ALERT_COOLDOWN` | `15m` | How long after an alert its repeats are held back and counted, a duration or `off`. |
|
||||
| `SWWAF_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. |
|
||||
| `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Alert when one client makes more requests than this in a minute, a count. |
|
||||
| `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Alert when one client makes more requests than this in an hour, a count. |
|
||||
| `SWWAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Alert when one client's requests carry more bytes than this in a minute, a size. |
|
||||
| `SWWAF_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Alert when one client's requests carry more bytes than this in an hour, a size. |
|
||||
| `SWWAF_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Alert when the netblock around a client makes more requests than this in a minute, a count. |
|
||||
| `SWWAF_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Alert when the netblock around a client makes more requests than this in an hour, a count. |
|
||||
| `SWWAF_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Alert when the netblock around a client carries more bytes than this in a minute, a size. |
|
||||
| `SWWAF_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Alert when the netblock around a client carries more bytes than this in an hour, a size. |
|
||||
| `SWWAF_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWWAF_ANOMALY_NET_` thresholds count, from 0 to 32. |
|
||||
| `SWWAF_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWWAF_ANOMALY_NET_` thresholds count, from 0 to 128. |
|
||||
| `SWWAF_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Alert when one AS number makes more requests than this in a minute, a count. |
|
||||
| `SWWAF_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Alert when one AS number makes more requests than this in an hour, a count. |
|
||||
| `SWWAF_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Alert when one AS number carries more bytes than this in a minute, a size. |
|
||||
| `SWWAF_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Alert when one AS number carries more bytes than this in an hour, a size. |
|
||||
| `SWWAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Alert when the whole service gets more requests than this in a minute, a count. |
|
||||
| `SWWAF_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Alert when the whole service gets more requests than this in an hour, a count. |
|
||||
| `SWWAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Alert when the whole service carries more bytes than this in a minute, a size. |
|
||||
| `SWWAF_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Alert when the whole service carries more bytes than this in an hour, a size. |
|
||||
| `SWWAF_WATCH_NETS` | empty | Named netblocks the `SWWAF_WATCH_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. |
|
||||
| `SWWAF_WATCH_REQUESTS_PER_MINUTE` | `off` | Alert when a watched netblock makes more requests than this in a minute, a count. |
|
||||
| `SWWAF_WATCH_REQUESTS_PER_HOUR` | `off` | Alert when a watched netblock makes more requests than this in an hour, a count. |
|
||||
| `SWWAF_WATCH_BYTES_PER_MINUTE` | `off` | Alert when a watched netblock carries more bytes than this in a minute, a size. |
|
||||
| `SWWAF_WATCH_BYTES_PER_HOUR` | `off` | Alert when a watched netblock carries more bytes than this in an hour, a size. |
|
||||
| Variable | Default | What it does |
|
||||
| ---------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `discriminator`. |
|
||||
| `SWRP_WAF_BACKEND` | required | `discriminator`'s URL, such as `http://discriminator:8080`. |
|
||||
| `SWRP_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. |
|
||||
| `SWRP_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. |
|
||||
| `SWRP_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. |
|
||||
| `SWRP_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `discriminator`'s answer to a question and to a report, in milliseconds. |
|
||||
| `SWRP_DEFAULT_DISPOSITION` | `allow` | What `rproxy` does with a request when `discriminator` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. |
|
||||
| `SWRP_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. |
|
||||
| `SWRP_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
|
||||
| `SWRP_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
|
||||
| `SWRP_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
|
||||
| `SWW_LISTEN_ADDR` | `:8080` | Where `discriminator` listens; leave the host part empty. |
|
||||
| `SWW_INSTANCE_NAME` | the container's host name | The name in every line `discriminator` logs, and in every alert and metric, such as `fsn1/discriminator`. |
|
||||
| `SWW_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
|
||||
| `SWW_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. |
|
||||
| `SWW_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. |
|
||||
| `SWW_MAX_BANS` | `5000` | The most bans `discriminator` keeps of its own; bans an admin made do not count. |
|
||||
| `SWW_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. |
|
||||
| `SWW_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients are exempt from rate limiting and the byte limits. |
|
||||
| `SWW_DENY_NETS` | empty | Netblocks whose clients are refused. |
|
||||
| `SWW_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; the next request is refused and bans the client. |
|
||||
| `SWW_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; the next request is refused and bans the client. |
|
||||
| `SWW_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; the next request is refused and bans the client. |
|
||||
| `SWW_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. |
|
||||
| `SWW_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes counted for a client in a minute, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||
| `SWW_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes counted for a client in an hour, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||
| `SWW_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes counted for a client in a day, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||
| `SWW_BYTES_COUNT` | `both` | What the byte limits and the byte anomaly thresholds count: `response`, the app's response bodies; `request`, the visitors' request bodies; or `both`. What the visitor sent over a WebSocket counts as request bytes, and what the app sent as response bytes. |
|
||||
| `SWW_ERROR_BURST_THRESHOLD` | `30` | The most refusals a client may get in a minute from a rule file, a trap path or the Core Rule Set, a count or `off`. The next one bans the client. |
|
||||
| `SWW_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. |
|
||||
| `SWW_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. |
|
||||
| `SWW_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A client that breaks a limit again within this time after its last such ban ended gets a ban three times as long. |
|
||||
| `SWW_MAX_BAN_DURATION` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. |
|
||||
| `SWW_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. |
|
||||
| `SWW_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
|
||||
| `SWW_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
|
||||
| `SWW_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
||||
| `SWW_LOOKUP_TIMEOUT` | `800ms` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take. |
|
||||
| `SWW_ADD_LOOKUP_HEADERS` | `false` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
||||
| `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
||||
| `SWW_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
||||
| `SWW_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. |
|
||||
| `SWW_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. |
|
||||
| `SWW_ROUTEWATCH_URL` | unset | The URL of a routewatch instance, which gives each client's `asn=` and `prefix=` tags. Unset, nothing asks routewatch. |
|
||||
| `SWW_ASN_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. |
|
||||
| `SWW_COUNTRY_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. |
|
||||
| `SWW_UNKNOWN_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. |
|
||||
| `SWW_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_TAG_LIMIT_PERCENT` (the lowest wins). |
|
||||
| `SWW_RULES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. |
|
||||
| `SWW_RULES_ENABLED` | `true` | `false` reads no rule file. |
|
||||
| `SWW_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. |
|
||||
| `SWW_WAF_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. |
|
||||
| `SWW_WAF_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. |
|
||||
| `SWW_WAF_ANOMALY_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. |
|
||||
| `SWW_WAF_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. |
|
||||
| `SWW_WAF_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
|
||||
| `SWW_WAF_BODY_LIMIT` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. |
|
||||
| `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
|
||||
| `SWW_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
|
||||
| `SWW_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. |
|
||||
| `SWW_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. |
|
||||
| `SWW_DNSBL_ZONES` | empty | DNS blocklists asked about each new client in the background, such as `dnsbl.dronebl.org`. |
|
||||
| `SWW_DNSBL_RESOLVER` | unset | The resolver the zones are asked through, an IP address with an optional port; unset, the host's resolver is used. |
|
||||
| `SWW_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account; set, a client that has committed an offence is checked in the background. |
|
||||
| `SWW_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. |
|
||||
| `SWW_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. |
|
||||
| `SWW_CROWDSEC_LAPI_URL` | unset | The local API of a CrowdSec engine you run, such as `http://172.17.0.1:8080`; its decision list is fetched every minute, and a listed client is banned. |
|
||||
| `SWW_CROWDSEC_LAPI_KEY` | unset | The key the engine gave for `smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it; set with the URL, never alone. |
|
||||
| `SWW_REPUTATION_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:<percent>` or `log`. |
|
||||
| `SWW_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. |
|
||||
| `SWW_REPUTATION_TIMEOUT` | `2s` | How long a DNS blocklist query or an AbuseIPDB check may take. |
|
||||
| `SWW_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. |
|
||||
| `SWW_STATE_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. |
|
||||
| `SWW_STATE_COUNTER_INTERVAL` | `15m` | How often every state file is written. |
|
||||
| `SWW_LOG_LEVEL` | `info` | The least severe of `discriminator`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. |
|
||||
| `SWW_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. |
|
||||
| `SWW_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `discriminator` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. |
|
||||
| `SWW_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. |
|
||||
| `SWW_LOG_REMOTE_BUFFER` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. |
|
||||
| `SWW_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. |
|
||||
| `SWW_LOG_REMOTE_APP_NAME` | `SWW_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. |
|
||||
| `SWW_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. |
|
||||
| `SWW_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. |
|
||||
| `SWW_METRICS_TOP_N` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. |
|
||||
| `SWW_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. |
|
||||
| `SWW_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer <token>`. |
|
||||
| `SWW_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. |
|
||||
| `SWW_ALERT_NTFY_URL` | unset | An ntfy topic each alert is published to, as its full URL, such as `https://ntfy.sh/my-alerts`. |
|
||||
| `SWW_ALERT_NTFY_TOKEN` | unset | An ntfy access token sent with each alert. |
|
||||
| `SWW_ALERT_EVENTS` | `ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. |
|
||||
| `SWW_ALERT_COOLDOWN` | `15m` | How long after an alert its repeats are held back and counted, a duration or `off`. |
|
||||
| `SWW_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. |
|
||||
| `SWW_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Alert when one client makes more requests than this in a minute, a count. |
|
||||
| `SWW_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Alert when one client makes more requests than this in an hour, a count. |
|
||||
| `SWW_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one client in a minute, a size. |
|
||||
| `SWW_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one client in an hour, a size. |
|
||||
| `SWW_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Alert when the netblock around a client makes more requests than this in a minute, a count. |
|
||||
| `SWW_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Alert when the netblock around a client makes more requests than this in an hour, a count. |
|
||||
| `SWW_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the netblock around a client in a minute, a size. |
|
||||
| `SWW_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the netblock around a client in an hour, a size. |
|
||||
| `SWW_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_ANOMALY_NET_` thresholds count, from 0 to 32. |
|
||||
| `SWW_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_ANOMALY_NET_` thresholds count, from 0 to 128. |
|
||||
| `SWW_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Alert when one AS number makes more requests than this in a minute, a count. |
|
||||
| `SWW_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Alert when one AS number makes more requests than this in an hour, a count. |
|
||||
| `SWW_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one AS number in a minute, a size. |
|
||||
| `SWW_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one AS number in an hour, a size. |
|
||||
| `SWW_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Alert when the whole service gets more requests than this in a minute, a count. |
|
||||
| `SWW_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Alert when the whole service gets more requests than this in an hour, a count. |
|
||||
| `SWW_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the whole service in a minute, a size. |
|
||||
| `SWW_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the whole service in an hour, a size. |
|
||||
| `SWW_WATCH_NETS` | empty | Named netblocks the `SWW_WATCH_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. |
|
||||
| `SWW_WATCH_REQUESTS_PER_MINUTE` | `off` | Alert when a watched netblock makes more requests than this in a minute, a count. |
|
||||
| `SWW_WATCH_REQUESTS_PER_HOUR` | `off` | Alert when a watched netblock makes more requests than this in an hour, a count. |
|
||||
| `SWW_WATCH_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for a watched netblock in a minute, a size. |
|
||||
| `SWW_WATCH_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for a watched netblock in an hour, a size. |
|
||||
|
||||
[`docs/settings.md`](docs/settings.md) gives the values each setting takes in
|
||||
full, and the limits that are fixed rather than settings.
|
||||
@@ -401,23 +409,22 @@ services:
|
||||
environment:
|
||||
SWW_COMMAND: discriminator
|
||||
# Monitoring skips every check.
|
||||
SWWAF_ALLOW_NETS: 192.0.2.10
|
||||
SWW_ALLOW_NETS: 192.0.2.10
|
||||
# The office and the automation host are never rate limited.
|
||||
SWWAF_RATE_LIMIT_EXEMPT_NETS: 203.0.113.0/24,198.51.100.7
|
||||
SWW_RATE_LIMIT_EXEMPT_NETS: 203.0.113.0/24,198.51.100.7
|
||||
# A netblock that sends nothing wanted.
|
||||
SWWAF_DENY_NETS: 198.51.100.128/25
|
||||
SWW_DENY_NETS: 198.51.100.128/25
|
||||
# Tighter limits for everyone else: a small static site.
|
||||
SWWAF_RATE_LIMIT_PER_MINUTE: "60"
|
||||
SWWAF_RATE_LIMIT_PER_HOUR: "600"
|
||||
SWWAF_RATE_LIMIT_PER_DAY: "3000"
|
||||
SWW_RATE_LIMIT_PER_MINUTE: "60"
|
||||
SWW_RATE_LIMIT_PER_HOUR: "600"
|
||||
SWW_RATE_LIMIT_PER_DAY: "3000"
|
||||
# Hosting networks and one country get a share of those limits.
|
||||
SWWAF_ASN_LIMIT_PERCENT: AS14061:50,AS16276:25
|
||||
SWWAF_COUNTRY_LIMIT_PERCENT: cn:25
|
||||
SWW_TAG_LIMIT_PERCENT: asn=14061:50,asn=16276:25,country=CN:25
|
||||
# Paths only scanners ask for; one request bans the client.
|
||||
SWWAF_TRAP_PATHS: /wp-login.php,/xmlrpc.php
|
||||
SWW_TRAP_PATHS: /wp-login.php,/xmlrpc.php
|
||||
# Answer a refused client with 429 rather than 403.
|
||||
SWWAF_BAN_RESPONSE: "429"
|
||||
SWWAF_LIMIT_BAN_DURATION: 30m
|
||||
SWW_BAN_RESPONSE: "429"
|
||||
SWW_LIMIT_BAN_DURATION: 30m
|
||||
volumes:
|
||||
- discriminator-state:/var/lib/sww
|
||||
|
||||
@@ -453,16 +460,16 @@ services:
|
||||
image: smallwebwaf
|
||||
environment:
|
||||
SWW_COMMAND: discriminator
|
||||
SWWAF_INSTANCE_NAME: fsn1/discriminator
|
||||
SWW_INSTANCE_NAME: fsn1/discriminator
|
||||
# Refuse nothing yet; log what enforce mode would have done.
|
||||
SWWAF_MODE: observe
|
||||
SWW_MODE: observe
|
||||
# Look addresses up in a local IPinfo Lite database, never GeoJS.
|
||||
SWWAF_LOOKUP_SOURCE: file
|
||||
SWWAF_LOOKUP_DB_PATH: /var/lib/ipinfo/ipinfo_lite.mmdb
|
||||
SWW_LOOKUP_SOURCE: file
|
||||
SWW_LOOKUP_DB_PATH: /var/lib/ipinfo/ipinfo_lite.mmdb
|
||||
# Alerts to an ntfy topic, with its token read from a file.
|
||||
SWWAF_ALERT_NTFY_URL: https://ntfy.example/smallwebwaf
|
||||
SWWAF_ALERT_NTFY_TOKEN_FILE: /etc/sww/tokens/ntfy
|
||||
SWWAF_ALERT_EVENTS: ban,permanent_ban,waf_block,source_failure,file_error
|
||||
SWW_ALERT_NTFY_URL: https://ntfy.example/smallwebwaf
|
||||
SWW_ALERT_NTFY_TOKEN_FILE: /etc/sww/tokens/ntfy
|
||||
SWW_ALERT_EVENTS: ban,permanent_ban,core_rule_set_match,source_failure,file_error
|
||||
volumes:
|
||||
- discriminator-state:/var/lib/sww
|
||||
- /srv/ipinfo:/var/lib/ipinfo:ro
|
||||
@@ -504,41 +511,45 @@ of the request reaches the app.
|
||||
|
||||
`discriminator` takes the checks:
|
||||
|
||||
3. The static lists, `SWWAF_ALLOW_NETS` and `SWWAF_DENY_NETS`.
|
||||
4. The bans.
|
||||
5. The lookup of the client's AS number and country.
|
||||
6. The country lists.
|
||||
7. The reputation sources.
|
||||
8. The client's limit percentage, from its AS number, its country or a
|
||||
reputation hit.
|
||||
9. The rate limits.
|
||||
10. The trap paths.
|
||||
11. The rule files.
|
||||
12. The header limits, and then the Core Rule Set.
|
||||
13. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||
3. The tag lists.
|
||||
4. The static lists, `SWW_ALLOW_NETS` and `SWW_DENY_NETS`.
|
||||
5. The bans.
|
||||
6. The lookup of the client's AS number and country.
|
||||
7. The country list.
|
||||
8. The reputation sources.
|
||||
9. The tag refusals of `SWW_TAG_DENY`.
|
||||
10. The client's limit percentage, from its tags, its AS number, its country or
|
||||
a reputation hit.
|
||||
11. The rate limits.
|
||||
12. The trap paths.
|
||||
13. The rule files.
|
||||
14. The header limits, and then the Core Rule Set.
|
||||
15. The tag refusals again, so that `exploitsignature` can refuse too.
|
||||
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||
prefix.
|
||||
|
||||
`rproxy` takes the rest, with `discriminator` counting at the end:
|
||||
|
||||
14. The request is passed to the app, streaming.
|
||||
15. After the answer, `rproxy` sends `discriminator` a report with the status
|
||||
and the bytes each way. For an allowed request, `discriminator` counts the
|
||||
bytes toward the byte limits and adds the request to the client's history.
|
||||
17. The request is passed to the app, streaming.
|
||||
18. After the answer, `rproxy` sends `discriminator` a report with the status
|
||||
and the body bytes each way. For an allowed request, `discriminator` counts
|
||||
the bytes `SWW_BYTES_COUNT` names toward the byte limits, bans the client if
|
||||
they take it past one, and adds the request to the client's history.
|
||||
|
||||
A refused visitor is answered with `SWWAF_BAN_RESPONSE`. A `block` rule or a
|
||||
Core Rule Set match answers `403` and bans nobody, since the Core Rule Set has
|
||||
false positives; such refusals count toward `SWWAF_ERROR_BURST_THRESHOLD`
|
||||
instead. A request over the header limits of step 12 is answered `431`, and bans
|
||||
nobody either. `rproxy` itself answers `405` to a method it does not pass on,
|
||||
and the status for a size or time limit, such as `413` for a body too large.
|
||||
A refused visitor is answered with `SWW_BAN_RESPONSE`. A `block` rule or a Core
|
||||
Rule Set match answers `403` and bans nobody, since the Core Rule Set has false
|
||||
positives; such refusals count toward `SWW_ERROR_BURST_THRESHOLD` instead. A
|
||||
request over the header limits of step 14 is answered `431`, and bans nobody
|
||||
either. `rproxy` itself answers `405` to a method it does not pass on, and the
|
||||
status for a size or time limit, such as `413` for a body too large.
|
||||
|
||||
### Clients and bans
|
||||
|
||||
- A client is one IPv4 address or one IPv6 netblock of
|
||||
`SWWAF_IPV6_GROUP_PREFIX`, since one abuser usually controls a whole /64.
|
||||
- A client is one IPv4 address or one IPv6 netblock of `SWW_IPV6_GROUP_PREFIX`,
|
||||
since one abuser usually controls a whole /64.
|
||||
- Each limit is counted over a sliding window with a few integers per client.
|
||||
The table of clients and the table of bans each have a bound
|
||||
(`SWWAF_MAX_TRACKED_CLIENTS`, `SWWAF_MAX_BANS`), so memory is bounded too.
|
||||
(`SWW_MAX_TRACKED_CLIENTS`, `SWW_MAX_BANS`), so memory is bounded too.
|
||||
- A ban refuses every request from its netblock until it ends. Requests refused
|
||||
under a ban count toward no limit. A permanent ban ends only when an admin
|
||||
lifts it.
|
||||
@@ -547,16 +558,36 @@ and the status for a size or time limit, such as `413` for a body too large.
|
||||
bans. An admin adds, keeps or lifts a ban by editing `bans.json` or through
|
||||
the ban endpoints (see [`docs/admin-endpoints.md`](docs/admin-endpoints.md)).
|
||||
|
||||
### Request tags
|
||||
|
||||
Each request gets tags from what `discriminator` knows of its client, written in
|
||||
the `tags` field of its log line:
|
||||
|
||||
- `asn=<number>` and `country=<code>`, from the lookup;
|
||||
- one tag per list of `SWW_TAG_LISTS` that holds the client, such as `torexit`
|
||||
or `vpn=mullvad`;
|
||||
- `crowdsec`, `dnsbl=<zone>` and `abuseipdb`, when that reputation source lists
|
||||
the client;
|
||||
- `exploitsignature`, when a rule file or the Core Rule Set matched the request;
|
||||
- `asn=<number>` and `prefix=<netblock>`, from a routewatch instance, while
|
||||
`SWW_ROUTEWATCH_URL` is set.
|
||||
|
||||
`SWW_TAG_DENY` refuses requests by tag, and `SWW_TAG_LIMIT_PERCENT` lowers a
|
||||
client's limits by tag. See "Request tags" in [`SPEC.md`](SPEC.md).
|
||||
|
||||
### Attack detection
|
||||
|
||||
- The Core Rule Set reads the method, the URL with its query string, and the
|
||||
headers of every request, and no body unless `SWWAF_WAF_BODY_LIMIT` is set. In
|
||||
headers of every request, and no body unless `SWW_WAF_BODY_LIMIT` is set. In
|
||||
front of a code forge, bodies carry what people write and publish, which the
|
||||
rules take for attacks.
|
||||
- `smallwebwaf` changes the Core Rule Set in six ways so that it lets ordinary
|
||||
- `smallwebwaf` changes the Core Rule Set in eight ways so that it lets ordinary
|
||||
requests to a code forge through, such as git over HTTP and sign-in through
|
||||
Git Credential Manager. No setting undoes them; "Attack detection" in
|
||||
[`SPEC.md`](SPEC.md) lists them.
|
||||
Git Credential Manager. On gitea's own paths, its searches, file names and
|
||||
branch names get through too. No setting undoes the changes; "Attack
|
||||
detection" in [`SPEC.md`](SPEC.md) lists them.
|
||||
- `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` switches single rules off for an app's own
|
||||
fields or paths.
|
||||
- Rule files are plain text, one rule to a line, edited by hand and read again
|
||||
while `discriminator` runs. The image ships one file of probes no real visitor
|
||||
sends (see [`docs/rules.md`](docs/rules.md)).
|
||||
@@ -564,7 +595,7 @@ and the status for a size or time limit, such as `413` for a body too large.
|
||||
### State files
|
||||
|
||||
`discriminator` keeps everything in memory, and the five JSON files in
|
||||
`SWWAF_STATE_DIR` hold a copy: `bans.json`, `clients.json`, `lookups.json`,
|
||||
`SWW_STATE_DIR` hold a copy: `bans.json`, `clients.json`, `lookups.json`,
|
||||
`reputation.json` and `alerts.json`. Each is written whole, to a temporary file
|
||||
renamed over the real one, so a crash leaves a complete file. `discriminator`
|
||||
notices a file edited on disk and takes the edit in; an edit that does not parse
|
||||
@@ -577,10 +608,10 @@ is set aside as `<name>.bad`, and never stops the running process (see
|
||||
come from Ubuntu's archive, and nixpkgs is fixed at one release. An app's
|
||||
`Dockerfile` installs a package with `nix-env -iA nixpkgs.<name>` and gets the
|
||||
same package each time it is built (see [`docs/image.md`](docs/image.md)).
|
||||
- runit, a service supervisor, runs `smallwebwaf` as the `sww` user (uid and
|
||||
gid 65532) and starts it again when it exits. The `run` script of
|
||||
`discriminator` gives the state directory to the `sww` user at each start, so
|
||||
a volume needs no change of owner.
|
||||
- runit, a service supervisor, runs `sww` as the `sww` user (uid and gid 65532)
|
||||
and starts it again when it exits. The `run` script of `discriminator` gives
|
||||
the state directory to the `sww` user at each start, so a volume needs no
|
||||
change of owner.
|
||||
- `rproxy` and `discriminator` use no token between them. `discriminator`'s own
|
||||
endpoints, under `/_sww/`, are for your admin tools and your metrics
|
||||
collector, at its own address on the private network; nothing routes the
|
||||
@@ -601,9 +632,10 @@ is set aside as `<name>.bad`, and never stops the running process (see
|
||||
- `internal/rproxy`: the reverse proxy in front of one app.
|
||||
- `internal/discriminator`: `discriminator`'s server, which runs the checks and
|
||||
answers its own endpoints.
|
||||
- `internal/ratelimit`, `internal/bans`, `internal/rules`, `internal/waf`,
|
||||
`internal/lookup`, `internal/reputation`, `internal/anomaly` and
|
||||
`internal/alerts`: one check or source each.
|
||||
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
|
||||
`internal/coreruleset`, `internal/lookup`, `internal/reputation`,
|
||||
`internal/routewatch`, `internal/anomaly` and `internal/alerts`: one check or
|
||||
source each.
|
||||
- `internal/state`, `internal/requestlog`, `internal/remotelog` and
|
||||
`internal/metrics`: the state files, the log lines, the syslog sender and the
|
||||
metrics.
|
||||
@@ -660,6 +692,15 @@ on the host.
|
||||
`make check`.
|
||||
- `script/docker`: builds the image, whose build runs the tests and the linter
|
||||
first. `make docker`.
|
||||
- `script/publish`: runs `script/docker`, whose build happens on the Docker host
|
||||
named by `DOCKER_HOST` (`ssh://root@fsn1app1` unless set). It then pushes the
|
||||
image to `PUBLISH_IMAGE` (`git.eeqj.de/sneak/smallwebwaf` unless set). It
|
||||
refuses a working tree with uncommitted changes. `make publish`. It pushes
|
||||
four tags:
|
||||
- the UTC date as `YYYY-MM-DD`;
|
||||
- the short commit hash;
|
||||
- the branch name;
|
||||
- `latest`.
|
||||
- `script/cibuild`: what CI runs: `script/bootstrap`, `script/check`, then the
|
||||
image build.
|
||||
- `script/precommit`: run by the git pre-commit hook; runs `script/check`.
|
||||
|
||||
+5
-5
@@ -1,8 +1,8 @@
|
||||
// Command sww is a web application firewall. `sww rproxy` is
|
||||
// the reverse proxy in front of one app, which asks discriminator about each request.
|
||||
// `sww discriminator` decides, for many sites, and keeps what lasts.
|
||||
// `sww healthcheck` is the container's health check. Everything it does
|
||||
// is in internal/sww.
|
||||
// Command sww is a web application firewall. `sww rproxy` is the reverse
|
||||
// proxy in front of one app, which asks discriminator about each request.
|
||||
// `sww discriminator` decides, for many sites, and keeps what lasts. `sww
|
||||
// healthcheck` is the container's health check. Everything it does is in
|
||||
// internal/sww.
|
||||
package main
|
||||
|
||||
import (
|
||||
|
||||
@@ -6,8 +6,8 @@
|
||||
# A real app names the smallwebwaf image by digest. This one takes the
|
||||
# image script/example-app has just built, or else the one `make docker`
|
||||
# builds.
|
||||
ARG SMALLWEBWAF_IMAGE=smallwebwaf
|
||||
FROM ${SMALLWEBWAF_IMAGE}
|
||||
ARG SWW_IMAGE=smallwebwaf
|
||||
FROM ${SWW_IMAGE}
|
||||
|
||||
# Packages the app needs, from the nixpkgs in the image.
|
||||
RUN nix-env -iA nixpkgs.busybox
|
||||
|
||||
@@ -5,14 +5,14 @@ The endpoints under `/_sww/`, what each takes and what each answers.
|
||||
"Admin endpoints" in [`SPEC.md`](../SPEC.md) says how their tokens work.
|
||||
|
||||
- `GET /_sww/healthz`: `200` and `ok`, to anyone.
|
||||
- `GET /_sww/metrics`: the metrics, for `SWWAF_METRICS_TOKEN` (see
|
||||
- `GET /_sww/metrics`: the metrics, for `SWW_METRICS_TOKEN` (see
|
||||
[`metrics.md`](metrics.md)).
|
||||
- `GET /_sww/bans`: every ban held, under `bans`, each as `bans.json` lists it.
|
||||
Needs `SWWAF_ADMIN_TOKEN`, as do the three below.
|
||||
Needs `SWW_ADMIN_TOKEN`, as do the three below.
|
||||
- `POST /_sww/bans`: bans a netblock. The body is a JSON object of `netblock`,
|
||||
`duration` and, if you like, `reason`. `netblock` is a netblock such as
|
||||
`203.0.113.0/24`, or a client's address, which bans its IPv4 netblock
|
||||
(`SWWAF_BAN_SCOPE_V4_PREFIX`) or its IPv6 group. `duration` is a duration such
|
||||
(`SWW_BAN_SCOPE_V4_PREFIX`) or its IPv6 group. `duration` is a duration such
|
||||
as `1h` or `7d`, or `permanent`. A bad body is answered `400`, saying what is
|
||||
wrong:
|
||||
- one that is not such an object, or has another field;
|
||||
@@ -38,7 +38,7 @@ in [`SPEC.md`](../SPEC.md)).
|
||||
|
||||
## Examples
|
||||
|
||||
With `SWWAF_ADMIN_TOKEN` set, the token in `$TOKEN`, and `discriminator` reached
|
||||
With `SWW_ADMIN_TOKEN` set, the token in `$TOKEN`, and `discriminator` reached
|
||||
as `discriminator:8080` on its network:
|
||||
|
||||
```sh
|
||||
|
||||
+7
-7
@@ -9,9 +9,9 @@ back or dropped.
|
||||
- `ban`: a ban `discriminator` made.
|
||||
- `permanent_ban`: a permanent ban it made, or an existing ban made permanent by
|
||||
a request.
|
||||
- `waf_block`: the Core Rule Set scored a request at or over the threshold. The
|
||||
request was refused unless the alert's `detail` gives `mode`: `detect` or
|
||||
`observe`.
|
||||
- `core_rule_set_match`: the Core Rule Set scored a request at or over the
|
||||
threshold. The request was refused unless the alert's `detail` gives `mode`:
|
||||
`detect` or `observe`.
|
||||
- `anomaly`: a count over an anomaly threshold.
|
||||
- `reputation_hit`: a request from a client on a blocklist, the CrowdSec
|
||||
decision list or a DNS blocklist, or with an AbuseIPDB score that is a hit.
|
||||
@@ -77,7 +77,7 @@ on one line:
|
||||
}
|
||||
```
|
||||
|
||||
- `instance`: `SWWAF_INSTANCE_NAME`. `time`: when the alert was raised. `event`:
|
||||
- `instance`: `SWW_INSTANCE_NAME`. `time`: when the alert was raised. `event`:
|
||||
one of the events above.
|
||||
- `host`: the site of the request that raised the alert, since one
|
||||
`discriminator` serves many sites. It is left out of `source_failure`,
|
||||
@@ -122,15 +122,15 @@ Slack gets the title in bold and the text below it, with `&`, `<` and `>`
|
||||
escaped. ntfy gets the text as the message, with these headers:
|
||||
|
||||
- `Title`: the title;
|
||||
- `Authorization`: the token, as `Bearer <token>`, while
|
||||
`SWWAF_ALERT_NTFY_TOKEN` is set;
|
||||
- `Authorization`: the token, as `Bearer <token>`, while `SWW_ALERT_NTFY_TOKEN`
|
||||
is set;
|
||||
- `Priority` and `Tags`: by event, as listed below.
|
||||
|
||||
The priority and tag by event:
|
||||
|
||||
- `ban`: priority `default`, tag `no_entry`.
|
||||
- `permanent_ban`: priority `high`, tag `no_entry`.
|
||||
- `waf_block`: priority `default`, tag `shield`.
|
||||
- `core_rule_set_match`: priority `default`, tag `shield`.
|
||||
- `anomaly`: priority `high`, tag `chart_with_upwards_trend`.
|
||||
- `reputation_hit`: priority `low`, tag `label`.
|
||||
- `source_failure` and `file_error`: priority `high`, tag `warning`.
|
||||
|
||||
+10
-6
@@ -5,6 +5,9 @@ What `GET /_sww/metrics` answers with, at `discriminator`'s own address.
|
||||
keeps the `instance` label as `exported_instance` unless the scrape sets
|
||||
`honor_labels: true`. `rproxy` has no metrics.
|
||||
|
||||
Every `sww_*request_bytes_total` and `sww_*response_bytes_total` series counts
|
||||
body bytes, never headers, whatever `SWW_BYTES_COUNT` says.
|
||||
|
||||
## Traffic
|
||||
|
||||
- `sww_requests_total`, `sww_request_bytes_total` and
|
||||
@@ -25,11 +28,12 @@ keeps the `instance` label as `exported_instance` unless the scrape sets
|
||||
## Attack detection
|
||||
|
||||
- `sww_rule_matches_total`, by `rule_id` and `action`, and `sww_rules_loaded`;
|
||||
- `sww_waf_matches_total`, by `mode` and `rule_id`, for the rules that fired.
|
||||
- `sww_core_rule_set_matches_total`, by `mode` and `rule_id`, for the rules that
|
||||
fired.
|
||||
|
||||
## Countries and AS numbers
|
||||
|
||||
For the `SWWAF_METRICS_TOP_N` busiest of each, the rest as `other`:
|
||||
For the `SWW_METRICS_TOP_N` busiest of each, the rest as `other`:
|
||||
|
||||
- `sww_country_requests_total`, `sww_country_request_bytes_total`,
|
||||
`sww_country_response_bytes_total` and `sww_country_list_refusals_total`, by
|
||||
@@ -40,8 +44,8 @@ For the `SWWAF_METRICS_TOP_N` busiest of each, the rest as `other`:
|
||||
## Lookups
|
||||
|
||||
- `sww_geojs_requests_total` and `sww_geojs_failures_total`;
|
||||
- `sww_geojs_unanswered_total`: the requests that needed an answer and went on
|
||||
without it;
|
||||
- `sww_requests_without_geojs_answer_total`: the requests that needed an answer
|
||||
and went on without it;
|
||||
- with the file source, `sww_lookup_database_last_read_timestamp_seconds` and
|
||||
`sww_lookup_database_read_failures_total`.
|
||||
|
||||
@@ -67,8 +71,8 @@ By `file`:
|
||||
## Remote log, alerts and the process
|
||||
|
||||
- while a syslog server is set, `sww_remote_log_lines_sent_total`,
|
||||
`sww_remote_log_lines_dropped_total` and `sww_remote_log_buffer_depth`;
|
||||
- by `destination`, `sww_alerts_sent_total`, `sww_alerts_failed_total`,
|
||||
`sww_remote_log_lines_dropped_total` and `sww_remote_log_lines_waiting`;
|
||||
- by `destination`, `sww_alerts_sent_total`, `sww_alert_send_failures_total`,
|
||||
`sww_alerts_suppressed_total` and `sww_alerts_dropped_total`;
|
||||
- Go's own `go_` metrics and the process's `process_` metrics.
|
||||
|
||||
|
||||
+80
-37
@@ -26,7 +26,7 @@ These questions write no line:
|
||||
A `report` line:
|
||||
|
||||
```
|
||||
{"type":"request","discriminator_line":"report","time":"2026-10-03T12:00:00.123Z","instance":"fsn1/discriminator","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"waf_score":0,"duration_total":3.217,"duration_checks":0.041,"duration_waf":0.029,"duration_upstream_total":3.104}
|
||||
{"type":"request","discriminator_line":"report","time":"2026-10-03T12:00:00.123Z","instance":"fsn1/discriminator","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"core_rule_set_score":0,"duration_total_ms":3.217,"duration_checks_ms":0.041,"duration_core_rule_set_ms":0.029,"duration_upstream_total_ms":3.104}
|
||||
```
|
||||
|
||||
### Every line
|
||||
@@ -37,14 +37,14 @@ Every one of these lines has:
|
||||
- the standard web log fields;
|
||||
- `request_id`, `peer_ip` and `client_group`;
|
||||
- `asn`, `as_name` and `country`;
|
||||
- `action` and `duration_total`.
|
||||
- `action` and `duration_total_ms`.
|
||||
|
||||
### Standard web log fields
|
||||
|
||||
- `time`: when `discriminator` had read the question's body, RFC 3339 with
|
||||
milliseconds, in UTC. The `report` line gives the same time as the `decision`
|
||||
line.
|
||||
- `instance`: `SWWAF_INSTANCE_NAME`.
|
||||
- `instance`: `SWW_INSTANCE_NAME`.
|
||||
- `client_ip`: the visitor's address, as `rproxy` worked it out.
|
||||
- `method`, `path` and `query`: the visitor's request.
|
||||
- `scheme`: how the visitor reached `rproxy` or the trusted proxy in front of
|
||||
@@ -82,8 +82,8 @@ Every one of these lines has:
|
||||
- GeoJS had not answered by the time `discriminator` answered.
|
||||
- `content_type`: the visitor's `Content-Type`.
|
||||
- `content_length`: the body length the visitor announced.
|
||||
- `request_headers`: the headers listed in `SWWAF_LOG_REQUEST_HEADERS`, by name
|
||||
in lower case.
|
||||
- `request_headers`: the headers listed in `SWW_LOG_REQUEST_HEADERS`, by name in
|
||||
lower case.
|
||||
- `has_authorization` and `has_cookie`: true when the request had those headers.
|
||||
Their values are never logged.
|
||||
|
||||
@@ -96,49 +96,92 @@ Every one of these lines has:
|
||||
|
||||
- `action`: what was done:
|
||||
- `forward`: answered `allow`, so that `rproxy` passes it to the app;
|
||||
- `denied`: refused for `SWWAF_DENY_NETS`, or for a reputation source with
|
||||
the action `deny`;
|
||||
- `banned`: refused under a ban, or for a `ban` rule, a trap path or a
|
||||
CrowdSec decision, each of which bans the client;
|
||||
- `country_denied`, `rate_limited`, `rule_blocked` and `waf_blocked`:
|
||||
refused by that check;
|
||||
- `too_large`: over a header limit;
|
||||
- `admin`: answered at one of `discriminator`'s own endpoints, or `404` for
|
||||
a path under `/_sww/`.
|
||||
- `client_denied`: refused for `SWW_DENY_NETS`, for a reputation source with
|
||||
the action `deny`, or for a tag in `SWW_TAG_DENY`;
|
||||
- `temporarily_banned`: refused under a ban that runs out, or for a `ban`
|
||||
rule, a trap path or a CrowdSec decision that made such a ban;
|
||||
- `permanently_banned`: refused under a ban that never runs out, including a
|
||||
ban this request made or made permanent;
|
||||
- `country_denied`, `client_is_rate_limited`, `blocked_by_block_rule` and
|
||||
`blocked_by_core_rule_set`: refused by that check;
|
||||
- `request_header_too_large`: over a header limit;
|
||||
- `own_endpoint`: answered at one of `discriminator`'s own endpoints, or
|
||||
`404` for a path under `/_sww/`.
|
||||
- `would_action`: in `observe` mode, what `enforce` mode would have done with a
|
||||
request it would have refused. `action` then names what was done, normally
|
||||
`forward`.
|
||||
- `limit_percent` and `limit_percent_setting`: the client's percentage of the
|
||||
rate limits and the setting that lowered it, when one did. `bytes_percent` and
|
||||
`bytes_percent_setting` do the same for the byte limits.
|
||||
- `heuristic`: the heuristic that refused the request, in `observe` mode too. It
|
||||
is left out when none did.
|
||||
- `heuristic_figures`: an object of the figures that heuristic acted on. Each
|
||||
heuristic and its figures:
|
||||
- `deny_nets`: `netblock`, the entry of `SWW_DENY_NETS` the client is in;
|
||||
- `active_ban`: `netblock`, the ban's netblock, and `ban_expires`;
|
||||
- `countries`: `country`, and `setting`, the country list that refused it;
|
||||
- `tag_deny`: `tag`, the tag of `SWW_TAG_DENY` the request carries;
|
||||
- `blocklists`: `lists`, the blocklist URLs that list the client;
|
||||
- `crowdsec`: `scenario`, the decision's scenario, and `expires`, when the
|
||||
decision ends;
|
||||
- `dnsbl`: `zones`, the DNS blocklist zones that list the client, their keys
|
||||
masked;
|
||||
- `abuseipdb`: `score`, and `min_score`, `SWW_ABUSEIPDB_MIN_SCORE`;
|
||||
- `rate_limits`: `window`, `count` and `limit`;
|
||||
- `trap_paths`: `path`, the trap path asked for;
|
||||
- `rule_files`: `rule_id` and `rule_action`, of the rule that refused it;
|
||||
- `header_limits`: `fields`, `fields_limit`, `cookies` and `cookies_limit`;
|
||||
- `core_rule_set`: `score`, and `threshold`, `SWW_WAF_ANOMALY_THRESHOLD`;
|
||||
- `refusal_limit`: `window`, `count` and `limit`, of the error burst;
|
||||
- `byte_limits`: `window`, `count` and `limit`, the count in bytes.
|
||||
- `after_answer_heuristic`: the heuristic that banned the client after
|
||||
`discriminator` answered the request, in `observe` mode too: `refusal_limit`
|
||||
or `byte_limits`.
|
||||
- `after_answer_heuristic_figures`: an object of the figures that heuristic
|
||||
acted on, as listed above.
|
||||
|
||||
`byte_limits` bans when the report comes, so only the `report` line names
|
||||
it.
|
||||
|
||||
- `rate_limit_percent` and `rate_limit_percent_setting`: the client's percentage
|
||||
of the rate limits and the setting that lowered it, when one did.
|
||||
`bytes_limit_percent` and `bytes_limit_percent_setting` do the same for the
|
||||
byte limits.
|
||||
- `counts`: the client's totals as rate limiting counts them, this request
|
||||
included. It is left out when rate limiting did not count the request. It
|
||||
holds:
|
||||
- `minute`, `hour` and `day`: requests;
|
||||
- `minute_bytes`, `hour_bytes` and `day_bytes`: bytes.
|
||||
- `minute_bytes`, `hour_bytes` and `day_bytes`: the body bytes
|
||||
`SWW_BYTES_COUNT` names, in that window.
|
||||
|
||||
A count can have a fraction (see "Counting method" in
|
||||
[`SPEC.md`](../SPEC.md)). The bytes of an allowed request are counted when
|
||||
its report comes. So on the `decision` line, the bytes are without this
|
||||
request's.
|
||||
[`SPEC.md`](../SPEC.md)). The bytes counted for an allowed request and its
|
||||
response are added when its report comes. So on the `decision` line, the
|
||||
three byte totals leave them out.
|
||||
|
||||
- `limit_hit`: the limit broken:
|
||||
- `minute`, `hour` or `day` for a rate limit;
|
||||
- `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit;
|
||||
- `error_burst`.
|
||||
- `minute_refusals` for the error burst (`SWW_ERROR_BURST_THRESHOLD`).
|
||||
|
||||
A request whose bytes broke a limit was passed on whole, so its `action`
|
||||
stays `forward`. Only its `report` line gives that limit.
|
||||
When the bytes counted for a request and its response break a limit, the
|
||||
request has been passed on whole, so its `action` stays `forward`. Only its
|
||||
`report` line gives that limit.
|
||||
|
||||
- `rule_ids`: the rule file rules that matched, in order, up to the one that
|
||||
refused.
|
||||
- `waf_score` and `waf_rule_ids`: the Core Rule Set's score and the rules that
|
||||
matched, for a request it inspected.
|
||||
- `core_rule_set_score` and `core_rule_set_rule_ids`: the Core Rule Set's score
|
||||
and the rules that matched, for a request it inspected.
|
||||
- `reputation`: the sources the client is on, in this order:
|
||||
- blocklist URLs;
|
||||
- the CrowdSec decision list's URL;
|
||||
- DNS blocklist zones;
|
||||
- `abuseipdb`.
|
||||
- `tags`: the request's tags, sorted, left out when it has none. Each tag and
|
||||
where it comes from is under "Request tags" in [`SPEC.md`](../SPEC.md).
|
||||
- `routewatch_tags_missing`: why the request got no tags from routewatch,
|
||||
present only while `SWW_ROUTEWATCH_URL` is set:
|
||||
- `no_answer_yet`: routewatch has not answered about this client yet;
|
||||
- `request_failed`: a query to routewatch failed within the last minute, and
|
||||
the process log says why;
|
||||
- `no_route`: routewatch knows no live route for the address.
|
||||
- `offence`: `limit`, for a request that broke a limit.
|
||||
- `ban_expires`: when the client's ban ends, for a request that made one or was
|
||||
refused under one: a time, or `permanent`.
|
||||
@@ -147,14 +190,14 @@ Every one of these lines has:
|
||||
|
||||
All in milliseconds:
|
||||
|
||||
- `duration_total`:
|
||||
- `duration_total_ms`:
|
||||
- on the `decision` line, from `discriminator` having read the question's
|
||||
body to the line;
|
||||
- on the `report` line, from `rproxy` having read the request's headers to
|
||||
the end of the answer.
|
||||
- `duration_checks`: to the end of the checks;
|
||||
- `duration_waf`: the part of that the Core Rule Set took;
|
||||
- `duration_upstream_total`: on the `report` line of an allowed request, from
|
||||
- `duration_checks_ms`: to the end of the checks;
|
||||
- `duration_core_rule_set_ms`: the part of that the Core Rule Set took;
|
||||
- `duration_upstream_total_ms`: on the `report` line of an allowed request, from
|
||||
the hand-over to the app to the end of the answer.
|
||||
|
||||
### Requests to its own endpoints
|
||||
@@ -167,8 +210,8 @@ these meanings where they differ:
|
||||
- `scheme`: `http`.
|
||||
- `time`: when the request's headers had been read.
|
||||
- `request_bytes`: the bytes read of a ban to add, and otherwise `0`.
|
||||
- `action`: `admin`.
|
||||
- `duration_total`: from the request's headers being read to the line.
|
||||
- `action`: `own_endpoint`.
|
||||
- `duration_total_ms`: from the request's headers being read to the line.
|
||||
|
||||
## `rproxy`'s line
|
||||
|
||||
@@ -186,10 +229,10 @@ these meanings where they differ:
|
||||
a WebSocket carried from the visitor.
|
||||
- `response_bytes`: the bytes of the answer's body, with what a WebSocket
|
||||
carried from the app.
|
||||
- `duration_request`: milliseconds from the request's headers being read until
|
||||
the app had been sent the whole request, or the answer started if that was
|
||||
sooner. It includes the wait for `discriminator`.
|
||||
- `duration_response`: milliseconds from then until the end of the answer.
|
||||
- `duration_request_ms`: milliseconds from the request's headers being read
|
||||
until the app had been sent the whole request, or the answer started if that
|
||||
was sooner. It includes the wait for `discriminator`.
|
||||
- `duration_response_ms`: milliseconds from then until the end of the answer.
|
||||
- `disposition`: what was done with the request: `allow`, `deny` or `close`. It
|
||||
is `discriminator`'s answer, or the value of `SWRP_DEFAULT_DISPOSITION`. It is
|
||||
left out when:
|
||||
@@ -203,5 +246,5 @@ these meanings where they differ:
|
||||
For example:
|
||||
|
||||
```
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","client_ip":"203.0.113.9","host":"app.example","method":"GET","path":"/","query":"","status":200,"request_bytes":0,"response_bytes":5120,"duration_request":1.204,"duration_response":3.104,"disposition":"allow"}
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","client_ip":"203.0.113.9","host":"app.example","method":"GET","path":"/","query":"","status":200,"request_bytes":0,"response_bytes":5120,"duration_request_ms":1.204,"duration_response_ms":3.104,"disposition":"allow"}
|
||||
```
|
||||
+225
-159
@@ -4,9 +4,9 @@ Every setting, its default and the values it takes, in the groups of "Settings"
|
||||
in [`SPEC.md`](../SPEC.md). That section gives the conventions, the settings
|
||||
that take `off`, the `_FILE` form, and what each group does and why.
|
||||
|
||||
The settings that start with `SWWAF_` are `discriminator`'s. Those that start
|
||||
with `SWRP_` are `rproxy`'s, and are under "`rproxy`" below. `SWW_COMMAND`
|
||||
chooses between the two (see "The image's command" below).
|
||||
The settings that start with `SWW_` are `discriminator`'s. Those that start with
|
||||
`SWRP_` are `rproxy`'s, and are under "`rproxy`" below. `SWW_COMMAND` chooses
|
||||
between the two (see "The image's command" below).
|
||||
|
||||
## Forms of values
|
||||
|
||||
@@ -28,270 +28,336 @@ chooses between the two (see "The image's command" below).
|
||||
|
||||
## Core
|
||||
|
||||
- `SWWAF_LISTEN_ADDR` (default `:8080`): where `discriminator` listens: `:` and
|
||||
a port, such as `:9000`. Leave the host out (see "Ports" in
|
||||
- `SWW_LISTEN_ADDR` (default `:8080`): where `discriminator` listens: `:` and a
|
||||
port, such as `:9000`. Leave the host out (see "Ports" in
|
||||
[`SPEC.md`](../SPEC.md)).
|
||||
- `SWWAF_INSTANCE_NAME` (default: the container's host name): the `instance`
|
||||
name in each line `discriminator` logs, and in each alert and metric, such as
|
||||
- `SWW_INSTANCE_NAME` (default: the container's host name): the `instance` name
|
||||
in each line `discriminator` logs, and in each alert and metric, such as
|
||||
`fsn1/discriminator`. Valid UTF-8. No control character while
|
||||
`SWWAF_ALERT_NTFY_URL` is set. 1 to 48 printable ASCII characters without a
|
||||
space while `SWWAF_LOG_REMOTE_URL` is set and `SWWAF_LOG_REMOTE_APP_NAME` is
|
||||
not.
|
||||
- `SWWAF_MODE` (default `enforce`): `enforce` or `observe`.
|
||||
- `SWWAF_IPV6_GROUP_PREFIX` (default `64`): the prefix length of an IPv6
|
||||
client's group, from 32 to 128.
|
||||
- `SWWAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept, a whole
|
||||
`SWW_ALERT_NTFY_URL` is set. 1 to 48 printable ASCII characters without a
|
||||
space while `SWW_LOG_REMOTE_URL` is set and `SWW_LOG_REMOTE_APP_NAME` is not.
|
||||
- `SWW_MODE` (default `enforce`): `enforce` or `observe`.
|
||||
- `SWW_IPV6_GROUP_PREFIX` (default `64`): the prefix length of an IPv6 client's
|
||||
group, from 32 to 128.
|
||||
- `SWW_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept, a whole
|
||||
number above zero.
|
||||
- `SWWAF_MAX_BANS` (default `5000`): how many of its own bans `discriminator`
|
||||
- `SWW_MAX_BANS` (default `5000`): how many of its own bans `discriminator`
|
||||
keeps, a whole number above zero.
|
||||
|
||||
## Tokens
|
||||
|
||||
- `SWWAF_ADMIN_TOKEN` (default unset): the token for the ban endpoints and
|
||||
- `SWW_ADMIN_TOKEN` (default unset): the token for the ban endpoints and
|
||||
`/_sww/clients/<ip>`. At least 32 characters.
|
||||
- `SWWAF_METRICS_TOKEN` (default unset): the token for `/_sww/metrics`. At least
|
||||
- `SWW_METRICS_TOKEN` (default unset): the token for `/_sww/metrics`. At least
|
||||
32 characters.
|
||||
|
||||
## State files
|
||||
|
||||
- `SWWAF_STATE_DIR` (default `/var/lib/sww`): the directory of the state files,
|
||||
an absolute path that exists and can be written.
|
||||
- `SWWAF_STATE_WRITE_DELAY` (default `10s`): how long `bans.json` waits to be
|
||||
- `SWW_STATE_DIR` (default `/var/lib/sww`): the directory of the state files, an
|
||||
absolute path that exists and can be written.
|
||||
- `SWW_STATE_WRITE_DELAY` (default `10s`): how long `bans.json` waits to be
|
||||
written after a ban is made, lifted or made permanent. A duration.
|
||||
- `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is
|
||||
- `SWW_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is
|
||||
written. A duration.
|
||||
|
||||
## Logging
|
||||
|
||||
- `SWWAF_LOG_LEVEL` (default `info`): the least severe of `discriminator`'s own
|
||||
- `SWW_LOG_LEVEL` (default `info`): the least severe of `discriminator`'s own
|
||||
messages that are written: `debug`, `info`, `warn` or `error`.
|
||||
- `SWWAF_LOG_REQUEST_HEADERS` (default
|
||||
- `SWW_LOG_REQUEST_HEADERS` (default
|
||||
`accept,accept-language,accept-encoding,content-type,origin,range`): the
|
||||
request headers recorded in the log line, in either case. `Host` and
|
||||
`Transfer-Encoding` are refused; the host is the field `host`.
|
||||
- `SWWAF_LOG_REMOTE_URL` (default unset): a syslog server that also gets every
|
||||
- `SWW_LOG_REMOTE_URL` (default unset): a syslog server that also gets every
|
||||
line written to stdout: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://`,
|
||||
each with a host and a port and nothing more.
|
||||
- `SWWAF_LOG_REMOTE_TLS_CA_FILE` (default unset): a file of PEM certificates
|
||||
that a `syslog+tls` server's certificate must chain to, instead of the host's
|
||||
own. A file that cannot be read, or holds no certificate, is refused.
|
||||
- `SWWAF_LOG_REMOTE_BUFFER` (default `10000`): the most lines held while they
|
||||
wait to be sent, a whole number above zero.
|
||||
- `SWWAF_LOG_REMOTE_FACILITY` (default `local0`): the syslog facility: `kern`,
|
||||
- `SWW_LOG_REMOTE_TLS_CA_FILE` (default unset): a file of PEM certificates that
|
||||
a `syslog+tls` server's certificate must chain to, instead of the host's own.
|
||||
A file that cannot be read, or holds no certificate, is refused.
|
||||
- `SWW_LOG_REMOTE_BUFFER` (default `10000`): the most lines held while they wait
|
||||
to be sent, a whole number above zero.
|
||||
- `SWW_LOG_REMOTE_FACILITY` (default `local0`): the syslog facility: `kern`,
|
||||
`user`, `mail`, `daemon`, `auth`, `syslog`, `lpr`, `news`, `uucp`, `cron`,
|
||||
`authpriv`, `ftp`, or `local0` to `local7`.
|
||||
- `SWWAF_LOG_REMOTE_APP_NAME` (default: `SWWAF_INSTANCE_NAME`): the app name in
|
||||
each syslog record, 1 to 48 printable ASCII characters without a space.
|
||||
- `SWW_LOG_REMOTE_APP_NAME` (default: `SWW_INSTANCE_NAME`): the app name in each
|
||||
syslog record, 1 to 48 printable ASCII characters without a space.
|
||||
|
||||
## Metrics
|
||||
|
||||
- `SWWAF_METRICS_TOP_N` (default `50`): how many AS numbers and how many
|
||||
countries get series of their own. A whole number above zero.
|
||||
- `SWW_METRICS_TOP_N` (default `50`): how many AS numbers and how many countries
|
||||
get series of their own. A whole number above zero.
|
||||
|
||||
## Static lists
|
||||
|
||||
- `SWWAF_ALLOW_NETS` (default empty): netblocks you trust, such as your
|
||||
monitoring (see step 3 in [`SPEC.md`](../SPEC.md)).
|
||||
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients are
|
||||
- `SWW_ALLOW_NETS` (default empty): netblocks you trust, such as your monitoring
|
||||
(see step 4 in [`SPEC.md`](../SPEC.md)).
|
||||
- `SWW_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients are
|
||||
exempt from rate limiting and the byte limits.
|
||||
- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are refused (see
|
||||
step 3 in [`SPEC.md`](../SPEC.md)).
|
||||
- `SWW_DENY_NETS` (default empty): netblocks whose clients are refused (see step
|
||||
4 in [`SPEC.md`](../SPEC.md)).
|
||||
|
||||
## Request rate limits, per client
|
||||
|
||||
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `120`, 2 a second): the most requests a
|
||||
The request past one of these limits is refused, and the client is banned.
|
||||
|
||||
- `SWW_RATE_LIMIT_PER_MINUTE` (default `120`, 2 a second): the most requests a
|
||||
client may make in a minute. A count or `off`. The default leaves room for a
|
||||
person opening two pages at once with an empty cache, up to 60 files each.
|
||||
- `SWWAF_RATE_LIMIT_PER_HOUR` (default `1500`): the most requests a client may
|
||||
- `SWW_RATE_LIMIT_PER_HOUR` (default `1500`): the most requests a client may
|
||||
make in an hour. A count or `off`. It is the main limit against scrapers. A
|
||||
person reading a site for an hour, a page every 20 seconds, makes about 900. A
|
||||
scraper making one request every 2 seconds makes 1,800 an hour, more than the
|
||||
limit.
|
||||
- `SWWAF_RATE_LIMIT_PER_DAY` (default `6000`): the most requests a client may
|
||||
make in a day. A count or `off`. The default is four hours at the hour limit.
|
||||
A scraper making one request every 10 seconds makes 8,640 a day, more than the
|
||||
- `SWW_RATE_LIMIT_PER_DAY` (default `6000`): the most requests a client may make
|
||||
in a day. A count or `off`. The default is four hours at the hour limit. A
|
||||
scraper making one request every 10 seconds makes 8,640 a day, more than the
|
||||
limit.
|
||||
- A CI runner asks its code forge for work all day and can pass the limits. Put
|
||||
the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. Do the same for a
|
||||
the runners' addresses in `SWW_RATE_LIMIT_EXEMPT_NETS`. Do the same for a
|
||||
network where many people share one address, such as an office.
|
||||
- Counting cannot catch a client that stays under all three limits (see "Request
|
||||
rate limits, per client" in [`SPEC.md`](../SPEC.md)).
|
||||
- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from
|
||||
rate limiting and the byte limits, such as `/assets/`.
|
||||
- `SWW_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from rate
|
||||
limiting and the byte limits, such as `/assets/`.
|
||||
|
||||
## Byte limits, per client
|
||||
|
||||
- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `off`), `SWWAF_BYTES_LIMIT_PER_HOUR`
|
||||
(default `50G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `100G`): the most
|
||||
bytes counted for a client in each window. A size or `off`.
|
||||
- `SWWAF_BYTES_COUNT` (default `both`): which body bytes the byte limits count:
|
||||
`response`, `request` or `both`.
|
||||
These count body bytes, never headers. What the visitor sent over a WebSocket
|
||||
counts as request bytes, and what the app sent as response bytes. The bytes
|
||||
counted for a request and its response are added to the client's total once the
|
||||
response has ended. The request that takes a client's total past a limit is not
|
||||
cut off; the client is banned.
|
||||
|
||||
- `SWW_BYTES_LIMIT_PER_MINUTE` (default `off`), `SWW_BYTES_LIMIT_PER_HOUR`
|
||||
(default `50G`) and `SWW_BYTES_LIMIT_PER_DAY` (default `100G`): the most bytes
|
||||
counted for a client in each window, a size or `off`. `SWW_BYTES_COUNT` says
|
||||
which bytes count: response bodies, request bodies or both. The request that
|
||||
passes the limit is not cut off, and the client is banned when its response
|
||||
ends.
|
||||
- `SWW_BYTES_COUNT` (default `both`): what the byte limits and the byte anomaly
|
||||
thresholds count: `response`, the app's response bodies; `request`, the
|
||||
visitors' request bodies; or `both`. What the visitor sent over a WebSocket
|
||||
counts as request bytes, and what the app sent as response bytes.
|
||||
|
||||
## Lookup of AS number and country
|
||||
|
||||
- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `off` stops
|
||||
- `SWW_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `off` stops
|
||||
the start while a setting that needs a lookup is set:
|
||||
- the two country lists;
|
||||
- `SWWAF_ADD_LOOKUP_HEADERS`;
|
||||
- the four biased threshold lists and `SWWAF_ASN_LIMIT_PERCENT_URL`;
|
||||
- `SWWAF_UNKNOWN_LIMIT_PERCENT` below `100`;
|
||||
- `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES`;
|
||||
- `SWW_ADD_LOOKUP_HEADERS`;
|
||||
- `SWW_TAG_DENY` or `SWW_TAG_LIMIT_PERCENT` naming a `country=` tag, or an
|
||||
`asn=` tag while `SWW_ROUTEWATCH_URL` is unset;
|
||||
- the two byte percentage lists and `SWW_ASN_LIMIT_PERCENT_URL`;
|
||||
- `SWW_UNKNOWN_LIMIT_PERCENT` below `100`;
|
||||
- an anomaly threshold per AS number.
|
||||
- `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its
|
||||
`.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`.
|
||||
- `SWWAF_LOOKUP_TIMEOUT` (default `800ms`): how long a request waits for its
|
||||
- `SWW_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its
|
||||
`.mmdb` form, for `SWW_LOOKUP_SOURCE=file`.
|
||||
- `SWW_LOOKUP_TIMEOUT` (default `800ms`): how long a request waits for its
|
||||
client's first answer from GeoJS, and how long a request to GeoJS may take. A
|
||||
duration; never `off`.
|
||||
- `SWWAF_ADD_LOOKUP_HEADERS` (default `false`): `true` has `rproxy` pass the app
|
||||
- `SWW_ADD_LOOKUP_HEADERS` (default `false`): `true` has `rproxy` pass the app
|
||||
the client's AS number in `X-Client-ASN` and its country in
|
||||
`X-Client-Country`.
|
||||
|
||||
## Country lists
|
||||
## Country list
|
||||
|
||||
- `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused,
|
||||
such as `cn,ru,kp`.
|
||||
- `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` (default empty): when set, the only
|
||||
countries whose clients get through, such as `us,de`. A country on both lists
|
||||
stops the start.
|
||||
- `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` (default empty): when set, the only
|
||||
countries whose clients get through, such as `us,de`. A country whose
|
||||
`country=` tag is in `SWW_TAG_DENY` stops the start.
|
||||
|
||||
## Request tags
|
||||
|
||||
Every request gets tags from what `discriminator` knows of its client. See
|
||||
"Request tags" in [`SPEC.md`](../SPEC.md).
|
||||
|
||||
- `SWW_TAG_LISTS` (default empty): address lists that each add a tag, as
|
||||
`name=URL` pairs, such as `torexit=https://example.com/tor.txt`. A name is the
|
||||
tag, and may hold `=`, such as `vpn=mullvad`. Each list is an `http` or
|
||||
`https` URL fetched like a blocklist. A name or URL listed twice is refused.
|
||||
So is a URL that is also in `SWW_BLOCKLIST_URLS`, `SWW_ASN_LIMIT_PERCENT_URL`,
|
||||
or the CrowdSec decision list.
|
||||
- `SWW_TAG_DENY` (default empty): tags whose requests are refused, such as
|
||||
`torexit,country=CN`. A tag listed twice, an `asn=` tag without a number, or a
|
||||
`country=` tag without a country, is refused.
|
||||
- `SWW_TAG_LIMIT_PERCENT` (default empty): tags that lower a client's limits, as
|
||||
tag and percentage pairs, such as `vpn:50,asn=14061:25`. A percentage is from
|
||||
`1` to `99`, of every rate limit and byte limit. The lowest percentage among a
|
||||
client's tags applies. A tag listed twice is refused. So is
|
||||
`exploitsignature`: a request gets that tag after the rate limits have counted
|
||||
it.
|
||||
|
||||
- `SWW_ROUTEWATCH_URL` (default unset): the `http` or `https` URL of a
|
||||
routewatch instance. [routewatch](https://git.eeqj.de/sneak/routewatch) is a
|
||||
service that gives an address's announced netblock and origin AS.
|
||||
`discriminator` asks it about each client, in the background, and adds the
|
||||
tags `asn=<number>` and `prefix=<netblock>` from its answer. While it is
|
||||
unset, nothing asks routewatch and no request gets a `prefix=` tag.
|
||||
|
||||
`SWW_TAG_DENY` and `SWW_TAG_LIMIT_PERCENT` accept tags in either case. A tag
|
||||
that no request can get stops the start. A request can get only these:
|
||||
|
||||
- a name in `SWW_TAG_LISTS`;
|
||||
- an `asn=` or `country=` tag;
|
||||
- a `prefix=` tag with a netblock, while `SWW_ROUTEWATCH_URL` is set. It matches
|
||||
only that exact announced netblock, such as `prefix=192.0.2.0/24`;
|
||||
- a `dnsbl=` tag of a zone in `SWW_DNSBL_ZONES`, in lower case, with a key
|
||||
masked as the `reputation` field shows it;
|
||||
- `abuseipdb`, while `SWW_ABUSEIPDB_KEY` is set;
|
||||
- `exploitsignature`, in `SWW_TAG_DENY` only, while `SWW_RULES_ENABLED` is
|
||||
`true`, or while neither `SWW_WAF_MODE` nor `SWW_WAF_ANOMALY_THRESHOLD` is
|
||||
`off`.
|
||||
|
||||
`crowdsec` stops the start in both settings, because a CrowdSec decision refuses
|
||||
the request first.
|
||||
|
||||
## Biased thresholds
|
||||
|
||||
- `SWWAF_ASN_LIMIT_PERCENT` (default empty): a list of AS numbers, each with `:`
|
||||
and a percentage, such as `AS14061:50,AS16276:50,AS45102:25`. An AS number
|
||||
listed twice is refused.
|
||||
- `SWWAF_COUNTRY_LIMIT_PERCENT` (default empty): the same by country, such as
|
||||
`cn:25,ru:50`.
|
||||
- `SWWAF_ASN_BYTES_PERCENT` and `SWWAF_COUNTRY_BYTES_PERCENT` (default empty):
|
||||
the same, for the byte limits alone.
|
||||
- `SWWAF_UNKNOWN_LIMIT_PERCENT` (default `100`): the percentage for a client the
|
||||
- `SWW_ASN_BYTES_PERCENT` and `SWW_COUNTRY_BYTES_PERCENT` (default empty): a
|
||||
percentage of the byte limits alone, for the AS numbers or countries listed,
|
||||
such as `AS14061:50` or `cn:25`. It takes the place of the percentage
|
||||
`SWW_TAG_LIMIT_PERCENT` gives their `asn=` or `country=` tag. An AS number or
|
||||
country listed twice is refused.
|
||||
- `SWW_UNKNOWN_LIMIT_PERCENT` (default `100`): the percentage for a client the
|
||||
lookup cannot place.
|
||||
- `SWWAF_ASN_LIMIT_PERCENT_URL` (default unset): the `http` or `https` URL of a
|
||||
- `SWW_ASN_LIMIT_PERCENT_URL` (default unset): the `http` or `https` URL of a
|
||||
text file with one AS number and percentage on each line, such as
|
||||
`AS64496:50`. A URL that is also in `SWWAF_BLOCKLIST_URLS` is refused.
|
||||
`AS64496:50`. Each entry counts like an `asn=` tag in `SWW_TAG_LIMIT_PERCENT`.
|
||||
A URL that is also in `SWW_BLOCKLIST_URLS` is refused.
|
||||
|
||||
## Attack detection
|
||||
|
||||
- `SWWAF_RULES_DIR` (default `/etc/sww/rules.d`): the directory of the rule
|
||||
files.
|
||||
- `SWWAF_RULES_ENABLED` (default `true`): `false` reads no rule file.
|
||||
- `SWWAF_TRAP_PATHS` (default empty): paths only scanners ask for, such as
|
||||
- `SWW_RULES_DIR` (default `/etc/sww/rules.d`): the directory of the rule files.
|
||||
- `SWW_RULES_ENABLED` (default `true`): `false` reads no rule file.
|
||||
- `SWW_TRAP_PATHS` (default empty): paths only scanners ask for, such as
|
||||
`/wp-login.php,/xmlrpc.php`. Each starts with `/` and holds no `?`.
|
||||
- `SWWAF_WAF_MODE` (default `block`): `off`, `detect` or `block`.
|
||||
- `SWWAF_WAF_PARANOIA_LEVEL` (default `1`): the Core Rule Set's paranoia level,
|
||||
- `SWW_WAF_MODE` (default `block`): `off`, `detect` or `block`.
|
||||
- `SWW_WAF_PARANOIA_LEVEL` (default `1`): the Core Rule Set's paranoia level,
|
||||
from 1 to 4.
|
||||
- `SWWAF_WAF_ANOMALY_THRESHOLD` (default `5`): the score at which a request is a
|
||||
- `SWW_WAF_ANOMALY_THRESHOLD` (default `5`): the score at which a request is a
|
||||
match. A whole number above zero, or `off`.
|
||||
- `SWWAF_WAF_DISABLED_RULES` (default
|
||||
- `SWW_WAF_DISABLED_RULES` (default
|
||||
`920340,920420,920440,920640,930130,930140`): the ids of Core Rule Set rules
|
||||
to switch off. Each a whole number above zero. An id from 900000 to 900999 is
|
||||
refused: those are the rules that set the Core Rule Set up, and
|
||||
`smallwebwaf`'s own.
|
||||
- `SWWAF_WAF_EXEMPT_PATHS` (default empty): path prefixes the Core Rule Set does
|
||||
- `SWW_WAF_EXEMPT_PATHS` (default empty): path prefixes the Core Rule Set does
|
||||
not inspect.
|
||||
- `SWWAF_WAF_BODY_LIMIT` (default `off`): `off`, or how much of a request body
|
||||
the Core Rule Set reads, a size of at most `64K`. A question carries no more
|
||||
of a body than 64 KiB, and the Core Rule Set reads at most 64 KiB less one
|
||||
byte of it, whatever the value (see "The answer" in [`SPEC.md`](../SPEC.md)).
|
||||
- `SWWAF_ERROR_BURST_THRESHOLD` (default `30`): the most refusals of one client
|
||||
in a minute (see "Attack detection" in `SPEC.md`). A count or `off`.
|
||||
- `SWW_WAF_BODY_LIMIT` (default `off`): `off`, or how much of a request body the
|
||||
Core Rule Set reads, a size of at most `64K`. A question carries no more of a
|
||||
body than 64 KiB, and the Core Rule Set reads at most 64 KiB less one byte of
|
||||
it, whatever the value (see "The answer" in [`SPEC.md`](../SPEC.md)).
|
||||
- `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` (default unset): a file of Core Rule Set
|
||||
exclusions, written as Coraza directives. It is loaded before the Core Rule
|
||||
Set's rules, so it holds only `SecAction` and `SecRule` lines that exclude
|
||||
with `ctl:ruleRemoveTargetById` or `ctl:ruleRemoveById`. A file that cannot be
|
||||
read or does not parse stops the start, whatever `SWW_WAF_MODE` is. So does a
|
||||
file that names a number from 900000 to 900999 outside a comment, alone or in
|
||||
a range. So does a directive that removes or changes rules as the file loads,
|
||||
such as `SecRuleRemoveById`, since none of the Core Rule Set's rules is loaded
|
||||
yet. It has no `_FILE` form.
|
||||
- `SWW_ERROR_BURST_THRESHOLD` (default `30`): the most refusals a client may get
|
||||
in a minute from a rule file, a trap path or the Core Rule Set, a count or
|
||||
`off`. The next one bans the client (see "Attack detection" in `SPEC.md`).
|
||||
|
||||
## Ban lengths, response and scope
|
||||
|
||||
- `SWWAF_BAN_RESPONSE` (default `403`): `403`, `429` or `close`.
|
||||
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken limit. A
|
||||
- `SWW_BAN_RESPONSE` (default `403`): `403`, `429` or `close`.
|
||||
- `SWW_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken limit. A
|
||||
duration.
|
||||
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): the time after a ban ends
|
||||
- `SWW_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): the time after a ban ends
|
||||
within which breaking a limit again lengthens the next ban. A duration.
|
||||
- `SWWAF_MAX_BAN_DURATION` (default `7d`): the longest ban for a broken limit;
|
||||
one that would be longer is permanent. A duration.
|
||||
- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
|
||||
- `SWW_MAX_BAN_DURATION` (default `7d`): the longest ban for a broken limit; one
|
||||
that would be longer is permanent. A duration.
|
||||
- `SWW_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
|
||||
attack. A duration.
|
||||
- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): how much a ban on an IPv4 client
|
||||
- `SWW_BAN_SCOPE_V4_PREFIX` (default `32`): how much a ban on an IPv4 client
|
||||
covers: the prefix length of its netblock, from 0 to 32.
|
||||
|
||||
## Reputation
|
||||
|
||||
- `SWWAF_BLOCKLIST_URLS` (default empty): blocklists to fetch, such as
|
||||
- `SWW_BLOCKLIST_URLS` (default empty): blocklists to fetch, such as
|
||||
`https://www.spamhaus.org/drop/drop.txt`. Each an `http` or `https` URL
|
||||
without a user or a fragment. A URL listed twice is refused.
|
||||
- `SWWAF_BLOCKLIST_REFRESH` (default `24h`): how long a list waits after its
|
||||
last try before it is fetched again. A duration of `1h` or more.
|
||||
- `SWWAF_BLOCKLIST_ACTION` (default `deny`): `deny`, `limit:<percent>` or `log`.
|
||||
- `SWWAF_DNSBL_ZONES` (default empty): the DNS blocklists to ask, such as
|
||||
- `SWW_BLOCKLIST_REFRESH` (default `24h`): how long a list waits after its last
|
||||
try before it is fetched again. A duration of `1h` or more.
|
||||
- `SWW_BLOCKLIST_ACTION` (default `deny`): `deny`, `limit:<percent>` or `log`.
|
||||
- `SWW_DNSBL_ZONES` (default empty): the DNS blocklists to ask, such as
|
||||
`dnsbl.dronebl.org`. Each a DNS name of at most 189 characters. A Spamhaus
|
||||
zone carries its key, as in `<key>.xbl.dq.spamhaus.net`. A zone listed twice
|
||||
is refused, whatever its case or key.
|
||||
- `SWWAF_DNSBL_RESOLVER` (default unset): the resolver to ask the zones through,
|
||||
- `SWW_DNSBL_RESOLVER` (default unset): the resolver to ask the zones through,
|
||||
such as `192.0.2.53` or `[2001:db8::53]:5353`. An IP address with an optional
|
||||
port, 53 when none is given. Unset, the host's resolver is used.
|
||||
- `SWWAF_ABUSEIPDB_KEY` (default unset): the key of your AbuseIPDB account. A
|
||||
- `SWW_ABUSEIPDB_KEY` (default unset): the key of your AbuseIPDB account. A
|
||||
control character in it is refused.
|
||||
- `SWWAF_ABUSEIPDB_MIN_SCORE` (default `75`): the least abuse confidence score,
|
||||
- `SWW_ABUSEIPDB_MIN_SCORE` (default `75`): the least abuse confidence score,
|
||||
from 0 to 100, that is a hit.
|
||||
- `SWWAF_ABUSEIPDB_DAILY_BUDGET` (default `900`): the most checks made in a day,
|
||||
a whole number above zero. AbuseIPDB's free accounts may make 1,000.
|
||||
- `SWWAF_CROWDSEC_LAPI_URL` (default unset): the local API of a CrowdSec engine
|
||||
- `SWW_ABUSEIPDB_DAILY_BUDGET` (default `900`): the most checks made in a day, a
|
||||
whole number above zero. AbuseIPDB's free accounts may make 1,000.
|
||||
- `SWW_CROWDSEC_LAPI_URL` (default unset): the local API of a CrowdSec engine
|
||||
you run, such as `http://172.17.0.1:8080`. An `http` or `https` URL without a
|
||||
user or a fragment; its decision list is at `v1/decisions` under it. Set
|
||||
without the key, it is refused. So is a decision list URL that is also in
|
||||
`SWWAF_BLOCKLIST_URLS`, or that is `SWWAF_ASN_LIMIT_PERCENT_URL`.
|
||||
- `SWWAF_CROWDSEC_LAPI_KEY` (default unset): the key the engine gave for
|
||||
`SWW_BLOCKLIST_URLS`, or that is `SWW_ASN_LIMIT_PERCENT_URL`.
|
||||
- `SWW_CROWDSEC_LAPI_KEY` (default unset): the key the engine gave for
|
||||
`smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it. Set without the
|
||||
URL, it is refused.
|
||||
- `SWWAF_REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:<percent>` or
|
||||
- `SWW_REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:<percent>` or
|
||||
`log`, for DNS blocklist and AbuseIPDB hits.
|
||||
- `SWWAF_REPUTATION_CACHE_TTL` (default `24h`): how long a zone's verdict or
|
||||
- `SWW_REPUTATION_CACHE_TTL` (default `24h`): how long a zone's verdict or
|
||||
AbuseIPDB's score is used. A duration.
|
||||
- `SWWAF_REPUTATION_TIMEOUT` (default `2s`): how long a query to a zone or a
|
||||
check with AbuseIPDB may take. A duration.
|
||||
- `SWW_REPUTATION_TIMEOUT` (default `2s`): how long a query to a zone or a check
|
||||
with AbuseIPDB may take. A duration.
|
||||
|
||||
## Alerting
|
||||
|
||||
- `SWWAF_ALERT_WEBHOOK_URL` (default unset): the webhook that receives each
|
||||
alert as a JSON object. An `http` or `https` URL without a user or a fragment.
|
||||
- `SWWAF_ALERT_WEBHOOK_HEADERS` (default empty): headers sent with each post,
|
||||
such as `Authorization:Bearer 0123456789abcdef`. A list of name, `:` and
|
||||
value. A value cannot hold a comma.
|
||||
- `SWWAF_ALERT_SLACK_WEBHOOK_URL` (default unset): the Slack incoming webhook
|
||||
that receives each alert as a message, checked as the webhook URL is.
|
||||
- `SWWAF_ALERT_NTFY_URL` (default unset): the ntfy topic that receives each
|
||||
alert, as the topic's full URL, such as `https://ntfy.sh/my-alerts`, checked
|
||||
as the webhook URL is.
|
||||
- `SWWAF_ALERT_NTFY_TOKEN` (default unset): an ntfy access token. A control
|
||||
- `SWW_ALERT_WEBHOOK_URL` (default unset): the webhook that receives each alert
|
||||
as a JSON object. An `http` or `https` URL without a user or a fragment.
|
||||
- `SWW_ALERT_WEBHOOK_HEADERS` (default empty): headers sent with each post, such
|
||||
as `Authorization:Bearer 0123456789abcdef`. A list of name, `:` and value. A
|
||||
value cannot hold a comma.
|
||||
- `SWW_ALERT_SLACK_WEBHOOK_URL` (default unset): the Slack incoming webhook that
|
||||
receives each alert as a message, checked as the webhook URL is.
|
||||
- `SWW_ALERT_NTFY_URL` (default unset): the ntfy topic that receives each alert,
|
||||
as the topic's full URL, such as `https://ntfy.sh/my-alerts`, checked as the
|
||||
webhook URL is.
|
||||
- `SWW_ALERT_NTFY_TOKEN` (default unset): an ntfy access token. A control
|
||||
character in it is refused.
|
||||
- `SWWAF_ALERT_EVENTS` (default
|
||||
`ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`):
|
||||
- `SWW_ALERT_EVENTS` (default
|
||||
`ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit,source_failure,file_error`):
|
||||
the event types alerts are sent for, from those in [`alerts.md`](alerts.md)
|
||||
but `summary`. Any other name is refused.
|
||||
- `SWWAF_ALERT_COOLDOWN` (default `15m`): how long after an alert is sent its
|
||||
- `SWW_ALERT_COOLDOWN` (default `15m`): how long after an alert is sent its
|
||||
repeats are held back. A duration or `off`.
|
||||
- `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): the most alerts sent in an hour. A
|
||||
- `SWW_ALERT_MAX_PER_HOUR` (default `60`): the most alerts sent in an hour. A
|
||||
whole number above zero, or `off`.
|
||||
|
||||
## Anomaly thresholds
|
||||
|
||||
Each scope has four thresholds: the most requests and the most bytes counted in
|
||||
a minute and in an hour. Requests are counts, bytes are sizes. Each is `off` by
|
||||
default. The four thresholds per client are:
|
||||
Each scope has four thresholds: the most requests and the most body bytes
|
||||
counted in a minute and in an hour. The bytes are those `SWW_BYTES_COUNT` names.
|
||||
Requests are counts, bytes are sizes. Each is `off` by default. The four
|
||||
thresholds per client are:
|
||||
|
||||
- `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`;
|
||||
- `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR`;
|
||||
- `SWWAF_ANOMALY_CLIENT_BYTES_PER_MINUTE`;
|
||||
- `SWWAF_ANOMALY_CLIENT_BYTES_PER_HOUR`.
|
||||
- `SWW_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`;
|
||||
- `SWW_ANOMALY_CLIENT_REQUESTS_PER_HOUR`;
|
||||
- `SWW_ANOMALY_CLIENT_BYTES_PER_MINUTE`;
|
||||
- `SWW_ANOMALY_CLIENT_BYTES_PER_HOUR`.
|
||||
|
||||
The other scopes' names have another prefix before the same four endings, such
|
||||
as `REQUESTS_PER_MINUTE`:
|
||||
|
||||
- `SWWAF_ANOMALY_NET_...`: per netblock around a client. Two settings give the
|
||||
- `SWW_ANOMALY_NET_...`: per netblock around a client. Two settings give the
|
||||
netblock's length:
|
||||
- `SWWAF_ANOMALY_NET_V4_PREFIX` (default `24`, from 0 to 32) for an IPv4
|
||||
- `SWW_ANOMALY_NET_V4_PREFIX` (default `24`, from 0 to 32) for an IPv4
|
||||
client;
|
||||
- `SWWAF_ANOMALY_NET_V6_PREFIX` (default `48`, from 0 to 128) for an IPv6
|
||||
- `SWW_ANOMALY_NET_V6_PREFIX` (default `48`, from 0 to 128) for an IPv6
|
||||
client.
|
||||
- `SWWAF_ANOMALY_ASN_...`: per AS number.
|
||||
- `SWWAF_ANOMALY_TOTAL_...`: the whole service.
|
||||
- `SWWAF_WATCH_...`: each netblock in `SWWAF_WATCH_NETS`, as a whole.
|
||||
`SWWAF_WATCH_NETS` (default empty) is a list of name, `=` and netblock, such
|
||||
as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. An item without `=` or a
|
||||
- `SWW_ANOMALY_ASN_...`: per AS number.
|
||||
- `SWW_ANOMALY_TOTAL_...`: the whole service.
|
||||
- `SWW_WATCH_...`: each netblock in `SWW_WATCH_NETS`, as a whole.
|
||||
`SWW_WATCH_NETS` (default empty) is a list of name, `=` and netblock, such as
|
||||
`office=203.0.113.0/24,scraper-x=198.51.100.0/22`. An item without `=` or a
|
||||
name, or with a bad netblock, is refused. So is a name used twice.
|
||||
|
||||
## Settings given as files
|
||||
@@ -312,7 +378,7 @@ Then mount its directory into the container read-only, and name the file:
|
||||
docker run \
|
||||
-v /srv/app/tokens:/etc/sww/tokens:ro \
|
||||
-e SWW_COMMAND=discriminator \
|
||||
-e SWWAF_METRICS_TOKEN_FILE=/etc/sww/tokens/metrics \
|
||||
-e SWW_METRICS_TOKEN_FILE=/etc/sww/tokens/metrics \
|
||||
...
|
||||
```
|
||||
|
||||
@@ -336,10 +402,10 @@ A few limits are fixed rather than settings:
|
||||
|
||||
## `discriminator`'s answer and `rproxy`'s wait
|
||||
|
||||
`SWWAF_LOOKUP_TIMEOUT` is the longest a question waits on an outside source.
|
||||
Keep it well under `rproxy`'s `SWRP_TIMEOUT_MS`, as the default is. No question
|
||||
waits for `SWWAF_REPUTATION_TIMEOUT`, since the DNS blocklists and AbuseIPDB are
|
||||
asked in the background (see "Answering in time" under "`discriminator`" in
|
||||
`SWW_LOOKUP_TIMEOUT` is the longest a question waits on an outside source. Keep
|
||||
it well under `rproxy`'s `SWRP_TIMEOUT_MS`, as the default is. No question waits
|
||||
for `SWW_REPUTATION_TIMEOUT`, since the DNS blocklists and AbuseIPDB are asked
|
||||
in the background (see "Answering in time" under "`discriminator`" in
|
||||
[`SPEC.md`](../SPEC.md)).
|
||||
|
||||
## `rproxy`
|
||||
@@ -392,14 +458,14 @@ check checks: `rproxy` or `discriminator` (see "Deployment" in
|
||||
|
||||
Nothing reads these names any more:
|
||||
|
||||
- `SWWAF_UPSTREAM_URL` is `SWRP_UPSTREAM_URL`.
|
||||
- `SWWAF_TRUSTED_PROXIES` is `SWRP_TRUSTED_PROXIES`.
|
||||
- `SWWAF_REQUEST_MAX_BYTES` is `SWRP_MAX_REQUEST_SIZE`.
|
||||
- `SWWAF_RESPONSE_MAX_BYTES` is `SWRP_MAX_RESPONSE_SIZE`.
|
||||
- `SWWAF_CLIENT_REQUEST_TIMEOUT` and `SWWAF_UPSTREAM_REQUEST_TIMEOUT` are
|
||||
- `SWW_UPSTREAM_URL` is `SWRP_UPSTREAM_URL`.
|
||||
- `SWW_TRUSTED_PROXIES` is `SWRP_TRUSTED_PROXIES`.
|
||||
- `SWW_REQUEST_MAX_BYTES` is `SWRP_MAX_REQUEST_SIZE`.
|
||||
- `SWW_RESPONSE_MAX_BYTES` is `SWRP_MAX_RESPONSE_SIZE`.
|
||||
- `SWW_CLIENT_REQUEST_TIMEOUT` and `SWW_UPSTREAM_REQUEST_TIMEOUT` are
|
||||
`SWRP_MAX_REQUEST_TIME_MS`, in milliseconds.
|
||||
- `SWWAF_CLIENT_RESPONSE_TIMEOUT` and `SWWAF_UPSTREAM_RESPONSE_TIMEOUT` are
|
||||
- `SWW_CLIENT_RESPONSE_TIMEOUT` and `SWW_UPSTREAM_RESPONSE_TIMEOUT` are
|
||||
`SWRP_MAX_RESPONSE_TIME_MS`, in milliseconds.
|
||||
- `SWWAF_CLIENT_IDLE_TIMEOUT` and `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` have
|
||||
no setting. `rproxy` closes a kept-open connection idle for 120 seconds, and
|
||||
- `SWW_CLIENT_IDLE_TIMEOUT` and `SWW_CLIENT_REQUEST_HEADER_MAX_BYTES` have no
|
||||
setting. `rproxy` closes a kept-open connection idle for 120 seconds, and
|
||||
takes a request line and headers of at most 32 KiB.
|
||||
+11
-10
@@ -1,13 +1,13 @@
|
||||
# State files
|
||||
|
||||
The five JSON files in `SWWAF_STATE_DIR`, and what each holds. "Persistent
|
||||
state" in [`SPEC.md`](../SPEC.md) says how they are written, read and edited.
|
||||
Each has a top-level `version` of 1.
|
||||
The five JSON files in `SWW_STATE_DIR`, and what each holds. "Persistent state"
|
||||
in [`SPEC.md`](../SPEC.md) says how they are written, read and edited. Each has
|
||||
a top-level `version` of 1.
|
||||
|
||||
## `bans.json`
|
||||
|
||||
Active and past bans: every ban an admin made, and up to `SWWAF_MAX_BANS` of
|
||||
those `discriminator` made. Per entry:
|
||||
Active and past bans: every ban an admin made, and up to `SWW_MAX_BANS` of those
|
||||
`discriminator` made. Per entry:
|
||||
|
||||
- `netblock`, `start` and `expires` (`null` for a permanent ban);
|
||||
- `cause`: `attack`, `limit`, `admin` or `crowdsec`;
|
||||
@@ -61,10 +61,11 @@ since it was first seen:
|
||||
- the first and last time seen;
|
||||
- the AS number, AS name and country last looked up, and when;
|
||||
- total requests, and how many were forwarded and how many refused;
|
||||
- bytes in each direction;
|
||||
- body bytes of its requests, and of the responses it was sent;
|
||||
- responses by status class;
|
||||
- offences by kind: `limit`, `attack`, `rule_blocked` and `waf_blocked`. A file
|
||||
may also hold `token_refused`, which is kept, and which nothing adds to.
|
||||
- offences by kind: `limit`, `attack`, `blocked_by_block_rule` and
|
||||
`blocked_by_core_rule_set`. A file may also hold `token_refused`, which is
|
||||
kept, and which nothing adds to.
|
||||
|
||||
## `lookups.json`
|
||||
|
||||
@@ -93,10 +94,10 @@ GeoJS answers, one per line:
|
||||
## Sizes
|
||||
|
||||
- `clients.json`: about 1 KiB per client, about 20 MiB at the default
|
||||
`SWWAF_MAX_TRACKED_CLIENTS`. Written every 15 minutes, that is under 2 GiB of
|
||||
`SWW_MAX_TRACKED_CLIENTS`. Written every 15 minutes, that is under 2 GiB of
|
||||
disk writes a day.
|
||||
- `bans.json`: about 1.2 KiB per ban, at most about 2.5 KiB, so about 6 MiB at
|
||||
the default `SWWAF_MAX_BANS`. Written every 15 minutes, a full file comes to
|
||||
the default `SWW_MAX_BANS`. Written every 15 minutes, a full file comes to
|
||||
about 600 MiB of writes a day, and a hundred new bans a day add about 600 MiB
|
||||
more.
|
||||
- `lookups.json`: about 150 bytes per answer, under 1.5 GiB of writes a day when
|
||||
|
||||
+35
-35
@@ -1,12 +1,12 @@
|
||||
// Package alerts sends alerts on bans, on traffic over an anomaly
|
||||
// threshold, on a source that fails and on a file with an error to each
|
||||
// destination set: to the webhook
|
||||
// SWWAF_ALERT_WEBHOOK_URL names, each as one JSON object, as "The JSON
|
||||
// SWW_ALERT_WEBHOOK_URL names, each as one JSON object, as "The JSON
|
||||
// object" in docs/alerts.md describes, to the Slack incoming
|
||||
// webhook SWWAF_ALERT_SLACK_WEBHOOK_URL names, as a message, and to the
|
||||
// ntfy topic SWWAF_ALERT_NTFY_URL names. A repeat within
|
||||
// SWWAF_ALERT_COOLDOWN is held back, and so is an alert past
|
||||
// SWWAF_ALERT_MAX_PER_HOUR, for the hour's summary. The others wait in a
|
||||
// webhook SWW_ALERT_SLACK_WEBHOOK_URL names, as a message, and to the
|
||||
// ntfy topic SWW_ALERT_NTFY_URL names. A repeat within
|
||||
// SWW_ALERT_COOLDOWN is held back, and so is an alert past
|
||||
// SWW_ALERT_MAX_PER_HOUR, for the hour's summary. The others wait in a
|
||||
// bounded queue of each destination's own, so that a destination that is
|
||||
// slow or unreachable holds up neither the others nor any request. The
|
||||
// state is written to alerts.json and read from it by the state package.
|
||||
@@ -34,7 +34,7 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// The events an alert is for, as SWWAF_ALERT_EVENTS names them.
|
||||
// The events an alert is for, as SWW_ALERT_EVENTS names them.
|
||||
const (
|
||||
// EventBan is a ban sww made.
|
||||
EventBan = "ban"
|
||||
@@ -44,10 +44,10 @@ const (
|
||||
// EventAnomaly is a count of requests or bytes over an anomaly
|
||||
// threshold.
|
||||
EventAnomaly = "anomaly"
|
||||
// EventWAFBlock is a request the Core Rule Set scored at or over
|
||||
// SWWAF_WAF_ANOMALY_THRESHOLD, refused in block mode, let through in
|
||||
// EventCoreRuleSetMatch is a request the Core Rule Set scored at or over
|
||||
// SWW_WAF_ANOMALY_THRESHOLD, refused in block mode, let through in
|
||||
// detect mode.
|
||||
EventWAFBlock = "waf_block"
|
||||
EventCoreRuleSetMatch = "core_rule_set_match"
|
||||
// EventReputationHit is a request whose client a blocklist, the
|
||||
// CrowdSec decision list or a DNSBL zone lists, or whose AbuseIPDB score
|
||||
// is a hit.
|
||||
@@ -61,18 +61,18 @@ const (
|
||||
// cannot be read, or a state file that cannot be written.
|
||||
EventFileError = "file_error"
|
||||
// EventSummary is the summary sent as an hour ends: of the alerts held
|
||||
// back in it past SWWAF_ALERT_MAX_PER_HOUR, and of the repeats held
|
||||
// back in it past SWW_ALERT_MAX_PER_HOUR, and of the repeats held
|
||||
// back by the cooldowns dropped as it ends, which no alert let through
|
||||
// has given. It is sent with SWWAF_ALERT_MAX_PER_HOUR off too, for
|
||||
// those repeats. SWWAF_ALERT_EVENTS does not name it.
|
||||
// has given. It is sent with SWW_ALERT_MAX_PER_HOUR off too, for
|
||||
// those repeats. SWW_ALERT_EVENTS does not name it.
|
||||
EventSummary = "summary"
|
||||
)
|
||||
|
||||
// Events returns every event SWWAF_ALERT_EVENTS can name, which is its
|
||||
// Events returns every event SWW_ALERT_EVENTS can name, which is its
|
||||
// default.
|
||||
func Events() []string {
|
||||
return []string{
|
||||
EventBan, EventPermanentBan, EventWAFBlock, EventAnomaly,
|
||||
EventBan, EventPermanentBan, EventCoreRuleSetMatch, EventAnomaly,
|
||||
EventReputationHit, EventSourceFailure, EventFileError,
|
||||
}
|
||||
}
|
||||
@@ -80,12 +80,12 @@ func Events() []string {
|
||||
// The destinations alerts are sent to, as the metrics and alerts.json
|
||||
// name them.
|
||||
const (
|
||||
// DestinationWebhook is the webhook SWWAF_ALERT_WEBHOOK_URL names.
|
||||
// DestinationWebhook is the webhook SWW_ALERT_WEBHOOK_URL names.
|
||||
DestinationWebhook = "webhook"
|
||||
// DestinationSlack is the Slack incoming webhook
|
||||
// SWWAF_ALERT_SLACK_WEBHOOK_URL names.
|
||||
// SWW_ALERT_SLACK_WEBHOOK_URL names.
|
||||
DestinationSlack = "slack"
|
||||
// DestinationNtfy is the ntfy topic SWWAF_ALERT_NTFY_URL names.
|
||||
// DestinationNtfy is the ntfy topic SWW_ALERT_NTFY_URL names.
|
||||
DestinationNtfy = "ntfy"
|
||||
)
|
||||
|
||||
@@ -121,26 +121,26 @@ var (
|
||||
// NtfyURL set, no alert is sent.
|
||||
type Params struct {
|
||||
// WebhookURL is where each alert is posted as JSON
|
||||
// (SWWAF_ALERT_WEBHOOK_URL), nil while it is unset. WebhookHeaders
|
||||
// are sent with each (SWWAF_ALERT_WEBHOOK_HEADERS).
|
||||
// (SWW_ALERT_WEBHOOK_URL), nil while it is unset. WebhookHeaders
|
||||
// are sent with each (SWW_ALERT_WEBHOOK_HEADERS).
|
||||
WebhookURL *url.URL
|
||||
WebhookHeaders http.Header
|
||||
// SlackURL is the Slack incoming webhook each alert is posted to as a
|
||||
// message (SWWAF_ALERT_SLACK_WEBHOOK_URL), nil while it is unset.
|
||||
// message (SWW_ALERT_SLACK_WEBHOOK_URL), nil while it is unset.
|
||||
SlackURL *url.URL
|
||||
// NtfyURL is the ntfy topic each alert is published to
|
||||
// (SWWAF_ALERT_NTFY_URL), nil while it is unset. NtfyToken, unless
|
||||
// empty, is sent with each as a bearer token (SWWAF_ALERT_NTFY_TOKEN).
|
||||
// (SWW_ALERT_NTFY_URL), nil while it is unset. NtfyToken, unless
|
||||
// empty, is sent with each as a bearer token (SWW_ALERT_NTFY_TOKEN).
|
||||
NtfyURL *url.URL
|
||||
NtfyToken string
|
||||
// Events are the events alerts are sent for (SWWAF_ALERT_EVENTS).
|
||||
// Events are the events alerts are sent for (SWW_ALERT_EVENTS).
|
||||
Events []string
|
||||
// Cooldown is how long a repeat of an alert is held back
|
||||
// (SWWAF_ALERT_COOLDOWN), 0 for no time. MaxPerHour is the most alerts
|
||||
// sent in an hour (SWWAF_ALERT_MAX_PER_HOUR), 0 for no limit.
|
||||
// (SWW_ALERT_COOLDOWN), 0 for no time. MaxPerHour is the most alerts
|
||||
// sent in an hour (SWW_ALERT_MAX_PER_HOUR), 0 for no limit.
|
||||
Cooldown time.Duration
|
||||
MaxPerHour int
|
||||
// Instance is SWWAF_INSTANCE_NAME, which every alert gives.
|
||||
// Instance is SWW_INSTANCE_NAME, which every alert gives.
|
||||
Instance string
|
||||
// Now tells the time of an alert, normally time.Now in UTC.
|
||||
Now func() time.Time
|
||||
@@ -301,17 +301,17 @@ func New(params Params) *Queue {
|
||||
}
|
||||
|
||||
if params.WebhookURL != nil {
|
||||
q.addDestination(DestinationWebhook, "SWWAF_ALERT_WEBHOOK_URL",
|
||||
q.addDestination(DestinationWebhook, "SWW_ALERT_WEBHOOK_URL",
|
||||
params.WebhookURL, q.webhookMessage)
|
||||
}
|
||||
|
||||
if params.SlackURL != nil {
|
||||
q.addDestination(DestinationSlack, "SWWAF_ALERT_SLACK_WEBHOOK_URL",
|
||||
q.addDestination(DestinationSlack, "SWW_ALERT_SLACK_WEBHOOK_URL",
|
||||
params.SlackURL, slackMessage)
|
||||
}
|
||||
|
||||
if params.NtfyURL != nil {
|
||||
q.addDestination(DestinationNtfy, "SWWAF_ALERT_NTFY_URL",
|
||||
q.addDestination(DestinationNtfy, "SWW_ALERT_NTFY_URL",
|
||||
params.NtfyURL, q.ntfyMessage)
|
||||
}
|
||||
|
||||
@@ -319,7 +319,7 @@ func New(params Params) *Queue {
|
||||
}
|
||||
|
||||
// Raise sends alert, which names its event and what is particular to it,
|
||||
// unless no destination is set or SWWAF_ALERT_EVENTS leaves its event
|
||||
// unless no destination is set or SWW_ALERT_EVENTS leaves its event
|
||||
// out. It gives alert the instance and the time. An alert that repeats
|
||||
// the last one let through less than Cooldown before is held back and
|
||||
// counted. The next one let through gives that count, unless an hour of
|
||||
@@ -363,7 +363,7 @@ func (q *Queue) Raise(alert Alert) {
|
||||
}
|
||||
|
||||
// WouldSend reports whether Raise would let an alert for event on
|
||||
// netblock through now: a destination is set, SWWAF_ALERT_EVENTS chooses
|
||||
// netblock through now: a destination is set, SWW_ALERT_EVENTS chooses
|
||||
// event, no alert for event on netblock was let through less than
|
||||
// Cooldown before, and fewer than MaxPerHour alerts have been let through
|
||||
// in the hour under way. Unlike Raise, it counts nothing.
|
||||
@@ -606,13 +606,13 @@ func (q *Queue) endHour(now time.Time) {
|
||||
|
||||
if heldBack > 0 {
|
||||
reasons = append(reasons, fmt.Sprintf("%d alerts held back in the hour from %s, "+
|
||||
"past the %d an hour SWWAF_ALERT_MAX_PER_HOUR allows", heldBack,
|
||||
"past the %d an hour SWW_ALERT_MAX_PER_HOUR allows", heldBack,
|
||||
q.hour.Start.Format(time.RFC3339), q.params.MaxPerHour))
|
||||
}
|
||||
|
||||
if repeats > 0 {
|
||||
reasons = append(reasons, fmt.Sprintf("%d repeats held back by "+
|
||||
"SWWAF_ALERT_COOLDOWN that no later alert gives", repeats))
|
||||
"SWW_ALERT_COOLDOWN that no later alert gives", repeats))
|
||||
}
|
||||
|
||||
if len(reasons) > 0 {
|
||||
@@ -871,7 +871,7 @@ func ntfyPriority(event string) string {
|
||||
return "high"
|
||||
case EventReputationHit:
|
||||
return "low"
|
||||
default: // ban, waf_block and summary
|
||||
default: // ban, core_rule_set_match and summary
|
||||
return "default"
|
||||
}
|
||||
}
|
||||
@@ -882,7 +882,7 @@ func ntfyTag(event string) string {
|
||||
switch event {
|
||||
case EventBan, EventPermanentBan:
|
||||
return "no_entry"
|
||||
case EventWAFBlock:
|
||||
case EventCoreRuleSetMatch:
|
||||
return "shield"
|
||||
case EventAnomaly:
|
||||
return "chart_with_upwards_trend"
|
||||
|
||||
@@ -132,7 +132,7 @@ func TestWouldSendOnlyForTheChosenEvents(t *testing.T) {
|
||||
|
||||
if q.WouldSend(alerts.EventBan, netblock(1)) ||
|
||||
q.WouldSend(alerts.EventPermanentBan, netblock(1)) {
|
||||
t.Error("a ban alert would be sent, though SWWAF_ALERT_EVENTS leaves it out")
|
||||
t.Error("a ban alert would be sent, though SWW_ALERT_EVENTS leaves it out")
|
||||
}
|
||||
|
||||
if !q.WouldSend(alerts.EventFileError, netip.Prefix{}) {
|
||||
@@ -184,8 +184,8 @@ func TestWouldSendWithOnlySlackOrOnlyNtfySet(t *testing.T) {
|
||||
onlyNtfy.NtfyURL = parseURL(ntfyURL)
|
||||
|
||||
for setting, params := range map[string]alerts.Params{
|
||||
"SWWAF_ALERT_SLACK_WEBHOOK_URL": onlySlack,
|
||||
"SWWAF_ALERT_NTFY_URL": onlyNtfy,
|
||||
"SWW_ALERT_SLACK_WEBHOOK_URL": onlySlack,
|
||||
"SWW_ALERT_NTFY_URL": onlyNtfy,
|
||||
} {
|
||||
if !alerts.New(params).WouldSend(alerts.EventBan, netblock(1)) {
|
||||
t.Errorf("a ban alert would not be sent with only %s set", setting)
|
||||
@@ -359,7 +359,7 @@ func TestAlertsPastTheHourlyLimitAreRolledIntoOneSummary(t *testing.T) {
|
||||
"as_name": "",
|
||||
"country": "",
|
||||
"reason": "3 alerts held back in the hour from 2000-01-01T00:00:00Z, " +
|
||||
"past the 2 an hour SWWAF_ALERT_MAX_PER_HOUR allows",
|
||||
"past the 2 an hour SWW_ALERT_MAX_PER_HOUR allows",
|
||||
"detail": map[string]any{
|
||||
"hour": started,
|
||||
"count": float64(3),
|
||||
@@ -525,7 +525,7 @@ func TestFailedRequestIsSentAgainWithBackoff(t *testing.T) {
|
||||
wantCounts(t, q, 0, int64(len(want)), 0, 0)
|
||||
|
||||
if !strings.Contains(log.String(),
|
||||
`"msg":"sending an alert to SWWAF_ALERT_WEBHOOK_URL failed"`) {
|
||||
`"msg":"sending an alert to SWW_ALERT_WEBHOOK_URL failed"`) {
|
||||
t.Errorf("process log %q names no failure", log.String())
|
||||
}
|
||||
|
||||
@@ -595,7 +595,7 @@ func TestRefusedAlertIsGivenUpAndTheNextSent(t *testing.T) {
|
||||
wantCounts(t, q, 1, 3, 0, 1)
|
||||
|
||||
if !strings.Contains(log.String(),
|
||||
`"msg":"gave up an alert SWWAF_ALERT_WEBHOOK_URL refused"`) {
|
||||
`"msg":"gave up an alert SWW_ALERT_WEBHOOK_URL refused"`) {
|
||||
t.Errorf("process log %q names no alert given up", log.String())
|
||||
}
|
||||
})
|
||||
@@ -619,7 +619,7 @@ func TestFailedRequestIsLoggedWithoutTheURL(t *testing.T) {
|
||||
|
||||
logged := log.String()
|
||||
if !strings.Contains(logged,
|
||||
`"msg":"sending an alert to SWWAF_ALERT_WEBHOOK_URL failed"`) ||
|
||||
`"msg":"sending an alert to SWW_ALERT_WEBHOOK_URL failed"`) ||
|
||||
strings.Contains(logged, "alerts.example") || strings.Contains(logged, "team=ops") {
|
||||
t.Errorf("process log %q names no failure, or names the URL", logged)
|
||||
}
|
||||
@@ -782,8 +782,8 @@ func TestSlackAndNtfyAreSentTheSummaryAndTheRepeatsHeldBack(t *testing.T) {
|
||||
}
|
||||
|
||||
const summary = "1 alerts held back in the hour from 2000-01-01T00:00:00Z, " +
|
||||
"past the 1 an hour SWWAF_ALERT_MAX_PER_HOUR allows; 1 repeats held back " +
|
||||
"by SWWAF_ALERT_COOLDOWN that no later alert gives\nsuppressed repeats: 1"
|
||||
"past the 1 an hour SWW_ALERT_MAX_PER_HOUR allows; 1 repeats held back " +
|
||||
"by SWW_ALERT_COOLDOWN that no later alert gives\nsuppressed repeats: 1"
|
||||
|
||||
wantSlackMessage(t, slack[1], "*"+instance+": summary*\n"+summary)
|
||||
wantNtfyMessage(t, ntfy[1], instance+": summary", "default bar_chart", summary)
|
||||
@@ -901,7 +901,7 @@ func TestDestinationThatDoesNotAnswerHoldsUpNeitherOther(t *testing.T) {
|
||||
|
||||
logged := log.String()
|
||||
if !strings.Contains(logged,
|
||||
`"msg":"sending an alert to SWWAF_ALERT_SLACK_WEBHOOK_URL failed"`) ||
|
||||
`"msg":"sending an alert to SWW_ALERT_SLACK_WEBHOOK_URL failed"`) ||
|
||||
strings.Contains(logged, "hooks.slack.example") ||
|
||||
strings.Contains(logged, "T0123") {
|
||||
t.Errorf("process log %q names no failure, or names the URL", logged)
|
||||
@@ -1254,7 +1254,7 @@ type eventMessage struct {
|
||||
title, priorityAndTag, text string
|
||||
}
|
||||
|
||||
// anAlertForEachEvent returns an alert for each event SWWAF_ALERT_EVENTS
|
||||
// anAlertForEachEvent returns an alert for each event SWW_ALERT_EVENTS
|
||||
// names, the first in observe mode, each with its message.
|
||||
func anAlertForEachEvent() []eventMessage {
|
||||
return []eventMessage{
|
||||
@@ -1281,12 +1281,12 @@ func anAlertForEachEvent() []eventMessage {
|
||||
},
|
||||
{
|
||||
alerts.Alert{
|
||||
Event: alerts.EventWAFBlock, Client: netip.MustParseAddr("192.0.2.1"),
|
||||
Event: alerts.EventCoreRuleSetMatch, Client: netip.MustParseAddr("192.0.2.1"),
|
||||
Netblock: netip.MustParsePrefix("192.0.2.1/32"),
|
||||
Reason: "refused by the Core Rule Set",
|
||||
Host: "app.example",
|
||||
},
|
||||
"fsn1app1/blog: waf_block", "default shield",
|
||||
"fsn1app1/blog: core_rule_set_match", "default shield",
|
||||
"refused by the Core Rule Set\nhost: app.example\nclient: 192.0.2.1\n" +
|
||||
"netblock: 192.0.2.1/32",
|
||||
},
|
||||
|
||||
+15
-15
@@ -28,22 +28,22 @@ const maxCounters = 20000
|
||||
// alerts name them.
|
||||
const (
|
||||
// ScopeClient is one client: an IPv4 address, or an IPv6 netblock of
|
||||
// SWWAF_IPV6_GROUP_PREFIX.
|
||||
// SWW_IPV6_GROUP_PREFIX.
|
||||
ScopeClient = "client"
|
||||
// ScopeNet is the netblock around a client, SWWAF_ANOMALY_NET_V4_PREFIX
|
||||
// or SWWAF_ANOMALY_NET_V6_PREFIX long.
|
||||
// ScopeNet is the netblock around a client, SWW_ANOMALY_NET_V4_PREFIX
|
||||
// or SWW_ANOMALY_NET_V6_PREFIX long.
|
||||
ScopeNet = "net"
|
||||
// ScopeASN is an AS number.
|
||||
ScopeASN = "asn"
|
||||
// ScopeTotal is the whole service.
|
||||
ScopeTotal = "total"
|
||||
// ScopeWatch is a named netblock of SWWAF_WATCH_NETS.
|
||||
ScopeWatch = "watch"
|
||||
// ScopeWatchedNet is a named netblock of SWW_WATCH_NETS.
|
||||
ScopeWatchedNet = "watched_net"
|
||||
)
|
||||
|
||||
// Scopes returns every scope.
|
||||
func Scopes() []string {
|
||||
return []string{ScopeClient, ScopeNet, ScopeASN, ScopeTotal, ScopeWatch}
|
||||
return []string{ScopeClient, ScopeNet, ScopeASN, ScopeTotal, ScopeWatchedNet}
|
||||
}
|
||||
|
||||
// The windows a counter counts in, as the alerts name them.
|
||||
@@ -62,7 +62,7 @@ type Thresholds struct {
|
||||
BytesPerHour int64
|
||||
}
|
||||
|
||||
// NamedNetblock is a netblock SWWAF_WATCH_NETS names.
|
||||
// NamedNetblock is a netblock SWW_WATCH_NETS names.
|
||||
type NamedNetblock struct {
|
||||
Name string
|
||||
Netblock netip.Prefix
|
||||
@@ -70,14 +70,14 @@ type NamedNetblock struct {
|
||||
|
||||
// Params are what New needs.
|
||||
type Params struct {
|
||||
// The thresholds of each scope: SWWAF_ANOMALY_CLIENT_*,
|
||||
// SWWAF_ANOMALY_NET_*, SWWAF_ANOMALY_ASN_*, SWWAF_ANOMALY_TOTAL_* and
|
||||
// SWWAF_WATCH_*.
|
||||
// The thresholds of each scope: SWW_ANOMALY_CLIENT_*,
|
||||
// SWW_ANOMALY_NET_*, SWW_ANOMALY_ASN_*, SWW_ANOMALY_TOTAL_* and
|
||||
// SWW_WATCH_*.
|
||||
Client, Net, ASN, Total, Watch Thresholds
|
||||
// NetV4Prefix and NetV6Prefix are the lengths of the netblock around a
|
||||
// client (SWWAF_ANOMALY_NET_V4_PREFIX and SWWAF_ANOMALY_NET_V6_PREFIX).
|
||||
// client (SWW_ANOMALY_NET_V4_PREFIX and SWW_ANOMALY_NET_V6_PREFIX).
|
||||
NetV4Prefix, NetV6Prefix int
|
||||
// NamedNetblocks are SWWAF_WATCH_NETS.
|
||||
// NamedNetblocks are SWW_WATCH_NETS.
|
||||
NamedNetblocks []NamedNetblock
|
||||
// Alerts receive the anomaly alerts.
|
||||
Alerts *alerts.Queue
|
||||
@@ -107,13 +107,13 @@ type Request struct {
|
||||
Host string
|
||||
// Client is the client's address, and ClientGroup the client it is
|
||||
// counted as: its IPv4 address, or the IPv6 netblock of
|
||||
// SWWAF_IPV6_GROUP_PREFIX its address is in.
|
||||
// SWW_IPV6_GROUP_PREFIX its address is in.
|
||||
Client netip.Addr
|
||||
ClientGroup netip.Prefix
|
||||
// ASN, ASName and Country are the client's as looked up, each "" when
|
||||
// unknown.
|
||||
ASN, ASName, Country string
|
||||
// Bytes are the request's bytes, as SWWAF_BYTES_COUNT counts them.
|
||||
// Bytes are the request's bytes, as SWW_BYTES_COUNT counts them.
|
||||
Bytes int64
|
||||
}
|
||||
|
||||
@@ -257,7 +257,7 @@ func (c *Counters) scopesOf(r Request) []scope {
|
||||
for _, named := range p.NamedNetblocks {
|
||||
if named.Netblock.Contains(client) {
|
||||
all = append(all, scope{
|
||||
key{scope: ScopeWatch, netblock: named.Netblock, name: named.Name}, p.Watch,
|
||||
key{scope: ScopeWatchedNet, netblock: named.Netblock, name: named.Name}, p.Watch,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
+59
-14
@@ -13,7 +13,7 @@ func TestBanWithoutACauseIsAnAdmins(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
netblock := netip.MustParsePrefix("203.0.113.0/24")
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{{Netblock: netblock, Start: midnight()}})
|
||||
|
||||
if got := ledger.Bans(netblock)[0].Cause; got != bans.CauseAdmin {
|
||||
@@ -24,9 +24,9 @@ func TestBanWithoutACauseIsAnAdmins(t *testing.T) {
|
||||
func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
rules := defaultRules()
|
||||
rules.MaxBans = 1
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBans = 1
|
||||
ledger := bans.New(params)
|
||||
adminsOnly := netip.MustParsePrefix("198.51.100.0/24")
|
||||
both := netip.MustParsePrefix("203.0.113.1/32")
|
||||
second := netip.MustParsePrefix("203.0.113.2/32")
|
||||
@@ -64,9 +64,9 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
|
||||
func TestEarlierBansCountABanDroppedAfterAnAdminsBan(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
rules := defaultRules()
|
||||
rules.MaxBans = 2
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBans = 2
|
||||
ledger := bans.New(params)
|
||||
netblock := netip.MustParsePrefix("203.0.113.1/32")
|
||||
other := netip.MustParsePrefix("203.0.113.2/32")
|
||||
|
||||
@@ -93,7 +93,7 @@ func TestEarlierBansCountABanDroppedAfterAnAdminsBan(t *testing.T) {
|
||||
func TestEarlierBansCountABanAnAdminKeepsOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.1/32")
|
||||
|
||||
// The netblock breaks a limit twice, the second time after its first
|
||||
@@ -117,7 +117,7 @@ func TestEarlierBansCountABanAnAdminKeepsOnce(t *testing.T) {
|
||||
func TestReasonOfTheBansSwwMakes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
|
||||
limit, _ := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.1/32"), midnight(),
|
||||
bans.Notes{Kind: "requests", Limit: 1000, Window: "minute"})
|
||||
@@ -150,7 +150,7 @@ func TestLiftedBanForALimitRefusesNothingAndMakesNoBanLonger(t *testing.T) {
|
||||
Lifted: midnight().Add(10 * time.Minute),
|
||||
}
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{lifted})
|
||||
|
||||
// While it would still last, it refuses nothing, and a limit broken
|
||||
@@ -181,7 +181,7 @@ func TestLiftedBanForAnAttackRefusesNothingAndMakesNoBanLonger(t *testing.T) {
|
||||
|
||||
// A permanent ban for a clear sign of attack, lifted.
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{{
|
||||
Netblock: netblock,
|
||||
Start: midnight(),
|
||||
@@ -211,7 +211,7 @@ func TestLiftedBanForAnAttackRefusesNothingAndMakesNoBanLonger(t *testing.T) {
|
||||
func TestLoadEditCountsTheBansAnAdminMade(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
made, _ := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.1/32"), midnight(),
|
||||
bans.Notes{})
|
||||
atStart := bans.Ban{
|
||||
@@ -246,7 +246,7 @@ func TestAdminsBanIsMadeWhileAnotherLasts(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
netblock := netip.MustParsePrefix("203.0.113.0/24")
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
|
||||
// An hour's ban for a broken limit.
|
||||
ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
||||
@@ -292,7 +292,7 @@ func TestLiftLiftsEveryActiveBanCoveringTheClient(t *testing.T) {
|
||||
wide := netip.MustParsePrefix("203.0.113.0/24")
|
||||
other := netip.MustParsePrefix("203.0.113.10/32")
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{
|
||||
// Ended an hour ago.
|
||||
{
|
||||
@@ -341,3 +341,48 @@ func TestLiftLiftsEveryActiveBanCoveringTheClient(t *testing.T) {
|
||||
|
||||
wantChanged(t, ledger, false)
|
||||
}
|
||||
|
||||
func TestPermanentBanOnAWiderNetblockIsReturnedOverATemporaryBan(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
client := netip.MustParseAddr("203.0.113.9")
|
||||
ledger := bans.New(defaultParams())
|
||||
|
||||
// The temporary ban is made first, so that its netblock is looked at first.
|
||||
ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"), midnight(), bans.Notes{})
|
||||
ledger.BanForAdmin(netip.MustParsePrefix("203.0.113.0/24"), midnight(), time.Time{},
|
||||
"for good")
|
||||
|
||||
wantPermanentBan(t, ledger, client, midnight().Add(time.Minute))
|
||||
}
|
||||
|
||||
func TestOlderPermanentBanIsReturnedOverANewerTemporaryBan(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
netblock := netip.MustParsePrefix("203.0.113.0/24")
|
||||
ledger := bans.New(defaultParams())
|
||||
|
||||
ledger.BanForAdmin(netblock, midnight(), time.Time{}, "for good")
|
||||
ledger.BanForAdmin(netblock, midnight().Add(time.Minute), midnight().Add(time.Hour),
|
||||
"for an hour")
|
||||
|
||||
wantPermanentBan(t, ledger, netblock.Addr(), midnight().Add(2*time.Minute))
|
||||
}
|
||||
|
||||
// wantPermanentBan checks that Find and Check return a permanent ban for
|
||||
// client at now.
|
||||
func wantPermanentBan(
|
||||
t *testing.T, ledger *bans.Ledger, client netip.Addr, now time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
for name, look := range map[string]func(netip.Addr, time.Time) (bans.Ban, bool, bool){
|
||||
"Find": ledger.Find,
|
||||
"Check": ledger.Check,
|
||||
} {
|
||||
ban, banned, _ := look(client, now)
|
||||
if !banned || !ban.Permanent() {
|
||||
t.Errorf("%s returned %+v and banned %v, want a permanent ban", name, ban, banned)
|
||||
}
|
||||
}
|
||||
}
|
||||
+43
-21
@@ -40,8 +40,8 @@ const repeatFactor = 3
|
||||
// maxTextBytes is how much of each text in a ban's notes is kept.
|
||||
const maxTextBytes = 256
|
||||
|
||||
// Rules are how long a ban lasts, and how many bans are held.
|
||||
type Rules struct {
|
||||
// Params are how long a ban lasts, and how many bans are held.
|
||||
type Params struct {
|
||||
// LimitBanDuration is how long a first ban for a broken limit lasts.
|
||||
LimitBanDuration time.Duration
|
||||
// LimitBanRepeatWindow is how soon after the end of the netblock's
|
||||
@@ -123,7 +123,7 @@ type Notes struct {
|
||||
LimitPercentSetting string `json:"limit_percent_setting,omitempty"`
|
||||
// RuleID and Target are, for a ban for a clear sign of attack, the id
|
||||
// of the rule file rule that matched, and its target; TrapPath is, for
|
||||
// one for a request for a path in SWWAF_TRAP_PATHS, that path as the
|
||||
// one for a request for a path in SWW_TRAP_PATHS, that path as the
|
||||
// client sent it.
|
||||
RuleID string `json:"rule_id,omitempty"`
|
||||
Target string `json:"target,omitempty"`
|
||||
@@ -180,7 +180,7 @@ type Request struct {
|
||||
|
||||
// Ledger holds the bans. It is safe for concurrent use.
|
||||
type Ledger struct {
|
||||
rules Rules
|
||||
params Params
|
||||
// changed receives a value when a ban is made, unless one is waiting
|
||||
// already.
|
||||
changed chan struct{}
|
||||
@@ -190,7 +190,7 @@ type Ledger struct {
|
||||
// Find make each netblock they find the most recently seen.
|
||||
netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
|
||||
// held is how many bans netblocks holds whose cause is not CauseAdmin,
|
||||
// at most rules.MaxBans.
|
||||
// at most params.MaxBans.
|
||||
held int
|
||||
// made is how many bans have been made since the start, by cause: by
|
||||
// the ledger, and by an admin, through BanForAdmin or in an edit of
|
||||
@@ -199,13 +199,13 @@ type Ledger struct {
|
||||
// v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6
|
||||
// netblocks that have been banned. Check looks for a ban at each of
|
||||
// them, so that a ban read from bans.json refuses every client in its
|
||||
// netblock even when it was made with another SWWAF_BAN_SCOPE_V4_PREFIX,
|
||||
// netblock even when it was made with another SWW_BAN_SCOPE_V4_PREFIX,
|
||||
// or another length of an IPv6 client's netblock.
|
||||
v4Lengths, v6Lengths []int
|
||||
}
|
||||
|
||||
// New returns a Ledger with no ban yet.
|
||||
func New(rules Rules) *Ledger {
|
||||
func New(params Params) *Ledger {
|
||||
// The ledger drops bans itself, and never those whose cause is
|
||||
// CauseAdmin, however many there are, so the LRU has no limit of its
|
||||
// own: it keeps the netblocks in the order they were last seen.
|
||||
@@ -215,7 +215,7 @@ func New(rules Rules) *Ledger {
|
||||
}
|
||||
|
||||
return &Ledger{
|
||||
rules: rules,
|
||||
params: params,
|
||||
changed: make(chan struct{}, 1),
|
||||
netblocks: netblocks,
|
||||
made: map[string]int{},
|
||||
@@ -274,16 +274,27 @@ func (l *Ledger) Find(client netip.Addr, now time.Time) (Ban, bool, bool) {
|
||||
|
||||
// activeBan returns the ban in bans, a netblock's bans oldest first, that
|
||||
// is active at now, or nil when none is. If several are, it returns the
|
||||
// one that started last. Every ban is looked at, since a ban an admin adds
|
||||
// to bans.json can start before the netblock's others and outlast them.
|
||||
// newest permanent one, or the newest when none is permanent. Every ban is
|
||||
// looked at, since a ban an admin adds to bans.json can start before the
|
||||
// netblock's others and outlast them.
|
||||
func activeBan(bans []Ban, now time.Time) *Ban {
|
||||
var newest *Ban
|
||||
|
||||
for i := len(bans) - 1; i >= 0; i-- {
|
||||
if bans[i].ActiveAt(now) {
|
||||
if !bans[i].ActiveAt(now) {
|
||||
continue
|
||||
}
|
||||
|
||||
if bans[i].Permanent() {
|
||||
return &bans[i]
|
||||
}
|
||||
|
||||
if newest == nil {
|
||||
newest = &bans[i]
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
return newest
|
||||
}
|
||||
|
||||
// BanForLimit bans netblock at now for a broken limit, with notes, and
|
||||
@@ -743,13 +754,16 @@ func (l *Ledger) markChanged() {
|
||||
}
|
||||
|
||||
// active returns the ban active at now on a netblock client is in, or
|
||||
// nil.
|
||||
// nil. If several netblocks have one, it returns a permanent ban when
|
||||
// there is one.
|
||||
func (l *Ledger) active(client netip.Addr, now time.Time) *Ban {
|
||||
lengths := l.v6Lengths
|
||||
if client.Is4() {
|
||||
lengths = l.v4Lengths
|
||||
}
|
||||
|
||||
var first *Ban
|
||||
|
||||
for _, length := range lengths {
|
||||
bans, found := l.netblocks.Get(netip.PrefixFrom(client, length).Masked())
|
||||
if !found {
|
||||
@@ -757,12 +771,20 @@ func (l *Ledger) active(client netip.Addr, now time.Time) *Ban {
|
||||
}
|
||||
|
||||
ban := activeBan(*bans, now)
|
||||
if ban != nil {
|
||||
if ban == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
if ban.Permanent() {
|
||||
return ban
|
||||
}
|
||||
|
||||
if first == nil {
|
||||
first = ban
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
return first
|
||||
}
|
||||
|
||||
// covering returns the bans of each netblock held that client is in,
|
||||
@@ -790,7 +812,7 @@ func (l *Ledger) covering(client netip.Addr) []*[]Ban {
|
||||
// unless ban's cause is CauseAdmin, which does not count toward MaxBans.
|
||||
func (l *Ledger) add(ban Ban) {
|
||||
counted := ban.Cause != CauseAdmin
|
||||
if counted && l.held == l.rules.MaxBans {
|
||||
if counted && l.held == l.params.MaxBans {
|
||||
l.dropOne()
|
||||
}
|
||||
|
||||
@@ -825,7 +847,7 @@ func (l *Ledger) add(ban Ban) {
|
||||
// new ban longer. A ban an admin adds to bans.json can start after
|
||||
// another and end before it, so that one is looked for among them all.
|
||||
func (l *Ledger) limitExpiry(held []Ban, now time.Time) time.Time {
|
||||
length := l.rules.LimitBanDuration
|
||||
length := l.params.LimitBanDuration
|
||||
|
||||
var last *Ban
|
||||
|
||||
@@ -836,18 +858,18 @@ func (l *Ledger) limitExpiry(held []Ban, now time.Time) time.Time {
|
||||
}
|
||||
}
|
||||
|
||||
if last != nil && now.Sub(last.Expires) <= l.rules.LimitBanRepeatWindow {
|
||||
if last != nil && now.Sub(last.Expires) <= l.params.LimitBanRepeatWindow {
|
||||
lastLength := last.Expires.Sub(last.Start)
|
||||
// This is repeatFactor * lastLength > MaxBanDuration, written so
|
||||
// that it cannot overflow.
|
||||
if lastLength > l.rules.MaxBanDuration/repeatFactor {
|
||||
if lastLength > l.params.MaxBanDuration/repeatFactor {
|
||||
return time.Time{}
|
||||
}
|
||||
|
||||
length = repeatFactor * lastLength
|
||||
}
|
||||
|
||||
if length > l.rules.MaxBanDuration {
|
||||
if length > l.params.MaxBanDuration {
|
||||
return time.Time{}
|
||||
}
|
||||
|
||||
@@ -865,7 +887,7 @@ func (l *Ledger) attackExpiry(held []Ban, now time.Time) time.Time {
|
||||
}
|
||||
}
|
||||
|
||||
return now.Add(l.rules.AttackBanDuration)
|
||||
return now.Add(l.params.AttackBanDuration)
|
||||
}
|
||||
|
||||
// dropOne drops the earliest ban whose cause is not CauseAdmin of the
|
||||
|
||||
+27
-27
@@ -15,7 +15,7 @@ const day = 24 * time.Hour
|
||||
func TestRepeatsTripleUntilPermanent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
now := midnight()
|
||||
|
||||
@@ -62,7 +62,7 @@ func TestRepeatWindowRunsOut(t *testing.T) {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
||||
@@ -80,9 +80,9 @@ func TestRepeatWindowRunsOut(t *testing.T) {
|
||||
func TestFirstBanLongerThanTheMaximumIsPermanent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
rules := defaultRules()
|
||||
rules.LimitBanDuration = rules.MaxBanDuration + time.Hour
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.LimitBanDuration = params.MaxBanDuration + time.Hour
|
||||
ledger := bans.New(params)
|
||||
|
||||
ban, _ := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"), midnight(),
|
||||
bans.Notes{})
|
||||
@@ -97,9 +97,9 @@ func TestLongestBanSetFarOffDoesNotOverflow(t *testing.T) {
|
||||
// With bans of up to 100,000 days, the 14th ban in a row, of 3^13
|
||||
// hours, is within the maximum, and three times as long would not fit
|
||||
// in a time.Duration. The 15th is permanent.
|
||||
rules := defaultRules()
|
||||
rules.MaxBanDuration = 100000 * day
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBanDuration = 100000 * day
|
||||
ledger := bans.New(params)
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
now := midnight()
|
||||
|
||||
@@ -121,7 +121,7 @@ func TestLongestBanSetFarOffDoesNotOverflow(t *testing.T) {
|
||||
func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
first, made := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
||||
@@ -146,7 +146,7 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
|
||||
func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
ban, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{Requests: 5})
|
||||
|
||||
@@ -178,7 +178,7 @@ func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
|
||||
func TestFindCountsNothing(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
ban, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{Requests: 5})
|
||||
|
||||
@@ -200,9 +200,9 @@ func TestFindCountsNothing(t *testing.T) {
|
||||
func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
rules := defaultRules()
|
||||
rules.MaxBans = 3
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBans = 3
|
||||
ledger := bans.New(params)
|
||||
a := netip.MustParsePrefix("203.0.113.1/32")
|
||||
b := netip.MustParsePrefix("203.0.113.2/32")
|
||||
c := netip.MustParsePrefix("203.0.113.3/32")
|
||||
@@ -239,9 +239,9 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
|
||||
|
||||
// With room for one ban, the netblock's ended ban goes to make room for
|
||||
// its new one, whose notes still count it.
|
||||
rules := defaultRules()
|
||||
rules.MaxBans = 1
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBans = 1
|
||||
ledger := bans.New(params)
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
||||
@@ -258,7 +258,7 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
|
||||
func TestRequestDuringAnAttackBanMakesItPermanent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
notes := bans.Notes{RuleID: "env-file", Target: "path"}
|
||||
|
||||
@@ -304,7 +304,7 @@ func TestRequestDuringAnAttackBanMakesItPermanent(t *testing.T) {
|
||||
func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
// A ban for a broken limit before does not count.
|
||||
@@ -336,7 +336,7 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
|
||||
func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
||||
wantChanged(t, ledger, true)
|
||||
@@ -380,7 +380,7 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
|
||||
func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
now := midnight()
|
||||
|
||||
@@ -414,7 +414,7 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
|
||||
func TestAttackBanDoesNotLengthenTheNextBanForALimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
// Three times the seven days would be permanent; a limit broken as the
|
||||
@@ -436,7 +436,7 @@ func TestAttackBanDoesNotLengthenTheNextBanForALimit(t *testing.T) {
|
||||
func TestRequestTextsAreCutTo256Bytes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
long := strings.Repeat("a", 300)
|
||||
request := bans.Request{
|
||||
@@ -458,7 +458,7 @@ func TestRequestTextsAreCutTo256Bytes(t *testing.T) {
|
||||
func TestLookupFillsTheNotesOfTheNetblocksBansWithoutOne(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
other := netip.MustParsePrefix("198.51.100.7/32")
|
||||
|
||||
@@ -494,9 +494,9 @@ func TestLookupFillsTheNotesOfTheNetblocksBansWithoutOne(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// defaultRules are the rules at the settings' defaults.
|
||||
func defaultRules() bans.Rules {
|
||||
return bans.Rules{
|
||||
// defaultParams are the params at the settings' defaults.
|
||||
func defaultParams() bans.Params {
|
||||
return bans.Params{
|
||||
LimitBanDuration: time.Hour,
|
||||
LimitBanRepeatWindow: day,
|
||||
MaxBanDuration: 7 * day,
|
||||
|
||||
@@ -15,7 +15,7 @@ const scenario = "crowdsecurity/ssh-bf"
|
||||
func TestCrowdSecBanLastsUntilTheDecisionEnds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
expires := midnight().Add(4 * time.Hour)
|
||||
|
||||
@@ -49,7 +49,7 @@ func TestCrowdSecBanLastsUntilTheDecisionEnds(t *testing.T) {
|
||||
func TestCrowdSecBanIsNeverMadePermanent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
expires := midnight().Add(4 * time.Hour)
|
||||
ledger.BanForCrowdSec(netblock, midnight(), expires, scenario, bans.Notes{})
|
||||
@@ -72,7 +72,7 @@ func TestCrowdSecBanIsNeverMadePermanent(t *testing.T) {
|
||||
func TestCrowdSecBanIsCountedAndDoesNotLengthenTheNextBanForALimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
// Three times the three days would be permanent; a limit broken as the
|
||||
|
||||
@@ -14,7 +14,7 @@ import (
|
||||
func TestChangedAfterABanIsMade(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
wantChanged(t, ledger, false)
|
||||
@@ -38,7 +38,7 @@ func TestChangedAfterABanIsMade(t *testing.T) {
|
||||
func TestSnapshotListsEveryBanByNetblock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
v6 := netip.MustParsePrefix("2001:db8::/64")
|
||||
high := netip.MustParsePrefix("203.0.113.10/32")
|
||||
low := netip.MustParsePrefix("203.0.113.9/32")
|
||||
@@ -67,14 +67,14 @@ func TestSnapshotListsEveryBanByNetblock(t *testing.T) {
|
||||
func TestLoadedBansCarryOn(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
before := bans.New(defaultRules())
|
||||
before := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
ban, _ := before.BanForLimit(netblock, midnight(), bans.Notes{Limit: 1})
|
||||
|
||||
// Loaded into a new ledger, as across a restart, the ban still refuses
|
||||
// while it lasts, and once it has ended a broken limit bans for three
|
||||
// times as long, with the loaded ban counted among the earlier ones.
|
||||
after := bans.New(defaultRules())
|
||||
after := bans.New(defaultParams())
|
||||
after.Load(before.Snapshot())
|
||||
|
||||
_, banned, _ := after.Check(netblock.Addr(), ban.Expires.Add(-time.Second))
|
||||
@@ -96,7 +96,7 @@ func TestLoadedBanRefusesEveryClientInItsNetblock(t *testing.T) {
|
||||
// Two entries as an admin might write them, with addresses not masked
|
||||
// to their lengths, the IPv6 one shorter than the /64 an IPv6 client's
|
||||
// ban covers, beside a ban the ledger makes on one IPv4 address.
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{
|
||||
{Netblock: netip.MustParsePrefix("203.0.113.9/24"), Start: midnight()},
|
||||
{Netblock: netip.MustParsePrefix("2001:db8::1/48"), Start: midnight()},
|
||||
@@ -145,7 +145,7 @@ func TestPermanentBanStartedBeforeAnEndedOneRefuses(t *testing.T) {
|
||||
Expires: midnight().Add(time.Hour),
|
||||
}
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{permanent, ended})
|
||||
|
||||
now := midnight().Add(2 * time.Hour)
|
||||
@@ -189,7 +189,7 @@ func TestNextBanWorkedOutFromTheBanThatEndedLast(t *testing.T) {
|
||||
Expires: midnight().Add(2 * time.Hour),
|
||||
}
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{nineHours, admins})
|
||||
|
||||
// Once both have ended, a limit broken within the repeat window bans
|
||||
@@ -219,9 +219,9 @@ func TestLoadKeepsAtMostMaxBansDroppingTheEarliest(t *testing.T) {
|
||||
Cause: bans.CauseLimit,
|
||||
}
|
||||
|
||||
rules := defaultRules()
|
||||
rules.MaxBans = 1
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBans = 1
|
||||
ledger := bans.New(params)
|
||||
ledger.Load([]bans.Ban{later, earlier})
|
||||
|
||||
held := ledger.Snapshot()
|
||||
@@ -235,9 +235,9 @@ func TestLoadReplacesTheBansHeld(t *testing.T) {
|
||||
|
||||
// Room for three bans, so that the second load, were it added to the
|
||||
// two bans held, would drop none of them to make room.
|
||||
rules := defaultRules()
|
||||
rules.MaxBans = 3
|
||||
ledger := bans.New(rules)
|
||||
params := defaultParams()
|
||||
params.MaxBans = 3
|
||||
ledger := bans.New(params)
|
||||
kept := bans.Ban{
|
||||
Netblock: netip.MustParsePrefix("2001:db8::/64"),
|
||||
Start: midnight(),
|
||||
@@ -285,7 +285,7 @@ func TestLoadCutsTheTextsTo256Bytes(t *testing.T) {
|
||||
}},
|
||||
}
|
||||
|
||||
ledger := bans.New(defaultRules())
|
||||
ledger := bans.New(defaultParams())
|
||||
ledger.Load([]bans.Ban{ban})
|
||||
|
||||
cut := long[:256]
|
||||
|
||||
+682
-279
File diff suppressed because it is too large.
Load diff
+415
-154
@@ -23,98 +23,100 @@ import (
|
||||
|
||||
// The settings, by name.
|
||||
const (
|
||||
listenAddr = "SWWAF_LISTEN_ADDR"
|
||||
mode = "SWWAF_MODE"
|
||||
ipv6GroupPrefix = "SWWAF_IPV6_GROUP_PREFIX"
|
||||
maxTrackedClients = "SWWAF_MAX_TRACKED_CLIENTS"
|
||||
allowNets = "SWWAF_ALLOW_NETS"
|
||||
rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS"
|
||||
denyNets = "SWWAF_DENY_NETS"
|
||||
rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE"
|
||||
rateLimitPerHour = "SWWAF_RATE_LIMIT_PER_HOUR"
|
||||
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
|
||||
rateLimitExemptPaths = "SWWAF_RATE_LIMIT_EXEMPT_PATHS"
|
||||
bytesLimitPerMinute = "SWWAF_BYTES_LIMIT_PER_MINUTE"
|
||||
bytesLimitPerHour = "SWWAF_BYTES_LIMIT_PER_HOUR"
|
||||
bytesLimitPerDay = "SWWAF_BYTES_LIMIT_PER_DAY"
|
||||
bytesCount = "SWWAF_BYTES_COUNT"
|
||||
lookupSource = "SWWAF_LOOKUP_SOURCE"
|
||||
lookupDBPath = "SWWAF_LOOKUP_DB_PATH"
|
||||
lookupTimeout = "SWWAF_LOOKUP_TIMEOUT"
|
||||
addLookupHeaders = "SWWAF_ADD_LOOKUP_HEADERS"
|
||||
deniedCountries = "SWWAF_DENIED_COUNTRIES"
|
||||
allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
|
||||
asnLimitPercent = "SWWAF_ASN_LIMIT_PERCENT"
|
||||
countryLimitPercent = "SWWAF_COUNTRY_LIMIT_PERCENT"
|
||||
asnBytesPercent = "SWWAF_ASN_BYTES_PERCENT"
|
||||
countryBytesPercent = "SWWAF_COUNTRY_BYTES_PERCENT"
|
||||
unknownLimitPercent = "SWWAF_UNKNOWN_LIMIT_PERCENT"
|
||||
asnLimitPercentURL = "SWWAF_ASN_LIMIT_PERCENT_URL"
|
||||
blocklistURLs = "SWWAF_BLOCKLIST_URLS"
|
||||
blocklistRefresh = "SWWAF_BLOCKLIST_REFRESH"
|
||||
blocklistAction = "SWWAF_BLOCKLIST_ACTION"
|
||||
dnsblZones = "SWWAF_DNSBL_ZONES"
|
||||
dnsblResolver = "SWWAF_DNSBL_RESOLVER"
|
||||
abuseIPDBKey = "SWWAF_ABUSEIPDB_KEY"
|
||||
abuseIPDBMinScore = "SWWAF_ABUSEIPDB_MIN_SCORE"
|
||||
abuseIPDBDailyBudget = "SWWAF_ABUSEIPDB_DAILY_BUDGET"
|
||||
reputationAction = "SWWAF_REPUTATION_ACTION"
|
||||
reputationCacheTTL = "SWWAF_REPUTATION_CACHE_TTL"
|
||||
reputationTimeout = "SWWAF_REPUTATION_TIMEOUT"
|
||||
crowdSecURL = "SWWAF_CROWDSEC_LAPI_URL"
|
||||
crowdSecKey = "SWWAF_CROWDSEC_LAPI_KEY"
|
||||
banResponse = "SWWAF_BAN_RESPONSE"
|
||||
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
|
||||
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
|
||||
maxBanDuration = "SWWAF_MAX_BAN_DURATION"
|
||||
attackBanDuration = "SWWAF_ATTACK_BAN_DURATION"
|
||||
maxBans = "SWWAF_MAX_BANS"
|
||||
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
|
||||
stateDir = "SWWAF_STATE_DIR"
|
||||
stateWriteDelay = "SWWAF_STATE_WRITE_DELAY"
|
||||
stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL"
|
||||
adminToken = "SWWAF_ADMIN_TOKEN" //nolint:gosec // the setting's name
|
||||
metricsToken = "SWWAF_METRICS_TOKEN" //nolint:gosec // the setting's name
|
||||
metricsTopN = "SWWAF_METRICS_TOP_N"
|
||||
instanceName = "SWWAF_INSTANCE_NAME"
|
||||
logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS"
|
||||
logLevel = "SWWAF_LOG_LEVEL"
|
||||
rulesDir = "SWWAF_RULES_DIR"
|
||||
rulesEnabled = "SWWAF_RULES_ENABLED"
|
||||
wafMode = "SWWAF_WAF_MODE"
|
||||
wafParanoiaLevel = "SWWAF_WAF_PARANOIA_LEVEL"
|
||||
wafAnomalyThreshold = "SWWAF_WAF_ANOMALY_THRESHOLD"
|
||||
wafDisabledRules = "SWWAF_WAF_DISABLED_RULES"
|
||||
wafExemptPaths = "SWWAF_WAF_EXEMPT_PATHS"
|
||||
wafBodyLimit = "SWWAF_WAF_BODY_LIMIT"
|
||||
trapPaths = "SWWAF_TRAP_PATHS"
|
||||
errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD"
|
||||
logRemoteURL = "SWWAF_LOG_REMOTE_URL"
|
||||
logRemoteTLSCAFile = "SWWAF_LOG_REMOTE_TLS_CA_FILE"
|
||||
logRemoteBuffer = "SWWAF_LOG_REMOTE_BUFFER"
|
||||
logRemoteFacility = "SWWAF_LOG_REMOTE_FACILITY"
|
||||
logRemoteAppName = "SWWAF_LOG_REMOTE_APP_NAME"
|
||||
alertWebhookURL = "SWWAF_ALERT_WEBHOOK_URL"
|
||||
alertWebhookHeaders = "SWWAF_ALERT_WEBHOOK_HEADERS"
|
||||
alertSlackWebhookURL = "SWWAF_ALERT_SLACK_WEBHOOK_URL"
|
||||
alertNtfyURL = "SWWAF_ALERT_NTFY_URL"
|
||||
alertNtfyToken = "SWWAF_ALERT_NTFY_TOKEN" //nolint:gosec // the setting's name
|
||||
alertEvents = "SWWAF_ALERT_EVENTS"
|
||||
alertCooldown = "SWWAF_ALERT_COOLDOWN"
|
||||
alertMaxPerHour = "SWWAF_ALERT_MAX_PER_HOUR"
|
||||
anomalyNetV4Prefix = "SWWAF_ANOMALY_NET_V4_PREFIX"
|
||||
anomalyNetV6Prefix = "SWWAF_ANOMALY_NET_V6_PREFIX"
|
||||
watchNets = "SWWAF_WATCH_NETS"
|
||||
listenAddr = "SWW_LISTEN_ADDR"
|
||||
mode = "SWW_MODE"
|
||||
ipv6GroupPrefix = "SWW_IPV6_GROUP_PREFIX"
|
||||
maxTrackedClients = "SWW_MAX_TRACKED_CLIENTS"
|
||||
allowNets = "SWW_ALLOW_NETS"
|
||||
rateLimitExemptNets = "SWW_RATE_LIMIT_EXEMPT_NETS"
|
||||
denyNets = "SWW_DENY_NETS"
|
||||
rateLimitPerMinute = "SWW_RATE_LIMIT_PER_MINUTE"
|
||||
rateLimitPerHour = "SWW_RATE_LIMIT_PER_HOUR"
|
||||
rateLimitPerDay = "SWW_RATE_LIMIT_PER_DAY"
|
||||
rateLimitExemptPaths = "SWW_RATE_LIMIT_EXEMPT_PATHS"
|
||||
bytesLimitPerMinute = "SWW_BYTES_LIMIT_PER_MINUTE"
|
||||
bytesLimitPerHour = "SWW_BYTES_LIMIT_PER_HOUR"
|
||||
bytesLimitPerDay = "SWW_BYTES_LIMIT_PER_DAY"
|
||||
bytesCount = "SWW_BYTES_COUNT"
|
||||
lookupSource = "SWW_LOOKUP_SOURCE"
|
||||
lookupDBPath = "SWW_LOOKUP_DB_PATH"
|
||||
lookupTimeout = "SWW_LOOKUP_TIMEOUT"
|
||||
addLookupHeaders = "SWW_ADD_LOOKUP_HEADERS"
|
||||
allowedCountries = "SWW_EXCLUSIVELY_ALLOWED_COUNTRIES"
|
||||
tagLists = "SWW_TAG_LISTS"
|
||||
tagDeny = "SWW_TAG_DENY"
|
||||
tagLimitPercent = "SWW_TAG_LIMIT_PERCENT"
|
||||
asnBytesPercent = "SWW_ASN_BYTES_PERCENT"
|
||||
countryBytesPercent = "SWW_COUNTRY_BYTES_PERCENT"
|
||||
unknownLimitPercent = "SWW_UNKNOWN_LIMIT_PERCENT"
|
||||
asnLimitPercentURL = "SWW_ASN_LIMIT_PERCENT_URL"
|
||||
blocklistURLs = "SWW_BLOCKLIST_URLS"
|
||||
blocklistRefresh = "SWW_BLOCKLIST_REFRESH"
|
||||
blocklistAction = "SWW_BLOCKLIST_ACTION"
|
||||
dnsblZones = "SWW_DNSBL_ZONES"
|
||||
dnsblResolver = "SWW_DNSBL_RESOLVER"
|
||||
abuseIPDBKey = "SWW_ABUSEIPDB_KEY"
|
||||
abuseIPDBMinScore = "SWW_ABUSEIPDB_MIN_SCORE"
|
||||
abuseIPDBDailyBudget = "SWW_ABUSEIPDB_DAILY_BUDGET"
|
||||
reputationAction = "SWW_REPUTATION_ACTION"
|
||||
reputationCacheTTL = "SWW_REPUTATION_CACHE_TTL"
|
||||
reputationTimeout = "SWW_REPUTATION_TIMEOUT"
|
||||
routeWatchURL = "SWW_ROUTEWATCH_URL"
|
||||
crowdSecURL = "SWW_CROWDSEC_LAPI_URL"
|
||||
crowdSecKey = "SWW_CROWDSEC_LAPI_KEY"
|
||||
banResponse = "SWW_BAN_RESPONSE"
|
||||
limitBanDuration = "SWW_LIMIT_BAN_DURATION"
|
||||
limitBanRepeatWindow = "SWW_LIMIT_BAN_REPEAT_WINDOW"
|
||||
maxBanDuration = "SWW_MAX_BAN_DURATION"
|
||||
attackBanDuration = "SWW_ATTACK_BAN_DURATION"
|
||||
maxBans = "SWW_MAX_BANS"
|
||||
banScopeV4Prefix = "SWW_BAN_SCOPE_V4_PREFIX"
|
||||
stateDir = "SWW_STATE_DIR"
|
||||
stateWriteDelay = "SWW_STATE_WRITE_DELAY"
|
||||
stateCounterInterval = "SWW_STATE_COUNTER_INTERVAL"
|
||||
adminToken = "SWW_ADMIN_TOKEN" //nolint:gosec // the setting's name
|
||||
metricsToken = "SWW_METRICS_TOKEN" //nolint:gosec // the setting's name
|
||||
metricsTopN = "SWW_METRICS_TOP_N"
|
||||
instanceName = "SWW_INSTANCE_NAME"
|
||||
logRequestHeaders = "SWW_LOG_REQUEST_HEADERS"
|
||||
logLevel = "SWW_LOG_LEVEL"
|
||||
rulesDir = "SWW_RULES_DIR"
|
||||
rulesEnabled = "SWW_RULES_ENABLED"
|
||||
wafMode = "SWW_WAF_MODE"
|
||||
wafParanoiaLevel = "SWW_WAF_PARANOIA_LEVEL"
|
||||
wafAnomalyThreshold = "SWW_WAF_ANOMALY_THRESHOLD"
|
||||
wafDisabledRules = "SWW_WAF_DISABLED_RULES"
|
||||
wafExemptPaths = "SWW_WAF_EXEMPT_PATHS"
|
||||
wafBodyLimit = "SWW_WAF_BODY_LIMIT"
|
||||
exclusionsFile = "SWW_CORE_RULE_SET_EXCLUSIONS_FILE"
|
||||
trapPaths = "SWW_TRAP_PATHS"
|
||||
errorBurstThreshold = "SWW_ERROR_BURST_THRESHOLD"
|
||||
logRemoteURL = "SWW_LOG_REMOTE_URL"
|
||||
logRemoteTLSCAFile = "SWW_LOG_REMOTE_TLS_CA_FILE"
|
||||
logRemoteBuffer = "SWW_LOG_REMOTE_BUFFER"
|
||||
logRemoteFacility = "SWW_LOG_REMOTE_FACILITY"
|
||||
logRemoteAppName = "SWW_LOG_REMOTE_APP_NAME"
|
||||
alertWebhookURL = "SWW_ALERT_WEBHOOK_URL"
|
||||
alertWebhookHeaders = "SWW_ALERT_WEBHOOK_HEADERS"
|
||||
alertSlackWebhookURL = "SWW_ALERT_SLACK_WEBHOOK_URL"
|
||||
alertNtfyURL = "SWW_ALERT_NTFY_URL"
|
||||
alertNtfyToken = "SWW_ALERT_NTFY_TOKEN" //nolint:gosec // the setting's name
|
||||
alertEvents = "SWW_ALERT_EVENTS"
|
||||
alertCooldown = "SWW_ALERT_COOLDOWN"
|
||||
alertMaxPerHour = "SWW_ALERT_MAX_PER_HOUR"
|
||||
anomalyNetV4Prefix = "SWW_ANOMALY_NET_V4_PREFIX"
|
||||
anomalyNetV6Prefix = "SWW_ANOMALY_NET_V6_PREFIX"
|
||||
watchNets = "SWW_WATCH_NETS"
|
||||
)
|
||||
|
||||
// The anomaly thresholds: each of the prefixes below, which name a scope,
|
||||
// followed by each of the four ends.
|
||||
const (
|
||||
anomalyClient = "SWWAF_ANOMALY_CLIENT_"
|
||||
anomalyNet = "SWWAF_ANOMALY_NET_"
|
||||
anomalyASN = "SWWAF_ANOMALY_ASN_"
|
||||
anomalyTotal = "SWWAF_ANOMALY_TOTAL_"
|
||||
watch = "SWWAF_WATCH_"
|
||||
anomalyClient = "SWW_ANOMALY_CLIENT_"
|
||||
anomalyNet = "SWW_ANOMALY_NET_"
|
||||
anomalyASN = "SWW_ANOMALY_ASN_"
|
||||
anomalyTotal = "SWW_ANOMALY_TOTAL_"
|
||||
watch = "SWW_WATCH_"
|
||||
|
||||
requestsPerMinute = "REQUESTS_PER_MINUTE"
|
||||
requestsPerHour = "REQUESTS_PER_HOUR"
|
||||
@@ -155,21 +157,21 @@ func loggedAnomalyDefaults() map[string]string {
|
||||
return logged
|
||||
}
|
||||
|
||||
// defaultAlertEvents is the default of SWWAF_ALERT_EVENTS, and
|
||||
// defaultAlertCooldown that of SWWAF_ALERT_COOLDOWN.
|
||||
// defaultAlertEvents is the default of SWW_ALERT_EVENTS, and
|
||||
// defaultAlertCooldown that of SWW_ALERT_COOLDOWN.
|
||||
const (
|
||||
defaultAlertEvents = "ban,permanent_ban,waf_block,anomaly,reputation_hit," +
|
||||
defaultAlertEvents = "ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit," +
|
||||
"source_failure,file_error"
|
||||
defaultAlertCooldown = "15m"
|
||||
)
|
||||
|
||||
// defaultReputationCacheTTL is the default of SWWAF_REPUTATION_CACHE_TTL.
|
||||
// defaultReputationCacheTTL is the default of SWW_REPUTATION_CACHE_TTL.
|
||||
const defaultReputationCacheTTL = "24h"
|
||||
|
||||
// defaultWAFDisabledRules is the default of SWWAF_WAF_DISABLED_RULES.
|
||||
// defaultWAFDisabledRules is the default of SWW_WAF_DISABLED_RULES.
|
||||
const defaultWAFDisabledRules = "920340,920420,920440,920640,930130,930140"
|
||||
|
||||
// defaultLogRequestHeaders is the default of SWWAF_LOG_REQUEST_HEADERS.
|
||||
// defaultLogRequestHeaders is the default of SWW_LOG_REQUEST_HEADERS.
|
||||
const defaultLogRequestHeaders = "accept,accept-language,accept-encoding," +
|
||||
"content-type,origin,range"
|
||||
|
||||
@@ -194,8 +196,8 @@ const (
|
||||
otherToken = "fedcba9876543210fedcba9876543210"
|
||||
)
|
||||
|
||||
// instance is an SWWAF_INSTANCE_NAME that is a valid app name too, and
|
||||
// remoteURL an SWWAF_LOG_REMOTE_URL, for the tests that send the lines.
|
||||
// instance is an SWW_INSTANCE_NAME that is a valid app name too, and
|
||||
// remoteURL an SWW_LOG_REMOTE_URL, for the tests that send the lines.
|
||||
const (
|
||||
instance = "fsn1app1/blog"
|
||||
remoteURL = "syslog+udp://192.0.2.1:514"
|
||||
@@ -207,7 +209,10 @@ const off = "off"
|
||||
// enabled is true, as a setting's value.
|
||||
const enabled = "true"
|
||||
|
||||
// defaultLookupSource is the default of SWWAF_LOOKUP_SOURCE, and
|
||||
// disabled is false, as a setting's value.
|
||||
const disabled = "false"
|
||||
|
||||
// defaultLookupSource is the default of SWW_LOOKUP_SOURCE, and
|
||||
// fileSource the source that is the lookup database.
|
||||
const (
|
||||
defaultLookupSource = "geojs"
|
||||
@@ -273,9 +278,13 @@ func TestDefaults(t *testing.T) {
|
||||
wantNetblocks(t, cfg.AllowNets)
|
||||
wantNetblocks(t, cfg.RateLimitExemptNets)
|
||||
wantNetblocks(t, cfg.DenyNets)
|
||||
wantCountries(t, deniedCountries, cfg.DeniedCountries)
|
||||
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries)
|
||||
|
||||
if len(cfg.TagLists) != 0 || len(cfg.TagDeny) != 0 || len(cfg.TagLimitPercent) != 0 {
|
||||
t.Errorf("tag settings %v, %v and %v by default, want all empty",
|
||||
cfg.TagLists, cfg.TagDeny, cfg.TagLimitPercent)
|
||||
}
|
||||
|
||||
hostname, err := os.Hostname()
|
||||
if err != nil || hostname == "" || cfg.InstanceName != hostname {
|
||||
t.Errorf("%s is %q, want the host's name %q (%v)", instanceName,
|
||||
@@ -326,7 +335,6 @@ func TestValuesAsSet(t *testing.T) {
|
||||
rateLimitPerMinute: "60",
|
||||
rateLimitPerHour: "600",
|
||||
rateLimitPerDay: "6000",
|
||||
deniedCountries: "cn, RU,kp,Xk",
|
||||
allowedCountries: "de",
|
||||
banResponse: "429",
|
||||
limitBanDuration: "15m",
|
||||
@@ -341,7 +349,7 @@ func TestValuesAsSet(t *testing.T) {
|
||||
metricsToken: token,
|
||||
metricsTopN: "10",
|
||||
rulesDir: "/srv/waf-rules",
|
||||
rulesEnabled: "false",
|
||||
rulesEnabled: disabled,
|
||||
})
|
||||
|
||||
wantSettings(t, cfg, config.Config{
|
||||
@@ -369,7 +377,6 @@ func TestValuesAsSet(t *testing.T) {
|
||||
wantNetblocks(t, cfg.AllowNets, "192.0.2.7/32")
|
||||
wantNetblocks(t, cfg.RateLimitExemptNets, "2001:db8::/48", "10.9.8.7/32")
|
||||
wantNetblocks(t, cfg.DenyNets, "198.51.100.0/24")
|
||||
wantCountries(t, deniedCountries, cfg.DeniedCountries, "CN", "RU", "KP", "XK")
|
||||
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
|
||||
}
|
||||
|
||||
@@ -539,13 +546,14 @@ func TestCoreRuleSetSettings(t *testing.T) {
|
||||
environment{
|
||||
wafMode: config.WAFModeDetect, wafParanoiaLevel: "4", wafAnomalyThreshold: "10",
|
||||
wafDisabledRules: "942100, 920350", wafExemptPaths: "/api/, /static/",
|
||||
wafBodyLimit: "32K",
|
||||
wafBodyLimit: "32K", exclusionsFile: "/etc/sww/crs-exclusions.conf",
|
||||
},
|
||||
config.Config{
|
||||
WAFMode: config.WAFModeDetect, WAFParanoiaLevel: 4, WAFAnomalyThreshold: 10,
|
||||
WAFDisabledRules: []int{942100, 920350},
|
||||
WAFExemptPaths: []string{"/api/", "/static/"},
|
||||
WAFBodyLimit: 32 << 10,
|
||||
WAFDisabledRules: []int{942100, 920350},
|
||||
WAFExemptPaths: []string{"/api/", "/static/"},
|
||||
WAFBodyLimit: 32 << 10,
|
||||
CoreRuleSetExclusionsFile: "/etc/sww/crs-exclusions.conf",
|
||||
},
|
||||
},
|
||||
{
|
||||
@@ -565,7 +573,8 @@ func TestCoreRuleSetSettings(t *testing.T) {
|
||||
WAFMode: cfg.WAFMode, WAFParanoiaLevel: cfg.WAFParanoiaLevel,
|
||||
WAFAnomalyThreshold: cfg.WAFAnomalyThreshold,
|
||||
WAFDisabledRules: cfg.WAFDisabledRules, WAFExemptPaths: cfg.WAFExemptPaths,
|
||||
WAFBodyLimit: cfg.WAFBodyLimit,
|
||||
WAFBodyLimit: cfg.WAFBodyLimit,
|
||||
CoreRuleSetExclusionsFile: cfg.CoreRuleSetExclusionsFile,
|
||||
}
|
||||
if !reflect.DeepEqual(got, tc.want) {
|
||||
t.Errorf("%v gave\n%+v\nwant\n%+v", tc.env, got, tc.want)
|
||||
@@ -1094,15 +1103,15 @@ func TestInstanceNameNotUTF8StopsTheStart(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
|
||||
func TestCodeInAllowedCountriesAndTagDenyStopsTheStart(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(environment{
|
||||
deniedCountries: "cn,ru",
|
||||
tagDeny: "country=cn,country=ru",
|
||||
allowedCountries: "de,RU",
|
||||
}.lookupEnv)
|
||||
if err == nil {
|
||||
t.Fatal("ru on both country lists was accepted")
|
||||
t.Fatal("ru in SWW_EXCLUSIVELY_ALLOWED_COUNTRIES and SWW_TAG_DENY was accepted")
|
||||
}
|
||||
|
||||
if !strings.HasPrefix(err.Error(), allowedCountries+": ") ||
|
||||
@@ -1173,12 +1182,11 @@ func TestSettingNeedingLookupsStopsTheStartWhileTheyAreOff(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for name, value := range map[string]string{
|
||||
deniedCountries: "kp",
|
||||
allowedCountries: "de",
|
||||
addLookupHeaders: enabled,
|
||||
tagDeny: "country=KP",
|
||||
tagLimitPercent: "asn=64496:50",
|
||||
asnLimitPercentURL: asnURL,
|
||||
asnLimitPercent: "AS64496:50",
|
||||
countryLimitPercent: "cn:25",
|
||||
asnBytesPercent: "AS64496:50",
|
||||
countryBytesPercent: "cn:25",
|
||||
unknownLimitPercent: "99",
|
||||
@@ -1201,12 +1209,13 @@ func TestSettingNeedingLookupsStopsTheStartWhileTheyAreOff(t *testing.T) {
|
||||
}
|
||||
|
||||
// Set empty, the lists need nothing looked up, and nor does
|
||||
// SWWAF_UNKNOWN_LIMIT_PERCENT at 100, which lowers no limit, an anomaly
|
||||
// threshold per AS number that is off, any other anomaly threshold, or
|
||||
// a blocklist.
|
||||
// SWW_UNKNOWN_LIMIT_PERCENT at 100, which lowers no limit, a tag the
|
||||
// lookup does not give, an anomaly threshold per AS number that is off,
|
||||
// any other anomaly threshold, or a blocklist.
|
||||
env := environment{
|
||||
lookupSource: off, deniedCountries: "", allowedCountries: "",
|
||||
asnLimitPercent: "", countryLimitPercent: "", asnBytesPercent: "",
|
||||
lookupSource: off, allowedCountries: "", asnBytesPercent: "",
|
||||
tagLists: "torexit=https://example.com/tor.txt,vpn=https://example.com/vpn.txt",
|
||||
tagDeny: torExitTag, tagLimitPercent: "vpn:50",
|
||||
countryBytesPercent: "", unknownLimitPercent: "100", asnLimitPercentURL: "",
|
||||
blocklistURLs: dropURL,
|
||||
}
|
||||
@@ -1352,34 +1361,251 @@ func TestInvalidAnomalySettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// torExitTag, zenTag and exploitTag are tags these tests name.
|
||||
const (
|
||||
torExitTag = "torexit"
|
||||
zenTag = "dnsbl=zen.example"
|
||||
exploitTag = "exploitsignature"
|
||||
)
|
||||
|
||||
func TestTagSettingsAsSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const torURL = "https://example.com/tor.txt"
|
||||
|
||||
// Names and tags in either case, an AS number with leading zeros, and a
|
||||
// name holding a =.
|
||||
cfg := fromEnvironment(t, environment{
|
||||
tagLists: "TorExit=" + torURL +
|
||||
", vpn=mullvad=https://example.com/mullvad.txt",
|
||||
dnsblZones: "Zen.example",
|
||||
tagDeny: "TorExit, Country=kp, asn=014061",
|
||||
tagLimitPercent: "vpn=mullvad :50, asn=014061:25, COUNTRY=cn:1, dnsbl=zen.example:99",
|
||||
})
|
||||
|
||||
wantLists := []config.TagList{
|
||||
{Name: torExitTag, URL: torURL},
|
||||
{Name: "vpn=mullvad", URL: "https://example.com/mullvad.txt"},
|
||||
}
|
||||
if !slices.Equal(cfg.TagLists, wantLists) {
|
||||
t.Errorf("%s gave %+v, want %+v", tagLists, cfg.TagLists, wantLists)
|
||||
}
|
||||
|
||||
wantDeny := []string{torExitTag, "country=KP", "asn=14061"}
|
||||
if !slices.Equal(cfg.TagDeny, wantDeny) {
|
||||
t.Errorf("%s gave %v, want %v", tagDeny, cfg.TagDeny, wantDeny)
|
||||
}
|
||||
|
||||
wantPercents := map[string]int64{
|
||||
"vpn=mullvad": 50, "asn=14061": 25, "country=CN": 1, zenTag: 99,
|
||||
}
|
||||
if !maps.Equal(cfg.TagLimitPercent, wantPercents) {
|
||||
t.Errorf("%s gave %v, want %v", tagLimitPercent, cfg.TagLimitPercent, wantPercents)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRouteWatchURLAndPrefixTags(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const rw = "https://routewatch.example"
|
||||
|
||||
if cfg := fromEnvironment(t, environment{}); cfg.RouteWatchURL != "" {
|
||||
t.Errorf("default %s is %q, want unset", routeWatchURL, cfg.RouteWatchURL)
|
||||
}
|
||||
|
||||
cfg := fromEnvironment(t, environment{
|
||||
routeWatchURL: rw, tagDeny: "prefix=2001:DB8::/32",
|
||||
tagLimitPercent: "prefix=192.0.2.0/24:50",
|
||||
})
|
||||
wantDeny := []string{"prefix=2001:db8::/32"}
|
||||
|
||||
if cfg.RouteWatchURL != rw || !slices.Equal(cfg.TagDeny, wantDeny) ||
|
||||
cfg.TagLimitPercent["prefix=192.0.2.0/24"] != 50 {
|
||||
t.Errorf("gave %q, %v, %v", cfg.RouteWatchURL, cfg.TagDeny, cfg.TagLimitPercent)
|
||||
}
|
||||
|
||||
// routewatch gives asn= tags too, so an asn= tag needs no lookups while it is set.
|
||||
fromEnvironment(t, environment{
|
||||
lookupSource: off, routeWatchURL: rw, tagDeny: "asn=64496",
|
||||
})
|
||||
|
||||
for _, stopped := range []struct {
|
||||
env environment
|
||||
name string
|
||||
}{
|
||||
{environment{routeWatchURL: "ftp://routewatch.example"}, routeWatchURL},
|
||||
{environment{routeWatchURL: "routewatch.example"}, routeWatchURL},
|
||||
{environment{tagDeny: "prefix=192.0.2.0/24"}, tagDeny},
|
||||
{environment{routeWatchURL: rw, tagDeny: "prefix=nowhere"}, tagDeny},
|
||||
{environment{routeWatchURL: rw, tagDeny: "prefix=192.0.2.5/24"}, tagDeny},
|
||||
{
|
||||
environment{routeWatchURL: rw, tagLimitPercent: "prefix=192.0.2.0/33:50"},
|
||||
tagLimitPercent,
|
||||
},
|
||||
{environment{lookupSource: off, routeWatchURL: rw, tagDeny: "country=DE"}, tagDeny},
|
||||
{environment{lookupSource: off, tagDeny: "asn=64496"}, tagDeny},
|
||||
} {
|
||||
_, err := config.FromEnvironment(stopped.env.lookupEnv)
|
||||
if err == nil || !strings.HasPrefix(err.Error(), stopped.name+": ") {
|
||||
t.Errorf("%v gave the error %v, want one naming %s", stopped.env, err, stopped.name)
|
||||
}
|
||||
}
|
||||
|
||||
badURL := environment{routeWatchURL: "ftp://routewatch.example"}
|
||||
_, err := config.FromEnvironment(badURL.lookupEnv)
|
||||
|
||||
want := routeWatchURL + `: "ftp://routewatch.example" is not an http or https URL ` +
|
||||
"without a user or a fragment, such as https://routewatch.example"
|
||||
if err == nil || err.Error() != want {
|
||||
t.Errorf("error %v, want %s", err, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagThatNoRequestCanGetIsToldApartFromOneItCan(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
list = "torexit=https://example.com/tor.txt"
|
||||
key = "abc123"
|
||||
zone = key + ".xbl.dq.spamhaus.net"
|
||||
masked = "********.xbl.dq.spamhaus.net"
|
||||
)
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
env environment
|
||||
want string // "" is accepted
|
||||
}{
|
||||
{"a list's name", environment{tagLists: list, tagDeny: torExitTag}, ""},
|
||||
{
|
||||
"a mistyped list's name", environment{tagLists: list, tagDeny: "torexti"},
|
||||
`"torexti"`,
|
||||
},
|
||||
{"a masked zone", environment{dnsblZones: zone, tagDeny: "dnsbl=" + masked}, ""},
|
||||
{
|
||||
"a zone with its key", environment{dnsblZones: zone, tagDeny: "dnsbl=" + zone},
|
||||
masked,
|
||||
},
|
||||
{"another zone", environment{dnsblZones: zone, tagDeny: zenTag}, zenTag},
|
||||
{
|
||||
"abuseipdb with a key",
|
||||
environment{abuseIPDBKey: "k", tagLimitPercent: "abuseipdb:50"}, "",
|
||||
},
|
||||
{"crowdsec, though a list has its name", environment{
|
||||
tagLists: "crowdsec=https://example.com/c.txt", tagDeny: "crowdsec",
|
||||
}, `"crowdsec" cannot be used`},
|
||||
{exploitTag, environment{tagDeny: exploitTag}, ""},
|
||||
{
|
||||
exploitTag + " with rule files off",
|
||||
environment{tagDeny: exploitTag, rulesEnabled: disabled}, "",
|
||||
},
|
||||
{
|
||||
exploitTag + " with the Core Rule Set off",
|
||||
environment{tagDeny: exploitTag, wafMode: off}, "",
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(tc.env.lookupEnv)
|
||||
|
||||
switch {
|
||||
case tc.want == "" && err != nil:
|
||||
t.Errorf("error %v, want none", err)
|
||||
case tc.want != "" && (err == nil || !strings.Contains(err.Error(), tc.want)):
|
||||
t.Errorf("error %v, want one holding %s", err, tc.want)
|
||||
case err != nil && strings.Contains(err.Error(), key):
|
||||
t.Errorf("error %v shows the zone's key", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestErrorOfATagSettingLeavesOutAZoneKey(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
tag = "dnsbl=" + spamhaus
|
||||
emptyItem = "has an empty item in its list"
|
||||
twice = `"` + spamhausMasked + `" is listed twice`
|
||||
)
|
||||
|
||||
for _, tc := range []struct{ name, value, want string }{
|
||||
{tagDeny, tag + "," + tag, twice},
|
||||
{tagDeny, tag + ",,", emptyItem},
|
||||
{
|
||||
tagLimitPercent, tag,
|
||||
`"` + spamhausMasked + `" is not a tag, : and a percentage, ` +
|
||||
"such as vpn:50 or asn=14061:25",
|
||||
},
|
||||
{
|
||||
tagLimitPercent, "50:" + tag,
|
||||
`"` + spamhausMasked + `" is not a percentage from 1 to 99`,
|
||||
},
|
||||
{tagLimitPercent, tag + ":50," + tag + " :25", twice},
|
||||
{tagLimitPercent, tag + ":50,,", emptyItem},
|
||||
} {
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
|
||||
|
||||
want := tc.name + ": " + tc.want
|
||||
if err == nil || err.Error() != want {
|
||||
t.Errorf("%s=%s gave the error %v, want %s", tc.name, tc.value, err, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagListURLFetchedAsAnotherListStopsTheStart(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const lapi = "http://172.17.0.1:8080"
|
||||
|
||||
for _, tc := range []struct {
|
||||
name, listURL string
|
||||
env environment
|
||||
}{
|
||||
{"a blocklist", dropURL, environment{blocklistURLs: dropURL}},
|
||||
{"the file of AS:percent lines", asnURL, environment{asnLimitPercentURL: asnURL}},
|
||||
{
|
||||
"the CrowdSec decision list", lapi + "/v1/decisions",
|
||||
environment{"SWW_CROWDSEC_LAPI_URL": lapi, "SWW_CROWDSEC_LAPI_KEY": "key"},
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tc.env[tagLists] = "dup=" + tc.listURL
|
||||
|
||||
want := tagLists + `: "` + tc.listURL + `" is in SWW_BLOCKLIST_URLS, is ` +
|
||||
"SWW_ASN_LIMIT_PERCENT_URL or is the CrowdSec decision list too"
|
||||
|
||||
_, err := config.FromEnvironment(tc.env.lookupEnv)
|
||||
if err == nil || err.Error() != want {
|
||||
t.Errorf("error %v, want %s", err, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestBiasedThresholdsAsSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := fromEnvironment(t, environment{})
|
||||
if len(cfg.ASNLimitPercent) != 0 || len(cfg.CountryLimitPercent) != 0 ||
|
||||
len(cfg.ASNBytesPercent) != 0 || len(cfg.CountryBytesPercent) != 0 ||
|
||||
if len(cfg.ASNBytesPercent) != 0 || len(cfg.CountryBytesPercent) != 0 ||
|
||||
cfg.UnknownLimitPercent != 100 {
|
||||
t.Errorf("biased thresholds %v, %v, %v, %v and %d by default, "+
|
||||
"want four empty lists and 100", cfg.ASNLimitPercent, cfg.CountryLimitPercent,
|
||||
t.Errorf("biased thresholds %v, %v and %d by default, "+
|
||||
"want two empty lists and 100",
|
||||
cfg.ASNBytesPercent, cfg.CountryBytesPercent, cfg.UnknownLimitPercent)
|
||||
}
|
||||
|
||||
// AS numbers and countries in either case, an AS number with leading
|
||||
// zeros, 0 and 100.
|
||||
cfg = fromEnvironment(t, environment{
|
||||
asnLimitPercent: "AS14061:50, as16276:0,AS045102:100",
|
||||
countryLimitPercent: "cn:25,RU:50",
|
||||
asnBytesPercent: "as16276:75",
|
||||
countryBytesPercent: "ru:10",
|
||||
unknownLimitPercent: "0",
|
||||
})
|
||||
|
||||
for name, tc := range map[string]struct{ got, want map[string]int64 }{
|
||||
asnLimitPercent: {
|
||||
cfg.ASNLimitPercent,
|
||||
map[string]int64{"AS14061": 50, "AS16276": 0, "AS45102": 100},
|
||||
},
|
||||
countryLimitPercent: {cfg.CountryLimitPercent, map[string]int64{"CN": 25, "RU": 50}},
|
||||
asnBytesPercent: {cfg.ASNBytesPercent, map[string]int64{"AS16276": 75}},
|
||||
countryBytesPercent: {cfg.CountryBytesPercent, map[string]int64{"RU": 10}},
|
||||
} {
|
||||
@@ -1403,16 +1629,16 @@ func TestInvalidBiasedThresholdStopsTheStartSayingWhatIsWrong(t *testing.T) {
|
||||
)
|
||||
|
||||
for _, tc := range []struct{ name, value, want string }{
|
||||
{asnLimitPercent, "14061:50", `"14061"` + notASN},
|
||||
{asnLimitPercent, "AS4294967296:50", `"AS4294967296"` + notASN},
|
||||
{asnLimitPercent, "AS14061", `"AS14061"` + notItem},
|
||||
{asnLimitPercent, "AS14061:101", `"101"` + notPercent},
|
||||
{asnLimitPercent, "AS14061:50,as14061:25", `"as14061" is listed twice`},
|
||||
{asnBytesPercent, "14061:50", `"14061"` + notASN},
|
||||
{asnBytesPercent, "AS4294967296:50", `"AS4294967296"` + notASN},
|
||||
{asnBytesPercent, "AS14061", `"AS14061"` + notItem},
|
||||
{asnBytesPercent, "AS14061:101", `"101"` + notPercent},
|
||||
{asnBytesPercent, "AS14061:50,as14061:25", `"as14061" is listed twice`},
|
||||
{
|
||||
countryLimitPercent, "nk:25",
|
||||
countryBytesPercent, "nk:25",
|
||||
`"nk" is not a two-letter country code such as de or kp`,
|
||||
},
|
||||
{countryLimitPercent, "cn:25,CN:50", `"CN" is listed twice`},
|
||||
{countryBytesPercent, "cn:25,CN:50", `"CN" is listed twice`},
|
||||
{asnBytesPercent, "AS14061:-1", `"-1"` + notPercent},
|
||||
{countryBytesPercent, "cn:50%", `"50%"` + notPercent},
|
||||
{unknownLimitPercent, "101", `"101"` + notPercent},
|
||||
@@ -1439,7 +1665,7 @@ const (
|
||||
asnURL = "https://lists.example/asn.txt"
|
||||
)
|
||||
|
||||
// The actions of SWWAF_BLOCKLIST_ACTION, as Config gives them.
|
||||
// The actions of SWW_BLOCKLIST_ACTION, as Config gives them.
|
||||
const (
|
||||
actionDeny = "deny"
|
||||
actionLimit = "limit"
|
||||
@@ -1535,7 +1761,7 @@ func TestASNLimitPercentURLThatIsABlocklistStopsTheStart(t *testing.T) {
|
||||
blocklistURLs: dropURL + "," + asnURL, asnLimitPercentURL: asnURL,
|
||||
}.lookupEnv)
|
||||
|
||||
want := asnLimitPercentURL + `: "` + asnURL + `" is in SWWAF_BLOCKLIST_URLS too`
|
||||
want := asnLimitPercentURL + `: "` + asnURL + `" is in SWW_BLOCKLIST_URLS too`
|
||||
if err == nil || err.Error() != want {
|
||||
t.Errorf("error %v, want %s", err, want)
|
||||
}
|
||||
@@ -1838,7 +2064,7 @@ func TestInvalidCrowdSecSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
|
||||
notLAPIURL = " is not an http or https URL without a user or a fragment, " +
|
||||
"such as http://172.17.0.1:8080"
|
||||
anotherList = `gives the decision list "` + lapi + `/v1/decisions", which is in ` +
|
||||
`SWWAF_BLOCKLIST_URLS or is SWWAF_ASN_LIMIT_PERCENT_URL too`
|
||||
`SWW_BLOCKLIST_URLS or is SWW_ASN_LIMIT_PERCENT_URL too`
|
||||
)
|
||||
|
||||
for _, tc := range []struct {
|
||||
@@ -2000,23 +2226,37 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
|
||||
{lookupSource, "ipinfo"}, {lookupSource, "GeoJS"}, {lookupSource, ""},
|
||||
{lookupTimeout, off}, {lookupTimeout, "0s"}, {lookupTimeout, "1"},
|
||||
{addLookupHeaders, "yes"},
|
||||
{deniedCountries, "nk"},
|
||||
{deniedCountries, "kp,,ir"},
|
||||
{deniedCountries, "prk"},
|
||||
{deniedCountries, "408"},
|
||||
{deniedCountries, "k"},
|
||||
{deniedCountries, "eu"},
|
||||
{deniedCountries, "un"},
|
||||
{deniedCountries, "su"},
|
||||
{allowedCountries, "nk"},
|
||||
{allowedCountries, "kp,,ir"},
|
||||
{allowedCountries, "prk"},
|
||||
{allowedCountries, "408"},
|
||||
{allowedCountries, "k"},
|
||||
{allowedCountries, "eu"},
|
||||
{allowedCountries, "un"},
|
||||
{allowedCountries, "su"},
|
||||
{allowedCountries, "ac"},
|
||||
{allowedCountries, "uk"},
|
||||
{allowedCountries, "zz"},
|
||||
{allowedCountries, "de,germany"},
|
||||
{asnLimitPercent, "AS14061:50,,AS16276:50"}, {asnLimitPercent, "ASX:50"},
|
||||
{asnLimitPercent, "AS14061:"}, {asnLimitPercent, "AS14061 :50"},
|
||||
{asnLimitPercent, "AS14061:1.5"}, {asnLimitPercent, "AS-1:50"},
|
||||
{countryLimitPercent, "cn"}, {countryLimitPercent, "cn:"},
|
||||
{countryLimitPercent, "cn:25:50"}, {countryLimitPercent, "china:25"},
|
||||
{asnBytesPercent, "AS14061:50,,AS16276:50"}, {asnBytesPercent, "ASX:50"},
|
||||
{asnBytesPercent, "AS14061:"}, {asnBytesPercent, "AS14061 :50"},
|
||||
{asnBytesPercent, "AS14061:1.5"}, {asnBytesPercent, "AS-1:50"},
|
||||
{countryBytesPercent, "cn"}, {countryBytesPercent, "cn:"},
|
||||
{countryBytesPercent, "cn:25:50"}, {countryBytesPercent, "china:25"},
|
||||
{tagDeny, "a,,b"}, {tagDeny, "country=nk"}, {tagDeny, "asn=abc"},
|
||||
{tagDeny, "asn="}, {tagDeny, "torexit,TOREXIT"}, {tagDeny, "torexti"},
|
||||
{tagDeny, zenTag}, {tagDeny, "abuseipdb"}, {tagDeny, "crowdsec"},
|
||||
{tagLimitPercent, "torexti:50"}, {tagLimitPercent, "dnsbl=zen.example:50"},
|
||||
{tagLimitPercent, "abuseipdb:50"}, {tagLimitPercent, "crowdsec:50"},
|
||||
{tagLimitPercent, "vpn :50"}, {tagLimitPercent, "vpn"}, {tagLimitPercent, ":50"},
|
||||
{tagLimitPercent, "vpn:"}, {tagLimitPercent, "vpn:0"}, {tagLimitPercent, "vpn:100"},
|
||||
{tagLimitPercent, "vpn:1.5"}, {tagLimitPercent, "vpn:50,VPN:25"},
|
||||
{tagLimitPercent, "exploitsignature:50"}, {tagLimitPercent, "asn=abc:50"},
|
||||
{tagLists, torExitTag}, {tagLists, "=https://example.com/a.txt"},
|
||||
{tagLists, "a,b=https://example.com/a.txt"},
|
||||
{tagLists, "a=ftp://example.com/a.txt"},
|
||||
{tagLists, "a=https://example.com/a.txt,A=https://example.com/b.txt"},
|
||||
{tagLists, "a=https://example.com/a.txt,b=https://example.com/a.txt"},
|
||||
{asnBytesPercent, "AS14061:101"}, {countryBytesPercent, "su:50"},
|
||||
{unknownLimitPercent, ""}, {unknownLimitPercent, "-1"},
|
||||
{unknownLimitPercent, "50%"},
|
||||
@@ -2028,6 +2268,25 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
|
||||
{logLevel, "INFO"}, {logLevel, "warning"}, {logLevel, "trace"}, {logLevel, ""},
|
||||
{rulesEnabled, "yes"}, {rulesEnabled, "True"},
|
||||
})
|
||||
|
||||
wantExploitTagStopsTheStartWithRulesAndCoreRuleSetOff(t)
|
||||
}
|
||||
|
||||
// wantExploitTagStopsTheStartWithRulesAndCoreRuleSetOff checks that with rule
|
||||
// files off, and either setting turning off the Core Rule Set, the tag
|
||||
// exploitsignature in SWW_TAG_DENY stops the start.
|
||||
func wantExploitTagStopsTheStartWithRulesAndCoreRuleSetOff(t *testing.T) {
|
||||
t.Helper()
|
||||
|
||||
for _, name := range []string{wafMode, wafAnomalyThreshold} {
|
||||
_, err := config.FromEnvironment(environment{
|
||||
tagDeny: exploitTag, rulesEnabled: disabled, name: off,
|
||||
}.lookupEnv)
|
||||
if err == nil || !strings.HasPrefix(err.Error(), tagDeny+": ") {
|
||||
t.Errorf("%s=off with rule files off gave the error %v, want one naming %s",
|
||||
name, err, tagDeny)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestInvalidBanOrStateValueStopsTheStart(t *testing.T) {
|
||||
@@ -2282,11 +2541,11 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
lookupSource: defaultLookupSource,
|
||||
lookupDBPath: "",
|
||||
lookupTimeout: "800ms",
|
||||
addLookupHeaders: "false",
|
||||
deniedCountries: "",
|
||||
addLookupHeaders: disabled,
|
||||
allowedCountries: "",
|
||||
asnLimitPercent: "",
|
||||
countryLimitPercent: "",
|
||||
tagLists: "",
|
||||
tagDeny: "",
|
||||
tagLimitPercent: "",
|
||||
asnBytesPercent: "",
|
||||
countryBytesPercent: "",
|
||||
unknownLimitPercent: "100",
|
||||
@@ -2302,6 +2561,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
reputationAction: "limit:25",
|
||||
reputationCacheTTL: defaultReputationCacheTTL,
|
||||
reputationTimeout: "2s",
|
||||
routeWatchURL: "",
|
||||
crowdSecURL: "",
|
||||
crowdSecKey: "",
|
||||
banResponse: "403",
|
||||
@@ -2328,6 +2588,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
wafDisabledRules: defaultWAFDisabledRules,
|
||||
wafExemptPaths: "",
|
||||
wafBodyLimit: off,
|
||||
exclusionsFile: "",
|
||||
trapPaths: "",
|
||||
errorBurstThreshold: "30",
|
||||
logRemoteURL: "",
|
||||
|
||||
@@ -1,20 +1,25 @@
|
||||
// Package waf runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
|
||||
// Package coreruleset runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
|
||||
// method, the URL with its query and the headers of a request, and on its
|
||||
// body while SWWAF_WAF_BODY_LIMIT is set, with the six changes sww
|
||||
// makes to it, as "Attack detection" under "Settings" in SPEC.md describes
|
||||
// them. It reads no response.
|
||||
// body while SWW_WAF_BODY_LIMIT is set, with the eight changes sww
|
||||
// makes to it and the exclusions in SWW_CORE_RULE_SET_EXCLUSIONS_FILE, as
|
||||
// "Attack detection" under "Settings" in SPEC.md describes them. It reads no
|
||||
// response.
|
||||
//
|
||||
// sww writes only to its state directory, so Coraza is built with
|
||||
// its no_fs_access tag, as the Dockerfile and script/build build it: of a
|
||||
// file in a multipart body, Coraza then counts the bytes instead of
|
||||
// writing them to the system's temporary directory.
|
||||
package waf
|
||||
package coreruleset
|
||||
|
||||
import (
|
||||
_ "embed"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"os"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
@@ -26,15 +31,17 @@ import (
|
||||
)
|
||||
|
||||
// directives are the Core Rule Set as sww runs it, with the
|
||||
// paranoia level for %d, and bodyDirectives for %s while
|
||||
// SWWAF_WAF_BODY_LIMIT is set. Each rule sww adds has an id from
|
||||
// 900000 to 900999, the ids the Core Rule Set keeps for the rules that set
|
||||
// it up, which SWWAF_WAF_DISABLED_RULES refuses, so that no setting
|
||||
// switches one off. Coraza joins a line ending in \ to the next, without
|
||||
// the spaces at the start of the next.
|
||||
// paranoia level for %d, bodyDirectives for the first %s while
|
||||
// SWW_WAF_BODY_LIMIT is set, the exclusions file for the second, and
|
||||
// giteaExclusions for the third.
|
||||
// Each rule sww adds has an id from 900000 to 900999, the ids the
|
||||
// Core Rule Set keeps for the rules that set it up, which
|
||||
// SWW_WAF_DISABLED_RULES and the exclusions file refuse, so that no
|
||||
// setting switches one off. Coraza joins a line ending in \ to the next,
|
||||
// without the spaces at the start of the next.
|
||||
const directives = `
|
||||
# The engine only detects. sww compares the request's anomaly
|
||||
# score with SWWAF_WAF_ANOMALY_THRESHOLD itself, in block and detect mode
|
||||
# score with SWW_WAF_ANOMALY_THRESHOLD itself, in block and detect mode
|
||||
# alike. It reads no body, unless bodyDirectives switch that on.
|
||||
SecRuleEngine DetectionOnly
|
||||
SecRequestBodyAccess Off
|
||||
@@ -69,6 +76,14 @@ SecArgumentsLimit 1000
|
||||
SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
|
||||
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
|
||||
|
||||
# The exclusions file comes before the rules, where the Core Rule Set puts
|
||||
# its own file of exclusions that run before the rules, so that they run
|
||||
# before the rules of phase 1 too.
|
||||
%s
|
||||
|
||||
# The eighth change comes before the rules for the same reason.
|
||||
%s
|
||||
|
||||
# The sixth: only the rules for requests are loaded, and no response is
|
||||
# inspected.
|
||||
Include @owasp_crs/REQUEST-*.conf
|
||||
@@ -96,10 +111,19 @@ SecRuleUpdateTargetById 932260 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
|
||||
!ARGS:artifactName|!ARGS:redirect_to"
|
||||
|
||||
# The fifth, for Referer: it is not checked for a Unix command without
|
||||
# arguments, or for Java starting a process. The cookies are left out in
|
||||
# Inspect.
|
||||
# arguments, for Java starting a process, or for a Java class name. 944130
|
||||
# refused every page opened from the page of java.lang.String.md. The
|
||||
# cookies are left out in Inspect.
|
||||
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
|
||||
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
|
||||
SecRuleUpdateTargetById 944130 "!REQUEST_HEADERS:Referer"
|
||||
|
||||
# The seventh: a container image push that reuses a layer names the image
|
||||
# it takes the layer from in the query parameter from, which is not
|
||||
# checked for command names. 932260 refused the push of a layer from
|
||||
# owner/python3-app.
|
||||
SecRule REQUEST_FILENAME "@rx ^/v2/.+/blobs/uploads/$" \
|
||||
"id:900500,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932260;ARGS:from"
|
||||
`
|
||||
|
||||
// bodyDirectives have the Core Rule Set read the part of a request body
|
||||
@@ -144,6 +168,13 @@ SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
|
||||
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
|
||||
`
|
||||
|
||||
// giteaExclusions are the eighth change: the fields of gitea's own
|
||||
// requests that skip the rules that refused them there, as Coraza
|
||||
// directives.
|
||||
//
|
||||
//go:embed gitea.conf
|
||||
var giteaExclusions string
|
||||
|
||||
// cookiesNotRead are the cookies the Core Rule Set reads a request
|
||||
// without, the rest of the fifth change.
|
||||
//
|
||||
@@ -152,14 +183,102 @@ var cookiesNotRead = []string{"gitea_flash", "redirect_to"}
|
||||
|
||||
// Params are what New needs.
|
||||
type Params struct {
|
||||
// ParanoiaLevel is SWWAF_WAF_PARANOIA_LEVEL, from 1 to 4.
|
||||
// ParanoiaLevel is SWW_WAF_PARANOIA_LEVEL, from 1 to 4.
|
||||
ParanoiaLevel int
|
||||
// DisabledRules are the ids of the rules switched off
|
||||
// (SWWAF_WAF_DISABLED_RULES).
|
||||
// (SWW_WAF_DISABLED_RULES).
|
||||
DisabledRules []int
|
||||
// BodyLimit is the most of a request body the Core Rule Set reads
|
||||
// (SWWAF_WAF_BODY_LIMIT), 0 while it is off and it reads none.
|
||||
// (SWW_WAF_BODY_LIMIT), 0 while it is off and it reads none.
|
||||
BodyLimit int64
|
||||
// ExclusionsFile is the path of a file of exclusions loaded before the
|
||||
// Core Rule Set's rules (SWW_CORE_RULE_SET_EXCLUSIONS_FILE), "" while
|
||||
// there is none.
|
||||
ExclusionsFile string
|
||||
}
|
||||
|
||||
// The ids of sww's own rules, which an exclusions file may not
|
||||
// name.
|
||||
const (
|
||||
firstOwnRuleID = 900000
|
||||
lastOwnRuleID = 900999
|
||||
)
|
||||
|
||||
// numberOrRange matches a number, or a range of numbers such as
|
||||
// 900000-900999. Coraza reads a number with a + before it too, so the
|
||||
// end of a range may have one.
|
||||
var numberOrRange = regexp.MustCompile(`(\d+)(?:-\+?(\d+))?`)
|
||||
|
||||
// errSwwRuleID is the error of an exclusions file that names an
|
||||
// id of sww's own rules.
|
||||
var errSwwRuleID = errors.New("an id of sww's own rules")
|
||||
|
||||
// errChangesNoRule is the error of an exclusions file that holds a
|
||||
// directive that removes or changes rules as it loads, such as
|
||||
// SecRuleRemoveById. The Core Rule Set's rules load after the file, so
|
||||
// the directive finds none of them.
|
||||
var errChangesNoRule = errors.New("the Core Rule Set's rules load after the file; " +
|
||||
"switch a rule off with SWW_WAF_DISABLED_RULES, or on some requests " +
|
||||
"with a ctl exclusion")
|
||||
|
||||
// directivesIn returns the directives in text as Coraza reads them: it
|
||||
// skips blank lines and comment lines, and joins a line ending in \ to
|
||||
// the next, without the spaces around either.
|
||||
func directivesIn(text string) []string {
|
||||
var directives []string
|
||||
|
||||
joined := ""
|
||||
|
||||
for line := range strings.Lines(text) {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
|
||||
if start, found := strings.CutSuffix(line, `\`); found {
|
||||
joined += start
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
directives = append(directives, joined+line)
|
||||
joined = ""
|
||||
}
|
||||
|
||||
return directives
|
||||
}
|
||||
|
||||
// checkExclusions returns an error if exclusions, the text of an
|
||||
// exclusions file, names an id of sww's own rules, alone or in a
|
||||
// range, or holds a directive whose name starts with SecRuleRemove or
|
||||
// SecRuleUpdate.
|
||||
func checkExclusions(exclusions string) error {
|
||||
for _, directive := range directivesIn(exclusions) {
|
||||
for _, match := range numberOrRange.FindAllStringSubmatch(directive, -1) {
|
||||
first, _ := strconv.Atoi(match[1])
|
||||
|
||||
last := first
|
||||
if match[2] != "" {
|
||||
last, _ = strconv.Atoi(match[2])
|
||||
}
|
||||
|
||||
if first <= lastOwnRuleID && last >= firstOwnRuleID {
|
||||
return fmt.Errorf("the exclusions file names %s: %w", match[0],
|
||||
errSwwRuleID)
|
||||
}
|
||||
}
|
||||
|
||||
// Coraza takes the directive's name up to the first space, without
|
||||
// regard to case.
|
||||
name, _, _ := strings.Cut(directive, " ")
|
||||
if lower := strings.ToLower(name); strings.HasPrefix(lower, "secruleremove") ||
|
||||
strings.HasPrefix(lower, "secruleupdate") {
|
||||
return fmt.Errorf("%s in the exclusions file changes no rule: %w", name,
|
||||
errChangesNoRule)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// CoreRuleSet is the Core Rule Set, ready to inspect requests. It is safe
|
||||
@@ -169,16 +288,35 @@ type CoreRuleSet struct {
|
||||
bodyLimit int64
|
||||
}
|
||||
|
||||
// New returns the Core Rule Set with the six changes, at params'
|
||||
// paranoia level, without the rules it switches off, and reading request
|
||||
// bodies up to params' limit.
|
||||
// New returns the Core Rule Set with the eight changes, at params'
|
||||
// paranoia level, without the rules it switches off, reading request
|
||||
// bodies up to params' limit, and with params' exclusions file loaded
|
||||
// before the Core Rule Set's rules. An exclusions file is an error if it
|
||||
// cannot be read or does not parse, or if checkExclusions refuses it.
|
||||
func New(params Params) (*CoreRuleSet, error) {
|
||||
body := ""
|
||||
if params.BodyLimit > 0 {
|
||||
body = fmt.Sprintf(bodyDirectives, params.BodyLimit)
|
||||
}
|
||||
|
||||
text := fmt.Sprintf(directives, params.ParanoiaLevel, body)
|
||||
exclusions := ""
|
||||
|
||||
if params.ExclusionsFile != "" {
|
||||
file, err := os.ReadFile(params.ExclusionsFile)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read the exclusions file: %w", err)
|
||||
}
|
||||
|
||||
exclusions = string(file)
|
||||
|
||||
err = checkExclusions(exclusions)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
|
||||
giteaExclusions)
|
||||
|
||||
if len(params.DisabledRules) > 0 {
|
||||
ids := make([]string, len(params.DisabledRules))
|
||||
@@ -266,7 +404,7 @@ func (c *CoreRuleSet) Inspect(
|
||||
}
|
||||
|
||||
// readBody reads body, the body of the request in tx, which has run on
|
||||
// the request's headers, while SWWAF_WAF_BODY_LIMIT is set and the body is
|
||||
// the request's headers, while SWW_WAF_BODY_LIMIT is set and the body is
|
||||
// of a kind the Core Rule Set reads: form data and multipart, of which it
|
||||
// reads the first c.bodyLimit bytes, and JSON and XML, which it reads only
|
||||
// when they are no longer than that, since they cannot be read in part.
|
||||
@@ -1,20 +1,21 @@
|
||||
package waf_test
|
||||
package coreruleset_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
)
|
||||
|
||||
// defaultDisabledRules are the rules SWWAF_WAF_DISABLED_RULES switches off
|
||||
// defaultDisabledRules are the rules SWW_WAF_DISABLED_RULES switches off
|
||||
// by default.
|
||||
//
|
||||
//nolint:gochecknoglobals // a constant cannot be a list
|
||||
@@ -22,10 +23,12 @@ var defaultDisabledRules = []int{920340, 920420, 920440, 920640, 930130, 930140}
|
||||
|
||||
// newCoreRuleSet returns the Core Rule Set at paranoia level level, with
|
||||
// the rules in disabled switched off.
|
||||
func newCoreRuleSet(t *testing.T, level int, disabled ...int) *waf.CoreRuleSet {
|
||||
func newCoreRuleSet(t *testing.T, level int, disabled ...int) *coreruleset.CoreRuleSet {
|
||||
t.Helper()
|
||||
|
||||
crs, err := waf.New(waf.Params{ParanoiaLevel: level, DisabledRules: disabled})
|
||||
crs, err := coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: level, DisabledRules: disabled,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("load the Core Rule Set: %v", err)
|
||||
}
|
||||
@@ -48,7 +51,7 @@ func get(target string, headers ...string) request {
|
||||
|
||||
// inspect returns what crs finds in r, sent to git.example by a browser,
|
||||
// whose Host, User-Agent and Accept r.headers may replace.
|
||||
func inspect(t *testing.T, crs *waf.CoreRuleSet, r request) waf.Result {
|
||||
func inspect(t *testing.T, crs *coreruleset.CoreRuleSet, r request) coreruleset.Result {
|
||||
t.Helper()
|
||||
|
||||
result, _ := inspectBody(t, crs, r, "")
|
||||
@@ -59,8 +62,8 @@ func inspect(t *testing.T, crs *waf.CoreRuleSet, r request) waf.Result {
|
||||
// inspectBody is inspect for r with body, which is announced with its
|
||||
// Content-Length unless it is "", and returns what crs read of body too.
|
||||
func inspectBody(
|
||||
t *testing.T, crs *waf.CoreRuleSet, r request, body string,
|
||||
) (waf.Result, string) {
|
||||
t *testing.T, crs *coreruleset.CoreRuleSet, r request, body string,
|
||||
) (coreruleset.Result, string) {
|
||||
t.Helper()
|
||||
|
||||
req := httptest.NewRequestWithContext(t.Context(), r.method,
|
||||
@@ -95,7 +98,9 @@ func inspectBody(
|
||||
}
|
||||
|
||||
// wantResult checks what crs finds in r.
|
||||
func wantResult(t *testing.T, crs *waf.CoreRuleSet, r request, want waf.Result) {
|
||||
func wantResult(
|
||||
t *testing.T, crs *coreruleset.CoreRuleSet, r request, want coreruleset.Result,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
if got := inspect(t, crs, r); !reflect.DeepEqual(got, want) {
|
||||
@@ -105,28 +110,29 @@ func wantResult(t *testing.T, crs *waf.CoreRuleSet, r request, want waf.Result)
|
||||
|
||||
// matched is the result of a request that the rules ids match, each of
|
||||
// them a critical one, which adds 5 to the score.
|
||||
func matched(ids ...int) waf.Result {
|
||||
func matched(ids ...int) coreruleset.Result {
|
||||
const critical = 5
|
||||
|
||||
return waf.Result{RuleIDs: ids, Score: critical * len(ids)}
|
||||
return coreruleset.Result{RuleIDs: ids, Score: critical * len(ids)}
|
||||
}
|
||||
|
||||
// atDefaults returns the Core Rule Set as sww runs it by default.
|
||||
func atDefaults(t *testing.T) *waf.CoreRuleSet {
|
||||
func atDefaults(t *testing.T) *coreruleset.CoreRuleSet {
|
||||
t.Helper()
|
||||
|
||||
return newCoreRuleSet(t, 1, defaultDisabledRules...)
|
||||
}
|
||||
|
||||
// wantChange checks that crs lets through passes, a gitea request one of
|
||||
// the six changes is for, and still finds result in refused, a request
|
||||
// the eight changes is for, and still finds result in refused, a request
|
||||
// like it that the change is not for.
|
||||
func wantChange(
|
||||
t *testing.T, crs *waf.CoreRuleSet, passes, refused request, result waf.Result,
|
||||
t *testing.T, crs *coreruleset.CoreRuleSet, passes, refused request,
|
||||
result coreruleset.Result,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
wantResult(t, crs, passes, waf.Result{})
|
||||
wantResult(t, crs, passes, coreruleset.Result{})
|
||||
wantResult(t, crs, refused, result)
|
||||
}
|
||||
|
||||
@@ -161,12 +167,12 @@ func TestExpectAndContentEncodingAreAllowed(t *testing.T) {
|
||||
|
||||
wantResult(t, crs,
|
||||
request{http.MethodPost, push, []string{pushType, length, "Expect: 100-continue"}},
|
||||
waf.Result{})
|
||||
coreruleset.Result{})
|
||||
wantResult(t, crs,
|
||||
request{http.MethodPost, fetch, []string{
|
||||
fetchType, length, "Content-Encoding: gzip",
|
||||
}},
|
||||
waf.Result{})
|
||||
coreruleset.Result{})
|
||||
|
||||
// Every other header on the Core Rule Set's list stays refused.
|
||||
for _, header := range []string{
|
||||
@@ -195,7 +201,7 @@ func TestTransferEncodingIsRead(t *testing.T) {
|
||||
"Content-Type: application/x-git-receive-pack-request",
|
||||
"Transfer-Encoding: chunked",
|
||||
}},
|
||||
waf.Result{})
|
||||
coreruleset.Result{})
|
||||
}
|
||||
|
||||
func TestMoreParametersThanCorazaKeepsIsAMatch(t *testing.T) {
|
||||
@@ -249,7 +255,7 @@ func TestParametersGiteaSendsNamesInSkipTheListsOfFilesPathsAndCommands(t *testi
|
||||
for _, value := range []struct {
|
||||
name string
|
||||
// result is what a parameter that is not one of gitea's gets.
|
||||
result waf.Result
|
||||
result coreruleset.Result
|
||||
}{
|
||||
// A file on the list of system files.
|
||||
{".gitignore", matched(930120)},
|
||||
@@ -270,7 +276,7 @@ func TestParametersGiteaSendsNamesInSkipTheListsOfFilesPathsAndCommands(t *testi
|
||||
wantChange(t, crs, get("/?path=|cat%20/etc/passwd"), get("/?q=|cat%20/etc/passwd"),
|
||||
matched(930120, 932160))
|
||||
wantResult(t, crs, get("/?path=../../etc/passwd"),
|
||||
waf.Result{RuleIDs: []int{930100, 930110}, Score: 20})
|
||||
coreruleset.Result{RuleIDs: []int{930100, 930110}, Score: 20})
|
||||
wantResult(t, crs, get("/?path=1'%20OR%20'1'='1"), matched(942100))
|
||||
}
|
||||
|
||||
@@ -308,13 +314,14 @@ func TestCookiesGiteaFlashAndRedirectToAreNotRead(t *testing.T) {
|
||||
matched(930120))
|
||||
}
|
||||
|
||||
func TestRefererIsNotCheckedForACommandOrJavaStartingAProcess(t *testing.T) {
|
||||
func TestRefererIsNotCheckedForACommandOrJava(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
search = "https://git.example/explore/repos?q=env"
|
||||
runtimeJava = "https://git.example/openjdk/jdk/src/branch/master/src/" +
|
||||
"java.base/share/classes/java/lang/Runtime.java"
|
||||
stringMD = "https://git.example/owner/repo/src/branch/main/java.lang.String.md"
|
||||
)
|
||||
|
||||
crs := atDefaults(t)
|
||||
@@ -323,6 +330,8 @@ func TestRefererIsNotCheckedForACommandOrJavaStartingAProcess(t *testing.T) {
|
||||
matched(932340))
|
||||
wantChange(t, crs, get("/", "Referer: "+runtimeJava),
|
||||
get("/", "X-Page: "+runtimeJava), matched(944110))
|
||||
wantChange(t, crs, get("/", "Referer: "+stringMD), get("/", "X-Page: "+stringMD),
|
||||
matched(944130))
|
||||
|
||||
// It is still checked for script and SQL injection.
|
||||
wantResult(t, crs,
|
||||
@@ -337,7 +346,7 @@ func TestEmptyHeaderIsRead(t *testing.T) {
|
||||
|
||||
// An empty User-Agent is a notice, which adds 2.
|
||||
wantResult(t, atDefaults(t), get("/", "User-Agent: "),
|
||||
waf.Result{RuleIDs: []int{920330}, Score: 2})
|
||||
coreruleset.Result{RuleIDs: []int{920330}, Score: 2})
|
||||
}
|
||||
|
||||
func TestParanoiaLevel(t *testing.T) {
|
||||
@@ -346,7 +355,7 @@ func TestParanoiaLevel(t *testing.T) {
|
||||
// Accept-Charset is refused from paranoia level 2.
|
||||
r := get("/", "Accept-Charset: utf-8")
|
||||
|
||||
wantResult(t, newCoreRuleSet(t, 1), r, waf.Result{})
|
||||
wantResult(t, newCoreRuleSet(t, 1), r, coreruleset.Result{})
|
||||
wantResult(t, newCoreRuleSet(t, 2), r, matched(920451))
|
||||
}
|
||||
|
||||
@@ -358,11 +367,195 @@ func TestEachDisabledRuleIsSwitchedOff(t *testing.T) {
|
||||
r := request{http.MethodTrace, "/", []string{"Host: 192.0.2.1"}}
|
||||
|
||||
wantResult(t, newCoreRuleSet(t, 1), r,
|
||||
waf.Result{RuleIDs: []int{911100, 920350}, Score: 8})
|
||||
wantResult(t, newCoreRuleSet(t, 1, 920350, 911100), r, waf.Result{})
|
||||
coreruleset.Result{RuleIDs: []int{911100, 920350}, Score: 8})
|
||||
wantResult(t, newCoreRuleSet(t, 1, 920350, 911100), r, coreruleset.Result{})
|
||||
}
|
||||
|
||||
// bodyLimit is SWWAF_WAF_BODY_LIMIT in the tests that read bodies.
|
||||
func TestImagePushThatReusesALayerIsNotCheckedForCommandNames(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
crs := atDefaults(t)
|
||||
reuse := "?mount=sha256:0000&from=owner/python3-app"
|
||||
|
||||
wantChange(t, crs, get("/v2/owner/app/blobs/uploads/"+reuse), get("/upload/"+reuse),
|
||||
matched(932260))
|
||||
|
||||
// A command after the image's name is still refused.
|
||||
wantResult(t, crs,
|
||||
get("/v2/owner/app/blobs/uploads/?from=owner/app;cat%20/etc/passwd"),
|
||||
matched(930120, 932160))
|
||||
}
|
||||
|
||||
// withExclusions returns the Core Rule Set at the defaults, with the
|
||||
// exclusions file at path.
|
||||
func withExclusions(path string) (*coreruleset.CoreRuleSet, error) {
|
||||
return coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, ExclusionsFile: path,
|
||||
})
|
||||
}
|
||||
|
||||
// exclusionsFile writes text to a file of exclusions and returns its path.
|
||||
func exclusionsFile(t *testing.T, text string) string {
|
||||
t.Helper()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "exclusions.conf")
|
||||
|
||||
err := os.WriteFile(path, []byte(text), 0o600)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
return path
|
||||
}
|
||||
|
||||
// backupFileExclusion is an exclusions file that switches 920500, a rule
|
||||
// of phase 1, off for the path of a backup file under /files/.
|
||||
const backupFileExclusion = `SecRule REQUEST_FILENAME "@beginsWith /files/" ` +
|
||||
`"id:1,phase:1,pass,nolog,ctl:ruleRemoveTargetById=920500;REQUEST_FILENAME"` + "\n"
|
||||
|
||||
func TestExclusionsFileIsLoadedBeforeTheRules(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
crs, err := withExclusions(exclusionsFile(t, backupFileExclusion))
|
||||
if err != nil {
|
||||
t.Fatalf("load the exclusions: %v", err)
|
||||
}
|
||||
|
||||
wantChange(t, crs, get("/files/notes.txt~"), get("/notes.txt~"), matched(920500))
|
||||
}
|
||||
|
||||
func TestRuleSwitchedOffAndNamedInTheExclusionsFileIsOff(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
crs, err := coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: 1, DisabledRules: []int{920500},
|
||||
ExclusionsFile: exclusionsFile(t, backupFileExclusion),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("load the exclusions: %v", err)
|
||||
}
|
||||
|
||||
wantResult(t, crs, get("/notes.txt~"), coreruleset.Result{})
|
||||
}
|
||||
|
||||
func TestExclusionsFileMayNameAnyNumberInAComment(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := withExclusions(exclusionsFile(t,
|
||||
"# 942100: GET /issues/900123\n"+backupFileExclusion))
|
||||
if err != nil {
|
||||
t.Errorf("the exclusions did not load: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExclusionsFileThatCannotBeUsedIsAnError(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
missing := filepath.Join(t.TempDir(), "missing.conf")
|
||||
|
||||
_, err := withExclusions(missing)
|
||||
if err == nil {
|
||||
t.Errorf("%s loaded", missing)
|
||||
}
|
||||
|
||||
for _, tc := range []struct{ text, want string }{
|
||||
{"SecNoSuchDirective On\n", "unknown directive"},
|
||||
// sww's own rules may not be switched off or changed, by
|
||||
// id or in a range, on one line or split over several.
|
||||
{"SecRuleRemoveById 899999-901000\n", "names 899999-901000"},
|
||||
{
|
||||
"SecAction \"id:1,phase:1,pass,nolog,ctl:ruleRemoveById=899999-901000\"\n",
|
||||
"names 899999-901000",
|
||||
},
|
||||
{
|
||||
"SecAction \"id:1,phase:1,pass,nolog,ctl:ruleRemoveById=899999-+901000\"\n",
|
||||
"names 899999-+901000",
|
||||
},
|
||||
{
|
||||
"SecAction \"id:1,phase:1,pass,nolog,ctl:ruleRemoveTargetById=900500;ARGS:from\"\n",
|
||||
"names 900500",
|
||||
},
|
||||
{"SecRuleRemoveById 899000-\\\n 901000\n", "names 899000-901000"},
|
||||
{
|
||||
"SecAction \"id:1,phase:1,pass,nolog,\\\n" +
|
||||
" ctl:ruleRemoveById=9005\\\n 00\"\n",
|
||||
"names 900500",
|
||||
},
|
||||
// The Core Rule Set's rules load after the file.
|
||||
{"SecRuleRemoveById 942100\n", "SecRuleRemoveById in the exclusions file"},
|
||||
{
|
||||
"SecRuleUpdateTargetById 942100 \"!ARGS:id\"\n",
|
||||
"SecRuleUpdateTargetById in the exclusions file",
|
||||
},
|
||||
} {
|
||||
_, err := withExclusions(exclusionsFile(t, tc.text))
|
||||
if err == nil || !strings.Contains(err.Error(), tc.want) {
|
||||
t.Errorf("%q gave the error %v, want one with %q", tc.text, err, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// giteaRequests returns the recorded ordinary gitea requests in testdata.
|
||||
// Each has the Referer of a page that names the same, as a page opened
|
||||
// from it has.
|
||||
func giteaRequests(t *testing.T) []request {
|
||||
t.Helper()
|
||||
|
||||
text, err := os.ReadFile(filepath.Join("testdata", "gitea-requests.txt"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var requests []request
|
||||
|
||||
for line := range strings.Lines(string(text)) {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
|
||||
target, header, _ := strings.Cut(line, " ")
|
||||
|
||||
headers := []string{"Referer: http://git.example" + target}
|
||||
if header != "" {
|
||||
headers = append(headers, header)
|
||||
}
|
||||
|
||||
requests = append(requests, get(target, headers...))
|
||||
}
|
||||
|
||||
return requests
|
||||
}
|
||||
|
||||
func TestOrdinaryGiteaRequestsPassAndAttacksAreRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const threshold = 5 // SWW_WAF_ANOMALY_THRESHOLD's default
|
||||
|
||||
crs := atDefaults(t)
|
||||
|
||||
for _, r := range giteaRequests(t) {
|
||||
if got := inspect(t, crs, r); got.Score >= threshold {
|
||||
t.Errorf("%s %q refused: %+v", r.target, r.headers[1:], got)
|
||||
}
|
||||
}
|
||||
|
||||
// In q too, off gitea's search pages.
|
||||
for _, parameter := range []string{"id", "q"} {
|
||||
for _, attack := range []string{
|
||||
"1'%20OR%20'1'='1",
|
||||
"..%2Fcustom%2Fconf%2Fapp.ini",
|
||||
"%3Cscript%3Ealert(1)%3C%2Fscript%3E",
|
||||
} {
|
||||
target := "/owner/repo?" + parameter + "=" + attack
|
||||
if got := inspect(t, crs, get(target)); got.Score < threshold {
|
||||
t.Errorf("%s let through: %+v", target, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// bodyLimit is SWW_WAF_BODY_LIMIT in the tests that read bodies.
|
||||
const bodyLimit = 8 << 10
|
||||
|
||||
// The Content-Type headers of the kinds of body the Core Rule Set reads.
|
||||
@@ -378,10 +571,10 @@ const injection = "1' OR '1'='1"
|
||||
|
||||
// readingBodies returns the Core Rule Set as sww runs it by
|
||||
// default, but reading bodies up to bodyLimit.
|
||||
func readingBodies(t *testing.T) *waf.CoreRuleSet {
|
||||
func readingBodies(t *testing.T) *coreruleset.CoreRuleSet {
|
||||
t.Helper()
|
||||
|
||||
crs, err := waf.New(waf.Params{
|
||||
crs, err := coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyLimit: bodyLimit,
|
||||
})
|
||||
if err != nil {
|
||||
@@ -415,7 +608,8 @@ func padded(head, tail string, n int) string {
|
||||
// wantBody checks what crs finds in r with body, and that what it read of
|
||||
// body is read.
|
||||
func wantBody(
|
||||
t *testing.T, crs *waf.CoreRuleSet, r request, body string, want waf.Result,
|
||||
t *testing.T, crs *coreruleset.CoreRuleSet, r request, body string,
|
||||
want coreruleset.Result,
|
||||
read string,
|
||||
) {
|
||||
t.Helper()
|
||||
@@ -439,7 +633,7 @@ func TestBodiesAreReadOnlyWhileBodyLimitIsSet(t *testing.T) {
|
||||
{jsonBody, `{"q":"` + injection + `"}`},
|
||||
{xmlBody, "<q>" + injection + "</q>"},
|
||||
} {
|
||||
wantBody(t, off, post(tc.header), tc.body, waf.Result{}, "")
|
||||
wantBody(t, off, post(tc.header), tc.body, coreruleset.Result{}, "")
|
||||
wantBody(t, on, post(tc.header), tc.body, matched(942100), tc.body)
|
||||
}
|
||||
}
|
||||
@@ -462,7 +656,7 @@ func TestFormDataAndMultipartAreReadUpToTheLimit(t *testing.T) {
|
||||
} {
|
||||
wantBody(t, crs, post(tc.header), tc.attackFirst, matched(942100),
|
||||
tc.attackFirst[:bodyLimit+1])
|
||||
wantBody(t, crs, post(tc.header), tc.attackLast, waf.Result{},
|
||||
wantBody(t, crs, post(tc.header), tc.attackLast, coreruleset.Result{},
|
||||
tc.attackLast[:bodyLimit+1])
|
||||
}
|
||||
|
||||
@@ -472,7 +666,7 @@ func TestFormDataAndMultipartAreReadUpToTheLimit(t *testing.T) {
|
||||
wantBody(t, crs, post(formData), atLimit, matched(930120, 932160), atLimit)
|
||||
|
||||
pastLimit := padded("pad=", "&q=/etc/passwd", bodyLimit+1)
|
||||
wantBody(t, crs, post(formData), pastLimit, waf.Result{}, pastLimit)
|
||||
wantBody(t, crs, post(formData), pastLimit, coreruleset.Result{}, pastLimit)
|
||||
}
|
||||
|
||||
func TestJSONAndXMLAreReadOnlyWhenNoLargerThanTheLimit(t *testing.T) {
|
||||
@@ -488,7 +682,7 @@ func TestJSONAndXMLAreReadOnlyWhenNoLargerThanTheLimit(t *testing.T) {
|
||||
wantBody(t, crs, post(tc.header), fits, matched(942100), fits)
|
||||
|
||||
larger := padded(tc.head, tc.tail, bodyLimit+1)
|
||||
wantBody(t, crs, post(tc.header), larger, waf.Result{}, larger)
|
||||
wantBody(t, crs, post(tc.header), larger, coreruleset.Result{}, larger)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -506,7 +700,7 @@ func TestOtherBodiesAreNotRead(t *testing.T) {
|
||||
"Content-Type: text/plain",
|
||||
"Content-Type: application/x-git-receive-pack-request",
|
||||
} {
|
||||
wantBody(t, crs, post(header), body, waf.Result{}, "")
|
||||
wantBody(t, crs, post(header), body, coreruleset.Result{}, "")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -534,8 +728,8 @@ func TestContentEncodingIsRefusedOnTheKindsOfBodyTheCoreRuleSetReads(t *testing.
|
||||
// It is allowed on a body of any other kind, and on every body while no
|
||||
// body is read.
|
||||
fetch := "Content-Type: application/x-git-upload-pack-request"
|
||||
wantBody(t, crs, post(fetch, gzip), "a", waf.Result{}, "")
|
||||
wantBody(t, atDefaults(t), post(formData, gzip), "a", waf.Result{}, "")
|
||||
wantBody(t, crs, post(fetch, gzip), "a", coreruleset.Result{}, "")
|
||||
wantBody(t, atDefaults(t), post(formData, gzip), "a", coreruleset.Result{}, "")
|
||||
}
|
||||
|
||||
func TestParametersGiteaSendsNamesInAreLeftOutAmongFormFieldsToo(t *testing.T) {
|
||||
@@ -546,18 +740,18 @@ func TestParametersGiteaSendsNamesInAreLeftOutAmongFormFieldsToo(t *testing.T) {
|
||||
|
||||
for _, tc := range []struct {
|
||||
body string
|
||||
want waf.Result
|
||||
want coreruleset.Result
|
||||
}{
|
||||
{"path=.gitignore", waf.Result{}},
|
||||
{"path=.gitignore", coreruleset.Result{}},
|
||||
{"q=.gitignore", matched(930120)},
|
||||
{"redirect_uri=" + local, waf.Result{}},
|
||||
{"redirect_uri=" + local, coreruleset.Result{}},
|
||||
{"next=" + local, matched(931100, 934110)},
|
||||
} {
|
||||
wantBody(t, crs, post(formData), tc.body, tc.want, tc.body)
|
||||
}
|
||||
|
||||
body := field("path", ".gitignore") + end
|
||||
wantBody(t, crs, post(multipart), body, waf.Result{}, body)
|
||||
wantBody(t, crs, post(multipart), body, coreruleset.Result{}, body)
|
||||
|
||||
body = field("q", ".gitignore") + end
|
||||
wantBody(t, crs, post(multipart), body, matched(930120), body)
|
||||
@@ -681,7 +875,7 @@ func TestCorazaWritesNoFile(t *testing.T) {
|
||||
func TestBodyLimitOf1GLoads(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := waf.New(waf.Params{ParanoiaLevel: 1, BodyLimit: 1 << 30})
|
||||
_, err := coreruleset.New(coreruleset.Params{ParanoiaLevel: 1, BodyLimit: 1 << 30})
|
||||
if err != nil {
|
||||
t.Errorf("load the Core Rule Set reading bodies up to 1G: %v", err)
|
||||
}
|
||||
@@ -0,0 +1,283 @@
|
||||
# The eighth change: on gitea's paths, some fields skip the rules that refused
|
||||
# ordinary gitea requests there. They are the fields where gitea sends a search,
|
||||
# a file name or a branch name. Each exclusion runs with the request, before the
|
||||
# rules it switches off, those of phase 1 included, so this file is loaded before
|
||||
# the Core Rule Set's rules. Each comment names a request the rule refused.
|
||||
|
||||
# gitea's searches of code, of issues and pull requests, and of commits send the
|
||||
# text searched for in q, which skips these rules there:
|
||||
# 930110: ../
|
||||
# 930120: .gitignore
|
||||
# 931100: http://127.0.0.1:8080
|
||||
# 932120: Invoke-WebRequest
|
||||
# 932130: $(pwd)
|
||||
# 932140: if (a == b) {
|
||||
# 932160: /bin/sh
|
||||
# 932235: `make check`
|
||||
# 932250: ls -la
|
||||
# 932260: python3
|
||||
# 932380: `whoami`
|
||||
# 933100: <?php echo
|
||||
# 933130: $_GET[
|
||||
# 933135: ${VAR}
|
||||
# 934100: process.env
|
||||
# 934110: http://127.0.0.1:8080
|
||||
# 934130: __proto__
|
||||
# 941100: <script>
|
||||
# 941110: <script>
|
||||
# 941160: <img src=
|
||||
# 941180: document.cookie
|
||||
# 941390: fetch(
|
||||
# 942100: SELECT * FROM users WHERE
|
||||
# 942140: information_schema
|
||||
# 942190: union select
|
||||
# 942230: if (a == b) {
|
||||
# 944100: java.lang.Runtime
|
||||
# 944110: java.lang.Runtime
|
||||
# 944130: java.lang.Runtime
|
||||
SecRule REQUEST_FILENAME "@rx ^/(?:explore/code|issues(?:/search)?|pulls|\
|
||||
[^/]+/-/code|\
|
||||
[^/]+/[^/]+/(?:search|issues(?:/search)?|pulls|\
|
||||
commits/(?:branch|tag|commit)/.+/search)|\
|
||||
api/v1/repos/(?:issues/search|[^/]+/[^/]+/issues))$" \
|
||||
"id:900600,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=930110;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=931100;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=932120;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=932130;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=932140;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=932235;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=932250;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=932380;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=933100;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=933130;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=933135;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=934100;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=934110;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=934130;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=941100;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=941110;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=941160;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=941180;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=941390;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=942100;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=942140;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=942190;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=942230;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=944100;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=944110;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=944130;ARGS:q"
|
||||
|
||||
# A page opened from the results of one of those searches has the search's
|
||||
# address in Referer, which skips these rules then:
|
||||
# 941110: <script>
|
||||
# 941120: onClick={
|
||||
# 941160: <img src=
|
||||
# 942100: LIKE '%foo%'
|
||||
# 944100: java.lang.Runtime
|
||||
SecRule REQUEST_HEADERS:Referer "@rx ^https?://[^/]+/(?:explore/code|\
|
||||
issues(?:/search)?|pulls|[^/]+/-/code|\
|
||||
[^/]+/[^/]+/(?:search|issues(?:/search)?|pulls|\
|
||||
commits/(?:branch|tag|commit)/.+/search)|\
|
||||
api/v1/repos/(?:issues/search|[^/]+/[^/]+/issues))\?" \
|
||||
"id:900601,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=941110;REQUEST_HEADERS:Referer,\
|
||||
ctl:ruleRemoveTargetById=941120;REQUEST_HEADERS:Referer,\
|
||||
ctl:ruleRemoveTargetById=941160;REQUEST_HEADERS:Referer,\
|
||||
ctl:ruleRemoveTargetById=942100;REQUEST_HEADERS:Referer,\
|
||||
ctl:ruleRemoveTargetById=944100;REQUEST_HEADERS:Referer"
|
||||
|
||||
# gitea's searches of repositories, users, branches, tags and packages send a
|
||||
# name in q, which skips these rules there:
|
||||
# 930120: .profile
|
||||
# 932120: powershell-scripts
|
||||
# 932260: python3-app
|
||||
# 944130: org.apache.commons.logging
|
||||
SecRule REQUEST_FILENAME "@rx ^/(?:explore/repos|user/search_candidates|\
|
||||
[^/]+/-/packages|[^/]+/[^/]+/(?:branches|tags)|api/v1/repos/search)$" \
|
||||
"id:900602,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=932120;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=944130;ARGS:q"
|
||||
|
||||
# gitea's other searches for a name send it in q too, which skips only these
|
||||
# rules there:
|
||||
# 930120: .profile
|
||||
# 932120: powershell-scripts
|
||||
# 932260: python3-app
|
||||
# The searches:
|
||||
# - the dashboard's filter of repositories;
|
||||
# - the repositories on a user's or an organisation's page, whose path has
|
||||
# one segment;
|
||||
# - the author filter of the issue and pull request lists;
|
||||
# - the lists of users and organisations, and the API's search for users.
|
||||
SecRule REQUEST_FILENAME "@rx ^/(?:[^/]+|repo/search|\
|
||||
explore/(?:users|organizations)|api/v1/users/search|\
|
||||
[^/]+/[^/]+/(?:issues|pulls)/posters)$" \
|
||||
"id:900603,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=932120;ARGS:q,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:q"
|
||||
|
||||
# These requests name a file in the path, which skips these rules there:
|
||||
# 920500: notes.txt~
|
||||
# 934100: process.env.d.ts
|
||||
# 941130: docs/a b.xhtml
|
||||
# 944130: java.lang.String.md
|
||||
# The requests:
|
||||
# - a file's page, raw file and download;
|
||||
# - its blame, history, edit, preview and delete pages;
|
||||
# - the API's call for its contents;
|
||||
# - a release attachment's download.
|
||||
SecRule REQUEST_FILENAME \
|
||||
"@rx ^/(?:api/v1/repos/)?[^/]+/[^/]+/(?:src|raw|media|blame|commits|_edit|_preview|_delete|contents|releases/download)/" \
|
||||
"id:900610,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=920500;REQUEST_FILENAME,\
|
||||
ctl:ruleRemoveTargetById=934100;REQUEST_FILENAME,\
|
||||
ctl:ruleRemoveTargetById=941130;REQUEST_FILENAME,\
|
||||
ctl:ruleRemoveTargetById=944130;REQUEST_FILENAME"
|
||||
|
||||
# The requests below name a file in a query parameter. These rules refused
|
||||
# them for the file named:
|
||||
# 930120: .gitignore
|
||||
# 932120: install-script.sh
|
||||
# 932130: [draft].md
|
||||
# 932160: bin/docker-entrypoint
|
||||
# 932260: docker-compose.yml
|
||||
# 934100: process.env.d.ts
|
||||
# 934130: __proto__.json
|
||||
# 941130: index.xhtml
|
||||
# 941180: document.write.js
|
||||
# 942140: information_schema.sql
|
||||
# 944110: java/lang/Runtime.java
|
||||
# 944130: java.lang.String.md
|
||||
# Each request skips the rules that refused it. The fourth change already has
|
||||
# path and redirect_to skip 930120, 932160 and 932260 on every path.
|
||||
|
||||
# A file's commit graph, in file.
|
||||
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/graph$" \
|
||||
"id:900620,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:file,\
|
||||
ctl:ruleRemoveTargetById=932120;ARGS:file,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:file,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:file,\
|
||||
ctl:ruleRemoveTargetById=934100;ARGS:file,\
|
||||
ctl:ruleRemoveTargetById=934130;ARGS:file,\
|
||||
ctl:ruleRemoveTargetById=941130;ARGS:file,\
|
||||
ctl:ruleRemoveTargetById=941180;ARGS:file,\
|
||||
ctl:ruleRemoveTargetById=942140;ARGS:file,\
|
||||
ctl:ruleRemoveTargetById=944110;ARGS:file,\
|
||||
ctl:ruleRemoveTargetById=944130;ARGS:file"
|
||||
|
||||
# The API's list of a file's commits, in path.
|
||||
SecRule REQUEST_FILENAME "@rx ^/api/v1/repos/[^/]+/[^/]+/commits$" \
|
||||
"id:900621,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=932120;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=934100;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=934130;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=941130;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=941180;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=942140;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=944110;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=944130;ARGS:path"
|
||||
|
||||
# The API's call for the contents of several files, in body, as JSON.
|
||||
SecRule REQUEST_FILENAME "@rx ^/api/v1/repos/[^/]+/[^/]+/file-contents$" \
|
||||
"id:900622,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:body,\
|
||||
ctl:ruleRemoveTargetById=932120;ARGS:body,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:body,\
|
||||
ctl:ruleRemoveTargetById=934100;ARGS:body,\
|
||||
ctl:ruleRemoveTargetById=934130;ARGS:body,\
|
||||
ctl:ruleRemoveTargetById=941130;ARGS:body,\
|
||||
ctl:ruleRemoveTargetById=941180;ARGS:body,\
|
||||
ctl:ruleRemoveTargetById=942140;ARGS:body,\
|
||||
ctl:ruleRemoveTargetById=944110;ARGS:body,\
|
||||
ctl:ruleRemoveTargetById=944130;ARGS:body"
|
||||
|
||||
# An issue opened from a template, in template.
|
||||
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/issues/new$" \
|
||||
"id:900623,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:template,\
|
||||
ctl:ruleRemoveTargetById=932120;ARGS:template,\
|
||||
ctl:ruleRemoveTargetById=932130;ARGS:template,\
|
||||
ctl:ruleRemoveTargetById=932160;ARGS:template,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:template,\
|
||||
ctl:ruleRemoveTargetById=934100;ARGS:template,\
|
||||
ctl:ruleRemoveTargetById=934130;ARGS:template,\
|
||||
ctl:ruleRemoveTargetById=941130;ARGS:template,\
|
||||
ctl:ruleRemoveTargetById=941180;ARGS:template,\
|
||||
ctl:ruleRemoveTargetById=942140;ARGS:template,\
|
||||
ctl:ruleRemoveTargetById=944110;ARGS:template,\
|
||||
ctl:ruleRemoveTargetById=944130;ARGS:template"
|
||||
|
||||
# The sign-in page on the way to a file's edit page, in redirect_to.
|
||||
SecRule REQUEST_FILENAME "@rx ^/user/login$" \
|
||||
"id:900624,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=932120;ARGS:redirect_to,\
|
||||
ctl:ruleRemoveTargetById=932130;ARGS:redirect_to,\
|
||||
ctl:ruleRemoveTargetById=934100;ARGS:redirect_to,\
|
||||
ctl:ruleRemoveTargetById=934130;ARGS:redirect_to,\
|
||||
ctl:ruleRemoveTargetById=941130;ARGS:redirect_to,\
|
||||
ctl:ruleRemoveTargetById=941180;ARGS:redirect_to,\
|
||||
ctl:ruleRemoveTargetById=942140;ARGS:redirect_to,\
|
||||
ctl:ruleRemoveTargetById=944110;ARGS:redirect_to,\
|
||||
ctl:ruleRemoveTargetById=944130;ARGS:redirect_to"
|
||||
|
||||
# An attachment uploaded through the API to a release, an issue or a comment,
|
||||
# in name.
|
||||
SecRule REQUEST_FILENAME \
|
||||
"@rx ^/api/v1/repos/[^/]+/[^/]+/(?:releases|issues|issues/comments)/[0-9]+/assets$" \
|
||||
"id:900625,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=930120;ARGS:name,\
|
||||
ctl:ruleRemoveTargetById=932120;ARGS:name,\
|
||||
ctl:ruleRemoveTargetById=932260;ARGS:name,\
|
||||
ctl:ruleRemoveTargetById=934100;ARGS:name,\
|
||||
ctl:ruleRemoveTargetById=934130;ARGS:name,\
|
||||
ctl:ruleRemoveTargetById=941130;ARGS:name,\
|
||||
ctl:ruleRemoveTargetById=941180;ARGS:name,\
|
||||
ctl:ruleRemoveTargetById=942140;ARGS:name,\
|
||||
ctl:ruleRemoveTargetById=944130;ARGS:name"
|
||||
|
||||
# A diff's hidden lines, fetched when shown, in path.
|
||||
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/blob_excerpt/[0-9a-f]+$" \
|
||||
"id:900626,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=932120;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=934100;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=934130;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=941130;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=941180;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=942140;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=944110;ARGS:path,\
|
||||
ctl:ruleRemoveTargetById=944130;ARGS:path"
|
||||
|
||||
# The requests below name a branch or tag in a query parameter, which skips
|
||||
# the list of command names (932260) there. It refused names such as
|
||||
# docker-build, python3-migration and feat/docker-support.
|
||||
|
||||
# Deleting or restoring a branch, in name.
|
||||
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/branches/(?:delete|restore)$" \
|
||||
"id:900630,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932260;ARGS:name"
|
||||
|
||||
# A release started from a tag, in tag.
|
||||
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/releases/new$" \
|
||||
"id:900631,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932260;ARGS:tag"
|
||||
|
||||
# A branch protection rule opened for editing, in rule_name.
|
||||
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/settings/branches/edit$" \
|
||||
"id:900632,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932260;ARGS:rule_name"
|
||||
|
||||
# A directory listing's last commits, in refSubUrl.
|
||||
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/lastcommit/" \
|
||||
"id:900633,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932260;ARGS:refSubUrl"
|
||||
|
||||
# An artifact upload from actions/upload-artifact@v3 sends the header
|
||||
# Content-Range, which the Core Rule Set refused (920450).
|
||||
SecRule REQUEST_FILENAME \
|
||||
"@rx ^/api/actions_pipeline/_apis/pipelines/workflows/[0-9]+/artifacts/[^/]+/upload$" \
|
||||
"id:900640,phase:1,pass,nolog,\
|
||||
ctl:ruleRemoveTargetById=920450;REQUEST_HEADERS_NAMES:Content-Range"
|
||||
+393
@@ -0,0 +1,393 @@
|
||||
# Ordinary gitea 1.27 requests that the Core Rule Set must let through at the
|
||||
# defaults. Each line is a path and query, and may end with one header after a
|
||||
# space. The tests of internal/coreruleset inspect each as a GET. The Core Rule Set reads
|
||||
# the query and the headers the same way for any method.
|
||||
|
||||
# Searches with ordinary words and symbols.
|
||||
/owner/repo/search?q=package.json
|
||||
/owner/repo/search?q=.gitignore
|
||||
/owner/repo/search?q=docker-compose.yml
|
||||
/owner/repo/search?q=python3
|
||||
/owner/repo/search?q=ssh%20key
|
||||
/owner/repo/search?q=feat%2Fdocker-support
|
||||
/owner/repo/search?q=ls%20-la
|
||||
/owner/repo/search?q=sed%20-i
|
||||
/owner/repo/search?q=%2Fbin%2Fsh
|
||||
/owner/repo/search?q=%60make%20check%60
|
||||
/owner/repo/search?q=%60whoami%60
|
||||
/owner/repo/search?q=fetch%28
|
||||
/owner/repo/search?q=%24%7BVAR%7D
|
||||
/owner/repo/search?q=%24%28pwd%29
|
||||
/owner/repo/search?q=process.env
|
||||
/owner/repo/search?q=%3Cimg%20src%3D
|
||||
/owner/repo/search?q=%3Cscript%3E
|
||||
/owner/repo/search?q=SELECT%20%2A%20FROM%20users%20WHERE
|
||||
/owner/repo/search?q=union%20select
|
||||
/owner/repo/search?q=http%3A%2F%2F127.0.0.1%3A8080
|
||||
/owner/repo/search?q=..%2F
|
||||
/owner/repo/search?q=%2Fetc%2Fpasswd
|
||||
/owner/repo/search?q=eval%28
|
||||
/owner/repo/search?q=chmod%20%2Bx
|
||||
/owner/repo/search?q=echo%20%24PATH
|
||||
/owner/repo/search?q=java.lang.Runtime
|
||||
/owner/repo/search?q=document.cookie
|
||||
/owner/repo/search?q=alert%281%29
|
||||
/owner/repo/search?q=__proto__
|
||||
/owner/repo/search?q=C%3A%5CWindows%5CSystem32
|
||||
/owner/repo/search?q=Invoke-WebRequest
|
||||
/owner/repo/search?q=if%20%28a%20%3D%3D%20b%29%20%7B
|
||||
/owner/repo/search?q=sudo%20apt%20install
|
||||
/owner/repo/search?q=~%2F.ssh%2Fconfig
|
||||
/owner/repo/search?q=.htaccess
|
||||
/owner/repo/search?q=%24_GET%5B
|
||||
/owner/repo/search?q=%24_POST
|
||||
/owner/repo/search?q=%24_SERVER%5B%27REQUEST_URI%27%5D
|
||||
/owner/repo/search?q=%3C%3Fphp%20echo
|
||||
/owner/repo/search?q=information_schema
|
||||
/owner/repo/search?q=LIKE%20%27%25foo%25%27
|
||||
/owner/repo/search?q=onClick%3D%7B
|
||||
/explore/code?q=SELECT%20%2A%20FROM%20users%20WHERE
|
||||
/explore/code?q=%3Cimg%20src%3D
|
||||
/owner/-/code?q=SELECT%20%2A%20FROM%20users%20WHERE
|
||||
/owner/-/code?q=%3Cimg%20src%3D
|
||||
/owner/repo/issues?type=all&state=open&q=SELECT%20%2A%20FROM%20users%20WHERE
|
||||
/owner/repo/issues?type=all&state=open&q=%3Cimg%20src%3D
|
||||
/owner/repo/issues/search?q=SELECT%20%2A%20FROM%20users%20WHERE
|
||||
/owner/repo/issues/search?q=%3Cimg%20src%3D
|
||||
/owner/repo/pulls?q=SELECT%20%2A%20FROM%20users%20WHERE
|
||||
/owner/repo/pulls?q=%3Cimg%20src%3D
|
||||
/issues?type=your_repositories&q=SELECT%20%2A%20FROM%20users%20WHERE
|
||||
/issues?type=your_repositories&q=%3Cimg%20src%3D
|
||||
/issues/search?q=SELECT%20%2A%20FROM%20users%20WHERE
|
||||
/issues/search?q=%3Cimg%20src%3D
|
||||
/pulls?type=your_repositories&q=SELECT%20%2A%20FROM%20users%20WHERE
|
||||
/pulls?type=your_repositories&q=%3Cimg%20src%3D
|
||||
/owner/repo/commits/branch/main/search?q=SELECT%20%2A%20FROM%20users%20WHERE
|
||||
/owner/repo/commits/branch/main/search?q=%3Cimg%20src%3D
|
||||
/api/v1/repos/issues/search?q=SELECT%20%2A%20FROM%20users%20WHERE
|
||||
/api/v1/repos/issues/search?q=%3Cimg%20src%3D
|
||||
/api/v1/repos/owner/repo/issues?q=SELECT%20%2A%20FROM%20users%20WHERE
|
||||
/api/v1/repos/owner/repo/issues?q=%3Cimg%20src%3D
|
||||
|
||||
# Searches for the names of repositories, users, branches, tags and packages.
|
||||
/explore/repos?q=python3-app
|
||||
/explore/repos?q=.profile
|
||||
/explore/repos?q=powershell-scripts
|
||||
/api/v1/repos/search?q=docker-build
|
||||
/user/search_candidates?q=docker-ci
|
||||
/owner/repo/branches?q=feat%2Fdocker-support
|
||||
/owner/repo/tags?q=docker-v1.0
|
||||
/owner/-/packages?q=python3-requests
|
||||
/owner/-/packages?q=org.apache.commons.logging
|
||||
/repo/search?q=python3-app
|
||||
/repo/search?q=docker-build
|
||||
/repo/search?q=powershell-scripts
|
||||
/repo/search?q=.profile
|
||||
/owner?tab=repositories&q=python3-app
|
||||
/org?q=python3-app
|
||||
/owner/repo/issues/posters?q=docker-ci
|
||||
/owner/repo/pulls/posters?q=docker-ci
|
||||
/explore/users?q=docker-ci
|
||||
/explore/organizations?q=docker-ci
|
||||
/api/v1/users/search?q=docker-ci
|
||||
|
||||
# Pages, raw files and edit pages of files, and the API's call for their
|
||||
# contents.
|
||||
/owner/repo/src/branch/main/package.json
|
||||
/owner/repo/raw/branch/main/package.json
|
||||
/owner/repo/_edit/main/package.json
|
||||
/api/v1/repos/owner/repo/contents/package.json
|
||||
/owner/repo/src/branch/main/.gitignore
|
||||
/owner/repo/raw/branch/main/.gitignore
|
||||
/owner/repo/_edit/main/.gitignore
|
||||
/api/v1/repos/owner/repo/contents/.gitignore
|
||||
/owner/repo/src/branch/main/docker-compose.yml
|
||||
/owner/repo/raw/branch/main/docker-compose.yml
|
||||
/owner/repo/_edit/main/docker-compose.yml
|
||||
/api/v1/repos/owner/repo/contents/docker-compose.yml
|
||||
/owner/repo/src/branch/main/Dockerfile
|
||||
/owner/repo/raw/branch/main/Dockerfile
|
||||
/owner/repo/_edit/main/Dockerfile
|
||||
/api/v1/repos/owner/repo/contents/Dockerfile
|
||||
/owner/repo/src/branch/main/.htaccess
|
||||
/owner/repo/raw/branch/main/.htaccess
|
||||
/owner/repo/_edit/main/.htaccess
|
||||
/api/v1/repos/owner/repo/contents/.htaccess
|
||||
/owner/repo/src/branch/main/rootfs/etc/passwd
|
||||
/owner/repo/raw/branch/main/rootfs/etc/passwd
|
||||
/owner/repo/_edit/main/rootfs/etc/passwd
|
||||
/api/v1/repos/owner/repo/contents/rootfs/etc/passwd
|
||||
/owner/repo/src/branch/main/bin/docker-entrypoint
|
||||
/owner/repo/raw/branch/main/bin/docker-entrypoint
|
||||
/owner/repo/_edit/main/bin/docker-entrypoint
|
||||
/api/v1/repos/owner/repo/contents/bin/docker-entrypoint
|
||||
/owner/repo/src/branch/main/scripts/python3-setup.sh
|
||||
/owner/repo/raw/branch/main/scripts/python3-setup.sh
|
||||
/owner/repo/_edit/main/scripts/python3-setup.sh
|
||||
/api/v1/repos/owner/repo/contents/scripts/python3-setup.sh
|
||||
/owner/repo/src/branch/main/src/java.base/share/classes/java/lang/Runtime.java
|
||||
/owner/repo/raw/branch/main/src/java.base/share/classes/java/lang/Runtime.java
|
||||
/owner/repo/_edit/main/src/java.base/share/classes/java/lang/Runtime.java
|
||||
/api/v1/repos/owner/repo/contents/src/java.base/share/classes/java/lang/Runtime.java
|
||||
/owner/repo/src/branch/main/java.lang.String.md
|
||||
/owner/repo/raw/branch/main/java.lang.String.md
|
||||
/owner/repo/_edit/main/java.lang.String.md
|
||||
/api/v1/repos/owner/repo/contents/java.lang.String.md
|
||||
/owner/repo/src/branch/main/META-INF/services/org.apache.commons.logging.LogFactory
|
||||
/owner/repo/raw/branch/main/META-INF/services/org.apache.commons.logging.LogFactory
|
||||
/owner/repo/_edit/main/META-INF/services/org.apache.commons.logging.LogFactory
|
||||
/api/v1/repos/owner/repo/contents/META-INF/services/org.apache.commons.logging.LogFactory
|
||||
/owner/repo/src/branch/main/process.env.d.ts
|
||||
/owner/repo/raw/branch/main/process.env.d.ts
|
||||
/owner/repo/_edit/main/process.env.d.ts
|
||||
/api/v1/repos/owner/repo/contents/process.env.d.ts
|
||||
/owner/repo/src/branch/main/document.write.js
|
||||
/owner/repo/raw/branch/main/document.write.js
|
||||
/owner/repo/_edit/main/document.write.js
|
||||
/api/v1/repos/owner/repo/contents/document.write.js
|
||||
/owner/repo/src/branch/main/test/fixtures/__proto__.json
|
||||
/owner/repo/raw/branch/main/test/fixtures/__proto__.json
|
||||
/owner/repo/_edit/main/test/fixtures/__proto__.json
|
||||
/api/v1/repos/owner/repo/contents/test/fixtures/__proto__.json
|
||||
/owner/repo/src/branch/main/install-script.sh
|
||||
/owner/repo/raw/branch/main/install-script.sh
|
||||
/owner/repo/_edit/main/install-script.sh
|
||||
/api/v1/repos/owner/repo/contents/install-script.sh
|
||||
/owner/repo/src/branch/main/src/backend/catalog/information_schema.sql
|
||||
/owner/repo/raw/branch/main/src/backend/catalog/information_schema.sql
|
||||
/owner/repo/_edit/main/src/backend/catalog/information_schema.sql
|
||||
/api/v1/repos/owner/repo/contents/src/backend/catalog/information_schema.sql
|
||||
/owner/repo/src/branch/main/docs/a%20b.xhtml
|
||||
/owner/repo/raw/branch/main/docs/a%20b.xhtml
|
||||
/owner/repo/_edit/main/docs/a%20b.xhtml
|
||||
/api/v1/repos/owner/repo/contents/docs/a%20b.xhtml
|
||||
/owner/repo/src/branch/main/%5Bdraft%5D.md
|
||||
/owner/repo/raw/branch/main/%5Bdraft%5D.md
|
||||
/owner/repo/_edit/main/%5Bdraft%5D.md
|
||||
/api/v1/repos/owner/repo/contents/%5Bdraft%5D.md
|
||||
/owner/repo/src/branch/main/notes.txt~
|
||||
/owner/repo/raw/branch/main/notes.txt~
|
||||
/owner/repo/_edit/main/notes.txt~
|
||||
/api/v1/repos/owner/repo/contents/notes.txt~
|
||||
|
||||
# The download, blame, history, delete and preview pages of the files above
|
||||
# whose names the Core Rule Set checks in the path, and their downloads as
|
||||
# release attachments.
|
||||
/owner/repo/media/branch/main/notes.txt~
|
||||
/owner/repo/blame/branch/main/notes.txt~
|
||||
/owner/repo/commits/branch/main/notes.txt~
|
||||
/owner/repo/_delete/main/notes.txt~
|
||||
/owner/repo/media/branch/main/process.env.d.ts
|
||||
/owner/repo/blame/branch/main/process.env.d.ts
|
||||
/owner/repo/commits/branch/main/process.env.d.ts
|
||||
/owner/repo/_delete/main/process.env.d.ts
|
||||
/owner/repo/media/branch/main/docs/a%20b.xhtml
|
||||
/owner/repo/blame/branch/main/docs/a%20b.xhtml
|
||||
/owner/repo/commits/branch/main/docs/a%20b.xhtml
|
||||
/owner/repo/_delete/main/docs/a%20b.xhtml
|
||||
/owner/repo/media/branch/main/java.lang.String.md
|
||||
/owner/repo/blame/branch/main/java.lang.String.md
|
||||
/owner/repo/commits/branch/main/java.lang.String.md
|
||||
/owner/repo/_delete/main/java.lang.String.md
|
||||
/owner/repo/_preview/main/notes.txt~
|
||||
/owner/repo/_preview/main/process.env.d.ts
|
||||
/owner/repo/_preview/main/java.lang.String.md
|
||||
/owner/repo/releases/download/v1.0/notes.txt~
|
||||
/owner/repo/releases/download/v1.0/process.env.d.ts
|
||||
/owner/repo/releases/download/v1.0/java.lang.String.md
|
||||
/owner/repo/releases/download/v1.0/a%20b.xhtml
|
||||
|
||||
# File names in the query string: the commit graph of a file, the API's lists
|
||||
# of its commits and of its contents, an issue opened from a template, the
|
||||
# sign-in page on the way to its edit page, and a diff's hidden lines.
|
||||
/owner/repo/graph?file=package.json
|
||||
/api/v1/repos/owner/repo/commits?path=package.json
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22package.json%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fpackage.json
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fpackage.json
|
||||
/owner/repo/graph?file=.gitignore
|
||||
/api/v1/repos/owner/repo/commits?path=.gitignore
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22.gitignore%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2F.gitignore
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2F.gitignore
|
||||
/owner/repo/graph?file=docker-compose.yml
|
||||
/api/v1/repos/owner/repo/commits?path=docker-compose.yml
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22docker-compose.yml%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fdocker-compose.yml
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fdocker-compose.yml
|
||||
/owner/repo/graph?file=.htaccess
|
||||
/api/v1/repos/owner/repo/commits?path=.htaccess
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22.htaccess%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2F.htaccess
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2F.htaccess
|
||||
/owner/repo/graph?file=rootfs%2Fetc%2Fpasswd
|
||||
/api/v1/repos/owner/repo/commits?path=rootfs%2Fetc%2Fpasswd
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22rootfs%2Fetc%2Fpasswd%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Frootfs%2Fetc%2Fpasswd
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Frootfs%2Fetc%2Fpasswd
|
||||
/owner/repo/graph?file=bin%2Fdocker-entrypoint
|
||||
/api/v1/repos/owner/repo/commits?path=bin%2Fdocker-entrypoint
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22bin%2Fdocker-entrypoint%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fbin%2Fdocker-entrypoint
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fbin%2Fdocker-entrypoint
|
||||
/owner/repo/graph?file=scripts%2Fpython3-setup.sh
|
||||
/api/v1/repos/owner/repo/commits?path=scripts%2Fpython3-setup.sh
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22scripts%2Fpython3-setup.sh%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fscripts%2Fpython3-setup.sh
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fscripts%2Fpython3-setup.sh
|
||||
/owner/repo/graph?file=src%2Fjava.base%2Fshare%2Fclasses%2Fjava%2Flang%2FRuntime.java
|
||||
/api/v1/repos/owner/repo/commits?path=src%2Fjava.base%2Fshare%2Fclasses%2Fjava%2Flang%2FRuntime.java
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22src%2Fjava.base%2Fshare%2Fclasses%2Fjava%2Flang%2FRuntime.java%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fsrc%2Fjava.base%2Fshare%2Fclasses%2Fjava%2Flang%2FRuntime.java
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fsrc%2Fjava.base%2Fshare%2Fclasses%2Fjava%2Flang%2FRuntime.java
|
||||
/owner/repo/graph?file=java.lang.String.md
|
||||
/api/v1/repos/owner/repo/commits?path=java.lang.String.md
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22java.lang.String.md%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fjava.lang.String.md
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fjava.lang.String.md
|
||||
/owner/repo/graph?file=META-INF%2Fservices%2Forg.apache.commons.logging.LogFactory
|
||||
/api/v1/repos/owner/repo/commits?path=META-INF%2Fservices%2Forg.apache.commons.logging.LogFactory
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22META-INF%2Fservices%2Forg.apache.commons.logging.LogFactory%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2FMETA-INF%2Fservices%2Forg.apache.commons.logging.LogFactory
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2FMETA-INF%2Fservices%2Forg.apache.commons.logging.LogFactory
|
||||
/owner/repo/graph?file=process.env.d.ts
|
||||
/api/v1/repos/owner/repo/commits?path=process.env.d.ts
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22process.env.d.ts%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fprocess.env.d.ts
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fprocess.env.d.ts
|
||||
/owner/repo/graph?file=document.write.js
|
||||
/api/v1/repos/owner/repo/commits?path=document.write.js
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22document.write.js%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fdocument.write.js
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fdocument.write.js
|
||||
/owner/repo/graph?file=test%2Ffixtures%2F__proto__.json
|
||||
/api/v1/repos/owner/repo/commits?path=test%2Ffixtures%2F__proto__.json
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22test%2Ffixtures%2F__proto__.json%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Ftest%2Ffixtures%2F__proto__.json
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Ftest%2Ffixtures%2F__proto__.json
|
||||
/owner/repo/graph?file=install-script.sh
|
||||
/api/v1/repos/owner/repo/commits?path=install-script.sh
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22install-script.sh%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Finstall-script.sh
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Finstall-script.sh
|
||||
/owner/repo/graph?file=src%2Fbackend%2Fcatalog%2Finformation_schema.sql
|
||||
/api/v1/repos/owner/repo/commits?path=src%2Fbackend%2Fcatalog%2Finformation_schema.sql
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22src%2Fbackend%2Fcatalog%2Finformation_schema.sql%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fsrc%2Fbackend%2Fcatalog%2Finformation_schema.sql
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fsrc%2Fbackend%2Fcatalog%2Finformation_schema.sql
|
||||
/owner/repo/graph?file=docs%2Fa%20b.xhtml
|
||||
/api/v1/repos/owner/repo/commits?path=docs%2Fa%20b.xhtml
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22docs%2Fa%20b.xhtml%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fdocs%2Fa%20b.xhtml
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fdocs%2Fa%20b.xhtml
|
||||
/owner/repo/graph?file=%5Bdraft%5D.md
|
||||
/api/v1/repos/owner/repo/commits?path=%5Bdraft%5D.md
|
||||
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22%5Bdraft%5D.md%22%5D%7D
|
||||
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2F%5Bdraft%5D.md
|
||||
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2F%5Bdraft%5D.md
|
||||
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=process.env.d.ts&style=unified&direction=down&wiki=false
|
||||
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=java.lang.String.md&style=unified&direction=down&wiki=false
|
||||
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=install-script.sh&style=unified&direction=down&wiki=false
|
||||
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=information_schema.sql&style=unified&direction=down&wiki=false
|
||||
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=document.write.js&style=unified&direction=down&wiki=false
|
||||
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=__proto__.json&style=unified&direction=down&wiki=false
|
||||
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=src%2Fjava.base%2Fshare%2Fclasses%2Fjava%2Flang%2FRuntime.java&style=unified&direction=down&wiki=false
|
||||
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=index.xhtml&style=unified&direction=down&wiki=false
|
||||
|
||||
# File names of attachments uploaded through the API, to a release, an issue and
|
||||
# a comment.
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=package.json
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=.gitignore
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=docker-compose.yml
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=.htaccess
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=docker-entrypoint
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=python3-setup.sh
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=Runtime.java
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=org.apache.commons.logging.LogFactory
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=%5Bdraft%5D.md
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=notes.txt~
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=install-script.sh
|
||||
/api/v1/repos/owner/repo/issues/1/assets?name=install-script.sh
|
||||
/api/v1/repos/owner/repo/issues/comments/1/assets?name=install-script.sh
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=process.env.d.ts
|
||||
/api/v1/repos/owner/repo/issues/1/assets?name=process.env.d.ts
|
||||
/api/v1/repos/owner/repo/issues/comments/1/assets?name=process.env.d.ts
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=java.lang.String.md
|
||||
/api/v1/repos/owner/repo/issues/1/assets?name=java.lang.String.md
|
||||
/api/v1/repos/owner/repo/issues/comments/1/assets?name=java.lang.String.md
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=information_schema.sql
|
||||
/api/v1/repos/owner/repo/issues/1/assets?name=information_schema.sql
|
||||
/api/v1/repos/owner/repo/issues/comments/1/assets?name=information_schema.sql
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=document.write.js
|
||||
/api/v1/repos/owner/repo/issues/1/assets?name=document.write.js
|
||||
/api/v1/repos/owner/repo/issues/comments/1/assets?name=document.write.js
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=__proto__.json
|
||||
/api/v1/repos/owner/repo/issues/1/assets?name=__proto__.json
|
||||
/api/v1/repos/owner/repo/issues/comments/1/assets?name=__proto__.json
|
||||
/api/v1/repos/owner/repo/releases/1/assets?name=index.xhtml
|
||||
/api/v1/repos/owner/repo/issues/1/assets?name=index.xhtml
|
||||
/api/v1/repos/owner/repo/issues/comments/1/assets?name=index.xhtml
|
||||
|
||||
# An artifact upload from actions/upload-artifact@v3, which sends Content-Range.
|
||||
/api/actions_pipeline/_apis/pipelines/workflows/1/artifacts/9b7c1f3a/upload?itemPath=artifact%2Freport.txt Content-Range: bytes 0-9/10
|
||||
|
||||
# Branch and tag names in the query string: deleting and restoring a branch,
|
||||
# starting a release from a tag, a branch protection rule, a directory listing's
|
||||
# last commits, and a container image push that reuses a layer.
|
||||
/owner/repo/branches/delete?name=docker-build&page=1
|
||||
/owner/repo/branches/restore?branch_id=1&name=docker-build&page=1
|
||||
/owner/repo/releases/new?tag=docker-build
|
||||
/owner/repo/settings/branches/edit?rule_name=docker-build
|
||||
/owner/repo/lastcommit/main/src?refSubUrl=branch%2Fdocker-build
|
||||
/v2/owner/app/blobs/uploads/?mount=sha256%3A0000&from=owner%2Fdocker-build
|
||||
/owner/repo/branches/delete?name=feat%2Fdocker-support&page=1
|
||||
/owner/repo/branches/restore?branch_id=1&name=feat%2Fdocker-support&page=1
|
||||
/owner/repo/releases/new?tag=feat%2Fdocker-support
|
||||
/owner/repo/settings/branches/edit?rule_name=feat%2Fdocker-support
|
||||
/owner/repo/lastcommit/main/src?refSubUrl=branch%2Ffeat%2Fdocker-support
|
||||
/v2/owner/app/blobs/uploads/?mount=sha256%3A0000&from=owner%2Ffeat%2Fdocker-support
|
||||
/owner/repo/branches/delete?name=renovate%2Fdocker-build-push-action-6.x&page=1
|
||||
/owner/repo/branches/restore?branch_id=1&name=renovate%2Fdocker-build-push-action-6.x&page=1
|
||||
/owner/repo/releases/new?tag=renovate%2Fdocker-build-push-action-6.x
|
||||
/owner/repo/settings/branches/edit?rule_name=renovate%2Fdocker-build-push-action-6.x
|
||||
/owner/repo/lastcommit/main/src?refSubUrl=branch%2Frenovate%2Fdocker-build-push-action-6.x
|
||||
/v2/owner/app/blobs/uploads/?mount=sha256%3A0000&from=owner%2Frenovate%2Fdocker-build-push-action-6.x
|
||||
/owner/repo/branches/delete?name=python3-migration&page=1
|
||||
/owner/repo/branches/restore?branch_id=1&name=python3-migration&page=1
|
||||
/owner/repo/releases/new?tag=python3-migration
|
||||
/owner/repo/settings/branches/edit?rule_name=python3-migration
|
||||
/owner/repo/lastcommit/main/src?refSubUrl=branch%2Fpython3-migration
|
||||
/v2/owner/app/blobs/uploads/?mount=sha256%3A0000&from=owner%2Fpython3-migration
|
||||
/owner/repo/branches/delete?name=ansible-roles&page=1
|
||||
/owner/repo/branches/restore?branch_id=1&name=ansible-roles&page=1
|
||||
/owner/repo/releases/new?tag=ansible-roles
|
||||
/owner/repo/settings/branches/edit?rule_name=ansible-roles
|
||||
/owner/repo/lastcommit/main/src?refSubUrl=branch%2Fansible-roles
|
||||
/v2/owner/app/blobs/uploads/?mount=sha256%3A0000&from=owner%2Fansible-roles
|
||||
/owner/repo/branches/delete?name=whoami&page=1
|
||||
/owner/repo/branches/restore?branch_id=1&name=whoami&page=1
|
||||
/owner/repo/releases/new?tag=whoami
|
||||
/owner/repo/settings/branches/edit?rule_name=whoami
|
||||
/owner/repo/lastcommit/main/src?refSubUrl=branch%2Fwhoami
|
||||
/v2/owner/app/blobs/uploads/?mount=sha256%3A0000&from=owner%2Fwhoami
|
||||
|
||||
# Pages a signed-in user opens.
|
||||
/
|
||||
/owner/repo
|
||||
/owner/repo/issues
|
||||
/owner/repo/pulls
|
||||
/owner/repo/releases
|
||||
/owner/repo/wiki
|
||||
/owner/repo/activity
|
||||
/owner/repo/branches
|
||||
/owner/repo/tags
|
||||
/owner/repo/settings
|
||||
/owner/repo/actions
|
||||
/owner/repo/compare/main...main
|
||||
/owner/repo/commits/branch/main
|
||||
/owner/repo/archive/main.zip
|
||||
/notifications
|
||||
/user/settings
|
||||
/user/settings/applications
|
||||
/explore/repos
|
||||
/-/admin
|
||||
@@ -0,0 +1,58 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic abuseipdb. It reads SWW_ABUSEIPDB_KEY,
|
||||
// SWW_ABUSEIPDB_MIN_SCORE and SWW_REPUTATION_ACTION.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// abuseIPDB adds the tag abuseipdb and notes AbuseIPDB, as noteHit does, with
|
||||
// the score, when its score of the client is a hit, and refuses the request
|
||||
// when SWW_REPUTATION_ACTION is deny, as dnsbl does for a zone. While
|
||||
// SWW_ABUSEIPDB_KEY is unset it does nothing. A client without a score is
|
||||
// checked in the background, by the request's address, if its history counts
|
||||
// an offence, and the request does not wait for the answer. The score is then
|
||||
// used for each address of the client. Its figures are the score and
|
||||
// SWW_ABUSEIPDB_MIN_SCORE.
|
||||
type abuseIPDB struct{}
|
||||
|
||||
func (abuseIPDB) name() string { return "abuseipdb" }
|
||||
|
||||
func (abuseIPDB) check(ctx context.Context, rq *request, _ time.Time) *verdict {
|
||||
if rq.h.config.AbuseIPDBKey == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
client := rq.h.clientGroup(rq.client)
|
||||
held, _ := rq.h.limiter.Client(client)
|
||||
offender := held.History.Offences != ratelimit.Offences{}
|
||||
|
||||
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
|
||||
if !hit {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.abuseIPDBHit = true
|
||||
rq.addTag(config.AbuseIPDBTag)
|
||||
rq.noteHit(bans.ReputationHit{Source: reputation.AbuseIPDBSource, Score: &score},
|
||||
"scored by AbuseIPDB at or over SWW_ABUSEIPDB_MIN_SCORE")
|
||||
|
||||
if rq.h.config.ReputationAction != deny {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionClientDenied,
|
||||
figures: map[string]any{
|
||||
figureScore: score, "min_score": rq.h.config.AbuseIPDBMinScore,
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -42,7 +42,7 @@ var (
|
||||
|
||||
// answerHealth answers a request for the health endpoint.
|
||||
func (rq *request) answerHealth() {
|
||||
rq.line.Action = requestlog.ActionAdmin
|
||||
rq.line.Action = requestlog.ActionOwnEndpoint
|
||||
// Set here rather than left to Go's server, which would set it only
|
||||
// after the log line has taken the response's headers.
|
||||
rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
||||
@@ -51,11 +51,11 @@ func (rq *request) answerHealth() {
|
||||
|
||||
// answerAdmin answers a request for one of discriminator's own endpoints. Each
|
||||
// needs a token, sent as Authorization: Bearer <token>: the metrics
|
||||
// SWWAF_METRICS_TOKEN, the others SWWAF_ADMIN_TOKEN. A request without it is
|
||||
// SWW_METRICS_TOKEN, the others SWW_ADMIN_TOKEN. A request without it is
|
||||
// refused with 401. An endpoint whose token is unset answers 404, as any other
|
||||
// path does.
|
||||
func (rq *request) answerAdmin() {
|
||||
rq.line.Action = requestlog.ActionAdmin
|
||||
rq.line.Action = requestlog.ActionOwnEndpoint
|
||||
|
||||
token, answer := rq.endpoint()
|
||||
|
||||
|
||||
@@ -17,8 +17,8 @@ import (
|
||||
)
|
||||
|
||||
const (
|
||||
adminToken = "SWWAF_ADMIN_TOKEN" //nolint:gosec // the setting's name
|
||||
// adminSecret is the SWWAF_ADMIN_TOKEN the tests set, and adminBearer
|
||||
adminToken = "SWW_ADMIN_TOKEN" //nolint:gosec // the setting's name
|
||||
// adminSecret is the SWW_ADMIN_TOKEN the tests set, and adminBearer
|
||||
// how a request carries it.
|
||||
adminSecret = "fedcba9876543210fedcba9876543210"
|
||||
adminBearer = "Bearer " + adminSecret
|
||||
@@ -121,7 +121,7 @@ func TestBanAddedListedAndLiftedThroughTheEndpoints(t *testing.T) {
|
||||
wantBans(t, s.admin(http.MethodPost, discriminator.BansPath, banOtherClient,
|
||||
http.StatusOK), want)
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
if line.BanExpires != requestlog.FormatTime(expires) {
|
||||
t.Errorf("the ban ends at %s, want %s", line.BanExpires, expires)
|
||||
}
|
||||
@@ -276,7 +276,7 @@ func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) {
|
||||
// limit of two a minute, and bans it.
|
||||
s.get(otherClient, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(otherClient, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(otherClient, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(otherClient, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
// Asked about by its address in IPv6 form too.
|
||||
for _, addr := range []string{otherClient, "::ffff:" + otherClient} {
|
||||
@@ -337,12 +337,12 @@ func TestAdminEndpointsNeedTheTokenInObserveMode(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// adminEndpoint is a request to an endpoint SWWAF_ADMIN_TOKEN opens.
|
||||
// adminEndpoint is a request to an endpoint SWW_ADMIN_TOKEN opens.
|
||||
type adminEndpoint struct {
|
||||
method, path, body string
|
||||
}
|
||||
|
||||
// adminEndpoints returns a request to each endpoint SWWAF_ADMIN_TOKEN
|
||||
// adminEndpoints returns a request to each endpoint SWW_ADMIN_TOKEN
|
||||
// opens: listing the bans, banning otherClient for an hour, lifting the
|
||||
// bans on otherClient, and asking about otherClient.
|
||||
func adminEndpoints() []adminEndpoint {
|
||||
@@ -356,7 +356,7 @@ func adminEndpoints() []adminEndpoint {
|
||||
|
||||
// admin sends a request with method for path, with body, to discriminator's own
|
||||
// address, with the admin token, and checks that it is answered with status, its
|
||||
// log line's action admin. It returns the answer.
|
||||
// log line's action own_endpoint. It returns the answer.
|
||||
func (s *sender) admin(method, path, body string, status int) answer {
|
||||
s.t.Helper()
|
||||
|
||||
@@ -380,7 +380,7 @@ func (s *sender) adminRequest(
|
||||
|
||||
line := s.out.endedLines(s.t, s.sent+1)[s.sent]
|
||||
s.sent++
|
||||
wantLine(s.t, line, status, requestlog.ActionAdmin)
|
||||
wantLine(s.t, line, status, requestlog.ActionOwnEndpoint)
|
||||
|
||||
return got
|
||||
}
|
||||
|
||||
@@ -15,8 +15,8 @@ import (
|
||||
)
|
||||
|
||||
const (
|
||||
alertWebhookURL = "SWWAF_ALERT_WEBHOOK_URL"
|
||||
alertMaxPerHour = "SWWAF_ALERT_MAX_PER_HOUR"
|
||||
alertWebhookURL = "SWW_ALERT_WEBHOOK_URL"
|
||||
alertMaxPerHour = "SWW_ALERT_MAX_PER_HOUR"
|
||||
// alertInstance is the instance every alert of these tests gives.
|
||||
alertInstance = "fsn1app1/blog"
|
||||
)
|
||||
@@ -31,14 +31,14 @@ func TestBanForABrokenLimitRaisesABanAlertWithItsNotes(t *testing.T) {
|
||||
start := clk.Now()
|
||||
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
netblock := netip.MustParsePrefix("203.0.113.0/24")
|
||||
ban := server.Ledger.Bans(netblock)[0]
|
||||
|
||||
// A request refused under the ban raises no other alert.
|
||||
clk.advance(time.Minute)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
wantAlerts(t, queue, banAlert(alerts.EventBan, start, client, bans.Ban{
|
||||
Netblock: netblock, Cause: bans.CauseLimit,
|
||||
@@ -62,22 +62,24 @@ func TestAttackBanRaisesABanAlertThenAPermanentBanAlert(t *testing.T) {
|
||||
|
||||
// The probe bans the client for seven days, and its next request makes
|
||||
// the ban permanent. The request after that changes nothing.
|
||||
s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(client, "/.env", http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
attackBan := server.Ledger.Bans(netblock)[0]
|
||||
|
||||
clk.advance(time.Minute)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionPermanentlyBanned)
|
||||
|
||||
permanentBan := server.Ledger.Bans(netblock)[0]
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionPermanentlyBanned)
|
||||
|
||||
// Another client's probe after its first ban has run out without a
|
||||
// request makes a permanent ban at once.
|
||||
s.request(otherClient, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(otherClient, "/.env", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
clk.advance(7 * 24 * time.Hour)
|
||||
s.request(otherClient, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(otherClient, "/.env", http.StatusForbidden,
|
||||
requestlog.ActionPermanentlyBanned)
|
||||
|
||||
otherBans := server.Ledger.Bans(other)
|
||||
|
||||
|
||||
@@ -21,11 +21,11 @@ import (
|
||||
// The anomaly thresholds: the prefix of a scope followed by the end of a
|
||||
// count.
|
||||
const (
|
||||
anomalyClient = "SWWAF_ANOMALY_CLIENT_"
|
||||
anomalyNet = "SWWAF_ANOMALY_NET_"
|
||||
anomalyASN = "SWWAF_ANOMALY_ASN_"
|
||||
anomalyTotal = "SWWAF_ANOMALY_TOTAL_"
|
||||
anomalyWatch = "SWWAF_WATCH_"
|
||||
anomalyClient = "SWW_ANOMALY_CLIENT_"
|
||||
anomalyNet = "SWW_ANOMALY_NET_"
|
||||
anomalyASN = "SWW_ANOMALY_ASN_"
|
||||
anomalyTotal = "SWW_ANOMALY_TOTAL_"
|
||||
anomalyWatch = "SWW_WATCH_"
|
||||
|
||||
requestsPerMinute = "REQUESTS_PER_MINUTE"
|
||||
requestsPerHour = "REQUESTS_PER_HOUR"
|
||||
@@ -35,9 +35,9 @@ const (
|
||||
|
||||
// The other anomaly settings.
|
||||
const (
|
||||
anomalyNetV4Prefix = "SWWAF_ANOMALY_NET_V4_PREFIX"
|
||||
anomalyNetV6Prefix = "SWWAF_ANOMALY_NET_V6_PREFIX"
|
||||
watchNets = "SWWAF_WATCH_NETS"
|
||||
anomalyNetV4Prefix = "SWW_ANOMALY_NET_V4_PREFIX"
|
||||
anomalyNetV6Prefix = "SWW_ANOMALY_NET_V6_PREFIX"
|
||||
watchNets = "SWW_WATCH_NETS"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -73,7 +73,7 @@ func TestEachScopeAndWindowOverItsThresholdAlertsOncePerCooldown(t *testing.T) {
|
||||
{anomalyASN, anomaly.ScopeASN, netip.Prefix{}, asnDE, map[string]any{"asn": asnDE}},
|
||||
{anomalyTotal, anomaly.ScopeTotal, netip.Prefix{}, "the whole service", nil},
|
||||
{
|
||||
anomalyWatch, anomaly.ScopeWatch, netip.MustParsePrefix(clientsNet),
|
||||
anomalyWatch, anomaly.ScopeWatchedNet, netip.MustParsePrefix(clientsNet),
|
||||
"the named netblock office, " + clientsNet, map[string]any{"name": "office"},
|
||||
},
|
||||
} {
|
||||
@@ -164,9 +164,9 @@ func TestEveryRequestIsCountedWhateverIsDoneWithIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
allowed = "192.0.2.7" // in SWWAF_ALLOW_NETS
|
||||
exempt = "192.0.2.10" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
|
||||
denied = "192.0.2.20" // in SWWAF_DENY_NETS
|
||||
allowed = "192.0.2.7" // in SWW_ALLOW_NETS
|
||||
exempt = "192.0.2.10" // in SWW_RATE_LIMIT_EXEMPT_NETS
|
||||
denied = "192.0.2.20" // in SWW_DENY_NETS
|
||||
)
|
||||
|
||||
s, _, _, queue := startAppWithAlerts(t, readAndAnswer, map[string]string{
|
||||
@@ -187,7 +187,7 @@ func TestEveryRequestIsCountedWhateverIsDoneWithIt(t *testing.T) {
|
||||
{allowed, "/", http.StatusOK, requestlog.ActionForward},
|
||||
{exempt, "/", http.StatusOK, requestlog.ActionForward},
|
||||
{client, "/static/app.js", http.StatusOK, requestlog.ActionForward},
|
||||
{denied, "/", http.StatusForbidden, requestlog.ActionDenied},
|
||||
{denied, "/", http.StatusForbidden, requestlog.ActionClientDenied},
|
||||
} {
|
||||
for range 3 {
|
||||
s.request(sent.from, sent.path, sent.status, sent.action)
|
||||
|
||||
@@ -26,7 +26,7 @@ const (
|
||||
// answerTimeLimit is the default of rproxy's SWRP_TIMEOUT_MS, the time
|
||||
// discriminator has to answer.
|
||||
answerTimeLimit = 2 * time.Second
|
||||
// defaultLookupWait is the default of SWWAF_LOOKUP_TIMEOUT.
|
||||
// defaultLookupWait is the default of SWW_LOOKUP_TIMEOUT.
|
||||
defaultLookupWait = 800 * time.Millisecond
|
||||
// The header limits SPEC.md gives.
|
||||
cookieLimit = 50
|
||||
@@ -84,13 +84,25 @@ func TestSixteenThousandCookiesAreRefusedWellWithinTheTimeLimit(t *testing.T) {
|
||||
wantDisposition(t, got, tooLarge, discriminator.DispositionDeny)
|
||||
|
||||
line := s.out.requestLine(t)
|
||||
wantLine(t, line, tooLarge, requestlog.ActionTooLarge)
|
||||
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
|
||||
|
||||
if line.DurationWAF != nil {
|
||||
t.Errorf("the Core Rule Set read the request, for %f ms", *line.DurationWAF)
|
||||
if line.DurationCoreRuleSetMS != nil {
|
||||
t.Errorf("the Core Rule Set read the request, for %f ms", *line.DurationCoreRuleSetMS)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeaderLimitRefusalNamesItsHeuristicCountsAndLimits(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := startDiscriminator(t, "", nil)
|
||||
req := s.question(siteA, client, "/")
|
||||
fillHeaders(req, headerFieldLimit+1)
|
||||
do(t, req)
|
||||
|
||||
wantHeuristic(t, s.out.requestLine(t), "header_limits",
|
||||
`{"fields": 101, "fields_limit": 100, "cookies": 0, "cookies_limit": 50}`)
|
||||
}
|
||||
|
||||
func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -145,7 +157,7 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
|
||||
|
||||
if tc.refused {
|
||||
wantDisposition(t, got, tooLarge, discriminator.DispositionDeny)
|
||||
wantLine(t, line, tooLarge, requestlog.ActionTooLarge)
|
||||
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
|
||||
|
||||
return
|
||||
}
|
||||
@@ -153,7 +165,7 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
|
||||
wantDisposition(t, got, http.StatusOK, discriminator.DispositionAllow)
|
||||
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
if line.DurationWAF == nil {
|
||||
if line.DurationCoreRuleSetMS == nil {
|
||||
t.Error("the Core Rule Set did not read a request within the limits")
|
||||
}
|
||||
})
|
||||
@@ -166,7 +178,7 @@ func TestHeaderLimitsHoldWhateverTheCoreRuleSetSettingsSay(t *testing.T) {
|
||||
for _, env := range []map[string]string{
|
||||
{wafMode: off},
|
||||
{wafMode: "detect"},
|
||||
{wafMode: block, "SWWAF_WAF_EXEMPT_PATHS": "/"},
|
||||
{wafMode: block, "SWW_WAF_EXEMPT_PATHS": "/"},
|
||||
} {
|
||||
t.Run(fmt.Sprint(env), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -207,9 +219,10 @@ func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
|
||||
wantDisposition(t, got, http.StatusOK, discriminator.DispositionAllow)
|
||||
|
||||
line := s.out.requestLine(t)
|
||||
if line.WouldAction != requestlog.ActionTooLarge || line.DurationWAF != nil {
|
||||
t.Errorf("would_action %q and duration_waf %v, want %q and none",
|
||||
line.WouldAction, line.DurationWAF, requestlog.ActionTooLarge)
|
||||
if line.WouldAction != requestlog.ActionRequestHeaderTooLarge ||
|
||||
line.DurationCoreRuleSetMS != nil {
|
||||
t.Errorf("would_action %q and duration_core_rule_set_ms %v, want %q and none",
|
||||
line.WouldAction, line.DurationCoreRuleSetMS, requestlog.ActionRequestHeaderTooLarge)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -228,7 +241,7 @@ func TestRequestsOverTheHeaderLimitsCountTowardTheRateLimits(t *testing.T) {
|
||||
wantDisposition(t, ask(), http.StatusForbidden, discriminator.DispositionDeny)
|
||||
|
||||
lines := s.out.requestLines(t, 2)
|
||||
wantLine(t, lines[1], http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
wantLine(t, lines[1], http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
}
|
||||
|
||||
func TestTheHeaderLimitsComeAfterTheTrapPathsAndTheRuleFiles(t *testing.T) {
|
||||
@@ -237,8 +250,8 @@ func TestTheHeaderLimitsComeAfterTheTrapPathsAndTheRuleFiles(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name, path, action string
|
||||
}{
|
||||
{"a trap path", "/wp-login.php", requestlog.ActionBanned},
|
||||
{"a block rule", "/blocked", requestlog.ActionRuleBlocked},
|
||||
{"a trap path", "/wp-login.php", requestlog.ActionTemporarilyBanned},
|
||||
{"a block rule", "/blocked", requestlog.ActionBlockedByBlockRule},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -268,10 +281,10 @@ func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) {
|
||||
wantDisposition(t, do(t, req), tooLarge, discriminator.DispositionDeny)
|
||||
|
||||
line := s.out.requestLine(t)
|
||||
wantLine(t, line, tooLarge, requestlog.ActionTooLarge)
|
||||
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
|
||||
|
||||
if line.LimitPercent != nil {
|
||||
t.Errorf("limit percentage %d, want none", *line.LimitPercent)
|
||||
if line.RateLimitPercent != nil {
|
||||
t.Errorf("limit percentage %d, want none", *line.RateLimitPercent)
|
||||
}
|
||||
|
||||
// The client is under no ban, and GeoJS places it in a country.
|
||||
@@ -292,7 +305,8 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(t *testing.T
|
||||
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
||||
|
||||
wantDisposition(t, do(t, req), http.StatusForbidden, discriminator.DispositionDeny)
|
||||
wantLine(t, s.out.requestLine(t), http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
wantLine(t, s.out.requestLine(t), http.StatusForbidden,
|
||||
requestlog.ActionClientIsRateLimited)
|
||||
}
|
||||
|
||||
func TestObserveModeWaitsForGeoJSForARequestOverTheHeaderLimits(t *testing.T) {
|
||||
@@ -323,7 +337,7 @@ func newDiscriminatorInBubble(
|
||||
) (*discriminator.Server, *output) {
|
||||
t.Helper()
|
||||
|
||||
settings := map[string]string{deniedCountries: "KP"}
|
||||
settings := map[string]string{tagDeny: "country=KP"}
|
||||
maps.Copy(settings, env)
|
||||
|
||||
server, out, _ := newDiscriminator(t, unansweredGeoJSURL, time.Now, settings)
|
||||
@@ -399,34 +413,38 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
|
||||
action string
|
||||
}{
|
||||
{
|
||||
name: "SWWAF_DENY_NETS", env: map[string]string{denyNets: client},
|
||||
prepare: leaveAsItIs, status: forbidden, action: requestlog.ActionDenied,
|
||||
name: "SWW_DENY_NETS", env: map[string]string{denyNets: client},
|
||||
prepare: leaveAsItIs, status: forbidden, action: requestlog.ActionClientDenied,
|
||||
},
|
||||
{
|
||||
name: "a ban",
|
||||
prepare: banClient, status: forbidden, action: requestlog.ActionBanned,
|
||||
prepare: banClient, status: forbidden, action: requestlog.ActionTemporarilyBanned,
|
||||
},
|
||||
{
|
||||
name: "the rate limits", env: map[string]string{rateLimitPerMinute: "1"},
|
||||
prepare: countClient, status: forbidden, action: requestlog.ActionRateLimited,
|
||||
prepare: countClient, status: forbidden,
|
||||
action: requestlog.ActionClientIsRateLimited,
|
||||
},
|
||||
{
|
||||
name: "the rate limits, with a biased threshold",
|
||||
env: map[string]string{rateLimitPerMinute: "1", unknownLimitPercent: "50"},
|
||||
prepare: countClient, status: forbidden, action: requestlog.ActionRateLimited,
|
||||
prepare: countClient, status: forbidden,
|
||||
action: requestlog.ActionClientIsRateLimited,
|
||||
},
|
||||
{
|
||||
name: "a blocklist", env: map[string]string{blocklistURLs: dropURL},
|
||||
prepare: listClient, status: forbidden, action: requestlog.ActionDenied,
|
||||
prepare: listClient, status: forbidden, action: requestlog.ActionClientDenied,
|
||||
},
|
||||
{
|
||||
name: "the CrowdSec decision list",
|
||||
env: map[string]string{crowdSecURL: lapi, crowdSecKey: bouncerKey},
|
||||
prepare: decideOnClient, status: forbidden, action: requestlog.ActionBanned,
|
||||
prepare: decideOnClient, status: forbidden,
|
||||
action: requestlog.ActionTemporarilyBanned,
|
||||
},
|
||||
{
|
||||
name: "the header limits", cookie: cookies(cookieLimit + 1),
|
||||
prepare: leaveAsItIs, status: tooLarge, action: requestlog.ActionTooLarge,
|
||||
prepare: leaveAsItIs, status: tooLarge,
|
||||
action: requestlog.ActionRequestHeaderTooLarge,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -530,7 +548,7 @@ func TestDiscriminatorAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
|
||||
|
||||
return discriminator.New(params)
|
||||
}, held, time.Now, map[string]string{
|
||||
deniedCountries: "KP", wafMode: block,
|
||||
tagDeny: "country=KP", wafMode: block,
|
||||
dnsblZones: dnsblZone, dnsblResolver: resolver.LocalAddr().String(),
|
||||
abuseIPDBKey: accountKey,
|
||||
blocklistURLs: held + "/drop.txt", asnLimitPercentURL: held + "/asn.txt",
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic active_ban, and the code that makes bans, which several
|
||||
// heuristics share. active_ban reads SWW_MODE; the bans are the ban ledger's.
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"context"
|
||||
"net/netip"
|
||||
"time"
|
||||
|
||||
@@ -12,17 +15,21 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// banResponse is a refusal answered with SWWAF_BAN_RESPONSE, and logged
|
||||
// banResponse is a refusal answered with SWW_BAN_RESPONSE, and logged
|
||||
// with action.
|
||||
func (rq *request) banResponse(action string) *refusal {
|
||||
return &refusal{status: rq.h.config.BanResponse, action: action}
|
||||
}
|
||||
|
||||
// banned reports whether a ban on a netblock the client is in covers the
|
||||
// request at now, and notes for the log line when that ban ends. A
|
||||
// request that makes the ban permanent, or in observe mode would have,
|
||||
// raises the alert for it.
|
||||
func (rq *request) banned(now time.Time) bool {
|
||||
// activeBan refuses a client a ban covers: a ban on a netblock the client is
|
||||
// in, in force at now. It notes for the log line when that ban ends. A request
|
||||
// that makes the ban permanent, or in observe mode would have, raises the alert
|
||||
// for it. Its figures are the ban's netblock and when it ends.
|
||||
type activeBan struct{}
|
||||
|
||||
func (activeBan) name() string { return "active_ban" }
|
||||
|
||||
func (activeBan) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
check := rq.h.ledger.Check
|
||||
if rq.h.config.Observe {
|
||||
check = rq.h.ledger.Find // the ban refuses nothing, and stays as it is
|
||||
@@ -38,90 +45,39 @@ func (rq *request) banned(now time.Time) bool {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return banned
|
||||
}
|
||||
|
||||
// limitBroken counts the request for the rate limits at now, notes the
|
||||
// client's counts for the log line, and reports whether the request takes
|
||||
// the client over a rate limit, as its limit percentage lowers it, which
|
||||
// breaks it.
|
||||
func (rq *request) limitBroken(now time.Time) bool {
|
||||
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now,
|
||||
rq.limitPercent.percent)
|
||||
rq.line.Counts = counts
|
||||
|
||||
if over {
|
||||
rq.banForLimit(now, hit, rq.h.config.BanResponse)
|
||||
if !banned {
|
||||
return nil
|
||||
}
|
||||
|
||||
return over
|
||||
}
|
||||
|
||||
// countBytes counts the request's bytes, as countedBytes gives them, for
|
||||
// the byte limits, once its report has come, and notes the client's byte totals
|
||||
// for the log line; its requests stay there as the rate limits counted them.
|
||||
// Only an allowed request has them counted, and only one the rate limits
|
||||
// counted; in observe mode, not one that enforce mode would have refused. Bytes
|
||||
// that take the client over a byte limit, as its limit percentage for the byte
|
||||
// limits lowers it, break it; the response was passed on whole.
|
||||
func (rq *request) countBytes() {
|
||||
if !rq.counted || rq.line.WouldAction != "" {
|
||||
return
|
||||
}
|
||||
|
||||
now := rq.h.now()
|
||||
|
||||
counts, hit, over := rq.h.limiter.CountBytes(rq.h.clientGroup(rq.client), now,
|
||||
rq.countedBytes(), rq.bytesPercent.percent)
|
||||
rq.line.Counts.MinuteBytes = counts.MinuteBytes
|
||||
rq.line.Counts.HourBytes = counts.HourBytes
|
||||
rq.line.Counts.DayBytes = counts.DayBytes
|
||||
|
||||
if over {
|
||||
rq.banForLimit(now, hit, rq.out.status)
|
||||
return &verdict{
|
||||
action: banAction(ban.Permanent()),
|
||||
figures: map[string]any{
|
||||
figureNetblock: ban.Netblock.String(), "ban_expires": rq.line.BanExpires,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// countRefusal counts the request for the error burst once it has been
|
||||
// answered, if discriminator refused it after a rule file match, a trap path or a
|
||||
// Core Rule Set match, and in observe mode if enforce mode would have: more than
|
||||
// SWWAF_ERROR_BURST_THRESHOLD such refusals of the client within a minute
|
||||
// break a limit. A client in SWWAF_ALLOW_NETS, which the checks skip, is
|
||||
// not counted, and nothing is while the threshold is off.
|
||||
func (rq *request) countRefusal() {
|
||||
cfg := rq.h.config
|
||||
if cfg.ErrorBurstThreshold == 0 {
|
||||
return
|
||||
// banAction is the action of a request refused under a ban or for making one:
|
||||
// permanent says the ban never runs out.
|
||||
func banAction(permanent bool) string {
|
||||
if permanent {
|
||||
return requestlog.ActionPermanentlyBanned
|
||||
}
|
||||
|
||||
if !rq.attack && !rq.ruleBlocked && !rq.wafBlocked {
|
||||
return
|
||||
return requestlog.ActionTemporarilyBanned
|
||||
}
|
||||
|
||||
// limitFigures are the figures of a heuristic that broke hit, a limit: its
|
||||
// window, the count and the limit.
|
||||
func limitFigures(hit ratelimit.Hit) map[string]any {
|
||||
return map[string]any{
|
||||
figureWindow: hit.Window, figureCount: hit.Count, figureLimit: hit.Limit,
|
||||
}
|
||||
|
||||
now := rq.h.now()
|
||||
|
||||
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
|
||||
cfg.ErrorBurstThreshold)
|
||||
if !over {
|
||||
return
|
||||
}
|
||||
|
||||
// What the client was sent, or in observe mode would have been.
|
||||
status := rq.out.status
|
||||
|
||||
switch rq.line.WouldAction {
|
||||
case requestlog.ActionRuleBlocked, requestlog.ActionWAFBlocked:
|
||||
status = http.StatusForbidden
|
||||
case requestlog.ActionBanned:
|
||||
status = cfg.BanResponse
|
||||
}
|
||||
|
||||
rq.banForLimit(now, hit, status)
|
||||
}
|
||||
|
||||
// countedBytes returns the request's bytes, once it has ended, as the
|
||||
// byte limits and the anomaly thresholds count them: the response's body
|
||||
// bytes, the request's, or both, as SWWAF_BYTES_COUNT says.
|
||||
// bytes, the request's, or both, as SWW_BYTES_COUNT says.
|
||||
func (rq *request) countedBytes() int64 {
|
||||
response, request := rq.out.bytes, rq.requestBytes
|
||||
|
||||
@@ -138,7 +94,7 @@ func (rq *request) countedBytes() int64 {
|
||||
// banForLimit bans the client's netblock at now for a broken limit, the
|
||||
// one hit names, notes the offence for the log line and counts the hit in
|
||||
// the metrics. status is what the client was sent, or is sent:
|
||||
// SWWAF_BAN_RESPONSE for a request over a rate limit, what the report says for
|
||||
// SWW_BAN_RESPONSE for a request over a rate limit, what the report says for
|
||||
// one whose bytes broke a byte limit, the refusal for one that broke the
|
||||
// error burst. The ban's notes give the client's limit percentage for a
|
||||
// rate limit or a byte limit; the error burst is not lowered. The ban sets
|
||||
@@ -150,7 +106,7 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
|
||||
case ratelimit.KindBytes:
|
||||
rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals
|
||||
case ratelimit.KindRefusals:
|
||||
rq.line.LimitHit = requestlog.LimitHitErrorBurst
|
||||
rq.line.LimitHit = requestlog.LimitHitMinuteRefusals
|
||||
default:
|
||||
rq.line.LimitHit = hit.Window
|
||||
}
|
||||
@@ -205,11 +161,11 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
|
||||
// attack, which notes name: the ban rule that matched, or the trap path
|
||||
// asked for. It fills in the rest of the notes. In observe mode it makes
|
||||
// no ban, and raises the alert for the ban it would have made, if that
|
||||
// alert would be sent.
|
||||
func (rq *request) banForAttack(now time.Time, notes bans.Notes) {
|
||||
// alert would be sent. It returns the action for that ban.
|
||||
func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
|
||||
netblock := rq.h.netblock(rq.client)
|
||||
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseAttack) {
|
||||
return
|
||||
return banAction(rq.h.ledger.WouldBePermanent(netblock, now, bans.CauseAttack))
|
||||
}
|
||||
|
||||
notes.ASN = rq.line.ASN
|
||||
@@ -225,7 +181,7 @@ func (rq *request) banForAttack(now time.Time, notes bans.Notes) {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return
|
||||
return banAction(ban.Permanent())
|
||||
}
|
||||
|
||||
ban, made := rq.h.ledger.BanForAttack(netblock, now, notes)
|
||||
@@ -234,16 +190,18 @@ func (rq *request) banForAttack(now time.Time, notes bans.Notes) {
|
||||
if made {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return banAction(ban.Permanent())
|
||||
}
|
||||
|
||||
// banForCrowdSec bans the client's netblock at now until decision,
|
||||
// CrowdSec's decision on the client, ends. In observe mode it makes no
|
||||
// ban, and raises the alert for the ban it would have made, if that alert
|
||||
// would be sent.
|
||||
func (rq *request) banForCrowdSec(now time.Time, decision reputation.Decision) {
|
||||
// would be sent. It returns the action for that ban.
|
||||
func (rq *request) banForCrowdSec(now time.Time, decision reputation.Decision) string {
|
||||
netblock := rq.h.netblock(rq.client)
|
||||
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseCrowdSec) {
|
||||
return
|
||||
return banAction(rq.h.ledger.WouldBePermanent(netblock, now, bans.CauseCrowdSec))
|
||||
}
|
||||
|
||||
notes := bans.Notes{
|
||||
@@ -262,7 +220,7 @@ func (rq *request) banForCrowdSec(now time.Time, decision reputation.Decision) {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return
|
||||
return banAction(ban.Permanent())
|
||||
}
|
||||
|
||||
ban, made := rq.h.ledger.BanForCrowdSec(netblock, now, decision.Expires,
|
||||
@@ -272,12 +230,14 @@ func (rq *request) banForCrowdSec(now time.Time, decision reputation.Decision) {
|
||||
if made {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return banAction(ban.Permanent())
|
||||
}
|
||||
|
||||
// wouldAlertBan reports whether the alert for a ban on netblock for cause
|
||||
// made at now would be sent. In observe mode the ban the request would
|
||||
// have made is worked out only then, at most once per
|
||||
// SWWAF_ALERT_COOLDOWN and never with no webhook set: its notes count the
|
||||
// SWW_ALERT_COOLDOWN and never with no webhook set: its notes count the
|
||||
// netblock's requests, which can mean going through every client.
|
||||
func (rq *request) wouldAlertBan(
|
||||
netblock netip.Prefix, now time.Time, cause string,
|
||||
@@ -343,7 +303,7 @@ func (rq *request) netblockRequests(netblock netip.Prefix) int64 {
|
||||
}
|
||||
|
||||
// netblock is the netblock a ban on client covers: its IPv4 address,
|
||||
// widened to SWWAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
|
||||
// widened to SWW_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
|
||||
// counts it in.
|
||||
func (h *handler) netblock(client netip.Addr) netip.Prefix {
|
||||
addr := client.Unmap()
|
||||
|
||||
@@ -26,7 +26,7 @@ const (
|
||||
userAgent = "ban-test/1.0"
|
||||
// permanent is the log line's ban_expires for a permanent ban.
|
||||
permanent = "permanent"
|
||||
// banResponseClose is the SWWAF_BAN_RESPONSE that closes the connection.
|
||||
// banResponseClose is the SWW_BAN_RESPONSE that closes the connection.
|
||||
banResponseClose = "close"
|
||||
)
|
||||
|
||||
@@ -40,7 +40,7 @@ func TestBrokenLimitBansTheClient(t *testing.T) {
|
||||
// client for an hour, the default.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
if line.LimitHit != minute || line.Offence != requestlog.OffenceLimit ||
|
||||
line.BanExpires != expires {
|
||||
t.Errorf("log line has limit_hit %q, offence %q and ban_expires %q, "+
|
||||
@@ -51,7 +51,7 @@ func TestBrokenLimitBansTheClient(t *testing.T) {
|
||||
// Every request while the ban lasts is refused.
|
||||
clk.advance(time.Hour - time.Second)
|
||||
|
||||
line = s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
line = s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
if line.BanExpires != expires || line.Offence != "" || line.LimitHit != "" {
|
||||
t.Errorf("log line has ban_expires %q, offence %q and limit_hit %q, "+
|
||||
"want %s and neither of the others", line.BanExpires, line.Offence,
|
||||
@@ -78,7 +78,9 @@ func TestBanLengthsFollowTheSettings(t *testing.T) {
|
||||
breakLimit := func() string {
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
return s.get(client, http.StatusForbidden, requestlog.ActionRateLimited).BanExpires
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
return line.BanExpires
|
||||
}
|
||||
wantExpires := func(got string, length time.Duration) {
|
||||
t.Helper()
|
||||
@@ -89,19 +91,19 @@ func TestBanLengthsFollowTheSettings(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A first ban lasts SWWAF_LIMIT_BAN_DURATION; one within
|
||||
// SWWAF_LIMIT_BAN_REPEAT_WINDOW after it ended, three times as long.
|
||||
// A first ban lasts SWW_LIMIT_BAN_DURATION; one within
|
||||
// SWW_LIMIT_BAN_REPEAT_WINDOW after it ended, three times as long.
|
||||
wantExpires(breakLimit(), 10*time.Minute)
|
||||
clk.advance(10*time.Minute + time.Hour)
|
||||
wantExpires(breakLimit(), 30*time.Minute)
|
||||
|
||||
// Later than that, SWWAF_LIMIT_BAN_DURATION again.
|
||||
// Later than that, SWW_LIMIT_BAN_DURATION again.
|
||||
clk.advance(30*time.Minute + time.Hour + time.Second)
|
||||
wantExpires(breakLimit(), 10*time.Minute)
|
||||
clk.advance(10 * time.Minute)
|
||||
wantExpires(breakLimit(), 30*time.Minute)
|
||||
|
||||
// 90 minutes would be longer than SWWAF_MAX_BAN_DURATION: the ban is
|
||||
// 90 minutes would be longer than SWW_MAX_BAN_DURATION: the ban is
|
||||
// permanent.
|
||||
clk.advance(30 * time.Minute)
|
||||
|
||||
@@ -111,7 +113,7 @@ func TestBanLengthsFollowTheSettings(t *testing.T) {
|
||||
}
|
||||
|
||||
clk.advance(365 * 24 * time.Hour)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionPermanentlyBanned)
|
||||
}
|
||||
|
||||
func TestBanIsNotCountedAndResetsTheCounters(t *testing.T) {
|
||||
@@ -123,10 +125,10 @@ func TestBanIsNotCountedAndResetsTheCounters(t *testing.T) {
|
||||
// client for an hour.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
for range 3 {
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
// Later the same day the client has its whole allowance again: the
|
||||
@@ -135,7 +137,7 @@ func TestBanIsNotCountedAndResetsTheCounters(t *testing.T) {
|
||||
clk.advance(time.Hour)
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
banned := server.Ledger.Bans(netip.MustParsePrefix(client + "/32"))
|
||||
if len(banned) != 2 || banned[0].Notes.Refused != 3 {
|
||||
@@ -148,8 +150,8 @@ func TestBanCoversTheClientsNetblock(t *testing.T) {
|
||||
|
||||
// In the IPv4 cases, client breaks the limit; these two are next to it.
|
||||
const (
|
||||
allowed = "203.0.113.60" // in SWWAF_ALLOW_NETS
|
||||
exempt = "203.0.113.50" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
|
||||
allowed = "203.0.113.60" // in SWW_ALLOW_NETS
|
||||
exempt = "203.0.113.50" // in SWW_RATE_LIMIT_EXEMPT_NETS
|
||||
)
|
||||
|
||||
for _, tc := range []struct {
|
||||
@@ -164,7 +166,7 @@ func TestBanCoversTheClientsNetblock(t *testing.T) {
|
||||
nil, []string{otherClient, exempt},
|
||||
},
|
||||
{
|
||||
"the IPv4 netblock SWWAF_BAN_SCOPE_V4_PREFIX sets",
|
||||
"the IPv4 netblock SWW_BAN_SCOPE_V4_PREFIX sets",
|
||||
map[string]string{banScopeV4Prefix: "24"}, client,
|
||||
[]string{otherClient, exempt}, []string{"203.0.112.9", allowed},
|
||||
},
|
||||
@@ -173,7 +175,7 @@ func TestBanCoversTheClientsNetblock(t *testing.T) {
|
||||
[]string{"2001:db8:5::ffff:1"}, []string{"2001:db8:5:1::1"},
|
||||
},
|
||||
{
|
||||
"the IPv6 netblock SWWAF_IPV6_GROUP_PREFIX sets",
|
||||
"the IPv6 netblock SWW_IPV6_GROUP_PREFIX sets",
|
||||
map[string]string{ipv6GroupPrefix: "48"}, "2001:db8:7::1",
|
||||
[]string{"2001:db8:7:ffff::1"}, []string{"2001:db8:8::1"},
|
||||
},
|
||||
@@ -190,10 +192,10 @@ func TestBanCoversTheClientsNetblock(t *testing.T) {
|
||||
s, _, _ := startWithClock(t, "", env)
|
||||
|
||||
s.get(tc.breaker, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(tc.breaker, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(tc.breaker, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
for _, sent := range tc.refused {
|
||||
s.get(sent, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(sent, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
for _, sent := range tc.let {
|
||||
@@ -211,14 +213,14 @@ func TestBannedClientIsRefusedBeforeItsCountryIsLookedUp(t *testing.T) {
|
||||
lookupTimeout: "1h",
|
||||
rateLimitPerMinute: "1",
|
||||
banScopeV4Prefix: "24",
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
|
||||
// fromDE's ban covers otherClient, which is refused unasked about.
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
line := s.get(otherClient, http.StatusForbidden, requestlog.ActionBanned)
|
||||
line := s.get(otherClient, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
if line.Country != "" {
|
||||
t.Errorf("log line has country %q, want none", line.Country)
|
||||
}
|
||||
@@ -228,13 +230,29 @@ func TestBannedClientIsRefusedBeforeItsCountryIsLookedUp(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefusalUnderABanNamesItsHeuristicNetblockAndEnd(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, _ := startWithClock(t, "", map[string]string{
|
||||
rateLimitPerMinute: "1", banScopeV4Prefix: "24",
|
||||
})
|
||||
expires := requestlog.FormatTime(clk.Now().Add(time.Hour))
|
||||
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
line := s.get(otherClient, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
wantHeuristic(t, line, "active_ban",
|
||||
`{"netblock": "`+clientsNet+`", "ban_expires": "`+expires+`"}`)
|
||||
}
|
||||
|
||||
func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const denied = "192.0.2.50" // in SWWAF_DENY_NETS
|
||||
const denied = "192.0.2.50" // in SWW_DENY_NETS
|
||||
|
||||
for _, tc := range []struct {
|
||||
setting string // "" leaves SWWAF_BAN_RESPONSE at its default
|
||||
setting string // "" leaves SWW_BAN_RESPONSE at its default
|
||||
status int // 0 is the connection closed without an answer
|
||||
}{
|
||||
{"", http.StatusForbidden},
|
||||
@@ -250,7 +268,7 @@ func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
|
||||
lookupTimeout: "1h",
|
||||
rateLimitPerMinute: "1",
|
||||
denyNets: denied,
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
}
|
||||
|
||||
if tc.setting != "" {
|
||||
@@ -259,11 +277,11 @@ func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
|
||||
|
||||
s, _, _ := startWithClock(t, geojsURL, env)
|
||||
|
||||
s.get(denied, tc.status, requestlog.ActionDenied)
|
||||
s.get(fromKP, tc.status, requestlog.ActionCountryDenied)
|
||||
s.get(denied, tc.status, requestlog.ActionClientDenied)
|
||||
s.get(fromKP, tc.status, requestlog.ActionClientDenied)
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(fromDE, tc.status, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, tc.status, requestlog.ActionBanned)
|
||||
s.get(fromDE, tc.status, requestlog.ActionClientIsRateLimited)
|
||||
s.get(fromDE, tc.status, requestlog.ActionTemporarilyBanned)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -275,15 +293,15 @@ func TestBanNotes(t *testing.T) {
|
||||
s, clk, server := startWithClock(t, geojsURL, map[string]string{
|
||||
lookupTimeout: "1h",
|
||||
rateLimitPerMinute: "1",
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
start := clk.Now()
|
||||
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
s.request(fromDE, "/repo/commits?page=2",
|
||||
http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
|
||||
http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
netblock := netip.MustParsePrefix(fromDE + "/32")
|
||||
want := bans.Ban{
|
||||
@@ -326,7 +344,7 @@ func TestBanNotes(t *testing.T) {
|
||||
// The next ban counts this one among the earlier.
|
||||
clk.advance(time.Hour)
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
got = ledger.Bans(netblock)
|
||||
if len(got) != 2 || got[1].Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
|
||||
@@ -344,12 +362,12 @@ func TestMaxBansDropsTheBanOfTheNetblockSeenLongestAgo(t *testing.T) {
|
||||
|
||||
// One ban is held, so otherClient's ban drops client's.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
s.get(otherClient, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(otherClient, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(otherClient, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(otherClient, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(otherClient, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
// clock is the time a test sets, by which sww counts requests and
|
||||
|
||||
@@ -17,74 +17,84 @@ type percentage struct {
|
||||
}
|
||||
|
||||
// biasedThresholdsSet reports whether a biased threshold can lower a
|
||||
// client's limits: one of its lists is not empty,
|
||||
// SWWAF_UNKNOWN_LIMIT_PERCENT is below 100, or SWWAF_ASN_LIMIT_PERCENT_URL
|
||||
// is set. The client's lookup is then needed before its request goes on.
|
||||
// client's limits by its lookup: SWW_ASN_BYTES_PERCENT or
|
||||
// SWW_COUNTRY_BYTES_PERCENT is not empty, SWW_UNKNOWN_LIMIT_PERCENT is
|
||||
// below 100, or SWW_ASN_LIMIT_PERCENT_URL is set. The client's lookup is
|
||||
// then needed before its request goes on.
|
||||
func biasedThresholdsSet(cfg *config.Config) bool {
|
||||
return len(cfg.ASNLimitPercent) > 0 || len(cfg.CountryLimitPercent) > 0 ||
|
||||
len(cfg.ASNBytesPercent) > 0 || len(cfg.CountryBytesPercent) > 0 ||
|
||||
return len(cfg.ASNBytesPercent) > 0 || len(cfg.CountryBytesPercent) > 0 ||
|
||||
cfg.UnknownLimitPercent < whole || cfg.ASNLimitPercentURL != ""
|
||||
}
|
||||
|
||||
// limitPercentages returns the client's limit percentages, for the rate
|
||||
// limits and for the byte limits, by its AS number and country as looked
|
||||
// up, each "" when unknown, and the blocklists, DNSBL zones and AbuseIPDB
|
||||
// that list it. Each is the lowest of those the settings give it, the
|
||||
// first of them in the order below when several are lowest: the
|
||||
// percentage SWWAF_ASN_LIMIT_PERCENT gives its AS number, the one the file
|
||||
// SWWAF_ASN_LIMIT_PERCENT_URL names gives it, the one
|
||||
// SWWAF_COUNTRY_LIMIT_PERCENT gives its country, for a client without a
|
||||
// country, unless its request skipped the wait for GeoJS,
|
||||
// SWWAF_UNKNOWN_LIMIT_PERCENT, for a client a blocklist lists, the
|
||||
// percentage of SWWAF_BLOCKLIST_ACTION while it is limit, and for a client
|
||||
// a DNSBL zone's verdict lists, or whose AbuseIPDB score is a hit, the
|
||||
// percentage of SWWAF_REPUTATION_ACTION while it is limit. For the byte
|
||||
// limits, SWWAF_ASN_BYTES_PERCENT and SWWAF_COUNTRY_BYTES_PERCENT take the
|
||||
// place of the first three for an AS number or a country they list.
|
||||
// limits and for the byte limits, by its tags, its AS number and country
|
||||
// as looked up, each "" when unknown, and the blocklists, DNSBL zones and
|
||||
// AbuseIPDB that list it. Each is the lowest of those the settings give
|
||||
// it, the first of them in the order below when several are lowest: the
|
||||
// percentage SWW_TAG_LIMIT_PERCENT gives its asn= tag, the one the file
|
||||
// SWW_ASN_LIMIT_PERCENT_URL names gives its AS number, the one
|
||||
// SWW_TAG_LIMIT_PERCENT gives its country= tag, the lowest it gives its
|
||||
// other tags, for a client without a country, unless its request skipped
|
||||
// the wait for GeoJS, SWW_UNKNOWN_LIMIT_PERCENT, for a client a blocklist
|
||||
// lists, the percentage of SWW_BLOCKLIST_ACTION while it is limit, and
|
||||
// for a client a DNSBL zone's verdict lists, or whose AbuseIPDB score is
|
||||
// a hit, the percentage of SWW_REPUTATION_ACTION while it is limit. For
|
||||
// the byte limits, SWW_ASN_BYTES_PERCENT and SWW_COUNTRY_BYTES_PERCENT
|
||||
// take the place of the first three for an AS number or a country they
|
||||
// list.
|
||||
func (rq *request) limitPercentages() (percentage, percentage) {
|
||||
cfg := rq.h.config
|
||||
asn, country := rq.line.ASN, rq.line.Country
|
||||
|
||||
unknown := percentage{percent: whole}
|
||||
if country == "" && !rq.skippedLookupWait {
|
||||
unknown = percentage{cfg.UnknownLimitPercent, "SWWAF_UNKNOWN_LIMIT_PERCENT"}
|
||||
unknown = percentage{cfg.UnknownLimitPercent, "SWW_UNKNOWN_LIMIT_PERCENT"}
|
||||
}
|
||||
|
||||
fetched := percentage{percent: whole}
|
||||
if percent, listed := rq.h.lists.ASNLimitPercent(asn); listed {
|
||||
fetched = percentage{percent, "SWWAF_ASN_LIMIT_PERCENT_URL"}
|
||||
fetched = percentage{percent, "SWW_ASN_LIMIT_PERCENT_URL"}
|
||||
}
|
||||
|
||||
blocklisted := percentage{percent: whole}
|
||||
if rq.blocklisted && cfg.BlocklistAction == "limit" {
|
||||
blocklisted = percentage{cfg.BlocklistLimitPercent, "SWWAF_BLOCKLIST_ACTION"}
|
||||
blocklisted = percentage{cfg.BlocklistLimitPercent, "SWW_BLOCKLIST_ACTION"}
|
||||
}
|
||||
|
||||
reputationListed := percentage{percent: whole}
|
||||
if (rq.dnsblListed || rq.abuseIPDBHit) && cfg.ReputationAction == "limit" {
|
||||
reputationListed = percentage{cfg.ReputationLimitPercent, "SWWAF_REPUTATION_ACTION"}
|
||||
reputationListed = percentage{cfg.ReputationLimitPercent, "SWW_REPUTATION_ACTION"}
|
||||
}
|
||||
|
||||
asnRequests := lowest(given(cfg.ASNLimitPercent, asn, "SWWAF_ASN_LIMIT_PERCENT"),
|
||||
fetched)
|
||||
countryRequests := given(cfg.CountryLimitPercent, country,
|
||||
"SWWAF_COUNTRY_LIMIT_PERCENT")
|
||||
const tagSetting = "SWW_TAG_LIMIT_PERCENT"
|
||||
|
||||
asnRequests := lowest(given(cfg.TagLimitPercent, asnTag(asn), tagSetting), fetched)
|
||||
countryRequests := given(cfg.TagLimitPercent, countryTag(country), tagSetting)
|
||||
|
||||
otherTags := percentage{percent: whole}
|
||||
|
||||
for _, tag := range rq.line.Tags {
|
||||
if tag != asnTag(asn) && tag != countryTag(country) {
|
||||
otherTags = lowest(otherTags, given(cfg.TagLimitPercent, tag, tagSetting))
|
||||
}
|
||||
}
|
||||
|
||||
asnBytes, countryBytes := asnRequests, countryRequests
|
||||
if _, listed := cfg.ASNBytesPercent[asn]; listed {
|
||||
asnBytes = given(cfg.ASNBytesPercent, asn, "SWWAF_ASN_BYTES_PERCENT")
|
||||
asnBytes = given(cfg.ASNBytesPercent, asn, "SWW_ASN_BYTES_PERCENT")
|
||||
}
|
||||
|
||||
if _, listed := cfg.CountryBytesPercent[country]; listed {
|
||||
countryBytes = given(cfg.CountryBytesPercent, country, "SWWAF_COUNTRY_BYTES_PERCENT")
|
||||
countryBytes = given(cfg.CountryBytesPercent, country, "SWW_COUNTRY_BYTES_PERCENT")
|
||||
}
|
||||
|
||||
return lowest(asnRequests, countryRequests, unknown, blocklisted, reputationListed),
|
||||
lowest(asnBytes, countryBytes, unknown, blocklisted, reputationListed)
|
||||
return lowest(asnRequests, countryRequests, otherTags, unknown, blocklisted,
|
||||
reputationListed),
|
||||
lowest(asnBytes, countryBytes, otherTags, unknown, blocklisted, reputationListed)
|
||||
}
|
||||
|
||||
// given returns the percentage percents, the setting named setting, gives
|
||||
// code, an AS number or a country, or whole when it does not list code.
|
||||
// code, an AS number, a country or a tag, or whole when it does not list code.
|
||||
func given(percents map[string]int64, code, setting string) percentage {
|
||||
percent, listed := percents[code]
|
||||
if !listed {
|
||||
|
||||
@@ -22,11 +22,10 @@ import (
|
||||
|
||||
// The biased thresholds.
|
||||
const (
|
||||
asnLimitPercent = "SWWAF_ASN_LIMIT_PERCENT"
|
||||
countryLimitPercent = "SWWAF_COUNTRY_LIMIT_PERCENT"
|
||||
asnBytesPercent = "SWWAF_ASN_BYTES_PERCENT"
|
||||
countryBytesPercent = "SWWAF_COUNTRY_BYTES_PERCENT"
|
||||
unknownLimitPercent = "SWWAF_UNKNOWN_LIMIT_PERCENT"
|
||||
tagLimitPercent = "SWW_TAG_LIMIT_PERCENT"
|
||||
asnBytesPercent = "SWW_ASN_BYTES_PERCENT"
|
||||
countryBytesPercent = "SWW_COUNTRY_BYTES_PERCENT"
|
||||
unknownLimitPercent = "SWW_UNKNOWN_LIMIT_PERCENT"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -35,6 +34,11 @@ const (
|
||||
asnDEHalf = asnDE + ":50"
|
||||
asnDEQuarter = asnDE + ":25"
|
||||
countryDEHalf = "de:50"
|
||||
// The same, as SWW_TAG_LIMIT_PERCENT names fromDE's tags.
|
||||
asnTagDEHalf = "asn=64496:50"
|
||||
asnTagDEQuarter = "asn=64496:25"
|
||||
countryTagDEHalf = "country=DE:50"
|
||||
countryTagDEQuarter = "country=DE:25"
|
||||
// noCountry is in an AS of its own, AS64500, and in no country.
|
||||
noCountry = "192.0.2.80"
|
||||
// fourAMinute is the rate limit these tests set: half of it is 2
|
||||
@@ -54,8 +58,8 @@ func TestEachBiasedThresholdLowersTheRateLimits(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
setting, value, from string
|
||||
}{
|
||||
{asnLimitPercent, asnDEHalf, fromDE},
|
||||
{countryLimitPercent, countryDEHalf, fromDE},
|
||||
{tagLimitPercent, asnTagDEHalf, fromDE},
|
||||
{tagLimitPercent, countryTagDEHalf, fromDE},
|
||||
{unknownLimitPercent, "50", unplaced},
|
||||
} {
|
||||
t.Run(tc.setting, func(t *testing.T) {
|
||||
@@ -72,17 +76,19 @@ func TestEachBiasedThresholdLowersTheRateLimits(t *testing.T) {
|
||||
}{
|
||||
{http.StatusOK, requestlog.ActionForward},
|
||||
{http.StatusOK, requestlog.ActionForward},
|
||||
{http.StatusForbidden, requestlog.ActionRateLimited},
|
||||
{http.StatusForbidden, requestlog.ActionClientIsRateLimited},
|
||||
} {
|
||||
line := s.get(tc.from, sent.status, sent.action)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
"50 from "+tc.setting)
|
||||
}
|
||||
|
||||
// fromKP, which no setting lists, has the whole limit.
|
||||
for range 3 {
|
||||
line := s.get(fromKP, http.StatusOK, requestlog.ActionForward)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
none)
|
||||
}
|
||||
})
|
||||
@@ -96,8 +102,8 @@ func TestEachBiasedThresholdLowersTheByteLimits(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
setting, value, from string
|
||||
}{
|
||||
{asnLimitPercent, asnDEHalf, fromDE},
|
||||
{countryLimitPercent, "DE:50", fromDE},
|
||||
{tagLimitPercent, asnTagDEHalf, fromDE},
|
||||
{tagLimitPercent, "Country=de:50", fromDE},
|
||||
{unknownLimitPercent, "50", unplaced},
|
||||
{asnBytesPercent, "as64496:50", fromDE},
|
||||
{countryBytesPercent, countryDEHalf, fromDE},
|
||||
@@ -115,7 +121,8 @@ func TestEachBiasedThresholdLowersTheByteLimits(t *testing.T) {
|
||||
t.Errorf("log line has limit_hit %q, want %s", line.LimitHit, minuteBytes)
|
||||
}
|
||||
|
||||
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting,
|
||||
"50 from "+tc.setting)
|
||||
|
||||
// fromKP, which no setting lists, has the whole limit.
|
||||
@@ -124,7 +131,8 @@ func TestEachBiasedThresholdLowersTheByteLimits(t *testing.T) {
|
||||
t.Errorf("log line for %s has limit_hit %q, want none", fromKP, line.LimitHit)
|
||||
}
|
||||
|
||||
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting, none)
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting, none)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -151,13 +159,13 @@ func TestBytesPercentSettingsTakeThePlaceOfTheOthersForByteLimits(t *testing.T)
|
||||
},
|
||||
{
|
||||
"raising the byte limits back",
|
||||
map[string]string{asnLimitPercent: asnDEHalf, asnBytesPercent: asnDE + ":100"},
|
||||
"50 from " + asnLimitPercent, none, "",
|
||||
map[string]string{tagLimitPercent: asnTagDEHalf, asnBytesPercent: asnDE + ":100"},
|
||||
"50 from " + tagLimitPercent, none, "",
|
||||
},
|
||||
{
|
||||
"raising the byte limits back, by country",
|
||||
map[string]string{countryLimitPercent: countryDEHalf, countryBytesPercent: "de:100"},
|
||||
"50 from " + countryLimitPercent, none, "",
|
||||
map[string]string{tagLimitPercent: countryTagDEHalf, countryBytesPercent: "de:100"},
|
||||
"50 from " + tagLimitPercent, none, "",
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -173,9 +181,11 @@ func TestBytesPercentSettingsTakeThePlaceOfTheOthersForByteLimits(t *testing.T)
|
||||
t.Errorf("log line has limit_hit %q, want %q", line.LimitHit, tc.limitHit)
|
||||
}
|
||||
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
tc.limitPercent)
|
||||
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting,
|
||||
tc.bytesPercent)
|
||||
})
|
||||
}
|
||||
@@ -184,19 +194,19 @@ func TestBytesPercentSettingsTakeThePlaceOfTheOthersForByteLimits(t *testing.T)
|
||||
func TestZeroPercentIsAZeroAllowance(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithLookups(t, map[string]string{asnLimitPercent: asnDE + ":0"})
|
||||
s, _, _ := startWithLookups(t, map[string]string{unknownLimitPercent: "0"})
|
||||
|
||||
// The first request breaks the limit, and bans the client; the log line
|
||||
// gives the 0.
|
||||
line := s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
if line.fields["limit_percent"] != float64(0) ||
|
||||
line.fields["limit_percent_setting"] != asnLimitPercent {
|
||||
t.Errorf("log line has limit_percent %v from %v, want 0 from %s",
|
||||
line.fields["limit_percent"], line.fields["limit_percent_setting"],
|
||||
asnLimitPercent)
|
||||
line := s.get(unplaced, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
if line.fields["rate_limit_percent"] != float64(0) ||
|
||||
line.fields["rate_limit_percent_setting"] != unknownLimitPercent {
|
||||
t.Errorf("log line has rate_limit_percent %v from %v, want 0 from %s",
|
||||
line.fields["rate_limit_percent"], line.fields["rate_limit_percent_setting"],
|
||||
unknownLimitPercent)
|
||||
}
|
||||
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(unplaced, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestLowestPercentageApplies(t *testing.T) {
|
||||
@@ -206,34 +216,34 @@ func TestLowestPercentageApplies(t *testing.T) {
|
||||
name string
|
||||
env map[string]string
|
||||
from string
|
||||
// want is the log line's limit_percent, as percentText gives it.
|
||||
// want is the log line's rate_limit_percent, as percentText gives it.
|
||||
want string
|
||||
}{
|
||||
{
|
||||
"the country's",
|
||||
map[string]string{asnLimitPercent: asnDEHalf, countryLimitPercent: "de:25"},
|
||||
fromDE, "25 from " + countryLimitPercent,
|
||||
map[string]string{tagLimitPercent: asnTagDEHalf + "," + countryTagDEQuarter},
|
||||
fromDE, "25 from " + tagLimitPercent,
|
||||
},
|
||||
{
|
||||
"the AS number's",
|
||||
map[string]string{asnLimitPercent: asnDEQuarter, countryLimitPercent: countryDEHalf},
|
||||
fromDE, "25 from " + asnLimitPercent,
|
||||
map[string]string{tagLimitPercent: asnTagDEQuarter + "," + countryTagDEHalf},
|
||||
fromDE, "25 from " + tagLimitPercent,
|
||||
},
|
||||
{
|
||||
"the AS number's, the first of two alike",
|
||||
map[string]string{asnLimitPercent: asnDEQuarter, countryLimitPercent: "de:25"},
|
||||
fromDE, "25 from " + asnLimitPercent,
|
||||
map[string]string{tagLimitPercent: asnTagDEQuarter + "," + countryTagDEQuarter},
|
||||
fromDE, "25 from " + tagLimitPercent,
|
||||
},
|
||||
{
|
||||
"that for a client without a country",
|
||||
map[string]string{asnLimitPercent: "AS64500:50", unknownLimitPercent: "25"},
|
||||
map[string]string{tagLimitPercent: "asn=64500:50", unknownLimitPercent: "25"},
|
||||
noCountry, "25 from " + unknownLimitPercent,
|
||||
},
|
||||
{
|
||||
// SWWAF_UNKNOWN_LIMIT_PERCENT is left at its default, 100.
|
||||
// SWW_UNKNOWN_LIMIT_PERCENT is left at its default, 100.
|
||||
"the AS number's, for a client without a country",
|
||||
map[string]string{asnLimitPercent: "AS64500:25"},
|
||||
noCountry, "25 from " + asnLimitPercent,
|
||||
map[string]string{tagLimitPercent: "asn=64500:25"},
|
||||
noCountry, "25 from " + tagLimitPercent,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -246,8 +256,9 @@ func TestLowestPercentageApplies(t *testing.T) {
|
||||
// A quarter of 4 requests a minute: the second breaks the limit.
|
||||
s.get(tc.from, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line := s.get(tc.from, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
line := s.get(tc.from, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
tc.want)
|
||||
})
|
||||
}
|
||||
@@ -265,7 +276,7 @@ func TestUnknownLimitPercentGivesEveryClientWithoutACountryItsPercentage(t *test
|
||||
for _, from := range []string{unplaced, "10.0.0.8"} {
|
||||
s.get(from, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(from, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(from, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(from, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
}
|
||||
|
||||
// One in a country has the whole limit.
|
||||
@@ -289,8 +300,9 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownLimitPercent(t *testing.T) {
|
||||
serveFromDE(t, server, http.MethodGet, http.NoBody)
|
||||
|
||||
line := out.requestLine(t)
|
||||
wantLine(t, line, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantLine(t, line, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
"0 from "+unknownLimitPercent)
|
||||
})
|
||||
}
|
||||
@@ -304,8 +316,12 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
|
||||
setting, value string
|
||||
waits bool
|
||||
}{
|
||||
{asnLimitPercent, asnDEHalf, true},
|
||||
{countryLimitPercent, countryDEHalf, true},
|
||||
{tagLimitPercent, asnTagDEHalf, true},
|
||||
{tagLimitPercent, countryTagDEHalf, true},
|
||||
{tagDeny, denyKP, true},
|
||||
// A tag the lookup does not give needs no answer.
|
||||
{tagLimitPercent, "vpn:50", false},
|
||||
{tagDeny, "vpn", false},
|
||||
{asnBytesPercent, asnDEHalf, true},
|
||||
{countryBytesPercent, countryDEHalf, true},
|
||||
{unknownLimitPercent, "99", true},
|
||||
@@ -320,7 +336,9 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
|
||||
// says, with a GeoJS that never answers.
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
server, out, _ := newDiscriminator(t, unansweredGeoJSURL, time.Now,
|
||||
map[string]string{lookupTimeout: timeout.String(), tc.setting: tc.value})
|
||||
map[string]string{
|
||||
lookupTimeout: timeout.String(), tagLists: "vpn=" + vpnList, tc.setting: tc.value,
|
||||
})
|
||||
began := time.Now()
|
||||
|
||||
serveFromDE(t, server, http.MethodPost, strings.NewReader("ab"))
|
||||
@@ -363,15 +381,15 @@ func TestBanForALoweredLimitGivesThePercentageInItsNotesAndItsAlert(t *testing.T
|
||||
{
|
||||
// A quarter of 12 requests a minute is 3: the fourth breaks it.
|
||||
"a rate limit",
|
||||
map[string]string{rateLimitPerMinute: "12", asnLimitPercent: asnDEQuarter},
|
||||
3, http.StatusForbidden, requestlog.ActionRateLimited,
|
||||
"requests per minute over the limit of 3", "25 from " + asnLimitPercent,
|
||||
map[string]string{rateLimitPerMinute: "12", tagLimitPercent: asnTagDEQuarter},
|
||||
3, http.StatusForbidden, requestlog.ActionClientIsRateLimited,
|
||||
"requests per minute over the limit of 3", "25 from " + tagLimitPercent,
|
||||
},
|
||||
{
|
||||
// The byte limits' percentage, not the rate limits'.
|
||||
"a byte limit",
|
||||
map[string]string{
|
||||
bytesLimitPerMinute: twoUploads, asnLimitPercent: asnDEQuarter,
|
||||
bytesLimitPerMinute: twoUploads, tagLimitPercent: asnTagDEQuarter,
|
||||
asnBytesPercent: asnDEHalf,
|
||||
},
|
||||
0, http.StatusOK, requestlog.ActionForward,
|
||||
@@ -487,7 +505,7 @@ func wantPercent(t *testing.T, what string, percent *int64, setting, want string
|
||||
}
|
||||
|
||||
// percentText gives a limit percentage and the setting that gave it as
|
||||
// text, such as "50 from SWWAF_ASN_LIMIT_PERCENT", or none when both are
|
||||
// text, such as "50 from SWW_TAG_LIMIT_PERCENT", or none when both are
|
||||
// left out.
|
||||
func percentText(percent *int64, setting string) string {
|
||||
switch {
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic blocklists. It reads SWW_BLOCKLIST_ACTION, and the blocklists
|
||||
// through the reputation lists.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// blocklists notes the blocklists that list the client, as noteListed does,
|
||||
// and refuses the request when SWW_BLOCKLIST_ACTION is deny. Being limit, it
|
||||
// lowers the client's limits instead (see limitPercentages), and being log, it
|
||||
// does nothing more. Its figure is the blocklists that list the client.
|
||||
type blocklists struct{}
|
||||
|
||||
func (blocklists) name() string { return "blocklists" }
|
||||
|
||||
func (blocklists) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
listedBy := rq.h.lists.ListedBy(rq.client)
|
||||
rq.blocklisted = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a blocklist")
|
||||
|
||||
if !rq.refusedByBlocklists(listedBy) {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionClientDenied,
|
||||
figures: map[string]any{"lists": listedBy},
|
||||
}
|
||||
}
|
||||
|
||||
// refusedByBlocklists reports whether the heuristic blocklists refuses a
|
||||
// client, given listedBy, the blocklists that list it: it does when one lists
|
||||
// it and SWW_BLOCKLIST_ACTION is deny.
|
||||
func (rq *request) refusedByBlocklists(listedBy []string) bool {
|
||||
return len(listedBy) > 0 && rq.h.config.BlocklistAction == deny
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic byte_limits. It reads SWW_BYTES_COUNT, and the byte limits
|
||||
// and biased thresholds through the limiter and limitPercentages.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// byteLimits counts the request's bytes, as countedBytes gives them, for the
|
||||
// byte limits, once its report has come, and notes the client's byte totals for
|
||||
// the log line; its requests stay there as the rate limits counted them. Only
|
||||
// an allowed request has them counted, and only one the rate limits counted; in
|
||||
// observe mode, not one that enforce mode would have refused. Bytes that take
|
||||
// the client over a byte limit, as its limit percentage for the byte limits
|
||||
// lowers it, break it, which bans the client; the response was passed on whole.
|
||||
// Its figures are the window, the count and the limit.
|
||||
type byteLimits struct{}
|
||||
|
||||
func (byteLimits) name() string { return "byte_limits" }
|
||||
|
||||
func (byteLimits) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
if !rq.counted || rq.line.WouldAction != "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
counts, hit, over := rq.h.limiter.CountBytes(rq.h.clientGroup(rq.client), now,
|
||||
rq.countedBytes(), rq.bytesPercent.percent)
|
||||
rq.line.Counts.MinuteBytes = counts.MinuteBytes
|
||||
rq.line.Counts.HourBytes = counts.HourBytes
|
||||
rq.line.Counts.DayBytes = counts.DayBytes
|
||||
|
||||
if !over {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.banForLimit(now, hit, rq.out.status)
|
||||
|
||||
return &verdict{figures: limitFigures(hit)}
|
||||
}
|
||||
@@ -19,13 +19,13 @@ import (
|
||||
|
||||
// The byte limit settings.
|
||||
const (
|
||||
bytesLimitPerMinute = "SWWAF_BYTES_LIMIT_PER_MINUTE"
|
||||
bytesLimitPerHour = "SWWAF_BYTES_LIMIT_PER_HOUR"
|
||||
bytesLimitPerDay = "SWWAF_BYTES_LIMIT_PER_DAY"
|
||||
bytesCount = "SWWAF_BYTES_COUNT"
|
||||
bytesLimitPerMinute = "SWW_BYTES_LIMIT_PER_MINUTE"
|
||||
bytesLimitPerHour = "SWW_BYTES_LIMIT_PER_HOUR"
|
||||
bytesLimitPerDay = "SWW_BYTES_LIMIT_PER_DAY"
|
||||
bytesCount = "SWW_BYTES_COUNT"
|
||||
)
|
||||
|
||||
// The values of SWWAF_BYTES_COUNT.
|
||||
// The values of SWW_BYTES_COUNT.
|
||||
const (
|
||||
countResponse = "response"
|
||||
countRequest = "request"
|
||||
@@ -40,7 +40,7 @@ const (
|
||||
// byteLimit is the byte limit these tests set, as a setting: a request
|
||||
// with a body and its answer, 100 bytes, go over it.
|
||||
byteLimit = "99"
|
||||
// minuteBytes is limit_hit for SWWAF_BYTES_LIMIT_PER_MINUTE.
|
||||
// minuteBytes is limit_hit for SWW_BYTES_LIMIT_PER_MINUTE.
|
||||
minuteBytes = "minute_bytes"
|
||||
)
|
||||
|
||||
@@ -90,7 +90,7 @@ func TestEachByteLimitBansOnceTheResponseHasEnded(t *testing.T) {
|
||||
line.BanExpires, tc.window, expires)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
wantMetric(t, s.scrape(), `sww_rate_limit_hits_total{`+
|
||||
`instance="`+alertInstance+`",kind="bytes",window="`+tc.window+`"}`, 1)
|
||||
@@ -98,6 +98,21 @@ func TestEachByteLimitBansOnceTheResponseHasEnded(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestByteLimitBanNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{bytesLimitPerMinute: byteLimit})
|
||||
|
||||
line, _ := s.download()
|
||||
wantAfterAnswerHeuristic(t, line, "", "null")
|
||||
|
||||
// Two answers of 70 bytes are over the limit of 99.
|
||||
line, _ = s.download()
|
||||
wantHeuristic(t, line, "", "null")
|
||||
wantAfterAnswerHeuristic(t, line, "byte_limits",
|
||||
`{"window": "minute", "count": 140, "limit": 99}`)
|
||||
}
|
||||
|
||||
func TestResponseOverAByteLimitByItselfIsPassedOnWhole(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -110,7 +125,7 @@ func TestResponseOverAByteLimitByItselfIsPassedOnWhole(t *testing.T) {
|
||||
len(got.body), got.err, line.LimitHit, answerBytes, minuteBytes)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestBytesOfAnAnswerThatBreaksOffAreCounted(t *testing.T) {
|
||||
@@ -132,7 +147,7 @@ func TestBytesOfAnAnswerThatBreaksOffAreCounted(t *testing.T) {
|
||||
answerBytes, minuteBytes, expires)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestWebSocketBytesAreCountedOnceItCloses(t *testing.T) {
|
||||
@@ -161,7 +176,7 @@ func TestWebSocketBytesAreCountedOnceItCloses(t *testing.T) {
|
||||
line.LimitHit, line.Counts.MinuteBytes, minuteBytes, tc.counted)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -203,7 +218,7 @@ func TestBytesCountSaysWhichBytesCount(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -212,8 +227,8 @@ func TestByteLimitsLeaveOutWhatTheRateLimitsLeaveOut(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
allowed = "192.0.2.7" // in SWWAF_ALLOW_NETS
|
||||
exempt = "192.0.2.10" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
|
||||
allowed = "192.0.2.7" // in SWW_ALLOW_NETS
|
||||
exempt = "192.0.2.10" // in SWW_RATE_LIMIT_EXEMPT_NETS
|
||||
)
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{
|
||||
@@ -371,7 +386,7 @@ func TestObserveModeLeavesOutTheBytesOfARequestEnforceModeRefuses(t *testing.T)
|
||||
// counted, and the byte limit is not broken. Its line gives the bytes
|
||||
// counted before it.
|
||||
line := s.upload()
|
||||
wantWouldAction(t, line, requestlog.ActionRateLimited)
|
||||
wantWouldAction(t, line, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
if line.LimitHit != minute || line.Counts.MinuteBytes != bodyBytes+answerBytes {
|
||||
t.Errorf("log line has limit_hit %q and minute_bytes %v, want minute and %d",
|
||||
|
||||
@@ -43,7 +43,7 @@ func scheme(r *http.Request, isQuestion bool) string {
|
||||
|
||||
// clientGroup is the client a request is counted toward: its IPv4
|
||||
// address, or its IPv6 group, the netblock its IPv6 address is in of the
|
||||
// length SWWAF_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser
|
||||
// length SWW_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser
|
||||
// usually holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
|
||||
func (h *handler) clientGroup(addr netip.Addr) netip.Prefix {
|
||||
addr = addr.Unmap()
|
||||
|
||||
@@ -1,26 +1,34 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic core_rule_set. It reads SWW_WAF_MODE, SWW_WAF_EXEMPT_PATHS
|
||||
// and SWW_WAF_ANOMALY_THRESHOLD.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
)
|
||||
|
||||
// checkCoreRuleSet inspects the request with the Core Rule Set, unless
|
||||
// SWWAF_WAF_MODE is off or SWWAF_WAF_EXEMPT_PATHS exempts its path, as
|
||||
// coreRuleSet inspects the request with the Core Rule Set, unless
|
||||
// SWW_WAF_MODE is off or SWW_WAF_EXEMPT_PATHS exempts its path, as
|
||||
// pathExempt decides, and notes the rules it matched and its score in the
|
||||
// log line, and the rules in the metrics. A score at or over
|
||||
// SWWAF_WAF_ANOMALY_THRESHOLD is a match: it raises the waf_block alert,
|
||||
// and in block mode refuses the request, which is an offence its client's
|
||||
// history counts, and so returns ActionWAFBlocked. It returns "" for a
|
||||
// request it does not refuse.
|
||||
func (rq *request) checkCoreRuleSet() string {
|
||||
// SWW_WAF_ANOMALY_THRESHOLD is a match: it adds the tag exploitsignature,
|
||||
// raises the core_rule_set_match alert, and in block mode refuses the request with 403,
|
||||
// which is an offence its client's history counts. Its figures are the score
|
||||
// and the threshold.
|
||||
type coreRuleSet struct{}
|
||||
|
||||
func (coreRuleSet) name() string { return "core_rule_set" }
|
||||
|
||||
func (coreRuleSet) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
cfg := rq.h.config
|
||||
if cfg.WAFMode == config.WAFModeOff || pathExempt(rq.in, cfg.WAFExemptPaths) {
|
||||
return ""
|
||||
return nil
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
@@ -30,36 +38,40 @@ func (rq *request) checkCoreRuleSet() string {
|
||||
result, read, _ := rq.h.coreRuleSet.Inspect(rq.in, rq.client, rq.in.Body)
|
||||
rq.requestBytes = int64(len(read))
|
||||
|
||||
rq.line.DurationWAF = new(requestlog.Milliseconds(time.Since(start)))
|
||||
rq.line.WAFRuleIDs = result.RuleIDs
|
||||
rq.line.WAFScore = &result.Score
|
||||
rq.line.DurationCoreRuleSetMS = new(requestlog.Milliseconds(time.Since(start)))
|
||||
rq.line.CoreRuleSetRuleIDs = result.RuleIDs
|
||||
rq.line.CoreRuleSetScore = &result.Score
|
||||
|
||||
for _, id := range result.RuleIDs {
|
||||
rq.h.metrics.WAFMatched(cfg.WAFMode, id)
|
||||
rq.h.metrics.CoreRuleSetMatched(cfg.WAFMode, id)
|
||||
}
|
||||
|
||||
threshold := cfg.WAFAnomalyThreshold
|
||||
if threshold == 0 || result.Score < threshold {
|
||||
return ""
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.alertWAFBlock(result)
|
||||
rq.addTag(config.ExploitSignatureTag)
|
||||
rq.alertCoreRuleSetMatch(result)
|
||||
|
||||
if cfg.WAFMode == config.WAFModeDetect {
|
||||
return ""
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.wafBlocked = true
|
||||
rq.blockedByCoreRuleSet = true
|
||||
|
||||
return requestlog.ActionWAFBlocked
|
||||
return &verdict{
|
||||
action: requestlog.ActionBlockedByCoreRuleSet,
|
||||
figures: map[string]any{figureScore: result.Score, "threshold": threshold},
|
||||
}
|
||||
}
|
||||
|
||||
// alertWAFBlock raises the waf_block alert for the request, which the Core
|
||||
// Rule Set scored at result, at or over SWWAF_WAF_ANOMALY_THRESHOLD. Its
|
||||
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
|
||||
// which the Core Rule Set scored at result, at or over SWW_WAF_ANOMALY_THRESHOLD. Its
|
||||
// detail gives the rule ids, the score, the method and the path with the
|
||||
// query, and, for a request that is not refused for it, the mode: detect,
|
||||
// or observe in observe mode.
|
||||
func (rq *request) alertWAFBlock(result waf.Result) {
|
||||
func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
|
||||
detail := map[string]any{
|
||||
"rule_ids": result.RuleIDs,
|
||||
"score": result.Score,
|
||||
@@ -75,14 +87,14 @@ func (rq *request) alertWAFBlock(result waf.Result) {
|
||||
}
|
||||
|
||||
rq.h.alerts.Raise(alerts.Alert{
|
||||
Event: alerts.EventWAFBlock,
|
||||
Event: alerts.EventCoreRuleSetMatch,
|
||||
Host: rq.site,
|
||||
Client: rq.client,
|
||||
Netblock: rq.h.clientGroup(rq.client),
|
||||
ASN: rq.line.ASN,
|
||||
ASName: rq.line.ASName,
|
||||
Country: rq.line.Country,
|
||||
Reason: "scored by the Core Rule Set at or over SWWAF_WAF_ANOMALY_THRESHOLD",
|
||||
Reason: "scored by the Core Rule Set at or over SWW_WAF_ANOMALY_THRESHOLD",
|
||||
Detail: detail,
|
||||
})
|
||||
}
|
||||
@@ -14,13 +14,13 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// The Core Rule Set's settings the tests set, besides SWWAF_WAF_MODE, and
|
||||
// The Core Rule Set's settings the tests set, besides SWW_WAF_MODE, and
|
||||
// its two modes that inspect requests.
|
||||
const (
|
||||
wafAnomalyThreshold = "SWWAF_WAF_ANOMALY_THRESHOLD"
|
||||
wafDisabledRules = "SWWAF_WAF_DISABLED_RULES"
|
||||
wafExemptPaths = "SWWAF_WAF_EXEMPT_PATHS"
|
||||
wafBodyLimit = "SWWAF_WAF_BODY_LIMIT"
|
||||
wafAnomalyThreshold = "SWW_WAF_ANOMALY_THRESHOLD"
|
||||
wafDisabledRules = "SWW_WAF_DISABLED_RULES"
|
||||
wafExemptPaths = "SWW_WAF_EXEMPT_PATHS"
|
||||
wafBodyLimit = "SWW_WAF_BODY_LIMIT"
|
||||
block = "block"
|
||||
detect = "detect"
|
||||
)
|
||||
@@ -28,7 +28,7 @@ const (
|
||||
// formData is the type of a form's body.
|
||||
const formData = "application/x-www-form-urlencoded"
|
||||
|
||||
// sizeLimit is SWWAF_WAF_BODY_LIMIT in the tests that read bodies, 1K as a
|
||||
// sizeLimit is SWW_WAF_BODY_LIMIT in the tests that read bodies, 1K as a
|
||||
// setting.
|
||||
const (
|
||||
sizeLimit = 1 << 10
|
||||
@@ -37,30 +37,72 @@ const (
|
||||
|
||||
// sqlInjection asks for / with an SQL injection in its query, which only
|
||||
// the Core Rule Set's rule 942100 matches, with a score of 5, the default
|
||||
// SWWAF_WAF_ANOMALY_THRESHOLD.
|
||||
// SWW_WAF_ANOMALY_THRESHOLD.
|
||||
const sqlInjection = "/?id=1'%20OR%20'1'='1"
|
||||
|
||||
// wantWAF checks the request log line's waf_rule_ids and waf_score, and
|
||||
// that it has duration_waf, or with no score, that it has none of the
|
||||
// three: the Core Rule Set did not inspect the request.
|
||||
func wantWAF(t *testing.T, line logLine, score *int, ruleIDs ...int) {
|
||||
// wantCoreRuleSet checks the request log line's core_rule_set_rule_ids and
|
||||
// core_rule_set_score, and that it has duration_core_rule_set_ms, or with no
|
||||
// score, that it has none of the three: the Core Rule Set did not inspect the
|
||||
// request.
|
||||
func wantCoreRuleSet(t *testing.T, line logLine, score *int, ruleIDs ...int) {
|
||||
t.Helper()
|
||||
|
||||
if !slices.Equal(line.WAFRuleIDs, ruleIDs) {
|
||||
t.Errorf("log line has waf_rule_ids %v, want %v", line.WAFRuleIDs, ruleIDs)
|
||||
if !slices.Equal(line.CoreRuleSetRuleIDs, ruleIDs) {
|
||||
t.Errorf("log line has core_rule_set_rule_ids %v, want %v",
|
||||
line.CoreRuleSetRuleIDs, ruleIDs)
|
||||
}
|
||||
|
||||
switch {
|
||||
case score == nil && (line.WAFScore != nil || line.DurationWAF != nil):
|
||||
t.Errorf("log line has waf_score %v and duration_waf %v, want neither",
|
||||
line.fields["waf_score"], line.fields["duration_waf"])
|
||||
case score != nil && (line.WAFScore == nil || *line.WAFScore != *score):
|
||||
t.Errorf("log line has waf_score %v, want %d", line.fields["waf_score"], *score)
|
||||
case score != nil && line.DurationWAF == nil:
|
||||
t.Error("log line has no duration_waf")
|
||||
case score == nil &&
|
||||
(line.CoreRuleSetScore != nil || line.DurationCoreRuleSetMS != nil):
|
||||
t.Errorf("log line has core_rule_set_score %v and "+
|
||||
"duration_core_rule_set_ms %v, want neither",
|
||||
line.fields["core_rule_set_score"], line.fields["duration_core_rule_set_ms"])
|
||||
case score != nil &&
|
||||
(line.CoreRuleSetScore == nil || *line.CoreRuleSetScore != *score):
|
||||
t.Errorf("log line has core_rule_set_score %v, want %d",
|
||||
line.fields["core_rule_set_score"], *score)
|
||||
case score != nil && line.DurationCoreRuleSetMS == nil:
|
||||
t.Error("log line has no duration_core_rule_set_ms")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoreRuleSetMatchAddsTheExploitSignatureTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
mode string
|
||||
status int
|
||||
action string
|
||||
}{
|
||||
{block, block, http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet},
|
||||
{detect, detect, http.StatusOK, requestlog.ActionForward},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{wafMode: tc.mode})
|
||||
|
||||
wantTags(t, s.request(client, sqlInjection, tc.status, tc.action),
|
||||
"exploitsignature")
|
||||
wantTags(t, s.get(client, http.StatusOK, requestlog.ActionForward))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetectedAttackIsRefusedByTagDenyLast(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
wafMode: detect, tagDeny: "exploitsignature",
|
||||
})
|
||||
|
||||
line := s.request(client, sqlInjection, http.StatusForbidden,
|
||||
requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, "tag_deny", `{"tag": "exploitsignature"}`)
|
||||
}
|
||||
|
||||
func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -97,19 +139,29 @@ func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(t *test
|
||||
mode, action string
|
||||
status int
|
||||
}{
|
||||
{block, requestlog.ActionWAFBlocked, http.StatusForbidden},
|
||||
{block, requestlog.ActionBlockedByCoreRuleSet, http.StatusForbidden},
|
||||
{detect, requestlog.ActionForward, http.StatusOK},
|
||||
} {
|
||||
s, _, _ := startWithClock(t, "", map[string]string{wafMode: tc.mode})
|
||||
|
||||
line := s.requestWithHeader(client, attack.path, attack.header,
|
||||
tc.status, tc.action)
|
||||
wantWAF(t, line, &attack.score, attack.ruleIDs...)
|
||||
wantCoreRuleSet(t, line, &attack.score, attack.ruleIDs...)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoreRuleSetRefusalNamesItsHeuristicScoreAndThreshold(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
|
||||
|
||||
line := s.request(client, sqlInjection, http.StatusForbidden,
|
||||
requestlog.ActionBlockedByCoreRuleSet)
|
||||
wantHeuristic(t, line, "core_rule_set", `{"score": 5, "threshold": 5}`)
|
||||
}
|
||||
|
||||
func TestOrdinaryRequestIsInspectedAndPassed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -117,7 +169,7 @@ func TestOrdinaryRequestIsInspectedAndPassed(t *testing.T) {
|
||||
|
||||
line := s.request(client, "/owner/repo/src/branch/main/README.md?display=source",
|
||||
http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, new(0))
|
||||
wantCoreRuleSet(t, line, new(0))
|
||||
}
|
||||
|
||||
func TestCoreRuleSetIsNotRunWhenOffOrForAnExemptClientPathOrRuleFileRefusal(
|
||||
@@ -125,7 +177,7 @@ func TestCoreRuleSetIsNotRunWhenOffOrForAnExemptClientPathOrRuleFileRefusal(
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
const allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
|
||||
const allowed = "192.0.2.60" // in SWW_ALLOW_NETS
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
wafMode: block,
|
||||
@@ -134,29 +186,29 @@ func TestCoreRuleSetIsNotRunWhenOffOrForAnExemptClientPathOrRuleFileRefusal(
|
||||
rulesDir: writeRules(t, testRules),
|
||||
})
|
||||
|
||||
// A client in SWWAF_ALLOW_NETS, and a path SWWAF_WAF_EXEMPT_PATHS
|
||||
// A client in SWW_ALLOW_NETS, and a path SWW_WAF_EXEMPT_PATHS
|
||||
// exempts, are not inspected.
|
||||
line := s.request(allowed, sqlInjection, http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, nil)
|
||||
wantCoreRuleSet(t, line, nil)
|
||||
line = s.request(client, "/api/v1/repos?id=1'%20OR%20'1'='1", http.StatusOK,
|
||||
requestlog.ActionForward)
|
||||
wantWAF(t, line, nil)
|
||||
wantCoreRuleSet(t, line, nil)
|
||||
|
||||
// The prefix is matched as rate limit exempt paths are: a path that
|
||||
// goes up and out of it is inspected.
|
||||
line = s.request(client, "/api/../?id=1'%20OR%20'1'='1", http.StatusForbidden,
|
||||
requestlog.ActionWAFBlocked)
|
||||
wantWAF(t, line, new(25), 930100, 930110, 942100)
|
||||
requestlog.ActionBlockedByCoreRuleSet)
|
||||
wantCoreRuleSet(t, line, new(25), 930100, 930110, 942100)
|
||||
|
||||
// A request a rule file refuses is not inspected.
|
||||
line = s.request(otherClient, "/blocked?id=1'%20OR%20'1'='1", http.StatusForbidden,
|
||||
requestlog.ActionRuleBlocked)
|
||||
wantWAF(t, line, nil)
|
||||
requestlog.ActionBlockedByBlockRule)
|
||||
wantCoreRuleSet(t, line, nil)
|
||||
|
||||
// With SWWAF_WAF_MODE off, no request is.
|
||||
// With SWW_WAF_MODE off, no request is.
|
||||
s, _, _ = startWithClock(t, "", map[string]string{wafMode: off})
|
||||
line = s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, nil)
|
||||
wantCoreRuleSet(t, line, nil)
|
||||
}
|
||||
|
||||
func TestAnomalyThreshold(t *testing.T) {
|
||||
@@ -170,13 +222,14 @@ func TestAnomalyThreshold(t *testing.T) {
|
||||
})
|
||||
|
||||
line := s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, new(5), 942100)
|
||||
wantCoreRuleSet(t, line, new(5), 942100)
|
||||
}
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
wafMode: block, wafAnomalyThreshold: "5",
|
||||
})
|
||||
s.request(client, sqlInjection, http.StatusForbidden, requestlog.ActionWAFBlocked)
|
||||
s.request(client, sqlInjection, http.StatusForbidden,
|
||||
requestlog.ActionBlockedByCoreRuleSet)
|
||||
}
|
||||
|
||||
func TestDisabledRulesSwitchOffWhatGiteaWouldBeRefused(t *testing.T) {
|
||||
@@ -219,7 +272,7 @@ func TestDisabledRulesSwitchOffWhatGiteaWouldBeRefused(t *testing.T) {
|
||||
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
|
||||
line, _ := s.requestWithBody(request.method, client, request.path,
|
||||
request.header, body, http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, new(0))
|
||||
wantCoreRuleSet(t, line, new(0))
|
||||
|
||||
// A list given replaces the default.
|
||||
s, _, _ = startWithClock(t, "", map[string]string{
|
||||
@@ -227,12 +280,12 @@ func TestDisabledRulesSwitchOffWhatGiteaWouldBeRefused(t *testing.T) {
|
||||
})
|
||||
score := 5 * len(request.ruleIDs)
|
||||
line, _ = s.requestWithBody(request.method, client, request.path,
|
||||
request.header, body, http.StatusForbidden, requestlog.ActionWAFBlocked)
|
||||
wantWAF(t, line, &score, request.ruleIDs...)
|
||||
request.header, body, http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet)
|
||||
wantCoreRuleSet(t, line, &score, request.ruleIDs...)
|
||||
|
||||
// And switches off the rules it lists.
|
||||
line = s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, new(0))
|
||||
wantCoreRuleSet(t, line, new(0))
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -248,14 +301,14 @@ func TestAttackInAFormBodyIsRefusedOnlyWhileBodiesAreRead(t *testing.T) {
|
||||
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
|
||||
line, _ := s.requestWithBody(http.MethodPost, client, "/", header, body,
|
||||
http.StatusOK, requestlog.ActionForward)
|
||||
wantWAF(t, line, new(0))
|
||||
wantCoreRuleSet(t, line, new(0))
|
||||
|
||||
s, _, _ = startWithClock(t, "", map[string]string{
|
||||
wafMode: block, wafBodyLimit: sizeLimitSetting,
|
||||
})
|
||||
line, _ = s.requestWithBody(http.MethodPost, client, "/", header, body,
|
||||
http.StatusForbidden, requestlog.ActionWAFBlocked)
|
||||
wantWAF(t, line, new(5), 942100)
|
||||
http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet)
|
||||
wantCoreRuleSet(t, line, new(5), 942100)
|
||||
}
|
||||
|
||||
func TestBodiesReachTheAppAsSentWhileBodiesAreRead(t *testing.T) {
|
||||
@@ -345,32 +398,33 @@ func TestCoreRuleSetRefusalIsAnOffenceAndCountsTowardTheErrorBurst(t *testing.T)
|
||||
})
|
||||
|
||||
for range 2 {
|
||||
s.request(client, sqlInjection, http.StatusForbidden, requestlog.ActionWAFBlocked)
|
||||
s.request(client, sqlInjection, http.StatusForbidden,
|
||||
requestlog.ActionBlockedByCoreRuleSet)
|
||||
}
|
||||
|
||||
// The third refusal in a minute breaks the error burst, and bans the
|
||||
// client.
|
||||
line := s.request(client, sqlInjection, http.StatusForbidden,
|
||||
requestlog.ActionWAFBlocked)
|
||||
if line.LimitHit != requestlog.LimitHitErrorBurst ||
|
||||
requestlog.ActionBlockedByCoreRuleSet)
|
||||
if line.LimitHit != requestlog.LimitHitMinuteRefusals ||
|
||||
line.Offence != requestlog.OffenceLimit {
|
||||
t.Errorf("log line has limit_hit %q and offence %q, want error_burst and limit",
|
||||
t.Errorf("log line has limit_hit %q and offence %q, want minute_refusals and limit",
|
||||
line.LimitHit, line.Offence)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
want := ratelimit.Offences{Limit: 1, WAFBlocked: 3}
|
||||
want := ratelimit.Offences{Limit: 1, BlockedByCoreRuleSet: 3}
|
||||
if offences := historyOf(t, server, client).Offences; offences != want {
|
||||
t.Errorf("history counts the offences %+v, want %+v", offences, want)
|
||||
}
|
||||
|
||||
metrics := s.scrape()
|
||||
wantMetric(t, metrics,
|
||||
`sww_waf_matches_total{instance="app",mode="block",rule_id="942100"}`, 3)
|
||||
`sww_core_rule_set_matches_total{instance="app",mode="block",rule_id="942100"}`, 3)
|
||||
wantMetric(t, metrics,
|
||||
`sww_offences_total{instance="app",kind="waf_blocked"}`, 3)
|
||||
wantMetric(t, metrics, `sww_requests_total{action="waf_blocked",`+
|
||||
`sww_offences_total{instance="app",kind="blocked_by_core_rule_set"}`, 3)
|
||||
wantMetric(t, metrics, `sww_requests_total{action="blocked_by_core_rule_set",`+
|
||||
`instance="app",status_class="4xx"}`, 3)
|
||||
}
|
||||
|
||||
@@ -394,9 +448,9 @@ func TestDetectModeMatchIsNoOffenceAndNotCountedTowardTheErrorBurst(t *testing.T
|
||||
|
||||
metrics := s.scrape()
|
||||
wantMetric(t, metrics,
|
||||
`sww_waf_matches_total{instance="app",mode="detect",rule_id="942100"}`, 3)
|
||||
`sww_core_rule_set_matches_total{instance="app",mode="detect",rule_id="942100"}`, 3)
|
||||
wantNoSeries(t, metrics,
|
||||
`sww_offences_total{instance="app",kind="waf_blocked"}`)
|
||||
`sww_offences_total{instance="app",kind="blocked_by_core_rule_set"}`)
|
||||
}
|
||||
|
||||
func TestObserveModeLogsWhatTheCoreRuleSetWouldDo(t *testing.T) {
|
||||
@@ -405,17 +459,17 @@ func TestObserveModeLogsWhatTheCoreRuleSetWouldDo(t *testing.T) {
|
||||
s, _, server := startWithClock(t, "", map[string]string{wafMode: block, mode: observe})
|
||||
|
||||
line := s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionWAFBlocked)
|
||||
wantWAF(t, line, new(5), 942100)
|
||||
wantWouldAction(t, line, requestlog.ActionBlockedByCoreRuleSet)
|
||||
wantCoreRuleSet(t, line, new(5), 942100)
|
||||
|
||||
// It is an offence as in enforce mode.
|
||||
want := ratelimit.Offences{WAFBlocked: 1}
|
||||
want := ratelimit.Offences{BlockedByCoreRuleSet: 1}
|
||||
if offences := historyOf(t, server, client).Offences; offences != want {
|
||||
t.Errorf("history counts the offences %+v, want %+v", offences, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoreRuleSetMatchRaisesTheWAFBlockAlert(t *testing.T) {
|
||||
func TestCoreRuleSetMatchRaisesTheCoreRuleSetMatchAlert(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
@@ -429,7 +483,7 @@ func TestCoreRuleSetMatchRaisesTheWAFBlockAlert(t *testing.T) {
|
||||
}{
|
||||
{
|
||||
"block", map[string]string{wafMode: block},
|
||||
http.StatusForbidden, requestlog.ActionWAFBlocked, "",
|
||||
http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet, "",
|
||||
},
|
||||
{
|
||||
"detect", map[string]string{wafMode: detect},
|
||||
@@ -464,11 +518,11 @@ func TestCoreRuleSetMatchRaisesTheWAFBlockAlert(t *testing.T) {
|
||||
wantAlerts(t, queue, alerts.Alert{
|
||||
Instance: alertInstance,
|
||||
Time: clk.Now(),
|
||||
Event: alerts.EventWAFBlock,
|
||||
Event: alerts.EventCoreRuleSetMatch,
|
||||
Host: appHost,
|
||||
Client: netip.MustParseAddr(client),
|
||||
Netblock: netip.MustParsePrefix(client + "/32"),
|
||||
Reason: "scored by the Core Rule Set at or over SWWAF_WAF_ANOMALY_THRESHOLD",
|
||||
Reason: "scored by the Core Rule Set at or over SWW_WAF_ANOMALY_THRESHOLD",
|
||||
Detail: detail,
|
||||
})
|
||||
|
||||
|
||||
@@ -1,21 +1,35 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic countries. It reads SWW_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// countryDenied reports whether the country lists refuse the request, by
|
||||
// the client's country as it was looked up. A client without a country,
|
||||
// or whose country cannot be found, is refused only by
|
||||
// SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||
func (rq *request) countryDenied() bool {
|
||||
denied := rq.h.config.DeniedCountries
|
||||
// countries refuses a request when SWW_EXCLUSIVELY_ALLOWED_COUNTRIES is set and
|
||||
// does not hold the client's country as it was looked up. A client without a
|
||||
// country, or whose country cannot be found, is refused. Its figures are the
|
||||
// country and the setting that refused it.
|
||||
type countries struct{}
|
||||
|
||||
func (countries) name() string { return "countries" }
|
||||
|
||||
func (countries) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
allowed := rq.h.config.ExclusivelyAllowedCountries
|
||||
country := rq.line.Country
|
||||
|
||||
if slices.Contains(denied, country) {
|
||||
return true
|
||||
if len(allowed) == 0 || slices.Contains(allowed, country) {
|
||||
return nil
|
||||
}
|
||||
|
||||
return len(allowed) > 0 && !slices.Contains(allowed, country)
|
||||
return &verdict{
|
||||
action: requestlog.ActionCountryDenied,
|
||||
figures: map[string]any{
|
||||
"country": country, "setting": "SWW_EXCLUSIVELY_ALLOWED_COUNTRIES",
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -33,13 +33,13 @@ func TestCountryLists(t *testing.T) {
|
||||
env map[string]string
|
||||
refused []string
|
||||
}{
|
||||
{"denied", map[string]string{deniedCountries: "kp"}, []string{fromKP}},
|
||||
{
|
||||
"exclusively allowed", map[string]string{allowedCountries: "DE"},
|
||||
[]string{fromKP, unplaced},
|
||||
},
|
||||
{
|
||||
"both", map[string]string{deniedCountries: "kp", allowedCountries: "de,fr"},
|
||||
"with a country in SWW_TAG_DENY",
|
||||
map[string]string{tagDeny: denyKP, allowedCountries: "de,fr"},
|
||||
[]string{fromKP, unplaced},
|
||||
},
|
||||
} {
|
||||
@@ -90,6 +90,23 @@ func TestCountryLists(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCountryRefusalNamesItsHeuristicCountryAndSetting(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
geojsURL, _ := startGeoJS(t)
|
||||
s, _, _ := startWithClock(t, geojsURL, map[string]string{
|
||||
lookupTimeout: "1h", allowedCountries: "de",
|
||||
})
|
||||
|
||||
line := s.get(fromKP, http.StatusForbidden, requestlog.ActionCountryDenied)
|
||||
wantHeuristic(t, line, "countries",
|
||||
`{"country": "KP", "setting": "SWW_EXCLUSIVELY_ALLOWED_COUNTRIES"}`)
|
||||
|
||||
line = s.get(unplaced, http.StatusForbidden, requestlog.ActionCountryDenied)
|
||||
wantHeuristic(t, line, "countries",
|
||||
`{"country": "", "setting": "SWW_EXCLUSIVELY_ALLOWED_COUNTRIES"}`)
|
||||
}
|
||||
|
||||
func TestCountryRefusalKeepsARequestFromTheApp(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -100,9 +117,9 @@ func TestCountryRefusalKeepsARequestFromTheApp(t *testing.T) {
|
||||
})
|
||||
geojsURL, _ := startGeoJS(t)
|
||||
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
lookupTimeout: "1h",
|
||||
deniedCountries: "kp",
|
||||
trustedProxies: trustLocalhost,
|
||||
lookupTimeout: "1h",
|
||||
allowedCountries: "de",
|
||||
})
|
||||
|
||||
req := newRequest(t, http.MethodPost, addr, "/", strings.NewReader("a body"))
|
||||
@@ -185,7 +202,7 @@ func TestPrivateAddressIsNeverLookedUp(t *testing.T) {
|
||||
env map[string]string
|
||||
}{
|
||||
{"no setting needs the lookup", nil},
|
||||
{"a country list is set", map[string]string{deniedCountries: "kp"}},
|
||||
{"a tag setting needs it", map[string]string{tagDeny: denyKP}},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -245,11 +262,11 @@ func TestExclusiveListRefusesAPrivateAddressUnlessAllowed(t *testing.T) {
|
||||
action string
|
||||
}{
|
||||
{
|
||||
"not in SWWAF_ALLOW_NETS", "",
|
||||
"not in SWW_ALLOW_NETS", "",
|
||||
http.StatusForbidden, requestlog.ActionCountryDenied,
|
||||
},
|
||||
{
|
||||
"in SWWAF_ALLOW_NETS", "10.0.0.7,fd00::/8",
|
||||
"in SWW_ALLOW_NETS", "10.0.0.7,fd00::/8",
|
||||
http.StatusOK, requestlog.ActionForward,
|
||||
},
|
||||
} {
|
||||
@@ -279,7 +296,7 @@ func TestExclusiveListRefusesAPrivateAddressUnlessAllowed(t *testing.T) {
|
||||
// startGeoJS starts a stand-in for GeoJS, which places fromDE and fromKP,
|
||||
// each in an AS of its own, and no other address. It returns its URL, and
|
||||
// what returns the addresses it has been asked about. A test that needs
|
||||
// the stand-in to be asked or to answer sets SWWAF_LOOKUP_TIMEOUT to an
|
||||
// the stand-in to be asked or to answer sets SWW_LOOKUP_TIMEOUT to an
|
||||
// hour, whether or not a request waits for the answer: on the default, a
|
||||
// hold-up of the test process can abandon the request to the stand-in, and
|
||||
// leave the client unknown.
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic crowdsec. It reads SWW_CROWDSEC_LAPI_URL, and the decisions
|
||||
// through the reputation lists.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// crowdSec refuses a request when a decision of the CrowdSec decision list on
|
||||
// the client is in force at now. It then adds the tag crowdsec, notes the
|
||||
// list, as noteHit does, and bans the client until that decision ends. Its
|
||||
// figures are the decision's scenario and when it ends.
|
||||
type crowdSec struct{}
|
||||
|
||||
func (crowdSec) name() string { return "crowdsec" }
|
||||
|
||||
func (crowdSec) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
decision, listed := rq.crowdSecDecisionInForce(now)
|
||||
if !listed {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.addTag(config.CrowdSecTag)
|
||||
rq.noteHit(bans.ReputationHit{Source: rq.h.config.CrowdSecDecisionsURL},
|
||||
"listed by the CrowdSec decision list")
|
||||
|
||||
return &verdict{
|
||||
action: rq.banForCrowdSec(now, decision),
|
||||
figures: map[string]any{
|
||||
"scenario": decision.Scenario,
|
||||
"expires": requestlog.FormatTime(decision.Expires),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// crowdSecDecisionInForce returns the decision of the CrowdSec decision list on
|
||||
// the client that is in force at now, and reports whether there is one, which
|
||||
// the heuristic crowdsec refuses the request for.
|
||||
func (rq *request) crowdSecDecisionInForce(
|
||||
now time.Time,
|
||||
) (reputation.Decision, bool) {
|
||||
return rq.h.lists.CrowdSecDecision(rq.client, now)
|
||||
}
|
||||
@@ -18,8 +18,8 @@ import (
|
||||
// test puts in the copy of its decision list, at decisionsURL, that it
|
||||
// needs, as reputation.json would at start.
|
||||
const (
|
||||
crowdSecURL = "SWWAF_CROWDSEC_LAPI_URL"
|
||||
crowdSecKey = "SWWAF_CROWDSEC_LAPI_KEY"
|
||||
crowdSecURL = "SWW_CROWDSEC_LAPI_URL"
|
||||
crowdSecKey = "SWW_CROWDSEC_LAPI_KEY"
|
||||
lapi = "http://crowdsec.example:8080"
|
||||
decisionsURL = lapi + "/v1/decisions"
|
||||
bouncerKey = "crowdsec-key-0123456789abcdef"
|
||||
@@ -39,7 +39,7 @@ func TestClientTheCrowdSecDecisionListListsIsBannedUntilTheDecisionEnds(t *testi
|
||||
expires := requestlog.FormatTime(fetched.Add(4 * time.Hour))
|
||||
|
||||
// Its first request is refused, and bans it until the decision ends.
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
wantReputation(t, line, decisionsURL)
|
||||
|
||||
if line.BanExpires != expires {
|
||||
@@ -72,7 +72,7 @@ func TestClientTheCrowdSecDecisionListListsIsBannedUntilTheDecisionEnds(t *testi
|
||||
// ban, and once it has ended the client is let through.
|
||||
clk.advance(4*time.Hour - time.Second)
|
||||
|
||||
line = s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
line = s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
wantReputation(t, line)
|
||||
|
||||
if line.BanExpires != expires {
|
||||
@@ -95,6 +95,22 @@ func TestClientTheCrowdSecDecisionListListsIsBannedUntilTheDecisionEnds(t *testi
|
||||
float64(fetched.Unix()))
|
||||
}
|
||||
|
||||
func TestCrowdSecBanNamesItsHeuristicScenarioAndEnd(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server, _ := startWithAlerts(t, map[string]string{
|
||||
crowdSecURL: lapi, crowdSecKey: bouncerKey,
|
||||
})
|
||||
loadDecisions(t, server, clk.Now(), `[{"duration": "4h0m0s", `+
|
||||
`"scenario": "crowdsecurity/ssh-bf", "scope": "Ip", "type": "ban", `+
|
||||
`"value": "`+client+`"}]`)
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
wantHeuristic(t, line, "crowdsec", `{"scenario": "crowdsecurity/ssh-bf", `+
|
||||
`"expires": "`+requestlog.FormatTime(clk.Now().Add(4*time.Hour))+`"}`)
|
||||
wantTags(t, line, "crowdsec")
|
||||
}
|
||||
|
||||
func TestEndedCrowdSecDecisionNoLongerBansThoughTheCopyStillHoldsIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -113,9 +129,9 @@ func TestEndedCrowdSecDecisionNoLongerBansThoughTheCopyStillHoldsIt(t *testing.T
|
||||
// Just before its end, the decision bans a client in the netblock, and
|
||||
// one on an IPv6 address bans the address's group, the /64.
|
||||
clk.advance(time.Minute - time.Nanosecond)
|
||||
s.get("198.51.100.7", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get("2001:db8::9", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get("2001:db8::5", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get("198.51.100.7", http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
s.get("2001:db8::9", http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
s.get("2001:db8::5", http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
// Once it has ended, it bans no other client, and the bans it made end
|
||||
// with it.
|
||||
@@ -148,7 +164,7 @@ func TestObserveModeForwardsAClientTheCrowdSecDecisionListListsAndAlertsTheBan(
|
||||
`"value": "`+client+`"}]`)
|
||||
|
||||
line := s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionBanned)
|
||||
wantWouldAction(t, line, requestlog.ActionTemporarilyBanned)
|
||||
wantReputation(t, line, decisionsURL)
|
||||
|
||||
if held := server.Ledger.Snapshot(); len(held) != 0 {
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic deny_nets. It reads SWW_DENY_NETS.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// denyNets refuses a client in SWW_DENY_NETS. Its figure is the netblock of
|
||||
// the setting the client is in.
|
||||
type denyNets struct{}
|
||||
|
||||
func (denyNets) name() string { return "deny_nets" }
|
||||
|
||||
func (denyNets) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
for _, netblock := range rq.h.config.DenyNets {
|
||||
if netblock.Contains(rq.client) {
|
||||
return &verdict{
|
||||
action: requestlog.ActionClientDenied,
|
||||
figures: map[string]any{figureNetblock: netblock.String()},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package discriminator
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -39,7 +40,7 @@ const (
|
||||
// headers and body.
|
||||
DispositionDeny = "deny"
|
||||
// DispositionClose tells rproxy to close the visitor's connection
|
||||
// without an answer, which SWWAF_BAN_RESPONSE asks for when it is close.
|
||||
// without an answer, which SWW_BAN_RESPONSE asks for when it is close.
|
||||
// discriminator closing the question's connection instead would read as
|
||||
// discriminator being down, and rproxy would then let the request through.
|
||||
DispositionClose = "close"
|
||||
@@ -52,13 +53,6 @@ const ReportPath = "/_sww/report"
|
||||
// QuestionBodyMaxBytes is the most of a visitor's body a question carries.
|
||||
const QuestionBodyMaxBytes = 64 << 10
|
||||
|
||||
// The header limits: the most cookies and header fields a request may have
|
||||
// when the checks reach the Core Rule Set. Real clients never reach them.
|
||||
const (
|
||||
maxCookies = 50
|
||||
maxHeaderFields = 100
|
||||
)
|
||||
|
||||
const (
|
||||
// maxAwaitingReport is the most answered requests held until their
|
||||
// reports come. Past it, the request answered longest ago is dropped.
|
||||
@@ -95,12 +89,12 @@ var (
|
||||
//
|
||||
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
|
||||
type Report struct {
|
||||
RequestID string `json:"request_id"`
|
||||
Status int `json:"status"`
|
||||
RequestBytes int64 `json:"request_bytes"`
|
||||
ResponseBytes int64 `json:"response_bytes"`
|
||||
DurationTotal float64 `json:"duration_total"`
|
||||
DurationUpstreamTotal float64 `json:"duration_upstream_total"`
|
||||
RequestID string `json:"request_id"`
|
||||
Status int `json:"status"`
|
||||
RequestBytes int64 `json:"request_bytes"`
|
||||
ResponseBytes int64 `json:"response_bytes"`
|
||||
DurationTotalMS float64 `json:"duration_total_ms"`
|
||||
DurationUpstreamTotalMS float64 `json:"duration_upstream_total_ms"`
|
||||
}
|
||||
|
||||
// New returns the server `sww discriminator` runs. It answers
|
||||
@@ -129,6 +123,7 @@ func New(params Params) *Server {
|
||||
Lists: h.lists,
|
||||
DNSBL: h.dnsbl,
|
||||
AbuseIPDB: h.abuseIPDB,
|
||||
RouteWatch: h.routeWatch,
|
||||
Metrics: h.metrics,
|
||||
}
|
||||
}
|
||||
@@ -148,48 +143,20 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
// refusedWhateverTheAnswer reports whether the request is one that does not wait
|
||||
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
|
||||
// its client: it is over the header limits, a blocklist lists its client while
|
||||
// SWWAF_BLOCKLIST_ACTION is deny, the CrowdSec decision list lists it, or it is
|
||||
// over a whole rate limit. In observe mode, which refuses nothing, none is.
|
||||
// rproxy lets a request through when discriminator answers too late.
|
||||
// its client: the heuristic crowdsec, header_limits, blocklists or rate_limits
|
||||
// will refuse it, as a function in that heuristic's file says. In observe mode,
|
||||
// which refuses nothing, none is. rproxy lets a request through when
|
||||
// discriminator answers too late.
|
||||
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
|
||||
if rq.h.config.Observe {
|
||||
return false
|
||||
}
|
||||
|
||||
lists := rq.h.lists
|
||||
_, decided := lists.CrowdSecDecision(rq.client, now)
|
||||
_, decided := rq.crowdSecDecisionInForce(now)
|
||||
|
||||
return decided || rq.overHeaderLimits() ||
|
||||
(rq.h.config.BlocklistAction == deny && len(lists.ListedBy(rq.client)) > 0) ||
|
||||
(rq.countedByRateLimits() &&
|
||||
rq.h.limiter.OverARateLimit(rq.h.clientGroup(rq.client), now))
|
||||
}
|
||||
|
||||
// overHeaderLimits reports whether discriminator refuses the request for its
|
||||
// headers: it has more than maxHeaderFields fields, each value of a header sent
|
||||
// twice being one, or more than maxCookies cookies, each part of a Cookie header
|
||||
// between semicolons being one unless it is empty or only spaces and tabs. The
|
||||
// Core Rule Set runs each of its rules on each cookie, so that thousands of small
|
||||
// cookies keep it busy for over a second. It trims only spaces and control
|
||||
// characters, of which a header value can hold only tabs, so a part of Unicode
|
||||
// spaces is a cookie to it.
|
||||
func (rq *request) overHeaderLimits() bool {
|
||||
fields, cookies := 0, 0
|
||||
|
||||
for _, values := range rq.in.Header {
|
||||
fields += len(values)
|
||||
}
|
||||
|
||||
for _, value := range rq.in.Header.Values("Cookie") {
|
||||
for part := range strings.SplitSeq(value, ";") {
|
||||
if strings.Trim(part, " \t") != "" {
|
||||
cookies++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return fields > maxHeaderFields || cookies > maxCookies
|
||||
return decided || overHeaderLimits(rq.headerCounts()) ||
|
||||
rq.refusedByBlocklists(rq.h.lists.ListedBy(rq.client)) ||
|
||||
rq.overAWholeRateLimit(now)
|
||||
}
|
||||
|
||||
// answerQuestion runs the checks on the visitor's request a question
|
||||
@@ -220,18 +187,18 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
if refused == nil && !underOwnPrefix {
|
||||
// In observe mode, what enforce mode would have refused counts here.
|
||||
defer rq.countRefusal()
|
||||
defer rq.runAfterAnswer(r.Context())
|
||||
|
||||
rq.allow()
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// The last deferred runs first: countRefusal before addToHistory, so that a
|
||||
// broken error burst is in the client's history.
|
||||
// The last deferred runs first: runAfterAnswer before addToHistory, so that
|
||||
// a broken refusal limit is in the client's history.
|
||||
defer rq.addToHistory()
|
||||
defer rq.countAnomalies()
|
||||
defer rq.countRefusal()
|
||||
defer rq.runAfterAnswer(r.Context())
|
||||
|
||||
if refused != nil {
|
||||
rq.deny(*refused)
|
||||
@@ -241,7 +208,7 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
// The app never sees this prefix. discriminator's own endpoints are at its
|
||||
// own address, so this answers as a path with no endpoint does.
|
||||
rq.line.Action = requestlog.ActionAdmin
|
||||
rq.line.Action = requestlog.ActionOwnEndpoint
|
||||
rq.out.Header().Set(DispositionHeader, DispositionDeny)
|
||||
http.Error(rq.out, http.StatusText(http.StatusNotFound), http.StatusNotFound)
|
||||
}
|
||||
@@ -306,10 +273,9 @@ func (h *handler) refuseQuestion(w http.ResponseWriter, reason string) {
|
||||
|
||||
// asTheVisitorSentIt returns a copy of r, a question, as the checks and the
|
||||
// log line are to see it: without the headers rproxy adds for discriminator,
|
||||
// with
|
||||
// body, which has been read of r, and with the length the visitor announced,
|
||||
// as -1 when it announced none. The question's own length covers only the part
|
||||
// of the body it carries.
|
||||
// with body, which has been read of r, and with the length the visitor
|
||||
// announced, as -1 when it announced none. The question's own length covers
|
||||
// only the part of the body it carries.
|
||||
func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request {
|
||||
sent := r.Clone(r.Context())
|
||||
announced := sent.Header.Get(ContentLengthHeader)
|
||||
@@ -336,7 +302,7 @@ func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request {
|
||||
}
|
||||
|
||||
// allow answers a question with DispositionAllow and, while
|
||||
// SWWAF_ADD_LOOKUP_HEADERS is set, the client's AS number and country.
|
||||
// SWW_ADD_LOOKUP_HEADERS is set, the client's AS number and country.
|
||||
func (rq *request) allow() {
|
||||
rq.allowed = true
|
||||
|
||||
@@ -362,7 +328,7 @@ func (rq *request) deny(r refusal) {
|
||||
rq.refused = &r
|
||||
|
||||
if r.status == 0 {
|
||||
// SWWAF_BAN_RESPONSE is close. Written past rq.out, since the log line's
|
||||
// SWW_BAN_RESPONSE is close. Written past rq.out, since the log line's
|
||||
// status stays 0: the visitor is sent nothing.
|
||||
rq.out.Header().Set(DispositionHeader, DispositionClose)
|
||||
rq.out.ResponseWriter.WriteHeader(http.StatusOK)
|
||||
@@ -414,7 +380,7 @@ func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
|
||||
// Remove reports true to one report only, of several that name the id. A
|
||||
// question still being handled is held as nil.
|
||||
if found && held != nil && h.awaitingReport.Remove(report.RequestID) {
|
||||
h.finishWith(held, report)
|
||||
h.finishWith(r.Context(), held, report)
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
@@ -439,7 +405,7 @@ func readReport(w http.ResponseWriter, r *http.Request) (Report, error) {
|
||||
|
||||
for _, name := range []string{
|
||||
"request_id", "status", "request_bytes", "response_bytes",
|
||||
"duration_total", "duration_upstream_total",
|
||||
"duration_total_ms", "duration_upstream_total_ms",
|
||||
} {
|
||||
value, there := fields[name]
|
||||
if !there || string(value) == "null" {
|
||||
@@ -465,11 +431,11 @@ func (report *Report) checkRanges() error {
|
||||
return fmt.Errorf("%w: request_bytes", errReportValue)
|
||||
case report.ResponseBytes < 0 || report.ResponseBytes > maxReportedBytes:
|
||||
return fmt.Errorf("%w: response_bytes", errReportValue)
|
||||
case report.DurationTotal < 0 || report.DurationTotal > maxReportedMilliseconds:
|
||||
return fmt.Errorf("%w: duration_total", errReportValue)
|
||||
case report.DurationUpstreamTotal < 0 ||
|
||||
report.DurationUpstreamTotal > maxReportedMilliseconds:
|
||||
return fmt.Errorf("%w: duration_upstream_total", errReportValue)
|
||||
case report.DurationTotalMS < 0 || report.DurationTotalMS > maxReportedMilliseconds:
|
||||
return fmt.Errorf("%w: duration_total_ms", errReportValue)
|
||||
case report.DurationUpstreamTotalMS < 0 ||
|
||||
report.DurationUpstreamTotalMS > maxReportedMilliseconds:
|
||||
return fmt.Errorf("%w: duration_upstream_total_ms", errReportValue)
|
||||
}
|
||||
|
||||
return nil
|
||||
@@ -477,11 +443,11 @@ func (report *Report) checkRanges() error {
|
||||
|
||||
// finishWith finishes a request with what its report says, and writes the
|
||||
// request's last log line, which holds its decision's fields and the report's.
|
||||
// For an allowed request it first counts the request's bytes, and adds it to its
|
||||
// client's history, the anomaly counters and the metrics, with the status and
|
||||
// bytes the report gives. A denied request's counting was finished at its
|
||||
// decision.
|
||||
func (h *handler) finishWith(held *decision, report Report) {
|
||||
// For an allowed request it first runs the heuristics of onReport, and adds the
|
||||
// request to its client's history, the anomaly counters and the metrics, with
|
||||
// the status and bytes the report gives. A denied request's counting was
|
||||
// finished at its decision. ctx is the report's own context.
|
||||
func (h *handler) finishWith(ctx context.Context, held *decision, report Report) {
|
||||
rq := &request{
|
||||
decision: *held,
|
||||
h: h,
|
||||
@@ -493,18 +459,18 @@ func (h *handler) finishWith(held *decision, report Report) {
|
||||
rq.line.Status = report.Status
|
||||
rq.line.RequestBytes = report.RequestBytes
|
||||
rq.line.ResponseBytes = report.ResponseBytes
|
||||
rq.line.DurationTotal = report.DurationTotal
|
||||
rq.line.DurationTotalMS = report.DurationTotalMS
|
||||
|
||||
if rq.allowed {
|
||||
rq.line.DurationUpstreamTotal = &report.DurationUpstreamTotal
|
||||
rq.line.DurationUpstreamTotalMS = &report.DurationUpstreamTotalMS
|
||||
|
||||
rq.countBytes()
|
||||
rq.runBanningHeuristics(ctx, onReport())
|
||||
rq.countAnomalies()
|
||||
rq.addToHistory()
|
||||
|
||||
rq.h.metrics.RequestReported(&rq.line,
|
||||
time.Duration(report.DurationTotal*float64(time.Millisecond)),
|
||||
time.Duration(report.DurationUpstreamTotal*float64(time.Millisecond)))
|
||||
time.Duration(report.DurationTotalMS*float64(time.Millisecond)),
|
||||
time.Duration(report.DurationUpstreamTotalMS*float64(time.Millisecond)))
|
||||
}
|
||||
|
||||
rq.writeLine()
|
||||
|
||||
@@ -39,7 +39,7 @@ const (
|
||||
// reportBodyLimit is the longest body SPEC.md lets a report have.
|
||||
reportBodyLimit = 1 << 20
|
||||
// anomalyClientPerMinute is the anomaly threshold these tests set.
|
||||
anomalyClientPerMinute = "SWWAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE"
|
||||
anomalyClientPerMinute = "SWW_ANOMALY_CLIENT_REQUESTS_PER_MINUTE"
|
||||
)
|
||||
|
||||
// The fields of a report, as "discriminator" in SPEC.md names them.
|
||||
@@ -48,8 +48,8 @@ const (
|
||||
reportStatus = "status"
|
||||
reportRequestBytes = "request_bytes"
|
||||
reportResponseBytes = "response_bytes"
|
||||
reportDuration = "duration_total"
|
||||
reportUpstreamDuration = "duration_upstream_total"
|
||||
reportDuration = "duration_total_ms"
|
||||
reportUpstreamDuration = "duration_upstream_total_ms"
|
||||
)
|
||||
|
||||
// Discriminator is a discriminator running for one test, on a clock set to midnight.
|
||||
@@ -181,7 +181,7 @@ func (s *Discriminator) history() ratelimit.History {
|
||||
}
|
||||
|
||||
// metrics returns what the metrics endpoint answers, for a server started with
|
||||
// the token as SWWAF_METRICS_TOKEN.
|
||||
// the token as SWW_METRICS_TOKEN.
|
||||
func (s *Discriminator) metrics() string {
|
||||
s.t.Helper()
|
||||
|
||||
@@ -414,7 +414,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
|
||||
|
||||
geojsURL, asked, release := startHeldGeoJS(t)
|
||||
s := startDiscriminator(t, geojsURL, map[string]string{
|
||||
lookupTimeout: "1h", deniedCountries: "kp", bytesLimitPerMinute: byteLimit,
|
||||
lookupTimeout: "1h", tagDeny: denyKP, bytesLimitPerMinute: byteLimit,
|
||||
})
|
||||
|
||||
first := s.question(siteA, client, "/")
|
||||
@@ -639,7 +639,7 @@ func TestQuestionDeniedGetsTheBanResponse(t *testing.T) {
|
||||
got := s.ask(client, "/")
|
||||
wantDisposition(t, got, status, discriminator.DispositionDeny)
|
||||
wantBody(t, got, http.StatusText(status)+"\n")
|
||||
wantLine(t, s.out.requestLine(t), status, requestlog.ActionDenied)
|
||||
wantLine(t, s.out.requestLine(t), status, requestlog.ActionClientDenied)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -655,8 +655,8 @@ func TestBanResponseCloseIsAnAnswerOfItsOwn(t *testing.T) {
|
||||
wantDisposition(t, got, http.StatusOK, discriminator.DispositionClose)
|
||||
wantBody(t, got, "")
|
||||
// The visitor is sent nothing.
|
||||
wantLine(t, s.out.requestLine(t), 0, requestlog.ActionDenied)
|
||||
wantMetric(t, s.metrics(), `sww_requests_total{action="denied",instance="`+
|
||||
wantLine(t, s.out.requestLine(t), 0, requestlog.ActionClientDenied)
|
||||
wantMetric(t, s.metrics(), `sww_requests_total{action="client_denied",instance="`+
|
||||
alertInstance+`",status_class="none"}`, 1)
|
||||
}
|
||||
|
||||
@@ -671,12 +671,12 @@ func TestBlockRuleIsDeniedWith403AndABanRuleWithTheBanResponse(t *testing.T) {
|
||||
got := s.ask(client, "/blocked")
|
||||
wantDisposition(t, got, http.StatusForbidden, discriminator.DispositionDeny)
|
||||
wantLine(t, s.out.requestLine(t), http.StatusForbidden,
|
||||
requestlog.ActionRuleBlocked)
|
||||
requestlog.ActionBlockedByBlockRule)
|
||||
|
||||
got = s.ask(client, "/.env")
|
||||
wantDisposition(t, got, http.StatusTooManyRequests, discriminator.DispositionDeny)
|
||||
wantLine(t, s.out.requestLines(t, 2)[1], http.StatusTooManyRequests,
|
||||
requestlog.ActionBanned)
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(t *testing.T) {
|
||||
@@ -693,7 +693,7 @@ func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(t *testing.T) {
|
||||
wantDisposition(t, got, http.StatusNotFound, discriminator.DispositionDeny)
|
||||
}
|
||||
|
||||
wantLine(t, s.out.requestLine(t), http.StatusNotFound, requestlog.ActionAdmin)
|
||||
wantLine(t, s.out.requestLine(t), http.StatusNotFound, requestlog.ActionOwnEndpoint)
|
||||
|
||||
got := s.ask(otherClient, discriminator.MetricsPath)
|
||||
wantDisposition(t, got, http.StatusForbidden, discriminator.DispositionDeny)
|
||||
@@ -709,7 +709,7 @@ func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) {
|
||||
|
||||
line := s.out.requestLine(t)
|
||||
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionDenied)
|
||||
wantWouldAction(t, line, requestlog.ActionClientDenied)
|
||||
}
|
||||
|
||||
func TestObserveModeRefusalAnsweredAllowCountsForTheErrorBurst(t *testing.T) {
|
||||
@@ -725,9 +725,9 @@ func TestObserveModeRefusalAnsweredAllowCountsForTheErrorBurst(t *testing.T) {
|
||||
}
|
||||
|
||||
lines := s.out.requestLines(t, 2)
|
||||
if lines[0].LimitHit != "" || lines[1].LimitHit != requestlog.LimitHitErrorBurst {
|
||||
if lines[0].LimitHit != "" || lines[1].LimitHit != requestlog.LimitHitMinuteRefusals {
|
||||
t.Errorf("log lines have limit_hit %q and %q, want none and %q",
|
||||
lines[0].LimitHit, lines[1].LimitHit, requestlog.LimitHitErrorBurst)
|
||||
lines[0].LimitHit, lines[1].LimitHit, requestlog.LimitHitMinuteRefusals)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -744,9 +744,9 @@ func TestDeniedQuestionCountsForTheErrorBurst(t *testing.T) {
|
||||
}
|
||||
|
||||
lines := s.out.requestLines(t, 2)
|
||||
if lines[0].LimitHit != "" || lines[1].LimitHit != requestlog.LimitHitErrorBurst {
|
||||
if lines[0].LimitHit != "" || lines[1].LimitHit != requestlog.LimitHitMinuteRefusals {
|
||||
t.Errorf("log lines have limit_hit %q and %q, want none and %q",
|
||||
lines[0].LimitHit, lines[1].LimitHit, requestlog.LimitHitErrorBurst)
|
||||
lines[0].LimitHit, lines[1].LimitHit, requestlog.LimitHitMinuteRefusals)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -765,7 +765,7 @@ func TestAllowCarriesTheLookupHeadersWhileTheSettingIsSet(t *testing.T) {
|
||||
|
||||
geojsURL, _ := startGeoJS(t)
|
||||
s := startDiscriminator(t, geojsURL, map[string]string{
|
||||
addLookupHeaders: tc.setting, lookupTimeout: "1h", deniedCountries: "kp",
|
||||
addLookupHeaders: tc.setting, lookupTimeout: "1h", tagDeny: denyKP,
|
||||
})
|
||||
|
||||
got := s.ask(fromDE, "/")
|
||||
@@ -894,14 +894,14 @@ func TestDeniedRequestIsFinishedAtItsDecision(t *testing.T) {
|
||||
t.Errorf("history is %+v, want the refused request alone", history)
|
||||
}
|
||||
|
||||
const series = `sww_requests_total{action="rate_limited",instance="` +
|
||||
const series = `sww_requests_total{action="client_is_rate_limited",instance="` +
|
||||
alertInstance + `",status_class="4xx"}`
|
||||
|
||||
wantMetric(t, s.metrics(), series, 1)
|
||||
|
||||
got := s.report(discriminator.Report{
|
||||
RequestID: denied.Header.Get(requestIDHeader), Status: http.StatusForbidden,
|
||||
ResponseBytes: 1000, DurationTotal: 2,
|
||||
ResponseBytes: 1000, DurationTotalMS: 2,
|
||||
})
|
||||
wantStatus(t, got, http.StatusNoContent)
|
||||
|
||||
@@ -935,7 +935,7 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
|
||||
got := s.report(discriminator.Report{
|
||||
RequestID: id, Status: http.StatusBadGateway,
|
||||
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
|
||||
DurationTotal: 5, DurationUpstreamTotal: 4,
|
||||
DurationTotalMS: 5, DurationUpstreamTotalMS: 4,
|
||||
})
|
||||
wantStatus(t, got, http.StatusNoContent)
|
||||
|
||||
@@ -951,7 +951,8 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
|
||||
unscraped := startDiscriminator(t, "", map[string]string{metricsToken: token})
|
||||
|
||||
got = unscraped.report(discriminator.Report{
|
||||
RequestID: unscraped.allowed(client), Status: http.StatusBadGateway, DurationTotal: 5,
|
||||
RequestID: unscraped.allowed(client), Status: http.StatusBadGateway,
|
||||
DurationTotalMS: 5,
|
||||
})
|
||||
wantStatus(t, got, http.StatusNoContent)
|
||||
wantMetric(t, unscraped.metrics(),
|
||||
@@ -1189,8 +1190,8 @@ func wantReportsFields(t *testing.T, line logLine) {
|
||||
if line.DiscriminatorLine != requestlog.DiscriminatorLineReport ||
|
||||
line.Status != http.StatusCreated ||
|
||||
line.RequestBytes != bodyBytes || line.ResponseBytes != answerBytes ||
|
||||
line.DurationTotal != 12.5 || line.DurationUpstreamTotal == nil ||
|
||||
*line.DurationUpstreamTotal != 10.25 {
|
||||
line.DurationTotalMS != 12.5 || line.DurationUpstreamTotalMS == nil ||
|
||||
*line.DurationUpstreamTotalMS != 10.25 {
|
||||
t.Errorf("the report's line lacks the report's fields: %+v", line.Line)
|
||||
}
|
||||
}
|
||||
@@ -1205,8 +1206,8 @@ func wantSameDecision(t *testing.T, reported, decided logLine) {
|
||||
same.Status = decided.Status
|
||||
same.RequestBytes = decided.RequestBytes
|
||||
same.ResponseBytes = decided.ResponseBytes
|
||||
same.DurationTotal = decided.DurationTotal
|
||||
same.DurationUpstreamTotal = decided.DurationUpstreamTotal
|
||||
same.DurationTotalMS = decided.DurationTotalMS
|
||||
same.DurationUpstreamTotalMS = decided.DurationUpstreamTotalMS
|
||||
same.Counts.MinuteBytes = decided.Counts.MinuteBytes
|
||||
same.Counts.HourBytes = decided.Counts.HourBytes
|
||||
same.Counts.DayBytes = decided.Counts.DayBytes
|
||||
@@ -1224,7 +1225,7 @@ func wantDecisionsFields(t *testing.T, line logLine, id, query, accept string) {
|
||||
|
||||
if line.DiscriminatorLine != requestlog.DiscriminatorLineDecision ||
|
||||
line.Status != http.StatusOK || line.ResponseBytes != 0 ||
|
||||
line.DurationUpstreamTotal != nil || line.DurationChecks == nil ||
|
||||
line.DurationUpstreamTotalMS != nil || line.DurationChecksMS == nil ||
|
||||
line.RequestID != id || line.Query != query ||
|
||||
!line.HasAuthorization || !line.HasCookie ||
|
||||
line.RequestHeaders["accept"] != accept || len(line.RuleIDs) != 1 ||
|
||||
@@ -1260,7 +1261,7 @@ func TestReportWritesTheLastLineWithTheDecisionsAndTheReportsFields(t *testing.T
|
||||
got := s.report(discriminator.Report{
|
||||
RequestID: id, Status: http.StatusCreated,
|
||||
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
|
||||
DurationTotal: 12.5, DurationUpstreamTotal: 10.25,
|
||||
DurationTotalMS: 12.5, DurationUpstreamTotalMS: 10.25,
|
||||
})
|
||||
wantStatus(t, got, http.StatusNoContent)
|
||||
|
||||
@@ -1298,14 +1299,14 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
|
||||
}{
|
||||
{
|
||||
"deny", map[string]string{denyNets: client}, "/", http.StatusForbidden,
|
||||
requestlog.ActionDenied,
|
||||
requestlog.ActionClientDenied,
|
||||
},
|
||||
{
|
||||
"close", map[string]string{denyNets: client, banResponse: banResponseClose}, "/",
|
||||
0, requestlog.ActionDenied,
|
||||
0, requestlog.ActionClientDenied,
|
||||
},
|
||||
{"own prefix", nil, discriminator.HealthPath, http.StatusNotFound,
|
||||
requestlog.ActionAdmin},
|
||||
requestlog.ActionOwnEndpoint},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -1317,7 +1318,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
|
||||
do(t, req)
|
||||
|
||||
got := s.report(discriminator.Report{
|
||||
RequestID: id, Status: tc.status, ResponseBytes: 8, DurationTotal: 3,
|
||||
RequestID: id, Status: tc.status, ResponseBytes: 8, DurationTotalMS: 3,
|
||||
})
|
||||
wantStatus(t, got, http.StatusNoContent)
|
||||
|
||||
@@ -1325,8 +1326,8 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
|
||||
wantLine(t, line, tc.status, tc.action)
|
||||
|
||||
if line.DiscriminatorLine != requestlog.DiscriminatorLineReport ||
|
||||
line.RequestID != id || line.ResponseBytes != 8 || line.DurationTotal != 3 ||
|
||||
line.DurationUpstreamTotal != nil {
|
||||
line.RequestID != id || line.ResponseBytes != 8 || line.DurationTotalMS != 3 ||
|
||||
line.DurationUpstreamTotalMS != nil {
|
||||
t.Errorf("the report's line is %+v", line.Line)
|
||||
}
|
||||
})
|
||||
@@ -1485,7 +1486,7 @@ func TestALargeBodyAnnouncedAndLongHeadersAreNoReasonToRefuse(t *testing.T) {
|
||||
func TestLogLevelHoldsBackNoRequestLine(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := startDiscriminator(t, "", map[string]string{"SWWAF_LOG_LEVEL": "error"})
|
||||
s := startDiscriminator(t, "", map[string]string{"SWW_LOG_LEVEL": "error"})
|
||||
|
||||
// A refused question would write a process warning, held back at error.
|
||||
req := s.question(siteA, client, "/")
|
||||
@@ -1583,9 +1584,10 @@ func TestCoreRuleSetAlertNamesTheSite(t *testing.T) {
|
||||
wantDisposition(t, got, http.StatusForbidden, discriminator.DispositionDeny)
|
||||
|
||||
waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
if len(waiting) != 1 || waiting[0].Event != alerts.EventWAFBlock ||
|
||||
if len(waiting) != 1 || waiting[0].Event != alerts.EventCoreRuleSetMatch ||
|
||||
waiting[0].Host != siteA {
|
||||
t.Errorf("alerts waiting are %+v, want a waf_block naming %s", waiting, siteA)
|
||||
t.Errorf("alerts waiting are %+v, want a core_rule_set_match naming %s",
|
||||
waiting, siteA)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic dnsbl. It reads SWW_REPUTATION_ACTION, and the DNSBL zones'
|
||||
// verdicts through the DNSBL checker.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// dnsbl notes the DNSBL zones whose verdict lists the client, as noteListed
|
||||
// does, adds the tag dnsbl=<zone> for each, the zone's key masked, and refuses
|
||||
// the request when SWW_REPUTATION_ACTION is deny. Being
|
||||
// limit, it lowers the client's limits instead (see limitPercentages), and
|
||||
// being log, it does nothing more. A zone without a verdict on the client is
|
||||
// asked about it in the background, and the request does not wait for the
|
||||
// answer. Its figure is the zones that list the client, their keys masked.
|
||||
type dnsbl struct{}
|
||||
|
||||
func (dnsbl) name() string { return "dnsbl" }
|
||||
|
||||
func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict {
|
||||
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
|
||||
rq.dnsblListed = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a DNSBL zone")
|
||||
|
||||
for _, zone := range listedBy {
|
||||
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
|
||||
}
|
||||
|
||||
if !rq.dnsblListed || rq.h.config.ReputationAction != deny {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionClientDenied,
|
||||
figures: map[string]any{"zones": listedBy},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic header_limits. It reads no setting: its limits are fixed.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// The header limits: the most cookies and header fields a request may have
|
||||
// when the checks reach the Core Rule Set. Real clients never reach them.
|
||||
const (
|
||||
maxCookies = 50
|
||||
maxHeaderFields = 100
|
||||
)
|
||||
|
||||
// headerLimits refuses a request over the header limits, with 431. Its figures
|
||||
// are the request's header fields and cookies, and the limit of each.
|
||||
type headerLimits struct{}
|
||||
|
||||
func (headerLimits) name() string { return "header_limits" }
|
||||
|
||||
func (headerLimits) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
fields, cookies := rq.headerCounts()
|
||||
if !overHeaderLimits(fields, cookies) {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionRequestHeaderTooLarge,
|
||||
figures: map[string]any{
|
||||
"fields": fields, "fields_limit": maxHeaderFields,
|
||||
"cookies": cookies, "cookies_limit": maxCookies,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// overHeaderLimits reports whether a request with these counts of header fields
|
||||
// and cookies, as headerCounts counts them, is over the header limits: it has
|
||||
// more than maxHeaderFields fields or more than maxCookies cookies.
|
||||
func overHeaderLimits(fields, cookies int) bool {
|
||||
return fields > maxHeaderFields || cookies > maxCookies
|
||||
}
|
||||
|
||||
// headerCounts returns the request's header fields, each value of a header sent
|
||||
// twice being one, and its cookies, each part of a Cookie header between
|
||||
// semicolons being one unless it is empty or only spaces and tabs. The Core
|
||||
// Rule Set runs each of its rules on each cookie, so that thousands of small
|
||||
// cookies keep it busy for over a second. It trims only spaces and control
|
||||
// characters, of which a header value can hold only tabs, so a part of Unicode
|
||||
// spaces is a cookie to it.
|
||||
func (rq *request) headerCounts() (int, int) {
|
||||
fields, cookies := 0, 0
|
||||
|
||||
for _, values := range rq.in.Header {
|
||||
fields += len(values)
|
||||
}
|
||||
|
||||
for _, value := range rq.in.Header.Values("Cookie") {
|
||||
for part := range strings.SplitSeq(value, ";") {
|
||||
if strings.Trim(part, " \t") != "" {
|
||||
cookies++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return fields, cookies
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
package discriminator
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// To add a heuristic, write one file with a type that implements heuristic,
|
||||
// and add one line for it to one of the lists below. Its settings go in
|
||||
// config.Config, and the comment at the top of its file names them.
|
||||
|
||||
// heuristic is one of the checks that decide what discriminator does with a
|
||||
// request.
|
||||
type heuristic interface {
|
||||
// name is the heuristic's name, which the log line gives.
|
||||
name() string
|
||||
// check looks at the request at now, and returns nil to let it through.
|
||||
// ctx is the request's own context.
|
||||
check(ctx context.Context, rq *request, now time.Time) *verdict
|
||||
}
|
||||
|
||||
// verdict is a heuristic refusing a request, or banning its client: the action
|
||||
// the log line names, one of requestlog's Action constants, and the figures the
|
||||
// heuristic acted on, by name, such as the count and the limit.
|
||||
type verdict struct {
|
||||
action string
|
||||
figures map[string]any
|
||||
}
|
||||
|
||||
// The names of the figures that more than one heuristic gives.
|
||||
const (
|
||||
figureWindow = "window"
|
||||
figureCount = "count"
|
||||
figureLimit = "limit"
|
||||
figureScore = "score"
|
||||
figureNetblock = "netblock"
|
||||
)
|
||||
|
||||
// The lists of heuristics, each in the order its heuristics run. A client in
|
||||
// SWW_ALLOW_NETS skips them all.
|
||||
|
||||
// beforeLookup are the heuristics that run before the client's AS number and
|
||||
// country are looked up, so that a client they refuse is not looked up.
|
||||
func beforeLookup() []heuristic {
|
||||
return []heuristic{
|
||||
tagLists{},
|
||||
routeWatchTags{},
|
||||
denyNets{},
|
||||
activeBan{},
|
||||
}
|
||||
}
|
||||
|
||||
// afterLookup are the heuristics that run once the lookup has been made.
|
||||
// tag_deny runs twice: before the rate limits, on the tags known by then, and
|
||||
// last, on the tag rule_files and core_rule_set can add.
|
||||
func afterLookup() []heuristic {
|
||||
return []heuristic{
|
||||
countries{},
|
||||
blocklists{},
|
||||
crowdSec{},
|
||||
dnsbl{},
|
||||
abuseIPDB{},
|
||||
tagDeny{},
|
||||
rateLimits{},
|
||||
trapPaths{},
|
||||
ruleFiles{},
|
||||
headerLimits{},
|
||||
coreRuleSet{},
|
||||
tagDeny{},
|
||||
}
|
||||
}
|
||||
|
||||
// afterAnswer are the heuristics that run on a request once discriminator has
|
||||
// answered it, before its decision line is written. They ban the client, and do
|
||||
// not change what the request got: their verdict's action is not used.
|
||||
func afterAnswer() []heuristic {
|
||||
return []heuristic{
|
||||
refusalLimit{},
|
||||
}
|
||||
}
|
||||
|
||||
// onReport are the heuristics that run on an allowed request when rproxy's
|
||||
// report of it comes, before its report line is written. Like afterAnswer, they
|
||||
// only ban.
|
||||
func onReport() []heuristic {
|
||||
return []heuristic{
|
||||
byteLimits{},
|
||||
}
|
||||
}
|
||||
|
||||
// runAfterAnswer runs the heuristics of afterAnswer on the request, at the time
|
||||
// it is called.
|
||||
func (rq *request) runAfterAnswer(ctx context.Context) {
|
||||
rq.runBanningHeuristics(ctx, afterAnswer())
|
||||
}
|
||||
|
||||
// runHeuristics runs list, beforeLookup or afterLookup, on the request at now,
|
||||
// and returns the first verdict, or nil when none gives one. The log line's
|
||||
// heuristic and heuristic_figures name the heuristic that gave it, and give its
|
||||
// figures. ctx is the request's own context.
|
||||
func (rq *request) runHeuristics(
|
||||
ctx context.Context, list []heuristic, now time.Time,
|
||||
) *verdict {
|
||||
name, v := rq.firstVerdict(ctx, list, now)
|
||||
if v != nil {
|
||||
rq.line.Heuristic = name
|
||||
rq.line.HeuristicFigures = v.figures
|
||||
}
|
||||
|
||||
return v
|
||||
}
|
||||
|
||||
// runBanningHeuristics runs list, afterAnswer or onReport, on the request at
|
||||
// the time it is called. The log line's after_answer_heuristic and
|
||||
// after_answer_heuristic_figures name the heuristic that gave a verdict, and
|
||||
// give its figures, so that heuristic and heuristic_figures still name the one
|
||||
// that refused the request. ctx is the request's own context.
|
||||
func (rq *request) runBanningHeuristics(ctx context.Context, list []heuristic) {
|
||||
name, v := rq.firstVerdict(ctx, list, rq.h.now())
|
||||
if v != nil {
|
||||
rq.line.AfterAnswerHeuristic = name
|
||||
rq.line.AfterAnswerHeuristicFigures = v.figures
|
||||
}
|
||||
}
|
||||
|
||||
// firstVerdict runs list on the request at now, in order, until a heuristic
|
||||
// gives a verdict, and returns that heuristic's name and its verdict, or "" and
|
||||
// nil when none gives one. ctx is the request's own context.
|
||||
func (rq *request) firstVerdict(
|
||||
ctx context.Context, list []heuristic, now time.Time,
|
||||
) (string, *verdict) {
|
||||
for _, each := range list {
|
||||
v := each.check(ctx, rq, now)
|
||||
if v != nil {
|
||||
return each.name(), v
|
||||
}
|
||||
}
|
||||
|
||||
return "", nil
|
||||
}
|
||||
@@ -20,7 +20,7 @@ func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
|
||||
s, clk, server := startWithClock(t, geojsURL, map[string]string{
|
||||
lookupTimeout: "1h",
|
||||
rateLimitPerMinute: "2",
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
start := clk.Now()
|
||||
|
||||
@@ -31,9 +31,9 @@ func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
clk.advance(time.Second)
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
clk.advance(time.Second)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
want := ratelimit.History{
|
||||
FirstSeen: start,
|
||||
|
||||
@@ -4,13 +4,15 @@ import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
)
|
||||
|
||||
// The headers in which an answer of allow gives rproxy the client's AS number
|
||||
// and country, for the app, while SWWAF_ADD_LOOKUP_HEADERS is set. Go writes
|
||||
// and country, for the app, while SWW_ADD_LOOKUP_HEADERS is set. Go writes
|
||||
// every header name in this form, as it sends it and as it receives it, so
|
||||
// X-Client-ASN arrives as X-Client-Asn; header names are not case-sensitive.
|
||||
const (
|
||||
@@ -19,8 +21,9 @@ const (
|
||||
)
|
||||
|
||||
// lookUp looks up the client's AS number and country, in the lookup
|
||||
// database or through GeoJS, and notes them for the log line, unless
|
||||
// SWWAF_LOOKUP_SOURCE is off or the client is on a private, loopback or
|
||||
// database or through GeoJS, and notes them for the log line and as the
|
||||
// request's asn= and country= tags, unless
|
||||
// SWW_LOOKUP_SOURCE is off or the client is on a private, loopback or
|
||||
// link-local address, which no lookup can place. The lookup database
|
||||
// answers at once. With GeoJS, while a setting needs the answer, such as a
|
||||
// country list or a biased threshold, a new client's request waits for it,
|
||||
@@ -47,6 +50,34 @@ func (rq *request) lookUp(ctx context.Context, now time.Time) {
|
||||
rq.line.ASN = rq.lookupAnswer.ASN
|
||||
rq.line.ASName = rq.lookupAnswer.ASName
|
||||
rq.line.Country = rq.lookupAnswer.Country
|
||||
|
||||
if tag := asnTag(rq.line.ASN); tag != "" {
|
||||
rq.addTag(tag)
|
||||
}
|
||||
|
||||
if tag := countryTag(rq.line.Country); tag != "" {
|
||||
rq.addTag(tag)
|
||||
}
|
||||
}
|
||||
|
||||
// asnTag returns the tag of asn, an AS number as the lookup gives it, such as
|
||||
// asn=64496 for AS64496, or "" for no AS number.
|
||||
func asnTag(asn string) string {
|
||||
if asn == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
return config.ASNTagPrefix + strings.TrimPrefix(asn, "AS")
|
||||
}
|
||||
|
||||
// countryTag returns the tag of country, a code as the lookup gives it, such
|
||||
// as country=DE for DE, or "" for no country.
|
||||
func countryTag(country string) string {
|
||||
if country == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
return config.CountryTagPrefix + country
|
||||
}
|
||||
|
||||
// addLookup adds answer, an answer about a client from the lookup
|
||||
|
||||
@@ -20,7 +20,7 @@ import (
|
||||
// and country.
|
||||
type asnAndCountry struct{ asn, asName, country string }
|
||||
|
||||
// fileSource is the SWWAF_LOOKUP_SOURCE that looks clients up in the
|
||||
// fileSource is the SWW_LOOKUP_SOURCE that looks clients up in the
|
||||
// lookup database.
|
||||
const fileSource = "file"
|
||||
|
||||
@@ -40,7 +40,7 @@ func TestEveryClientIsLookedUpWithoutWaitingWhileNoSettingNeedsIt(t *testing.T)
|
||||
// comes too. None waits for GeoJS.
|
||||
for _, line := range []logLine{
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited),
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited),
|
||||
s.get(fromKP, http.StatusOK, requestlog.ActionForward),
|
||||
} {
|
||||
got := asnAndCountry{line.ASN, line.ASName, line.Country}
|
||||
@@ -94,9 +94,9 @@ func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) {
|
||||
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 58, 0, time.UTC)}
|
||||
addr, out, server := startProxyWithClock(t, app.URL, geojsURL, clk.Now,
|
||||
map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
lookupTimeout: "1h",
|
||||
deniedCountries: "kp",
|
||||
trustedProxies: trustLocalhost,
|
||||
lookupTimeout: "1h",
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
|
||||
requestFromDE := func() *http.Request {
|
||||
@@ -179,7 +179,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(t *testing.
|
||||
|
||||
for _, line := range []logLine{
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited),
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited),
|
||||
} {
|
||||
if got := (asnAndCountry{line.ASN, line.ASName, line.Country}); got != want {
|
||||
t.Errorf("log line has %+v, want %+v", got, want)
|
||||
@@ -264,7 +264,7 @@ func TestClientsAreLookedUpInTheLookupDatabaseAndGeoJSIsNotAsked(t *testing.T) {
|
||||
want asnAndCountry
|
||||
}{
|
||||
{s.get(fromDE, http.StatusOK, requestlog.ActionForward), de},
|
||||
{s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited), de},
|
||||
{s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited), de},
|
||||
{
|
||||
s.get(fromKP, http.StatusForbidden, requestlog.ActionCountryDenied),
|
||||
asnAndCountry{asnKP, asNameKP, "KP"},
|
||||
|
||||
@@ -17,9 +17,9 @@ import (
|
||||
)
|
||||
|
||||
const (
|
||||
metricsToken = "SWWAF_METRICS_TOKEN" //nolint:gosec // the setting's name
|
||||
metricsTopN = "SWWAF_METRICS_TOP_N"
|
||||
// token is the SWWAF_METRICS_TOKEN the tests set, and bearer how a
|
||||
metricsToken = "SWW_METRICS_TOKEN" //nolint:gosec // the setting's name
|
||||
metricsTopN = "SWW_METRICS_TOP_N"
|
||||
// token is the SWW_METRICS_TOKEN the tests set, and bearer how a
|
||||
// request carries it.
|
||||
token = "0123456789abcdef0123456789abcdef"
|
||||
bearer = "Bearer " + token
|
||||
@@ -91,7 +91,7 @@ func TestMetricsCountTheTraffic(t *testing.T) {
|
||||
out.endedLines(t, 2)
|
||||
|
||||
forward := `{action="forward",instance="app",status_class="2xx"}`
|
||||
notFound := `{action="admin",instance="app",status_class="4xx"}`
|
||||
notFound := `{action="own_endpoint",instance="app",status_class="4xx"}`
|
||||
|
||||
// The request for the metrics is itself under way.
|
||||
metrics := scrape(t, server.Addr)
|
||||
@@ -113,7 +113,7 @@ func TestMetricsCountTheTraffic(t *testing.T) {
|
||||
func TestMetricsCountLimitsAndBans(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const denied = "192.0.2.50" // in SWWAF_DENY_NETS
|
||||
const denied = "192.0.2.50" // in SWW_DENY_NETS
|
||||
|
||||
s, clk, _ := startWithClock(t, "", map[string]string{
|
||||
metricsToken: token,
|
||||
@@ -124,16 +124,16 @@ func TestMetricsCountLimitsAndBans(t *testing.T) {
|
||||
maxBanDuration: "2h",
|
||||
})
|
||||
|
||||
// SWWAF_BAN_RESPONSE=close sends no status at all.
|
||||
s.get(denied, 0, requestlog.ActionDenied)
|
||||
// SWW_BAN_RESPONSE=close sends no status at all.
|
||||
s.get(denied, 0, requestlog.ActionClientDenied)
|
||||
|
||||
// A first broken limit bans for an hour.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, 0, requestlog.ActionRateLimited)
|
||||
s.get(client, 0, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
metrics := s.scrape()
|
||||
wantMetric(t, metrics,
|
||||
`sww_requests_total{action="denied",instance="app",status_class="none"}`, 1)
|
||||
`sww_requests_total{action="client_denied",instance="app",status_class="none"}`, 1)
|
||||
wantMetric(t, metrics, `sww_rate_limit_hits_total{instance="app",`+
|
||||
`kind="requests",window="minute"}`, 1)
|
||||
wantMetric(t, metrics, `sww_offences_total{instance="app",kind="limit"}`, 1)
|
||||
@@ -145,9 +145,9 @@ func TestMetricsCountLimitsAndBans(t *testing.T) {
|
||||
wantMetric(t, s.scrape(), `sww_active_bans{instance="app"}`, 0)
|
||||
|
||||
// A limit broken again right after would ban for three hours, longer
|
||||
// than SWWAF_MAX_BAN_DURATION, so the ban is permanent.
|
||||
// than SWW_MAX_BAN_DURATION, so the ban is permanent.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, 0, requestlog.ActionRateLimited)
|
||||
s.get(client, 0, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
metrics = s.scrape()
|
||||
wantMetric(t, metrics, `sww_rate_limit_hits_total{instance="app",`+
|
||||
@@ -195,10 +195,10 @@ func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) {
|
||||
_, _ = io.WriteString(w, "hello")
|
||||
})
|
||||
env := map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
metricsToken: token,
|
||||
metricsTopN: "2",
|
||||
deniedCountries: "kp",
|
||||
trustedProxies: trustLocalhost,
|
||||
metricsToken: token,
|
||||
metricsTopN: "2",
|
||||
allowedCountries: "de,fr",
|
||||
}
|
||||
geojsURL, _ := startGeoJS(t)
|
||||
addr, out, server := startProxyWithClock(t, app.URL, geojsURL, time.Now, env)
|
||||
@@ -322,18 +322,18 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
|
||||
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
||||
addr, _, server := startProxyWithClock(t, app.URL, geojs.URL, time.Now,
|
||||
map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
metricsToken: token,
|
||||
deniedCountries: "kp",
|
||||
trustedProxies: trustLocalhost,
|
||||
metricsToken: token,
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
|
||||
// GeoJS fails, so the client counts as coming from an unknown country,
|
||||
// which SWWAF_DENIED_COUNTRIES does not refuse.
|
||||
// GeoJS fails, so the client has no country= tag, and SWW_TAG_DENY does
|
||||
// not refuse it.
|
||||
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
|
||||
req.Header.Set(forwardedFor, fromDE)
|
||||
wantStatus(t, do(t, req), http.StatusOK)
|
||||
|
||||
// The client stops waiting for GeoJS once SWWAF_LOOKUP_TIMEOUT is up, so
|
||||
// The client stops waiting for GeoJS once SWW_LOOKUP_TIMEOUT is up, so
|
||||
// GeoJS's failure can come after its request has ended.
|
||||
deadline := time.Now().Add(waitLimit)
|
||||
metrics := scrape(t, server.Addr)
|
||||
@@ -347,7 +347,7 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
|
||||
|
||||
wantMetric(t, metrics, `sww_geojs_requests_total{instance="app"}`, 1)
|
||||
wantMetric(t, metrics, `sww_geojs_failures_total{instance="app"}`, 1)
|
||||
wantMetric(t, metrics, `sww_geojs_unanswered_total{instance="app"}`, 1)
|
||||
wantMetric(t, metrics, `sww_requests_without_geojs_answer_total{instance="app"}`, 1)
|
||||
}
|
||||
|
||||
// keptAnswer returns GeoJS's answer that the client at addr is in
|
||||
|
||||
@@ -7,23 +7,123 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// observe is the value of SWWAF_MODE for observe mode.
|
||||
// observe is the value of SWW_MODE for observe mode.
|
||||
const observe = "observe"
|
||||
|
||||
func TestBanActionSaysWhetherTheBanRunsOut(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const trapPath = "/wp-login.php"
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
cause string
|
||||
// endsIn is how long after the request the ban ends: negative for a
|
||||
// ban that has ended, and zero for a permanent ban.
|
||||
endsIn time.Duration
|
||||
path string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
"temporary ban", bans.CauseAdmin, time.Hour, "/",
|
||||
requestlog.ActionTemporarilyBanned,
|
||||
},
|
||||
{"permanent ban", bans.CauseAdmin, 0, "/", requestlog.ActionPermanentlyBanned},
|
||||
{
|
||||
"request that makes a ban permanent", bans.CauseAttack, time.Hour, "/",
|
||||
requestlog.ActionPermanentlyBanned,
|
||||
},
|
||||
{
|
||||
"trap path after an attack ban ended", bans.CauseAttack, -time.Hour,
|
||||
trapPath, requestlog.ActionPermanentlyBanned,
|
||||
},
|
||||
{
|
||||
"first trap path after another ban ended", bans.CauseAdmin, -time.Hour,
|
||||
trapPath, requestlog.ActionTemporarilyBanned,
|
||||
},
|
||||
} {
|
||||
for _, modeValue := range []string{"enforce", observe} {
|
||||
t.Run(tc.name+" in "+modeValue, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server := startWithClock(t, "", map[string]string{
|
||||
trapPaths: trapPathList, mode: modeValue,
|
||||
})
|
||||
|
||||
ban := bans.Ban{
|
||||
Netblock: netip.MustParsePrefix(client + "/32"),
|
||||
Start: clk.Now().Add(-2 * time.Hour),
|
||||
Cause: tc.cause,
|
||||
}
|
||||
if tc.endsIn != 0 {
|
||||
ban.Expires = clk.Now().Add(tc.endsIn)
|
||||
}
|
||||
|
||||
server.Ledger.Load([]bans.Ban{ban})
|
||||
|
||||
if modeValue == observe {
|
||||
line := s.request(client, tc.path, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, tc.want)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
s.request(client, tc.path, http.StatusForbidden, tc.want)
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestObserveModeNamesAPermanentBanWhoseAlertWouldBeSent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server, queue := startWithAlerts(t, map[string]string{
|
||||
trapPaths: trapPathList, mode: observe,
|
||||
})
|
||||
|
||||
server.Ledger.Load([]bans.Ban{{
|
||||
Netblock: netip.MustParsePrefix(client + "/32"),
|
||||
Start: clk.Now().Add(-2 * time.Hour),
|
||||
Expires: clk.Now().Add(-time.Hour),
|
||||
Cause: bans.CauseAttack,
|
||||
}})
|
||||
|
||||
line := s.request(client, "/wp-login.php", http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionPermanentlyBanned)
|
||||
|
||||
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
if len(waiting) != 1 || waiting[0].Event != alerts.EventPermanentBan {
|
||||
t.Errorf("alerts waiting are %+v, want one %s", waiting, alerts.EventPermanentBan)
|
||||
}
|
||||
}
|
||||
|
||||
func TestObserveModeNamesTheHeuristicThatWouldHaveRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
mode: observe, denyNets: client,
|
||||
})
|
||||
|
||||
line := s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, "deny_nets", `{"netblock": "`+client+`/32"}`)
|
||||
}
|
||||
|
||||
func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
denied = "192.0.2.50" // in SWWAF_DENY_NETS
|
||||
denied = "192.0.2.50" // in SWW_DENY_NETS
|
||||
banned = otherClient // under a ban read from bans.json
|
||||
)
|
||||
|
||||
for _, tc := range []struct {
|
||||
setting string // "" leaves SWWAF_MODE at its default
|
||||
setting string // "" leaves SWW_MODE at its default
|
||||
observe bool
|
||||
}{
|
||||
{"", false},
|
||||
@@ -38,7 +138,7 @@ func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
|
||||
lookupTimeout: "1h",
|
||||
rateLimitPerMinute: "1",
|
||||
denyNets: denied,
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
}
|
||||
|
||||
if tc.setting != "" {
|
||||
@@ -57,10 +157,10 @@ func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
for _, sent := range []struct{ from, refusal string }{
|
||||
{denied, requestlog.ActionDenied},
|
||||
{banned, requestlog.ActionBanned},
|
||||
{fromKP, requestlog.ActionCountryDenied},
|
||||
{fromDE, requestlog.ActionRateLimited},
|
||||
{denied, requestlog.ActionClientDenied},
|
||||
{banned, requestlog.ActionTemporarilyBanned},
|
||||
{fromKP, requestlog.ActionClientDenied},
|
||||
{fromDE, requestlog.ActionClientIsRateLimited},
|
||||
} {
|
||||
if !tc.observe {
|
||||
line := s.get(sent.from, http.StatusForbidden, sent.refusal)
|
||||
@@ -98,7 +198,7 @@ func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
|
||||
|
||||
for range 2 {
|
||||
line := s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionRateLimited)
|
||||
wantWouldAction(t, line, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
if line.LimitHit != minute || line.Offence != requestlog.OffenceLimit ||
|
||||
line.BanExpires != "" {
|
||||
@@ -111,7 +211,7 @@ func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
|
||||
// The ban read from bans.json refuses nothing, and so counts no
|
||||
// refusal in its notes, but is kept.
|
||||
line := s.get(otherClient, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionBanned)
|
||||
wantWouldAction(t, line, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
if line.BanExpires != requestlog.FormatTime(kept.Expires) {
|
||||
t.Errorf("log line has ban_expires %q, want %s", line.BanExpires,
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
package discriminator
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
@@ -15,12 +16,13 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
"sneak.berlin/go/smallwebwaf/internal/routewatch"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
)
|
||||
|
||||
// adminPrefix starts the path of each of discriminator's own endpoints. The app never
|
||||
@@ -32,15 +34,15 @@ const adminPrefix = "/_sww/"
|
||||
const HealthPath = "/_sww/healthz"
|
||||
|
||||
// MetricsPath is where the metrics are, for a request that carries
|
||||
// SWWAF_METRICS_TOKEN.
|
||||
// SWW_METRICS_TOKEN.
|
||||
const MetricsPath = "/_sww/metrics"
|
||||
|
||||
// BansPath is where an admin lists and adds bans, and, followed by / and
|
||||
// a client's address, lifts them, with SWWAF_ADMIN_TOKEN.
|
||||
// a client's address, lifts them, with SWW_ADMIN_TOKEN.
|
||||
const BansPath = "/_sww/bans"
|
||||
|
||||
// ClientsPath is where an admin asks what discriminator knows of a client, by
|
||||
// the client's address after it, with SWWAF_ADMIN_TOKEN.
|
||||
// the client's address after it, with SWW_ADMIN_TOKEN.
|
||||
const ClientsPath = "/_sww/clients/"
|
||||
|
||||
// Params are what New needs.
|
||||
@@ -53,14 +55,17 @@ type Params struct {
|
||||
// ProcessLog receives the process's own messages.
|
||||
ProcessLog *slog.Logger
|
||||
// GeoJSURL is where clients' AS numbers and countries are looked up
|
||||
// while SWWAF_LOOKUP_SOURCE is geojs, normally lookup.URL.
|
||||
// while SWW_LOOKUP_SOURCE is geojs, normally lookup.URL.
|
||||
GeoJSURL string
|
||||
// AbuseIPDBURL is where clients are checked with AbuseIPDB while
|
||||
// SWWAF_ABUSEIPDB_KEY is set, normally reputation.AbuseIPDBURL.
|
||||
// SWW_ABUSEIPDB_KEY is set, normally reputation.AbuseIPDBURL.
|
||||
AbuseIPDBURL string
|
||||
// LookupFile is the lookup database they are looked up in while
|
||||
// SWWAF_LOOKUP_SOURCE is file, and nil otherwise.
|
||||
// SWW_LOOKUP_SOURCE is file, and nil otherwise.
|
||||
LookupFile *lookup.File
|
||||
// CoreRuleSet is what NewCoreRuleSet returns: what inspects requests
|
||||
// while SWW_WAF_MODE is not off, and nil while it is.
|
||||
CoreRuleSet *coreruleset.CoreRuleSet
|
||||
// Now tells the time by which requests are counted for the rate
|
||||
// limits, bans are made and run out, and GeoJS's answers are kept,
|
||||
// normally time.Now in UTC, the time the state files give.
|
||||
@@ -72,16 +77,17 @@ type Params struct {
|
||||
// permanent, for each count over an anomaly threshold, for each request
|
||||
// whose client a blocklist, the CrowdSec decision list, a DNSBL zone or
|
||||
// AbuseIPDB lists, for each request the Core Rule Set scores at or over
|
||||
// SWWAF_WAF_ANOMALY_THRESHOLD, and for GeoJS failing, a fetch of a list
|
||||
// SWW_WAF_ANOMALY_THRESHOLD, and for GeoJS failing, a fetch of a list
|
||||
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
|
||||
// or the day's AbuseIPDB checks used up.
|
||||
Alerts *alerts.Queue
|
||||
}
|
||||
|
||||
// Server is the server discriminator runs, with the parts whose state the state
|
||||
// files keep, the lookup database, nil unless SWWAF_LOOKUP_SOURCE is file, the
|
||||
// files keep, the lookup database, nil unless SWW_LOOKUP_SOURCE is file, the
|
||||
// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts,
|
||||
// AbuseIPDB's scores and checks spent, and the metrics.
|
||||
// AbuseIPDB's scores and checks spent, and the metrics. RouteWatch, nil while
|
||||
// SWW_ROUTEWATCH_URL is unset, keeps its answers in memory only.
|
||||
type Server struct {
|
||||
*http.Server
|
||||
|
||||
@@ -93,6 +99,7 @@ type Server struct {
|
||||
Lists *reputation.Lists
|
||||
DNSBL *reputation.DNSBL
|
||||
AbuseIPDB *reputation.AbuseIPDB
|
||||
RouteWatch *routewatch.Client
|
||||
Metrics *metrics.Metrics
|
||||
}
|
||||
|
||||
@@ -122,7 +129,7 @@ func newHandler(params Params) *handler {
|
||||
BytesPerHour: params.Config.BytesLimitPerHour,
|
||||
BytesPerDay: params.Config.BytesLimitPerDay,
|
||||
}, params.Config.MaxTrackedClients),
|
||||
ledger: bans.New(bans.Rules{
|
||||
ledger: bans.New(bans.Params{
|
||||
LimitBanDuration: params.Config.LimitBanDuration,
|
||||
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
|
||||
MaxBanDuration: params.Config.MaxBanDuration,
|
||||
@@ -145,17 +152,24 @@ func newHandler(params Params) *handler {
|
||||
dnsbl: dnsbl,
|
||||
abuseIPDB: abuseIPDB,
|
||||
rules: params.Rules,
|
||||
coreRuleSet: newCoreRuleSet(params.Config),
|
||||
coreRuleSet: params.CoreRuleSet,
|
||||
alerts: params.Alerts,
|
||||
}
|
||||
if params.Config.RouteWatchURL != "" {
|
||||
h.routeWatch = routewatch.New(routewatch.Params{
|
||||
URL: params.Config.RouteWatchURL, Now: params.Now, ProcessLog: params.ProcessLog,
|
||||
})
|
||||
}
|
||||
|
||||
h.geojs = lookup.New(lookup.Params{
|
||||
URL: params.GeoJSURL,
|
||||
Timeout: params.Config.LookupTimeout,
|
||||
// The country lists, the headers and the biased thresholds act on
|
||||
// the answer before the request goes on.
|
||||
Wait: len(params.Config.DeniedCountries) > 0 ||
|
||||
len(params.Config.ExclusivelyAllowedCountries) > 0 ||
|
||||
params.Config.AddLookupHeaders || biasedThresholdsSet(params.Config),
|
||||
// The country list, the headers, the tag settings that name an asn=
|
||||
// or country= tag and the biased thresholds act on the answer before
|
||||
// the request goes on.
|
||||
Wait: len(params.Config.ExclusivelyAllowedCountries) > 0 ||
|
||||
params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() ||
|
||||
biasedThresholdsSet(params.Config),
|
||||
Answered: h.addLookup,
|
||||
Now: params.Now,
|
||||
ProcessLog: params.ProcessLog,
|
||||
@@ -171,7 +185,7 @@ func newHandler(params Params) *handler {
|
||||
// newReputation returns the lists fetched from URLs, the DNSBL zones'
|
||||
// verdicts and AbuseIPDB's scores, as the settings in params name them,
|
||||
// with none fetched, asked for or checked yet, and adds their metrics to
|
||||
// m, AbuseIPDB's while SWWAF_ABUSEIPDB_KEY is set.
|
||||
// m, AbuseIPDB's while SWW_ABUSEIPDB_KEY is set.
|
||||
func newReputation(
|
||||
params Params, m *metrics.Metrics,
|
||||
) (*reputation.Lists, *reputation.DNSBL, *reputation.AbuseIPDB) {
|
||||
@@ -179,6 +193,7 @@ func newReputation(
|
||||
lists := reputation.New(reputation.Params{
|
||||
BlocklistURLs: cfg.BlocklistURLs, Refresh: cfg.BlocklistRefresh,
|
||||
ASNLimitPercentURL: cfg.ASNLimitPercentURL,
|
||||
TagLists: cfg.TagLists,
|
||||
CrowdSecDecisionsURL: cfg.CrowdSecDecisionsURL, CrowdSecKey: cfg.CrowdSecKey,
|
||||
Now: params.Now, ProcessLog: params.ProcessLog, Alerts: params.Alerts,
|
||||
})
|
||||
@@ -203,29 +218,34 @@ func newReputation(
|
||||
return lists, dnsbl, abuseIPDB
|
||||
}
|
||||
|
||||
// newCoreRuleSet returns the Core Rule Set at SWWAF_WAF_PARANOIA_LEVEL,
|
||||
// without the rules SWWAF_WAF_DISABLED_RULES switches off, reading bodies
|
||||
// up to SWWAF_WAF_BODY_LIMIT, or nil while SWWAF_WAF_MODE is off. It reads at
|
||||
// most QuestionBodyMaxBytes less one byte: a question's body of
|
||||
// NewCoreRuleSet returns the Core Rule Set at SWW_WAF_PARANOIA_LEVEL,
|
||||
// without the rules SWW_WAF_DISABLED_RULES switches off, reading bodies
|
||||
// up to SWW_WAF_BODY_LIMIT, with the exclusions in
|
||||
// SWW_CORE_RULE_SET_EXCLUSIONS_FILE, or nil while SWW_WAF_MODE is off. It
|
||||
// reads at most QuestionBodyMaxBytes less one byte: a question's body of
|
||||
// QuestionBodyMaxBytes may be cut short, and the Core Rule Set does not read a
|
||||
// JSON or XML body over its limit.
|
||||
func newCoreRuleSet(cfg *config.Config) *waf.CoreRuleSet {
|
||||
if cfg.WAFMode == config.WAFModeOff {
|
||||
return nil
|
||||
// JSON or XML body over its limit. An exclusions file that coreruleset.New refuses is
|
||||
// an error in every mode. While the mode is off, a set file is loaded with the
|
||||
// Core Rule Set only to check it.
|
||||
func NewCoreRuleSet(cfg *config.Config) (*coreruleset.CoreRuleSet, error) {
|
||||
if cfg.WAFMode == config.WAFModeOff && cfg.CoreRuleSetExclusionsFile == "" {
|
||||
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
|
||||
}
|
||||
|
||||
coreRuleSet, err := waf.New(waf.Params{
|
||||
coreRuleSet, err := coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: cfg.WAFParanoiaLevel, DisabledRules: cfg.WAFDisabledRules,
|
||||
BodyLimit: min(cfg.WAFBodyLimit, QuestionBodyMaxBytes-1),
|
||||
BodyLimit: min(cfg.WAFBodyLimit, QuestionBodyMaxBytes-1),
|
||||
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
|
||||
})
|
||||
if err != nil {
|
||||
// The Core Rule Set is built in, and the settings cannot break it:
|
||||
// the paranoia level is from 1 to 4, and the id of no rule switches
|
||||
// nothing off.
|
||||
panic(err)
|
||||
return nil, fmt.Errorf("SWW_CORE_RULE_SET_EXCLUSIONS_FILE: %w", err)
|
||||
}
|
||||
|
||||
return coreRuleSet
|
||||
if cfg.WAFMode == config.WAFModeOff {
|
||||
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
|
||||
}
|
||||
|
||||
return coreRuleSet, nil
|
||||
}
|
||||
|
||||
// handler is discriminator's handler. It holds what every request shares; what
|
||||
@@ -244,8 +264,9 @@ type handler struct {
|
||||
lists *reputation.Lists
|
||||
dnsbl *reputation.DNSBL
|
||||
abuseIPDB *reputation.AbuseIPDB
|
||||
routeWatch *routewatch.Client
|
||||
rules *rules.Files
|
||||
coreRuleSet *waf.CoreRuleSet
|
||||
coreRuleSet *coreruleset.CoreRuleSet
|
||||
alerts *alerts.Queue
|
||||
// awaitingReport holds the decision on each answered question by its id
|
||||
// until its report comes, and nil for a question still being handled.
|
||||
|
||||
@@ -55,33 +55,35 @@ const (
|
||||
|
||||
// The settings the tests set.
|
||||
const (
|
||||
mode = "SWWAF_MODE"
|
||||
mode = "SWW_MODE"
|
||||
trustedProxies = "SWRP_TRUSTED_PROXIES"
|
||||
ipv6GroupPrefix = "SWWAF_IPV6_GROUP_PREFIX"
|
||||
maxTrackedClients = "SWWAF_MAX_TRACKED_CLIENTS"
|
||||
allowNets = "SWWAF_ALLOW_NETS"
|
||||
rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS"
|
||||
denyNets = "SWWAF_DENY_NETS"
|
||||
rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE"
|
||||
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
|
||||
rateLimitExemptPaths = "SWWAF_RATE_LIMIT_EXEMPT_PATHS"
|
||||
lookupSource = "SWWAF_LOOKUP_SOURCE"
|
||||
lookupDBPath = "SWWAF_LOOKUP_DB_PATH"
|
||||
lookupTimeout = "SWWAF_LOOKUP_TIMEOUT"
|
||||
addLookupHeaders = "SWWAF_ADD_LOOKUP_HEADERS"
|
||||
deniedCountries = "SWWAF_DENIED_COUNTRIES"
|
||||
allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
|
||||
banResponse = "SWWAF_BAN_RESPONSE"
|
||||
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
|
||||
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
|
||||
maxBanDuration = "SWWAF_MAX_BAN_DURATION"
|
||||
maxBans = "SWWAF_MAX_BANS"
|
||||
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
|
||||
instanceName = "SWWAF_INSTANCE_NAME"
|
||||
logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS"
|
||||
attackBanDuration = "SWWAF_ATTACK_BAN_DURATION"
|
||||
rulesDir = "SWWAF_RULES_DIR"
|
||||
wafMode = "SWWAF_WAF_MODE"
|
||||
ipv6GroupPrefix = "SWW_IPV6_GROUP_PREFIX"
|
||||
maxTrackedClients = "SWW_MAX_TRACKED_CLIENTS"
|
||||
allowNets = "SWW_ALLOW_NETS"
|
||||
rateLimitExemptNets = "SWW_RATE_LIMIT_EXEMPT_NETS"
|
||||
denyNets = "SWW_DENY_NETS"
|
||||
rateLimitPerMinute = "SWW_RATE_LIMIT_PER_MINUTE"
|
||||
rateLimitPerDay = "SWW_RATE_LIMIT_PER_DAY"
|
||||
rateLimitExemptPaths = "SWW_RATE_LIMIT_EXEMPT_PATHS"
|
||||
lookupSource = "SWW_LOOKUP_SOURCE"
|
||||
lookupDBPath = "SWW_LOOKUP_DB_PATH"
|
||||
lookupTimeout = "SWW_LOOKUP_TIMEOUT"
|
||||
addLookupHeaders = "SWW_ADD_LOOKUP_HEADERS"
|
||||
tagDeny = "SWW_TAG_DENY"
|
||||
// denyKP is the value of SWW_TAG_DENY that refuses fromKP's country.
|
||||
denyKP = "country=KP"
|
||||
allowedCountries = "SWW_EXCLUSIVELY_ALLOWED_COUNTRIES"
|
||||
banResponse = "SWW_BAN_RESPONSE"
|
||||
limitBanDuration = "SWW_LIMIT_BAN_DURATION"
|
||||
limitBanRepeatWindow = "SWW_LIMIT_BAN_REPEAT_WINDOW"
|
||||
maxBanDuration = "SWW_MAX_BAN_DURATION"
|
||||
maxBans = "SWW_MAX_BANS"
|
||||
banScopeV4Prefix = "SWW_BAN_SCOPE_V4_PREFIX"
|
||||
instanceName = "SWW_INSTANCE_NAME"
|
||||
logRequestHeaders = "SWW_LOG_REQUEST_HEADERS"
|
||||
attackBanDuration = "SWW_ATTACK_BAN_DURATION"
|
||||
rulesDir = "SWW_RULES_DIR"
|
||||
wafMode = "SWW_WAF_MODE"
|
||||
)
|
||||
|
||||
// off is the value that switches a setting off.
|
||||
@@ -294,8 +296,8 @@ func startProxyWithGeoJS(
|
||||
|
||||
// startProxyWithClock is startProxyWithGeoJS with requests counted and
|
||||
// bans made by the time now tells, and returns discriminator's server as well, whose
|
||||
// Addr is where discriminator listens. Unless env sets SWWAF_RULES_DIR, it is an empty
|
||||
// directory, of no rules, and unless it sets SWWAF_INSTANCE_NAME, that is app,
|
||||
// Addr is where discriminator listens. Unless env sets SWW_RULES_DIR, it is an empty
|
||||
// directory, of no rules, and unless it sets SWW_INSTANCE_NAME, that is app,
|
||||
// the label instance of every metric.
|
||||
func startProxyWithClock(
|
||||
t *testing.T, appURL, geojsURL string, now func() time.Time,
|
||||
@@ -391,13 +393,13 @@ func startRProxy(
|
||||
}
|
||||
|
||||
// newDiscriminator makes discriminator's server, without starting it, and
|
||||
// returns it, what
|
||||
// it writes, and the queue of the alerts it raises, as the settings in env make
|
||||
// it. No alert is sent from the queue: they wait in it, for the test to look at.
|
||||
// returns it, what it writes, and the queue of the alerts it raises, as the
|
||||
// settings in env make it. No alert is sent from the queue: they wait in it,
|
||||
// for the test to look at.
|
||||
// With no geojsURL, there is no stand-in for GeoJS to look clients up at, and
|
||||
// SWWAF_LOOKUP_SOURCE is off unless env sets it. While it is file, the lookup
|
||||
// database SWWAF_LOOKUP_DB_PATH names is read. Clients are checked with AbuseIPDB
|
||||
// at abuseIPDBURL while env sets SWWAF_ABUSEIPDB_KEY. SWWAF_WAF_MODE is off unless
|
||||
// SWW_LOOKUP_SOURCE is off unless env sets it. While it is file, the lookup
|
||||
// database SWW_LOOKUP_DB_PATH names is read. Clients are checked with AbuseIPDB
|
||||
// at abuseIPDBURL while env sets SWW_ABUSEIPDB_KEY. SWW_WAF_MODE is off unless
|
||||
// env sets it, so that only the tests of the Core Rule Set have their requests
|
||||
// inspected by it.
|
||||
func newDiscriminator(
|
||||
@@ -464,6 +466,11 @@ func newDiscriminatorWith(
|
||||
}
|
||||
}
|
||||
|
||||
coreRuleSet, err := discriminator.NewCoreRuleSet(cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("Core Rule Set: %v", err)
|
||||
}
|
||||
|
||||
server := build(discriminator.Params{
|
||||
Config: cfg,
|
||||
RequestLog: out,
|
||||
@@ -471,6 +478,7 @@ func newDiscriminatorWith(
|
||||
GeoJSURL: geojsURL,
|
||||
AbuseIPDBURL: abuseIPDBURL,
|
||||
LookupFile: lookupFile,
|
||||
CoreRuleSet: coreRuleSet,
|
||||
Now: now,
|
||||
Rules: ruleFiles,
|
||||
Alerts: alertQueue,
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic rate_limits. It reads SWW_RATE_LIMIT_EXEMPT_NETS,
|
||||
// SWW_RATE_LIMIT_EXEMPT_PATHS and SWW_BAN_RESPONSE, and the rate limits and
|
||||
// biased thresholds through the limiter and limitPercentages.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// rateLimits counts the request for the rate limits, unless
|
||||
// countedByRateLimits exempts it, by the client's limit percentages, which it
|
||||
// notes for the log line with the client's counts. A request that takes the
|
||||
// client over a rate limit, as its limit percentage lowers it, breaks it: it is
|
||||
// refused, and bans the client. A request exempt from the rate limits is exempt
|
||||
// from the byte limits too. Its figures are the window, the count and the
|
||||
// limit.
|
||||
type rateLimits struct{}
|
||||
|
||||
func (rateLimits) name() string { return "rate_limits" }
|
||||
|
||||
func (rateLimits) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
rq.counted = rq.countedByRateLimits()
|
||||
if !rq.counted {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.limitPercent, rq.bytesPercent = rq.limitPercentages()
|
||||
rq.line.RateLimitPercent, rq.line.RateLimitPercentSetting = rq.limitPercent.logged()
|
||||
rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting = rq.bytesPercent.logged()
|
||||
|
||||
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now,
|
||||
rq.limitPercent.percent)
|
||||
rq.line.Counts = counts
|
||||
|
||||
if !over {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.banForLimit(now, hit, rq.h.config.BanResponse)
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionClientIsRateLimited, figures: limitFigures(hit),
|
||||
}
|
||||
}
|
||||
|
||||
// overAWholeRateLimit reports whether the heuristic rate_limits will refuse the
|
||||
// request at now, whatever its client's limit percentages: the rate limits
|
||||
// count it, and one more request from its client takes it over a whole rate
|
||||
// limit. It counts nothing.
|
||||
func (rq *request) overAWholeRateLimit(now time.Time) bool {
|
||||
return rq.countedByRateLimits() &&
|
||||
rq.h.limiter.OverARateLimit(rq.h.clientGroup(rq.client), now)
|
||||
}
|
||||
|
||||
// countedByRateLimits reports whether the rate limits count the request: its
|
||||
// client is not in SWW_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt
|
||||
// under SWW_RATE_LIMIT_EXEMPT_PATHS.
|
||||
func (rq *request) countedByRateLimits() bool {
|
||||
cfg := rq.h.config
|
||||
|
||||
return !isInside(rq.client, cfg.RateLimitExemptNets) &&
|
||||
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
|
||||
}
|
||||
@@ -10,7 +10,7 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// minute is the window of SWWAF_RATE_LIMIT_PER_MINUTE, as a log line's
|
||||
// minute is the window of SWW_RATE_LIMIT_PER_MINUTE, as a log line's
|
||||
// limit_hit names it.
|
||||
const minute = "minute"
|
||||
|
||||
@@ -59,7 +59,7 @@ func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
|
||||
if sent.want == http.StatusOK {
|
||||
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||
} else {
|
||||
wantLine(t, line, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
wantLine(t, line, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
if line.LimitHit != minute {
|
||||
t.Errorf("log line has limit_hit %q, want minute", line.LimitHit)
|
||||
@@ -72,10 +72,21 @@ func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimitRefusalNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{rateLimitPerMinute: "1"})
|
||||
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
wantHeuristic(t, line, "rate_limits", `{"window": "minute", "count": 2, "limit": 1}`)
|
||||
}
|
||||
|
||||
func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// With SWWAF_IPV6_GROUP_PREFIX at 48, the first two addresses, in two
|
||||
// With SWW_IPV6_GROUP_PREFIX at 48, the first two addresses, in two
|
||||
// /64s of one /48, are one client, and the second's request breaks the
|
||||
// limit; the third, in the next /48, is another client.
|
||||
const (
|
||||
@@ -91,7 +102,7 @@ func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) {
|
||||
status int
|
||||
action string
|
||||
}{
|
||||
{rateLimitPerMinute, "1", http.StatusForbidden, requestlog.ActionRateLimited},
|
||||
{rateLimitPerMinute, "1", http.StatusForbidden, requestlog.ActionClientIsRateLimited},
|
||||
{bytesLimitPerMinute, byteLimit, http.StatusOK, requestlog.ActionForward},
|
||||
} {
|
||||
t.Run(tc.setting, func(t *testing.T) {
|
||||
@@ -118,14 +129,14 @@ func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) {
|
||||
func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const denied = "192.0.2.50" // in SWWAF_DENY_NETS
|
||||
const denied = "192.0.2.50" // in SWW_DENY_NETS
|
||||
|
||||
geojsURL, _ := startGeoJS(t)
|
||||
s, _, server := startWithClock(t, geojsURL, map[string]string{
|
||||
rateLimitPerMinute: "1",
|
||||
rateLimitExemptPaths: "/assets/,/favicon.ico",
|
||||
denyNets: denied,
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
|
||||
// The answers are kept before the requests, so that none waits for
|
||||
@@ -149,14 +160,17 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
|
||||
|
||||
// A path outside every prefix is counted: /assets is not under
|
||||
// /assets/, and breaks the limit.
|
||||
s.request(client, "/assets", http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.request(client, "/assets", http.StatusForbidden,
|
||||
requestlog.ActionClientIsRateLimited)
|
||||
|
||||
// A ban, SWWAF_DENY_NETS and the country lists still refuse a path
|
||||
// under a prefix.
|
||||
s.request(client, "/assets/app.js", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(denied, "/assets/app.js", http.StatusForbidden, requestlog.ActionDenied)
|
||||
// A ban, SWW_DENY_NETS and SWW_TAG_DENY still refuse a path under a
|
||||
// prefix.
|
||||
s.request(client, "/assets/app.js", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
s.request(denied, "/assets/app.js", http.StatusForbidden,
|
||||
requestlog.ActionClientDenied)
|
||||
s.request(fromKP, "/assets/app.js",
|
||||
http.StatusForbidden, requestlog.ActionCountryDenied)
|
||||
http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
}
|
||||
|
||||
func TestRateLimitCountsPathsThatAreNotExempt(t *testing.T) {
|
||||
@@ -200,7 +214,7 @@ func TestRateLimitCountsPathsThatAreNotExempt(t *testing.T) {
|
||||
// Counted, the second request breaks the limit of one request
|
||||
// a minute.
|
||||
s.request(client, sent, http.StatusOK, requestlog.ActionForward)
|
||||
s.request(client, sent, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.request(client, sent, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic refusal_limit. It reads SWW_ERROR_BURST_THRESHOLD and
|
||||
// SWW_BAN_RESPONSE.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// refusalLimit counts the request once it has been answered, if discriminator
|
||||
// refused it after a rule file match, a trap path or a Core Rule Set match, and
|
||||
// in observe mode if enforce mode would have: more than
|
||||
// SWW_ERROR_BURST_THRESHOLD such refusals of the client within a minute break
|
||||
// a limit, which bans the client. A client in SWW_ALLOW_NETS, which the
|
||||
// heuristics skip, is not counted, and nothing is while the threshold is off.
|
||||
// Its figures are the window, the count and the limit.
|
||||
type refusalLimit struct{}
|
||||
|
||||
func (refusalLimit) name() string { return "refusal_limit" }
|
||||
|
||||
func (refusalLimit) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
cfg := rq.h.config
|
||||
if cfg.ErrorBurstThreshold == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
if !rq.attack && !rq.blockedByBlockRule && !rq.blockedByCoreRuleSet {
|
||||
return nil
|
||||
}
|
||||
|
||||
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
|
||||
cfg.ErrorBurstThreshold)
|
||||
if !over {
|
||||
return nil
|
||||
}
|
||||
|
||||
// What the client was sent, or in observe mode would have been.
|
||||
status := rq.out.status
|
||||
|
||||
switch rq.line.WouldAction {
|
||||
case requestlog.ActionBlockedByBlockRule, requestlog.ActionBlockedByCoreRuleSet:
|
||||
status = http.StatusForbidden
|
||||
case requestlog.ActionTemporarilyBanned, requestlog.ActionPermanentlyBanned:
|
||||
status = cfg.BanResponse
|
||||
}
|
||||
|
||||
rq.banForLimit(now, hit, status)
|
||||
|
||||
return &verdict{figures: limitFigures(hit)}
|
||||
}
|
||||
+55
-23
@@ -15,7 +15,7 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
const errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD"
|
||||
const errorBurstThreshold = "SWW_ERROR_BURST_THRESHOLD"
|
||||
|
||||
// refused is a request the tests here send, which discriminator refuses after a rule
|
||||
// file match.
|
||||
@@ -37,9 +37,10 @@ func (r refused) send(s *sender, from string) logLine {
|
||||
switch r {
|
||||
case blockRule:
|
||||
return s.request(from, blockedPath, http.StatusForbidden,
|
||||
requestlog.ActionRuleBlocked)
|
||||
requestlog.ActionBlockedByBlockRule)
|
||||
case banRule:
|
||||
return s.request(from, probePath, http.StatusForbidden, requestlog.ActionBanned)
|
||||
return s.request(from, probePath, http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
s.t.Fatalf("no request for the refusal %d", r)
|
||||
@@ -48,7 +49,7 @@ func (r refused) send(s *sender, from string) logLine {
|
||||
}
|
||||
|
||||
// startForErrorBurst is startWithClock with testRules, the metrics token and
|
||||
// SWWAF_ERROR_BURST_THRESHOLD at threshold, and the settings in env.
|
||||
// SWW_ERROR_BURST_THRESHOLD at threshold, and the settings in env.
|
||||
func startForErrorBurst(
|
||||
t *testing.T, threshold string, env map[string]string,
|
||||
) (*sender, *clock, *discriminator.Server) {
|
||||
@@ -71,9 +72,20 @@ func TestErrorBurstBreaksAtOneOverTheThreshold(t *testing.T) {
|
||||
name string
|
||||
// refusals are four, one over the threshold of three.
|
||||
refusals []refused
|
||||
// nextAction is the action of the client's next request: the next
|
||||
// request after a ban for a ban rule makes it permanent.
|
||||
nextAction string
|
||||
}{
|
||||
{"block rule", []refused{blockRule, blockRule, blockRule, blockRule}},
|
||||
{"a ban rule last", []refused{blockRule, blockRule, blockRule, banRule}},
|
||||
{
|
||||
"block rule",
|
||||
[]refused{blockRule, blockRule, blockRule, blockRule},
|
||||
requestlog.ActionTemporarilyBanned,
|
||||
},
|
||||
{
|
||||
"a ban rule last",
|
||||
[]refused{blockRule, blockRule, blockRule, banRule},
|
||||
requestlog.ActionPermanentlyBanned,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -95,17 +107,36 @@ func TestErrorBurstBreaksAtOneOverTheThreshold(t *testing.T) {
|
||||
// The fourth is answered as the others were, breaks the error
|
||||
// burst, and bans the client.
|
||||
line := tc.refusals[3].send(s, client)
|
||||
if line.LimitHit != requestlog.LimitHitErrorBurst ||
|
||||
if line.LimitHit != requestlog.LimitHitMinuteRefusals ||
|
||||
line.Offence != requestlog.OffenceLimit {
|
||||
t.Errorf("log line has limit_hit %q and offence %q, want error_burst "+
|
||||
t.Errorf("log line has limit_hit %q and offence %q, want minute_refusals "+
|
||||
"and limit", line.LimitHit, line.Offence)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, tc.nextAction)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefusalLimitBanNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startForErrorBurst(t, "1", nil)
|
||||
|
||||
const ruleFigures = `{"rule_id": "blocked", "rule_action": "block"}`
|
||||
|
||||
line := blockRule.send(s, client)
|
||||
wantHeuristic(t, line, "rule_files", ruleFigures)
|
||||
wantAfterAnswerHeuristic(t, line, "", "null")
|
||||
|
||||
// The second refusal is the request refusal_limit bans for, and the line
|
||||
// names both.
|
||||
line = blockRule.send(s, client)
|
||||
wantHeuristic(t, line, "rule_files", ruleFigures)
|
||||
wantAfterAnswerHeuristic(t, line, "refusal_limit",
|
||||
`{"window": "minute", "count": 2, "limit": 1}`)
|
||||
}
|
||||
|
||||
func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -150,7 +181,7 @@ func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) {
|
||||
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
|
||||
}
|
||||
|
||||
wantOffences := ratelimit.Offences{Limit: 1, RuleBlocked: 3}
|
||||
wantOffences := ratelimit.Offences{Limit: 1, BlockedByBlockRule: 3}
|
||||
if offences := historyOf(t, server, client).Offences; offences != wantOffences {
|
||||
t.Errorf("history counts the offences %+v, want %+v", offences, wantOffences)
|
||||
}
|
||||
@@ -160,7 +191,7 @@ func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) {
|
||||
`kind="refusals",window="minute"}`, 1)
|
||||
wantMetric(t, metrics, `sww_offences_total{instance="app",kind="limit"}`, 1)
|
||||
wantMetric(t, metrics,
|
||||
`sww_offences_total{instance="app",kind="rule_blocked"}`, 3)
|
||||
`sww_offences_total{instance="app",kind="blocked_by_block_rule"}`, 3)
|
||||
wantMetric(t, metrics, `sww_bans_made_total{cause="limit",instance="app"}`, 1)
|
||||
}
|
||||
|
||||
@@ -172,7 +203,7 @@ func TestErrorBurstIsNotLoweredForAClientWithLowerLimits(t *testing.T) {
|
||||
lookupTimeout: "1h",
|
||||
errorBurstThreshold: "2",
|
||||
rulesDir: writeRules(t, testRules),
|
||||
countryLimitPercent: countryDEHalf,
|
||||
tagLimitPercent: countryTagDEHalf,
|
||||
})
|
||||
|
||||
// Half of the threshold would be one, which the second refusal is over.
|
||||
@@ -246,7 +277,8 @@ func TestErrorBurstOffOrAtItsDefault(t *testing.T) {
|
||||
line = blockRule.send(s, client)
|
||||
}
|
||||
|
||||
if broken := line.LimitHit == requestlog.LimitHitErrorBurst; broken != tc.broken {
|
||||
broken := line.LimitHit == requestlog.LimitHitMinuteRefusals
|
||||
if broken != tc.broken {
|
||||
t.Errorf("the 31st refusal broke the error burst: %t, want %t",
|
||||
broken, tc.broken)
|
||||
}
|
||||
@@ -258,15 +290,15 @@ func TestErrorBurstCountsEachClientTheChecksApplyTo(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
|
||||
exempt = "192.0.2.50" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
|
||||
allowed = "192.0.2.60" // in SWW_ALLOW_NETS
|
||||
exempt = "192.0.2.50" // in SWW_RATE_LIMIT_EXEMPT_NETS
|
||||
)
|
||||
|
||||
s, _, _ := startForErrorBurst(t, "1", map[string]string{
|
||||
allowNets: allowed, rateLimitExemptNets: exempt,
|
||||
})
|
||||
|
||||
// A client in SWWAF_ALLOW_NETS is not refused, and so not counted.
|
||||
// A client in SWW_ALLOW_NETS is not refused, and so not counted.
|
||||
for range 3 {
|
||||
s.request(allowed, blockedPath, http.StatusOK, requestlog.ActionForward)
|
||||
}
|
||||
@@ -275,11 +307,11 @@ func TestErrorBurstCountsEachClientTheChecksApplyTo(t *testing.T) {
|
||||
blockRule.send(s, exempt)
|
||||
|
||||
line := blockRule.send(s, exempt)
|
||||
if line.LimitHit != requestlog.LimitHitErrorBurst {
|
||||
t.Errorf("log line has limit_hit %q, want error_burst", line.LimitHit)
|
||||
if line.LimitHit != requestlog.LimitHitMinuteRefusals {
|
||||
t.Errorf("log line has limit_hit %q, want minute_refusals", line.LimitHit)
|
||||
}
|
||||
|
||||
s.get(exempt, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(exempt, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestErrorBurstBanSetsTheRefusalsBackToZero(t *testing.T) {
|
||||
@@ -322,7 +354,7 @@ func TestObserveModeLogsAndAlertsTheErrorBurst(t *testing.T) {
|
||||
for range 2 {
|
||||
line := s.request(otherClient, blockedPath, http.StatusOK,
|
||||
requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionBanned)
|
||||
wantWouldAction(t, line, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
if line.LimitHit != "" {
|
||||
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
|
||||
@@ -334,10 +366,10 @@ func TestObserveModeLogsAndAlertsTheErrorBurst(t *testing.T) {
|
||||
s.request(client, blockedPath, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line := s.request(client, blockedPath, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionRuleBlocked)
|
||||
wantWouldAction(t, line, requestlog.ActionBlockedByBlockRule)
|
||||
|
||||
if line.LimitHit != requestlog.LimitHitErrorBurst || line.BanExpires != "" {
|
||||
t.Errorf("log line has limit_hit %q and ban_expires %q, want error_burst "+
|
||||
if line.LimitHit != requestlog.LimitHitMinuteRefusals || line.BanExpires != "" {
|
||||
t.Errorf("log line has limit_hit %q and ban_expires %q, want minute_refusals "+
|
||||
"and none", line.LimitHit, line.BanExpires)
|
||||
}
|
||||
|
||||
@@ -1,91 +1,17 @@
|
||||
package discriminator
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
// What the reputation heuristics share: blocklists, crowdsec, dnsbl and
|
||||
// abuseipdb.
|
||||
|
||||
import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
)
|
||||
|
||||
// deny is the SWWAF_BLOCKLIST_ACTION and the SWWAF_REPUTATION_ACTION that
|
||||
// deny is the SWW_BLOCKLIST_ACTION and the SWW_REPUTATION_ACTION that
|
||||
// refuses the requests of a client a source lists.
|
||||
const deny = "deny"
|
||||
|
||||
// blocklistDenied notes the blocklists that list the client, as
|
||||
// noteListed does, and reports whether SWWAF_BLOCKLIST_ACTION, being deny,
|
||||
// refuses the request. Being limit, it lowers the client's limits instead
|
||||
// (see limitPercentages), and being log, it does nothing more.
|
||||
func (rq *request) blocklistDenied() bool {
|
||||
listedBy := rq.h.lists.ListedBy(rq.client)
|
||||
rq.blocklisted = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a blocklist")
|
||||
|
||||
return rq.blocklisted && rq.h.config.BlocklistAction == deny
|
||||
}
|
||||
|
||||
// crowdSecBanned reports whether a decision of the CrowdSec decision list
|
||||
// on the client is in force at now. If one is, it notes the list, as
|
||||
// noteHit does, and bans the client until that decision ends.
|
||||
func (rq *request) crowdSecBanned(now time.Time) bool {
|
||||
decision, listed := rq.h.lists.CrowdSecDecision(rq.client, now)
|
||||
if !listed {
|
||||
return false
|
||||
}
|
||||
|
||||
rq.noteHit(bans.ReputationHit{Source: rq.h.config.CrowdSecDecisionsURL},
|
||||
"listed by the CrowdSec decision list")
|
||||
rq.banForCrowdSec(now, decision)
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// dnsblDenied notes the DNSBL zones whose verdict lists the client, as
|
||||
// noteListed does, and reports whether SWWAF_REPUTATION_ACTION, being
|
||||
// deny, refuses the request. Being limit, it lowers the client's limits
|
||||
// instead (see limitPercentages), and being log, it does nothing more. A
|
||||
// zone without a verdict on the client is asked about it in the
|
||||
// background, and the request does not wait for the answer. ctx is the
|
||||
// request's own context.
|
||||
func (rq *request) dnsblDenied(ctx context.Context) bool {
|
||||
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
|
||||
rq.dnsblListed = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a DNSBL zone")
|
||||
|
||||
return rq.dnsblListed && rq.h.config.ReputationAction == deny
|
||||
}
|
||||
|
||||
// abuseIPDBDenied notes AbuseIPDB, as noteHit does, with the score, when
|
||||
// its score of the client is a hit, and reports whether
|
||||
// SWWAF_REPUTATION_ACTION, being deny, refuses the request, as dnsblDenied
|
||||
// does for a zone. While SWWAF_ABUSEIPDB_KEY is unset it does nothing. A
|
||||
// client without a score is checked in the background, by the request's
|
||||
// address, if its history counts an offence, and the request does not
|
||||
// wait for the answer. The score is then used for each address of the
|
||||
// client. ctx is the request's own context.
|
||||
func (rq *request) abuseIPDBDenied(ctx context.Context) bool {
|
||||
if rq.h.config.AbuseIPDBKey == "" {
|
||||
return false
|
||||
}
|
||||
|
||||
client := rq.h.clientGroup(rq.client)
|
||||
held, _ := rq.h.limiter.Client(client)
|
||||
offender := held.History.Offences != ratelimit.Offences{}
|
||||
|
||||
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
|
||||
if !hit {
|
||||
return false
|
||||
}
|
||||
|
||||
rq.abuseIPDBHit = true
|
||||
rq.noteHit(bans.ReputationHit{Source: reputation.AbuseIPDBSource, Score: &score},
|
||||
"scored by AbuseIPDB at or over SWWAF_ABUSEIPDB_MIN_SCORE")
|
||||
|
||||
return rq.h.config.ReputationAction == deny
|
||||
}
|
||||
|
||||
// noteListed notes each of sources, the URLs of the blocklists or the
|
||||
// DNSBL zones, their keys masked, that list the client, as noteHit does,
|
||||
// with reason.
|
||||
|
||||
@@ -23,12 +23,12 @@ import (
|
||||
|
||||
// The reputation settings.
|
||||
const (
|
||||
blocklistURLs = "SWWAF_BLOCKLIST_URLS"
|
||||
blocklistAction = "SWWAF_BLOCKLIST_ACTION"
|
||||
asnLimitPercentURL = "SWWAF_ASN_LIMIT_PERCENT_URL"
|
||||
blocklistURLs = "SWW_BLOCKLIST_URLS"
|
||||
blocklistAction = "SWW_BLOCKLIST_ACTION"
|
||||
asnLimitPercentURL = "SWW_ASN_LIMIT_PERCENT_URL"
|
||||
)
|
||||
|
||||
// The actions of SWWAF_BLOCKLIST_ACTION and SWWAF_REPUTATION_ACTION:
|
||||
// The actions of SWW_BLOCKLIST_ACTION and SWW_REPUTATION_ACTION:
|
||||
// limitHalf gives a listed client half of every limit, and limitQuarter a
|
||||
// quarter.
|
||||
const (
|
||||
@@ -49,12 +49,12 @@ const (
|
||||
func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
forward, denied := requestlog.ActionForward, requestlog.ActionDenied
|
||||
forward, denied := requestlog.ActionForward, requestlog.ActionClientDenied
|
||||
|
||||
for _, tc := range []struct {
|
||||
action string
|
||||
// statuses and actions are those of a listed client's three
|
||||
// requests, and percent their limit_percent, as percentText gives it.
|
||||
// requests, and percent their rate_limit_percent, as percentText gives it.
|
||||
statuses []int
|
||||
actions []string
|
||||
percent string
|
||||
@@ -66,7 +66,7 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
|
||||
{
|
||||
// Half of 4 requests a minute: the third breaks the limit.
|
||||
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||
[]string{forward, forward, requestlog.ActionRateLimited},
|
||||
[]string{forward, forward, requestlog.ActionClientIsRateLimited},
|
||||
"50 from " + blocklistAction,
|
||||
},
|
||||
{
|
||||
@@ -90,8 +90,8 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
|
||||
for i := range 3 {
|
||||
line := s.get(from, tc.statuses[i], tc.actions[i])
|
||||
wantReputation(t, line, dropURL)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent,
|
||||
line.LimitPercentSetting, tc.percent)
|
||||
wantPercent(t, "rate_limit_percent", line.RateLimitPercent,
|
||||
line.RateLimitPercentSetting, tc.percent)
|
||||
|
||||
// A request refused for the list is not counted.
|
||||
counted := line.fields["counts"] != nil
|
||||
@@ -106,7 +106,8 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
|
||||
for range 3 {
|
||||
line := s.get(unplaced, http.StatusOK, forward)
|
||||
wantReputation(t, line)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
none)
|
||||
}
|
||||
|
||||
@@ -118,16 +119,66 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlocklistsComeAfterTheCountryListsAndSkipAllowNets(t *testing.T) {
|
||||
func TestReputationRefusalNamesItsHeuristicAndWhatListedTheClient(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
heuristic string
|
||||
env map[string]string
|
||||
load func(*testing.T, *discriminator.Server)
|
||||
figures string
|
||||
}{
|
||||
{
|
||||
"blocklists",
|
||||
map[string]string{blocklistURLs: dropURL, blocklistAction: actionDeny},
|
||||
func(t *testing.T, server *discriminator.Server) {
|
||||
t.Helper()
|
||||
loadLists(t, server, map[string][]string{dropURL: {client}})
|
||||
},
|
||||
`{"lists": ["` + dropURL + `"]}`,
|
||||
},
|
||||
{
|
||||
"dnsbl",
|
||||
map[string]string{
|
||||
dnsblZones: dnsblZone, dnsblResolver: noResolver,
|
||||
reputationAction: actionDeny,
|
||||
},
|
||||
func(_ *testing.T, server *discriminator.Server) {
|
||||
loadVerdicts(server, map[string][]string{client: {dnsblZone}})
|
||||
},
|
||||
`{"zones": ["` + dnsblZone + `"]}`,
|
||||
},
|
||||
{
|
||||
"abuseipdb",
|
||||
map[string]string{abuseIPDBKey: accountKey, reputationAction: actionDeny},
|
||||
func(_ *testing.T, server *discriminator.Server) {
|
||||
loadScores(server, map[string]int64{client: 80})
|
||||
},
|
||||
`{"score": 80, "min_score": 75}`,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.heuristic, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, tc.env)
|
||||
tc.load(t, server)
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, tc.heuristic, tc.figures)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlocklistsComeAfterAllowedCountriesAndSkipAllowNets(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, queue := startWithLookups(t, map[string]string{
|
||||
blocklistURLs: dropURL, deniedCountries: "kp", allowNets: fromDE,
|
||||
blocklistURLs: dropURL, allowedCountries: "de", allowNets: fromDE,
|
||||
})
|
||||
loadLists(t, server, map[string][]string{dropURL: {fromDE, fromKP}})
|
||||
|
||||
// fromKP's country refuses it before the list is looked at, and fromDE,
|
||||
// in SWWAF_ALLOW_NETS, is not checked at all: neither is noted, nor
|
||||
// in SWW_ALLOW_NETS, is not checked at all: neither is noted, nor
|
||||
// alerted.
|
||||
wantReputation(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionCountryDenied))
|
||||
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward))
|
||||
@@ -143,7 +194,7 @@ func TestObserveModeForwardsAClientABlocklistDeniesAndAlertsIt(t *testing.T) {
|
||||
loadLists(t, server, map[string][]string{dropURL: {fromDE}})
|
||||
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionDenied)
|
||||
wantWouldAction(t, line, requestlog.ActionClientDenied)
|
||||
wantReputation(t, line, dropURL)
|
||||
|
||||
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
@@ -157,31 +208,33 @@ func TestBlocklistLimitTakesPartInTheLowestPercentageOfEveryLimit(t *testing.T)
|
||||
|
||||
for _, tc := range []struct {
|
||||
action, asnPercent string
|
||||
// want is the upload's limit_percent and bytes_percent, as
|
||||
// want is the upload's rate_limit_percent and bytes_limit_percent, as
|
||||
// percentText gives them, and limitHit its limit_hit.
|
||||
want, limitHit string
|
||||
}{
|
||||
{limitHalf, asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
|
||||
{limitQuarter, asnDEHalf, "25 from " + blocklistAction, minuteBytes},
|
||||
{limitHalf, asnTagDEQuarter, "25 from " + tagLimitPercent, minuteBytes},
|
||||
{limitQuarter, asnTagDEHalf, "25 from " + blocklistAction, minuteBytes},
|
||||
// The AS number's, the first of two alike.
|
||||
{limitQuarter, asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
|
||||
{actionLog, asnDE + ":100", none, ""},
|
||||
{limitQuarter, asnTagDEQuarter, "25 from " + tagLimitPercent, minuteBytes},
|
||||
{actionLog, "asn=64511:25", none, ""},
|
||||
} {
|
||||
t.Run(tc.action+" "+tc.asnPercent, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
bytesLimitPerMinute: twoUploads, blocklistURLs: dropURL,
|
||||
blocklistAction: tc.action, asnLimitPercent: tc.asnPercent,
|
||||
blocklistAction: tc.action, tagLimitPercent: tc.asnPercent,
|
||||
})
|
||||
loadLists(t, server, map[string][]string{dropURL: {fromDE}})
|
||||
|
||||
// The upload's 100 bytes are over 49, a quarter of 199, and 99,
|
||||
// half of it, and within 199.
|
||||
line := s.uploadFrom(fromDE)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
tc.want)
|
||||
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting,
|
||||
tc.want)
|
||||
|
||||
if line.LimitHit != tc.limitHit {
|
||||
@@ -196,7 +249,7 @@ func TestASNLimitPercentFileCountsAsTheSettingDoesTheLowerWinning(t *testing.T)
|
||||
|
||||
const (
|
||||
fromURL = "25 from " + asnLimitPercentURL
|
||||
fromSetting = "25 from " + asnLimitPercent
|
||||
fromSetting = "25 from " + tagLimitPercent
|
||||
)
|
||||
|
||||
for _, tc := range []struct {
|
||||
@@ -210,21 +263,21 @@ func TestASNLimitPercentFileCountsAsTheSettingDoesTheLowerWinning(t *testing.T)
|
||||
{"the file's alone", nil, asnDEQuarter, fromURL, fromURL},
|
||||
{
|
||||
"the file's, lower than the setting's",
|
||||
map[string]string{asnLimitPercent: asnDEHalf}, asnDEQuarter, fromURL, fromURL,
|
||||
map[string]string{tagLimitPercent: asnTagDEHalf}, asnDEQuarter, fromURL, fromURL,
|
||||
},
|
||||
{
|
||||
"the setting's, lower than the file's",
|
||||
map[string]string{asnLimitPercent: asnDEQuarter}, asnDEHalf,
|
||||
map[string]string{tagLimitPercent: asnTagDEQuarter}, asnDEHalf,
|
||||
fromSetting, fromSetting,
|
||||
},
|
||||
{
|
||||
"the setting's, the first of two alike",
|
||||
map[string]string{asnLimitPercent: asnDEQuarter}, asnDEQuarter,
|
||||
map[string]string{tagLimitPercent: asnTagDEQuarter}, asnDEQuarter,
|
||||
fromSetting, fromSetting,
|
||||
},
|
||||
{"none, for an AS number the file does not list", nil, asnKP + ":25", none, none},
|
||||
{
|
||||
"SWWAF_ASN_BYTES_PERCENT's in place of the file's for the byte limits",
|
||||
"SWW_ASN_BYTES_PERCENT's in place of the file's for the byte limits",
|
||||
map[string]string{asnBytesPercent: asnDE + ":100"}, asnDEQuarter, fromURL, none,
|
||||
},
|
||||
} {
|
||||
@@ -238,9 +291,11 @@ func TestASNLimitPercentFileCountsAsTheSettingDoesTheLowerWinning(t *testing.T)
|
||||
loadLists(t, server, map[string][]string{asnURL: {"# by AS number", tc.file}})
|
||||
|
||||
line := s.uploadFrom(fromDE)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
tc.limitPercent)
|
||||
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting,
|
||||
tc.bytesPercent)
|
||||
})
|
||||
}
|
||||
@@ -312,9 +367,9 @@ func TestEachBlocklistThatListsAClientRaisesAnAlertOncePerCooldownAndIsCounted(
|
||||
|
||||
// The DNSBL settings.
|
||||
const (
|
||||
dnsblZones = "SWWAF_DNSBL_ZONES"
|
||||
dnsblResolver = "SWWAF_DNSBL_RESOLVER"
|
||||
reputationAction = "SWWAF_REPUTATION_ACTION"
|
||||
dnsblZones = "SWW_DNSBL_ZONES"
|
||||
dnsblResolver = "SWW_DNSBL_RESOLVER"
|
||||
reputationAction = "SWW_REPUTATION_ACTION"
|
||||
)
|
||||
|
||||
// The DNSBL zones these tests name, which are never asked about the
|
||||
@@ -330,12 +385,12 @@ const (
|
||||
func TestEachReputationActionForAClientADNSBLZoneLists(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
forward, denied := requestlog.ActionForward, requestlog.ActionDenied
|
||||
forward, denied := requestlog.ActionForward, requestlog.ActionClientDenied
|
||||
|
||||
for _, tc := range []struct {
|
||||
action string
|
||||
// statuses and actions are those of a listed client's three
|
||||
// requests, and percent their limit_percent, as percentText gives it.
|
||||
// requests, and percent their rate_limit_percent, as percentText gives it.
|
||||
statuses []int
|
||||
actions []string
|
||||
percent string
|
||||
@@ -347,7 +402,7 @@ func TestEachReputationActionForAClientADNSBLZoneLists(t *testing.T) {
|
||||
{
|
||||
// Half of 4 requests a minute: the third breaks the limit.
|
||||
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||
[]string{forward, forward, requestlog.ActionRateLimited},
|
||||
[]string{forward, forward, requestlog.ActionClientIsRateLimited},
|
||||
"50 from " + reputationAction,
|
||||
},
|
||||
{
|
||||
@@ -370,10 +425,10 @@ func TestEachReputationActionForAClientADNSBLZoneLists(t *testing.T) {
|
||||
for _, from := range []string{fromDE, fromKP} {
|
||||
for i := range 3 {
|
||||
line := s.get(from, tc.statuses[i], tc.actions[i])
|
||||
// In the order SWWAF_DNSBL_ZONES names them.
|
||||
// In the order SWW_DNSBL_ZONES names them.
|
||||
wantReputation(t, line, listedBy[from]...)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent,
|
||||
line.LimitPercentSetting, tc.percent)
|
||||
wantPercent(t, "rate_limit_percent", line.RateLimitPercent,
|
||||
line.RateLimitPercentSetting, tc.percent)
|
||||
|
||||
// A request refused for the verdict is not counted.
|
||||
counted := line.fields["counts"] != nil
|
||||
@@ -388,7 +443,8 @@ func TestEachReputationActionForAClientADNSBLZoneLists(t *testing.T) {
|
||||
for range 3 {
|
||||
line := s.get(unplaced, http.StatusOK, forward)
|
||||
wantReputation(t, line)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
none)
|
||||
}
|
||||
|
||||
@@ -416,9 +472,9 @@ func TestDNSBLZonesComeAfterTheBlocklistsAndSkipAllowNets(t *testing.T) {
|
||||
loadVerdicts(server, map[string][]string{fromKP: {dnsblZone}, fromDE: {dnsblZone}})
|
||||
|
||||
// The blocklist refuses fromKP before its verdict is looked at, and
|
||||
// fromDE, in SWWAF_ALLOW_NETS, is not checked at all: neither is noted
|
||||
// fromDE, in SWW_ALLOW_NETS, is not checked at all: neither is noted
|
||||
// for the zone, nor alerted, nor asked about.
|
||||
wantReputation(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionDenied),
|
||||
wantReputation(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionClientDenied),
|
||||
dropURL)
|
||||
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward))
|
||||
|
||||
@@ -432,6 +488,46 @@ func TestDNSBLZonesComeAfterTheBlocklistsAndSkipAllowNets(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestReputationSourcesTagTheRequestWhateverTheirAction(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
env map[string]string
|
||||
load func(*discriminator.Server)
|
||||
want []string
|
||||
}{
|
||||
{
|
||||
"dnsbl",
|
||||
map[string]string{
|
||||
dnsblZones: dnsblZone, dnsblResolver: noResolver, reputationAction: actionLog,
|
||||
},
|
||||
func(server *discriminator.Server) {
|
||||
loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}})
|
||||
},
|
||||
[]string{"asn=64496", "country=DE", "dnsbl=" + dnsblZone},
|
||||
},
|
||||
{
|
||||
abuseipdb,
|
||||
map[string]string{abuseIPDBKey: accountKey, reputationAction: actionLog},
|
||||
func(server *discriminator.Server) {
|
||||
loadScores(server, map[string]int64{fromDE: 80})
|
||||
},
|
||||
[]string{abuseipdb, "asn=64496", "country=DE"},
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, tc.env)
|
||||
tc.load(server)
|
||||
|
||||
wantTags(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||
tc.want...)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestObserveModeForwardsAClientADNSBLZoneDeniesAndAlertsIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -442,7 +538,7 @@ func TestObserveModeForwardsAClientADNSBLZoneDeniesAndAlertsIt(t *testing.T) {
|
||||
loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}})
|
||||
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionDenied)
|
||||
wantWouldAction(t, line, requestlog.ActionClientDenied)
|
||||
wantReputation(t, line, dnsblZone)
|
||||
|
||||
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
@@ -456,7 +552,7 @@ func TestReputationLimitTakesPartInTheLowestPercentageOfEveryLimit(t *testing.T)
|
||||
|
||||
for _, tc := range []struct {
|
||||
blocklistAction, reputationAction string
|
||||
// want is the request's limit_percent and bytes_percent, as
|
||||
// want is the request's rate_limit_percent and bytes_limit_percent, as
|
||||
// percentText gives them.
|
||||
want string
|
||||
}{
|
||||
@@ -481,9 +577,11 @@ func TestReputationLimitTakesPartInTheLowestPercentageOfEveryLimit(t *testing.T)
|
||||
// Named by the blocklist, then by the zone.
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
wantReputation(t, line, dropURL, dnsblZone)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
tc.want)
|
||||
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting,
|
||||
tc.want)
|
||||
})
|
||||
}
|
||||
@@ -609,7 +707,7 @@ func TestRequestFromAClientWithoutAVerdictHasTheZoneAskedAboutIt(t *testing.T) {
|
||||
|
||||
// The AbuseIPDB settings, and accountKey, the key the tests set.
|
||||
const (
|
||||
abuseIPDBKey = "SWWAF_ABUSEIPDB_KEY"
|
||||
abuseIPDBKey = "SWW_ABUSEIPDB_KEY"
|
||||
accountKey = "abuseipdb-key-0123456789abcdef"
|
||||
)
|
||||
|
||||
@@ -651,8 +749,8 @@ func TestOnlyAClientThatHasCommittedAnOffenceIsCheckedWithAbuseIPDB(t *testing.T
|
||||
s.get(fromDE, http.StatusOK, forward)
|
||||
s.get(fromDE, http.StatusOK, forward)
|
||||
s.get(fromKP, http.StatusOK, forward)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
wantAbuseIPDBChecks(t, server, 0)
|
||||
|
||||
// Once the ban has ended, fromDE's first request has it checked in the
|
||||
@@ -691,7 +789,7 @@ func TestIPv6ClientCostsOneAbuseIPDBCheckWhicheverOfItsAddressesSends(t *testing
|
||||
s.get(addresses[0], http.StatusOK, forward)
|
||||
}
|
||||
|
||||
s.get(addresses[0], http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(addresses[0], http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
clk.advance(time.Hour)
|
||||
|
||||
// Once the ban has ended, which set its counters back to zero, a
|
||||
@@ -724,20 +822,20 @@ func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(t *test
|
||||
want ratelimit.Offences
|
||||
}{
|
||||
{
|
||||
"a block rule", blockedPath, http.StatusForbidden, requestlog.ActionRuleBlocked,
|
||||
ratelimit.Offences{RuleBlocked: 1},
|
||||
"a block rule", blockedPath, http.StatusForbidden,
|
||||
requestlog.ActionBlockedByBlockRule, ratelimit.Offences{BlockedByBlockRule: 1},
|
||||
},
|
||||
{
|
||||
"the Core Rule Set", sqlInjection, http.StatusForbidden,
|
||||
requestlog.ActionWAFBlocked, ratelimit.Offences{WAFBlocked: 1},
|
||||
requestlog.ActionBlockedByCoreRuleSet, ratelimit.Offences{BlockedByCoreRuleSet: 1},
|
||||
},
|
||||
{
|
||||
"a ban rule", probePath, http.StatusForbidden, requestlog.ActionBanned,
|
||||
"a ban rule", probePath, http.StatusForbidden, requestlog.ActionTemporarilyBanned,
|
||||
ratelimit.Offences{Attack: 1},
|
||||
},
|
||||
{
|
||||
"a trap path", "/xmlrpc.php", http.StatusForbidden, requestlog.ActionBanned,
|
||||
ratelimit.Offences{Attack: 1},
|
||||
"a trap path", "/xmlrpc.php", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned, ratelimit.Offences{Attack: 1},
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -768,12 +866,12 @@ func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(t *test
|
||||
func TestEachReputationActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
forward, denied := requestlog.ActionForward, requestlog.ActionDenied
|
||||
forward, denied := requestlog.ActionForward, requestlog.ActionClientDenied
|
||||
|
||||
for _, tc := range []struct {
|
||||
action string
|
||||
// statuses and actions are those of fromDE's three requests, and
|
||||
// percent their limit_percent, as percentText gives it.
|
||||
// percent their rate_limit_percent, as percentText gives it.
|
||||
statuses []int
|
||||
actions []string
|
||||
percent string
|
||||
@@ -785,7 +883,7 @@ func TestEachReputationActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(t *test
|
||||
{
|
||||
// Half of 4 requests a minute: the third breaks the limit.
|
||||
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||
[]string{forward, forward, requestlog.ActionRateLimited},
|
||||
[]string{forward, forward, requestlog.ActionClientIsRateLimited},
|
||||
"50 from " + reputationAction,
|
||||
},
|
||||
{
|
||||
@@ -800,13 +898,14 @@ func TestEachReputationActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(t *test
|
||||
rateLimitPerMinute: fourAMinute, abuseIPDBKey: accountKey,
|
||||
reputationAction: tc.action,
|
||||
})
|
||||
// At SWWAF_ABUSEIPDB_MIN_SCORE, 75 by default, and just under it.
|
||||
// At SWW_ABUSEIPDB_MIN_SCORE, 75 by default, and just under it.
|
||||
loadScores(server, map[string]int64{fromDE: 75, fromKP: 74})
|
||||
|
||||
for i := range 3 {
|
||||
line := s.get(fromDE, tc.statuses[i], tc.actions[i])
|
||||
wantReputation(t, line, abuseipdb)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
tc.percent)
|
||||
|
||||
// A request refused for the score is not counted.
|
||||
@@ -820,7 +919,8 @@ func TestEachReputationActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(t *test
|
||||
for range 3 {
|
||||
line := s.get(fromKP, http.StatusOK, forward)
|
||||
wantReputation(t, line)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
none)
|
||||
}
|
||||
|
||||
@@ -851,7 +951,7 @@ func TestAbuseIPDBHitRaisesAnAlertWithTheScoreOncePerCooldownAndIsCounted(
|
||||
// The alert is made as a DNSBL zone's is, with the score besides.
|
||||
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit ||
|
||||
waiting[0].Reason != "scored by AbuseIPDB at or over SWWAF_ABUSEIPDB_MIN_SCORE" ||
|
||||
waiting[0].Reason != "scored by AbuseIPDB at or over SWW_ABUSEIPDB_MIN_SCORE" ||
|
||||
waiting[0].Detail["source"] != abuseipdb || waiting[0].Detail["score"] != int64(90) ||
|
||||
queue.Suppressed() != 1 {
|
||||
t.Errorf("alerts waiting %+v, %d held back, want AbuseIPDB's reputation_hit "+
|
||||
@@ -882,7 +982,7 @@ func TestWithoutAnAbuseIPDBKeyNoClientIsCheckedNorAScoreUsed(t *testing.T) {
|
||||
// fromDE's score is not used, and once it has committed an offence it
|
||||
// is not checked either.
|
||||
wantReputation(t, s.get(fromDE, http.StatusOK, forward))
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
clk.advance(time.Hour)
|
||||
wantReputation(t, s.get(fromDE, http.StatusOK, forward))
|
||||
wantAbuseIPDBChecks(t, server, 0)
|
||||
@@ -906,10 +1006,10 @@ func TestBanNotesNameEachReputationSourceThatListedTheClient(t *testing.T) {
|
||||
}{
|
||||
{"for a broken rate limit", func(s *sender, from string) {
|
||||
s.get(from, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(from, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
s.get(from, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
}},
|
||||
{"for a clear sign of attack", func(s *sender, from string) {
|
||||
s.request(from, probePath, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(from, probePath, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -922,7 +1022,7 @@ func TestBanNotesNameEachReputationSourceThatListedTheClient(t *testing.T) {
|
||||
abuseIPDBKey: accountKey, reputationAction: actionLog,
|
||||
})
|
||||
// Every source lists client, and none otherClient, whose score is
|
||||
// under SWWAF_ABUSEIPDB_MIN_SCORE, 75 by default.
|
||||
// under SWW_ABUSEIPDB_MIN_SCORE, 75 by default.
|
||||
loadLists(t, server, map[string][]string{dropURL: {client}})
|
||||
loadVerdicts(server, map[string][]string{client: {dnsblZone}, otherClient: nil})
|
||||
loadScores(server, map[string]int64{client: score, otherClient: 74})
|
||||
@@ -971,7 +1071,7 @@ func verdictsFetched() time.Time {
|
||||
}
|
||||
|
||||
// loadVerdicts puts into server's DNSBL, for each client listedBy names,
|
||||
// a verdict of each zone SWWAF_DNSBL_ZONES names, fetched at
|
||||
// a verdict of each zone SWW_DNSBL_ZONES names, fetched at
|
||||
// verdictsFetched, as reputation.json would at start: one that lists the
|
||||
// client from each zone listedBy gives for it, and one that does not from
|
||||
// each other zone.
|
||||
|
||||
@@ -44,7 +44,7 @@ type decision struct {
|
||||
lookedUp bool
|
||||
lookupAnswer lookup.Answer
|
||||
// skippedLookupWait is true for a request that did not wait for GeoJS and
|
||||
// found no answer kept. SWWAF_UNKNOWN_LIMIT_PERCENT does not apply to it,
|
||||
// found no answer kept. SWW_UNKNOWN_LIMIT_PERCENT does not apply to it,
|
||||
// so that not waiting never lowers its client's limits.
|
||||
skippedLookupWait bool
|
||||
// counted is true for a request the rate limits counted, whose bytes
|
||||
@@ -54,10 +54,10 @@ type decision struct {
|
||||
counted bool
|
||||
limitPercent, bytesPercent percentage
|
||||
// attack is true for a request that matched a ban rule or asked for a
|
||||
// trap path, ruleBlocked for one a block rule refused, and wafBlocked for
|
||||
// one the Core Rule Set refused, each an offence its client's history
|
||||
// counts.
|
||||
attack, ruleBlocked, wafBlocked bool
|
||||
// trap path, blockedByBlockRule for one a block rule refused, and
|
||||
// blockedByCoreRuleSet for one the Core Rule Set refused, each an offence
|
||||
// its client's history counts.
|
||||
attack, blockedByBlockRule, blockedByCoreRuleSet bool
|
||||
// reputation is the reputation sources that list the client, for the
|
||||
// notes of a ban the request makes.
|
||||
reputation []bans.ReputationHit
|
||||
@@ -138,6 +138,15 @@ func (h *handler) startRequest(
|
||||
return rq
|
||||
}
|
||||
|
||||
// addTag adds tag to the request's tags, which stay sorted and hold each tag
|
||||
// once.
|
||||
func (rq *request) addTag(tag string) {
|
||||
at, found := slices.BinarySearch(rq.line.Tags, tag)
|
||||
if !found {
|
||||
rq.line.Tags = slices.Insert(rq.line.Tags, at, tag)
|
||||
}
|
||||
}
|
||||
|
||||
// requestHeaders returns the headers of r that names lists, by name in
|
||||
// lower case, each with its values joined by ", ". Authorization, Cookie
|
||||
// and Set-Cookie are never among them, whatever names says.
|
||||
@@ -160,58 +169,49 @@ func requestHeaders(r *http.Request, names []string) map[string]string {
|
||||
}
|
||||
|
||||
// check is the one place where a request can be refused. It returns nil to let
|
||||
// the request through. The checks of checkClient decide: a request they refuse
|
||||
// is answered with SWWAF_BAN_RESPONSE, 403 for a block rule or the Core Rule Set,
|
||||
// or 431 for the header limits. In observe mode nothing is refused, and the log
|
||||
// line names what enforce mode would have done. ctx is the request's own context.
|
||||
// the request through. The heuristics of checkClient decide: a request they
|
||||
// refuse is answered with SWW_BAN_RESPONSE, 403 for a block rule or the Core
|
||||
// Rule Set, or 431 for the header limits. In observe mode nothing is refused,
|
||||
// and the log line names what enforce mode would have done. ctx is the
|
||||
// request's own context.
|
||||
func (rq *request) check(ctx context.Context) *refusal {
|
||||
action := rq.checkClient(ctx)
|
||||
v := rq.checkClient(ctx)
|
||||
if v == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
action := v.action
|
||||
|
||||
switch {
|
||||
case action == "":
|
||||
return nil
|
||||
case rq.h.config.Observe:
|
||||
rq.line.WouldAction = action
|
||||
|
||||
return nil
|
||||
case action == requestlog.ActionRuleBlocked || action == requestlog.ActionWAFBlocked:
|
||||
case action == requestlog.ActionBlockedByBlockRule ||
|
||||
action == requestlog.ActionBlockedByCoreRuleSet:
|
||||
return &refusal{status: http.StatusForbidden, action: action}
|
||||
case action == requestlog.ActionTooLarge:
|
||||
case action == requestlog.ActionRequestHeaderTooLarge:
|
||||
return &refusal{status: http.StatusRequestHeaderFieldsTooLarge, action: action}
|
||||
default:
|
||||
return rq.banResponse(action)
|
||||
}
|
||||
}
|
||||
|
||||
// checkClient runs the checks on the request's client, and returns the
|
||||
// action of the first that refuses the request, or "" when none does. A
|
||||
// client in SWWAF_ALLOW_NETS skips them, and is not looked up. For any
|
||||
// other client, SWWAF_DENY_NETS comes first, then a ban on its netblock,
|
||||
// so that a client either refuses is not looked up, then the lookup of
|
||||
// its AS number and country, then the country lists, then the blocklists,
|
||||
// then the CrowdSec decision list, which bans the client it lists, then
|
||||
// the DNSBL zones' verdicts, and then AbuseIPDB's score; a request any of
|
||||
// them refuses is not counted for the rate limits. Then come the
|
||||
// rate limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the
|
||||
// request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that
|
||||
// every other request is counted, each of them by the client's limit
|
||||
// percentages, then SWWAF_TRAP_PATHS, then the rule files, then the header
|
||||
// limits, and last the Core Rule Set. A request exempt from the rate limits
|
||||
// is exempt from the byte limits too. ctx is the request's own context.
|
||||
func (rq *request) checkClient(ctx context.Context) string {
|
||||
cfg := rq.h.config
|
||||
if isInside(rq.client, cfg.AllowNets) {
|
||||
return ""
|
||||
// checkClient runs the heuristics of beforeLookup on the request, then looks
|
||||
// up the client's AS number and country, then runs those of afterLookup. It
|
||||
// returns the first verdict, or nil when none gives one. A client in
|
||||
// SWW_ALLOW_NETS skips them all, and is not looked up. ctx is the request's
|
||||
// own context.
|
||||
func (rq *request) checkClient(ctx context.Context) *verdict {
|
||||
if isInside(rq.client, rq.h.config.AllowNets) {
|
||||
return nil
|
||||
}
|
||||
|
||||
now := rq.h.now()
|
||||
|
||||
if isInside(rq.client, cfg.DenyNets) {
|
||||
return requestlog.ActionDenied
|
||||
}
|
||||
|
||||
if rq.banned(now) {
|
||||
return requestlog.ActionBanned
|
||||
v := rq.runHeuristics(ctx, beforeLookup(), now)
|
||||
if v != nil {
|
||||
return v
|
||||
}
|
||||
|
||||
rq.lookUp(ctx, now)
|
||||
@@ -220,63 +220,11 @@ func (rq *request) checkClient(ctx context.Context) string {
|
||||
// again: a request counted at the time it arrived could be counted
|
||||
// after a later one of its client's, which the rate limits would take
|
||||
// for the clock set back.
|
||||
now = rq.h.now()
|
||||
|
||||
if rq.countryDenied() {
|
||||
return requestlog.ActionCountryDenied
|
||||
}
|
||||
|
||||
if rq.blocklistDenied() {
|
||||
return requestlog.ActionDenied
|
||||
}
|
||||
|
||||
if rq.crowdSecBanned(now) {
|
||||
return requestlog.ActionBanned
|
||||
}
|
||||
|
||||
if rq.dnsblDenied(ctx) || rq.abuseIPDBDenied(ctx) {
|
||||
return requestlog.ActionDenied
|
||||
}
|
||||
|
||||
rq.counted = rq.countedByRateLimits()
|
||||
if rq.counted {
|
||||
rq.limitPercent, rq.bytesPercent = rq.limitPercentages()
|
||||
rq.line.LimitPercent, rq.line.LimitPercentSetting = rq.limitPercent.logged()
|
||||
rq.line.BytesPercent, rq.line.BytesPercentSetting = rq.bytesPercent.logged()
|
||||
}
|
||||
|
||||
if rq.counted && rq.limitBroken(now) {
|
||||
return requestlog.ActionRateLimited
|
||||
}
|
||||
|
||||
if rq.trapPath(now) {
|
||||
return requestlog.ActionBanned
|
||||
}
|
||||
|
||||
action := rq.checkRules(now)
|
||||
if action != "" {
|
||||
return action
|
||||
}
|
||||
|
||||
if rq.overHeaderLimits() {
|
||||
return requestlog.ActionTooLarge
|
||||
}
|
||||
|
||||
return rq.checkCoreRuleSet()
|
||||
}
|
||||
|
||||
// countedByRateLimits reports whether the rate limits count the request: its
|
||||
// client is not in SWWAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt
|
||||
// under SWWAF_RATE_LIMIT_EXEMPT_PATHS.
|
||||
func (rq *request) countedByRateLimits() bool {
|
||||
cfg := rq.h.config
|
||||
|
||||
return !isInside(rq.client, cfg.RateLimitExemptNets) &&
|
||||
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
|
||||
return rq.runHeuristics(ctx, afterLookup(), rq.h.now())
|
||||
}
|
||||
|
||||
// pathExempt reports whether r is exempt under prefixes,
|
||||
// SWWAF_RATE_LIMIT_EXEMPT_PATHS or SWWAF_WAF_EXEMPT_PATHS: whether its
|
||||
// SWW_RATE_LIMIT_EXEMPT_PATHS or SWW_WAF_EXEMPT_PATHS: whether its
|
||||
// path as sent, the path the app receives, not percent-decoded, starts
|
||||
// with one of prefixes, so that /%61ssets/x is not under /assets/ for an
|
||||
// app whose router matches the path as received. A request whose decoded
|
||||
@@ -314,8 +262,8 @@ func (rq *request) finish() {
|
||||
}
|
||||
|
||||
duration := time.Since(rq.start)
|
||||
line.DurationTotal = requestlog.Milliseconds(duration)
|
||||
line.DurationChecks = timing(rq.start, rq.checked)
|
||||
line.DurationTotalMS = requestlog.Milliseconds(duration)
|
||||
line.DurationChecksMS = timing(rq.start, rq.checked)
|
||||
|
||||
// Counted before the log line is written, so that the metrics count
|
||||
// every request whose line is out. An allowed request is the exception:
|
||||
@@ -356,15 +304,15 @@ func timing(start, end time.Time) *float64 {
|
||||
// when it comes.
|
||||
func (rq *request) addToHistory() {
|
||||
request := ratelimit.Request{
|
||||
Forwarded: rq.allowed,
|
||||
Refused: rq.refused != nil,
|
||||
Status: rq.out.status,
|
||||
RequestBytes: rq.requestBytes,
|
||||
ResponseBytes: rq.out.bytes,
|
||||
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
|
||||
Attack: rq.attack,
|
||||
RuleBlocked: rq.ruleBlocked,
|
||||
WAFBlocked: rq.wafBlocked,
|
||||
Forwarded: rq.allowed,
|
||||
Refused: rq.refused != nil,
|
||||
Status: rq.out.status,
|
||||
RequestBytes: rq.requestBytes,
|
||||
ResponseBytes: rq.out.bytes,
|
||||
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
|
||||
Attack: rq.attack,
|
||||
BlockedByBlockRule: rq.blockedByBlockRule,
|
||||
BlockedByCoreRuleSet: rq.blockedByCoreRuleSet,
|
||||
}
|
||||
|
||||
rq.h.limiter.AddToHistory(rq.h.clientGroup(rq.client), rq.h.now(), request)
|
||||
@@ -378,9 +326,9 @@ func (rq *request) addToHistory() {
|
||||
|
||||
// countAnomalies counts the request, which has ended, and its bytes, as
|
||||
// countedBytes gives them, for the anomaly thresholds, whatever was done
|
||||
// with it: a request refused, one from a client in SWWAF_ALLOW_NETS or
|
||||
// SWWAF_RATE_LIMIT_EXEMPT_NETS, and one for a path in
|
||||
// SWWAF_RATE_LIMIT_EXEMPT_PATHS are counted too. It is counted for its
|
||||
// with it: a request refused, one from a client in SWW_ALLOW_NETS or
|
||||
// SWW_RATE_LIMIT_EXEMPT_NETS, and one for a path in
|
||||
// SWW_RATE_LIMIT_EXEMPT_PATHS are counted too. It is counted for its
|
||||
// client's AS number when answerAtTheEnd gives one. With every anomaly
|
||||
// threshold off, the default, it does nothing.
|
||||
func (rq *request) countAnomalies() {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package discriminator_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"math"
|
||||
"net/http"
|
||||
@@ -18,7 +19,7 @@ import (
|
||||
const (
|
||||
// requestIDHeader carries the request's id.
|
||||
requestIDHeader = "X-Request-ID"
|
||||
// instance is the SWWAF_INSTANCE_NAME a test sets.
|
||||
// instance is the SWW_INSTANCE_NAME a test sets.
|
||||
instance = "fsn1app1/blog"
|
||||
// ipv6Client is a client on IPv6, and ipv6Group the netblock the rate
|
||||
// limits count it as.
|
||||
@@ -155,9 +156,9 @@ func wantBareLine(t *testing.T, line logLine) {
|
||||
|
||||
// withTimings returns want with the timings of line.
|
||||
func withTimings(line logLine, want requestlog.Line) requestlog.Line {
|
||||
want.DurationTotal = line.DurationTotal
|
||||
want.DurationChecks = line.DurationChecks
|
||||
want.DurationUpstreamTotal = line.DurationUpstreamTotal
|
||||
want.DurationTotalMS = line.DurationTotalMS
|
||||
want.DurationChecksMS = line.DurationChecksMS
|
||||
want.DurationUpstreamTotalMS = line.DurationUpstreamTotalMS
|
||||
|
||||
return want
|
||||
}
|
||||
@@ -195,7 +196,7 @@ func TestHasAuthorizationAndHasCookieEachComeFromTheirOwnHeader(t *testing.T) {
|
||||
func TestTimingsAreInOrder(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const denied = "192.0.2.50" // in SWWAF_DENY_NETS
|
||||
const denied = "192.0.2.50" // in SWW_DENY_NETS
|
||||
|
||||
app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
// The pause sets the times apart; a hold-up of the test only lengthens
|
||||
@@ -225,10 +226,10 @@ func TestTimingsAreInOrder(t *testing.T) {
|
||||
|
||||
// A request passed to the app has every timing; one refused, none of
|
||||
// the app's; the health check, which runs no check, only the total.
|
||||
wantTimings(t, forwarded, "duration_total", "duration_checks",
|
||||
"duration_upstream_total")
|
||||
wantTimings(t, refused, "duration_total", "duration_checks")
|
||||
wantTimings(t, health, "duration_total")
|
||||
wantTimings(t, forwarded, "duration_total_ms", "duration_checks_ms",
|
||||
"duration_upstream_total_ms")
|
||||
wantTimings(t, refused, "duration_total_ms", "duration_checks_ms")
|
||||
wantTimings(t, health, "duration_total_ms")
|
||||
|
||||
if t.Failed() {
|
||||
return
|
||||
@@ -236,9 +237,9 @@ func TestTimingsAreInOrder(t *testing.T) {
|
||||
|
||||
// In whole microseconds, as they are logged, so that the sum below is
|
||||
// exact.
|
||||
total := microseconds(forwarded.DurationTotal)
|
||||
checks := microseconds(*forwarded.DurationChecks)
|
||||
upstream := microseconds(*forwarded.DurationUpstreamTotal)
|
||||
total := microseconds(forwarded.DurationTotalMS)
|
||||
checks := microseconds(*forwarded.DurationChecksMS)
|
||||
upstream := microseconds(*forwarded.DurationUpstreamTotalMS)
|
||||
|
||||
// The checks end before the request is handed to the app.
|
||||
if checks+upstream > total {
|
||||
@@ -246,9 +247,9 @@ func TestTimingsAreInOrder(t *testing.T) {
|
||||
total, checks, upstream)
|
||||
}
|
||||
|
||||
if *refused.DurationChecks > refused.DurationTotal {
|
||||
if *refused.DurationChecksMS > refused.DurationTotalMS {
|
||||
t.Errorf("refused request's checks took %v of %v milliseconds",
|
||||
*refused.DurationChecks, refused.DurationTotal)
|
||||
*refused.DurationChecksMS, refused.DurationTotalMS)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -272,6 +273,48 @@ func wantTimings(t *testing.T, line logLine, want ...string) {
|
||||
}
|
||||
}
|
||||
|
||||
// wantHeuristic checks that line names heuristic as the one that refused the
|
||||
// request, with figures, a JSON object. A heuristic of "" with figures "null" is
|
||||
// a line that names none.
|
||||
func wantHeuristic(t *testing.T, line logLine, heuristic, figures string) {
|
||||
t.Helper()
|
||||
|
||||
if line.Heuristic != heuristic ||
|
||||
!reflect.DeepEqual(line.HeuristicFigures, decodeFigures(t, figures)) {
|
||||
t.Errorf("log line has heuristic %q with heuristic_figures %v, want %q with %s",
|
||||
line.Heuristic, line.HeuristicFigures, heuristic, figures)
|
||||
}
|
||||
}
|
||||
|
||||
// wantAfterAnswerHeuristic checks after_answer_heuristic and
|
||||
// after_answer_heuristic_figures as wantHeuristic checks heuristic and
|
||||
// heuristic_figures.
|
||||
func wantAfterAnswerHeuristic(t *testing.T, line logLine, heuristic, figures string) {
|
||||
t.Helper()
|
||||
|
||||
if line.AfterAnswerHeuristic != heuristic ||
|
||||
!reflect.DeepEqual(line.AfterAnswerHeuristicFigures, decodeFigures(t, figures)) {
|
||||
t.Errorf("log line has after_answer_heuristic %q with "+
|
||||
"after_answer_heuristic_figures %v, want %q with %s",
|
||||
line.AfterAnswerHeuristic, line.AfterAnswerHeuristicFigures, heuristic, figures)
|
||||
}
|
||||
}
|
||||
|
||||
// decodeFigures decodes figures, a JSON object of a heuristic's figures, as
|
||||
// the log line's figures are decoded.
|
||||
func decodeFigures(t *testing.T, figures string) map[string]any {
|
||||
t.Helper()
|
||||
|
||||
var decoded map[string]any
|
||||
|
||||
err := json.Unmarshal([]byte(figures), &decoded)
|
||||
if err != nil {
|
||||
t.Fatalf("decode %s: %v", figures, err)
|
||||
}
|
||||
|
||||
return decoded
|
||||
}
|
||||
|
||||
// microseconds is a timing in whole microseconds.
|
||||
func microseconds(milliseconds float64) int64 {
|
||||
return int64(math.Round(milliseconds * 1000))
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic routewatch_tags. It reads SWW_ROUTEWATCH_URL.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
)
|
||||
|
||||
// routeWatchTags adds to the request the tags asn=<number> and prefix=<netblock>
|
||||
// from what routewatch answered about the client. It refuses nothing, and the
|
||||
// request does not wait for routewatch. A request that gets no tags from it has
|
||||
// the reason in the log line's routewatch_tags_missing. While SWW_ROUTEWATCH_URL
|
||||
// is unset it does nothing. Answers are kept by client and asked for by the
|
||||
// request's address, and a route without an origin AS gives only the prefix= tag.
|
||||
type routeWatchTags struct{}
|
||||
|
||||
func (routeWatchTags) name() string { return "routewatch_tags" }
|
||||
|
||||
func (routeWatchTags) check(ctx context.Context, rq *request, _ time.Time) *verdict {
|
||||
if rq.h.routeWatch == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
route, missing := rq.h.routeWatch.Route(ctx, rq.h.clientGroup(rq.client), rq.client)
|
||||
if missing != "" {
|
||||
rq.line.RouteWatchTagsMissing = string(missing)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.addTag(config.PrefixTagPrefix + route.Netblock.String())
|
||||
|
||||
if route.ASN != 0 {
|
||||
rq.addTag(config.ASNTagPrefix + strconv.FormatUint(uint64(route.ASN), 10))
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,245 @@
|
||||
package discriminator_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/discriminator"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// routeWatchURL is SWW_ROUTEWATCH_URL, and tagsMissing the log line's field
|
||||
// that says why a request got no tags from routewatch.
|
||||
const (
|
||||
routeWatchURL = "SWW_ROUTEWATCH_URL"
|
||||
tagsMissing = "routewatch_tags_missing"
|
||||
fromV6 = "2001:db8:1::5"
|
||||
prefixV6 = "2001:db8::/32"
|
||||
prefixDE = "203.0.113.0/24"
|
||||
prefixKP = "198.51.100.0/24"
|
||||
)
|
||||
|
||||
// standInRoute is what the stand-in for routewatch answers about an address.
|
||||
type standInRoute struct {
|
||||
Netblock string `json:"netblock"`
|
||||
ASN int `json:"asn"`
|
||||
}
|
||||
|
||||
// startRouteWatch starts a stand-in for routewatch's IP info API that knows the
|
||||
// routes given by address, and returns its URL.
|
||||
func startRouteWatch(t *testing.T, routes map[string]standInRoute) string {
|
||||
t.Helper()
|
||||
|
||||
server := httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
address := strings.TrimPrefix(r.URL.Path, "/api/v1/ip/")
|
||||
|
||||
result := map[string]any{"ip": address}
|
||||
if route, known := routes[address]; known {
|
||||
result["netblock"], result["asn"] = route.Netblock, route.ASN
|
||||
}
|
||||
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"status": "ok",
|
||||
"data": map[string]any{"results": []any{result}},
|
||||
})
|
||||
}))
|
||||
t.Cleanup(server.Close)
|
||||
|
||||
return server.URL
|
||||
}
|
||||
|
||||
// waitForRouteWatch waits until routewatch's answer about from, in the client
|
||||
// group of the given length, is kept, so that the next request gets its tags.
|
||||
func waitForRouteWatch(
|
||||
t *testing.T, server *discriminator.Server, from string, bits int,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
addr := netip.MustParseAddr(from)
|
||||
client := netip.PrefixFrom(addr, bits).Masked()
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
|
||||
for {
|
||||
_, missing := server.RouteWatch.Route(context.Background(), client, addr)
|
||||
if missing == "" {
|
||||
return
|
||||
}
|
||||
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("routewatch's answer about %s not kept: %q", from, missing)
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func wantTagsMissing(t *testing.T, line logLine, want string) {
|
||||
t.Helper()
|
||||
|
||||
got, _ := line.fields[tagsMissing].(string)
|
||||
if got != want {
|
||||
t.Errorf("log line has %s %q, want %q", tagsMissing, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRouteWatchTagsAreOnTheLogLineWithTheLookupsTags(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
url := startRouteWatch(t, map[string]standInRoute{fromDE: {prefixDE, 64496}})
|
||||
s, server, _ := startWithLookups(t, map[string]string{routeWatchURL: url})
|
||||
|
||||
waitForRouteWatch(t, server, fromDE, 32)
|
||||
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
wantTags(t, line, "asn=64496", "country=DE", "prefix="+prefixDE)
|
||||
wantTagsMissing(t, line, "")
|
||||
}
|
||||
|
||||
func TestRouteWatchTagsAreOnALineDenyNetsRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
url := startRouteWatch(t, map[string]standInRoute{fromKP: {prefixKP, 64511}})
|
||||
s, _, server := startWithClock(t, "", map[string]string{
|
||||
lookupSource: off, routeWatchURL: url, denyNets: fromKP,
|
||||
})
|
||||
|
||||
waitForRouteWatch(t, server, fromKP, 32)
|
||||
|
||||
line := s.get(fromKP, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, "deny_nets", `{"netblock": "`+fromKP+`/32"}`)
|
||||
wantTags(t, line, "asn=64511", "prefix="+prefixKP)
|
||||
}
|
||||
|
||||
func TestRouteWithoutAnOriginASGivesOnlyThePrefixTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
url := startRouteWatch(t, map[string]standInRoute{fromDE: {prefixDE, 0}})
|
||||
s, _, server := startWithClock(t, "", map[string]string{
|
||||
lookupSource: off, routeWatchURL: url,
|
||||
})
|
||||
|
||||
waitForRouteWatch(t, server, fromDE, 32)
|
||||
|
||||
wantTags(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||
"prefix="+prefixDE)
|
||||
}
|
||||
|
||||
func TestUnsetRouteWatchChangesNothing(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, server := startWithClock(t, "", map[string]string{lookupSource: off})
|
||||
|
||||
if server.RouteWatch != nil {
|
||||
t.Error("RouteWatch is set while SWW_ROUTEWATCH_URL is unset")
|
||||
}
|
||||
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
if _, present := line.fields[tagsMissing]; present {
|
||||
t.Errorf("log line has %s", tagsMissing)
|
||||
}
|
||||
|
||||
if _, present := line.fields["tags"]; present {
|
||||
t.Errorf("log line has tags %v", line.fields["tags"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnreachableRouteWatchLetsTheRequestThroughAndSaysWhy(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// A stand-in that is closed at once, so that the port refuses connections.
|
||||
dead := httptest.NewServer(http.NotFoundHandler())
|
||||
url := dead.URL
|
||||
|
||||
dead.Close()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
lookupSource: off, routeWatchURL: url,
|
||||
})
|
||||
|
||||
wantTagsMissing(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||
"no_answer_yet")
|
||||
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
|
||||
for {
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
missing, _ := line.fields[tagsMissing].(string)
|
||||
if missing == "request_failed" {
|
||||
return
|
||||
}
|
||||
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("still %q, want request_failed", missing)
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddressWithoutARouteHasNoTagsAndTheLineSaysSo(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
lookupSource: off, routeWatchURL: startRouteWatch(t, nil),
|
||||
})
|
||||
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
|
||||
for {
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
missing, _ := line.fields[tagsMissing].(string)
|
||||
if missing == "no_route" {
|
||||
return
|
||||
}
|
||||
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("still %q, want no_route", missing)
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagDenyRefusesOnARouteWatchPrefix(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
url := startRouteWatch(t, map[string]standInRoute{
|
||||
fromDE: {prefixDE, 64496}, fromKP: {prefixKP, 64511},
|
||||
})
|
||||
s, _, server := startWithClock(t, "", map[string]string{
|
||||
lookupSource: off, routeWatchURL: url, tagDeny: "prefix=" + prefixKP,
|
||||
})
|
||||
|
||||
waitForRouteWatch(t, server, fromKP, 32)
|
||||
waitForRouteWatch(t, server, fromDE, 32)
|
||||
|
||||
wantHeuristic(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionClientDenied),
|
||||
"tag_deny", `{"tag": "prefix=`+prefixKP+`"}`)
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
}
|
||||
|
||||
func TestTagLimitPercentWorksWithAnIPv6Prefix(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
url := startRouteWatch(t, map[string]standInRoute{fromV6: {prefixV6, 64496}})
|
||||
s, _, server := startWithClock(t, "", map[string]string{
|
||||
lookupSource: off, routeWatchURL: url, rateLimitPerMinute: "2",
|
||||
tagLimitPercent: "prefix=" + prefixV6 + ":50",
|
||||
})
|
||||
|
||||
waitForRouteWatch(t, server, fromV6, 64)
|
||||
|
||||
// Half of two is one: the second request breaks the limit.
|
||||
line := s.get(fromV6, http.StatusOK, requestlog.ActionForward)
|
||||
wantTags(t, line, "asn=64496", "prefix="+prefixV6)
|
||||
s.get(fromV6, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic rule_files. It reads the rules of the rule files, which
|
||||
// SWW_RULES_DIR and SWW_RULES_ENABLED name.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// ruleFiles checks the request against the rules of the rule files at now,
|
||||
// notes the ids of those it matches in the log line, adds the tag
|
||||
// exploitsignature when it matches any, and refuses the request
|
||||
// when the last rule it matches refuses it: with ActionBlockedByBlockRule for a
|
||||
// block rule, and for a ban rule with the action for the ban it makes. A ban
|
||||
// rule bans the client's netblock for a clear sign of attack, or in observe
|
||||
// mode raises the alert for the ban
|
||||
// it would have made. Either rule's match is noted as an offence, for the
|
||||
// client's history. Its figures are the rule's id and action.
|
||||
type ruleFiles struct{}
|
||||
|
||||
func (ruleFiles) name() string { return "rule_files" }
|
||||
|
||||
func (ruleFiles) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
matched := rq.h.rules.Match(rq.in)
|
||||
|
||||
for _, rule := range matched {
|
||||
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
|
||||
rq.h.metrics.RuleMatched(rule.ID, rule.Action)
|
||||
}
|
||||
|
||||
if len(matched) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.addTag(config.ExploitSignatureTag)
|
||||
|
||||
// Only the last rule matched can refuse the request.
|
||||
last := matched[len(matched)-1]
|
||||
figures := map[string]any{"rule_id": last.ID, "rule_action": last.Action}
|
||||
|
||||
switch last.Action {
|
||||
case rules.ActionBlock:
|
||||
rq.blockedByBlockRule = true
|
||||
|
||||
return &verdict{action: requestlog.ActionBlockedByBlockRule, figures: figures}
|
||||
case rules.ActionBan:
|
||||
rq.attack = true
|
||||
|
||||
return &verdict{
|
||||
action: rq.banForAttack(now, bans.Notes{RuleID: last.ID, Target: last.Target}),
|
||||
figures: figures,
|
||||
}
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
@@ -34,16 +34,17 @@ func TestEachRuleAction(t *testing.T) {
|
||||
line := s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
wantRuleIDs(t, line, "noted")
|
||||
|
||||
// A block rule refuses with 403, whatever SWWAF_BAN_RESPONSE is, and
|
||||
// A block rule refuses with 403, whatever SWW_BAN_RESPONSE is, and
|
||||
// bans no one.
|
||||
line = s.request(client, "/blocked", http.StatusForbidden,
|
||||
requestlog.ActionRuleBlocked)
|
||||
requestlog.ActionBlockedByBlockRule)
|
||||
wantRuleIDs(t, line, "noted", "blocked")
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
// A ban rule refuses with SWWAF_BAN_RESPONSE, and bans the client for
|
||||
// A ban rule refuses with SWW_BAN_RESPONSE, and bans the client for
|
||||
// seven days, the default.
|
||||
line = s.request(client, "/.env", http.StatusTooManyRequests, requestlog.ActionBanned)
|
||||
line = s.request(client, "/.env", http.StatusTooManyRequests,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
wantRuleIDs(t, line, "noted", "probe")
|
||||
|
||||
if line.BanExpires != requestlog.FormatTime(start.Add(7*24*time.Hour)) {
|
||||
@@ -82,7 +83,7 @@ func TestEachRuleAction(t *testing.T) {
|
||||
// against the rules, and makes the ban permanent.
|
||||
clk.advance(time.Hour)
|
||||
|
||||
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
|
||||
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionPermanentlyBanned)
|
||||
wantRuleIDs(t, line)
|
||||
|
||||
if line.BanExpires != permanent {
|
||||
@@ -90,7 +91,37 @@ func TestEachRuleAction(t *testing.T) {
|
||||
}
|
||||
|
||||
clk.advance(365 * 24 * time.Hour)
|
||||
s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusTooManyRequests, requestlog.ActionPermanentlyBanned)
|
||||
}
|
||||
|
||||
func TestEachRuleActionAddsTheExploitSignatureTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
rulesDir: writeRules(t, "noted path log ^/noted$\n"+testRules),
|
||||
})
|
||||
|
||||
wantTags(t, s.request(client, "/noted", http.StatusOK, requestlog.ActionForward),
|
||||
"exploitsignature")
|
||||
wantTags(t, s.request(client, "/blocked", http.StatusForbidden,
|
||||
requestlog.ActionBlockedByBlockRule), "exploitsignature")
|
||||
wantTags(t, s.get(client, http.StatusOK, requestlog.ActionForward))
|
||||
}
|
||||
|
||||
func TestRuleRefusalNamesItsHeuristicAndTheRule(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
rulesDir: writeRules(t, "noted path log ^/\n"+testRules),
|
||||
})
|
||||
|
||||
line := s.request(client, "/blocked", http.StatusForbidden,
|
||||
requestlog.ActionBlockedByBlockRule)
|
||||
wantHeuristic(t, line, "rule_files", `{"rule_id": "blocked", "rule_action": "block"}`)
|
||||
|
||||
line = s.request(client, "/.env", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
wantHeuristic(t, line, "rule_files", `{"rule_id": "probe", "rule_action": "ban"}`)
|
||||
}
|
||||
|
||||
func TestNextClearSignOfAttackAfterABanBansPermanently(t *testing.T) {
|
||||
@@ -101,8 +132,9 @@ func TestNextClearSignOfAttackAfterABanBansPermanently(t *testing.T) {
|
||||
attackBanDuration: "1h",
|
||||
})
|
||||
|
||||
// The first probe bans for SWWAF_ATTACK_BAN_DURATION.
|
||||
line := s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
// The first probe bans for SWW_ATTACK_BAN_DURATION.
|
||||
line := s.request(client, "/.env", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
if line.BanExpires != requestlog.FormatTime(clk.Now().Add(time.Hour)) {
|
||||
t.Errorf("log line has ban_expires %q, want an hour on", line.BanExpires)
|
||||
}
|
||||
@@ -112,7 +144,8 @@ func TestNextClearSignOfAttackAfterABanBansPermanently(t *testing.T) {
|
||||
clk.advance(time.Hour)
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line = s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
line = s.request(client, "/.env", http.StatusForbidden,
|
||||
requestlog.ActionPermanentlyBanned)
|
||||
if line.BanExpires != permanent {
|
||||
t.Errorf("log line has ban_expires %q, want permanent", line.BanExpires)
|
||||
}
|
||||
@@ -122,8 +155,8 @@ func TestRulesComeAfterTheOtherChecks(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
|
||||
exempt = "192.0.2.50" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
|
||||
allowed = "192.0.2.60" // in SWW_ALLOW_NETS
|
||||
exempt = "192.0.2.50" // in SWW_RATE_LIMIT_EXEMPT_NETS
|
||||
)
|
||||
|
||||
s, _, server := startWithClock(t, "", map[string]string{
|
||||
@@ -133,14 +166,15 @@ func TestRulesComeAfterTheOtherChecks(t *testing.T) {
|
||||
rateLimitPerMinute: "1",
|
||||
})
|
||||
|
||||
// A client in SWWAF_ALLOW_NETS is not checked.
|
||||
// A client in SWW_ALLOW_NETS is not checked.
|
||||
line := s.request(allowed, "/.env", http.StatusOK, requestlog.ActionForward)
|
||||
wantRuleIDs(t, line)
|
||||
|
||||
// A probe over the rate limit breaks the limit before any rule sees
|
||||
// it.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
line = s.request(client, "/.env", http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
line = s.request(client, "/.env", http.StatusForbidden,
|
||||
requestlog.ActionClientIsRateLimited)
|
||||
wantRuleIDs(t, line)
|
||||
|
||||
limitBan := server.Ledger.Bans(netip.MustParsePrefix(client + "/32"))
|
||||
@@ -150,7 +184,7 @@ func TestRulesComeAfterTheOtherChecks(t *testing.T) {
|
||||
|
||||
// A client the rate limits do not apply to is still checked.
|
||||
s.get(exempt, http.StatusOK, requestlog.ActionForward)
|
||||
s.request(exempt, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(exempt, "/.env", http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestObserveModeLogsWhatTheRulesWouldDo(t *testing.T) {
|
||||
@@ -162,11 +196,11 @@ func TestObserveModeLogsWhatTheRulesWouldDo(t *testing.T) {
|
||||
})
|
||||
|
||||
line := s.request(client, "/blocked", http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionRuleBlocked)
|
||||
wantWouldAction(t, line, requestlog.ActionBlockedByBlockRule)
|
||||
wantRuleIDs(t, line, "blocked")
|
||||
|
||||
line = s.request(client, "/.env", http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionBanned)
|
||||
wantWouldAction(t, line, requestlog.ActionTemporarilyBanned)
|
||||
wantRuleIDs(t, line, "probe")
|
||||
|
||||
if line.BanExpires != "" {
|
||||
@@ -190,9 +224,10 @@ func TestMetricsCountRuleMatchesAndBansForAnAttack(t *testing.T) {
|
||||
metricsToken: token,
|
||||
})
|
||||
|
||||
s.request(client, "/blocked", http.StatusForbidden, requestlog.ActionRuleBlocked)
|
||||
s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(client, "/blocked", http.StatusForbidden,
|
||||
requestlog.ActionBlockedByBlockRule)
|
||||
s.request(client, "/.env", http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionPermanentlyBanned)
|
||||
|
||||
metrics := s.scrape()
|
||||
wantMetric(t, metrics,
|
||||
@@ -200,10 +235,10 @@ func TestMetricsCountRuleMatchesAndBansForAnAttack(t *testing.T) {
|
||||
wantMetric(t, metrics,
|
||||
`sww_rule_matches_total{action="ban",instance="app",rule_id="probe"}`, 1)
|
||||
wantMetric(t, metrics, `sww_rules_loaded{instance="app"}`, 2)
|
||||
wantMetric(t, metrics, `sww_requests_total{action="rule_blocked",`+
|
||||
wantMetric(t, metrics, `sww_requests_total{action="blocked_by_block_rule",`+
|
||||
`instance="app",status_class="4xx"}`, 1)
|
||||
wantMetric(t, metrics,
|
||||
`sww_offences_total{instance="app",kind="rule_blocked"}`, 1)
|
||||
`sww_offences_total{instance="app",kind="blocked_by_block_rule"}`, 1)
|
||||
wantMetric(t, metrics, `sww_offences_total{instance="app",kind="attack"}`, 1)
|
||||
wantMetric(t, metrics, `sww_bans_made_total{cause="attack",instance="app"}`, 1)
|
||||
wantMetric(t, metrics, `sww_bans_made_total{cause="limit",instance="app"}`, 0)
|
||||
|
||||
@@ -1,64 +0,0 @@
|
||||
package discriminator
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// trapPath reports whether the request asks for a path in
|
||||
// SWWAF_TRAP_PATHS: its path as a path rule sees it, without the query,
|
||||
// as sent or once percent-decoded, is one of them. Such a request is a
|
||||
// clear sign of attack, as a ban rule's match is: it bans the client's
|
||||
// netblock, or in observe mode raises the alert for the ban it would have
|
||||
// made. The ban's notes give the path as sent.
|
||||
func (rq *request) trapPath(now time.Time) bool {
|
||||
path := rules.Path(rq.in)
|
||||
if !slices.Contains(rq.h.config.TrapPaths, path) &&
|
||||
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
|
||||
return false
|
||||
}
|
||||
|
||||
rq.attack = true
|
||||
rq.banForAttack(now, bans.Notes{TrapPath: path})
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// checkRules checks the request against the rules of the rule files at
|
||||
// now, notes the ids of those it matches in the log line, and returns the
|
||||
// action of the rule that refuses it, ActionRuleBlocked for a block rule
|
||||
// and ActionBanned for a ban rule, or "" when none does. A ban rule bans
|
||||
// the client's netblock for a clear sign of attack, or in observe mode
|
||||
// raises the alert for the ban it would have made. Either rule's match
|
||||
// is noted as an offence, for the client's history.
|
||||
func (rq *request) checkRules(now time.Time) string {
|
||||
matched := rq.h.rules.Match(rq.in)
|
||||
|
||||
for _, rule := range matched {
|
||||
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
|
||||
rq.h.metrics.RuleMatched(rule.ID, rule.Action)
|
||||
}
|
||||
|
||||
if len(matched) == 0 {
|
||||
return ""
|
||||
}
|
||||
|
||||
// Only the last rule matched can refuse the request.
|
||||
switch last := matched[len(matched)-1]; last.Action {
|
||||
case rules.ActionBlock:
|
||||
rq.ruleBlocked = true
|
||||
|
||||
return requestlog.ActionRuleBlocked
|
||||
case rules.ActionBan:
|
||||
rq.attack = true
|
||||
rq.banForAttack(now, bans.Notes{RuleID: last.ID, Target: last.Target})
|
||||
|
||||
return requestlog.ActionBanned
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
@@ -26,19 +26,18 @@ func TestAllowNetsSkipEveryCheck(t *testing.T) {
|
||||
calls.Add(1)
|
||||
})
|
||||
geojsURL, asked := startGeoJS(t)
|
||||
// fromKP is in SWWAF_ALLOW_NETS, and in SWWAF_DENY_NETS too, which
|
||||
// fromKP is in SWW_ALLOW_NETS, and in SWW_DENY_NETS too, which
|
||||
// comes after it.
|
||||
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
allowNets: "198.51.100.0/24",
|
||||
denyNets: fromKP,
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
rateLimitPerMinute: "1",
|
||||
})
|
||||
|
||||
// Neither SWWAF_DENY_NETS, the country lists nor the limit of one
|
||||
// request a minute refuses the client, and its country is not looked
|
||||
// up.
|
||||
// Neither SWW_DENY_NETS, SWW_TAG_DENY nor the limit of one request a
|
||||
// minute refuses the client, and its country is not looked up.
|
||||
wantAnswers(t, addr, out, []sentRequest{
|
||||
{fromKP, http.StatusOK, requestlog.ActionForward},
|
||||
{fromKP, http.StatusOK, requestlog.ActionForward},
|
||||
@@ -69,7 +68,7 @@ func TestRequestFromAllowNetsIsNotCounted(t *testing.T) {
|
||||
{listedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{listedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{unlistedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
|
||||
{unlistedAddr, http.StatusForbidden, requestlog.ActionClientIsRateLimited},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -83,9 +82,9 @@ func TestDenyNetsRefuseBeforeTheLookup(t *testing.T) {
|
||||
})
|
||||
geojsURL, asked := startGeoJS(t)
|
||||
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
denyNets: "203.0.113.0/24",
|
||||
deniedCountries: "kp",
|
||||
trustedProxies: trustLocalhost,
|
||||
denyNets: "203.0.113.0/24",
|
||||
tagDeny: denyKP,
|
||||
})
|
||||
|
||||
req := newRequest(t, http.MethodPost, addr, "/", strings.NewReader("a body"))
|
||||
@@ -93,7 +92,7 @@ func TestDenyNetsRefuseBeforeTheLookup(t *testing.T) {
|
||||
wantStatus(t, do(t, req), http.StatusForbidden)
|
||||
|
||||
line := out.endedLine(t)
|
||||
wantLine(t, line, http.StatusForbidden, requestlog.ActionDenied)
|
||||
wantLine(t, line, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
|
||||
if len(asked()) != 0 {
|
||||
t.Errorf("GeoJS was asked about %v, want nothing", asked())
|
||||
@@ -104,6 +103,15 @@ func TestDenyNetsRefuseBeforeTheLookup(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestDenyNetsRefusalNamesItsHeuristicAndNetblock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{denyNets: clientsNet})
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, "deny_nets", `{"netblock": "`+clientsNet+`"}`)
|
||||
}
|
||||
|
||||
func TestRequestRefusedByDenyNetsIsNotCounted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -117,10 +125,10 @@ func TestRequestRefusedByDenyNetsIsNotCounted(t *testing.T) {
|
||||
// listedAddr's refused requests are not counted, so the first request
|
||||
// from unlistedAddr is within the limit of one a minute.
|
||||
wantAnswers(t, addr, out, []sentRequest{
|
||||
{listedAddr, http.StatusForbidden, requestlog.ActionDenied},
|
||||
{listedAddr, http.StatusForbidden, requestlog.ActionDenied},
|
||||
{listedAddr, http.StatusForbidden, requestlog.ActionClientDenied},
|
||||
{listedAddr, http.StatusForbidden, requestlog.ActionClientDenied},
|
||||
{unlistedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
|
||||
{unlistedAddr, http.StatusForbidden, requestlog.ActionClientIsRateLimited},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -133,19 +141,19 @@ func TestRateLimitExemptNetsAreNeitherCountedNorRefused(t *testing.T) {
|
||||
trustedProxies: trustLocalhost,
|
||||
lookupTimeout: "1h",
|
||||
rateLimitExemptNets: listedAddr + "," + fromKP,
|
||||
deniedCountries: "kp",
|
||||
tagDeny: denyKP,
|
||||
rateLimitPerMinute: "1",
|
||||
})
|
||||
|
||||
// listedAddr's requests are neither refused nor counted, so the first
|
||||
// request from unlistedAddr is within the limit of one a minute. The
|
||||
// country lists still refuse an exempt client.
|
||||
// request from unlistedAddr is within the limit of one a minute.
|
||||
// SWW_TAG_DENY still refuses an exempt client.
|
||||
wantAnswers(t, addr, out, []sentRequest{
|
||||
{listedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{listedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{unlistedAddr, http.StatusOK, requestlog.ActionForward},
|
||||
{unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
|
||||
{fromKP, http.StatusForbidden, requestlog.ActionCountryDenied},
|
||||
{unlistedAddr, http.StatusForbidden, requestlog.ActionClientIsRateLimited},
|
||||
{fromKP, http.StatusForbidden, requestlog.ActionClientDenied},
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic tag_deny. It reads SWW_TAG_DENY.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// tagDeny refuses a request that carries a tag in SWW_TAG_DENY. Its figure is
|
||||
// the first such tag, in sorted order.
|
||||
type tagDeny struct{}
|
||||
|
||||
func (tagDeny) name() string { return "tag_deny" }
|
||||
|
||||
func (tagDeny) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
for _, tag := range rq.line.Tags {
|
||||
if slices.Contains(rq.h.config.TagDeny, tag) {
|
||||
return &verdict{
|
||||
action: requestlog.ActionClientDenied,
|
||||
figures: map[string]any{"tag": tag},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
package discriminator_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
func TestTagDenyRefusesALookupTagAndNamesTheTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// The setting takes a country code in either case; the request's tag is
|
||||
// in capitals.
|
||||
s, _, _ := startWithLookups(t, map[string]string{tagDeny: "Country=kp"})
|
||||
|
||||
line := s.get(fromKP, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, "tag_deny", `{"tag": "country=KP"}`)
|
||||
wantTags(t, line, "asn=64511", "country=KP")
|
||||
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
}
|
||||
|
||||
func TestTagDenyRefusesAnASNTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithLookups(t, map[string]string{tagDeny: "asn=64511"})
|
||||
|
||||
wantHeuristic(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionClientDenied),
|
||||
"tag_deny", `{"tag": "asn=64511"}`)
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
}
|
||||
|
||||
func TestTagDenyRefusesAListTagBeforeTheRateLimitsCount(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
tagLists: "torexit=" + torList, tagDeny: "torexit", rateLimitPerMinute: "1",
|
||||
})
|
||||
loadLists(t, server, map[string][]string{torList: {fromDE}})
|
||||
|
||||
// Each is refused for its tag: none is counted to break the limit of one.
|
||||
for range 3 {
|
||||
wantHeuristic(t, s.get(fromDE, http.StatusForbidden, requestlog.ActionClientDenied),
|
||||
"tag_deny", `{"tag": "torexit"}`)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagDenyRefusesAMixedCaseZoneByItsLowerCaseTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const mixedCaseZone = "DNSBL.example"
|
||||
|
||||
// The zone's tag is in lower case, however SWW_DNSBL_ZONES writes it.
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
dnsblZones: mixedCaseZone, dnsblResolver: noResolver,
|
||||
reputationAction: actionLog, tagDeny: "dnsbl=" + dnsblZone,
|
||||
})
|
||||
loadVerdicts(server, map[string][]string{fromDE: {mixedCaseZone}})
|
||||
|
||||
wantHeuristic(t, s.get(fromDE, http.StatusForbidden, requestlog.ActionClientDenied),
|
||||
"tag_deny", `{"tag": "dnsbl=`+dnsblZone+`"}`)
|
||||
}
|
||||
|
||||
func TestTagDenyRefusesAnExploitSignatureOfARuleThatOnlyLogs(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
rulesDir: writeRules(t, "noted path log ^/\n"), tagDeny: "exploitsignature",
|
||||
})
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
wantHeuristic(t, line, "tag_deny", `{"tag": "exploitsignature"}`)
|
||||
wantTags(t, line, "exploitsignature")
|
||||
}
|
||||
|
||||
func TestTagLimitPercentLowersALimitTheLowestWinning(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
rateLimitPerMinute: fourAMinute,
|
||||
tagLists: "vpn=" + vpnList + ",tor=" + torList,
|
||||
tagLimitPercent: "vpn:50,tor:25",
|
||||
})
|
||||
loadLists(t, server, map[string][]string{vpnList: {fromDE}, torList: {fromDE}})
|
||||
|
||||
// A quarter of 4 requests a minute: the second breaks the limit.
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line := s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
wantPercent(t, "rate_limit_percent",
|
||||
line.RateLimitPercent, line.RateLimitPercentSetting,
|
||||
"25 from "+tagLimitPercent)
|
||||
}
|
||||
|
||||
func TestTagLimitPercentLowersAByteLimitByAListTag(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
bytesLimitPerMinute: twoUploads,
|
||||
tagLists: "vpn=" + vpnList,
|
||||
tagLimitPercent: "vpn:50",
|
||||
})
|
||||
loadLists(t, server, map[string][]string{vpnList: {fromDE}})
|
||||
|
||||
// The upload's 100 bytes are over half of 199, 99.
|
||||
line := s.uploadFrom(fromDE)
|
||||
if line.LimitHit != minuteBytes {
|
||||
t.Errorf("log line has limit_hit %q, want %s", line.LimitHit, minuteBytes)
|
||||
}
|
||||
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting,
|
||||
"50 from "+tagLimitPercent)
|
||||
|
||||
// fromKP, which the list does not hold, has the whole limit.
|
||||
line = s.uploadFrom(fromKP)
|
||||
if line.LimitHit != "" {
|
||||
t.Errorf("log line for %s has limit_hit %q, want none", fromKP, line.LimitHit)
|
||||
}
|
||||
|
||||
wantPercent(t, "bytes_limit_percent",
|
||||
line.BytesLimitPercent, line.BytesLimitPercentSetting, none)
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic tag_lists. It reads the address lists of SWW_TAG_LISTS through
|
||||
// the reputation lists.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// tagLists adds to the request one tag for each list of SWW_TAG_LISTS that
|
||||
// holds the client, the tag being the list's name. It refuses nothing, and runs
|
||||
// first so that every refusal's log line carries these tags.
|
||||
type tagLists struct{}
|
||||
|
||||
func (tagLists) name() string { return "tag_lists" }
|
||||
|
||||
func (tagLists) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
for _, name := range rq.h.lists.TagsOf(rq.client) {
|
||||
rq.addTag(name)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
package discriminator_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// tagLists is SWW_TAG_LISTS, and vpnList and torList lists it names.
|
||||
const (
|
||||
tagLists = "SWW_TAG_LISTS"
|
||||
vpnList = "https://lists.example/vpn.txt"
|
||||
torList = "https://lists.example/torexit.txt"
|
||||
)
|
||||
|
||||
// wantTags checks the log line's tags.
|
||||
func wantTags(t *testing.T, line logLine, want ...string) {
|
||||
t.Helper()
|
||||
|
||||
got, _ := line.fields["tags"].([]any)
|
||||
|
||||
tags := make([]string, 0, len(got))
|
||||
for _, tag := range got {
|
||||
text, _ := tag.(string)
|
||||
tags = append(tags, text)
|
||||
}
|
||||
|
||||
if !slices.Equal(tags, want) {
|
||||
t.Errorf("log line has tags %v, want %v", tags, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLookupAndTagListsGiveTheRequestItsTagsSorted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
tagLists: "vpn=mullvad=" + vpnList + ",TorExit=" + torList,
|
||||
})
|
||||
loadLists(t, server, map[string][]string{
|
||||
vpnList: {fromDE}, torList: {fromDE + "/32", fromKP},
|
||||
})
|
||||
|
||||
wantTags(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||
"asn=64496", "country=DE", "torexit", "vpn=mullvad")
|
||||
wantTags(t, s.get(fromKP, http.StatusOK, requestlog.ActionForward),
|
||||
"asn=64511", "country=KP", "torexit")
|
||||
}
|
||||
|
||||
func TestRequestWithoutATagLeavesTheFieldOut(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// With no lookup, a client is not placed and no setting gives it a tag.
|
||||
s, _, _ := startWithClock(t, "", map[string]string{lookupSource: "off"})
|
||||
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
if _, present := line.fields["tags"]; present {
|
||||
t.Errorf("log line has tags %v, want none", line.fields["tags"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagListsTagEvenARequestTheCountryListRefuses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
tagLists: "vpn=" + vpnList, allowedCountries: "de",
|
||||
})
|
||||
loadLists(t, server, map[string][]string{vpnList: {fromKP}})
|
||||
|
||||
wantTags(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionCountryDenied),
|
||||
"asn=64511", "country=KP", "vpn")
|
||||
}
|
||||
|
||||
func TestTagListsTagARequestRefusedBeforeTheLookup(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
tagLists: "vpn=" + vpnList, denyNets: fromKP, rateLimitPerMinute: "1",
|
||||
})
|
||||
loadLists(t, server, map[string][]string{vpnList: {fromDE, fromKP}})
|
||||
|
||||
line := s.get(fromKP, http.StatusForbidden, requestlog.ActionClientDenied)
|
||||
if line.Heuristic != "deny_nets" {
|
||||
t.Errorf("log line has heuristic %q, want deny_nets", line.Heuristic)
|
||||
}
|
||||
|
||||
wantTags(t, line, "vpn")
|
||||
|
||||
// The second request breaks the limit of one a minute, which bans the client.
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||
|
||||
line = s.get(fromDE, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
if line.Heuristic != "active_ban" {
|
||||
t.Errorf("log line has heuristic %q, want active_ban", line.Heuristic)
|
||||
}
|
||||
|
||||
wantTags(t, line, "vpn")
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
package discriminator
|
||||
|
||||
// The heuristic trap_paths. It reads SWW_TRAP_PATHS.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// trapPaths refuses a request for a path in SWW_TRAP_PATHS: its path as a
|
||||
// path rule sees it, without the query, as sent or once percent-decoded, is one
|
||||
// of them. Such a request is a clear sign of attack, as a ban rule's match is:
|
||||
// it bans the client's netblock, or in observe mode raises the alert for the
|
||||
// ban it would have made. The ban's notes give the path as sent, and so does
|
||||
// its figure.
|
||||
type trapPaths struct{}
|
||||
|
||||
func (trapPaths) name() string { return "trap_paths" }
|
||||
|
||||
func (trapPaths) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
path := rules.Path(rq.in)
|
||||
if !slices.Contains(rq.h.config.TrapPaths, path) &&
|
||||
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.attack = true
|
||||
|
||||
return &verdict{
|
||||
action: rq.banForAttack(now, bans.Notes{TrapPath: path}),
|
||||
figures: map[string]any{"path": path},
|
||||
}
|
||||
}
|
||||
@@ -14,14 +14,14 @@ import (
|
||||
// trapPaths is the setting's name, and trapPathList what the tests set it
|
||||
// to.
|
||||
const (
|
||||
trapPaths = "SWWAF_TRAP_PATHS"
|
||||
trapPaths = "SWW_TRAP_PATHS"
|
||||
trapPathList = "/wp-login.php,/xmlrpc.php"
|
||||
)
|
||||
|
||||
func TestTrapPathBansAsABanRuleDoes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
|
||||
const allowed = "192.0.2.60" // in SWW_ALLOW_NETS
|
||||
|
||||
// A block rule for the same path: the trap path comes first.
|
||||
s, clk, server := startWithClock(t, "", map[string]string{
|
||||
@@ -38,12 +38,12 @@ func TestTrapPathBansAsABanRuleDoes(t *testing.T) {
|
||||
s.request(otherClient, path, http.StatusOK, requestlog.ActionForward)
|
||||
}
|
||||
|
||||
// A client in SWWAF_ALLOW_NETS is not checked.
|
||||
// A client in SWW_ALLOW_NETS is not checked.
|
||||
s.request(allowed, "/xmlrpc.php", http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
// The query is not part of the path.
|
||||
line := s.request(client, "/wp-login.php?redirect_to=x", http.StatusTooManyRequests,
|
||||
requestlog.ActionBanned)
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
wantRuleIDs(t, line)
|
||||
|
||||
if line.BanExpires != requestlog.FormatTime(start.Add(7*24*time.Hour)) {
|
||||
@@ -77,12 +77,22 @@ func TestTrapPathBansAsABanRuleDoes(t *testing.T) {
|
||||
}
|
||||
|
||||
// The next request is refused under the ban, and makes it permanent.
|
||||
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
|
||||
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionPermanentlyBanned)
|
||||
if line.BanExpires != permanent {
|
||||
t.Errorf("log line has ban_expires %q, want permanent", line.BanExpires)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTrapPathRefusalNamesItsHeuristicAndPath(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{trapPaths: trapPathList})
|
||||
|
||||
line := s.request(client, "/wp-login.php?redirect_to=x", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
wantHeuristic(t, line, "trap_paths", `{"path": "/wp-login.php"}`)
|
||||
}
|
||||
|
||||
func TestTrapPathMatchesThePathDecodedOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -90,7 +100,8 @@ func TestTrapPathMatchesThePathDecodedOnce(t *testing.T) {
|
||||
|
||||
// %2577 decodes once to %77, not to w.
|
||||
s.request(otherClient, "/%2577p-login.php", http.StatusOK, requestlog.ActionForward)
|
||||
s.request(client, "/%77p-login.php", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(client, "/%77p-login.php", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
|
||||
// The ban's notes give the path as the client sent it.
|
||||
got := server.Ledger.Bans(netip.MustParsePrefix(client + "/32"))
|
||||
@@ -103,11 +114,12 @@ func TestTrapPathsNeedNoRuleFiles(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
trapPaths: trapPathList,
|
||||
"SWWAF_RULES_ENABLED": "false",
|
||||
trapPaths: trapPathList,
|
||||
"SWW_RULES_ENABLED": "false",
|
||||
})
|
||||
|
||||
s.request(client, "/xmlrpc.php", http.StatusForbidden, requestlog.ActionBanned)
|
||||
s.request(client, "/xmlrpc.php", http.StatusForbidden,
|
||||
requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
func TestObserveModeLogsWhatATrapPathWouldDo(t *testing.T) {
|
||||
@@ -119,7 +131,7 @@ func TestObserveModeLogsWhatATrapPathWouldDo(t *testing.T) {
|
||||
})
|
||||
|
||||
line := s.request(client, "/xmlrpc.php", http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionBanned)
|
||||
wantWouldAction(t, line, requestlog.ActionTemporarilyBanned)
|
||||
|
||||
// No ban was made.
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
+17
-7
@@ -24,7 +24,7 @@ const quietTime = 2 * time.Second
|
||||
// FileParams are what OpenFile needs.
|
||||
type FileParams struct {
|
||||
// Path is the lookup database, the IPinfo Lite file in its .mmdb form
|
||||
// (SWWAF_LOOKUP_DB_PATH).
|
||||
// (SWW_LOOKUP_DB_PATH).
|
||||
Path string
|
||||
// Now tells the time, normally time.Now.
|
||||
Now func() time.Time
|
||||
@@ -133,8 +133,11 @@ func (f *File) Watch(ctx context.Context) {
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
f.params.ProcessLog.Error("cannot watch the lookup database for replacements",
|
||||
"error", err.Error())
|
||||
f.params.ProcessLog.Error(
|
||||
"cannot watch the lookup database for replacements",
|
||||
"error",
|
||||
err.Error(),
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
@@ -194,7 +197,10 @@ func (f *File) readAgain() {
|
||||
f.params.Alerts.Raise(alerts.Alert{
|
||||
Event: alerts.EventFileError,
|
||||
Reason: kept,
|
||||
Detail: map[string]any{"file": f.params.Path, "error": err.Error()},
|
||||
Detail: map[string]any{
|
||||
"file": f.params.Path,
|
||||
"error": err.Error(),
|
||||
},
|
||||
})
|
||||
f.params.ProcessLog.Error(kept, "error", err.Error())
|
||||
|
||||
@@ -212,7 +218,11 @@ func (f *File) use(reader *maxminddb.Reader) {
|
||||
f.lastRead = f.params.Now()
|
||||
f.mu.Unlock()
|
||||
|
||||
f.params.ProcessLog.Info("read the lookup database", "file", f.params.Path)
|
||||
f.params.ProcessLog.Info(
|
||||
"read the lookup database",
|
||||
"file",
|
||||
f.params.Path,
|
||||
)
|
||||
}
|
||||
|
||||
// read reads the lookup database at path. The whole file is read into
|
||||
@@ -221,12 +231,12 @@ func (f *File) use(reader *maxminddb.Reader) {
|
||||
func read(path string) (*maxminddb.Reader, error) {
|
||||
data, err := os.ReadFile(path) //nolint:gosec // the file the admin names
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("SWWAF_LOOKUP_DB_PATH cannot be read: %w", err)
|
||||
return nil, fmt.Errorf("SWW_LOOKUP_DB_PATH cannot be read: %w", err)
|
||||
}
|
||||
|
||||
reader, err := maxminddb.OpenBytes(data)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("SWWAF_LOOKUP_DB_PATH %s is not a .mmdb file: %w", path, err)
|
||||
return nil, fmt.Errorf("SWW_LOOKUP_DB_PATH %s is not a .mmdb file: %w", path, err)
|
||||
}
|
||||
|
||||
return reader, nil
|
||||
|
||||
@@ -99,9 +99,9 @@ func TestFileThatCannotBeReadIsAnError(t *testing.T) {
|
||||
writeFile(t, notDatabase, "not a lookup database\n")
|
||||
|
||||
for path, want := range map[string]string{
|
||||
missing: "SWWAF_LOOKUP_DB_PATH cannot be read: open " + missing +
|
||||
missing: "SWW_LOOKUP_DB_PATH cannot be read: open " + missing +
|
||||
": no such file or directory",
|
||||
notDatabase: "SWWAF_LOOKUP_DB_PATH " + notDatabase +
|
||||
notDatabase: "SWW_LOOKUP_DB_PATH " + notDatabase +
|
||||
" is not a .mmdb file: error opening database: invalid MaxMind DB file",
|
||||
} {
|
||||
_, err := OpenFile(FileParams{
|
||||
@@ -216,7 +216,7 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(t *testing.T)
|
||||
Reason: "the lookup database cannot be read, and the one read before stays in use",
|
||||
Detail: map[string]any{
|
||||
"file": path,
|
||||
"error": "SWWAF_LOOKUP_DB_PATH " + path + " is not a .mmdb file: " +
|
||||
"error": "SWW_LOOKUP_DB_PATH " + path + " is not a .mmdb file: " +
|
||||
"error opening database: invalid MaxMind DB file",
|
||||
},
|
||||
})
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// Package lookup looks up each client's AS number and country, through
|
||||
// the GeoJS web service or in the lookup database, the IPinfo Lite file
|
||||
// SWWAF_LOOKUP_DB_PATH names. GeoJS's answers are kept in memory, for at
|
||||
// SWW_LOOKUP_DB_PATH names. GeoJS's answers are kept in memory, for at
|
||||
// most 100,000 clients and for 7 days each, and are written to
|
||||
// lookups.json and read from it by the state package.
|
||||
package lookup
|
||||
@@ -66,7 +66,7 @@ type Params struct {
|
||||
URL string
|
||||
// Timeout is how long a request waits for its client's first answer,
|
||||
// and how long a request to GeoJS may take before it is abandoned
|
||||
// (SWWAF_LOOKUP_TIMEOUT).
|
||||
// (SWW_LOOKUP_TIMEOUT).
|
||||
Timeout time.Duration
|
||||
// Wait is true when a setting needs each request's answer before the
|
||||
// request goes on. Otherwise no request waits for one.
|
||||
@@ -198,7 +198,7 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
|
||||
|
||||
answer, found := g.kept(client)
|
||||
if !found {
|
||||
g.metrics.GeoJSUnanswered.Inc()
|
||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
||||
}
|
||||
|
||||
w, waiting := g.waiting[client]
|
||||
@@ -296,7 +296,7 @@ func (g *GeoJS) answerOrWait(
|
||||
}
|
||||
|
||||
if w == nil {
|
||||
g.metrics.GeoJSUnanswered.Inc()
|
||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
||||
|
||||
return Answer{}, nil // too many clients wait already
|
||||
}
|
||||
@@ -307,7 +307,7 @@ func (g *GeoJS) answerOrWait(
|
||||
}
|
||||
|
||||
if w.late {
|
||||
g.metrics.GeoJSUnanswered.Inc()
|
||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
||||
|
||||
return Answer{}, nil
|
||||
}
|
||||
|
||||
@@ -846,7 +846,7 @@ func wantAsked(t *testing.T, geojs *standIn, i int, want ...string) {
|
||||
func wantUnanswered(t *testing.T, m *metrics.Metrics, want float64) {
|
||||
t.Helper()
|
||||
|
||||
got := testutil.ToFloat64(m.GeoJSUnanswered)
|
||||
got := testutil.ToFloat64(m.RequestsWithoutGeoJSAnswer)
|
||||
if got != want {
|
||||
t.Errorf("%v requests went without an answer, want %v", got, want)
|
||||
}
|
||||
|
||||
@@ -25,7 +25,7 @@ type busiest struct {
|
||||
requests *prometheus.CounterVec
|
||||
requestBytes *prometheus.CounterVec
|
||||
responseBytes *prometheus.CounterVec
|
||||
// refused are, by country, the requests the country lists refused; nil
|
||||
// refused are, by country, the requests the country list refused; nil
|
||||
// by AS number.
|
||||
refused *prometheus.CounterVec
|
||||
|
||||
@@ -42,7 +42,7 @@ type busiest struct {
|
||||
func newCountries(topN int) *busiest {
|
||||
countries := newBusiest(topN, "country", "the client's country")
|
||||
countries.refused = counterVec("sww_country_list_refusals_total",
|
||||
"Requests the country lists refused, by the client's country.",
|
||||
"Requests SWW_EXCLUSIVELY_ALLOWED_COUNTRIES refused, by the client's country.",
|
||||
[]string{"country"})
|
||||
|
||||
return countries
|
||||
@@ -88,7 +88,7 @@ func (b *busiest) Collect(ch chan<- prometheus.Metric) {
|
||||
}
|
||||
|
||||
// vecs returns the metrics: by AS number, those of requests and bytes; by
|
||||
// country, the refusals by the country lists as well.
|
||||
// country, the refusals by the country list as well.
|
||||
func (b *busiest) vecs() []*prometheus.CounterVec {
|
||||
vecs := []*prometheus.CounterVec{b.requests, b.requestBytes, b.responseBytes}
|
||||
if b.refused != nil {
|
||||
|
||||
+39
-39
@@ -28,15 +28,15 @@ type Metrics struct {
|
||||
registry prometheus.Registerer
|
||||
handler http.Handler
|
||||
|
||||
inFlight prometheus.Gauge
|
||||
requests *prometheus.CounterVec
|
||||
requestBytes *prometheus.CounterVec
|
||||
responseBytes *prometheus.CounterVec
|
||||
requestDuration prometheus.Histogram
|
||||
upstreamDuration prometheus.Histogram
|
||||
rateLimitHits *prometheus.CounterVec
|
||||
offences *prometheus.CounterVec
|
||||
wafMatches *prometheus.CounterVec
|
||||
inFlight prometheus.Gauge
|
||||
requests *prometheus.CounterVec
|
||||
requestBytes *prometheus.CounterVec
|
||||
responseBytes *prometheus.CounterVec
|
||||
requestDuration prometheus.Histogram
|
||||
upstreamDuration prometheus.Histogram
|
||||
rateLimitHits *prometheus.CounterVec
|
||||
offences *prometheus.CounterVec
|
||||
coreRuleSetMatches *prometheus.CounterVec
|
||||
// ruleMatches are made by AddRules, and reputationHits by
|
||||
// AddReputation.
|
||||
ruleMatches *prometheus.CounterVec
|
||||
@@ -45,12 +45,12 @@ type Metrics struct {
|
||||
asns *busiest
|
||||
|
||||
// GeoJSRequests are the requests to GeoJS, and GeoJSFailures those
|
||||
// that failed. GeoJSUnanswered are the requests that needed their
|
||||
// client's answer, for a setting that acts on it, and went on without
|
||||
// it because GeoJS had not given it in time.
|
||||
GeoJSRequests prometheus.Counter
|
||||
GeoJSFailures prometheus.Counter
|
||||
GeoJSUnanswered prometheus.Counter
|
||||
// that failed. RequestsWithoutGeoJSAnswer are the requests that needed
|
||||
// their client's answer, for a setting that acts on it, and went on
|
||||
// without it because GeoJS had not given it in time.
|
||||
GeoJSRequests prometheus.Counter
|
||||
GeoJSFailures prometheus.Counter
|
||||
RequestsWithoutGeoJSAnswer prometheus.Counter
|
||||
|
||||
stateFileWrites *prometheus.CounterVec
|
||||
stateFileWriteFailures *prometheus.CounterVec
|
||||
@@ -62,8 +62,8 @@ type Metrics struct {
|
||||
|
||||
// New returns the metrics, with the Go runtime's and the process's own.
|
||||
// topN is how many countries and how many AS numbers get series of their
|
||||
// own (SWWAF_METRICS_TOP_N). Every metric carries instanceName
|
||||
// (SWWAF_INSTANCE_NAME) as its label instance.
|
||||
// own (SWW_METRICS_TOP_N). Every metric carries instanceName
|
||||
// (SWW_INSTANCE_NAME) as its label instance.
|
||||
//
|
||||
//nolint:funlen // a few lines for each metric, a list that grows with them
|
||||
func New(topN int, instanceName string) *Metrics {
|
||||
@@ -101,8 +101,8 @@ func New(topN int, instanceName string) *Metrics {
|
||||
[]string{"window", "kind"}),
|
||||
offences: counterVec("sww_offences_total",
|
||||
"Offences, by kind.", []string{"kind"}),
|
||||
wafMatches: counterVec("sww_waf_matches_total",
|
||||
"Requests that matched a rule of the Core Rule Set, by SWWAF_WAF_MODE "+
|
||||
coreRuleSetMatches: counterVec("sww_core_rule_set_matches_total",
|
||||
"Requests that matched a rule of the Core Rule Set, by SWW_WAF_MODE "+
|
||||
"and the rule's id.", []string{"mode", "rule_id"}),
|
||||
countries: newCountries(topN),
|
||||
asns: newASNs(topN),
|
||||
@@ -114,8 +114,8 @@ func New(topN int, instanceName string) *Metrics {
|
||||
Name: "sww_geojs_failures_total",
|
||||
Help: "Requests to GeoJS that failed.",
|
||||
}),
|
||||
GeoJSUnanswered: prometheus.NewCounter(prometheus.CounterOpts{
|
||||
Name: "sww_geojs_unanswered_total",
|
||||
RequestsWithoutGeoJSAnswer: prometheus.NewCounter(prometheus.CounterOpts{
|
||||
Name: "sww_requests_without_geojs_answer_total",
|
||||
Help: "Requests that needed their client's answer from GeoJS and " +
|
||||
"went on without it, because GeoJS had not given it in time.",
|
||||
}),
|
||||
@@ -139,9 +139,9 @@ func New(topN int, instanceName string) *Metrics {
|
||||
collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}),
|
||||
m.inFlight, m.requests, m.requestBytes, m.responseBytes,
|
||||
m.requestDuration, m.upstreamDuration,
|
||||
m.rateLimitHits, m.offences, m.wafMatches,
|
||||
m.rateLimitHits, m.offences, m.coreRuleSetMatches,
|
||||
m.countries, m.asns,
|
||||
m.GeoJSRequests, m.GeoJSFailures, m.GeoJSUnanswered,
|
||||
m.GeoJSRequests, m.GeoJSFailures, m.RequestsWithoutGeoJSAnswer,
|
||||
m.stateFileWrites, m.stateFileWriteFailures,
|
||||
m.stateFileLastWrite, m.stateFileSize,
|
||||
m.stateFileEditsTakenIn, m.stateFileEditsSetAside,
|
||||
@@ -214,13 +214,13 @@ func (m *Metrics) AddRules(ruleFiles *rules.Files) {
|
||||
}
|
||||
|
||||
// AddRemoteLog adds the metrics of sending the log lines to
|
||||
// SWWAF_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
|
||||
// SWW_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
|
||||
// lines sent, those dropped, and those waiting in the buffer.
|
||||
func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
|
||||
m.registry.MustRegister(
|
||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||
Name: "sww_remote_log_lines_sent_total",
|
||||
Help: "Log lines sent to SWWAF_LOG_REMOTE_URL.",
|
||||
Help: "Log lines sent to SWW_LOG_REMOTE_URL.",
|
||||
}, func() float64 {
|
||||
return float64(remote.Sent())
|
||||
}),
|
||||
@@ -232,7 +232,7 @@ func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
|
||||
return float64(remote.Dropped())
|
||||
}),
|
||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||
Name: "sww_remote_log_buffer_depth",
|
||||
Name: "sww_remote_log_lines_waiting",
|
||||
Help: "Log lines in the buffer, waiting to be sent.",
|
||||
}, func() float64 {
|
||||
return float64(remote.Depth())
|
||||
@@ -313,7 +313,7 @@ func (m *Metrics) AddReputation(lists *reputation.Lists, dnsbl *reputation.DNSBL
|
||||
// AddAbuseIPDB adds the metrics of AbuseIPDB, with the source abuseipdb,
|
||||
// read from abuseIPDB as the metrics are asked for: the checks made, those
|
||||
// that failed, and how many checks the day's budget has left. It is
|
||||
// called once, after AddReputation, while SWWAF_ABUSEIPDB_KEY is set.
|
||||
// called once, after AddReputation, while SWW_ABUSEIPDB_KEY is set.
|
||||
func (m *Metrics) AddAbuseIPDB(abuseIPDB *reputation.AbuseIPDB) {
|
||||
source := prometheus.Labels{sourceLabel: reputation.AbuseIPDBSource}
|
||||
|
||||
@@ -322,7 +322,7 @@ func (m *Metrics) AddAbuseIPDB(abuseIPDB *reputation.AbuseIPDB) {
|
||||
m.registry.MustRegister(
|
||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||
Name: "sww_reputation_daily_budget_remaining",
|
||||
Help: "Checks of the day's SWWAF_ABUSEIPDB_DAILY_BUDGET not yet spent.",
|
||||
Help: "Checks of the day's SWW_ABUSEIPDB_DAILY_BUDGET not yet spent.",
|
||||
ConstLabels: source,
|
||||
}, func() float64 {
|
||||
return float64(abuseIPDB.BudgetLeft())
|
||||
@@ -355,7 +355,7 @@ func (m *Metrics) AddAlerts(queue *alerts.Queue) {
|
||||
return float64(queue.Counts(name).Sent)
|
||||
}),
|
||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||
Name: "sww_alerts_failed_total",
|
||||
Name: "sww_alert_send_failures_total",
|
||||
Help: "Requests to the destination that failed.",
|
||||
ConstLabels: destination,
|
||||
}, func() float64 {
|
||||
@@ -363,8 +363,8 @@ func (m *Metrics) AddAlerts(queue *alerts.Queue) {
|
||||
}),
|
||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||
Name: "sww_alerts_suppressed_total",
|
||||
Help: "Alerts held back: repeats within SWWAF_ALERT_COOLDOWN, and " +
|
||||
"alerts past SWWAF_ALERT_MAX_PER_HOUR, for the hour's summary.",
|
||||
Help: "Alerts held back: repeats within SWW_ALERT_COOLDOWN, and " +
|
||||
"alerts past SWW_ALERT_MAX_PER_HOUR, for the hour's summary.",
|
||||
ConstLabels: destination,
|
||||
}, func() float64 {
|
||||
return float64(queue.Suppressed())
|
||||
@@ -424,10 +424,10 @@ func (m *Metrics) LimitHit(hit ratelimit.Hit) {
|
||||
// them.
|
||||
func (m *Metrics) Offences(r ratelimit.Request) {
|
||||
for kind, committed := range map[string]bool{
|
||||
"limit": r.BrokeLimit,
|
||||
"attack": r.Attack,
|
||||
"rule_blocked": r.RuleBlocked,
|
||||
"waf_blocked": r.WAFBlocked,
|
||||
"limit": r.BrokeLimit,
|
||||
"attack": r.Attack,
|
||||
"blocked_by_block_rule": r.BlockedByBlockRule,
|
||||
"blocked_by_core_rule_set": r.BlockedByCoreRuleSet,
|
||||
} {
|
||||
if committed {
|
||||
m.offences.WithLabelValues(kind).Inc()
|
||||
@@ -441,10 +441,10 @@ func (m *Metrics) RuleMatched(id, action string) {
|
||||
m.ruleMatches.WithLabelValues(id, action).Inc()
|
||||
}
|
||||
|
||||
// WAFMatched counts a request that matched the Core Rule Set's rule id,
|
||||
// with SWWAF_WAF_MODE at mode.
|
||||
func (m *Metrics) WAFMatched(mode string, id int) {
|
||||
m.wafMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc()
|
||||
// CoreRuleSetMatched counts a request that matched the Core Rule Set's rule id,
|
||||
// with SWW_WAF_MODE at mode.
|
||||
func (m *Metrics) CoreRuleSetMatched(mode string, id int) {
|
||||
m.coreRuleSetMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc()
|
||||
}
|
||||
|
||||
// StateFileWritten counts a write of the state file name, of size bytes,
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
// and bytes counted over a minute, an hour and a day, as the "Counting
|
||||
// method" section of SPEC.md describes, which tell when a request takes
|
||||
// the client over a rate limit or a byte limit, and each client's history
|
||||
// since it was first seen. At most SWWAF_MAX_TRACKED_CLIENTS clients are
|
||||
// since it was first seen. At most SWW_MAX_TRACKED_CLIENTS clients are
|
||||
// kept, in memory, and written to clients.json and read from it by the
|
||||
// state package.
|
||||
package ratelimit
|
||||
@@ -122,12 +122,13 @@ type Responses struct {
|
||||
type Offences struct {
|
||||
// Limit is its requests that broke a rate limit, a byte limit or the
|
||||
// error burst, Attack those that were a clear sign of attack, a match
|
||||
// of a ban rule or a request for a trap path, RuleBlocked those a block
|
||||
// rule refused, and WAFBlocked those the Core Rule Set refused.
|
||||
Limit int64 `json:"limit"`
|
||||
Attack int64 `json:"attack"`
|
||||
RuleBlocked int64 `json:"rule_blocked"`
|
||||
WAFBlocked int64 `json:"waf_blocked"`
|
||||
// of a ban rule or a request for a trap path, BlockedByBlockRule those a
|
||||
// block rule refused, and BlockedByCoreRuleSet those the Core Rule Set
|
||||
// refused.
|
||||
Limit int64 `json:"limit"`
|
||||
Attack int64 `json:"attack"`
|
||||
BlockedByBlockRule int64 `json:"blocked_by_block_rule"`
|
||||
BlockedByCoreRuleSet int64 `json:"blocked_by_core_rule_set"`
|
||||
// TokenRefused is what a clients.json written by an earlier version gives,
|
||||
// which must still load. Nothing adds to it.
|
||||
TokenRefused int64 `json:"token_refused,omitempty"`
|
||||
@@ -147,16 +148,16 @@ type Request struct {
|
||||
ResponseBytes int64
|
||||
// BrokeLimit is true for a request that broke a rate limit, a byte
|
||||
// limit or the error burst, Attack for one that matched a ban rule or
|
||||
// asked for a trap path, RuleBlocked for one a block rule refused, and
|
||||
// WAFBlocked for one the Core Rule Set refused.
|
||||
BrokeLimit bool
|
||||
Attack bool
|
||||
RuleBlocked bool
|
||||
WAFBlocked bool
|
||||
// asked for a trap path, BlockedByBlockRule for one a block rule refused,
|
||||
// and BlockedByCoreRuleSet for one the Core Rule Set refused.
|
||||
BrokeLimit bool
|
||||
Attack bool
|
||||
BlockedByBlockRule bool
|
||||
BlockedByCoreRuleSet bool
|
||||
}
|
||||
|
||||
// New returns a Limiter for limits, with no client counted yet, whose
|
||||
// table holds at most maxClients clients (SWWAF_MAX_TRACKED_CLIENTS). Past
|
||||
// table holds at most maxClients clients (SWW_MAX_TRACKED_CLIENTS). Past
|
||||
// it, the least recently seen client is dropped, with its history, and
|
||||
// starts afresh if it comes back.
|
||||
func New(limits Limits, maxClients int) *Limiter {
|
||||
@@ -323,12 +324,12 @@ func (l *Limiter) AddToHistory(client netip.Prefix, now time.Time, r Request) {
|
||||
h.Offences.Attack++
|
||||
}
|
||||
|
||||
if r.RuleBlocked {
|
||||
h.Offences.RuleBlocked++
|
||||
if r.BlockedByBlockRule {
|
||||
h.Offences.BlockedByBlockRule++
|
||||
}
|
||||
|
||||
if r.WAFBlocked {
|
||||
h.Offences.WAFBlocked++
|
||||
if r.BlockedByCoreRuleSet {
|
||||
h.Offences.BlockedByCoreRuleSet++
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ import (
|
||||
const limit = 3
|
||||
|
||||
// tableSize is the most clients the tests' tables hold, the default of
|
||||
// SWWAF_MAX_TRACKED_CLIENTS.
|
||||
// SWW_MAX_TRACKED_CLIENTS.
|
||||
const tableSize = 20000
|
||||
|
||||
// whole is the percentage of each limit a client gets when nothing lowers
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
// Package remotelog sends the lines sww writes on stdout to the
|
||||
// remote log endpoint, SWWAF_LOG_REMOTE_URL, as the "Request log" section
|
||||
// remote log endpoint, SWW_LOG_REMOTE_URL, as the "Request log" section
|
||||
// of SPEC.md describes: each line as the message of an RFC 5424 syslog
|
||||
// record, over UDP, TCP or TLS. Lines wait in a bounded buffer, so a slow
|
||||
// or unreachable endpoint never holds up a request or stdout.
|
||||
@@ -24,7 +24,7 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// The forms of SWWAF_LOG_REMOTE_URL, by its scheme.
|
||||
// The forms of SWW_LOG_REMOTE_URL, by its scheme.
|
||||
const (
|
||||
SchemeUDP = "syslog+udp"
|
||||
SchemeTCP = "syslog+tcp"
|
||||
@@ -56,18 +56,18 @@ const (
|
||||
|
||||
// Params are what New needs.
|
||||
type Params struct {
|
||||
// URL is the endpoint (SWWAF_LOG_REMOTE_URL): SchemeUDP, SchemeTCP or
|
||||
// URL is the endpoint (SWW_LOG_REMOTE_URL): SchemeUDP, SchemeTCP or
|
||||
// SchemeTLS, a host and a port.
|
||||
URL *url.URL
|
||||
// RootCAs are the certificates a SchemeTLS endpoint's certificate
|
||||
// must chain to (SWWAF_LOG_REMOTE_TLS_CA_FILE), nil for the host's.
|
||||
// must chain to (SWW_LOG_REMOTE_TLS_CA_FILE), nil for the host's.
|
||||
RootCAs *x509.CertPool
|
||||
// Buffer is the most lines held while they wait to be sent
|
||||
// (SWWAF_LOG_REMOTE_BUFFER).
|
||||
// (SWW_LOG_REMOTE_BUFFER).
|
||||
Buffer int
|
||||
// Facility is the number of the records' syslog facility
|
||||
// (SWWAF_LOG_REMOTE_FACILITY), and AppName their APP-NAME
|
||||
// (SWWAF_LOG_REMOTE_APP_NAME).
|
||||
// (SWW_LOG_REMOTE_FACILITY), and AppName their APP-NAME
|
||||
// (SWW_LOG_REMOTE_APP_NAME).
|
||||
Facility int
|
||||
AppName string
|
||||
}
|
||||
@@ -243,7 +243,7 @@ func (s *Sender) send(ctx context.Context, processLog *slog.Logger) net.Conn {
|
||||
}
|
||||
}
|
||||
|
||||
processLog.Warn("sending to SWWAF_LOG_REMOTE_URL failed",
|
||||
processLog.Warn("sending to SWW_LOG_REMOTE_URL failed",
|
||||
"error", err.Error(), "connecting_again_in", delay.String())
|
||||
|
||||
select {
|
||||
|
||||
@@ -586,7 +586,7 @@ func wantRetries(t *testing.T, logged *output, delays ...string) {
|
||||
var fields map[string]any
|
||||
|
||||
err := json.Unmarshal([]byte(line), &fields)
|
||||
if err != nil || fields["msg"] != "sending to SWWAF_LOG_REMOTE_URL failed" {
|
||||
if err != nil || fields["msg"] != "sending to SWW_LOG_REMOTE_URL failed" {
|
||||
t.Fatalf("logged %q", line)
|
||||
}
|
||||
|
||||
|
||||
@@ -58,17 +58,17 @@ type Checks struct {
|
||||
// AbuseIPDBParams are what NewAbuseIPDB needs.
|
||||
type AbuseIPDBParams struct {
|
||||
// URL is where clients are checked, normally AbuseIPDBURL, with Key,
|
||||
// the account's key (SWWAF_ABUSEIPDB_KEY).
|
||||
// the account's key (SWW_ABUSEIPDB_KEY).
|
||||
URL string
|
||||
Key string
|
||||
// MinScore is the least score that is a hit (SWWAF_ABUSEIPDB_MIN_SCORE),
|
||||
// MinScore is the least score that is a hit (SWW_ABUSEIPDB_MIN_SCORE),
|
||||
// and DailyBudget the most checks made in a day, in UTC
|
||||
// (SWWAF_ABUSEIPDB_DAILY_BUDGET).
|
||||
// (SWW_ABUSEIPDB_DAILY_BUDGET).
|
||||
MinScore int64
|
||||
DailyBudget int
|
||||
// CacheTTL is how long a score is used after it was fetched
|
||||
// (SWWAF_REPUTATION_CACHE_TTL), and Timeout how long a check may take
|
||||
// (SWWAF_REPUTATION_TIMEOUT).
|
||||
// (SWW_REPUTATION_CACHE_TTL), and Timeout how long a check may take
|
||||
// (SWW_REPUTATION_TIMEOUT).
|
||||
CacheTTL time.Duration
|
||||
Timeout time.Duration
|
||||
// Now tells the time, normally time.Now in UTC.
|
||||
|
||||
@@ -53,15 +53,15 @@ type Verdict struct {
|
||||
// DNSBLParams are what NewDNSBL needs.
|
||||
type DNSBLParams struct {
|
||||
// Zones are the DNSBL zones clients are asked about in
|
||||
// (SWWAF_DNSBL_ZONES).
|
||||
// (SWW_DNSBL_ZONES).
|
||||
Zones []string
|
||||
// Resolver is the resolver they are asked through
|
||||
// (SWWAF_DNSBL_RESOLVER), or, while it is the zero AddrPort, the
|
||||
// (SWW_DNSBL_RESOLVER), or, while it is the zero AddrPort, the
|
||||
// host's, as /etc/resolv.conf names it.
|
||||
Resolver netip.AddrPort
|
||||
// CacheTTL is how long a verdict is used after it was fetched
|
||||
// (SWWAF_REPUTATION_CACHE_TTL), and Timeout how long a query may take
|
||||
// (SWWAF_REPUTATION_TIMEOUT).
|
||||
// (SWW_REPUTATION_CACHE_TTL), and Timeout how long a query may take
|
||||
// (SWW_REPUTATION_TIMEOUT).
|
||||
CacheTTL time.Duration
|
||||
Timeout time.Duration
|
||||
// Now tells the time, normally time.Now in UTC.
|
||||
@@ -127,13 +127,13 @@ func NewDNSBL(params DNSBLParams) *DNSBL {
|
||||
}
|
||||
}
|
||||
|
||||
// Zones returns the zones, in the order SWWAF_DNSBL_ZONES names them.
|
||||
// Zones returns the zones, in the order SWW_DNSBL_ZONES names them.
|
||||
func (d *DNSBL) Zones() []string {
|
||||
return slices.Clone(d.params.Zones)
|
||||
}
|
||||
|
||||
// ListedBy returns the zones whose verdict on addr, a client's address,
|
||||
// lists it, in the order SWWAF_DNSBL_ZONES names them, each with its key
|
||||
// lists it, in the order SWW_DNSBL_ZONES names them, each with its key
|
||||
// masked, as config.MaskZoneKey masks it, since they go to the request
|
||||
// log, the alerts and the metrics. A verdict is used until CacheTTL has
|
||||
// passed since it was fetched. Each zone without one is asked about addr
|
||||
@@ -210,7 +210,7 @@ func (d *DNSBL) Snapshot() []Verdict {
|
||||
}
|
||||
|
||||
// Load keeps verdicts, read from reputation.json, in place of those it
|
||||
// keeps, but for those of a zone SWWAF_DNSBL_ZONES does not name, and,
|
||||
// keeps, but for those of a zone SWW_DNSBL_ZONES does not name, and,
|
||||
// past maxVerdicts, those fetched longest ago. One fetched CacheTTL ago or
|
||||
// more is neither used nor written, as for any verdict.
|
||||
func (d *DNSBL) Load(verdicts []Verdict) {
|
||||
|
||||
@@ -40,8 +40,8 @@ const (
|
||||
// zone and otherZone are the DNSBL zones the tests name.
|
||||
zone = "dnsbl.example"
|
||||
otherZone = "other.example"
|
||||
// cacheTTL is the tests' SWWAF_REPUTATION_CACHE_TTL, and timeout their
|
||||
// SWWAF_REPUTATION_TIMEOUT: a second, the least time /etc/resolv.conf
|
||||
// cacheTTL is the tests' SWW_REPUTATION_CACHE_TTL, and timeout their
|
||||
// SWW_REPUTATION_TIMEOUT: a second, the least time /etc/resolv.conf
|
||||
// can have Go's resolver wait for one server, so that it is the
|
||||
// DNSBL's own timeout that ends a query, whatever that file says.
|
||||
cacheTTL = 24 * time.Hour
|
||||
@@ -419,7 +419,7 @@ func TestNeitherAVerdictOfAZoneNotNamedNorOnePastItsTTLIsKept(t *testing.T) {
|
||||
dnsbl := reputation.NewDNSBL(p)
|
||||
|
||||
// The last verdict still in use, one fetched a TTL ago, and one of a
|
||||
// zone SWWAF_DNSBL_ZONES does not name.
|
||||
// zone SWW_DNSBL_ZONES does not name.
|
||||
inUse := reputation.Verdict{
|
||||
Zone: zone, Client: netip.MustParseAddr(listed), Listed: true,
|
||||
Fetched: now.Add(-cacheTTL + time.Nanosecond),
|
||||
@@ -471,7 +471,7 @@ func TestAtMost100000VerdictsKeptTheOneFetchedLongestAgoDroppedFirst(t *testing.
|
||||
}
|
||||
|
||||
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||
func TestQueriesGoToTheResolverSWWAFDNSBLResolverNames(t *testing.T) {
|
||||
func TestQueriesGoToTheResolverSWWDNSBLResolverNames(t *testing.T) {
|
||||
resolver := &resolverStandIn{answers: map[string]answer{
|
||||
listedName: {addrs: []string{listing}},
|
||||
}}
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
// Package reputation fetches the lists the settings name by URL: the
|
||||
// blocklists of SWWAF_BLOCKLIST_URLS, the file of AS:percent lines
|
||||
// SWWAF_ASN_LIMIT_PERCENT_URL names, and the decision list of the CrowdSec
|
||||
// engine SWWAF_CROWDSEC_LAPI_URL names. It keeps the last good copy of
|
||||
// blocklists of SWW_BLOCKLIST_URLS, the file of AS:percent lines
|
||||
// SWW_ASN_LIMIT_PERCENT_URL names, the address lists of SWW_TAG_LISTS, and
|
||||
// the decision list of the CrowdSec engine SWW_CROWDSEC_LAPI_URL names. It
|
||||
// keeps the last good copy of
|
||||
// each, whole, comment lines included, which is used while a fetch fails,
|
||||
// and when each was last tried. It also asks the DNSBL zones of
|
||||
// SWWAF_DNSBL_ZONES about clients, and keeps their verdicts, and checks
|
||||
// SWW_DNSBL_ZONES about clients, and keeps their verdicts, and checks
|
||||
// clients with AbuseIPDB, and keeps their scores and the checks spent
|
||||
// today. The state package writes all of these to reputation.json and
|
||||
// reads them from it, so that a restart keeps them too.
|
||||
@@ -66,18 +67,20 @@ type List struct {
|
||||
|
||||
// Params are what New needs.
|
||||
type Params struct {
|
||||
// BlocklistURLs are the blocklists (SWWAF_BLOCKLIST_URLS), and
|
||||
// BlocklistURLs are the blocklists (SWW_BLOCKLIST_URLS), and
|
||||
// ASNLimitPercentURL the file of AS:percent lines
|
||||
// (SWWAF_ASN_LIMIT_PERCENT_URL), "" while it is unset.
|
||||
// (SWW_ASN_LIMIT_PERCENT_URL), "" while it is unset.
|
||||
BlocklistURLs []string
|
||||
ASNLimitPercentURL string
|
||||
// TagLists are the address lists that add a tag (SWW_TAG_LISTS).
|
||||
TagLists []config.TagList
|
||||
// CrowdSecDecisionsURL is the CrowdSec decision list, "" while
|
||||
// SWWAF_CROWDSEC_LAPI_URL is unset, fetched with CrowdSecKey
|
||||
// (SWWAF_CROWDSEC_LAPI_KEY).
|
||||
// SWW_CROWDSEC_LAPI_URL is unset, fetched with CrowdSecKey
|
||||
// (SWW_CROWDSEC_LAPI_KEY).
|
||||
CrowdSecDecisionsURL string
|
||||
CrowdSecKey string
|
||||
// Refresh is how long after a list was last fetched or tried it is
|
||||
// fetched again (SWWAF_BLOCKLIST_REFRESH), but for the CrowdSec decision
|
||||
// fetched again (SWW_BLOCKLIST_REFRESH), but for the CrowdSec decision
|
||||
// list, which is fetched again crowdSecRefresh after.
|
||||
Refresh time.Duration
|
||||
// Now tells the time, normally time.Now in UTC.
|
||||
@@ -112,11 +115,11 @@ type list struct {
|
||||
failures int
|
||||
}
|
||||
|
||||
// entries are what the lines of a copy say: for a blocklist, the netblocks
|
||||
// it names, with the lengths among them, for the file of AS:percent lines,
|
||||
// the percentage it gives each AS number, and for the CrowdSec decision
|
||||
// list, the decision on each netblock that ends last, with the lengths
|
||||
// among them.
|
||||
// entries are what the lines of a copy say: for a blocklist or a tag list, the
|
||||
// netblocks it names, with the lengths among them, for the file of AS:percent
|
||||
// lines, the percentage it gives each AS number, and for the CrowdSec decision
|
||||
// list, the decision on each netblock that ends last, with the lengths among
|
||||
// them.
|
||||
type entries struct {
|
||||
netblocks map[netip.Prefix]bool
|
||||
lengths []int
|
||||
@@ -152,14 +155,18 @@ func New(params Params) *Lists {
|
||||
}
|
||||
|
||||
// URLs returns the URL of every list: the blocklists' in the order
|
||||
// SWWAF_BLOCKLIST_URLS names them, then SWWAF_ASN_LIMIT_PERCENT_URL, then
|
||||
// the CrowdSec decision list's.
|
||||
// SWW_BLOCKLIST_URLS names them, then SWW_ASN_LIMIT_PERCENT_URL, then the
|
||||
// tag lists', then the CrowdSec decision list's.
|
||||
func (l *Lists) URLs() []string {
|
||||
urls := slices.Clone(l.params.BlocklistURLs)
|
||||
if l.params.ASNLimitPercentURL != "" {
|
||||
urls = append(urls, l.params.ASNLimitPercentURL)
|
||||
}
|
||||
|
||||
for _, list := range l.params.TagLists {
|
||||
urls = append(urls, list.URL)
|
||||
}
|
||||
|
||||
if l.params.CrowdSecDecisionsURL != "" {
|
||||
urls = append(urls, l.params.CrowdSecDecisionsURL)
|
||||
}
|
||||
@@ -168,7 +175,7 @@ func (l *Lists) URLs() []string {
|
||||
}
|
||||
|
||||
// ListedBy returns the URLs of the blocklists whose copy lists addr, in
|
||||
// the order SWWAF_BLOCKLIST_URLS names them.
|
||||
// the order SWW_BLOCKLIST_URLS names them.
|
||||
func (l *Lists) ListedBy(addr netip.Addr) []string {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
@@ -184,6 +191,22 @@ func (l *Lists) ListedBy(addr netip.Addr) []string {
|
||||
return listedBy
|
||||
}
|
||||
|
||||
// TagsOf returns the names of the tag lists whose copy holds addr.
|
||||
func (l *Lists) TagsOf(addr netip.Addr) []string {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
var names []string
|
||||
|
||||
for _, list := range l.params.TagLists {
|
||||
if l.lists[list.URL].entries.contain(addr) {
|
||||
names = append(names, list.Name)
|
||||
}
|
||||
}
|
||||
|
||||
return names
|
||||
}
|
||||
|
||||
// ASNLimitPercent returns the percentage the copy of the file of
|
||||
// AS:percent lines gives asn, and whether it lists asn.
|
||||
func (l *Lists) ASNLimitPercent(asn string) (int64, bool) {
|
||||
@@ -481,10 +504,10 @@ func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) {
|
||||
}
|
||||
|
||||
// parse reads the lines of the list at listURL, fetched at fetched: those
|
||||
// of a blocklist, of the file of AS:percent lines, or of the CrowdSec
|
||||
// decision list. In the first two, anything after a ; or a # on a line is
|
||||
// left out, and so is a line left blank. Any other line that does not read
|
||||
// is an error naming it by its number.
|
||||
// of a blocklist or a tag list, of the file of AS:percent lines, or of the
|
||||
// CrowdSec decision list. In all but the decision list, anything after a ; or
|
||||
// a # on a line is left out, and so is a line left blank. Any other line that
|
||||
// does not read is an error naming it by its number.
|
||||
func (l *Lists) parse(
|
||||
listURL string, lines []string, fetched time.Time,
|
||||
) (entries, error) {
|
||||
@@ -498,8 +521,8 @@ func (l *Lists) parse(
|
||||
}
|
||||
}
|
||||
|
||||
// parseNetblocks reads a blocklist's lines, each an address or a netblock
|
||||
// as the settings take them.
|
||||
// parseNetblocks reads the lines of a blocklist or a tag list, each an address
|
||||
// or a netblock as the settings take them.
|
||||
func parseNetblocks(lines []string) (entries, error) {
|
||||
found := entries{netblocks: map[netip.Prefix]bool{}}
|
||||
|
||||
@@ -551,7 +574,7 @@ func parseNetblock(text string) (netip.Prefix, bool) {
|
||||
}
|
||||
|
||||
// parsePercents reads the lines of the file of AS:percent lines, each an
|
||||
// AS number, : and a percentage, as SWWAF_ASN_LIMIT_PERCENT takes them. An
|
||||
// AS number, : and a percentage, such as AS64496:50. An
|
||||
// AS number listed more than once gets the lowest of its percentages.
|
||||
func parsePercents(lines []string) (entries, error) {
|
||||
found := entries{percents: map[string]int64{}}
|
||||
|
||||
@@ -34,8 +34,8 @@ const (
|
||||
dropURL = "https://lists.example/drop.txt"
|
||||
torURL = "https://lists.example/tor.txt"
|
||||
asnURL = "https://lists.example/asn.txt"
|
||||
// refresh is the tests' SWWAF_BLOCKLIST_REFRESH, and cooldown their
|
||||
// SWWAF_ALERT_COOLDOWN, longer than it.
|
||||
// refresh is the tests' SWW_BLOCKLIST_REFRESH, and cooldown their
|
||||
// SWW_ALERT_COOLDOWN, longer than it.
|
||||
refresh = 24 * time.Hour
|
||||
cooldown = 48 * time.Hour
|
||||
// drop is a blocklist as the Spamhaus DROP list is written, with an
|
||||
@@ -116,7 +116,7 @@ func TestClientIsListedByEachBlocklistThatListsIt(t *testing.T) {
|
||||
}}
|
||||
lists := start(t, servers, params(torURL, dropURL))
|
||||
|
||||
// In the order SWWAF_BLOCKLIST_URLS names them.
|
||||
// In the order SWW_BLOCKLIST_URLS names them.
|
||||
wantListedBy(t, lists, "203.0.113.9", torURL, dropURL)
|
||||
wantListedBy(t, lists, "203.0.113.8", dropURL)
|
||||
})
|
||||
|
||||
@@ -19,32 +19,45 @@ const (
|
||||
// ActionForward is a request discriminator allowed, which rproxy passes to
|
||||
// the app.
|
||||
ActionForward = "forward"
|
||||
// ActionTooLarge is a request over the header limits.
|
||||
ActionTooLarge = "too_large"
|
||||
// ActionRateLimited is a request refused because it took its client
|
||||
// over a rate limit, which bans the client.
|
||||
ActionRateLimited = "rate_limited"
|
||||
// ActionBanned is a request refused because a ban covers its client,
|
||||
// or because it matched a ban rule, asked for a trap path or the
|
||||
// CrowdSec decision list lists its client, each of which bans the
|
||||
// client.
|
||||
ActionBanned = "banned"
|
||||
// ActionRuleBlocked is a request refused because it matched a block
|
||||
// rule.
|
||||
ActionRuleBlocked = "rule_blocked"
|
||||
// ActionWAFBlocked is a request refused because the Core Rule Set
|
||||
// scored it at or over SWWAF_WAF_ANOMALY_THRESHOLD.
|
||||
ActionWAFBlocked = "waf_blocked"
|
||||
// ActionDenied is a request refused because its client is in
|
||||
// SWWAF_DENY_NETS, in a blocklist while SWWAF_BLOCKLIST_ACTION is deny,
|
||||
|
||||
// ActionRequestHeaderTooLarge is a request over the header limits.
|
||||
ActionRequestHeaderTooLarge = "request_header_too_large"
|
||||
|
||||
// ActionClientIsRateLimited is a request refused because it took its
|
||||
// client over a rate limit, which bans the client.
|
||||
ActionClientIsRateLimited = "client_is_rate_limited"
|
||||
|
||||
// ActionTemporarilyBanned is a request refused because a ban that runs
|
||||
// out covers its client, or because it matched a ban rule, asked for a
|
||||
// trap path or the CrowdSec decision list lists its client, and so made
|
||||
// such a ban.
|
||||
ActionTemporarilyBanned = "temporarily_banned"
|
||||
|
||||
// ActionPermanentlyBanned is a request refused because a ban that never
|
||||
// runs out covers its client, or because the request made such a ban or
|
||||
// made its client's ban permanent.
|
||||
ActionPermanentlyBanned = "permanently_banned"
|
||||
|
||||
// ActionBlockedByBlockRule is a request refused because it matched a
|
||||
// block rule.
|
||||
ActionBlockedByBlockRule = "blocked_by_block_rule"
|
||||
|
||||
// ActionBlockedByCoreRuleSet is a request refused because the Core Rule
|
||||
// Set scored it at or over SWW_WAF_ANOMALY_THRESHOLD.
|
||||
ActionBlockedByCoreRuleSet = "blocked_by_core_rule_set"
|
||||
|
||||
// ActionClientDenied is a request refused because its client is in
|
||||
// SWW_DENY_NETS, in a blocklist while SWW_BLOCKLIST_ACTION is deny,
|
||||
// or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while
|
||||
// SWWAF_REPUTATION_ACTION is deny.
|
||||
ActionDenied = "denied"
|
||||
// ActionCountryDenied is a request refused for its client's country.
|
||||
// SWW_REPUTATION_ACTION is deny, or that carries a tag in SWW_TAG_DENY.
|
||||
ActionClientDenied = "client_denied"
|
||||
|
||||
// ActionCountryDenied is a request refused because its client's country
|
||||
// is not in SWW_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||
ActionCountryDenied = "country_denied"
|
||||
// ActionAdmin is a request discriminator answered at one of its own
|
||||
// ActionOwnEndpoint is a request discriminator answered at one of its own
|
||||
// endpoints, or a question about a path under /_sww/.
|
||||
ActionAdmin = "admin"
|
||||
ActionOwnEndpoint = "own_endpoint"
|
||||
)
|
||||
|
||||
// OffenceLimit is the offence a request line names for a request that
|
||||
@@ -52,9 +65,9 @@ const (
|
||||
// limit.
|
||||
const OffenceLimit = "limit"
|
||||
|
||||
// LimitHitErrorBurst is the limit_hit a request line names for a request
|
||||
// LimitHitMinuteRefusals is the limit_hit a request line names for a request
|
||||
// that broke the error burst.
|
||||
const LimitHitErrorBurst = "error_burst"
|
||||
const LimitHitMinuteRefusals = "minute_refusals"
|
||||
|
||||
// The discriminator_line a request line names, which says which of
|
||||
// discriminator's two lines about a question it is.
|
||||
@@ -112,7 +125,7 @@ type Line struct {
|
||||
ContentType string `json:"content_type,omitempty"`
|
||||
// ContentLength is the length of its body the request announced.
|
||||
ContentLength int64 `json:"content_length,omitempty"`
|
||||
// RequestHeaders are the headers SWWAF_LOG_REQUEST_HEADERS names that
|
||||
// RequestHeaders are the headers SWW_LOG_REQUEST_HEADERS names that
|
||||
// the request carried, by name in lower case.
|
||||
RequestHeaders map[string]string `json:"request_headers,omitempty"`
|
||||
HasAuthorization bool `json:"has_authorization,omitempty"`
|
||||
@@ -126,20 +139,31 @@ type Line struct {
|
||||
// The decision.
|
||||
Action string `json:"action"`
|
||||
// WouldAction is, in observe mode, the action enforce mode would have
|
||||
// taken with a request it would have refused: ActionDenied,
|
||||
// ActionBanned, ActionCountryDenied, ActionRateLimited,
|
||||
// ActionRuleBlocked, ActionWAFBlocked, or, in discriminator, ActionTooLarge
|
||||
// for the header limits.
|
||||
// taken with a request it would have refused: ActionClientDenied,
|
||||
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
|
||||
// ActionClientIsRateLimited, ActionBlockedByBlockRule,
|
||||
// ActionBlockedByCoreRuleSet, or, in discriminator,
|
||||
// ActionRequestHeaderTooLarge for the header limits.
|
||||
WouldAction string `json:"would_action,omitempty"`
|
||||
// LimitPercent and LimitPercentSetting are, for a request the rate
|
||||
// limits counted whose client a biased threshold gives a percentage of
|
||||
// the rate limits below 100, that percentage and the setting that gave
|
||||
// it. BytesPercent and BytesPercentSetting are the same for the byte
|
||||
// limits.
|
||||
LimitPercent *int64 `json:"limit_percent,omitempty"`
|
||||
LimitPercentSetting string `json:"limit_percent_setting,omitempty"`
|
||||
BytesPercent *int64 `json:"bytes_percent,omitempty"`
|
||||
BytesPercentSetting string `json:"bytes_percent_setting,omitempty"`
|
||||
// Heuristic is the name of the heuristic that refused the request, in
|
||||
// observe mode too, and HeuristicFigures are the figures it acted on, by
|
||||
// name.
|
||||
Heuristic string `json:"heuristic,omitempty"`
|
||||
HeuristicFigures map[string]any `json:"heuristic_figures,omitempty"`
|
||||
// AfterAnswerHeuristic and AfterAnswerHeuristicFigures are the same for
|
||||
// the heuristic that banned the client after discriminator answered the
|
||||
// request.
|
||||
AfterAnswerHeuristic string `json:"after_answer_heuristic,omitempty"`
|
||||
AfterAnswerHeuristicFigures map[string]any `json:"after_answer_heuristic_figures,omitempty"` //nolint:lll // a struct tag cannot be split
|
||||
// RateLimitPercent and RateLimitPercentSetting are, for a request the
|
||||
// rate limits counted whose client a biased threshold gives a percentage
|
||||
// of the rate limits below 100, that percentage and the setting that gave
|
||||
// it. BytesLimitPercent and BytesLimitPercentSetting are the same for the
|
||||
// byte limits.
|
||||
RateLimitPercent *int64 `json:"rate_limit_percent,omitempty"`
|
||||
RateLimitPercentSetting string `json:"rate_limit_percent_setting,omitempty"`
|
||||
BytesLimitPercent *int64 `json:"bytes_limit_percent,omitempty"`
|
||||
BytesLimitPercentSetting string `json:"bytes_limit_percent_setting,omitempty"`
|
||||
// Counts are, for a request the rate limits counted, the client's
|
||||
// requests as they counted them with this one, and its bytes as the
|
||||
// byte limits counted them, with this request's once it has ended if
|
||||
@@ -147,36 +171,42 @@ type Line struct {
|
||||
Counts ratelimit.Counts `json:"counts,omitzero"`
|
||||
// RuleIDs are the ids of the rule file rules the request matched.
|
||||
RuleIDs []string `json:"rule_ids,omitempty"`
|
||||
// WAFRuleIDs are the ids of the Core Rule Set's rules the request
|
||||
// matched, and WAFScore its anomaly score, nil for a request the Core
|
||||
// Rule Set did not inspect.
|
||||
WAFRuleIDs []int `json:"waf_rule_ids,omitempty"`
|
||||
WAFScore *int `json:"waf_score,omitempty"`
|
||||
// CoreRuleSetRuleIDs are the ids of the Core Rule Set's rules the request
|
||||
// matched, and CoreRuleSetScore its anomaly score, nil for a request the
|
||||
// Core Rule Set did not inspect.
|
||||
CoreRuleSetRuleIDs []int `json:"core_rule_set_rule_ids,omitempty"`
|
||||
CoreRuleSetScore *int `json:"core_rule_set_score,omitempty"`
|
||||
// LimitHit is the window whose limit the request went over, named as
|
||||
// Counts names its count: minute, hour or day for a rate limit, and
|
||||
// minute_bytes, hour_bytes or day_bytes for a byte limit; or
|
||||
// LimitHitErrorBurst for the error burst.
|
||||
// LimitHitMinuteRefusals for the error burst.
|
||||
LimitHit string `json:"limit_hit,omitempty"`
|
||||
// Reputation are the URLs of the blocklists that list the client, then
|
||||
// that of the CrowdSec decision list when it does, then the DNSBL zones
|
||||
// whose verdict lists it, their keys masked, then abuseipdb when its
|
||||
// score is a hit.
|
||||
Reputation []string `json:"reputation,omitempty"`
|
||||
// Tags are the request's tags, sorted.
|
||||
Tags []string `json:"tags,omitempty"`
|
||||
// RouteWatchTagsMissing, while SWW_ROUTEWATCH_URL is set, says why the
|
||||
// request got no tags from routewatch: no_answer_yet, request_failed or
|
||||
// no_route.
|
||||
RouteWatchTagsMissing string `json:"routewatch_tags_missing,omitempty"`
|
||||
// Offence is the offence the request was held as, OffenceLimit.
|
||||
Offence string `json:"offence,omitempty"`
|
||||
// BanExpires is when the ban the request made, or was refused under,
|
||||
// ends: a time, or "permanent".
|
||||
BanExpires string `json:"ban_expires,omitempty"`
|
||||
|
||||
// The timings, in milliseconds. DurationChecks is the time until the
|
||||
// checks were done, and DurationWAF the part of it the Core Rule Set
|
||||
// took. DurationUpstreamTotal is what the report of an allowed request gives,
|
||||
// The timings. DurationChecksMS is the time until the checks were done, and
|
||||
// DurationCoreRuleSetMS the part of it the Core Rule Set took.
|
||||
// DurationUpstreamTotalMS is what the report of an allowed request gives,
|
||||
// from the hand-over to the app until the end of its answer. Each but
|
||||
// DurationTotal is nil for a request that did not get that far.
|
||||
DurationTotal float64 `json:"duration_total"`
|
||||
DurationChecks *float64 `json:"duration_checks,omitempty"`
|
||||
DurationWAF *float64 `json:"duration_waf,omitempty"`
|
||||
DurationUpstreamTotal *float64 `json:"duration_upstream_total,omitempty"`
|
||||
// DurationTotalMS is nil for a request that did not get that far.
|
||||
DurationTotalMS float64 `json:"duration_total_ms"`
|
||||
DurationChecksMS *float64 `json:"duration_checks_ms,omitempty"`
|
||||
DurationCoreRuleSetMS *float64 `json:"duration_core_rule_set_ms,omitempty"`
|
||||
DurationUpstreamTotalMS *float64 `json:"duration_upstream_total_ms,omitempty"`
|
||||
}
|
||||
|
||||
// RProxyLine is rproxy's short line in the request log about one request. The
|
||||
@@ -196,11 +226,11 @@ type RProxyLine struct {
|
||||
Status int `json:"status"`
|
||||
RequestBytes int64 `json:"request_bytes"`
|
||||
ResponseBytes int64 `json:"response_bytes"`
|
||||
// DurationRequest runs in milliseconds from the request's headers being read
|
||||
// until the app has been sent the whole request, or the answer has started
|
||||
// if that is sooner. DurationResponse runs from then until the answer ends.
|
||||
DurationRequest float64 `json:"duration_request"`
|
||||
DurationResponse float64 `json:"duration_response"`
|
||||
// DurationRequestMS runs from the request's headers being read until the app
|
||||
// has been sent the whole request, or the answer has started if that is
|
||||
// sooner. DurationResponseMS runs from then until the answer ends.
|
||||
DurationRequestMS float64 `json:"duration_request_ms"`
|
||||
DurationResponseMS float64 `json:"duration_response_ms"`
|
||||
// Disposition is the valid disposition discriminator answered or, while
|
||||
// DefaultDisposition is true, the value of SWRP_DEFAULT_DISPOSITION. It is ""
|
||||
// when neither gave one.
|
||||
@@ -245,25 +275,37 @@ func FormatTime(t time.Time) string {
|
||||
|
||||
// Milliseconds is d in milliseconds, to the microsecond.
|
||||
func Milliseconds(d time.Duration) float64 {
|
||||
return float64(d.Microseconds()) / float64(time.Millisecond/time.Microsecond)
|
||||
return float64(
|
||||
d.Microseconds(),
|
||||
) / float64(
|
||||
time.Millisecond/time.Microsecond,
|
||||
)
|
||||
}
|
||||
|
||||
// NewProcessLogger returns the logger for the process's own messages:
|
||||
// JSON lines on w, marked "type":"process", with the time in the same form
|
||||
// as a request line's, and instanceName, SWWAF_INSTANCE_NAME, as instance.
|
||||
// It writes only the messages at level, SWWAF_LOG_LEVEL, or more severe;
|
||||
// as a request line's, and instanceName, SWW_INSTANCE_NAME, as instance.
|
||||
// It writes only the messages at level, SWW_LOG_LEVEL, or more severe;
|
||||
// the request lines Write writes are never held back.
|
||||
func NewProcessLogger(w io.Writer, instanceName string, level slog.Level) *slog.Logger {
|
||||
func NewProcessLogger(
|
||||
w io.Writer,
|
||||
instanceName string,
|
||||
level slog.Level,
|
||||
) *slog.Logger {
|
||||
handler := slog.NewJSONHandler(w, &slog.HandlerOptions{
|
||||
Level: level,
|
||||
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
|
||||
if attr.Key == slog.TimeKey && len(groups) == 0 {
|
||||
return slog.String(slog.TimeKey, FormatTime(attr.Value.Time()))
|
||||
return slog.String(
|
||||
slog.TimeKey,
|
||||
FormatTime(attr.Value.Time()),
|
||||
)
|
||||
}
|
||||
|
||||
return attr
|
||||
},
|
||||
})
|
||||
|
||||
return slog.New(handler).With("type", "process", "instance", instanceName)
|
||||
return slog.New(handler).
|
||||
With("type", "process", "instance", instanceName)
|
||||
}
|
||||
@@ -18,11 +18,11 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
|
||||
err := requestlog.Write(&out, &requestlog.Line{
|
||||
Time: requestlog.FormatTime(time.Date(2026, 10, 3, 12, 0, 0, 0, time.UTC)),
|
||||
ClientIP: "203.0.113.9",
|
||||
Status: 200,
|
||||
Action: requestlog.ActionForward,
|
||||
DurationTotal: requestlog.Milliseconds(1500 * time.Microsecond),
|
||||
Time: requestlog.FormatTime(time.Date(2026, 10, 3, 12, 0, 0, 0, time.UTC)),
|
||||
ClientIP: "203.0.113.9",
|
||||
Status: 200,
|
||||
Action: requestlog.ActionForward,
|
||||
DurationTotalMS: requestlog.Milliseconds(1500 * time.Microsecond),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("write: %v", err)
|
||||
@@ -43,7 +43,7 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
|
||||
want := map[string]any{
|
||||
"type": "request", "time": "2026-10-03T12:00:00.000Z",
|
||||
"client_ip": "203.0.113.9", "status": 200.0, "action": "forward",
|
||||
"duration_total": 1.5,
|
||||
"duration_total_ms": 1.5,
|
||||
}
|
||||
for name, value := range want {
|
||||
if fields[name] != value {
|
||||
@@ -55,9 +55,9 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
|
||||
"forwarded_for", "content_type", "content_length", "request_headers",
|
||||
"has_authorization", "has_cookie", "websocket", "response_content_type",
|
||||
"upstream_status", "cache_control", "location", "aborted", "counts",
|
||||
"limit_hit", "offence", "ban_expires", "duration_checks",
|
||||
"limit_hit", "offence", "ban_expires", "duration_checks_ms",
|
||||
"duration_upstream_connect", "duration_upstream_first_byte",
|
||||
"duration_upstream_total",
|
||||
"duration_upstream_total_ms",
|
||||
}
|
||||
for _, name := range unset {
|
||||
_, present := fields[name]
|
||||
@@ -82,7 +82,7 @@ func TestWriteRProxyLineWritesOneJSONLineWithOnlyItsFields(t *testing.T) {
|
||||
|
||||
const want = `{"type":"request","time":"","request_id":"an-id","client_ip":"",` +
|
||||
`"host":"","method":"","path":"","query":"","status":0,"request_bytes":0,` +
|
||||
`"response_bytes":0,"duration_request":0,"duration_response":0,` +
|
||||
`"response_bytes":0,"duration_request_ms":0,"duration_response_ms":0,` +
|
||||
`"disposition":"allow"}` + "\n"
|
||||
if out.String() != want {
|
||||
t.Errorf("wrote %s, want %s", out.String(), want)
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
// Package routewatch asks a routewatch instance about client addresses, in the
|
||||
// background, and keeps its answers in memory. A routewatch instance knows, for
|
||||
// an address, the netblock it is announced in and the AS that originates it.
|
||||
package routewatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||
)
|
||||
|
||||
const (
|
||||
// answerKept is how long an answer is used after it came.
|
||||
answerKept = time.Hour
|
||||
// failureKept is how long routewatch is not asked again after a query
|
||||
// fails, so that a dead routewatch is not asked on every request.
|
||||
failureKept = time.Minute
|
||||
// queryTimeout is how long one query may take. routewatch answers only
|
||||
// after a reverse DNS lookup it allows 2 seconds, so this is longer.
|
||||
queryTimeout = 5 * time.Second
|
||||
// maxAnswers is how many answers are kept. Past it, the one that came
|
||||
// longest ago is dropped.
|
||||
maxAnswers = 100000
|
||||
// maxQueries is how many queries may be under way at once. Past it, a
|
||||
// client is not asked about until its next request, so that a swarm of
|
||||
// new addresses cannot fill the memory.
|
||||
maxQueries = 100
|
||||
// maxAnswerBytes is the most of an answer that is read.
|
||||
maxAnswerBytes = 64 << 10
|
||||
)
|
||||
|
||||
// Missing is why Route gives no route. The request log's field
|
||||
// routewatch_tags_missing gives these values.
|
||||
type Missing string
|
||||
|
||||
const (
|
||||
// NoAnswerYet is a client routewatch has not answered about yet.
|
||||
NoAnswerYet Missing = "no_answer_yet"
|
||||
// RequestFailed is no answer because a recent query failed.
|
||||
RequestFailed Missing = "request_failed"
|
||||
// NoRoute is routewatch knowing no live route for the address.
|
||||
NoRoute Missing = "no_route"
|
||||
)
|
||||
|
||||
var (
|
||||
errStatus = errors.New("routewatch answered")
|
||||
errNoResult = errors.New("the answer holds no result")
|
||||
errNotNetblock = errors.New("the answer's netblock is not a netblock")
|
||||
)
|
||||
|
||||
// Route is what routewatch knows of an address: the netblock it is announced
|
||||
// in, and the AS that originates it.
|
||||
type Route struct {
|
||||
Netblock netip.Prefix
|
||||
ASN uint32
|
||||
}
|
||||
|
||||
// Params are what New needs.
|
||||
type Params struct {
|
||||
// URL is where routewatch is (SWW_ROUTEWATCH_URL).
|
||||
URL string
|
||||
// Now tells the time, normally time.Now in UTC.
|
||||
Now func() time.Time
|
||||
// ProcessLog receives the query that fails, and why.
|
||||
ProcessLog *slog.Logger
|
||||
}
|
||||
|
||||
// Client asks a routewatch instance about clients, in the background, and keeps
|
||||
// its answers. It is safe for concurrent use.
|
||||
type Client struct {
|
||||
params Params
|
||||
httpClient *http.Client
|
||||
|
||||
mu sync.Mutex
|
||||
// answers are by client. Each is added as it comes and never moved up,
|
||||
// so that the one that came longest ago is the first dropped.
|
||||
answers *simplelru.LRU[netip.Prefix, answer]
|
||||
// asking are the clients whose query is under way.
|
||||
asking map[netip.Prefix]bool
|
||||
// retryAt is when routewatch may be asked again after a query failed.
|
||||
retryAt time.Time
|
||||
}
|
||||
|
||||
// answer is one kept answer: the route, or why there is none, and when it came.
|
||||
type answer struct {
|
||||
route Route
|
||||
missing Missing
|
||||
came time.Time
|
||||
}
|
||||
|
||||
// New returns a Client with no answer yet.
|
||||
func New(params Params) *Client {
|
||||
answers, err := simplelru.NewLRU[netip.Prefix, answer](maxAnswers, nil)
|
||||
if err != nil {
|
||||
panic(err) // NewLRU fails only for a size below one
|
||||
}
|
||||
|
||||
return &Client{
|
||||
params: params,
|
||||
httpClient: &http.Client{},
|
||||
answers: answers,
|
||||
asking: map[netip.Prefix]bool{},
|
||||
}
|
||||
}
|
||||
|
||||
// Route returns what routewatch answered about client, an IPv4 address or an
|
||||
// IPv6 group, or why it has no route. It never waits for a query. A client
|
||||
// without an answer is asked about in the background, by addr, the address its
|
||||
// request came from, unless its query is under way, a query failed less than
|
||||
// failureKept ago, or maxQueries are under way. ctx is the context of the
|
||||
// client's request, and a query goes on after the request ends.
|
||||
func (c *Client) Route(
|
||||
ctx context.Context, client netip.Prefix, addr netip.Addr,
|
||||
) (Route, Missing) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
now := c.params.Now()
|
||||
|
||||
kept, found := c.answers.Peek(client)
|
||||
if found && now.Sub(kept.came) < answerKept {
|
||||
return kept.route, kept.missing
|
||||
}
|
||||
|
||||
if now.Before(c.retryAt) {
|
||||
return Route{}, RequestFailed
|
||||
}
|
||||
|
||||
if !c.asking[client] && len(c.asking) < maxQueries {
|
||||
c.asking[client] = true
|
||||
|
||||
go c.ask(context.WithoutCancel(ctx), client, addr)
|
||||
}
|
||||
|
||||
return Route{}, NoAnswerYet
|
||||
}
|
||||
|
||||
// ask asks routewatch about addr, keeps the answer as client's, and notes the
|
||||
// query as no longer under way. A query that fails gives no answer, and
|
||||
// routewatch is not asked again for failureKept. Only the first failure of
|
||||
// those under way is logged.
|
||||
func (c *Client) ask(ctx context.Context, client netip.Prefix, addr netip.Addr) {
|
||||
route, missing, err := c.query(ctx, addr)
|
||||
now := c.params.Now()
|
||||
|
||||
c.mu.Lock()
|
||||
|
||||
delete(c.asking, client)
|
||||
|
||||
firstFailure := err != nil && !now.Before(c.retryAt)
|
||||
|
||||
if err == nil {
|
||||
c.answers.Add(client, answer{route: route, missing: missing, came: now})
|
||||
} else {
|
||||
c.retryAt = now.Add(failureKept)
|
||||
}
|
||||
|
||||
c.mu.Unlock()
|
||||
|
||||
if firstFailure {
|
||||
c.params.ProcessLog.Warn("asking routewatch failed", "error", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
// query asks routewatch's IP info API about addr. An answer other than 200, one
|
||||
// that cannot be read, and none within queryTimeout, fail. A result with no
|
||||
// netblock is an address without a live route.
|
||||
func (c *Client) query(ctx context.Context, addr netip.Addr) (Route, Missing, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, queryTimeout)
|
||||
defer cancel()
|
||||
|
||||
endpoint, err := url.JoinPath(c.params.URL, "api/v1/ip", addr.String())
|
||||
if err != nil {
|
||||
return Route{}, "", fmt.Errorf("make the URL: %w", err)
|
||||
}
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, http.NoBody)
|
||||
if err != nil {
|
||||
return Route{}, "", fmt.Errorf("make the request: %w", err)
|
||||
}
|
||||
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
res, err := c.httpClient.Do(req)
|
||||
if err != nil {
|
||||
// Do's error names the URL, which holds the client's address, which
|
||||
// is not to be logged: only what went wrong is kept.
|
||||
return Route{}, "", fmt.Errorf("ask routewatch: %w", errors.Unwrap(err))
|
||||
}
|
||||
|
||||
defer func() {
|
||||
_ = res.Body.Close()
|
||||
}()
|
||||
|
||||
if res.StatusCode != http.StatusOK {
|
||||
return Route{}, "", fmt.Errorf("%w %s", errStatus, res.Status)
|
||||
}
|
||||
|
||||
return readAnswer(io.LimitReader(res.Body, maxAnswerBytes))
|
||||
}
|
||||
|
||||
// readAnswer reads routewatch's answer about one address.
|
||||
func readAnswer(body io.Reader) (Route, Missing, error) {
|
||||
var decoded struct {
|
||||
Data struct {
|
||||
Results []struct {
|
||||
Netblock string `json:"netblock"`
|
||||
ASN uint32 `json:"asn"`
|
||||
} `json:"results"`
|
||||
} `json:"data"`
|
||||
}
|
||||
|
||||
err := json.NewDecoder(body).Decode(&decoded)
|
||||
if err != nil {
|
||||
return Route{}, "", fmt.Errorf("read the answer: %w", err)
|
||||
}
|
||||
|
||||
if len(decoded.Data.Results) == 0 {
|
||||
return Route{}, "", errNoResult
|
||||
}
|
||||
|
||||
result := decoded.Data.Results[0]
|
||||
if result.Netblock == "" {
|
||||
return Route{}, NoRoute, nil
|
||||
}
|
||||
|
||||
netblock, err := netip.ParsePrefix(result.Netblock)
|
||||
if err != nil {
|
||||
return Route{}, "", fmt.Errorf("%w: %q", errNotNetblock, result.Netblock)
|
||||
}
|
||||
|
||||
return Route{Netblock: netblock.Masked(), ASN: result.ASN}, "", nil
|
||||
}
|
||||
@@ -0,0 +1,333 @@
|
||||
package routewatch_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/routewatch"
|
||||
)
|
||||
|
||||
const (
|
||||
clientAddr = "192.0.2.1"
|
||||
routedAnswer = `{"status":"ok","data":{"results":[` +
|
||||
`{"ip":"192.0.2.1","netblock":"192.0.2.0/24","asn":64496}]}}`
|
||||
noRouteAnswer = `{"status":"ok","data":{"results":[{"ip":"192.0.2.1"}]}}`
|
||||
// slowAnswer is longer than routewatch allows a reverse DNS lookup.
|
||||
slowAnswer = 2500 * time.Millisecond
|
||||
// waitLimit is longer than the Client allows a query.
|
||||
waitLimit = 10 * time.Second
|
||||
)
|
||||
|
||||
// clock is a time the tests move by hand.
|
||||
type clock struct {
|
||||
mu sync.Mutex
|
||||
now time.Time
|
||||
}
|
||||
|
||||
func (c *clock) Now() time.Time {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
return c.now
|
||||
}
|
||||
|
||||
func (c *clock) advance(by time.Duration) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
c.now = c.now.Add(by)
|
||||
}
|
||||
|
||||
// fixture is a Client asking a stand-in for routewatch. queries counts what the
|
||||
// stand-in was asked, paths holds the last path it was asked for, and log holds
|
||||
// the process log.
|
||||
type fixture struct {
|
||||
t *testing.T
|
||||
server *httptest.Server
|
||||
client *routewatch.Client
|
||||
clk *clock
|
||||
queries *atomic.Int32
|
||||
path *atomic.Value
|
||||
log *syncBuffer
|
||||
}
|
||||
|
||||
// syncBuffer is a buffer the background queries can log to while a test reads it.
|
||||
type syncBuffer struct {
|
||||
mu sync.Mutex
|
||||
buf bytes.Buffer
|
||||
}
|
||||
|
||||
func (b *syncBuffer) Write(p []byte) (int, error) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
|
||||
return b.buf.Write(p)
|
||||
}
|
||||
|
||||
func (b *syncBuffer) String() string {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
|
||||
return b.buf.String()
|
||||
}
|
||||
|
||||
func newFixture(t *testing.T, standIn http.HandlerFunc) *fixture {
|
||||
t.Helper()
|
||||
|
||||
f := &fixture{
|
||||
t: t,
|
||||
clk: &clock{now: time.Date(2026, 11, 9, 0, 0, 0, 0, time.UTC)},
|
||||
queries: &atomic.Int32{},
|
||||
path: &atomic.Value{},
|
||||
log: &syncBuffer{},
|
||||
}
|
||||
|
||||
f.server = httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
f.queries.Add(1)
|
||||
f.path.Store(r.URL.Path)
|
||||
standIn(w, r)
|
||||
}))
|
||||
t.Cleanup(f.server.Close)
|
||||
|
||||
f.client = routewatch.New(routewatch.Params{
|
||||
URL: f.server.URL + "/", Now: f.clk.Now,
|
||||
ProcessLog: slog.New(slog.NewTextHandler(f.log, nil)),
|
||||
})
|
||||
|
||||
return f
|
||||
}
|
||||
|
||||
func answering(body string) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(body))
|
||||
}
|
||||
}
|
||||
|
||||
func (f *fixture) route() (routewatch.Route, routewatch.Missing) {
|
||||
addr := netip.MustParseAddr(clientAddr)
|
||||
client := netip.PrefixFrom(addr, addr.BitLen())
|
||||
|
||||
return f.client.Route(context.Background(), client, addr)
|
||||
}
|
||||
|
||||
// waitFor asks until the answer is why, or, while why is "", until there is a
|
||||
// route, and returns the route.
|
||||
func (f *fixture) waitFor(why routewatch.Missing) routewatch.Route {
|
||||
f.t.Helper()
|
||||
|
||||
deadline := time.Now().Add(waitLimit)
|
||||
|
||||
for {
|
||||
route, missing := f.route()
|
||||
if missing == why {
|
||||
return route
|
||||
}
|
||||
|
||||
if time.Now().After(deadline) {
|
||||
f.t.Fatalf("still %q after %s, want %q", missing, waitLimit, why)
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// wantFailureLoggedWithoutTheClient waits for the process log to name a failed
|
||||
// query, and checks that it does not name the client.
|
||||
func (f *fixture) wantFailureLoggedWithoutTheClient() {
|
||||
f.t.Helper()
|
||||
|
||||
// The failure is noted before it is logged.
|
||||
deadline := time.Now().Add(waitLimit)
|
||||
for !strings.Contains(f.log.String(), "asking routewatch failed") {
|
||||
if time.Now().After(deadline) {
|
||||
f.t.Fatalf("process log %q names no failure", f.log.String())
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
|
||||
if strings.Contains(f.log.String(), clientAddr) {
|
||||
f.t.Errorf("process log %q names the client", f.log.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRouteGivesRoutewatchsAnswerAndAsksByAddress(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, answering(routedAnswer))
|
||||
|
||||
if _, missing := f.route(); missing != routewatch.NoAnswerYet {
|
||||
t.Errorf("before the answer: %q, want %q", missing, routewatch.NoAnswerYet)
|
||||
}
|
||||
|
||||
route := f.waitFor("")
|
||||
|
||||
want := routewatch.Route{Netblock: netip.MustParsePrefix("192.0.2.0/24"), ASN: 64496}
|
||||
if route != want {
|
||||
t.Errorf("route %+v, want %+v", route, want)
|
||||
}
|
||||
|
||||
if got := f.path.Load(); got != "/api/v1/ip/"+clientAddr {
|
||||
t.Errorf("asked for path %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnswerAfterASlowReverseDNSLookupIsKept(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
time.Sleep(slowAnswer)
|
||||
|
||||
_, _ = w.Write([]byte(routedAnswer))
|
||||
})
|
||||
|
||||
f.route()
|
||||
|
||||
route := f.waitFor("")
|
||||
|
||||
want := routewatch.Route{Netblock: netip.MustParsePrefix("192.0.2.0/24"), ASN: 64496}
|
||||
if route != want {
|
||||
t.Errorf("route %+v, want %+v", route, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddressWithoutARouteIsKeptAsNoRoute(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, answering(noRouteAnswer))
|
||||
|
||||
f.route()
|
||||
f.waitFor(routewatch.NoRoute)
|
||||
|
||||
if got := f.queries.Load(); got != 1 {
|
||||
t.Errorf("%d queries, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnswerRoutewatchCannotBeTakenFromFails(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := map[string]http.HandlerFunc{
|
||||
"500": func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
},
|
||||
"not JSON": answering("no"),
|
||||
"no results": answering(`{"status":"ok","data":{"results":[]}}`),
|
||||
"bad netblock": answering(
|
||||
`{"status":"ok","data":{"results":[{"netblock":"nowhere","asn":1}]}}`),
|
||||
}
|
||||
for name, standIn := range tests {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, standIn)
|
||||
|
||||
f.route()
|
||||
f.waitFor(routewatch.RequestFailed)
|
||||
f.wantFailureLoggedWithoutTheClient()
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefusedConnectionFailsTheQueryAndTheLogOmitsTheClient(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, answering(routedAnswer))
|
||||
|
||||
f.server.Close()
|
||||
|
||||
f.route()
|
||||
f.waitFor(routewatch.RequestFailed)
|
||||
f.wantFailureLoggedWithoutTheClient()
|
||||
}
|
||||
|
||||
func TestFailureIsKeptForAMinuteSoRoutewatchIsNotAskedPerRequest(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
})
|
||||
|
||||
f.route()
|
||||
f.waitFor(routewatch.RequestFailed)
|
||||
f.route()
|
||||
|
||||
if got := f.queries.Load(); got != 1 {
|
||||
t.Errorf("%d queries while the failure is kept, want 1", got)
|
||||
}
|
||||
|
||||
f.clk.advance(time.Minute)
|
||||
f.route()
|
||||
|
||||
deadline := time.Now().Add(waitLimit)
|
||||
for f.queries.Load() != 2 {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("%d queries after the failure expired, want 2", f.queries.Load())
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHangingRoutewatchNeverDelaysAndFailsTheQuery(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newFixture(t, func(_ http.ResponseWriter, r *http.Request) {
|
||||
<-r.Context().Done()
|
||||
})
|
||||
|
||||
start := time.Now()
|
||||
|
||||
f.route()
|
||||
|
||||
if elapsed := time.Since(start); elapsed > time.Second {
|
||||
t.Errorf("Route took %s while routewatch hangs", elapsed)
|
||||
}
|
||||
|
||||
f.waitFor(routewatch.RequestFailed)
|
||||
f.wantFailureLoggedWithoutTheClient()
|
||||
}
|
||||
|
||||
func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
release := make(chan struct{})
|
||||
f := newFixture(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
<-release
|
||||
|
||||
_, _ = w.Write([]byte(routedAnswer))
|
||||
})
|
||||
|
||||
for range 50 {
|
||||
f.route()
|
||||
}
|
||||
|
||||
close(release)
|
||||
f.waitFor("")
|
||||
|
||||
for range 50 {
|
||||
f.route()
|
||||
}
|
||||
|
||||
if got := f.queries.Load(); got != 1 {
|
||||
t.Errorf("%d queries for repeated requests, want 1", got)
|
||||
}
|
||||
|
||||
f.clk.advance(time.Hour)
|
||||
f.waitFor(routewatch.NoAnswerYet)
|
||||
f.waitFor("")
|
||||
|
||||
if got := f.queries.Load(); got != 2 {
|
||||
t.Errorf("%d queries after the answer expired, want 2", got)
|
||||
}
|
||||
}
|
||||
@@ -371,8 +371,8 @@ func (rq *request) finish(ctx context.Context) {
|
||||
Status: rq.out.status,
|
||||
RequestBytes: requestBytes,
|
||||
ResponseBytes: responseBytes,
|
||||
DurationRequest: requestlog.Milliseconds(requestEnd.Sub(rq.start)),
|
||||
DurationResponse: requestlog.Milliseconds(end.Sub(requestEnd)),
|
||||
DurationRequestMS: requestlog.Milliseconds(requestEnd.Sub(rq.start)),
|
||||
DurationResponseMS: requestlog.Milliseconds(end.Sub(requestEnd)),
|
||||
Disposition: rq.disposition,
|
||||
DefaultDisposition: rq.byDefault,
|
||||
})
|
||||
@@ -385,14 +385,14 @@ func (rq *request) finish(ctx context.Context) {
|
||||
}
|
||||
|
||||
report := &discriminator.Report{
|
||||
RequestID: rq.id,
|
||||
Status: min(rq.out.status, reportStatusMax),
|
||||
RequestBytes: requestBytes,
|
||||
ResponseBytes: responseBytes,
|
||||
DurationTotal: requestlog.Milliseconds(end.Sub(rq.start)),
|
||||
RequestID: rq.id,
|
||||
Status: min(rq.out.status, reportStatusMax),
|
||||
RequestBytes: requestBytes,
|
||||
ResponseBytes: responseBytes,
|
||||
DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)),
|
||||
}
|
||||
if !rq.upstreamStart.IsZero() {
|
||||
report.DurationUpstreamTotal = requestlog.Milliseconds(end.Sub(rq.upstreamStart))
|
||||
report.DurationUpstreamTotalMS = requestlog.Milliseconds(end.Sub(rq.upstreamStart))
|
||||
}
|
||||
|
||||
rq.h.sendReport(ctx, report)
|
||||
|
||||
Loaded 100 of 114 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user