Compare commits
2
Commits
896ca69a82
...
790f5df069
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
790f5df069 | ||
 clawbot
|
d7c62f2854 |
No files matched your search
+56
-44
@@ -12,8 +12,8 @@ configured only by environment variables, that provides:
|
|||||||
- R1: rate limits per minute, per hour and per day
|
- R1: rate limits per minute, per hour and per day
|
||||||
- R2: netblocks that bypass rate limiting
|
- R2: netblocks that bypass rate limiting
|
||||||
- R3: alerts on attacks through webhook, Slack or ntfy
|
- R3: alerts on attacks through webhook, Slack or ntfy
|
||||||
- R4: alerts on anomalies: requests or bytes per minute or hour, for given IPs
|
- R4: alerts on anomalies: requests made or bytes transferred per minute or
|
||||||
or netblocks, crossing a threshold
|
hour, for given IPs or netblocks, crossing a threshold
|
||||||
- R5: temporary blocks for abusers, permanent bans for repeat offenders
|
- R5: temporary blocks for abusers, permanent bans for repeat offenders
|
||||||
- R6: one or more RBLs or IP reputation APIs
|
- R6: one or more RBLs or IP reputation APIs
|
||||||
- R7: AS number lookup
|
- R7: AS number lookup
|
||||||
@@ -24,14 +24,19 @@ configured only by environment variables, that provides:
|
|||||||
|
|
||||||
## Verdict
|
## Verdict
|
||||||
|
|
||||||
Nothing existing is suitable. No surveyed tool meets R8 (scaling limits by AS
|
Nothing existing is suitable. No surveyed tool meets R8, which asks for limits
|
||||||
number or country) or the bytes half of R4 at all, and the tools that come
|
scaled by AS number or country. None meets R4's alerts on bytes transferred. The
|
||||||
closest on the rest fail R10: they need several containers, YAML or a web UI.
|
tools that come closest on the rest fail R10: they need several containers, YAML
|
||||||
The recommendation is to build a small single-binary sidecar; `SPEC.md` in this
|
or a web UI. The recommendation is to build a small single-binary sidecar;
|
||||||
directory begins that spec. It reuses the mature parts of the surveyed field as
|
`SPEC.md` in this directory begins that spec.
|
||||||
libraries and data sources (Coraza and the OWASP Core Rule Set for attack
|
|
||||||
detection, DNSBLs, AbuseIPDB, optionally a CrowdSec decision feed) instead of
|
The sidecar reuses the mature parts of the surveyed field instead of
|
||||||
reimplementing them.
|
reimplementing them. It takes these as libraries and data sources:
|
||||||
|
|
||||||
|
- Coraza and the OWASP Core Rule Set, for attack detection
|
||||||
|
- DNSBLs
|
||||||
|
- AbuseIPDB
|
||||||
|
- optionally, a CrowdSec decision feed
|
||||||
|
|
||||||
Closest existing options, and why each still falls short:
|
Closest existing options, and why each still falls short:
|
||||||
|
|
||||||
@@ -45,13 +50,16 @@ Closest existing options, and why each still falls short:
|
|||||||
external blacklist URLs, optional CrowdSec), R7 partly (AS number used for
|
external blacklist URLs, optional CrowdSec), R7 partly (AS number used for
|
||||||
blacklist and whitelist decisions), R9 (ModSecurity with the Core Rule
|
blacklist and whitelist decisions), R9 (ModSecurity with the Core Rule
|
||||||
Set, or Coraza plugin), env-var settings.
|
Set, or Coraza plugin), env-var settings.
|
||||||
- Fails: R8 (AS number and country can only allow or deny, never scale a
|
- Fails:
|
||||||
limit), R4 (no volume or byte threshold alerts in the free edition;
|
- R8: AS number and country can only allow or deny, never scale a limit.
|
||||||
reporting is a paid feature), R5 escalation, and R10 in spirit: since 1.6
|
- R4: no alerts on request-count or byte thresholds in the free edition;
|
||||||
it needs a `bunkerweb` container plus a `bw-scheduler` container and a
|
reporting is a paid feature.
|
||||||
database, or the all-in-one image that bundles nginx, scheduler, UI and
|
- R5: no escalation for repeat offenders.
|
||||||
Redis in one container. It is designed to be the front door for many
|
- R10: since 1.6 it is not one plain sidecar. It needs a `bunkerweb`
|
||||||
sites, not a per-app sidecar. AGPL-3.0.
|
container, a `bw-scheduler` container and a database, or the
|
||||||
|
all-in-one image that bundles nginx, the scheduler, the UI and Redis.
|
||||||
|
- It is designed as the proxy in front of many sites, not as a sidecar for
|
||||||
|
one app. AGPL-3.0.
|
||||||
- Unverified: whether several rates (minute, hour, day) can be stacked on
|
- Unverified: whether several rates (minute, hour, day) can be stacked on
|
||||||
the same URL pattern; the documented design is one rate per URL pattern.
|
the same URL pattern; the documented design is one rate per URL pattern.
|
||||||
- CrowdSec is the strongest detection and ban engine, but it is not a proxy.
|
- CrowdSec is the strongest detection and ban engine, but it is not a proxy.
|
||||||
@@ -66,16 +74,20 @@ Closest existing options, and why each still falls short:
|
|||||||
listed AS numbers is possible, but each is a hand-written YAML scenario,
|
listed AS numbers is possible, but each is a hand-written YAML scenario,
|
||||||
it reacts after the fact by banning, and it is not an inline limiter that
|
it reacts after the fact by banning, and it is not an inline limiter that
|
||||||
answers 429.
|
answers 429.
|
||||||
- Fails: R10 (needs the security engine container with persistent state, log
|
- Fails:
|
||||||
acquisition from traefik, a bouncer such as the traefik plugin, and YAML
|
- R10, because it needs:
|
||||||
for acquisition, profiles, scenarios and notifications), bytes half of R4.
|
- the security engine container with persistent state
|
||||||
|
- log acquisition from traefik
|
||||||
|
- a bouncer such as the traefik plugin
|
||||||
|
- YAML for acquisition, profiles, scenarios and notifications
|
||||||
|
- R4's alerts on bytes.
|
||||||
- CrowdSec plus traefik's own `rateLimit` middleware is the best combination
|
- CrowdSec plus traefik's own `rateLimit` middleware is the best combination
|
||||||
with no new code. It gives inline limiting (one window per middleware, keyed
|
with no new code. It gives inline limiting (one window per middleware, keyed
|
||||||
by IP, with `sourceCriterion` exclusions), bans with escalation, alerts and
|
by IP, with `sourceCriterion` exclusions), bans with escalation, alerts and
|
||||||
reputation. It still fails R8, the bytes half of R4, R1's three stacked
|
reputation lookups. It still fails R8, R4's alerts on bytes, and R10 entirely:
|
||||||
windows only by chaining three middlewares per router, and R10 entirely: the
|
the configuration lives in traefik labels and CrowdSec YAML on each host, not
|
||||||
configuration lives in traefik labels and CrowdSec YAML on each host, not in
|
in one sidecar's environment. It meets R1's three windows only by chaining
|
||||||
one sidecar's environment.
|
three middlewares per router.
|
||||||
|
|
||||||
## Candidate by candidate
|
## Candidate by candidate
|
||||||
|
|
||||||
@@ -87,10 +99,10 @@ Closest existing options, and why each still falls short:
|
|||||||
cached, streamed decision list (recommended), standalone against the
|
cached, streamed decision list (recommended), standalone against the
|
||||||
central API, or AppSec only. Supports captcha remediation.
|
central API, or AppSec only. Supports captcha remediation.
|
||||||
- Covers: R5, R6, R9, R3 and R7 through the engine (see Verdict).
|
- Covers: R5, R6, R9, R3 and R7 through the engine (see Verdict).
|
||||||
- Misses: the plugin itself does no rate limiting; R8; R4 bytes.
|
- Misses: the plugin itself does no rate limiting; R8; R4's alerts on bytes.
|
||||||
- Configuration: traefik static config to load the plugin, dynamic config or
|
- Configuration: traefik static config to load the plugin, dynamic config or
|
||||||
labels for the middleware, CrowdSec YAML for everything else.
|
labels for the middleware, CrowdSec YAML for everything else.
|
||||||
- Sidecar fit: no. It lives inside traefik, plus a separate engine
|
- Sidecar fit: no. It runs inside traefik and needs a separate engine
|
||||||
container.
|
container.
|
||||||
- Maturity: widely used (about 900 stars, listed in the traefik plugin
|
- Maturity: widely used (about 900 stars, listed in the traefik plugin
|
||||||
catalog, documented by CrowdSec itself), actively maintained. Traefik
|
catalog, documented by CrowdSec itself), actively maintained. Traefik
|
||||||
@@ -115,7 +127,7 @@ Closest existing options, and why each still falls short:
|
|||||||
- What it is: nginx-derived (Tengine) reverse proxy with a proprietary
|
- What it is: nginx-derived (Tengine) reverse proxy with a proprietary
|
||||||
semantic detection engine and a web console. About 22,000 stars.
|
semantic detection engine and a web console. About 22,000 stars.
|
||||||
- Covers: R9 well, basic per-IP rate limiting with block duration, bot
|
- Covers: R9 well, basic per-IP rate limiting with block duration, bot
|
||||||
challenge, authentication gate.
|
challenge, authentication in front of the app.
|
||||||
- Misses: R8, R7, R4; R1 has one window per rule. Country blocking, alert
|
- Misses: R8, R7, R4; R1 has one window per rule. Country blocking, alert
|
||||||
messages, syslog forwarding and the better reputation database are paid
|
messages, syslog forwarding and the better reputation database are paid
|
||||||
(Lite about 10 USD per month, Pro about 100 USD per month); the free
|
(Lite about 10 USD per month, Pro about 100 USD per month); the free
|
||||||
@@ -128,7 +140,7 @@ Closest existing options, and why each still falls short:
|
|||||||
- Coraza (`corazawaf/coraza`) and Coraza-based proxies
|
- Coraza (`corazawaf/coraza`) and Coraza-based proxies
|
||||||
- What it is: a Go library that implements the ModSecurity rule language and
|
- What it is: a Go library that implements the ModSecurity rule language and
|
||||||
runs the OWASP Core Rule Set. OWASP project, actively maintained, the
|
runs the OWASP Core Rule Set. OWASP project, actively maintained, the
|
||||||
successor path now that ModSecurity is in maintenance only.
|
successor to ModSecurity, which is now in maintenance only.
|
||||||
- Packagings: `coraza-caddy` (Caddy module), `coraza-spoa` (HAProxy),
|
- Packagings: `coraza-caddy` (Caddy module), `coraza-spoa` (HAProxy),
|
||||||
`coraza-proxy-wasm` (Envoy), a traefik WASM plugin, and
|
`coraza-proxy-wasm` (Envoy), a traefik WASM plugin, and
|
||||||
`coreruleset/coraza-crs-docker` (Caddy plus Coraza plus the Core Rule Set,
|
`coreruleset/coraza-crs-docker` (Caddy plus Coraza plus the Core Rule Set,
|
||||||
@@ -142,7 +154,7 @@ Closest existing options, and why each still falls short:
|
|||||||
- ModSecurity Core Rule Set containers (`owasp/modsecurity-crs`)
|
- ModSecurity Core Rule Set containers (`owasp/modsecurity-crs`)
|
||||||
- What it is: official images of Apache or nginx with ModSecurity and the
|
- What it is: official images of Apache or nginx with ModSecurity and the
|
||||||
Core Rule Set, as a reverse proxy configured by env vars (`BACKEND`,
|
Core Rule Set, as a reverse proxy configured by env vars (`BACKEND`,
|
||||||
paranoia level, anomaly thresholds, many more). Current images carry
|
paranoia level, anomaly thresholds, many more). Current images contain
|
||||||
ModSecurity 2.9.x on Apache and Core Rule Set 4.x.
|
ModSecurity 2.9.x on Apache and Core Rule Set 4.x.
|
||||||
- Covers: R9, and R10 (single container, env vars, one backend).
|
- Covers: R9, and R10 (single container, env vars, one backend).
|
||||||
- Misses: R1 to R8.
|
- Misses: R1 to R8.
|
||||||
@@ -152,8 +164,8 @@ Closest existing options, and why each still falls short:
|
|||||||
- Anubis (`TecharoHQ/anubis`), 1.27 current
|
- Anubis (`TecharoHQ/anubis`), 1.27 current
|
||||||
- What it is: a single-binary reverse proxy that makes browsers solve a
|
- What it is: a single-binary reverse proxy that makes browsers solve a
|
||||||
proof-of-work challenge before passing them to `TARGET`. Aimed at
|
proof-of-work challenge before passing them to `TARGET`. Aimed at
|
||||||
scrapers, which is likely a large share of unwanted traffic on a public
|
scrapers, which are likely a large share of the unwanted traffic on a
|
||||||
code forge.
|
public code forge.
|
||||||
- Covers: R10 well (one container, env vars for listener, target,
|
- Covers: R10 well (one container, env vars for listener, target,
|
||||||
difficulty, cookies). Policy rules can match on path, user agent, headers,
|
difficulty, cookies). Policy rules can match on path, user agent, headers,
|
||||||
IP ranges, and with the vendor's hosted data service also AS number and
|
IP ranges, and with the vendor's hosted data service also AS number and
|
||||||
@@ -180,13 +192,13 @@ Closest existing options, and why each still falls short:
|
|||||||
temporary ban, no escalation, state lost on traefik restart) and R2
|
temporary ban, no escalation, state lost on traefik restart) and R2
|
||||||
(allowlist). Nothing else.
|
(allowlist). Nothing else.
|
||||||
- `juitde/traefik-plugin-fail2ban`, fail2ban connector plugins, GeoBlock
|
- `juitde/traefik-plugin-fail2ban`, fail2ban connector plugins, GeoBlock
|
||||||
plugins (country allow or deny through an external lookup API): same
|
plugins (country allow or deny through an external lookup API): each adds
|
||||||
shape, each one small feature, configured in traefik, state in memory.
|
one small feature, is configured in traefik and keeps its state in memory.
|
||||||
- Sidecar fit: no; they live inside traefik.
|
- Sidecar fit: no; they live inside traefik.
|
||||||
- Traefik built-in middlewares
|
- Traefik built-in middlewares
|
||||||
- `rateLimit` (one average-and-burst window per middleware, optional Redis
|
- `rateLimit` (one average-and-burst window per middleware, optional Redis
|
||||||
in traefik 3.x), `inFlightReq`, `ipAllowList`. No bans, alerts, reputation
|
in traefik 3.x), `inFlightReq`, `ipAllowList`. No bans, alerts, reputation
|
||||||
or AS number awareness.
|
lookups or AS number lookups.
|
||||||
- caddy-waf (`fabriziosalmi/caddy-waf`), 0.4.x
|
- caddy-waf (`fabriziosalmi/caddy-waf`), 0.4.x
|
||||||
- What it is: a Caddy module with regex rules and anomaly scoring, per-IP
|
- What it is: a Caddy module with regex rules and anomaly scoring, per-IP
|
||||||
and per-path rate limiting with one configurable window, IP and DNS
|
and per-path rate limiting with one configurable window, IP and DNS
|
||||||
@@ -198,12 +210,12 @@ Closest existing options, and why each still falls short:
|
|||||||
- open-appsec (Check Point)
|
- open-appsec (Check Point)
|
||||||
- Machine-learning WAF agent attached to nginx, Kong, Envoy or similar, with
|
- Machine-learning WAF agent attached to nginx, Kong, Envoy or similar, with
|
||||||
a declarative policy file or the vendor's cloud console. Covers R9 only;
|
a declarative policy file or the vendor's cloud console. Covers R9 only;
|
||||||
rate limiting and richer features are in paid tiers. Not a sidecar in the
|
rate limiting and richer features are in paid tiers. It attaches to an
|
||||||
required sense.
|
existing proxy, so it is not the sidecar R10 asks for.
|
||||||
- iocaine
|
- iocaine
|
||||||
- Serves generated garbage pages to clients the fronting proxy classifies as
|
- Serves generated garbage pages to clients the fronting proxy classifies as
|
||||||
scrapers. Not a limiter, WAF or ban tool; out of scope except as a
|
scrapers. Not a limiter, WAF or ban tool; out of scope, and only of
|
||||||
curiosity for scraper traffic.
|
passing interest for scraper traffic.
|
||||||
- Pangolin
|
- Pangolin
|
||||||
- A tunnelled access platform that bundles traefik and optionally CrowdSec.
|
- A tunnelled access platform that bundles traefik and optionally CrowdSec.
|
||||||
Replaces the ingress rather than adding a sidecar; out of scope.
|
Replaces the ingress rather than adding a sidecar; out of scope.
|
||||||
@@ -214,17 +226,17 @@ Closest existing options, and why each still falls short:
|
|||||||
Everyone else offers one window per rule or none.
|
Everyone else offers one window per rule or none.
|
||||||
- R2 bypass netblocks: BunkerWeb, CrowdSec, traefik, fail2ban plugins.
|
- R2 bypass netblocks: BunkerWeb, CrowdSec, traefik, fail2ban plugins.
|
||||||
- R3 alerts: CrowdSec (best), BunkerWeb (denied requests only), SafeLine (paid).
|
- R3 alerts: CrowdSec (best), BunkerWeb (denied requests only), SafeLine (paid).
|
||||||
- R4 volume and byte anomalies: none. CrowdSec can approximate request-count
|
- R4 alerts on request counts and bytes: none. CrowdSec can approximate alerts
|
||||||
anomalies with a custom scenario; nothing handles bytes.
|
on request counts with a custom scenario; no surveyed tool alerts on bytes.
|
||||||
- R5 temporary then permanent: CrowdSec only. BunkerWeb has fixed or permanent,
|
- R5 temporary then permanent: CrowdSec only. BunkerWeb's ban is either one
|
||||||
not both by history.
|
fixed length or permanent, and does not change with a client's history.
|
||||||
- R6 reputation: CrowdSec, BunkerWeb, go-away, caddy-waf (static lists).
|
- R6 reputation: CrowdSec, BunkerWeb, go-away, caddy-waf (static lists).
|
||||||
- R7 AS number lookup: CrowdSec (enrichment), BunkerWeb and caddy-waf (allow or
|
- R7 AS number lookup: CrowdSec (enrichment), BunkerWeb and caddy-waf (allow or
|
||||||
deny), Anubis (hosted service).
|
deny), Anubis (hosted service).
|
||||||
- R8 biased thresholds: none.
|
- R8 biased thresholds: none.
|
||||||
- R9 attack detection: Coraza or ModSecurity with the Core Rule Set (open,
|
- R9 attack detection: Coraza or ModSecurity with the Core Rule Set (open,
|
||||||
standard), CrowdSec AppSec, SafeLine (closed engine), BunkerWeb (wraps the
|
standard), CrowdSec AppSec, SafeLine (closed engine), BunkerWeb (wraps Coraza
|
||||||
first).
|
or ModSecurity).
|
||||||
- R10 env-var sidecar: `owasp/modsecurity-crs`, `coraza-crs-docker`, Anubis.
|
- R10 env-var sidecar: `owasp/modsecurity-crs`, `coraza-crs-docker`, Anubis.
|
||||||
None of these covers anything beyond its one job.
|
None of these covers anything beyond its one job.
|
||||||
|
|
||||||
|
|||||||
@@ -25,11 +25,11 @@ without an answer in time, `rproxy` passes the request to the app.
|
|||||||
It protects against:
|
It protects against:
|
||||||
|
|
||||||
- Scrapers that fetch every page of a site, all day. Each client gets a limit on
|
- Scrapers that fetch every page of a site, all day. Each client gets a limit on
|
||||||
requests and on bytes per minute, per hour and per day. The defaults leave
|
requests and on body bytes per minute, per hour and per day. The defaults
|
||||||
room for a person reading a site for hours. They catch a scraper that keeps up
|
leave room for a person reading a site for hours. They catch a scraper that
|
||||||
one request every 2 seconds. A client over a limit is refused and banned: for
|
keeps up one request every 2 seconds. A client that breaks a limit is banned:
|
||||||
a while at first, for three times as long at each repeat, and permanently when
|
for a while at first, for three times as long at each repeat, and permanently
|
||||||
it keeps at it.
|
when it keeps at it.
|
||||||
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
|
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
|
||||||
default rule file bans such a client at its first probe, and its next request
|
default rule file bans such a client at its first probe, and its next request
|
||||||
during the ban makes the ban permanent. Your own rules, or a list of trap
|
during the ban makes the ban permanent. Your own rules, or a list of trap
|
||||||
@@ -210,128 +210,133 @@ account, a key, a token or an alert destination stays off until you give one.
|
|||||||
percentage is a whole number from 0 to 100. A path prefix starts with `/`.
|
percentage is a whole number from 0 to 100. A path prefix starts with `/`.
|
||||||
- `off` switches a limit, a threshold or a timeout off where the table says a
|
- `off` switches a limit, a threshold or a timeout off where the table says a
|
||||||
setting takes it.
|
setting takes it.
|
||||||
|
- The byte limits and the byte anomaly thresholds count only body bytes: the
|
||||||
|
app's response bodies, the visitors' request bodies, or both, as
|
||||||
|
`SWW_BYTES_COUNT` says. Headers are never counted. What the visitor sent over
|
||||||
|
a WebSocket counts as request bytes, and what the app sent as response bytes.
|
||||||
- An invalid setting stops the process at start with one log line naming the
|
- An invalid setting stops the process at start with one log line naming the
|
||||||
variable and what is wrong with the value. The settings in effect are logged
|
variable and what is wrong with the value. The settings in effect are logged
|
||||||
on one line at start, with tokens, keys and webhook paths shown as `********`.
|
on one line at start, with tokens, keys and webhook paths shown as `********`.
|
||||||
|
|
||||||
Every setting the code reads, with its default:
|
Every setting the code reads, with its default:
|
||||||
|
|
||||||
| Variable | Default | What it does |
|
| Variable | Default | What it does |
|
||||||
| ---------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
| ---------------------------------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `discriminator`. |
|
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `discriminator`. |
|
||||||
| `SWRP_WAF_BACKEND` | required | `discriminator`'s URL, such as `http://discriminator:8080`. |
|
| `SWRP_WAF_BACKEND` | required | `discriminator`'s URL, such as `http://discriminator:8080`. |
|
||||||
| `SWRP_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. |
|
| `SWRP_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. |
|
||||||
| `SWRP_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. |
|
| `SWRP_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. |
|
||||||
| `SWRP_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. |
|
| `SWRP_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. |
|
||||||
| `SWRP_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `discriminator`'s answer to a question and to a report, in milliseconds. |
|
| `SWRP_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `discriminator`'s answer to a question and to a report, in milliseconds. |
|
||||||
| `SWRP_DEFAULT_DISPOSITION` | `allow` | What `rproxy` does with a request when `discriminator` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. |
|
| `SWRP_DEFAULT_DISPOSITION` | `allow` | What `rproxy` does with a request when `discriminator` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. |
|
||||||
| `SWRP_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. |
|
| `SWRP_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. |
|
||||||
| `SWRP_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`; a larger one is answered `502`, or cut off once it has started. |
|
| `SWRP_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
|
||||||
| `SWRP_MAX_REQUEST_TIME_MS` | `60000` | How long the visitor may take to send its request line and headers, and then the whole request to reach the app, in milliseconds or `off`. |
|
| `SWRP_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
|
||||||
| `SWRP_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
|
| `SWRP_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
|
||||||
| `SWW_LISTEN_ADDR` | `:8080` | Where `discriminator` listens; leave the host part empty. |
|
| `SWW_LISTEN_ADDR` | `:8080` | Where `discriminator` listens; leave the host part empty. |
|
||||||
| `SWW_INSTANCE_NAME` | the container's host name | The name in every line `discriminator` logs, and in every alert and metric, such as `fsn1/discriminator`. |
|
| `SWW_INSTANCE_NAME` | the container's host name | The name in every line `discriminator` logs, and in every alert and metric, such as `fsn1/discriminator`. |
|
||||||
| `SWW_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
|
| `SWW_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
|
||||||
| `SWW_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. |
|
| `SWW_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. |
|
||||||
| `SWW_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. |
|
| `SWW_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. |
|
||||||
| `SWW_MAX_BANS` | `5000` | The most bans `discriminator` keeps of its own; bans an admin made do not count. |
|
| `SWW_MAX_BANS` | `5000` | The most bans `discriminator` keeps of its own; bans an admin made do not count. |
|
||||||
| `SWW_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. |
|
| `SWW_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. |
|
||||||
| `SWW_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients the request and byte limits neither count nor refuse. |
|
| `SWW_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients are exempt from rate limiting and the byte limits. |
|
||||||
| `SWW_DENY_NETS` | empty | Netblocks whose clients are refused. |
|
| `SWW_DENY_NETS` | empty | Netblocks whose clients are refused. |
|
||||||
| `SWW_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; one more bans it. |
|
| `SWW_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; the next request is refused and bans the client. |
|
||||||
| `SWW_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; one more bans it. |
|
| `SWW_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; the next request is refused and bans the client. |
|
||||||
| `SWW_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; one more bans it. |
|
| `SWW_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; the next request is refused and bans the client. |
|
||||||
| `SWW_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes whose requests the request and byte limits neither count nor refuse, such as `/assets/`. |
|
| `SWW_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. |
|
||||||
| `SWW_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes a client's requests may carry in a minute, a size or `off`; the request that takes a client past any byte limit bans it. |
|
| `SWW_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes counted for a client in a minute, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||||
| `SWW_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes a client's requests may carry in an hour, a size or `off`. |
|
| `SWW_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes counted for a client in an hour, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||||
| `SWW_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes a client's requests may carry in a day, a size or `off`. |
|
| `SWW_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes counted for a client in a day, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||||
| `SWW_BYTES_COUNT` | `both` | Which body bytes the byte limits count: `response`, `request` or `both`. |
|
| `SWW_BYTES_COUNT` | `both` | What the byte limits and the byte anomaly thresholds count: `response`, the app's response bodies; `request`, the visitors' request bodies; or `both`. What the visitor sent over a WebSocket counts as request bytes, and what the app sent as response bytes. |
|
||||||
| `SWW_ERROR_BURST_THRESHOLD` | `30` | The most refusals by a rule file, a trap path or the Core Rule Set a client may get in a minute, a count or `off`; one more bans it. |
|
| `SWW_ERROR_BURST_THRESHOLD` | `30` | The most refusals a client may get in a minute from a rule file, a trap path or the Core Rule Set, a count or `off`. The next one bans the client. |
|
||||||
| `SWW_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. |
|
| `SWW_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. |
|
||||||
| `SWW_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. |
|
| `SWW_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. |
|
||||||
| `SWW_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A limit broken again within this time after the last such ban ended gets a ban three times as long. |
|
| `SWW_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A client that breaks a limit again within this time after its last such ban ended gets a ban three times as long. |
|
||||||
| `SWW_MAX_BAN_DURATION` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. |
|
| `SWW_MAX_BAN_DURATION` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. |
|
||||||
| `SWW_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. |
|
| `SWW_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. |
|
||||||
| `SWW_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
|
| `SWW_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
|
||||||
| `SWW_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs`, a web service, `file`, or `off`. |
|
| `SWW_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
|
||||||
| `SWW_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
| `SWW_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
||||||
| `SWW_LOOKUP_TIMEOUT` | `800ms` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take. |
|
| `SWW_LOOKUP_TIMEOUT` | `800ms` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take. |
|
||||||
| `SWW_ADD_LOOKUP_HEADERS` | `false` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
| `SWW_ADD_LOOKUP_HEADERS` | `false` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
||||||
| `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
| `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
||||||
| `SWW_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
| `SWW_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
||||||
| `SWW_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. |
|
| `SWW_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. |
|
||||||
| `SWW_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. |
|
| `SWW_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. |
|
||||||
| `SWW_ASN_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. |
|
| `SWW_ROUTEWATCH_URL` | unset | The URL of a routewatch instance, which gives each client's `asn=` and `prefix=` tags. Unset, nothing asks routewatch. |
|
||||||
| `SWW_COUNTRY_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. |
|
| `SWW_ASN_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. |
|
||||||
| `SWW_UNKNOWN_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. |
|
| `SWW_COUNTRY_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. |
|
||||||
| `SWW_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_TAG_LIMIT_PERCENT` (the lowest wins). |
|
| `SWW_UNKNOWN_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. |
|
||||||
| `SWW_RULES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. |
|
| `SWW_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_TAG_LIMIT_PERCENT` (the lowest wins). |
|
||||||
| `SWW_RULES_ENABLED` | `true` | `false` reads no rule file. |
|
| `SWW_RULES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. |
|
||||||
| `SWW_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. |
|
| `SWW_RULES_ENABLED` | `true` | `false` reads no rule file. |
|
||||||
| `SWW_WAF_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. |
|
| `SWW_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. |
|
||||||
| `SWW_WAF_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. |
|
| `SWW_WAF_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. |
|
||||||
| `SWW_WAF_ANOMALY_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. |
|
| `SWW_WAF_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. |
|
||||||
| `SWW_WAF_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. |
|
| `SWW_WAF_ANOMALY_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. |
|
||||||
| `SWW_WAF_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
|
| `SWW_WAF_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. |
|
||||||
| `SWW_WAF_BODY_LIMIT` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. |
|
| `SWW_WAF_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
|
||||||
| `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
|
| `SWW_WAF_BODY_LIMIT` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. |
|
||||||
| `SWW_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
|
| `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
|
||||||
| `SWW_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. |
|
| `SWW_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
|
||||||
| `SWW_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. |
|
| `SWW_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. |
|
||||||
| `SWW_DNSBL_ZONES` | empty | DNS blocklists asked about each new client in the background, such as `dnsbl.dronebl.org`. |
|
| `SWW_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. |
|
||||||
| `SWW_DNSBL_RESOLVER` | unset | The resolver the zones are asked through, an IP address with an optional port; unset, the host's resolver is used. |
|
| `SWW_DNSBL_ZONES` | empty | DNS blocklists asked about each new client in the background, such as `dnsbl.dronebl.org`. |
|
||||||
| `SWW_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account; set, a client that has committed an offence is checked in the background. |
|
| `SWW_DNSBL_RESOLVER` | unset | The resolver the zones are asked through, an IP address with an optional port; unset, the host's resolver is used. |
|
||||||
| `SWW_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. |
|
| `SWW_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account; set, a client that has committed an offence is checked in the background. |
|
||||||
| `SWW_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. |
|
| `SWW_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. |
|
||||||
| `SWW_CROWDSEC_LAPI_URL` | unset | The local API of a CrowdSec engine you run, such as `http://172.17.0.1:8080`; its decision list is fetched every minute, and a listed client is banned. |
|
| `SWW_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. |
|
||||||
| `SWW_CROWDSEC_LAPI_KEY` | unset | The key the engine gave for `smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it; set with the URL, never alone. |
|
| `SWW_CROWDSEC_LAPI_URL` | unset | The local API of a CrowdSec engine you run, such as `http://172.17.0.1:8080`; its decision list is fetched every minute, and a listed client is banned. |
|
||||||
| `SWW_REPUTATION_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:<percent>` or `log`. |
|
| `SWW_CROWDSEC_LAPI_KEY` | unset | The key the engine gave for `smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it; set with the URL, never alone. |
|
||||||
| `SWW_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. |
|
| `SWW_REPUTATION_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:<percent>` or `log`. |
|
||||||
| `SWW_REPUTATION_TIMEOUT` | `2s` | How long a DNS blocklist query or an AbuseIPDB check may take. |
|
| `SWW_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. |
|
||||||
| `SWW_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. |
|
| `SWW_REPUTATION_TIMEOUT` | `2s` | How long a DNS blocklist query or an AbuseIPDB check may take. |
|
||||||
| `SWW_STATE_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. |
|
| `SWW_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. |
|
||||||
| `SWW_STATE_COUNTER_INTERVAL` | `15m` | How often every state file is written. |
|
| `SWW_STATE_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. |
|
||||||
| `SWW_LOG_LEVEL` | `info` | The least severe of `discriminator`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. |
|
| `SWW_STATE_COUNTER_INTERVAL` | `15m` | How often every state file is written. |
|
||||||
| `SWW_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. |
|
| `SWW_LOG_LEVEL` | `info` | The least severe of `discriminator`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. |
|
||||||
| `SWW_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `discriminator` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. |
|
| `SWW_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. |
|
||||||
| `SWW_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. |
|
| `SWW_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `discriminator` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. |
|
||||||
| `SWW_LOG_REMOTE_BUFFER` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. |
|
| `SWW_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. |
|
||||||
| `SWW_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. |
|
| `SWW_LOG_REMOTE_BUFFER` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. |
|
||||||
| `SWW_LOG_REMOTE_APP_NAME` | `SWW_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. |
|
| `SWW_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. |
|
||||||
| `SWW_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. |
|
| `SWW_LOG_REMOTE_APP_NAME` | `SWW_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. |
|
||||||
| `SWW_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. |
|
| `SWW_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. |
|
||||||
| `SWW_METRICS_TOP_N` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. |
|
| `SWW_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. |
|
||||||
| `SWW_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. |
|
| `SWW_METRICS_TOP_N` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. |
|
||||||
| `SWW_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer <token>`. |
|
| `SWW_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. |
|
||||||
| `SWW_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. |
|
| `SWW_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer <token>`. |
|
||||||
| `SWW_ALERT_NTFY_URL` | unset | An ntfy topic each alert is published to, as its full URL, such as `https://ntfy.sh/my-alerts`. |
|
| `SWW_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. |
|
||||||
| `SWW_ALERT_NTFY_TOKEN` | unset | An ntfy access token sent with each alert. |
|
| `SWW_ALERT_NTFY_URL` | unset | An ntfy topic each alert is published to, as its full URL, such as `https://ntfy.sh/my-alerts`. |
|
||||||
| `SWW_ALERT_EVENTS` | `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. |
|
| `SWW_ALERT_NTFY_TOKEN` | unset | An ntfy access token sent with each alert. |
|
||||||
| `SWW_ALERT_COOLDOWN` | `15m` | How long after an alert its repeats are held back and counted, a duration or `off`. |
|
| `SWW_ALERT_EVENTS` | `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. |
|
||||||
| `SWW_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. |
|
| `SWW_ALERT_COOLDOWN` | `15m` | How long after an alert its repeats are held back and counted, a duration or `off`. |
|
||||||
| `SWW_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Alert when one client makes more requests than this in a minute, a count. |
|
| `SWW_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. |
|
||||||
| `SWW_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Alert when one client makes more requests than this in an hour, a count. |
|
| `SWW_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Alert when one client makes more requests than this in a minute, a count. |
|
||||||
| `SWW_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Alert when one client's requests carry more bytes than this in a minute, a size. |
|
| `SWW_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Alert when one client makes more requests than this in an hour, a count. |
|
||||||
| `SWW_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Alert when one client's requests carry more bytes than this in an hour, a size. |
|
| `SWW_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one client in a minute, a size. |
|
||||||
| `SWW_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Alert when the netblock around a client makes more requests than this in a minute, a count. |
|
| `SWW_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one client in an hour, a size. |
|
||||||
| `SWW_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Alert when the netblock around a client makes more requests than this in an hour, a count. |
|
| `SWW_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Alert when the netblock around a client makes more requests than this in a minute, a count. |
|
||||||
| `SWW_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Alert when the netblock around a client carries more bytes than this in a minute, a size. |
|
| `SWW_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Alert when the netblock around a client makes more requests than this in an hour, a count. |
|
||||||
| `SWW_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Alert when the netblock around a client carries more bytes than this in an hour, a size. |
|
| `SWW_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the netblock around a client in a minute, a size. |
|
||||||
| `SWW_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_ANOMALY_NET_` thresholds count, from 0 to 32. |
|
| `SWW_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the netblock around a client in an hour, a size. |
|
||||||
| `SWW_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_ANOMALY_NET_` thresholds count, from 0 to 128. |
|
| `SWW_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_ANOMALY_NET_` thresholds count, from 0 to 32. |
|
||||||
| `SWW_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Alert when one AS number makes more requests than this in a minute, a count. |
|
| `SWW_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_ANOMALY_NET_` thresholds count, from 0 to 128. |
|
||||||
| `SWW_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Alert when one AS number makes more requests than this in an hour, a count. |
|
| `SWW_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Alert when one AS number makes more requests than this in a minute, a count. |
|
||||||
| `SWW_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Alert when one AS number carries more bytes than this in a minute, a size. |
|
| `SWW_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Alert when one AS number makes more requests than this in an hour, a count. |
|
||||||
| `SWW_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Alert when one AS number carries more bytes than this in an hour, a size. |
|
| `SWW_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one AS number in a minute, a size. |
|
||||||
| `SWW_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Alert when the whole service gets more requests than this in a minute, a count. |
|
| `SWW_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one AS number in an hour, a size. |
|
||||||
| `SWW_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Alert when the whole service gets more requests than this in an hour, a count. |
|
| `SWW_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Alert when the whole service gets more requests than this in a minute, a count. |
|
||||||
| `SWW_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Alert when the whole service carries more bytes than this in a minute, a size. |
|
| `SWW_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Alert when the whole service gets more requests than this in an hour, a count. |
|
||||||
| `SWW_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Alert when the whole service carries more bytes than this in an hour, a size. |
|
| `SWW_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the whole service in a minute, a size. |
|
||||||
| `SWW_WATCH_NETS` | empty | Named netblocks the `SWW_WATCH_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. |
|
| `SWW_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the whole service in an hour, a size. |
|
||||||
| `SWW_WATCH_REQUESTS_PER_MINUTE` | `off` | Alert when a watched netblock makes more requests than this in a minute, a count. |
|
| `SWW_WATCH_NETS` | empty | Named netblocks the `SWW_WATCH_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. |
|
||||||
| `SWW_WATCH_REQUESTS_PER_HOUR` | `off` | Alert when a watched netblock makes more requests than this in an hour, a count. |
|
| `SWW_WATCH_REQUESTS_PER_MINUTE` | `off` | Alert when a watched netblock makes more requests than this in a minute, a count. |
|
||||||
| `SWW_WATCH_BYTES_PER_MINUTE` | `off` | Alert when a watched netblock carries more bytes than this in a minute, a size. |
|
| `SWW_WATCH_REQUESTS_PER_HOUR` | `off` | Alert when a watched netblock makes more requests than this in an hour, a count. |
|
||||||
| `SWW_WATCH_BYTES_PER_HOUR` | `off` | Alert when a watched netblock carries more bytes than this in an hour, a size. |
|
| `SWW_WATCH_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for a watched netblock in a minute, a size. |
|
||||||
|
| `SWW_WATCH_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for a watched netblock in an hour, a size. |
|
||||||
|
|
||||||
[`docs/settings.md`](docs/settings.md) gives the values each setting takes in
|
[`docs/settings.md`](docs/settings.md) gives the values each setting takes in
|
||||||
full, and the limits that are fixed rather than settings.
|
full, and the limits that are fixed rather than settings.
|
||||||
@@ -527,8 +532,9 @@ of the request reaches the app.
|
|||||||
|
|
||||||
17. The request is passed to the app, streaming.
|
17. The request is passed to the app, streaming.
|
||||||
18. After the answer, `rproxy` sends `discriminator` a report with the status
|
18. After the answer, `rproxy` sends `discriminator` a report with the status
|
||||||
and the bytes each way. For an allowed request, `discriminator` counts the
|
and the body bytes each way. For an allowed request, `discriminator` counts
|
||||||
bytes toward the byte limits and adds the request to the client's history.
|
the bytes `SWW_BYTES_COUNT` names toward the byte limits, bans the client if
|
||||||
|
they take it past one, and adds the request to the client's history.
|
||||||
|
|
||||||
A refused visitor is answered with `SWW_BAN_RESPONSE`. A `block` rule or a Core
|
A refused visitor is answered with `SWW_BAN_RESPONSE`. A `block` rule or a Core
|
||||||
Rule Set match answers `403` and bans nobody, since the Core Rule Set has false
|
Rule Set match answers `403` and bans nobody, since the Core Rule Set has false
|
||||||
@@ -562,7 +568,9 @@ the `tags` field of its log line:
|
|||||||
or `vpn=mullvad`;
|
or `vpn=mullvad`;
|
||||||
- `crowdsec`, `dnsbl=<zone>` and `abuseipdb`, when that reputation source lists
|
- `crowdsec`, `dnsbl=<zone>` and `abuseipdb`, when that reputation source lists
|
||||||
the client;
|
the client;
|
||||||
- `exploitsignature`, when a rule file or the Core Rule Set matched the request.
|
- `exploitsignature`, when a rule file or the Core Rule Set matched the request;
|
||||||
|
- `asn=<number>` and `prefix=<netblock>`, from a routewatch instance, while
|
||||||
|
`SWW_ROUTEWATCH_URL` is set.
|
||||||
|
|
||||||
`SWW_TAG_DENY` refuses requests by tag, and `SWW_TAG_LIMIT_PERCENT` lowers a
|
`SWW_TAG_DENY` refuses requests by tag, and `SWW_TAG_LIMIT_PERCENT` lowers a
|
||||||
client's limits by tag. See "Request tags" in [`SPEC.md`](SPEC.md).
|
client's limits by tag. See "Request tags" in [`SPEC.md`](SPEC.md).
|
||||||
|
|||||||
@@ -22,8 +22,8 @@ to them.
|
|||||||
key or a token stays off until the operator supplies it.
|
key or a token stays off until the operator supplies it.
|
||||||
- It keeps a site answering while the part that decides is down, slow or
|
- It keeps a site answering while the part that decides is down, slow or
|
||||||
restarting (see "Without an answer from `discriminator`").
|
restarting (see "Without an answer from `discriminator`").
|
||||||
- It limits each client's requests and bytes per minute, hour and day. It bounds
|
- It limits each client's requests and body bytes per minute, hour and day. It
|
||||||
the size and duration of every request and answer. It detects attacks.
|
bounds the size and duration of every request and answer. It detects attacks.
|
||||||
- It bans abusers: briefly at first, for seven days on a clear sign of attack,
|
- It bans abusers: briefly at first, for seven days on a clear sign of attack,
|
||||||
and permanently when they keep at it.
|
and permanently when they keep at it.
|
||||||
- It looks up each client's AS number and country. It can refuse whole countries
|
- It looks up each client's AS number and country. It can refuse whole countries
|
||||||
@@ -144,7 +144,9 @@ request log's `heuristic` field gives that name. `refusal_limit` and
|
|||||||
around its address, a /64 by default, since one abuser usually controls a
|
around its address, a /64 by default, since one abuser usually controls a
|
||||||
whole /64. A client is therefore one IPv4 address or one IPv6 group.
|
whole /64. A client is therefore one IPv4 address or one IPv6 group.
|
||||||
3. The tag lists (`tag_lists`; see "Request tags"). Each list that holds the
|
3. The tag lists (`tag_lists`; see "Request tags"). Each list that holds the
|
||||||
client adds its tag. A client in `SWW_ALLOW_NETS` gets no tag.
|
client adds its tag. Next, `routewatch_tags` adds the client's `asn=` and
|
||||||
|
`prefix=` tags while `SWW_ROUTEWATCH_URL` is set. A client in
|
||||||
|
`SWW_ALLOW_NETS` gets no tag.
|
||||||
4. The static lists. A client in `SWW_ALLOW_NETS` skips steps 5 to 15, the byte
|
4. The static lists. A client in `SWW_ALLOW_NETS` skips steps 5 to 15, the byte
|
||||||
limits and the error burst. A client in `SWW_DENY_NETS` is refused
|
limits and the error burst. A client in `SWW_DENY_NETS` is refused
|
||||||
(`deny_nets`); a client in both lists is let through.
|
(`deny_nets`); a client in both lists is let through.
|
||||||
@@ -154,8 +156,8 @@ request log's `heuristic` field gives that name. `refusal_limit` and
|
|||||||
up to `SWW_LOOKUP_TIMEOUT`, only when one of these needs it:
|
up to `SWW_LOOKUP_TIMEOUT`, only when one of these needs it:
|
||||||
- the country list;
|
- the country list;
|
||||||
- a biased threshold;
|
- a biased threshold;
|
||||||
- `SWW_TAG_DENY` or `SWW_TAG_LIMIT_PERCENT` naming an `asn=` or `country=`
|
- `SWW_TAG_DENY` or `SWW_TAG_LIMIT_PERCENT` naming a `country=` tag, or an
|
||||||
tag;
|
`asn=` tag while `SWW_ROUTEWATCH_URL` is unset;
|
||||||
- `SWW_ADD_LOOKUP_HEADERS`.
|
- `SWW_ADD_LOOKUP_HEADERS`.
|
||||||
|
|
||||||
A request that ends before the answer comes is logged without it.
|
A request that ends before the answer comes is logged without it.
|
||||||
@@ -192,11 +194,12 @@ request log's `heuristic` field gives that name. `refusal_limit` and
|
|||||||
sent is removed.
|
sent is removed.
|
||||||
18. After the visitor's answer has ended, `rproxy` sends the report (see "The
|
18. After the visitor's answer has ended, `rproxy` sends the report (see "The
|
||||||
report"). `discriminator` then counts:
|
report"). `discriminator` then counts:
|
||||||
- For a request it allowed, the response bytes, the request bytes or both
|
- For a request it allowed, the body bytes of the response, of the request
|
||||||
are added to the byte counters, as `SWW_BYTES_COUNT` says. For a
|
or of both are added to the byte counters, as `SWW_BYTES_COUNT` says. For
|
||||||
WebSocket, what it carried each way counts once it closes. A client whose
|
a WebSocket, what it carried each way counts once it closes. A client
|
||||||
total passes a byte limit times the percentage has broken that limit and
|
whose total passes a byte limit times the percentage has broken that limit
|
||||||
is banned (`byte_limits`). The response in progress is not cut off.
|
and is banned (`byte_limits`). The request that broke it has already been
|
||||||
|
passed on whole; nothing is cut off.
|
||||||
- For a request it allowed, the client's history and the anomaly counters
|
- For a request it allowed, the client's history and the anomaly counters
|
||||||
are updated, and the alert thresholds are evaluated.
|
are updated, and the alert thresholds are evaluated.
|
||||||
|
|
||||||
@@ -259,7 +262,8 @@ to a ban, each by its own rule:
|
|||||||
clear sign of attack bans it permanently at once.
|
clear sign of attack bans it permanently at once.
|
||||||
- A broken limit, one of:
|
- A broken limit, one of:
|
||||||
- a request over a rate limit;
|
- a request over a rate limit;
|
||||||
- a response that takes the client's byte total past a byte limit;
|
- the bytes counted for a request and its response taking the client's total
|
||||||
|
past a byte limit;
|
||||||
- more than `SWW_ERROR_BURST_THRESHOLD` refusals within a minute.
|
- more than `SWW_ERROR_BURST_THRESHOLD` refusals within a minute.
|
||||||
|
|
||||||
The bans for a broken limit:
|
The bans for a broken limit:
|
||||||
@@ -448,10 +452,15 @@ the same way.
|
|||||||
|
|
||||||
### Byte limits, per client
|
### Byte limits, per client
|
||||||
|
|
||||||
By default there is no minute limit, so a client can download in a burst at full
|
The byte limits count body bytes: the app's response bodies, the visitors'
|
||||||
speed. The hour limit defaults to 50 GiB and the day limit to 100 GiB. They
|
request bodies, or both, as `SWW_BYTES_COUNT` says. Headers are never counted.
|
||||||
allow several downloads of a few GiB each. A response's bytes are counted when
|
What the visitor sent over a WebSocket counts as request bytes, and what the app
|
||||||
it ends (see step 18).
|
sent as response bytes. The bytes counted for a request and its response are
|
||||||
|
added to the client's total once the response has ended (see step 18). So the
|
||||||
|
request that takes a client past a limit is passed on whole, and the ban refuses
|
||||||
|
the client's requests from then on. By default there is no minute limit, so a
|
||||||
|
client can download in a burst at full speed. The hour limit defaults to 50 GiB
|
||||||
|
and the day limit to 100 GiB. They allow several downloads of a few GiB each.
|
||||||
|
|
||||||
### Lookup of AS number and country
|
### Lookup of AS number and country
|
||||||
|
|
||||||
@@ -547,6 +556,19 @@ Each tag and where it comes from:
|
|||||||
- `exploitsignature`, when a rule file or the Core Rule Set matched the request.
|
- `exploitsignature`, when a rule file or the Core Rule Set matched the request.
|
||||||
For the Core Rule Set a match is a score at or over
|
For the Core Rule Set a match is a score at or over
|
||||||
`SWW_WAF_ANOMALY_THRESHOLD`.
|
`SWW_WAF_ANOMALY_THRESHOLD`.
|
||||||
|
- `asn=<number>` and `prefix=<netblock>` from a routewatch instance
|
||||||
|
(`routewatch_tags`), while `SWW_ROUTEWATCH_URL` is set:
|
||||||
|
- The tag `prefix=` holds the netblock the client's address is announced in,
|
||||||
|
such as `prefix=192.0.2.0/24`.
|
||||||
|
- An `asn=` tag has the same spelling as the lookup's. A request carries it
|
||||||
|
once, and carries both when the two sources disagree.
|
||||||
|
- These tags come before `deny_nets`, so a line that check refused has them.
|
||||||
|
- `discriminator` asks routewatch in the background and never waits for it.
|
||||||
|
An answer is kept for 1 hour, in memory only.
|
||||||
|
- After a query fails, routewatch is not asked for 1 minute.
|
||||||
|
- A request without these tags gives the reason in the log field
|
||||||
|
`routewatch_tags_missing` (see
|
||||||
|
[`docs/request-log.md`](docs/request-log.md)).
|
||||||
|
|
||||||
A client in `SWW_ALLOW_NETS` skips the checks and gets no tag. Every tag is in
|
A client in `SWW_ALLOW_NETS` skips the checks and gets no tag. Every tag is in
|
||||||
lower case, except a country code, which is in capitals. `SWW_TAG_DENY` and
|
lower case, except a country code, which is in capitals. `SWW_TAG_DENY` and
|
||||||
@@ -953,7 +975,8 @@ which the metrics show. They reveal a widely distributed scraper, thousands of
|
|||||||
addresses at low rates each, which the per-client limits do not see. A low
|
addresses at low rates each, which the per-client limits do not see. A low
|
||||||
percentage for its `asn=` tag in `SWW_TAG_LIMIT_PERCENT` is the response.
|
percentage for its `asn=` tag in `SWW_TAG_LIMIT_PERCENT` is the response.
|
||||||
|
|
||||||
- Requests and bytes are counted per minute and per hour, for each of:
|
- Requests and body bytes, as `SWW_BYTES_COUNT` counts them, are counted per
|
||||||
|
minute and per hour, for each of:
|
||||||
- a client;
|
- a client;
|
||||||
- the surrounding netblock;
|
- the surrounding netblock;
|
||||||
- an AS number;
|
- an AS number;
|
||||||
@@ -1263,9 +1286,9 @@ short.
|
|||||||
### Answering in time
|
### Answering in time
|
||||||
|
|
||||||
`rproxy` stops waiting for `discriminator` after `SWRP_TIMEOUT_MS`, by default 2
|
`rproxy` stops waiting for `discriminator` after `SWRP_TIMEOUT_MS`, by default 2
|
||||||
seconds, and `SWRP_DEFAULT_DISPOSITION` then decides. So at the defaults,
|
seconds, and `SWRP_DEFAULT_DISPOSITION` then decides. The longest a question
|
||||||
`discriminator` answers a request well within that time, whatever the request
|
waits on an outside source is `SWW_LOOKUP_TIMEOUT`, by default 800 milliseconds,
|
||||||
and whatever its outside sources do.
|
well within that time.
|
||||||
|
|
||||||
GeoJS is the only outside source a question can wait on. A question waits only
|
GeoJS is the only outside source a question can wait on. A question waits only
|
||||||
when step 6 says so, and for at most `SWW_LOOKUP_TIMEOUT`, by default 800
|
when step 6 says so, and for at most `SWW_LOOKUP_TIMEOUT`, by default 800
|
||||||
@@ -1361,9 +1384,9 @@ answer.
|
|||||||
For a request it holds, `discriminator` writes the request's last log line (see
|
For a request it holds, `discriminator` writes the request's last log line (see
|
||||||
"The log"). For a request it allowed, it first:
|
"The log"). For a request it allowed, it first:
|
||||||
|
|
||||||
- counts the bytes as `SWW_BYTES_COUNT` says, toward the client's byte limits at
|
- counts the body bytes `SWW_BYTES_COUNT` names, toward the client's byte limits
|
||||||
the percentages it gave that request;
|
at the percentages it gave that request;
|
||||||
- bans a client that passes a byte limit;
|
- bans a client whose total passes a byte limit;
|
||||||
- adds the request with its status to the client's history, the anomaly counters
|
- adds the request with its status to the client's history, the anomaly counters
|
||||||
and the metrics.
|
and the metrics.
|
||||||
|
|
||||||
|
|||||||
@@ -5,6 +5,9 @@ What `GET /_sww/metrics` answers with, at `discriminator`'s own address.
|
|||||||
keeps the `instance` label as `exported_instance` unless the scrape sets
|
keeps the `instance` label as `exported_instance` unless the scrape sets
|
||||||
`honor_labels: true`. `rproxy` has no metrics.
|
`honor_labels: true`. `rproxy` has no metrics.
|
||||||
|
|
||||||
|
Every `sww_*request_bytes_total` and `sww_*response_bytes_total` series counts
|
||||||
|
body bytes, never headers, whatever `SWW_BYTES_COUNT` says.
|
||||||
|
|
||||||
## Traffic
|
## Traffic
|
||||||
|
|
||||||
- `sww_requests_total`, `sww_request_bytes_total` and
|
- `sww_requests_total`, `sww_request_bytes_total` and
|
||||||
|
|||||||
+14
-6
@@ -145,20 +145,22 @@ Every one of these lines has:
|
|||||||
included. It is left out when rate limiting did not count the request. It
|
included. It is left out when rate limiting did not count the request. It
|
||||||
holds:
|
holds:
|
||||||
- `minute`, `hour` and `day`: requests;
|
- `minute`, `hour` and `day`: requests;
|
||||||
- `minute_bytes`, `hour_bytes` and `day_bytes`: bytes.
|
- `minute_bytes`, `hour_bytes` and `day_bytes`: the body bytes
|
||||||
|
`SWW_BYTES_COUNT` names, in that window.
|
||||||
|
|
||||||
A count can have a fraction (see "Counting method" in
|
A count can have a fraction (see "Counting method" in
|
||||||
[`SPEC.md`](../SPEC.md)). The bytes of an allowed request are counted when
|
[`SPEC.md`](../SPEC.md)). The bytes counted for an allowed request and its
|
||||||
its report comes. So on the `decision` line, the bytes are without this
|
response are added when its report comes. So on the `decision` line, the
|
||||||
request's.
|
three byte totals leave them out.
|
||||||
|
|
||||||
- `limit_hit`: the limit broken:
|
- `limit_hit`: the limit broken:
|
||||||
- `minute`, `hour` or `day` for a rate limit;
|
- `minute`, `hour` or `day` for a rate limit;
|
||||||
- `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit;
|
- `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit;
|
||||||
- `error_burst`.
|
- `error_burst`.
|
||||||
|
|
||||||
A request whose bytes broke a limit was passed on whole, so its `action`
|
When the bytes counted for a request and its response break a limit, the
|
||||||
stays `forward`. Only its `report` line gives that limit.
|
request has been passed on whole, so its `action` stays `forward`. Only its
|
||||||
|
`report` line gives that limit.
|
||||||
|
|
||||||
- `rule_ids`: the rule file rules that matched, in order, up to the one that
|
- `rule_ids`: the rule file rules that matched, in order, up to the one that
|
||||||
refused.
|
refused.
|
||||||
@@ -171,6 +173,12 @@ Every one of these lines has:
|
|||||||
- `abuseipdb`.
|
- `abuseipdb`.
|
||||||
- `tags`: the request's tags, sorted, left out when it has none. Each tag and
|
- `tags`: the request's tags, sorted, left out when it has none. Each tag and
|
||||||
where it comes from is under "Request tags" in [`SPEC.md`](../SPEC.md).
|
where it comes from is under "Request tags" in [`SPEC.md`](../SPEC.md).
|
||||||
|
- `routewatch_tags_missing`: why the request got no tags from routewatch,
|
||||||
|
present only while `SWW_ROUTEWATCH_URL` is set:
|
||||||
|
- `no_answer_yet`: routewatch has not answered about this client yet;
|
||||||
|
- `request_failed`: a query to routewatch failed within the last minute, and
|
||||||
|
the process log says why;
|
||||||
|
- `no_route`: routewatch knows no live route for the address.
|
||||||
- `offence`: `limit`, for a request that broke a limit.
|
- `offence`: `limit`, for a request that broke a limit.
|
||||||
- `ban_expires`: when the client's ban ends, for a request that made one or was
|
- `ban_expires`: when the client's ban ends, for a request that made one or was
|
||||||
refused under one: a time, or `permanent`.
|
refused under one: a time, or `permanent`.
|
||||||
|
|||||||
+31
-8
@@ -98,6 +98,8 @@ between the two (see "The image's command" below).
|
|||||||
|
|
||||||
## Request rate limits, per client
|
## Request rate limits, per client
|
||||||
|
|
||||||
|
The request past one of these limits is refused, and the client is banned.
|
||||||
|
|
||||||
- `SWW_RATE_LIMIT_PER_MINUTE` (default `120`, 2 a second): the most requests a
|
- `SWW_RATE_LIMIT_PER_MINUTE` (default `120`, 2 a second): the most requests a
|
||||||
client may make in a minute. A count or `off`. The default leaves room for a
|
client may make in a minute. A count or `off`. The default leaves room for a
|
||||||
person opening two pages at once with an empty cache, up to 60 files each.
|
person opening two pages at once with an empty cache, up to 60 files each.
|
||||||
@@ -120,11 +122,22 @@ between the two (see "The image's command" below).
|
|||||||
|
|
||||||
## Byte limits, per client
|
## Byte limits, per client
|
||||||
|
|
||||||
|
These count body bytes, never headers. What the visitor sent over a WebSocket
|
||||||
|
counts as request bytes, and what the app sent as response bytes. The bytes
|
||||||
|
counted for a request and its response are added to the client's total once the
|
||||||
|
response has ended. The request that takes a client's total past a limit is not
|
||||||
|
cut off; the client is banned.
|
||||||
|
|
||||||
- `SWW_BYTES_LIMIT_PER_MINUTE` (default `off`), `SWW_BYTES_LIMIT_PER_HOUR`
|
- `SWW_BYTES_LIMIT_PER_MINUTE` (default `off`), `SWW_BYTES_LIMIT_PER_HOUR`
|
||||||
(default `50G`) and `SWW_BYTES_LIMIT_PER_DAY` (default `100G`): the most bytes
|
(default `50G`) and `SWW_BYTES_LIMIT_PER_DAY` (default `100G`): the most bytes
|
||||||
counted for a client in each window. A size or `off`.
|
counted for a client in each window, a size or `off`. `SWW_BYTES_COUNT` says
|
||||||
- `SWW_BYTES_COUNT` (default `both`): which body bytes the byte limits count:
|
which bytes count: response bodies, request bodies or both. The request that
|
||||||
`response`, `request` or `both`.
|
passes the limit is not cut off, and the client is banned when its response
|
||||||
|
ends.
|
||||||
|
- `SWW_BYTES_COUNT` (default `both`): what the byte limits and the byte anomaly
|
||||||
|
thresholds count: `response`, the app's response bodies; `request`, the
|
||||||
|
visitors' request bodies; or `both`. What the visitor sent over a WebSocket
|
||||||
|
counts as request bytes, and what the app sent as response bytes.
|
||||||
|
|
||||||
## Lookup of AS number and country
|
## Lookup of AS number and country
|
||||||
|
|
||||||
@@ -173,11 +186,19 @@ Every request gets tags from what `discriminator` knows of its client. See
|
|||||||
`exploitsignature`: a request gets that tag after the rate limits have counted
|
`exploitsignature`: a request gets that tag after the rate limits have counted
|
||||||
it.
|
it.
|
||||||
|
|
||||||
|
- `SWW_ROUTEWATCH_URL` (default unset): the `http` or `https` URL of a
|
||||||
|
routewatch instance. `discriminator` asks it about each client, in the
|
||||||
|
background, and adds the tags `asn=<number>` and `prefix=<netblock>` from its
|
||||||
|
answer. While it is unset, nothing asks routewatch and no request gets a
|
||||||
|
`prefix=` tag.
|
||||||
|
|
||||||
`SWW_TAG_DENY` and `SWW_TAG_LIMIT_PERCENT` accept tags in either case. A tag
|
`SWW_TAG_DENY` and `SWW_TAG_LIMIT_PERCENT` accept tags in either case. A tag
|
||||||
that no request can get stops the start. A request can get only these:
|
that no request can get stops the start. A request can get only these:
|
||||||
|
|
||||||
- a name in `SWW_TAG_LISTS`;
|
- a name in `SWW_TAG_LISTS`;
|
||||||
- an `asn=` or `country=` tag;
|
- an `asn=` or `country=` tag;
|
||||||
|
- a `prefix=` tag with a netblock, while `SWW_ROUTEWATCH_URL` is set. It matches
|
||||||
|
only that exact announced netblock, such as `prefix=192.0.2.0/24`;
|
||||||
- a `dnsbl=` tag of a zone in `SWW_DNSBL_ZONES`, in lower case, with a key
|
- a `dnsbl=` tag of a zone in `SWW_DNSBL_ZONES`, in lower case, with a key
|
||||||
masked as the `reputation` field shows it;
|
masked as the `reputation` field shows it;
|
||||||
- `abuseipdb`, while `SWW_ABUSEIPDB_KEY` is set;
|
- `abuseipdb`, while `SWW_ABUSEIPDB_KEY` is set;
|
||||||
@@ -233,8 +254,9 @@ the request first.
|
|||||||
a range. So does a directive that removes or changes rules as the file loads,
|
a range. So does a directive that removes or changes rules as the file loads,
|
||||||
such as `SecRuleRemoveById`, since none of the Core Rule Set's rules is loaded
|
such as `SecRuleRemoveById`, since none of the Core Rule Set's rules is loaded
|
||||||
yet. It has no `_FILE` form.
|
yet. It has no `_FILE` form.
|
||||||
- `SWW_ERROR_BURST_THRESHOLD` (default `30`): the most refusals of one client in
|
- `SWW_ERROR_BURST_THRESHOLD` (default `30`): the most refusals a client may get
|
||||||
a minute (see "Attack detection" in `SPEC.md`). A count or `off`.
|
in a minute from a rule file, a trap path or the Core Rule Set, a count or
|
||||||
|
`off`. The next one bans the client (see "Attack detection" in `SPEC.md`).
|
||||||
|
|
||||||
## Ban lengths, response and scope
|
## Ban lengths, response and scope
|
||||||
|
|
||||||
@@ -311,9 +333,10 @@ the request first.
|
|||||||
|
|
||||||
## Anomaly thresholds
|
## Anomaly thresholds
|
||||||
|
|
||||||
Each scope has four thresholds: the most requests and the most bytes counted in
|
Each scope has four thresholds: the most requests and the most body bytes
|
||||||
a minute and in an hour. Requests are counts, bytes are sizes. Each is `off` by
|
counted in a minute and in an hour. The bytes are those `SWW_BYTES_COUNT` names.
|
||||||
default. The four thresholds per client are:
|
Requests are counts, bytes are sizes. Each is `off` by default. The four
|
||||||
|
thresholds per client are:
|
||||||
|
|
||||||
- `SWW_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`;
|
- `SWW_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`;
|
||||||
- `SWW_ANOMALY_CLIENT_REQUESTS_PER_HOUR`;
|
- `SWW_ANOMALY_CLIENT_REQUESTS_PER_HOUR`;
|
||||||
|
|||||||
+1
-1
@@ -61,7 +61,7 @@ since it was first seen:
|
|||||||
- the first and last time seen;
|
- the first and last time seen;
|
||||||
- the AS number, AS name and country last looked up, and when;
|
- the AS number, AS name and country last looked up, and when;
|
||||||
- total requests, and how many were forwarded and how many refused;
|
- total requests, and how many were forwarded and how many refused;
|
||||||
- bytes in each direction;
|
- body bytes of its requests, and of the responses it was sent;
|
||||||
- responses by status class;
|
- responses by status class;
|
||||||
- offences by kind: `limit`, `attack`, `rule_blocked` and `waf_blocked`. A file
|
- offences by kind: `limit`, `attack`, `rule_blocked` and `waf_blocked`. A file
|
||||||
may also hold `token_refused`, which is kept, and which nothing adds to.
|
may also hold `token_refused`, which is kept, and which nothing adds to.
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
|
"maps"
|
||||||
"math"
|
"math"
|
||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
@@ -106,6 +107,9 @@ type Config struct {
|
|||||||
TagLists []TagList
|
TagLists []TagList
|
||||||
TagDeny []string
|
TagDeny []string
|
||||||
TagLimitPercent map[string]int64
|
TagLimitPercent map[string]int64
|
||||||
|
// RouteWatchURL is where the routewatch instance is that adds the asn=
|
||||||
|
// and prefix= tags, "" for none (SWW_ROUTEWATCH_URL).
|
||||||
|
RouteWatchURL string
|
||||||
// The biased thresholds. ASNBytesPercent and CountryBytesPercent give
|
// The biased thresholds. ASNBytesPercent and CountryBytesPercent give
|
||||||
// those they list a percentage of the byte limits in place of the one
|
// those they list a percentage of the byte limits in place of the one
|
||||||
// TagLimitPercent gives their asn= or country= tag
|
// TagLimitPercent gives their asn= or country= tag
|
||||||
@@ -475,7 +479,9 @@ var (
|
|||||||
"is not a tag, : and a percentage, such as vpn:50 or asn=14061:25")
|
"is not a tag, : and a percentage, such as vpn:50 or asn=14061:25")
|
||||||
errNotTagPercent = errors.New("is not a percentage from 1 to 99")
|
errNotTagPercent = errors.New("is not a percentage from 1 to 99")
|
||||||
errNotASNTag = errors.New("is not asn= and a number, such as asn=14061")
|
errNotASNTag = errors.New("is not asn= and a number, such as asn=14061")
|
||||||
errExploitTag = errors.New(
|
errNotPrefixTag = errors.New(
|
||||||
|
"is not prefix= and a netblock, such as prefix=192.0.2.0/24")
|
||||||
|
errExploitTag = errors.New(
|
||||||
"cannot lower limits: the rate limits count a request before its " +
|
"cannot lower limits: the rate limits count a request before its " +
|
||||||
"rules are matched")
|
"rules are matched")
|
||||||
errCrowdSecTag = errors.New(
|
errCrowdSecTag = errors.New(
|
||||||
@@ -483,8 +489,9 @@ var (
|
|||||||
errNoRequestGetsTag = errors.New(
|
errNoRequestGetsTag = errors.New(
|
||||||
"is a tag no request can get: it is not a name in SWW_TAG_LISTS, an " +
|
"is a tag no request can get: it is not a name in SWW_TAG_LISTS, an " +
|
||||||
"asn= or country= tag, a dnsbl= tag of a zone in SWW_DNSBL_ZONES, " +
|
"asn= or country= tag, a dnsbl= tag of a zone in SWW_DNSBL_ZONES, " +
|
||||||
"abuseipdb with SWW_ABUSEIPDB_KEY set, or exploitsignature with " +
|
"a prefix= tag with SWW_ROUTEWATCH_URL set, abuseipdb with " +
|
||||||
"rule files or the Core Rule Set on")
|
"SWW_ABUSEIPDB_KEY set, or exploitsignature with rule files or the " +
|
||||||
|
"Core Rule Set on")
|
||||||
)
|
)
|
||||||
|
|
||||||
// FromEnvironment reads the settings with lookupEnv, normally
|
// FromEnvironment reads the settings with lookupEnv, normally
|
||||||
@@ -524,6 +531,7 @@ func FromEnvironment(
|
|||||||
TagLists: env.tagLists("SWW_TAG_LISTS"),
|
TagLists: env.tagLists("SWW_TAG_LISTS"),
|
||||||
TagDeny: env.tags("SWW_TAG_DENY"),
|
TagDeny: env.tags("SWW_TAG_DENY"),
|
||||||
TagLimitPercent: env.tagPercents("SWW_TAG_LIMIT_PERCENT"),
|
TagLimitPercent: env.tagPercents("SWW_TAG_LIMIT_PERCENT"),
|
||||||
|
RouteWatchURL: env.listURL("SWW_ROUTEWATCH_URL"),
|
||||||
ASNBytesPercent: env.percents("SWW_ASN_BYTES_PERCENT", ParseASN),
|
ASNBytesPercent: env.percents("SWW_ASN_BYTES_PERCENT", ParseASN),
|
||||||
CountryBytesPercent: env.percents("SWW_COUNTRY_BYTES_PERCENT", parseCountry),
|
CountryBytesPercent: env.percents("SWW_COUNTRY_BYTES_PERCENT", parseCountry),
|
||||||
UnknownLimitPercent: env.percent("SWW_UNKNOWN_LIMIT_PERCENT", "100"),
|
UnknownLimitPercent: env.percent("SWW_UNKNOWN_LIMIT_PERCENT", "100"),
|
||||||
@@ -1064,14 +1072,21 @@ func (e *environment) checkCountriesAndLookups(cfg *Config) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// routewatch gives asn= tags too.
|
||||||
|
onlyLookupGives := func(tag string) bool {
|
||||||
|
return isLookupTag(tag) &&
|
||||||
|
(cfg.RouteWatchURL == "" || !strings.HasPrefix(tag, ASNTagPrefix))
|
||||||
|
}
|
||||||
|
|
||||||
for _, setting := range []struct {
|
for _, setting := range []struct {
|
||||||
name string
|
name string
|
||||||
set bool
|
set bool
|
||||||
}{
|
}{
|
||||||
{"SWW_EXCLUSIVELY_ALLOWED_COUNTRIES", len(cfg.ExclusivelyAllowedCountries) > 0},
|
{"SWW_EXCLUSIVELY_ALLOWED_COUNTRIES", len(cfg.ExclusivelyAllowedCountries) > 0},
|
||||||
{"SWW_ADD_LOOKUP_HEADERS", cfg.AddLookupHeaders},
|
{"SWW_ADD_LOOKUP_HEADERS", cfg.AddLookupHeaders},
|
||||||
{"SWW_TAG_DENY", slices.ContainsFunc(cfg.TagDeny, isLookupTag)},
|
{"SWW_TAG_DENY", slices.ContainsFunc(cfg.TagDeny, onlyLookupGives)},
|
||||||
{"SWW_TAG_LIMIT_PERCENT", tagLimitNeedsLookups(cfg)},
|
{"SWW_TAG_LIMIT_PERCENT", slices.ContainsFunc(
|
||||||
|
slices.Collect(maps.Keys(cfg.TagLimitPercent)), onlyLookupGives)},
|
||||||
{"SWW_ASN_BYTES_PERCENT", len(cfg.ASNBytesPercent) > 0},
|
{"SWW_ASN_BYTES_PERCENT", len(cfg.ASNBytesPercent) > 0},
|
||||||
{"SWW_COUNTRY_BYTES_PERCENT", len(cfg.CountryBytesPercent) > 0},
|
{"SWW_COUNTRY_BYTES_PERCENT", len(cfg.CountryBytesPercent) > 0},
|
||||||
{"SWW_UNKNOWN_LIMIT_PERCENT", cfg.UnknownLimitPercent < 100},
|
{"SWW_UNKNOWN_LIMIT_PERCENT", cfg.UnknownLimitPercent < 100},
|
||||||
@@ -1171,6 +1186,10 @@ func requestCanGetTag(cfg *Config, tag string) bool {
|
|||||||
return cfg.AbuseIPDBKey != ""
|
return cfg.AbuseIPDBKey != ""
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if strings.HasPrefix(tag, PrefixTagPrefix) {
|
||||||
|
return cfg.RouteWatchURL != ""
|
||||||
|
}
|
||||||
|
|
||||||
zone, isZoneTag := strings.CutPrefix(tag, DNSBLTagPrefix)
|
zone, isZoneTag := strings.CutPrefix(tag, DNSBLTagPrefix)
|
||||||
|
|
||||||
return isZoneTag && slices.ContainsFunc(cfg.DNSBLZones, func(each string) bool {
|
return isZoneTag && slices.ContainsFunc(cfg.DNSBLZones, func(each string) bool {
|
||||||
@@ -1923,6 +1942,7 @@ const (
|
|||||||
ASNTagPrefix = "asn="
|
ASNTagPrefix = "asn="
|
||||||
CountryTagPrefix = "country="
|
CountryTagPrefix = "country="
|
||||||
DNSBLTagPrefix = "dnsbl="
|
DNSBLTagPrefix = "dnsbl="
|
||||||
|
PrefixTagPrefix = "prefix="
|
||||||
AbuseIPDBTag = "abuseipdb"
|
AbuseIPDBTag = "abuseipdb"
|
||||||
CrowdSecTag = "crowdsec"
|
CrowdSecTag = "crowdsec"
|
||||||
// ExploitSignatureTag is the tag of a request a rule file or the Core
|
// ExploitSignatureTag is the tag of a request a rule file or the Core
|
||||||
@@ -1999,9 +2019,10 @@ func parseTags(value string) ([]string, error) {
|
|||||||
|
|
||||||
// parseTag reads a tag in either case, with spaces around it, and returns it
|
// parseTag reads a tag in either case, with spaces around it, and returns it
|
||||||
// as a request carries it: in lower case, but for a country code, which is in
|
// as a request carries it: in lower case, but for a country code, which is in
|
||||||
// capitals, and an asn= tag's number without leading zeros. An asn= or
|
// capitals, an asn= tag's number without leading zeros, and a prefix= tag's
|
||||||
// country= tag must name a number or a country, so that a mistyped one stops
|
// netblock as routewatch writes it. An asn=, country= or prefix= tag must name
|
||||||
// the start instead of never matching.
|
// a number, a country or a netblock, so that a mistyped one stops the start
|
||||||
|
// instead of never matching.
|
||||||
func parseTag(value string) (string, error) {
|
func parseTag(value string) (string, error) {
|
||||||
tag := strings.ToLower(strings.TrimSpace(value))
|
tag := strings.ToLower(strings.TrimSpace(value))
|
||||||
|
|
||||||
@@ -2023,6 +2044,15 @@ func parseTag(value string) (string, error) {
|
|||||||
return CountryTagPrefix + country, nil
|
return CountryTagPrefix + country, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if text, isPrefix := strings.CutPrefix(tag, PrefixTagPrefix); isPrefix {
|
||||||
|
netblock, err := netip.ParsePrefix(text)
|
||||||
|
if err != nil || netblock != netblock.Masked() {
|
||||||
|
return "", fmt.Errorf("%q %w", value, errNotPrefixTag)
|
||||||
|
}
|
||||||
|
|
||||||
|
return PrefixTagPrefix + netblock.String(), nil
|
||||||
|
}
|
||||||
|
|
||||||
return tag, nil
|
return tag, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -61,6 +61,7 @@ const (
|
|||||||
reputationAction = "SWW_REPUTATION_ACTION"
|
reputationAction = "SWW_REPUTATION_ACTION"
|
||||||
reputationCacheTTL = "SWW_REPUTATION_CACHE_TTL"
|
reputationCacheTTL = "SWW_REPUTATION_CACHE_TTL"
|
||||||
reputationTimeout = "SWW_REPUTATION_TIMEOUT"
|
reputationTimeout = "SWW_REPUTATION_TIMEOUT"
|
||||||
|
routeWatchURL = "SWW_ROUTEWATCH_URL"
|
||||||
crowdSecURL = "SWW_CROWDSEC_LAPI_URL"
|
crowdSecURL = "SWW_CROWDSEC_LAPI_URL"
|
||||||
crowdSecKey = "SWW_CROWDSEC_LAPI_KEY"
|
crowdSecKey = "SWW_CROWDSEC_LAPI_KEY"
|
||||||
banResponse = "SWW_BAN_RESPONSE"
|
banResponse = "SWW_BAN_RESPONSE"
|
||||||
@@ -1403,6 +1404,54 @@ func TestTagSettingsAsSet(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRouteWatchURLAndPrefixTags(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const rw = "https://routewatch.example"
|
||||||
|
|
||||||
|
if cfg := fromEnvironment(t, environment{}); cfg.RouteWatchURL != "" {
|
||||||
|
t.Errorf("default %s is %q, want unset", routeWatchURL, cfg.RouteWatchURL)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := fromEnvironment(t, environment{
|
||||||
|
routeWatchURL: rw, tagDeny: "prefix=2001:DB8::/32",
|
||||||
|
tagLimitPercent: "prefix=192.0.2.0/24:50",
|
||||||
|
})
|
||||||
|
wantDeny := []string{"prefix=2001:db8::/32"}
|
||||||
|
|
||||||
|
if cfg.RouteWatchURL != rw || !slices.Equal(cfg.TagDeny, wantDeny) ||
|
||||||
|
cfg.TagLimitPercent["prefix=192.0.2.0/24"] != 50 {
|
||||||
|
t.Errorf("gave %q, %v, %v", cfg.RouteWatchURL, cfg.TagDeny, cfg.TagLimitPercent)
|
||||||
|
}
|
||||||
|
|
||||||
|
// routewatch gives asn= tags too, so an asn= tag needs no lookups while it is set.
|
||||||
|
fromEnvironment(t, environment{
|
||||||
|
lookupSource: off, routeWatchURL: rw, tagDeny: "asn=64496",
|
||||||
|
})
|
||||||
|
|
||||||
|
for _, stopped := range []struct {
|
||||||
|
env environment
|
||||||
|
name string
|
||||||
|
}{
|
||||||
|
{environment{routeWatchURL: "ftp://routewatch.example"}, routeWatchURL},
|
||||||
|
{environment{routeWatchURL: "routewatch.example"}, routeWatchURL},
|
||||||
|
{environment{tagDeny: "prefix=192.0.2.0/24"}, tagDeny},
|
||||||
|
{environment{routeWatchURL: rw, tagDeny: "prefix=nowhere"}, tagDeny},
|
||||||
|
{environment{routeWatchURL: rw, tagDeny: "prefix=192.0.2.5/24"}, tagDeny},
|
||||||
|
{
|
||||||
|
environment{routeWatchURL: rw, tagLimitPercent: "prefix=192.0.2.0/33:50"},
|
||||||
|
tagLimitPercent,
|
||||||
|
},
|
||||||
|
{environment{lookupSource: off, routeWatchURL: rw, tagDeny: "country=DE"}, tagDeny},
|
||||||
|
{environment{lookupSource: off, tagDeny: "asn=64496"}, tagDeny},
|
||||||
|
} {
|
||||||
|
_, err := config.FromEnvironment(stopped.env.lookupEnv)
|
||||||
|
if err == nil || !strings.HasPrefix(err.Error(), stopped.name+": ") {
|
||||||
|
t.Errorf("%v gave the error %v, want one naming %s", stopped.env, err, stopped.name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestTagThatNoRequestCanGetIsToldApartFromOneItCan(t *testing.T) {
|
func TestTagThatNoRequestCanGetIsToldApartFromOneItCan(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -2503,6 +2552,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
|||||||
reputationAction: "limit:25",
|
reputationAction: "limit:25",
|
||||||
reputationCacheTTL: defaultReputationCacheTTL,
|
reputationCacheTTL: defaultReputationCacheTTL,
|
||||||
reputationTimeout: "2s",
|
reputationTimeout: "2s",
|
||||||
|
routeWatchURL: "",
|
||||||
crowdSecURL: "",
|
crowdSecURL: "",
|
||||||
crowdSecKey: "",
|
crowdSecKey: "",
|
||||||
banResponse: "403",
|
banResponse: "403",
|
||||||
|
|||||||
@@ -123,6 +123,7 @@ func New(params Params) *Server {
|
|||||||
Lists: h.lists,
|
Lists: h.lists,
|
||||||
DNSBL: h.dnsbl,
|
DNSBL: h.dnsbl,
|
||||||
AbuseIPDB: h.abuseIPDB,
|
AbuseIPDB: h.abuseIPDB,
|
||||||
|
RouteWatch: h.routeWatch,
|
||||||
Metrics: h.metrics,
|
Metrics: h.metrics,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -44,6 +44,7 @@ const (
|
|||||||
func beforeLookup() []heuristic {
|
func beforeLookup() []heuristic {
|
||||||
return []heuristic{
|
return []heuristic{
|
||||||
tagLists{},
|
tagLists{},
|
||||||
|
routeWatchTags{},
|
||||||
denyNets{},
|
denyNets{},
|
||||||
activeBan{},
|
activeBan{},
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/routewatch"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
)
|
)
|
||||||
@@ -85,7 +86,8 @@ type Params struct {
|
|||||||
// Server is the server discriminator runs, with the parts whose state the state
|
// Server is the server discriminator runs, with the parts whose state the state
|
||||||
// files keep, the lookup database, nil unless SWW_LOOKUP_SOURCE is file, the
|
// files keep, the lookup database, nil unless SWW_LOOKUP_SOURCE is file, the
|
||||||
// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts,
|
// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts,
|
||||||
// AbuseIPDB's scores and checks spent, and the metrics.
|
// AbuseIPDB's scores and checks spent, and the metrics. RouteWatch, nil while
|
||||||
|
// SWW_ROUTEWATCH_URL is unset, keeps its answers in memory only.
|
||||||
type Server struct {
|
type Server struct {
|
||||||
*http.Server
|
*http.Server
|
||||||
|
|
||||||
@@ -97,6 +99,7 @@ type Server struct {
|
|||||||
Lists *reputation.Lists
|
Lists *reputation.Lists
|
||||||
DNSBL *reputation.DNSBL
|
DNSBL *reputation.DNSBL
|
||||||
AbuseIPDB *reputation.AbuseIPDB
|
AbuseIPDB *reputation.AbuseIPDB
|
||||||
|
RouteWatch *routewatch.Client
|
||||||
Metrics *metrics.Metrics
|
Metrics *metrics.Metrics
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -152,6 +155,12 @@ func newHandler(params Params) *handler {
|
|||||||
coreRuleSet: params.CoreRuleSet,
|
coreRuleSet: params.CoreRuleSet,
|
||||||
alerts: params.Alerts,
|
alerts: params.Alerts,
|
||||||
}
|
}
|
||||||
|
if params.Config.RouteWatchURL != "" {
|
||||||
|
h.routeWatch = routewatch.New(routewatch.Params{
|
||||||
|
URL: params.Config.RouteWatchURL, Now: params.Now, ProcessLog: params.ProcessLog,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
h.geojs = lookup.New(lookup.Params{
|
h.geojs = lookup.New(lookup.Params{
|
||||||
URL: params.GeoJSURL,
|
URL: params.GeoJSURL,
|
||||||
Timeout: params.Config.LookupTimeout,
|
Timeout: params.Config.LookupTimeout,
|
||||||
@@ -255,6 +264,7 @@ type handler struct {
|
|||||||
lists *reputation.Lists
|
lists *reputation.Lists
|
||||||
dnsbl *reputation.DNSBL
|
dnsbl *reputation.DNSBL
|
||||||
abuseIPDB *reputation.AbuseIPDB
|
abuseIPDB *reputation.AbuseIPDB
|
||||||
|
routeWatch *routewatch.Client
|
||||||
rules *rules.Files
|
rules *rules.Files
|
||||||
coreRuleSet *waf.CoreRuleSet
|
coreRuleSet *waf.CoreRuleSet
|
||||||
alerts *alerts.Queue
|
alerts *alerts.Queue
|
||||||
|
|||||||
@@ -0,0 +1,42 @@
|
|||||||
|
package discriminator
|
||||||
|
|
||||||
|
// The heuristic routewatch_tags. It reads SWW_ROUTEWATCH_URL.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"strconv"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// routeWatchTags adds to the request the tags asn=<number> and prefix=<netblock>
|
||||||
|
// from what routewatch answered about the client. It refuses nothing, and the
|
||||||
|
// request does not wait for routewatch. A request that gets no tags from it has
|
||||||
|
// the reason in the log line's routewatch_tags_missing. While SWW_ROUTEWATCH_URL
|
||||||
|
// is unset it does nothing. Answers are kept by client and asked for by the
|
||||||
|
// request's address, and a route without an origin AS gives only the prefix= tag.
|
||||||
|
type routeWatchTags struct{}
|
||||||
|
|
||||||
|
func (routeWatchTags) name() string { return "routewatch_tags" }
|
||||||
|
|
||||||
|
func (routeWatchTags) check(ctx context.Context, rq *request, _ time.Time) *verdict {
|
||||||
|
if rq.h.routeWatch == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
route, missing := rq.h.routeWatch.Route(ctx, rq.h.clientGroup(rq.client), rq.client)
|
||||||
|
if missing != "" {
|
||||||
|
rq.line.RouteWatchTagsMissing = string(missing)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
rq.addTag(config.PrefixTagPrefix + route.Netblock.String())
|
||||||
|
|
||||||
|
if route.ASN != 0 {
|
||||||
|
rq.addTag(config.ASNTagPrefix + strconv.FormatUint(uint64(route.ASN), 10))
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,231 @@
|
|||||||
|
package discriminator_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/netip"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/discriminator"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
|
)
|
||||||
|
|
||||||
|
// routeWatchURL is SWW_ROUTEWATCH_URL, and tagsMissing the log line's field
|
||||||
|
// that says why a request got no tags from routewatch.
|
||||||
|
const (
|
||||||
|
routeWatchURL = "SWW_ROUTEWATCH_URL"
|
||||||
|
tagsMissing = "routewatch_tags_missing"
|
||||||
|
fromV6 = "2001:db8:1::5"
|
||||||
|
prefixV6 = "2001:db8::/32"
|
||||||
|
prefixDE = "203.0.113.0/24"
|
||||||
|
prefixKP = "198.51.100.0/24"
|
||||||
|
)
|
||||||
|
|
||||||
|
// standInRoute is what the stand-in for routewatch answers about an address.
|
||||||
|
type standInRoute struct {
|
||||||
|
Netblock string `json:"netblock"`
|
||||||
|
ASN int `json:"asn"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// startRouteWatch starts a stand-in for routewatch's IP info API that knows the
|
||||||
|
// routes given by address, and returns its URL.
|
||||||
|
func startRouteWatch(t *testing.T, routes map[string]standInRoute) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
address := strings.TrimPrefix(r.URL.Path, "/api/v1/ip/")
|
||||||
|
|
||||||
|
result := map[string]any{"ip": address}
|
||||||
|
if route, known := routes[address]; known {
|
||||||
|
result["netblock"], result["asn"] = route.Netblock, route.ASN
|
||||||
|
}
|
||||||
|
|
||||||
|
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||||
|
"status": "ok",
|
||||||
|
"data": map[string]any{"results": []any{result}},
|
||||||
|
})
|
||||||
|
}))
|
||||||
|
t.Cleanup(server.Close)
|
||||||
|
|
||||||
|
return server.URL
|
||||||
|
}
|
||||||
|
|
||||||
|
// waitForRouteWatch waits until routewatch's answer about from, in the client
|
||||||
|
// group of the given length, is kept, so that the next request gets its tags.
|
||||||
|
func waitForRouteWatch(
|
||||||
|
t *testing.T, server *discriminator.Server, from string, bits int,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
addr := netip.MustParseAddr(from)
|
||||||
|
client := netip.PrefixFrom(addr, bits).Masked()
|
||||||
|
deadline := time.Now().Add(5 * time.Second)
|
||||||
|
|
||||||
|
for {
|
||||||
|
_, missing := server.RouteWatch.Route(context.Background(), client, addr)
|
||||||
|
if missing == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
t.Fatalf("routewatch's answer about %s not kept: %q", from, missing)
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func wantTagsMissing(t *testing.T, line logLine, want string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
got, _ := line.fields[tagsMissing].(string)
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("log line has %s %q, want %q", tagsMissing, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRouteWatchTagsAreOnTheLogLineWithTheLookupsTags(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
url := startRouteWatch(t, map[string]standInRoute{fromDE: {prefixDE, 64496}})
|
||||||
|
s, server, _ := startWithLookups(t, map[string]string{routeWatchURL: url})
|
||||||
|
|
||||||
|
waitForRouteWatch(t, server, fromDE, 32)
|
||||||
|
|
||||||
|
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||||
|
wantTags(t, line, "asn=64496", "country=DE", "prefix="+prefixDE)
|
||||||
|
wantTagsMissing(t, line, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRouteWatchTagsAreOnALineDenyNetsRefused(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
url := startRouteWatch(t, map[string]standInRoute{fromKP: {prefixKP, 64511}})
|
||||||
|
s, _, server := startWithClock(t, "", map[string]string{
|
||||||
|
lookupSource: off, routeWatchURL: url, denyNets: fromKP,
|
||||||
|
})
|
||||||
|
|
||||||
|
waitForRouteWatch(t, server, fromKP, 32)
|
||||||
|
|
||||||
|
line := s.get(fromKP, http.StatusForbidden, requestlog.ActionDenied)
|
||||||
|
wantHeuristic(t, line, "deny_nets", `{"netblock": "`+fromKP+`/32"}`)
|
||||||
|
wantTags(t, line, "asn=64511", "prefix="+prefixKP)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUnsetRouteWatchChangesNothing(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s, _, server := startWithClock(t, "", map[string]string{lookupSource: off})
|
||||||
|
|
||||||
|
if server.RouteWatch != nil {
|
||||||
|
t.Error("RouteWatch is set while SWW_ROUTEWATCH_URL is unset")
|
||||||
|
}
|
||||||
|
|
||||||
|
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||||
|
if _, present := line.fields[tagsMissing]; present {
|
||||||
|
t.Errorf("log line has %s", tagsMissing)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, present := line.fields["tags"]; present {
|
||||||
|
t.Errorf("log line has tags %v", line.fields["tags"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUnreachableRouteWatchLetsTheRequestThroughAndSaysWhy(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// A stand-in that is closed at once, so that the port refuses connections.
|
||||||
|
dead := httptest.NewServer(http.NotFoundHandler())
|
||||||
|
url := dead.URL
|
||||||
|
|
||||||
|
dead.Close()
|
||||||
|
|
||||||
|
s, _, _ := startWithClock(t, "", map[string]string{
|
||||||
|
lookupSource: off, routeWatchURL: url,
|
||||||
|
})
|
||||||
|
|
||||||
|
wantTagsMissing(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||||
|
"no_answer_yet")
|
||||||
|
|
||||||
|
deadline := time.Now().Add(5 * time.Second)
|
||||||
|
|
||||||
|
for {
|
||||||
|
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||||
|
|
||||||
|
missing, _ := line.fields[tagsMissing].(string)
|
||||||
|
if missing == "request_failed" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
t.Fatalf("still %q, want request_failed", missing)
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAddressWithoutARouteHasNoTagsAndTheLineSaysSo(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s, _, _ := startWithClock(t, "", map[string]string{
|
||||||
|
lookupSource: off, routeWatchURL: startRouteWatch(t, nil),
|
||||||
|
})
|
||||||
|
|
||||||
|
deadline := time.Now().Add(5 * time.Second)
|
||||||
|
|
||||||
|
for {
|
||||||
|
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||||
|
|
||||||
|
missing, _ := line.fields[tagsMissing].(string)
|
||||||
|
if missing == "no_route" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
t.Fatalf("still %q, want no_route", missing)
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTagDenyRefusesOnARouteWatchPrefix(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
url := startRouteWatch(t, map[string]standInRoute{
|
||||||
|
fromDE: {prefixDE, 64496}, fromKP: {prefixKP, 64511},
|
||||||
|
})
|
||||||
|
s, _, server := startWithClock(t, "", map[string]string{
|
||||||
|
lookupSource: off, routeWatchURL: url, tagDeny: "prefix=" + prefixKP,
|
||||||
|
})
|
||||||
|
|
||||||
|
waitForRouteWatch(t, server, fromKP, 32)
|
||||||
|
waitForRouteWatch(t, server, fromDE, 32)
|
||||||
|
|
||||||
|
wantHeuristic(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionDenied),
|
||||||
|
"tag_deny", `{"tag": "prefix=`+prefixKP+`"}`)
|
||||||
|
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTagLimitPercentWorksWithAnIPv6Prefix(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
url := startRouteWatch(t, map[string]standInRoute{fromV6: {prefixV6, 64496}})
|
||||||
|
s, _, server := startWithClock(t, "", map[string]string{
|
||||||
|
lookupSource: off, routeWatchURL: url, rateLimitPerMinute: "2",
|
||||||
|
tagLimitPercent: "prefix=" + prefixV6 + ":50",
|
||||||
|
})
|
||||||
|
|
||||||
|
waitForRouteWatch(t, server, fromV6, 64)
|
||||||
|
|
||||||
|
// Half of two is one: the second request breaks the limit.
|
||||||
|
line := s.get(fromV6, http.StatusOK, requestlog.ActionForward)
|
||||||
|
wantTags(t, line, "asn=64496", "prefix="+prefixV6)
|
||||||
|
s.get(fromV6, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||||
|
}
|
||||||
@@ -182,6 +182,10 @@ type Line struct {
|
|||||||
Reputation []string `json:"reputation,omitempty"`
|
Reputation []string `json:"reputation,omitempty"`
|
||||||
// Tags are the request's tags, sorted.
|
// Tags are the request's tags, sorted.
|
||||||
Tags []string `json:"tags,omitempty"`
|
Tags []string `json:"tags,omitempty"`
|
||||||
|
// RouteWatchTagsMissing, while SWW_ROUTEWATCH_URL is set, says why the
|
||||||
|
// request got no tags from routewatch: no_answer_yet, request_failed or
|
||||||
|
// no_route.
|
||||||
|
RouteWatchTagsMissing string `json:"routewatch_tags_missing,omitempty"`
|
||||||
// Offence is the offence the request was held as, OffenceLimit.
|
// Offence is the offence the request was held as, OffenceLimit.
|
||||||
Offence string `json:"offence,omitempty"`
|
Offence string `json:"offence,omitempty"`
|
||||||
// BanExpires is when the ban the request made, or was refused under,
|
// BanExpires is when the ban the request made, or was refused under,
|
||||||
|
|||||||
@@ -0,0 +1,242 @@
|
|||||||
|
// Package routewatch asks a routewatch instance about client addresses, in the
|
||||||
|
// background, and keeps its answers in memory. A routewatch instance knows, for
|
||||||
|
// an address, the netblock it is announced in and the AS that originates it.
|
||||||
|
package routewatch
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/netip"
|
||||||
|
"net/url"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// answerKept is how long an answer is used after it came.
|
||||||
|
answerKept = time.Hour
|
||||||
|
// failureKept is how long routewatch is not asked again after a query
|
||||||
|
// fails, so that a dead routewatch is not asked on every request.
|
||||||
|
failureKept = time.Minute
|
||||||
|
// queryTimeout is how long one query may take.
|
||||||
|
queryTimeout = 2 * time.Second
|
||||||
|
// maxAnswers is how many answers are kept. Past it, the one that came
|
||||||
|
// longest ago is dropped.
|
||||||
|
maxAnswers = 100000
|
||||||
|
// maxQueries is how many queries may be under way at once. Past it, a
|
||||||
|
// client is not asked about until its next request, so that a swarm of
|
||||||
|
// new addresses cannot fill the memory.
|
||||||
|
maxQueries = 100
|
||||||
|
// maxAnswerBytes is the most of an answer that is read.
|
||||||
|
maxAnswerBytes = 64 << 10
|
||||||
|
)
|
||||||
|
|
||||||
|
// Missing is why Route gives no route. The request log's field
|
||||||
|
// routewatch_tags_missing gives these values.
|
||||||
|
type Missing string
|
||||||
|
|
||||||
|
const (
|
||||||
|
// NoAnswerYet is a client routewatch has not answered about yet.
|
||||||
|
NoAnswerYet Missing = "no_answer_yet"
|
||||||
|
// RequestFailed is no answer because a recent query failed.
|
||||||
|
RequestFailed Missing = "request_failed"
|
||||||
|
// NoRoute is routewatch knowing no live route for the address.
|
||||||
|
NoRoute Missing = "no_route"
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
errStatus = errors.New("routewatch answered")
|
||||||
|
errNoResult = errors.New("the answer holds no result")
|
||||||
|
errNotNetblock = errors.New("the answer's netblock is not a netblock")
|
||||||
|
)
|
||||||
|
|
||||||
|
// Route is what routewatch knows of an address: the netblock it is announced
|
||||||
|
// in, and the AS that originates it.
|
||||||
|
type Route struct {
|
||||||
|
Netblock netip.Prefix
|
||||||
|
ASN uint32
|
||||||
|
}
|
||||||
|
|
||||||
|
// Params are what New needs.
|
||||||
|
type Params struct {
|
||||||
|
// URL is where routewatch is (SWW_ROUTEWATCH_URL).
|
||||||
|
URL string
|
||||||
|
// Now tells the time, normally time.Now in UTC.
|
||||||
|
Now func() time.Time
|
||||||
|
// ProcessLog receives the query that fails, and why.
|
||||||
|
ProcessLog *slog.Logger
|
||||||
|
}
|
||||||
|
|
||||||
|
// Client asks a routewatch instance about clients, in the background, and keeps
|
||||||
|
// its answers. It is safe for concurrent use.
|
||||||
|
type Client struct {
|
||||||
|
params Params
|
||||||
|
httpClient *http.Client
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
// answers are by client. Each is added as it comes and never moved up,
|
||||||
|
// so that the one that came longest ago is the first dropped.
|
||||||
|
answers *simplelru.LRU[netip.Prefix, answer]
|
||||||
|
// asking are the clients whose query is under way.
|
||||||
|
asking map[netip.Prefix]bool
|
||||||
|
// retryAt is when routewatch may be asked again after a query failed.
|
||||||
|
retryAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// answer is one kept answer: the route, or why there is none, and when it came.
|
||||||
|
type answer struct {
|
||||||
|
route Route
|
||||||
|
missing Missing
|
||||||
|
came time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// New returns a Client with no answer yet.
|
||||||
|
func New(params Params) *Client {
|
||||||
|
answers, err := simplelru.NewLRU[netip.Prefix, answer](maxAnswers, nil)
|
||||||
|
if err != nil {
|
||||||
|
panic(err) // NewLRU fails only for a size below one
|
||||||
|
}
|
||||||
|
|
||||||
|
return &Client{
|
||||||
|
params: params,
|
||||||
|
httpClient: &http.Client{},
|
||||||
|
answers: answers,
|
||||||
|
asking: map[netip.Prefix]bool{},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Route returns what routewatch answered about client, an IPv4 address or an
|
||||||
|
// IPv6 group, or why it has no route. It never waits for a query. A client
|
||||||
|
// without an answer is asked about in the background, by addr, the address its
|
||||||
|
// request came from, unless its query is under way, a query failed less than
|
||||||
|
// failureKept ago, or maxQueries are under way. ctx is the context of the
|
||||||
|
// client's request, and a query goes on after the request ends.
|
||||||
|
func (c *Client) Route(
|
||||||
|
ctx context.Context, client netip.Prefix, addr netip.Addr,
|
||||||
|
) (Route, Missing) {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
|
||||||
|
now := c.params.Now()
|
||||||
|
|
||||||
|
kept, found := c.answers.Peek(client)
|
||||||
|
if found && now.Sub(kept.came) < answerKept {
|
||||||
|
return kept.route, kept.missing
|
||||||
|
}
|
||||||
|
|
||||||
|
if now.Before(c.retryAt) {
|
||||||
|
return Route{}, RequestFailed
|
||||||
|
}
|
||||||
|
|
||||||
|
if !c.asking[client] && len(c.asking) < maxQueries {
|
||||||
|
c.asking[client] = true
|
||||||
|
|
||||||
|
go c.ask(context.WithoutCancel(ctx), client, addr)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Route{}, NoAnswerYet
|
||||||
|
}
|
||||||
|
|
||||||
|
// ask asks routewatch about addr, keeps the answer as client's, and notes the
|
||||||
|
// query as no longer under way. A query that fails gives no answer, and
|
||||||
|
// routewatch is not asked again for failureKept. Only the first failure of
|
||||||
|
// those under way is logged.
|
||||||
|
func (c *Client) ask(ctx context.Context, client netip.Prefix, addr netip.Addr) {
|
||||||
|
route, missing, err := c.query(ctx, addr)
|
||||||
|
now := c.params.Now()
|
||||||
|
|
||||||
|
c.mu.Lock()
|
||||||
|
|
||||||
|
delete(c.asking, client)
|
||||||
|
|
||||||
|
firstFailure := err != nil && !now.Before(c.retryAt)
|
||||||
|
|
||||||
|
if err == nil {
|
||||||
|
c.answers.Add(client, answer{route: route, missing: missing, came: now})
|
||||||
|
} else {
|
||||||
|
c.retryAt = now.Add(failureKept)
|
||||||
|
}
|
||||||
|
|
||||||
|
c.mu.Unlock()
|
||||||
|
|
||||||
|
if firstFailure {
|
||||||
|
c.params.ProcessLog.Warn("asking routewatch failed", "error", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// query asks routewatch's IP info API about addr. An answer other than 200, one
|
||||||
|
// that cannot be read, and none within queryTimeout, fail. A result with no
|
||||||
|
// netblock is an address without a live route.
|
||||||
|
func (c *Client) query(ctx context.Context, addr netip.Addr) (Route, Missing, error) {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, queryTimeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
endpoint, err := url.JoinPath(c.params.URL, "api/v1/ip", addr.String())
|
||||||
|
if err != nil {
|
||||||
|
return Route{}, "", fmt.Errorf("make the URL: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, http.NoBody)
|
||||||
|
if err != nil {
|
||||||
|
return Route{}, "", fmt.Errorf("make the request: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
req.Header.Set("Accept", "application/json")
|
||||||
|
|
||||||
|
res, err := c.httpClient.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
// Do's error names the URL, which holds the client's address, which
|
||||||
|
// is not to be logged: only what went wrong is kept.
|
||||||
|
return Route{}, "", fmt.Errorf("ask routewatch: %w", errors.Unwrap(err))
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() {
|
||||||
|
_ = res.Body.Close()
|
||||||
|
}()
|
||||||
|
|
||||||
|
if res.StatusCode != http.StatusOK {
|
||||||
|
return Route{}, "", fmt.Errorf("%w %s", errStatus, res.Status)
|
||||||
|
}
|
||||||
|
|
||||||
|
return readAnswer(io.LimitReader(res.Body, maxAnswerBytes))
|
||||||
|
}
|
||||||
|
|
||||||
|
// readAnswer reads routewatch's answer about one address.
|
||||||
|
func readAnswer(body io.Reader) (Route, Missing, error) {
|
||||||
|
var decoded struct {
|
||||||
|
Data struct {
|
||||||
|
Results []struct {
|
||||||
|
Netblock string `json:"netblock"`
|
||||||
|
ASN uint32 `json:"asn"`
|
||||||
|
} `json:"results"`
|
||||||
|
} `json:"data"`
|
||||||
|
}
|
||||||
|
|
||||||
|
err := json.NewDecoder(body).Decode(&decoded)
|
||||||
|
if err != nil {
|
||||||
|
return Route{}, "", fmt.Errorf("read the answer: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(decoded.Data.Results) == 0 {
|
||||||
|
return Route{}, "", errNoResult
|
||||||
|
}
|
||||||
|
|
||||||
|
result := decoded.Data.Results[0]
|
||||||
|
if result.Netblock == "" {
|
||||||
|
return Route{}, NoRoute, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
netblock, err := netip.ParsePrefix(result.Netblock)
|
||||||
|
if err != nil {
|
||||||
|
return Route{}, "", fmt.Errorf("%w: %q", errNotNetblock, result.Netblock)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Route{Netblock: netblock.Masked(), ASN: result.ASN}, "", nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,290 @@
|
|||||||
|
package routewatch_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/netip"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/routewatch"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
clientAddr = "192.0.2.1"
|
||||||
|
routedAnswer = `{"status":"ok","data":{"results":[` +
|
||||||
|
`{"ip":"192.0.2.1","netblock":"192.0.2.0/24","asn":64496}]}}`
|
||||||
|
noRouteAnswer = `{"status":"ok","data":{"results":[{"ip":"192.0.2.1"}]}}`
|
||||||
|
waitLimit = 5 * time.Second
|
||||||
|
)
|
||||||
|
|
||||||
|
// clock is a time the tests move by hand.
|
||||||
|
type clock struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
now time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *clock) Now() time.Time {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
|
||||||
|
return c.now
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *clock) advance(by time.Duration) {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
|
||||||
|
c.now = c.now.Add(by)
|
||||||
|
}
|
||||||
|
|
||||||
|
// fixture is a Client asking a stand-in for routewatch. queries counts what the
|
||||||
|
// stand-in was asked, paths holds the last path it was asked for, and log holds
|
||||||
|
// the process log.
|
||||||
|
type fixture struct {
|
||||||
|
t *testing.T
|
||||||
|
client *routewatch.Client
|
||||||
|
clk *clock
|
||||||
|
queries *atomic.Int32
|
||||||
|
path *atomic.Value
|
||||||
|
log *syncBuffer
|
||||||
|
}
|
||||||
|
|
||||||
|
// syncBuffer is a buffer the background queries can log to while a test reads it.
|
||||||
|
type syncBuffer struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
buf bytes.Buffer
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *syncBuffer) Write(p []byte) (int, error) {
|
||||||
|
b.mu.Lock()
|
||||||
|
defer b.mu.Unlock()
|
||||||
|
|
||||||
|
return b.buf.Write(p)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *syncBuffer) String() string {
|
||||||
|
b.mu.Lock()
|
||||||
|
defer b.mu.Unlock()
|
||||||
|
|
||||||
|
return b.buf.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
func newFixture(t *testing.T, standIn http.HandlerFunc) *fixture {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
f := &fixture{
|
||||||
|
t: t,
|
||||||
|
clk: &clock{now: time.Date(2026, 11, 9, 0, 0, 0, 0, time.UTC)},
|
||||||
|
queries: &atomic.Int32{},
|
||||||
|
path: &atomic.Value{},
|
||||||
|
log: &syncBuffer{},
|
||||||
|
}
|
||||||
|
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
f.queries.Add(1)
|
||||||
|
f.path.Store(r.URL.Path)
|
||||||
|
standIn(w, r)
|
||||||
|
}))
|
||||||
|
t.Cleanup(server.Close)
|
||||||
|
|
||||||
|
f.client = routewatch.New(routewatch.Params{
|
||||||
|
URL: server.URL + "/", Now: f.clk.Now,
|
||||||
|
ProcessLog: slog.New(slog.NewTextHandler(f.log, nil)),
|
||||||
|
})
|
||||||
|
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
|
||||||
|
func answering(body string) http.HandlerFunc {
|
||||||
|
return func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
_, _ = w.Write([]byte(body))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fixture) route() (routewatch.Route, routewatch.Missing) {
|
||||||
|
addr := netip.MustParseAddr(clientAddr)
|
||||||
|
client := netip.PrefixFrom(addr, addr.BitLen())
|
||||||
|
|
||||||
|
return f.client.Route(context.Background(), client, addr)
|
||||||
|
}
|
||||||
|
|
||||||
|
// waitFor asks until the answer is why, or, while why is "", until there is a
|
||||||
|
// route, and returns the route.
|
||||||
|
func (f *fixture) waitFor(why routewatch.Missing) routewatch.Route {
|
||||||
|
f.t.Helper()
|
||||||
|
|
||||||
|
deadline := time.Now().Add(waitLimit)
|
||||||
|
|
||||||
|
for {
|
||||||
|
route, missing := f.route()
|
||||||
|
if missing == why {
|
||||||
|
return route
|
||||||
|
}
|
||||||
|
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
f.t.Fatalf("still %q after %s, want %q", missing, waitLimit, why)
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRouteGivesRoutewatchsAnswerAndAsksByAddress(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
f := newFixture(t, answering(routedAnswer))
|
||||||
|
|
||||||
|
if _, missing := f.route(); missing != routewatch.NoAnswerYet {
|
||||||
|
t.Errorf("before the answer: %q, want %q", missing, routewatch.NoAnswerYet)
|
||||||
|
}
|
||||||
|
|
||||||
|
route := f.waitFor("")
|
||||||
|
|
||||||
|
want := routewatch.Route{Netblock: netip.MustParsePrefix("192.0.2.0/24"), ASN: 64496}
|
||||||
|
if route != want {
|
||||||
|
t.Errorf("route %+v, want %+v", route, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := f.path.Load(); got != "/api/v1/ip/"+clientAddr {
|
||||||
|
t.Errorf("asked for path %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAddressWithoutARouteIsKeptAsNoRoute(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
f := newFixture(t, answering(noRouteAnswer))
|
||||||
|
|
||||||
|
f.route()
|
||||||
|
f.waitFor(routewatch.NoRoute)
|
||||||
|
|
||||||
|
if got := f.queries.Load(); got != 1 {
|
||||||
|
t.Errorf("%d queries, want 1", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAnswerRoutewatchCannotBeTakenFromFails(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := map[string]http.HandlerFunc{
|
||||||
|
"500": func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusInternalServerError)
|
||||||
|
},
|
||||||
|
"not JSON": answering("no"),
|
||||||
|
"no results": answering(`{"status":"ok","data":{"results":[]}}`),
|
||||||
|
"bad netblock": answering(
|
||||||
|
`{"status":"ok","data":{"results":[{"netblock":"nowhere","asn":1}]}}`),
|
||||||
|
}
|
||||||
|
for name, standIn := range tests {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
f := newFixture(t, standIn)
|
||||||
|
|
||||||
|
f.route()
|
||||||
|
f.waitFor(routewatch.RequestFailed)
|
||||||
|
|
||||||
|
// The failure is noted before it is logged.
|
||||||
|
deadline := time.Now().Add(waitLimit)
|
||||||
|
for !strings.Contains(f.log.String(), "asking routewatch failed") {
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
t.Fatalf("process log %q names no failure", f.log.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.Contains(f.log.String(), clientAddr) {
|
||||||
|
t.Errorf("process log %q names the client", f.log.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFailureIsKeptForAMinuteSoRoutewatchIsNotAskedPerRequest(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
f := newFixture(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusInternalServerError)
|
||||||
|
})
|
||||||
|
|
||||||
|
f.route()
|
||||||
|
f.waitFor(routewatch.RequestFailed)
|
||||||
|
f.route()
|
||||||
|
|
||||||
|
if got := f.queries.Load(); got != 1 {
|
||||||
|
t.Errorf("%d queries while the failure is kept, want 1", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
f.clk.advance(time.Minute)
|
||||||
|
f.route()
|
||||||
|
|
||||||
|
deadline := time.Now().Add(waitLimit)
|
||||||
|
for f.queries.Load() != 2 {
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
t.Fatalf("%d queries after the failure expired, want 2", f.queries.Load())
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHangingRoutewatchNeverDelaysAndFailsTheQuery(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
f := newFixture(t, func(_ http.ResponseWriter, r *http.Request) {
|
||||||
|
<-r.Context().Done()
|
||||||
|
})
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
|
||||||
|
f.route()
|
||||||
|
|
||||||
|
if elapsed := time.Since(start); elapsed > time.Second {
|
||||||
|
t.Errorf("Route took %s while routewatch hangs", elapsed)
|
||||||
|
}
|
||||||
|
|
||||||
|
f.waitFor(routewatch.RequestFailed)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
release := make(chan struct{})
|
||||||
|
f := newFixture(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
<-release
|
||||||
|
|
||||||
|
_, _ = w.Write([]byte(routedAnswer))
|
||||||
|
})
|
||||||
|
|
||||||
|
for range 50 {
|
||||||
|
f.route()
|
||||||
|
}
|
||||||
|
|
||||||
|
close(release)
|
||||||
|
f.waitFor("")
|
||||||
|
|
||||||
|
for range 50 {
|
||||||
|
f.route()
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := f.queries.Load(); got != 1 {
|
||||||
|
t.Errorf("%d queries for repeated requests, want 1", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
f.clk.advance(time.Hour)
|
||||||
|
f.waitFor(routewatch.NoAnswerYet)
|
||||||
|
f.waitFor("")
|
||||||
|
|
||||||
|
if got := f.queries.Load(); got != 2 {
|
||||||
|
t.Errorf("%d queries after the answer expired, want 2", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in new issue
Block a user