Compare commits

...
Author SHA1 Message Date
sneak 9d3f212b0d waf: a heuristic says itself that it refuses whatever the lookup answers (closes #446)
check / check (push) Waiting to run
refusedWhateverTheAnswer asks each heuristic of afterLookup that
implements the new refusesWithoutLookup interface, instead of naming
blocklists and rate_limits. The comment on adding a heuristic names the
places outside its file that may change with it.

Model: opus-5-5
2026-11-19 21:59:05 +00:00
clawbot 51989235d6 rproxy: the body wrappers hold only what they call (closes #431)
check / check (push) Waiting to run
requestBody and responseBody are read on goroutines other than the
handler's. They held the whole request; now they hold only the
functions they call and, for responseBody, the request's context.

Model: opus-5-5
2026-11-19 22:43:57 +01:00
clawbot e15edef583 Move what rproxy and waf send each other into wafprotocol (closes #425)
check / check (push) Waiting to run
internal/wafprotocol writes and reads the question, the answer and the
report, and declares once each header, path and limit both daemons use.
rproxy no longer imports internal/waf. Nothing on the wire changes.

Model: opus-5-5
2026-11-19 22:27:16 +01:00
clawbot 06faf0877c dirwatch: one loop that watches a directory, for rules and lookup (closes #429)
check / check (push) Waiting to run
rules and lookup each had the same loop: watch the directory, start a
timer at once, restart it at each change that counts, read again when it
fires. It now lives once in internal/dirwatch, with its quiet time and
its tests. Each package keeps its own log messages, its readAgain, and
its test for which changes count. internal/state keeps its own copy.

Model: opus-5-5
2026-11-19 22:13:55 +01:00
clawbot b2b1716907 sww: the three commands as cobra commands (closes #440)
check / check (push) Waiting to run
Main builds a cobra root command with rproxy, waf and healthcheck under
it, in place of the switch on os.Args and the hand-kept usage text.
Exit statuses stay: 2 with no command or an unknown one, otherwise what
the command returns. cobra's completion command is off. `sww help` and
`sww --help` now print the commands and exit 0.

Model: opus-5-5
2026-11-19 21:59:25 +01:00
clawbot 56a57125f1 waf: each heuristic holds its own settings and parts (closes #441)
check / check (push) Waiting to run
Every heuristic is a struct built once in newHandler with the settings
and collaborators it reads; no check reads the handler. The six lists
are handler fields built there. Code several heuristics share takes what
it needs as arguments or lives on a struct they hold: banMaker for bans
and their alerts, reputationHitNoter for reputation hits,
biasedThresholds for limit percentages. The handler keeps blocklists,
rateLimits, trapPaths, ruleFiles and coreRuleSet as fields too, since
refusedWhateverTheAnswer and findBesideLookup use them.

Model: opus-5-5
2026-11-19 21:43:55 +01:00
clawbot 9a8ed745bd Constructors take Params: ratelimit.New and processlog.New (closes #450)
check / check (push) Waiting to run
ratelimit.New takes ratelimit.Params (Limits, MaxClients) in place of
two arguments by position. The process logger moves out of requestlog
into internal/processlog, whose New takes Params (Output, InstanceName,
Level); its two tests move with it unchanged. geojs.GeoJS keeps its
params whole, as DNSBL, AbuseIPDB and routewatch.Client do. SPEC and
README list the new package.

Model: opus-5-5
2026-11-19 21:28:01 +01:00
clawbot 0a230c86ce rproxy tests: leave room for a busy host in five timing tests (closes #451)
check / check (push) Waiting to run
Three tests that wait out a 2- or 3-second request time limit now use 5
seconds, so the request reaches the app before it runs out under load.
The WebSocket test also uses 5-second limits, and renews its
connection's deadline after the wait.

TestFiveFailedAsksKeepTheNextRequestFromAskingWAF stops the breaker's
clock like TestAnswersThatCannotBePassedOnOpenTheBreaker.

Model: opus-5-5
2026-11-19 21:14:01 +01:00
clawbot 40386dce6e waf: wait as long as it takes for a test request's answer (closes #454)
check / check (push) Waiting to run
requestWithBody gave the answer 10 seconds, and endedLines gave the log
line 10 seconds. On a busy host the Core Rule Set's inspection of a
request with 1000 query parameters, under the race detector, took
longer. Both now wait until the answer or line comes; go test's
-timeout ends a test where it never does.

Model: opus-5-5
2026-11-19 20:46:01 +01:00
clawbot 433d5cc0a9 bans: one ledger method makes a request's ban, one place decides observe mode (closes #444)
check / check (push) Waiting to run
The ledger's BanForLimit, WouldBanForLimit, BanForAttack and
WouldBanForAttack become Ban and WouldBan, which take the cause and the
notes in one bans.Grounds. The waf's banForLimit and banForAttack fill in
what differs and call banClient, the one function that tests observe
mode on the way to a ban. Behaviour is unchanged; tests change only
where they called the four methods.

Model: opus-5-5
2026-11-19 20:29:53 +01:00
clawbot 419ee1c441 No part imports config (closes #447)
check / check (push) Waiting to run
TagList, MaskZoneKey, ParseNetblock, ParseASN and ParsePercent move
from config to reputation; config imports them from there. rules and
config check header names with httpguts.ValidHeaderFieldName, and
IsHeaderName goes. A new test pins that check to the characters
IsHeaderName took, for the empty name and every single-byte name.

Model: opus-5-5
2026-11-19 20:14:27 +01:00
clawbot aaa28f541b rproxy, sww: keep the breaker and stop tests from failing under load (closes #474)
check / check (push) Waiting to run
The circuit breaker reads the time through a clock the handler holds,
which is time.Now outside tests. The test that answers waf's
questions with answers rproxy cannot pass on stops that clock, so how
long its requests take on a slow host no longer decides whether the
breaker opens or turns the next request away.

The test that stops rproxy mid-request sets its read deadline from the
stop, not from the send, so a slow wait for the app no longer shortens
it.

Model: opus-5-5
2026-11-19 19:59:34 +01:00
clawbot 98d7432077 waf: the steps after the answer as plain statements (closes #445)
check / check (push) Waiting to run
banAndCount runs the heuristics that only ban, then countAnomalies,
then addToHistory. A refused request's decision and an allowed
request's report both call it, each with its own list. answerQuestion
keeps one defer, which writes the log line and holds the decision for
the report.

A check that panics now leaves the request out of its client's history
and the anomaly counts, as the issue states.

Model: opus-5-5
2026-11-19 19:45:13 +01:00
clawbot 11ac8aee50 config: export the setting names waf logs (closes #448)
check / check (push) Waiting to run
waf wrote eight setting names as string literals into log lines and
ban notes, and config wrote each again where it reads the setting. A
rename that missed one would log a setting that does not exist. config
now exports a constant for each name, used by its reader and by waf.

Model: opus-5-5
2026-11-19 19:29:49 +01:00
clawbot e7cc0adbdb waf: end the asks to GeoJS, the DNSBL zones, AbuseIPDB and routewatch at the stop (closes #439)
check / check (push) Waiting to run
Each of the four keeps the cancel function of every ask it started, and a
wait group of their goroutines. Its new method (GeoJS.StopAsking,
DNSBL.StopAsking, AbuseIPDB.StopChecking, routewatch Client.StopAsking)
refuses further asks, ends those under way, and returns once their
goroutines have. An ask ended that way is no failure: no alert, no failure
count, no wait before the next. Each provider in internal/sww adds the
method as a stop hook, so the last write of the state files follows it.

Model: opus-5-5
2026-11-19 19:02:12 +01:00
clawbot 805c64a256 config: give the fixed-value settings named types (closes #427)
check / check (push) Waiting to run
BytesCounted, LookupSource, BlocklistAction, DNSBLAbuseIPDBAction and
CoreRuleSetMode become named string types declared in config, with
constants for their values. Every comparison in waf and sww uses the
constants, and the private duplicates fileSource and deny go. The two
action settings share ReputationAction. The values and the strings
written to logs, metrics and alerts are unchanged.

Model: opus-5-5
2026-11-19 18:32:29 +01:00
clawbot 967ccdf75e sww: build the Core Rule Set from the settings (closes #434)
check / check (push) Failing after 4m0s
The body of waf.NewCoreRuleSet moves into newCoreRuleSet in
internal/sww, beside the providers of the other parts; waf no longer
exports it. The waf test helper builds the Core Rule Set with
coreruleset.New while the mode is not off.

Model: opus-5-5
2026-11-19 18:15:33 +01:00
clawbot 809d2664d7 waf: ban the clients on AbuseIPDB's blacklist (closes #411)
check / check (push) Successful in 8m16s
While SWW_WAF_ABUSEIPDB_KEY is set, waf fetches AbuseIPDB's blacklist at
the start and every SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS
(60), cut at SWW_WAF_ABUSEIPDB_THRESHOLD (90, from 25 to 100). Each
listed client's netblock gets a ban of SWW_WAF_ABUSEIPDB_BAN_DAYS (30)
with the new cause abuseipdb. The blacklist itself is not kept.

These bans do not count toward SWW_WAF_MAX_BANS, so a 10,000-address
blacklist cannot push out the bans for limits and attacks. An ended
abuseipdb ban is forgotten at each fetch and each read of bans.json. A
netblock with an active ban is left alone, and so is one whose abuseipdb
ban an admin lifted, until that ban would have ended. Observe mode makes
no ban.

Model: opus-5-5
2026-11-19 12:42:41 +01:00
clawbot 01e6b5d47e sww: wire the daemons with fx (closes #414)
check / check (push) Successful in 3m54s
go.uber.org/fx builds the parts of waf and rproxy in internal/sww, one
provider per part. Each background loop is a lifecycle hook added where
its part is built. waf.Params takes every part built, and waf.New builds
none. GeoJS calls back into waf, so waf exports AddLookup and
GeoJSFailed, and clientGroup and banNetblock take the settings.

newServer, loadStateFiles, the inBackground list and serveRProxy are
gone. serve is the one hook that serves both daemons, and reports a
failure to run on a channel of its own; fx's signal handling is not
used. Once fx has stopped, run takes each daemon's last step: waf writes
the state files, and rproxy waits for its reports.

Model: opus-5-5
2026-11-19 11:58:20 +01:00
clawbot 5099448f79 Keep the sww and rproxy tests from failing on a busy host (closes #420)
check / check (push) Failing after 2m12s
The rproxy package's tests and sww's startRProxyProcess give rproxy a
minute for waf's answers unless a test sets
SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. At the 2-second default, a slow host
made rproxy drop questions and reports.

TestMainRunsWAF runs waf with the Core Rule Set off. Main keeps waf's
500 ms limit for its checks, and under the race detector the Core Rule
Set's inspection could pass it and get 503.

Model: opus-5-5
2026-11-19 11:43:01 +01:00
clawbot eb48796f27 README: compose example for the open internet, with common waf settings (closes #381)
check / check (push) Failing after 4m5s
A new example puts one gitea behind traefik, rproxy and waf. waf sets
nine settings most sites use, then eight more commented out, each with
what it does and when to set it.

Model: opus-5-5
2026-11-19 11:16:15 +01:00
clawbot 9f42460f70 Say the anomaly thresholds only send an alert (closes #412)
check / check (push) Failing after 16m9s
Each SWW_WAF_ANOMALY_ threshold's description now says crossing it sends
one alert to the alert destinations and bans, rate limits and blocks no
client. The prefix and watched-netblock settings say their thresholds
only alert. README's configuration notes and docs/settings.md explain
"anomaly" once.

Model: opus-5-5
2026-11-19 10:58:27 +01:00
clawbot 9cb8f362bc sww: lift waf's check time limit and rproxy's wait in tests (closes #418)
check / check (push) Failing after 1m52s
Under heavy host load the sww tests got 503 where they expected 200 or
403: waf refused requests whose trap paths, rule files and Core Rule Set
took longer than 500 ms, and rproxy gave up on waf after 2 s.

waf.Params and sww.Params gain BesideLookupTimeLimit; zero keeps the
500 ms limit, so the binary is unchanged. The sww tests' run gives waf
an hour, and startRProxy sets SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS to
60000, as the rproxy package's tests do. waf's own tests set the limit
through the new field instead of SetBesideLookupTimeLimit, which is gone.

Model: opus-5-5
2026-11-19 10:44:53 +01:00
clawbot 48edab5dad Test rproxy's 32 KiB header limit at its edge (closes #415)
check / check (push) Successful in 5m6s
The limit was already a constant with no setting. These tests pin it:
a request line and headers of exactly 32 KiB reach the app, and one
byte more is answered 431 with the connection closed and waf not asked.

Model: opus-5-5
2026-11-19 10:20:07 +01:00
sneak dadd5768d5 Merge pull request 'next into main' (#223) from next into main
check / check (push) Failing after 11m39s
Reviewed-on: #223
2026-11-19 03:43:19 +01:00
sneak e6a081134a Merge branch 'main' into next
check / check (push) Failing after 1m52s
2026-11-19 03:43:10 +01:00
sneak 4cb91a4870 Merge pull request 'Set the images' own OCI labels instead of Ubuntu's' (#224) from issue-216-oci-labels into next
check / check (push) Failing after 1m58s
Reviewed-on: #224
2026-11-19 03:41:57 +01:00
sneak b9972af6a0 Merge branch 'next' into issue-216-oci-labels
check / check (push) Successful in 7m10s
2026-11-19 03:41:52 +01:00
clawbot c0d9d5787e reputation: run the DNSBL resolver test in parallel (closes #406)
check / check (push) Failing after 2m25s
TestQueriesGoToTheResolverSWWDNSBLResolverNames queries a real UDP
stand-in outside any synctest bubble. A parallel test starts only once
every test that is not parallel has ended, so it can no longer overlap
the bubbled DNSBL tests, and no other parallel test of the package
queries a resolver.

The other tests that run one at a time stay so, each for the reason its
comment gives: environment variables, TMPDIR, timing, Go's resolver in
synctest bubbles, and garbage collection hiding a leaked connection.
Every subtest inside a parallel test already calls t.Parallel().

Model: opus-5-5
2026-11-17 06:57:44 +01:00
sneak a93243a294 Set the images' own OCI labels instead of Ubuntu's (closes #216)
check / check (push) Failing after 1m53s
The image sets title, description, version, created, revision and
source; the last three known only at build time come from build
arguments that default to unknown, and script/docker and
script/cibuild pass them. The example app sets its own title and
description.

Model: opus-5-5
2026-11-14 17:58:09 +00:00
sneak f52b1f9ab5 Merge pull request 'next into main' (#205) from next into main
check / check (push) Successful in 3m37s
Reviewed-on: #205
2026-11-07 02:52:39 +01:00
111 changed files with 6389 additions and 3062 deletions

No files matched your search

+14
View File
@@ -161,6 +161,20 @@ RUN git checkout --quiet --detach b4b2c785308b1ce785b6155c7fe5f16879080193 \
# ubuntu 26.04, 2026-09-27 # ubuntu 26.04, 2026-09-27
FROM ubuntu@sha256:f144425ff09be612d6d9ad965196e9cdc23dae1f42110a8a11a3e9a8198759f7 FROM ubuntu@sha256:f144425ff09be612d6d9ad965196e9cdc23dae1f42110a8a11a3e9a8198759f7
# The image's own labels, in place of the ones it would inherit from
# Ubuntu's. script/docker and script/cibuild pass the build arguments;
# a build that passes none labels the image unknown rather than leave
# Ubuntu's values.
ARG VERSION=unknown
ARG CREATED=unknown
ARG REVISION=unknown
LABEL org.opencontainers.image.title="smallwebwaf" \
org.opencontainers.image.description="A web application firewall for self-hosted web apps: sww rproxy in front of an app, and sww waf, which decides." \
org.opencontainers.image.version="${VERSION}" \
org.opencontainers.image.created="${CREATED}" \
org.opencontainers.image.revision="${REVISION}" \
org.opencontainers.image.source="https://git.eeqj.de/sneak/smallwebwaf"
# runit's install creates its _runit-log user with minsysusers, which # runit's install creates its _runit-log user with minsysusers, which
# reads this file in place of runit's /usr/lib/sysusers.d/runit.conf. # reads this file in place of runit's /usr/lib/sysusers.d/runit.conf.
# runit's line leaves out the shell, and minsysusers prints a Perl # runit's line leaves out the shell, and minsysusers prints a Perl
+279 -125
View File
@@ -75,7 +75,8 @@ Together, `rproxy` and `waf` protect against:
- give the clients of chosen networks and countries a fraction of every - give the clients of chosen networks and countries a fraction of every
limit, so that they are banned sooner; limit, so that they are banned sooner;
- refuse or limit clients on a blocklist, a DNS blocklist, or with a high - refuse or limit clients on a blocklist, a DNS blocklist, or with a high
AbuseIPDB score. AbuseIPDB score;
- ban the clients on AbuseIPDB's blacklist before they send a request.
For the operator, it: For the operator, it:
@@ -284,129 +285,136 @@ account, a key, a token or an alert destination stays off until you give one.
`SWW_WAF_BYTES_COUNTED` says. Headers are never counted. What the visitor sent `SWW_WAF_BYTES_COUNTED` says. Headers are never counted. What the visitor sent
over a WebSocket counts as request bytes, and what the app sent as response over a WebSocket counts as request bytes, and what the app sent as response
bytes. bytes.
- An anomaly is more requests or bytes than an `SWW_WAF_ANOMALY_` threshold
allows. Crossing a threshold sends one `anomaly` alert to the alert
destinations, and does nothing else: no client is banned, rate limited or
blocked.
- An invalid setting stops the process at start with one log line naming the - An invalid setting stops the process at start with one log line naming the
variable and what is wrong with the value. The settings in effect are logged variable and what is wrong with the value. The settings in effect are logged
on one line at start, with tokens, keys and webhook paths shown as `********`. on one line at start, with tokens, keys and webhook paths shown as `********`.
Every setting the code reads, with its default: Every setting the code reads, with its default:
| Variable | Default | What it does | | Variable | Default | What it does |
| ------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | --------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `waf`. | | `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `waf`. |
| `SWW_RPROXY_WAF_BACKEND` | required | `waf`'s URL, such as `http://waf:8080`. | | `SWW_RPROXY_WAF_BACKEND` | required | `waf`'s URL, such as `http://waf:8080`. |
| `SWW_RPROXY_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. | | `SWW_RPROXY_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. |
| `SWW_RPROXY_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. | | `SWW_RPROXY_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. |
| `SWW_RPROXY_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. | | `SWW_RPROXY_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. |
| `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `waf`'s answer to a question and to a report, in milliseconds. | | `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `waf`'s answer to a question and to a report, in milliseconds. |
| `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` | `allow` | What `rproxy` does with a request when `waf` gives no answer in time, and while the circuit breaker for asks to `waf` is open: `allow` passes it to the app, `deny` answers `503`. | | `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` | `allow` | What `rproxy` does with a request when `waf` gives no answer in time, and while the circuit breaker for asks to `waf` is open: `allow` passes it to the app, `deny` answers `503`. |
| `SWW_RPROXY_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. | | `SWW_RPROXY_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. |
| `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. | | `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
| `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. | | `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
| `SWW_RPROXY_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. | | `SWW_RPROXY_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
| `SWW_WAF_LISTEN_ADDR` | `:8080` | Where `waf` listens; leave the host part empty. | | `SWW_WAF_LISTEN_ADDR` | `:8080` | Where `waf` listens; leave the host part empty. |
| `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `host1/waf`. | | `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `host1/waf`. |
| `SWW_WAF_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. | | `SWW_WAF_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
| `SWW_WAF_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. | | `SWW_WAF_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. |
| `SWW_WAF_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. | | `SWW_WAF_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. |
| `SWW_WAF_MAX_BANS` | `5000` | The most bans `waf` keeps of its own; bans an admin made do not count. | | `SWW_WAF_MAX_BANS` | `5000` | The most bans `waf` keeps for broken limits and attacks; bans an admin made and bans for AbuseIPDB's blacklist do not count. |
| `SWW_WAF_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. | | `SWW_WAF_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. |
| `SWW_WAF_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients are exempt from rate limiting and the byte limits. | | `SWW_WAF_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients are exempt from rate limiting and the byte limits. |
| `SWW_WAF_DENY_NETS` | empty | Netblocks whose clients are refused. | | `SWW_WAF_DENY_NETS` | empty | Netblocks whose clients are refused. |
| `SWW_WAF_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; the next request is refused and bans the client. | | `SWW_WAF_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; the next request is refused and bans the client. |
| `SWW_WAF_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; the next request is refused and bans the client. | | `SWW_WAF_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; the next request is refused and bans the client. |
| `SWW_WAF_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; the next request is refused and bans the client. | | `SWW_WAF_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; the next request is refused and bans the client. |
| `SWW_WAF_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. | | `SWW_WAF_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. |
| `SWW_WAF_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes counted for a client in a minute, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. | | `SWW_WAF_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes counted for a client in a minute, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
| `SWW_WAF_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes counted for a client in an hour, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. | | `SWW_WAF_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes counted for a client in an hour, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
| `SWW_WAF_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes counted for a client in a day, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. | | `SWW_WAF_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes counted for a client in a day, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
| `SWW_WAF_BYTES_COUNTED` | `both` | What the byte limits and the byte anomaly thresholds count: `response`, the app's response bodies; `request`, the visitors' request bodies; or `both`. What the visitor sent over a WebSocket counts as request bytes, and what the app sent as response bytes. | | `SWW_WAF_BYTES_COUNTED` | `both` | What the byte limits and the byte anomaly thresholds count: `response`, the app's response bodies; `request`, the visitors' request bodies; or `both`. What the visitor sent over a WebSocket counts as request bytes, and what the app sent as response bytes. |
| `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE` | `30` | The most refusals a client may get in a minute from a rule file, a trap path or the Core Rule Set, a count or `off`. The next one bans the client. | | `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE` | `30` | The most refusals a client may get in a minute from a rule file, a trap path or the Core Rule Set, a count or `off`. The next one bans the client. |
| `SWW_WAF_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. | | `SWW_WAF_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. |
| `SWW_WAF_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. | | `SWW_WAF_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. |
| `SWW_WAF_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A client that breaks a limit again within this time after its last such ban ended gets a ban three times as long. | | `SWW_WAF_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A client that breaks a limit again within this time after its last such ban ended gets a ban three times as long. |
| `SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. | | `SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. |
| `SWW_WAF_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. | | `SWW_WAF_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. |
| `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. | | `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
| `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. | | `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
| `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. | | `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, in milliseconds. The request to GeoJS goes on, and a later answer is kept. | | `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, in milliseconds. The request to GeoJS goes on, and a later answer is kept. |
| `SWW_WAF_ADD_LOOKUP_HEADERS` | `true`, or `false` while the lookup source is `off` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. | | `SWW_WAF_ADD_LOOKUP_HEADERS` | `true`, or `false` while the lookup source is `off` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
| `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. | | `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
| `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. | | `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
| `SWW_WAF_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. | | `SWW_WAF_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. |
| `SWW_WAF_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. | | `SWW_WAF_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. |
| `SWW_WAF_ROUTEWATCH_URL` | unset | The URL of a routewatch instance, which gives each client's `asn=` and `prefix=` tags. Unset, nothing asks routewatch. | | `SWW_WAF_ROUTEWATCH_URL` | unset | The URL of a routewatch instance, which gives each client's `asn=` and `prefix=` tags. Unset, nothing asks routewatch. |
| `SWW_WAF_ASN_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. | | `SWW_WAF_ASN_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. |
| `SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. | | `SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. |
| `SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. | | `SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. |
| `SWW_WAF_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_WAF_TAG_LIMIT_PERCENT` (the lowest wins). | | `SWW_WAF_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_WAF_TAG_LIMIT_PERCENT` (the lowest wins). |
| `SWW_WAF_RULE_FILES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. | | `SWW_WAF_RULE_FILES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. |
| `SWW_WAF_RULE_FILES_ENABLED` | `true` | `false` reads no rule file. | | `SWW_WAF_RULE_FILES_ENABLED` | `true` | `false` reads no rule file. |
| `SWW_WAF_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. | | `SWW_WAF_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. |
| `SWW_WAF_CORE_RULE_SET_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. | | `SWW_WAF_CORE_RULE_SET_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. |
| `SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. | | `SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. |
| `SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. | | `SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. |
| `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. | | `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. |
| `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. | | `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
| `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` | `off` | How much of a request body the Core Rule Set reads, a size up to `8K`; `off` reads none. | | `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` | `off` | How much of a request body the Core Rule Set reads, a size up to `8K`; `off` reads none. |
| `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. | | `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
| `SWW_WAF_APPS` | unset | The apps each hostname serves, such as `git.example.com:gitea;blog.example.com:wordpress`; only those apps' fixes apply there. "Attack detection" lists the apps. Listing `forgejo` brings gitea's fixes too. Unset, every app's fixes apply to every hostname. | | `SWW_WAF_APPS` | unset | The apps each hostname serves, such as `git.example.com:gitea;blog.example.com:wordpress`; only those apps' fixes apply there. "Attack detection" lists the apps. Listing `forgejo` brings gitea's fixes too. Unset, every app's fixes apply to every hostname. |
| `SWW_WAF_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. | | `SWW_WAF_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
| `SWW_WAF_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. | | `SWW_WAF_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. |
| `SWW_WAF_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. | | `SWW_WAF_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. |
| `SWW_WAF_DNSBL_ZONES` | empty | DNS blocklists asked about each new client in the background, such as `dnsbl.dronebl.org`. | | `SWW_WAF_DNSBL_ZONES` | empty | DNS blocklists asked about each new client in the background, such as `dnsbl.dronebl.org`. |
| `SWW_WAF_DNSBL_RESOLVER` | unset | The resolver the zones are asked through, an IP address with an optional port; unset, the host's resolver is used. | | `SWW_WAF_DNSBL_RESOLVER` | unset | The resolver the zones are asked through, an IP address with an optional port; unset, the host's resolver is used. |
| `SWW_WAF_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account; set, a client that has committed an offence is checked in the background. | | `SWW_WAF_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account. Set, a client that has committed an offence is checked in the background, and the clients on AbuseIPDB's blacklist are banned. |
| `SWW_WAF_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. | | `SWW_WAF_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. |
| `SWW_WAF_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. | | `SWW_WAF_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. |
| `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:<percent>` or `log`. | | `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:<percent>` or `log`. |
| `SWW_WAF_IP_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. | | `SWW_WAF_IP_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. |
| `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS` | `2000` | How long a DNS blocklist query or an AbuseIPDB check may take, in milliseconds. | | `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS` | `2000` | How long a DNS blocklist query or an AbuseIPDB check may take, in milliseconds. |
| `SWW_WAF_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. | | `SWW_WAF_ABUSEIPDB_THRESHOLD` | `90` | The least AbuseIPDB score, from 25 to 100, at which a client on its blacklist is banned. Do not set it below 50 to 75: lower scores list ordinary visitors. |
| `SWW_WAF_BANS_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. | | `SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS` | `60` | How often AbuseIPDB's blacklist is fetched again, in minutes. |
| `SWW_WAF_STATE_WRITE_INTERVAL` | `15m` | How often every state and cache file is written. | | `SWW_WAF_ABUSEIPDB_BAN_DAYS` | `30` | How long a client on AbuseIPDB's blacklist is banned, in days. |
| `SWW_WAF_CACHE_DIR` | `/var/cache/sww` | The directory of the cache files, an absolute path. | | `SWW_WAF_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. |
| `SWW_WAF_LOG_LEVEL` | `info` | The least severe of `waf`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. | | `SWW_WAF_BANS_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. |
| `SWW_WAF_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. | | `SWW_WAF_STATE_WRITE_INTERVAL` | `15m` | How often every state and cache file is written. |
| `SWW_WAF_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `waf` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. | | `SWW_WAF_CACHE_DIR` | `/var/cache/sww` | The directory of the cache files, an absolute path. |
| `SWW_WAF_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. | | `SWW_WAF_LOG_LEVEL` | `info` | The least severe of `waf`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. |
| `SWW_WAF_LOG_REMOTE_BUFFER_LINES` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. | | `SWW_WAF_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. |
| `SWW_WAF_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. | | `SWW_WAF_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `waf` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. |
| `SWW_WAF_LOG_REMOTE_APP_NAME` | `SWW_WAF_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. | | `SWW_WAF_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. |
| `SWW_WAF_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. | | `SWW_WAF_LOG_REMOTE_BUFFER_LINES` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. |
| `SWW_WAF_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. | | `SWW_WAF_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. |
| `SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. | | `SWW_WAF_LOG_REMOTE_APP_NAME` | `SWW_WAF_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. |
| `SWW_WAF_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. | | `SWW_WAF_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. |
| `SWW_WAF_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer <token>`. | | `SWW_WAF_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. |
| `SWW_WAF_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. | | `SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. |
| `SWW_WAF_ALERT_NTFY_URL` | unset | An ntfy topic each alert is published to, as its full URL, such as `https://ntfy.sh/my-alerts`. | | `SWW_WAF_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. |
| `SWW_WAF_ALERT_NTFY_TOKEN` | unset | An ntfy access token sent with each alert. | | `SWW_WAF_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer <token>`. |
| `SWW_WAF_ALERT_EVENTS` | `ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. | | `SWW_WAF_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. |
| `SWW_WAF_ALERT_COOLDOWN` | `15m` | How long after an alert its repeats are held back and counted, a duration or `off`. | | `SWW_WAF_ALERT_NTFY_URL` | unset | An ntfy topic each alert is published to, as its full URL, such as `https://ntfy.sh/my-alerts`. |
| `SWW_WAF_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. | | `SWW_WAF_ALERT_NTFY_TOKEN` | unset | An ntfy access token sent with each alert. |
| `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Alert when one client makes more requests than this in a minute, a count. | | `SWW_WAF_ALERT_EVENTS` | `ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. |
| `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Alert when one client makes more requests than this in an hour, a count. | | `SWW_WAF_ALERT_COOLDOWN` | `15m` | How long after an alert its repeats are held back and counted, a duration or `off`. |
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one client in a minute, a size. | | `SWW_WAF_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. |
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one client in an hour, a size. | | `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when one client makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Alert when the netblock around a client makes more requests than this in a minute, a count. | | `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when one client makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Alert when the netblock around a client makes more requests than this in an hour, a count. | | `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one client in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the netblock around a client in a minute, a size. | | `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one client in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the netblock around a client in an hour, a size. | | `SWW_WAF_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when the netblock around a client makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 32. | | `SWW_WAF_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when the netblock around a client makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 128. | | `SWW_WAF_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the netblock around a client in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Alert when one AS number makes more requests than this in a minute, a count. | | `SWW_WAF_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the netblock around a client in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Alert when one AS number makes more requests than this in an hour, a count. | | `SWW_WAF_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 32. Those thresholds only send alerts. |
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one AS number in a minute, a size. | | `SWW_WAF_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 128. Those thresholds only send alerts. |
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one AS number in an hour, a size. | | `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when one AS number makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Alert when the whole service gets more requests than this in a minute, a count. | | `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when one AS number makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Alert when the whole service gets more requests than this in an hour, a count. | | `SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one AS number in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the whole service in a minute, a size. | | `SWW_WAF_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one AS number in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the whole service in an hour, a size. | | `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when the whole service gets more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_WATCHED_NETS` | empty | Named netblocks the `SWW_WAF_ANOMALY_WATCHED_NET_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. | | `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when the whole service gets more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_MINUTE` | `off` | Alert when a watched netblock makes more requests than this in a minute, a count. | | `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the whole service in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_HOUR` | `off` | Alert when a watched netblock makes more requests than this in an hour, a count. | | `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the whole service in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for a watched netblock in a minute, a size. | | `SWW_WAF_ANOMALY_WATCHED_NETS` | empty | Named netblocks the `SWW_WAF_ANOMALY_WATCHED_NET_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. Those thresholds only send alerts. |
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for a watched netblock in an hour, a size. | | `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when a watched netblock makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when a watched netblock makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for a watched netblock in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for a watched netblock in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
[`docs/settings.md`](docs/settings.md) gives the values each setting takes in [`docs/settings.md`](docs/settings.md) gives the values each setting takes in
full, and the limits that are fixed rather than settings. full, and the limits that are fixed rather than settings.
@@ -416,8 +424,8 @@ full, and the limits that are fixed rather than settings.
Each example is a complete `compose.yaml`. `smallwebwaf` is the tag Each example is a complete `compose.yaml`. `smallwebwaf` is the tag
`make docker` gives the image; on another host, use the name you pushed it `make docker` gives the image; on another host, use the name you pushed it
under, by digest. Docker Compose puts the services on a private network of its under, by digest. Docker Compose puts the services on a private network of its
own, where each reaches the others by service name. Each example publishes own, where each reaches the others by service name. Each example but "On the
`rproxy`'s port on the host, for trying it out. open internet" publishes `rproxy`'s port on the host, for trying it out.
### Smallest setup ### Smallest setup
@@ -449,6 +457,142 @@ volumes:
waf-state: waf-state:
``` ```
### On the open internet
One gitea at `git.example.com`, with traefik in front of `rproxy`. traefik gets
the certificate from Let's Encrypt and is the only service with published ports.
`waf` sets what most sites set. More `waf` settings follow, commented out, each
with what it does and when to set it.
```yaml
services:
traefik:
# traefik v3.7.19, 2026-11-13
image: traefik@sha256:a6c0194dda77bcb804824580f36fa7edc56e315b1acfe933366c8b52844a78a4
restart: unless-stopped
command:
- --providers.docker=true
- --providers.docker.exposedByDefault=false
- --entryPoints.web.address=:80
- --entryPoints.websecure.address=:443
# Redirect plain HTTP to HTTPS.
- --entryPoints.web.http.redirections.entryPoint.to=websecure
- --entryPoints.web.http.redirections.entryPoint.scheme=https
# A Let's Encrypt certificate for each site, by the TLS-ALPN
# challenge on port 443.
- --entryPoints.websecure.http.tls.certResolver=le
- --certificatesResolvers.le.acme.email=hostmaster@example.com
- --certificatesResolvers.le.acme.storage=/letsencrypt/acme.json
- --certificatesResolvers.le.acme.tlsChallenge=true
# Give a request 10 minutes to arrive, rather than 1, for the
# large git pushes the rproxy lines below accept.
# - --entryPoints.websecure.transport.respondingTimeouts.readTimeout=600s
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- acme:/letsencrypt
app:
# Your app: here a gitea, which listens on port 3000.
image: registry.example/app@sha256:0000000000000000000000000000000000000000000000000000000000000000
restart: unless-stopped
rproxy:
image: smallwebwaf
restart: unless-stopped
environment:
SWW_RPROXY_WAF_BACKEND: http://waf:8080
SWW_RPROXY_UPSTREAM_URL: http://app:3000
# Accept git pushes up to 1 GiB, rather than 100 MiB, sent in
# up to 10 minutes, rather than 1, with traefik's readTimeout
# line above.
# SWW_RPROXY_MAX_REQUEST_SIZE: 1G
# SWW_RPROXY_MAX_REQUEST_TIME_MS: "600000"
labels:
- traefik.enable=true
- traefik.http.routers.git.rule=Host(`git.example.com`)
- traefik.http.routers.git.entryPoints=websecure
- traefik.http.services.git.loadBalancer.server.port=8080
waf:
image: smallwebwaf
restart: unless-stopped
environment:
SWW_COMMAND: waf
# The name in every log line, alert and metric. The default,
# the container's host name, is a random id under Compose.
SWW_WAF_INSTANCE_NAME: host1/waf
# The app each hostname serves. Only that app's fixes to the
# Core Rule Set apply there.
SWW_WAF_APPS: git.example.com:gitea
# Monitoring skips every check.
SWW_WAF_ALLOW_NETS: 192.0.2.10
# The office and the CI runner, which would exceed the rate
# limits, are never rate limited.
SWW_WAF_RATE_LIMIT_EXEMPT_NETS: 203.0.113.0/24,198.51.100.7
# WordPress paths, which only scanners ask a gitea for. One
# request for one bans the client.
SWW_WAF_TRAP_PATHS: /wp-login.php,/xmlrpc.php
# Spamhaus's list of netblocks hijacked or leased by
# criminals. Their clients are refused.
SWW_WAF_BLOCKLIST_URLS: https://www.spamhaus.org/drop/drop.txt
# The token for the ban endpoints, to lift a ban without
# editing bans.json.
SWW_WAF_ADMIN_TOKEN_FILE: /etc/sww/tokens/admin
# Alerts, such as one for each ban, to an ntfy topic.
SWW_WAF_ALERT_NTFY_URL: https://ntfy.example.com/smallwebwaf
# Refuse nothing and ban no one, for the first days. Each log
# line says what enforce mode would have done.
# SWW_WAF_MODE: observe
# The access token for an ntfy topic that is not public.
# SWW_WAF_ALERT_NTFY_TOKEN_FILE: /etc/sww/tokens/ntfy
# The token for /_sww/metrics, for Prometheus. Unset, the
# metrics are off.
# SWW_WAF_METRICS_TOKEN_FILE: /etc/sww/tokens/metrics
# 600 requests an hour rather than 1500, to ban slower
# scrapers. A fast human reader can reach it too.
# SWW_WAF_RATE_LIMIT_PER_HOUR: "600"
# A quarter of every limit for clients in two hosting
# networks, where scrapers run rather than people.
# SWW_WAF_TAG_LIMIT_PERCENT: asn=14061:25,asn=16276:25
# For a site only for people in a few countries. Clients
# elsewhere are refused, and so are any the lookup cannot place.
# SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES: de,at,ch
# Close a refused request's connection rather than answer
# 403. traefik then answers 502, so a scraper cannot tell a
# ban from a broken site.
# SWW_WAF_BAN_RESPONSE: close
# A copy of every log line to a syslog server.
# SWW_WAF_LOG_REMOTE_URL: syslog+tls://logs.example.com:6514
volumes:
- waf-state:/var/lib/sww
- /srv/sww/tokens:/etc/sww/tokens:ro
volumes:
acme:
waf-state:
```
Change these before the first start:
- `git.example.com`, in the router rule and in `SWW_WAF_APPS`;
- the email address for Let's Encrypt;
- the ntfy topic's URL;
- the app's image and port;
- the netblocks, which are documentation addresses.
Make the admin token file first, as for any token file:
```sh
mkdir -p /srv/sww/tokens
openssl rand -hex 32 > /srv/sww/tokens/admin
chown 65532:65532 /srv/sww/tokens/admin
chmod 0400 /srv/sww/tokens/admin
```
### Custom per-address limits ### Custom per-address limits
For a site with no API: For a site with no API:
@@ -626,8 +770,10 @@ status for a size or time limit, such as `413` for a body too large.
- A client is one IPv4 address or one IPv6 netblock of - A client is one IPv4 address or one IPv6 netblock of
`SWW_WAF_IPV6_GROUP_PREFIX`, since one abuser usually controls a whole /64. `SWW_WAF_IPV6_GROUP_PREFIX`, since one abuser usually controls a whole /64.
- Each limit is counted over a sliding window with a few integers per client. - Each limit is counted over a sliding window with a few integers per client.
The table of clients and the table of bans each have a bound The table of clients and the bans for limits and attacks each have a bound
(`SWW_WAF_MAX_TRACKED_CLIENTS`, `SWW_WAF_MAX_BANS`), so memory is bounded too. (`SWW_WAF_MAX_TRACKED_CLIENTS`, `SWW_WAF_MAX_BANS`). The bans for AbuseIPDB's
blacklist are bounded by the clients it listed over the last
`SWW_WAF_ABUSEIPDB_BAN_DAYS` days. So memory is bounded too.
- A ban refuses every request from its netblock until it ends. Requests refused - A ban refuses every request from its netblock until it ends. Requests refused
under a ban count toward no limit. A permanent ban ends only when an admin under a ban count toward no limit. A permanent ban ends only when an admin
lifts it. lifts it.
@@ -750,17 +896,25 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
- `cmd/sww`: the binary, which only calls `internal/sww`. - `cmd/sww`: the binary, which only calls `internal/sww`.
- `internal/sww`: the process, as the command it is given, and the health check. - `internal/sww`: the process, as the command it is given, and the health check.
It is the only package that imports `go.uber.org/fx`, which builds the parts
of `waf` and `rproxy`.
- `internal/config`: reads the settings, the one place they are read. - `internal/config`: reads the settings, the one place they are read.
- `internal/rproxy`: the reverse proxy in front of one app. - `internal/rproxy`: the reverse proxy in front of one app.
- `internal/waf`: `waf`'s server, which runs the checks and answers its own - `internal/waf`: `waf`'s server, which runs the checks and answers its own
endpoints. endpoints.
- `internal/wafprotocol`: the question, the answer and the report that `rproxy`
and `waf` send each other.
- `internal/ratelimit`, `internal/bans`, `internal/rules`, - `internal/ratelimit`, `internal/bans`, `internal/rules`,
`internal/coreruleset`, `internal/geojs`, `internal/lookup`, `internal/coreruleset`, `internal/geojs`, `internal/lookup`,
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and `internal/reputation`, `internal/routewatch`, `internal/anomaly` and
`internal/alerts`: one check or source each. `internal/alerts`: one check or source each.
- `internal/state`, `internal/requestlog`, `internal/remotelog` and - `internal/dirwatch`: watches a directory, so that `internal/rules` and
`internal/metrics`: the state and cache files, the log lines, the syslog `internal/lookup` read their files again when they change.
sender and the metrics. - `internal/state`: the state and cache files.
- `internal/requestlog`: the request log.
- `internal/processlog`: the process's own messages.
- `internal/remotelog`: the syslog sender.
- `internal/metrics`: the metrics.
## Rationale ## Rationale
+64 -24
View File
@@ -69,15 +69,21 @@ never as a time.
that command's settings, and runs `internal/rproxy` until `SIGTERM` or that command's settings, and runs `internal/rproxy` until `SIGTERM` or
`SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check. `SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
With no command, or one it does not know, the binary prints the commands With no command, or one it does not know, the binary prints the commands
and exits with status 2. and exits with status 2. `sww help` and `sww --help` print them and exit
with status 0. `internal/sww` is the only package that imports
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
and stops their background work.
- `internal/config`: reads the settings, the one place they are read. - `internal/config`: reads the settings, the one place they are read.
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and holds - `internal/waf`: `waf`'s server. It runs the checks in `check`, and answers
`waf`'s question, answer and report. It answers `waf`'s own endpoints in `waf`'s own endpoints in `answerOwnEndpoint`. It counts bytes, refusals
`answerOwnEndpoint`. It counts bytes, refusals and anomalies, and writes and anomalies, and writes the log lines.
the log lines.
- `internal/rproxy`: the reverse proxy in front of one app that asks `waf` - `internal/rproxy`: the reverse proxy in front of one app that asks `waf`
about each request. It passes a request to the app with the standard about each request. It passes a request to the app with the standard
library's `httputil.ReverseProxy`, within the time and size limits. library's `httputil.ReverseProxy`, within the time and size limits.
- `internal/wafprotocol`: the question, the answer and the report that
`rproxy` and `waf` send each other (see "`waf`"). It writes and reads each
one, and declares every header, path and limit both sides use.
`internal/rproxy` imports it, not `internal/waf`.
- `internal/ratelimit`: the table of clients. It counts requests, bytes and - `internal/ratelimit`: the table of clients. It counts requests, bytes and
refusals, and keeps each client's history. refusals, and keeps each client's history.
- `internal/bans`: the bans. It holds each netblock's bans with their notes, - `internal/bans`: the bans. It holds each netblock's bans with their notes,
@@ -91,8 +97,12 @@ never as a time.
imports no other package of this repository: `waf` gives it what it needs imports no other package of this repository: `waf` gives it what it needs
to count and report failures. to count and report failures.
- `internal/lookup`: looks clients up in the lookup database. - `internal/lookup`: looks clients up in the lookup database.
- `internal/dirwatch`: watches a directory, and calls a function once it has
gone a short time without a change. `internal/rules` and `internal/lookup`
read their files again this way.
- `internal/reputation`: fetches the lists, asks the DNS blocklists and - `internal/reputation`: fetches the lists, asks the DNS blocklists and
AbuseIPDB in the background, and keeps their answers. AbuseIPDB in the background, and keeps their answers. It also fetches
AbuseIPDB's blacklist, which `internal/waf` bans.
- `internal/routewatch`: asks a routewatch instance about clients in the - `internal/routewatch`: asks a routewatch instance about clients in the
background, and keeps its answers. background, and keeps its answers.
- `internal/anomaly`: the anomaly counters and their alerts. - `internal/anomaly`: the anomaly counters and their alerts.
@@ -101,8 +111,10 @@ never as a time.
its own. its own.
- `internal/state`: reads the state and cache files at start, takes in edits - `internal/state`: reads the state and cache files at start, takes in edits
while running, and writes the files when due and at the stop. while running, and writes the files when due and at the stop.
- `internal/requestlog` and `internal/remotelog`: the lines on stdout, and - `internal/requestlog`: the request log, as JSON lines on stdout.
sending them to the syslog server. - `internal/processlog`: the process lines on stdout, which hold the
process's own messages.
- `internal/remotelog`: sends the lines on stdout to the syslog server.
- `internal/metrics`: the metrics, in the Prometheus text format. - `internal/metrics`: the metrics, in the Prometheus text format.
- `Dockerfile`: the lint and test phases, then the image. - `Dockerfile`: the lint and test phases, then the image.
`deploy/example-app` is an app built on the image, with `rproxy` in its `deploy/example-app` is an app built on the image, with `rproxy` in its
@@ -121,6 +133,7 @@ never as a time.
- `github.com/fsnotify/fsnotify` to notice edited files; - `github.com/fsnotify/fsnotify` to notice edited files;
- `github.com/prometheus/client_golang` for the metrics; - `github.com/prometheus/client_golang` for the metrics;
- `github.com/spf13/viper` to read the settings from the environment; - `github.com/spf13/viper` to read the settings from the environment;
- `github.com/spf13/cobra` for the commands and their help text;
- `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases; - `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases;
- remote log sending written in this project, since the standard library's - remote log sending written in this project, since the standard library's
`log/syslog` is frozen and writes only the older syslog format. `log/syslog` is frozen and writes only the older syslog format.
@@ -238,13 +251,15 @@ apply.
load. load.
- Memory is bounded. When the table of clients holds - Memory is bounded. When the table of clients holds
`SWW_WAF_MAX_TRACKED_CLIENTS`, the least recently seen client is dropped `SWW_WAF_MAX_TRACKED_CLIENTS`, the least recently seen client is dropped
first, with its history. The bans are bounded by `SWW_WAF_MAX_BANS` (see first, with its history. The bans for broken limits and clear signs of attack
"Bans"). The GeoJS answers, the reputation verdicts and scores, the DNS are bounded by `SWW_WAF_MAX_BANS` (see "Bans"). Those for AbuseIPDB's
queries under way and the anomaly counters have fixed bounds, listed under blacklist are bounded by the clients it listed over the last
"Fixed limits" in `docs/settings.md`. Past its bound, the GeoJS answer or `SWW_WAF_ABUSEIPDB_BAN_DAYS` days (see "Reputation"). The GeoJS answers, the
anomaly counter used longest ago goes first, and the verdict or score fetched reputation verdicts and scores, the DNS queries under way and the anomaly
longest ago. Past the bound on DNS queries under way, a zone is asked at the counters have fixed bounds, listed under "Fixed limits" in `docs/settings.md`.
client's next request. Past its bound, the GeoJS answer or anomaly counter used longest ago goes
first, and the verdict or score fetched longest ago. Past the bound on DNS
queries under way, a zone is asked at the client's next request.
## Bans ## Bans
@@ -299,6 +314,9 @@ to a ban, each by its own rule:
Once it ends, only new requests can break a limit again. The client's Once it ends, only new requests can break a limit again. The client's
history keeps its totals. history keeps its totals.
A client on AbuseIPDB's blacklist is banned before it sends a request, with the
cause `abuseipdb` (see "Reputation").
A Core Rule Set match refuses only the request it matched, with `403` and A Core Rule Set match refuses only the request it matched, with `403` and
without a ban, because the Core Rule Set has false positives. A `block` rule without a ban, because the Core Rule Set has false positives. A `block` rule
does the same. Those refusals count toward the refusal limit, so a client that does the same. Those refusals count toward the refusal limit, so a client that
@@ -325,11 +343,12 @@ does not count toward a longer ban; it refuses nothing. Deleting its entry from
A permanent ban does not run out. It ends when an admin lifts it, or when `waf` A permanent ban does not run out. It ends when an admin lifts it, or when `waf`
drops it from a full table, as below. When `waf` holds `SWW_WAF_MAX_BANS` bans drops it from a full table, as below. When `waf` holds `SWW_WAF_MAX_BANS` bans
of its own, it drops the earliest ban of one netblock: the netblock that has for broken limits and clear signs of attack, it drops the earliest such ban of
gone longest without a request. A past, active or permanent ban is dropped one netblock: the netblock that has gone longest without a request. A past,
alike. A scanner whose ban was dropped that way is refused and banned again by active or permanent ban is dropped alike. A scanner whose ban was dropped that
its next probe. A ban whose cause is `admin` is never dropped and does not count way is refused and banned again by its next probe. Bans whose cause is `admin`
toward `SWW_WAF_MAX_BANS`, so such bans can never fill the table. or `abuseipdb` do not count toward `SWW_WAF_MAX_BANS` and are never dropped to
make room, so they can never push out the others.
A request refused under a ban that runs out is logged with the action A request refused under a ban that runs out is logged with the action
`temporarily_banned`. Under a permanent ban the action is `permanently_banned`. `temporarily_banned`. Under a permanent ban the action is `permanently_banned`.
@@ -2036,6 +2055,26 @@ host, sharing one address, would often fetch too close together.
- AbuseIPDB answers other than `200`; - AbuseIPDB answers other than `200`;
- the answer gives no score; - the answer gives no score;
- no answer comes within `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`. - no answer comes within `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`.
- While `SWW_WAF_ABUSEIPDB_KEY` is set, AbuseIPDB's blacklist is fetched at the
start and every `SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS` minutes
(default `60`). It lists the clients AbuseIPDB scores at
`SWW_WAF_ABUSEIPDB_THRESHOLD` (default `90`) or more. The blacklist itself is
not kept.
- Each client on it is banned for `SWW_WAF_ABUSEIPDB_BAN_DAYS` days (default
`30`), with the cause `abuseipdb`. The ban covers the client's netblock,
as any ban does.
- A netblock that already has an active ban is left as it is. So is one
whose `abuseipdb` ban an admin lifted, until that ban would have ended.
- An `abuseipdb` ban that has ended is forgotten at the next of these:
- a fetch of the blacklist in `enforce` mode;
- a read of `bans.json`, at the start or after an admin's edit.
- In `observe` mode the blacklist makes no ban.
- The threshold takes 25 to 100. It should not be set below 50 to 75, since
lower scores list ordinary visitors. AbuseIPDB serves free accounts the
blacklist at the threshold 100, whatever is asked for.
- AbuseIPDB limits the fetches of the blacklist a day: 5 for a free account.
A fetch over the limit fails, and is logged and alerted. The default
refresh fetches it 24 times a day.
- `SWW_WAF_DNSBL_ABUSEIPDB_ACTION`, for DNS blocklist and AbuseIPDB hits, is - `SWW_WAF_DNSBL_ABUSEIPDB_ACTION`, for DNS blocklist and AbuseIPDB hits, is
`limit:25` by default. Such verdicts are less certain than a blocklist, so a `limit:25` by default. Such verdicts are less certain than a blocklist, so a
listed client gets a quarter of every request and byte limit rather than a listed client gets a quarter of every request and byte limit rather than a
@@ -2050,9 +2089,10 @@ host, sharing one address, would often fetch too close together.
- Alerts go to a webhook as JSON, and to Slack and ntfy as formatted messages. - Alerts go to a webhook as JSON, and to Slack and ntfy as formatted messages.
`docs/alerts.md` lists the events, the JSON object and the messages. `docs/alerts.md` lists the events, the JSON object and the messages.
`SWW_WAF_ALERT_EVENTS` chooses which of the seven event types are sent; the `SWW_WAF_ALERT_EVENTS` chooses which of the seven event types are sent; the
hourly `summary` is sent whatever it says. The bans an admin makes raise no hourly `summary` is sent whatever it says. The bans an admin makes, and the
alert. In `observe` mode, a request that would have made a ban still raises bans for AbuseIPDB's blacklist, raise no alert. In `observe` mode, a request
the ban's alert, with `mode` set to `observe` in its `detail`. that would have made a ban still raises the ban's alert, with `mode` set to
`observe` in its `detail`.
- A repeat is an alert with the same event, and the same netblock, source, file - A repeat is an alert with the same event, and the same netblock, source, file
or anomaly scope, as one sent less than `SWW_WAF_ALERT_COOLDOWN` before. An or anomaly scope, as one sent less than `SWW_WAF_ALERT_COOLDOWN` before. An
anomaly scope is one of the things counted under "Anomaly thresholds". A anomaly scope is one of the things counted under "Anomaly thresholds". A
@@ -2200,7 +2240,7 @@ lists the files, what each holds, and their sizes.
these stop the process with a message naming the file and position: these stop the process with a message naming the file and position:
- a file that does not parse, or has an unknown `version`; - a file that does not parse, or has an unknown `version`;
- an entry without a field it needs; - an entry without a field it needs;
- a ban whose cause is not one of the four; - a ban whose cause is not `limit`, `attack`, `admin` or `abuseipdb`;
- a ban whose `expires` is not after its `start`. - a ban whose `expires` is not after its `start`.
Starting with empty state would silently forgive every repeat offender. Starting with empty state would silently forgive every repeat offender.
+4
View File
@@ -9,6 +9,10 @@
ARG SWW_IMAGE=smallwebwaf ARG SWW_IMAGE=smallwebwaf
FROM ${SWW_IMAGE} FROM ${SWW_IMAGE}
# Its own title and description, in place of the smallwebwaf image's.
LABEL org.opencontainers.image.title="smallwebwaf-example-app" \
org.opencontainers.image.description="An example app built on the smallwebwaf image."
# Packages the app needs, from the nixpkgs in the image. # Packages the app needs, from the nixpkgs in the image.
RUN nix-env -iA nixpkgs.busybox RUN nix-env -iA nixpkgs.busybox
+3 -2
View File
@@ -6,7 +6,7 @@ back or dropped.
## Events ## Events
- `ban`: a ban `waf` made. - `ban`: a ban `waf` made for a broken limit or a clear sign of attack.
- `permanent_ban`: a permanent ban it made, or an existing ban made permanent by - `permanent_ban`: a permanent ban it made, or an existing ban made permanent by
a request. a request.
- `core_rule_set_match`: the Core Rule Set scored a request at or over the - `core_rule_set_match`: the Core Rule Set scored a request at or over the
@@ -68,7 +68,8 @@ on one line:
"earlier_bans": { "earlier_bans": {
"limit": 0, "limit": 0,
"attack": 0, "attack": 0,
"admin": 0 "admin": 0,
"abuseipdb": 0
} }
} }
}, },
+14 -2
View File
@@ -42,8 +42,8 @@ chooses between the two (see "The image's command" below).
client's group, from 32 to 128. client's group, from 32 to 128.
- `SWW_WAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept, a - `SWW_WAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept, a
whole number above zero. whole number above zero.
- `SWW_WAF_MAX_BANS` (default `5000`): how many of its own bans `waf` keeps, a - `SWW_WAF_MAX_BANS` (default `5000`): the most bans `waf` keeps for broken
whole number above zero. limits and attacks, a whole number above zero.
## Tokens ## Tokens
@@ -338,6 +338,13 @@ tag that no request can get stops the start. A request can get only these:
or AbuseIPDB's score is used. A duration. or AbuseIPDB's score is used. A duration.
- `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS` (default `2000`): how long a query to a - `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS` (default `2000`): how long a query to a
zone or a check with AbuseIPDB may take. A time in milliseconds. zone or a check with AbuseIPDB may take. A time in milliseconds.
- `SWW_WAF_ABUSEIPDB_THRESHOLD` (default `90`): the least abuse confidence score
of the clients on AbuseIPDB's blacklist, a whole number from 25 to 100. It
should not be set below 50 to 75, since lower scores list ordinary visitors.
- `SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS` (default `60`): how often
the blacklist is fetched again, in minutes. A whole number above zero.
- `SWW_WAF_ABUSEIPDB_BAN_DAYS` (default `30`): how long the ban of a client on
the blacklist lasts, in days. A whole number above zero.
## Alerting ## Alerting
@@ -364,6 +371,11 @@ tag that no request can get stops the start. A request can get only these:
## Anomaly thresholds ## Anomaly thresholds
An anomaly is more requests or bytes than one of these thresholds allows.
Crossing a threshold sends one `anomaly` alert to the destinations under
"Alerting", and does nothing else: no client is banned, rate limited or blocked.
`SWW_WAF_ALERT_COOLDOWN` holds back repeats of that alert.
Each scope has four thresholds: the most requests and the most body bytes Each scope has four thresholds: the most requests and the most body bytes
counted in a minute and in an hour. The bytes are those `SWW_WAF_BYTES_COUNTED` counted in a minute and in an hour. The bytes are those `SWW_WAF_BYTES_COUNTED`
names. Requests are counts, bytes are sizes. Each is `off` by default. The four names. Requests are counts, bytes are sizes. Each is `off` by default. The four
+19 -5
View File
@@ -21,11 +21,22 @@ top-level `version` of 1.
## `bans.json` ## `bans.json`
Active and past bans: every ban an admin made, and up to `SWW_WAF_MAX_BANS` of Active and past bans:
those `waf` made. Per entry:
- every ban an admin made;
- up to `SWW_WAF_MAX_BANS` of those `waf` made for a broken limit or an attack;
- the bans for AbuseIPDB's blacklist.
A ban for AbuseIPDB's blacklist that has ended is forgotten at the next of
these:
- a fetch of the blacklist in `enforce` mode;
- a read of `bans.json`, at the start or after an admin's edit.
Per entry:
- `netblock`, `start` and `expires` (`null` for a permanent ban); - `netblock`, `start` and `expires` (`null` for a permanent ban);
- `cause`: `attack`, `limit` or `admin`; - `cause`: `attack`, `limit`, `admin` or `abuseipdb`;
- `reason`: a short reason; - `reason`: a short reason;
- `lifted`: when an admin lifted it, if one did; - `lifted`: when an admin lifted it, if one did;
- `notes`: the notes below. - `notes`: the notes below.
@@ -60,7 +71,8 @@ ban. The notes:
- for a clear sign of attack, `rule_id` and `target`, or `trap_path`; - for a clear sign of attack, `rule_id` and `target`, or `trap_path`;
- `reputation`: the sources the client was on, each with its `source` and, for - `reputation`: the sources the client was on, each with its `source` and, for
AbuseIPDB, its `score`; AbuseIPDB, its `score`;
- `request`: the request that caused the ban, each text cut to 256 bytes: - `request`: the request that caused the ban, each text cut to 256 bytes, and
empty for a ban for AbuseIPDB's blacklist:
- `time`, `method` and `host`; - `time`, `method` and `host`;
- `path`, with its query string; - `path`, with its query string;
- `status` and `user_agent`; - `status` and `user_agent`;
@@ -151,7 +163,9 @@ A cache file. routewatch's answers from the last hour, one per line:
- `bans.json`: about 1.2 KiB per ban, at most about 2.5 KiB, so about 6 MiB at - `bans.json`: about 1.2 KiB per ban, at most about 2.5 KiB, so about 6 MiB at
the default `SWW_WAF_MAX_BANS`. Written every 15 minutes, a full file comes to the default `SWW_WAF_MAX_BANS`. Written every 15 minutes, a full file comes to
about 600 MiB of writes a day, and a hundred new bans a day add about 600 MiB about 600 MiB of writes a day, and a hundred new bans a day add about 600 MiB
more. more. A ban for AbuseIPDB's blacklist takes about 1 KiB. A free account's
blacklist lists 10,000 clients, so it adds at least 10 MiB, and 1 GiB of
writes a day.
- `lookups.json`: about 150 bytes per answer, under 1.5 GiB of writes a day when - `lookups.json`: about 150 bytes per answer, under 1.5 GiB of writes a day when
full. full.
- `routewatch.json`: about 120 bytes per answer, at most about 12 MiB. - `routewatch.json`: about 120 bytes per answer, at most about 12 MiB.
+7 -1
View File
@@ -11,7 +11,10 @@ require (
github.com/maxmind/mmdbwriter v1.2.0 github.com/maxmind/mmdbwriter v1.2.0
github.com/oschwald/maxminddb-golang/v2 v2.7.0 github.com/oschwald/maxminddb-golang/v2 v2.7.0
github.com/prometheus/client_golang v1.24.1 github.com/prometheus/client_golang v1.24.1
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0 github.com/spf13/viper v1.21.0
go.uber.org/fx v1.24.0
golang.org/x/net v0.60.0
golang.org/x/sys v0.48.0 golang.org/x/sys v0.48.0
) )
@@ -24,6 +27,7 @@ require (
github.com/goccy/go-yaml v1.19.2 // indirect github.com/goccy/go-yaml v1.19.2 // indirect
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/kaptinlin/go-i18n v0.1.4 // indirect github.com/kaptinlin/go-i18n v0.1.4 // indirect
github.com/kaptinlin/jsonschema v0.4.6 // indirect github.com/kaptinlin/jsonschema v0.4.6 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect github.com/kylelemons/godebug v1.1.0 // indirect
@@ -44,9 +48,11 @@ require (
github.com/tidwall/match v1.1.1 // indirect github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.1 // indirect github.com/tidwall/pretty v1.2.1 // indirect
github.com/valllabh/ocsf-schema-golang v1.0.3 // indirect github.com/valllabh/ocsf-schema-golang v1.0.3 // indirect
go.uber.org/dig v1.19.0 // indirect
go.uber.org/multierr v1.10.0 // indirect
go.uber.org/zap v1.26.0 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect go.yaml.in/yaml/v3 v3.0.5 // indirect
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
golang.org/x/net v0.60.0 // indirect
golang.org/x/sync v0.23.0 // indirect golang.org/x/sync v0.23.0 // indirect
golang.org/x/text v0.42.0 // indirect golang.org/x/text v0.42.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect google.golang.org/protobuf v1.36.11 // indirect
+17
View File
@@ -14,6 +14,7 @@ github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68 h1:KLEjvBBSw
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8= github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8=
github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk= github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk=
github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk= github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0= github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk= github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8= github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
@@ -34,6 +35,8 @@ github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 h1:c7gcN
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU= github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y= github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y=
github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk= github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk=
github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs= github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs=
@@ -72,6 +75,7 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY= github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8= github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc= github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik= github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw= github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
@@ -80,6 +84,9 @@ github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg= github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY= github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo= github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk= github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU= github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
@@ -97,10 +104,19 @@ github.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4=
github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU= github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/valllabh/ocsf-schema-golang v1.0.3 h1:eR8k/3jP/OOqB8LRCtdJ4U+vlgd/gk5y3KMXoodrsrw= github.com/valllabh/ocsf-schema-golang v1.0.3 h1:eR8k/3jP/OOqB8LRCtdJ4U+vlgd/gk5y3KMXoodrsrw=
github.com/valllabh/ocsf-schema-golang v1.0.3/go.mod h1:sZ3as9xqm1SSK5feFWIR2CuGeGRhsM7TR1MbpBctzPk= github.com/valllabh/ocsf-schema-golang v1.0.3/go.mod h1:sZ3as9xqm1SSK5feFWIR2CuGeGRhsM7TR1MbpBctzPk=
go.uber.org/dig v1.19.0 h1:BACLhebsYdpQ7IROQ1AGPjrXcP5dF80U3gKoFzbaq/4=
go.uber.org/dig v1.19.0/go.mod h1:Us0rSJiThwCv2GteUN0Q7OKvU7n5J4dxZ9JKUXozFdE=
go.uber.org/fx v1.24.0 h1:wE8mruvpg2kiiL1Vqd0CC+tr0/24XIB10Iwp2lLWzkg=
go.uber.org/fx v1.24.0/go.mod h1:AmDeGyS+ZARGKM4tlH4FY2Jr63VjbEDJHtqXTGP5hbo=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ= go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ= go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw= go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg= go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M= go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M=
@@ -119,6 +135,7 @@ golang.org/x/tools v0.50.0 h1:c2ifzfcuY7L90lZ2aKd8S4K2NpASF08SZx9ZuJkHmSU=
golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0= golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE= rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
+169
View File
@@ -0,0 +1,169 @@
package bans_test
import (
"net/netip"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
)
func TestBlacklistedNetblockIsBannedUntilTheGivenEnd(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
now := midnight()
expires := now.Add(30 * day)
made := ledger.BanForAbuseIPDB(
now, expires, map[netip.Prefix]int64{netblock: 95},
)
if made != 1 || ledger.Made(bans.CauseAbuseIPDB) != 1 {
t.Fatalf("made %d bans, %d counted, want 1 and 1",
made, ledger.Made(bans.CauseAbuseIPDB))
}
ban, banned, madePermanent := ledger.Check(netblock.Addr(), now)
if !banned || madePermanent || !ban.Expires.Equal(expires) ||
ban.Cause != bans.CauseAbuseIPDB ||
ban.Reason != "on AbuseIPDB's blacklist with a score of 95" ||
len(ban.Notes.Reputation) != 1 ||
ban.Notes.Reputation[0].Source != "abuseipdb" ||
*ban.Notes.Reputation[0].Score != 95 {
t.Errorf(
"ban %+v, banned %t, made permanent %t, want the blacklist's "+
"ban until %s",
ban,
banned,
madePermanent,
expires,
)
}
if _, banned, _ := ledger.Check(netblock.Addr(), expires); banned {
t.Error("the ban refuses a request at its end")
}
}
func TestBlacklistLeavesANetblockWithAnActiveBanAsItIs(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
now := midnight()
scores := map[netip.Prefix]int64{netblock: 100}
limitBan, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 0 {
t.Errorf("made %d bans over a ban for a limit, want none", made)
}
// Once that ban has ended the netblock is banned, and the blacklist
// fetched again makes no second ban while the first lasts.
later := limitBan.Expires
if made := ledger.BanForAbuseIPDB(later, later.Add(30*day), scores); made != 1 {
t.Errorf("made %d bans after the ban for a limit, want 1", made)
}
later = later.Add(time.Hour)
if made := ledger.BanForAbuseIPDB(later, later.Add(30*day), scores); made != 0 {
t.Errorf("made %d bans while the blacklist's lasts, want none", made)
}
wantBans(t, ledger, map[netip.Prefix]int{netblock: 2})
if got := ledger.Bans(netblock)[1].Notes.EarlierBans; got != (bans.EarlierBans{
Limit: 1,
}) {
t.Errorf("earlier bans %+v, want 1 for a limit", got)
}
}
func TestLiftedBlacklistBanHoldsUntilItsEnd(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
now := midnight()
expires := now.Add(30 * day)
scores := map[netip.Prefix]int64{netblock: 100}
ledger.BanForAbuseIPDB(now, expires, scores)
ledger.Lift(netblock.Addr(), now)
// The admin lifted it: the blacklist fetched again until its end bans
// nothing.
later := expires.Add(-time.Hour)
if made := ledger.BanForAbuseIPDB(later, later.Add(30*day), scores); made != 0 {
t.Errorf("made %d bans before the lifted ban's end, want none", made)
}
// From its end on, the blacklist bans the netblock again, and the
// lifted ban, ended, is forgotten.
if made := ledger.BanForAbuseIPDB(expires, expires.Add(30*day), scores); made != 1 {
t.Errorf("made %d bans at the lifted ban's end, want 1", made)
}
wantBans(t, ledger, map[netip.Prefix]int{netblock: 1})
}
func TestEndedBlacklistBansAreForgottenAtTheNextFetch(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultParams())
listed := netip.MustParsePrefix("203.0.113.9/32")
limited := netip.MustParsePrefix("203.0.113.10/32")
now := midnight()
expires := now.Add(30 * day)
ledger.BanForAbuseIPDB(now, expires, map[netip.Prefix]int64{listed: 100})
ledger.Ban(limited, now, bans.Grounds{Cause: bans.CauseLimit})
// listed is off the blacklist when it is fetched after the ban's end.
if made := ledger.BanForAbuseIPDB(expires, expires.Add(30*day), nil); made != 0 {
t.Errorf("made %d bans for an empty blacklist, want none", made)
}
wantBans(t, ledger, map[netip.Prefix]int{listed: 0, limited: 1})
}
func TestBlacklistBansDoNotCountTowardMaxBans(t *testing.T) {
t.Parallel()
params := defaultParams()
params.MaxBans = 1
ledger := bans.New(params)
limited := netip.MustParsePrefix("203.0.113.1/32")
attacker := netip.MustParsePrefix("203.0.113.2/32")
now := midnight()
scores := map[netip.Prefix]int64{}
for i := range 3 {
scores[netip.PrefixFrom(netip.AddrFrom4([4]byte{198, 51, 100, byte(i)}), 32)] = 100
}
ledger.Ban(limited, now, bans.Grounds{Cause: bans.CauseLimit})
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 3 {
t.Errorf("made %d bans for 3 netblocks, want 3", made)
}
wantBans(t, ledger, map[netip.Prefix]int{limited: 1})
// The ban for an attack takes the place of the one for a limit, and no
// blacklist ban goes.
ledger.Ban(attacker, now, bans.Grounds{Cause: bans.CauseAttack})
want := map[netip.Prefix]int{limited: 0, attacker: 1}
for netblock := range scores {
want[netblock] = 1
}
wantBans(t, ledger, want)
}
+64 -23
View File
@@ -61,7 +61,7 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 2}) wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 2})
// A new ban drops the ban sww made, and only that one. // A new ban drops the ban sww made, and only that one.
ledger.BanForLimit(second, midnight(), bans.Notes{}) ledger.Ban(second, midnight(), bans.Grounds{Cause: bans.CauseLimit})
wantBans( wantBans(
t, t,
ledger, ledger,
@@ -77,7 +77,7 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
} }
// And the next drops that one. // And the next drops that one.
ledger.BanForLimit(third, midnight(), bans.Notes{}) ledger.Ban(third, midnight(), bans.Grounds{Cause: bans.CauseLimit})
wantBans( wantBans(
t, t,
ledger, ledger,
@@ -98,20 +98,28 @@ func TestEarlierBansCountABanDroppedAfterAnAdminsBan(t *testing.T) {
// twice, each time after its last ban has ended. // twice, each time after its last ban has ended.
ledger.BanForAdmin(netblock, midnight(), midnight().Add(time.Minute), ledger.BanForAdmin(netblock, midnight(), midnight().Add(time.Minute),
"probes for logins") "probes for logins")
first, _ := ledger.BanForLimit( first, _ := ledger.Ban(
netblock, netblock,
midnight().Add(time.Minute), midnight().Add(time.Minute),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
) )
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
// A ban on another netblock fills the ledger, and the netblock's first // A ban on another netblock fills the ledger, and the netblock's first
// ban for a limit, its earliest that is not an admin's, goes. // ban for a limit, its earliest that is not an admin's, goes.
ledger.BanForLimit(other, second.Start, bans.Notes{}) ledger.Ban(other, second.Start, bans.Grounds{Cause: bans.CauseLimit})
wantBans(t, ledger, map[netip.Prefix]int{netblock: 2, other: 1}) wantBans(t, ledger, map[netip.Prefix]int{netblock: 2, other: 1})
// The netblock's next ban still counts it. // The netblock's next ban still counts it.
third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{}) third, _ := ledger.Ban(
netblock,
second.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 2, Admin: 1}) { if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 2, Admin: 1}) {
t.Errorf("the third ban for a limit has earlier bans %+v, "+ t.Errorf("the third ban for a limit has earlier bans %+v, "+
"want 2 for a limit and 1 an admin's", third.Notes.EarlierBans) "want 2 for a limit and 1 an admin's", third.Notes.EarlierBans)
@@ -127,15 +135,27 @@ func TestEarlierBansCountABanAnAdminKeepsOnce(t *testing.T) {
// The netblock breaks a limit twice, the second time after its first // The netblock breaks a limit twice, the second time after its first
// ban has ended, and the second ban's notes count the first. An admin // ban has ended, and the second ban's notes count the first. An admin
// then keeps the first ban. // then keeps the first ban.
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{}) first, _ := ledger.Ban(
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{}) netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
kept := first kept := first
kept.Cause = bans.CauseAdmin kept.Cause = bans.CauseAdmin
ledger.LoadEdit([]bans.Ban{kept, second}) ledger.LoadEdit([]bans.Ban{kept, second})
// The netblock's next ban counts the kept ban once, as an admin's. // The netblock's next ban counts the kept ban once, as an admin's.
third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{}) third, _ := ledger.Ban(
netblock,
second.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Admin: 1}) { if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Admin: 1}) {
t.Errorf("the third ban for a limit has earlier bans %+v, "+ t.Errorf("the third ban for a limit has earlier bans %+v, "+
"want 1 for a limit and 1 an admin's", third.Notes.EarlierBans) "want 1 for a limit and 1 an admin's", third.Notes.EarlierBans)
@@ -147,20 +167,37 @@ func TestReasonOfTheBansSwwMakes(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
limit, _ := ledger.BanForLimit( limit, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.1/32"), netip.MustParsePrefix("203.0.113.1/32"),
midnight(), midnight(),
bans.Notes{Kind: "requests", Limit: 1000, Window: "minute"}, bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{
Kind: "requests",
Limit: 1000,
Window: "minute",
},
},
) )
byteLimit, _ := ledger.BanForLimit( byteLimit, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.3/32"), netip.MustParsePrefix("203.0.113.3/32"),
midnight(), midnight(),
bans.Notes{Kind: "bytes", Limit: 10 << 30, Window: "hour"}, bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{
Kind: "bytes",
Limit: 10 << 30,
Window: "hour",
},
},
) )
attack, _ := ledger.BanForAttack( attack, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.2/32"), netip.MustParsePrefix("203.0.113.2/32"),
midnight(), midnight(),
bans.Notes{RuleID: "git-dir", Target: "path"}, bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "git-dir", Target: "path"},
},
) )
for _, tc := range []struct{ got, want string }{ for _, tc := range []struct{ got, want string }{
@@ -200,7 +237,7 @@ func TestLiftedBanForALimitRefusesNothingAndMakesNoBanLonger(t *testing.T) {
t.Error("the lifted ban refuses") t.Error("the lifted ban refuses")
} }
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
if ban.Expires.Sub(ban.Start) != time.Hour || if ban.Expires.Sub(ban.Start) != time.Hour ||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) { ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
t.Errorf( t.Errorf(
@@ -251,7 +288,11 @@ func TestLiftedBanForAnAttackRefusesNothingAndMakesNoBanLonger(
} }
// The next clear sign of attack bans for seven days, as a first does. // The next clear sign of attack bans for seven days, as a first does.
ban, _ := ledger.BanForAttack(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseAttack},
)
if ban.Expires.Sub(ban.Start) != 7*day { if ban.Expires.Sub(ban.Start) != 7*day {
t.Errorf( t.Errorf(
"the next ban for an attack ends at %s, want seven days on", "the next ban for an attack ends at %s, want seven days on",
@@ -264,10 +305,10 @@ func TestLoadEditCountsTheBansAnAdminMade(t *testing.T) {
t.Parallel() t.Parallel()
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
made, _ := ledger.BanForLimit( made, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.1/32"), netip.MustParsePrefix("203.0.113.1/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
atStart := bans.Ban{ atStart := bans.Ban{
Netblock: netip.MustParsePrefix("203.0.113.2/32"), Netblock: netip.MustParsePrefix("203.0.113.2/32"),
@@ -311,7 +352,7 @@ func TestAdminsBanIsMadeWhileAnotherLasts(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
// An hour's ban for a broken limit. // An hour's ban for a broken limit.
ledger.BanForLimit(netblock, midnight(), bans.Notes{}) ledger.Ban(netblock, midnight(), bans.Grounds{Cause: bans.CauseLimit})
wantChanged(t, ledger, true) wantChanged(t, ledger, true)
// A minute later an admin bans the netblock for good, named by an // A minute later an admin bans the netblock for good, named by an
@@ -431,10 +472,10 @@ func TestPermanentBanOnAWiderNetblockIsReturnedOverATemporaryBan(
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
// The temporary ban is made first, so that its netblock is looked at first. // The temporary ban is made first, so that its netblock is looked at first.
ledger.BanForLimit( ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
ledger.BanForAdmin( ledger.BanForAdmin(
netip.MustParsePrefix("203.0.113.0/24"), netip.MustParsePrefix("203.0.113.0/24"),
+215 -116
View File
@@ -1,8 +1,9 @@
// Package bans is the ban ledger: the bans sww makes on the // Package bans is the ban ledger: the bans sww makes on the
// netblocks of clients that break a rate limit, a byte limit or the refusal // netblocks of clients that break a rate limit, a byte limit or the refusal
// limit or show a clear sign of attack, and those an admin makes, with their // limit or show a clear sign of attack, or are on AbuseIPDB's blacklist, and
// notes, as the "Bans" section of SPEC.md describes. The bans are kept in // those an admin makes, with their notes, as the "Bans" section of SPEC.md
// memory, and written to bans.json and read from it by the state package. // describes. The bans are kept in memory, and written to bans.json and read
// from it by the state package.
package bans package bans
import ( import (
@@ -26,6 +27,9 @@ const (
// CauseAdmin is a ban an admin made, or one sww made that an // CauseAdmin is a ban an admin made, or one sww made that an
// admin keeps. It is never dropped. // admin keeps. It is never dropped.
CauseAdmin = "admin" CauseAdmin = "admin"
// CauseAbuseIPDB is a ban sww made for a client on AbuseIPDB's
// blacklist. It is forgotten once it has ended.
CauseAbuseIPDB = "abuseipdb"
) )
// repeatFactor is how many times as long as the netblock's last ban a ban // repeatFactor is how many times as long as the netblock's last ban a ban
@@ -50,10 +54,11 @@ type Params struct {
// AttackBanDuration is how long a first ban for a clear sign of attack // AttackBanDuration is how long a first ban for a clear sign of attack
// lasts. // lasts.
AttackBanDuration time.Duration AttackBanDuration time.Duration
// MaxBans is the most bans held whose cause is not CauseAdmin, at // MaxBans is the most bans held whose cause is CauseLimit or
// least one. Past it, the earliest such ban of the netblock that has // CauseAttack, at least one. Past it, the earliest such ban of the
// gone longest without a request is dropped. Bans whose cause is // netblock that has gone longest without a request is dropped. Bans
// CauseAdmin are held besides, and never dropped. // whose cause is CauseAdmin or CauseAbuseIPDB are held besides, and
// never dropped this way.
MaxBans int MaxBans int
} }
@@ -63,10 +68,11 @@ type Ban struct {
Start time.Time Start time.Time
// Expires is when the ban ends, zero for a permanent ban. // Expires is when the ban ends, zero for a permanent ban.
Expires time.Time Expires time.Time
// Cause is CauseLimit, CauseAttack or CauseAdmin. // Cause is CauseLimit, CauseAttack, CauseAdmin or CauseAbuseIPDB.
Cause string Cause string
// Reason is a short text: for a ban sww made, the limit broken or // Reason is a short text: for a ban sww made, the limit broken, the
// the rule that matched; for an admin's, what the admin wrote. // rule that matched or AbuseIPDB's score; for an admin's, what the
// admin wrote.
Reason string Reason string
// Lifted is when an admin lifted the ban, zero while no admin has. A // Lifted is when an admin lifted the ban, zero while no admin has. A
// lifted ban refuses nothing, and does not make the netblock's next // lifted ban refuses nothing, and does not make the netblock's next
@@ -127,7 +133,8 @@ type Notes struct {
// log's reputation names them. It is left out when none did. // log's reputation names them. It is left out when none did.
Reputation []ReputationHit `json:"reputation,omitempty"` Reputation []ReputationHit `json:"reputation,omitempty"`
// Request is the request that broke the limit, or whose bytes broke // Request is the request that broke the limit, or whose bytes broke
// it, or that was the clear sign of attack. // it, or that was the clear sign of attack. A ban for AbuseIPDB's
// blacklist has none, and leaves it zero.
Request Request `json:"request"` Request Request `json:"request"`
// Requests is how many requests the netblock has sent since it was // Requests is how many requests the netblock has sent since it was
// first seen, and Refused how many of them the ban has refused so // first seen, and Refused how many of them the ban has refused so
@@ -150,9 +157,10 @@ type ReputationHit struct {
// EarlierBans counts a netblock's bans before a ban, by cause. // EarlierBans counts a netblock's bans before a ban, by cause.
type EarlierBans struct { type EarlierBans struct {
Limit int `json:"limit"` Limit int `json:"limit"`
Attack int `json:"attack"` Attack int `json:"attack"`
Admin int `json:"admin"` Admin int `json:"admin"`
AbuseIPDB int `json:"abuseipdb"`
} }
// Request is a request in a ban's notes. Each text is cut to 256 bytes. // Request is a request in a ban's notes. Each text is cut to 256 bytes.
@@ -180,8 +188,8 @@ type Ledger struct {
// netblocks holds each banned netblock's bans, oldest first. Check and // netblocks holds each banned netblock's bans, oldest first. Check and
// Find make each netblock they find the most recently seen. // Find make each netblock they find the most recently seen.
netblocks *simplelru.LRU[netip.Prefix, *[]Ban] netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
// held is how many bans netblocks holds whose cause is not CauseAdmin, // held is how many bans netblocks holds whose cause is CauseLimit or
// at most params.MaxBans. // CauseAttack, at most params.MaxBans.
held int held int
// made is how many bans have been made since the start, by cause: by // made is how many bans have been made since the start, by cause: by
// the ledger, and by an admin, through BanForAdmin or in an edit of // the ledger, and by an admin, through BanForAdmin or in an edit of
@@ -198,8 +206,9 @@ type Ledger struct {
// New returns a Ledger with no ban yet. // New returns a Ledger with no ban yet.
func New(params Params) *Ledger { func New(params Params) *Ledger {
// The ledger drops bans itself, and never those whose cause is // The ledger drops bans itself, and never those whose cause is
// CauseAdmin, however many there are, so the LRU has no limit of its // CauseAdmin or CauseAbuseIPDB to make room, however many there are, so
// own: it keeps the netblocks in the order they were last seen. // the LRU has no limit of its own: it keeps the netblocks in the order
// they were last seen.
netblocks, err := simplelru.NewLRU[netip.Prefix, *[]Ban]( netblocks, err := simplelru.NewLRU[netip.Prefix, *[]Ban](
math.MaxInt, math.MaxInt,
nil, nil,
@@ -291,81 +300,50 @@ func activeBan(bans []Ban, now time.Time) *Ban {
return newest return newest
} }
// BanForLimit bans netblock at now for a broken limit, with notes, and // Grounds are why a request makes a ban: its cause, CauseLimit or
// returns the ban, and true. A first ban lasts LimitBanDuration. A ban // CauseAttack, and its notes.
// made within LimitBanRepeatWindow after the netblock's ban that ended type Grounds struct {
// last, other than one for a clear sign of attack or a lifted one, lasts Cause string
// repeatFactor times as long as that one. A ban that would be longer Notes Notes
// than LimitBanPermanentAbove is permanent instead. If a ban on netblock is still
// active, as when two of its requests break a limit at once, that ban is
// returned with false, and no other is made. The ledger fills in the
// notes' Refused and EarlierBans itself, and gives the ban the reason
// "<Kind> per <Window> over the limit of <Limit>", from the notes, such
// as "requests per minute over the limit of 1000".
func (l *Ledger) BanForLimit(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(
netblock,
now,
CauseLimit,
limitReason(notes),
notes,
true,
)
} }
// WouldBanForLimit returns what BanForLimit would, without making the ban: // Ban bans netblock at now on grounds, and returns the ban, and true. If
// what observe mode would have done. // a ban on netblock is still active, as when two of its requests break a
func (l *Ledger) WouldBanForLimit( // limit at once, that ban is returned with false, and no other is made.
netblock netip.Prefix, now time.Time, notes Notes, // The ledger fills in the notes' Refused and EarlierBans itself.
//
// A ban for a broken limit:
// - lasts LimitBanDuration the first time;
// - lasts repeatFactor times as long as the netblock's ban that ended
// last, when made within LimitBanRepeatWindow after it, unless that
// ban was for a clear sign of attack or was lifted;
// - is permanent instead when it would be longer than
// LimitBanPermanentAbove;
// - has the reason "<Kind> per <Window> over the limit of <Limit>", from
// the notes, such as "requests per minute over the limit of 1000".
//
// A ban for a clear sign of attack:
// - lasts AttackBanDuration the first time;
// - is permanent once the netblock has had one that was not lifted;
// - has the reason "matched the rule <RuleID>", or "asked for the trap
// path <TrapPath>".
func (l *Ledger) Ban(
netblock netip.Prefix, now time.Time, grounds Grounds,
) (Ban, bool) { ) (Ban, bool) {
return l.ban( return l.ban(netblock, now, grounds, true)
netblock,
now,
CauseLimit,
limitReason(notes),
notes,
false,
)
} }
// BanForAttack bans netblock at now for a clear sign of attack, with // WouldBan returns what Ban would, without making the ban: what observe
// notes, and returns the ban, and whether it made it, as BanForLimit // mode would have done.
// does. A first ban lasts AttackBanDuration; once the netblock has had func (l *Ledger) WouldBan(
// one that was not lifted, the next is permanent. Its reason is "matched netblock netip.Prefix, now time.Time, grounds Grounds,
// the rule <RuleID>", or "asked for the trap path <TrapPath>".
func (l *Ledger) BanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) { ) (Ban, bool) {
return l.ban( return l.ban(netblock, now, grounds, false)
netblock,
now,
CauseAttack,
attackReason(notes),
notes,
true,
)
}
// WouldBanForAttack returns what BanForAttack would, without making the
// ban: what observe mode would have done.
func (l *Ledger) WouldBanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(
netblock,
now,
CauseAttack,
attackReason(notes),
notes,
false,
)
} }
// WouldBePermanent reports whether a ban on netblock for cause, CauseLimit // WouldBePermanent reports whether a ban on netblock for cause, CauseLimit
// or CauseAttack, made at now would be permanent, as BanForLimit or // or CauseAttack, made at now would be permanent, as Ban would make it. It
// BanForAttack would make it. It works out nothing else of the ban. // works out nothing else of the ban.
func (l *Ledger) WouldBePermanent( func (l *Ledger) WouldBePermanent(
netblock netip.Prefix, now time.Time, cause string, netblock netip.Prefix, now time.Time, cause string,
) bool { ) bool {
@@ -402,10 +380,9 @@ func attackReason(notes Notes) string {
// BanForAdmin bans netblock at now for an admin, with reason, until // BanForAdmin bans netblock at now for an admin, with reason, until
// expires, or for good when expires is zero, and returns the ban, whose // expires, or for good when expires is zero, and returns the ban, whose
// cause is CauseAdmin. Unlike BanForLimit and BanForAttack, it makes the // cause is CauseAdmin. Unlike Ban, it makes the ban even while another
// ban even while another on netblock is active, since the admin asked // on netblock is active, since the admin asked for this one. The ledger
// for this one. The ledger fills in the notes' EarlierBans, and counts // fills in the notes' EarlierBans, and counts the ban among those made.
// the ban among those made.
func (l *Ledger) BanForAdmin( func (l *Ledger) BanForAdmin(
netblock netip.Prefix, now, expires time.Time, reason string, netblock netip.Prefix, now, expires time.Time, reason string,
) Ban { ) Ban {
@@ -429,6 +406,88 @@ func (l *Ledger) BanForAdmin(
return ban return ban
} }
// BanForAbuseIPDB bans each netblock in scores at now until expires, and
// returns how many bans it made. scores are the netblocks of the clients
// on AbuseIPDB's blacklist, each with the highest score of its clients.
// A netblock is left as it is while a ban on it is active, or while a ban
// for the blacklist that an admin lifted has not reached its end: the
// admin's lift holds that long. Each ban whose cause is CauseAbuseIPDB
// and that has ended is forgotten first, so that the bans held for the
// blacklist are those of the clients listed within one ban's length.
func (l *Ledger) BanForAbuseIPDB(
now, expires time.Time, scores map[netip.Prefix]int64,
) int {
l.mu.Lock()
defer l.mu.Unlock()
changed := l.forgetEndedAbuseIPDBBans(now)
made := 0
for netblock, score := range scores {
netblock = netblock.Masked()
var notes Notes
if held, found := l.netblocks.Peek(netblock); found {
if activeBan(*held, now) != nil ||
liftedAbuseIPDBBan(*held, now) {
continue
}
notes.EarlierBans = earlierBans(*held)
}
// The source is named as the request log names AbuseIPDB.
notes.Reputation = []ReputationHit{
{Source: CauseAbuseIPDB, Score: &score},
}
l.add(Ban{
Netblock: netblock,
Start: now,
Expires: expires,
Cause: CauseAbuseIPDB,
Reason: fmt.Sprintf(
"on AbuseIPDB's blacklist with a score of %d", score,
),
Notes: notes,
})
made++
}
l.made[CauseAbuseIPDB] += made
if changed || made > 0 {
l.markChanged()
}
return made
}
// ForgetEndedAbuseIPDBBans forgets the bans whose cause is CauseAbuseIPDB
// and that have ended at now, as BanForAbuseIPDB does first. The state
// package calls it each time it reads bans.json, so that these bans are
// also forgotten while no fetch of the blacklist reaches the ledger.
func (l *Ledger) ForgetEndedAbuseIPDBBans(now time.Time) {
l.mu.Lock()
defer l.mu.Unlock()
if l.forgetEndedAbuseIPDBBans(now) {
l.markChanged()
}
}
// liftedAbuseIPDBBan reports whether bans, a netblock's, hold a ban whose
// cause is CauseAbuseIPDB that an admin lifted, and whose end is not
// reached at now.
func liftedAbuseIPDBBan(bans []Ban, now time.Time) bool {
return slices.ContainsFunc(bans, func(ban Ban) bool {
return ban.Cause == CauseAbuseIPDB && !ban.Lifted.IsZero() &&
(ban.Permanent() || now.Before(ban.Expires))
})
}
// Lift lifts, at now, every ban active then on a netblock client is in, // Lift lifts, at now, every ban active then on a netblock client is in,
// as an admin does, and returns those bans. A lifted ban is kept, refuses // as an admin does, and returns those bans. A lifted ban is kept, refuses
// nothing, and does not make the netblock's next ban longer. // nothing, and does not make the netblock's next ban longer.
@@ -515,9 +574,9 @@ func (l *Ledger) AddLookup(
} }
// Made returns how many bans for cause have been made since the start: // Made returns how many bans for cause have been made since the start:
// for CauseLimit and CauseAttack, by the ledger; for CauseAdmin, by an // for CauseLimit, CauseAttack and CauseAbuseIPDB, by the ledger; for
// admin, with BanForAdmin or in an edit of bans.json, as LoadEdit counts // CauseAdmin, by an admin, with BanForAdmin or in an edit of bans.json, as
// them. The bans read from bans.json at the start are not among them. // LoadEdit counts them. The bans read from bans.json at the start are not among them.
func (l *Ledger) Made(cause string) int { func (l *Ledger) Made(cause string) int {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
@@ -574,8 +633,8 @@ func (l *Ledger) Snapshot() []Ban {
// seen. A ban without a cause is an admin's, and gets CauseAdmin. Each // seen. A ban without a cause is an admin's, and gets CauseAdmin. Each
// netblock is masked to its length, so that 203.0.113.9/24 is // netblock is masked to its length, so that 203.0.113.9/24 is
// 203.0.113.0/24, and each text in the notes is cut to 256 bytes. Past // 203.0.113.0/24, and each text in the notes is cut to 256 bytes. Past
// MaxBans the earliest bans whose cause is not CauseAdmin are dropped, as // MaxBans the earliest bans that count toward it are dropped, as when they
// when they are made. // are made.
func (l *Ledger) Load(bans []Ban) { func (l *Ledger) Load(bans []Ban) {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
@@ -639,15 +698,42 @@ func (l *Ledger) holds(netblock netip.Prefix, start time.Time) bool {
}) })
} }
// ban bans netblock at now for cause, CauseLimit or CauseAttack, with // forgetEndedAbuseIPDBBans forgets each ban whose cause is CauseAbuseIPDB
// reason and notes, as BanForLimit and BanForAttack describe, and returns // and whose end is reached at now, lifted or not, and each netblock left
// the ban, and whether it made it. Unless keep is true, the ban is not // without a ban. It reports whether it forgot one.
// made, only returned: it is the ban that would have been made. func (l *Ledger) forgetEndedAbuseIPDBBans(now time.Time) bool {
forgot := false
for _, netblock := range l.netblocks.Keys() {
bans, _ := l.netblocks.Peek(netblock)
kept := slices.DeleteFunc(*bans, func(ban Ban) bool {
return ban.Cause == CauseAbuseIPDB && !ban.Permanent() &&
!now.Before(ban.Expires)
})
if len(kept) == len(*bans) {
continue
}
forgot = true
if len(kept) == 0 {
l.netblocks.Remove(netblock)
} else {
*bans = kept
}
}
return forgot
}
// ban bans netblock at now on grounds, as Ban describes, and returns the
// ban, and whether it made it. Unless keep is true, the ban is not made,
// only returned: it is the ban that would have been made.
func (l *Ledger) ban( func (l *Ledger) ban(
netblock netip.Prefix, netblock netip.Prefix,
now time.Time, now time.Time,
cause, reason string, grounds Grounds,
notes Notes,
keep bool, keep bool,
) (Ban, bool) { ) (Ban, bool) {
l.mu.Lock() l.mu.Lock()
@@ -656,6 +742,8 @@ func (l *Ledger) ban(
// held are the netblock's bans, none of them active. // held are the netblock's bans, none of them active.
var held []Ban var held []Ban
notes := grounds.Notes
bans, found := l.netblocks.Get(netblock) bans, found := l.netblocks.Get(netblock)
if found { if found {
active := activeBan(*bans, now) active := activeBan(*bans, now)
@@ -671,15 +759,19 @@ func (l *Ledger) ban(
ban := Ban{ ban := Ban{
Netblock: netblock, Netblock: netblock,
Start: now, Start: now,
Cause: cause, Cause: grounds.Cause,
Reason: reason,
Notes: notes, Notes: notes,
} }
if cause == CauseAttack { switch grounds.Cause {
ban.Expires = l.attackExpiry(held, now) case CauseLimit:
} else { ban.Reason = limitReason(notes)
ban.Expires = l.limitExpiry(held, now) ban.Expires = l.limitExpiry(held, now)
case CauseAttack:
ban.Reason = attackReason(notes)
ban.Expires = l.attackExpiry(held, now)
default:
panic("bans: a request's ban has the cause " + grounds.Cause)
} }
if !keep { if !keep {
@@ -687,7 +779,7 @@ func (l *Ledger) ban(
} }
l.add(ban) l.add(ban)
l.made[cause]++ l.made[grounds.Cause]++
l.markChanged() l.markChanged()
return ban, true return ban, true
@@ -714,13 +806,16 @@ func earlierBans(held []Ban) EarlierBans {
fromHere.Attack++ fromHere.Attack++
case CauseAdmin: case CauseAdmin:
fromHere.Admin++ fromHere.Admin++
case CauseAbuseIPDB:
fromHere.AbuseIPDB++
} }
notes := ban.Notes.EarlierBans notes := ban.Notes.EarlierBans
count := EarlierBans{ count := EarlierBans{
Limit: notes.Limit + fromHere.Limit, Limit: notes.Limit + fromHere.Limit,
Attack: notes.Attack + fromHere.Attack, Attack: notes.Attack + fromHere.Attack,
Admin: notes.Admin + fromHere.Admin, Admin: notes.Admin + fromHere.Admin,
AbuseIPDB: notes.AbuseIPDB + fromHere.AbuseIPDB,
} }
// Of counts that name as many bans, the earlier ban's is taken: it // Of counts that name as many bans, the earlier ban's is taken: it
@@ -735,7 +830,7 @@ func earlierBans(held []Ban) EarlierBans {
// total returns how many bans e counts, of all causes. // total returns how many bans e counts, of all causes.
func (e EarlierBans) total() int { func (e EarlierBans) total() int {
return e.Limit + e.Attack + e.Admin return e.Limit + e.Attack + e.Admin + e.AbuseIPDB
} }
// markChanged has Changed receive a value, unless one is waiting already. // markChanged has Changed receive a value, unless one is waiting already.
@@ -806,9 +901,9 @@ func (l *Ledger) covering(client netip.Addr) []*[]Ban {
// add adds ban to its netblock's bans, after the last, and makes its // add adds ban to its netblock's bans, after the last, and makes its
// netblock the most recently seen. With MaxBans held, it drops one first, // netblock the most recently seen. With MaxBans held, it drops one first,
// unless ban's cause is CauseAdmin, which does not count toward MaxBans. // unless ban does not count toward MaxBans.
func (l *Ledger) add(ban Ban) { func (l *Ledger) add(ban Ban) {
counted := ban.Cause != CauseAdmin counted := countsTowardMaxBans(ban)
if counted && l.held == l.params.MaxBans { if counted && l.held == l.params.MaxBans {
l.dropOne() l.dropOne()
} }
@@ -889,7 +984,13 @@ func (l *Ledger) attackExpiry(held []Ban, now time.Time) time.Time {
return now.Add(l.params.AttackBanDuration) return now.Add(l.params.AttackBanDuration)
} }
// dropOne drops the earliest ban whose cause is not CauseAdmin of the // countsTowardMaxBans reports whether ban counts toward MaxBans: its cause
// is CauseLimit or CauseAttack.
func countsTowardMaxBans(ban Ban) bool {
return ban.Cause == CauseLimit || ban.Cause == CauseAttack
}
// dropOne drops the earliest ban that counts toward MaxBans of the
// netblock that has gone longest without a request, of those that hold // netblock that has gone longest without a request, of those that hold
// such a ban, and the netblock with it if that was its only ban. It is // such a ban, and the netblock with it if that was its only ban. It is
// called with at least one such ban held. It looks at each netblock once // called with at least one such ban held. It looks at each netblock once
@@ -898,11 +999,9 @@ func (l *Ledger) dropOne() {
for range l.netblocks.Len() { for range l.netblocks.Len() {
netblock, bans, _ := l.netblocks.GetOldest() netblock, bans, _ := l.netblocks.GetOldest()
i := slices.IndexFunc(*bans, func(ban Ban) bool { i := slices.IndexFunc(*bans, countsTowardMaxBans)
return ban.Cause != CauseAdmin
})
if i < 0 { if i < 0 {
// Its bans are all an admin's, and never dropped. Get makes // None of its bans is dropped to make room. Get makes
// it the most recently seen, so that the next netblock is // it the most recently seen, so that the next netblock is
// looked at; when it was seen matters only for dropping a // looked at; when it was seen matters only for dropping a
// ban, and a ban added to it makes it the most recently seen // ban, and a ban added to it makes it the most recently seen
+121 -51
View File
@@ -22,7 +22,11 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
// Each ban is followed by another as soon as it ends: 1, 3, 9, 27 and // Each ban is followed by another as soon as it ends: 1, 3, 9, 27 and
// 81 hours. // 81 hours.
for i, hours := range []int{1, 3, 9, 27, 81} { for i, hours := range []int{1, 3, 9, 27, 81} {
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
length := time.Duration(hours) * time.Hour length := time.Duration(hours) * time.Hour
if !ban.Expires.Equal(now.Add(length)) || if !ban.Expires.Equal(now.Add(length)) ||
@@ -42,7 +46,7 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
// The sixth would last 243 hours, more than seven days: it is // The sixth would last 243 hours, more than seven days: it is
// permanent, and never ends. // permanent, and never ends.
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
if !ban.Permanent() { if !ban.Permanent() {
t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires) t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires)
} }
@@ -71,15 +75,15 @@ func TestRepeatWindowRunsOut(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.BanForLimit( first, _ := ledger.Ban(
netblock, netblock,
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
second, _ := ledger.BanForLimit( second, _ := ledger.Ban(
netblock, netblock,
first.Expires.Add(tc.gap), first.Expires.Add(tc.gap),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
if second.Expires.Sub(second.Start) != tc.want || if second.Expires.Sub(second.Start) != tc.want ||
@@ -104,10 +108,10 @@ func TestFirstBanLongerThanLimitBanPermanentAboveIsPermanent(t *testing.T) {
params.LimitBanDuration = params.LimitBanPermanentAbove + time.Hour params.LimitBanDuration = params.LimitBanPermanentAbove + time.Hour
ledger := bans.New(params) ledger := bans.New(params)
ban, _ := ledger.BanForLimit( ban, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
if !ban.Permanent() { if !ban.Permanent() {
t.Errorf("first ban ends at %s, want a permanent one", ban.Expires) t.Errorf("first ban ends at %s, want a permanent one", ban.Expires)
@@ -127,7 +131,11 @@ func TestLimitBanPermanentAboveSetFarOffDoesNotOverflow(t *testing.T) {
now := midnight() now := midnight()
for i := range 14 { for i := range 14 {
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
if !ban.Expires.After(ban.Start) { if !ban.Expires.After(ban.Start) {
t.Fatalf( t.Fatalf(
"ban %d starts at %s and ends at %s", "ban %d starts at %s and ends at %s",
@@ -140,7 +148,7 @@ func TestLimitBanPermanentAboveSetFarOffDoesNotOverflow(t *testing.T) {
now = ban.Expires now = ban.Expires
} }
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
if !ban.Permanent() { if !ban.Permanent() {
t.Errorf("15th ban ends at %s, want a permanent one", ban.Expires) t.Errorf("15th ban ends at %s, want a permanent one", ban.Expires)
} }
@@ -152,15 +160,19 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
first, made := ledger.BanForLimit(netblock, midnight(), bans.Notes{}) first, made := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
if !made { if !made {
t.Error("the first ban was not made") t.Error("the first ban was not made")
} }
again, made := ledger.BanForLimit( again, made := ledger.Ban(
netblock, netblock,
midnight().Add(time.Minute), midnight().Add(time.Minute),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
if made || !reflect.DeepEqual(again, first) || if made || !reflect.DeepEqual(again, first) ||
@@ -175,10 +187,10 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
) )
} }
again, made = ledger.BanForAttack( again, made = ledger.Ban(
netblock, netblock,
midnight().Add(time.Minute), midnight().Add(time.Minute),
bans.Notes{}, bans.Grounds{Cause: bans.CauseAttack},
) )
if made || !reflect.DeepEqual(again, first) { if made || !reflect.DeepEqual(again, first) {
t.Errorf( t.Errorf(
@@ -195,10 +207,10 @@ func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := ledger.BanForLimit( ban, _ := ledger.Ban(
netblock, netblock,
midnight(), midnight(),
bans.Notes{Requests: 5}, bans.Grounds{Cause: bans.CauseLimit, Notes: bans.Notes{Requests: 5}},
) )
for range 3 { for range 3 {
@@ -237,10 +249,10 @@ func TestFindCountsNothing(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := ledger.BanForLimit( ban, _ := ledger.Ban(
netblock, netblock,
midnight(), midnight(),
bans.Notes{Requests: 5}, bans.Grounds{Cause: bans.CauseLimit, Notes: bans.Notes{Requests: 5}},
) )
got, banned, _ := ledger.Find( got, banned, _ := ledger.Find(
@@ -287,24 +299,24 @@ func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(
d := netip.MustParsePrefix("2001:db8::/64") d := netip.MustParsePrefix("2001:db8::/64")
now := midnight() now := midnight()
first, _ := ledger.BanForLimit(a, now, bans.Notes{}) first, _ := ledger.Ban(a, now, bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(b, now, bans.Notes{}) ledger.Ban(b, now, bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(c, now, bans.Notes{}) ledger.Ban(c, now, bans.Grounds{Cause: bans.CauseLimit})
// A request from a makes b the netblock seen longest ago, and its ban // A request from a makes b the netblock seen longest ago, and its ban
// goes to make room for d's. // goes to make room for d's.
ledger.Check(a.Addr(), now) ledger.Check(a.Addr(), now)
ledger.BanForLimit(d, now, bans.Notes{}) ledger.Ban(d, now, bans.Grounds{Cause: bans.CauseLimit})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1}) wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1})
// a is banned again once its ban has ended; c, seen longest ago, goes. // a is banned again once its ban has ended; c, seen longest ago, goes.
ledger.BanForLimit(a, first.Expires, bans.Notes{}) ledger.Ban(a, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
wantBans(t, ledger, map[netip.Prefix]int{a: 2, c: 0, d: 1}) wantBans(t, ledger, map[netip.Prefix]int{a: 2, c: 0, d: 1})
// With d seen since, a is seen longest ago, and its earlier ban goes // With d seen since, a is seen longest ago, and its earlier ban goes
// first. // first.
ledger.Check(d.Addr(), first.Expires) ledger.Check(d.Addr(), first.Expires)
ledger.BanForLimit(b, first.Expires, bans.Notes{}) ledger.Ban(b, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1}) wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1})
if !ledger.Bans(a)[0].Start.Equal(first.Expires) { if !ledger.Bans(a)[0].Start.Equal(first.Expires) {
@@ -325,8 +337,16 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
ledger := bans.New(params) ledger := bans.New(params)
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{}) first, _ := ledger.Ban(
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{}) netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
held := ledger.Bans(netblock) held := ledger.Bans(netblock)
if len(held) != 1 || !reflect.DeepEqual(held[0], second) || if len(held) != 1 || !reflect.DeepEqual(held[0], second) ||
@@ -343,7 +363,11 @@ func TestRequestDuringAnAttackBanMakesItPermanent(t *testing.T) {
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
notes := bans.Notes{RuleID: "env-file", Target: "path"} notes := bans.Notes{RuleID: "env-file", Target: "path"}
ban, _ := ledger.BanForAttack(netblock, midnight(), notes) ban, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
)
if !ban.Expires.Equal(midnight().Add(7*day)) || if !ban.Expires.Equal(midnight().Add(7*day)) ||
ban.Cause != bans.CauseAttack || ban.Cause != bans.CauseAttack ||
ban.Notes.RuleID != "env-file" || ban.Notes.RuleID != "env-file" ||
@@ -418,8 +442,16 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
// A ban for a broken limit before does not count. // A ban for a broken limit before does not count.
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{}) first, _ := ledger.Ban(
second, _ := ledger.BanForAttack(netblock, first.Expires, bans.Notes{}) netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseAttack},
)
if second.Expires.Sub(second.Start) != 7*day { if second.Expires.Sub(second.Start) != 7*day {
t.Fatalf("the first ban for an attack lasts %s, want 7 days", t.Fatalf("the first ban for an attack lasts %s, want 7 days",
@@ -435,10 +467,10 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
// Its notes show the earlier ban for an attack that makes it permanent, // Its notes show the earlier ban for an attack that makes it permanent,
// beside the one for a limit. // beside the one for a limit.
third, _ := ledger.BanForAttack( third, _ := ledger.Ban(
netblock, netblock,
second.Expires.Add(30*day), second.Expires.Add(30*day),
bans.Notes{}, bans.Grounds{Cause: bans.CauseAttack},
) )
if !third.Permanent() || if !third.Permanent() ||
third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Attack: 1}) { third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Attack: 1}) {
@@ -452,14 +484,18 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{}) first, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
wantChanged(t, ledger, true) wantChanged(t, ledger, true)
// While the first ban lasts, none would be made. // While the first ban lasts, none would be made.
during, would := ledger.WouldBanForAttack( during, would := ledger.WouldBan(
netblock, netblock,
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseAttack},
) )
if would || !reflect.DeepEqual(during, first) { if would || !reflect.DeepEqual(during, first) {
t.Errorf( t.Errorf(
@@ -472,13 +508,19 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
// As it ends, a clear sign of attack would ban for seven days, and a // As it ends, a clear sign of attack would ban for seven days, and a
// limit broken again for three hours, but neither is made. // limit broken again for three hours, but neither is made.
limitNotes := bans.Notes{Kind: "requests", Limit: 1, Window: "minute"} limitGrounds := bans.Grounds{
attack, wouldAttack := ledger.WouldBanForAttack(netblock, first.Expires, Cause: bans.CauseLimit,
bans.Notes{RuleID: "git-dir"}) Notes: bans.Notes{Kind: "requests", Limit: 1, Window: "minute"},
limit, wouldLimit := ledger.WouldBanForLimit( }
attack, wouldAttack := ledger.WouldBan(netblock, first.Expires,
bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "git-dir"},
})
limit, wouldLimit := ledger.WouldBan(
netblock, netblock,
first.Expires, first.Expires,
limitNotes, limitGrounds,
) )
if !wouldAttack || !attack.Expires.Equal(first.Expires.Add(7*day)) || if !wouldAttack || !attack.Expires.Equal(first.Expires.Add(7*day)) ||
@@ -505,7 +547,7 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
wantChanged(t, ledger, false) wantChanged(t, ledger, false)
// The ban made is the one that would have been. // The ban made is the one that would have been.
made, _ := ledger.BanForLimit(netblock, first.Expires, limitNotes) made, _ := ledger.Ban(netblock, first.Expires, limitGrounds)
if !reflect.DeepEqual(made, limit) { if !reflect.DeepEqual(made, limit) {
t.Errorf("the ban made is %+v, want %+v", made, limit) t.Errorf("the ban made is %+v, want %+v", made, limit)
} }
@@ -526,7 +568,11 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
t.Fatalf("ban %d for a limit would be permanent", i+1) t.Fatalf("ban %d for a limit would be permanent", i+1)
} }
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
now = ban.Expires now = ban.Expires
} }
@@ -539,7 +585,11 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
} }
// Once a first ban for an attack has ended, the next would be permanent. // Once a first ban for an attack has ended, the next would be permanent.
attack, _ := ledger.BanForAttack(netblock, now, bans.Notes{}) attack, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseAttack},
)
if !ledger.WouldBePermanent(netblock, attack.Expires, bans.CauseAttack) { if !ledger.WouldBePermanent(netblock, attack.Expires, bans.CauseAttack) {
t.Error("a second ban for an attack would not be permanent") t.Error("a second ban for an attack would not be permanent")
} }
@@ -553,8 +603,16 @@ func TestAttackBanDoesNotLengthenTheNextBanForALimit(t *testing.T) {
// Three times the seven days would be permanent; a limit broken as the // Three times the seven days would be permanent; a limit broken as the
// ban for an attack ends bans for an hour, as a first broken limit does. // ban for an attack ends bans for an hour, as a first broken limit does.
attack, _ := ledger.BanForAttack(netblock, midnight(), bans.Notes{}) attack, _ := ledger.Ban(
limit, _ := ledger.BanForLimit(netblock, attack.Expires, bans.Notes{}) netblock,
midnight(),
bans.Grounds{Cause: bans.CauseAttack},
)
limit, _ := ledger.Ban(
netblock,
attack.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if limit.Expires.Sub(limit.Start) != time.Hour || if limit.Expires.Sub(limit.Start) != time.Hour ||
limit.Cause != bans.CauseLimit { limit.Cause != bans.CauseLimit {
@@ -578,10 +636,13 @@ func TestRequestTextsAreCutTo256Bytes(t *testing.T) {
Time: midnight(), Method: long, Host: long, Path: long, Status: 403, UserAgent: long, Time: midnight(), Method: long, Host: long, Path: long, Status: 403, UserAgent: long,
} }
ban, _ := ledger.BanForLimit( ban, _ := ledger.Ban(
netblock, netblock,
midnight(), midnight(),
bans.Notes{Request: request}, bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{Request: request},
},
) )
cut := long[:256] cut := long[:256]
@@ -607,11 +668,20 @@ func TestLookupFillsTheNotesOfTheNetblocksBansWithoutOne(t *testing.T) {
// A ban made with the client's lookup, one made before it came, after // A ban made with the client's lookup, one made before it came, after
// the first ended, and one on another netblock. // the first ended, and one on another netblock.
ledger.BanForLimit(netblock, midnight(), bans.Notes{ ledger.Ban(netblock, midnight(), bans.Grounds{
ASN: "AS64497", ASName: "Other Net", Country: "FR", Cause: bans.CauseLimit,
Notes: bans.Notes{
ASN: "AS64497",
ASName: "Other Net",
Country: "FR",
},
}) })
ledger.BanForLimit(netblock, midnight().Add(time.Hour), bans.Notes{}) ledger.Ban(
ledger.BanForLimit(other, midnight(), bans.Notes{}) netblock,
midnight().Add(time.Hour),
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.Ban(other, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.AddLookup(netblock, "AS64496", "Example Net", "DE") ledger.AddLookup(netblock, "AS64496", "Example Net", "DE")
+36 -21
View File
@@ -19,25 +19,29 @@ func TestChangedAfterABanIsMade(t *testing.T) {
wantChanged(t, ledger, false) wantChanged(t, ledger, false)
ledger.BanForLimit(netblock, midnight(), bans.Notes{}) ledger.Ban(netblock, midnight(), bans.Grounds{Cause: bans.CauseLimit})
wantChanged(t, ledger, true) wantChanged(t, ledger, true)
// A limit broken during the ban makes no other, and a refusal changes // A limit broken during the ban makes no other, and a refusal changes
// only the counts in the notes, which wait for the interval's write. // only the counts in the notes, which wait for the interval's write.
ledger.BanForLimit(netblock, midnight().Add(time.Minute), bans.Notes{}) ledger.Ban(
netblock,
midnight().Add(time.Minute),
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.Check(netblock.Addr(), midnight().Add(time.Minute)) ledger.Check(netblock.Addr(), midnight().Add(time.Minute))
wantChanged(t, ledger, false) wantChanged(t, ledger, false)
// Two bans before the value is read leave one. // Two bans before the value is read leave one.
ledger.BanForLimit( ledger.Ban(
netip.MustParsePrefix("203.0.113.10/32"), netip.MustParsePrefix("203.0.113.10/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
ledger.BanForLimit( ledger.Ban(
netip.MustParsePrefix("203.0.113.11/32"), netip.MustParsePrefix("203.0.113.11/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
wantChanged(t, ledger, true) wantChanged(t, ledger, true)
wantChanged(t, ledger, false) wantChanged(t, ledger, false)
@@ -51,10 +55,14 @@ func TestSnapshotListsEveryBanByNetblock(t *testing.T) {
high := netip.MustParsePrefix("203.0.113.10/32") high := netip.MustParsePrefix("203.0.113.10/32")
low := netip.MustParsePrefix("203.0.113.9/32") low := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.BanForLimit(v6, midnight(), bans.Notes{}) first, _ := ledger.Ban(
ledger.BanForLimit(high, midnight(), bans.Notes{}) v6,
ledger.BanForLimit(low, midnight(), bans.Notes{}) midnight(),
ledger.BanForLimit(v6, first.Expires, bans.Notes{}) bans.Grounds{Cause: bans.CauseLimit},
)
ledger.Ban(high, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(low, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(v6, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
snapshot := ledger.Snapshot() snapshot := ledger.Snapshot()
@@ -80,7 +88,10 @@ func TestLoadedBansCarryOn(t *testing.T) {
before := bans.New(defaultParams()) before := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := before.BanForLimit(netblock, midnight(), bans.Notes{Limit: 1}) ban, _ := before.Ban(netblock, midnight(), bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{Limit: 1},
})
// Loaded into a new ledger, as across a restart, the ban still refuses // Loaded into a new ledger, as across a restart, the ban still refuses
// while it lasts, and once it has ended a broken limit bans for three // while it lasts, and once it has ended a broken limit bans for three
@@ -96,7 +107,11 @@ func TestLoadedBansCarryOn(t *testing.T) {
t.Error("the loaded ban does not refuse") t.Error("the loaded ban does not refuse")
} }
again, _ := after.BanForLimit(netblock, ban.Expires, bans.Notes{}) again, _ := after.Ban(
netblock,
ban.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if again.Expires.Sub(again.Start) != 3*time.Hour || if again.Expires.Sub(again.Start) != 3*time.Hour ||
again.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) { again.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
t.Errorf( t.Errorf(
@@ -124,10 +139,10 @@ func TestLoadedBanRefusesEveryClientInItsNetblock(t *testing.T) {
Start: midnight(), Start: midnight(),
}, },
}) })
ledger.BanForLimit( ledger.Ban(
netip.MustParsePrefix("198.51.100.7/32"), netip.MustParsePrefix("198.51.100.7/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
for client, want := range map[string]bool{ for client, want := range map[string]bool{
@@ -200,7 +215,7 @@ func TestPermanentBanStartedBeforeAnEndedOneRefuses(t *testing.T) {
} }
// A limit broken now makes no shorter ban over the permanent one. // A limit broken now makes no shorter ban over the permanent one.
ban, _ = ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ = ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
if !ban.Permanent() || len(ledger.Bans(netblock)) != 2 { if !ban.Permanent() || len(ledger.Bans(netblock)) != 2 {
t.Errorf( t.Errorf(
"a broken limit returned %+v and left the netblock %d bans, "+ "a broken limit returned %+v and left the netblock %d bans, "+
@@ -236,10 +251,10 @@ func TestNextBanWorkedOutFromTheBanThatEndedLast(t *testing.T) {
// Once both have ended, a limit broken within the repeat window bans // Once both have ended, a limit broken within the repeat window bans
// for three times the 9 hours, and the notes count the two bans // for three times the 9 hours, and the notes count the two bans
// before the 9-hour one and it, for a limit, and the admin's. // before the 9-hour one and it, for a limit, and the admin's.
ban, _ := ledger.BanForLimit( ban, _ := ledger.Ban(
netblock, netblock,
nineHours.Expires.Add(time.Hour), nineHours.Expires.Add(time.Hour),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
if ban.Expires.Sub(ban.Start) != 27*time.Hour || if ban.Expires.Sub(ban.Start) != 27*time.Hour ||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 3, Admin: 1}) { ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 3, Admin: 1}) {
@@ -313,15 +328,15 @@ func TestLoadReplacesTheBansHeld(t *testing.T) {
} }
// The ledger holds one ban, so it makes two more without dropping any. // The ledger holds one ban, so it makes two more without dropping any.
first, _ := ledger.BanForLimit( first, _ := ledger.Ban(
netip.MustParsePrefix("198.51.100.7/32"), netip.MustParsePrefix("198.51.100.7/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
second, _ := ledger.BanForLimit( second, _ := ledger.Ban(
netip.MustParsePrefix("198.51.100.8/32"), netip.MustParsePrefix("198.51.100.8/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
want := []bans.Ban{first, second, kept} want := []bans.Ban{first, second, kept}
+4 -25
View File
@@ -19,6 +19,8 @@ import (
"time" "time"
"github.com/spf13/viper" "github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/reputation"
) )
// off is the value that switches a timeout, a size limit or a rate limit // off is the value that switches a timeout, a size limit or a rate limit
@@ -40,9 +42,7 @@ var (
errNotPositive = errors.New("must be more than zero, or off") errNotPositive = errors.New("must be more than zero, or off")
errNotMilliseconds = errors.New( errNotMilliseconds = errors.New(
"is not a whole number of milliseconds above zero, such as 2000") "is not a whole number of milliseconds above zero, such as 2000")
errEmptyItem = errors.New("has an empty item in its list") errEmptyItem = errors.New("has an empty item in its list")
errNotNetblock = errors.New(
"is not a netblock such as 10.0.0.0/8, or an address")
errNotListenAddr = errors.New( errNotListenAddr = errors.New(
"is not an address to listen on, such as :8080") "is not an address to listen on, such as :8080")
errNotCommand = errors.New("is not rproxy or waf") errNotCommand = errors.New("is not rproxy or waf")
@@ -253,7 +253,7 @@ func parseNetblocks(value string) ([]netip.Prefix, error) {
netblocks := make([]netip.Prefix, 0, len(items)) netblocks := make([]netip.Prefix, 0, len(items))
for _, item := range items { for _, item := range items {
netblock, err := ParseNetblock(item) netblock, err := reputation.ParseNetblock(item)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -264,27 +264,6 @@ func parseNetblocks(value string) ([]netip.Prefix, error) {
return netblocks, nil return netblocks, nil
} }
// ParseNetblock reads a netblock in CIDR form, such as 10.0.0.0/8. A bare
// address is a netblock of that address alone, a /32 or a /128. A
// blocklist's lines are read with it too.
func ParseNetblock(value string) (netip.Prefix, error) {
if strings.Contains(value, "/") {
netblock, err := netip.ParsePrefix(value)
if err != nil {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netblock.Masked(), nil
}
addr, err := netip.ParseAddr(value)
if err != nil || addr.Zone() != "" {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netip.PrefixFrom(addr, addr.BitLen()), nil
}
// parseListenAddr checks an address to listen on: an optional host and a // parseListenAddr checks an address to listen on: an optional host and a
// port number. // port number.
func parseListenAddr(value string) (string, error) { func parseListenAddr(value string) (string, error) {
+229 -151
View File
@@ -20,11 +20,26 @@ import (
"unicode/utf8" "unicode/utf8"
"github.com/spf13/viper" "github.com/spf13/viper"
"golang.org/x/net/http/httpguts"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/coreruleset" "sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/remotelog" "sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// The names of the settings that waf writes into log lines and ban notes,
// as the setting that refused a request or lowered a client's limits.
const (
ExclusivelyAllowedCountriesSetting = "SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
TagLimitPercentSetting = "SWW_WAF_TAG_LIMIT_PERCENT"
ASNBytesLimitPercentSetting = "SWW_WAF_ASN_BYTES_LIMIT_PERCENT"
CountryBytesLimitPercentSetting = "SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT"
UnknownCountryLimitPercentSetting = "SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT"
ASNLimitPercentURLSetting = "SWW_WAF_ASN_LIMIT_PERCENT_URL"
BlocklistActionSetting = "SWW_WAF_BLOCKLIST_ACTION"
DNSBLAbuseIPDBActionSetting = "SWW_WAF_DNSBL_ABUSEIPDB_ACTION"
) )
// WAF is the settings of `sww waf`. A time or size limit of zero is off. // WAF is the settings of `sww waf`. A time or size limit of zero is off.
@@ -75,7 +90,7 @@ type WAF struct {
BytesLimitPerMinute int64 BytesLimitPerMinute int64
BytesLimitPerHour int64 BytesLimitPerHour int64
BytesLimitPerDay int64 BytesLimitPerDay int64
BytesCounted string BytesCounted BytesCounted
// LookupSource is where each client's AS number and country are looked // LookupSource is where each client's AS number and country are looked
// up (SWW_WAF_LOOKUP_SOURCE): geojs, file, or off for nowhere. // up (SWW_WAF_LOOKUP_SOURCE): geojs, file, or off for nowhere.
// LookupDBPath is the lookup database, the IPinfo Lite file looked up in // LookupDBPath is the lookup database, the IPinfo Lite file looked up in
@@ -86,7 +101,7 @@ type WAF struct {
// AddLookupHeaders is true when the app is passed the client's AS number // AddLookupHeaders is true when the app is passed the client's AS number
// and country in headers (SWW_WAF_ADD_LOOKUP_HEADERS), by default unless // and country in headers (SWW_WAF_ADD_LOOKUP_HEADERS), by default unless
// LookupSource is off. // LookupSource is off.
LookupSource string LookupSource LookupSource
LookupDBPath string LookupDBPath string
ASNAndCountryLookupTimeout time.Duration ASNAndCountryLookupTimeout time.Duration
AddLookupHeaders bool AddLookupHeaders bool
@@ -100,7 +115,7 @@ type WAF struct {
// with a tag that percentage of every rate limit and byte limit, from 1 // with a tag that percentage of every rate limit and byte limit, from 1
// to 99 (SWW_WAF_TAG_LIMIT_PERCENT). All hold tags as requests carry them: in // to 99 (SWW_WAF_TAG_LIMIT_PERCENT). All hold tags as requests carry them: in
// lower case, a country code in capitals. // lower case, a country code in capitals.
TagLists []TagList TagLists []reputation.TagList
TagDeny []string TagDeny []string
TagLimitPercent map[string]int64 TagLimitPercent map[string]int64
// RouteWatchURL is where the routewatch instance is that adds the asn= // RouteWatchURL is where the routewatch instance is that adds the asn=
@@ -131,7 +146,7 @@ type WAF struct {
// BlocklistLimitPercent is the percentage of every limit it gets. // BlocklistLimitPercent is the percentage of every limit it gets.
BlocklistURLs []string BlocklistURLs []string
BlocklistRefresh time.Duration BlocklistRefresh time.Duration
BlocklistAction string BlocklistAction ReputationAction
BlocklistLimitPercent int64 BlocklistLimitPercent int64
// DNSBLZones are the DNSBL zones clients are asked about // DNSBLZones are the DNSBL zones clients are asked about
// (SWW_WAF_DNSBL_ZONES), through DNSBLResolver (SWW_WAF_DNSBL_RESOLVER), or // (SWW_WAF_DNSBL_ZONES), through DNSBLResolver (SWW_WAF_DNSBL_RESOLVER), or
@@ -154,10 +169,18 @@ type WAF struct {
AbuseIPDBKey string AbuseIPDBKey string
AbuseIPDBMinScore int64 AbuseIPDBMinScore int64
AbuseIPDBDailyBudget int AbuseIPDBDailyBudget int
DNSBLAbuseIPDBAction string DNSBLAbuseIPDBAction ReputationAction
DNSBLAbuseIPDBLimitPercent int64 DNSBLAbuseIPDBLimitPercent int64
IPReputationCacheTTL time.Duration IPReputationCacheTTL time.Duration
DNSBLAbuseIPDBTimeout time.Duration DNSBLAbuseIPDBTimeout time.Duration
// While AbuseIPDBKey is set, AbuseIPDB's blacklist of the clients it
// scores AbuseIPDBThreshold or more (SWW_WAF_ABUSEIPDB_THRESHOLD), from
// 25 to 100, is fetched at the start and every AbuseIPDBBlacklistRefresh
// (SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS). Each client on it
// is banned for AbuseIPDBBanDuration (SWW_WAF_ABUSEIPDB_BAN_DAYS).
AbuseIPDBThreshold int64
AbuseIPDBBlacklistRefresh time.Duration
AbuseIPDBBanDuration time.Duration
// BanResponse is the status a refused client is answered with, 403 // BanResponse is the status a refused client is answered with, 403
// or 429, or 0 to close the connection without an answer // or 429, or 0 to close the connection without an answer
// (SWW_WAF_BAN_RESPONSE). It answers a banned client, a request that // (SWW_WAF_BAN_RESPONSE). It answers a banned client, a request that
@@ -229,7 +252,7 @@ type WAF struct {
// reads none. // reads none.
// CoreRuleSetExclusionsFile is the path of the file of its exclusions // CoreRuleSetExclusionsFile is the path of the file of its exclusions
// (SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE), "" while there is none. // (SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE), "" while there is none.
CoreRuleSetMode string CoreRuleSetMode CoreRuleSetMode
CoreRuleSetParanoiaLevel int CoreRuleSetParanoiaLevel int
CoreRuleSetScoreThreshold int CoreRuleSetScoreThreshold int
CoreRuleSetDisabledRules []int CoreRuleSetDisabledRules []int
@@ -308,20 +331,66 @@ type WAF struct {
settings []slog.Attr settings []slog.Attr
} }
// BytesCounted is which body bytes count toward the byte limits and the
// byte anomaly thresholds, a value of SWW_WAF_BYTES_COUNTED.
type BytesCounted string
// The values of SWW_WAF_BYTES_COUNTED.
const (
// BytesCountedResponse counts the response's body bytes.
BytesCountedResponse BytesCounted = "response"
// BytesCountedRequest counts the request's body bytes.
BytesCountedRequest BytesCounted = "request"
// BytesCountedBoth counts the body bytes of both.
BytesCountedBoth BytesCounted = "both"
)
// LookupSource is where each client's AS number and country are looked up,
// a value of SWW_WAF_LOOKUP_SOURCE.
type LookupSource string
// The values of SWW_WAF_LOOKUP_SOURCE.
const (
// LookupSourceGeoJS asks GeoJS.
LookupSourceGeoJS LookupSource = "geojs"
// LookupSourceFile looks clients up in the lookup database, the file
// SWW_WAF_LOOKUP_DB_PATH names.
LookupSourceFile LookupSource = "file"
// LookupSourceOff looks no client up.
LookupSourceOff LookupSource = "off"
)
// ReputationAction is what is done with a client a reputation source
// lists, a value of SWW_WAF_BLOCKLIST_ACTION and of
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION.
type ReputationAction string
// The values of SWW_WAF_BLOCKLIST_ACTION and SWW_WAF_DNSBL_ABUSEIPDB_ACTION.
const (
// ReputationActionDeny refuses the client's requests.
ReputationActionDeny ReputationAction = "deny"
// ReputationActionLimit gives the client a percentage of every limit,
// written limit:<percent> in the setting.
ReputationActionLimit ReputationAction = "limit"
// ReputationActionLog only notes the hit in the log line and raises an
// alert.
ReputationActionLog ReputationAction = "log"
)
// CoreRuleSetMode is what the Core Rule Set does, a value of
// SWW_WAF_CORE_RULE_SET_MODE.
type CoreRuleSetMode string
// The values of SWW_WAF_CORE_RULE_SET_MODE. // The values of SWW_WAF_CORE_RULE_SET_MODE.
const ( const (
// CoreRuleSetModeOff runs no request through the Core Rule Set. // CoreRuleSetModeOff runs no request through the Core Rule Set.
CoreRuleSetModeOff = off CoreRuleSetModeOff CoreRuleSetMode = "off"
// CoreRuleSetModeDetect logs and alerts a match, and refuses nothing. // CoreRuleSetModeDetect logs and alerts a match, and refuses nothing.
CoreRuleSetModeDetect = "detect" CoreRuleSetModeDetect CoreRuleSetMode = "detect"
// CoreRuleSetModeBlock refuses a match with 403. // CoreRuleSetModeBlock refuses a match with 403.
CoreRuleSetModeBlock = "block" CoreRuleSetModeBlock CoreRuleSetMode = "block"
) )
// fileSource is the SWW_WAF_LOOKUP_SOURCE that looks clients up in the
// lookup database, the file SWW_WAF_LOOKUP_DB_PATH names.
const fileSource = "file"
const ( const (
day = 24 * time.Hour day = 24 * time.Hour
ipv4Bits = 32 ipv4Bits = 32
@@ -330,6 +399,9 @@ const (
// netblock a provider is usually given: a shorter one would make one // netblock a provider is usually given: a shorter one would make one
// client of the customers of several providers. // client of the customers of several providers.
minIPv6GroupPrefix = 32 minIPv6GroupPrefix = 32
// minAbuseIPDBThreshold is the lowest score AbuseIPDB's blacklist can
// be cut at.
minAbuseIPDBThreshold = 25
// minTokenLength is the fewest characters a token may have. // minTokenLength is the fewest characters a token may have.
minTokenLength = 32 minTokenLength = 32
// maxParanoiaLevel is the Core Rule Set's highest paranoia level. // maxParanoiaLevel is the Core Rule Set's highest paranoia level.
@@ -440,11 +512,14 @@ var (
"is not a whole number above zero, such as 60, or off", "is not a whole number above zero, such as 60, or off",
) )
errNotUTF8 = errors.New("is not valid UTF-8") errNotUTF8 = errors.New("is not valid UTF-8")
errNotASN = errors.New("is not an AS number such as AS64496")
errNotPercentItem = errors.New( errNotPercentItem = errors.New(
"is not a code, : and a percentage, such as AS64496:50 or cn:25") "is not a code, : and a percentage, such as AS64496:50 or cn:25")
errNotPercent = errors.New( errNotAbuseIPDBThreshold = errors.New(
"is not a percentage, a whole number from 0 to 100", "is not a whole number from 25 to 100, the scores AbuseIPDB's " +
"blacklist can be cut at",
)
errNotWholeUnits = errors.New(
"is not a whole number above zero, such as 30, that fits a duration",
) )
errListedTwice = errors.New("is listed twice") errListedTwice = errors.New("is listed twice")
errNotListURL = errors.New( errNotListURL = errors.New(
@@ -540,10 +615,13 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
), ),
BytesLimitPerHour: env.size("SWW_WAF_BYTES_LIMIT_PER_HOUR", "50G"), BytesLimitPerHour: env.size("SWW_WAF_BYTES_LIMIT_PER_HOUR", "50G"),
BytesLimitPerDay: env.size("SWW_WAF_BYTES_LIMIT_PER_DAY", "100G"), BytesLimitPerDay: env.size("SWW_WAF_BYTES_LIMIT_PER_DAY", "100G"),
BytesCounted: env.bytesCounted("SWW_WAF_BYTES_COUNTED", "both"), BytesCounted: env.bytesCounted(
"SWW_WAF_BYTES_COUNTED",
BytesCountedBoth,
),
LookupSource: env.lookupSource( LookupSource: env.lookupSource(
"SWW_WAF_LOOKUP_SOURCE", "SWW_WAF_LOOKUP_SOURCE",
"geojs", LookupSourceGeoJS,
), ),
LookupDBPath: env.value("SWW_WAF_LOOKUP_DB_PATH", ""), LookupDBPath: env.value("SWW_WAF_LOOKUP_DB_PATH", ""),
ASNAndCountryLookupTimeout: env.milliseconds( ASNAndCountryLookupTimeout: env.milliseconds(
@@ -551,24 +629,24 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
"800", "800",
), ),
ExclusivelyAllowedCountries: env.countries( ExclusivelyAllowedCountries: env.countries(
"SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""), ExclusivelyAllowedCountriesSetting, ""),
TagLists: env.tagLists("SWW_WAF_TAG_LISTS"), TagLists: env.tagLists("SWW_WAF_TAG_LISTS"),
TagDeny: env.tags("SWW_WAF_TAG_DENY"), TagDeny: env.tags("SWW_WAF_TAG_DENY"),
TagLimitPercent: env.tagPercents("SWW_WAF_TAG_LIMIT_PERCENT"), TagLimitPercent: env.tagPercents(TagLimitPercentSetting),
RouteWatchURL: env.routeWatchURL("SWW_WAF_ROUTEWATCH_URL"), RouteWatchURL: env.routeWatchURL("SWW_WAF_ROUTEWATCH_URL"),
ASNBytesLimitPercent: env.percents( ASNBytesLimitPercent: env.percents(
"SWW_WAF_ASN_BYTES_LIMIT_PERCENT", ASNBytesLimitPercentSetting,
ParseASN, reputation.ParseASN,
), ),
CountryBytesLimitPercent: env.percents( CountryBytesLimitPercent: env.percents(
"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT", CountryBytesLimitPercentSetting,
parseCountry, parseCountry,
), ),
UnknownCountryLimitPercent: env.percent( UnknownCountryLimitPercent: env.percent(
"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT", UnknownCountryLimitPercentSetting,
"100", "100",
), ),
ASNLimitPercentURL: env.listURL("SWW_WAF_ASN_LIMIT_PERCENT_URL"), ASNLimitPercentURL: env.listURL(ASNLimitPercentURLSetting),
BlocklistURLs: env.listURLs("SWW_WAF_BLOCKLIST_URLS"), BlocklistURLs: env.listURLs("SWW_WAF_BLOCKLIST_URLS"),
BlocklistRefresh: env.refresh("SWW_WAF_BLOCKLIST_REFRESH", "24h"), BlocklistRefresh: env.refresh("SWW_WAF_BLOCKLIST_REFRESH", "24h"),
DNSBLZones: env.zones("SWW_WAF_DNSBL_ZONES"), DNSBLZones: env.zones("SWW_WAF_DNSBL_ZONES"),
@@ -587,6 +665,20 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
"SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS", "SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS",
"2000", "2000",
), ),
AbuseIPDBThreshold: env.abuseIPDBThreshold(
"SWW_WAF_ABUSEIPDB_THRESHOLD",
"90",
),
AbuseIPDBBlacklistRefresh: env.wholeUnits(
"SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS",
"60",
time.Minute,
),
AbuseIPDBBanDuration: env.wholeUnits(
"SWW_WAF_ABUSEIPDB_BAN_DAYS",
"30",
day,
),
BanResponse: env.banResponse("SWW_WAF_BAN_RESPONSE", "403"), BanResponse: env.banResponse("SWW_WAF_BAN_RESPONSE", "403"),
LimitBanDuration: env.durationNotOff( LimitBanDuration: env.durationNotOff(
"SWW_WAF_LIMIT_BAN_DURATION", "SWW_WAF_LIMIT_BAN_DURATION",
@@ -726,11 +818,11 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
cfg.InstanceName, cfg.LogRemoteURL != nil) cfg.InstanceName, cfg.LogRemoteURL != nil)
// The app gets the headers by default, unless no client is looked up. // The app gets the headers by default, unless no client is looked up.
cfg.AddLookupHeaders = env.boolean("SWW_WAF_ADD_LOOKUP_HEADERS", cfg.AddLookupHeaders = env.boolean("SWW_WAF_ADD_LOOKUP_HEADERS",
strconv.FormatBool(cfg.LookupSource != off)) strconv.FormatBool(cfg.LookupSource != LookupSourceOff))
cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action( cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action(
"SWW_WAF_BLOCKLIST_ACTION", "deny") BlocklistActionSetting, string(ReputationActionDeny))
cfg.DNSBLAbuseIPDBAction, cfg.DNSBLAbuseIPDBLimitPercent = env.action( cfg.DNSBLAbuseIPDBAction, cfg.DNSBLAbuseIPDBLimitPercent = env.action(
"SWW_WAF_DNSBL_ABUSEIPDB_ACTION", "limit:25") DNSBLAbuseIPDBActionSetting, "limit:25")
env.checkInstanceNameForNtfy(cfg.InstanceName, cfg.AlertNtfyURL != nil) env.checkInstanceNameForNtfy(cfg.InstanceName, cfg.AlertNtfyURL != nil)
env.checkLookupDBPath(cfg) env.checkLookupDBPath(cfg)
@@ -826,9 +918,12 @@ func (e *environment) count(name, defaultValue string) int64 {
// bytesCounted reads the setting that is which body bytes count toward the // bytesCounted reads the setting that is which body bytes count toward the
// byte limits and the byte anomaly thresholds: response, request or both. // byte limits and the byte anomaly thresholds: response, request or both.
func (e *environment) bytesCounted(name, defaultValue string) string { func (e *environment) bytesCounted(
value := e.value(name, defaultValue) name string, defaultValue BytesCounted,
if value != "response" && value != "request" && value != "both" { ) BytesCounted {
value := BytesCounted(e.value(name, string(defaultValue)))
if value != BytesCountedResponse && value != BytesCountedRequest &&
value != BytesCountedBoth {
e.check(name, fmt.Errorf("%q %w", value, errNotBytesCounted)) e.check(name, fmt.Errorf("%q %w", value, errNotBytesCounted))
} }
@@ -873,8 +968,10 @@ func (e *environment) trapPaths(name string) []string {
// coreRuleSetMode reads the setting that is what the Core Rule Set does: off, // coreRuleSetMode reads the setting that is what the Core Rule Set does: off,
// detect or block. // detect or block.
func (e *environment) coreRuleSetMode(name, defaultValue string) string { func (e *environment) coreRuleSetMode(
mode := e.value(name, defaultValue) name string, defaultValue CoreRuleSetMode,
) CoreRuleSetMode {
mode := CoreRuleSetMode(e.value(name, string(defaultValue)))
if mode != CoreRuleSetModeOff && mode != CoreRuleSetModeDetect && if mode != CoreRuleSetModeOff && mode != CoreRuleSetModeDetect &&
mode != CoreRuleSetModeBlock { mode != CoreRuleSetModeBlock {
e.check(name, fmt.Errorf("%q %w", mode, errNotCoreRuleSetMode)) e.check(name, fmt.Errorf("%q %w", mode, errNotCoreRuleSetMode))
@@ -925,7 +1022,7 @@ func (e *environment) percents(
} }
// tagLists reads SWW_WAF_TAG_LISTS, which is empty by default. // tagLists reads SWW_WAF_TAG_LISTS, which is empty by default.
func (e *environment) tagLists(name string) []TagList { func (e *environment) tagLists(name string) []reputation.TagList {
lists, err := parseTagLists(e.value(name, "")) lists, err := parseTagLists(e.value(name, ""))
e.check(name, err) e.check(name, err)
@@ -951,12 +1048,41 @@ func (e *environment) tagPercents(name string) map[string]int64 {
// percent reads a setting that is a percentage, from 0 to 100. // percent reads a setting that is a percentage, from 0 to 100.
func (e *environment) percent(name, defaultValue string) int64 { func (e *environment) percent(name, defaultValue string) int64 {
percent, err := ParsePercent(e.value(name, defaultValue)) percent, err := reputation.ParsePercent(e.value(name, defaultValue))
e.check(name, err) e.check(name, err)
return percent return percent
} }
// abuseIPDBThreshold reads the setting that is the least score of the
// clients on AbuseIPDB's blacklist: a whole number from 25 to 100, the
// range AbuseIPDB takes.
func (e *environment) abuseIPDBThreshold(name, defaultValue string) int64 {
value := e.value(name, defaultValue)
threshold, err := strconv.ParseInt(value, 10, 64)
if err != nil || threshold < minAbuseIPDBThreshold || threshold > 100 {
e.check(name, fmt.Errorf("%q %w", value, errNotAbuseIPDBThreshold))
}
return threshold
}
// wholeUnits reads a setting that is a whole number above zero of unit,
// such as minutes or days, and returns it as a duration.
func (e *environment) wholeUnits(
name, defaultValue string, unit time.Duration,
) time.Duration {
value := e.value(name, defaultValue)
n, err := strconv.ParseInt(value, 10, 64)
if err != nil || n <= 0 || n > math.MaxInt64/int64(unit) {
e.check(name, fmt.Errorf("%q %w", value, errNotWholeUnits))
}
return time.Duration(n) * unit
}
// listURL reads a setting that is the URL a list is fetched from, "" while // listURL reads a setting that is the URL a list is fetched from, "" while
// it is unset or empty. // it is unset or empty.
func (e *environment) listURL(name string) string { func (e *environment) listURL(name string) string {
@@ -995,25 +1121,29 @@ func (e *environment) refresh(name, defaultValue string) time.Duration {
// action reads a setting that is what is done with a client a blocklist // action reads a setting that is what is done with a client a blocklist
// or a DNSBL zone lists: deny, log, or limit:<percent>, which it returns // or a DNSBL zone lists: deny, log, or limit:<percent>, which it returns
// as limit and the percentage. // as limit and the percentage.
func (e *environment) action(name, defaultValue string) (string, int64) { func (e *environment) action(
value := e.value(name, defaultValue) name, defaultValue string,
if value == "deny" || value == "log" { ) (ReputationAction, int64) {
value := ReputationAction(e.value(name, defaultValue))
if value == ReputationActionDeny || value == ReputationActionLog {
return value, 0 return value, 0
} }
percentText, isLimit := strings.CutPrefix(value, "limit:") percentText, isLimit := strings.CutPrefix(
string(value), string(ReputationActionLimit)+":",
)
percent, err := ParsePercent(percentText) percent, err := reputation.ParsePercent(percentText)
if !isLimit || err != nil { if !isLimit || err != nil {
e.check(name, fmt.Errorf("%q %w", value, errNotAction)) e.check(name, fmt.Errorf("%q %w", value, errNotAction))
} }
return "limit", percent return ReputationActionLimit, percent
} }
// zones reads the setting that is the list of DNSBL zones. It is empty by // zones reads the setting that is the list of DNSBL zones. It is empty by
// default. The log shows each zone with its key masked, as MaskZoneKey // default. The log shows each zone with its key masked, as
// masks it. // reputation.MaskZoneKey masks it.
func (e *environment) zones(name string) []string { func (e *environment) zones(name string) []string {
value, _ := e.lookup(name) value, _ := e.lookup(name)
zones, err := parseZones(value) zones, err := parseZones(value)
@@ -1021,7 +1151,7 @@ func (e *environment) zones(name string) []string {
logged := make([]string, len(zones)) logged := make([]string, len(zones))
for i, zone := range zones { for i, zone := range zones {
logged[i] = MaskZoneKey(zone) logged[i] = reputation.MaskZoneKey(zone)
} }
e.settings = append( e.settings = append(
@@ -1059,9 +1189,12 @@ func (e *environment) routeWatchURL(name string) string {
// lookupSource reads the setting that is where clients are looked up: // lookupSource reads the setting that is where clients are looked up:
// geojs, file, or off. // geojs, file, or off.
func (e *environment) lookupSource(name, defaultValue string) string { func (e *environment) lookupSource(
source := e.value(name, defaultValue) name string, defaultValue LookupSource,
if source != "geojs" && source != fileSource && source != off { ) LookupSource {
source := LookupSource(e.value(name, string(defaultValue)))
if source != LookupSourceGeoJS && source != LookupSourceFile &&
source != LookupSourceOff {
e.check(name, fmt.Errorf("%q %w", source, errNotLookupSource)) e.check(name, fmt.Errorf("%q %w", source, errNotLookupSource))
} }
@@ -1073,12 +1206,12 @@ func (e *environment) lookupSource(name, defaultValue string) string {
// one source at a time. // one source at a time.
func (e *environment) checkLookupDBPath(cfg *WAF) { func (e *environment) checkLookupDBPath(cfg *WAF) {
switch { switch {
case cfg.LookupSource == fileSource && cfg.LookupDBPath == "": case cfg.LookupSource == LookupSourceFile && cfg.LookupDBPath == "":
e.check("SWW_WAF_LOOKUP_SOURCE", fmt.Errorf( e.check("SWW_WAF_LOOKUP_SOURCE", fmt.Errorf(
"is file while SWW_WAF_LOOKUP_DB_PATH is unset; %w", "is file while SWW_WAF_LOOKUP_DB_PATH is unset; %w",
errNeedsDBPath, errNeedsDBPath,
)) ))
case cfg.LookupSource != fileSource && cfg.LookupDBPath != "": case cfg.LookupSource != LookupSourceFile && cfg.LookupDBPath != "":
e.check("SWW_WAF_LOOKUP_DB_PATH", fmt.Errorf( e.check("SWW_WAF_LOOKUP_DB_PATH", fmt.Errorf(
"is set while SWW_WAF_LOOKUP_SOURCE is %s; %w", "is set while SWW_WAF_LOOKUP_SOURCE is %s; %w",
cfg.LookupSource, cfg.LookupSource,
@@ -1098,12 +1231,12 @@ func (e *environment) checkLookupDBPath(cfg *WAF) {
func (e *environment) checkCountriesAndLookups(cfg *WAF) { func (e *environment) checkCountriesAndLookups(cfg *WAF) {
for _, country := range cfg.ExclusivelyAllowedCountries { for _, country := range cfg.ExclusivelyAllowedCountries {
if slices.Contains(cfg.TagDeny, CountryTagPrefix+country) { if slices.Contains(cfg.TagDeny, CountryTagPrefix+country) {
e.check("SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", e.check(ExclusivelyAllowedCountriesSetting,
fmt.Errorf("%q %w", country, errOnBothLists)) fmt.Errorf("%q %w", country, errOnBothLists))
} }
} }
if cfg.LookupSource != off { if cfg.LookupSource != LookupSourceOff {
return return
} }
@@ -1117,15 +1250,15 @@ func (e *environment) checkCountriesAndLookups(cfg *WAF) {
name string name string
set bool set bool
}{ }{
{"SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", len(cfg.ExclusivelyAllowedCountries) > 0}, {ExclusivelyAllowedCountriesSetting, len(cfg.ExclusivelyAllowedCountries) > 0},
{"SWW_WAF_ADD_LOOKUP_HEADERS", cfg.AddLookupHeaders}, {"SWW_WAF_ADD_LOOKUP_HEADERS", cfg.AddLookupHeaders},
{"SWW_WAF_TAG_DENY", slices.ContainsFunc(cfg.TagDeny, onlyLookupGives)}, {"SWW_WAF_TAG_DENY", slices.ContainsFunc(cfg.TagDeny, onlyLookupGives)},
{"SWW_WAF_TAG_LIMIT_PERCENT", slices.ContainsFunc( {TagLimitPercentSetting, slices.ContainsFunc(
slices.Collect(maps.Keys(cfg.TagLimitPercent)), onlyLookupGives)}, slices.Collect(maps.Keys(cfg.TagLimitPercent)), onlyLookupGives)},
{"SWW_WAF_ASN_BYTES_LIMIT_PERCENT", len(cfg.ASNBytesLimitPercent) > 0}, {ASNBytesLimitPercentSetting, len(cfg.ASNBytesLimitPercent) > 0},
{"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT", len(cfg.CountryBytesLimitPercent) > 0}, {CountryBytesLimitPercentSetting, len(cfg.CountryBytesLimitPercent) > 0},
{"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT", cfg.UnknownCountryLimitPercent < 100}, {UnknownCountryLimitPercentSetting, cfg.UnknownCountryLimitPercent < 100},
{"SWW_WAF_ASN_LIMIT_PERCENT_URL", cfg.ASNLimitPercentURL != ""}, {ASNLimitPercentURLSetting, cfg.ASNLimitPercentURL != ""},
{"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE", cfg.AnomalyASN.RequestsPerMinute > 0}, {"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE", cfg.AnomalyASN.RequestsPerMinute > 0},
{"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR", cfg.AnomalyASN.RequestsPerHour > 0}, {"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR", cfg.AnomalyASN.RequestsPerHour > 0},
{"SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE", cfg.AnomalyASN.BytesPerMinute > 0}, {"SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE", cfg.AnomalyASN.BytesPerMinute > 0},
@@ -1145,7 +1278,7 @@ func (e *environment) checkCountriesAndLookups(cfg *WAF) {
// blocklist too: the file at a URL is fetched as one list or the other. // blocklist too: the file at a URL is fetched as one list or the other.
func (e *environment) checkASNLimitPercentURL(cfg *WAF) { func (e *environment) checkASNLimitPercentURL(cfg *WAF) {
if slices.Contains(cfg.BlocklistURLs, cfg.ASNLimitPercentURL) { if slices.Contains(cfg.BlocklistURLs, cfg.ASNLimitPercentURL) {
e.check("SWW_WAF_ASN_LIMIT_PERCENT_URL", e.check(ASNLimitPercentURLSetting,
fmt.Errorf("%q %w", cfg.ASNLimitPercentURL, errInBlocklistURLs)) fmt.Errorf("%q %w", cfg.ASNLimitPercentURL, errInBlocklistURLs))
} }
} }
@@ -1172,14 +1305,18 @@ func (e *environment) checkTags(cfg *WAF) {
} }
for tag := range cfg.TagLimitPercent { for tag := range cfg.TagLimitPercent {
e.check("SWW_WAF_TAG_LIMIT_PERCENT", checkTag(cfg, tag)) e.check(TagLimitPercentSetting, checkTag(cfg, tag))
} }
} }
// checkTag returns an error when tag would never apply, as checkTags says. // checkTag returns an error when tag would never apply, as checkTags says.
func checkTag(cfg *WAF, tag string) error { func checkTag(cfg *WAF, tag string) error {
if !requestCanGetTag(cfg, tag) { if !requestCanGetTag(cfg, tag) {
return fmt.Errorf("%q %w", MaskZoneKey(tag), errNoRequestGetsTag) return fmt.Errorf(
"%q %w",
reputation.MaskZoneKey(tag),
errNoRequestGetsTag,
)
} }
return nil return nil
@@ -1187,7 +1324,7 @@ func checkTag(cfg *WAF, tag string) error {
// requestCanGetTag reports whether anything cfg sets up can give a request tag. // requestCanGetTag reports whether anything cfg sets up can give a request tag.
func requestCanGetTag(cfg *WAF, tag string) bool { func requestCanGetTag(cfg *WAF, tag string) bool {
isListName := func(list TagList) bool { return list.Name == tag } isListName := func(list reputation.TagList) bool { return list.Name == tag }
if isLookupTag(tag) || slices.ContainsFunc(cfg.TagLists, isListName) { if isLookupTag(tag) || slices.ContainsFunc(cfg.TagLists, isListName) {
return true return true
} }
@@ -1211,7 +1348,7 @@ func requestCanGetTag(cfg *WAF, tag string) bool {
return isZoneTag && return isZoneTag &&
slices.ContainsFunc(cfg.DNSBLZones, func(each string) bool { slices.ContainsFunc(cfg.DNSBLZones, func(each string) bool {
return strings.ToLower(MaskZoneKey(each)) == zone return strings.ToLower(reputation.MaskZoneKey(each)) == zone
}) })
} }
@@ -1709,7 +1846,9 @@ func parseNamedNetblocks(value string) ([]anomaly.NamedNetblock, error) {
return nil, fmt.Errorf("%q %w", item, errNotNamedNetblock) return nil, fmt.Errorf("%q %w", item, errNotNamedNetblock)
} }
netblock, err := ParseNetblock(strings.TrimSpace(netblockText)) netblock, err := reputation.ParseNetblock(
strings.TrimSpace(netblockText),
)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -1915,21 +2054,6 @@ func parseCountry(value string) (string, error) {
return country, nil return country, nil
} }
// ParseASN reads an AS number such as AS64496, in either case, and
// returns it as the lookup gives it: AS and the number, in capitals and
// without leading zeros. The file SWW_WAF_ASN_LIMIT_PERCENT_URL names is
// read with it too.
func ParseASN(value string) (string, error) {
digits, hasAS := strings.CutPrefix(strings.ToUpper(value), "AS")
number, err := strconv.ParseUint(digits, 10, 32)
if !hasAS || err != nil {
return "", fmt.Errorf("%q %w", value, errNotASN)
}
return "AS" + strconv.FormatUint(number, 10), nil
}
// The tags a request can get that the settings can name, as waf // The tags a request can get that the settings can name, as waf
// writes them. The prefixes are followed by a value. // writes them. The prefixes are followed by a value.
const ( const (
@@ -1943,24 +2067,17 @@ const (
ExploitSignatureTag = "exploitsignature" ExploitSignatureTag = "exploitsignature"
) )
// TagList is an address list of SWW_WAF_TAG_LISTS: the tag it adds to the
// requests of the clients it holds, and the URL it is fetched from.
type TagList struct {
Name string
URL string
}
// parseTagLists reads a comma-separated list of name=URL pairs, such as // parseTagLists reads a comma-separated list of name=URL pairs, such as
// torexit=https://example.com/tor.txt, and returns each list with its name // torexit=https://example.com/tor.txt, and returns each list with its name
// in lower case. A name can hold a =, as vpn=mullvad does, and ends at the // in lower case. A name can hold a =, as vpn=mullvad does, and ends at the
// = before http. A name or a URL listed twice is an error. // = before http. A name or a URL listed twice is an error.
func parseTagLists(value string) ([]TagList, error) { func parseTagLists(value string) ([]reputation.TagList, error) {
items, err := parseList(value) items, err := parseList(value)
if err != nil { if err != nil {
return nil, err return nil, err
} }
lists := make([]TagList, 0, len(items)) lists := make([]reputation.TagList, 0, len(items))
for _, item := range items { for _, item := range items {
end := strings.Index(item, "=http") end := strings.Index(item, "=http")
@@ -1968,9 +2085,12 @@ func parseTagLists(value string) ([]TagList, error) {
return nil, fmt.Errorf("%q %w", item, errNotTagList) return nil, fmt.Errorf("%q %w", item, errNotTagList)
} }
list := TagList{Name: strings.ToLower(item[:end]), URL: item[end+1:]} list := reputation.TagList{
Name: strings.ToLower(item[:end]),
URL: item[end+1:],
}
if slices.ContainsFunc(lists, func(earlier TagList) bool { if slices.ContainsFunc(lists, func(earlier reputation.TagList) bool {
return earlier.Name == list.Name || earlier.URL == list.URL return earlier.Name == list.Name || earlier.URL == list.URL
}) { }) {
return nil, fmt.Errorf("%q %w", item, errListedTwice) return nil, fmt.Errorf("%q %w", item, errListedTwice)
@@ -1983,7 +2103,8 @@ func parseTagLists(value string) ([]TagList, error) {
} }
// parseTags reads a comma-separated list of tags, which parseTag reads. A // parseTags reads a comma-separated list of tags, which parseTag reads. A
// tag listed twice is an error, which shows the tag as MaskZoneKey does. // tag listed twice is an error, which shows the tag as
// reputation.MaskZoneKey does.
func parseTags(value string) ([]string, error) { func parseTags(value string) ([]string, error) {
items, err := parseList(value) items, err := parseList(value)
if err != nil { if err != nil {
@@ -2003,7 +2124,7 @@ func parseTags(value string) ([]string, error) {
if slices.Contains(tags, tag) { if slices.Contains(tags, tag) {
return nil, fmt.Errorf( return nil, fmt.Errorf(
"%q %w", "%q %w",
MaskZoneKey(item), reputation.MaskZoneKey(item),
errListedTwice, errListedTwice,
) )
} }
@@ -2072,7 +2193,7 @@ func parseTagPercents(value string) (map[string]int64, error) {
if colon <= 0 { if colon <= 0 {
return nil, fmt.Errorf( return nil, fmt.Errorf(
"%q %w", "%q %w",
MaskZoneKey(item), reputation.MaskZoneKey(item),
errNotTagPercentItem, errNotTagPercentItem,
) )
} }
@@ -2086,7 +2207,7 @@ func parseTagPercents(value string) (map[string]int64, error) {
if err != nil || percent < 1 || percent > 99 { if err != nil || percent < 1 || percent > 99 {
return nil, fmt.Errorf( return nil, fmt.Errorf(
"%q %w", "%q %w",
MaskZoneKey(item[colon+1:]), reputation.MaskZoneKey(item[colon+1:]),
errNotTagPercent, errNotTagPercent,
) )
} }
@@ -2098,7 +2219,11 @@ func parseTagPercents(value string) (map[string]int64, error) {
if _, listed := percents[tag]; listed { if _, listed := percents[tag]; listed {
// tag, since item[:colon] may end in a space, which MaskZoneKey // tag, since item[:colon] may end in a space, which MaskZoneKey
// would not mask. // would not mask.
return nil, fmt.Errorf("%q %w", MaskZoneKey(tag), errListedTwice) return nil, fmt.Errorf(
"%q %w",
reputation.MaskZoneKey(tag),
errListedTwice,
)
} }
percents[tag] = percent percents[tag] = percent
@@ -2132,7 +2257,7 @@ func parsePercents(
return nil, err return nil, err
} }
percent, err := ParsePercent(percentText) percent, err := reputation.ParsePercent(percentText)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -2147,38 +2272,6 @@ func parsePercents(
return percents, nil return percents, nil
} }
// ParsePercent reads a percentage, a whole number from 0 to 100. The file
// SWW_WAF_ASN_LIMIT_PERCENT_URL names is read with it too.
func ParsePercent(value string) (int64, error) {
percent, err := strconv.ParseInt(value, 10, 64)
if err != nil || percent < 0 || percent > 100 {
return 0, fmt.Errorf("%q %w", value, errNotPercent)
}
return percent, nil
}
// headerNameChars are the characters RFC 9110 allows in a header name:
// letters, digits and these marks.
const headerNameChars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" +
"0123456789!#$%&'*+-.^_`|~"
// IsHeaderName reports whether name can be a header name: one or more of
// the characters RFC 9110 allows in one.
func IsHeaderName(name string) bool {
if name == "" {
return false
}
for _, char := range name {
if !strings.ContainsRune(headerNameChars, char) {
return false
}
}
return true
}
// parseHeaderNames reads a comma-separated list of header names in either // parseHeaderNames reads a comma-separated list of header names in either
// case, and returns them in lower case. Host and Transfer-Encoding are // case, and returns them in lower case. Host and Transfer-Encoding are
// refused: Go's HTTP server takes them out of the request's headers. // refused: Go's HTTP server takes them out of the request's headers.
@@ -2191,7 +2284,7 @@ func parseHeaderNames(value string) ([]string, error) {
headers := make([]string, 0, len(items)) headers := make([]string, 0, len(items))
for _, item := range items { for _, item := range items {
if !IsHeaderName(item) { if !httpguts.ValidHeaderFieldName(item) {
return nil, fmt.Errorf("%q %w", item, errNotHeaderName) return nil, fmt.Errorf("%q %w", item, errNotHeaderName)
} }
@@ -2347,8 +2440,8 @@ const (
// other name for one that does not exist, so that the zone would list no // other name for one that does not exist, so that the zone would list no
// client. A zone listed twice is an error, whatever the case of its // client. A zone listed twice is an error, whatever the case of its
// letters, which DNS names ignore, and whatever its key, since // letters, which DNS names ignore, and whatever its key, since
// MaskZoneKey shows two keys of one zone alike. An error shows a zone as // reputation.MaskZoneKey shows two keys of one zone alike. An error shows a
// MaskZoneKey does. // zone as it does.
func parseZones(value string) ([]string, error) { func parseZones(value string) ([]string, error) {
zones, err := parseList(value) zones, err := parseList(value)
if err != nil { if err != nil {
@@ -2358,11 +2451,14 @@ func parseZones(value string) ([]string, error) {
} }
for i, zone := range zones { for i, zone := range zones {
shown := MaskZoneKey(zone) shown := reputation.MaskZoneKey(zone)
listedBefore := slices.ContainsFunc( listedBefore := slices.ContainsFunc(
zones[:i], zones[:i],
func(earlier string) bool { func(earlier string) bool {
return strings.EqualFold(MaskZoneKey(earlier), shown) return strings.EqualFold(
reputation.MaskZoneKey(earlier),
shown,
)
}, },
) )
@@ -2379,24 +2475,6 @@ func parseZones(value string) ([]string, error) {
return zones, nil return zones, nil
} }
// MaskZoneKey returns zone with ******** in place of its key, if it is a
// zone of Spamhaus's keyed query service, a name under dq.spamhaus.net,
// such as <key>.xbl.dq.spamhaus.net, whose first label is the key. Any
// other zone it returns as it is. A zone is shown so wherever it leaves
// the process: in the log, the alerts and the metrics.
func MaskZoneKey(zone string) string {
// DNS names ignore case, and a name may be written with a dot at its
// end.
name := strings.TrimSuffix(strings.ToLower(zone), ".")
if !strings.HasSuffix(name, ".dq.spamhaus.net") {
return zone
}
_, rest, _ := strings.Cut(zone, ".")
return masked + "." + rest
}
// isZone reports whether each label of zone is as parseZones takes it. It // isZone reports whether each label of zone is as parseZones takes it. It
// also checks the hostnames of SWW_WAF_APPS, which are DNS names too. // also checks the hostnames of SWW_WAF_APPS, which are DNS names too.
func isZone(zone string) bool { func isZone(zone string) bool {
@@ -2452,7 +2530,7 @@ func parseWebhookHeaders(value string) (http.Header, string, error) {
name, headerValue, found := strings.Cut(item, ":") name, headerValue, found := strings.Cut(item, ":")
name = strings.TrimSpace(name) name = strings.TrimSpace(name)
if !found || !IsHeaderName(name) || if !found || !httpguts.ValidHeaderFieldName(name) ||
strings.ContainsAny(headerValue, "\r\n\x00") { strings.ContainsAny(headerValue, "\r\n\x00") {
return nil, "", fmt.Errorf( return nil, "", fmt.Errorf(
"item %d %w", "item %d %w",
+110 -25
View File
@@ -18,10 +18,12 @@ import (
"time" "time"
"github.com/spf13/viper" "github.com/spf13/viper"
"golang.org/x/net/http/httpguts"
"sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/reputation"
) )
// The settings, by name. // The settings, by name.
@@ -61,6 +63,9 @@ const (
abuseIPDBKey = "SWW_WAF_ABUSEIPDB_KEY" abuseIPDBKey = "SWW_WAF_ABUSEIPDB_KEY"
abuseIPDBMinScore = "SWW_WAF_ABUSEIPDB_MIN_SCORE" abuseIPDBMinScore = "SWW_WAF_ABUSEIPDB_MIN_SCORE"
abuseIPDBDailyBudget = "SWW_WAF_ABUSEIPDB_DAILY_BUDGET" abuseIPDBDailyBudget = "SWW_WAF_ABUSEIPDB_DAILY_BUDGET"
abuseIPDBThreshold = "SWW_WAF_ABUSEIPDB_THRESHOLD"
abuseIPDBBlacklistRefresh = "SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS"
abuseIPDBBanDays = "SWW_WAF_ABUSEIPDB_BAN_DAYS"
dnsblAbuseIPDBAction = "SWW_WAF_DNSBL_ABUSEIPDB_ACTION" dnsblAbuseIPDBAction = "SWW_WAF_DNSBL_ABUSEIPDB_ACTION"
ipReputationCacheTTL = "SWW_WAF_IP_REPUTATION_CACHE_TTL" ipReputationCacheTTL = "SWW_WAF_IP_REPUTATION_CACHE_TTL"
dnsblAbuseIPDBTimeout = "SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS" dnsblAbuseIPDBTimeout = "SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS"
@@ -611,7 +616,7 @@ func TestCoreRuleSetSettings(t *testing.T) {
}, },
{ {
environment{ environment{
coreRuleSetMode: config.CoreRuleSetModeDetect, coreRuleSetMode: string(config.CoreRuleSetModeDetect),
coreRuleSetParanoiaLevel: "4", coreRuleSetParanoiaLevel: "4",
coreRuleSetScoreThreshold: "10", coreRuleSetScoreThreshold: "10",
coreRuleSetDisabledRules: "942100, 920350", coreRuleSetDisabledRules: "942100, 920350",
@@ -861,6 +866,32 @@ func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
} }
} }
// config and rules check header names with httpguts.ValidHeaderFieldName.
// It takes a name of one or more of the characters RFC 9110 allows in one,
// as the check it replaced did: letters, digits and these marks.
func TestHeaderNameCheckTakesTheRFC9110Characters(t *testing.T) {
t.Parallel()
const marks = "!#$%&'*+-.^_`|~"
if httpguts.ValidHeaderFieldName("") {
t.Error("the empty name is taken")
}
for b := range 256 {
char := byte(b)
want := 'a' <= char && char <= 'z' || 'A' <= char && char <= 'Z' ||
'0' <= char && char <= '9' || strings.IndexByte(marks, char) >= 0
// string(char) would be the UTF-8 of U+0080 to U+00FF, two bytes,
// for a byte above 0x7f.
name := string([]byte{char})
if got := httpguts.ValidHeaderFieldName(name); got != want {
t.Errorf("the name %q is taken: %t, want %t", name, got, want)
}
}
}
func TestRemoteLogSettingsDefaults(t *testing.T) { func TestRemoteLogSettingsDefaults(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1691,7 +1722,7 @@ func TestTagSettingsAsSet(t *testing.T) {
tagLimitPercent: "vpn=mullvad :50, asn=014061:25, COUNTRY=cn:1, dnsbl=zen.example:99", tagLimitPercent: "vpn=mullvad :50, asn=014061:25, COUNTRY=cn:1, dnsbl=zen.example:99",
}) })
wantLists := []config.TagList{ wantLists := []reputation.TagList{
{Name: torExitTag, URL: torURL}, {Name: torExitTag, URL: torURL},
{Name: "vpn=mullvad", URL: "https://example.com/mullvad.txt"}, {Name: "vpn=mullvad", URL: "https://example.com/mullvad.txt"},
} }
@@ -2030,8 +2061,9 @@ func TestReputationSettingsAsSet(t *testing.T) {
} }
for _, tc := range []struct { for _, tc := range []struct {
value, action string value string
percent int64 action config.ReputationAction
percent int64
}{ }{
{actionDeny, actionDeny, 0}, {actionDeny, actionDeny, 0},
{actionLog, actionLog, 0}, {actionLog, actionLog, 0},
@@ -2164,7 +2196,7 @@ func TestDNSBLSettingsAsSet(t *testing.T) {
// want are the zones, resolver, action and percent config.WAF gives. // want are the zones, resolver, action and percent config.WAF gives.
want []string want []string
wantResolver string wantResolver string
wantAction string wantAction config.ReputationAction
wantPercent int64 wantPercent int64
}{ }{
{ {
@@ -2284,25 +2316,6 @@ func TestInvalidDNSBLSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
} }
} }
func TestMaskZoneKeyMasksTheFirstLabelOfAZoneUnderDqSpamhausNet(
t *testing.T,
) {
t.Parallel()
for zone, want := range map[string]string{
spamhaus: spamhausMasked,
spamhaus + ".": spamhausMasked + ".",
"KEY.ZEN.DQ.SPAMHAUS.NET": "********.ZEN.DQ.SPAMHAUS.NET",
dronebl: dronebl,
"dq.spamhaus.net": "dq.spamhaus.net",
spamhaus + ".example": spamhaus + ".example",
} {
if got := config.MaskZoneKey(zone); got != want {
t.Errorf("MaskZoneKey(%q) is %q, want %q", zone, got, want)
}
}
}
func TestDNSBLZoneKeyIsLoggedMaskedAndNeverShown(t *testing.T) { func TestDNSBLZoneKeyIsLoggedMaskedAndNeverShown(t *testing.T) {
t.Parallel() t.Parallel()
@@ -2399,6 +2412,75 @@ func TestAbuseIPDBSettingsAsSet(t *testing.T) {
} }
} }
func TestAbuseIPDBBlacklistSettingsAsSet(t *testing.T) {
t.Parallel()
cfg := wafFromEnvironment(t, environment{})
if cfg.AbuseIPDBThreshold != 90 ||
cfg.AbuseIPDBBlacklistRefresh != time.Hour ||
cfg.AbuseIPDBBanDuration != 30*24*time.Hour {
t.Errorf(
"by default, the threshold %d, the refresh %s and the ban %s, "+
"want 90, 1h and 720h",
cfg.AbuseIPDBThreshold,
cfg.AbuseIPDBBlacklistRefresh,
cfg.AbuseIPDBBanDuration,
)
}
cfg = wafFromEnvironment(t, environment{
abuseIPDBThreshold: "25", abuseIPDBBlacklistRefresh: "1440",
abuseIPDBBanDays: "1",
})
if cfg.AbuseIPDBThreshold != 25 ||
cfg.AbuseIPDBBlacklistRefresh != 24*time.Hour ||
cfg.AbuseIPDBBanDuration != 24*time.Hour {
t.Errorf(
"set, the threshold %d, the refresh %s and the ban %s, "+
"want 25, 24h and 24h",
cfg.AbuseIPDBThreshold,
cfg.AbuseIPDBBlacklistRefresh,
cfg.AbuseIPDBBanDuration,
)
}
}
func TestInvalidAbuseIPDBBlacklistSettingStopsTheStartSayingWhatIsWrong(
t *testing.T,
) {
t.Parallel()
const (
notThreshold = " is not a whole number from 25 to 100, the scores " +
"AbuseIPDB's blacklist can be cut at"
notWhole = " is not a whole number above zero, such as 30, that fits " +
"a duration"
)
for _, tc := range []struct{ name, value, want string }{
{abuseIPDBThreshold, "24", `"24"` + notThreshold},
{abuseIPDBThreshold, "101", `"101"` + notThreshold},
{abuseIPDBThreshold, off, `"off"` + notThreshold},
{abuseIPDBBlacklistRefresh, "0", `"0"` + notWhole},
{abuseIPDBBlacklistRefresh, "1h", `"1h"` + notWhole},
{abuseIPDBBanDays, "-1", `"-1"` + notWhole},
{abuseIPDBBanDays, "200000", `"200000"` + notWhole},
} {
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel()
_, err := config.WAFFromEnvironment(
environment{tc.name: tc.value}.viper(),
)
want := tc.name + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
})
}
}
func TestInvalidAbuseIPDBSettingStopsTheStartSayingWhatIsWrong( func TestInvalidAbuseIPDBSettingStopsTheStartSayingWhatIsWrong(
t *testing.T, t *testing.T,
) { ) {
@@ -2951,6 +3033,9 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
abuseIPDBKey: "", abuseIPDBKey: "",
abuseIPDBMinScore: "75", abuseIPDBMinScore: "75",
abuseIPDBDailyBudget: "900", abuseIPDBDailyBudget: "900",
abuseIPDBThreshold: "90",
abuseIPDBBlacklistRefresh: "60",
abuseIPDBBanDays: "30",
dnsblAbuseIPDBAction: "limit:25", dnsblAbuseIPDBAction: "limit:25",
ipReputationCacheTTL: defaultIPReputationCacheTTL, ipReputationCacheTTL: defaultIPReputationCacheTTL,
dnsblAbuseIPDBTimeout: "2000", dnsblAbuseIPDBTimeout: "2000",
@@ -2974,7 +3059,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
logLevel: "info", logLevel: "info",
ruleFilesDir: "/etc/sww/rules.d", ruleFilesDir: "/etc/sww/rules.d",
ruleFilesEnabled: "true", ruleFilesEnabled: "true",
coreRuleSetMode: config.CoreRuleSetModeBlock, coreRuleSetMode: string(config.CoreRuleSetModeBlock),
coreRuleSetParanoiaLevel: "1", coreRuleSetParanoiaLevel: "1",
coreRuleSetScoreThreshold: "5", coreRuleSetScoreThreshold: "5",
coreRuleSetDisabledRules: defaultCoreRuleSetDisabledRules, coreRuleSetDisabledRules: defaultCoreRuleSetDisabledRules,
+90
View File
@@ -0,0 +1,90 @@
// Package dirwatch watches a directory, and calls a function once the
// directory has gone a short time without a change after one. The rule
// files and the lookup database are read again this way.
package dirwatch
import (
"context"
"time"
"github.com/fsnotify/fsnotify"
)
// quietTime is how long the directory must go without a change before the
// function is called, so that a file still being written, such as one
// saved in place, appended to or copied in with scp, is read only once
// whole.
const quietTime = 2 * time.Second
// Watcher watches one directory. The directory is watched, rather than a
// file in it, so that a file replaced by a rename is seen.
type Watcher struct {
watcher *fsnotify.Watcher
}
// New starts watching dir, or returns why it cannot.
func New(dir string) (*Watcher, error) {
watcher, err := fsnotify.NewWatcher()
if err != nil {
return nil, err
}
err = watcher.Add(dir)
if err != nil {
_ = watcher.Close()
return nil, err
}
return &Watcher{watcher: watcher}, nil
}
// Close stops watching the directory.
func (w *Watcher) Close() error {
return w.watcher.Close()
}
// CallWhenQuiet calls changed each time the directory has gone quietTime
// without a change, after a change to a path for which counts is true, until
// ctx is done. It also calls failed with each error the watch reports. The
// wait starts at once, as if for a change, so that a change made before the
// watch began is taken in too.
func (w *Watcher) CallWhenQuiet(
ctx context.Context,
counts func(path string) bool,
changed func(),
failed func(error),
) {
callWhenQuiet(
ctx, w.watcher.Events, w.watcher.Errors, counts, changed, failed,
)
}
// callWhenQuiet is CallWhenQuiet, for the changes from events and the
// errors from errs.
func callWhenQuiet(
ctx context.Context,
events <-chan fsnotify.Event,
errs <-chan error,
counts func(path string) bool,
changed func(),
failed func(error),
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if counts(event.Name) {
quiet.Reset(quietTime)
}
case <-quiet.C:
changed()
case err := <-errs:
failed(err)
}
}
}
+167
View File
@@ -0,0 +1,167 @@
package dirwatch
import (
"context"
"errors"
"path/filepath"
"sync/atomic"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
)
// errWatch is an error the watch reports.
var errWatch = errors.New("too many changes")
func TestDirectoryThatCannotBeWatchedIsAnError(t *testing.T) {
t.Parallel()
_, err := New(filepath.Join(t.TempDir(), "missing"))
if err == nil {
t.Error("a missing directory is watched, want an error")
}
}
// The tests below run callWhenQuiet in a synctest bubble, where time is a
// clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once callWhenQuiet waits again, so that every call
// due by then is made. The test sends the changes itself, as the watch of
// a directory cannot run in a bubble.
func TestSecondChangeStartsTheWaitAgain(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
// The second part starts the wait again.
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
time.Sleep(time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestCalledOnceQuietAfterTheStart(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
// An edit saved before the watch began sends no change, and is
// taken in by this call.
loop := start(t, everyPath)
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
// Long after, it has been called only once.
time.Sleep(time.Hour)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestChangeThatDoesNotCountIsNoChange(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, func(path string) bool {
return path == "ipinfo_lite.mmdb"
})
// The wait that starts with the watch ends with a call.
time.Sleep(quietTime)
synctest.Wait()
loop.changes <- fsnotify.Event{Name: "other.mmdb", Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
loop.changes <- fsnotify.Event{Name: "ipinfo_lite.mmdb", Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 2)
})
}
func TestErrorsOfTheWatchHandedOn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.errs <- errWatch
synctest.Wait()
if !errors.Is(loop.failure, errWatch) {
t.Errorf("failed with %v, want %v", loop.failure, errWatch)
}
})
}
// loop is a callWhenQuiet that a test runs.
type loop struct {
// changes and errs send it changes and errors.
changes chan fsnotify.Event
errs chan error
// calls counts its calls of changed, and failure is the last error it
// handed to failed.
calls atomic.Int32
failure error
}
// start runs callWhenQuiet, with counts, until the test ends.
func start(t *testing.T, counts func(path string) bool) *loop {
t.Helper()
l := &loop{changes: make(chan fsnotify.Event), errs: make(chan error)}
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
callWhenQuiet(ctx, l.changes, l.errs, counts,
func() { l.calls.Add(1) },
func(err error) { l.failure = err },
)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return l
}
// everyPath counts a change to any path.
func everyPath(string) bool {
return true
}
// wantCalls checks how many times changed has been called.
func (l *loop) wantCalls(t *testing.T, want int32) {
t.Helper()
got := l.calls.Load()
if got != want {
t.Errorf("changed called %d times, want %d", got, want)
}
}
+60 -37
View File
@@ -84,12 +84,7 @@ type Params struct {
// one request to GeoJS is under way at a time, and it asks about every // one request to GeoJS is under way at a time, and it asks about every
// client waiting, up to maxPerRequest. It is safe for concurrent use. // client waiting, up to maxPerRequest. It is safe for concurrent use.
type GeoJS struct { type GeoJS struct {
url string params Params
answered func(Answer)
now func() time.Time
asked func()
failed func(err error, askingAgainIn time.Duration)
wentWithoutAnswer func()
// httpClient follows no redirect, so that visitors' addresses go to // httpClient follows no redirect, so that visitors' addresses go to
// GeoJS alone: a redirect is a failure. // GeoJS alone: a redirect is a failure.
httpClient *http.Client httpClient *http.Client
@@ -99,8 +94,13 @@ type GeoJS struct {
// waiting are the clients without an answer: those to ask GeoJS about, // waiting are the clients without an answer: those to ask GeoJS about,
// and those it is being asked about. // and those it is being asked about.
waiting map[netip.Prefix]*wait waiting map[netip.Prefix]*wait
// asking is true while a request to GeoJS is under way. // asking is true while a request to GeoJS is under way, and endAsking
asking bool // ends it. stopped is true once StopAsking has been called, and
// askGoroutines waits for the goroutine that makes the requests.
asking bool
endAsking context.CancelFunc
stopped bool
askGoroutines sync.WaitGroup
// retryDelay is how long GeoJS is left alone after its last failure, // retryDelay is how long GeoJS is left alone after its last failure,
// zero after an answer; retryAt is when it may be asked again. // zero after an answer; retryAt is when it may be asked again.
retryDelay time.Duration retryDelay time.Duration
@@ -140,12 +140,7 @@ func New(params Params) *GeoJS {
} }
return &GeoJS{ return &GeoJS{
url: params.URL, params: params,
answered: params.Answered,
now: params.Now,
asked: params.Asked,
failed: params.Failed,
wentWithoutAnswer: params.WentWithoutAnswer,
httpClient: &http.Client{ httpClient: &http.Client{
CheckRedirect: func(*http.Request, []*http.Request) error { CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse return http.ErrUseLastResponse
@@ -188,7 +183,7 @@ func (g *GeoJS) LookUp(
answer, found := g.kept(client) answer, found := g.kept(client)
if !found { if !found {
g.wentWithoutAnswer() g.params.WentWithoutAnswer()
} }
w, waiting := g.waiting[client] w, waiting := g.waiting[client]
@@ -210,6 +205,23 @@ func (g *GeoJS) LookUpWithoutWaiting(
return answer return answer
} }
// StopAsking ends the request to GeoJS under way, and returns once the
// goroutine that made it has. A request it ends is no failure: Failed is not
// called, and GeoJS is not left alone after it. After StopAsking, GeoJS is
// asked nothing.
func (g *GeoJS) StopAsking() {
g.mu.Lock()
g.stopped = true
if g.asking {
g.endAsking()
}
g.mu.Unlock()
g.askGoroutines.Wait()
}
// Kept returns client's answer, if one is kept, without asking GeoJS. // Kept returns client's answer, if one is kept, without asking GeoJS.
func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) { func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) {
g.mu.Lock() g.mu.Lock()
@@ -251,7 +263,7 @@ func (g *GeoJS) Load(answers []Answer) {
g.answers.Purge() g.answers.Purge()
now := g.now() now := g.params.Now()
for _, answer := range answers { for _, answer := range answers {
if now.Sub(answer.Answered) < keepFor { if now.Sub(answer.Answered) < keepFor {
@@ -288,18 +300,19 @@ func (g *GeoJS) answerOrWait(
} }
if w == nil { if w == nil {
g.wentWithoutAnswer() g.params.WentWithoutAnswer()
return Answer{}, nil // too many clients wait already return Answer{}, nil // too many clients wait already
} }
if !g.asking { if !g.asking {
// GeoJS is left alone after a failure, so no answer can come. // GeoJS is left alone after a failure, or asked nothing after
// StopAsking, so no answer can come.
w.late = true w.late = true
} }
if w.late { if w.late {
g.wentWithoutAnswer() g.params.WentWithoutAnswer()
return Answer{}, nil return Answer{}, nil
} }
@@ -310,7 +323,7 @@ func (g *GeoJS) answerOrWait(
// kept returns client's answer, if GeoJS gave it less than keepFor ago, // kept returns client's answer, if GeoJS gave it less than keepFor ago,
// and notes that it was used. // and notes that it was used.
func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) { func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
now := g.now() now := g.params.Now()
kept, found := g.answers.Get(client) kept, found := g.answers.Get(client)
if !found || now.Sub(kept.Answered) >= keepFor { if !found || now.Sub(kept.Answered) >= keepFor {
@@ -323,22 +336,28 @@ func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
} }
// ask starts asking GeoJS about the waiting clients, unless a request to // ask starts asking GeoJS about the waiting clients, unless a request to
// it is under way or it is left alone after a failure. The requests to // it is under way, it is left alone after a failure, or StopAsking has
// GeoJS are for every client waiting, so they go on when the caller whose // been called. The requests to GeoJS are for every client waiting, so they
// ctx is given stops waiting. // go on when the caller whose ctx is given stops waiting, until StopAsking.
func (g *GeoJS) ask(ctx context.Context) { func (g *GeoJS) ask(ctx context.Context) {
if g.asking || g.now().Before(g.retryAt) { if g.asking || g.stopped || g.params.Now().Before(g.retryAt) {
return return
} }
g.asking = true askCtx, end := context.WithCancel(context.WithoutCancel(ctx))
g.asking, g.endAsking = true, end
go g.askAboutWaiting(context.WithoutCancel(ctx)) g.askGoroutines.Go(func() {
defer end()
g.askAboutWaiting(askCtx)
})
} }
// askAboutWaiting asks GeoJS about the waiting clients, one request at a // askAboutWaiting asks GeoJS about the waiting clients, one request at a
// time, until none is left or GeoJS fails. Each answer kept is given to // time, until none is left, GeoJS fails, or StopAsking ends ctx. Each
// Answered, outside the lock, since Answered takes locks of its own. // answer kept is given to Answered, outside the lock, since Answered takes
// locks of its own.
func (g *GeoJS) askAboutWaiting(ctx context.Context) { func (g *GeoJS) askAboutWaiting(ctx context.Context) {
for { for {
clients := g.nextClients() clients := g.nextClients()
@@ -347,11 +366,15 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
} }
given, err := g.request(ctx, clients) given, err := g.request(ctx, clients)
if err != nil && ctx.Err() != nil {
continue // ended by StopAsking, which is no failure
}
kept, answered := g.keep(clients, given, err) kept, answered := g.keep(clients, given, err)
if g.answered != nil { if g.params.Answered != nil {
for _, answer := range kept { for _, answer := range kept {
g.answered(answer) g.params.Answered(answer)
} }
} }
@@ -362,13 +385,13 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
} }
// nextClients returns up to maxPerRequest of the waiting clients. When // nextClients returns up to maxPerRequest of the waiting clients. When
// none is waiting, it returns none and notes that no request to GeoJS is // none is waiting, or StopAsking has been called, it returns none and
// under way. // notes that no request to GeoJS is under way.
func (g *GeoJS) nextClients() []netip.Prefix { func (g *GeoJS) nextClients() []netip.Prefix {
g.mu.Lock() g.mu.Lock()
defer g.mu.Unlock() defer g.mu.Unlock()
if len(g.waiting) == 0 { if len(g.waiting) == 0 || g.stopped {
g.asking = false g.asking = false
return nil return nil
@@ -400,7 +423,7 @@ func (g *GeoJS) keep(
g.mu.Lock() g.mu.Lock()
defer g.mu.Unlock() defer g.mu.Unlock()
now := g.now() now := g.params.Now()
kept := make([]Answer, 0, len(clients)) kept := make([]Answer, 0, len(clients))
leftOut := 0 leftOut := 0
@@ -444,7 +467,7 @@ func (g *GeoJS) keep(
w.late = true w.late = true
} }
g.failed(err, g.retryDelay) g.params.Failed(err, g.retryDelay)
return kept, false return kept, false
} }
@@ -474,7 +497,7 @@ func (g *GeoJS) request(
req, err := http.NewRequestWithContext( req, err := http.NewRequestWithContext(
ctx, ctx,
http.MethodGet, http.MethodGet,
g.url, g.params.URL,
http.NoBody, http.NoBody,
) )
if err != nil { if err != nil {
@@ -483,7 +506,7 @@ func (g *GeoJS) request(
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",") req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
g.asked() g.params.Asked()
res, err := g.httpClient.Do(req) res, err := g.httpClient.Do(req)
if err != nil { if err != nil {
+38
View File
@@ -318,6 +318,44 @@ func TestCallerWhoseContextEndsEarlyDoesNotStopTheRequest(t *testing.T) {
}) })
} }
func TestStopAskingEndsTheRequestUnderWayAsNoFailureAndAsksNoMore(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
service, _, g, reports := startWithReports()
clients := newClients()
service.set(hanging)
g.LookUpWithoutWaiting(t.Context(), clients())
waitForRequests(t, service, 1)
// The request is ended, not waited out, and is no failure.
began := time.Now()
g.StopAsking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopAsking returned after %s, want at once", waited)
}
reports.wantFailures(t)
// A new client is then not asked about, and its lookup does not
// wait.
service.set(answering)
wantCountry(t, g, clients(), "")
if waited := time.Since(began); waited != 0 {
t.Errorf("the lookup returned after %s, want at once", waited)
}
waitForRequests(t, service, 1)
reports.wantFailures(t)
})
}
func TestRequestOver10SecondsIsAFailure(t *testing.T) { func TestRequestOver10SecondsIsAFailure(t *testing.T) {
t.Parallel() t.Parallel()
+18 -49
View File
@@ -13,18 +13,13 @@ import (
"sync" "sync"
"time" "time"
"github.com/fsnotify/fsnotify"
"github.com/oschwald/maxminddb-golang/v2" "github.com/oschwald/maxminddb-golang/v2"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
) )
// quietTime is how long the lookup database must go without a change
// before it is read again, so that a file still being copied in is read
// only once whole.
const quietTime = 2 * time.Second
// FileParams are what OpenFile needs. // FileParams are what OpenFile needs.
type FileParams struct { type FileParams struct {
// Path is the lookup database, the IPinfo Lite file in its .mmdb form // Path is the lookup database, the IPinfo Lite file in its .mmdb form
@@ -122,20 +117,13 @@ func (f *File) ReadFailures() int {
} }
// Watch watches the directory of the lookup database until ctx is done, // Watch watches the directory of the lookup database until ctx is done,
// and reads the file again once it has gone without a change for // and reads the file again once it has gone without a change for a short
// quietTime, after it is replaced, written or removed, and after Watch // time, after it is replaced, written or removed, and after Watch starts
// starts watching. If the directory cannot be watched, that is logged, and // watching, as dirwatch describes. A change to another file in its
// the database read at start stays in use. // directory does not count. If the directory cannot be watched, that is
// logged, and the database read at start stays in use.
func (f *File) Watch(ctx context.Context) { func (f *File) Watch(ctx context.Context) {
watcher, err := fsnotify.NewWatcher() watcher, err := dirwatch.New(filepath.Dir(f.params.Path))
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(filepath.Dir(f.params.Path))
}
if err != nil { if err != nil {
f.params.ProcessLog.Error( f.params.ProcessLog.Error(
"cannot watch the lookup database for replacements", "cannot watch the lookup database for replacements",
@@ -146,41 +134,22 @@ func (f *File) Watch(ctx context.Context) {
return return
} }
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the lookup database for replacements", f.params.ProcessLog.Info("watching the lookup database for replacements",
"file", f.params.Path) "file", f.params.Path)
f.readAfterChanges(ctx, watcher.Events, watcher.Errors) watcher.CallWhenQuiet(ctx, f.isDatabase, f.readAgain, func(err error) {
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
})
} }
// readAfterChanges reads the lookup database again once quietTime has // isDatabase reports whether path names the lookup database.
// passed without a change to it from events, until ctx is done, and logs func (f *File) isDatabase(path string) bool {
// the errors from errs. A change to another file in its directory does not return filepath.Clean(path) == filepath.Clean(f.params.Path)
// count. The wait starts at once, as if for a change, so that a file
// replaced after OpenFile read it, and before its directory was watched,
// is read too.
func (f *File) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
path := filepath.Clean(f.params.Path)
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if filepath.Clean(event.Name) == path {
quiet.Reset(quietTime)
}
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
}
}
} }
// readAgain reads the lookup database again, in place of the one in use, // readAgain reads the lookup database again, in place of the one in use,
+35 -141
View File
@@ -1,7 +1,6 @@
package lookup package lookup
import ( import (
"context"
"log/slog" "log/slog"
"net/netip" "net/netip"
"net/url" "net/url"
@@ -12,7 +11,6 @@ import (
"testing/synctest" "testing/synctest"
"time" "time"
"github.com/fsnotify/fsnotify"
"github.com/maxmind/mmdbwriter/mmdbtype" "github.com/maxmind/mmdbwriter/mmdbtype"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
@@ -133,66 +131,48 @@ func TestFileThatCannotBeReadIsAnError(t *testing.T) {
} }
} }
// The tests below run readAfterChanges in a synctest bubble, where time is func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
// a clock of the test's own: time.Sleep moves it on at once, and t.Parallel()
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
func TestReplacementCopiedOverTheFileInTwoPartsIsReadOnlyWhole( dir := t.TempDir()
t *testing.T, path := filepath.Join(dir, "ipinfo_lite.mmdb")
) { writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
for changed, want := range map[string]bool{
path: true,
dir + "/./ipinfo_lite.mmdb": true,
filepath.Join(dir, "other.mmdb"): false,
} {
if f.isDatabase(changed) != want {
t.Errorf(
"a change to %s counts: %t, want %t",
changed,
!want,
want,
)
}
}
}
// The tests below run in a synctest bubble, where time is a clock of the
// test's own, so that the time a file is read can be checked. The wait for
// the directory to go quiet before a reading is tested in dirwatch.
func TestReplacementIsReadAgain(t *testing.T) {
t.Parallel() t.Parallel()
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
dir := t.TempDir() path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE") writeDatabase(t, path, "DE")
queue := newQueue() queue := newQueue()
f := openFile(t, path, queue) f := openFile(t, path, queue)
changes := watch(t, f)
other := filepath.Join(dir, "replacement.mmdb") writeDatabase(t, path, "KP")
writeDatabase(t, other, "KP") time.Sleep(time.Hour)
f.readAgain()
//nolint:gosec // a file the test wrote
replacement, err := os.ReadFile(other)
if err != nil {
t.Fatalf("read %s: %v", other, err)
}
// The file in use is overwritten in place, and keeps giving what
// it gave. Its first part alone is not a .mmdb file.
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create %s: %v", path, err)
}
defer func() {
_ = file.Close()
}()
half := len(replacement) / 2
write(t, file, replacement[:half])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
// The second part starts the wait again.
write(t, file, replacement[half:])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "KP") wantCountry(t, f, "KP")
if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 { if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 {
@@ -219,15 +199,10 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
queue := newQueue() queue := newQueue()
f := openFile(t, path, queue) f := openFile(t, path, queue)
read := f.LastRead() read := f.LastRead()
changes := watch(t, f)
writeFile(t, path, "not a lookup database\n") writeFile(t, path, "not a lookup database\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
// Long after, the replacement has been read once.
time.Sleep(time.Hour) time.Sleep(time.Hour)
synctest.Wait() f.readAgain()
wantCountry(t, f, "DE") wantCountry(t, f, "DE")
if !f.LastRead().Equal(read) || f.ReadFailures() != 1 { if !f.LastRead().Equal(read) || f.ReadFailures() != 1 {
@@ -240,7 +215,7 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
} }
wantAlerts(t, queue, alerts.Alert{ wantAlerts(t, queue, alerts.Alert{
Time: read.Add(quietTime), Time: read.Add(time.Hour),
Event: alerts.EventFileError, Event: alerts.EventFileError,
Reason: "the lookup database cannot be read, and the one read before stays in use", Reason: "the lookup database cannot be read, and the one read before stays in use",
Detail: map[string]any{ Detail: map[string]any{
@@ -252,55 +227,6 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
}) })
} }
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
changes := watch(t, f)
// The wait that starts with the watch ends with a reading.
time.Sleep(quietTime)
synctest.Wait()
writeDatabase(t, path, "KP")
changes <- fsnotify.Event{Name: filepath.Join(dir, "other.mmdb"), Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "DE")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
func TestReplacementSavedBeforeTheWatchStartsIsRead(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
// Saved after OpenFile read the file, and before its directory was
// watched, so that no change is seen for it.
writeDatabase(t, path, "KP")
watch(t, f)
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
// newQueue returns a queue of alerts for a webhook that is never sent // newQueue returns a queue of alerts for a webhook that is never sent
// them, so that they wait in it for the test to look at. // them, so that they wait in it for the test to look at.
func newQueue() *alerts.Queue { func newQueue() *alerts.Queue {
@@ -344,28 +270,6 @@ func openFile(t *testing.T, path string, queue *alerts.Queue) *File {
return f return f
} }
// watch runs f's readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func watch(t *testing.T, f *File) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
f.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// wantCountry checks the country f gives testClient. // wantCountry checks the country f gives testClient.
func wantCountry(t *testing.T, f *File, want string) { func wantCountry(t *testing.T, f *File, want string) {
t.Helper() t.Helper()
@@ -401,13 +305,3 @@ func writeFile(t *testing.T, path, content string) {
t.Fatalf("write %s: %v", path, err) t.Fatalf("write %s: %v", path, err)
} }
} }
// write writes data to the end of file.
func write(t *testing.T, file *os.File, data []byte) {
t.Helper()
_, err := file.Write(data)
if err != nil {
t.Fatalf("write: %v", err)
}
}
+5 -4
View File
@@ -14,7 +14,6 @@ import (
"github.com/prometheus/client_golang/prometheus/promhttp" "github.com/prometheus/client_golang/prometheus/promhttp"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/remotelog" "sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
@@ -185,7 +184,7 @@ func (m *Metrics) AddBansAndClients(
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time, ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
) { ) {
for _, cause := range []string{ for _, cause := range []string{
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseAbuseIPDB,
} { } {
m.registry.MustRegister( m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{ prometheus.NewCounterFunc(prometheus.CounterOpts{
@@ -301,7 +300,7 @@ const sourceLabel = "source"
// AddReputation adds the metrics of the lists fetched from URLs and of the // AddReputation adds the metrics of the lists fetched from URLs and of the
// DNSBL zones, by source, each list's URL or each zone, its key masked as // DNSBL zones, by source, each list's URL or each zone, its key masked as
// config.MaskZoneKey masks it: the requests whose client a blocklist, a // reputation.MaskZoneKey masks it: the requests whose client a blocklist, a
// zone's verdict or AbuseIPDB's score lists, which ReputationHit counts, // zone's verdict or AbuseIPDB's score lists, which ReputationHit counts,
// and, read from lists and dnsbl as the metrics are asked for, for a list, // and, read from lists and dnsbl as the metrics are asked for, for a list,
// the fetches that failed and when the copy in use was fetched, and for a // the fetches that failed and when the copy in use was fetched, and for a
@@ -320,7 +319,9 @@ func (m *Metrics) AddReputation(
m.registry.MustRegister(m.reputationHits) m.registry.MustRegister(m.reputationHits)
for _, zone := range dnsbl.Zones() { for _, zone := range dnsbl.Zones() {
source := prometheus.Labels{sourceLabel: config.MaskZoneKey(zone)} source := prometheus.Labels{
sourceLabel: reputation.MaskZoneKey(zone),
}
m.addReputationQueries( m.addReputationQueries(
source, source,
+44
View File
@@ -0,0 +1,44 @@
// Package processlog builds the logger for the process's own messages,
// which it writes as JSON lines marked "type":"process". The request log
// is package requestlog's.
package processlog
import (
"io"
"log/slog"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// Params are what New needs.
type Params struct {
// Output is where the lines are written.
Output io.Writer
// InstanceName, SWW_WAF_INSTANCE_NAME, is each line's instance.
InstanceName string
// Level, SWW_WAF_LOG_LEVEL, is the least severe level written.
Level slog.Level
}
// New returns the logger for the process's own messages: JSON lines on
// params.Output, marked "type":"process", with the time in the same form
// as a request line's. It writes only the messages at params.Level or more
// severe; the request lines requestlog.Write writes are never held back.
func New(params Params) *slog.Logger {
handler := slog.NewJSONHandler(params.Output, &slog.HandlerOptions{
Level: params.Level,
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(
slog.TimeKey,
requestlog.FormatTime(attr.Value.Time()),
)
}
return attr
},
})
return slog.New(handler).
With("type", "process", "instance", params.InstanceName)
}
+105
View File
@@ -0,0 +1,105 @@
package processlog_test
import (
"bytes"
"encoding/json"
"log/slog"
"slices"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/processlog"
)
// instanceName is the instance the tests' loggers are built with.
const instanceName = "host1/blog"
func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processlog.New(processlog.Params{
Output: &out,
InstanceName: instanceName,
Level: slog.LevelInfo,
}).
Info("starting",
"version", "v1")
var fields map[string]any
err := json.Unmarshal(out.Bytes(), &fields)
if err != nil {
t.Fatalf("decode %q: %v", out.String(), err)
}
if fields["type"] != "process" ||
fields["instance"] != "host1/blog" ||
fields["msg"] != "starting" ||
fields["level"] != "INFO" ||
fields["version"] != "v1" {
t.Errorf("process line %v", fields)
}
timeText, _ := fields["time"].(string)
logged, err := time.Parse(time.RFC3339, timeText)
if err != nil || !strings.HasSuffix(timeText, "Z") ||
len(timeText) != len("2006-01-02T15:04:05.000Z") ||
time.Since(logged) > time.Minute {
t.Errorf(
"process line time %q, want now in UTC with milliseconds",
timeText,
)
}
}
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
t.Parallel()
levels := []slog.Level{
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
}
for i, level := range levels {
t.Run(level.String(), func(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processLog := processlog.New(processlog.Params{
Output: &out,
InstanceName: instanceName,
Level: level,
})
for _, at := range levels {
processLog.Log(t.Context(), at, "message")
}
var got, want []string
for line := range strings.Lines(out.String()) {
var fields struct {
Level string `json:"level"`
}
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("decode %q: %v", line, err)
}
got = append(got, fields.Level)
}
for _, written := range levels[i:] {
want = append(want, written.String())
}
if !slices.Equal(got, want) {
t.Errorf("lines at %v, want %v", got, want)
}
})
}
}
+7 -4
View File
@@ -11,7 +11,7 @@ import (
func TestHistoryKeepsEveryRequest(t *testing.T) { func TestHistoryKeepsEveryRequest(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -57,7 +57,7 @@ func TestHistoryKeepsEveryRequest(t *testing.T) {
func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) { func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("198.51.100.7/32") other := netip.MustParsePrefix("198.51.100.7/32")
start := midnight() start := midnight()
@@ -100,7 +100,10 @@ func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
func TestResetKeepsTheHistory(t *testing.T) { func TestResetKeepsTheHistory(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -123,7 +126,7 @@ func TestResetKeepsTheHistory(t *testing.T) {
func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) { func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
for client, requests := range map[string]int{ for client, requests := range map[string]int{
"198.51.100.9/32": 2, "198.51.100.9/32": 2,
+18 -6
View File
@@ -156,16 +156,28 @@ type Request struct {
BlockedByCoreRuleSet bool BlockedByCoreRuleSet bool
} }
// New returns a Limiter for limits, with no client counted yet, whose // Params are what New needs.
// table holds at most maxClients clients (SWW_WAF_MAX_TRACKED_CLIENTS). Past type Params struct {
// it, the least recently seen client is dropped, with its history, and // Limits are what each client's requests and bytes are counted against.
// starts afresh if it comes back. Limits Limits
func New(limits Limits, maxClients int) *Limiter { // MaxClients is the most clients the table holds,
clients, err := simplelru.NewLRU[netip.Prefix, *Client](maxClients, nil) // SWW_WAF_MAX_TRACKED_CLIENTS. Past it, the least recently seen client
// is dropped, with its history, and starts afresh if it comes back.
MaxClients int
}
// New returns a Limiter with no client counted yet.
func New(params Params) *Limiter {
clients, err := simplelru.NewLRU[netip.Prefix, *Client](
params.MaxClients,
nil,
)
if err != nil { if err != nil {
panic(err) // NewLRU fails only for a size below one panic(err) // NewLRU fails only for a size below one
} }
limits := params.Limits
return &Limiter{ return &Limiter{
windows: [3]window{ windows: [3]window{
{ {
+63 -39
View File
@@ -33,10 +33,10 @@ func TestOverARateLimitSaysWhatTheNextRequestWouldDoAndCountsNothing(
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
now := midnight() now := midnight()
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerMinute: limit, PerDay: limit + 1}, Limits: ratelimit.Limits{PerMinute: limit, PerDay: limit + 1},
tableSize, MaxClients: tableSize,
) })
if limiter.OverARateLimit(client, now) { if limiter.OverARateLimit(client, now) {
t.Error("a client never seen is over a rate limit") t.Error("a client never seen is over a rate limit")
@@ -72,7 +72,7 @@ func TestOverARateLimitSaysWhatTheNextRequestWouldDoAndCountsNothing(
wantCount(t, limiter, client, now, minute) wantCount(t, limiter, client, now, minute)
// A limit that is off is never gone over. // A limit that is off is never gone over.
off := ratelimit.New(ratelimit.Limits{}, tableSize) off := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
off.Count(client, now, whole) off.Count(client, now, whole)
if off.OverARateLimit(client, now) { if off.OverARateLimit(client, now) {
@@ -95,7 +95,10 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
t.Run(tc.window, func(t *testing.T) { t.Run(tc.window, func(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(tc.limits, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: tc.limits,
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
quarter := tc.length / 4 quarter := tc.length / 4
@@ -126,10 +129,10 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) { func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerMinute: limit, PerHour: limit}, Limits: ratelimit.Limits{PerMinute: limit, PerHour: limit},
tableSize, MaxClients: tableSize,
) })
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -159,10 +162,13 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) { func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerMinute: 5, BytesPerDay: math.MaxInt64}, Limits: ratelimit.Limits{
tableSize, PerMinute: 5,
) BytesPerDay: math.MaxInt64,
},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -203,10 +209,10 @@ func TestZeroPercentIsAZeroAllowanceAndALimitOffStaysOff(t *testing.T) {
t.Parallel() t.Parallel()
// Only the hour has limits: the minute's and the day's are off. // Only the hour has limits: the minute's and the day's are off.
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerHour: limit, BytesPerHour: 1000}, Limits: ratelimit.Limits{PerHour: limit, BytesPerHour: 1000},
tableSize, MaxClients: tableSize,
) })
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -253,7 +259,10 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
t.Run(tc.window, func(t *testing.T) { t.Run(tc.window, func(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(tc.limits, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: tc.limits,
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
// 600 bytes are within the limit, 600 more over it. // 600 bytes are within the limit, 600 more over it.
@@ -287,10 +296,10 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) { func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000}, Limits: ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000},
tableSize, MaxClients: tableSize,
) })
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("203.0.113.10/32") other := netip.MustParsePrefix("203.0.113.10/32")
start := midnight() start := midnight()
@@ -314,7 +323,7 @@ func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
func TestCountGivesTheBytesInEachWindow(t *testing.T) { func TestCountGivesTheBytesInEachWindow(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -340,7 +349,10 @@ func TestCountGivesTheBytesInEachWindow(t *testing.T) {
func TestResetSetsTheBytesBackToZero(t *testing.T) { func TestResetSetsTheBytesBackToZero(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000}, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{BytesPerDay: 1000},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -356,7 +368,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
) { ) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -411,7 +423,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
func TestCountGivesTheRequestsInEachWindow(t *testing.T) { func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -438,10 +450,10 @@ func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
func TestResetSetsTheCountsBackToZero(t *testing.T) { func TestResetSetsTheCountsBackToZero(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerMinute: limit, PerDay: limit}, Limits: ratelimit.Limits{PerMinute: limit, PerDay: limit},
tableSize, MaxClients: tableSize,
) })
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -463,7 +475,10 @@ func TestResetSetsTheCountsBackToZero(t *testing.T) {
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) { func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -482,10 +497,10 @@ func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
func TestRefusedRequestsCount(t *testing.T) { func TestRefusedRequestsCount(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit}, Limits: ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit},
tableSize, MaxClients: tableSize,
) })
refused := netip.MustParsePrefix("203.0.113.9/32") refused := netip.MustParsePrefix("203.0.113.9/32")
within := netip.MustParsePrefix("203.0.113.10/32") within := netip.MustParsePrefix("203.0.113.10/32")
start := midnight() start := midnight()
@@ -518,7 +533,10 @@ func TestRefusedRequestsCount(t *testing.T) {
func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) { func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -534,7 +552,10 @@ func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) { func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -562,7 +583,10 @@ func TestKeepsAtMostMaxClientsDroppingTheLeastRecentlySeen(t *testing.T) {
const maxClients = 3 const maxClients = 3
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1}, maxClients) limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: 1},
MaxClients: maxClients,
})
now := midnight() now := midnight()
clients := make([]netip.Prefix, maxClients+1) clients := make([]netip.Prefix, maxClients+1)
+12 -6
View File
@@ -19,7 +19,7 @@ func TestSnapshotListsTheClientsByAddress(t *testing.T) {
"2001:db8::/64", "2001:db8::/64",
} }
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
for _, i := range []int{2, 3, 0, 1} { for _, i := range []int{2, 3, 0, 1} {
limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole) limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole)
} }
@@ -52,7 +52,10 @@ func TestLoadedCountsCarryOn(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
before := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize) before := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
for range limit { for range limit {
wantCount(t, before, client, start, "") wantCount(t, before, client, start, "")
} }
@@ -60,7 +63,10 @@ func TestLoadedCountsCarryOn(t *testing.T) {
// Loaded into a new limiter, as across a restart, the client has no // Loaded into a new limiter, as across a restart, the client has no
// fresh allowance. // fresh allowance.
later := start.Add(time.Minute) later := start.Add(time.Minute)
after := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize) after := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
after.Load(before.Snapshot(), later) after.Load(before.Snapshot(), later)
wantCount(t, after, client, later, hour) wantCount(t, after, client, later, hour)
} }
@@ -71,7 +77,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
limiter.Count(client, start, whole) limiter.Count(client, start, whole)
limiter.CountBytes(client, start, 5, whole) limiter.CountBytes(client, start, 5, whole)
limiter.CountRefusal(client, start, limit) limiter.CountRefusal(client, start, limit)
@@ -80,7 +86,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
loaded := func(now time.Time) ratelimit.Client { loaded := func(now time.Time) ratelimit.Client {
t.Helper() t.Helper()
after := ratelimit.New(ratelimit.Limits{}, tableSize) after := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
after.Load(limiter.Snapshot(), now) after.Load(limiter.Snapshot(), now)
return after.Snapshot()[0] return after.Snapshot()[0]
@@ -142,7 +148,7 @@ func TestLoadDropsTheLeastRecentlySeenFirst(t *testing.T) {
addr = addr.Next() addr = addr.Next()
} }
limiter := ratelimit.New(ratelimit.Limits{}, maxClients) limiter := ratelimit.New(ratelimit.Params{MaxClients: maxClients})
limiter.Load(clients, midnight()) limiter.Load(clients, midnight())
got := limiter.Snapshot() got := limiter.Snapshot()
+194 -18
View File
@@ -11,6 +11,7 @@ import (
"net/netip" "net/netip"
"net/url" "net/url"
"slices" "slices"
"strconv"
"sync" "sync"
"time" "time"
@@ -23,6 +24,9 @@ const (
// AbuseIPDBURL is where clients are checked: the check endpoint of // AbuseIPDBURL is where clients are checked: the check endpoint of
// AbuseIPDB's API. // AbuseIPDB's API.
AbuseIPDBURL = "https://api.abuseipdb.com/api/v2/check" AbuseIPDBURL = "https://api.abuseipdb.com/api/v2/check"
// AbuseIPDBBlacklistURL is where AbuseIPDB's blacklist is fetched from:
// the blacklist endpoint of AbuseIPDB's API.
AbuseIPDBBlacklistURL = "https://api.abuseipdb.com/api/v2/blacklist"
// AbuseIPDBSource is how the request log, the alerts and the metrics // AbuseIPDBSource is how the request log, the alerts and the metrics
// name AbuseIPDB. // name AbuseIPDB.
AbuseIPDBSource = "abuseipdb" AbuseIPDBSource = "abuseipdb"
@@ -32,6 +36,8 @@ const (
var ( var (
errNoScore = errors.New("the answer gives no abuseConfidenceScore") errNoScore = errors.New("the answer gives no abuseConfidenceScore")
errNoBlacklist = errors.New("the answer gives no data")
errNotAddress = errors.New("is not an address")
errBudgetUsedUp = errors.New( errBudgetUsedUp = errors.New(
"checks spent; none is made until the day ends at 00:00 UTC") "checks spent; none is made until the day ends at 00:00 UTC")
) )
@@ -53,6 +59,13 @@ type ChecksSpent struct {
Spent int Spent int
} }
// BlacklistEntry is a client on AbuseIPDB's blacklist: its address and its
// abuse confidence score.
type BlacklistEntry struct {
Addr netip.Addr
Score int64
}
// AbuseIPDBParams are what NewAbuseIPDB needs. // AbuseIPDBParams are what NewAbuseIPDB needs.
type AbuseIPDBParams struct { type AbuseIPDBParams struct {
// URL is where clients are checked, normally AbuseIPDBURL, with Key, // URL is where clients are checked, normally AbuseIPDBURL, with Key,
@@ -69,10 +82,19 @@ type AbuseIPDBParams struct {
// take (SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS). // take (SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS).
CacheTTL time.Duration CacheTTL time.Duration
Timeout time.Duration Timeout time.Duration
// BlacklistURL is where the blacklist is fetched from, normally
// AbuseIPDBBlacklistURL, with the clients scored Threshold or more
// (SWW_WAF_ABUSEIPDB_THRESHOLD), every BlacklistRefresh
// (SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS). Blacklisted
// receives the clients on each blacklist fetched.
BlacklistURL string
Threshold int64
BlacklistRefresh time.Duration
Blacklisted func([]BlacklistEntry)
// Now tells the time, normally time.Now in UTC. // Now tells the time, normally time.Now in UTC.
Now func() time.Time Now func() time.Time
// ProcessLog receives each check that fails, and why, and the day's // ProcessLog receives each check or fetch of the blacklist that fails,
// budget used up. // and why, and the day's budget used up.
ProcessLog *slog.Logger ProcessLog *slog.Logger
// Alerts receive a source_failure alert for each. // Alerts receive a source_failure alert for each.
Alerts *alerts.Queue Alerts *alerts.Queue
@@ -88,8 +110,12 @@ type AbuseIPDB struct {
// scores are by client. Each is added as it is fetched and never moved // scores are by client. Each is added as it is fetched and never moved
// up, so that the one fetched longest ago is the first dropped. // up, so that the one fetched longest ago is the first dropped.
scores *simplelru.LRU[netip.Prefix, Score] scores *simplelru.LRU[netip.Prefix, Score]
// checking are the clients whose check is under way. // checking are the clients whose check is under way, each with the
checking map[netip.Prefix]bool // function that ends it. stopped is true once StopChecking has been
// called, and checkGoroutines waits for the goroutine of each check.
checking map[netip.Prefix]context.CancelFunc
stopped bool
checkGoroutines sync.WaitGroup
// day is the day, in UTC, of the checks spent counts. // day is the day, in UTC, of the checks spent counts.
day civil.Date day civil.Date
spent int spent int
@@ -112,7 +138,7 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
params: params, params: params,
httpClient: &http.Client{}, httpClient: &http.Client{},
scores: scores, scores: scores,
checking: map[netip.Prefix]bool{}, checking: map[netip.Prefix]context.CancelFunc{},
} }
} }
@@ -122,11 +148,11 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
// addresses its request comes from. A client without one is checked in // addresses its request comes from. A client without one is checked in
// the background, by addr, the address its request came from, if // the background, by addr, the address its request came from, if
// offender, if it has committed an offence, unless its check is under // offender, if it has committed an offence, unless its check is under
// way, a check failed less than failureDelay ago, or the day's checks // way, a check failed less than failureDelay ago, the day's checks have
// have used up DailyBudget; Hit never waits for a check. The check that // used up DailyBudget, or StopChecking has been called; Hit never waits
// uses the budget up is logged and raised as a source_failure alert. ctx // for a check. The check that uses the budget up is logged and raised as a
// is the context of the client's request, and a check goes on after the // source_failure alert. ctx is the context of the client's request, and a
// request ends. // check goes on after the request ends, until StopChecking.
func (a *AbuseIPDB) Hit( func (a *AbuseIPDB) Hit(
ctx context.Context, client netip.Prefix, addr netip.Addr, offender bool, ctx context.Context, client netip.Prefix, addr netip.Addr, offender bool,
) (int64, bool) { ) (int64, bool) {
@@ -145,14 +171,21 @@ func (a *AbuseIPDB) Hit(
a.day, a.spent = today, 0 a.day, a.spent = today, 0
} }
check := offender && !a.checking[client] && !now.Before(a.retryAt) && _, checking := a.checking[client]
a.spent < a.params.DailyBudget
check := offender && !checking && !a.stopped &&
!now.Before(a.retryAt) && a.spent < a.params.DailyBudget
if check { if check {
a.checking[client] = true checkCtx, end := context.WithCancel(context.WithoutCancel(ctx))
a.checking[client] = end
a.checks++ a.checks++
a.spent++ a.spent++
go a.check(context.WithoutCancel(ctx), client, addr) a.checkGoroutines.Go(func() {
defer end()
a.check(checkCtx, client, addr)
})
} }
usedUp := check && a.spent == a.params.DailyBudget usedUp := check && a.spent == a.params.DailyBudget
@@ -167,6 +200,24 @@ func (a *AbuseIPDB) Hit(
return 0, false return 0, false
} }
// StopChecking ends the checks under way, and returns once their goroutines
// have. A check it ends gives no score and is no failure: it is neither
// counted, logged nor raised as an alert, and checks are not left alone
// after it. The ended check stays spent from the day's budget. After
// StopChecking, Hit checks no client.
func (a *AbuseIPDB) StopChecking() {
a.mu.Lock()
a.stopped = true
for _, end := range a.checking {
end()
}
a.mu.Unlock()
a.checkGoroutines.Wait()
}
// Checked returns how many checks were made. // Checked returns how many checks were made.
func (a *AbuseIPDB) Checked() int { func (a *AbuseIPDB) Checked() int {
a.mu.Lock() a.mu.Lock()
@@ -256,10 +307,45 @@ func (a *AbuseIPDB) LoadChecksSpent(checks ChecksSpent) {
a.day, a.spent = checks.Day, checks.Spent a.day, a.spent = checks.Day, checks.Spent
} }
// RunBlacklist fetches AbuseIPDB's blacklist at once, and again every
// BlacklistRefresh, until ctx is done, and gives the clients on each copy
// fetched to Blacklisted. While Key is unset it does nothing. A fetch that
// fails is logged and raised as a source_failure alert, unless it was cut
// off as ctx ended.
func (a *AbuseIPDB) RunBlacklist(ctx context.Context) {
if a.params.Key == "" {
return
}
for {
listed, err := a.fetchBlacklist(ctx)
switch {
case ctx.Err() != nil:
return
case err != nil:
a.alert("fetching AbuseIPDB's blacklist failed", err)
default:
a.params.Blacklisted(listed)
}
timer := time.NewTimer(a.params.BlacklistRefresh)
select {
case <-ctx.Done():
timer.Stop()
return
case <-timer.C:
}
}
}
// check checks client with AbuseIPDB by addr, one of its addresses, keeps // check checks client with AbuseIPDB by addr, one of its addresses, keeps
// the score as client's, and notes the check as no longer under way. A // the score as client's, and notes the check as no longer under way. A
// check that fails gives no score: it is counted, logged and raised as a // check that fails gives no score: it is counted, logged and raised as a
// source_failure alert, and no client is checked for failureDelay. // source_failure alert, and no client is checked for failureDelay. A check
// ended as ctx ends, by StopChecking, is no failure.
func (a *AbuseIPDB) check( func (a *AbuseIPDB) check(
ctx context.Context, ctx context.Context,
client netip.Prefix, client netip.Prefix,
@@ -267,24 +353,26 @@ func (a *AbuseIPDB) check(
) { ) {
score, err := a.ask(ctx, addr) score, err := a.ask(ctx, addr)
now := a.params.Now() now := a.params.Now()
isFailure := err != nil && ctx.Err() == nil
a.mu.Lock() a.mu.Lock()
delete(a.checking, client) delete(a.checking, client)
if err == nil { switch {
case err == nil:
a.scores.Add( a.scores.Add(
client, client,
Score{Client: client, Score: score, Fetched: now}, Score{Client: client, Score: score, Fetched: now},
) )
} else { case isFailure:
a.failures++ a.failures++
a.retryAt = now.Add(failureDelay) a.retryAt = now.Add(failureDelay)
} }
a.mu.Unlock() a.mu.Unlock()
if err != nil { if isFailure {
a.alert("checking a client with AbuseIPDB failed", err) a.alert("checking a client with AbuseIPDB failed", err)
} }
} }
@@ -344,6 +432,94 @@ func (a *AbuseIPDB) ask(
return *answer.Data.AbuseConfidenceScore, nil return *answer.Data.AbuseConfidenceScore, nil
} }
// fetchBlacklist fetches the clients on AbuseIPDB's blacklist scored
// Threshold or more, sending the key in the header Key. An answer other
// than 200, one longer than maxListBytes or without the list, an entry
// whose address does not read, and none within fetchTimeout, fail.
func (a *AbuseIPDB) fetchBlacklist(
ctx context.Context,
) ([]BlacklistEntry, error) {
ctx, cancel := context.WithTimeout(ctx, fetchTimeout)
defer cancel()
query := url.Values{
"confidenceMinimum": {strconv.FormatInt(a.params.Threshold, 10)},
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet,
a.params.BlacklistURL+"?"+query.Encode(), http.NoBody)
if err != nil {
return nil, fmt.Errorf("make the request: %w", err)
}
req.Header.Set("Key", a.params.Key)
req.Header.Set("Accept", "application/json")
res, err := a.httpClient.Do(req)
if err != nil {
return nil, fmt.Errorf("fetch the blacklist: %w", errors.Unwrap(err))
}
defer func() {
_ = res.Body.Close()
}()
if res.StatusCode != http.StatusOK {
return nil, fmt.Errorf("%w %s", errStatus, res.Status)
}
body, err := io.ReadAll(io.LimitReader(res.Body, maxListBytes+1))
if err != nil {
return nil, fmt.Errorf("read the blacklist: %w", err)
}
if len(body) > maxListBytes {
return nil, errTooLong
}
return parseBlacklist(body)
}
// parseBlacklist reads body, AbuseIPDB's answer with its blacklist.
func parseBlacklist(body []byte) ([]BlacklistEntry, error) {
var answer struct {
Data *[]struct {
IPAddress string `json:"ipAddress"`
AbuseConfidenceScore int64 `json:"abuseConfidenceScore"`
} `json:"data"`
}
err := json.Unmarshal(body, &answer)
if err != nil {
return nil, fmt.Errorf("read the blacklist: %w", err)
}
if answer.Data == nil {
return nil, errNoBlacklist
}
listed := make([]BlacklistEntry, 0, len(*answer.Data))
for i, entry := range *answer.Data {
addr, err := netip.ParseAddr(entry.IPAddress)
if err != nil || addr.Zone() != "" {
// The error leaves out what the entry holds, as it does a
// client's address.
return nil, fmt.Errorf(
"entry %d's ipAddress %w",
i+1,
errNotAddress,
)
}
listed = append(listed, BlacklistEntry{
Addr: addr.Unmap(), Score: entry.AbuseConfidenceScore,
})
}
return listed, nil
}
// alert raises a source_failure alert from AbuseIPDB with reason and err, // alert raises a source_failure alert from AbuseIPDB with reason and err,
// and logs them. // and logs them.
func (a *AbuseIPDB) alert(reason string, err error) { func (a *AbuseIPDB) alert(reason string, err error) {
+37
View File
@@ -394,6 +394,43 @@ func TestCheckNotAnsweredWithinTheTimeoutFailsAndHitNeverWaits(
}) })
} }
func TestStopCheckingEndsTheCheckUnderWayAsNoFailureAndChecksNoMore(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
queue := newQueue()
p := abuseIPDBParams()
p.Alerts = queue
abuseIPDB := &abuseIPDBStandIn{hanging: true}
checker := newAbuseIPDB(abuseIPDB, p)
wantScore(t, checker, suspect, true, 0, false)
synctest.Wait()
// The check is ended, not waited out, and is no failure.
began := time.Now()
checker.StopChecking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopChecking returned after %s, want at once", waited)
}
wantFailures(t, checker, 0)
if got := waiting(queue); len(got) != 0 {
t.Errorf("alerts waiting %+v, want none", got)
}
// An offender without a score is then not checked.
wantScore(t, checker, other, true, 0, false)
synctest.Wait()
wantChecked(t, abuseIPDB, suspect)
})
}
func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) { func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) {
t.Parallel() t.Parallel()
+252
View File
@@ -0,0 +1,252 @@
package reputation_test
import (
"context"
"fmt"
"io"
"net/http"
"net/netip"
"slices"
"strings"
"sync"
"testing"
"testing/synctest"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// The tests of the blacklist run in synctest bubbles, as those of the
// checks do, with AbuseIPDB a stand-in reached without the network.
// blacklistURL is where the tests fetch the blacklist from.
const blacklistURL = "https://abuseipdb.example/api/v2/blacklist"
func TestBlacklistIsFetchedAtOnceAndEveryRefresh(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
abuseIPDB := &blacklistStandIn{body: `{"meta":{},"data":[` +
`{"ipAddress":"203.0.113.9","abuseConfidenceScore":100},` +
`{"ipAddress":"::ffff:198.51.100.7","abuseConfidenceScore":95},` +
`{"ipAddress":"2001:db8::9","abuseConfidenceScore":90}]}`}
got := runBlacklist(t, abuseIPDB, key, newQueue())
want := []reputation.BlacklistEntry{
{Addr: netip.MustParseAddr("203.0.113.9"), Score: 100},
{Addr: netip.MustParseAddr("198.51.100.7"), Score: 95},
{Addr: netip.MustParseAddr("2001:db8::9"), Score: 90},
}
synctest.Wait()
got.want(t, want)
time.Sleep(time.Hour - time.Nanosecond)
synctest.Wait()
got.want(t, want)
time.Sleep(time.Nanosecond)
synctest.Wait()
got.want(t, want, want)
// Each fetch asks for the clients scored 90 or more, with the key
// in the header Key alone.
abuseIPDB.mu.Lock()
defer abuseIPDB.mu.Unlock()
for _, req := range abuseIPDB.asked {
if req.URL.String() != blacklistURL+"?confidenceMinimum=90" ||
req.Header.Get("Key") != key {
t.Errorf(
"asked %s with the key %q, want %s?confidenceMinimum=90 "+
"with %q",
req.URL,
req.Header.Get("Key"),
blacklistURL,
key,
)
}
}
})
}
func TestFailedBlacklistFetchIsAlertedAndTriedAgainAtTheNextRefresh(
t *testing.T,
) {
t.Parallel()
for _, tc := range []struct{ name, body, error string }{
{
"a refusal", "",
"the server answered 429 Too Many Requests",
},
{
"an answer without the list", `{"errors":[]}`,
"the answer gives no data",
},
{
"an entry that is not an address",
`{"data":[{"ipAddress":"203.0.113.9"},{"ipAddress":"example"}]}`,
"entry 2's ipAddress is not an address",
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
abuseIPDB := &blacklistStandIn{body: tc.body}
if tc.body == "" {
abuseIPDB.status = http.StatusTooManyRequests
}
queue := newQueue()
got := runBlacklist(t, abuseIPDB, key, queue)
synctest.Wait()
got.want(t)
alerted := waiting(queue)
if len(alerted) != 1 ||
alerted[0].Event != alerts.EventSourceFailure ||
alerted[0].Reason != "fetching AbuseIPDB's blacklist failed" ||
alerted[0].Detail["source"] != reputation.AbuseIPDBSource ||
alerted[0].Detail["error"] != tc.error {
t.Errorf(
"alerts %+v, want the failure %q",
alerted,
tc.error,
)
}
abuseIPDB.answer(`{"data":[]}`)
time.Sleep(time.Hour)
synctest.Wait()
got.want(t, []reputation.BlacklistEntry{})
})
})
}
}
func TestBlacklistIsNotFetchedWithoutAKey(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
abuseIPDB := &blacklistStandIn{body: `{"data":[]}`}
got := runBlacklist(t, abuseIPDB, "", newQueue())
time.Sleep(time.Hour)
synctest.Wait()
got.want(t)
if len(abuseIPDB.asked) != 0 {
t.Errorf("fetched the blacklist %d times, want none",
len(abuseIPDB.asked))
}
})
}
// blacklistStandIn is a stand-in for AbuseIPDB that answers each fetch of
// the blacklist with status, 200 when it is zero, and body, and keeps the
// requests.
type blacklistStandIn struct {
mu sync.Mutex
status int
body string
asked []*http.Request
}
// RoundTrip has the stand-in answer req, in place of the network.
func (s *blacklistStandIn) RoundTrip(
req *http.Request,
) (*http.Response, error) {
s.mu.Lock()
s.asked = append(s.asked, req)
status, body := s.status, s.body
s.mu.Unlock()
if status == 0 {
status = http.StatusOK
}
return &http.Response{
StatusCode: status,
Status: fmt.Sprintf("%d %s", status, http.StatusText(status)),
Header: http.Header{},
Body: io.NopCloser(strings.NewReader(body)),
Request: req,
}, nil
}
// answer has the stand-in answer each fetch with 200 and body.
func (s *blacklistStandIn) answer(body string) {
s.mu.Lock()
defer s.mu.Unlock()
s.status, s.body = 0, body
}
// blacklisted holds the clients on each blacklist given to Blacklisted.
type blacklisted struct {
mu sync.Mutex
fetched [][]reputation.BlacklistEntry
}
// want checks the clients on each blacklist given so far.
func (b *blacklisted) want(
t *testing.T,
want ...[]reputation.BlacklistEntry,
) {
t.Helper()
b.mu.Lock()
defer b.mu.Unlock()
if !slices.EqualFunc(b.fetched, want, slices.Equal) {
t.Errorf("blacklists %v, want %v", b.fetched, want)
}
}
// runBlacklist runs RunBlacklist, until the test ends, with the key
// accountKey, the threshold 90 and a refresh of an hour, fetching from
// abuseIPDB and raising alerts into queue, and returns what it gives
// Blacklisted.
func runBlacklist(
t *testing.T, abuseIPDB *blacklistStandIn, accountKey string,
queue *alerts.Queue,
) *blacklisted {
t.Helper()
got := &blacklisted{}
p := abuseIPDBParams()
p.Key = accountKey
p.BlacklistURL = blacklistURL
p.Threshold = 90
p.BlacklistRefresh = time.Hour
p.Alerts = queue
p.Blacklisted = func(listed []reputation.BlacklistEntry) {
got.mu.Lock()
defer got.mu.Unlock()
got.fetched = append(got.fetched, listed)
}
checker := reputation.NewAbuseIPDB(p)
checker.SetTransport(abuseIPDB)
ctx, cancel := context.WithCancel(t.Context())
done := make(chan struct{})
go func() {
checker.RunBlacklist(ctx)
close(done)
}()
t.Cleanup(func() {
cancel()
<-done
})
return got
}
+67 -17
View File
@@ -16,7 +16,6 @@ import (
"github.com/hashicorp/golang-lru/v2/simplelru" "github.com/hashicorp/golang-lru/v2/simplelru"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
) )
const ( const (
@@ -32,6 +31,8 @@ const (
// it fails, and no client is checked with AbuseIPDB after a check // it fails, and no client is checked with AbuseIPDB after a check
// fails, so that a source refusing them is not asked on every request. // fails, so that a source refusing them is not asked on every request.
failureDelay = time.Minute failureDelay = time.Minute
// maskedKey is what MaskZoneKey shows in place of a zone's key.
maskedKey = "********"
) )
var ( var (
@@ -82,8 +83,12 @@ type DNSBL struct {
// verdicts are by query. Each is added as it is fetched and never moved // verdicts are by query. Each is added as it is fetched and never moved
// up, so that the one fetched longest ago is the first dropped. // up, so that the one fetched longest ago is the first dropped.
verdicts *simplelru.LRU[query, Verdict] verdicts *simplelru.LRU[query, Verdict]
// asking are the queries under way. // asking are the queries under way, each with the function that ends
asking map[query]bool // it. stopped is true once StopAsking has been called, and
// queryGoroutines waits for the goroutine of each query.
asking map[query]context.CancelFunc
stopped bool
queryGoroutines sync.WaitGroup
// queries and failures count, by zone, the queries made and those that // queries and failures count, by zone, the queries made and those that
// failed, and retryAt is when a zone whose last query failed may be // failed, and retryAt is when a zone whose last query failed may be
// asked again. // asked again.
@@ -120,13 +125,31 @@ func NewDNSBL(params DNSBLParams) *DNSBL {
params: params, params: params,
resolver: resolver, resolver: resolver,
verdicts: verdicts, verdicts: verdicts,
asking: map[query]bool{}, asking: map[query]context.CancelFunc{},
queries: map[string]int{}, queries: map[string]int{},
failures: map[string]int{}, failures: map[string]int{},
retryAt: map[string]time.Time{}, retryAt: map[string]time.Time{},
} }
} }
// MaskZoneKey returns zone with ******** in place of its key, if it is a
// zone of Spamhaus's keyed query service, a name under dq.spamhaus.net,
// such as <key>.xbl.dq.spamhaus.net, whose first label is the key. Any
// other zone it returns as it is. A zone is shown so wherever it leaves
// the process: in the log, the alerts and the metrics.
func MaskZoneKey(zone string) string {
// DNS names ignore case, and a name may be written with a dot at its
// end.
name := strings.TrimSuffix(strings.ToLower(zone), ".")
if !strings.HasSuffix(name, ".dq.spamhaus.net") {
return zone
}
_, rest, _ := strings.Cut(zone, ".")
return maskedKey + "." + rest
}
// Zones returns the zones, in the order SWW_WAF_DNSBL_ZONES names them. // Zones returns the zones, in the order SWW_WAF_DNSBL_ZONES names them.
func (d *DNSBL) Zones() []string { func (d *DNSBL) Zones() []string {
return slices.Clone(d.params.Zones) return slices.Clone(d.params.Zones)
@@ -134,13 +157,14 @@ func (d *DNSBL) Zones() []string {
// ListedBy returns the zones whose verdict on addr, a client's address, // ListedBy returns the zones whose verdict on addr, a client's address,
// lists it, in the order SWW_WAF_DNSBL_ZONES names them, each with its key // lists it, in the order SWW_WAF_DNSBL_ZONES names them, each with its key
// masked, as config.MaskZoneKey masks it, since they go to the request // masked, as MaskZoneKey masks it, since they go to the request
// log, the alerts and the metrics. A verdict is used until CacheTTL has // log, the alerts and the metrics. A verdict is used until CacheTTL has
// passed since it was fetched. Each zone without one is asked about addr // passed since it was fetched. Each zone without one is asked about addr
// in the background, unless a query about addr to it is under way, the // in the background, unless a query about addr to it is under way, the
// zone is left alone after a failure, or maxQueries are under way; // zone is left alone after a failure, maxQueries are under way, or
// ListedBy never waits for a query. ctx is the context of the client's // StopAsking has been called; ListedBy never waits for a query. ctx is the
// request, and a query goes on after the request ends. // context of the client's request, and a query goes on after the request
// ends, until StopAsking.
func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string { func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
d.mu.Lock() d.mu.Lock()
defer d.mu.Unlock() defer d.mu.Unlock()
@@ -153,23 +177,47 @@ func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
q := query{zone: zone, client: addr} q := query{zone: zone, client: addr}
kept, found := d.verdicts.Peek(q) kept, found := d.verdicts.Peek(q)
_, asking := d.asking[q]
switch { switch {
case found && now.Sub(kept.Fetched) < d.params.CacheTTL: case found && now.Sub(kept.Fetched) < d.params.CacheTTL:
if kept.Listed { if kept.Listed {
listedBy = append(listedBy, config.MaskZoneKey(zone)) listedBy = append(listedBy, MaskZoneKey(zone))
} }
case !d.asking[q] && !now.Before(d.retryAt[zone]) && len(d.asking) < maxQueries: case !asking && !d.stopped && !now.Before(d.retryAt[zone]) &&
d.asking[q] = true len(d.asking) < maxQueries:
queryCtx, end := context.WithCancel(context.WithoutCancel(ctx))
d.asking[q] = end
d.queries[zone]++ d.queries[zone]++
go d.ask(context.WithoutCancel(ctx), q) d.queryGoroutines.Go(func() {
defer end()
d.ask(queryCtx, q)
})
} }
} }
return listedBy return listedBy
} }
// StopAsking ends the queries under way, and returns once their
// goroutines have. A query it ends gives no verdict and is no failure: it
// is neither counted, logged nor raised as an alert, and the zone is not
// left alone after it. After StopAsking, ListedBy asks no zone.
func (d *DNSBL) StopAsking() {
d.mu.Lock()
d.stopped = true
for _, end := range d.asking {
end()
}
d.mu.Unlock()
d.queryGoroutines.Wait()
}
// Queries returns how many queries were made to zone. // Queries returns how many queries were made to zone.
func (d *DNSBL) Queries(zone string) int { func (d *DNSBL) Queries(zone string) int {
d.mu.Lock() d.mu.Lock()
@@ -238,30 +286,32 @@ func (d *DNSBL) Load(verdicts []Verdict) {
// query as no longer under way. A query that fails gives no verdict: it // query as no longer under way. A query that fails gives no verdict: it
// is counted, logged and raised as a source_failure alert, which show the // is counted, logged and raised as a source_failure alert, which show the
// zone with its key masked, and the zone is not asked again for // zone with its key masked, and the zone is not asked again for
// failureDelay. // failureDelay. A query ended as ctx ends, by StopAsking, is no failure.
func (d *DNSBL) ask(ctx context.Context, q query) { func (d *DNSBL) ask(ctx context.Context, q query) {
listed, err := d.lookUp(ctx, q) listed, err := d.lookUp(ctx, q)
now := d.params.Now() now := d.params.Now()
isFailure := err != nil && ctx.Err() == nil
d.mu.Lock() d.mu.Lock()
delete(d.asking, q) delete(d.asking, q)
if err == nil { switch {
case err == nil:
d.verdicts.Add(q, Verdict{ d.verdicts.Add(q, Verdict{
Zone: q.zone, Client: q.client, Listed: listed, Fetched: now, Zone: q.zone, Client: q.client, Listed: listed, Fetched: now,
}) })
} else { case isFailure:
d.failures[q.zone]++ d.failures[q.zone]++
d.retryAt[q.zone] = now.Add(failureDelay) d.retryAt[q.zone] = now.Add(failureDelay)
} }
d.mu.Unlock() d.mu.Unlock()
if err != nil { if isFailure {
const failed = "asking a DNSBL zone failed" const failed = "asking a DNSBL zone failed"
shown := config.MaskZoneKey(q.zone) shown := MaskZoneKey(q.zone)
// Raised before it is logged, so that the alert is there once the // Raised before it is logged, so that the alert is there once the
// log line is. // log line is.
+72 -5
View File
@@ -31,10 +31,12 @@ import (
// this package: Go's resolver counts the queries under way in one // this package: Go's resolver counts the queries under way in one
// sync.WaitGroup for the whole process, and the process fails when // sync.WaitGroup for the whole process, and the process fails when
// queries from two bubbles, or from a bubble and from outside one, are // queries from two bubbles, or from a bubble and from outside one, are
// under way at once. TestMain has the resolver make its configuration, // under way at once. The one test outside a bubble is parallel, so it
// which it makes on its first query, outside every bubble, since the // starts only once every test that is not parallel has ended; no other
// configuration holds a channel, which the bubble it was made in would // parallel test of this package queries a resolver. TestMain has the
// keep to itself. // resolver make its configuration, which it makes on its first query,
// outside every bubble, since the configuration holds a channel, which
// the bubble it was made in would keep to itself.
const ( const (
// zone and otherZone are the DNSBL zones the tests name. // zone and otherZone are the DNSBL zones the tests name.
@@ -208,6 +210,45 @@ func TestQueryNotAnsweredWithinTheTimeoutFails(t *testing.T) {
}) })
} }
//
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestStopAskingEndsTheQueryUnderWayAsNoFailureAndAsksNoMore(
t *testing.T,
) {
synctest.Test(t, func(t *testing.T) {
queue := newQueue()
p := dnsblParams(zone)
p.Alerts = queue
resolver := &resolverStandIn{hanging: true}
dnsbl := newDNSBL(resolver, p)
wantZones(t, dnsbl, listed)
synctest.Wait()
// The query is ended, not waited out, and is no failure.
began := time.Now()
dnsbl.StopAsking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopAsking returned after %s, want at once", waited)
}
wantQueries(t, dnsbl, 1, 0)
if got := waiting(queue); len(got) != 0 {
t.Errorf("alerts waiting %+v, want none", got)
}
// A client without a verdict is then asked about in no zone.
wantZones(t, dnsbl, unlisted)
synctest.Wait()
wantQueries(t, dnsbl, 1, 0)
wantAsked(t, resolver, listedName)
waitForTheResolver()
})
}
// //
//nolint:paralleltest // one at a time, as the comment at the top of this file says //nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestZoneThatFailsOrRefusesGivesNoVerdictAndIsLeftAloneForAMinute( func TestZoneThatFailsOrRefusesGivesNoVerdictAndIsLeftAloneForAMinute(
@@ -525,8 +566,9 @@ func TestAtMost100000VerdictsKeptTheOneFetchedLongestAgoDroppedFirst(
} }
} }
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestQueriesGoToTheResolverSWWDNSBLResolverNames(t *testing.T) { func TestQueriesGoToTheResolverSWWDNSBLResolverNames(t *testing.T) {
t.Parallel()
resolver := &resolverStandIn{answers: map[string]answer{ resolver := &resolverStandIn{answers: map[string]answer{
listedName: {addrs: []string{listing}}, listedName: {addrs: []string{listing}},
}} }}
@@ -818,3 +860,28 @@ func wantAsked(t *testing.T, resolver *resolverStandIn, want ...string) {
t.Errorf("asked about %v, want %v", got, want) t.Errorf("asked about %v, want %v", got, want)
} }
} }
func TestMaskZoneKeyMasksTheFirstLabelOfAZoneUnderDqSpamhausNet(
t *testing.T,
) {
t.Parallel()
const (
dronebl = "dnsbl.dronebl.org"
spamhaus = "abcdefghijklmnopqrstuvwxyz.xbl.dq.spamhaus.net"
spamhausMasked = "********.xbl.dq.spamhaus.net"
)
for zone, want := range map[string]string{
spamhaus: spamhausMasked,
spamhaus + ".": spamhausMasked + ".",
"KEY.ZEN.DQ.SPAMHAUS.NET": "********.ZEN.DQ.SPAMHAUS.NET",
dronebl: dronebl,
"dq.spamhaus.net": "dq.spamhaus.net",
spamhaus + ".example": spamhaus + ".example",
} {
if got := reputation.MaskZoneKey(zone); got != want {
t.Errorf("MaskZoneKey(%q) is %q, want %q", zone, got, want)
}
}
}
+77 -10
View File
@@ -7,7 +7,10 @@
// SWW_WAF_DNSBL_ZONES about clients, and keeps their verdicts, and checks // SWW_WAF_DNSBL_ZONES about clients, and keeps their verdicts, and checks
// clients with AbuseIPDB, and keeps their scores and the checks spent // clients with AbuseIPDB, and keeps their scores and the checks spent
// today. The state package writes all of these to files and reads them // today. The state package writes all of these to files and reads them
// from there, so that a restart keeps them too. // from there, so that a restart keeps them too. It fetches AbuseIPDB's
// blacklist, which it does not keep. Last, it reads the netblocks, AS
// numbers and percentages that both the settings and the lists hold, and
// masks the keys of DNSBL zones.
package reputation package reputation
import ( import (
@@ -19,12 +22,12 @@ import (
"net/http" "net/http"
"net/netip" "net/netip"
"slices" "slices"
"strconv"
"strings" "strings"
"sync" "sync"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
) )
const ( const (
@@ -39,15 +42,28 @@ const (
) )
var ( var (
errStatus = errors.New("the server answered") errStatus = errors.New("the server answered")
errTooLong = errors.New("the list is longer than 16 MiB") errTooLong = errors.New("the list is longer than 16 MiB")
errNotNetblock = errors.New( errNotNetblockLine = errors.New(
"is not an address or a netblock, such as 192.0.2.0/24", "is not an address or a netblock, such as 192.0.2.0/24",
) )
errNotASNPercent = errors.New( errNotASNPercent = errors.New(
"is not an AS number, : and a percentage, such as AS64496:50") "is not an AS number, : and a percentage, such as AS64496:50")
errNotNetblock = errors.New(
"is not a netblock such as 10.0.0.0/8, or an address")
errNotASN = errors.New("is not an AS number such as AS64496")
errNotPercent = errors.New(
"is not a percentage, a whole number from 0 to 100",
)
) )
// TagList is an address list of SWW_WAF_TAG_LISTS: the tag it adds to the
// requests of the clients it holds, and the URL it is fetched from.
type TagList struct {
Name string
URL string
}
// List is a list as lists.json holds it: the URL it is fetched from, // List is a list as lists.json holds it: the URL it is fetched from,
// when it was last tried, the fetch failed or not, and its last good copy: // when it was last tried, the fetch failed or not, and its last good copy:
// when that was fetched, and its lines, as fetched, comment lines // when that was fetched, and its lines, as fetched, comment lines
@@ -67,7 +83,7 @@ type Params struct {
BlocklistURLs []string BlocklistURLs []string
ASNLimitPercentURL string ASNLimitPercentURL string
// TagLists are the address lists that add a tag (SWW_WAF_TAG_LISTS). // TagLists are the address lists that add a tag (SWW_WAF_TAG_LISTS).
TagLists []config.TagList TagLists []TagList
// Refresh is how long after a list was last fetched or tried it is // Refresh is how long after a list was last fetched or tried it is
// fetched again (SWW_WAF_BLOCKLIST_REFRESH). // fetched again (SWW_WAF_BLOCKLIST_REFRESH).
Refresh time.Duration Refresh time.Duration
@@ -464,7 +480,11 @@ func parseNetblocks(lines []string) (entries, error) {
netblock, ok := parseNetblock(text) netblock, ok := parseNetblock(text)
if !ok { if !ok {
return entries{}, fmt.Errorf("line %d %w", i+1, errNotNetblock) return entries{}, fmt.Errorf(
"line %d %w",
i+1,
errNotNetblockLine,
)
} }
found.netblocks[netblock] = true found.netblocks[netblock] = true
@@ -484,7 +504,7 @@ func parseNetblocks(lines []string) (entries, error) {
// stands for, 192.0.2.0/24, and a mapped netblock shorter than /96, which // stands for, 192.0.2.0/24, and a mapped netblock shorter than /96, which
// stands for none, is refused. // stands for none, is refused.
func parseNetblock(text string) (netip.Prefix, bool) { func parseNetblock(text string) (netip.Prefix, bool) {
netblock, err := config.ParseNetblock(text) netblock, err := ParseNetblock(text)
if err != nil { if err != nil {
return netip.Prefix{}, false return netip.Prefix{}, false
} }
@@ -519,9 +539,9 @@ func parsePercents(lines []string) (entries, error) {
} }
asnText, percentText, _ := strings.Cut(text, ":") asnText, percentText, _ := strings.Cut(text, ":")
asn, asnErr := config.ParseASN(asnText) asn, asnErr := ParseASN(asnText)
percent, percentErr := config.ParsePercent(percentText) percent, percentErr := ParsePercent(percentText)
if asnErr != nil || percentErr != nil { if asnErr != nil || percentErr != nil {
return entries{}, fmt.Errorf("line %d %w", i+1, errNotASNPercent) return entries{}, fmt.Errorf("line %d %w", i+1, errNotASNPercent)
} }
@@ -535,6 +555,53 @@ func parsePercents(lines []string) (entries, error) {
return found, nil return found, nil
} }
// ParseNetblock reads a netblock in CIDR form, such as 10.0.0.0/8. A bare
// address is a netblock of that address alone, a /32 or a /128. A
// blocklist's lines are read with it too.
func ParseNetblock(value string) (netip.Prefix, error) {
if strings.Contains(value, "/") {
netblock, err := netip.ParsePrefix(value)
if err != nil {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netblock.Masked(), nil
}
addr, err := netip.ParseAddr(value)
if err != nil || addr.Zone() != "" {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netip.PrefixFrom(addr, addr.BitLen()), nil
}
// ParseASN reads an AS number such as AS64496, in either case, and
// returns it as the lookup gives it: AS and the number, in capitals and
// without leading zeros. The file SWW_WAF_ASN_LIMIT_PERCENT_URL names is
// read with it too.
func ParseASN(value string) (string, error) {
digits, hasAS := strings.CutPrefix(strings.ToUpper(value), "AS")
number, err := strconv.ParseUint(digits, 10, 32)
if !hasAS || err != nil {
return "", fmt.Errorf("%q %w", value, errNotASN)
}
return "AS" + strconv.FormatUint(number, 10), nil
}
// ParsePercent reads a percentage, a whole number from 0 to 100. The file
// SWW_WAF_ASN_LIMIT_PERCENT_URL names is read with it too.
func ParsePercent(value string) (int64, error) {
percent, err := strconv.ParseInt(value, 10, 64)
if err != nil || percent < 0 || percent > 100 {
return 0, fmt.Errorf("%q %w", value, errNotPercent)
}
return percent, nil
}
// withoutComment returns line without anything after a ; or a #, and // withoutComment returns line without anything after a ; or a #, and
// without the spaces around what is left. // without the spaces around what is left.
func withoutComment(line string) string { func withoutComment(line string) string {
+4 -33
View File
@@ -1,7 +1,7 @@
// Package requestlog writes the lines sww prints on stdout: one // Package requestlog writes the request log sww prints on stdout: JSON
// JSON object per request, two from waf and a short one from rproxy, // lines marked "type":"request", at most two from waf and a short one
// marked "type":"request", and the process's own messages as JSON lines marked // from rproxy for each request. Package processlog writes the process's
// "type":"process". // own messages.
package requestlog package requestlog
import ( import (
@@ -9,7 +9,6 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"io" "io"
"log/slog"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
@@ -300,31 +299,3 @@ func Milliseconds(d time.Duration) float64 {
time.Millisecond/time.Microsecond, time.Millisecond/time.Microsecond,
) )
} }
// NewProcessLogger returns the logger for the process's own messages:
// JSON lines on w, marked "type":"process", with the time in the same form
// as a request line's, and instanceName, SWW_WAF_INSTANCE_NAME, as instance.
// It writes only the messages at level, SWW_WAF_LOG_LEVEL, or more severe;
// the request lines Write writes are never held back.
func NewProcessLogger(
w io.Writer,
instanceName string,
level slog.Level,
) *slog.Logger {
handler := slog.NewJSONHandler(w, &slog.HandlerOptions{
Level: level,
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(
slog.TimeKey,
FormatTime(attr.Value.Time()),
)
}
return attr
},
})
return slog.New(handler).
With("type", "process", "instance", instanceName)
}
-87
View File
@@ -3,8 +3,6 @@ package requestlog_test
import ( import (
"bytes" "bytes"
"encoding/json" "encoding/json"
"log/slog"
"slices"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -134,88 +132,3 @@ func TestWriteRProxyLineWritesOneJSONLineWithOnlyItsFields(t *testing.T) {
t.Errorf("wrote %s, want %s", out.String(), want) t.Errorf("wrote %s, want %s", out.String(), want)
} }
} }
func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
t.Parallel()
var out bytes.Buffer
requestlog.NewProcessLogger(&out, "host1/blog", slog.LevelInfo).
Info("starting",
"version", "v1")
var fields map[string]any
err := json.Unmarshal(out.Bytes(), &fields)
if err != nil {
t.Fatalf("decode %q: %v", out.String(), err)
}
if fields["type"] != "process" ||
fields["instance"] != "host1/blog" ||
fields["msg"] != "starting" ||
fields["level"] != "INFO" ||
fields["version"] != "v1" {
t.Errorf("process line %v", fields)
}
timeText, _ := fields["time"].(string)
logged, err := time.Parse(time.RFC3339, timeText)
if err != nil || !strings.HasSuffix(timeText, "Z") ||
len(timeText) != len("2006-01-02T15:04:05.000Z") ||
time.Since(logged) > time.Minute {
t.Errorf(
"process line time %q, want now in UTC with milliseconds",
timeText,
)
}
}
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
t.Parallel()
levels := []slog.Level{
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
}
for i, level := range levels {
t.Run(level.String(), func(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processLog := requestlog.NewProcessLogger(
&out,
"host1/blog",
level,
)
for _, at := range levels {
processLog.Log(t.Context(), at, "message")
}
var got, want []string
for line := range strings.Lines(out.String()) {
var fields struct {
Level string `json:"level"`
}
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("decode %q: %v", line, err)
}
got = append(got, fields.Level)
}
for _, written := range levels[i:] {
want = append(want, written.String())
}
if !slices.Equal(got, want) {
t.Errorf("lines at %v, want %v", got, want)
}
})
}
}
+43 -12
View File
@@ -99,8 +99,12 @@ type Client struct {
// answers are by client. Each is added as it comes and never moved up, // answers are by client. Each is added as it comes and never moved up,
// so that the one that came longest ago is the first dropped. // so that the one that came longest ago is the first dropped.
answers *simplelru.LRU[netip.Prefix, Answer] answers *simplelru.LRU[netip.Prefix, Answer]
// asking are the clients whose query is under way. // asking are the clients whose query is under way, each with the
asking map[netip.Prefix]bool // function that ends it. stopped is true once StopAsking has been
// called, and queryGoroutines waits for the goroutine of each query.
asking map[netip.Prefix]context.CancelFunc
stopped bool
queryGoroutines sync.WaitGroup
// retryAt is when routewatch may be asked again after a query failed. // retryAt is when routewatch may be asked again after a query failed.
retryAt time.Time retryAt time.Time
} }
@@ -116,7 +120,7 @@ func New(params Params) *Client {
params: params, params: params,
httpClient: &http.Client{}, httpClient: &http.Client{},
answers: answers, answers: answers,
asking: map[netip.Prefix]bool{}, asking: map[netip.Prefix]context.CancelFunc{},
} }
} }
@@ -124,8 +128,9 @@ func New(params Params) *Client {
// IPv6 group, or why it has no route. It never waits for a query. A client // IPv6 group, or why it has no route. It never waits for a query. A client
// without an answer is asked about in the background, by addr, the address its // without an answer is asked about in the background, by addr, the address its
// request came from, unless its query is under way, a query failed less than // request came from, unless its query is under way, a query failed less than
// failureKept ago, or maxQueries are under way. ctx is the context of the // failureKept ago, maxQueries are under way, or StopAsking has been called. ctx
// client's request, and a query goes on after the request ends. // is the context of the client's request, and a query goes on after the request
// ends, until StopAsking.
func (c *Client) Route( func (c *Client) Route(
ctx context.Context, client netip.Prefix, addr netip.Addr, ctx context.Context, client netip.Prefix, addr netip.Addr,
) (Route, Missing) { ) (Route, Missing) {
@@ -147,15 +152,38 @@ func (c *Client) Route(
return Route{}, RequestFailed return Route{}, RequestFailed
} }
if !c.asking[client] && len(c.asking) < maxQueries { _, asking := c.asking[client]
c.asking[client] = true if !asking && !c.stopped && len(c.asking) < maxQueries {
queryCtx, end := context.WithCancel(context.WithoutCancel(ctx))
c.asking[client] = end
go c.ask(context.WithoutCancel(ctx), client, addr) c.queryGoroutines.Go(func() {
defer end()
c.ask(queryCtx, client, addr)
})
} }
return Route{}, NoAnswerYet return Route{}, NoAnswerYet
} }
// StopAsking ends the queries under way, and returns once their goroutines
// have. A query it ends gives no answer and is no failure: it is not logged,
// and routewatch is not left alone after it. After StopAsking, Route asks
// routewatch nothing.
func (c *Client) StopAsking() {
c.mu.Lock()
c.stopped = true
for _, end := range c.asking {
end()
}
c.mu.Unlock()
c.queryGoroutines.Wait()
}
// Snapshot returns every answer still in use, sorted by client, as // Snapshot returns every answer still in use, sorted by client, as
// routewatch.json lists them. // routewatch.json lists them.
func (c *Client) Snapshot() []Answer { func (c *Client) Snapshot() []Answer {
@@ -205,7 +233,8 @@ func (c *Client) Load(answers []Answer) {
// ask asks routewatch about addr, keeps the answer as client's, and notes the // ask asks routewatch about addr, keeps the answer as client's, and notes the
// query as no longer under way. A query that fails gives no answer, and // query as no longer under way. A query that fails gives no answer, and
// routewatch is not asked again for failureKept. Only the first failure of // routewatch is not asked again for failureKept. Only the first failure of
// those under way is logged. // those under way is logged. A query ended as ctx ends, by StopAsking, is no
// failure.
func (c *Client) ask( func (c *Client) ask(
ctx context.Context, ctx context.Context,
client netip.Prefix, client netip.Prefix,
@@ -213,18 +242,20 @@ func (c *Client) ask(
) { ) {
route, err := c.query(ctx, addr) route, err := c.query(ctx, addr)
now := c.params.Now() now := c.params.Now()
isFailure := err != nil && ctx.Err() == nil
c.mu.Lock() c.mu.Lock()
delete(c.asking, client) delete(c.asking, client)
firstFailure := err != nil && !now.Before(c.retryAt) firstFailure := isFailure && !now.Before(c.retryAt)
if err == nil { switch {
case err == nil:
c.answers.Add(client, Answer{ c.answers.Add(client, Answer{
Client: client, Netblock: route.Netblock, ASN: route.ASN, Answered: now, Client: client, Netblock: route.Netblock, ASN: route.ASN, Answered: now,
}) })
} else { case isFailure:
c.retryAt = now.Add(failureKept) c.retryAt = now.Add(failureKept)
} }
+48
View File
@@ -361,6 +361,54 @@ func TestHangingRoutewatchNeverDelaysAndFailsTheQuery(t *testing.T) {
}) })
} }
func TestStopAskingEndsTheQueryUnderWayAsNoFailureAndAsksNoMore(
t *testing.T,
) {
t.Parallel()
// In a synctest bubble, as TestHangingRoutewatchNeverDelaysAndFailsTheQuery
// says.
synctest.Test(t, func(t *testing.T) {
f := &fixture{t: t, log: &syncBuffer{}}
f.client = routewatch.New(routewatch.Params{
URL: silentURL, Now: time.Now,
ProcessLog: slog.New(slog.NewTextHandler(f.log, nil)),
})
f.route()
synctest.Wait()
// The query is ended, not waited out.
began := time.Now()
f.client.StopAsking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopAsking returned after %s, want at once", waited)
}
// A query started now would fail within waitLimit. None is, and the
// one ended is no failure: nothing is logged, and routewatch is not
// left alone.
for range 2 {
if _, missing := f.route(); missing != routewatch.NoAnswerYet {
t.Errorf(
"Route gave %q, want %q",
missing,
routewatch.NoAnswerYet,
)
}
time.Sleep(waitLimit)
synctest.Wait()
}
if f.log.String() != "" {
t.Errorf("process log %q, want nothing", f.log.String())
}
})
}
func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) { func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) {
t.Parallel() t.Parallel()
+16 -7
View File
@@ -2,6 +2,7 @@ package rproxy
import ( import (
"bufio" "bufio"
"context"
"errors" "errors"
"io" "io"
"net" "net"
@@ -22,7 +23,9 @@ type requestBody struct {
// body is the visitor's body, ending in an *http.MaxBytesError past // body is the visitor's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_REQUEST_SIZE. // SWW_RPROXY_MAX_REQUEST_SIZE.
body io.ReadCloser body io.ReadCloser
rq *request // refuse refuses the request with a status, as the request's own refuse
// does.
refuse func(status int)
// held is the part read before the request goes to the app, which Read gives // held is the part read before the request goes to the app, which Read gives
// first. // first.
held []byte held []byte
@@ -68,7 +71,7 @@ func (b *requestBody) Read(p []byte) (int, error) {
case errors.Is(err, io.EOF): case errors.Is(err, io.EOF):
b.received.Store(true) b.received.Store(true)
case errors.As(err, &tooLarge): case errors.As(err, &tooLarge):
b.rq.refuse(http.StatusRequestEntityTooLarge) b.refuse(http.StatusRequestEntityTooLarge)
} }
return n, err return n, err
@@ -86,7 +89,13 @@ type responseBody struct {
// body is the app's body, ending in an *http.MaxBytesError past // body is the app's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_RESPONSE_SIZE. // SWW_RPROXY_MAX_RESPONSE_SIZE.
body io.ReadCloser body io.ReadCloser
rq *request // refuse refuses the request with a status, as the request's own refuse
// does.
refuse func(status int)
// responseReceived is called once the app's whole answer has arrived.
responseReceived func()
// ctx is the visitor's request's context.
ctx context.Context //nolint:containedctx // the body ends with the request
} }
// Read reads from the app's body. // Read reads from the app's body.
@@ -100,15 +109,15 @@ func (b *responseBody) Read(p []byte) (int, error) {
switch { switch {
case errors.Is(err, io.EOF): case errors.Is(err, io.EOF):
b.rq.responseReceived() b.responseReceived()
case errors.As(err, &tooLarge): case errors.As(err, &tooLarge):
b.rq.refuse(http.StatusBadGateway) b.refuse(http.StatusBadGateway)
return n, errResponseTooLarge return n, errResponseTooLarge
case b.rq.in.Context().Err() == nil: case b.ctx.Err() == nil:
// The answer broke off, not because the visitor went away. If a time // The answer broke off, not because the visitor went away. If a time
// limit cut it, that refusal came first and is the one kept. // limit cut it, that refusal came first and is the one kept.
b.rq.refuse(http.StatusBadGateway) b.refuse(http.StatusBadGateway)
} }
return n, err return n, err
-10
View File
@@ -7,16 +7,6 @@ import (
"strings" "strings"
) )
// The headers rproxy sets on the requests it passes to the app. Go writes every
// header name in this form, as it sends it and as it receives it, so
// X-Client-ASN arrives as X-Client-Asn, and Del removes a visitor's own whatever
// their case; header names are not case-sensitive.
const (
requestIDHeader = "X-Request-ID"
asnHeader = "X-Client-Asn"
countryHeader = "X-Client-Country"
)
// peerAddress is the address of the request's TCP peer, normally traefik, without // peerAddress is the address of the request's TCP peer, normally traefik, without
// any IPv6 zone, which waf refuses. // any IPv6 zone, which waf refuses.
func peerAddress(r *http.Request) netip.Addr { func peerAddress(r *http.Request) netip.Addr {
+6
View File
@@ -2,6 +2,12 @@ package rproxy
import "time" import "time"
// StopBreakerClock has the circuit breaker of s read at as the time, so that how
// long requests take cannot matter to it. It is called before s serves.
func (s *Server) StopBreakerClock(at time.Time) {
s.h.breakerClock = func() time.Time { return at }
}
// Breaker is the circuit breaker, for tests that give it the time. // Breaker is the circuit breaker, for tests that give it the time.
type Breaker struct { type Breaker struct {
b breaker b breaker
+75 -13
View File
@@ -18,9 +18,9 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// post is the start of a POST whose headers end with what follows it. // post is the start of a POST whose headers end with what follows it.
@@ -51,11 +51,11 @@ func TestServerHasTheFixedLimits(t *testing.T) {
server := rproxy.New(rproxy.Params{ server := rproxy.New(rproxy.Params{
Config: cfg, Config: cfg,
RequestLog: io.Discard, RequestLog: io.Discard,
ProcessLog: requestlog.NewProcessLogger( ProcessLog: processlog.New(processlog.Params{
io.Discard, Output: io.Discard,
"test", InstanceName: "test",
slog.LevelInfo, Level: slog.LevelInfo,
), }),
}) })
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 || if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
@@ -71,6 +71,59 @@ func TestServerHasTheFixedLimits(t *testing.T) {
} }
} }
// getOfSize is a GET whose request line and headers, the blank line that ends
// them included, are size bytes long.
func getOfSize(size int) string {
const start, end = "GET / HTTP/1.1\r\nHost: site.example\r\nX-Pad: ", "\r\n\r\n"
return start + strings.Repeat("x", size-len(start)-len(end)) + end
}
func TestARequestLineAndHeadersOf32KiBReachTheApp(t *testing.T) {
t.Parallel()
standIn := startWAF(t, allow)
app := startApp(t, greet)
addr, _ := startRProxy(t, standIn.URL, app.URL, nil)
conn := dial(t, addr)
send(t, conn, getOfSize(32<<10))
if status := readStatus(t, conn); status != http.StatusOK {
t.Errorf("status %d, want the app's 200", status)
}
}
func TestARequestLineAndHeadersOver32KiBAreRefusedAndTheConnectionClosed(
t *testing.T,
) {
t.Parallel()
standIn := startWAF(t, allow)
app := startApp(t, func(http.ResponseWriter, *http.Request) {
t.Error("the app was asked")
})
addr, _ := startRProxy(t, standIn.URL, app.URL, nil)
conn := dial(t, addr)
send(t, conn, getOfSize(32<<10+1))
// io.ReadAll returns without an error only once rproxy has closed the
// connection.
text, err := io.ReadAll(conn)
if err != nil || !strings.HasPrefix(string(text), "HTTP/1.1 431 ") {
t.Errorf(
"read %q (%v), want 431 and the connection closed",
text,
err,
)
}
if len(standIn.questions) != 0 {
t.Error("waf was asked")
}
}
func TestABodyAnnouncedOverItsLimitIsRefusedOnceWAFAllowsIt(t *testing.T) { func TestABodyAnnouncedOverItsLimitIsRefusedOnceWAFAllowsIt(t *testing.T) {
t.Parallel() t.Parallel()
@@ -439,7 +492,7 @@ func TestASettingOfOffSetsNoLimit(t *testing.T) {
strings.NewReader("a body"))), strings.NewReader("a body"))),
) )
if got := get(t, addr, waf.HealthPath); got.status != http.StatusOK { if got := get(t, addr, wafprotocol.HealthPath); got.status != http.StatusOK {
t.Errorf("answered %+v, want rproxy's own 200", got) t.Errorf("answered %+v, want rproxy's own 200", got)
} }
}) })
@@ -817,10 +870,14 @@ func TestAnAnswerRProxyGivesItselfHasTenSecondsToBeSentAtLeast(
func TestAVisitorTooSlowToSendTheRestIsAnswered408(t *testing.T) { func TestAVisitorTooSlowToSendTheRestIsAnswered408(t *testing.T) {
t.Parallel() t.Parallel()
// rproxy is waiting for the rest of the body well within this limit, even on
// a busy host, and answers when it runs out, well before dial's deadline.
const limit = 5 * time.Second
standIn := startWAF(t, allow) standIn := startWAF(t, allow)
app := startApp(t, greet) app := startApp(t, greet)
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{ addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: "3000", maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10),
}) })
conn := dial(t, addr) conn := dial(t, addr)
@@ -1209,7 +1266,9 @@ func TestAVisitorThatStopsALittleShortIsReadUntilTheRequestTimeLimit(
) { ) {
t.Parallel() t.Parallel()
const limit = 2 * time.Second // The app's answer arrives well within this limit, even on a busy host, and
// the connection closes when it runs out, well before dial's deadline.
const limit = 5 * time.Second
standIn := startWAF(t, allow) standIn := startWAF(t, allow)
app := startApp(t, answerBeforeTheBody) app := startApp(t, answerBeforeTheBody)
@@ -1374,17 +1433,20 @@ func TestTheRequestTimeLimitDoesNotCutAnAnswerOnceTheAppHasTheRequest(
const answer = "the end of a long answer" const answer = "the end of a long answer"
// The request reaches the app well within this limit, even on a busy host.
const limit = 5 * time.Second
standIn := startWAF(t, allow) standIn := startWAF(t, allow)
app := startApp(t, func(w http.ResponseWriter, r *http.Request) { app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
_, _ = io.Copy(io.Discard, r.Body) _, _ = io.Copy(io.Discard, r.Body)
_ = http.NewResponseController(w).Flush() _ = http.NewResponseController(w).Flush()
time.Sleep(2100 * time.Millisecond) time.Sleep(limit + 100*time.Millisecond)
_, _ = io.WriteString(w, answer) _, _ = io.WriteString(w, answer)
}) })
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{ addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: "2000", maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10),
}) })
got := visit( got := visit(
@@ -1418,7 +1480,7 @@ func TestAVisitorThatStopsALittleShortGetsAnAnswerRProxyGivesAtOnce(
{"a deny", post, http.StatusForbidden}, {"a deny", post, http.StatusForbidden},
{ {
"the health check", "the health check",
"GET " + waf.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n", "GET " + wafprotocol.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n",
http.StatusOK, http.StatusOK,
}, },
} { } {
+13 -13
View File
@@ -12,11 +12,11 @@ import (
"strings" "strings"
"testing" "testing"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// questionBodyMaxBytes is the most of a body a question carries. // questionBodyMaxBytes is the most of a body a question carries.
const questionBodyMaxBytes = waf.QuestionBodyMaxBytes const questionBodyMaxBytes = wafprotocol.QuestionBodyMaxBytes
// chunked is the encoding of a body whose length the visitor did not announce. // chunked is the encoding of a body whose length the visitor did not announce.
const chunked = "chunked" const chunked = "chunked"
@@ -42,9 +42,9 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
req.Header.Set(forwardedForHeader, "198.51.100.1") req.Header.Set(forwardedForHeader, "198.51.100.1")
req.Header.Set(forwardedProtoHeader, "gopher") req.Header.Set(forwardedProtoHeader, "gopher")
req.Header.Set(requestIDHeader, "the-visitor's") req.Header.Set(requestIDHeader, "the-visitor's")
req.Header.Set(waf.ClientHeader, "198.51.100.2") req.Header.Set(wafprotocol.ClientHeader, "198.51.100.2")
req.Header.Set(waf.PeerHeader, "198.51.100.3") req.Header.Set(wafprotocol.PeerHeader, "198.51.100.3")
req.Header.Set(waf.ContentLengthHeader, "12345") req.Header.Set(wafprotocol.ContentLengthHeader, "12345")
// Go's client then sends no User-Agent, so that one in the question would be // Go's client then sends no User-Agent, so that one in the question would be
// Go's own. // Go's own.
req.Header["User-Agent"] = nil req.Header["User-Agent"] = nil
@@ -73,8 +73,8 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
want.Set(forwardedForHeader, "198.51.100.1") want.Set(forwardedForHeader, "198.51.100.1")
want.Set(forwardedProtoHeader, "http") want.Set(forwardedProtoHeader, "http")
want.Set(requestIDHeader, asked.header.Get(requestIDHeader)) want.Set(requestIDHeader, asked.header.Get(requestIDHeader))
want.Set(waf.ClientHeader, localhost) want.Set(wafprotocol.ClientHeader, localhost)
want.Set(waf.PeerHeader, localhost) want.Set(wafprotocol.PeerHeader, localhost)
if !maps.EqualFunc(asked.header, want, slices.Equal) { if !maps.EqualFunc(asked.header, want, slices.Equal) {
t.Errorf( t.Errorf(
@@ -124,8 +124,8 @@ func TestQuestionNamesTheVisitorATrustedProxySpeaksFor(t *testing.T) {
wantGreeting(t, visit(t, req)) wantGreeting(t, visit(t, req))
asked := standIn.question(t) asked := standIn.question(t)
if asked.header.Get(waf.ClientHeader) != tc.client || if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
asked.header.Get(waf.PeerHeader) != localhost || asked.header.Get(wafprotocol.PeerHeader) != localhost ||
asked.header.Get(forwardedForHeader) != visitor || asked.header.Get(forwardedForHeader) != visitor ||
asked.header.Get(forwardedProtoHeader) != tc.proto { asked.header.Get(forwardedProtoHeader) != tc.proto {
t.Errorf("the question's headers are %v", asked.header) t.Errorf("the question's headers are %v", asked.header)
@@ -191,7 +191,7 @@ func TestQuestionNamesTheClientAsTheForwardedAddressesSay(t *testing.T) {
req.Header[forwardedForHeader] = tc.header req.Header[forwardedForHeader] = tc.header
wantGreeting(t, visit(t, req)) wantGreeting(t, visit(t, req))
if got := standIn.question(t).header.Get(waf.ClientHeader); got != if got := standIn.question(t).header.Get(wafprotocol.ClientHeader); got !=
tc.client { tc.client {
t.Errorf("the question names %s, want %s", got, tc.client) t.Errorf("the question names %s, want %s", got, tc.client)
} }
@@ -282,7 +282,7 @@ func wantQuestionBody(
) )
} }
values := asked.header.Values(waf.ContentLengthHeader) values := asked.header.Values(wafprotocol.ContentLengthHeader)
if announced { if announced {
if !slices.Equal(values, []string{strconv.Itoa(size)}) || if !slices.Equal(values, []string{strconv.Itoa(size)}) ||
@@ -586,8 +586,8 @@ func TestAQuestionNamesNoZone(t *testing.T) {
server.Handler.ServeHTTP(httptest.NewRecorder(), req) server.Handler.ServeHTTP(httptest.NewRecorder(), req)
asked := standIn.question(t) asked := standIn.question(t)
if asked.header.Get(waf.ClientHeader) != tc.client || if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
asked.header.Get(waf.PeerHeader) != tc.peer { asked.header.Get(wafprotocol.PeerHeader) != tc.peer {
t.Errorf("the question's headers are %v", asked.header) t.Errorf("the question's headers are %v", asked.header)
} }
+19 -16
View File
@@ -18,17 +18,13 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// flushAfterEachWrite has ReverseProxy pass on each part of the app's answer as // flushAfterEachWrite has ReverseProxy pass on each part of the app's answer as
// soon as it arrives. // soon as it arrives.
const flushAfterEachWrite time.Duration = -1 const flushAfterEachWrite time.Duration = -1
// reportStatusMax is the highest status waf takes in a report. An app can
// send a higher one, up to 999, and the visitor gets that as it is.
const reportStatusMax = 599
// plainText is the Content-Type of the answers rproxy gives itself. // plainText is the Content-Type of the answers rproxy gives itself.
const plainText = "text/plain; charset=utf-8" const plainText = "text/plain; charset=utf-8"
@@ -126,8 +122,8 @@ func (h *handler) newRequest(
if r.Body != http.NoBody { if r.Body != http.NoBody {
rq.body = &requestBody{ rq.body = &requestBody{
body: limitBody(r.Body, h.config.MaxRequestSize), body: limitBody(r.Body, h.config.MaxRequestSize),
rq: rq, refuse: rq.refuse,
} }
} }
@@ -154,7 +150,7 @@ func (rq *request) readFirstPartOfBody() bool {
_ = rq.rc.SetReadDeadline(rq.requestDeadline()) _ = rq.rc.SetReadDeadline(rq.requestDeadline())
held, err := io.ReadAll( held, err := io.ReadAll(
io.LimitReader(rq.body, waf.QuestionBodyMaxBytes), io.LimitReader(rq.body, wafprotocol.QuestionBodyMaxBytes),
) )
// From here on, the request timer ends a read of the rest of the body, so that // From here on, the request timer ends a read of the rest of the body, so that
// the answer can name the side rproxy was waiting on. // the answer can name the side rproxy was waiting on.
@@ -228,16 +224,17 @@ func (rq *request) rewrite(pr *httputil.ProxyRequest) {
setPathAsSent(pr.Out.URL, pr.In) setPathAsSent(pr.Out.URL, pr.In)
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted) setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
pr.Out.Header.Set(requestIDHeader, rq.id) pr.Out.Header.Set(wafprotocol.RequestIDHeader, rq.id)
pr.Out.Header.Del(asnHeader) // Del removes a visitor's own, whatever their case.
pr.Out.Header.Del(countryHeader) pr.Out.Header.Del(wafprotocol.ASNHeader)
pr.Out.Header.Del(wafprotocol.CountryHeader)
if rq.asn != "" { if rq.asn != "" {
pr.Out.Header.Set(asnHeader, rq.asn) pr.Out.Header.Set(wafprotocol.ASNHeader, rq.asn)
} }
if rq.country != "" { if rq.country != "" {
pr.Out.Header.Set(countryHeader, rq.country) pr.Out.Header.Set(wafprotocol.CountryHeader, rq.country)
} }
} }
@@ -294,7 +291,12 @@ func (rq *request) modifyResponse(res *http.Response) error {
// the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts. // the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
rq.startResponseTimer(requestEnd) rq.startResponseTimer(requestEnd)
res.Body = &responseBody{body: limitBody(res.Body, maxBytes), rq: rq} res.Body = &responseBody{
body: limitBody(res.Body, maxBytes),
refuse: rq.refuse,
responseReceived: rq.responseReceived,
ctx: rq.in.Context(),
}
rq.out.afterStatus = rq.setResponseDeadline rq.out.afterStatus = rq.setResponseDeadline
return nil return nil
@@ -470,9 +472,10 @@ func (rq *request) finish(ctx context.Context) {
return return
} }
report := &waf.Report{ // An app can send a status up to 999, and the visitor gets it as it is.
report := &wafprotocol.Report{
RequestID: rq.id, RequestID: rq.id,
Status: min(rq.out.status, reportStatusMax), Status: min(rq.out.status, wafprotocol.ReportStatusMax),
RequestBytes: requestBytes, RequestBytes: requestBytes,
ResponseBytes: responseBytes, ResponseBytes: responseBytes,
DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)), DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)),
+15 -14
View File
@@ -24,7 +24,7 @@ import (
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// How rproxy keeps connections to the app and to waf open between // How rproxy keeps connections to the app and to waf open between
@@ -47,10 +47,6 @@ const (
idleTimeout = 120 * time.Second idleTimeout = 120 * time.Second
) )
// ownEndpointPrefix starts the path of every request for sww itself, which
// never reaches the app.
const ownEndpointPrefix = "/_sww/"
// allowedMethods lists the methods rproxy passes on, as an Allow header gives // allowedMethods lists the methods rproxy passes on, as an Allow header gives
// them. It answers any other method 405 itself. // them. It answers any other method 405 itself.
// The WebDAV ones are for Nextcloud. // The WebDAV ones are for Nextcloud.
@@ -92,8 +88,11 @@ func New(params Params) *Server {
transport: newTransport(), transport: newTransport(),
wafTransport: newWAFTransport(), wafTransport: newWAFTransport(),
reportURL: (&url.URL{ reportURL: (&url.URL{
Scheme: backend.Scheme, Host: backend.Host, Path: waf.ReportPath, Scheme: backend.Scheme,
Host: backend.Host,
Path: wafprotocol.ReportPath,
}).String(), }).String(),
breakerClock: time.Now,
} }
return &Server{ return &Server{
@@ -133,8 +132,10 @@ type handler struct {
wafTransport http.RoundTripper wafTransport http.RoundTripper
// reportURL is where the reports go. // reportURL is where the reports go.
reportURL string reportURL string
// breaker is the circuit breaker for asks to waf. // breaker is the circuit breaker for asks to waf, and breakerClock gives it
breaker breaker // the time.
breaker breaker
breakerClock func() time.Time
// reports waits for the reports being sent. reportsMu guards starting one // reports waits for the reports being sent. reportsMu guards starting one
// against reportsStopped, which is set once nothing is to wait for another. // against reportsStopped, which is set once nothing is to wait for another.
reports sync.WaitGroup reports sync.WaitGroup
@@ -176,7 +177,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
defer rq.readRestOfBody() defer rq.readRestOfBody()
defer rq.finish(r.Context()) defer rq.finish(r.Context())
if r.Method == http.MethodGet && r.URL.Path == waf.HealthPath { if r.Method == http.MethodGet && r.URL.Path == wafprotocol.HealthPath {
rq.answerHealth() rq.answerHealth()
return return
@@ -209,9 +210,9 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rq.noteAnswer(answer) rq.noteAnswer(answer)
switch rq.disposition { switch rq.disposition {
case waf.DispositionAllow: case wafprotocol.DispositionAllow:
rq.passToApp(r.Context()) rq.passToApp(r.Context())
case waf.DispositionClose: case wafprotocol.DispositionClose:
// This panic has Go's server close the connection without an answer, // This panic has Go's server close the connection without an answer,
// and log nothing; the deferred finish still runs. // and log nothing; the deferred finish still runs.
panic(http.ErrAbortHandler) panic(http.ErrAbortHandler)
@@ -221,8 +222,8 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
} }
// passToApp passes the request on to the app, unless a limit or its path keeps // passToApp passes the request on to the app, unless a limit or its path keeps
// it from the app. waf answers a path under ownEndpointPrefix itself, so // it from the app. waf answers a path under wafprotocol.OwnEndpointPrefix
// such a request gets here only when // itself, so such a request gets here only when
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it. // SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it.
// ctx is the request's own context. // ctx is the request's own context.
func (rq *request) passToApp(ctx context.Context) { func (rq *request) passToApp(ctx context.Context) {
@@ -231,7 +232,7 @@ func (rq *request) passToApp(ctx context.Context) {
switch { switch {
case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes): case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes):
rq.answer(http.StatusRequestEntityTooLarge) rq.answer(http.StatusRequestEntityTooLarge)
case strings.HasPrefix(rq.in.URL.Path, ownEndpointPrefix): case strings.HasPrefix(rq.in.URL.Path, wafprotocol.OwnEndpointPrefix):
rq.answer(http.StatusNotFound) rq.answer(http.StatusNotFound)
default: default:
rq.forward(ctx) rq.forward(ctx)
+75 -40
View File
@@ -21,9 +21,9 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -56,7 +56,7 @@ const (
// The headers the tests set and look at. // The headers the tests set and look at.
const ( const (
dispositionHeader = waf.DispositionHeader dispositionHeader = wafprotocol.DispositionHeader
forwardedForHeader = "X-Forwarded-For" forwardedForHeader = "X-Forwarded-For"
forwardedProtoHeader = "X-Forwarded-Proto" forwardedProtoHeader = "X-Forwarded-Proto"
requestIDHeader = "X-Request-ID" requestIDHeader = "X-Request-ID"
@@ -72,9 +72,9 @@ const (
// What the stand-in for waf answers. // What the stand-in for waf answers.
const ( const (
dispositionAllow = waf.DispositionAllow dispositionAllow = wafprotocol.DispositionAllow
dispositionDeny = waf.DispositionDeny dispositionDeny = wafprotocol.DispositionDeny
dispositionClose = waf.DispositionClose dispositionClose = wafprotocol.DispositionClose
// deniedBody is the body of a deny. // deniedBody is the body of a deny.
deniedBody = "refused by waf" deniedBody = "refused by waf"
) )
@@ -175,7 +175,7 @@ type WAF struct {
*httptest.Server *httptest.Server
questions chan question questions chan question
reports chan waf.Report reports chan wafprotocol.Report
} }
// startWAF starts a stand-in for waf. // startWAF starts a stand-in for waf.
@@ -184,11 +184,11 @@ func startWAF(t *testing.T, answer http.HandlerFunc) *WAF {
s := &WAF{ s := &WAF{
questions: make(chan question, 10), questions: make(chan question, 10),
reports: make(chan waf.Report, 10), reports: make(chan wafprotocol.Report, 10),
} }
s.Server = httptest.NewServer(http.HandlerFunc( s.Server = httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) == "" { if r.Header.Get(wafprotocol.ClientHeader) == "" {
s.takeReport(t, w, r) s.takeReport(t, w, r)
return return
@@ -217,11 +217,11 @@ func (s *WAF) takeReport(
) { ) {
t.Helper() t.Helper()
var report waf.Report var report wafprotocol.Report
err := json.NewDecoder(r.Body).Decode(&report) err := json.NewDecoder(r.Body).Decode(&report)
if err != nil || r.Method != http.MethodPost || if err != nil || r.Method != http.MethodPost ||
r.URL.Path != waf.ReportPath { r.URL.Path != wafprotocol.ReportPath {
t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err) t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err)
} }
@@ -242,7 +242,7 @@ func (s *WAF) wantNoReport(t *testing.T) {
} }
// report waits for a report. // report waits for a report.
func (s *WAF) report(t *testing.T) waf.Report { func (s *WAF) report(t *testing.T) wafprotocol.Report {
t.Helper() t.Helper()
select { select {
@@ -251,7 +251,7 @@ func (s *WAF) report(t *testing.T) waf.Report {
case <-time.After(waitLimit): case <-time.After(waitLimit):
t.Fatal("no report") t.Fatal("no report")
return waf.Report{} return wafprotocol.Report{}
} }
} }
@@ -279,7 +279,7 @@ func startWAFAnsweringReports(
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) != "" { if r.Header.Get(wafprotocol.ClientHeader) != "" {
answer(w, r) answer(w, r)
return return
@@ -324,7 +324,8 @@ func greet(w http.ResponseWriter, r *http.Request) {
// startRProxy starts rproxy with waf at backend and the app at appURL, and // startRProxy starts rproxy with waf at backend and the app at appURL, and
// the settings in env on top of the defaults. It returns where rproxy listens // the settings in env on top of the defaults. It returns where rproxy listens
// and what it writes. // and what it writes. SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS is a minute unless env
// sets it, so that a busy host cannot make rproxy drop a question or a report.
func startRProxy( func startRProxy(
t *testing.T, backend, appURL string, env map[string]string, t *testing.T, backend, appURL string, env map[string]string,
) (string, *output) { ) (string, *output) {
@@ -355,8 +356,22 @@ func startServerLogging(
) (string, *rproxy.Server) { ) (string, *rproxy.Server) {
t.Helper() t.Helper()
server := newServer(t, backend, appURL, env, requestLog, processLog)
return serve(t, server), server
}
// newServer returns rproxy, not yet serving, with its request lines going to
// requestLog and its process lines to processLog.
func newServer(
t *testing.T, backend, appURL string, env map[string]string,
requestLog, processLog io.Writer,
) *rproxy.Server {
t.Helper()
settings := map[string]string{ settings := map[string]string{
"SWW_RPROXY_WAF_BACKEND": backend, "SWW_RPROXY_UPSTREAM_URL": appURL, "SWW_RPROXY_WAF_BACKEND": backend, "SWW_RPROXY_UPSTREAM_URL": appURL,
wafBackendTimeout: "60000",
} }
maps.Copy(settings, env) maps.Copy(settings, env)
@@ -365,15 +380,20 @@ func startServerLogging(
t.Fatalf("settings %v: %v", settings, err) t.Fatalf("settings %v: %v", settings, err)
} }
server := rproxy.New(rproxy.Params{ return rproxy.New(rproxy.Params{
Config: cfg, Config: cfg,
RequestLog: requestLog, RequestLog: requestLog,
ProcessLog: requestlog.NewProcessLogger( ProcessLog: processlog.New(processlog.Params{
processLog, Output: processLog,
"test", InstanceName: "test",
slog.LevelInfo, Level: slog.LevelInfo,
), }),
}) })
}
// serve has server serve until the test ends, and returns its address.
func serve(t *testing.T, server *rproxy.Server) string {
t.Helper()
listener, err := (&net.ListenConfig{}).Listen( listener, err := (&net.ListenConfig{}).Listen(
t.Context(), t.Context(),
@@ -394,7 +414,7 @@ func startServerLogging(
server.WaitForReports() server.WaitForReports()
}) })
return listener.Addr().String(), server return listener.Addr().String()
} }
// answered is the visitor's answer. // answered is the visitor's answer.
@@ -809,7 +829,7 @@ func TestAnAnswerRProxyHoldsWholeIsSentBeforeItsLogLine(t *testing.T) {
}{ }{
{"a deny", "GET /" + host, deniedBody, http.StatusForbidden}, {"a deny", "GET /" + host, deniedBody, http.StatusForbidden},
{"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest}, {"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest},
{"the health check", "GET " + waf.HealthPath + host, "ok\n", http.StatusOK}, {"the health check", "GET " + wafprotocol.HealthPath + host, "ok\n", http.StatusOK},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1432,14 +1452,18 @@ func TestFiveFailedAsksKeepTheNextRequestFromAskingWAF(t *testing.T) {
greet(w, r) greet(w, r)
}) })
addr, out := startRProxy( out := &output{}
t, server := newServer(
standIn.URL, t, standIn.URL, app.URL,
app.URL, map[string]string{wafBackendTimeout: "200"},
map[string]string{ out, out,
wafBackendTimeout: "200",
},
) )
// The breaker's clock stands still, as in
// TestAnswersThatCannotBePassedOnOpenTheBreaker: on a slow host, the five
// failures could take more than five seconds, and the next request could
// come more than a second later.
server.StopBreakerClock(time.Now())
addr := serve(t, server)
// An ask can time out before it reaches the stand-in, so these // An ask can time out before it reaches the stand-in, so these
// may leave fewer than five questions, or leave some late. // may leave fewer than five questions, or leave some late.
@@ -1530,9 +1554,13 @@ func TestAnswersThatCannotBePassedOnOpenTheBreaker(t *testing.T) {
tooLarge(w, r) tooLarge(w, r)
}) })
app := startApp(t, greet) app := startApp(t, greet)
addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{ out := &output{}
wafBackendTimeout: "60000", server := newServer(t, standIn.URL, app.URL, nil, out, out)
}) // The breaker's clock stands still. On a slow host, the five failures could
// otherwise take more than the five seconds that open the breaker. The next
// request could come more than a second later, when it may ask waf again.
server.StopBreakerClock(time.Now())
addr := serve(t, server)
for range 5 { for range 5 {
if got := get(t, addr, "/"); got.status != http.StatusBadGateway { if got := get(t, addr, "/"); got.status != http.StatusBadGateway {
@@ -1574,19 +1602,19 @@ func TestHealthCheckIsAnsweredWhileWAFIsDown(t *testing.T) {
app := startApp(t, greet) app := startApp(t, greet)
addr, out := startRProxy(t, nowhere, app.URL, nil) addr, out := startRProxy(t, nowhere, app.URL, nil)
got := get(t, addr, waf.HealthPath) got := get(t, addr, wafprotocol.HealthPath)
if got.status != http.StatusOK || got.body != "ok\n" { if got.status != http.StatusOK || got.body != "ok\n" {
t.Errorf("answered %+v, want ok", got) t.Errorf("answered %+v, want ok", got)
} }
line := out.requestLine(t) line := out.requestLine(t)
if line["path"] != waf.HealthPath || line["disposition"] != nil { if line["path"] != wafprotocol.HealthPath || line["disposition"] != nil {
t.Errorf("line %v, want one with no disposition", line) t.Errorf("line %v, want one with no disposition", line)
} }
// With waf up too, it is not asked. // With waf up too, it is not asked.
addr, _ = startRProxy(t, standIn.URL, app.URL, nil) addr, _ = startRProxy(t, standIn.URL, app.URL, nil)
get(t, addr, waf.HealthPath) get(t, addr, wafprotocol.HealthPath)
if len(standIn.questions) != 0 { if len(standIn.questions) != 0 {
t.Error("waf was asked about the health check") t.Error("waf was asked about the health check")
@@ -1747,7 +1775,7 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{}) release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) != "" { if r.Header.Get(wafprotocol.ClientHeader) != "" {
allow(w, r) allow(w, r)
return return
@@ -1807,11 +1835,15 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) { func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) {
t.Parallel() t.Parallel()
// The app's answer to the upgrade arrives well within these limits, even on a
// busy host.
const limit = 5 * time.Second
standIn := startWAF(t, allow) standIn := startWAF(t, allow)
app := startApp(t, echoAfterUpgrade) app := startApp(t, echoAfterUpgrade)
addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{ addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: "2000", maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10),
maxResponseTime: "2000", maxResponseTime: strconv.FormatInt(limit.Milliseconds(), 10),
}) })
conn := dial(t, addr) conn := dial(t, addr)
@@ -1836,7 +1868,10 @@ func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) {
} }
// Past both time limits, the connection still carries what is sent. // Past both time limits, the connection still carries what is sent.
time.Sleep(2100 * time.Millisecond) time.Sleep(limit + 100*time.Millisecond)
// dial's deadline ran from before the upgrade and this wait.
_ = conn.SetDeadline(time.Now().Add(waitLimit))
const message = "still here\n" const message = "still here\n"
+36 -85
View File
@@ -1,18 +1,16 @@
package rproxy package rproxy
import ( import (
"bytes"
"context" "context"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"io" "io"
"maps" "maps"
"net/http" "net/http"
"strings" "strings"
"time"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// answerBodyMaxBytes is the most of the body of waf's answer that is read. // answerBodyMaxBytes is the most of the body of waf's answer that is read.
@@ -41,15 +39,12 @@ type wafAnswer struct {
// notes nothing, and no report is due for it: waf holds no decision to report // notes nothing, and no report is due for it: waf holds no decision to report
// on. // on.
func (rq *request) noteAnswer(answer *wafAnswer) { func (rq *request) noteAnswer(answer *wafAnswer) {
disposition := answer.header.Get(waf.DispositionHeader) read, valid := wafprotocol.ReadAnswer(answer.header)
if valid {
switch disposition { rq.disposition = read.Disposition
case waf.DispositionAllow, waf.DispositionDeny,
waf.DispositionClose:
rq.disposition = disposition
rq.answered = true rq.answered = true
rq.asn = answer.header.Get(asnHeader) rq.asn = read.ASN
rq.country = answer.header.Get(countryHeader) rq.country = read.Country
} }
} }
@@ -80,7 +75,7 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition
rq.wafNoAnswer = true rq.wafNoAnswer = true
if rq.disposition == waf.DispositionAllow { if rq.disposition == wafprotocol.DispositionAllow {
rq.passToApp(ctx) rq.passToApp(ctx)
} else { } else {
rq.answer(http.StatusServiceUnavailable) rq.answer(http.StatusServiceUnavailable)
@@ -97,7 +92,7 @@ func (rq *request) askUnlessBreakerOpen(
ctx context.Context, ctx context.Context,
) (*wafAnswer, error) { ) (*wafAnswer, error) {
b := &rq.h.breaker b := &rq.h.breaker
if !b.mayAsk(time.Now()) { if !b.mayAsk(rq.h.breakerClock()) {
return nil, errBreakerOpen return nil, errBreakerOpen
} }
@@ -107,7 +102,7 @@ func (rq *request) askUnlessBreakerOpen(
case err == nil: case err == nil:
b.noteAnswer() b.noteAnswer()
case ctx.Err() == nil: case ctx.Err() == nil:
b.noteFailure(time.Now()) b.noteFailure(rq.h.breakerClock())
} }
return answer, err return answer, err
@@ -120,7 +115,21 @@ func (rq *request) ask(ctx context.Context) (*wafAnswer, error) {
ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout) ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout)
defer cancel() defer cancel()
question, err := rq.newQuestion(ctx) var held []byte
if rq.body != nil {
held = rq.body.held
}
question, err := wafprotocol.NewQuestion(ctx, wafprotocol.QuestionParams{
Visitor: rq.in,
PathAsSent: rules.Path(rq.in),
Body: held,
WAFBackend: rq.h.config.WAFBackend,
Client: rq.client,
Peer: rq.peer,
RequestID: rq.id,
Scheme: scheme(rq.in, rq.peerTrusted),
})
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -177,69 +186,13 @@ func canBeAsked(r *http.Request) bool {
len(r.Header.Values("User-Agent")) <= 1 len(r.Header.Values("User-Agent")) <= 1
} }
// newQuestion returns the question about the request, as "The question" in
// SPEC.md gives it: the visitor's request as the visitor sent it, with the part
// of the body it holds and rproxy's headers.
func (rq *request) newQuestion(ctx context.Context) (*http.Request, error) {
in := rq.in
var held []byte
if rq.body != nil {
held = rq.body.held
}
question, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
ctx,
in.Method,
rq.h.config.WAFBackend.String(),
bytes.NewReader(held),
)
if err != nil {
return nil, fmt.Errorf("make the question: %w", err)
}
backend := question.URL
target := *in.URL
target.Scheme = backend.Scheme
target.Host = backend.Host
setPathAsSent(&target, in)
question.URL = &target
question.Host = in.Host
// The length NewRequestWithContext set is that of what is held. This line makes
// a body whose length the visitor did not announce go chunked, as "The question"
// in SPEC.md requires.
question.TransferEncoding = in.TransferEncoding
question.Header = in.Header.Clone()
question.Header.Set(waf.ClientHeader, rq.client.String())
question.Header.Set(waf.PeerHeader, rq.peer.String())
question.Header.Del(waf.ContentLengthHeader)
announced := in.Header.Get("Content-Length")
if announced != "" {
question.Header.Set(waf.ContentLengthHeader, announced)
}
question.Header.Set(requestIDHeader, rq.id)
question.Header.Set("X-Forwarded-Proto", scheme(in, rq.peerTrusted))
// Go's client adds a User-Agent of its own to a request without the key.
_, sent := question.Header["User-Agent"]
if !sent {
question.Header["User-Agent"] = nil
}
return question, nil
}
// answerWith gives the visitor waf's answer, less its disposition. // answerWith gives the visitor waf's answer, less its disposition.
func (rq *request) answerWith(answer *wafAnswer) { func (rq *request) answerWith(answer *wafAnswer) {
rq.prepareOwnAnswer(rq.requestDeadline()) rq.prepareOwnAnswer(rq.requestDeadline())
header := rq.out.Header() header := rq.out.Header()
maps.Copy(header, answer.header) maps.Copy(header, answer.header)
header.Del(waf.DispositionHeader) header.Del(wafprotocol.DispositionHeader)
// Go's server adds a Content-Type it guesses from the body to a response whose // Go's server adds a Content-Type it guesses from the body to a response whose
// headers have no Content-Type key. The key with no value sends none. // headers have no Content-Type key. The key with no value sends none.
@@ -253,7 +206,10 @@ func (rq *request) answerWith(answer *wafAnswer) {
// sendReport sends waf report on a goroutine of its own, so that the // sendReport sends waf report on a goroutine of its own, so that the
// visitor does not wait for it. The report goes on after ctx, the request's own // visitor does not wait for it. The report goes on after ctx, the request's own
// context, has ended. // context, has ended.
func (h *handler) sendReport(ctx context.Context, report *waf.Report) { func (h *handler) sendReport(
ctx context.Context,
report *wafprotocol.Report,
) {
h.reportsMu.Lock() h.reportsMu.Lock()
defer h.reportsMu.Unlock() defer h.reportsMu.Unlock()
@@ -271,23 +227,18 @@ func (h *handler) sendReport(ctx context.Context, report *waf.Report) {
// postReport posts report to waf, and waits for its answer for up to // postReport posts report to waf, and waits for its answer for up to
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. // SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS.
func (h *handler) postReport(ctx context.Context, report *waf.Report) error { func (h *handler) postReport(
body, err := json.Marshal(report) ctx context.Context,
if err != nil { report *wafprotocol.Report,
return fmt.Errorf("encode the report: %w", err) ) error {
}
ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout) ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout)
defer cancel() defer cancel()
req, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL req, err := wafprotocol.NewReportRequest(ctx, h.reportURL, report)
ctx, http.MethodPost, h.reportURL, bytes.NewReader(body))
if err != nil { if err != nil {
return fmt.Errorf("make the request: %w", err) return err
} }
req.Header.Set("Content-Type", "application/json")
res, err := h.wafTransport.RoundTrip(req) res, err := h.wafTransport.RoundTrip(req)
if err != nil { if err != nil {
return fmt.Errorf("send the report: %w", err) return fmt.Errorf("send the report: %w", err)
+21 -50
View File
@@ -18,12 +18,11 @@ import (
"slices" "slices"
"strings" "strings"
"sync/atomic" "sync/atomic"
"time"
"github.com/fsnotify/fsnotify" "golang.org/x/net/http/httpguts"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/dirwatch"
) )
// The actions a rule takes when it matches. // The actions a rule takes when it matches.
@@ -40,12 +39,6 @@ const (
// extension ends the name of every rule file. // extension ends the name of every rule file.
const extension = ".rules" const extension = ".rules"
// quietTime is how long SWW_WAF_RULE_FILES_DIR must go without a change before
// the rule files are read again, so that a file still being written, such
// as one saved in place, appended to or copied in with scp, is read only
// once whole.
const quietTime = 2 * time.Second
// headerTarget starts the target that is one request header, // headerTarget starts the target that is one request header,
// header:<Name>. // header:<Name>.
const headerTarget = "header:" const headerTarget = "header:"
@@ -176,26 +169,18 @@ func (f *Files) Len() int {
} }
// Watch watches Dir until ctx is done, and reads the rule files again // Watch watches Dir until ctx is done, and reads the rule files again
// once Dir has had no change for quietTime, after one is edited, added or // once Dir has had no change for a short time, after one is edited, added
// removed, and after Watch starts watching. If they then hold an error, // or removed, and after Watch starts watching, as dirwatch describes. If
// the rules stay as they were, the error is logged with its file and // they then hold an error, the rules stay as they were, the error is
// line, and the files are read again after the next change. If Dir cannot // logged with its file and line, and the files are read again after the
// be watched, that is logged, and the rules stay as they were loaded. // next change. If Dir cannot be watched, that is logged, and the rules
// While Enabled is false, Watch returns at once. // stay as they were loaded. While Enabled is false, Watch returns at once.
func (f *Files) Watch(ctx context.Context) { func (f *Files) Watch(ctx context.Context) {
if !f.params.Enabled { if !f.params.Enabled {
return return
} }
watcher, err := fsnotify.NewWatcher() watcher, err := dirwatch.New(f.params.Dir)
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(f.params.Dir)
}
if err != nil { if err != nil {
f.params.ProcessLog.Error("cannot watch the rule files for edits", f.params.ProcessLog.Error("cannot watch the rule files for edits",
"error", err.Error()) "error", err.Error())
@@ -203,36 +188,22 @@ func (f *Files) Watch(ctx context.Context) {
return return
} }
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the rule files for edits", f.params.ProcessLog.Info("watching the rule files for edits",
"directory", f.params.Dir) "directory", f.params.Dir)
f.readAfterChanges(ctx, watcher.Events, watcher.Errors) // A change to any file in Dir counts.
} watcher.CallWhenQuiet(ctx,
func(string) bool { return true },
// readAfterChanges reads the rule files again once quietTime has passed f.readAgain,
// without a change from events, until ctx is done, and logs the errors func(err error) {
// from errs. The wait starts at once, as if for a change, so that an edit
// saved after Load read the files, and before Dir was watched, is taken
// in too.
func (f *Files) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case <-events:
quiet.Reset(quietTime)
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the rule files failed", f.params.ProcessLog.Warn("watching the rule files failed",
"error", err.Error()) "error", err.Error())
} },
} )
} }
// readAgain reads the rule files again, in place of the rules loaded, or // readAgain reads the rule files again, in place of the rules loaded, or
@@ -378,7 +349,7 @@ func parse(line string) (Rule, bool, error) {
rule.Target, rule.Target,
errNotTarget, errNotTarget,
) )
case isHeader && !config.IsHeaderName(headerName): case isHeader && !httpguts.ValidHeaderFieldName(headerName):
return Rule{}, false, fmt.Errorf( return Rule{}, false, fmt.Errorf(
"the target %q %w", "the target %q %w",
rule.Target, rule.Target,
-165
View File
@@ -1,165 +0,0 @@
package rules
import (
"context"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
"sneak.berlin/go/smallwebwaf/internal/alerts"
)
// The tests below run readAfterChanges in a synctest bubble, where time is
// a clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
func TestFileWrittenInTwoPartsTakenInOnlyWhole(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
changes := run(t, files)
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create: %v", err)
}
defer func() {
_ = file.Close()
}()
// The first part ends in the middle of a ban rule's regex, which,
// read then, would ban every request.
write(t, file, "first path block ^/first\nprobe path ban ^/")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/anything")
// The second part starts the wait again.
write(t, file, `\.env$`+"\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env", "probe")
wantMatched(t, files, "/anything")
})
}
func TestEditSavedBeforeTheWatchStartsTakenIn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
// Saved after Load read the files, and before the directory was
// watched, so that no change is seen for it.
writeFile(t, path, "first path block ^/edited\n")
run(t, files)
time.Sleep(quietTime)
synctest.Wait()
wantMatched(t, files, "/edited", "first")
})
}
// load loads the rules in dir.
func load(t *testing.T, dir string) *Files {
t.Helper()
files, err := Load(Params{
Dir: dir, Enabled: true, ProcessLog: slog.New(slog.DiscardHandler),
Alerts: alerts.New(alerts.Params{}),
})
if err != nil {
t.Fatalf("load: %v", err)
}
return files
}
// run runs files' readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func run(t *testing.T, files *Files) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
files.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// writeFile writes content to the file at path.
func writeFile(t *testing.T, path, content string) {
t.Helper()
err := os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("write %s: %v", path, err)
}
}
// write writes text to the end of file.
func write(t *testing.T, file *os.File, text string) {
t.Helper()
_, err := file.WriteString(text)
if err != nil {
t.Fatalf("write: %v", err)
}
}
// wantMatched checks the ids of the rules that a GET request for path
// matches, in order.
func wantMatched(t *testing.T, files *Files, path string, want ...string) {
t.Helper()
r := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"http://app.example"+path, nil)
matched := files.Match(r)
got := make([]string, 0, len(matched))
for _, rule := range matched {
got = append(got, rule.ID)
}
if !slices.Equal(got, want) {
t.Errorf("%s matched %v, want %v", path, got, want)
}
}
+6 -3
View File
@@ -104,7 +104,7 @@ var (
errVersion = errors.New("unknown version") errVersion = errors.New("unknown version")
// errMissing is for an entry without a field it needs. // errMissing is for an entry without a field it needs.
errMissing = errors.New("has no") errMissing = errors.New("has no")
errCause = errors.New("is not limit, attack or admin") errCause = errors.New("is not limit, attack, admin or abuseipdb")
errExpires = errors.New("is not after its start") errExpires = errors.New("is not after its start")
errDestination = errors.New("is not webhook, slack or ntfy") errDestination = errors.New("is not webhook, slack or ntfy")
errScope = errors.New( errScope = errors.New(
@@ -638,6 +638,8 @@ func (f *Files) takeInState(
f.params.Ledger.Load(held) f.params.Ledger.Load(held)
} }
f.params.Ledger.ForgetEndedAbuseIPDBBans(f.params.Now())
entries = len(held) entries = len(held)
case clientsJSON: case clientsJSON:
var file clientsFile var file clientsFile
@@ -967,7 +969,7 @@ func (e BanEntry) ban() bans.Ban {
// each expires is read again as written. An expires not after the start // each expires is read again as written. An expires not after the start
// is refused too, since the netblock's next ban for a broken limit, three // is refused too, since the netblock's next ban for a broken limit, three
// times as long as this one, would then refuse nothing; and so is a cause // times as long as this one, would then refuse nothing; and so is a cause
// other than limit, attack or admin, most likely misspelt. // other than limit, attack, admin or abuseipdb, most likely misspelt.
func (f *bansFile) check(data []byte) error { func (f *bansFile) check(data []byte) error {
var written struct { var written struct {
Bans []struct { Bans []struct {
@@ -991,7 +993,8 @@ func (f *bansFile) check(data []byte) error {
case entry.Expires != nil && !entry.Expires.After(entry.Start): case entry.Expires != nil && !entry.Expires.After(entry.Start):
return fmt.Errorf("entry %d's expires %w", i+1, errExpires) return fmt.Errorf("entry %d's expires %w", i+1, errExpires)
case entry.Cause != "" && entry.Cause != bans.CauseLimit && case entry.Cause != "" && entry.Cause != bans.CauseLimit &&
entry.Cause != bans.CauseAttack && entry.Cause != bans.CauseAdmin: entry.Cause != bans.CauseAttack && entry.Cause != bans.CauseAdmin &&
entry.Cause != bans.CauseAbuseIPDB:
return fmt.Errorf( return fmt.Errorf(
"entry %d's cause %q %w", "entry %d's cause %q %w",
i+1, i+1,
+67 -21
View File
@@ -114,7 +114,8 @@ const permanentBansJSON = `{
"earlier_bans": { "earlier_bans": {
"limit": 3, "limit": 3,
"attack": 1, "attack": 1,
"admin": 1 "admin": 1,
"abuseipdb": 0
} }
} }
} }
@@ -1190,7 +1191,7 @@ func TestBanWithAnotherCauseStopsTheStart(t *testing.T) {
`"expires": "2026-10-06T04:00:00Z", "cause": "limit"}, `+ `"expires": "2026-10-06T04:00:00Z", "cause": "limit"}, `+
`{"netblock": "203.0.113.12/32", "start": "2026-10-06T00:00:00Z", `+ `{"netblock": "203.0.113.12/32", "start": "2026-10-06T00:00:00Z", `+
`"expires": null, "cause": "atack"}]}`, `"expires": null, "cause": "atack"}]}`,
`: entry 4's cause "atack" is not limit, attack or admin`) `: entry 4's cause "atack" is not limit, attack, admin or abuseipdb`)
} }
func TestBanThatDoesNotEndAfterItsStartStopsTheStart(t *testing.T) { func TestBanThatDoesNotEndAfterItsStartStopsTheStart(t *testing.T) {
@@ -1283,18 +1284,18 @@ func TestBansWrittenOnceBansWriteDelayAfterABan(t *testing.T) {
// A second ban, made while the first waits to be written, puts the // A second ban, made while the first waits to be written, puts the
// write off no further, and is written with it. // write off no further, and is written with it.
first, _ := params.Ledger.BanForLimit( first, _ := params.Ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
time.Sleep(5 * time.Second) time.Sleep(5 * time.Second)
second, _ := params.Ledger.BanForLimit( second, _ := params.Ledger.Ban(
netip.MustParsePrefix("203.0.113.10/32"), netip.MustParsePrefix("203.0.113.10/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
time.Sleep(5*time.Second - time.Nanosecond) time.Sleep(5*time.Second - time.Nanosecond)
@@ -1357,16 +1358,16 @@ func TestEditJustBeforeAScheduledWriteSurvivesIt(t *testing.T) {
run(t, load(t, params).Run) run(t, load(t, params).Run)
// A ban, and bans.json written with it. // A ban, and bans.json written with it.
params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"), params.Ledger.Ban(netip.MustParsePrefix("203.0.113.9/32"),
midnight(), bans.Notes{}) midnight(), bans.Grounds{Cause: bans.CauseLimit})
time.Sleep(params.BansWriteDelay) time.Sleep(params.BansWriteDelay)
synctest.Wait() synctest.Wait()
// A second ban is to be written BansWriteDelay later. Just before // A second ban is to be written BansWriteDelay later. Just before
// then, an admin saves bans.json with the first ban lifted and // then, an admin saves bans.json with the first ban lifted and
// another added. // another added.
params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.10/32"), params.Ledger.Ban(netip.MustParsePrefix("203.0.113.10/32"),
midnight(), bans.Notes{}) midnight(), bans.Grounds{Cause: bans.CauseLimit})
time.Sleep(params.BansWriteDelay - time.Nanosecond) time.Sleep(params.BansWriteDelay - time.Nanosecond)
synctest.Wait() synctest.Wait()
edit(t, dir, bansJSON, permanentBansJSON) edit(t, dir, bansJSON, permanentBansJSON)
@@ -1482,10 +1483,10 @@ func TestFailedWriteLeavesTheFileAsItWas(t *testing.T) {
t.Fatalf("mkdir: %v", err) t.Fatalf("mkdir: %v", err)
} }
params.Ledger.BanForLimit( params.Ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
params.Limiter.Count( params.Limiter.Count(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
@@ -1555,10 +1556,10 @@ func TestWritesAreCountedInTheMetrics(t *testing.T) {
t.Fatalf("mkdir: %v", err) t.Fatalf("mkdir: %v", err)
} }
params.Ledger.BanForLimit( params.Ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
err = files.WriteAll() err = files.WriteAll()
@@ -2090,6 +2091,39 @@ func TestLiftedBanReadAtTheStart(t *testing.T) {
wantLiftedBanKept(t, load(t, params), dir, params.Ledger) wantLiftedBanKept(t, load(t, params), dir, params.Ledger)
} }
func TestEndedAbuseIPDBBanForgottenWhenBansJSONIsReadAtTheStart(
t *testing.T,
) {
t.Parallel()
ended := netip.MustParsePrefix("203.0.113.9/32")
lasting := netip.MustParsePrefix("203.0.113.10/32")
limited := netip.MustParsePrefix("203.0.113.11/32")
// At midnight, the start's time, the first ban has ended and the second
// lasts. The third has ended too, but is for a limit.
dir := t.TempDir()
edit(t, dir, bansJSON, `{"version": 1, "bans": [`+
`{"netblock": "203.0.113.9/32", "start": "2026-09-06T00:00:00Z", `+
`"expires": "2026-10-06T00:00:00Z", "cause": "abuseipdb"}, `+
`{"netblock": "203.0.113.10/32", "start": "2026-09-07T00:00:00Z", `+
`"expires": "2026-10-07T00:00:00Z", "cause": "abuseipdb"}, `+
`{"netblock": "203.0.113.11/32", "start": "2026-10-05T00:00:00Z", `+
`"expires": "2026-10-05T01:00:00Z", "cause": "limit"}]}`)
params := newParams(dir)
load(t, params)
for netblock, want := range map[netip.Prefix]int{
ended: 0, lasting: 1, limited: 1,
} {
if got := len(params.Ledger.Bans(netblock)); got != want {
t.Errorf("the ledger holds %d bans on %s, want %d",
got, netblock, want)
}
}
}
func TestBanLiftedByAnEditWhileRunning(t *testing.T) { func TestBanLiftedByAnEditWhileRunning(t *testing.T) {
t.Parallel() t.Parallel()
@@ -2101,7 +2135,11 @@ func TestBanLiftedByAnEditWhileRunning(t *testing.T) {
// The ban that liftedBansJSON lifts, before it is lifted. // The ban that liftedBansJSON lifts, before it is lifted.
netblock := netip.MustParsePrefix(liftedClient + "/32") netblock := netip.MustParsePrefix(liftedClient + "/32")
params.Ledger.BanForLimit(netblock, midnight(), bans.Notes{}) params.Ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
_, banned, _ := params.Ledger.Find(netblock.Addr(), afterLifting()) _, banned, _ := params.Ledger.Find(netblock.Addr(), afterLifting())
if !banned { if !banned {
@@ -2414,7 +2452,7 @@ func newParams(dir string) state.Params {
AttackBanDuration: 7 * 24 * time.Hour, AttackBanDuration: 7 * 24 * time.Hour,
MaxBans: 5000, MaxBans: 5000,
}), }),
Limiter: ratelimit.New(ratelimit.Limits{}, 20000), Limiter: ratelimit.New(ratelimit.Params{MaxClients: 20000}),
GeoJS: geojs.New(geojs.Params{Now: midnight}), GeoJS: geojs.New(geojs.Params{Now: midnight}),
Lists: reputation.New(reputation.Params{ Lists: reputation.New(reputation.Params{
BlocklistURLs: []string{ BlocklistURLs: []string{
@@ -2476,11 +2514,15 @@ func fill(params state.Params) {
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
params.Ledger.Load([]bans.Ban{permanentBan()}) params.Ledger.Load([]bans.Ban{permanentBan()})
params.Ledger.BanForLimit(client, now, bans.Notes{ params.Ledger.Ban(client, now, bans.Grounds{
ASN: asn, ASName: asName, Country: "DE", Limit: 1, Cause: bans.CauseLimit,
Notes: bans.Notes{ASN: asn, ASName: asName, Country: "DE", Limit: 1},
}) })
params.Ledger.BanForAttack(netip.MustParsePrefix("192.0.2.1/32"), now, params.Ledger.Ban(netip.MustParsePrefix("192.0.2.1/32"), now,
bans.Notes{RuleID: "env-file", Target: "path"}) bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "env-file", Target: "path"},
})
for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} { for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} {
params.Limiter.Count(netip.MustParsePrefix(c), now, whole) params.Limiter.Count(netip.MustParsePrefix(c), now, whole)
@@ -2666,7 +2708,11 @@ func wantLiftedBanKept(
} }
// Were the lifted ban counted, the next would last three hours. // Were the lifted ban counted, the next would last three hours.
ban, _ := ledger.BanForLimit(netblock, afterLifting(), bans.Notes{}) ban, _ := ledger.Ban(
netblock,
afterLifting(),
bans.Grounds{Cause: bans.CauseLimit},
)
if ban.Expires.Sub(ban.Start) != time.Hour { if ban.Expires.Sub(ban.Start) != time.Hour {
t.Errorf( t.Errorf(
"the next ban lasts %s, want 1h", "the next ban lasts %s, want 1h",
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"github.com/spf13/viper" "github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// healthCheckTimeout bounds the whole health check. // healthCheckTimeout bounds the whole health check.
@@ -116,7 +116,7 @@ func askHealthEndpoint(ctx context.Context, listenAddr string) error {
health := "http://" + net.JoinHostPort( health := "http://" + net.JoinHostPort(
"127.0.0.1", "127.0.0.1",
port, port,
) + waf.HealthPath ) + wafprotocol.HealthPath
req, err := http.NewRequestWithContext( req, err := http.NewRequestWithContext(
ctx, ctx,
+198
View File
@@ -0,0 +1,198 @@
package sww
import (
"context"
"errors"
"io"
"log/slog"
"net"
"net/http"
"time"
"go.uber.org/fx"
"go.uber.org/fx/fxevent"
)
// shutdownTimeout is how long requests in progress may take to finish
// once sww is told to stop, before their connections are closed.
// runit and docker wait a little longer before they kill the process.
const shutdownTimeout = 5 * time.Second
// requestLog is where a daemon's server writes the request log: stdout, and
// the remote log sender while there is one. fx tells parts apart by type, so
// it has a type of its own.
type requestLog struct {
io.Writer
}
// servingFailed gets why serving failed, for run, which then stops the
// daemon. fx tells parts apart by type, so it has a type of its own.
type servingFailed chan error
// utcNow is the time the state files, the alerts and the rate limits go by.
func utcNow() time.Time {
return time.Now().UTC()
}
// startError is why a daemon cannot start: the message its process log gives,
// and the error it logs with it.
type startError struct {
message string
err error
}
func (e startError) Error() string {
return e.message + ": " + e.err.Error()
}
func (e startError) Unwrap() error {
return e.err
}
// cannotStart logs why a daemon cannot start, with the message of the
// startError in err, and returns its exit status.
func cannotStart(processLog *slog.Logger, err error) int {
message, cause := "cannot start", err
failed, ok := errors.AsType[startError](err)
if ok {
message, cause = failed.message, failed.err
}
processLog.Error(message, "error", cause.Error())
return 1
}
// listen listens on address, which the setting names.
func listen(address, setting string) (net.Listener, error) {
listener, err := (&net.ListenConfig{}).Listen(
context.Background(), "tcp", address)
if err != nil {
return nil, startError{"cannot listen on " + setting, err}
}
return listener, nil
}
// background returns the hook that runs loop on a goroutine of its own from
// the start, and at the stop ends it and waits for it to return.
func background(loop func(context.Context)) fx.Hook {
running, stop := context.WithCancel(context.Background())
ended := make(chan struct{})
return fx.StartStopHook(
func() {
go func() {
loop(running)
close(ended)
}()
},
func() {
stop()
<-ended
},
)
}
// serve serves requests on listener from the start. At the stop it gives the
// requests in progress shutdownTimeout to finish, then closes their
// connections. When serving fails, it logs why and sends the error to
// failed.
func serve(
lc fx.Lifecycle, server *http.Server, listener net.Listener,
processLog *slog.Logger, failed servingFailed,
) {
served := make(chan error, 1)
lc.Append(fx.Hook{
OnStart: func(context.Context) error {
go func() {
err := server.Serve(listener)
if !errors.Is(err, http.ErrServerClosed) {
processLog.Error("serving failed", "error", err.Error())
failed <- err
}
served <- err
}()
return nil
},
OnStop: func(ctx context.Context) error {
processLog.Info("stopping")
shutdownCtx, cancel := context.WithTimeout(ctx, shutdownTimeout)
defer cancel()
err := server.Shutdown(shutdownCtx)
if err != nil {
processLog.Warn("requests still in progress were cut off",
"error", err.Error())
_ = server.Close()
}
err = <-served
if !errors.Is(err, http.ErrServerClosed) {
return err
}
return nil
},
})
}
// run starts app, which serves until ctx is done or serving fails, as
// failed tells. Then it stops app and takes the daemon's last step. It
// returns the process's exit status, 1 when serving failed, a part could
// not stop in order or the last step failed; each logs why.
func run(
ctx context.Context, app *fx.App, failed servingFailed,
processLog *slog.Logger, lastStep func() error,
) int {
// The hooks only start goroutines, and end them at the stop, so neither
// is cut short by a deadline or by ctx.
err := app.Start(context.WithoutCancel(ctx))
if err != nil {
return cannotStart(processLog, err)
}
status := 0
select {
case <-ctx.Done():
case <-failed:
status = 1
}
err = app.Stop(context.WithoutCancel(ctx))
if err != nil {
status = 1
}
err = lastStep()
if err != nil {
status = 1
}
if status == 0 {
processLog.Info("stopped")
}
return status
}
// fxEventLogger returns what fx.WithLogger takes to write fx's own events to
// processLog at debug, which the daemons' log levels normally hold back. Its
// errors are at debug too: the daemon logs each one itself.
func fxEventLogger(processLog *slog.Logger) func() fxevent.Logger {
return func() fxevent.Logger {
logger := &fxevent.SlogLogger{Logger: processLog}
logger.UseLogLevel(slog.LevelDebug)
logger.UseErrorLevel(slog.LevelDebug)
return logger
}
}
+48 -69
View File
@@ -2,14 +2,15 @@ package sww
import ( import (
"context" "context"
"errors"
"log/slog" "log/slog"
"net" "net"
"net/http" "net/http"
"os" "os"
"go.uber.org/fx"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
) )
@@ -22,34 +23,39 @@ import (
// setting for either. // setting for either.
func RunRProxy(ctx context.Context, params Params) int { func RunRProxy(ctx context.Context, params Params) int {
hostname, _ := os.Hostname() // "" when the host has no name to give hostname, _ := os.Hostname() // "" when the host has no name to give
processLog := requestlog.NewProcessLogger( processLog := processlog.New(processlog.Params{
params.Stdout, Output: params.Stdout,
hostname, InstanceName: hostname,
slog.LevelInfo, Level: slog.LevelInfo,
) })
cfg, err := config.RProxyFromEnvironment(params.Viper) cfg, err := config.RProxyFromEnvironment(params.Viper)
if err != nil { if err != nil {
processLog.Error("invalid setting", "error", err.Error()) return cannotStart(processLog, startError{"invalid setting", err})
return 1
} }
server := rproxy.New(rproxy.Params{ var (
Config: cfg, server *rproxy.Server
RequestLog: params.Stdout, listener net.Listener
ProcessLog: processLog, )
})
listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", cfg.ListenAddr) failed := make(servingFailed, 1)
app := fx.New(
fx.Supply(cfg, processLog, requestLog{params.Stdout}, failed),
fx.Provide(
newRProxyServer,
func(server *rproxy.Server) *http.Server { return server.Server },
newRProxyListener,
),
fx.Invoke(serve),
fx.Populate(&server, &listener),
fx.WithLogger(fxEventLogger(processLog)),
)
err = app.Err()
if err != nil { if err != nil {
processLog.Error( return cannotStart(processLog, err)
"cannot listen on SWW_RPROXY_LISTEN_ADDR",
"error",
err.Error(),
)
return 1
} }
processLog.Info("starting", processLog.Info("starting",
@@ -57,52 +63,25 @@ func RunRProxy(ctx context.Context, params Params) int {
"address", listener.Addr().String(), "address", listener.Addr().String(),
"settings", cfg) "settings", cfg)
return serveRProxy(ctx, server, listener, processLog) return run(ctx, app, failed, processLog, func() error {
server.WaitForReports()
return nil
})
} }
// serveRProxy serves requests on listener until ctx is done, and then stops as // newRProxyServer returns the server rproxy runs.
// RunRProxy says. func newRProxyServer(
func serveRProxy( cfg *config.RProxy, out requestLog, processLog *slog.Logger,
ctx context.Context, server *rproxy.Server, listener net.Listener, ) *rproxy.Server {
processLog *slog.Logger, return rproxy.New(rproxy.Params{
) int { Config: cfg,
served := make(chan error, 1) RequestLog: out,
ProcessLog: processLog,
go func() { })
served <- server.Serve(listener) }
}()
// newRProxyListener listens on SWW_RPROXY_LISTEN_ADDR.
select { func newRProxyListener(cfg *config.RProxy) (net.Listener, error) {
case err := <-served: return listen(cfg.ListenAddr, "SWW_RPROXY_LISTEN_ADDR")
processLog.Error("serving failed", "error", err.Error())
return 1
case <-ctx.Done():
}
processLog.Info("stopping")
shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx),
shutdownTimeout)
defer cancel()
err := server.Shutdown(shutdownCtx)
if err != nil {
processLog.Warn("requests still in progress were cut off",
"error", err.Error())
_ = server.Close()
}
err = <-served
if !errors.Is(err, http.ErrServerClosed) {
processLog.Error("serving failed", "error", err.Error())
return 1
}
server.WaitForReports()
processLog.Info("stopped")
return 0
} }
+30 -27
View File
@@ -21,7 +21,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/sww" "sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -51,7 +51,9 @@ func runRProxy(ctx context.Context, env map[string]string, out *output) int {
// startRProxy runs rproxy with waf at backend and the app at appURL, and // startRProxy runs rproxy with waf at backend and the app at appURL, and
// returns its URL and what it writes. It believes X-Forwarded-For from the // returns its URL and what it writes. It believes X-Forwarded-For from the
// tests. It stops as the test ends, and that it stopped in order is checked. // tests. It waits a minute for waf's answer, so that a slow answer on a busy
// host is not taken for no answer. It stops as the test ends, and that it
// stopped in order is checked.
func startRProxy(t *testing.T, backend, appURL string) (string, *output) { func startRProxy(t *testing.T, backend, appURL string) (string, *output) {
t.Helper() t.Helper()
@@ -65,6 +67,7 @@ func startRProxy(t *testing.T, backend, appURL string) (string, *output) {
"SWW_RPROXY_UPSTREAM_URL": appURL, "SWW_RPROXY_UPSTREAM_URL": appURL,
"SWW_RPROXY_TRUSTED_PROXIES": localhost + "/32", "SWW_RPROXY_TRUSTED_PROXIES": localhost + "/32",
wafBackend: backend, wafBackend: backend,
wafBackendTimeout: "60000",
}, out) }, out)
}() }()
@@ -161,7 +164,7 @@ func TestABanThroughOneSiteRefusesTheClientOnTheOther(t *testing.T) {
// One request's three lines carry one id, rproxy's with waf's answer. // One request's three lines carry one id, rproxy's with waf's answer.
line := firstOut.line(t, "path", bannedProbe) line := firstOut.line(t, "path", bannedProbe)
if line["disposition"] != waf.DispositionDeny || if line["disposition"] != wafprotocol.DispositionDeny ||
line["client_ip"] != placed { line["client_ip"] != placed {
t.Errorf("rproxy's line is %v", line) t.Errorf("rproxy's line is %v", line)
} }
@@ -246,7 +249,7 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
wantStatus(t, site+"/after-stop", placed, http.StatusOK) wantStatus(t, site+"/after-stop", placed, http.StatusOK)
line := out.line(t, "path", "/after-stop") line := out.line(t, "path", "/after-stop")
if line["disposition"] != waf.DispositionAllow || if line["disposition"] != wafprotocol.DispositionAllow ||
line["waf_backend_no_answer"] != true { line["waf_backend_no_answer"] != true {
t.Errorf("rproxy's line is %v", line) t.Errorf("rproxy's line is %v", line)
} }
@@ -434,7 +437,7 @@ func TestAChunkedBodyStaysABodyAtWAF(t *testing.T) {
const smuggledID = "smuggled-request" const smuggledID = "smuggled-request"
smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" + smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" +
waf.ClientHeader + ": " + otherClient + "\r\n" + wafprotocol.ClientHeader + ": " + otherClient + "\r\n" +
"X-Request-ID: " + smuggledID + "\r\n\r\n" "X-Request-ID: " + smuggledID + "\r\n\r\n"
status := sendRaw( status := sendRaw(
@@ -549,7 +552,9 @@ func TestAStatusWAFDoesNotTakeIsReportedAs599(t *testing.T) {
// startRProxyProcess runs `sww rproxy` in a process of its own, with the // startRProxyProcess runs `sww rproxy` in a process of its own, with the
// settings in env, each as NAME=value. It returns where rproxy listens and a // settings in env, each as NAME=value. It returns where rproxy listens and a
// function that sends rproxy SIGTERM and fails the test unless it exits with // function that sends rproxy SIGTERM and fails the test unless it exits with
// status 0 within waitLimit. // status 0 within waitLimit. It waits a minute for waf's answers unless env
// sets SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, so that a busy host cannot make it
// drop a question.
func startRProxyProcess(t *testing.T, env ...string) (string, func()) { func startRProxyProcess(t *testing.T, env ...string) (string, func()) {
t.Helper() t.Helper()
@@ -565,7 +570,8 @@ func startRProxyProcess(t *testing.T, env ...string) (string, func()) {
cmd.Env = append(os.Environ(), cmd.Env = append(os.Environ(),
commandForMain+"=rproxy", commandForMain+"=rproxy",
"SWW_RPROXY_LISTEN_ADDR="+localhost+":0") "SWW_RPROXY_LISTEN_ADDR="+localhost+":0",
wafBackendTimeout+"=60000")
cmd.Env = append(cmd.Env, env...) cmd.Env = append(cmd.Env, env...)
exited := startProcess(t, cmd) exited := startProcess(t, cmd)
@@ -599,10 +605,11 @@ func TestRProxyAsksAWAFThatSpeaksHTTP2OverHTTP1(t *testing.T) {
protocols := make(chan string, 10) protocols := make(chan string, 10)
backend := httptest.NewUnstartedServer(http.HandlerFunc( backend := httptest.NewUnstartedServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) != "" { if r.Header.Get(wafprotocol.ClientHeader) != "" {
protocols <- r.Proto protocols <- r.Proto
w.Header().Set(waf.DispositionHeader, waf.DispositionDeny) w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionDeny)
w.WriteHeader(http.StatusForbidden) w.WriteHeader(http.StatusForbidden)
} }
})) }))
@@ -684,8 +691,9 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{}) release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) != "" { if r.Header.Get(wafprotocol.ClientHeader) != "" {
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow) w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
return return
} }
@@ -749,8 +757,7 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
} }
} }
// sendAndKeepOpen sends request to rproxy at addr, and returns the connection. A // sendAndKeepOpen sends request to rproxy at addr, and returns the connection.
// read from it ends with an error once waitLimit is up.
func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn { func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
t.Helper() t.Helper()
@@ -763,11 +770,6 @@ func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
_ = conn.Close() _ = conn.Close()
}) })
err = conn.SetDeadline(time.Now().Add(waitLimit))
if err != nil {
t.Fatalf("set the deadline: %v", err)
}
_, err = io.WriteString(conn, request) _, err = io.WriteString(conn, request)
if err != nil { if err != nil {
t.Fatalf("send: %v", err) t.Fatalf("send: %v", err)
@@ -779,10 +781,7 @@ func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) { func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
t.Parallel() t.Parallel()
const ( const cutOffAfter = 5 * time.Second
cutOffAfter = 5 * time.Second
upstreamSetting = "SWW_RPROXY_UPSTREAM_URL"
)
asked := make(chan struct{}) asked := make(chan struct{})
release := make(chan struct{}) release := make(chan struct{})
@@ -796,7 +795,8 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) { func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow) w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
})) }))
t.Cleanup(backend.Close) t.Cleanup(backend.Close)
@@ -804,9 +804,9 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
defer stop() defer stop()
env := map[string]string{ env := map[string]string{
rproxyListenAddr: localhost + ":0", rproxyListenAddr: localhost + ":0",
upstreamSetting: app.URL, rproxyUpstreamURL: app.URL,
wafBackend: backend.URL, wafBackend: backend.URL,
} }
out := &output{} out := &output{}
exited := make(chan int, 1) exited := make(chan int, 1)
@@ -832,6 +832,9 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
stop() stop()
// Counted from the stop, so a slow wait for the app cannot shorten it.
_ = conn.SetReadDeadline(stopped.Add(cutOffAfter + waitLimit))
text, err := io.ReadAll(conn) text, err := io.ReadAll(conn)
if err != nil || len(text) != 0 { if err != nil || len(text) != 0 {
t.Errorf("read %q (%v), want the connection cut off", text, err) t.Errorf("read %q (%v), want the connection cut off", text, err)
@@ -929,7 +932,7 @@ func TestMainRunsRProxy(t *testing.T) {
addr, stop := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9") addr, stop := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9")
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
"http://"+addr+waf.HealthPath, http.NoBody) "http://"+addr+wafprotocol.HealthPath, http.NoBody)
if err != nil { if err != nil {
t.Fatalf("new request: %v", err) t.Fatalf("new request: %v", err)
} }
+52 -419
View File
@@ -4,59 +4,30 @@
// stop, and then stops in an orderly way, writing the state and cache files. // stop, and then stops in an orderly way, writing the state and cache files.
// `sww rproxy` reads rproxy's settings alone, and runs the reverse proxy in // `sww rproxy` reads rproxy's settings alone, and runs the reverse proxy in
// front of one app that asks waf about each request. `sww healthcheck` is the // front of one app that asks waf about each request. `sww healthcheck` is the
// container's health check. // container's health check. fx builds the parts of each daemon, starts their
// background work and stops it.
package sww package sww
import ( import (
"context" "context"
"errors" "errors"
"fmt"
"io" "io"
"log/slog"
"net"
"net/http"
"os" "os"
"os/signal" "os/signal"
"syscall" "syscall"
"time" "time"
"github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/state"
"sneak.berlin/go/smallwebwaf/internal/waf"
) )
// shutdownTimeout is how long requests in progress may take to finish
// once sww is told to stop, before their connections are closed.
// runit and docker wait a little longer before they kill the process.
const shutdownTimeout = 5 * time.Second
// remoteLogStopTimeout is how long, as waf stops, the log lines still
// waiting are sent to SWW_WAF_LOG_REMOTE_URL before they are given up. stdout has
// carried them.
const remoteLogStopTimeout = 2 * time.Second
// usageStatus is the exit status with no command, or one sww does not // usageStatus is the exit status with no command, or one sww does not
// know. // know.
const usageStatus = 2 const usageStatus = 2
// usage is what sww prints then. var errNoCommand = errors.New("no command given")
const usage = `usage: sww <command>
commands:
rproxy the reverse proxy in front of one app
waf decides for rproxy, and keeps bans, counters and history
healthcheck the container's health check
`
// Params are what RunWAF and RunRProxy need from the process. // Params are what RunWAF and RunRProxy need from the process.
type Params struct { type Params struct {
@@ -67,18 +38,16 @@ type Params struct {
Viper *viper.Viper Viper *viper.Viper
// Stdout receives the request log and the process's own messages. // Stdout receives the request log and the process's own messages.
Stdout io.Writer Stdout io.Writer
// BesideLookupTimeLimit is the waf.Params.BesideLookupTimeLimit RunWAF
// gives waf. Main leaves it zero, waf's own limit; tests lengthen it.
BesideLookupTimeLimit time.Duration
} }
// Main runs the command os.Args names, and returns the process's exit status. // Main runs the command os.Args names, and returns the process's exit status.
// `sww rproxy` and `sww waf` run until SIGTERM or SIGINT. // `sww rproxy` and `sww waf` run until SIGTERM or SIGINT.
// With no command, or one it does not know, it prints the commands and returns // With no command, or one it does not know, cobra prints the error and the
// usageStatus. // commands to stderr, and Main returns usageStatus.
func Main(version string) int { func Main(version string) int {
args := os.Args[1:]
if len(args) == 0 {
return printUsage(os.Stderr)
}
ctx, stop := signal.NotifyContext(context.Background(), ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGTERM, os.Interrupt) syscall.SIGTERM, os.Interrupt)
defer stop() defer stop()
@@ -89,390 +58,54 @@ func Main(version string) int {
Stdout: os.Stdout, Stdout: os.Stdout,
} }
switch args[0] { // status is what the command that ran returns.
case "rproxy": status := 0
return RunRProxy(ctx, params)
case "waf": root := &cobra.Command{
return RunWAF(ctx, params) Use: "sww",
case "healthcheck": Short: "a web application firewall",
return HealthCheck(ctx, args[1:], params.Viper, os.Stderr) Args: cobra.NoArgs,
default: RunE: func(*cobra.Command, []string) error {
return printUsage(os.Stderr) return errNoCommand
},
} }
} root.CompletionOptions.DisableDefaultCmd = true
// printUsage writes the commands to w, and returns usageStatus. root.AddCommand(
func printUsage(w io.Writer) int { &cobra.Command{
_, _ = fmt.Fprint(w, usage) Use: "rproxy",
Short: "the reverse proxy in front of one app",
return usageStatus Run: func(cmd *cobra.Command, _ []string) {
} status = RunRProxy(cmd.Context(), params)
},
// RunWAF is `sww waf`: it reads the settings, the rule files, },
// the lookup database and the state and cache files, then serves requests until &cobra.Command{
// ctx is done. It returns the process's exit status, 1 when waf cannot start. Use: "waf",
func RunWAF(ctx context.Context, params Params) int { Short: "decides for rproxy, and keeps bans, counters and history",
// Until the settings are read, the one message is an invalid setting's Run: func(cmd *cobra.Command, _ []string) {
// error, which every SWW_WAF_LOG_LEVEL lets through. status = RunWAF(cmd.Context(), params)
processLog := requestlog.NewProcessLogger(params.Stdout, },
config.InstanceName(params.Viper), slog.LevelError) },
&cobra.Command{
cfg, err := config.WAFFromEnvironment(params.Viper) Use: "healthcheck",
if err != nil { Short: "the container's health check",
return cannotStart(processLog, "invalid setting", err) // HealthCheck refuses every argument itself, flags included.
} DisableFlagParsing: true,
Run: func(cmd *cobra.Command, args []string) {
// While SWW_WAF_LOG_REMOTE_URL is set, every line on stdout from here on status = HealthCheck(
// is sent there too. cmd.Context(),
stdout := params.Stdout args,
params.Viper,
var remote *remotelog.Sender cmd.ErrOrStderr(),
)
if cfg.LogRemoteURL != nil { },
remote = newRemoteLogSender(cfg) },
stdout = io.MultiWriter(params.Stdout, remote)
}
processLog = requestlog.NewProcessLogger(
stdout,
cfg.InstanceName,
cfg.LogLevel,
) )
if remote != nil { err := root.ExecuteContext(ctx)
stopSending := startSending(ctx, remote, processLog)
defer stopSending()
}
return runWAFWithSettings(
ctx,
params.Version,
cfg,
stdout,
remote,
processLog,
)
}
// runWAFWithSettings is the rest of RunWAF once the settings are read and
// the process log goes where they say. remote is nil unless
// SWW_WAF_LOG_REMOTE_URL is set.
func runWAFWithSettings(
ctx context.Context,
version string,
cfg *config.WAF,
stdout io.Writer,
remote *remotelog.Sender,
processLog *slog.Logger,
) int {
// The state files and the alerts give times in UTC.
now := func() time.Time { return time.Now().UTC() }
alertQueue := newAlertQueue(cfg, now, processLog)
ruleFiles, err := rules.Load(rules.Params{
Dir: cfg.RuleFilesDir,
Enabled: cfg.RuleFilesEnabled,
ProcessLog: processLog,
Alerts: alertQueue,
})
if err != nil { if err != nil {
return cannotStart(processLog, "cannot use the rule files", err) return usageStatus
} }
crs, err := waf.NewCoreRuleSet(cfg) return status
if err != nil {
return cannotStart(processLog, "cannot use the Core Rule Set", err)
}
server, err := newServer(
cfg,
stdout,
processLog,
now,
ruleFiles,
alertQueue,
crs,
)
if err != nil {
return cannotStart(processLog, "cannot use the lookup database", err)
}
if remote != nil {
server.Metrics.AddRemoteLog(remote)
}
files, err := loadStateFiles(cfg, server, alertQueue, now, processLog)
if err != nil {
return cannotStart(
processLog,
"cannot use the state and cache files",
err,
)
}
listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", cfg.ListenAddr)
if err != nil {
return cannotStart(
processLog,
"cannot listen on SWW_WAF_LISTEN_ADDR",
err,
)
}
processLog.Info("starting",
"version", version,
"address", listener.Addr().String(),
"settings", cfg)
return serve(
ctx,
server,
listener,
files,
ruleFiles,
alertQueue,
processLog,
)
}
// cannotStart logs why waf cannot start, and returns its exit status.
func cannotStart(processLog *slog.Logger, message string, err error) int {
processLog.Error(message, "error", err.Error())
return 1
}
// newServer returns the server waf runs, with the metrics of the alerts,
// after reading the lookup database while SWW_WAF_LOOKUP_SOURCE is file. A lookup
// database that cannot be read is an error.
func newServer(
cfg *config.WAF, stdout io.Writer, processLog *slog.Logger,
now func() time.Time, ruleFiles *rules.Files, alertQueue *alerts.Queue,
crs *coreruleset.CoreRuleSet,
) (*waf.Server, error) {
var lookupFile *lookup.File
if cfg.LookupSource == "file" {
var err error
lookupFile, err = lookup.OpenFile(lookup.FileParams{
Path: cfg.LookupDBPath,
Now: now,
ProcessLog: processLog,
Alerts: alertQueue,
})
if err != nil {
return nil, err
}
}
server := waf.New(waf.Params{
Config: cfg,
RequestLog: stdout,
ProcessLog: processLog,
GeoJSURL: geojs.URL,
AbuseIPDBURL: reputation.AbuseIPDBURL,
LookupFile: lookupFile,
CoreRuleSet: crs,
Now: now,
Rules: ruleFiles,
Alerts: alertQueue,
})
server.Metrics.AddAlerts(alertQueue)
if lookupFile != nil {
server.Metrics.AddLookupFile(
lookupFile.LastRead,
lookupFile.ReadFailures,
)
}
return server, nil
}
// loadStateFiles reads the state and cache files into the parts of server and
// into alertQueue, as state.Load does.
func loadStateFiles(
cfg *config.WAF, server *waf.Server, alertQueue *alerts.Queue,
now func() time.Time, processLog *slog.Logger,
) (*state.Files, error) {
return state.Load(state.Params{
Dir: cfg.StateDir,
CacheDir: cfg.CacheDir,
BansWriteDelay: cfg.BansWriteDelay,
WriteInterval: cfg.StateWriteInterval,
Ledger: server.Ledger,
Limiter: server.Limiter,
GeoJS: server.GeoJS,
Lists: server.Lists,
DNSBL: server.DNSBL,
AbuseIPDB: server.AbuseIPDB,
RouteWatch: server.RouteWatch,
Alerts: alertQueue,
Anomalies: server.Anomalies,
Now: now,
ProcessLog: processLog,
Metrics: server.Metrics,
})
}
// newAlertQueue returns the queue of the alerts to the webhook, Slack and
// ntfy, with the settings for them.
func newAlertQueue(
cfg *config.WAF, now func() time.Time, processLog *slog.Logger,
) *alerts.Queue {
return alerts.New(alerts.Params{
WebhookURL: cfg.AlertWebhookURL,
WebhookHeaders: cfg.AlertWebhookHeaders,
SlackURL: cfg.AlertSlackWebhookURL,
NtfyURL: cfg.AlertNtfyURL,
NtfyToken: cfg.AlertNtfyToken,
Events: cfg.AlertEvents,
Cooldown: cfg.AlertCooldown,
MaxPerHour: cfg.AlertMaxPerHour,
Instance: cfg.InstanceName,
Now: now,
ProcessLog: processLog,
})
}
// newRemoteLogSender returns a sender of the log lines to
// SWW_WAF_LOG_REMOTE_URL, with the settings for it.
func newRemoteLogSender(cfg *config.WAF) *remotelog.Sender {
return remotelog.New(remotelog.Params{
URL: cfg.LogRemoteURL,
RootCAs: cfg.LogRemoteTLSCAs,
BufferLines: cfg.LogRemoteBufferLines,
Facility: cfg.LogRemoteFacility,
AppName: cfg.LogRemoteAppName,
})
}
// startSending runs remote until the function it returns is called, which
// then waits at most remoteLogStopTimeout for the lines still waiting to
// be sent. Sending goes on after ctx is done, so that the lines written
// while sww stops are sent too.
func startSending(
ctx context.Context, remote *remotelog.Sender, processLog *slog.Logger,
) func() {
sending, stop := context.WithCancel(context.WithoutCancel(ctx))
sent := make(chan struct{})
go func() {
remote.Run(sending, processLog)
close(sent)
}()
return func() {
stop()
select {
case <-sent:
case <-time.After(remoteLogStopTimeout):
}
}
}
// serve serves requests on listener, writes the state and cache files as they
// are due, takes in an admin's edits of them, reads the rule files again as
// they change, and the lookup database when it is replaced, fetches the
// lists the settings name by URL as they are due, and sends the alerts,
// until ctx is done. Then it gives the requests in progress
// shutdownTimeout to finish, and writes every state and cache file,
// alerts.json with the alerts still waiting.
func serve(
ctx context.Context, server *waf.Server, listener net.Listener,
files *state.Files, ruleFiles *rules.Files, alertQueue *alerts.Queue,
processLog *slog.Logger,
) int {
served := make(chan error, 1)
go func() {
served <- server.Serve(listener)
}()
writing, stopWriting := context.WithCancel(ctx)
defer stopWriting()
written := inBackground(func() { files.Run(writing) })
watched := inBackground(func() { files.Watch(writing) })
rulesWatched := inBackground(func() { ruleFiles.Watch(writing) })
lookupFileWatched := inBackground(func() {
if server.LookupFile != nil {
server.LookupFile.Watch(writing)
}
})
listsFetched := inBackground(func() { server.Lists.Run(writing) })
alertsSent := inBackground(func() { alertQueue.Run(writing) })
select {
case err := <-served:
processLog.Error("serving failed", "error", err.Error())
return 1
case <-ctx.Done():
}
processLog.Info("stopping")
shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx),
shutdownTimeout)
defer cancel()
err := server.Shutdown(shutdownCtx)
if err != nil {
processLog.Warn("requests still in progress were cut off",
"error", err.Error())
_ = server.Close()
}
err = <-served
if !errors.Is(err, http.ErrServerClosed) {
processLog.Error("serving failed", "error", err.Error())
return 1
}
// Run and Watch have ended, so nothing else reads or writes the
// files, and no alert is being sent, so that alerts.json keeps every
// alert not yet sent. Every request has ended too, but for one cut off
// because Shutdown timed out, which adds to its client's history only as
// it ends. That can be after this write, and then that request is missing
// from clients.json.
<-written
<-watched
<-rulesWatched
<-lookupFileWatched
<-listsFetched
<-alertsSent
err = files.WriteAll()
if err != nil {
processLog.Error(
"writing the state and cache files failed",
"error",
err.Error(),
)
return 1
}
processLog.Info("stopped")
return 0
}
// inBackground runs task on a goroutine of its own, and returns a channel
// that is closed once task has returned.
func inBackground(task func()) <-chan struct{} {
done := make(chan struct{})
go func() {
task()
close(done)
}()
return done
} }
+8 -3
View File
@@ -23,7 +23,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/sww" "sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -173,6 +173,9 @@ func run(ctx context.Context, env map[string]string, out *output) int {
Version: testVersion, Version: testVersion,
Viper: v, Viper: v,
Stdout: out, Stdout: out,
// The race detector and a busy host can slow the Core Rule Set's
// inspections past waf's own limit.
BesideLookupTimeLimit: time.Hour,
}) })
} }
@@ -657,7 +660,9 @@ func TestCoreRuleSetExclusionsFileThatCannotBeUsedStopsTheStartWhileTheCoreRuleS
status := run(t.Context(), map[string]string{ status := run(t.Context(), map[string]string{
listenAddr: localhost + ":0", stateDir: t.TempDir(), ruleFilesDir: t.TempDir(), listenAddr: localhost + ":0", stateDir: t.TempDir(), ruleFilesDir: t.TempDir(),
"SWW_WAF_CORE_RULE_SET_MODE": config.CoreRuleSetModeOff, "SWW_WAF_CORE_RULE_SET_MODE": string(
config.CoreRuleSetModeOff,
),
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE": path, "SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE": path,
}, out) }, out)
if status != 1 { if status != 1 {
@@ -1808,7 +1813,7 @@ func statusFrom(t *testing.T, url, from string) int {
} }
req.Header.Set("X-Forwarded-For", from) req.Header.Set("X-Forwarded-For", from)
req.Header.Set(waf.ClientHeader, from) req.Header.Set(wafprotocol.ClientHeader, from)
req.Header.Set("X-Request-ID", rand.Text()) req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{} transport := &http.Transport{}
+560
View File
@@ -0,0 +1,560 @@
package sww
import (
"context"
"fmt"
"io"
"log/slog"
"net"
"net/http"
"time"
"go.uber.org/fx"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/routewatch"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/state"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// remoteLogStopTimeout is how long, as waf stops, the log lines still
// waiting are sent to SWW_WAF_LOG_REMOTE_URL before they are given up. stdout has
// carried them.
const remoteLogStopTimeout = 2 * time.Second
// RunWAF is `sww waf`: it reads the settings, the rule files,
// the lookup database and the state and cache files, then serves requests until
// ctx is done. Then it gives the requests in progress shutdownTimeout to
// finish, ends its background work and writes every state and cache file,
// alerts.json with the alerts still waiting. It returns the process's exit
// status, 1 when waf cannot start or stop in order.
func RunWAF(ctx context.Context, params Params) int {
// Until the settings are read, the one message is an invalid setting's
// error, which every SWW_WAF_LOG_LEVEL lets through.
processLog := processlog.New(processlog.Params{
Output: params.Stdout,
InstanceName: config.InstanceName(params.Viper),
Level: slog.LevelError,
})
cfg, err := config.WAFFromEnvironment(params.Viper)
if err != nil {
return cannotStart(processLog, startError{"invalid setting", err})
}
// While SWW_WAF_LOG_REMOTE_URL is set, every line on stdout from here on
// is sent there too. The sender runs before any part is built, so that it
// sends why waf cannot start.
stdout := params.Stdout
var remote *remotelog.Sender
if cfg.LogRemoteURL != nil {
remote = newRemoteLogSender(cfg)
stdout = io.MultiWriter(params.Stdout, remote)
}
processLog = processlog.New(processlog.Params{
Output: stdout,
InstanceName: cfg.InstanceName,
Level: cfg.LogLevel,
})
if remote != nil {
stopSending := startSending(ctx, remote, processLog)
defer stopSending()
}
var (
files *state.Files
listener net.Listener
)
failed := make(servingFailed, 1)
app := fx.New(
fx.Supply(
params,
cfg,
processLog,
requestLog{stdout},
remote,
failed,
),
fx.Provide(
newAlertQueue, newRuleFiles, newCoreRuleSet, newLookupFile,
newLimiter, newLedger, newAnomalyCounters,
newLists, newDNSBL, newAbuseIPDB, newRouteWatch,
newMetrics, newGeoJS, newWAFServer,
func(server *waf.Server) *http.Server { return server.Server },
newWAFListener, newStateFiles,
),
// fx stops hooks in the reverse of the order they were added in. The
// state files are read before serving is set up, so that serving
// stops before their loops do.
fx.Populate(&files),
fx.Invoke(serve),
fx.Populate(&listener),
fx.WithLogger(fxEventLogger(processLog)),
)
err = app.Err()
if err != nil {
return cannotStart(processLog, err)
}
processLog.Info("starting",
"version", params.Version,
"address", listener.Addr().String(),
"settings", cfg)
// The last write follows the stop of every hook, and so the end of every
// loop that changes the state or raises an alert, and of every ask of
// GeoJS, the DNSBL zones, AbuseIPDB and routewatch. A hook of the state
// files would stop before the loops of the parts built ahead of them.
return run(ctx, app, failed, processLog, func() error {
err := files.WriteAll()
if err != nil {
processLog.Error("writing the state and cache files failed",
"error", err.Error())
}
return err
})
}
// newAlertQueue returns the queue of the alerts to the webhook, Slack and
// ntfy, with the settings for them. Until waf stops, it sends them.
func newAlertQueue(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
) *alerts.Queue {
alertQueue := alerts.New(alerts.Params{
WebhookURL: cfg.AlertWebhookURL,
WebhookHeaders: cfg.AlertWebhookHeaders,
SlackURL: cfg.AlertSlackWebhookURL,
NtfyURL: cfg.AlertNtfyURL,
NtfyToken: cfg.AlertNtfyToken,
Events: cfg.AlertEvents,
Cooldown: cfg.AlertCooldown,
MaxPerHour: cfg.AlertMaxPerHour,
Instance: cfg.InstanceName,
Now: utcNow,
ProcessLog: processLog,
})
lc.Append(background(alertQueue.Run))
return alertQueue
}
// newRuleFiles reads the rule files, and reads them again as they change
// until waf stops.
func newRuleFiles(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
alertQueue *alerts.Queue,
) (*rules.Files, error) {
ruleFiles, err := rules.Load(rules.Params{
Dir: cfg.RuleFilesDir,
Enabled: cfg.RuleFilesEnabled,
ProcessLog: processLog,
Alerts: alertQueue,
})
if err != nil {
return nil, startError{"cannot use the rule files", err}
}
lc.Append(background(ruleFiles.Watch))
return ruleFiles, nil
}
// newCoreRuleSet returns the Core Rule Set at SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL,
// without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies
// up to SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE, with the exclusions in
// SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS,
// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. An exclusions file that
// coreruleset.New refuses is an error in every mode. While the mode is off, a
// set file is loaded with the Core Rule Set only to check it.
func newCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff &&
cfg.CoreRuleSetExclusionsFile == "" {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
}
crs, err := coreruleset.New(coreruleset.Params{
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
DisabledRules: cfg.CoreRuleSetDisabledRules,
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
Apps: cfg.Apps,
})
if err != nil {
return nil, startError{
"cannot use the Core Rule Set",
fmt.Errorf("SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE: %w", err),
}
}
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
}
return crs, nil
}
// newLookupFile reads the lookup database while SWW_WAF_LOOKUP_SOURCE is
// file, and reads it again when it is replaced until waf stops. It returns
// nil while the source is anything else.
func newLookupFile(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
alertQueue *alerts.Queue,
) (*lookup.File, error) {
if cfg.LookupSource != config.LookupSourceFile {
return nil, nil //nolint:nilnil // no lookup database, and no error
}
lookupFile, err := lookup.OpenFile(lookup.FileParams{
Path: cfg.LookupDBPath,
Now: utcNow,
ProcessLog: processLog,
Alerts: alertQueue,
})
if err != nil {
return nil, startError{"cannot use the lookup database", err}
}
lc.Append(background(lookupFile.Watch))
return lookupFile, nil
}
// newLimiter returns the table of clients, with the rate and byte limits.
func newLimiter(cfg *config.WAF) *ratelimit.Limiter {
return ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{
PerMinute: cfg.RateLimitPerMinute,
PerHour: cfg.RateLimitPerHour,
PerDay: cfg.RateLimitPerDay,
BytesPerMinute: cfg.BytesLimitPerMinute,
BytesPerHour: cfg.BytesLimitPerHour,
BytesPerDay: cfg.BytesLimitPerDay,
},
MaxClients: cfg.MaxTrackedClients,
})
}
// newLedger returns the ledger of the bans, with the settings for them.
func newLedger(cfg *config.WAF) *bans.Ledger {
return bans.New(bans.Params{
LimitBanDuration: cfg.LimitBanDuration,
LimitBanRepeatWindow: cfg.LimitBanRepeatWindow,
LimitBanPermanentAbove: cfg.LimitBanPermanentAbove,
AttackBanDuration: cfg.AttackBanDuration,
MaxBans: cfg.MaxBans,
})
}
// newAnomalyCounters returns the counters of the anomaly thresholds.
func newAnomalyCounters(
cfg *config.WAF, alertQueue *alerts.Queue,
) *anomaly.Counters {
return anomaly.New(anomaly.Params{
Client: cfg.AnomalyClient,
Net: cfg.AnomalyNet,
ASN: cfg.AnomalyASN,
Total: cfg.AnomalyTotal,
WatchedNet: cfg.AnomalyWatchedNet,
NetV4Prefix: cfg.AnomalyNetV4Prefix,
NetV6Prefix: cfg.AnomalyNetV6Prefix,
NamedNetblocks: cfg.AnomalyWatchedNets,
Alerts: alertQueue,
})
}
// newLists returns the lists the settings name by URL, none fetched yet.
// Until waf stops, it fetches them as they are due.
func newLists(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
alertQueue *alerts.Queue,
) *reputation.Lists {
lists := reputation.New(reputation.Params{
BlocklistURLs: cfg.BlocklistURLs,
Refresh: cfg.BlocklistRefresh,
ASNLimitPercentURL: cfg.ASNLimitPercentURL,
TagLists: cfg.TagLists,
Now: utcNow,
ProcessLog: processLog,
Alerts: alertQueue,
})
lc.Append(background(lists.Run))
return lists
}
// newDNSBL returns what asks the DNSBL zones, none asked yet. At the stop it
// ends the queries under way, as reputation.DNSBL.StopAsking says.
func newDNSBL(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
alertQueue *alerts.Queue,
) *reputation.DNSBL {
dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
Zones: cfg.DNSBLZones,
Resolver: cfg.DNSBLResolver,
CacheTTL: cfg.IPReputationCacheTTL,
Timeout: cfg.DNSBLAbuseIPDBTimeout,
Now: utcNow,
ProcessLog: processLog,
Alerts: alertQueue,
})
lc.Append(fx.StopHook(dnsbl.StopAsking))
return dnsbl
}
// newAbuseIPDB returns what checks clients with AbuseIPDB while
// SWW_WAF_ABUSEIPDB_KEY is set, none checked yet. Until waf stops, it
// fetches AbuseIPDB's blacklist as it is due, and bans the clients on it in
// ledger, as waf.BanAbuseIPDBBlacklist says. At the stop it ends the checks
// under way, as reputation.AbuseIPDB.StopChecking says.
func newAbuseIPDB(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
alertQueue *alerts.Queue, ledger *bans.Ledger,
) *reputation.AbuseIPDB {
abuseIPDB := reputation.NewAbuseIPDB(reputation.AbuseIPDBParams{
URL: reputation.AbuseIPDBURL,
Key: cfg.AbuseIPDBKey,
MinScore: cfg.AbuseIPDBMinScore,
DailyBudget: cfg.AbuseIPDBDailyBudget,
CacheTTL: cfg.IPReputationCacheTTL,
Timeout: cfg.DNSBLAbuseIPDBTimeout,
BlacklistURL: reputation.AbuseIPDBBlacklistURL,
Threshold: cfg.AbuseIPDBThreshold,
BlacklistRefresh: cfg.AbuseIPDBBlacklistRefresh,
Blacklisted: func(listed []reputation.BlacklistEntry) {
waf.BanAbuseIPDBBlacklist(
cfg, ledger, processLog, utcNow(), listed,
)
},
Now: utcNow,
ProcessLog: processLog,
Alerts: alertQueue,
})
lc.Append(background(abuseIPDB.RunBlacklist))
lc.Append(fx.StopHook(abuseIPDB.StopChecking))
return abuseIPDB
}
// newRouteWatch returns what asks routewatch about clients, or nil while
// SWW_WAF_ROUTEWATCH_URL is unset. At the stop it ends the queries under
// way, as routewatch.Client.StopAsking says.
func newRouteWatch(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
) *routewatch.Client {
if cfg.RouteWatchURL == "" {
return nil
}
routeWatch := routewatch.New(routewatch.Params{
URL: cfg.RouteWatchURL,
Now: utcNow,
ProcessLog: processLog,
})
lc.Append(fx.StopHook(routeWatch.StopAsking))
return routeWatch
}
// newMetrics returns the metrics, with those read from the other parts as
// the metrics are asked for: AbuseIPDB's while SWW_WAF_ABUSEIPDB_KEY is set,
// the lookup database's while there is one, and the remote log sender's
// while there is one.
func newMetrics(
cfg *config.WAF, ledger *bans.Ledger, limiter *ratelimit.Limiter,
ruleFiles *rules.Files, lists *reputation.Lists,
dnsbl *reputation.DNSBL, abuseIPDB *reputation.AbuseIPDB,
alertQueue *alerts.Queue, lookupFile *lookup.File,
remote *remotelog.Sender,
) *metrics.Metrics {
m := metrics.New(cfg.MetricsBusiestCountriesAndASNs, cfg.InstanceName)
m.AddBansAndClients(ledger, limiter, utcNow)
m.AddRules(ruleFiles)
m.AddReputation(lists, dnsbl)
if cfg.AbuseIPDBKey != "" {
m.AddAbuseIPDB(abuseIPDB)
}
m.AddAlerts(alertQueue)
if lookupFile != nil {
m.AddLookupFile(lookupFile.LastRead, lookupFile.ReadFailures)
}
if remote != nil {
m.AddRemoteLog(remote)
}
return m
}
// newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each
// answer goes to the limiter and the ledger, as waf.AddLookup says, and each
// failure is handled as waf.GeoJSFailed says. At the stop it ends the request
// under way, as geojs.GeoJS.StopAsking says.
func newGeoJS(
lc fx.Lifecycle,
cfg *config.WAF,
m *metrics.Metrics,
limiter *ratelimit.Limiter,
ledger *bans.Ledger,
alertQueue *alerts.Queue,
processLog *slog.Logger,
) *geojs.GeoJS {
geoJS := geojs.New(geojs.Params{
URL: geojs.URL,
Answered: func(answer geojs.Answer) {
waf.AddLookup(cfg, limiter, ledger, answer)
},
Now: utcNow,
Asked: m.GeoJSRequests.Inc,
Failed: func(err error, askingAgainIn time.Duration) {
waf.GeoJSFailed(processLog, alertQueue, m, err, askingAgainIn)
},
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
})
lc.Append(fx.StopHook(geoJS.StopAsking))
return geoJS
}
// newWAFServer returns the server waf runs, with its parts and the time
// limit params gives.
func newWAFServer(
params Params, cfg *config.WAF, out requestLog, processLog *slog.Logger,
m *metrics.Metrics, limiter *ratelimit.Limiter, ledger *bans.Ledger,
anomalies *anomaly.Counters, lists *reputation.Lists,
dnsbl *reputation.DNSBL, abuseIPDB *reputation.AbuseIPDB,
routeWatch *routewatch.Client, geoJS *geojs.GeoJS,
lookupFile *lookup.File, crs *coreruleset.CoreRuleSet,
ruleFiles *rules.Files, alertQueue *alerts.Queue,
) *waf.Server {
return waf.New(waf.Params{
Config: cfg,
RequestLog: out,
ProcessLog: processLog,
Metrics: m,
Limiter: limiter,
Ledger: ledger,
Anomalies: anomalies,
Lists: lists,
DNSBL: dnsbl,
AbuseIPDB: abuseIPDB,
RouteWatch: routeWatch,
GeoJS: geoJS,
LookupFile: lookupFile,
CoreRuleSet: crs,
Now: utcNow,
Rules: ruleFiles,
Alerts: alertQueue,
BesideLookupTimeLimit: params.BesideLookupTimeLimit,
})
}
// newWAFListener listens on SWW_WAF_LISTEN_ADDR.
func newWAFListener(cfg *config.WAF) (net.Listener, error) {
return listen(cfg.ListenAddr, "SWW_WAF_LISTEN_ADDR")
}
// newStateFiles reads the state and cache files into the parts and the
// alert queue, as state.Load does. Until waf stops, it writes them as they
// are due and takes in an admin's edits of them.
func newStateFiles(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
m *metrics.Metrics, ledger *bans.Ledger, limiter *ratelimit.Limiter,
geoJS *geojs.GeoJS, lists *reputation.Lists, dnsbl *reputation.DNSBL,
abuseIPDB *reputation.AbuseIPDB, routeWatch *routewatch.Client,
alertQueue *alerts.Queue, anomalies *anomaly.Counters,
) (*state.Files, error) {
files, err := state.Load(state.Params{
Dir: cfg.StateDir,
CacheDir: cfg.CacheDir,
BansWriteDelay: cfg.BansWriteDelay,
WriteInterval: cfg.StateWriteInterval,
Ledger: ledger,
Limiter: limiter,
GeoJS: geoJS,
Lists: lists,
DNSBL: dnsbl,
AbuseIPDB: abuseIPDB,
RouteWatch: routeWatch,
Alerts: alertQueue,
Anomalies: anomalies,
Now: utcNow,
ProcessLog: processLog,
Metrics: m,
})
if err != nil {
return nil, startError{"cannot use the state and cache files", err}
}
lc.Append(background(files.Run))
lc.Append(background(files.Watch))
return files, nil
}
// newRemoteLogSender returns a sender of the log lines to
// SWW_WAF_LOG_REMOTE_URL, with the settings for it.
func newRemoteLogSender(cfg *config.WAF) *remotelog.Sender {
return remotelog.New(remotelog.Params{
URL: cfg.LogRemoteURL,
RootCAs: cfg.LogRemoteTLSCAs,
BufferLines: cfg.LogRemoteBufferLines,
Facility: cfg.LogRemoteFacility,
AppName: cfg.LogRemoteAppName,
})
}
// startSending runs remote until the function it returns is called, which
// then waits at most remoteLogStopTimeout for the lines still waiting to
// be sent. Sending goes on after ctx is done, so that the lines written
// while sww stops are sent too.
func startSending(
ctx context.Context, remote *remotelog.Sender, processLog *slog.Logger,
) func() {
sending, stop := context.WithCancel(context.WithoutCancel(ctx))
sent := make(chan struct{})
go func() {
remote.Run(sending, processLog)
close(sent)
}()
return func() {
stop()
select {
case <-sent:
case <-time.After(remoteLogStopTimeout):
}
}
}
+106 -38
View File
@@ -4,6 +4,7 @@ import (
"bytes" "bytes"
"context" "context"
"crypto/rand" "crypto/rand"
"errors"
"io" "io"
"net/http" "net/http"
"os" "os"
@@ -20,7 +21,7 @@ import (
_ "time/tzdata" _ "time/tzdata"
"sneak.berlin/go/smallwebwaf/internal/sww" "sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// commandForMain, while set in the environment, has this test binary run Main // commandForMain, while set in the environment, has this test binary run Main
@@ -90,6 +91,8 @@ func TestMainRunsWAF(t *testing.T) {
cmd := exec.CommandContext(t.Context(), binary) cmd := exec.CommandContext(t.Context(), binary)
cmd.Stdout = out cmd.Stdout = out
// The Core Rule Set is off. Main keeps waf's own limit for its checks,
// and on a busy host the Core Rule Set's inspection can pass it.
cmd.Env = append(os.Environ(), cmd.Env = append(os.Environ(),
"TZ="+zone, "TZ="+zone,
commandForMain+"="+wafCommand, commandForMain+"="+wafCommand,
@@ -97,13 +100,14 @@ func TestMainRunsWAF(t *testing.T) {
stateDir+"="+dir, stateDir+"="+dir,
cacheDir+"="+dir, cacheDir+"="+dir,
ruleFilesDir+"="+t.TempDir(), ruleFilesDir+"="+t.TempDir(),
lookupSource+"=off") lookupSource+"=off",
"SWW_WAF_CORE_RULE_SET_MODE=off")
exited := startProcess(t, cmd) exited := startProcess(t, cmd)
starting := out.lineBeforeExit(t, "starting", exited) starting := out.lineBeforeExit(t, "starting", exited)
addr, _ := starting["address"].(string) addr, _ := starting["address"].(string)
ask(t, addr, "/", http.StatusOK, waf.DispositionAllow) ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow)
// The files are written as sww stops. // The files are written as sww stops.
err = cmd.Process.Signal(syscall.SIGTERM) err = cmd.Process.Signal(syscall.SIGTERM)
@@ -161,46 +165,104 @@ func wantUTC(t *testing.T, where string, value any) {
func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) { func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) {
t.Parallel() t.Parallel()
// completion is cobra's own command, which sww turns off.
for _, command := range []string{"", "serve", "completion"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, command)
if code != 2 {
t.Errorf("exit status %d, want 2", code)
}
wantTheCommands(t, "stderr", stderr)
if stdout != "" {
t.Errorf("stdout %q, want nothing", stdout)
}
})
}
}
func TestMainHelpPrintsTheCommands(t *testing.T) {
t.Parallel()
for _, command := range []string{"help", "--help"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, _ := runMain(t, command)
if code != 0 {
t.Errorf("exit status %d, want 0", code)
}
wantTheCommands(t, "stdout", stdout)
})
}
}
// wantTheCommands checks that text, what Main wrote to where, names each
// command.
func wantTheCommands(t *testing.T, where, text string) {
t.Helper()
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(text, name) {
t.Errorf("%s %q does not name %s", where, text, name)
}
}
}
func TestMainHealthCheckRefusesAnArgument(t *testing.T) {
t.Parallel()
for _, argument := range []string{"now", "--x", "-h"} {
t.Run("argument "+argument, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, "healthcheck "+argument)
want := "sww healthcheck: unexpected argument \"" +
argument + "\"\n"
if code != 1 || stderr != want || stdout != "" {
t.Errorf("exit status %d, stdout %q and stderr %q, "+
"want 1, nothing and %q", code, stdout, stderr, want)
}
})
}
}
// runMain runs Main with command in a process of its own, and returns its
// exit status and what it wrote to stdout and stderr.
func runMain(t *testing.T, command string) (int, string, string) {
t.Helper()
binary, err := os.Executable() binary, err := os.Executable()
if err != nil { if err != nil {
t.Fatalf("find the test binary: %v", err) t.Fatalf("find the test binary: %v", err)
} }
for _, command := range []string{"", "serve"} { ctx, cancel := context.WithTimeout(t.Context(), waitLimit)
t.Run("command "+command, func(t *testing.T) { defer cancel()
t.Parallel()
ctx, cancel := context.WithTimeout(t.Context(), waitLimit) var stdout, stderr bytes.Buffer
defer cancel()
var stdout, stderr bytes.Buffer //nolint:gosec // this test binary
cmd := exec.CommandContext(ctx, binary)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
cmd := exec.CommandContext(ctx, binary) cmd.Env = append(os.Environ(), commandForMain+"="+command)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
cmd.Env = append(os.Environ(), commandForMain+"="+command) err = cmd.Run()
err := cmd.Run() var exitErr *exec.ExitError
if err != nil && !errors.As(err, &exitErr) {
if code := cmd.ProcessState.ExitCode(); code != 2 { t.Fatalf("run: %v", err)
t.Errorf("exit status %d (%v), want 2", code, err)
}
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(stderr.String(), name) {
t.Errorf(
"stderr %q does not name %s",
stderr.String(),
name,
)
}
}
if stdout.Len() != 0 {
t.Errorf("stdout %q, want nothing", stdout.String())
}
})
} }
return cmd.ProcessState.ExitCode(), stdout.String(), stderr.String()
} }
// ask sends waf at addr rproxy's question about a request for path from // ask sends waf at addr rproxy's question about a request for path from
@@ -214,7 +276,7 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
t.Fatalf("new request: %v", err) t.Fatalf("new request: %v", err)
} }
req.Header.Set(waf.ClientHeader, placed) req.Header.Set(wafprotocol.ClientHeader, placed)
req.Header.Set("X-Request-ID", rand.Text()) req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{} transport := &http.Transport{}
@@ -229,11 +291,11 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
_ = res.Body.Close() _ = res.Body.Close()
if res.StatusCode != status || if res.StatusCode != status ||
res.Header.Get(waf.DispositionHeader) != disposition { res.Header.Get(wafprotocol.DispositionHeader) != disposition {
t.Errorf( t.Errorf(
"answered %d with disposition %q, want %d with %q", "answered %d with disposition %q, want %d with %q",
res.StatusCode, res.StatusCode,
res.Header.Get(waf.DispositionHeader), res.Header.Get(wafprotocol.DispositionHeader),
status, status,
disposition, disposition,
) )
@@ -284,12 +346,18 @@ func TestWAFKeepsItsBansAcrossARestartAndNeedsNoApp(t *testing.T) {
} }
runWAF(t, env, func(addr string) { runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusOK, waf.DispositionAllow) ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow)
// The default rule file bans a probe for this. // The default rule file bans a probe for this.
ask(t, addr, "/.env", http.StatusForbidden, waf.DispositionDeny) ask(
t,
addr,
"/.env",
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
}) })
runWAF(t, env, func(addr string) { runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusForbidden, waf.DispositionDeny) ask(t, addr, "/", http.StatusForbidden, wafprotocol.DispositionDeny)
}) })
} }
+20 -9
View File
@@ -22,31 +22,42 @@ import (
// an offence, and the request does not wait for the answer. The score is then // an offence, and the request does not wait for the answer. The score is then
// used for each address of the client. Its figures are the score and // used for each address of the client. Its figures are the score and
// SWW_WAF_ABUSEIPDB_MIN_SCORE. // SWW_WAF_ABUSEIPDB_MIN_SCORE.
type abuseIPDB struct{} type abuseIPDB struct {
abuseIPDB *reputation.AbuseIPDB
// limiter is the table of clients the rate limits count, whose histories
// say which clients are offenders.
limiter *ratelimit.Limiter
keySet bool // SWW_WAF_ABUSEIPDB_KEY is set
minScore int64 // SWW_WAF_ABUSEIPDB_MIN_SCORE
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
hitNoter reputationHitNoter
}
func (abuseIPDB) name() string { return "abuseipdb" } func (abuseIPDB) name() string { return "abuseipdb" }
func (abuseIPDB) check( func (a abuseIPDB) check(
ctx context.Context, ctx context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
if rq.h.config.AbuseIPDBKey == "" { if !a.keySet {
return nil return nil
} }
client := rq.h.clientGroup(rq.client) client := clientGroup(a.ipv6GroupPrefix, rq.client)
held, _ := rq.h.limiter.Client(client) held, _ := a.limiter.Client(client)
offender := held.History.Offences != ratelimit.Offences{} offender := held.History.Offences != ratelimit.Offences{}
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender) score, hit := a.abuseIPDB.Hit(ctx, client, rq.client, offender)
if !hit { if !hit {
return nil return nil
} }
rq.abuseIPDBHit = true rq.abuseIPDBHit = true
rq.addTag(config.AbuseIPDBTag) rq.addTag(config.AbuseIPDBTag)
rq.noteHit( a.hitNoter.noteHit(
rq,
bans.ReputationHit{ bans.ReputationHit{
Source: reputation.AbuseIPDBSource, Source: reputation.AbuseIPDBSource,
Score: &score, Score: &score,
@@ -54,14 +65,14 @@ func (abuseIPDB) check(
"scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE", "scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE",
) )
if rq.h.config.DNSBLAbuseIPDBAction != deny { if a.action != config.ReputationActionDeny {
return nil return nil
} }
return &verdict{ return &verdict{
action: requestlog.ActionClientDenied, action: requestlog.ActionClientDenied,
figures: map[string]any{ figures: map[string]any{
figureScore: score, "min_score": rq.h.config.AbuseIPDBMinScore, figureScore: score, "min_score": a.minScore,
}, },
} }
} }
+48
View File
@@ -0,0 +1,48 @@
package waf
// The bans for AbuseIPDB's blacklist. They read SWW_WAF_ABUSEIPDB_BAN_DAYS and
// SWW_WAF_MODE; the blacklist is fetched by the reputation package.
import (
"log/slog"
"net/netip"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// BanAbuseIPDBBlacklist bans in ledger the netblock of each client on
// AbuseIPDB's blacklist, listed, for SWW_WAF_ABUSEIPDB_BAN_DAYS from now, as
// the ledger's BanForAbuseIPDB describes. It logs to processLog how many
// clients were listed and how many bans were made. Clients in one netblock
// make one ban, with the highest of their scores. In observe mode it makes
// no ban.
func BanAbuseIPDBBlacklist(
cfg *config.WAF, ledger *bans.Ledger, processLog *slog.Logger,
now time.Time, listed []reputation.BlacklistEntry,
) {
scores := make(map[netip.Prefix]int64, len(listed))
for _, client := range listed {
netblock := banNetblock(
cfg.BanScopeV4Prefix, cfg.IPv6GroupPrefix, client.Addr,
)
scores[netblock] = max(scores[netblock], client.Score)
}
made := 0
if !cfg.Observe {
made = ledger.BanForAbuseIPDB(
now, now.Add(cfg.AbuseIPDBBanDuration), scores,
)
}
processLog.Info(
"fetched AbuseIPDB's blacklist",
"clients", len(listed),
"bans_made", made,
)
}
+100
View File
@@ -0,0 +1,100 @@
package waf_test
import (
"context"
"net/http"
"net/netip"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// abuseIPDBBanDays is the setting of how long a ban for AbuseIPDB's
// blacklist lasts, in days.
const abuseIPDBBanDays = "SWW_WAF_ABUSEIPDB_BAN_DAYS"
func TestClientOnTheFetchedAbuseIPDBBlacklistIsRefused(t *testing.T) {
t.Parallel()
s, clk, server := startWithClock(t, "", map[string]string{
abuseIPDBKey: accountKey,
})
ctx, cancel := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
server.AbuseIPDB.RunBlacklist(ctx)
close(stopped)
}()
t.Cleanup(func() {
cancel()
<-stopped
})
waitUntil(func() bool {
active, _ := server.Ledger.Count(clk.Now())
return active == 1
})
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
s.get(otherClient, http.StatusOK, requestlog.ActionForward)
}
func TestClientsOnTheAbuseIPDBBlacklistAreBannedForTheBanDays(t *testing.T) {
t.Parallel()
s, clk, server := startWithClock(t, "", map[string]string{
abuseIPDBBanDays: "2", banScopeV4Prefix: "24",
})
// Two clients in one netblock make one ban, with the higher score.
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{
{Addr: netip.MustParseAddr(client), Score: 95},
{Addr: netip.MustParseAddr(otherClient), Score: 100},
{Addr: netip.MustParseAddr("2001:db8:5::1"), Score: 90},
})
held := server.Ledger.Bans(netip.MustParsePrefix(clientsNet))
if len(held) != 1 || held[0].Cause != bans.CauseAbuseIPDB ||
*held[0].Notes.Reputation[0].Score != 100 {
t.Errorf("bans on %s %+v, want one for the blacklist, with 100",
clientsNet, held)
}
// Every client of a banned netblock is refused, the IPv6 address's /64
// included.
for _, from := range []string{"203.0.113.200", "2001:db8:5::ffff:1"} {
s.get(from, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
}
s.get("2001:db8:5:1::1", http.StatusOK, requestlog.ActionForward)
clk.advance(2 * 24 * time.Hour)
s.get(client, http.StatusOK, requestlog.ActionForward)
}
func TestAbuseIPDBBlacklistMakesNoBanInObserveMode(t *testing.T) {
t.Parallel()
s, clk, server := startWithClock(
t,
"",
map[string]string{mode: "observe"},
)
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{
{Addr: netip.MustParseAddr(client), Score: 100},
})
if active, _ := server.Ledger.Count(clk.Now()); active != 0 {
t.Errorf("%d bans active, want none", active)
}
s.get(client, http.StatusOK, requestlog.ActionForward)
}
+11 -5
View File
@@ -139,7 +139,7 @@ func (rq *request) addBan() {
) )
if err == nil { if err == nil {
netblock, err = rq.h.banNetblock(toAdd.Netblock) netblock, err = rq.h.parseBanNetblock(toAdd.Netblock)
} }
if err == nil { if err == nil {
@@ -190,12 +190,12 @@ func (rq *request) readBanToAdd() (banToAdd, error) {
return toAdd, nil return toAdd, nil
} }
// banNetblock reads value, a netblock such as 203.0.113.0/24, or a // parseBanNetblock reads value, a netblock such as 203.0.113.0/24, or a
// client's address, which stands for the netblock a ban on that client // client's address, which stands for the netblock a ban on that client
// covers. An IPv4-mapped netblock, such as ::ffff:203.0.113.0/120, is // covers. An IPv4-mapped netblock, such as ::ffff:203.0.113.0/120, is
// refused, since a client's address is looked up as IPv4 and a ban on it // refused, since a client's address is looked up as IPv4 and a ban on it
// would refuse nothing, and so is a value with a zone. // would refuse nothing, and so is a value with a zone.
func (h *handler) banNetblock(value string) (netip.Prefix, error) { func (h *handler) parseBanNetblock(value string) (netip.Prefix, error) {
netblock, err := netip.ParsePrefix(value) netblock, err := netip.ParsePrefix(value)
if err == nil { if err == nil {
if netblock.Addr().Is4In6() { if netblock.Addr().Is4In6() {
@@ -224,7 +224,11 @@ func (h *handler) banNetblock(value string) (netip.Prefix, error) {
return netip.Prefix{}, fmt.Errorf("netblock %q %w", value, errZone) return netip.Prefix{}, fmt.Errorf("netblock %q %w", value, errZone)
} }
return h.netblock(addr), nil return banNetblock(
h.config.BanScopeV4Prefix,
h.config.IPv6GroupPrefix,
addr,
), nil
} }
// expiry returns when a ban made at now for duration ends: duration // expiry returns when a ban made at now for duration ends: duration
@@ -295,7 +299,9 @@ func (rq *request) showClient() {
Bans: state.BanEntries(rq.h.ledger.Covering(addr)), Bans: state.BanEntries(rq.h.ledger.Covering(addr)),
} }
client, seen := rq.h.limiter.Client(rq.h.clientGroup(addr)) client, seen := rq.h.limiter.Client(
clientGroup(rq.h.config.IPv6GroupPrefix, addr),
)
if seen { if seen {
answer.Client = &client answer.Client = &client
} }
+2 -2
View File
@@ -37,10 +37,10 @@ func TestAdminEndpointsAreOffWhileTheTokenIsUnset(t *testing.T) {
"", "",
map[string]string{metricsToken: token}, map[string]string{metricsToken: token},
) )
server.Ledger.BanForLimit( server.Ledger.Ban(
netip.MustParsePrefix(otherClient+"/32"), netip.MustParsePrefix(otherClient+"/32"),
clk.Now(), clk.Now(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
before := server.Ledger.Snapshot() before := server.Ledger.Snapshot()
+3 -5
View File
@@ -130,11 +130,9 @@ func TestObserveModeRaisesTheBanAlertsItWouldHave(t *testing.T) {
// A ban for a clear sign of attack, which a request under it would make // A ban for a clear sign of attack, which a request under it would make
// permanent. // permanent.
group := netip.MustParsePrefix(ipv6Group) group := netip.MustParsePrefix(ipv6Group)
attackBan, _ := server.Ledger.BanForAttack( attackBan, _ := server.Ledger.Ban(group, start, bans.Grounds{
group, Cause: bans.CauseAttack, Notes: bans.Notes{RuleID: "probe"},
start, })
bans.Notes{RuleID: "probe"},
)
// The third request breaks the limit, and so does the fourth, within the // The third request breaks the limit, and so does the fourth, within the
// cooldown, which raises nothing. The probe is a clear sign of attack. // cooldown, which raises nothing. The probe is a clear sign of attack.
+70 -37
View File
@@ -26,6 +26,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -84,7 +85,7 @@ func TestSixteenThousandCookiesAreRefusedBeforeTheCoreRuleSetReadsThem(
got := do(t, req) got := do(t, req)
wantDisposition(t, got, tooLarge, waf.DispositionDeny) wantDisposition(t, got, tooLarge, wafprotocol.DispositionDeny)
line := s.out.requestLine(t) line := s.out.requestLine(t)
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge) wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
@@ -164,9 +165,14 @@ func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
line := s.out.requestLine(t) line := s.out.requestLine(t)
if tc.status == http.StatusOK { if tc.status == http.StatusOK {
wantDisposition(t, got, tc.status, waf.DispositionAllow) wantDisposition(
t,
got,
tc.status,
wafprotocol.DispositionAllow,
)
} else { } else {
wantDisposition(t, got, tc.status, waf.DispositionDeny) wantDisposition(t, got, tc.status, wafprotocol.DispositionDeny)
} }
wantLine(t, line, tc.status, tc.action) wantLine(t, line, tc.status, tc.action)
@@ -219,9 +225,14 @@ func TestAHeaderLineOverTheLimitIsRefusedWith431(t *testing.T) {
got := do(t, req) got := do(t, req)
if tc.refused { if tc.refused {
wantDisposition(t, got, tooLarge, waf.DispositionDeny) wantDisposition(
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
} else { } else {
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow) wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
} }
}) })
} }
@@ -244,10 +255,7 @@ func TestRequestLineRefusalNamesItsHeuristicLengthAndLimit(t *testing.T) {
func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) { func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
t.Parallel() t.Parallel()
repeated := make([]string, headerFieldLimit) repeated := slices.Repeat([]string{"1"}, headerFieldLimit)
for i := range repeated {
repeated[i] = "1"
}
for _, tc := range []struct { for _, tc := range []struct {
name string name string
@@ -294,7 +302,12 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
line := s.out.requestLine(t) line := s.out.requestLine(t)
if tc.refused { if tc.refused {
wantDisposition(t, got, tooLarge, waf.DispositionDeny) wantDisposition(
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
wantLine( wantLine(
t, t,
line, line,
@@ -305,7 +318,12 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
return return
} }
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow) wantDisposition(
t,
got,
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantLine(t, line, http.StatusOK, requestlog.ActionForward) wantLine(t, line, http.StatusOK, requestlog.ActionForward)
if line.DurationCoreRuleSetMS == nil { if line.DurationCoreRuleSetMS == nil {
@@ -332,7 +350,12 @@ func TestHeaderLimitsHoldWhateverTheCoreRuleSetSettingsSay(t *testing.T) {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny) wantDisposition(
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
}) })
} }
} }
@@ -348,7 +371,12 @@ func TestClientInTheAllowNetsSkipsTheHeaderLimits(t *testing.T) {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow) wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
} }
func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) { func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
@@ -364,7 +392,7 @@ func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
got := do(t, req) got := do(t, req)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow) wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
line := s.out.requestLine(t) line := s.out.requestLine(t)
if line.WouldAction != requestlog.ActionRequestHeaderTooLarge || if line.WouldAction != requestlog.ActionRequestHeaderTooLarge ||
@@ -386,14 +414,14 @@ func TestTheRateLimitsDoNotCountRequestsOverTheHeaderLimits(t *testing.T) {
for range 2 { for range 2 {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny) wantDisposition(t, do(t, req), tooLarge, wafprotocol.DispositionDeny)
} }
wantDisposition( wantDisposition(
t, t,
s.ask(client, "/"), s.ask(client, "/"),
http.StatusOK, http.StatusOK,
waf.DispositionAllow, wafprotocol.DispositionAllow,
) )
} }
@@ -466,7 +494,12 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
req := s.question(siteA, client, tc.path) req := s.question(siteA, client, tc.path)
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny) wantDisposition(
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
wantLine( wantLine(
t, t,
s.out.requestLine(t), s.out.requestLine(t),
@@ -503,11 +536,16 @@ func TestARequestOverTheHeaderLimitsRunsNoOtherCheck(t *testing.T) {
over := s.question(siteA, client, "/") over := s.question(siteA, client, "/")
over.Header.Set(cookieHeader, cookies(cookieLimit+1)) over.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, over), tooLarge, waf.DispositionDeny) wantDisposition(t, do(t, over), tooLarge, wafprotocol.DispositionDeny)
within := s.question(siteA, client, "/") within := s.question(siteA, client, "/")
within.Header.Set(cookieHeader, cookies(cookieLimit)) within.Header.Set(cookieHeader, cookies(cookieLimit))
wantDisposition(t, do(t, within), http.StatusOK, waf.DispositionAllow) wantDisposition(
t,
do(t, within),
http.StatusOK,
wafprotocol.DispositionAllow,
)
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
@@ -564,7 +602,7 @@ func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) {
t, t,
s.ask(fromDE, "/"), s.ask(fromDE, "/"),
http.StatusForbidden, http.StatusForbidden,
waf.DispositionDeny, wafprotocol.DispositionDeny,
) )
line := s.out.requestLine(t) line := s.out.requestLine(t)
@@ -601,7 +639,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
t, t,
s.ask(client, "/"), s.ask(client, "/"),
http.StatusForbidden, http.StatusForbidden,
waf.DispositionDeny, wafprotocol.DispositionDeny,
) )
line := s.out.requestLine(t) line := s.out.requestLine(t)
@@ -630,7 +668,7 @@ func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
got := do(t, req) got := do(t, req)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow) wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" { if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" {
t.Errorf( t.Errorf(
@@ -672,7 +710,7 @@ func askDirectly(
http.NoBody, http.NoBody,
) )
req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234") req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234")
req.Header.Set(waf.ClientHeader, client) req.Header.Set(wafprotocol.ClientHeader, client)
req.Header.Set(requestIDHeader, "the-id") req.Header.Set(requestIDHeader, "the-id")
change(req) change(req)
@@ -686,10 +724,10 @@ func askDirectly(
// banClient bans client at now. // banClient bans client at now.
func banClient(_ *testing.T, server *waf.Server, now time.Time) { func banClient(_ *testing.T, server *waf.Server, now time.Time) {
server.Ledger.BanForLimit( server.Ledger.Ban(
netip.MustParsePrefix(client+"/32"), netip.MustParsePrefix(client+"/32"),
now, now,
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
} }
@@ -877,18 +915,13 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
held, _, _ := startHeldGeoJS(t) held, _, _ := startHeldGeoJS(t)
resolver, releaseResolver := startHeldResolver(t) resolver, releaseResolver := startHeldResolver(t)
server, out, _ := newWAFWith(t, func( server, out, _ := newWAFWith(t, waf.New,
params waf.Params, silentGeoJSURL, silentAbuseIPDBURL, time.Now, map[string]string{
) *waf.Server { tagDeny: "country=KP", coreRuleSetMode: block,
params.AbuseIPDBURL = silentAbuseIPDBURL dnsblZones: dnsblZone, dnsblResolver: resolver,
abuseIPDBKey: accountKey,
return waf.New(params) blocklistURLs: held + "/drop.txt", asnLimitPercentURL: held + "/asn.txt",
}, silentGeoJSURL, time.Now, map[string]string{ })
tagDeny: "country=KP", coreRuleSetMode: block,
dnsblZones: dnsblZone, dnsblResolver: resolver,
abuseIPDBKey: accountKey,
blocklistURLs: held + "/drop.txt", asnLimitPercentURL: held + "/asn.txt",
})
// The lists are fetched outside the synctest bubble below: a wait on the // The lists are fetched outside the synctest bubble below: a wait on the
// network inside it would keep its clock from moving on. // network inside it would keep its clock from moving on.
+164 -99
View File
@@ -1,7 +1,7 @@
package waf package waf
// The heuristic active_ban, and the code that makes bans, which several // The heuristic active_ban, and banMaker, which makes the bans of several
// heuristics share. active_ban reads SWW_WAF_MODE; the bans are the ban ledger's. // heuristics. active_ban reads SWW_WAF_MODE; the bans are the ban ledger's.
import ( import (
"context" "context"
@@ -10,6 +10,8 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -24,18 +26,22 @@ func (rq *request) banResponse(action string) *refusal {
// in, in force at now. It notes for the log line when that ban ends. A request // in, in force at now. It notes for the log line when that ban ends. A request
// that makes the ban permanent, or in observe mode would have, raises the alert // that makes the ban permanent, or in observe mode would have, raises the alert
// for it. Its figures are the ban's netblock and when it ends. // for it. Its figures are the ban's netblock and when it ends.
type activeBan struct{} type activeBan struct {
banMaker banMaker
}
func (activeBan) name() string { return "active_ban" } func (activeBan) name() string { return "active_ban" }
func (activeBan) check( func (a activeBan) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
now time.Time, now time.Time,
) *verdict { ) *verdict {
check := rq.h.ledger.Check ledger := a.banMaker.ledger
if rq.h.config.Observe {
check = rq.h.ledger.Find // the ban refuses nothing, and stays as it is check := ledger.Check
if a.banMaker.observe {
check = ledger.Find // the ban refuses nothing, and stays as it is
} }
ban, banned, madePermanent := check(rq.client, now) ban, banned, madePermanent := check(rq.client, now)
@@ -45,7 +51,7 @@ func (activeBan) check(
if madePermanent { if madePermanent {
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
rq.alertBan(ban) a.banMaker.alertBan(rq, ban)
} }
if !banned { if !banned {
@@ -80,31 +86,55 @@ func limitFigures(hit ratelimit.Hit) map[string]any {
// countedBytes returns the request's bytes, once it has ended, as the // countedBytes returns the request's bytes, once it has ended, as the
// byte limits and the anomaly thresholds count them: the response's body // byte limits and the anomaly thresholds count them: the response's body
// bytes, the request's, or both, as SWW_WAF_BYTES_COUNTED says. // bytes, the request's, or both, as bytesCounted, SWW_WAF_BYTES_COUNTED,
func (rq *request) countedBytes() int64 { // says.
func (rq *request) countedBytes(bytesCounted config.BytesCounted) int64 {
response, request := rq.out.bytes, rq.requestBytes response, request := rq.out.bytes, rq.requestBytes
switch rq.h.config.BytesCounted { switch bytesCounted {
case "response": case config.BytesCountedResponse:
return response return response
case "request": case config.BytesCountedRequest:
return request return request
default: // both case config.BytesCountedBoth:
return response + request
} }
return response + request
} }
// banForLimit bans the client's netblock at now for a broken limit, the // banMaker makes the bans of the heuristics that ban a client, in the ban
// ledger, and raises the alert for each. In observe mode it makes no ban, and
// raises the alert for the ban it would have made.
type banMaker struct {
ledger *bans.Ledger
// limiter is the table of clients the rate limits count. A ban for a limit
// sets the client's counters back to zero, and a ban's notes give how many
// requests its netblock has sent.
limiter *ratelimit.Limiter
metrics *metrics.Metrics
alerts *alerts.Queue
// observe is true in observe mode (SWW_WAF_MODE).
observe bool
// banResponse is SWW_WAF_BAN_RESPONSE, the status the notes of a ban for a
// clear sign of attack give.
banResponse int
// banScopeV4Prefix is SWW_WAF_BAN_SCOPE_V4_PREFIX and ipv6GroupPrefix
// SWW_WAF_IPV6_GROUP_PREFIX, which set the netblock a ban covers.
banScopeV4Prefix int
ipv6GroupPrefix int
}
// banForLimit bans the netblock of rq's client at now for a broken limit, the
// one hit names, notes the offence for the log line and counts the hit in // one hit names, notes the offence for the log line and counts the hit in
// the metrics. status is what the client was sent, or is sent: // the metrics. status is what the client was sent, or is sent:
// SWW_WAF_BAN_RESPONSE for a request over a rate limit, what the report says for // SWW_WAF_BAN_RESPONSE for a request over a rate limit, what the report says for
// one whose bytes broke a byte limit, the refusal for one that broke the // one whose bytes broke a byte limit, the refusal for one that broke the
// refusal limit. The ban's notes give the client's limit percentage for a // refusal limit. The ban's notes give the client's limit percentage for a
// rate limit or a byte limit; the refusal limit is not lowered. The ban sets // rate limit or a byte limit; the refusal limit is not lowered. The ban sets
// the client's counters back to zero. In observe mode it makes no ban and // the client's counters back to zero. In observe mode, banClient says what
// sets nothing back, and raises the alert for the ban it would have made, // happens instead.
// if that alert would be sent. func (b banMaker) banForLimit(
func (rq *request) banForLimit( rq *request,
now time.Time, now time.Time,
hit ratelimit.Hit, hit ratelimit.Hit,
status int, status int,
@@ -119,25 +149,14 @@ func (rq *request) banForLimit(
} }
rq.line.Offence = requestlog.OffenceLimit rq.line.Offence = requestlog.OffenceLimit
rq.h.metrics.LimitHit(hit)
netblock := rq.h.netblock(rq.client) b.metrics.LimitHit(hit)
if rq.h.config.Observe &&
!rq.wouldAlertBan(netblock, now, bans.CauseLimit) {
return
}
notes := bans.Notes{ notes := bans.Notes{
ASN: rq.line.ASN, Kind: hit.Kind,
ASName: rq.line.ASName, Limit: hit.Limit,
Country: rq.line.Country, Window: hit.Window,
Kind: hit.Kind, Count: hit.Count,
Limit: hit.Limit,
Window: hit.Window,
Count: hit.Count,
Reputation: rq.reputation,
Request: rq.noted(now, status),
Requests: rq.netblockRequests(netblock),
} }
switch hit.Kind { switch hit.Kind {
@@ -147,86 +166,130 @@ func (rq *request) banForLimit(
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged() notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
} }
if rq.h.config.Observe { b.banClient(
ban, wouldBan := rq.h.ledger.WouldBanForLimit(netblock, now, notes) rq,
if wouldBan { now,
rq.alertBan(ban) status,
} bans.Grounds{Cause: bans.CauseLimit, Notes: notes},
)
return
}
ban, made := rq.h.ledger.BanForLimit(netblock, now, notes)
rq.h.limiter.Reset(rq.h.clientGroup(rq.client))
rq.line.BanExpires = banExpires(ban)
if made {
rq.alertBan(ban)
}
} }
// banForAttack bans the client's netblock at now for a clear sign of // banForAttack bans the netblock of rq's client at now for a clear sign of
// attack, which notes name: the ban rule that matched, or the trap path // attack, which notes name: the ban rule that matched, or the trap path
// asked for. It fills in the rest of the notes. In observe mode it makes // asked for. It returns the action for that ban. In observe mode, banClient
// no ban, and raises the alert for the ban it would have made, if that // says what happens instead.
// alert would be sent. It returns the action for that ban. func (b banMaker) banForAttack(
func (rq *request) banForAttack(now time.Time, notes bans.Notes) string { rq *request, now time.Time, notes bans.Notes,
netblock := rq.h.netblock(rq.client) ) string {
if rq.h.config.Observe && return b.banClient(
!rq.wouldAlertBan(netblock, now, bans.CauseAttack) { rq,
return banAction( now,
rq.h.ledger.WouldBePermanent(netblock, now, bans.CauseAttack), b.banResponse,
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
)
}
// banClient bans the netblock of rq's client at now on grounds, and returns
// the action for that ban. It adds to the notes what the request tells of the
// client, with status, what the client was sent. A ban for a broken limit
// sets the client's counters back to zero.
//
// In observe mode it makes no ban and sets nothing back. It raises the
// alert for the ban it would have made, if that alert would be sent.
func (b banMaker) banClient(
rq *request,
now time.Time,
status int,
grounds bans.Grounds,
) string {
netblock := banNetblock(b.banScopeV4Prefix, b.ipv6GroupPrefix, rq.client)
if b.observe {
if !b.wouldAlertBan(netblock, now, grounds.Cause) {
return banAction(
b.ledger.WouldBePermanent(netblock, now, grounds.Cause),
)
}
grounds.Notes = b.withClientNotes(
rq,
grounds.Notes,
now,
status,
netblock,
) )
}
notes.ASN = rq.line.ASN ban, wouldBan := b.ledger.WouldBan(netblock, now, grounds)
notes.ASName = rq.line.ASName
notes.Country = rq.line.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, rq.h.config.BanResponse)
notes.Requests = rq.netblockRequests(netblock)
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForAttack(netblock, now, notes)
if wouldBan { if wouldBan {
rq.alertBan(ban) b.alertBan(rq, ban)
} }
return banAction(ban.Permanent()) return banAction(ban.Permanent())
} }
ban, made := rq.h.ledger.BanForAttack(netblock, now, notes) grounds.Notes = b.withClientNotes(
rq,
grounds.Notes,
now,
status,
netblock,
)
ban, made := b.ledger.Ban(netblock, now, grounds)
if grounds.Cause == bans.CauseLimit {
b.limiter.Reset(clientGroup(b.ipv6GroupPrefix, rq.client))
}
rq.line.BanExpires = banExpires(ban) rq.line.BanExpires = banExpires(ban)
if made { if made {
rq.alertBan(ban) b.alertBan(rq, ban)
} }
return banAction(ban.Permanent()) return banAction(ban.Permanent())
} }
// withClientNotes is notes with what rq tells of its client added: its AS,
// country and reputation, the request at now with status, and how many
// requests netblock, the client's, has sent.
func (b banMaker) withClientNotes(
rq *request,
notes bans.Notes,
now time.Time,
status int,
netblock netip.Prefix,
) bans.Notes {
notes.ASN = rq.line.ASN
notes.ASName = rq.line.ASName
notes.Country = rq.line.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, status)
notes.Requests = b.netblockRequests(netblock)
return notes
}
// wouldAlertBan reports whether the alert for a ban on netblock for cause // wouldAlertBan reports whether the alert for a ban on netblock for cause
// made at now would be sent. In observe mode the ban the request would // made at now would be sent. In observe mode the ban the request would
// have made is worked out only then, at most once per // have made is worked out only then, at most once per
// SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the // SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the
// netblock's requests, which can mean going through every client. // netblock's requests, which can mean going through every client.
func (rq *request) wouldAlertBan( func (b banMaker) wouldAlertBan(
netblock netip.Prefix, now time.Time, cause string, netblock netip.Prefix, now time.Time, cause string,
) bool { ) bool {
event := alerts.EventBan event := alerts.EventBan
if rq.h.ledger.WouldBePermanent(netblock, now, cause) { if b.ledger.WouldBePermanent(netblock, now, cause) {
event = alerts.EventPermanentBan event = alerts.EventPermanentBan
} }
return rq.h.alerts.WouldSend(event, netblock) return b.alerts.WouldSend(event, netblock)
} }
// alertBan raises the alert for ban, which the request made, or made // alertBan raises the alert for ban, which rq made, or made permanent:
// permanent: permanent_ban for a permanent ban, ban for another. Its // permanent_ban for a permanent ban, ban for another. Its detail gives the
// detail gives the ban's cause, when it ends, and its notes, and in // ban's cause, when it ends, and its notes, and in observe mode, where ban is
// observe mode, where ban is the ban that would have been made, or made // the ban that would have been made, or made permanent, mode, observe.
// permanent, mode, observe. func (b banMaker) alertBan(rq *request, ban bans.Ban) {
func (rq *request) alertBan(ban bans.Ban) {
event := alerts.EventBan event := alerts.EventBan
if ban.Permanent() { if ban.Permanent() {
event = alerts.EventPermanentBan event = alerts.EventPermanentBan
@@ -235,11 +298,11 @@ func (rq *request) alertBan(ban bans.Ban) {
detail := map[string]any{ detail := map[string]any{
"cause": ban.Cause, "ban_expires": banExpires(ban), "notes": ban.Notes, "cause": ban.Cause, "ban_expires": banExpires(ban), "notes": ban.Notes,
} }
if rq.h.config.Observe { if b.observe {
detail["mode"] = "observe" detail["mode"] = "observe"
} }
rq.h.alerts.Raise(alerts.Alert{ b.alerts.Raise(alerts.Alert{
Event: event, Event: event,
Host: rq.site, Host: rq.site,
Client: rq.client, Client: rq.client,
@@ -252,6 +315,13 @@ func (rq *request) alertBan(ban bans.Ban) {
}) })
} }
// netblockRequests is how many requests netblock has sent since it was
// first seen, the request being banned for included: the histories count it
// only once it has ended.
func (b banMaker) netblockRequests(netblock netip.Prefix) int64 {
return b.limiter.Requests(netblock) + 1
}
// noted is the request, at now, with status, what the client was sent, or // noted is the request, at now, with status, what the client was sent, or
// in observe mode would have been, as the notes of the ban it makes keep // in observe mode would have been, as the notes of the ban it makes keep
// it. // it.
@@ -266,23 +336,18 @@ func (rq *request) noted(now time.Time, status int) bans.Request {
} }
} }
// netblockRequests is how many requests netblock has sent since it was // banNetblock is the netblock a ban on client covers: its IPv4 address,
// first seen, this one included: the histories count it only once it has // widened to banScopeV4Prefix, SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group
// ended. // clientGroup counts it in, of length ipv6GroupPrefix.
func (rq *request) netblockRequests(netblock netip.Prefix) int64 { func banNetblock(
return rq.h.limiter.Requests(netblock) + 1 banScopeV4Prefix, ipv6GroupPrefix int, client netip.Addr,
} ) netip.Prefix {
// netblock is the netblock a ban on client covers: its IPv4 address,
// widened to SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
// counts it in.
func (h *handler) netblock(client netip.Addr) netip.Prefix {
addr := client.Unmap() addr := client.Unmap()
if addr.Is4() { if addr.Is4() {
return netip.PrefixFrom(addr, h.config.BanScopeV4Prefix).Masked() return netip.PrefixFrom(addr, banScopeV4Prefix).Masked()
} }
return h.clientGroup(addr) return clientGroup(ipv6GroupPrefix, addr)
} }
// banExpires is when ban ends, as the log line gives it: a time, or // banExpires is when ban ends, as the log line gives it: a time, or
+4 -5
View File
@@ -548,6 +548,10 @@ func (s *sender) requestWithHeader(
// body is sent as it is after the headers, header holding its // body is sent as it is after the headers, header holding its
// Content-Length or Transfer-Encoding. header may hold several lines, // Content-Length or Transfer-Encoding. header may hold several lines,
// separated by "\r\n". It returns the whole answer. // separated by "\r\n". It returns the whole answer.
//
// It waits for the answer as long as it takes, since a busy host can slow
// the Core Rule Set past any shorter wait. go test's -timeout ends a test
// whose answer never comes.
func (s *sender) requestWithBody( func (s *sender) requestWithBody(
method, from, path, header, body string, status int, action string, method, from, path, header, body string, status int, action string,
) (logLine, answer) { ) (logLine, answer) {
@@ -566,11 +570,6 @@ func (s *sender) requestWithBody(
send(s.t, conn, method+" "+path+" HTTP/1.1\r\nHost: "+appHost+"\r\n"+ send(s.t, conn, method+" "+path+" HTTP/1.1\r\nHost: "+appHost+"\r\n"+
forwardedFor+": "+from+"\r\n"+header+"\r\n"+body) forwardedFor+": "+from+"\r\n"+header+"\r\n"+body)
err := conn.SetReadDeadline(time.Now().Add(waitLimit))
if err != nil {
s.t.Fatalf("set read deadline: %v", err)
}
var got answer var got answer
res, err := http.ReadResponse(bufio.NewReader(conn), nil) res, err := http.ReadResponse(bufio.NewReader(conn), nil)
+12 -12
View File
@@ -54,7 +54,7 @@ func (rq *request) runBesideLookup(
) *verdict { ) *verdict {
var running atomic.Pointer[string] var running atomic.Pointer[string]
running.Store(new(trapPaths{}.name())) running.Store(new(rq.h.trapPaths.name()))
found := make(chan *besideLookupFindings, 1) found := make(chan *besideLookupFindings, 1)
@@ -92,7 +92,7 @@ func (rq *request) runBesideLookup(
return v return v
} }
return rq.runHeuristics(ctx, afterBesideLookup(), rq.h.now()) return rq.runHeuristics(ctx, rq.h.afterBesideLookup, rq.h.now())
} }
// findBesideLookup does the work of the heuristics of besideLookup on the // findBesideLookup does the work of the heuristics of besideLookup on the
@@ -105,39 +105,39 @@ func (rq *request) findBesideLookup(
) *besideLookupFindings { ) *besideLookupFindings {
findings := &besideLookupFindings{} findings := &besideLookupFindings{}
findings.trapPath = rq.trapPath() findings.trapPath = rq.h.trapPaths.trapPath(rq)
if findings.trapPath != "" { if findings.trapPath != "" {
findings.refuses = true findings.refuses = true
return findings return findings
} }
running.Store(new(ruleFiles{}.name())) running.Store(new(rq.h.ruleFiles.name()))
findings.rules = rq.h.rules.Match(rq.in) findings.rules = rq.h.ruleFiles.rules.Match(rq.in)
if refusedByRule(findings.rules) { if refusedByRule(findings.rules) {
findings.refuses = true findings.refuses = true
return findings return findings
} }
running.Store(new(coreRuleSet{}.name())) coreRuleSet := rq.h.coreRuleSet
running.Store(new(coreRuleSet.name()))
cfg := rq.h.config if coreRuleSet.mode == config.CoreRuleSetModeOff ||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff || pathExempt(rq.in, coreRuleSet.exemptPaths) {
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
return findings return findings
} }
start := time.Now() start := time.Now()
// A question's body has been read into memory, so that reading it here // A question's body has been read into memory, so that reading it here
// cannot fail. // cannot fail.
findings.inspection, findings.read, _ = rq.h.coreRuleSet.Inspect( findings.inspection, findings.read, _ = coreRuleSet.inspector.Inspect(
rq.in, rq.client, rq.in.Body, rq.in, rq.client, rq.in.Body,
) )
findings.inspectionTime = time.Since(start) findings.inspectionTime = time.Since(start)
findings.inspected = true findings.inspected = true
findings.refuses = rq.refusedByCoreRuleSet(findings.inspection) findings.refuses = coreRuleSet.refusedByCoreRuleSet(findings.inspection)
return findings return findings
} }
@@ -151,7 +151,7 @@ func (rq *request) applyFindings(
) *verdict { ) *verdict {
rq.findings = findings rq.findings = findings
return rq.runHeuristics(ctx, besideLookup(), rq.h.now()) return rq.runHeuristics(ctx, rq.h.besideLookup, rq.h.now())
} }
// timedOut refuses the request for the heuristics of besideLookup not having // timedOut refuses the request for the heuristics of besideLookup not having
+2 -2
View File
@@ -73,10 +73,10 @@ func newWAFHoldingTheCoreRuleSet(
server, out, queue := newWAFWith(t, func(params waf.Params) *waf.Server { server, out, queue := newWAFWith(t, func(params waf.Params) *waf.Server {
params.CoreRuleSet = crs params.CoreRuleSet = crs
params.BesideLookupTimeLimit = waf.BesideLookupTimeLimit
return waf.New(params) return waf.New(params)
}, silentGeoJSURL, time.Now, settings) }, silentGeoJSURL, abuseIPDBURL, time.Now, settings)
server.SetBesideLookupTimeLimit(waf.BesideLookupTimeLimit)
return server, out, queue return server, out, queue
} }
+44 -22
View File
@@ -2,6 +2,7 @@ package waf
import ( import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
) )
// whole is the percentage of each limit a client gets when no biased // whole is the percentage of each limit a client gets when no biased
@@ -29,6 +30,25 @@ func biasedThresholdsSet(cfg *config.WAF) bool {
cfg.ASNLimitPercentURL != "" cfg.ASNLimitPercentURL != ""
} }
// biasedThresholds are the settings and the lists by which limitPercentages
// lowers a client's limits.
type biasedThresholds struct {
// lists hold the file SWW_WAF_ASN_LIMIT_PERCENT_URL names.
lists *reputation.Lists
tagLimitPercent map[string]int64 // SWW_WAF_TAG_LIMIT_PERCENT
asnBytesLimitPercent map[string]int64 // SWW_WAF_ASN_BYTES_LIMIT_PERCENT
countryBytesLimitPercent map[string]int64 // SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT
unknownCountryLimitPercent int64 // SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT
// blocklistAction and blocklistLimitPercent are SWW_WAF_BLOCKLIST_ACTION
// and its percentage.
blocklistAction config.ReputationAction
blocklistLimitPercent int64
// dnsblAbuseIPDBAction and dnsblAbuseIPDBLimitPercent are
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION and its percentage.
dnsblAbuseIPDBAction config.ReputationAction
dnsblAbuseIPDBLimitPercent int64
}
// limitPercentages returns the client's limit percentages, for the rate // limitPercentages returns the client's limit percentages, for the rate
// limits and for the byte limits, by its tags, its AS number and country // limits and for the byte limits, by its tags, its AS number and country
// as looked up, each "" when unknown, and the blocklists, DNSBL zones and // as looked up, each "" when unknown, and the blocklists, DNSBL zones and
@@ -45,48 +65,50 @@ func biasedThresholdsSet(cfg *config.WAF) bool {
// the byte limits, SWW_WAF_ASN_BYTES_LIMIT_PERCENT and // the byte limits, SWW_WAF_ASN_BYTES_LIMIT_PERCENT and
// SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT take the place of the first three for // SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT take the place of the first three for
// an AS number or a country they list. // an AS number or a country they list.
func (rq *request) limitPercentages() (percentage, percentage) { func (b biasedThresholds) limitPercentages(
cfg := rq.h.config rq *request,
) (percentage, percentage) {
asn, country := rq.line.ASN, rq.line.Country asn, country := rq.line.ASN, rq.line.Country
unknown := percentage{percent: whole} unknown := percentage{percent: whole}
if country == "" && !rq.skippedLookupWait { if country == "" && !rq.skippedLookupWait {
unknown = percentage{ unknown = percentage{
cfg.UnknownCountryLimitPercent, b.unknownCountryLimitPercent,
"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT", config.UnknownCountryLimitPercentSetting,
} }
} }
fetched := percentage{percent: whole} fetched := percentage{percent: whole}
if percent, listed := rq.h.lists.ASNLimitPercent(asn); listed { if percent, listed := b.lists.ASNLimitPercent(asn); listed {
fetched = percentage{percent, "SWW_WAF_ASN_LIMIT_PERCENT_URL"} fetched = percentage{percent, config.ASNLimitPercentURLSetting}
} }
blocklisted := percentage{percent: whole} blocklisted := percentage{percent: whole}
if rq.blocklisted && cfg.BlocklistAction == "limit" { if rq.blocklisted &&
b.blocklistAction == config.ReputationActionLimit {
blocklisted = percentage{ blocklisted = percentage{
cfg.BlocklistLimitPercent, b.blocklistLimitPercent,
"SWW_WAF_BLOCKLIST_ACTION", config.BlocklistActionSetting,
} }
} }
dnsblAbuseIPDBListed := percentage{percent: whole} dnsblAbuseIPDBListed := percentage{percent: whole}
if (rq.dnsblListed || rq.abuseIPDBHit) && if (rq.dnsblListed || rq.abuseIPDBHit) &&
cfg.DNSBLAbuseIPDBAction == "limit" { b.dnsblAbuseIPDBAction == config.ReputationActionLimit {
dnsblAbuseIPDBListed = percentage{ dnsblAbuseIPDBListed = percentage{
cfg.DNSBLAbuseIPDBLimitPercent, b.dnsblAbuseIPDBLimitPercent,
"SWW_WAF_DNSBL_ABUSEIPDB_ACTION", config.DNSBLAbuseIPDBActionSetting,
} }
} }
const tagSetting = "SWW_WAF_TAG_LIMIT_PERCENT" const tagSetting = config.TagLimitPercentSetting
asnRequests := lowest( asnRequests := lowest(
given(cfg.TagLimitPercent, asnTag(asn), tagSetting), given(b.tagLimitPercent, asnTag(asn), tagSetting),
fetched, fetched,
) )
countryRequests := given( countryRequests := given(
cfg.TagLimitPercent, b.tagLimitPercent,
countryTag(country), countryTag(country),
tagSetting, tagSetting,
) )
@@ -97,25 +119,25 @@ func (rq *request) limitPercentages() (percentage, percentage) {
if tag != asnTag(asn) && tag != countryTag(country) { if tag != asnTag(asn) && tag != countryTag(country) {
otherTags = lowest( otherTags = lowest(
otherTags, otherTags,
given(cfg.TagLimitPercent, tag, tagSetting), given(b.tagLimitPercent, tag, tagSetting),
) )
} }
} }
asnBytes, countryBytes := asnRequests, countryRequests asnBytes, countryBytes := asnRequests, countryRequests
if _, listed := cfg.ASNBytesLimitPercent[asn]; listed { if _, listed := b.asnBytesLimitPercent[asn]; listed {
asnBytes = given( asnBytes = given(
cfg.ASNBytesLimitPercent, b.asnBytesLimitPercent,
asn, asn,
"SWW_WAF_ASN_BYTES_LIMIT_PERCENT", config.ASNBytesLimitPercentSetting,
) )
} }
if _, listed := cfg.CountryBytesLimitPercent[country]; listed { if _, listed := b.countryBytesLimitPercent[country]; listed {
countryBytes = given( countryBytes = given(
cfg.CountryBytesLimitPercent, b.countryBytesLimitPercent,
country, country,
"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT", config.CountryBytesLimitPercentSetting,
) )
} }
+2 -1
View File
@@ -19,6 +19,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// The biased thresholds. // The biased thresholds.
@@ -589,7 +590,7 @@ func serveFromDE(
t.Helper() t.Helper()
req := httptest.NewRequestWithContext(t.Context(), method, "/", body) req := httptest.NewRequestWithContext(t.Context(), method, "/", body)
req.Header.Set(waf.ClientHeader, fromDE) req.Header.Set(wafprotocol.ClientHeader, fromDE)
req.Header.Set(requestIDHeader, "the-id") req.Header.Set(requestIDHeader, "the-id")
server.Handler.ServeHTTP(httptest.NewRecorder(), req) server.Handler.ServeHTTP(httptest.NewRecorder(), req)
+22 -7
View File
@@ -7,6 +7,8 @@ import (
"context" "context"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -14,20 +16,24 @@ import (
// and refuses the request when SWW_WAF_BLOCKLIST_ACTION is deny. Being limit, it // and refuses the request when SWW_WAF_BLOCKLIST_ACTION is deny. Being limit, it
// lowers the client's limits instead (see limitPercentages), and being log, it // lowers the client's limits instead (see limitPercentages), and being log, it
// does nothing more. Its figure is the blocklists that list the client. // does nothing more. Its figure is the blocklists that list the client.
type blocklists struct{} type blocklists struct {
lists *reputation.Lists
action config.ReputationAction // SWW_WAF_BLOCKLIST_ACTION
hitNoter reputationHitNoter
}
func (blocklists) name() string { return "blocklists" } func (blocklists) name() string { return "blocklists" }
func (blocklists) check( func (b blocklists) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
listedBy := rq.h.lists.ListedBy(rq.client) listedBy := b.lists.ListedBy(rq.client)
rq.blocklisted = len(listedBy) > 0 rq.blocklisted = len(listedBy) > 0
rq.noteListed(listedBy, "listed by a blocklist") b.hitNoter.noteListed(rq, listedBy, "listed by a blocklist")
if !rq.refusedByBlocklists(listedBy) { if !b.refusedByBlocklists(listedBy) {
return nil return nil
} }
@@ -37,9 +43,18 @@ func (blocklists) check(
} }
} }
// refusesWhateverTheLookupAnswers reports whether blocklists refuses rq, as
// refusedByBlocklists says. The lookup's answer plays no part in it.
func (b blocklists) refusesWhateverTheLookupAnswers(
rq *request,
_ time.Time,
) bool {
return b.refusedByBlocklists(b.lists.ListedBy(rq.client))
}
// refusedByBlocklists reports whether the heuristic blocklists refuses a // refusedByBlocklists reports whether the heuristic blocklists refuses a
// client, given listedBy, the blocklists that list it: it does when one lists // client, given listedBy, the blocklists that list it: it does when one lists
// it and SWW_WAF_BLOCKLIST_ACTION is deny. // it and SWW_WAF_BLOCKLIST_ACTION is deny.
func (rq *request) refusedByBlocklists(listedBy []string) bool { func (b blocklists) refusedByBlocklists(listedBy []string) bool {
return len(listedBy) > 0 && rq.h.config.BlocklistAction == deny return len(listedBy) > 0 && b.action == config.ReputationActionDeny
} }
+14 -6
View File
@@ -6,6 +6,9 @@ package waf
import ( import (
"context" "context"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
) )
// byteLimits counts the request's bytes, as countedBytes gives them, for the // byteLimits counts the request's bytes, as countedBytes gives them, for the
@@ -16,11 +19,16 @@ import (
// the client over a byte limit, as its limit percentage for the byte limits // the client over a byte limit, as its limit percentage for the byte limits
// lowers it, break it, which bans the client; the response was passed on whole. // lowers it, break it, which bans the client; the response was passed on whole.
// Its figures are the window, the count and the limit. // Its figures are the window, the count and the limit.
type byteLimits struct{} type byteLimits struct {
limiter *ratelimit.Limiter
bytesCounted config.BytesCounted // SWW_WAF_BYTES_COUNTED
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
banMaker banMaker
}
func (byteLimits) name() string { return "byte_limits" } func (byteLimits) name() string { return "byte_limits" }
func (byteLimits) check( func (b byteLimits) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
now time.Time, now time.Time,
@@ -29,10 +37,10 @@ func (byteLimits) check(
return nil return nil
} }
counts, hit, over := rq.h.limiter.CountBytes( counts, hit, over := b.limiter.CountBytes(
rq.h.clientGroup(rq.client), clientGroup(b.ipv6GroupPrefix, rq.client),
now, now,
rq.countedBytes(), rq.countedBytes(b.bytesCounted),
rq.bytesLimitPercent.percent, rq.bytesLimitPercent.percent,
) )
rq.line.Counts.MinuteBytes = counts.MinuteBytes rq.line.Counts.MinuteBytes = counts.MinuteBytes
@@ -43,7 +51,7 @@ func (byteLimits) check(
return nil return nil
} }
rq.banForLimit(now, hit, rq.out.status) b.banMaker.banForLimit(rq, now, hit, rq.out.status)
return &verdict{figures: limitFigures(hit)} return &verdict{figures: limitFigures(hit)}
} }
+8 -8
View File
@@ -5,6 +5,8 @@ import (
"net/http" "net/http"
"net/netip" "net/netip"
"slices" "slices"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// peerAddress is the address of the request's TCP peer. // peerAddress is the address of the request's TCP peer.
@@ -17,14 +19,11 @@ func peerAddress(r *http.Request) netip.Addr {
return addrPort.Addr().Unmap() return addrPort.Addr().Unmap()
} }
// requestIDHeader carries the request's id, from rproxy.
const requestIDHeader = "X-Request-ID"
// requestID is the request's id: the one a question gives, or for any other // requestID is the request's id: the one a question gives, or for any other
// request a new random one. // request a new random one.
func requestID(r *http.Request, isQuestion bool) string { func requestID(r *http.Request, isQuestion bool) string {
if isQuestion { if isQuestion {
return r.Header.Get(requestIDHeader) return r.Header.Get(wafprotocol.RequestIDHeader)
} }
return rand.Text() return rand.Text()
@@ -43,12 +42,13 @@ func scheme(r *http.Request, isQuestion bool) string {
// clientGroup is the client a request is counted toward: its IPv4 // clientGroup is the client a request is counted toward: its IPv4
// address, or its IPv6 group, the netblock its IPv6 address is in of the // address, or its IPv6 group, the netblock its IPv6 address is in of the
// length SWW_WAF_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser // length ipv6GroupPrefix, SWW_WAF_IPV6_GROUP_PREFIX, a /64 by default, since
// usually holds a whole /64. An IPv4 address in IPv6 form counts as IPv4. // one abuser usually holds a whole /64. An IPv4 address in IPv6 form counts as
func (h *handler) clientGroup(addr netip.Addr) netip.Prefix { // IPv4.
func clientGroup(ipv6GroupPrefix int, addr netip.Addr) netip.Prefix {
addr = addr.Unmap() addr = addr.Unmap()
if addr.Is6() { if addr.Is6() {
return netip.PrefixFrom(addr, h.config.IPv6GroupPrefix).Masked() return netip.PrefixFrom(addr, ipv6GroupPrefix).Masked()
} }
return netip.PrefixFrom(addr, addr.BitLen()) return netip.PrefixFrom(addr, addr.BitLen())
+31 -19
View File
@@ -10,6 +10,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset" "sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -22,11 +23,22 @@ import (
// exploitsignature, raises the core_rule_set_match alert, and in block mode // exploitsignature, raises the core_rule_set_match alert, and in block mode
// refuses the request with 403, which is an offence its client's history // refuses the request with 403, which is an offence its client's history
// counts. Its figures are the score and the threshold. // counts. Its figures are the score and the threshold.
type coreRuleSet struct{} type coreRuleSet struct {
// inspector is the Core Rule Set, nil while SWW_WAF_CORE_RULE_SET_MODE is
// off.
inspector CoreRuleSetInspector
metrics *metrics.Metrics
alerts *alerts.Queue
mode config.CoreRuleSetMode // SWW_WAF_CORE_RULE_SET_MODE
exemptPaths []string // SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS
scoreThreshold int // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD
observe bool // SWW_WAF_MODE is observe
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
}
func (coreRuleSet) name() string { return "core_rule_set" } func (coreRuleSet) name() string { return "core_rule_set" }
func (coreRuleSet) check( func (c coreRuleSet) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
@@ -36,7 +48,6 @@ func (coreRuleSet) check(
return nil return nil
} }
cfg := rq.h.config
result := findings.inspection result := findings.inspection
rq.requestBytes = int64(len(findings.read)) rq.requestBytes = int64(len(findings.read))
@@ -47,18 +58,18 @@ func (coreRuleSet) check(
rq.line.CoreRuleSetScore = &result.Score rq.line.CoreRuleSetScore = &result.Score
for _, id := range result.RuleIDs { for _, id := range result.RuleIDs {
rq.h.metrics.CoreRuleSetMatched(cfg.CoreRuleSetMode, id) c.metrics.CoreRuleSetMatched(string(c.mode), id)
} }
threshold := cfg.CoreRuleSetScoreThreshold threshold := c.scoreThreshold
if threshold == 0 || result.Score < threshold { if threshold == 0 || result.Score < threshold {
return nil return nil
} }
rq.addTag(config.ExploitSignatureTag) rq.addTag(config.ExploitSignatureTag)
rq.alertCoreRuleSetMatch(result) c.alertCoreRuleSetMatch(rq, result)
if !rq.refusedByCoreRuleSet(result) { if !c.refusedByCoreRuleSet(result) {
return nil return nil
} }
@@ -76,20 +87,21 @@ func (coreRuleSet) check(
// refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core // refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core
// Rule Set found result of: whether the score is at or over // Rule Set found result of: whether the score is at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode. // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode.
func (rq *request) refusedByCoreRuleSet(result coreruleset.Result) bool { func (c coreRuleSet) refusedByCoreRuleSet(result coreruleset.Result) bool {
cfg := rq.h.config threshold := c.scoreThreshold
threshold := cfg.CoreRuleSetScoreThreshold
return cfg.CoreRuleSetMode == config.CoreRuleSetModeBlock && return c.mode == config.CoreRuleSetModeBlock &&
threshold != 0 && result.Score >= threshold threshold != 0 && result.Score >= threshold
} }
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request, // alertCoreRuleSetMatch raises the core_rule_set_match alert for rq, which the
// which the Core Rule Set scored at result, at or over // Core Rule Set scored at result, at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
// score, the method and the path with the query, and, for a request that is // score, the method and the path with the query, and, for a request that is
// not refused for it, the mode: detect, or observe in observe mode. // not refused for it, the mode: detect, or observe in observe mode.
func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) { func (c coreRuleSet) alertCoreRuleSetMatch(
rq *request, result coreruleset.Result,
) {
detail := map[string]any{ detail := map[string]any{
"rule_ids": result.RuleIDs, "rule_ids": result.RuleIDs,
"score": result.Score, "score": result.Score,
@@ -98,17 +110,17 @@ func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
} }
switch { switch {
case rq.h.config.CoreRuleSetMode == config.CoreRuleSetModeDetect: case c.mode == config.CoreRuleSetModeDetect:
detail["mode"] = config.CoreRuleSetModeDetect detail["mode"] = string(config.CoreRuleSetModeDetect)
case rq.h.config.Observe: case c.observe:
detail["mode"] = "observe" detail["mode"] = "observe"
} }
rq.h.alerts.Raise(alerts.Alert{ c.alerts.Raise(alerts.Alert{
Event: alerts.EventCoreRuleSetMatch, Event: alerts.EventCoreRuleSetMatch,
Host: rq.site, Host: rq.site,
Client: rq.client, Client: rq.client,
Netblock: rq.h.clientGroup(rq.client), Netblock: clientGroup(c.ipv6GroupPrefix, rq.client),
ASN: rq.line.ASN, ASN: rq.line.ASN,
ASName: rq.line.ASName, ASName: rq.line.ASName,
Country: rq.line.Country, Country: rq.line.Country,
+3 -3
View File
@@ -12,7 +12,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and // The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and
@@ -395,9 +395,9 @@ func TestGiteasFixesApplyOnlyToAHostnameThatServesGitea(t *testing.T) {
}) })
wantDisposition(t, do(t, s.question(siteA, client, fileName)), wantDisposition(t, do(t, s.question(siteA, client, fileName)),
http.StatusOK, waf.DispositionAllow) http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, do(t, s.question(siteB, client, fileName)), wantDisposition(t, do(t, s.question(siteB, client, fileName)),
http.StatusForbidden, waf.DispositionDeny) http.StatusForbidden, wafprotocol.DispositionDeny)
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward) wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward)
+8 -4
View File
@@ -7,6 +7,7 @@ import (
"slices" "slices"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -14,16 +15,18 @@ import (
// does not hold the client's country as it was looked up. A client without a // does not hold the client's country as it was looked up. A client without a
// country, or whose country cannot be found, is refused. Its figures are the // country, or whose country cannot be found, is refused. Its figures are the
// country and the setting that refused it. // country and the setting that refused it.
type countries struct{} type countries struct {
allowed []string // SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES
}
func (countries) name() string { return "countries" } func (countries) name() string { return "countries" }
func (countries) check( func (c countries) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
allowed := rq.h.config.ExclusivelyAllowedCountries allowed := c.allowed
country := rq.line.Country country := rq.line.Country
if len(allowed) == 0 || slices.Contains(allowed, country) { if len(allowed) == 0 || slices.Contains(allowed, country) {
@@ -33,7 +36,8 @@ func (countries) check(
return &verdict{ return &verdict{
action: requestlog.ActionCountryDenied, action: requestlog.ActionCountryDenied,
figures: map[string]any{ figures: map[string]any{
"country": country, "setting": "SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", "country": country,
"setting": config.ExclusivelyAllowedCountriesSetting,
}, },
} }
} }
+10 -3
View File
@@ -4,6 +4,7 @@ package waf
import ( import (
"context" "context"
"net/netip"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
@@ -11,12 +12,18 @@ import (
// denyNets refuses a client in SWW_WAF_DENY_NETS. Its figure is the netblock of // denyNets refuses a client in SWW_WAF_DENY_NETS. Its figure is the netblock of
// the setting the client is in. // the setting the client is in.
type denyNets struct{} type denyNets struct {
netblocks []netip.Prefix // SWW_WAF_DENY_NETS
}
func (denyNets) name() string { return "deny_nets" } func (denyNets) name() string { return "deny_nets" }
func (denyNets) check(_ context.Context, rq *request, _ time.Time) *verdict { func (d denyNets) check(
for _, netblock := range rq.h.config.DenyNets { _ context.Context,
rq *request,
_ time.Time,
) *verdict {
for _, netblock := range d.netblocks {
if netblock.Contains(rq.client) { if netblock.Contains(rq.client) {
return &verdict{ return &verdict{
action: requestlog.ActionClientDenied, action: requestlog.ActionClientDenied,
+47
View File
@@ -0,0 +1,47 @@
package waf
import (
"net/netip"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
func TestDenyNetsRefusesOnlyAClientInsideItsNetblocks(t *testing.T) {
t.Parallel()
heuristic := denyNets{
netblocks: []netip.Prefix{netip.MustParsePrefix("198.51.100.0/24")},
}
// The request has no handler: the heuristic reads only what it holds.
inside := &request{
decision: decision{client: netip.MustParseAddr("198.51.100.7")},
}
v := heuristic.check(t.Context(), inside, time.Time{})
if v == nil {
t.Fatal("a client inside the netblock was let through")
}
if v.action != requestlog.ActionClientDenied {
t.Errorf(
"action = %q, want %q",
v.action,
requestlog.ActionClientDenied,
)
}
if got := v.figures[figureNetblock]; got != "198.51.100.0/24" {
t.Errorf("netblock figure = %v, want 198.51.100.0/24", got)
}
outside := &request{
decision: decision{client: netip.MustParseAddr("203.0.113.7")},
}
if v := heuristic.check(t.Context(), outside, time.Time{}); v != nil {
t.Errorf("a client outside the netblock was refused: %+v", v)
}
}
+14 -5
View File
@@ -9,6 +9,7 @@ import (
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -19,20 +20,28 @@ import (
// being log, it does nothing more. A zone without a verdict on the client is // being log, it does nothing more. A zone without a verdict on the client is
// asked about it in the background, and the request does not wait for the // asked about it in the background, and the request does not wait for the
// answer. Its figure is the zones that list the client, their keys masked. // answer. Its figure is the zones that list the client, their keys masked.
type dnsbl struct{} type dnsbl struct {
dnsbl *reputation.DNSBL
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
hitNoter reputationHitNoter
}
func (dnsbl) name() string { return "dnsbl" } func (dnsbl) name() string { return "dnsbl" }
func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict { func (d dnsbl) check(
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client) ctx context.Context,
rq *request,
_ time.Time,
) *verdict {
listedBy := d.dnsbl.ListedBy(ctx, rq.client)
rq.dnsblListed = len(listedBy) > 0 rq.dnsblListed = len(listedBy) > 0
rq.noteListed(listedBy, "listed by a DNSBL zone") d.hitNoter.noteListed(rq, listedBy, "listed by a DNSBL zone")
for _, zone := range listedBy { for _, zone := range listedBy {
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone)) rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
} }
if !rq.dnsblListed || rq.h.config.DNSBLAbuseIPDBAction != deny { if !rq.dnsblListed || d.action != config.ReputationActionDeny {
return nil return nil
} }
+5 -5
View File
@@ -1,13 +1,13 @@
package waf package waf
import "time" import "sneak.berlin/go/smallwebwaf/internal/reputation"
// BesideLookupTimeLimit is besideLookupTimeLimit. // BesideLookupTimeLimit is besideLookupTimeLimit.
const BesideLookupTimeLimit = besideLookupTimeLimit const BesideLookupTimeLimit = besideLookupTimeLimit
// SetBesideLookupTimeLimit gives the heuristics of besideLookup limit on s in // BanAbuseIPDBBlacklist does on s what a fetch of AbuseIPDB's blacklist with
// place of besideLookupTimeLimit. // the clients on listed does.
func (s *Server) SetBesideLookupTimeLimit(limit time.Duration) { func (s *Server) BanAbuseIPDBBlacklist(listed []reputation.BlacklistEntry) {
h := s.Handler.(*handler) //nolint:forcetypeassert // New makes it one h := s.Handler.(*handler) //nolint:forcetypeassert // New makes it one
h.besideLookupLimit = limit BanAbuseIPDBBlacklist(h.config, h.ledger, h.processLog, h.now(), listed)
} }
+8 -3
View File
@@ -5,7 +5,7 @@ import (
"testing" "testing"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
func TestHealthEndpointCountsTowardNoLimit(t *testing.T) { func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
@@ -25,8 +25,13 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
) )
for range healthChecks { for range healthChecks {
got := s.ownEndpointRequest("", http.MethodGet, waf.HealthPath, "", got := s.ownEndpointRequest(
http.StatusOK) "",
http.MethodGet,
wafprotocol.HealthPath,
"",
http.StatusOK,
)
if string(got.body) != "ok\n" || if string(got.body) != "ok\n" ||
got.header.Get("Content-Type") != contentType { got.header.Get("Content-Type") != contentType {
+23 -77
View File
@@ -5,9 +5,20 @@ import (
"time" "time"
) )
// To add a heuristic, write one file with a type that implements heuristic, // To add a heuristic, write one file with a struct that implements heuristic
// and add one line for it to one of the lists below. Its settings go in // and holds the settings and parts it reads. Add one line for it to one of the
// config.WAF, and the comment at the top of its file names them. // lists that newHandler builds, which hands it those settings and parts. Its
// settings go in config.WAF, and the comment at the top of its file names them.
// A heuristic of afterLookup that can refuse a request whatever the lookup
// answers also implements refusesWithoutLookup.
//
// Three places outside its file may change with it:
// - waitForGeoJS in newHandler lists the settings that need the lookup's
// answer;
// - requestCanGetTag in internal/config/waf.go holds which setting can give a
// request which tag;
// - findBesideLookup does the work of each heuristic of besideLookup, by
// name.
// heuristic is one of the checks that decide what waf does with a // heuristic is one of the checks that decide what waf does with a
// request. // request.
@@ -19,6 +30,15 @@ type heuristic interface {
check(ctx context.Context, rq *request, now time.Time) *verdict check(ctx context.Context, rq *request, now time.Time) *verdict
} }
// refusesWithoutLookup is a heuristic of afterLookup that can tell, before the
// lookup answers, that it will refuse a request whatever the answer.
type refusesWithoutLookup interface {
// refusesWhateverTheLookupAnswers reports whether the heuristic will refuse
// rq at now, whatever the lookup answers about its client. It changes
// nothing.
refusesWhateverTheLookupAnswers(rq *request, now time.Time) bool
}
// verdict is a heuristic refusing a request, or banning its client: the action // verdict is a heuristic refusing a request, or banning its client: the action
// the log line names, one of requestlog's Action constants, and the figures the // the log line names, one of requestlog's Action constants, and the figures the
// heuristic acted on, by name, such as the count and the limit. // heuristic acted on, by name, such as the count and the limit.
@@ -36,80 +56,6 @@ const (
figureNetblock = "netblock" figureNetblock = "netblock"
) )
// The lists of heuristics, each in the order its heuristics run. A client in
// SWW_WAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number and
// country are looked up, so that a client they refuse is not looked up.
// header_limits comes first: it only measures and counts, and a request it
// refuses causes no other work.
func beforeLookup() []heuristic {
return []heuristic{
headerLimits{},
tagLists{},
routeWatchTags{},
denyNets{},
activeBan{},
}
}
// afterLookup are the heuristics that run once the lookup has been made.
func afterLookup() []heuristic {
return []heuristic{
countries{},
blocklists{},
dnsbl{},
abuseIPDB{},
tagDeny{},
rateLimits{},
}
}
// besideLookup are the heuristics that only read the request. Their work runs
// beside the lookup and afterLookup, in findBesideLookup, which changes nothing
// of the request. Their check then applies what that work found, and runs only
// once waf takes it.
func besideLookup() []heuristic {
return []heuristic{
trapPaths{},
ruleFiles{},
coreRuleSet{},
}
}
// afterBesideLookup are the heuristics that run once afterLookup and
// besideLookup have both let the request through. tag_deny runs a second time,
// on the tag rule_files and core_rule_set can add.
func afterBesideLookup() []heuristic {
return []heuristic{
tagDeny{},
}
}
// afterAnswer are the heuristics that run on a request once waf has
// answered it, before its decision line is written. They ban the client, and do
// not change what the request got: their verdict's action is not used.
func afterAnswer() []heuristic {
return []heuristic{
refusalLimit{},
}
}
// onReport are the heuristics that run on an allowed request when rproxy's
// report of it comes, before its report line is written. Like afterAnswer, they
// only ban.
func onReport() []heuristic {
return []heuristic{
byteLimits{},
}
}
// runAfterAnswer runs the heuristics of afterAnswer on the request, at the time
// it is called.
func (rq *request) runAfterAnswer(ctx context.Context) {
rq.runBanningHeuristics(ctx, afterAnswer())
}
// runHeuristics runs list, one of those checkClient runs, on the request at now, // runHeuristics runs list, one of those checkClient runs, on the request at now,
// and returns the first verdict, or nil when none gives one. The log line's // and returns the first verdict, or nil when none gives one. The log line's
// heuristic and heuristic_figures name the heuristic that gave it, and give its // heuristic and heuristic_figures name the heuristic that gave it, and give its
+2 -1
View File
@@ -11,6 +11,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) { func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
@@ -118,7 +119,7 @@ func TestOwnEndpointsAreNotInTheHistory(t *testing.T) {
s.ownEndpointRequest( s.ownEndpointRequest(
"", "",
http.MethodGet, http.MethodGet,
waf.HealthPath, wafprotocol.HealthPath,
"", "",
http.StatusOK, http.StatusOK,
) )
+31 -38
View File
@@ -2,23 +2,17 @@ package waf
import ( import (
"context" "context"
"net/http" "log/slog"
"net/netip" "net/netip"
"strings" "strings"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
) "sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
// The headers in which an answer of allow gives rproxy the client's AS number
// and country, for the app, while SWW_WAF_ADD_LOOKUP_HEADERS is true. Go writes
// every header name in this form, as it sends it and as it receives it, so
// X-Client-ASN arrives as X-Client-Asn; header names are not case-sensitive.
const (
asnHeader = "X-Client-Asn"
countryHeader = "X-Client-Country"
) )
// lookupResult is what the lookup gave for a request: whether its client was // lookupResult is what the lookup gave for a request: whether its client was
@@ -45,15 +39,16 @@ func (rq *request) startLookUp(
) <-chan lookupResult { ) <-chan lookupResult {
result := make(chan lookupResult, 1) result := make(chan lookupResult, 1)
if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) { if rq.h.config.LookupSource == config.LookupSourceOff ||
!canBePlaced(rq.client) {
result <- lookupResult{} result <- lookupResult{}
return result return result
} }
client := rq.h.clientGroup(rq.client) client := clientGroup(rq.h.config.IPv6GroupPrefix, rq.client)
if rq.h.config.LookupSource == "file" { if rq.h.config.LookupSource == config.LookupSourceFile {
result <- lookupResult{ result <- lookupResult{
lookedUp: true, answer: rq.h.lookupFile.LookUp(client), lookedUp: true, answer: rq.h.lookupFile.LookUp(client),
} }
@@ -128,22 +123,32 @@ func countryTag(country string) string {
return config.CountryTagPrefix + country return config.CountryTagPrefix + country
} }
// addLookup adds answer, an answer about a client from the lookup // AddLookup adds answer, an answer about a client from the lookup database
// database or GeoJS, to the client's history, and to the notes of the bans // or GeoJS, to the client's history in limiter, and to the notes of the bans
// on its netblock that have no AS number, AS name or country yet. // in ledger on its netblock that have no AS number, AS name or country yet.
func (h *handler) addLookup(answer geojs.Answer) { // cfg sets the netblock's size.
h.limiter.AddLookup(answer.Client, answer.Answered, func AddLookup(
answer.ASN, answer.ASName, answer.Country) cfg *config.WAF, limiter *ratelimit.Limiter, ledger *bans.Ledger,
h.ledger.AddLookup(h.netblock(answer.Client.Addr()), answer geojs.Answer,
) {
limiter.AddLookup(answer.Client, answer.Answered,
answer.ASN, answer.ASName, answer.Country) answer.ASN, answer.ASName, answer.Country)
netblock := banNetblock(
cfg.BanScopeV4Prefix, cfg.IPv6GroupPrefix, answer.Client.Addr(),
)
ledger.AddLookup(netblock, answer.ASN, answer.ASName, answer.Country)
} }
// geojsFailed logs a failure of GeoJS, raises a source_failure alert for it // GeoJSFailed logs a failure of GeoJS to processLog, raises a
// and counts it. askingAgainIn is how long GeoJS is left alone after it. // source_failure alert for it on alertQueue and counts it in m.
func (h *handler) geojsFailed(err error, askingAgainIn time.Duration) { // askingAgainIn is how long GeoJS is left alone after it.
h.processLog.Warn("asking GeoJS failed", func GeoJSFailed(
processLog *slog.Logger, alertQueue *alerts.Queue, m *metrics.Metrics,
err error, askingAgainIn time.Duration,
) {
processLog.Warn("asking GeoJS failed",
"error", err.Error(), "asking_again_in", askingAgainIn.String()) "error", err.Error(), "asking_again_in", askingAgainIn.String())
h.alerts.Raise(alerts.Alert{ alertQueue.Raise(alerts.Alert{
Event: alerts.EventSourceFailure, Event: alerts.EventSourceFailure,
Reason: "asking GeoJS failed", Reason: "asking GeoJS failed",
Detail: map[string]any{ Detail: map[string]any{
@@ -151,19 +156,7 @@ func (h *handler) geojsFailed(err error, askingAgainIn time.Duration) {
"asking_again_in": askingAgainIn.String(), "asking_again_in": askingAgainIn.String(),
}, },
}) })
h.metrics.GeoJSFailures.Inc() m.GeoJSFailures.Inc()
}
// setLookupHeaders sets the headers that give the client's AS number and
// country, leaving out one that is unknown.
func setLookupHeaders(header http.Header, asn, country string) {
if asn != "" {
header.Set(asnHeader, asn)
}
if country != "" {
header.Set(countryHeader, country)
}
} }
// canBePlaced reports whether a lookup can place addr: private, loopback // canBePlaced reports whether a lookup can place addr: private, loopback
+2 -1
View File
@@ -16,6 +16,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -66,7 +67,7 @@ func TestMetricsNeedTheTokenAndOtherPathsAreNotFound(t *testing.T) {
{http.MethodPost, waf.MetricsPath, bearer, http.StatusNotFound}, {http.MethodPost, waf.MetricsPath, bearer, http.StatusNotFound},
{http.MethodGet, waf.MetricsPath + "/", bearer, http.StatusNotFound}, {http.MethodGet, waf.MetricsPath + "/", bearer, http.StatusNotFound},
{http.MethodGet, "/_sww/bans", bearer, http.StatusNotFound}, {http.MethodGet, "/_sww/bans", bearer, http.StatusNotFound},
{http.MethodPost, waf.HealthPath, "", http.StatusNotFound}, {http.MethodPost, wafprotocol.HealthPath, "", http.StatusNotFound},
} { } {
got := s.ownEndpointRequest( got := s.ownEndpointRequest(
tc.authorization, tc.authorization,
+208 -180
View File
@@ -1,10 +1,11 @@
// Package waf is the server the waf command runs. It answers // Package waf is the server the waf command runs. It answers
// rproxy's questions, takes rproxy's reports and answers its own endpoints, and // rproxy's questions, takes rproxy's reports and answers its own endpoints, and
// writes a log line for each. // writes a log line for each. It also bans the clients on AbuseIPDB's
// blacklist each time the reputation package fetches it.
package waf package waf
import ( import (
"fmt" "cmp"
"io" "io"
"log/slog" "log/slog"
"net/http" "net/http"
@@ -27,14 +28,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
) )
// ownEndpointPrefix starts the path of each of waf's own endpoints. The app
// never sees a path under it.
const ownEndpointPrefix = "/_sww/"
// HealthPath is the health endpoint of waf and of rproxy, which the
// container's health check asks.
const HealthPath = "/_sww/healthz"
// MetricsPath is where the metrics are, for a request that carries // MetricsPath is where the metrics are, for a request that carries
// SWW_WAF_METRICS_TOKEN. // SWW_WAF_METRICS_TOKEN.
const MetricsPath = "/_sww/metrics" const MetricsPath = "/_sww/metrics"
@@ -56,17 +49,33 @@ type Params struct {
RequestLog io.Writer RequestLog io.Writer
// ProcessLog receives the process's own messages. // ProcessLog receives the process's own messages.
ProcessLog *slog.Logger ProcessLog *slog.Logger
// GeoJSURL is where clients' AS numbers and countries are looked up // Metrics count what waf does. Whoever builds them adds the metrics
// while SWW_WAF_LOOKUP_SOURCE is geojs, normally geojs.URL. // read from the other parts.
GeoJSURL string Metrics *metrics.Metrics
// AbuseIPDBURL is where clients are checked with AbuseIPDB while // Limiter is the table of clients the rate limits count, with each
// SWW_WAF_ABUSEIPDB_KEY is set, normally reputation.AbuseIPDBURL. // client's history.
AbuseIPDBURL string Limiter *ratelimit.Limiter
// Ledger holds the bans.
Ledger *bans.Ledger
// Anomalies are the counters of the anomaly thresholds.
Anomalies *anomaly.Counters
// Lists are the lists fetched from URLs, DNSBL the DNSBL zones'
// verdicts, and AbuseIPDB AbuseIPDB's scores and the checks spent.
Lists *reputation.Lists
DNSBL *reputation.DNSBL
AbuseIPDB *reputation.AbuseIPDB
// RouteWatch asks routewatch about clients, and is nil while
// SWW_WAF_ROUTEWATCH_URL is unset.
RouteWatch *routewatch.Client
// GeoJS looks clients' AS numbers and countries up while
// SWW_WAF_LOOKUP_SOURCE is geojs. It is given AddLookup as its Answered
// and GeoJSFailed as its Failed.
GeoJS *geojs.GeoJS
// LookupFile is the lookup database they are looked up in while // LookupFile is the lookup database they are looked up in while
// SWW_WAF_LOOKUP_SOURCE is file, and nil otherwise. // SWW_WAF_LOOKUP_SOURCE is file, and nil otherwise.
LookupFile *lookup.File LookupFile *lookup.File
// CoreRuleSet is what NewCoreRuleSet returns: what inspects requests // CoreRuleSet is what inspects requests while
// while SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is. // SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
CoreRuleSet CoreRuleSetInspector CoreRuleSet CoreRuleSetInspector
// Now tells the time by which requests are counted for the rate // Now tells the time by which requests are counted for the rate
// limits, bans are made and run out, and GeoJS's answers are kept, // limits, bans are made and run out, and GeoJS's answers are kept,
@@ -83,6 +92,11 @@ type Params struct {
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing, // failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
// or the day's AbuseIPDB checks used up. // or the day's AbuseIPDB checks used up.
Alerts *alerts.Queue Alerts *alerts.Queue
// BesideLookupTimeLimit is how long the heuristics of besideLookup have
// for a request. Zero means besideLookupTimeLimit. Tests that run the
// Core Rule Set lengthen it, since the race detector and a busy host can
// slow an inspection past besideLookupTimeLimit.
BesideLookupTimeLimit time.Duration
} }
// CoreRuleSetInspector inspects a request with the Core Rule Set, as // CoreRuleSetInspector inspects a request with the Core Rule Set, as
@@ -93,11 +107,8 @@ type CoreRuleSetInspector interface {
) (coreruleset.Result, []byte, error) ) (coreruleset.Result, []byte, error)
} }
// Server is the server waf runs, with the parts whose state the state // Server is the server waf runs, with the parts of its Params. The tests
// files keep, the lookup database, nil unless SWW_WAF_LOOKUP_SOURCE is file, the // read them.
// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts,
// AbuseIPDB's scores and checks spent, routewatch's answers, and the metrics.
// RouteWatch is nil while SWW_WAF_ROUTEWATCH_URL is unset.
type Server struct { type Server struct {
*http.Server *http.Server
@@ -113,188 +124,205 @@ type Server struct {
Metrics *metrics.Metrics Metrics *metrics.Metrics
} }
// newHandler returns the handler with what every request shares, as the // newHandler returns the handler with what every request shares, the parts
// settings in params make it. // in params among it. It builds the heuristics, each with the settings and
// parts it reads, and the lists they run in.
//
//nolint:funlen // a few lines for each heuristic, a list that grows with them
func newHandler(params Params) *handler { func newHandler(params Params) *handler {
m := metrics.New(
params.Config.MetricsBusiestCountriesAndASNs,
params.Config.InstanceName,
)
lists, dnsbl, abuseIPDB := newReputation(params, m)
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport) awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
if err != nil { if err != nil {
panic(err) // New fails only for a size below one panic(err) // New fails only for a size below one
} }
cfg := params.Config
h := &handler{ h := &handler{
config: params.Config, config: cfg,
requestLog: params.RequestLog, requestLog: params.RequestLog,
processLog: params.ProcessLog, processLog: params.ProcessLog,
now: params.Now, now: params.Now,
metrics: m, metrics: params.Metrics,
awaitingReport: awaitingReport, awaitingReport: awaitingReport,
limiter: ratelimit.New(ratelimit.Limits{ limiter: params.Limiter,
PerMinute: params.Config.RateLimitPerMinute, ledger: params.Ledger,
PerHour: params.Config.RateLimitPerHour, geojs: params.GeoJS,
PerDay: params.Config.RateLimitPerDay, anomalies: params.Anomalies,
BytesPerMinute: params.Config.BytesLimitPerMinute, lookupFile: params.LookupFile,
BytesPerHour: params.Config.BytesLimitPerHour, // The country list, the headers, the tag settings that name an asn=
BytesPerDay: params.Config.BytesLimitPerDay, // or country= tag and the biased thresholds act on the answer before
}, params.Config.MaxTrackedClients), // the request goes on.
ledger: bans.New(bans.Params{ waitForGeoJS: len(cfg.ExclusivelyAllowedCountries) > 0 ||
LimitBanDuration: params.Config.LimitBanDuration, cfg.AddLookupHeaders ||
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow, cfg.TagsNeedLookup() ||
LimitBanPermanentAbove: params.Config.LimitBanPermanentAbove, biasedThresholdsSet(cfg),
AttackBanDuration: params.Config.AttackBanDuration, besideLookupLimit: cmp.Or(
MaxBans: params.Config.MaxBans, params.BesideLookupTimeLimit,
}), besideLookupTimeLimit,
anomalies: anomaly.New(anomaly.Params{ ),
Client: params.Config.AnomalyClient,
Net: params.Config.AnomalyNet,
ASN: params.Config.AnomalyASN,
Total: params.Config.AnomalyTotal,
WatchedNet: params.Config.AnomalyWatchedNet,
NetV4Prefix: params.Config.AnomalyNetV4Prefix,
NetV6Prefix: params.Config.AnomalyNetV6Prefix,
NamedNetblocks: params.Config.AnomalyWatchedNets,
Alerts: params.Alerts,
}),
lookupFile: params.LookupFile,
lists: lists,
dnsbl: dnsbl,
abuseIPDB: abuseIPDB,
rules: params.Rules,
coreRuleSet: params.CoreRuleSet,
alerts: params.Alerts,
besideLookupLimit: besideLookupTimeLimit,
}
if params.Config.RouteWatchURL != "" {
h.routeWatch = routewatch.New(routewatch.Params{
URL: params.Config.RouteWatchURL, Now: params.Now, ProcessLog: params.ProcessLog,
})
} }
// The country list, the headers, the tag settings that name an asn= or sharedBanMaker := banMaker{
// country= tag and the biased thresholds act on the answer before the ledger: params.Ledger,
// request goes on. limiter: params.Limiter,
h.waitForGeoJS = len(params.Config.ExclusivelyAllowedCountries) > 0 || metrics: params.Metrics,
params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() || alerts: params.Alerts,
biasedThresholdsSet(params.Config) observe: cfg.Observe,
h.geojs = geojs.New(geojs.Params{ banResponse: cfg.BanResponse,
URL: params.GeoJSURL, banScopeV4Prefix: cfg.BanScopeV4Prefix,
Answered: h.addLookup, ipv6GroupPrefix: cfg.IPv6GroupPrefix,
Now: params.Now, }
Asked: m.GeoJSRequests.Inc, hitNoter := reputationHitNoter{
Failed: h.geojsFailed, metrics: params.Metrics,
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc, alerts: params.Alerts,
}) ipv6GroupPrefix: cfg.IPv6GroupPrefix,
m.AddBansAndClients(h.ledger, h.limiter, params.Now) }
m.AddRules(params.Rules)
h.trapPaths = trapPaths{paths: cfg.TrapPaths, banMaker: sharedBanMaker}
h.ruleFiles = ruleFiles{
rules: params.Rules,
metrics: params.Metrics,
banMaker: sharedBanMaker,
}
h.coreRuleSet = coreRuleSet{
inspector: params.CoreRuleSet,
metrics: params.Metrics,
alerts: params.Alerts,
mode: cfg.CoreRuleSetMode,
exemptPaths: cfg.CoreRuleSetExemptPaths,
scoreThreshold: cfg.CoreRuleSetScoreThreshold,
observe: cfg.Observe,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.beforeLookup = []heuristic{
headerLimits{},
tagLists{lists: params.Lists},
routeWatchTags{
routeWatch: params.RouteWatch,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
},
denyNets{netblocks: cfg.DenyNets},
activeBan{banMaker: sharedBanMaker},
}
h.afterLookup = []heuristic{
countries{allowed: cfg.ExclusivelyAllowedCountries},
blocklists{
lists: params.Lists,
action: cfg.BlocklistAction,
hitNoter: hitNoter,
},
dnsbl{
dnsbl: params.DNSBL,
action: cfg.DNSBLAbuseIPDBAction,
hitNoter: hitNoter,
},
abuseIPDB{
abuseIPDB: params.AbuseIPDB,
limiter: params.Limiter,
keySet: cfg.AbuseIPDBKey != "",
minScore: cfg.AbuseIPDBMinScore,
action: cfg.DNSBLAbuseIPDBAction,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
hitNoter: hitNoter,
},
tagDeny{tags: cfg.TagDeny},
rateLimits{
limiter: params.Limiter,
exemptNets: cfg.RateLimitExemptNets,
exemptPaths: cfg.RateLimitExemptPaths,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
biasedThresholds: biasedThresholds{
lists: params.Lists,
tagLimitPercent: cfg.TagLimitPercent,
asnBytesLimitPercent: cfg.ASNBytesLimitPercent,
countryBytesLimitPercent: cfg.CountryBytesLimitPercent,
unknownCountryLimitPercent: cfg.UnknownCountryLimitPercent,
blocklistAction: cfg.BlocklistAction,
blocklistLimitPercent: cfg.BlocklistLimitPercent,
dnsblAbuseIPDBAction: cfg.DNSBLAbuseIPDBAction,
dnsblAbuseIPDBLimitPercent: cfg.DNSBLAbuseIPDBLimitPercent,
},
banMaker: sharedBanMaker,
},
}
h.besideLookup = []heuristic{h.trapPaths, h.ruleFiles, h.coreRuleSet}
h.afterBesideLookup = []heuristic{tagDeny{tags: cfg.TagDeny}}
h.afterAnswer = []heuristic{
refusalLimit{
limiter: params.Limiter,
perMinute: cfg.RefusalsLimitPerMinute,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.onReport = []heuristic{
byteLimits{
limiter: params.Limiter,
bytesCounted: cfg.BytesCounted,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
return h return h
} }
// newReputation returns the lists fetched from URLs, the DNSBL zones'
// verdicts and AbuseIPDB's scores, as the settings in params name them,
// with none fetched, asked for or checked yet, and adds their metrics to
// m, AbuseIPDB's while SWW_WAF_ABUSEIPDB_KEY is set.
func newReputation(
params Params, m *metrics.Metrics,
) (*reputation.Lists, *reputation.DNSBL, *reputation.AbuseIPDB) {
cfg := params.Config
lists := reputation.New(reputation.Params{
BlocklistURLs: cfg.BlocklistURLs,
Refresh: cfg.BlocklistRefresh,
ASNLimitPercentURL: cfg.ASNLimitPercentURL,
TagLists: cfg.TagLists,
Now: params.Now,
ProcessLog: params.ProcessLog,
Alerts: params.Alerts,
})
dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
Zones: cfg.DNSBLZones, Resolver: cfg.DNSBLResolver,
CacheTTL: cfg.IPReputationCacheTTL,
Timeout: cfg.DNSBLAbuseIPDBTimeout, Now: params.Now, ProcessLog: params.ProcessLog,
Alerts: params.Alerts,
})
abuseIPDB := reputation.NewAbuseIPDB(reputation.AbuseIPDBParams{
URL: params.AbuseIPDBURL, Key: cfg.AbuseIPDBKey, MinScore: cfg.AbuseIPDBMinScore,
DailyBudget: cfg.AbuseIPDBDailyBudget, CacheTTL: cfg.IPReputationCacheTTL,
Timeout: cfg.DNSBLAbuseIPDBTimeout, Now: params.Now, ProcessLog: params.ProcessLog,
Alerts: params.Alerts,
})
m.AddReputation(lists, dnsbl)
if cfg.AbuseIPDBKey != "" {
m.AddAbuseIPDB(abuseIPDB)
}
return lists, dnsbl, abuseIPDB
}
// NewCoreRuleSet returns the Core Rule Set at SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL,
// without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies
// up to SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE, with the exclusions in
// SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS,
// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. An exclusions file that
// coreruleset.New refuses is an error in every mode. While the mode is off, a
// set file is loaded with the Core Rule Set only to check it.
func NewCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff &&
cfg.CoreRuleSetExclusionsFile == "" {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
}
coreRuleSet, err := coreruleset.New(coreruleset.Params{
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
DisabledRules: cfg.CoreRuleSetDisabledRules,
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
Apps: cfg.Apps,
})
if err != nil {
return nil, fmt.Errorf(
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE: %w",
err,
)
}
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
}
return coreRuleSet, nil
}
// handler is waf's handler. It holds what every request shares; what // handler is waf's handler. It holds what every request shares; what
// belongs to one request is in a request. // belongs to one request is in a request.
type handler struct { type handler struct {
config *config.WAF config *config.WAF
requestLog io.Writer requestLog io.Writer
processLog *slog.Logger processLog *slog.Logger
now func() time.Time now func() time.Time
metrics *metrics.Metrics metrics *metrics.Metrics
limiter *ratelimit.Limiter limiter *ratelimit.Limiter
ledger *bans.Ledger ledger *bans.Ledger
geojs *geojs.GeoJS geojs *geojs.GeoJS
anomalies *anomaly.Counters anomalies *anomaly.Counters
lookupFile *lookup.File lookupFile *lookup.File
lists *reputation.Lists
dnsbl *reputation.DNSBL
abuseIPDB *reputation.AbuseIPDB
routeWatch *routewatch.Client
rules *rules.Files
coreRuleSet CoreRuleSetInspector
alerts *alerts.Queue
// waitForGeoJS is true when a setting needs each request's answer from // waitForGeoJS is true when a setting needs each request's answer from
// GeoJS before the request goes on. // GeoJS before the request goes on.
waitForGeoJS bool waitForGeoJS bool
// besideLookupLimit is besideLookupTimeLimit. Only tests change it. // besideLookupLimit is how long the heuristics of besideLookup have for a
// request, as Params.BesideLookupTimeLimit says.
besideLookupLimit time.Duration besideLookupLimit time.Duration
// awaitingReport holds the decision on each answered question by its id // awaitingReport holds the decision on each answered question by its id
// until its report comes, and nil for a question still being handled. // until its report comes, and nil for a question still being handled.
awaitingReport *lru.Cache[string, *decision] awaitingReport *lru.Cache[string, *decision]
// The lists of heuristics, each in the order its heuristics run. A client
// in SWW_WAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number
// and country are looked up, so that a client they refuse is not looked
// up. header_limits comes first: it only measures and counts, and a
// request it refuses causes no other work.
beforeLookup []heuristic
// afterLookup are the heuristics that run once the lookup has been made.
afterLookup []heuristic
// besideLookup are the heuristics that only read the request. Their work
// runs beside the lookup and afterLookup, in findBesideLookup, which
// changes nothing of the request. Their check then applies what that work
// found, and runs only once waf takes it.
besideLookup []heuristic
// afterBesideLookup are the heuristics that run once afterLookup and
// besideLookup have both let the request through. tag_deny runs a second
// time, on the tag rule_files and core_rule_set can add.
afterBesideLookup []heuristic
// afterAnswer are the heuristics that run on a request once waf has
// answered it, before its decision line is written. They ban the client,
// and do not change what the request got: their verdict's action is not
// used.
afterAnswer []heuristic
// onReport are the heuristics that run on an allowed request when
// rproxy's report of it comes, before its report line is written. Like
// afterAnswer, they only ban.
onReport []heuristic
// The heuristics of besideLookup, whose work findBesideLookup does.
trapPaths trapPaths
ruleFiles ruleFiles
coreRuleSet coreRuleSet
} }
+192 -37
View File
@@ -16,10 +16,19 @@ import (
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/routewatch"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
@@ -185,11 +194,11 @@ func (o *output) requestLine(t *testing.T) logLine {
// endedLines waits for count requests that went through rproxy, or to waf's // endedLines waits for count requests that went through rproxy, or to waf's
// own endpoints, to have ended, and returns the last log line of each, in the // own endpoints, to have ended, and returns the last log line of each, in the
// order waf took them. A question's last line is the one its report writes. // order waf took them. A question's last line is the one its report writes.
// It waits as long as it takes, as requestWithBody does.
func (o *output) endedLines(t *testing.T, count int) []logLine { func (o *output) endedLines(t *testing.T, count int) []logLine {
t.Helper() t.Helper()
deadline := time.Now().Add(waitLimit) for {
for time.Now().Before(deadline) {
found := o.lastLines(t) found := o.lastLines(t)
ended := len(found) >= count ended := len(found) >= count
@@ -205,10 +214,6 @@ func (o *output) endedLines(t *testing.T, count int) []logLine {
time.Sleep(pollInterval) time.Sleep(pollInterval)
} }
t.Fatalf("fewer than %d requests have ended after %s", count, waitLimit)
return nil
} }
// lastLines returns the last request log line written so far of each request, // lastLines returns the last request log line written so far of each request,
@@ -390,11 +395,11 @@ func startRProxy(
server := rproxy.New(rproxy.Params{ server := rproxy.New(rproxy.Params{
Config: cfg, Config: cfg,
RequestLog: io.Discard, RequestLog: io.Discard,
ProcessLog: requestlog.NewProcessLogger( ProcessLog: processlog.New(processlog.Params{
io.Discard, Output: io.Discard,
"rproxy", InstanceName: "rproxy",
slog.LevelInfo, Level: slog.LevelInfo,
), }),
}) })
listener, err := (&net.ListenConfig{}).Listen( listener, err := (&net.ListenConfig{}).Listen(
@@ -424,7 +429,8 @@ func startRProxy(
// With no geojsURL, there is no stand-in for GeoJS to look clients up at, and // With no geojsURL, there is no stand-in for GeoJS to look clients up at, and
// SWW_WAF_LOOKUP_SOURCE is off unless env sets it. While it is file, the lookup // SWW_WAF_LOOKUP_SOURCE is off unless env sets it. While it is file, the lookup
// database SWW_WAF_LOOKUP_DB_PATH names is read. Clients are checked with AbuseIPDB // database SWW_WAF_LOOKUP_DB_PATH names is read. Clients are checked with AbuseIPDB
// at abuseIPDBURL while env sets SWW_WAF_ABUSEIPDB_KEY. // at abuseIPDBURL while env sets SWW_WAF_ABUSEIPDB_KEY, and its blacklist is
// fetched from abuseIPDBBlacklistURL.
// SWW_WAF_CORE_RULE_SET_MODE is off unless env sets it, so that only the tests of the // SWW_WAF_CORE_RULE_SET_MODE is off unless env sets it, so that only the tests of the
// Core Rule Set have their requests inspected by it. // Core Rule Set have their requests inspected by it.
func newWAF( func newWAF(
@@ -435,13 +441,15 @@ func newWAF(
) (*waf.Server, *output, *alerts.Queue) { ) (*waf.Server, *output, *alerts.Queue) {
t.Helper() t.Helper()
return newWAFWith(t, waf.New, geojsURL, now, env) return newWAFWith(t, waf.New, geojsURL, abuseIPDBURL, now, env)
} }
// newWAFWith is newWAF for the server build makes. // newWAFWith is newWAF for the server build makes, with clients checked with
// AbuseIPDB at abuseIPDBCheckURL.
func newWAFWith( func newWAFWith(
t *testing.T, build func(waf.Params) *waf.Server, t *testing.T, build func(waf.Params) *waf.Server,
geojsURL string, now func() time.Time, env map[string]string, geojsURL, abuseIPDBCheckURL string, now func() time.Time,
env map[string]string,
) (*waf.Server, *output, *alerts.Queue) { ) (*waf.Server, *output, *alerts.Queue) {
t.Helper() t.Helper()
@@ -460,11 +468,11 @@ func newWAFWith(
} }
out := &output{} out := &output{}
processLog := requestlog.NewProcessLogger( processLog := processlog.New(processlog.Params{
out, Output: out,
cfg.InstanceName, InstanceName: cfg.InstanceName,
cfg.LogLevel, Level: cfg.LogLevel,
) })
ruleFiles, err := rules.Load(rules.Params{ ruleFiles, err := rules.Load(rules.Params{
Dir: cfg.RuleFilesDir, Enabled: cfg.RuleFilesEnabled, ProcessLog: processLog, Dir: cfg.RuleFilesDir, Enabled: cfg.RuleFilesEnabled, ProcessLog: processLog,
@@ -494,30 +502,177 @@ func newWAFWith(
} }
} }
coreRuleSet, err := waf.NewCoreRuleSet(cfg) params := waf.Params{
Config: cfg,
RequestLog: out,
ProcessLog: processLog,
LookupFile: lookupFile,
Now: now,
Rules: ruleFiles,
Alerts: alertQueue,
// The tests of the limit set it back, and hold the Core Rule Set
// with a stand-in.
BesideLookupTimeLimit: time.Hour,
}
addCoreRuleSet(t, &params)
addCounters(&params)
addOutsideSources(&params, geojsURL, abuseIPDBCheckURL)
server := build(params)
return server, out, alertQueue
}
// addCoreRuleSet adds to params the Core Rule Set, with the settings for it,
// while SWW_WAF_CORE_RULE_SET_MODE is not off.
func addCoreRuleSet(t *testing.T, params *waf.Params) {
t.Helper()
cfg := params.Config
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
return
}
coreRuleSet, err := coreruleset.New(coreruleset.Params{
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
DisabledRules: cfg.CoreRuleSetDisabledRules,
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
Apps: cfg.Apps,
})
if err != nil { if err != nil {
t.Fatalf("Core Rule Set: %v", err) t.Fatalf("Core Rule Set: %v", err)
} }
server := build(waf.Params{ params.CoreRuleSet = coreRuleSet
Config: cfg, }
RequestLog: out,
ProcessLog: processLog, // addCounters adds to params the metrics, the table of clients, the ledger
GeoJSURL: geojsURL, // and the anomaly counters, as sww builds them from params' settings. The
AbuseIPDBURL: abuseIPDBURL, // metrics read the ledger, the table of clients and the rules.
LookupFile: lookupFile, func addCounters(params *waf.Params) {
CoreRuleSet: coreRuleSet, cfg := params.Config
Now: now,
Rules: ruleFiles, params.Metrics = metrics.New(
Alerts: alertQueue, cfg.MetricsBusiestCountriesAndASNs,
cfg.InstanceName,
)
params.Limiter = ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{
PerMinute: cfg.RateLimitPerMinute,
PerHour: cfg.RateLimitPerHour,
PerDay: cfg.RateLimitPerDay,
BytesPerMinute: cfg.BytesLimitPerMinute,
BytesPerHour: cfg.BytesLimitPerHour,
BytesPerDay: cfg.BytesLimitPerDay,
},
MaxClients: cfg.MaxTrackedClients,
})
params.Ledger = bans.New(bans.Params{
LimitBanDuration: cfg.LimitBanDuration,
LimitBanRepeatWindow: cfg.LimitBanRepeatWindow,
LimitBanPermanentAbove: cfg.LimitBanPermanentAbove,
AttackBanDuration: cfg.AttackBanDuration,
MaxBans: cfg.MaxBans,
})
params.Anomalies = anomaly.New(anomaly.Params{
Client: cfg.AnomalyClient,
Net: cfg.AnomalyNet,
ASN: cfg.AnomalyASN,
Total: cfg.AnomalyTotal,
WatchedNet: cfg.AnomalyWatchedNet,
NetV4Prefix: cfg.AnomalyNetV4Prefix,
NetV6Prefix: cfg.AnomalyNetV6Prefix,
NamedNetblocks: cfg.AnomalyWatchedNets,
Alerts: params.Alerts,
}) })
// The race detector and a busy host can slow the Core Rule Set's params.Metrics.AddBansAndClients(
// inspections past the time limit. The tests of the limit set it back, and params.Ledger,
// hold the Core Rule Set with a stand-in. params.Limiter,
server.SetBesideLookupTimeLimit(time.Hour) params.Now,
)
params.Metrics.AddRules(params.Rules)
}
return server, out, alertQueue // addOutsideSources adds to params the lists fetched from URLs, the DNSBL
// zones' verdicts, AbuseIPDB's scores from abuseIPDBCheckURL and its
// blacklist from abuseIPDBBlacklistURL, routewatch's answers while its
// setting is set, and GeoJS at geojsURL, as sww builds
// them from params' settings and the parts addCounters added. The metrics
// read the lists, the zones' verdicts, and AbuseIPDB's while its key is set.
func addOutsideSources(
params *waf.Params, geojsURL, abuseIPDBCheckURL string,
) {
cfg := params.Config
m, limiter, ledger := params.Metrics, params.Limiter, params.Ledger
processLog, alertQueue := params.ProcessLog, params.Alerts
params.Lists = reputation.New(reputation.Params{
BlocklistURLs: cfg.BlocklistURLs,
Refresh: cfg.BlocklistRefresh,
ASNLimitPercentURL: cfg.ASNLimitPercentURL,
TagLists: cfg.TagLists,
Now: params.Now,
ProcessLog: processLog,
Alerts: alertQueue,
})
params.DNSBL = reputation.NewDNSBL(reputation.DNSBLParams{
Zones: cfg.DNSBLZones,
Resolver: cfg.DNSBLResolver,
CacheTTL: cfg.IPReputationCacheTTL,
Timeout: cfg.DNSBLAbuseIPDBTimeout,
Now: params.Now,
ProcessLog: processLog,
Alerts: alertQueue,
})
params.AbuseIPDB = reputation.NewAbuseIPDB(reputation.AbuseIPDBParams{
URL: abuseIPDBCheckURL,
Key: cfg.AbuseIPDBKey,
MinScore: cfg.AbuseIPDBMinScore,
DailyBudget: cfg.AbuseIPDBDailyBudget,
CacheTTL: cfg.IPReputationCacheTTL,
Timeout: cfg.DNSBLAbuseIPDBTimeout,
BlacklistURL: abuseIPDBBlacklistURL,
Threshold: cfg.AbuseIPDBThreshold,
BlacklistRefresh: cfg.AbuseIPDBBlacklistRefresh,
Blacklisted: func(listed []reputation.BlacklistEntry) {
waf.BanAbuseIPDBBlacklist(
cfg, ledger, processLog, params.Now(), listed,
)
},
Now: params.Now,
ProcessLog: processLog,
Alerts: alertQueue,
})
if cfg.RouteWatchURL != "" {
params.RouteWatch = routewatch.New(routewatch.Params{
URL: cfg.RouteWatchURL,
Now: params.Now,
ProcessLog: processLog,
})
}
params.GeoJS = geojs.New(geojs.Params{
URL: geojsURL,
Answered: func(answer geojs.Answer) {
waf.AddLookup(cfg, limiter, ledger, answer)
},
Now: params.Now,
Asked: m.GeoJSRequests.Inc,
Failed: func(err error, askingAgainIn time.Duration) {
waf.GeoJSFailed(processLog, alertQueue, m, err, askingAgainIn)
},
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
})
m.AddReputation(params.Lists, params.DNSBL)
if cfg.AbuseIPDBKey != "" {
m.AddAbuseIPDB(params.AbuseIPDB)
}
} }
// newClient returns an HTTP client that sends requests as they are made, // newClient returns an HTTP client that sends requests as they are made,
+40 -22
View File
@@ -6,8 +6,10 @@ package waf
import ( import (
"context" "context"
"net/netip"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -18,56 +20,72 @@ import (
// refused, and bans the client. A request exempt from the rate limits is exempt // refused, and bans the client. A request exempt from the rate limits is exempt
// from the byte limits too. Its figures are the window, the count and the // from the byte limits too. Its figures are the window, the count and the
// limit. // limit.
type rateLimits struct{} type rateLimits struct {
limiter *ratelimit.Limiter
exemptNets []netip.Prefix // SWW_WAF_RATE_LIMIT_EXEMPT_NETS
exemptPaths []string // SWW_WAF_RATE_LIMIT_EXEMPT_PATHS
banResponse int // SWW_WAF_BAN_RESPONSE
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
biasedThresholds biasedThresholds
banMaker banMaker
}
func (rateLimits) name() string { return "rate_limits" } func (rateLimits) name() string { return "rate_limits" }
func (rateLimits) check( func (r rateLimits) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
now time.Time, now time.Time,
) *verdict { ) *verdict {
rq.counted = rq.countedByRateLimits() rq.counted = r.countedByRateLimits(rq)
if !rq.counted { if !rq.counted {
return nil return nil
} }
rq.rateLimitPercent, rq.bytesLimitPercent = rq.limitPercentages() rq.rateLimitPercent, rq.bytesLimitPercent =
r.biasedThresholds.limitPercentages(rq)
rq.line.RateLimitPercent, rq.line.RateLimitPercentSetting = rq.line.RateLimitPercent, rq.line.RateLimitPercentSetting =
rq.rateLimitPercent.logged() rq.rateLimitPercent.logged()
rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting = rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting =
rq.bytesLimitPercent.logged() rq.bytesLimitPercent.logged()
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now, counts, hit, over := r.limiter.Count(
rq.rateLimitPercent.percent) clientGroup(r.ipv6GroupPrefix, rq.client),
now,
rq.rateLimitPercent.percent,
)
rq.line.Counts = counts rq.line.Counts = counts
if !over { if !over {
return nil return nil
} }
rq.banForLimit(now, hit, rq.h.config.BanResponse) r.banMaker.banForLimit(rq, now, hit, r.banResponse)
return &verdict{ return &verdict{
action: requestlog.ActionClientIsRateLimited, figures: limitFigures(hit), action: requestlog.ActionClientIsRateLimited, figures: limitFigures(hit),
} }
} }
// overAWholeRateLimit reports whether the heuristic rate_limits will refuse the // refusesWhateverTheLookupAnswers reports whether the heuristic rate_limits
// request at now, whatever its client's limit percentages: the rate limits // will refuse rq at now, whatever its client's limit percentages, which the
// count it, and one more request from its client takes it over a whole rate // lookup's answer can lower: the rate limits count it, and one more request
// limit. It counts nothing. // from its client takes it over a whole rate limit. It counts nothing.
func (rq *request) overAWholeRateLimit(now time.Time) bool { func (r rateLimits) refusesWhateverTheLookupAnswers(
return rq.countedByRateLimits() && rq *request,
rq.h.limiter.OverARateLimit(rq.h.clientGroup(rq.client), now) now time.Time,
) bool {
return r.countedByRateLimits(rq) &&
r.limiter.OverARateLimit(
clientGroup(r.ipv6GroupPrefix, rq.client),
now,
)
} }
// countedByRateLimits reports whether the rate limits count the request: its // countedByRateLimits reports whether the rate limits count rq: its client is
// client is not in SWW_WAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt // not in SWW_WAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt under
// under SWW_WAF_RATE_LIMIT_EXEMPT_PATHS. // SWW_WAF_RATE_LIMIT_EXEMPT_PATHS.
func (rq *request) countedByRateLimits() bool { func (r rateLimits) countedByRateLimits(rq *request) bool {
cfg := rq.h.config return !isInside(rq.client, r.exemptNets) &&
!pathExempt(rq.in, r.exemptPaths)
return !isInside(rq.client, cfg.RateLimitExemptNets) &&
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
} }
+17 -8
View File
@@ -8,6 +8,7 @@ import (
"net/http" "net/http"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -18,17 +19,22 @@ import (
// a limit, which bans the client. A client in SWW_WAF_ALLOW_NETS, which the // a limit, which bans the client. A client in SWW_WAF_ALLOW_NETS, which the
// heuristics skip, is not counted, and nothing is while the limit is off. // heuristics skip, is not counted, and nothing is while the limit is off.
// Its figures are the window, the count and the limit. // Its figures are the window, the count and the limit.
type refusalLimit struct{} type refusalLimit struct {
limiter *ratelimit.Limiter
perMinute int64 // SWW_WAF_REFUSALS_LIMIT_PER_MINUTE
banResponse int // SWW_WAF_BAN_RESPONSE
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
banMaker banMaker
}
func (refusalLimit) name() string { return "refusal_limit" } func (refusalLimit) name() string { return "refusal_limit" }
func (refusalLimit) check( func (r refusalLimit) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
now time.Time, now time.Time,
) *verdict { ) *verdict {
cfg := rq.h.config if r.perMinute == 0 {
if cfg.RefusalsLimitPerMinute == 0 {
return nil return nil
} }
@@ -36,8 +42,11 @@ func (refusalLimit) check(
return nil return nil
} }
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now, hit, over := r.limiter.CountRefusal(
cfg.RefusalsLimitPerMinute) clientGroup(r.ipv6GroupPrefix, rq.client),
now,
r.perMinute,
)
if !over { if !over {
return nil return nil
} }
@@ -51,10 +60,10 @@ func (refusalLimit) check(
status = http.StatusForbidden status = http.StatusForbidden
case requestlog.ActionTemporarilyBanned, case requestlog.ActionTemporarilyBanned,
requestlog.ActionPermanentlyBanned: requestlog.ActionPermanentlyBanned:
status = cfg.BanResponse status = r.banResponse
} }
rq.banForLimit(now, hit, status) r.banMaker.banForLimit(rq, now, hit, status)
return &verdict{figures: limitFigures(hit)} return &verdict{figures: limitFigures(hit)}
} }
+20 -11
View File
@@ -5,26 +5,35 @@ package waf
import ( import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/metrics"
) )
// deny is the SWW_WAF_BLOCKLIST_ACTION and the SWW_WAF_DNSBL_ABUSEIPDB_ACTION that // reputationHitNoter notes the reputation sources that list a request's
// refuses the requests of a client a source lists. // client, for the reputation heuristics.
const deny = "deny" type reputationHitNoter struct {
metrics *metrics.Metrics
alerts *alerts.Queue
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
}
// noteListed notes each of sources, the URLs of the blocklists or the // noteListed notes each of sources, the URLs of the blocklists or the
// DNSBL zones, their keys masked, that list the client, as noteHit does, // DNSBL zones, their keys masked, that list rq's client, as noteHit does,
// with reason. // with reason.
func (rq *request) noteListed(sources []string, reason string) { func (n reputationHitNoter) noteListed(
rq *request, sources []string, reason string,
) {
for _, source := range sources { for _, source := range sources {
rq.noteHit(bans.ReputationHit{Source: source}, reason) n.noteHit(rq, bans.ReputationHit{Source: source}, reason)
} }
} }
// noteHit adds hit's source, which lists the client, to the log line's // noteHit adds hit's source, which lists rq's client, to the log line's
// reputation, and hit to the notes of a ban the request makes, counts the // reputation, and hit to the notes of a ban the request makes, counts the
// source in the metrics, and raises a reputation_hit alert with reason, // source in the metrics, and raises a reputation_hit alert with reason,
// whose detail gives hit's source and score. // whose detail gives hit's source and score.
func (rq *request) noteHit(hit bans.ReputationHit, reason string) { func (n reputationHitNoter) noteHit(
rq *request, hit bans.ReputationHit, reason string,
) {
detail := map[string]any{"source": hit.Source} detail := map[string]any{"source": hit.Source}
if hit.Score != nil { if hit.Score != nil {
detail["score"] = *hit.Score detail["score"] = *hit.Score
@@ -32,12 +41,12 @@ func (rq *request) noteHit(hit bans.ReputationHit, reason string) {
rq.line.Reputation = append(rq.line.Reputation, hit.Source) rq.line.Reputation = append(rq.line.Reputation, hit.Source)
rq.reputation = append(rq.reputation, hit) rq.reputation = append(rq.reputation, hit)
rq.h.metrics.ReputationHit(hit.Source) n.metrics.ReputationHit(hit.Source)
rq.h.alerts.Raise(alerts.Alert{ n.alerts.Raise(alerts.Alert{
Event: alerts.EventReputationHit, Event: alerts.EventReputationHit,
Host: rq.site, Host: rq.site,
Client: rq.client, Client: rq.client,
Netblock: rq.h.clientGroup(rq.client), Netblock: clientGroup(n.ipv6GroupPrefix, rq.client),
ASN: rq.line.ASN, ASN: rq.line.ASN,
ASName: rq.line.ASName, ASName: rq.line.ASName,
Country: rq.line.Country, Country: rq.line.Country,
+25 -8
View File
@@ -822,22 +822,39 @@ const abuseipdb = reputation.AbuseIPDBSource
// through which AbuseIPDB is asked. // through which AbuseIPDB is asked.
const abuseIPDBURL = "abuseipdb://stand-in/api/v2/check" const abuseIPDBURL = "abuseipdb://stand-in/api/v2/check"
// abuseIPDBBlacklistURL is where newWAF has AbuseIPDB's blacklist fetched
// from: at abuseIPDBStandIn too.
const abuseIPDBBlacklistURL = "abuseipdb://stand-in/api/v2/blacklist"
// abuseIPDBStandIn is a stand-in for AbuseIPDB that gives every client the // abuseIPDBStandIn is a stand-in for AbuseIPDB that gives every client the
// score 100, at once and without the network. // score 100, at once and without the network. Its blacklist lists client
// alone, with the score 100, to a fetch with accountKey that asks for the
// clients scored 90 or more, the default threshold; it refuses any other
// fetch of it with 400.
type abuseIPDBStandIn struct{} type abuseIPDBStandIn struct{}
// RoundTrip answers req with the score 100. // RoundTrip answers req with the score 100, or with the blacklist.
func (abuseIPDBStandIn) RoundTrip( func (abuseIPDBStandIn) RoundTrip(
req *http.Request, req *http.Request,
) (*http.Response, error) { ) (*http.Response, error) {
status, body := http.StatusOK, `{"data":{"abuseConfidenceScore":100}}`
if req.URL.Path == "/api/v2/blacklist" {
body = `{"data":[{"ipAddress":"` + client +
`","abuseConfidenceScore":100}]}`
if req.Header.Get("Key") != accountKey ||
req.URL.RawQuery != "confidenceMinimum=90" {
status, body = http.StatusBadRequest, `{"errors":[]}`
}
}
return &http.Response{ return &http.Response{
StatusCode: http.StatusOK, StatusCode: status,
Status: "200 OK", Status: fmt.Sprintf("%d %s", status, http.StatusText(status)),
Header: http.Header{}, Header: http.Header{},
Body: io.NopCloser( Body: io.NopCloser(strings.NewReader(body)),
strings.NewReader(`{"data":{"abuseConfidenceScore":100}}`), Request: req,
),
Request: req,
}, nil }, nil
} }
+25 -10
View File
@@ -122,8 +122,11 @@ func (h *handler) startRequest(
RequestID: requestID(r, isQuestion), RequestID: requestID(r, isQuestion),
PeerIP: peer.String(), PeerIP: peer.String(),
ForwardedFor: strings.Join(forwardedFor, ", "), ForwardedFor: strings.Join(forwardedFor, ", "),
ClientGroup: h.clientGroup(client).String(), ClientGroup: clientGroup(
ContentType: r.Header.Get("Content-Type"), h.config.IPv6GroupPrefix,
client,
).String(),
ContentType: r.Header.Get("Content-Type"),
RequestHeaders: requestHeaders( RequestHeaders: requestHeaders(
r, r,
h.config.LogRequestHeaders, h.config.LogRequestHeaders,
@@ -233,7 +236,7 @@ func (rq *request) checkClient(ctx context.Context) *verdict {
now := rq.h.now() now := rq.h.now()
v := rq.runHeuristics(ctx, beforeLookup(), now) v := rq.runHeuristics(ctx, rq.h.beforeLookup, now)
if v != nil { if v != nil {
return v return v
} }
@@ -265,7 +268,7 @@ func (rq *request) runAfterLookup(
// again: a request counted at the time it arrived could be counted // again: a request counted at the time it arrived could be counted
// after a later one of its client's, which the rate limits would take // after a later one of its client's, which the rate limits would take
// for the clock set back. // for the clock set back.
return rq.runHeuristics(ctx, afterLookup(), rq.h.now()) return rq.runHeuristics(ctx, rq.h.afterLookup, rq.h.now())
} }
// pathExempt reports whether r is exempt under prefixes, // pathExempt reports whether r is exempt under prefixes,
@@ -345,6 +348,16 @@ func timing(start, end time.Time) *float64 {
return new(requestlog.Milliseconds(end.Sub(start))) return new(requestlog.Milliseconds(end.Sub(start)))
} }
// banAndCount runs list, afterAnswer or onReport, whose heuristics only ban.
// It then counts the request for the anomaly thresholds and adds it to its
// client's history. list runs first, so that a refusal limit it finds broken
// is in that history. ctx is the request's own context.
func (rq *request) banAndCount(ctx context.Context, list []heuristic) {
rq.runBanningHeuristics(ctx, list)
rq.countAnomalies()
rq.addToHistory()
}
// addToHistory adds the request, which has ended, to its client's // addToHistory adds the request, which has ended, to its client's
// history, and counts its offences in the metrics, and then the lookup's // history, and counts its offences in the metrics, and then the lookup's
// answer about the client, as answerAtTheEnd gives it, to that history and // answer about the client, as answerAtTheEnd gives it, to that history and
@@ -365,7 +378,7 @@ func (rq *request) addToHistory() {
} }
rq.h.limiter.AddToHistory( rq.h.limiter.AddToHistory(
rq.h.clientGroup(rq.client), clientGroup(rq.h.config.IPv6GroupPrefix, rq.client),
rq.h.now(), rq.h.now(),
request, request,
) )
@@ -373,7 +386,7 @@ func (rq *request) addToHistory() {
answer, found := rq.answerAtTheEnd() answer, found := rq.answerAtTheEnd()
if found { if found {
rq.h.addLookup(answer) AddLookup(rq.h.config, rq.h.limiter, rq.h.ledger, answer)
} }
} }
@@ -394,11 +407,11 @@ func (rq *request) countAnomalies() {
rq.h.anomalies.Count(rq.h.now(), anomaly.Request{ rq.h.anomalies.Count(rq.h.now(), anomaly.Request{
Host: rq.site, Host: rq.site,
Client: rq.client, Client: rq.client,
ClientGroup: rq.h.clientGroup(rq.client), ClientGroup: clientGroup(rq.h.config.IPv6GroupPrefix, rq.client),
ASN: answer.ASN, ASN: answer.ASN,
ASName: answer.ASName, ASName: answer.ASName,
Country: answer.Country, Country: answer.Country,
Bytes: rq.countedBytes(), Bytes: rq.countedBytes(rq.h.config.BytesCounted),
}) })
} }
@@ -421,11 +434,13 @@ func (rq *request) answerAtTheEnd() (geojs.Answer, bool) {
return geojs.Answer{}, false return geojs.Answer{}, false
} }
if rq.h.config.LookupSource == "file" { if rq.h.config.LookupSource == config.LookupSourceFile {
return rq.lookupAnswer, true return rq.lookupAnswer, true
} }
return rq.h.geojs.Kept(rq.h.clientGroup(rq.client)) return rq.h.geojs.Kept(
clientGroup(rq.h.config.IPv6GroupPrefix, rq.client),
)
} }
// responseWriter is waf's answer to a request. It notes the status and // responseWriter is waf's answer to a request. It notes the status and
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -248,7 +248,7 @@ func TestTimingsAreInOrder(t *testing.T) {
wantStatus(t, do(t, req), http.StatusForbidden) wantStatus(t, do(t, req), http.StatusForbidden)
refused := out.endedLines(t, 2)[1] refused := out.endedLines(t, 2)[1]
wantStatus(t, get(t, server.Addr, waf.HealthPath), http.StatusOK) wantStatus(t, get(t, server.Addr, wafprotocol.HealthPath), http.StatusOK)
health := out.endedLines(t, 3)[2] health := out.endedLines(t, 3)[2]
// A request passed to the app has every timing; one refused, none of // A request passed to the app has every timing; one refused, none of
+10 -5
View File
@@ -8,6 +8,7 @@ import (
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/routewatch"
) )
// routeWatchTags adds to the request the tags asn=<number> and prefix=<netblock> // routeWatchTags adds to the request the tags asn=<number> and prefix=<netblock>
@@ -17,22 +18,26 @@ import (
// SWW_WAF_ROUTEWATCH_URL is unset it does nothing. Answers are kept by client and // SWW_WAF_ROUTEWATCH_URL is unset it does nothing. Answers are kept by client and
// asked for by the request's address, and a route without an origin AS gives only // asked for by the request's address, and a route without an origin AS gives only
// the prefix= tag. // the prefix= tag.
type routeWatchTags struct{} type routeWatchTags struct {
// routeWatch is nil while SWW_WAF_ROUTEWATCH_URL is unset.
routeWatch *routewatch.Client
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
}
func (routeWatchTags) name() string { return "routewatch_tags" } func (routeWatchTags) name() string { return "routewatch_tags" }
func (routeWatchTags) check( func (r routeWatchTags) check(
ctx context.Context, ctx context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
if rq.h.routeWatch == nil { if r.routeWatch == nil {
return nil return nil
} }
route, missing := rq.h.routeWatch.Route( route, missing := r.routeWatch.Route(
ctx, ctx,
rq.h.clientGroup(rq.client), clientGroup(r.ipv6GroupPrefix, rq.client),
rq.client, rq.client,
) )
if missing != "" { if missing != "" {
+10 -4
View File
@@ -9,6 +9,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
) )
@@ -22,11 +23,15 @@ import (
// mode raises the alert for the ban // mode raises the alert for the ban
// it would have made. Either rule's match is noted as an offence, for the // it would have made. Either rule's match is noted as an offence, for the
// client's history. Its figures are the rule's id and action. // client's history. Its figures are the rule's id and action.
type ruleFiles struct{} type ruleFiles struct {
rules *rules.Files
metrics *metrics.Metrics
banMaker banMaker
}
func (ruleFiles) name() string { return "rule_files" } func (ruleFiles) name() string { return "rule_files" }
func (ruleFiles) check( func (r ruleFiles) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
now time.Time, now time.Time,
@@ -35,7 +40,7 @@ func (ruleFiles) check(
for _, rule := range matched { for _, rule := range matched {
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID) rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
rq.h.metrics.RuleMatched(rule.ID, rule.Action) r.metrics.RuleMatched(rule.ID, rule.Action)
} }
if len(matched) == 0 { if len(matched) == 0 {
@@ -60,7 +65,8 @@ func (ruleFiles) check(
rq.attack = true rq.attack = true
return &verdict{ return &verdict{
action: rq.banForAttack( action: r.banMaker.banForAttack(
rq,
now, now,
bans.Notes{RuleID: last.ID, Target: last.Target}, bans.Notes{RuleID: last.ID, Target: last.Target},
), ),
+9 -3
View File
@@ -12,13 +12,19 @@ import (
// tagDeny refuses a request that carries a tag in SWW_WAF_TAG_DENY. Its figure is // tagDeny refuses a request that carries a tag in SWW_WAF_TAG_DENY. Its figure is
// the first such tag, in sorted order. // the first such tag, in sorted order.
type tagDeny struct{} type tagDeny struct {
tags []string // SWW_WAF_TAG_DENY
}
func (tagDeny) name() string { return "tag_deny" } func (tagDeny) name() string { return "tag_deny" }
func (tagDeny) check(_ context.Context, rq *request, _ time.Time) *verdict { func (t tagDeny) check(
_ context.Context,
rq *request,
_ time.Time,
) *verdict {
for _, tag := range rq.line.Tags { for _, tag := range rq.line.Tags {
if slices.Contains(rq.h.config.TagDeny, tag) { if slices.Contains(t.tags, tag) {
return &verdict{ return &verdict{
action: requestlog.ActionClientDenied, action: requestlog.ActionClientDenied,
figures: map[string]any{"tag": tag}, figures: map[string]any{"tag": tag},
Loaded 100 of 111 files, more files were not shown because too many files have changed in this diff. Show more