Compare commits

..
1 Commits
Author SHA1 Message Date
clawbot fc469d96a9 Leave SWWAF_RATE_LIMIT_EXEMPT_PATHS out of the request rate limits (closes #77)
check / check (push) Successful in 3m53s
A request is neither counted nor refused by the request rate limits
when its path, percent-decoded, starts with one of the comma-separated
prefixes in SWWAF_RATE_LIMIT_EXEMPT_PATHS. A request whose decoded path
contains .. or a backslash, or whose path as sent holds an encoded
slash, is never exempt, since an app may act on it as a path outside
every prefix, such as /assets/..%2Flogin as /login. The static lists,
bans and the country lists still apply. The setting is empty by
default, and a prefix that does not start with / stops the start.
README.md documents it.

Model: opus-5-5
2026-10-06 12:37:15 +00:00
44 changed files with 488 additions and 6049 deletions
-4
View File
@@ -167,10 +167,6 @@ RUN groupadd --system --gid 65532 smallwebwaf \
# smallwebwaf user at each start. # smallwebwaf user at each start.
RUN mkdir /var/lib/smallwebwaf RUN mkdir /var/lib/smallwebwaf
# The default rule file, in SWWAF_RULES_DIR by default, where an app's
# Dockerfile can copy rule files of its own beside it.
COPY share/rules.d/00-default.rules /etc/smallwebwaf/rules.d/00-default.rules
# runsvinit starts runit's runsvdir on /etc/service, where Ubuntu's sv # runsvinit starts runit's runsvdir on /etc/service, where Ubuntu's sv
# looks too. # looks too.
COPY --chmod=755 share/smallwebwaf.run /etc/service/smallwebwaf/run COPY --chmod=755 share/smallwebwaf.run /etc/service/smallwebwaf/run
+135 -426
View File
@@ -13,30 +13,24 @@ JSON log line for every request.
Status: the first two milestones are built Status: the first two milestones are built
(https://git.eeqj.de/sneak/smallwebwaf/issues/13 and (https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are nine parts of https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are seven parts of
milestone 3: the static lists, the bans that broken rate limits lead to, the ban milestone 3: the static lists, the bans that broken rate limits lead to, the
ledger with the bans you make, keep and lift, the JSON state files with your JSON state files with your edits taken in while it runs and the paths the rate
edits taken in while it runs and the paths the rate limits do not count, which limits do not count, which come next in the build order, `observe` mode, which
come next in the build order, `observe` mode and the rest of the request log's comes a little later, and the metrics endpoint and the header size and the idle
fields, which come a little later, and the metrics endpoint and the header size time as settings, which come last in it. `smallwebwaf` passes each request to
and the idle time as settings, which come last in it. So are two parts of the the app and the app's answer back, unchanged, within its timeouts and size
stage after it: the rule files, the first part, with the bans for a clear sign limits, works out each client's address, bans a client that sends too many
of attack, and remote log sending. `smallwebwaf` passes each request to the app requests, not counting those for the paths you choose, refuses a client that
and the app's answer back, unchanged, within its timeouts and size limits, works comes from a country you refuse or from a network you refuse, lets the networks
out each client's address, bans a client that sends too many requests, not you choose through, keeps its bans, each client's counters and history, and
counting those for the paths you choose, refuses a client that comes from a GeoJS's answers in JSON files across restarts, takes in your edits of those
country you refuse or from a network you refuse, lets the networks you choose files while it runs, writes a JSON log line for every request, serves Prometheus
through, checks each request against the rule files and bans a client whose metrics to a scraper that holds the metrics token, and in `observe` mode passes
request is a clear sign of attack, keeps its bans, each client's counters and on the requests it would refuse, logging what it would have done with them. It
history, and GeoJS's answers in JSON files across restarts, takes in your edits comes as the image the app's own image is built on. The rest of the design comes
of those files, such as a ban you make, keep or lift, and of the rule files after that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey
while it runs, writes a JSON log line for every request, sends its log lines to of existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
a syslog server too if you name one, serves Prometheus metrics to a scraper that
holds the metrics token, and in `observe` mode passes on the requests it would
refuse, logging what it would have done with them. It comes as the image the
app's own image is built on. The rest of the design comes after that, in the
order of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools
that led to the design is in [`EVALUATION.md`](EVALUATION.md).
## Getting started ## Getting started
@@ -59,8 +53,7 @@ works.
To work on the code, `make build` builds the binary alone, with Go installed, To work on the code, `make build` builds the binary alone, with Go installed,
and `make run` builds and runs it, listening on port 8080 in front of an app at and `make run` builds and runs it, listening on port 8080 in front of an app at
`SWWAF_UPSTREAM_URL`, by default `http://127.0.0.1:8081`, with its state files `SWWAF_UPSTREAM_URL`, by default `http://127.0.0.1:8081`, with its state files
in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of in `bin/state` unless `SWWAF_STATE_DIR` is set.
`share/rules.d` unless `SWWAF_RULES_DIR` is set.
## What it does so far ## What it does so far
@@ -74,9 +67,7 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
address outside `SWWAF_TRUSTED_PROXIES` is the client; if every address in it address outside `SWWAF_TRUSTED_PROXIES` is the client; if every address in it
is inside, the leftmost is, and with no header the peer is. The app sees what is inside, the leftmost is, and with no header the peer is. The app sees what
it would see from traefik directly: the same `Host`, the same it would see from traefik directly: the same `Host`, the same
`X-Forwarded-Proto`, and `X-Forwarded-For` with the peer added at the end. It `X-Forwarded-Proto`, and `X-Forwarded-For` with the peer added at the end.
also gets the request's id in `X-Request-ID`, the same id as in the request's
log line (see `request_id` in "Request log" below).
- Enforces the timeouts and the size limits below. A limit passed before the - Enforces the timeouts and the size limits below. A limit passed before the
response has started gets `smallwebwaf`'s own answer: `408` for a client too response has started gets `smallwebwaf`'s own answer: `408` for a client too
slow to send its request, `413` for a request body that is too large, `504` slow to send its request, `413` for a request body that is too large, `504`
@@ -92,13 +83,13 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
`SWWAF_BAN_RESPONSE`, `403` by default, before anything reaches the app, and `SWWAF_BAN_RESPONSE`, `403` by default, before anything reaches the app, and
bans the client. A request whose path starts with one of bans the client. A request whose path starts with one of
`SWWAF_RATE_LIMIT_EXEMPT_PATHS`, as that setting below describes, is neither `SWWAF_RATE_LIMIT_EXEMPT_PATHS`, as that setting below describes, is neither
counted nor refused by the rate limits; the static lists, bans, the country counted nor refused by the rate limits; the static lists, bans and the country
lists and the rule files still apply to it. A client is one IPv4 address, or lists still apply to it. A client is one IPv4 address, or one IPv6 /64, since
one IPv6 /64, since one abuser usually holds a whole /64. Each window is one abuser usually holds a whole /64. Each window is counted in two fixed
counted in two fixed buckets, the earlier one weighted by how much of it the buckets, the earlier one weighted by how much of it the window still covers.
window still covers. At most 20,000 clients are kept, the least recently seen At most 20,000 clients are kept, the least recently seen dropped first, with
dropped first, with their history, and a restart gives no client a fresh their history, and a restart gives no client a fresh allowance (see "State
allowance (see "State files" below). files" below).
- Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md) - Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md)
describes: the first ban lasts an hour, and a limit broken again within a day describes: the first ban lasts an hour, and a limit broken again within a day
of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and
@@ -112,28 +103,11 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
window and the requests counted in it, the request that broke it, the client's window and the requests counted in it, the request that broke it, the client's
country when it was looked up, the netblock's requests since it was first country when it was looked up, the netblock's requests since it was first
seen, how many of them the ban has refused, and how many bans the netblock had seen, how many of them the ban has refused, and how many bans the netblock had
before, for a broken limit, for a clear sign of attack and by an admin. At before. At most `SWWAF_MAX_BANS` bans are kept, past, active and permanent;
most `SWWAF_MAX_BANS` bans `smallwebwaf` made are kept, past, active and past that, the earliest ban of the netblock that has gone longest without a
permanent; past that, the earliest such ban of the netblock that has gone request is dropped first. `bans.json` shows the bans and their notes, a
longest without a request is dropped first. The bans whose cause is `admin`, restart lifts none, and you add or lift a ban by editing it (see "State files"
those you make or keep, are kept besides, and never dropped. `bans.json` shows below).
the bans and their notes, a restart lifts none, and you make, keep or lift a
ban by editing it (see "State files" below).
- Checks each request against the rules of the rule files (see "Rule files"
below) after the rate limits, and before its body is read. A `log` rule that
matches is noted in the log line; a `block` rule refuses the request with
`403`, and bans no one; a `ban` rule refuses it with `SWWAF_BAN_RESPONSE` and
bans the client's netblock for a clear sign of attack. Matching stops at the
first rule that refuses. A client in `SWWAF_ALLOW_NETS` is not checked.
- Bans a client for a clear sign of attack, as "Bans" in [`SPEC.md`](SPEC.md)
describes: the first such ban lasts `SWWAF_ATTACK_BAN_DURATION`, seven days by
default, and any request from the netblock while it lasts makes it permanent.
Once it has run out, the netblock is served like any other, but its next clear
sign of attack bans it permanently at once. Such a ban covers the same
netblock as a ban for a broken rate limit, does not set the client's counters
back to zero, and does not make the netblock's next ban for a broken limit
longer. Its notes give the id and the target of the rule that matched in place
of the limit.
- Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon - Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
as the client's country is known and before its body is read; such a request as the client's country is known and before its body is read; such a request
is not counted for the rate limits. While one of the country lists below is is not counted for the rate limits. While one of the country lists below is
@@ -144,26 +118,25 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
in `SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it. in `SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it.
- Checks the client's own address against the static lists, the three netblock - Checks the client's own address against the static lists, the three netblock
settings below, before anything else, its country included. A client in settings below, before anything else, its country included. A client in
`SWWAF_ALLOW_NETS` skips bans, the country lists, the rate limits and the rule `SWWAF_ALLOW_NETS` skips bans, the country lists and the rate limits, and is
files, and is not looked up; the timeouts and size limits still apply. A not looked up; the timeouts and size limits still apply. A client in
client in `SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE` before its `SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE` before its body is
body is read, and the request is not counted for the rate limits; an address read, and the request is not counted for the rate limits; an address in
in `SWWAF_ALLOW_NETS` too is let through. A client in `SWWAF_ALLOW_NETS` too is let through. A client in
`SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
limits; the country lists, the rule files and bans still apply to it. limits; the country lists and bans still apply to it.
- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests - In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
that `SWWAF_DENY_NETS`, a ban, the country lists, a rate limit or a rule would that `SWWAF_DENY_NETS`, a ban, the country lists or a rate limit would refuse:
refuse: it passes them to the app, and their log lines name what `enforce` it passes them to the app, and their log lines name what `enforce` mode would
mode would have done (see `would_action` in "Request log" below). The checks have done (see `would_action` in "Request log" below). The checks run, and
run, and requests are counted, as in `enforce` mode, with three differences: requests are counted, as in `enforce` mode, but a broken rate limit makes no
neither a broken rate limit nor a `ban` rule makes a ban; a broken rate limit ban and does not set the client's counters back to zero, so each request over
does not set the client's counters back to zero, so each request over the the limit is logged as one that would be refused. The bans in `bans.json` are
limit is logged as one that would be refused; and a request under a ban does kept, and refuse requests again when `smallwebwaf` next runs in `enforce`
not make it permanent. The bans in `bans.json` are kept, and refuse requests mode, as long as they last. The timeouts and size limits still apply, since
again when `smallwebwaf` next runs in `enforce` mode, as long as they last. they protect `smallwebwaf` and the app themselves, and a request for the
The timeouts and size limits still apply, since they protect `smallwebwaf` and metrics without the token is still answered `401`. It is for trying a
the app themselves, and a request for the metrics without the token is still configuration before enforcing it.
answered `401`. It is for trying a configuration before enforcing it.
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any - Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
check and without asking the app, for the image's health check. check and without asking the app, for the image's health check.
- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for - Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
@@ -175,25 +148,16 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
passed to the app: a banned client stays refused, and each counts toward the passed to the app: a banned client stays refused, and each counts toward the
client's rate limits. None of them reaches the app. client's rate limits. None of them reaches the app.
- Writes a line in the request log for each request (see "Request log" below). - Writes a line in the request log for each request (see "Request log" below).
- Sends every line it writes on stdout to a syslog server as well, while
`SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
below).
## Settings ## Settings
Each setting is an environment variable, or a file one names (see "Settings Each setting is an environment variable, and each has a default, so none has to
given as files" below), and each has a default, so none has to be set. A setting be set. A setting that is set but invalid stops the start with a message naming
that is set but invalid stops the start with a message naming it, and the it, and the effective settings are logged at start.
effective settings are logged at start.
- `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens. - `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens.
- `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app, as `http` or - `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app, as `http` or
`https`, a host and an optional port, and nothing more. `https`, a host and an optional port, and nothing more.
- `SWWAF_INSTANCE_NAME` (default: the host's name, which docker sets to the
first 12 characters of the container's id unless the deployment names one):
the name each request log line gives as `instance`. Set it, for example to
`fsn1app1/gitea`, for a name that stays the same when a deploy replaces the
container, and that tells instances apart when several log to one place.
- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to pass on the - `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to pass on the
requests `smallwebwaf` would refuse and log what it would have done (see "What requests `smallwebwaf` would refuse and log what it would have done (see "What
it does so far" above). it does so far" above).
@@ -222,8 +186,8 @@ effective settings are logged at start.
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. - `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. - `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the - `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
country lists, the rate limits and the rule files, such as your monitoring or country lists and the rate limits, such as your monitoring or your own
your own networks. networks.
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the - `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
rate limits do not apply to, such as a machine that talks to the app all day. rate limits do not apply to, such as a machine that talks to the app all day.
- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused. - `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
@@ -238,13 +202,11 @@ effective settings are logged at start.
`..` anywhere or a backslash, or whose path as sent holds an encoded slash `..` anywhere or a backslash, or whose path as sent holds an encoded slash
(`%2F` or `%2f`), is never exempt, since the app may act on it as a path (`%2F` or `%2f`), is never exempt, since the app may act on it as a path
outside every prefix: `/assets/..%2Flogin` as `/login`. Any other request is outside every prefix: `/assets/..%2Flogin` as `/login`. Any other request is
exempt when its path as sent, the path the app receives, before any query exempt when its path, percent-decoded and before any query string, starts with
string and not percent-decoded, starts with a prefix, character for character. a prefix, character for character. `/assets/` matches `/assets/app.js` and
`/assets/` matches `/assets/app.js` and `/assets/`, but not `/assets`, `/assets/`, but not `/assets`, `/Assets/app.js`, `/static/assets/app.js`,
`/Assets/app.js`, `/%61ssets/app.js`, `/static/assets/app.js`, `/static/../assets/app.js` or `/assets%2Fapp.js`. A prefix is written without
`/static/../assets/app.js` or `/assets%2Fapp.js`. A character the client sends percent-encoding, and there are no wildcards: `*` is a character like any
percent-encoded, such as a space, is written percent-encoded in a prefix, as
in `/my%20files/`, and there are no wildcards: `*` is a character like any
other. other.
- `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused, - `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused,
for example `cn,ru,kp`. for example `cn,ru,kp`.
@@ -253,22 +215,18 @@ effective settings are logged at start.
country cannot be found is refused too, so that new clients are not let in country cannot be found is refused too, so that new clients are not let in
whenever GeoJS stops answering. whenever GeoJS stops answering.
- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one - `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one
that is banned, breaks a rate limit, matches a `ban` rule, is in that is banned, breaks a rate limit, is in `SWWAF_DENY_NETS` or comes from a
`SWWAF_DENY_NETS` or comes from a refused country: `403`, `429`, or `close` to refused country: `403`, `429`, or `close` to close the connection without an
close the connection without an answer. Behind traefik, `close` does not leave answer. Behind traefik, `close` does not leave the client unanswered: traefik
the client unanswered: traefik answers `502`, as it does whenever its backend answers `502`, as it does whenever its backend drops a connection.
drops a connection. A `block` rule always answers `403`.
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate - `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
limit. limit.
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit broken again - `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit broken again
within this time after a ban ended, other than one for a clear sign of attack, within this time after a ban ended bans for three times as long as that ban.
bans for three times as long as that ban. - `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban that would be longer is
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken rate limit that permanent instead.
would be longer is permanent instead. - `SWWAF_MAX_BANS` (default `5000`): the most bans kept, past, active and
- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of permanent.
attack.
- `SWWAF_MAX_BANS` (default `5000`): the most bans `smallwebwaf` made that are
kept, past, active and permanent. The bans you make or keep are kept besides.
- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around - `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around
an IPv4 client that a ban covers, such as `24` to ban the surrounding /24. An an IPv4 client that a ban covers, such as `24` to ban the surrounding /24. An
IPv6 ban covers the client's /64. IPv6 ban covers the client's /64.
@@ -279,41 +237,12 @@ effective settings are logged at start.
`bans.json` is written, with every ban made in between. `bans.json` is written, with every ban made in between.
- `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is - `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is
written. written.
- `SWWAF_LOG_REQUEST_HEADERS` (default
`accept,accept-language,accept-encoding,content-type,origin,range`): the
request headers whose values the request log gives, in either case.
`Authorization`, `Cookie` and `Set-Cookie` are never logged, even when listed
(see "Request log" below). An entry naming `Host` or `Transfer-Encoding` stops
the start, since Go's HTTP server takes both out of the request; the request's
host is the field `host`.
- `SWWAF_METRICS_TOKEN` (default unset): the token a scraper sends for the - `SWWAF_METRICS_TOKEN` (default unset): the token a scraper sends for the
metrics, a long random value. While it is unset the metrics are off; one metrics, a long random value. While it is unset the metrics are off; one
shorter than 32 characters stops the start. The settings logged at start show shorter than 32 characters stops the start. The settings logged at start show
`********` in its place. Given as a file, it can be kept out of the app's `********` in its place.
reach (see "Settings given as files" below).
- `SWWAF_METRICS_TOP_N` (default `50`): how many countries get series of their - `SWWAF_METRICS_TOP_N` (default `50`): how many countries get series of their
own in the metrics by country; the others are counted as `other`. own in the metrics by country; the others are counted as `other`.
- `SWWAF_RULES_DIR` (default `/etc/smallwebwaf/rules.d`): the directory of the
rule files. A directory that does not exist stops the start.
- `SWWAF_RULES_ENABLED` (default `true`): `false` reads no rule file, and checks
no request against one.
- `SWWAF_LOG_REMOTE_URL` (default unset): a syslog server that every line on
stdout is also sent to, as `syslog+udp://`, `syslog+tcp://` or `syslog+tls://`
with a host and a port, such as `syslog+tls://logs.example:6514`. Unset or
empty, nothing is sent.
- `SWWAF_LOG_REMOTE_TLS_CA_FILE` (default unset): a file of PEM certificates,
which the certificate of a `syslog+tls` server must chain to instead of the
host's own. A file that cannot be read or holds no certificate stops the
start.
- `SWWAF_LOG_REMOTE_BUFFER` (default `10000`): the most lines held while they
wait to be sent.
- `SWWAF_LOG_REMOTE_FACILITY` (default `local0`): the syslog facility the lines
are sent with: `kern`, `user`, `mail`, `daemon`, `auth`, `syslog`, `lpr`,
`news`, `uucp`, `cron`, `authpriv`, `ftp`, or `local0` to `local7`.
- `SWWAF_LOG_REMOTE_APP_NAME` (default `SWWAF_INSTANCE_NAME`): the app name the
lines are sent with, 1 to 48 printable ASCII characters without a space. While
`SWWAF_LOG_REMOTE_URL` is set, an `SWWAF_INSTANCE_NAME` that is not such a
name stops the start too, unless this setting gives one that is.
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024 bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
@@ -323,116 +252,47 @@ ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
`su`, stops the start, and so does a code on both country lists. `off` switches `su`, stops the start, and so does a code on both country lists. `off` switches
a timeout, a size limit or a rate limit off; a timeout, a size limit or a rate limit off;
`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings, the state settings, `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings, the state settings
`SWWAF_METRICS_TOP_N` and `SWWAF_LOG_REMOTE_BUFFER` cannot be off. and `SWWAF_METRICS_TOP_N` cannot be off.
Several limits are fixed rather than settings. At most 20,000 clients are kept, Several limits are fixed rather than settings. At most 20,000 clients are kept,
with their counters and history, and an IPv6 client is counted by its /64. A new with their counters and history, and an IPv6 client is counted by its /64. A new
client waits at most a second for its country, and at most 100,000 answers from client waits at most a second for its country, and at most 100,000 answers from
GeoJS are kept, for 7 days each. GeoJS are kept, for 7 days each.
### Settings given as files
Any setting may instead be given as a file that holds its value: the variable
named like the setting with `_FILE` added, such as `SWWAF_METRICS_TOKEN_FILE`,
names the file. `smallwebwaf` reads the file once, at start, and its health
check reads only the files of `SWWAF_LISTEN_ADDR` and `SWWAF_UPSTREAM_URL`, each
time it runs. The file's contents are the value, less one newline at their end
so that a file written with `echo` or an editor works, and are checked as the
setting's own value would be. Setting both the setting and its `_FILE` form, or
naming a file that cannot be read, stops the start with a message naming the
variable. The settings logged at start name the file, and show a token given in
one as `********`, as they show one given directly.
`SWWAF_LOG_REMOTE_TLS_CA_FILE`, whose value names a file already, has no `_FILE`
form.
The app starts with the same environment variables as `smallwebwaf`, so it can
read a token given as one. A token given as a file is out of the app's reach
only while the `smallwebwaf` user alone can read the file: make it on the host,
owned by uid 65532, the `smallwebwaf` user, with mode `0400`, and mount the
directory that holds it into the container read-only; the container sees the
same owner and mode. For example, on the host:
```sh
mkdir -p /srv/app/tokens
openssl rand -hex 32 > /srv/app/tokens/metrics
chown 65532:65532 /srv/app/tokens/metrics
chmod 0400 /srv/app/tokens/metrics
```
and for the container, `-v /srv/app/tokens:/etc/smallwebwaf/tokens:ro` and
`-e SWWAF_METRICS_TOKEN_FILE=/etc/smallwebwaf/tokens/metrics`.
## Request log ## Request log
`smallwebwaf` writes one JSON object per line on stdout for every request, `smallwebwaf` writes one JSON object per line on stdout for every request,
refused ones included: refused ones included:
``` ```
{"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104} {"type":"request","time":"2026-10-03T12:00:00.123Z","client_ip":"203.0.113.9","peer_ip":"172.18.0.2","country":"DE","method":"GET","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"upstream_status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","action":"forward","duration_total":3.217,"duration_upstream_total":3.104}
``` ```
A field that does not apply to a request is left out of its line, apart from - `time` is when the request arrived, in UTC. `peer_ip` is the TCP peer,
`type`, the fields from `time` to `user_agent`, `request_id`, `peer_ip`, normally traefik. `path` and `query` are as the client sent them.
`client_group`, `country`, `action` and `duration_total`, which every line has.
- `time` is when the request arrived, in UTC. `instance` is
`SWWAF_INSTANCE_NAME`. `scheme` is the `X-Forwarded-Proto` a trusted proxy
sent, and otherwise `http`. `path` and `query` are as the client sent them.
- `request_id` is the `X-Request-ID` a trusted proxy sent, or a new random one
of 26 letters and digits when it sent none, or when the peer is not a trusted
proxy. A request passed to the app takes it there in `X-Request-ID`.
- `peer_ip` is the TCP peer, normally traefik. `forwarded_for` is the
`X-Forwarded-For` header as received, several lines of it joined with `, `.
`client_group` is the client as the rate limits count it: its IPv4 address as
a /32, or the /64 of its IPv6 address.
- `country` is the client's country as GeoJS places it. It is empty with neither - `country` is the client's country as GeoJS places it. It is empty with neither
country list set, for a client in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, for country list set, for a client in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, for
a client on a private, loopback or link-local address, when GeoJS cannot place a client on a private, loopback or link-local address, when GeoJS cannot place
the client or has not answered in time, and for a request whose client a ban the client or has not answered in time, and for a request whose client a ban
covers, even when the client's country is known. covers, even when the client's country is known.
- `content_type` is the request's `Content-Type`, and `content_length` the
length the request announced for its body, which is left out for none or zero.
- `request_headers` are the request's headers that `SWWAF_LOG_REQUEST_HEADERS`
names, by name in lower case, several lines of one joined with `, `.
`Authorization`, `Cookie` and `Set-Cookie` are never among them, whatever the
setting says: `has_authorization` and `has_cookie` are there instead, and
true, when the request has an `Authorization` or a `Cookie` header.
- `websocket` is there, and true, when the app switched the connection to
another protocol, as it does for a WebSocket.
- `status` is what the client was sent, `0` if nothing was; `upstream_status` is - `status` is what the client was sent, `0` if nothing was; `upstream_status` is
what the app answered, and is left out when the app did not answer. what the app answered, and is left out when the app did not answer.
- `response_content_type`, `cache_control` and `location` are the
`Content-Type`, `Cache-Control` and `Location` headers of the answer: the
app's, as passed on, or those of `smallwebwaf`'s own answer.
- `request_bytes` and `response_bytes` count body bytes. - `request_bytes` and `response_bytes` count body bytes.
- `action` is `forward` for a request passed to the app, `denied` for one - `action` is `forward` for a request passed to the app, `denied` for one
refused because its client is in `SWWAF_DENY_NETS`, `banned` for one refused refused because its client is in `SWWAF_DENY_NETS`, `banned` for one refused
because a ban covers its client or because it matched a `ban` rule, which bans because a ban covers its client, `country_denied` for one refused for its
its client, `country_denied` for one refused for its client's country, client's country, `rate_limited` for one that broke a rate limit and banned
`rate_limited` for one that broke a rate limit and banned its client, its client, `too_large` for a request or response over its size limit,
`rule_blocked` for one a `block` rule refused, `too_large` for a request or `timed_out` for one that ran out of time, `upstream_error` when the app could
response over its size limit, `timed_out` for one that ran out of time, not be reached or its answer broke off, and `admin` for one `smallwebwaf`
`upstream_error` when the app could not be reached or its answer broke off, answered at its own endpoint.
and `admin` for one `smallwebwaf` answered at its own endpoint.
- `would_action` is there in `observe` mode for a request that - `would_action` is there in `observe` mode for a request that
`SWWAF_DENY_NETS`, a ban, the country lists, a rate limit or a rule would have `SWWAF_DENY_NETS`, a ban, the country lists or a rate limit would have refused
refused in `enforce` mode, and names the action that refusal would have had: in `enforce` mode, and names the action that refusal would have had: `denied`,
`denied`, `banned`, `country_denied`, `rate_limited` or `rule_blocked`. `banned`, `country_denied` or `rate_limited`. `action` then names what was
`action` then names what was done: `forward` for a request passed to the app, done: `forward` for a request passed to the app, and another action, such as
and another action, such as `too_large`, for one a size or time limit refused. `too_large`, for one a size or time limit refused.
- `counts` gives the client's requests in the minute, the hour and the day as
the rate limits count them, this request included: in each window, those in
the bucket under way and a share of those in the bucket before, so a count can
have a fraction. For a request that broke a limit, they are the counts that
broke it. It is left out for a request the rate limits do not count: the
health check, one from a client in `SWWAF_ALLOW_NETS` or
`SWWAF_RATE_LIMIT_EXEMPT_NETS`, one for a path that
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a ban
or the country lists refuse, or would refuse in `observe` mode. The byte
totals come with the byte limits.
- `rule_ids` is there for a request that matched rules of the rule files, and
lists their ids in the order they matched, up to the one that refused it.
- `limit_hit` is there for a request that broke a rate limit, and names the - `limit_hit` is there for a request that broke a rate limit, and names the
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
went over several. `offence` is then `limit`. went over several. `offence` is then `limit`.
@@ -440,19 +300,10 @@ A field that does not apply to a request is left out of its line, apart from
or in `observe` mode would have been refused under one, and gives when the ban or in `observe` mode would have been refused under one, and gives when the ban
ends, in the same form as `time`, or `permanent`. ends, in the same form as `time`, or `permanent`.
- `aborted` is there, and true, when the client went away early. - `aborted` is there, and true, when the client went away early.
- The timings are in milliseconds, to the microsecond. `duration_total` runs - `duration_total` and `duration_upstream_total` are in milliseconds.
from when the request's headers had been read to when its line is written, and
`duration_checks` over the same start to when the checks were done; the health
check runs none, and its line has no `duration_checks`.
`duration_upstream_connect`, `duration_upstream_first_byte` and
`duration_upstream_total` are there for a request passed to the app, and run
from when it was handed to the app: until there was a connection to it, new or
kept open from an earlier request, until the first byte of its answer arrived,
and until the end. The first two are left out when that never happened, as for
an app that cannot be reached.
No body is logged, and no header but those above. `smallwebwaf`'s own messages No body and no other header is logged. `smallwebwaf`'s own messages (start, the
(start, the settings, stop, errors) share the stream as JSON lines marked settings, stop, errors) share the stream as JSON lines marked
`"type":"process"`. `"type":"process"`.
Go's HTTP server, on which `smallwebwaf` is built, reads a request's line and Go's HTTP server, on which `smallwebwaf` is built, reads a request's line and
@@ -462,32 +313,6 @@ which it answers `431`, headers slower than `SWWAF_CLIENT_REQUEST_TIMEOUT`,
whose connection it closes without an answer, and requests it cannot read at whose connection it closes without an answer, and requests it cannot read at
all, which it answers itself, mostly with `400`. all, which it answers itself, mostly with `400`.
### Sending the log to a syslog server
While `SWWAF_LOG_REMOTE_URL` is set, every line `smallwebwaf` writes on stdout,
request lines and its own, is also sent to that syslog server, as the message of
an RFC 5424 record: one record to a datagram over UDP, and over TCP and TLS each
record after its length in bytes and a space. A record gives the facility
`SWWAF_LOG_REMOTE_FACILITY` names, the severity informational, the time the line
was written, in the same form as a request line's `time`, the host's name, and
the app name `SWWAF_LOG_REMOTE_APP_NAME` gives. stdout is unchanged.
The lines wait in a buffer of `SWWAF_LOG_REMOTE_BUFFER` lines and are sent from
there, so a server that is slow or cannot be reached never holds up a request or
stdout. When the buffer is full, its oldest line is dropped to make room. A line
whose sending fails is dropped too, and the connection closed. That failure,
like a failed attempt to connect, is logged and followed by the next attempt to
connect a second later, twice as long after each further failure up to a minute,
and a second again after a connection that stayed up for a minute before it
failed. A line too long for one UDP datagram is dropped alone, with no wait and
nothing logged. UDP gives no sign of what arrives, and over TCP and TLS a line
sent on a connection the server has just closed can be lost before a failure
shows; such a loss is not counted.
As `smallwebwaf` stops, it sends the lines still waiting, on the connection open
or a new one, for at most two seconds, and gives up the rest; stdout has carried
them.
## State files ## State files
`smallwebwaf` keeps its state in memory and a copy of it in three JSON files in `smallwebwaf` keeps its state in memory and a copy of it in three JSON files in
@@ -495,14 +320,8 @@ them.
[`SPEC.md`](SPEC.md) describes. Each has a top-level `version`, 1, and lists its [`SPEC.md`](SPEC.md) describes. Each has a top-level `version`, 1, and lists its
entries by client address, with times in UTC. entries by client address, with times in UTC.
- `bans.json`: every ban with its notes, indented to be read. A permanent ban's - `bans.json`: every ban with its notes, indented to be read; a permanent ban's
`expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or `expires` is `null`.
`attack` for a clear sign of attack, for a ban `smallwebwaf` made, and `admin`
for one you made or keep. Its `reason` is a short text: for a ban
`smallwebwaf` made, the limit broken, such as
`requests per minute over the limit of 1000`, or the rule that matched, such
as `matched the rule env-file`; for yours, what you wrote. Its `lifted` is
when you lifted it, and is left out until you do.
- `clients.json`: each client's two buckets in the minute, the hour and the day, - `clients.json`: each client's two buckets in the minute, the hour and the day,
and its history: when it was first and last seen, its country as last looked and its history: when it was first and last seen, its country as last looked
up and when, its requests, how many were forwarded and how many refused (one up and when, its requests, how many were forwarded and how many refused (one
@@ -513,13 +332,12 @@ entries by client address, with times in UTC.
- `lookups.json`: GeoJS's answers, one to a line, with when GeoJS gave each and - `lookups.json`: GeoJS's answers, one to a line, with when GeoJS gave each and
when it was last used. when it was last used.
`bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made or made `bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, with every
permanent, with every such change in between, and every file every ban made in between, and every file every `SWWAF_STATE_COUNTER_INTERVAL` and
`SWWAF_STATE_COUNTER_INTERVAL` and when `smallwebwaf` stops. Each write goes to when `smallwebwaf` stops. Each write goes to a temporary file in the same
a temporary file in the same directory, which then replaces the file, so a crash directory, which then replaces the file, so a crash leaves the old file or the
leaves the old file or the new one, whole. A write that fails is logged, and new one, whole. A write that fails is logged, and tried again at the next write.
tried again at the next write. A hard kill loses what changed since the last A hard kill loses what changed since the last write.
write.
At start the files are read back: each client keeps its counts, so a restart At start the files are read back: each client keeps its counts, so a restart
gives it no fresh allowance, and each ban keeps refusing every client in its gives it no fresh allowance, and each ban keeps refusing every client in its
@@ -534,8 +352,7 @@ without a field it needs, named with the entry's place in the file: a ban's
`netblock`, `start` or `expires`, which is `null` for a permanent ban; a `netblock`, `start` or `expires`, which is `null` for a permanent ban; a
client's `client`, or the `start` of a window in which it has requests; an client's `client`, or the `start` of a window in which it has requests; an
answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
`answered`. So does a ban whose `cause` is not `limit`, `attack` or `admin`. The `answered`. The AS number and AS name come with their lookup.
AS number and AS name come with their lookup.
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
a state file as soon as you save it: what the file then holds replaces what a state file as soon as you save it: what the file then holds replaces what
@@ -544,21 +361,17 @@ yours by comparing the file with what it last read or wrote, and before it
writes a file it takes in any edit made since, so your edit is not overwritten; writes a file it takes in any edit made since, so your edit is not overwritten;
a change `smallwebwaf` made after you opened the file, such as a new ban, is a change `smallwebwaf` made after you opened the file, such as a new ban, is
lost when you save over it. An edit that would stop the start, because it does lost when you save over it. An edit that would stop the start, because it does
not parse, has another `version`, leaves out a field an entry needs or gives a not parse, has another `version` or leaves out a field an entry needs, does not
ban another `cause`, does not stop the running `smallwebwaf`: it keeps what it stop the running `smallwebwaf`: it keeps what it holds, and at the file's next
holds, and at the file's next write renames your file to `<name>.bad`, such as write renames your file to `<name>.bad`, such as `bans.json.bad`, writes the
`bans.json.bad`, writes the file again from memory, and logs the file and where file again from memory, and logs the file and where the error is. It waits for
the error is. It waits for that write because an editor's file can be read that write because an editor's file can be read before the editor has finished
before the editor has finished writing it. Mend the `.bad` file and move it writing it. Mend the `.bad` file and move it back. A file you remove is written
back. A file you remove is written again at its next write. again at its next write.
To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start` To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
and its `expires`, `null` for a ban that never ends; its `reason` and its and its `expires`, `null` for a ban that never ends; its `notes` may be left
`notes` may be left out, and so may its `cause`, which is then `admin`, and is out. This `bans.json` bans `203.0.113.0/24` for good:
written so at the file's next write. A ban whose `cause` is `admin` is never
dropped and does not count toward `SWWAF_MAX_BANS`. A ban whose `cause` is
`attack` becomes permanent at the first request it refuses; one whose `cause` is
`admin` does not. This `bans.json` bans `203.0.113.0/24` for good:
```json ```json
{ {
@@ -567,97 +380,14 @@ dropped and does not count toward `SWWAF_MAX_BANS`. A ban whose `cause` is
{ {
"netblock": "203.0.113.0/24", "netblock": "203.0.113.0/24",
"start": "2026-10-06T12:00:00Z", "start": "2026-10-06T12:00:00Z",
"expires": null, "expires": null
"reason": "probes for logins"
} }
] ]
} }
``` ```
To keep a ban `smallwebwaf` made, so that it is never dropped, set its `cause` To lift a ban, delete its entry. `smallwebwaf` then forgets the ban, so it does
to `admin`: `"cause": "admin"`. not make the netblock's next ban longer.
To lift a ban, add `lifted` to its entry, with the time you lift it, such as
`"lifted": "2026-10-06T13:00:00Z"`. From when the edit is taken in, the ban
refuses nothing, whatever time `lifted` gives, and does not make the netblock's
next ban longer; it is kept in `bans.json` with its notes, as any other ban is.
To forget a ban altogether, delete its entry: it then refuses nothing either,
and does not make the netblock's next ban longer.
## Rule files
`smallwebwaf` reads every `*.rules` file in `SWWAF_RULES_DIR`,
`/etc/smallwebwaf/rules.d` by default, in the order of their names, and checks
each request against their rules in that order, as "Rule files" in
[`SPEC.md`](SPEC.md) describes. A file whose name starts with `.`, such as an
editor's lock file `.#50-app.rules`, is not a rule file, as a shell's `*.rules`
would not match it. A rule is a line of four fields separated by spaces or tabs:
an id, a target, an action and a regex, which runs to the end of the line.
Spaces and tabs at the end of a line are not part of its regex, so a line with
only those after its action has no regex, and is not a rule. Blank lines and
lines that start with `#` are ignored.
```
# id target action regex
env-file path ban (?i)^/\.env(\.[a-z]+)?$
scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b
```
- The id is letters, digits, `-` and `_`, and no two rules share one. The
request log, the metrics and a ban's notes name the rule by it.
- The target is what the regex is matched against: `path` or `query`, as the
client sent it, before any decoding; `uri`, the path and the query together,
both as sent and once percent-decoded, so that an encoded probe does not slip
past; `method`; `host`; `user_agent`; `referer`; or `header:<Name>`, any one
request header but `Host` and `Transfer-Encoding`, which Go's HTTP server
takes out of every request; the request's host is the target `host`. A header
sent more than once is matched with its values joined by `, `, and one not
sent as empty text. No body is read.
- The action is `log`, `block` or `ban` (see "What it does so far" above). Keep
`ban` for requests no real visitor sends, and anchor a path at the site root
with `^/`: a file of the same name deeper in a site can be ordinary content,
such as a file in a repository on a code forge.
- The regex is in Go's syntax, RE2, which has no backreferences or lookaround,
and takes time linear in the text it reads. It matches anywhere in the target
unless anchored with `^` and `$`; `(?i)` at its front makes it ignore case.
A line that is not a rule, a `header:` name with a character no header name can
have, such as `header:User-Agent:`, a rule for the `Host` or the
`Transfer-Encoding` header, a regex that does not compile or an id used twice
stops the start with a message naming the file and the line, and so does a
`SWWAF_RULES_DIR` that does not exist. An empty directory is no error, and the
log says that it holds no rules. While it runs, `smallwebwaf` watches the
directory, and reads the rule files again once the directory has had no change
for 2 seconds after one is edited, added or removed, so that a file saved in
place, appended to or copied in with `scp` is read only once whole, unless its
writing stops for longer. It also reads them 2 seconds after it starts watching,
so that an edit saved while it started is not missed. If they then hold one of
those errors, the rules stay as they were, the earlier version of the edited
file included, the log names the file and the line, and the files are read again
after the next change.
The image ships one rule file, `share/rules.d/00-default.rules` here: rules that
ban probes no real visitor sends, for secrets, version control directories,
backups, logs and web shells at the site root, and the user agents of common
scanners; one that blocks `../` twice in a row in the path or the query; and one
that only notes a request without a user agent. An app's Dockerfile adds rules
of its own in a file beside it, named to sort after it, such as this
`50-gitea.rules` for an app that serves no WordPress:
```
wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/)
```
```dockerfile
COPY 50-gitea.rules /etc/smallwebwaf/rules.d/50-gitea.rules
```
A directory mounted over `/etc/smallwebwaf/rules.d` replaces the default file,
and single files mounted into it add to it. Docker does not show a single
mounted file being replaced, which is how many editors save, so rules to be
edited while `smallwebwaf` runs belong in a mounted directory, with a copy of
`00-default.rules` if its rules are to stay. To run without rules, mount an
empty directory or set `SWWAF_RULES_ENABLED=false`.
## Metrics ## Metrics
@@ -676,13 +406,8 @@ other request. No metric carries a client's address.
- `smallwebwaf_rate_limit_hits_total` by `window`, - `smallwebwaf_rate_limit_hits_total` by `window`,
`smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose `smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose
limit was passed, `smallwebwaf_offences_total` by `kind`, and limit was passed, `smallwebwaf_offences_total` by `kind`, and
`smallwebwaf_bans_made_total` by `cause`, `limit`, `attack` or `admin`, the `smallwebwaf_bans_made_total` by `cause`; `smallwebwaf_active_bans` and
last for the bans whose `cause` is `admin` that you add to `bans.json` while `smallwebwaf_permanent_bans`.
`smallwebwaf` runs; `smallwebwaf_active_bans` and
`smallwebwaf_permanent_bans`, neither of which counts a lifted ban.
- `smallwebwaf_rule_matches_total`: the requests that matched each rule, by
`rule_id` and `action`, the rule's own; and `smallwebwaf_rules_loaded`: the
rules read from the rule files.
- `smallwebwaf_country_requests_total`, - `smallwebwaf_country_requests_total`,
`smallwebwaf_country_request_bytes_total`, `smallwebwaf_country_request_bytes_total`,
`smallwebwaf_country_response_bytes_total`, and `smallwebwaf_country_response_bytes_total`, and
@@ -705,16 +430,11 @@ other request. No metric carries a client's address.
`smallwebwaf_state_file_edits_taken_in_total`: your edits taken in, and `smallwebwaf_state_file_edits_taken_in_total`: your edits taken in, and
`smallwebwaf_state_file_edits_set_aside_total`: those renamed to `<name>.bad` `smallwebwaf_state_file_edits_set_aside_total`: those renamed to `<name>.bad`
because they would stop the start. because they would stop the start.
- While `SWWAF_LOG_REMOTE_URL` is set,
`smallwebwaf_remote_log_lines_sent_total`: the lines sent to it;
`smallwebwaf_remote_log_lines_dropped_total`: those dropped, from a full
buffer or because their sending failed; and
`smallwebwaf_remote_log_buffer_depth`: those waiting in the buffer.
- Go's own `go_` metrics and the process's `process_` metrics. - Go's own `go_` metrics and the process's `process_` metrics.
The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
log") are not counted. The metrics of the features still to come, such as the log") are not counted. The metrics of the features still to come, such as the
Core Rule Set, come with them. rule files, come with them.
## Why ## Why
@@ -988,29 +708,25 @@ addresses are never sent to GeoJS.
## How the code is laid out ## How the code is laid out
- `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`. - `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`.
- `internal/smallwebwaf`: the process: it reads the settings, the rule files and - `internal/smallwebwaf`: the process: it reads the settings and the state
the state files, listens, serves requests until `SIGTERM` or `SIGINT`, and files, listens, serves requests until `SIGTERM` or `SIGINT`, and stops,
stops, writing the state files. Run as `smallwebwaf healthcheck`, it is the writing the state files. Run as `smallwebwaf healthcheck`, it is the image's
image's health check instead. health check instead.
- `internal/config`: reads the settings, the one place they are read. - `internal/config`: reads the settings, the one place they are read.
- `internal/proxy`: what happens to each request: it works out the client, runs - `internal/proxy`: what happens to each request: it works out the client, runs
the checks, passes the request to the app and the answer back with the the checks, passes the request to the app and the answer back with the
standard library's `httputil.ReverseProxy` within the timeouts and size standard library's `httputil.ReverseProxy` within the timeouts and size
limits, and writes the request's log line. Its `check` method is where a limits, and writes the request's log line. Its `check` method is where a
request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
a ban, for the country lists, for a rate limit, which bans the client, for a a ban, for the country lists, for a rate limit, which bans the client, and for
`block` or `ban` rule, the latter banning the client, and for an announced an announced body over the size limit; in `observe` mode, only for the size
body over the size limit; in `observe` mode, only for the size limit, with limit, with what it would have refused for noted in the log line. A request
what it would have refused for noted in the log line. A request under under `/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
`/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
instead of reaching the app. instead of reaching the app.
- `internal/metrics`: the metrics, counted as the other parts tell it what - `internal/metrics`: the metrics, counted as the other parts tell it what
happened, and served in the Prometheus text format. happened, and served in the Prometheus text format.
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how - `internal/bans`: the ban ledger: each netblock's bans with their notes, how
long a new ban lasts, when a ban for a clear sign of attack becomes permanent, long a new ban lasts, and which ban is dropped when `SWWAF_MAX_BANS` are held.
and which ban `smallwebwaf` made is dropped when `SWWAF_MAX_BANS` are held.
- `internal/rules`: reads the rule files at start and again as they change, and
tells which of their rules a request matches.
- `internal/lookup`: looks up each client's country through GeoJS, and keeps the - `internal/lookup`: looks up each client's country through GeoJS, and keeps the
answers. answers.
- `internal/ratelimit`: the table of clients: counts each client's requests, - `internal/ratelimit`: the table of clients: counts each client's requests,
@@ -1019,24 +735,18 @@ addresses are never sent to GeoJS.
one while running, and writes them when they are due and at the stop. one while running, and writes them when they are due and at the stop.
- `internal/requestlog`: the lines on stdout: the request log line and the - `internal/requestlog`: the lines on stdout: the request log line and the
process's own messages. process's own messages.
- `internal/remotelog`: sends the lines on stdout to `SWWAF_LOG_REMOTE_URL`,
each as a syslog record, from a buffer of its own. It is written with the
standard library alone, whose `log/syslog` writes only the older syslog
format.
- `Dockerfile`: the lint and test phases, then the image, whose last stage - `Dockerfile`: the lint and test phases, then the image, whose last stage
installs Ubuntu's packages, nixpkgs, `runsvinit` and `smallwebwaf`, with installs Ubuntu's packages, nixpkgs, `runsvinit` and `smallwebwaf`, with
`share/smallwebwaf.run` as runit's `run` script for `smallwebwaf` and `share/smallwebwaf.run` as runit's `run` script for `smallwebwaf`.
`share/rules.d/00-default.rules` as its default rule file.
- `deploy/example-app`: an app built on the image, which `script/example-app` - `deploy/example-app`: an app built on the image, which `script/example-app`
checks. checks.
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least table of clients to 20,000, the GeoJS answers to 100,000 and the banned
recently seen, and the banned netblocks in the order they were last seen, from netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen, and
which the ledger picks the ban to drop past `SWWAF_MAX_BANS`, and
`github.com/prometheus/client_golang` keeps the metrics and serves them, and `github.com/prometheus/client_golang` keeps the metrics and serves them, and
`github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file or a rule `github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file is saved.
file is saved. The country codes are the list in `internal/config/config.go`. The country codes are the list in `internal/config/config.go`.
## Entrypoints ## Entrypoints
@@ -1066,22 +776,21 @@ so that they run in minimal containers.
- `script/build`: builds `bin/smallwebwaf` on the host, with Go installed, for - `script/build`: builds `bin/smallwebwaf` on the host, with Go installed, for
working on the code by hand; `make build` runs it. working on the code by hand; `make build` runs it.
- `script/run`: builds `bin/smallwebwaf` with `script/build` and runs it, with - `script/run`: builds `bin/smallwebwaf` with `script/build` and runs it, with
its state files in `bin/state` unless `SWWAF_STATE_DIR` is set, and the rule its state files in `bin/state` unless `SWWAF_STATE_DIR` is set; `make run`
files of `share/rules.d` unless `SWWAF_RULES_DIR` is set; `make run` runs it. runs it.
- `script/example-app`: builds the image and, on it, the example app in - `script/example-app`: builds the image and, on it, the example app in
`deploy/example-app`, runs it with a volume for the state files, and checks `deploy/example-app`, runs it with a volume for the state files, and checks
that the health check passes, that a request reaches the app through that the health check passes, that a request reaches the app through
`smallwebwaf`, that a second request in a minute bans the client, that a probe `smallwebwaf`, that a second request in a minute bans the client, that
for `/.env` bans another client, whose next request makes the ban permanent, `sv stop` and `docker stop` stop it in order, and that a new container on the
that `sv stop` and `docker stop` stop it in order, and that a new container on same volume still refuses the banned client; then removes the containers, the
the same volume still refuses the banned client; then removes the containers, volume and both images. It needs network access, for nixpkgs' binary cache,
the volume and both images. It needs network access, for nixpkgs' binary and `script/check` does not run it; `make example-app` does.
cache, and `script/check` does not run it; `make example-app` does.
## TODO ## TODO
- The rest of the design, in the order of the build order in - The rest of milestone 3: the rest of the request log's fields; then the rest
[`SPEC.md`](SPEC.md). of the design, in the order of the build order in [`SPEC.md`](SPEC.md).
## Documents ## Documents
+2 -3
View File
@@ -32,7 +32,7 @@ from a directory of hand-editable text files.
- Defence against traffic floods that saturate the host's network link. That - Defence against traffic floods that saturate the host's network link. That
needs help upstream of the host. needs help upstream of the host.
- A web UI or a configuration file. Settings are environment variables. Apart - A web UI or a configuration file. Settings are environment variables. Apart
from settings given as files (the `_FILE` form of a setting, such as from settings given as files (the `_FILE` form of any setting, such as
`SWWAF_ADMIN_TOKEN_FILE`, and `SWWAF_LOG_REMOTE_TLS_CA_FILE`), its own state `SWWAF_ADMIN_TOKEN_FILE`, and `SWWAF_LOG_REMOTE_TLS_CA_FILE`), its own state
files and the lookup database, the only files read are the rule files, which files and the lookup database, the only files read are the rule files, which
hold one regex per line and nothing more elaborate. hold one regex per line and nothing more elaborate.
@@ -298,8 +298,7 @@ it.
- A list set to an empty value is an empty list, and replaces the default. - A list set to an empty value is an empty list, and replaces the default.
- Every setting may instead be given as a file holding the value, named by the - Every setting may instead be given as a file holding the value, named by the
setting's name with `_FILE` added, such as `SWWAF_ADMIN_TOKEN_FILE`, for setting's name with `_FILE` added, such as `SWWAF_ADMIN_TOKEN_FILE`, for
secrets and long lists. `SWWAF_LOG_REMOTE_TLS_CA_FILE`, whose value names a secrets and long lists.
file already, has no `_FILE` form.
- Settings, including those given as files, are read once at start; changing one - Settings, including those given as files, are read once at start; changing one
means restarting the container. The files `smallwebwaf` watches while it runs means restarting the container. The files `smallwebwaf` watches while it runs
are its state files, its rule files and the lookup database. are its state files, its rule files and the lookup database.
-186
View File
@@ -1,186 +0,0 @@
package bans_test
import (
"net/netip"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
)
func TestBanWithoutACauseIsAnAdmins(t *testing.T) {
t.Parallel()
netblock := netip.MustParsePrefix("203.0.113.0/24")
ledger := bans.New(defaultRules())
ledger.Load([]bans.Ban{{Netblock: netblock, Start: midnight()}})
if got := ledger.Bans(netblock)[0].Cause; got != bans.CauseAdmin {
t.Errorf("the ban's cause is %q, want admin", got)
}
}
func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
t.Parallel()
rules := defaultRules()
rules.MaxBans = 1
ledger := bans.New(rules)
adminsOnly := netip.MustParsePrefix("198.51.100.0/24")
both := netip.MustParsePrefix("203.0.113.1/32")
second := netip.MustParsePrefix("203.0.113.2/32")
third := netip.MustParsePrefix("203.0.113.3/32")
// Seen longest ago, a netblock with two of an admin's bans alone, and
// then one with an admin's ban before a ban smallwebwaf made: the one
// ban counted toward MaxBans.
ledger.Load([]bans.Ban{
{Netblock: adminsOnly, Start: midnight().Add(-3 * time.Hour), Cause: bans.CauseAdmin},
{Netblock: adminsOnly, Start: midnight().Add(-2 * time.Hour), Cause: bans.CauseAdmin},
{Netblock: both, Start: midnight().Add(-time.Hour), Cause: bans.CauseAdmin},
{
Netblock: both,
Start: midnight(),
Expires: midnight().Add(time.Hour),
Cause: bans.CauseLimit,
},
})
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 2})
// A new ban drops the ban smallwebwaf made, and only that one.
ledger.BanForLimit(second, midnight(), bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 1, second: 1})
if ledger.Bans(both)[0].Cause != bans.CauseAdmin {
t.Errorf("%s kept %+v, want the admin's ban", both, ledger.Bans(both))
}
// And the next drops that one.
ledger.BanForLimit(third, midnight(), bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 1, second: 0, third: 1})
}
func TestReasonOfTheBansSmallwebwafMakes(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
limit := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.1/32"), midnight(),
bans.Notes{Limit: 1000, Window: "minute"})
attack := ledger.BanForAttack(netip.MustParsePrefix("203.0.113.2/32"), midnight(),
bans.Notes{RuleID: "git-dir", Target: "path"})
for _, tc := range []struct{ got, want string }{
{limit.Reason, "requests per minute over the limit of 1000"},
{attack.Reason, "matched the rule git-dir"},
} {
if tc.got != tc.want {
t.Errorf("the reason is %q, want %q", tc.got, tc.want)
}
}
}
func TestLiftedBanForALimitRefusesNothingAndMakesNoBanLonger(t *testing.T) {
t.Parallel()
// An hour's ban lifted ten minutes after it started.
netblock := netip.MustParsePrefix("203.0.113.9/32")
lifted := bans.Ban{
Netblock: netblock,
Start: midnight(),
Expires: midnight().Add(time.Hour),
Cause: bans.CauseLimit,
Lifted: midnight().Add(10 * time.Minute),
}
ledger := bans.New(defaultRules())
ledger.Load([]bans.Ban{lifted})
// While it would still last, it refuses nothing, and a limit broken
// bans for an hour, as a first broken limit does; the lifted ban is
// kept, and counted among the earlier bans.
now := midnight().Add(30 * time.Minute)
_, banned := ledger.Check(netblock.Addr(), now)
if banned {
t.Error("the lifted ban refuses")
}
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
if ban.Expires.Sub(ban.Start) != time.Hour ||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
t.Errorf("the next ban lasts %s with earlier bans %+v, want 1h and 1 for a limit",
ban.Expires.Sub(ban.Start), ban.Notes.EarlierBans)
}
held := ledger.Bans(netblock)
if len(held) != 2 || held[0] != lifted {
t.Errorf("the ledger holds %+v, want the lifted ban and the new one", held)
}
}
func TestLiftedBanForAnAttackRefusesNothingAndMakesNoBanLonger(t *testing.T) {
t.Parallel()
// A permanent ban for a clear sign of attack, lifted.
netblock := netip.MustParsePrefix("203.0.113.9/32")
ledger := bans.New(defaultRules())
ledger.Load([]bans.Ban{{
Netblock: netblock,
Start: midnight(),
Cause: bans.CauseAttack,
Lifted: midnight().Add(time.Hour),
}})
now := midnight().Add(2 * time.Hour)
_, banned := ledger.Find(netblock.Addr(), now)
if banned {
t.Error("the lifted ban refuses")
}
active, permanent := ledger.Count(now)
if active != 0 || permanent != 0 {
t.Errorf("%d bans are active and %d permanent, want none", active, permanent)
}
// The next clear sign of attack bans for seven days, as a first does.
ban := ledger.BanForAttack(netblock, now, bans.Notes{})
if ban.Expires.Sub(ban.Start) != 7*day {
t.Errorf("the next ban for an attack ends at %s, want seven days on", ban.Expires)
}
}
func TestLoadEditCountsTheBansAnAdminMade(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
made := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.1/32"), midnight(),
bans.Notes{})
atStart := bans.Ban{
Netblock: netip.MustParsePrefix("203.0.113.2/32"),
Start: midnight(),
}
// The bans read at the start were made before it.
ledger.Load([]bans.Ban{made, atStart})
if got := ledger.Made(bans.CauseAdmin); got != 0 {
t.Fatalf("%d bans made by an admin after the start's, want none", got)
}
// The admin keeps the ban smallwebwaf made, keeps the one read at the
// start, and adds one without a cause: that one alone is made.
kept := made
kept.Cause = bans.CauseAdmin
added := bans.Ban{
Netblock: netip.MustParsePrefix("203.0.113.3/32"),
Start: midnight(),
}
ledger.LoadEdit([]bans.Ban{kept, atStart, added})
if ledger.Made(bans.CauseAdmin) != 1 || ledger.Made(bans.CauseLimit) != 1 {
t.Errorf("%d bans made by an admin and %d for a limit, want 1 of each",
ledger.Made(bans.CauseAdmin), ledger.Made(bans.CauseLimit))
}
}
+115 -309
View File
@@ -1,13 +1,10 @@
// Package bans is the ban ledger: the bans smallwebwaf makes on the // Package bans is the ban ledger: the bans smallwebwaf makes on the
// netblocks of clients that break a rate limit or show a clear sign of // netblocks of clients that break a rate limit, with their notes, as the
// attack, and those an admin makes, with their notes, as the "Bans" // "Bans" section of SPEC.md describes. The bans are kept in memory, and
// section of SPEC.md describes. The bans are kept in memory, and written // written to bans.json and read from it by the state package.
// to bans.json and read from it by the state package.
package bans package bans
import ( import (
"fmt"
"math"
"net/netip" "net/netip"
"slices" "slices"
"strings" "strings"
@@ -17,17 +14,6 @@ import (
"github.com/hashicorp/golang-lru/v2/simplelru" "github.com/hashicorp/golang-lru/v2/simplelru"
) )
// The causes of bans.
const (
// CauseLimit is a ban smallwebwaf made for a broken limit.
CauseLimit = "limit"
// CauseAttack is a ban smallwebwaf made for a clear sign of attack.
CauseAttack = "attack"
// CauseAdmin is a ban an admin made, or one smallwebwaf made that an
// admin keeps. It is never dropped.
CauseAdmin = "admin"
)
// repeatFactor is how many times as long as the netblock's last ban a ban // repeatFactor is how many times as long as the netblock's last ban a ban
// for a limit broken again within the repeat window lasts. // for a limit broken again within the repeat window lasts.
const repeatFactor = 3 const repeatFactor = 3
@@ -35,44 +21,32 @@ const repeatFactor = 3
// maxTextBytes is how much of each text in a ban's notes is kept. // maxTextBytes is how much of each text in a ban's notes is kept.
const maxTextBytes = 256 const maxTextBytes = 256
// Rules are how long a ban lasts, and how many bans are held. // Rules are how long a ban for a broken limit lasts, and how many bans
// are held.
type Rules struct { type Rules struct {
// LimitBanDuration is how long a first ban for a broken limit lasts. // LimitBanDuration is how long a first ban lasts.
LimitBanDuration time.Duration LimitBanDuration time.Duration
// LimitBanRepeatWindow is how soon after the end of the netblock's // LimitBanRepeatWindow is how soon after the end of the netblock's
// ban that ended last, other than one for a clear sign of attack, a // ban that ended last a broken limit counts as a repeat, which bans
// broken limit counts as a repeat, which bans for repeatFactor times as // for repeatFactor times as long as that ban.
// long as that ban.
LimitBanRepeatWindow time.Duration LimitBanRepeatWindow time.Duration
// MaxBanDuration is the longest ban for a broken limit; one that would // MaxBanDuration is the longest ban; a ban that would be longer is
// be longer is permanent instead. // permanent instead.
MaxBanDuration time.Duration MaxBanDuration time.Duration
// AttackBanDuration is how long a first ban for a clear sign of attack // MaxBans is the most bans held, at least one. Past it, the earliest
// lasts. // ban of the netblock that has gone longest without a request is
AttackBanDuration time.Duration // dropped.
// MaxBans is the most bans held whose cause is not CauseAdmin, at
// least one. Past it, the earliest such ban of the netblock that has
// gone longest without a request is dropped. Bans whose cause is
// CauseAdmin are held besides, and never dropped.
MaxBans int MaxBans int
} }
// Ban is a ban on a netblock. // Ban is a ban on a netblock for a broken limit, the only kind of ban
// smallwebwaf makes so far.
type Ban struct { type Ban struct {
Netblock netip.Prefix Netblock netip.Prefix
Start time.Time Start time.Time
// Expires is when the ban ends, zero for a permanent ban. // Expires is when the ban ends, zero for a permanent ban.
Expires time.Time Expires time.Time
// Cause is CauseLimit, CauseAttack or CauseAdmin. Notes Notes
Cause string
// Reason is a short text: for a ban smallwebwaf made, the limit broken
// or the rule that matched; for an admin's, what the admin wrote.
Reason string
// Lifted is when an admin lifted the ban, zero while no admin has. A
// lifted ban refuses nothing, and does not make the netblock's next
// ban longer.
Lifted time.Time
Notes Notes
} }
// Permanent reports whether the ban never runs out. // Permanent reports whether the ban never runs out.
@@ -80,10 +54,9 @@ func (b Ban) Permanent() bool {
return b.Expires.IsZero() return b.Expires.IsZero()
} }
// ActiveAt reports whether the ban refuses requests at now: it has not // ActiveAt reports whether the ban refuses requests at now.
// been lifted, and has not run out.
func (b Ban) ActiveAt(now time.Time) bool { func (b Ban) ActiveAt(now time.Time) bool {
return b.Lifted.IsZero() && (b.Permanent() || now.Before(b.Expires)) return b.Permanent() || now.Before(b.Expires)
} }
// Notes are what an admin needs to decide whether to lift a ban. The // Notes are what an admin needs to decide whether to lift a ban. The
@@ -93,36 +66,23 @@ func (b Ban) ActiveAt(now time.Time) bool {
type Notes struct { type Notes struct {
// Country is the client's country, when it was looked up. // Country is the client's country, when it was looked up.
Country string `json:"country"` Country string `json:"country"`
// Limit, Window and Count are, for a ban for a broken limit, the limit // Limit, Window and Count are the limit that was broken, its window,
// that was broken, its window, "minute", "hour" or "day", and the // "minute", "hour" or "day", and the count reached: the client's
// count reached: the client's requests in the window, the one that // requests in the window, the one that broke the limit included.
// broke the limit included. These are the requests that counted // These are the requests that counted toward the ban, and the window
// toward the ban, and the window is the time over which they came. // is the time over which they came.
Limit int64 `json:"limit,omitempty"` Limit int64 `json:"limit"`
Window string `json:"window,omitempty"` Window string `json:"window"`
Count float64 `json:"count,omitempty"` Count float64 `json:"count"`
// RuleID and Target are, for a ban for a clear sign of attack, the id // Request is the request that broke the limit.
// of the rule file rule that matched, and its target.
RuleID string `json:"rule_id,omitempty"`
Target string `json:"target,omitempty"`
// Request is the request that broke the limit, or that was the clear
// sign of attack.
Request Request `json:"request"` Request Request `json:"request"`
// Requests is how many requests the netblock has sent since it was // Requests is how many requests the netblock has sent since it was
// first seen, and Refused how many of them the ban has refused so // first seen, and Refused how many of them the ban has refused so
// far. Both go up with each request the ban refuses. // far. Both go up with each request the ban refuses.
Requests int64 `json:"requests"` Requests int64 `json:"requests"`
Refused int64 `json:"refused"` Refused int64 `json:"refused"`
// EarlierBans is how many bans the netblock had before this one, by // EarlierBans is how many bans the netblock had before this one.
// cause. EarlierBans int `json:"earlier_bans"`
EarlierBans EarlierBans `json:"earlier_bans"`
}
// EarlierBans counts a netblock's bans before a ban, by cause.
type EarlierBans struct {
Limit int `json:"limit"`
Attack int `json:"attack"`
Admin int `json:"admin"`
} }
// Request is a request in a ban's notes. Each text is cut to 256 bytes. // Request is a request in a ban's notes. Each text is cut to 256 bytes.
@@ -150,12 +110,10 @@ type Ledger struct {
// netblocks holds each banned netblock's bans, oldest first. Check and // netblocks holds each banned netblock's bans, oldest first. Check and
// Find make each netblock they find the most recently seen. // Find make each netblock they find the most recently seen.
netblocks *simplelru.LRU[netip.Prefix, *[]Ban] netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
// held is how many bans netblocks holds whose cause is not CauseAdmin, // held is how many bans netblocks holds, at most rules.MaxBans.
// at most rules.MaxBans.
held int held int
// made is how many bans have been made since the start, by cause: by // made is how many bans BanForLimit has made since the start.
// the ledger, and by an admin in an edit of bans.json. made int
made map[string]int
// v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6 // v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6
// netblocks that have been banned. Check looks for a ban at each of // netblocks that have been banned. Check looks for a ban at each of
// them, so that a ban read from bans.json refuses every client in its // them, so that a ban read from bans.json refuses every client in its
@@ -166,10 +124,9 @@ type Ledger struct {
// New returns a Ledger with no ban yet. // New returns a Ledger with no ban yet.
func New(rules Rules) *Ledger { func New(rules Rules) *Ledger {
// The ledger drops bans itself, and never those whose cause is // Every netblock held has a ban, so there are never more netblocks
// CauseAdmin, however many there are, so the LRU has no limit of its // than rules.MaxBans, and the LRU never drops one itself.
// own: it keeps the netblocks in the order they were last seen. netblocks, err := simplelru.NewLRU[netip.Prefix, *[]Ban](rules.MaxBans, nil)
netblocks, err := simplelru.NewLRU[netip.Prefix, *[]Ban](math.MaxInt, nil)
if err != nil { if err != nil {
panic(err) // NewLRU fails only for a size below one panic(err) // NewLRU fails only for a size below one
} }
@@ -178,21 +135,18 @@ func New(rules Rules) *Ledger {
rules: rules, rules: rules,
changed: make(chan struct{}, 1), changed: make(chan struct{}, 1),
netblocks: netblocks, netblocks: netblocks,
made: map[string]int{},
} }
} }
// Changed receives a value after a ban is made or made permanent, so that // Changed receives a value after a ban is made, so that bans.json can be
// bans.json can be written. Several changes before it is read leave one // written. Several bans made before it is read leave one value.
// value.
func (l *Ledger) Changed() <-chan struct{} { func (l *Ledger) Changed() <-chan struct{} {
return l.changed return l.changed
} }
// Check is called for a request from client, at now. It reports whether // Check is called for a request from client, at now. It reports whether
// a ban on a netblock client is in is active, and returns that ban, with // a ban on a netblock client is in is active, and returns that ban, with
// the request counted among those it refused. A ban for a clear sign of // the request counted among those it refused.
// attack is made permanent by the request: the netblock is malicious.
func (l *Ledger) Check(client netip.Addr, now time.Time) (Ban, bool) { func (l *Ledger) Check(client netip.Addr, now time.Time) (Ban, bool) {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
@@ -205,12 +159,6 @@ func (l *Ledger) Check(client netip.Addr, now time.Time) (Ban, bool) {
ban.Notes.Requests++ ban.Notes.Requests++
ban.Notes.Refused++ ban.Notes.Refused++
if ban.Cause == CauseAttack && !ban.Permanent() {
ban.Expires = time.Time{}
l.markChanged()
}
return *ban, true return *ban, true
} }
@@ -244,27 +192,54 @@ func activeBan(bans []Ban, now time.Time) *Ban {
// BanForLimit bans netblock at now for a broken limit, with notes, and // BanForLimit bans netblock at now for a broken limit, with notes, and
// returns the ban. A first ban lasts LimitBanDuration. A ban made within // returns the ban. A first ban lasts LimitBanDuration. A ban made within
// LimitBanRepeatWindow after the netblock's ban that ended last, other // LimitBanRepeatWindow after the netblock's ban that ended last lasts
// than one for a clear sign of attack or a lifted one, lasts repeatFactor // repeatFactor times as long as that one. A ban that would be longer
// times as long as that one. A ban that would be longer than // than MaxBanDuration is permanent instead. If a ban on netblock is still
// MaxBanDuration is permanent instead. If a ban on netblock is still
// active, as when two of its requests break a limit at once, that ban is // active, as when two of its requests break a limit at once, that ban is
// returned and no other is made. The ledger fills in the notes' Refused // returned and no other is made. The ledger fills in the notes' Refused
// and EarlierBans itself, and gives the ban the reason "requests per // and EarlierBans itself.
// <Window> over the limit of <Limit>", from the notes.
func (l *Ledger) BanForLimit(netblock netip.Prefix, now time.Time, notes Notes) Ban { func (l *Ledger) BanForLimit(netblock netip.Prefix, now time.Time, notes Notes) Ban {
reason := fmt.Sprintf("requests per %s over the limit of %d", l.mu.Lock()
notes.Window, notes.Limit) defer l.mu.Unlock()
return l.ban(netblock, now, CauseLimit, reason, notes) var last *Ban
}
// BanForAttack bans netblock at now for a clear sign of attack, with bans, found := l.netblocks.Get(netblock)
// notes, and returns the ban, as BanForLimit does. A first ban lasts if found {
// AttackBanDuration; once the netblock has had one that was not lifted, active := activeBan(*bans, now)
// the next is permanent. Its reason is "matched the rule <RuleID>". if active != nil {
func (l *Ledger) BanForAttack(netblock netip.Prefix, now time.Time, notes Notes) Ban { return *active
return l.ban(netblock, now, CauseAttack, "matched the rule "+notes.RuleID, notes) }
// No ban is active, so each has an end. A ban an admin adds to
// bans.json can start after another and end before it, so the
// ban that ended last is looked for among them all.
ended := slices.MaxFunc(*bans, func(a, b Ban) int {
return a.Expires.Compare(b.Expires)
})
last = &ended
// The netblock's first ban held counts the bans it had before that
// one, since dropped to make room, and each ban held adds one.
notes.EarlierBans = (*bans)[0].Notes.EarlierBans + len(*bans)
}
notes.Request = notes.Request.cut()
ban := Ban{
Netblock: netblock,
Start: now,
Expires: l.expiry(last, now),
Notes: notes,
}
l.add(ban)
l.made++
select {
case l.changed <- struct{}{}:
default: // a value is waiting already
}
return ban
} }
// Bans returns the bans held on netblock, oldest first. It is not a // Bans returns the bans held on netblock, oldest first. It is not a
@@ -281,19 +256,17 @@ func (l *Ledger) Bans(netblock netip.Prefix) []Ban {
return slices.Clone(*bans) return slices.Clone(*bans)
} }
// Made returns how many bans for cause have been made since the start: // Made returns how many bans the ledger has made since the start; bans
// for CauseLimit and CauseAttack, by the ledger; for CauseAdmin, by an // read from bans.json are not among them.
// admin in an edit of bans.json, as LoadEdit counts them. The bans read func (l *Ledger) Made() int {
// from bans.json at the start are not among them.
func (l *Ledger) Made(cause string) int {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
return l.made[cause] return l.made
} }
// Count returns how many of the bans held are active at now, and how many // Count returns how many of the bans held are active at now, and how many
// of those are permanent. A lifted ban is neither. // are permanent.
func (l *Ledger) Count(now time.Time) (int, int) { func (l *Ledger) Count(now time.Time) (int, int) {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
@@ -302,12 +275,10 @@ func (l *Ledger) Count(now time.Time) (int, int) {
for _, bans := range l.netblocks.Values() { for _, bans := range l.netblocks.Values() {
for _, ban := range *bans { for _, ban := range *bans {
if !ban.ActiveAt(now) { if ban.ActiveAt(now) {
continue active++
} }
active++
if ban.Permanent() { if ban.Permanent() {
permanent++ permanent++
} }
@@ -335,142 +306,30 @@ func (l *Ledger) Snapshot() []Ban {
return held return held
} }
// Load puts bans read from bans.json at the start into the ledger, in // Load puts bans read from bans.json into the ledger, in place of the
// place of the bans it holds, in the order they started, so that a // bans it holds, in the order they started, so that a netblock whose last
// netblock whose last ban started latest counts as the most recently // ban started latest counts as the most recently seen. Each netblock is
// seen. A ban without a cause is an admin's, and gets CauseAdmin. Each // masked to its length, so that 203.0.113.9/24 is 203.0.113.0/24, and
// netblock is masked to its length, so that 203.0.113.9/24 is // each text in the notes is cut to 256 bytes. Past MaxBans the earliest
// 203.0.113.0/24, and each text in the notes is cut to 256 bytes. Past // bans are dropped, as when they are made.
// MaxBans the earliest bans whose cause is not CauseAdmin are dropped, as
// when they are made.
func (l *Ledger) Load(bans []Ban) { func (l *Ledger) Load(bans []Ban) {
l.mu.Lock()
defer l.mu.Unlock()
l.load(bans)
}
// LoadEdit is Load for an admin's edit of bans.json, taken in while
// smallwebwaf runs. Each ban in it whose cause is CauseAdmin, and which
// the ledger did not hold, with the same netblock and start, is one the
// admin made, and is counted among the bans made.
func (l *Ledger) LoadEdit(bans []Ban) {
l.mu.Lock()
defer l.mu.Unlock()
l.made[CauseAdmin] += l.load(bans)
}
// load does what Load describes, and returns how many of bans are bans
// whose cause is CauseAdmin that the ledger did not hold before.
func (l *Ledger) load(bans []Ban) int {
bans = slices.Clone(bans) bans = slices.Clone(bans)
added := 0
for i := range bans {
ban := &bans[i]
ban.Netblock = ban.Netblock.Masked()
ban.Notes.Request = ban.Notes.Request.cut()
if ban.Cause == "" {
ban.Cause = CauseAdmin
}
if ban.Cause == CauseAdmin && !l.holds(ban.Netblock, ban.Start) {
added++
}
}
slices.SortStableFunc(bans, func(a, b Ban) int { slices.SortStableFunc(bans, func(a, b Ban) int {
return a.Start.Compare(b.Start) return a.Start.Compare(b.Start)
}) })
l.mu.Lock()
defer l.mu.Unlock()
l.netblocks.Purge() l.netblocks.Purge()
l.held = 0 l.held = 0
l.v4Lengths, l.v6Lengths = nil, nil l.v4Lengths, l.v6Lengths = nil, nil
for _, ban := range bans { for _, ban := range bans {
ban.Netblock = ban.Netblock.Masked()
ban.Notes.Request = ban.Notes.Request.cut()
l.add(ban) l.add(ban)
} }
return added
}
// holds reports whether the ledger holds a ban on netblock that started
// at start.
func (l *Ledger) holds(netblock netip.Prefix, start time.Time) bool {
bans, found := l.netblocks.Peek(netblock)
return found && slices.ContainsFunc(*bans, func(ban Ban) bool {
return ban.Start.Equal(start)
})
}
// ban bans netblock at now for cause, with reason and notes, as
// BanForLimit and BanForAttack describe, and returns the ban.
func (l *Ledger) ban(
netblock netip.Prefix, now time.Time, cause, reason string, notes Notes,
) Ban {
l.mu.Lock()
defer l.mu.Unlock()
// held are the netblock's bans, none of them active.
var held []Ban
bans, found := l.netblocks.Get(netblock)
if found {
active := activeBan(*bans, now)
if active != nil {
return *active
}
held = *bans
notes.EarlierBans = earlierBans(held)
}
notes.Request = notes.Request.cut()
ban := Ban{Netblock: netblock, Start: now, Cause: cause, Reason: reason, Notes: notes}
if cause == CauseAttack {
ban.Expires = l.attackExpiry(held, now)
} else {
ban.Expires = l.limitExpiry(held, now)
}
l.add(ban)
l.made[cause]++
l.markChanged()
return ban
}
// earlierBans returns how many bans a netblock with the bans held, oldest
// first, has had, by cause: the first ban held counts the bans the
// netblock had before that one, since dropped to make room, and each ban
// held adds one.
func earlierBans(held []Ban) EarlierBans {
earlier := held[0].Notes.EarlierBans
for _, ban := range held {
switch ban.Cause {
case CauseLimit:
earlier.Limit++
case CauseAttack:
earlier.Attack++
case CauseAdmin:
earlier.Admin++
}
}
return earlier
}
// markChanged has Changed receive a value, unless one is waiting already.
func (l *Ledger) markChanged() {
select {
case l.changed <- struct{}{}:
default: // a value is waiting already
}
} }
// active returns the ban active at now on a netblock client is in, or // active returns the ban active at now on a netblock client is in, or
@@ -497,11 +356,9 @@ func (l *Ledger) active(client netip.Addr, now time.Time) *Ban {
} }
// add adds ban to its netblock's bans, after the last, and makes its // add adds ban to its netblock's bans, after the last, and makes its
// netblock the most recently seen. With MaxBans held, it drops one first, // netblock the most recently seen. With MaxBans held, it drops one first.
// unless ban's cause is CauseAdmin, which does not count toward MaxBans.
func (l *Ledger) add(ban Ban) { func (l *Ledger) add(ban Ban) {
counted := ban.Cause != CauseAdmin if l.held == l.rules.MaxBans {
if counted && l.held == l.rules.MaxBans {
l.dropOne() l.dropOne()
} }
@@ -514,10 +371,7 @@ func (l *Ledger) add(ban Ban) {
} }
*bans = append(*bans, ban) *bans = append(*bans, ban)
l.held++
if counted {
l.held++
}
lengths := &l.v6Lengths lengths := &l.v6Lengths
if ban.Netblock.Addr().Is4() { if ban.Netblock.Addr().Is4() {
@@ -529,24 +383,12 @@ func (l *Ledger) add(ban Ban) {
} }
} }
// limitExpiry returns when a ban for a broken limit made at now ends, or // expiry returns when a ban for a broken limit made at now ends, or zero
// zero when it is permanent. held are the netblock's bans, none of them // when it is permanent. last is the netblock's ban that ended last, or nil
// active, of which the one that ended last, other than a ban for a clear // when it has none.
// sign of attack or a lifted one, can make the new ban longer. A ban an func (l *Ledger) expiry(last *Ban, now time.Time) time.Time {
// admin adds to bans.json can start after another and end before it, so
// that one is looked for among them all.
func (l *Ledger) limitExpiry(held []Ban, now time.Time) time.Time {
length := l.rules.LimitBanDuration length := l.rules.LimitBanDuration
var last *Ban
for i, ban := range held {
if ban.Cause != CauseAttack && ban.Lifted.IsZero() &&
(last == nil || ban.Expires.After(last.Expires)) {
last = &held[i]
}
}
if last != nil && now.Sub(last.Expires) <= l.rules.LimitBanRepeatWindow { if last != nil && now.Sub(last.Expires) <= l.rules.LimitBanRepeatWindow {
lastLength := last.Expires.Sub(last.Start) lastLength := last.Expires.Sub(last.Start)
// This is repeatFactor * lastLength > MaxBanDuration, written so // This is repeatFactor * lastLength > MaxBanDuration, written so
@@ -565,53 +407,17 @@ func (l *Ledger) limitExpiry(held []Ban, now time.Time) time.Time {
return now.Add(length) return now.Add(length)
} }
// attackExpiry returns when a ban for a clear sign of attack made at now // dropOne drops the earliest ban of the netblock that has gone longest
// ends. held are the netblock's bans, none of them active: if one of them // without a request, and the netblock with it if that was its only ban.
// is for a clear sign of attack too, and was not lifted, the new ban is
// permanent, and its end zero; otherwise it ends AttackBanDuration later.
func (l *Ledger) attackExpiry(held []Ban, now time.Time) time.Time {
for _, ban := range held {
if ban.Cause == CauseAttack && ban.Lifted.IsZero() {
return time.Time{}
}
}
return now.Add(l.rules.AttackBanDuration)
}
// dropOne drops the earliest ban whose cause is not CauseAdmin of the
// netblock that has gone longest without a request, of those that hold
// such a ban, and the netblock with it if that was its only ban. It is
// called with at least one such ban held. It looks at each netblock once
// at most, and drops nothing when none holds such a ban.
func (l *Ledger) dropOne() { func (l *Ledger) dropOne() {
for range l.netblocks.Len() { netblock, bans, _ := l.netblocks.GetOldest()
netblock, bans, _ := l.netblocks.GetOldest() if len(*bans) == 1 {
l.netblocks.Remove(netblock)
i := slices.IndexFunc(*bans, func(ban Ban) bool { } else {
return ban.Cause != CauseAdmin *bans = slices.Delete(*bans, 0, 1)
})
if i < 0 {
// Its bans are all an admin's, and never dropped. Get makes
// it the most recently seen, so that the next netblock is
// looked at; when it was seen matters only for dropping a
// ban, and a ban added to it makes it the most recently seen
// anyway.
l.netblocks.Get(netblock)
continue
}
if len(*bans) == 1 {
l.netblocks.Remove(netblock)
} else {
*bans = slices.Delete(*bans, i, i+1)
}
l.held--
return
} }
l.held--
} }
// cut returns r with each text cut to maxTextBytes and copied, so that // cut returns r with each text cut to maxTextBytes and copied, so that
+7 -103
View File
@@ -24,9 +24,8 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
ban := ledger.BanForLimit(netblock, now, bans.Notes{}) ban := ledger.BanForLimit(netblock, now, bans.Notes{})
length := time.Duration(hours) * time.Hour length := time.Duration(hours) * time.Hour
if !ban.Expires.Equal(now.Add(length)) || if !ban.Expires.Equal(now.Add(length)) || ban.Notes.EarlierBans != i {
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: i}) { t.Fatalf("ban %d lasts %s with %d earlier bans, want %d hours and %d",
t.Fatalf("ban %d lasts %s with earlier bans %+v, want %d hours and %d for a limit",
i+1, ban.Expires.Sub(now), ban.Notes.EarlierBans, hours, i) i+1, ban.Expires.Sub(now), ban.Notes.EarlierBans, hours, i)
} }
@@ -67,9 +66,8 @@ func TestRepeatWindowRunsOut(t *testing.T) {
first := ledger.BanForLimit(netblock, midnight(), bans.Notes{}) first := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second := ledger.BanForLimit(netblock, first.Expires.Add(tc.gap), bans.Notes{}) second := ledger.BanForLimit(netblock, first.Expires.Add(tc.gap), bans.Notes{})
if second.Expires.Sub(second.Start) != tc.want || if second.Expires.Sub(second.Start) != tc.want || second.Notes.EarlierBans != 1 {
second.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) { t.Errorf("second ban lasts %s with %d earlier bans, want %s and 1",
t.Errorf("second ban lasts %s with earlier bans %+v, want %s and 1 for a limit",
second.Expires.Sub(second.Start), second.Notes.EarlierBans, tc.want) second.Expires.Sub(second.Start), second.Notes.EarlierBans, tc.want)
} }
}) })
@@ -237,102 +235,9 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
second := ledger.BanForLimit(netblock, first.Expires, bans.Notes{}) second := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
held := ledger.Bans(netblock) held := ledger.Bans(netblock)
if len(held) != 1 || held[0] != second || if len(held) != 1 || held[0] != second || held[0].Notes.EarlierBans != 1 {
held[0].Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) { t.Errorf("the ledger holds %+v, want only the second ban, with 1 earlier ban",
t.Errorf("the ledger holds %+v, want only the second ban, "+ held)
"with 1 earlier ban for a limit", held)
}
}
func TestRequestDuringAnAttackBanMakesItPermanent(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
notes := bans.Notes{RuleID: "env-file", Target: "path"}
ban := ledger.BanForAttack(netblock, midnight(), notes)
if !ban.Expires.Equal(midnight().Add(7*day)) || ban.Cause != bans.CauseAttack ||
ban.Notes.RuleID != "env-file" || ledger.Made(bans.CauseAttack) != 1 ||
ledger.Made(bans.CauseLimit) != 0 {
t.Fatalf("the ban is %+v, with %d made for an attack and %d for a limit, "+
"want one for an attack, of seven days", ban,
ledger.Made(bans.CauseAttack), ledger.Made(bans.CauseLimit))
}
wantChanged(t, ledger, true)
// In observe mode the ban refuses nothing, and stays as it is.
got, _ := ledger.Find(netblock.Addr(), midnight().Add(time.Hour))
if got.Permanent() {
t.Fatal("a request found under the ban made it permanent")
}
// A request it refuses makes it permanent, and bans.json due.
got, _ = ledger.Check(netblock.Addr(), midnight().Add(time.Hour))
if !got.Permanent() || !ledger.Bans(netblock)[0].Permanent() {
t.Fatalf("after a request during the ban, it is %+v, want it permanent", got)
}
wantChanged(t, ledger, true)
_, banned := ledger.Check(netblock.Addr(), midnight().Add(100*365*day))
if !banned {
t.Error("the permanent ban ended")
}
}
func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
// A ban for a broken limit before does not count.
first := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second := ledger.BanForAttack(netblock, first.Expires, bans.Notes{})
if second.Expires.Sub(second.Start) != 7*day {
t.Fatalf("the first ban for an attack lasts %s, want 7 days",
second.Expires.Sub(second.Start))
}
// Once that has run out without a request, the netblock is served, and
// its next clear sign of attack bans it for good.
_, banned := ledger.Check(netblock.Addr(), second.Expires)
if banned {
t.Fatal("the ban did not end")
}
// Its notes show the earlier ban for an attack that makes it permanent,
// beside the one for a limit.
third := ledger.BanForAttack(netblock, second.Expires.Add(30*day), bans.Notes{})
if !third.Permanent() ||
third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Attack: 1}) {
t.Errorf("the next ban for an attack is %+v, want a permanent one, "+
"with 1 earlier ban for a limit and 1 for an attack", third)
}
}
func TestAttackBanDoesNotLengthenTheNextBanForALimit(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
// Three times the seven days would be permanent; a limit broken as the
// ban for an attack ends bans for an hour, as a first broken limit does.
attack := ledger.BanForAttack(netblock, midnight(), bans.Notes{})
limit := ledger.BanForLimit(netblock, attack.Expires, bans.Notes{})
if limit.Expires.Sub(limit.Start) != time.Hour || limit.Cause != bans.CauseLimit {
t.Errorf("the ban for a limit is %+v, want one of an hour", limit)
}
// And a request during the ban for a limit leaves it as it is.
got, _ := ledger.Check(netblock.Addr(), limit.Start)
if got.Permanent() {
t.Error("a request during a ban for a limit made it permanent")
} }
} }
@@ -364,7 +269,6 @@ func defaultRules() bans.Rules {
LimitBanDuration: time.Hour, LimitBanDuration: time.Hour,
LimitBanRepeatWindow: day, LimitBanRepeatWindow: day,
MaxBanDuration: 7 * day, MaxBanDuration: 7 * day,
AttackBanDuration: 7 * day,
MaxBans: 5000, MaxBans: 5000,
} }
} }
+10 -27
View File
@@ -82,9 +82,8 @@ func TestLoadedBansCarryOn(t *testing.T) {
} }
again := after.BanForLimit(netblock, ban.Expires, bans.Notes{}) again := after.BanForLimit(netblock, ban.Expires, bans.Notes{})
if again.Expires.Sub(again.Start) != 3*time.Hour || if again.Expires.Sub(again.Start) != 3*time.Hour || again.Notes.EarlierBans != 1 {
again.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) { t.Errorf("the next ban lasts %s with %d earlier bans, want 3h and 1",
t.Errorf("the next ban lasts %s with earlier bans %+v, want 3h and 1 for a limit",
again.Expires.Sub(again.Start), again.Notes.EarlierBans) again.Expires.Sub(again.Start), again.Notes.EarlierBans)
} }
} }
@@ -173,14 +172,13 @@ func TestNextBanWorkedOutFromTheBanThatEndedLast(t *testing.T) {
t.Parallel() t.Parallel()
// A 9-hour ban smallwebwaf made, the third in a row, and an admin's // A 9-hour ban smallwebwaf made, the third in a row, and an admin's
// 1-hour ban added to bans.json over it, with no cause and no notes. // 1-hour ban added to bans.json over it, with no notes.
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
nineHours := bans.Ban{ nineHours := bans.Ban{
Netblock: netblock, Netblock: netblock,
Start: midnight(), Start: midnight(),
Expires: midnight().Add(9 * time.Hour), Expires: midnight().Add(9 * time.Hour),
Cause: bans.CauseLimit, Notes: bans.Notes{EarlierBans: 2},
Notes: bans.Notes{EarlierBans: bans.EarlierBans{Limit: 2}},
} }
admins := bans.Ban{ admins := bans.Ban{
Netblock: netblock, Netblock: netblock,
@@ -193,12 +191,10 @@ func TestNextBanWorkedOutFromTheBanThatEndedLast(t *testing.T) {
// Once both have ended, a limit broken within the repeat window bans // Once both have ended, a limit broken within the repeat window bans
// for three times the 9 hours, and the notes count the two bans // for three times the 9 hours, and the notes count the two bans
// before the 9-hour one and it, for a limit, and the admin's. // before the 9-hour one, it, and the admin's.
ban := ledger.BanForLimit(netblock, nineHours.Expires.Add(time.Hour), bans.Notes{}) ban := ledger.BanForLimit(netblock, nineHours.Expires.Add(time.Hour), bans.Notes{})
if ban.Expires.Sub(ban.Start) != 27*time.Hour || if ban.Expires.Sub(ban.Start) != 27*time.Hour || ban.Notes.EarlierBans != 4 {
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 3, Admin: 1}) { t.Errorf("the next ban lasts %s with %d earlier bans, want 27h and 4",
t.Errorf("the next ban lasts %s with earlier bans %+v, "+
"want 27h, 3 for a limit and 1 an admin's",
ban.Expires.Sub(ban.Start), ban.Notes.EarlierBans) ban.Expires.Sub(ban.Start), ban.Notes.EarlierBans)
} }
} }
@@ -207,15 +203,10 @@ func TestLoadKeepsAtMostMaxBansDroppingTheEarliest(t *testing.T) {
t.Parallel() t.Parallel()
// bans.json lists the bans by netblock, not in the order they began. // bans.json lists the bans by netblock, not in the order they began.
later := bans.Ban{ later := bans.Ban{Netblock: netip.MustParsePrefix("203.0.113.1/32"), Start: midnight()}
Netblock: netip.MustParsePrefix("203.0.113.1/32"),
Start: midnight(),
Cause: bans.CauseLimit,
}
earlier := bans.Ban{ earlier := bans.Ban{
Netblock: netip.MustParsePrefix("203.0.113.2/32"), Netblock: netip.MustParsePrefix("203.0.113.2/32"),
Start: midnight().Add(-time.Hour), Start: midnight().Add(-time.Hour),
Cause: bans.CauseLimit,
} }
rules := defaultRules() rules := defaultRules()
@@ -237,17 +228,9 @@ func TestLoadReplacesTheBansHeld(t *testing.T) {
rules := defaultRules() rules := defaultRules()
rules.MaxBans = 3 rules.MaxBans = 3
ledger := bans.New(rules) ledger := bans.New(rules)
kept := bans.Ban{ kept := bans.Ban{Netblock: netip.MustParsePrefix("2001:db8::/64"), Start: midnight()}
Netblock: netip.MustParsePrefix("2001:db8::/64"),
Start: midnight(),
Cause: bans.CauseLimit,
}
ledger.Load([]bans.Ban{ ledger.Load([]bans.Ban{
{ {Netblock: netip.MustParsePrefix("203.0.113.0/24"), Start: midnight()},
Netblock: netip.MustParsePrefix("203.0.113.0/24"),
Start: midnight(),
Cause: bans.CauseLimit,
},
kept, kept,
}) })
+16 -340
View File
@@ -1,11 +1,9 @@
// Package config reads smallwebwaf's settings. Every setting is an // Package config reads smallwebwaf's settings. Every setting is an
// environment variable whose name starts with SWWAF_, or a file such a // environment variable whose name starts with SWWAF_, every setting has a
// variable names, every setting has a default, and this package is the // default, and this package is the one place they are read.
// one place they are read.
package config package config
import ( import (
"crypto/x509"
"errors" "errors"
"fmt" "fmt"
"log/slog" "log/slog"
@@ -14,15 +12,12 @@ import (
"net/http" "net/http"
"net/netip" "net/netip"
"net/url" "net/url"
"os"
"path/filepath" "path/filepath"
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
"time" "time"
"unicode/utf8" "unicode/utf8"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
) )
// Config is smallwebwaf's settings. A timeout, size or rate limit of zero // Config is smallwebwaf's settings. A timeout, size or rate limit of zero
@@ -32,14 +27,10 @@ type Config struct {
ListenAddr string ListenAddr string
// UpstreamURL is the app (SWWAF_UPSTREAM_URL). // UpstreamURL is the app (SWWAF_UPSTREAM_URL).
UpstreamURL *url.URL UpstreamURL *url.URL
// InstanceName is the name each request log line gives as instance
// (SWWAF_INSTANCE_NAME), by default the host's name, which docker sets
// to the first 12 characters of the container's id.
InstanceName string
// Observe is true in observe mode, when SWWAF_MODE is observe rather // Observe is true in observe mode, when SWWAF_MODE is observe rather
// than enforce: a request that SWWAF_DENY_NETS, a ban, the country // than enforce: a request that SWWAF_DENY_NETS, a ban, the country
// lists, a rate limit or a rule would refuse is passed to the app // lists or a rate limit would refuse is passed to the app instead, and
// instead, and no ban is made. // no ban is made.
Observe bool Observe bool
// TrustedProxies are the netblocks whose X-Forwarded-For is // TrustedProxies are the netblocks whose X-Forwarded-For is
// believed (SWWAF_TRUSTED_PROXIES). // believed (SWWAF_TRUSTED_PROXIES).
@@ -98,8 +89,7 @@ type Config struct {
// BanResponse is the status a refused client is answered with, 403 // BanResponse is the status a refused client is answered with, 403
// or 429, or 0 to close the connection without an answer // or 429, or 0 to close the connection without an answer
// (SWWAF_BAN_RESPONSE). It answers a banned client, a request that // (SWWAF_BAN_RESPONSE). It answers a banned client, a request that
// breaks a rate limit or matches a ban rule, SWWAF_DENY_NETS and the // breaks a rate limit, SWWAF_DENY_NETS and the country lists.
// country lists.
BanResponse int BanResponse int
// LimitBanDuration is the ban for a first broken rate limit // LimitBanDuration is the ban for a first broken rate limit
// (SWWAF_LIMIT_BAN_DURATION). A limit broken again within // (SWWAF_LIMIT_BAN_DURATION). A limit broken again within
@@ -110,9 +100,6 @@ type Config struct {
LimitBanDuration time.Duration LimitBanDuration time.Duration
LimitBanRepeatWindow time.Duration LimitBanRepeatWindow time.Duration
MaxBanDuration time.Duration MaxBanDuration time.Duration
// AttackBanDuration is the ban for a first clear sign of attack
// (SWWAF_ATTACK_BAN_DURATION). It cannot be off.
AttackBanDuration time.Duration
// MaxBans is the most bans held (SWWAF_MAX_BANS). // MaxBans is the most bans held (SWWAF_MAX_BANS).
MaxBans int MaxBans int
// BanScopeV4Prefix is the length of the netblock around an IPv4 // BanScopeV4Prefix is the length of the netblock around an IPv4
@@ -126,37 +113,15 @@ type Config struct {
StateDir string StateDir string
StateWriteDelay time.Duration StateWriteDelay time.Duration
StateCounterInterval time.Duration StateCounterInterval time.Duration
// LogRequestHeaders are the request headers whose values the request
// log gives, in lower case (SWWAF_LOG_REQUEST_HEADERS).
LogRequestHeaders []string
// MetricsToken is the bearer token a scraper sends for the metrics // MetricsToken is the bearer token a scraper sends for the metrics
// (SWWAF_METRICS_TOKEN), "" while it is unset and the metrics are off. // (SWWAF_METRICS_TOKEN), "" while it is unset and the metrics are off.
// MetricsTopN is how many countries get series of their own in the // MetricsTopN is how many countries get series of their own in the
// metrics (SWWAF_METRICS_TOP_N). // metrics (SWWAF_METRICS_TOP_N).
MetricsToken string MetricsToken string
MetricsTopN int MetricsTopN int
// RulesDir is the directory of the rule files (SWWAF_RULES_DIR), read
// unless RulesEnabled is false (SWWAF_RULES_ENABLED).
RulesDir string
RulesEnabled bool
// LogRemoteURL is where every line on stdout is also sent
// (SWWAF_LOG_REMOTE_URL), nil while it is unset and nothing is sent.
// LogRemoteTLSCAs are the certificates a syslog+tls endpoint's
// certificate must chain to (SWWAF_LOG_REMOTE_TLS_CA_FILE), nil while
// it is unset and the host's own are used. LogRemoteBuffer is the most
// lines held while they wait to be sent (SWWAF_LOG_REMOTE_BUFFER).
// LogRemoteFacility is the number of the syslog facility
// (SWWAF_LOG_REMOTE_FACILITY), and LogRemoteAppName the APP-NAME
// (SWWAF_LOG_REMOTE_APP_NAME, by default InstanceName), of the records
// the lines are sent in.
LogRemoteURL *url.URL
LogRemoteTLSCAs *x509.CertPool
LogRemoteBuffer int
LogRemoteFacility int
LogRemoteAppName string
// settings are the values read, as given or by default, and the // settings are the values read, as given or by default, for the
// files they were read from, for the log line at start. // log line at start.
settings []slog.Attr settings []slog.Attr
} }
@@ -174,10 +139,6 @@ const (
minTokenLength = 32 minTokenLength = 32
// masked is what the log shows for a token that is set. // masked is what the log shows for a token that is set.
masked = "********" masked = "********"
// defaultListenAddr and defaultUpstreamURL are the defaults of
// SWWAF_LISTEN_ADDR and SWWAF_UPSTREAM_URL.
defaultListenAddr = ":8080"
defaultUpstreamURL = "http://127.0.0.1:8081"
) )
var ( var (
@@ -198,11 +159,6 @@ var (
"such as http://127.0.0.1:8081") "such as http://127.0.0.1:8081")
errNotCountry = errors.New( errNotCountry = errors.New(
"is not a two-letter country code such as de or kp") "is not a two-letter country code such as de or kp")
errNotHeaderName = errors.New(
"is not a header name such as accept-language")
errHeaderTakenOut = errors.New(
"is taken out of every request by Go's HTTP server, so it can never " +
"be logged")
errOnBothLists = errors.New("is in SWWAF_DENIED_COUNTRIES too") errOnBothLists = errors.New("is in SWWAF_DENIED_COUNTRIES too")
errNotOver4K = errors.New("is not a size of more than 4K, such as 32K") errNotOver4K = errors.New("is not a size of more than 4K, such as 32K")
errNotDurationAboveZero = errors.New( errNotDurationAboveZero = errors.New(
@@ -218,29 +174,16 @@ var (
errNotMode = errors.New("is not enforce or observe") errNotMode = errors.New("is not enforce or observe")
errNotPathPrefix = errors.New( errNotPathPrefix = errors.New(
"is not a path prefix starting with /, such as /assets/") "is not a path prefix starting with /, such as /assets/")
errNotBoolean = errors.New("is not true or false")
errNotLogRemoteURL = errors.New(
"is not syslog+udp, syslog+tcp or syslog+tls with a host and a port, " +
"and nothing more, such as syslog+tls://logs.example:6514")
errNoCertificate = errors.New("holds no PEM certificate")
errNotFacility = errors.New("is not a syslog facility such as local0 or daemon")
errNotAppName = errors.New(
"is not 1 to 48 printable ASCII characters without a space, such as gitea")
errSetTwice = errors.New("set only one of them")
) )
// FromEnvironment reads the settings with lookupEnv, normally // FromEnvironment reads the settings with lookupEnv, normally
// os.LookupEnv. A setting may instead be given as a file: the variable // os.LookupEnv. A setting that is not set takes its default. A setting
// named by the setting's name with _FILE added names the file, which is // that is set but invalid is an error that names it.
// read now (see lookup). A setting that is not set takes its default. A
// setting that is set but invalid is an error that names it.
func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) { func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
env := &environment{lookupEnv: lookupEnv} env := &environment{lookupEnv: lookupEnv}
hostname, _ := os.Hostname() // "" when the host has no name to give
cfg := &Config{ cfg := &Config{
ListenAddr: env.address("SWWAF_LISTEN_ADDR", defaultListenAddr), ListenAddr: env.address("SWWAF_LISTEN_ADDR", ":8080"),
UpstreamURL: env.appURL("SWWAF_UPSTREAM_URL", defaultUpstreamURL), UpstreamURL: env.appURL("SWWAF_UPSTREAM_URL", "http://127.0.0.1:8081"),
InstanceName: env.value("SWWAF_INSTANCE_NAME", hostname),
Observe: env.observe("SWWAF_MODE", "enforce"), Observe: env.observe("SWWAF_MODE", "enforce"),
TrustedProxies: env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges), TrustedProxies: env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges),
ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"), ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"),
@@ -266,27 +209,15 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
LimitBanDuration: env.durationNotOff("SWWAF_LIMIT_BAN_DURATION", "1h"), LimitBanDuration: env.durationNotOff("SWWAF_LIMIT_BAN_DURATION", "1h"),
LimitBanRepeatWindow: env.durationNotOff("SWWAF_LIMIT_BAN_REPEAT_WINDOW", "24h"), LimitBanRepeatWindow: env.durationNotOff("SWWAF_LIMIT_BAN_REPEAT_WINDOW", "24h"),
MaxBanDuration: env.durationNotOff("SWWAF_MAX_BAN_DURATION", "7d"), MaxBanDuration: env.durationNotOff("SWWAF_MAX_BAN_DURATION", "7d"),
AttackBanDuration: env.durationNotOff("SWWAF_ATTACK_BAN_DURATION", "7d"),
MaxBans: env.numberNotOff("SWWAF_MAX_BANS", "5000"), MaxBans: env.numberNotOff("SWWAF_MAX_BANS", "5000"),
BanScopeV4Prefix: env.v4Prefix("SWWAF_BAN_SCOPE_V4_PREFIX", "32"), BanScopeV4Prefix: env.v4Prefix("SWWAF_BAN_SCOPE_V4_PREFIX", "32"),
StateDir: env.absolutePath("SWWAF_STATE_DIR", "/var/lib/smallwebwaf"), StateDir: env.absolutePath("SWWAF_STATE_DIR", "/var/lib/smallwebwaf"),
StateWriteDelay: env.durationNotOff("SWWAF_STATE_WRITE_DELAY", "10s"), StateWriteDelay: env.durationNotOff("SWWAF_STATE_WRITE_DELAY", "10s"),
StateCounterInterval: env.durationNotOff("SWWAF_STATE_COUNTER_INTERVAL", "15m"), StateCounterInterval: env.durationNotOff("SWWAF_STATE_COUNTER_INTERVAL", "15m"),
LogRequestHeaders: env.headerNames("SWWAF_LOG_REQUEST_HEADERS", MetricsToken: env.token("SWWAF_METRICS_TOKEN"),
"accept,accept-language,accept-encoding,content-type,origin,range"), MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
MetricsToken: env.token("SWWAF_METRICS_TOKEN"),
MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
RulesDir: env.value("SWWAF_RULES_DIR", "/etc/smallwebwaf/rules.d"),
RulesEnabled: env.boolean("SWWAF_RULES_ENABLED", "true"),
LogRemoteURL: env.logRemoteURL("SWWAF_LOG_REMOTE_URL"),
LogRemoteTLSCAs: env.certificates("SWWAF_LOG_REMOTE_TLS_CA_FILE"),
LogRemoteBuffer: env.numberNotOff("SWWAF_LOG_REMOTE_BUFFER", "10000"),
LogRemoteFacility: env.facility("SWWAF_LOG_REMOTE_FACILITY", "local0"),
} }
cfg.LogRemoteAppName = env.appName("SWWAF_LOG_REMOTE_APP_NAME",
cfg.InstanceName, cfg.LogRemoteURL != nil)
for _, country := range cfg.ExclusivelyAllowedCountries { for _, country := range cfg.ExclusivelyAllowedCountries {
if slices.Contains(cfg.DeniedCountries, country) { if slices.Contains(cfg.DeniedCountries, country) {
env.check("SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", env.check("SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES",
@@ -303,24 +234,6 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
return cfg, nil return cfg, nil
} }
// ListenAddrAndUpstreamURL reads only SWWAF_LISTEN_ADDR and
// SWWAF_UPSTREAM_URL, either of which may be given as a file, as
// FromEnvironment does. The health check needs no other setting, so it
// reads no other, nor a file that another names.
func ListenAddrAndUpstreamURL(
lookupEnv func(string) (string, bool),
) (string, *url.URL, error) {
env := &environment{lookupEnv: lookupEnv}
listenAddr := env.address("SWWAF_LISTEN_ADDR", defaultListenAddr)
upstreamURL := env.appURL("SWWAF_UPSTREAM_URL", defaultUpstreamURL)
if env.err != nil {
return "", nil, env.err
}
return listenAddr, upstreamURL, nil
}
// privateRanges are the private address ranges, the default trusted // privateRanges are the private address ranges, the default trusted
// proxies. // proxies.
const privateRanges = "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16" const privateRanges = "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
@@ -342,8 +255,8 @@ type environment struct {
// value returns a setting's value, or its default when it is not set, // value returns a setting's value, or its default when it is not set,
// and notes it for the log. // and notes it for the log.
func (e *environment) value(name, defaultValue string) string { func (e *environment) value(name, defaultValue string) string {
value, set := e.lookup(name) value, ok := e.lookupEnv(name)
if !set { if !ok {
value = defaultValue value = defaultValue
} }
@@ -352,37 +265,6 @@ func (e *environment) value(name, defaultValue string) string {
return value return value
} }
// lookup returns a setting's value and whether it is set: the value of the
// variable name, or the contents of the file that the variable name_FILE
// names, less one newline at their end. It notes that file's path for the
// log. Both variables set, or a file that cannot be read, is an error.
func (e *environment) lookup(name string) (string, bool) {
value, set := e.lookupEnv(name)
fileName := name + "_FILE"
path, inFile := e.lookupEnv(fileName)
if !inFile {
return value, set
}
if set {
e.check(name, fmt.Errorf("is set, and so is %s; %w", fileName, errSetTwice))
return value, set
}
e.settings = append(e.settings, slog.String(fileName, path))
contents, err := os.ReadFile(path) //nolint:gosec // a file the admin names
if err != nil {
e.check(fileName, fmt.Errorf("cannot be read: %w", err))
return "", false
}
return strings.TrimSuffix(string(contents), "\n"), true
}
// check keeps the first error, naming the setting it is about. // check keeps the first error, naming the setting it is about.
func (e *environment) check(name string, err error) { func (e *environment) check(name string, err error) {
if err != nil && e.err == nil { if err != nil && e.err == nil {
@@ -417,16 +299,6 @@ func (e *environment) observe(name, defaultValue string) bool {
return mode == "observe" return mode == "observe"
} }
// boolean reads a setting that is true or false.
func (e *environment) boolean(name, defaultValue string) bool {
value := e.value(name, defaultValue)
if value != "true" && value != "false" {
e.check(name, fmt.Errorf("%q %w", value, errNotBoolean))
}
return value == "true"
}
// netblocks reads a setting that is a list of netblocks. // netblocks reads a setting that is a list of netblocks.
func (e *environment) netblocks(name, defaultValue string) []netip.Prefix { func (e *environment) netblocks(name, defaultValue string) []netip.Prefix {
netblocks, err := parseNetblocks(e.value(name, defaultValue)) netblocks, err := parseNetblocks(e.value(name, defaultValue))
@@ -484,15 +356,6 @@ func (e *environment) countries(name, defaultValue string) []string {
return countries return countries
} }
// headerNames reads a setting that is a list of header names, and
// returns them in lower case.
func (e *environment) headerNames(name, defaultValue string) []string {
headers, err := parseHeaderNames(e.value(name, defaultValue))
e.check(name, err)
return headers
}
// durationNotOff reads a setting that is a duration and, unlike a // durationNotOff reads a setting that is a duration and, unlike a
// timeout, cannot be off. // timeout, cannot be off.
func (e *environment) durationNotOff(name, defaultValue string) time.Duration { func (e *environment) durationNotOff(name, defaultValue string) time.Duration {
@@ -541,7 +404,7 @@ func (e *environment) absolutePath(name, defaultValue string) string {
// switches off what it guards; set, it must be at least minTokenLength // switches off what it guards; set, it must be at least minTokenLength
// characters. Neither the log nor an error shows its value. // characters. Neither the log nor an error shows its value.
func (e *environment) token(name string) string { func (e *environment) token(name string) string {
value, set := e.lookup(name) value, set := e.lookupEnv(name)
if !set { if !set {
e.settings = append(e.settings, slog.String(name, "")) e.settings = append(e.settings, slog.String(name, ""))
@@ -557,80 +420,6 @@ func (e *environment) token(name string) string {
return value return value
} }
// logRemoteURL reads the setting that is where every log line is also
// sent. Unset or empty, it is nil, and nothing is sent.
func (e *environment) logRemoteURL(name string) *url.URL {
value := e.value(name, "")
if value == "" {
return nil
}
remote, err := parseLogRemoteURL(value)
e.check(name, err)
return remote
}
// certificates reads a setting that is the path of a file of PEM
// certificates. Unset or empty, it is nil. Its value names a file
// already, so, unlike the other settings, it has no _FILE form.
func (e *environment) certificates(name string) *x509.CertPool {
path, _ := e.lookupEnv(name)
e.settings = append(e.settings, slog.String(name, path))
if path == "" {
return nil
}
pem, err := os.ReadFile(path) //nolint:gosec // a file the admin names
if err != nil {
e.check(name, fmt.Errorf("cannot be read: %w", err))
return nil
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
e.check(name, fmt.Errorf("%q %w", path, errNoCertificate))
return nil
}
return pool
}
// facility reads a setting that is a syslog facility, and returns its
// number.
func (e *environment) facility(name, defaultValue string) int {
number, err := parseFacility(e.value(name, defaultValue))
e.check(name, err)
return number
}
// appName reads the setting that is the APP-NAME of the records the log
// lines are sent in, by default the instance name. Its value is checked
// when it is set, and, while lines are sent, when it is the instance name.
func (e *environment) appName(name, instanceName string, sending bool) string {
value, set := e.lookup(name)
if !set {
value = instanceName
}
e.settings = append(e.settings, slog.String(name, value))
switch {
case isAppName(value):
case set:
e.check(name, fmt.Errorf("%q %w", value, errNotAppName))
case sending:
e.check(name, fmt.Errorf("is unset, and SWWAF_INSTANCE_NAME %q, its default, %w",
value, errNotAppName))
}
return value
}
// parseDuration reads a duration in Go's syntax, such as 90s or 15m, a // parseDuration reads a duration in Go's syntax, such as 90s or 15m, a
// whole number of days such as 7d, or off. // whole number of days such as 7d, or off.
func parseDuration(value string) (time.Duration, error) { func parseDuration(value string) (time.Duration, error) {
@@ -910,58 +699,6 @@ func parseCountries(value string) ([]string, error) {
return countries, nil return countries, nil
} }
// headerNameChars are the characters RFC 9110 allows in a header name:
// letters, digits and these marks.
const headerNameChars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" +
"0123456789!#$%&'*+-.^_`|~"
// IsHeaderName reports whether name can be a header name: one or more of
// the characters RFC 9110 allows in one.
func IsHeaderName(name string) bool {
if name == "" {
return false
}
for _, char := range name {
if !strings.ContainsRune(headerNameChars, char) {
return false
}
}
return true
}
// parseHeaderNames reads a comma-separated list of header names in either
// case, and returns them in lower case. Host and Transfer-Encoding are
// refused: Go's HTTP server takes them out of the request's headers.
func parseHeaderNames(value string) ([]string, error) {
items, err := parseList(value)
if err != nil {
return nil, err
}
headers := make([]string, 0, len(items))
for _, item := range items {
if !IsHeaderName(item) {
return nil, fmt.Errorf("%q %w", item, errNotHeaderName)
}
header := strings.ToLower(item)
switch header {
case "host":
return nil, fmt.Errorf("%q %w; the request's host is the field host",
item, errHeaderTakenOut)
case "transfer-encoding":
return nil, fmt.Errorf("%q %w", item, errHeaderTakenOut)
}
headers = append(headers, header)
}
return headers, nil
}
// parseListenAddr checks an address to listen on: an optional host and a // parseListenAddr checks an address to listen on: an optional host and a
// port number. // port number.
func parseListenAddr(value string) (string, error) { func parseListenAddr(value string) (string, error) {
@@ -1004,64 +741,3 @@ func parseUpstreamURL(value string) (*url.URL, error) {
return upstream, nil return upstream, nil
} }
// parseLogRemoteURL reads where every log line is also sent:
// syslog+udp, syslog+tcp or syslog+tls, a host and a port from 1 to
// 65535, and nothing else.
func parseLogRemoteURL(value string) (*url.URL, error) {
remote, err := url.Parse(value)
if err != nil {
return nil, fmt.Errorf("%q %w", value, errNotLogRemoteURL)
}
schemes := []string{remotelog.SchemeUDP, remotelog.SchemeTCP, remotelog.SchemeTLS}
port, err := strconv.ParseUint(remote.Port(), 10, 16)
onlySchemeHostAndPort := slices.Contains(schemes, remote.Scheme) &&
remote.Hostname() != "" && err == nil && port != 0 &&
remote.User == nil && remote.Opaque == "" &&
(remote.Path == "" || remote.Path == "/") &&
remote.RawQuery == "" && remote.Fragment == ""
if !onlySchemeHostAndPort {
return nil, fmt.Errorf("%q %w", value, errNotLogRemoteURL)
}
return remote, nil
}
// parseFacility reads the name of a syslog facility, and returns its
// number, as RFC 5424 numbers them.
func parseFacility(value string) (int, error) {
//nolint:mnd // the facilities' numbers in RFC 5424
number, known := map[string]int{
"kern": 0, "user": 1, "mail": 2, "daemon": 3, "auth": 4, "syslog": 5,
"lpr": 6, "news": 7, "uucp": 8, "cron": 9, "authpriv": 10, "ftp": 11,
"local0": 16, "local1": 17, "local2": 18, "local3": 19,
"local4": 20, "local5": 21, "local6": 22, "local7": 23,
}[value]
if !known {
return 0, fmt.Errorf("%q %w", value, errNotFacility)
}
return number, nil
}
// appNameMaxLength is the most characters RFC 5424 allows in an
// APP-NAME.
const appNameMaxLength = 48
// isAppName reports whether value can be an APP-NAME: 1 to
// appNameMaxLength printable ASCII characters, none of them a space.
func isAppName(value string) bool {
if value == "" || len(value) > appNameMaxLength {
return false
}
for _, char := range []byte(value) {
if char < '!' || char > '~' {
return false
}
}
return true
}
+14 -446
View File
@@ -2,13 +2,10 @@ package config_test
import ( import (
"bytes" "bytes"
"crypto/x509"
"encoding/json" "encoding/json"
"log/slog" "log/slog"
"maps" "maps"
"net/netip" "net/netip"
"os"
"path/filepath"
"slices" "slices"
"strings" "strings"
"testing" "testing"
@@ -44,7 +41,6 @@ const (
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION" limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW" limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
maxBanDuration = "SWWAF_MAX_BAN_DURATION" maxBanDuration = "SWWAF_MAX_BAN_DURATION"
attackBanDuration = "SWWAF_ATTACK_BAN_DURATION"
maxBans = "SWWAF_MAX_BANS" maxBans = "SWWAF_MAX_BANS"
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX" banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
stateDir = "SWWAF_STATE_DIR" stateDir = "SWWAF_STATE_DIR"
@@ -52,45 +48,11 @@ const (
stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL" stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL"
metricsToken = "SWWAF_METRICS_TOKEN" //nolint:gosec // the setting's name metricsToken = "SWWAF_METRICS_TOKEN" //nolint:gosec // the setting's name
metricsTopN = "SWWAF_METRICS_TOP_N" metricsTopN = "SWWAF_METRICS_TOP_N"
instanceName = "SWWAF_INSTANCE_NAME"
logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS"
rulesDir = "SWWAF_RULES_DIR"
rulesEnabled = "SWWAF_RULES_ENABLED"
logRemoteURL = "SWWAF_LOG_REMOTE_URL"
logRemoteTLSCAFile = "SWWAF_LOG_REMOTE_TLS_CA_FILE"
logRemoteBuffer = "SWWAF_LOG_REMOTE_BUFFER"
logRemoteFacility = "SWWAF_LOG_REMOTE_FACILITY"
logRemoteAppName = "SWWAF_LOG_REMOTE_APP_NAME"
) )
// defaultLogRequestHeaders is the default of SWWAF_LOG_REQUEST_HEADERS.
const defaultLogRequestHeaders = "accept,accept-language,accept-encoding," +
"content-type,origin,range"
// testCA is a CA certificate, of which only that it reads matters here.
const testCA = `-----BEGIN CERTIFICATE-----
MIIBkzCCATmgAwIBAgIUeySaE27dnr6A2HijrMB13gTLUKIwCgYIKoZIzj0EAwIw
HjEcMBoGA1UEAwwTc21hbGx3ZWJ3YWYgdGVzdCBDQTAgFw0yNjEwMDYxNDI2MTNa
GA8yMTI2MDkxMjE0MjYxM1owHjEcMBoGA1UEAwwTc21hbGx3ZWJ3YWYgdGVzdCBD
QTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABKDEhcWKKhet2KgSdME+iEPxyEyn
2sd9IdElbt8DM2SfCdB2JsXo0C07UNZaywMPMfn/n8LNI/PKwu+N2uX7gfSjUzBR
MB0GA1UdDgQWBBRZo3BPLv0KbV4drw6JI1JIUKmoRzAfBgNVHSMEGDAWgBRZo3BP
Lv0KbV4drw6JI1JIUKmoRzAPBgNVHRMBAf8EBTADAQH/MAoGCCqGSM49BAMCA0gA
MEUCICC5k+76UpWoSwVbZA+atu5WcALEOJGqwOUWua3zemhcAiEA8Hfdxgwp0z2v
rlG9y/jrJb6ORy3kTLWo2EA0BA67vuI=
-----END CERTIFICATE-----
`
// token is a token of 32 characters, the shortest allowed. // token is a token of 32 characters, the shortest allowed.
const token = "0123456789abcdef0123456789abcdef" const token = "0123456789abcdef0123456789abcdef"
// instance is an SWWAF_INSTANCE_NAME that is a valid app name too, and
// remoteURL an SWWAF_LOG_REMOTE_URL, for the tests that send the lines.
const (
instance = "fsn1app1/gitea"
remoteURL = "syslog+udp://192.0.2.1:514"
)
// off switches a timeout, a size limit or a rate limit off. // off switches a timeout, a size limit or a rate limit off.
const off = "off" const off = "off"
@@ -139,7 +101,6 @@ func TestDefaults(t *testing.T) {
LimitBanDuration: time.Hour, LimitBanDuration: time.Hour,
LimitBanRepeatWindow: 24 * time.Hour, LimitBanRepeatWindow: 24 * time.Hour,
MaxBanDuration: 7 * 24 * time.Hour, MaxBanDuration: 7 * 24 * time.Hour,
AttackBanDuration: 7 * 24 * time.Hour,
MaxBans: 5000, MaxBans: 5000,
BanScopeV4Prefix: 32, BanScopeV4Prefix: 32,
StateDir: "/var/lib/smallwebwaf", StateDir: "/var/lib/smallwebwaf",
@@ -147,8 +108,6 @@ func TestDefaults(t *testing.T) {
StateCounterInterval: 15 * time.Minute, StateCounterInterval: 15 * time.Minute,
MetricsToken: "", MetricsToken: "",
MetricsTopN: 50, MetricsTopN: 50,
RulesDir: "/etc/smallwebwaf/rules.d",
RulesEnabled: true,
}) })
if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" { if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" {
@@ -163,18 +122,6 @@ func TestDefaults(t *testing.T) {
wantCountries(t, deniedCountries, cfg.DeniedCountries) wantCountries(t, deniedCountries, cfg.DeniedCountries)
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries) wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries)
hostname, err := os.Hostname()
if err != nil || hostname == "" || cfg.InstanceName != hostname {
t.Errorf("%s is %q, want the host's name %q (%v)", instanceName,
cfg.InstanceName, hostname, err)
}
wantHeaders := strings.Split(defaultLogRequestHeaders, ",")
if !slices.Equal(cfg.LogRequestHeaders, wantHeaders) {
t.Errorf("%s gave %v, want %v", logRequestHeaders, cfg.LogRequestHeaders,
wantHeaders)
}
if len(cfg.RateLimitExemptPaths) != 0 { if len(cfg.RateLimitExemptPaths) != 0 {
t.Errorf("%s gave %v, want none", rateLimitExemptPaths, cfg.RateLimitExemptPaths) t.Errorf("%s gave %v, want none", rateLimitExemptPaths, cfg.RateLimitExemptPaths)
} }
@@ -202,13 +149,13 @@ func TestValuesAsSet(t *testing.T) {
rateLimitPerMinute: "60", rateLimitPerMinute: "60",
rateLimitPerHour: "600", rateLimitPerHour: "600",
rateLimitPerDay: "6000", rateLimitPerDay: "6000",
rateLimitExemptPaths: "/assets/, /favicon.ico",
deniedCountries: "cn, RU,kp,Xk", deniedCountries: "cn, RU,kp,Xk",
allowedCountries: "de", allowedCountries: "de",
banResponse: "429", banResponse: "429",
limitBanDuration: "15m", limitBanDuration: "15m",
limitBanRepeatWindow: "2d", limitBanRepeatWindow: "2d",
maxBanDuration: "30d", maxBanDuration: "30d",
attackBanDuration: "1d",
maxBans: "100", maxBans: "100",
banScopeV4Prefix: "24", banScopeV4Prefix: "24",
stateDir: "/srv/waf-state", stateDir: "/srv/waf-state",
@@ -216,8 +163,6 @@ func TestValuesAsSet(t *testing.T) {
stateCounterInterval: "1h", stateCounterInterval: "1h",
metricsToken: token, metricsToken: token,
metricsTopN: "10", metricsTopN: "10",
rulesDir: "/srv/waf-rules",
rulesEnabled: "false",
}) })
wantSettings(t, cfg, config.Config{ wantSettings(t, cfg, config.Config{
@@ -238,7 +183,6 @@ func TestValuesAsSet(t *testing.T) {
LimitBanDuration: 15 * time.Minute, LimitBanDuration: 15 * time.Minute,
LimitBanRepeatWindow: 48 * time.Hour, LimitBanRepeatWindow: 48 * time.Hour,
MaxBanDuration: 30 * 24 * time.Hour, MaxBanDuration: 30 * 24 * time.Hour,
AttackBanDuration: 24 * time.Hour,
MaxBans: 100, MaxBans: 100,
BanScopeV4Prefix: 24, BanScopeV4Prefix: 24,
StateDir: "/srv/waf-state", StateDir: "/srv/waf-state",
@@ -246,8 +190,6 @@ func TestValuesAsSet(t *testing.T) {
StateCounterInterval: time.Hour, StateCounterInterval: time.Hour,
MetricsToken: token, MetricsToken: token,
MetricsTopN: 10, MetricsTopN: 10,
RulesDir: "/srv/waf-rules",
RulesEnabled: false,
}) })
if cfg.UpstreamURL.String() != "https://app.internal:8443/" { if cfg.UpstreamURL.String() != "https://app.internal:8443/" {
@@ -260,12 +202,6 @@ func TestValuesAsSet(t *testing.T) {
wantNetblocks(t, cfg.DenyNets, "198.51.100.0/24") wantNetblocks(t, cfg.DenyNets, "198.51.100.0/24")
wantCountries(t, deniedCountries, cfg.DeniedCountries, "CN", "RU", "KP", "XK") wantCountries(t, deniedCountries, cfg.DeniedCountries, "CN", "RU", "KP", "XK")
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE") wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
}
func TestRateLimitExemptPathsAsSet(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{rateLimitExemptPaths: "/assets/, /favicon.ico"})
if !slices.Equal(cfg.RateLimitExemptPaths, []string{"/assets/", "/favicon.ico"}) { if !slices.Equal(cfg.RateLimitExemptPaths, []string{"/assets/", "/favicon.ico"}) {
t.Errorf("%s gave %v, want /assets/ and /favicon.ico", t.Errorf("%s gave %v, want /assets/ and /favicon.ico",
@@ -286,192 +222,6 @@ func TestPathPrefixNotStartingWithSlashStopsTheStart(t *testing.T) {
} }
} }
func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{
instanceName: "fsn1app1/gitea",
logRequestHeaders: " Accept , X-Custom",
})
if cfg.InstanceName != "fsn1app1/gitea" ||
!slices.Equal(cfg.LogRequestHeaders, []string{"accept", "x-custom"}) {
t.Errorf("%s is %q and %s gives %v", instanceName, cfg.InstanceName,
logRequestHeaders, cfg.LogRequestHeaders)
}
}
func TestRemoteLogSettingsDefaults(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{instanceName: instance})
if cfg.LogRemoteURL != nil || cfg.LogRemoteTLSCAs != nil ||
cfg.LogRemoteBuffer != 10000 || cfg.LogRemoteFacility != 16 ||
cfg.LogRemoteAppName != instance {
t.Errorf("remote log settings %v, %v, %d, %d and %q, want no URL, no "+
"certificates, 10000, 16 and %s's %s", cfg.LogRemoteURL,
cfg.LogRemoteTLSCAs, cfg.LogRemoteBuffer, cfg.LogRemoteFacility,
cfg.LogRemoteAppName, instanceName, instance)
}
}
func TestRemoteLogSettingsAsSet(t *testing.T) {
t.Parallel()
caFile := filepath.Join(t.TempDir(), "ca.pem")
err := os.WriteFile(caFile, []byte(testCA), 0o600)
if err != nil {
t.Fatalf("write %s: %v", caFile, err)
}
cfg := fromEnvironment(t, environment{
logRemoteURL: "syslog+tls://logs.example:6514",
logRemoteTLSCAFile: caFile,
logRemoteBuffer: "500",
logRemoteFacility: "daemon",
logRemoteAppName: instance,
})
roots := x509.NewCertPool()
roots.AppendCertsFromPEM([]byte(testCA))
if cfg.LogRemoteURL.String() != "syslog+tls://logs.example:6514" ||
!roots.Equal(cfg.LogRemoteTLSCAs) || cfg.LogRemoteBuffer != 500 ||
cfg.LogRemoteFacility != 3 || cfg.LogRemoteAppName != instance {
t.Errorf("remote log settings %v, %v, %d, %d and %q", cfg.LogRemoteURL,
cfg.LogRemoteTLSCAs, cfg.LogRemoteBuffer, cfg.LogRemoteFacility,
cfg.LogRemoteAppName)
}
}
func TestRemoteLogURLForms(t *testing.T) {
t.Parallel()
for _, value := range []string{
"syslog+udp://192.0.2.1:514",
"syslog+tcp://[2001:db8::1]:514",
"syslog+tls://logs.example:6514/",
} {
cfg := fromEnvironment(t, environment{logRemoteURL: value})
if cfg.LogRemoteURL.String() != value {
t.Errorf("%s read as %v", value, cfg.LogRemoteURL)
}
}
cfg := fromEnvironment(t, environment{logRemoteURL: ""})
if cfg.LogRemoteURL != nil {
t.Errorf("set but empty, %s read as %v", logRemoteURL, cfg.LogRemoteURL)
}
}
func TestRemoteLogFacilitiesByNumber(t *testing.T) {
t.Parallel()
for name, number := range map[string]int{
"kern": 0, "user": 1, "auth": 4, "authpriv": 10, "ftp": 11,
"local0": 16, "local5": 21, "local7": 23,
} {
cfg := fromEnvironment(t, environment{logRemoteFacility: name})
if cfg.LogRemoteFacility != number {
t.Errorf("%s read as %d, want %d", name, cfg.LogRemoteFacility, number)
}
}
}
func TestInvalidRemoteLogSettingStopsTheStart(t *testing.T) {
t.Parallel()
for _, tc := range []struct{ name, value string }{
{logRemoteURL, "logs.example:514"},
{logRemoteURL, "syslog://logs.example:514"},
{logRemoteURL, "http://logs.example:514"},
{logRemoteURL, "syslog+udp://logs.example"},
{logRemoteURL, "syslog+tcp://:514"},
{logRemoteURL, "syslog+tcp://logs.example:0"},
{logRemoteURL, "syslog+tls://logs.example:65536"},
{logRemoteURL, "syslog+tls://user@logs.example:6514"},
{logRemoteURL, "syslog+tcp://logs.example:514/app"},
{logRemoteURL, "syslog+tcp://logs.example:514?tls=1"},
{logRemoteTLSCAFile, "/nonexistent/ca.pem"},
{logRemoteBuffer, off}, {logRemoteBuffer, "0"}, {logRemoteBuffer, "10K"},
{logRemoteFacility, "local8"}, {logRemoteFacility, "LOCAL0"},
{logRemoteFacility, "16"}, {logRemoteFacility, ""},
{logRemoteAppName, ""}, {logRemoteAppName, "my app"},
{logRemoteAppName, "gitéa"}, {logRemoteAppName, strings.Repeat("a", 49)},
} {
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
if err == nil || !strings.HasPrefix(err.Error(), tc.name+": ") {
t.Errorf("%s=%q: error %v, want one naming it", tc.name, tc.value, err)
}
}
}
func TestRemoteLogCAFileWithoutCertificateStopsTheStart(t *testing.T) {
t.Parallel()
caFile := filepath.Join(t.TempDir(), "ca.pem")
err := os.WriteFile(caFile, []byte("not a certificate\n"), 0o600)
if err != nil {
t.Fatalf("write %s: %v", caFile, err)
}
_, err = config.FromEnvironment(environment{logRemoteTLSCAFile: caFile}.lookupEnv)
want := logRemoteTLSCAFile + `: "` + caFile + `" holds no PEM certificate`
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
}
func TestInstanceNameNotAnAppNameStopsTheStartOnlyWhileSending(t *testing.T) {
t.Parallel()
const spaced = "fsn1 app1"
sending := environment{logRemoteURL: remoteURL, instanceName: spaced}
_, err := config.FromEnvironment(sending.lookupEnv)
want := logRemoteAppName + `: is unset, and ` + instanceName +
` "fsn1 app1", its default, is not 1 to 48 printable ASCII characters ` +
`without a space, such as gitea`
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
cfg := fromEnvironment(t, environment{instanceName: spaced})
if cfg.LogRemoteAppName != spaced {
t.Errorf("not sending, %s is %q", logRemoteAppName, cfg.LogRemoteAppName)
}
sending[logRemoteAppName] = instance
cfg = fromEnvironment(t, sending)
if cfg.LogRemoteAppName != instance {
t.Errorf("set to %s, %s is %q", instance, logRemoteAppName,
cfg.LogRemoteAppName)
}
}
func TestAppNameSetStopsTheStartWhileSending(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{
logRemoteURL: remoteURL,
instanceName: instance,
logRemoteAppName: "my app",
}.lookupEnv)
want := logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` +
`characters without a space, such as gitea`
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
}
func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) { func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
t.Parallel() t.Parallel()
@@ -572,8 +322,10 @@ func TestTrustedProxiesSetButEmptyTrustNothing(t *testing.T) {
func TestInvalidValueStopsTheStart(t *testing.T) { func TestInvalidValueStopsTheStart(t *testing.T) {
t.Parallel() t.Parallel()
wantStartStopped(t, []struct{ name, value string }{ for _, tc := range []struct{ name, value string }{
{listenAddr, "8080"}, {listenAddr, ":http"}, {listenAddr, ":65536"}, {listenAddr, "8080"},
{listenAddr, ":http"},
{listenAddr, ":65536"},
{upstreamURL, "127.0.0.1:8081"}, {upstreamURL, "127.0.0.1:8081"},
{upstreamURL, "ftp://127.0.0.1:8081"}, {upstreamURL, "ftp://127.0.0.1:8081"},
{upstreamURL, "http://"}, {upstreamURL, "http://"},
@@ -591,7 +343,8 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
{allowNets, "192.0.2.0/24,monitoring"}, {allowNets, "192.0.2.0/24,monitoring"},
{rateLimitExemptNets, "2001:db8::/129"}, {rateLimitExemptNets, "2001:db8::/129"},
{denyNets, "198.51.100.0/24,"}, {denyNets, "198.51.100.0/24,"},
{clientRequestTimeout, "60"}, {clientRequestTimeout, ""}, {clientRequestTimeout, "60"},
{clientRequestTimeout, ""},
{clientIdleTimeout, "0s"}, {clientIdleTimeout, "0s"},
{clientIdleTimeout, "2 minutes"}, {clientIdleTimeout, "2 minutes"},
{clientResponseTimeout, "1y"}, {clientResponseTimeout, "1y"},
@@ -608,7 +361,8 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
{rateLimitPerMinute, "1K"}, {rateLimitPerMinute, "1K"},
{rateLimitPerHour, "0"}, {rateLimitPerHour, "0"},
{rateLimitPerHour, "1.5"}, {rateLimitPerHour, "1.5"},
{rateLimitPerDay, "-1"}, {rateLimitPerDay, "lots"}, {rateLimitPerDay, "-1"},
{rateLimitPerDay, "lots"},
{rateLimitExemptPaths, "/assets/,,/static/"}, {rateLimitExemptPaths, "/assets/,,/static/"},
{deniedCountries, "nk"}, {deniedCountries, "nk"},
{deniedCountries, "kp,,ir"}, {deniedCountries, "kp,,ir"},
@@ -622,37 +376,17 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
{allowedCountries, "uk"}, {allowedCountries, "uk"},
{allowedCountries, "zz"}, {allowedCountries, "zz"},
{allowedCountries, "de,germany"}, {allowedCountries, "de,germany"},
{metricsTopN, off}, {metricsTopN, "0"}, {metricsTopN, "-1"},
{logRequestHeaders, "accept,,origin"}, {logRequestHeaders, "accept;origin"},
{logRequestHeaders, "accept language"}, {logRequestHeaders, "x-foo:"},
{logRequestHeaders, "host"}, {logRequestHeaders, "accept,Host"},
{logRequestHeaders, "transfer-encoding"}, {logRequestHeaders, "TRANSFER-ENCODING"},
{rulesEnabled, "yes"}, {rulesEnabled, "True"},
})
}
func TestInvalidBanOrStateValueStopsTheStart(t *testing.T) {
t.Parallel()
wantStartStopped(t, []struct{ name, value string }{
{banResponse, "404"}, {banResponse, "drop"}, {banResponse, ""}, {banResponse, "404"}, {banResponse, "drop"}, {banResponse, ""},
{limitBanDuration, off}, {limitBanDuration, "0s"}, {limitBanDuration, "1"}, {limitBanDuration, off}, {limitBanDuration, "0s"}, {limitBanDuration, "1"},
{limitBanRepeatWindow, off}, {limitBanRepeatWindow, "-1h"}, {limitBanRepeatWindow, off}, {limitBanRepeatWindow, "-1h"},
{maxBanDuration, off}, {maxBanDuration, "1w"}, {attackBanDuration, off}, {maxBanDuration, off}, {maxBanDuration, "1w"},
{maxBans, off}, {maxBans, "0"}, {maxBans, "5K"}, {maxBans, off}, {maxBans, "0"}, {maxBans, "5K"},
{banScopeV4Prefix, "33"}, {banScopeV4Prefix, "-1"}, {banScopeV4Prefix, "/24"}, {banScopeV4Prefix, "33"}, {banScopeV4Prefix, "-1"}, {banScopeV4Prefix, "/24"},
{stateDir, ""}, {stateDir, "state"}, {stateDir, "./var/lib/smallwebwaf"}, {stateDir, ""}, {stateDir, "state"}, {stateDir, "./var/lib/smallwebwaf"},
{stateWriteDelay, off}, {stateWriteDelay, "0s"}, {stateWriteDelay, off}, {stateWriteDelay, "0s"},
{stateCounterInterval, off}, {stateCounterInterval, "15"}, {stateCounterInterval, off}, {stateCounterInterval, "15"},
}) {metricsTopN, off}, {metricsTopN, "0"}, {metricsTopN, "-1"},
} } {
// wantStartStopped checks that each setting, set to its value, stops the
// start with an error that names the setting.
func wantStartStopped(t *testing.T, invalid []struct{ name, value string }) {
t.Helper()
for _, tc := range invalid {
t.Run(tc.name+"="+tc.value, func(t *testing.T) { t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -668,27 +402,6 @@ func wantStartStopped(t *testing.T, invalid []struct{ name, value string }) {
} }
} }
func TestHostOrTransferEncodingStopsTheStart(t *testing.T) {
t.Parallel()
// Only Host's message points to the field host.
for value, want := range map[string]string{
"Host": `"Host" is taken out of every request by Go's HTTP server, ` +
"so it can never be logged; the request's host is the field host",
"transfer-encoding": `"transfer-encoding" is taken out of every ` +
"request by Go's HTTP server, so it can never be logged",
} {
t.Run(value, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{logRequestHeaders: value}.lookupEnv)
if err == nil || err.Error() != logRequestHeaders+": "+want {
t.Errorf("error %v, want %s: %s", err, logRequestHeaders, want)
}
})
}
}
func TestShortTokenStopsTheStartWithoutShowingIt(t *testing.T) { func TestShortTokenStopsTheStartWithoutShowingIt(t *testing.T) {
t.Parallel() t.Parallel()
@@ -722,133 +435,6 @@ func TestTokenIsLoggedMasked(t *testing.T) {
} }
} }
func TestSettingFromFileLosesOneNewlineAndNoMore(t *testing.T) {
t.Parallel()
for contents, want := range map[string]string{
token: token,
token + "\n": token,
token + "\n\n": token + "\n",
token + " \n": token + " ",
} {
cfg := fromEnvironment(t, environment{
metricsToken + "_FILE": writeFile(t, contents),
})
if cfg.MetricsToken != want {
t.Errorf("file holding %q gave %s %q, want %q", contents, metricsToken,
cfg.MetricsToken, want)
}
}
}
func TestSettingFromFileIsCheckedAsTheSettingItself(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{
requestMaxBytes + "_FILE": writeFile(t, "lots\n"),
}.lookupEnv)
want := requestMaxBytes + `: "lots" is not a size such as 512K, 100M or 5G, or off`
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
_, err = config.FromEnvironment(environment{
logRemoteURL: remoteURL,
instanceName: instance,
logRemoteAppName + "_FILE": writeFile(t, "my app\n"),
}.lookupEnv)
want = logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` +
`characters without a space, such as gitea`
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
}
func TestSettingAndItsFileBothSetStopsTheStart(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{
metricsToken: token,
metricsToken + "_FILE": writeFile(t, token),
}.lookupEnv)
want := metricsToken + ": is set, and so is " + metricsToken +
"_FILE; set only one of them"
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
}
func TestUnreadableSettingFileStopsTheStart(t *testing.T) {
t.Parallel()
dir := t.TempDir()
for _, path := range []string{filepath.Join(dir, "missing"), dir} {
_, err := config.FromEnvironment(environment{metricsToken + "_FILE": path}.lookupEnv)
want := metricsToken + "_FILE: cannot be read: "
if err == nil || !strings.HasPrefix(err.Error(), want) {
t.Errorf("%s: error %v, want one starting %s", path, err, want)
}
}
}
func TestTokenFromFileIsLoggedMaskedWithTheFile(t *testing.T) {
t.Parallel()
path := writeFile(t, token+"\n")
cfg := fromEnvironment(t, environment{metricsToken + "_FILE": path})
var out bytes.Buffer
slog.New(slog.NewJSONHandler(&out, nil)).Info("starting", "settings", cfg)
var line struct {
Settings map[string]string `json:"settings"`
}
err := json.Unmarshal(out.Bytes(), &line)
if err != nil {
t.Fatalf("decode %s: %v", out.Bytes(), err)
}
if strings.Contains(out.String(), token) ||
line.Settings[metricsToken] != "********" ||
line.Settings[metricsToken+"_FILE"] != path {
t.Errorf("the token is not logged masked, with its file %s: %s", path,
out.String())
}
}
func TestRemoteLogCAFileIsNotReadAsAFileInItsTurn(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{
logRemoteTLSCAFile + "_FILE": writeFile(t, "/nonexistent/ca.pem\n"),
})
if cfg.LogRemoteTLSCAs != nil {
t.Errorf("%s_FILE gave certificates", logRemoteTLSCAFile)
}
}
// writeFile writes contents to a file in a directory of its own, removed
// when the test ends, and returns the file's path.
func writeFile(t *testing.T, contents string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "setting")
err := os.WriteFile(path, []byte(contents), 0o600)
if err != nil {
t.Fatalf("write %s: %v", path, err)
}
return path
}
func TestLogsEachSettingWithItsValue(t *testing.T) { func TestLogsEachSettingWithItsValue(t *testing.T) {
t.Parallel() t.Parallel()
@@ -867,8 +453,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
t.Fatalf("decode %s: %v", out.Bytes(), err) t.Fatalf("decode %s: %v", out.Bytes(), err)
} }
hostname, _ := os.Hostname()
want := map[string]string{ want := map[string]string{
listenAddr: ":8080", listenAddr: ":8080",
upstreamURL: "http://127.0.0.1:8081", upstreamURL: "http://127.0.0.1:8081",
@@ -895,7 +479,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
limitBanDuration: "1h", limitBanDuration: "1h",
limitBanRepeatWindow: "24h", limitBanRepeatWindow: "24h",
maxBanDuration: "7d", maxBanDuration: "7d",
attackBanDuration: "7d",
maxBans: "5000", maxBans: "5000",
banScopeV4Prefix: "32", banScopeV4Prefix: "32",
stateDir: "/var/lib/smallwebwaf", stateDir: "/var/lib/smallwebwaf",
@@ -903,15 +486,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
stateCounterInterval: "15m", stateCounterInterval: "15m",
metricsToken: "", metricsToken: "",
metricsTopN: "50", metricsTopN: "50",
instanceName: hostname,
logRequestHeaders: defaultLogRequestHeaders,
rulesDir: "/etc/smallwebwaf/rules.d",
rulesEnabled: "true",
logRemoteURL: "",
logRemoteTLSCAFile: "",
logRemoteBuffer: "10000",
logRemoteFacility: "local0",
logRemoteAppName: hostname,
} }
if !maps.Equal(line.Settings, want) { if !maps.Equal(line.Settings, want) {
t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want) t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
@@ -941,8 +515,8 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) {
wantBanSettings(t, got, want) wantBanSettings(t, got, want)
} }
// wantBanSettings checks the settings for bans, the state files, the // wantBanSettings checks the settings for bans, the state files and the
// metrics and the rule files. // metrics.
func wantBanSettings(t *testing.T, got *config.Config, want config.Config) { func wantBanSettings(t *testing.T, got *config.Config, want config.Config) {
t.Helper() t.Helper()
@@ -950,17 +524,11 @@ func wantBanSettings(t *testing.T, got *config.Config, want config.Config) {
got.LimitBanDuration != want.LimitBanDuration || got.LimitBanDuration != want.LimitBanDuration ||
got.LimitBanRepeatWindow != want.LimitBanRepeatWindow || got.LimitBanRepeatWindow != want.LimitBanRepeatWindow ||
got.MaxBanDuration != want.MaxBanDuration || got.MaxBanDuration != want.MaxBanDuration ||
got.AttackBanDuration != want.AttackBanDuration ||
got.MaxBans != want.MaxBans || got.MaxBans != want.MaxBans ||
got.BanScopeV4Prefix != want.BanScopeV4Prefix { got.BanScopeV4Prefix != want.BanScopeV4Prefix {
t.Errorf("ban settings\n%+v\nwant\n%+v", got, want) t.Errorf("ban settings\n%+v\nwant\n%+v", got, want)
} }
if got.RulesDir != want.RulesDir || got.RulesEnabled != want.RulesEnabled {
t.Errorf("rule files in %q, on: %t, want %q, %t",
got.RulesDir, got.RulesEnabled, want.RulesDir, want.RulesEnabled)
}
if got.StateDir != want.StateDir || if got.StateDir != want.StateDir ||
got.StateWriteDelay != want.StateWriteDelay || got.StateWriteDelay != want.StateWriteDelay ||
got.StateCounterInterval != want.StateCounterInterval { got.StateCounterInterval != want.StateCounterInterval {
+9 -67
View File
@@ -13,9 +13,7 @@ import (
"github.com/prometheus/client_golang/prometheus/promhttp" "github.com/prometheus/client_golang/prometheus/promhttp"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
) )
// Metrics are smallwebwaf's metrics. They are safe for concurrent use. // Metrics are smallwebwaf's metrics. They are safe for concurrent use.
@@ -32,9 +30,7 @@ type Metrics struct {
rateLimitHits *prometheus.CounterVec rateLimitHits *prometheus.CounterVec
sizeAndTimeLimitHits *prometheus.CounterVec sizeAndTimeLimitHits *prometheus.CounterVec
offences *prometheus.CounterVec offences *prometheus.CounterVec
// ruleMatches are made by AddRules. countries *countries
ruleMatches *prometheus.CounterVec
countries *countries
// GeoJSRequests are the requests to GeoJS, and GeoJSFailures those // GeoJSRequests are the requests to GeoJS, and GeoJSFailures those
// that failed. GeoJSUnanswered are the requests whose client counted // that failed. GeoJSUnanswered are the requests whose client counted
@@ -139,22 +135,19 @@ func New(topN int) *Metrics {
// AddBansAndClients adds the metrics read from the ledger and the table // AddBansAndClients adds the metrics read from the ledger and the table
// of clients as the metrics are asked for: the bans made since the start, // of clients as the metrics are asked for: the bans made since the start,
// by cause, the bans active and permanent at now, and the clients in the // the bans active and permanent at now, and the clients in the table.
// table.
func (m *Metrics) AddBansAndClients( func (m *Metrics) AddBansAndClients(
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time, ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
) { ) {
for _, cause := range []string{bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin} { m.registry.MustRegister(
m.registry.MustRegister(prometheus.NewCounterFunc(prometheus.CounterOpts{ // Every ban smallwebwaf makes so far is for a broken limit.
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "smallwebwaf_bans_made_total", Name: "smallwebwaf_bans_made_total",
Help: "Bans made, by cause.", Help: "Bans made, by cause.",
ConstLabels: prometheus.Labels{"cause": cause}, ConstLabels: prometheus.Labels{"cause": "limit"},
}, func() float64 { }, func() float64 {
return float64(ledger.Made(cause)) return float64(ledger.Made())
})) }),
}
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{ prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "smallwebwaf_active_bans", Name: "smallwebwaf_active_bans",
Help: "Bans active now, the permanent ones included.", Help: "Bans active now, the permanent ones included.",
@@ -165,7 +158,7 @@ func (m *Metrics) AddBansAndClients(
}), }),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{ prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "smallwebwaf_permanent_bans", Name: "smallwebwaf_permanent_bans",
Help: "Permanent bans not lifted.", Help: "Permanent bans.",
}, func() float64 { }, func() float64 {
_, permanent := ledger.Count(now()) _, permanent := ledger.Count(now())
@@ -180,51 +173,6 @@ func (m *Metrics) AddBansAndClients(
) )
} }
// AddRules adds the metrics of the rule files: the requests that matched
// each rule, which RuleMatched counts, and the rules loaded from
// ruleFiles, read as the metrics are asked for. It is called once, before
// RuleMatched.
func (m *Metrics) AddRules(ruleFiles *rules.Files) {
m.ruleMatches = counterVec("smallwebwaf_rule_matches_total",
"Requests that matched a rule of the rule files, by its id and action.",
[]string{"rule_id", "action"})
m.registry.MustRegister(m.ruleMatches,
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "smallwebwaf_rules_loaded",
Help: "Rules loaded from the rule files.",
}, func() float64 {
return float64(ruleFiles.Len())
}))
}
// AddRemoteLog adds the metrics of sending the log lines to
// SWWAF_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
// lines sent, those dropped, and those waiting in the buffer.
func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "smallwebwaf_remote_log_lines_sent_total",
Help: "Log lines sent to SWWAF_LOG_REMOTE_URL.",
}, func() float64 {
return float64(remote.Sent())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "smallwebwaf_remote_log_lines_dropped_total",
Help: "Log lines dropped: the oldest in a full buffer, and those " +
"whose sending failed.",
}, func() float64 {
return float64(remote.Dropped())
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "smallwebwaf_remote_log_buffer_depth",
Help: "Log lines in the buffer, waiting to be sent.",
}, func() float64 {
return float64(remote.Depth())
}),
)
}
// ServeHTTP answers with the metrics in the Prometheus text format. // ServeHTTP answers with the metrics in the Prometheus text format.
func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) { func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) {
m.handler.ServeHTTP(w, r) m.handler.ServeHTTP(w, r)
@@ -271,12 +219,6 @@ func (m *Metrics) RequestEnded(
} }
} }
// RuleMatched counts a request that matched the rule id, whose action is
// action.
func (m *Metrics) RuleMatched(id, action string) {
m.ruleMatches.WithLabelValues(id, action).Inc()
}
// StateFileWritten counts a write of the state file name, of size bytes, // StateFileWritten counts a write of the state file name, of size bytes,
// that ended with err. // that ended with err.
func (m *Metrics) StateFileWritten(name string, size int, err error) { func (m *Metrics) StateFileWritten(name string, size int, err error) {
+19 -49
View File
@@ -6,7 +6,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
) )
// banResponse is a refusal answered with SWWAF_BAN_RESPONSE, and logged // banResponse is a refusal answered with SWWAF_BAN_RESPONSE, and logged
@@ -31,17 +30,14 @@ func (rq *request) banned(now time.Time) bool {
return banned return banned
} }
// limitBroken counts the request for the rate limits at now, notes the // limitBroken counts the request for the rate limits at now, and reports
// client's counts for the log line, and reports whether the request takes // whether it takes the client over one. In enforce mode such a request
// the client over a limit. In enforce mode such a request bans the // bans the client's netblock, and sets the client's counters back to
// client's netblock, and sets the client's counters back to zero; in // zero; in observe mode it does neither.
// observe mode it does neither.
func (rq *request) limitBroken(now time.Time) bool { func (rq *request) limitBroken(now time.Time) bool {
group := clientGroup(rq.client) group := clientGroup(rq.client)
counts, hit, over := rq.h.limiter.Count(group, now) hit, over := rq.h.limiter.Count(group, now)
rq.line.Counts = counts
if !over { if !over {
return false return false
} }
@@ -55,12 +51,20 @@ func (rq *request) limitBroken(now time.Time) bool {
netblock := rq.netblock() netblock := rq.netblock()
ban := rq.h.ledger.BanForLimit(netblock, now, bans.Notes{ ban := rq.h.ledger.BanForLimit(netblock, now, bans.Notes{
Country: rq.line.Country, Country: rq.line.Country,
Limit: hit.Limit, Limit: hit.Limit,
Window: hit.Window, Window: hit.Window,
Count: hit.Requests, Count: hit.Requests,
Request: rq.noted(now), Request: bans.Request{
Requests: rq.netblockRequests(netblock), Time: now,
Method: rq.in.Method,
Host: rq.in.Host,
Path: rq.in.URL.RequestURI(),
Status: rq.h.config.BanResponse,
UserAgent: rq.in.UserAgent(),
},
// The histories count this request only once it has ended.
Requests: rq.h.limiter.Requests(netblock) + 1,
}) })
rq.h.limiter.Reset(group) rq.h.limiter.Reset(group)
rq.line.BanExpires = banExpires(ban) rq.line.BanExpires = banExpires(ban)
@@ -68,40 +72,6 @@ func (rq *request) limitBroken(now time.Time) bool {
return true return true
} }
// banForAttack bans the client's netblock at now for a clear sign of
// attack, the match of rule, a ban rule.
func (rq *request) banForAttack(now time.Time, rule rules.Rule) {
netblock := rq.netblock()
ban := rq.h.ledger.BanForAttack(netblock, now, bans.Notes{
Country: rq.line.Country,
RuleID: rule.ID,
Target: rule.Target,
Request: rq.noted(now),
Requests: rq.netblockRequests(netblock),
})
rq.line.BanExpires = banExpires(ban)
}
// noted is the request, refused at now with SWWAF_BAN_RESPONSE, as the
// notes of the ban it makes keep it.
func (rq *request) noted(now time.Time) bans.Request {
return bans.Request{
Time: now,
Method: rq.in.Method,
Host: rq.in.Host,
Path: rq.in.URL.RequestURI(),
Status: rq.h.config.BanResponse,
UserAgent: rq.in.UserAgent(),
}
}
// netblockRequests is how many requests netblock has sent since it was
// first seen, this one included: the histories count it only once it has
// ended.
func (rq *request) netblockRequests(netblock netip.Prefix) int64 {
return rq.h.limiter.Requests(netblock) + 1
}
// netblock is the netblock a ban on the client covers: its IPv4 address, // netblock is the netblock a ban on the client covers: its IPv4 address,
// widened to SWWAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup // widened to SWWAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
// counts it in. // counts it in.
+3 -5
View File
@@ -278,8 +278,6 @@ func TestBanNotes(t *testing.T) {
Netblock: netblock, Netblock: netblock,
Start: start, Start: start,
Expires: start.Add(time.Hour), Expires: start.Add(time.Hour),
Cause: bans.CauseLimit,
Reason: "requests per minute over the limit of 1",
Notes: bans.Notes{ Notes: bans.Notes{
Country: "DE", Country: "DE",
Limit: 1, Limit: 1,
@@ -297,7 +295,7 @@ func TestBanNotes(t *testing.T) {
// refused under the ban. // refused under the ban.
Requests: 4, Requests: 4,
Refused: 2, Refused: 2,
EarlierBans: bans.EarlierBans{}, EarlierBans: 0,
}, },
} }
@@ -314,8 +312,8 @@ func TestBanNotes(t *testing.T) {
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited) s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
got = ledger.Bans(netblock) got = ledger.Bans(netblock)
if len(got) != 2 || got[1].Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) { if len(got) != 2 || got[1].Notes.EarlierBans != 1 {
t.Errorf("bans %+v, want two, the second with one earlier ban for a limit", got) t.Errorf("bans %+v, want two, the second with one earlier ban", got)
} }
} }
-28
View File
@@ -1,7 +1,6 @@
package proxy package proxy
import ( import (
"crypto/rand"
"net/http" "net/http"
"net/netip" "net/netip"
"slices" "slices"
@@ -49,33 +48,6 @@ func clientAddress(
return client return client
} }
// requestIDHeader carries the request's id, from traefik and to the app.
const requestIDHeader = "X-Request-ID"
// requestID is the request's id: the one a trusted proxy sent, or a new
// random one. A peer outside the trusted proxies did not come through
// traefik, so the id it sends is its own claim, and is replaced.
func requestID(r *http.Request, peerTrusted bool) string {
id := r.Header.Get(requestIDHeader)
if !peerTrusted || id == "" {
id = rand.Text()
}
return id
}
// scheme is how the client reached traefik, as a trusted proxy says in
// X-Forwarded-Proto, or otherwise http, the only scheme smallwebwaf
// serves.
func scheme(r *http.Request, peerTrusted bool) string {
proto := r.Header.Get("X-Forwarded-Proto")
if !peerTrusted || proto == "" {
return "http"
}
return proto
}
// ipv6GroupPrefix is the length of the IPv6 netblock that is one client. // ipv6GroupPrefix is the length of the IPv6 netblock that is one client.
const ipv6GroupPrefix = 64 const ipv6GroupPrefix = 64
+13 -17
View File
@@ -14,14 +14,10 @@ const (
appHost = "app.example" appHost = "app.example"
// client is the client's address, as a proxy names it. // client is the client's address, as a proxy names it.
client = "203.0.113.9" client = "203.0.113.9"
// forwardedFor is the header that lists the client and its proxies, // forwardedFor is the header that lists the client and its proxies.
// and forwardedProto the one that gives the scheme the client used. forwardedFor = "X-Forwarded-For"
forwardedFor = "X-Forwarded-For" // secure is the scheme a client reached traefik with.
forwardedProto = "X-Forwarded-Proto"
// secure is the scheme a client reached traefik with, and plain the
// one smallwebwaf serves.
secure = "https" secure = "https"
plain = "http"
) )
// appHeaders is what the app tells about the headers it received. // appHeaders is what the app tells about the headers it received.
@@ -69,13 +65,13 @@ func TestClientAddressAndForwardedHeaders(t *testing.T) {
func clientAddressCases() []clientAddressCase { func clientAddressCases() []clientAddressCase {
trusted := map[string]string{trustedProxies: trustLocalhost} trusted := map[string]string{trustedProxies: trustLocalhost}
forged := http.Header{ forged := http.Header{
forwardedFor: {client}, forwardedFor: {client},
"X-Forwarded-Host": {"forged.example"}, "X-Forwarded-Host": {"forged.example"},
forwardedProto: {secure}, "X-Forwarded-Proto": {secure},
"X-Real-Ip": {client}, "X-Real-Ip": {client},
} }
replaced := appHeaders{ replaced := appHeaders{
ForwardedFor: localhost, ForwardedHost: appHost, ForwardedProto: plain, ForwardedFor: localhost, ForwardedHost: appHost, ForwardedProto: "http",
} }
return []clientAddressCase{{ return []clientAddressCase{{
@@ -91,10 +87,10 @@ func clientAddressCases() []clientAddressCase {
"outside the trusted proxies from the right", "outside the trusted proxies from the right",
env: trusted, env: trusted,
header: http.Header{ header: http.Header{
forwardedFor: {"198.51.100.7, " + client + ", 10.0.0.2"}, forwardedFor: {"198.51.100.7, " + client + ", 10.0.0.2"},
"X-Forwarded-Host": {appHost}, "X-Forwarded-Host": {appHost},
forwardedProto: {secure}, "X-Forwarded-Proto": {secure},
"X-Real-Ip": {client}, "X-Real-Ip": {client},
}, },
wantClient: client, wantClient: client,
wantApp: appHeaders{ wantApp: appHeaders{
@@ -142,7 +138,7 @@ func requestWithHeaders(
Host: r.Host, Host: r.Host,
ForwardedFor: r.Header.Get(forwardedFor), ForwardedFor: r.Header.Get(forwardedFor),
ForwardedHost: r.Header.Get("X-Forwarded-Host"), ForwardedHost: r.Header.Get("X-Forwarded-Host"),
ForwardedProto: r.Header.Get(forwardedProto), ForwardedProto: r.Header.Get("X-Forwarded-Proto"),
RealIP: r.Header.Get("X-Real-IP"), RealIP: r.Header.Get("X-Real-IP"),
}) })
}) })
+3 -12
View File
@@ -21,18 +21,14 @@ func TestHealthEndpointIsAnsweredBeforeAnyCheck(t *testing.T) {
// the last one would have it refused. // the last one would have it refused.
addr, out := startProxy(t, app.URL, map[string]string{rateLimitPerMinute: "1"}) addr, out := startProxy(t, app.URL, map[string]string{rateLimitPerMinute: "1"})
const ( const healthChecks = 3
healthChecks = 3
contentType = "text/plain; charset=utf-8"
)
for range healthChecks { for range healthChecks {
got := get(t, addr, proxy.HealthPath) got := get(t, addr, proxy.HealthPath)
wantStatus(t, got, http.StatusOK) wantStatus(t, got, http.StatusOK)
if string(got.body) != "ok\n" || got.header.Get("Content-Type") != contentType { if string(got.body) != "ok\n" {
t.Errorf("health endpoint answered %q with Content-Type %q, want ok "+ t.Errorf("health endpoint answered %q, want ok", got.body)
"with %q", got.body, got.header.Get("Content-Type"), contentType)
} }
} }
@@ -41,11 +37,6 @@ func TestHealthEndpointIsAnsweredBeforeAnyCheck(t *testing.T) {
lines := out.requestLines(t, healthChecks+1) lines := out.requestLines(t, healthChecks+1)
for _, line := range lines[:healthChecks] { for _, line := range lines[:healthChecks] {
wantLine(t, line, http.StatusOK, requestlog.ActionAdmin) wantLine(t, line, http.StatusOK, requestlog.ActionAdmin)
if line.ResponseContentType != contentType {
t.Errorf("health check's log line has response_content_type %q, "+
"want %q", line.ResponseContentType, contentType)
}
} }
wantLine(t, lines[healthChecks], http.StatusOK, requestlog.ActionForward) wantLine(t, lines[healthChecks], http.StatusOK, requestlog.ActionForward)
-24
View File
@@ -12,7 +12,6 @@ import (
"testing" "testing"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/proxy" "sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
@@ -244,29 +243,6 @@ func TestMetricsCountLimitsAndBans(t *testing.T) {
wantMetric(t, metrics, "smallwebwaf_tracked_clients", 3) wantMetric(t, metrics, "smallwebwaf_tracked_clients", 3)
} }
func TestMetricsCountTheBansAnAdminMakes(t *testing.T) {
t.Parallel()
const scraper = "192.0.2.200" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
s, clk, server := startWithClock(t, "", map[string]string{
metricsToken: token,
rateLimitExemptNets: scraper,
})
const admins = `smallwebwaf_bans_made_total{cause="admin"}`
wantMetric(t, s.scrape(scraper), admins, 0)
// As an admin's edit of bans.json that adds a ban is taken in.
server.Ledger.LoadEdit([]bans.Ban{{
Netblock: netip.MustParsePrefix(client + "/32"),
Start: clk.Now(),
}})
wantMetric(t, s.scrape(scraper), admins, 1)
}
func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) { func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) {
t.Parallel() t.Parallel()
-1
View File
@@ -95,7 +95,6 @@ func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
Netblock: netip.MustParsePrefix(otherClient + "/32"), Netblock: netip.MustParsePrefix(otherClient + "/32"),
Start: clk.Now(), Start: clk.Now(),
Expires: clk.Now().Add(time.Hour), Expires: clk.Now().Add(time.Hour),
Cause: bans.CauseAdmin,
} }
server.Ledger.Load([]bans.Ban{kept}) server.Ledger.Load([]bans.Ban{kept})
+15 -30
View File
@@ -6,8 +6,6 @@ import (
"errors" "errors"
"io" "io"
"net/http" "net/http"
"os"
"reflect"
"slices" "slices"
"strings" "strings"
"sync/atomic" "sync/atomic"
@@ -16,7 +14,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/proxy" "sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -118,34 +115,27 @@ func wantAnswer(t *testing.T, got answer, body []byte) {
} }
} }
// wantRequestFields checks the log line's fields about the request. Its // wantRequestFields checks the log line's fields about the request.
// time, its id and its timings are checked only for being there.
func wantRequestFields(t *testing.T, line logLine, host string, sent, received int) { func wantRequestFields(t *testing.T, line logLine, host string, sent, received int) {
t.Helper() t.Helper()
hostname, _ := os.Hostname() want := requestlog.Line{
Type: "request", Time: line.Time, ClientIP: localhost, PeerIP: localhost,
want := withTimings(line, requestlog.Line{ Method: http.MethodPatch, Host: host, Path: rawPath, Query: rawQuery,
Type: requestType, Time: line.Time, Instance: hostname, Protocol: "HTTP/1.1", Status: http.StatusTeapot,
ClientIP: localhost, Method: http.MethodPatch, Scheme: plain, Host: host, UpstreamStatus: http.StatusTeapot, RequestBytes: int64(sent),
Path: rawPath, Query: rawQuery, Protocol: protocol,
Status: http.StatusTeapot, RequestBytes: int64(sent),
ResponseBytes: int64(received), UserAgent: "test-agent", ResponseBytes: int64(received), UserAgent: "test-agent",
RequestID: line.RequestID, PeerIP: localhost, ClientGroup: localhost + "/32", Action: requestlog.ActionForward, DurationTotal: line.DurationTotal,
ContentLength: int64(sent), ResponseContentType: "text/plain; charset=utf-8", DurationUpstreamTotal: line.DurationUpstreamTotal,
UpstreamStatus: http.StatusTeapot, Action: requestlog.ActionForward, }
Counts: ratelimit.Counts{Minute: 1, Hour: 1, Day: 1}, if line.Line != want {
})
if !reflect.DeepEqual(line.Line, want) {
t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want) t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
} }
_, err := time.Parse(time.RFC3339, line.Time) _, err := time.Parse(time.RFC3339, line.Time)
if err != nil || line.RequestID == "" || line.DurationTotal <= 0 || if err != nil || line.DurationTotal <= 0 || line.DurationUpstreamTotal <= 0 {
line.DurationUpstreamTotal == nil || *line.DurationUpstreamTotal <= 0 { t.Errorf("log line has time %q and durations %v and %v",
t.Errorf("log line has time %q, request_id %q and durations %v and %v", line.Time, line.DurationTotal, line.DurationUpstreamTotal)
line.Time, line.RequestID, line.DurationTotal,
line.fields["duration_upstream_total"])
} }
} }
@@ -381,13 +371,8 @@ func TestAnswers502WhenTheAppCannotBeReached(t *testing.T) {
addr, out := startProxy(t, "http://"+localhost+":1", nil) addr, out := startProxy(t, "http://"+localhost+":1", nil)
wantStatus(t, get(t, addr, "/"), http.StatusBadGateway) wantStatus(t, get(t, addr, "/"), http.StatusBadGateway)
wantLine(t, out.requestLine(t), http.StatusBadGateway,
line := out.requestLine(t) requestlog.ActionUpstreamError)
wantLine(t, line, http.StatusBadGateway, requestlog.ActionUpstreamError)
// There never was a connection to the app, nor an answer from it.
wantTimings(t, line, "duration_total", "duration_checks",
"duration_upstream_total")
logged := slices.ContainsFunc(out.lines(t), func(line map[string]any) bool { logged := slices.ContainsFunc(out.lines(t), func(line map[string]any) bool {
return line["type"] == "process" && line["msg"] == "request to the app failed" return line["type"] == "process" && line["msg"] == "request to the app failed"
-13
View File
@@ -17,7 +17,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
) )
// How smallwebwaf keeps connections to the app open between requests. // How smallwebwaf keeps connections to the app open between requests.
@@ -52,9 +51,6 @@ type Params struct {
// limits, bans are made and run out, and GeoJS's answers are kept, // limits, bans are made and run out, and GeoJS's answers are kept,
// normally time.Now in UTC, the time the state files give. // normally time.Now in UTC, the time the state files give.
Now func() time.Time Now func() time.Time
// Rules are the rule files' rules, which each request is checked
// against.
Rules *rules.Files
} }
// Server is the server smallwebwaf runs, with the parts of the proxy // Server is the server smallwebwaf runs, with the parts of the proxy
@@ -94,7 +90,6 @@ func New(params Params) *Server {
LimitBanDuration: params.Config.LimitBanDuration, LimitBanDuration: params.Config.LimitBanDuration,
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow, LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
MaxBanDuration: params.Config.MaxBanDuration, MaxBanDuration: params.Config.MaxBanDuration,
AttackBanDuration: params.Config.AttackBanDuration,
MaxBans: params.Config.MaxBans, MaxBans: params.Config.MaxBans,
}), }),
geojs: lookup.New(lookup.Params{ geojs: lookup.New(lookup.Params{
@@ -103,10 +98,8 @@ func New(params Params) *Server {
ProcessLog: params.ProcessLog, ProcessLog: params.ProcessLog,
Metrics: m, Metrics: m,
}), }),
rules: params.Rules,
} }
m.AddBansAndClients(h.ledger, h.limiter, params.Now) m.AddBansAndClients(h.ledger, h.limiter, params.Now)
m.AddRules(params.Rules)
return &Server{ return &Server{
Server: &http.Server{ Server: &http.Server{
@@ -141,7 +134,6 @@ type handler struct {
limiter *ratelimit.Limiter limiter *ratelimit.Limiter
ledger *bans.Ledger ledger *bans.Ledger
geojs *lookup.GeoJS geojs *lookup.GeoJS
rules *rules.Files
} }
// newTransport returns what carries requests to the app. It never goes // newTransport returns what carries requests to the app. It never goes
@@ -168,9 +160,6 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// a health checker is never refused. It does not ask the app. // a health checker is never refused. It does not ask the app.
if r.Method == http.MethodGet && r.URL.Path == HealthPath { if r.Method == http.MethodGet && r.URL.Path == HealthPath {
rq.line.Action = requestlog.ActionAdmin rq.line.Action = requestlog.ActionAdmin
// Set here rather than left to Go's server, which would set it only
// after the log line has taken the response's headers.
rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8")
_, _ = io.WriteString(rq.out, "ok\n") _, _ = io.WriteString(rq.out, "ok\n")
return return
@@ -180,8 +169,6 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
defer rq.addToHistory() defer rq.addToHistory()
refused := rq.check(r.Context()) refused := rq.check(r.Context())
rq.checked = time.Now()
if refused != nil { if refused != nil {
rq.answer(*refused) rq.answer(*refused)
+4 -32
View File
@@ -17,7 +17,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/proxy" "sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
) )
const ( const (
@@ -36,10 +35,6 @@ const (
// localhost is where every test server listens, and so the address // localhost is where every test server listens, and so the address
// smallwebwaf sees each test's requests come from. // smallwebwaf sees each test's requests come from.
localhost = "127.0.0.1" localhost = "127.0.0.1"
// requestType is the type that marks a request log line.
requestType = "request"
// protocol is the protocol of every test's requests.
protocol = "HTTP/1.1"
) )
// shortTimeoutSetting is shortTimeout as a setting's value. // shortTimeoutSetting is shortTimeout as a setting's value.
@@ -73,10 +68,6 @@ const (
maxBanDuration = "SWWAF_MAX_BAN_DURATION" maxBanDuration = "SWWAF_MAX_BAN_DURATION"
maxBans = "SWWAF_MAX_BANS" maxBans = "SWWAF_MAX_BANS"
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX" banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
instanceName = "SWWAF_INSTANCE_NAME"
logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS"
attackBanDuration = "SWWAF_ATTACK_BAN_DURATION"
rulesDir = "SWWAF_RULES_DIR"
) )
// output collects what smallwebwaf writes on stdout. // output collects what smallwebwaf writes on stdout.
@@ -93,14 +84,6 @@ func (o *output) Write(p []byte) (int, error) {
return o.buf.Write(p) return o.buf.Write(p)
} }
// text returns everything written so far.
func (o *output) text() string {
o.mu.Lock()
defer o.mu.Unlock()
return o.buf.String()
}
// lines returns every line written so far, decoded. // lines returns every line written so far, decoded.
func (o *output) lines(t *testing.T) []map[string]any { func (o *output) lines(t *testing.T) []map[string]any {
t.Helper() t.Helper()
@@ -140,7 +123,7 @@ func (o *output) requestLines(t *testing.T, count int) []logLine {
var found []logLine var found []logLine
for _, fields := range o.lines(t) { for _, fields := range o.lines(t) {
if fields["type"] == requestType { if fields["type"] == "request" {
found = append(found, decodeLine(t, fields)) found = append(found, decodeLine(t, fields))
} }
} }
@@ -215,15 +198,14 @@ func startProxyWithGeoJS(
} }
// startProxyWithClock is startProxyWithGeoJS with requests counted and // startProxyWithClock is startProxyWithGeoJS with requests counted and
// bans made by the time now tells, and returns the server as well. Unless // bans made by the time now tells, and returns the server as well.
// env sets SWWAF_RULES_DIR, it is an empty directory, of no rules.
func startProxyWithClock( func startProxyWithClock(
t *testing.T, appURL, geojsURL string, now func() time.Time, t *testing.T, appURL, geojsURL string, now func() time.Time,
env map[string]string, env map[string]string,
) (string, *output, *proxy.Server) { ) (string, *output, *proxy.Server) {
t.Helper() t.Helper()
settings := map[string]string{"SWWAF_UPSTREAM_URL": appURL, rulesDir: t.TempDir()} settings := map[string]string{"SWWAF_UPSTREAM_URL": appURL}
maps.Copy(settings, env) maps.Copy(settings, env)
cfg, err := config.FromEnvironment(func(name string) (string, bool) { cfg, err := config.FromEnvironment(func(name string) (string, bool) {
@@ -236,22 +218,12 @@ func startProxyWithClock(
} }
out := &output{} out := &output{}
processLog := requestlog.NewProcessLogger(out)
ruleFiles, err := rules.Load(rules.Params{
Dir: cfg.RulesDir, Enabled: cfg.RulesEnabled, ProcessLog: processLog,
})
if err != nil {
t.Fatalf("rule files: %v", err)
}
server := proxy.New(proxy.Params{ server := proxy.New(proxy.Params{
Config: cfg, Config: cfg,
RequestLog: out, RequestLog: out,
ProcessLog: processLog, ProcessLog: requestlog.NewProcessLogger(out),
GeoJSURL: geojsURL, GeoJSURL: geojsURL,
Now: now, Now: now,
Rules: ruleFiles,
}) })
listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0") listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
+4 -14
View File
@@ -6,7 +6,6 @@ import (
"testing" "testing"
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -98,9 +97,8 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
s.get(client, http.StatusOK, requestlog.ActionForward) s.get(client, http.StatusOK, requestlog.ActionForward)
line := s.request(client, "/assets/app.js", http.StatusOK, requestlog.ActionForward) line := s.request(client, "/assets/app.js", http.StatusOK, requestlog.ActionForward)
if line.LimitHit != "" || line.Counts != (ratelimit.Counts{}) { if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q and counts %+v, want neither", t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
line.LimitHit, line.Counts)
} }
// A path outside every prefix is counted: /assets is not under // A path outside every prefix is counted: /assets is not under
@@ -121,10 +119,6 @@ func TestRateLimitCountsPathsThatAreNotExempt(t *testing.T) {
for _, sent := range []string{ for _, sent := range []string{
// A prefix matches only at the start of the path. // A prefix matches only at the start of the path.
"/static/assets/app.js", "/static/assets/app.js",
// A prefix matches the path as sent: a router that matches the
// path as received does not take /%61ssets/x for a path under
// /assets/.
"/%61ssets/x",
// .. once percent-decoded: an app may act on these as /login, the // .. once percent-decoded: an app may act on these as /login, the
// last as a path under /sneak/app/ or as /assets/x. // last as a path under /sneak/app/ or as /assets/x.
"/assets/../login", "/assets/../login",
@@ -132,14 +126,10 @@ func TestRateLimitCountsPathsThatAreNotExempt(t *testing.T) {
"/assets/..%2Flogin", "/assets/..%2Flogin",
"/assets/..;/login", "/assets/..;/login",
"/sneak/app/src/branch/main/..%2F..%2F..%2F..%2F..%2F..%2Fassets/x", "/sneak/app/src/branch/main/..%2F..%2F..%2F..%2F..%2F..%2Fassets/x",
// Not under /assets/ as sent: Go's router takes /assets%2Fx for one // An encoded slash or a backslash: Go's router takes /assets%2Fx
// path segment, not a path under /assets/. // for one path segment, not a path under /assets/.
"/assets%2Fx", "/assets%2Fx",
"/assets%2fx", "/assets%2fx",
// Under /assets/ as sent, but holding an encoded slash, in either
// case, or a backslash: never exempt, whatever the prefix.
"/assets/x%2Fy",
"/assets/x%2fy",
`/assets/x\y`, `/assets/x\y`,
} { } {
t.Run(sent, func(t *testing.T) { t.Run(sent, func(t *testing.T) {
+43 -138
View File
@@ -49,9 +49,7 @@ type request struct {
peer netip.Addr peer netip.Addr
peerTrusted bool peerTrusted bool
start time.Time start time.Time
// checked is when the checks were done, and upstreamStart when the // upstreamStart is when the request was handed to the app.
// request was handed to the app.
checked time.Time
upstreamStart time.Time upstreamStart time.Time
// cancel ends the request to the app. // cancel ends the request to the app.
cancel context.CancelFunc cancel context.CancelFunc
@@ -61,34 +59,26 @@ type request struct {
complete bool complete bool
// mu guards what follows. The timeouts run on goroutines of their // mu guards what follows. The timeouts run on goroutines of their
// own, and the transport starts and stops them, and notes the times // own, and the transport starts and stops them from its own; once
// below, from its own; once timersStopped is set, none of the timeouts // timersStopped is set, none of them acts any more.
// acts any more.
mu sync.Mutex mu sync.Mutex
timersStopped bool timersStopped bool
clientRequestTimer *time.Timer clientRequestTimer *time.Timer
upstreamRequestTimer *time.Timer upstreamRequestTimer *time.Timer
upstreamResponseTimer *time.Timer upstreamResponseTimer *time.Timer
// connected is when there was a connection to the app, requestSent // requestSent is when the app had been sent the whole request.
// when the app had been sent the whole request, and answerStarted requestSent time.Time
// when the first byte of its answer arrived.
connected time.Time
requestSent time.Time
answerStarted time.Time
} }
// newRequest starts handling r: it notes the time, counts the request as // newRequest starts handling r: it notes the time, counts the request as
// under way, works out the client, and starts the log line with what is // under way, and works out the client.
// known of the request.
func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request { func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
h.metrics.RequestStarted() h.metrics.RequestStarted()
start := time.Now() start := time.Now()
peer := peerAddress(r) peer := peerAddress(r)
trusted := h.config.TrustedProxies trusted := h.config.TrustedProxies
peerTrusted := isInside(peer, trusted) client := clientAddress(peer, r.Header.Values("X-Forwarded-For"), trusted)
forwardedFor := r.Header.Values("X-Forwarded-For")
client := clientAddress(peer, forwardedFor, trusted)
rq := &request{ rq := &request{
h: h, h: h,
@@ -97,37 +87,22 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
out: &responseWriter{ResponseWriter: w}, out: &responseWriter{ResponseWriter: w},
client: client, client: client,
peer: peer, peer: peer,
peerTrusted: peerTrusted, peerTrusted: isInside(peer, trusted),
start: start, start: start,
line: requestlog.Line{ line: requestlog.Line{
Time: requestlog.FormatTime(start), Time: requestlog.FormatTime(start),
Instance: h.config.InstanceName, ClientIP: client.String(),
ClientIP: client.String(), PeerIP: peer.String(),
Method: r.Method, Method: r.Method,
Scheme: scheme(r, peerTrusted), Host: r.Host,
Host: r.Host, Path: r.URL.EscapedPath(),
Path: r.URL.EscapedPath(), Query: r.URL.RawQuery,
Query: r.URL.RawQuery, Protocol: r.Proto,
Protocol: r.Proto, Referer: r.Referer(),
Referer: r.Referer(), UserAgent: r.UserAgent(),
UserAgent: r.UserAgent(), Action: requestlog.ActionForward,
RequestID: requestID(r, peerTrusted),
PeerIP: peer.String(),
ForwardedFor: strings.Join(forwardedFor, ", "),
ClientGroup: clientGroup(client).String(),
ContentType: r.Header.Get("Content-Type"),
RequestHeaders: requestHeaders(r, h.config.LogRequestHeaders),
HasAuthorization: len(r.Header.Values("Authorization")) > 0,
HasCookie: len(r.Header.Values("Cookie")) > 0,
Action: requestlog.ActionForward,
}, },
} }
// A length of -1 is a body whose length was not announced.
if r.ContentLength > 0 {
rq.line.ContentLength = r.ContentLength
}
if r.Body != http.NoBody { if r.Body != http.NoBody {
rq.body = &requestBody{body: limitBody(r.Body, h.config.RequestMaxBytes), rq: rq} rq.body = &requestBody{body: limitBody(r.Body, h.config.RequestMaxBytes), rq: rq}
} }
@@ -135,47 +110,23 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
return rq return rq
} }
// requestHeaders returns the headers of r that names lists, by name in
// lower case, each with its values joined by ", ". Authorization, Cookie
// and Set-Cookie are never among them, whatever names says.
func requestHeaders(r *http.Request, names []string) map[string]string {
headers := map[string]string{}
for _, name := range names {
switch name {
case "authorization", "cookie", "set-cookie":
continue
}
values := r.Header.Values(name)
if len(values) > 0 {
headers[name] = strings.Join(values, ", ")
}
}
return headers
}
// check is the one place where a request can be refused once its client // check is the one place where a request can be refused once its client
// is known, before its body is read or anything reaches the app. It // is known, before its body is read or anything reaches the app. It
// returns nil to let the request through. The checks of checkClient come // returns nil to let the request through. The checks of checkClient come
// first, answered with SWWAF_BAN_RESPONSE, or 403 for a block rule, and // first, answered with SWWAF_BAN_RESPONSE, and then the size limit, so
// then the size limit, so that a request the rate limits count is counted // that a request the rate limits count is counted even when it is
// even when it is refused for its size. In observe mode a request // refused for its size. In observe mode a request checkClient refuses
// checkClient refuses goes on to the size limit like any other. ctx is // goes on to the size limit like any other. ctx is the request's own
// the request's own context. // context.
func (rq *request) check(ctx context.Context) *refusal { func (rq *request) check(ctx context.Context) *refusal {
action := rq.checkClient(ctx) action := rq.checkClient(ctx)
if action != "" {
if !rq.h.config.Observe {
return rq.banResponse(action)
}
switch {
case action == "":
case rq.h.config.Observe:
// The log line names what enforce mode would have done. // The log line names what enforce mode would have done.
rq.line.WouldAction = action rq.line.WouldAction = action
case action == requestlog.ActionRuleBlocked:
return &refusal{status: http.StatusForbidden, action: action}
default:
return rq.banResponse(action)
} }
maxBytes := rq.h.config.RequestMaxBytes maxBytes := rq.h.config.RequestMaxBytes
@@ -198,8 +149,8 @@ func (rq *request) check(ctx context.Context) *refusal {
// request any of them refuses is not counted for the rate limits. Then // request any of them refuses is not counted for the rate limits. Then
// come the rate limits, unless the client is in // come the rate limits, unless the client is in
// SWWAF_RATE_LIMIT_EXEMPT_NETS or the request's path is exempt under // SWWAF_RATE_LIMIT_EXEMPT_NETS or the request's path is exempt under
// SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request is counted, // SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request is counted.
// and last the rule files. ctx is the request's own context. // ctx is the request's own context.
func (rq *request) checkClient(ctx context.Context) string { func (rq *request) checkClient(ctx context.Context) string {
cfg := rq.h.config cfg := rq.h.config
if isInside(rq.client, cfg.AllowNets) { if isInside(rq.client, cfg.AllowNets) {
@@ -226,30 +177,28 @@ func (rq *request) checkClient(ctx context.Context) string {
return requestlog.ActionRateLimited return requestlog.ActionRateLimited
} }
return rq.checkRules(now) return ""
} }
// pathExempt reports whether the rate limits leave out a request for u // pathExempt reports whether the rate limits leave out a request for u
// because of SWWAF_RATE_LIMIT_EXEMPT_PATHS: whether its path as sent, the // because of SWWAF_RATE_LIMIT_EXEMPT_PATHS: whether its path,
// path the app receives, not percent-decoded, starts with one of // percent-decoded, starts with one of prefixes. A request whose decoded
// prefixes, so that /%61ssets/x is not under /assets/ for an app whose // path contains .. anywhere or a backslash, or whose path as sent holds
// router matches the path as received. A request whose decoded path // an encoded slash (%2F or %2f), never is, since an app may act on it as
// contains .. anywhere or a backslash, or whose path as sent holds an // a path outside every prefix: /assets/..%2Flogin as /login, or
// encoded slash (%2F or %2f), never is, since an app may act on it as a // /assets%2Fx as one path segment, as Go's router does.
// path outside every prefix: /assets/..%2Flogin as /login, or /assets%2Fx
// as one path segment, as Go's router does.
func pathExempt(u *url.URL, prefixes []string) bool { func pathExempt(u *url.URL, prefixes []string) bool {
decoded := u.Path decoded := u.Path
// EscapedPath is the path as the app receives it, not decoded. // EscapedPath is the path as the app receives it, not decoded.
sent := u.EscapedPath() sent := strings.ToLower(u.EscapedPath())
if strings.Contains(decoded, "..") || strings.Contains(decoded, `\`) || if strings.Contains(decoded, "..") || strings.Contains(decoded, `\`) ||
strings.Contains(strings.ToLower(sent), "%2f") { strings.Contains(sent, "%2f") {
return false return false
} }
return slices.ContainsFunc(prefixes, func(prefix string) bool { return slices.ContainsFunc(prefixes, func(prefix string) bool {
return strings.HasPrefix(sent, prefix) return strings.HasPrefix(decoded, prefix)
}) })
} }
@@ -261,9 +210,7 @@ func (rq *request) forward(ctx context.Context) {
rq.cancel = cancel rq.cancel = cancel
ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{ ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{
GotConn: rq.gotConn, WroteRequest: rq.wroteRequest,
WroteRequest: rq.wroteRequest,
GotFirstResponseByte: rq.gotFirstResponseByte,
}) })
out := rq.in.WithContext(ctx) out := rq.in.WithContext(ctx)
@@ -286,8 +233,7 @@ func (rq *request) forward(ctx context.Context) {
} }
// rewrite makes the request the app receives: the client's request, // rewrite makes the request the app receives: the client's request,
// unchanged, sent to SWWAF_UPSTREAM_URL, with the forwarded headers and // unchanged, sent to SWWAF_UPSTREAM_URL, with the forwarded headers set.
// the request's id set.
func (rq *request) rewrite(pr *httputil.ProxyRequest) { func (rq *request) rewrite(pr *httputil.ProxyRequest) {
upstream := rq.h.config.UpstreamURL upstream := rq.h.config.UpstreamURL
pr.Out.URL.Scheme = upstream.Scheme pr.Out.URL.Scheme = upstream.Scheme
@@ -296,7 +242,6 @@ func (rq *request) rewrite(pr *httputil.ProxyRequest) {
// the query as the client sent it. // the query as the client sent it.
pr.Out.URL.RawQuery = pr.In.URL.RawQuery pr.Out.URL.RawQuery = pr.In.URL.RawQuery
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted) setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
pr.Out.Header.Set(requestIDHeader, rq.line.RequestID)
} }
// modifyResponse looks at the app's answer before ReverseProxy passes it // modifyResponse looks at the app's answer before ReverseProxy passes it
@@ -310,7 +255,6 @@ func (rq *request) modifyResponse(res *http.Response) error {
// connection it takes over, not through rq.out. // connection it takes over, not through rq.out.
rq.stopTimers() rq.stopTimers()
rq.out.status = res.StatusCode rq.out.status = res.StatusCode
rq.line.Websocket = true
return nil return nil
} }
@@ -406,10 +350,6 @@ func (rq *request) finish() {
line := &rq.line line := &rq.line
line.Status = rq.out.status line.Status = rq.out.status
line.ResponseBytes = rq.out.bytes line.ResponseBytes = rq.out.bytes
header := rq.out.Header()
line.ResponseContentType = header.Get("Content-Type")
line.CacheControl = header.Get("Cache-Control")
line.Location = header.Get("Location")
if rq.body != nil { if rq.body != nil {
line.RequestBytes = rq.body.bytes.Load() line.RequestBytes = rq.body.bytes.Load()
@@ -434,18 +374,12 @@ func (rq *request) finish() {
now := time.Now() now := time.Now()
duration := now.Sub(rq.start) duration := now.Sub(rq.start)
line.DurationTotal = requestlog.Milliseconds(duration) line.DurationTotal = requestlog.Milliseconds(duration)
line.DurationChecks = timing(rq.start, rq.checked)
var upstreamDuration time.Duration var upstreamDuration time.Duration
if !rq.upstreamStart.IsZero() { if !rq.upstreamStart.IsZero() {
upstreamDuration = now.Sub(rq.upstreamStart) upstreamDuration = now.Sub(rq.upstreamStart)
line.DurationUpstreamTotal = new(requestlog.Milliseconds(upstreamDuration)) line.DurationUpstreamTotal = requestlog.Milliseconds(upstreamDuration)
rq.mu.Lock()
line.DurationUpstreamConnect = timing(rq.upstreamStart, rq.connected)
line.DurationUpstreamFirstByte = timing(rq.upstreamStart, rq.answerStarted)
rq.mu.Unlock()
} }
// Counted before the log line is written, so that the metrics count // Counted before the log line is written, so that the metrics count
@@ -458,17 +392,6 @@ func (rq *request) finish() {
} }
} }
// timing is the time from start to end in milliseconds, for one of the
// log line's timings, or nil when end is zero: what it times never
// happened.
func timing(start, end time.Time) *float64 {
if end.IsZero() {
return nil
}
return new(requestlog.Milliseconds(end.Sub(start)))
}
// addToHistory adds the request, which has ended, to its client's // addToHistory adds the request, which has ended, to its client's
// history. // history.
func (rq *request) addToHistory() { func (rq *request) addToHistory() {
@@ -578,24 +501,6 @@ func (rq *request) bodyReceived() {
stopTimer(rq.clientRequestTimer) stopTimer(rq.clientRequestTimer)
} }
// gotConn is called once there is a connection to the app, a new one or
// one kept open from an earlier request.
func (rq *request) gotConn(httptrace.GotConnInfo) {
rq.mu.Lock()
defer rq.mu.Unlock()
rq.connected = time.Now()
}
// gotFirstResponseByte is called once the first byte of the app's answer
// has arrived.
func (rq *request) gotFirstResponseByte() {
rq.mu.Lock()
defer rq.mu.Unlock()
rq.answerStarted = time.Now()
}
// wroteRequest is called once the app has been sent the whole request: // wroteRequest is called once the app has been sent the whole request:
// the request timeouts end and SWWAF_UPSTREAM_RESPONSE_TIMEOUT starts. // the request timeouts end and SWWAF_UPSTREAM_RESPONSE_TIMEOUT starts.
func (rq *request) wroteRequest(info httptrace.WroteRequestInfo) { func (rq *request) wroteRequest(info httptrace.WroteRequestInfo) {
-368
View File
@@ -1,368 +0,0 @@
package proxy_test
import (
"io"
"maps"
"math"
"net/http"
"reflect"
"slices"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
const (
// requestIDHeader carries the request's id.
requestIDHeader = "X-Request-ID"
// instance is the SWWAF_INSTANCE_NAME a test sets.
instance = "fsn1app1/gitea"
// ipv6Client is a client on IPv6, and ipv6Group the netblock the rate
// limits count it as.
ipv6Client = "2001:db8::7"
ipv6Group = "2001:db8::/64"
)
func TestLogLineHasEachFieldWhereItApplies(t *testing.T) {
t.Parallel()
received := make(chan string, 2) // the request ids the app received
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
received <- r.Header.Get(requestIDHeader)
_, _ = io.Copy(io.Discard, r.Body)
if r.URL.Path != "/full" {
w.WriteHeader(http.StatusNoContent)
return
}
w.Header().Set("Content-Type", "text/html")
w.Header().Set("Cache-Control", "no-store")
w.Header().Set("Location", "/elsewhere")
w.WriteHeader(http.StatusFound)
_, _ = io.WriteString(w, "moved")
})
addr, out := startProxy(t, app.URL, map[string]string{
trustedProxies: trustLocalhost,
rateLimitExemptNets: localhost,
instanceName: instance,
logRequestHeaders: "Accept,x-custom,Authorization,cookie,SET-COOKIE",
})
// This request comes from ipv6Client through a trusted proxy, with a
// body and each header the log line looks at, and is answered with a
// redirect.
conn := dial(t, addr)
send(t, conn, "POST /full HTTP/1.1\r\nHost: "+appHost+"\r\n"+
forwardedFor+": 198.51.100.7, "+ipv6Client+"\r\n"+
forwardedProto+": "+secure+"\r\n"+requestIDHeader+": from-traefik\r\n"+
"Content-Type: application/x-www-form-urlencoded\r\nContent-Length: 3\r\n"+
"Accept: text/html\r\nX-Custom: one\r\nX-Custom: two\r\n"+
"Authorization: Bearer secret-token\r\nCookie: session=secret-cookie\r\n"+
"Set-Cookie: secret-set-cookie\r\n\r\na=b")
wantStatus(t, readResponse(t, conn), http.StatusFound)
// A request's log line can come after its answer: each is waited for
// before the next request, so that the lines are in order.
full := out.requestLines(t, 1)[0]
// This one comes from 127.0.0.1, which the rate limits do not count,
// with a body of 4 bytes whose length it does not announce, so that its
// request_bytes is not its content_length, and no header the log line
// looks at, and is answered with 204 and no header.
conn = dial(t, addr)
send(t, conn, "POST /bare HTTP/1.1\r\nHost: "+appHost+"\r\n"+
"Transfer-Encoding: chunked\r\n\r\n4\r\nbody\r\n0\r\n\r\n")
wantStatus(t, readResponse(t, conn), http.StatusNoContent)
bare := out.requestLines(t, 2)[1]
wantFullLine(t, full)
wantBareLine(t, bare)
for _, line := range []logLine{full, bare} {
got := <-received
if got != line.RequestID {
t.Errorf("the app received request id %q, the log line has %q",
got, line.RequestID)
}
}
if strings.Contains(out.text(), "secret") {
t.Errorf("a value of Authorization, Cookie or Set-Cookie is logged:\n%s",
out.text())
}
}
// wantFullLine checks the log line of the request with every header the
// line looks at. Its timings are checked by TestTimingsAreInOrder.
func wantFullLine(t *testing.T, line logLine) {
t.Helper()
headers := map[string]string{"accept": "text/html", "x-custom": "one, two"}
want := withTimings(line, requestlog.Line{
Type: requestType, Time: line.Time, Instance: instance,
ClientIP: ipv6Client, Method: http.MethodPost, Scheme: secure,
Host: appHost, Path: "/full", Protocol: protocol,
Status: http.StatusFound, RequestBytes: 3, ResponseBytes: 5,
RequestID: "from-traefik", PeerIP: localhost,
ForwardedFor: "198.51.100.7, " + ipv6Client, ClientGroup: ipv6Group,
ContentType: "application/x-www-form-urlencoded", ContentLength: 3,
RequestHeaders: headers, HasAuthorization: true, HasCookie: true,
ResponseContentType: "text/html", UpstreamStatus: http.StatusFound,
CacheControl: "no-store", Location: "/elsewhere",
Action: requestlog.ActionForward,
Counts: ratelimit.Counts{Minute: 1, Hour: 1, Day: 1},
})
if !reflect.DeepEqual(line.Line, want) {
t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
}
}
// wantBareLine checks the log line of the request with none of them, and
// that the fields that do not apply to it are left out.
func wantBareLine(t *testing.T, line logLine) {
t.Helper()
want := withTimings(line, requestlog.Line{
Type: requestType, Time: line.Time, Instance: instance,
ClientIP: localhost, Method: http.MethodPost, Scheme: plain,
Host: appHost, Path: "/bare", Protocol: protocol,
Status: http.StatusNoContent, RequestBytes: 4, RequestID: line.RequestID,
PeerIP: localhost, ClientGroup: localhost + "/32",
UpstreamStatus: http.StatusNoContent, Action: requestlog.ActionForward,
})
if !reflect.DeepEqual(line.Line, want) || line.RequestID == "" {
t.Errorf("log line\n%+v\nwant\n%+v, with a request id", line.Line, want)
}
for _, name := range []string{
"forwarded_for", "content_type", "content_length", "request_headers",
"has_authorization", "has_cookie", "websocket", "response_content_type",
"cache_control", "location", "counts",
} {
_, present := line.fields[name]
if present {
t.Errorf("log line has %s, which does not apply", name)
}
}
}
// withTimings returns want with the timings of line.
func withTimings(line logLine, want requestlog.Line) requestlog.Line {
want.DurationTotal = line.DurationTotal
want.DurationChecks = line.DurationChecks
want.DurationUpstreamConnect = line.DurationUpstreamConnect
want.DurationUpstreamFirstByte = line.DurationUpstreamFirstByte
want.DurationUpstreamTotal = line.DurationUpstreamTotal
return want
}
func TestHasAuthorizationAndHasCookieEachComeFromTheirOwnHeader(t *testing.T) {
t.Parallel()
const hasAuthorization, hasCookie = "has_authorization", "has_cookie"
for _, tc := range []struct{ header, field, other string }{
{"Authorization", hasAuthorization, hasCookie},
{"Cookie", hasCookie, hasAuthorization},
} {
t.Run("only "+tc.header, func(t *testing.T) {
t.Parallel()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
addr, out := startProxy(t, app.URL, nil)
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
req.Header.Set(tc.header, "secret")
wantStatus(t, do(t, req), http.StatusOK)
line := out.requestLine(t)
_, otherPresent := line.fields[tc.other]
if line.fields[tc.field] != true || otherPresent {
t.Errorf("log line has %s %v and %s %v, want true and none",
tc.field, line.fields[tc.field], tc.other, line.fields[tc.other])
}
})
}
}
func TestRequestIDAndSchemeComeOnlyFromATrustedProxy(t *testing.T) {
t.Parallel()
const sentID = "from-traefik"
sent := http.Header{requestIDHeader: {sentID}, forwardedProto: {secure}}
trusted := map[string]string{trustedProxies: trustLocalhost}
for _, tc := range []struct {
name string
env map[string]string
header http.Header
// wantID is the request id logged, "" for a new one.
wantID, wantScheme string
}{
{"a trusted proxy's are kept", trusted, sent, sentID, secure},
{"without them, the id is new and the scheme http", trusted, nil, "", plain},
{"another peer's are replaced", nil, sent, "", plain},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
received := make(chan string, 2)
app := startApp(t, func(_ http.ResponseWriter, r *http.Request) {
received <- r.Header.Get(requestIDHeader)
})
addr, out := startProxy(t, app.URL, tc.env)
// Two requests, so that two new ids can be told apart.
ids := make([]string, 0, 2)
for i := range 2 {
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
maps.Copy(req.Header, tc.header)
wantStatus(t, do(t, req), http.StatusOK)
line := out.requestLines(t, i+1)[i]
ids = append(ids, line.RequestID)
got := <-received
if line.RequestID != got || line.Scheme != tc.wantScheme {
t.Errorf("log line has request_id %q and scheme %q, and the "+
"app received id %q; want the same id and scheme %q",
line.RequestID, line.Scheme, got, tc.wantScheme)
}
}
switch {
case tc.wantID != "" && (ids[0] != tc.wantID || ids[1] != tc.wantID):
t.Errorf("request ids %q, want %q", ids, tc.wantID)
case tc.wantID == "" && (slices.Contains(ids, sentID) ||
slices.Contains(ids, "") || ids[0] == ids[1]):
t.Errorf("request ids %q, want two new ones", ids)
}
})
}
}
func TestTimingsAreInOrder(t *testing.T) {
t.Parallel()
const denied = "192.0.2.50" // in SWWAF_DENY_NETS
app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
// The pauses set the times apart; a hold-up of the test only
// lengthens them.
time.Sleep(time.Millisecond)
w.WriteHeader(http.StatusOK)
_ = http.NewResponseController(w).Flush()
time.Sleep(time.Millisecond)
_, _ = io.WriteString(w, "done")
})
addr, out := startProxy(t, app.URL, map[string]string{
trustedProxies: trustLocalhost,
denyNets: denied,
})
// Each log line is waited for before the next request, so that the
// lines are in order.
wantStatus(t, get(t, addr, "/"), http.StatusOK)
forwarded := out.requestLines(t, 1)[0]
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
req.Header.Set(forwardedFor, denied)
wantStatus(t, do(t, req), http.StatusForbidden)
refused := out.requestLines(t, 2)[1]
wantStatus(t, get(t, addr, proxy.HealthPath), http.StatusOK)
health := out.requestLines(t, 3)[2]
// A request passed to the app has every timing; one refused, none of
// the app's; the health check, which runs no check, only the total.
wantTimings(t, forwarded, "duration_total", "duration_checks",
"duration_upstream_connect", "duration_upstream_first_byte",
"duration_upstream_total")
wantTimings(t, refused, "duration_total", "duration_checks")
wantTimings(t, health, "duration_total")
if t.Failed() {
return
}
// In whole microseconds, as they are logged, so that the sum below is
// exact.
total := microseconds(forwarded.DurationTotal)
checks := microseconds(*forwarded.DurationChecks)
connect := microseconds(*forwarded.DurationUpstreamConnect)
firstByte := microseconds(*forwarded.DurationUpstreamFirstByte)
upstream := microseconds(*forwarded.DurationUpstreamTotal)
// The checks end before the request is handed to the app, and the
// connection comes before the answer, which the app ends after a
// pause.
if checks+upstream > total || connect >= firstByte || firstByte >= upstream {
t.Errorf("timings in microseconds: total %d, checks %d, connect %d, "+
"first byte %d, upstream total %d", total, checks, connect, firstByte,
upstream)
}
if *refused.DurationChecks > refused.DurationTotal {
t.Errorf("refused request's checks took %v of %v milliseconds",
*refused.DurationChecks, refused.DurationTotal)
}
}
// wantTimings checks that the timings named are the only ones line has.
func wantTimings(t *testing.T, line logLine, want ...string) {
t.Helper()
var got []string
for name := range line.fields {
if strings.HasPrefix(name, "duration_") {
got = append(got, name)
}
}
slices.Sort(got)
slices.Sort(want)
if !slices.Equal(got, want) {
t.Errorf("log line of %s has timings %v, want %v", line.Path, got, want)
}
}
// microseconds is a timing in whole microseconds.
func microseconds(milliseconds float64) int64 {
return int64(math.Round(milliseconds * 1000))
}
func TestLogsAnUpgradedConnection(t *testing.T) {
t.Parallel()
app := startApp(t, echoAfterUpgrade)
addr, out := startProxy(t, app.URL, nil)
conn := dial(t, addr)
send(t, conn, "GET /socket HTTP/1.1\r\nHost: app\r\n"+
"Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
wantStatus(t, readResponse(t, conn), http.StatusSwitchingProtocols)
_ = conn.Close()
line := out.requestLine(t)
if line.fields["websocket"] != true {
t.Errorf("log line has websocket %v, want true", line.fields["websocket"])
}
}
-233
View File
@@ -1,233 +0,0 @@
package proxy_test
import (
"net/http"
"net/netip"
"os"
"path/filepath"
"slices"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// testRules are the rules most tests here load: a block rule for
// /blocked and a ban rule for /.env.
const testRules = `
blocked path block ^/blocked$
probe path ban ^/\.env$
`
func TestEachRuleAction(t *testing.T) {
t.Parallel()
s, clk, server := startWithClock(t, "", map[string]string{
rulesDir: writeRules(t, "noted path log ^/\n"+testRules),
banResponse: "429",
})
start := clk.Now()
// A log rule notes its match, and lets the request through.
line := s.get(client, http.StatusOK, requestlog.ActionForward)
wantRuleIDs(t, line, "noted")
// A block rule refuses with 403, whatever SWWAF_BAN_RESPONSE is, and
// bans no one.
line = s.request(client, "/blocked", http.StatusForbidden,
requestlog.ActionRuleBlocked)
wantRuleIDs(t, line, "noted", "blocked")
s.get(client, http.StatusOK, requestlog.ActionForward)
// A ban rule refuses with SWWAF_BAN_RESPONSE, and bans the client for
// seven days, the default.
line = s.request(client, "/.env", http.StatusTooManyRequests, requestlog.ActionBanned)
wantRuleIDs(t, line, "noted", "probe")
if line.BanExpires != requestlog.FormatTime(start.Add(7*24*time.Hour)) {
t.Errorf("log line has ban_expires %q, want seven days on", line.BanExpires)
}
netblock := netip.MustParsePrefix(client + "/32")
want := bans.Ban{
Netblock: netblock,
Start: start,
Expires: start.Add(7 * 24 * time.Hour),
Cause: bans.CauseAttack,
Reason: "matched the rule probe",
Notes: bans.Notes{
RuleID: "probe",
Target: "path",
Request: bans.Request{
Time: start,
Method: http.MethodGet,
Host: appHost,
Path: "/.env",
Status: http.StatusTooManyRequests,
UserAgent: userAgent,
},
// The four requests up to and including the probe.
Requests: 4,
},
}
got := server.Ledger.Bans(netblock)
if len(got) != 1 || got[0] != want {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
// The next request is refused under the ban, without being checked
// against the rules, and makes the ban permanent.
clk.advance(time.Hour)
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
wantRuleIDs(t, line)
if line.BanExpires != permanent {
t.Errorf("log line has ban_expires %q, want permanent", line.BanExpires)
}
clk.advance(365 * 24 * time.Hour)
s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
}
func TestNextClearSignOfAttackAfterABanBansPermanently(t *testing.T) {
t.Parallel()
s, clk, _ := startWithClock(t, "", map[string]string{
rulesDir: writeRules(t, testRules),
attackBanDuration: "1h",
})
// The first probe bans for SWWAF_ATTACK_BAN_DURATION.
line := s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
if line.BanExpires != requestlog.FormatTime(clk.Now().Add(time.Hour)) {
t.Errorf("log line has ban_expires %q, want an hour on", line.BanExpires)
}
// Once that ban has run out without a request, the client is served,
// and its next probe bans it for good.
clk.advance(time.Hour)
s.get(client, http.StatusOK, requestlog.ActionForward)
line = s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
if line.BanExpires != permanent {
t.Errorf("log line has ban_expires %q, want permanent", line.BanExpires)
}
}
func TestRulesComeAfterTheOtherChecks(t *testing.T) {
t.Parallel()
const (
allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
exempt = "192.0.2.50" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
)
s, _, server := startWithClock(t, "", map[string]string{
rulesDir: writeRules(t, testRules),
allowNets: allowed,
rateLimitExemptNets: exempt,
rateLimitPerMinute: "1",
})
// A client in SWWAF_ALLOW_NETS is not checked.
line := s.request(allowed, "/.env", http.StatusOK, requestlog.ActionForward)
wantRuleIDs(t, line)
// A probe over the rate limit breaks the limit before any rule sees
// it.
s.get(client, http.StatusOK, requestlog.ActionForward)
line = s.request(client, "/.env", http.StatusForbidden, requestlog.ActionRateLimited)
wantRuleIDs(t, line)
limitBan := server.Ledger.Bans(netip.MustParsePrefix(client + "/32"))
if len(limitBan) != 1 || limitBan[0].Cause != bans.CauseLimit {
t.Errorf("bans %+v, want one for a broken limit", limitBan)
}
// A client the rate limits do not apply to is still checked.
s.get(exempt, http.StatusOK, requestlog.ActionForward)
s.request(exempt, "/.env", http.StatusForbidden, requestlog.ActionBanned)
}
func TestObserveModeLogsWhatTheRulesWouldDo(t *testing.T) {
t.Parallel()
s, _, server := startWithClock(t, "", map[string]string{
rulesDir: writeRules(t, testRules),
mode: observe,
})
line := s.request(client, "/blocked", http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionRuleBlocked)
wantRuleIDs(t, line, "blocked")
line = s.request(client, "/.env", http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionBanned)
wantRuleIDs(t, line, "probe")
if line.BanExpires != "" {
t.Errorf("log line has ban_expires %q, want none", line.BanExpires)
}
// No ban was made.
line = s.get(client, http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, "")
if got := server.Ledger.Snapshot(); len(got) != 0 {
t.Errorf("bans %+v, want none", got)
}
}
func TestMetricsCountRuleMatchesAndBansForAnAttack(t *testing.T) {
t.Parallel()
const scraper = "192.0.2.200"
s, _, _ := startWithClock(t, "", map[string]string{
rulesDir: writeRules(t, testRules),
metricsToken: token,
})
s.request(client, "/blocked", http.StatusForbidden, requestlog.ActionRuleBlocked)
s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
metrics := s.scrape(scraper)
wantMetric(t, metrics,
`smallwebwaf_rule_matches_total{action="block",rule_id="blocked"}`, 1)
wantMetric(t, metrics,
`smallwebwaf_rule_matches_total{action="ban",rule_id="probe"}`, 1)
wantMetric(t, metrics, "smallwebwaf_rules_loaded", 2)
wantMetric(t, metrics,
`smallwebwaf_requests_total{action="rule_blocked",status_class="4xx"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="attack"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit"}`, 0)
wantMetric(t, metrics, "smallwebwaf_permanent_bans", 1)
}
// writeRules writes content as a rule file into a new directory, and
// returns the directory.
func writeRules(t *testing.T, content string) string {
t.Helper()
dir := t.TempDir()
err := os.WriteFile(filepath.Join(dir, "test.rules"), []byte(content), 0o600)
if err != nil {
t.Fatalf("write the rule file: %v", err)
}
return dir
}
// wantRuleIDs checks the request log line's rule_ids.
func wantRuleIDs(t *testing.T, line logLine, want ...string) {
t.Helper()
if !slices.Equal(line.RuleIDs, want) {
t.Errorf("log line has rule_ids %v, want %v", line.RuleIDs, want)
}
}
-40
View File
@@ -1,40 +0,0 @@
package proxy
import (
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// checkRules checks the request against the rules of the rule files at
// now, notes the ids of those it matches in the log line, and returns the
// action of the rule that refuses it, ActionRuleBlocked for a block rule
// and ActionBanned for a ban rule, or "" when none does. In enforce mode
// a ban rule bans the client's netblock for a clear sign of attack.
func (rq *request) checkRules(now time.Time) string {
matched := rq.h.rules.Match(rq.in)
for _, rule := range matched {
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
rq.h.metrics.RuleMatched(rule.ID, rule.Action)
}
if len(matched) == 0 {
return ""
}
// Only the last rule matched can refuse the request.
switch last := matched[len(matched)-1]; last.Action {
case rules.ActionBlock:
return requestlog.ActionRuleBlocked
case rules.ActionBan:
if !rq.h.config.Observe {
rq.banForAttack(now, last)
}
return requestlog.ActionBanned
default:
return ""
}
}
+9 -22
View File
@@ -149,39 +149,26 @@ type Hit struct {
Requests float64 Requests float64
} }
// Counts are a client's requests in the minute, the hour and the day that
// end at a request, that request included.
type Counts struct {
Minute float64 `json:"minute"`
Hour float64 `json:"hour"`
Day float64 `json:"day"`
}
// Count counts a request from client at now, in every window, whether or // Count counts a request from client at now, in every window, whether or
// not it is refused, and returns the client's requests in each window. It // not it is refused. It reports whether the request takes the client over
// reports whether the request takes the client over a limit, and the // a limit, and the window whose limit it goes over, the shortest if it is
// window whose limit it goes over, the shortest if it is over several. // over several.
func (l *Limiter) Count(client netip.Prefix, now time.Time) (Counts, Hit, bool) { func (l *Limiter) Count(client netip.Prefix, now time.Time) (Hit, bool) {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
var ( var hit Hit
requests [3]float64
hit Hit
)
for i, b := range l.get(client).buckets() { for i, b := range l.get(client).buckets() {
w := l.windows[i] w := l.windows[i]
requests[i] = b.add(now, w.length) requests := b.add(now, w.length)
if hit.Window == "" && w.limit > 0 && requests[i] > float64(w.limit) { if hit.Window == "" && w.limit > 0 && requests > float64(w.limit) {
hit = Hit{Window: w.name, Limit: w.limit, Requests: requests[i]} hit = Hit{Window: w.name, Limit: w.limit, Requests: requests}
} }
} }
counts := Counts{Minute: requests[0], Hour: requests[1], Day: requests[2]} return hit, hit.Window != ""
return counts, hit, hit.Window != ""
} }
// Reset sets client's counts in every window back to zero. Its history // Reset sets client's counts in every window back to zero. Its history
+3 -26
View File
@@ -62,14 +62,14 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
start := midnight() start := midnight()
for range limit { for range limit {
_, _, over := limiter.Count(client, start) _, over := limiter.Count(client, start)
if over { if over {
t.Fatal("a request within the limit is over it") t.Fatal("a request within the limit is over it")
} }
} }
// Over both limits; the minute's is named, with the four requests. // Over both limits; the minute's is named, with the four requests.
_, hit, over := limiter.Count(client, start) hit, over := limiter.Count(client, start)
want := ratelimit.Hit{Window: minute, Limit: limit, Requests: limit + 1} want := ratelimit.Hit{Window: minute, Limit: limit, Requests: limit + 1}
if !over || hit != want { if !over || hit != want {
@@ -78,29 +78,6 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
} }
} }
func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range 3 {
limiter.Count(client, start)
}
// A quarter into the next hour, the minute has only this request. The
// hour still covers three quarters of the bucket before, with its three
// requests, which count 2.25, and this one: 3.25. The day covers all
// four.
counts, _, _ := limiter.Count(client, start.Add(time.Hour+time.Hour/4))
want := ratelimit.Counts{Minute: 1, Hour: 3.25, Day: 4}
if counts != want {
t.Errorf("counts %+v, want %+v", counts, want)
}
}
func TestResetSetsTheCountsBackToZero(t *testing.T) { func TestResetSetsTheCountsBackToZero(t *testing.T) {
t.Parallel() t.Parallel()
@@ -261,7 +238,7 @@ func wantCount(
) { ) {
t.Helper() t.Helper()
_, hit, _ := limiter.Count(client, now) hit, _ := limiter.Count(client, now)
if hit.Window != want { if hit.Window != want {
t.Errorf("request from %s at %s is over %q, want %q", t.Errorf("request from %s at %s is over %q, want %q",
client, now.Format(time.RFC3339), hit.Window, want) client, now.Format(time.RFC3339), hit.Window, want)
-310
View File
@@ -1,310 +0,0 @@
// Package remotelog sends the lines smallwebwaf writes on stdout to the
// remote log endpoint, SWWAF_LOG_REMOTE_URL, as the "Request log" section
// of SPEC.md describes: each line as the message of an RFC 5424 syslog
// record, over UDP, TCP or TLS. Lines wait in a bounded buffer, so a slow
// or unreachable endpoint never holds up a request or stdout.
package remotelog
import (
"bytes"
"context"
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"log/slog"
"net"
"net/url"
"os"
"strconv"
"sync/atomic"
"syscall"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// The forms of SWWAF_LOG_REMOTE_URL, by its scheme.
const (
SchemeUDP = "syslog+udp"
SchemeTCP = "syslog+tcp"
SchemeTLS = "syslog+tls"
)
// A record's priority is the number of its facility times the number of
// severities there are, plus the number of its severity. Every record's
// severity is informational.
const (
severities = 8
informational = 6
)
const (
// dialTimeout bounds connecting to the endpoint, the TLS handshake
// included.
dialTimeout = 10 * time.Second
// After a failed attempt to connect, or a connection on which a record
// fails, the next attempt to connect is made a second later, and
// retryDelayFactor times as long after each further failure in a row,
// up to a minute. A connection that fails after it has stayed up for
// resetRetryDelayAfter ends the row.
firstRetryDelay = time.Second
retryDelayFactor = 2
maxRetryDelay = time.Minute
resetRetryDelayAfter = time.Minute
)
// Params are what New needs.
type Params struct {
// URL is the endpoint (SWWAF_LOG_REMOTE_URL): SchemeUDP, SchemeTCP or
// SchemeTLS, a host and a port.
URL *url.URL
// RootCAs are the certificates a SchemeTLS endpoint's certificate
// must chain to (SWWAF_LOG_REMOTE_TLS_CA_FILE), nil for the host's.
RootCAs *x509.CertPool
// Buffer is the most lines held while they wait to be sent
// (SWWAF_LOG_REMOTE_BUFFER).
Buffer int
// Facility is the number of the records' syslog facility
// (SWWAF_LOG_REMOTE_FACILITY), and AppName their APP-NAME
// (SWWAF_LOG_REMOTE_APP_NAME).
Facility int
AppName string
}
// Sender sends lines to the endpoint. Write puts them in its buffer, and
// Run sends them from there.
type Sender struct {
url *url.URL
tlsConfig *tls.Config
// beforeTime and afterTime are the parts of every record's header
// before and after its time, as RFC 5424 lays the header out.
beforeTime string
afterTime string
// records is the buffer: each line's record, framed to be sent.
records chan []byte
sent atomic.Int64
dropped atomic.Int64
}
// New returns a Sender for the endpoint params.URL.
func New(params Params) *Sender {
hostname, err := os.Hostname()
if err != nil || hostname == "" {
hostname = "-" // RFC 5424's value for a field that has none
}
priority := params.Facility*severities + informational
return &Sender{
url: params.URL,
tlsConfig: &tls.Config{
RootCAs: params.RootCAs,
MinVersion: tls.VersionTLS12,
},
// The 1 is the version of the format. The process id, the message
// id and the structured data have no value.
beforeTime: "<" + strconv.Itoa(priority) + ">1 ",
afterTime: " " + hostname + " " + params.AppName + " - - - ",
records: make(chan []byte, params.Buffer),
}
}
// Write puts each line in p in the buffer, as the message of a record of
// its own, and never waits: when the buffer is full, the oldest record in
// it is dropped to make room. It is safe for concurrent use.
func (s *Sender) Write(p []byte) (int, error) {
at := requestlog.FormatTime(time.Now())
for line := range bytes.Lines(p) {
line = bytes.TrimSuffix(line, []byte("\n"))
if len(line) > 0 {
s.put(s.record(at, line))
}
}
return len(p), nil
}
// Sent is how many records have been sent.
func (s *Sender) Sent() int64 {
return s.sent.Load()
}
// Dropped is how many records were dropped: the oldest in a full buffer,
// and those whose sending failed.
func (s *Sender) Dropped() int64 {
return s.dropped.Load()
}
// Depth is how many records are in the buffer.
func (s *Sender) Depth() int {
return len(s.records)
}
// Run connects to the endpoint and sends each record as it comes into the
// buffer, until ctx is done. Then it sends the records still in the buffer,
// on the connection open at that time or, if there is none, on a new one,
// until none is left or one fails, and returns. How long it may take over
// that is for the caller to bound.
//
// A connection on which a record fails is closed and the record dropped.
// That failure, like a failed attempt to connect, is logged to processLog
// and followed by the next attempt after firstRetryDelay, retryDelayFactor
// times as long after each further failure in a row up to maxRetryDelay,
// and firstRetryDelay again after a connection that stayed up for
// resetRetryDelayAfter. Meanwhile the records wait in the buffer.
func (s *Sender) Run(ctx context.Context, processLog *slog.Logger) {
conn := s.send(ctx, processLog)
if conn == nil && len(s.records) > 0 {
conn, _ = s.dial(context.WithoutCancel(ctx))
}
if conn == nil {
return
}
defer func() {
_ = conn.Close()
}()
for {
select {
case record := <-s.records:
if s.write(conn, record) != nil {
return
}
default:
return
}
}
}
// record returns line as an RFC 5424 record made at the time at, framed
// for the endpoint: on its own over UDP, since each datagram holds one,
// and over TCP and TLS after its length in bytes and a space, the
// octet-counted framing of RFC 6587 and RFC 5425.
func (s *Sender) record(at string, line []byte) []byte {
record := make([]byte, 0, len(s.beforeTime)+len(at)+len(s.afterTime)+len(line))
record = append(record, s.beforeTime...)
record = append(record, at...)
record = append(record, s.afterTime...)
record = append(record, line...)
if s.url.Scheme == SchemeUDP {
return record
}
return append([]byte(strconv.Itoa(len(record))+" "), record...)
}
// put adds record to the buffer, first dropping the oldest record in it
// while it is full.
func (s *Sender) put(record []byte) {
for {
select {
case s.records <- record:
return
default:
}
select {
case <-s.records:
s.dropped.Add(1)
default:
}
}
}
// send connects to the endpoint and sends each record as it comes into
// the buffer, until ctx is done, and returns the connection then open, or
// nil.
func (s *Sender) send(ctx context.Context, processLog *slog.Logger) net.Conn {
delay := firstRetryDelay
for {
conn, err := s.dial(ctx)
if ctx.Err() != nil {
return conn
}
if err == nil {
connected := time.Now()
err = s.sendOn(ctx, conn)
if err == nil {
return conn
}
_ = conn.Close()
if time.Since(connected) >= resetRetryDelayAfter {
delay = firstRetryDelay
}
}
processLog.Warn("sending to SWWAF_LOG_REMOTE_URL failed",
"error", err.Error(), "connecting_again_in", delay.String())
select {
case <-time.After(delay):
case <-ctx.Done():
return nil
}
delay = min(retryDelayFactor*delay, maxRetryDelay)
}
}
// sendOn sends each record on conn as it comes into the buffer, until one
// fails, whose error it returns, or ctx is done.
func (s *Sender) sendOn(ctx context.Context, conn net.Conn) error {
for {
select {
case record := <-s.records:
err := s.write(conn, record)
if err != nil {
return err
}
case <-ctx.Done():
return nil
}
}
}
// write sends record on conn, and counts it as sent or, if that fails,
// as dropped. A record too long for one UDP datagram is dropped without
// an error, since the connection has not failed: a long request must not
// hold up the lines after it.
func (s *Sender) write(conn net.Conn, record []byte) error {
_, err := conn.Write(record)
if err != nil {
s.dropped.Add(1)
if errors.Is(err, syscall.EMSGSIZE) {
return nil
}
return fmt.Errorf("send a record: %w", err)
}
s.sent.Add(1)
return nil
}
// dial connects to the endpoint.
func (s *Sender) dial(ctx context.Context) (net.Conn, error) {
dialer := &net.Dialer{Timeout: dialTimeout}
switch s.url.Scheme {
case SchemeUDP:
return dialer.DialContext(ctx, "udp", s.url.Host)
case SchemeTLS:
tlsDialer := &tls.Dialer{NetDialer: dialer, Config: s.tlsConfig}
return tlsDialer.DialContext(ctx, "tcp", s.url.Host)
default:
return dialer.DialContext(ctx, "tcp", s.url.Host)
}
}
-640
View File
@@ -1,640 +0,0 @@
package remotelog_test
import (
"bufio"
"bytes"
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/json"
"fmt"
"io"
"log/slog"
"math/big"
"net"
"net/url"
"os"
"slices"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
"testing/synctest"
"time"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
)
// The tests run in a synctest bubble, where the time package runs on a
// clock of the test's own, which starts at 2000-01-01T00:00:00Z: a wait
// lasts exactly as long as it should, however slowly the test process
// runs, and synctest.Wait returns once the sender has done all it can
// before time passes. The endpoint is a listener on the loopback address.
// A test reads from it only once the records are on their way, and checks
// the sender's counts first, since a goroutine of the bubble that waits on
// the network keeps that clock from moving on. For the same reason the
// endpoint that refuses connections, a tlsEndpoint, runs outside the
// bubble: a sender connecting over TLS waits on the endpoint's answer.
const (
// started is the time a record made as a test starts gives.
started = "2000-01-01T00:00:00.000Z"
appName = "fsn1app1/gitea"
// local0 is the number of the default facility, and local0Info the
// priority of its records.
local0 = 16
local0Info = "<134>"
// loopback is where the endpoints listen.
loopback = "127.0.0.1:0"
)
func TestRecordsOverUDPGoOnePerDatagram(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
endpoint, err := (&net.ListenConfig{}).ListenPacket(t.Context(), "udp", loopback)
if err != nil {
t.Fatalf("listen: %v", err)
}
t.Cleanup(func() { _ = endpoint.Close() })
sender, _, _ := run(t, params(remotelog.SchemeUDP, endpoint.LocalAddr()))
_, _ = sender.Write([]byte(`{"type":"request"}` + "\n" + `{"type":"process"}` + "\n"))
synctest.Wait()
wantCounts(t, sender, 2, 0, 0)
for _, line := range []string{`{"type":"request"}`, `{"type":"process"}`} {
datagram := make([]byte, 1024)
n, _, err := endpoint.ReadFrom(datagram)
if err != nil {
t.Fatalf("read: %v", err)
}
want := record(t, local0Info, appName, line)
if string(datagram[:n]) != want {
t.Errorf("datagram %q, want %q", datagram[:n], want)
}
}
})
}
func TestRecordsOverTCPAreOctetCountedWithTheirFacility(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
endpoint := listen(t)
endpointParams := params(remotelog.SchemeTCP, endpoint.Addr())
endpointParams.Facility = 19 // local3
endpointParams.AppName = "gitea"
sender, _, _ := run(t, endpointParams)
_, _ = sender.Write([]byte("first\nsecond\n"))
synctest.Wait()
wantCounts(t, sender, 2, 0, 0)
frames := bufio.NewReader(accept(t, endpoint))
wantFrame(t, frames, record(t, "<158>", "gitea", "first"))
wantFrame(t, frames, record(t, "<158>", "gitea", "second"))
})
}
func TestStalledEndpointHoldsUpNoWriteAndOldestRecordsAreDropped(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
certificate, roots := testCertificate(t)
endpoint := listen(t)
endpointParams := params(remotelog.SchemeTLS, endpoint.Addr())
endpointParams.RootCAs = roots
endpointParams.Buffer = 3
sender, _, _ := run(t, endpointParams)
// The sender connects, and its TLS handshake waits for an answer
// the endpoint does not give yet.
conn := accept(t, endpoint)
var stdout bytes.Buffer
out := io.MultiWriter(&stdout, sender)
for i := range 5 {
_, _ = fmt.Fprintf(out, "line %d\n", i+1)
}
if stdout.String() != "line 1\nline 2\nline 3\nline 4\nline 5\n" {
t.Errorf("stdout has %q", stdout.String())
}
wantCounts(t, sender, 0, 2, 3)
// Once the endpoint answers, the three newest records are sent.
server := tls.Server(conn, &tls.Config{
Certificates: []tls.Certificate{certificate},
MinVersion: tls.VersionTLS12,
})
err := server.HandshakeContext(t.Context())
if err != nil {
t.Fatalf("handshake: %v", err)
}
synctest.Wait()
wantCounts(t, sender, 3, 2, 0)
frames := bufio.NewReader(server)
for _, line := range []string{"line 3", "line 4", "line 5"} {
wantFrame(t, frames, record(t, local0Info, appName, line))
}
})
}
func TestReconnectsWithBackoffAfterTheEndpointGoesAway(t *testing.T) {
t.Parallel()
certificate, roots := testCertificate(t)
endpoint := startTLSEndpoint(t, certificate)
synctest.Test(t, func(t *testing.T) {
endpointParams := params(remotelog.SchemeTLS, endpoint.addr)
endpointParams.RootCAs = roots
sender, logged, _ := run(t, endpointParams)
_, _ = sender.Write([]byte("one\n"))
synctest.Wait()
wantCounts(t, sender, 1, 0, 0)
conn := endpoint.next(t)
wantFrame(t, bufio.NewReader(conn), record(t, local0Info, appName, "one"))
// The endpoint goes away: it closes the connection, and refuses the
// next ones. The sender notices when a record fails, and tries to
// connect again a second later, then two seconds after that.
endpoint.refusing.Store(true)
_ = conn.Close()
writeUntilDropped(t, sender, 1)
sent := sender.Sent()
_, _ = sender.Write([]byte("two\n"))
time.Sleep(time.Second)
synctest.Wait()
endpoint.refusing.Store(false)
time.Sleep(2*time.Second - time.Nanosecond)
synctest.Wait()
wantCounts(t, sender, sent, 1, 1)
// The endpoint is back, and the record waiting is sent.
time.Sleep(time.Nanosecond)
synctest.Wait()
wantCounts(t, sender, sent+1, 1, 0)
conn = endpoint.next(t)
wantFrame(t, bufio.NewReader(conn), record(t, local0Info, appName, "two"))
wantRetries(t, logged, "1s", "2s")
})
}
func TestAConnectionClosedAtOnceIsMadeAgainAfterAGrowingDelay(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
endpoint := listen(t)
sender, logged, _ := run(t, params(remotelog.SchemeTCP, endpoint.Addr()))
// The endpoint closes each connection as soon as it takes it. The
// sender notices when a record fails, and connects again a second
// later, then two seconds after that, then four.
delays := []time.Duration{time.Second, 2 * time.Second, 4 * time.Second}
for i, delay := range delays {
_ = accept(t, endpoint).Close()
writeUntilDropped(t, sender, int64(i+1))
wantConnectedAgainAfter(t, sender, delay)
}
wantRetries(t, logged, "1s", "2s", "4s")
})
}
func TestTheDelayStartsAgainAfterAConnectionThatStayedUpAMinute(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
endpoint := listen(t)
sender, logged, _ := run(t, params(remotelog.SchemeTCP, endpoint.Addr()))
_ = accept(t, endpoint).Close()
writeUntilDropped(t, sender, 1)
wantConnectedAgainAfter(t, sender, time.Second)
// A connection that fails just short of a minute after it was made
// leaves the delay growing.
conn := accept(t, endpoint)
time.Sleep(time.Minute - time.Nanosecond)
_ = conn.Close()
writeUntilDropped(t, sender, 2)
wantConnectedAgainAfter(t, sender, 2*time.Second)
// One that fails a minute after it was made starts it again from a
// second.
conn = accept(t, endpoint)
time.Sleep(time.Minute)
_ = conn.Close()
writeUntilDropped(t, sender, 3)
wantConnectedAgainAfter(t, sender, time.Second)
wantRetries(t, logged, "1s", "2s", "1s")
})
}
func TestALineTooLongForADatagramIsDroppedAlone(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
endpoint, err := (&net.ListenConfig{}).ListenPacket(t.Context(), "udp", loopback)
if err != nil {
t.Fatalf("listen: %v", err)
}
t.Cleanup(func() { _ = endpoint.Close() })
sender, logged, _ := run(t, params(remotelog.SchemeUDP, endpoint.LocalAddr()))
// With its header, the first line's record is longer than the 65507
// bytes a UDP datagram over IPv4 holds. It is dropped, nothing is
// logged, and the next line is sent at once.
_, _ = sender.Write([]byte(strings.Repeat("x", 65507) + "\nnext\n"))
synctest.Wait()
wantCounts(t, sender, 1, 1, 0)
wantRetries(t, logged)
datagram := make([]byte, 1024)
n, _, err := endpoint.ReadFrom(datagram)
if err != nil {
t.Fatalf("read: %v", err)
}
want := record(t, local0Info, appName, "next")
if string(datagram[:n]) != want {
t.Errorf("datagram %q, want %q", datagram[:n], want)
}
})
}
func TestRecordsWaitingAtTheStopAreSent(t *testing.T) {
t.Parallel()
certificate, roots := testCertificate(t)
endpoint := startTLSEndpoint(t, certificate)
synctest.Test(t, func(t *testing.T) {
// The endpoint refuses the sender's first connection: it fails to
// connect, and waits a second to try again.
endpoint.refusing.Store(true)
endpointParams := params(remotelog.SchemeTLS, endpoint.addr)
endpointParams.RootCAs = roots
sender, logged, stop := run(t, endpointParams)
synctest.Wait()
wantRetries(t, logged, "1s")
_, _ = sender.Write([]byte("one\ntwo\n"))
endpoint.refusing.Store(false)
// Stopped before that second is over, it connects to send them.
stop()
wantCounts(t, sender, 2, 0, 0)
frames := bufio.NewReader(endpoint.next(t))
wantFrame(t, frames, record(t, local0Info, appName, "one"))
wantFrame(t, frames, record(t, local0Info, appName, "two"))
})
}
// output collects what the sender logs.
type output struct {
mu sync.Mutex
buf bytes.Buffer
}
// Write adds lines the sender logs.
func (o *output) Write(p []byte) (int, error) {
o.mu.Lock()
defer o.mu.Unlock()
return o.buf.Write(p)
}
// text returns everything logged so far.
func (o *output) text() string {
o.mu.Lock()
defer o.mu.Unlock()
return o.buf.String()
}
// params returns the settings of a Sender for the endpoint at addr, in
// the form scheme names: room for ten lines, the default facility, and
// appName.
func params(scheme string, addr net.Addr) remotelog.Params {
return remotelog.Params{
URL: &url.URL{Scheme: scheme, Host: addr.String()},
Buffer: 10,
Facility: local0,
AppName: appName,
}
}
// run runs a Sender with settings until the test ends or the function
// it returns is called, which waits for Run to return. It returns the
// Sender, and what it logs.
func run(t *testing.T, settings remotelog.Params) (*remotelog.Sender, *output, func()) {
t.Helper()
sender := remotelog.New(settings)
logged := &output{}
ctx, cancel := context.WithCancel(t.Context())
ran := make(chan struct{})
go func() {
sender.Run(ctx, slog.New(slog.NewJSONHandler(logged, nil)))
close(ran)
}()
stop := func() {
cancel()
<-ran
}
t.Cleanup(stop)
return sender, logged, stop
}
// listen returns a TCP listener on the loopback address, closed when the
// test ends.
func listen(t *testing.T) net.Listener {
t.Helper()
listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", loopback)
if err != nil {
t.Fatalf("listen: %v", err)
}
t.Cleanup(func() { _ = listener.Close() })
return listener
}
// accept returns the next connection to listener, closed when the test
// ends.
func accept(t *testing.T, listener net.Listener) net.Conn {
t.Helper()
conn, err := listener.Accept()
if err != nil {
t.Fatalf("accept: %v", err)
}
t.Cleanup(func() { _ = conn.Close() })
return conn
}
// tlsEndpoint is a syslog+tls endpoint on the loopback address, which a
// test starts outside its bubble. It keeps its listener until the test
// ends, and either takes each connection or refuses it.
type tlsEndpoint struct {
addr net.Addr
// refusing is set while the endpoint closes each connection before the
// TLS handshake, which fails the sender's attempt to connect.
refusing atomic.Bool
// conns are the connections it has taken, after the handshake.
conns chan net.Conn
}
// startTLSEndpoint starts a tlsEndpoint with certificate, which takes
// connections until it is told to refuse them.
func startTLSEndpoint(t *testing.T, certificate tls.Certificate) *tlsEndpoint {
t.Helper()
listener := listen(t)
endpoint := &tlsEndpoint{addr: listener.Addr(), conns: make(chan net.Conn, 10)}
config := &tls.Config{
Certificates: []tls.Certificate{certificate},
MinVersion: tls.VersionTLS12,
}
go func() {
for {
conn, err := listener.Accept()
if err != nil {
return
}
server := tls.Server(conn, config)
if endpoint.refusing.Load() || server.HandshakeContext(t.Context()) != nil {
_ = conn.Close()
continue
}
endpoint.conns <- server
}
}()
return endpoint
}
// next returns the next connection the endpoint has taken, closed when
// the test ends.
func (e *tlsEndpoint) next(t *testing.T) net.Conn {
t.Helper()
conn := <-e.conns
t.Cleanup(func() { _ = conn.Close() })
return conn
}
// record returns the record of line made as the test started, with the
// priority and the app name given.
func record(t *testing.T, priority, app, line string) string {
t.Helper()
hostname, err := os.Hostname()
if err != nil || hostname == "" {
hostname = "-"
}
return priority + "1 " + started + " " + hostname + " " + app + " - - - " + line
}
// wantFrame reads the next octet-counted frame from frames, and checks
// that it holds want.
func wantFrame(t *testing.T, frames *bufio.Reader, want string) {
t.Helper()
count, err := frames.ReadString(' ')
if err != nil {
t.Fatalf("read a frame's length: %v", err)
}
length, err := strconv.Atoi(strings.TrimSuffix(count, " "))
if err != nil {
t.Fatalf("frame starts %q, not with its length", count)
}
got := make([]byte, length)
_, err = io.ReadFull(frames, got)
if err != nil {
t.Fatalf("read a frame: %v", err)
}
if string(got) != want {
t.Errorf("frame %q, want %q", got, want)
}
}
// wantCounts checks the records sender has sent, dropped and holds in
// its buffer.
func wantCounts(
t *testing.T, sender *remotelog.Sender, sent, dropped int64, depth int,
) {
t.Helper()
if sender.Sent() != sent || sender.Dropped() != dropped || sender.Depth() != depth {
t.Fatalf("sent %d, dropped %d, %d in the buffer; want %d, %d and %d",
sender.Sent(), sender.Dropped(), sender.Depth(), sent, dropped, depth)
}
}
// writeUntilDropped writes a line at a time until the count of records
// sender has dropped reaches dropped. The records it sends on a
// connection the endpoint has closed are lost before one fails; how many
// depends on when the endpoint's host answers that the connection is
// gone.
func writeUntilDropped(t *testing.T, sender *remotelog.Sender, dropped int64) {
t.Helper()
for sender.Dropped() < dropped {
_, _ = sender.Write([]byte("lost\n"))
synctest.Wait()
}
}
// wantConnectedAgainAfter writes a line while the sender waits to connect
// again, and checks that it connects, and takes the line from the buffer,
// only once delay is over.
func wantConnectedAgainAfter(
t *testing.T, sender *remotelog.Sender, delay time.Duration,
) {
t.Helper()
_, _ = sender.Write([]byte("waiting\n"))
time.Sleep(delay - time.Nanosecond)
synctest.Wait()
if sender.Depth() != 1 {
t.Fatalf("connected again before %v", delay)
}
time.Sleep(time.Nanosecond)
synctest.Wait()
if sender.Depth() != 0 {
t.Fatalf("not connected again after %v", delay)
}
}
// wantRetries checks that the sender logged a failure, of an attempt to
// connect or of a connection, for each of delays, the time until the next
// attempt, in order, and logged nothing else.
func wantRetries(t *testing.T, logged *output, delays ...string) {
t.Helper()
var got []string
for line := range strings.Lines(logged.text()) {
var fields map[string]any
err := json.Unmarshal([]byte(line), &fields)
if err != nil || fields["msg"] != "sending to SWWAF_LOG_REMOTE_URL failed" {
t.Fatalf("logged %q", line)
}
delay, _ := fields["connecting_again_in"].(string)
got = append(got, delay)
}
if !slices.Equal(got, delays) {
t.Errorf("logged failures to connect again in %v, want %v", got, delays)
}
}
// testCertificate returns a certificate for 127.0.0.1 that is its own
// CA, and a pool that holds it. It is valid on the bubble's clock, which
// starts at 2000-01-01T00:00:00Z.
func testCertificate(t *testing.T) (tls.Certificate, *x509.CertPool) {
t.Helper()
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
t.Fatalf("generate a key: %v", err)
}
template := &x509.Certificate{
SerialNumber: big.NewInt(1),
Subject: pkix.Name{CommonName: "smallwebwaf test CA"},
NotBefore: time.Date(1999, 12, 31, 0, 0, 0, 0, time.UTC),
NotAfter: time.Date(2000, 1, 2, 0, 0, 0, 0, time.UTC),
IsCA: true,
BasicConstraintsValid: true,
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageDigitalSignature,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
IPAddresses: []net.IP{net.IPv4(127, 0, 0, 1)},
}
der, err := x509.CreateCertificate(rand.Reader, template, template,
&key.PublicKey, key)
if err != nil {
t.Fatalf("create a certificate: %v", err)
}
certificate, err := x509.ParseCertificate(der)
if err != nil {
t.Fatalf("parse the certificate: %v", err)
}
roots := x509.NewCertPool()
roots.AddCert(certificate)
return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key}, roots
}
+26 -81
View File
@@ -9,8 +9,6 @@ import (
"io" "io"
"log/slog" "log/slog"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
) )
// The action a request line names: what smallwebwaf did with the // The action a request line names: what smallwebwaf did with the
@@ -28,12 +26,8 @@ const (
// ActionRateLimited is a request refused because it took its client // ActionRateLimited is a request refused because it took its client
// over a rate limit, which bans the client. // over a rate limit, which bans the client.
ActionRateLimited = "rate_limited" ActionRateLimited = "rate_limited"
// ActionBanned is a request refused because a ban covers its client, // ActionBanned is a request refused because a ban covers its client.
// or because it matched a ban rule, which bans the client.
ActionBanned = "banned" ActionBanned = "banned"
// ActionRuleBlocked is a request refused because it matched a block
// rule.
ActionRuleBlocked = "rule_blocked"
// ActionDenied is a request refused because its client is in // ActionDenied is a request refused because its client is in
// SWWAF_DENY_NETS. // SWWAF_DENY_NETS.
ActionDenied = "denied" ActionDenied = "denied"
@@ -51,74 +45,32 @@ const OffenceLimit = "limit"
// timeLayout is RFC 3339 with milliseconds. // timeLayout is RFC 3339 with milliseconds.
const timeLayout = "2006-01-02T15:04:05.000Z07:00" const timeLayout = "2006-01-02T15:04:05.000Z07:00"
// Line is one request's line in the request log. The field names, and // Line is one request's line in the request log. The field names are
// their order, are those of the "Request log" section of SPEC.md. A field // those of the "Request log" section of SPEC.md.
// that may not apply to a request is left out of its line when it does
// not.
// //
//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case //nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
type Line struct { type Line struct {
Type string `json:"type"` Type string `json:"type"`
Time string `json:"time"`
// The standard web log fields. Scheme is how the client reached ClientIP string `json:"client_ip"`
// smallwebwaf, or the trusted proxy in front of it. PeerIP string `json:"peer_ip"`
Time string `json:"time"` Country string `json:"country"`
Instance string `json:"instance"` Method string `json:"method"`
ClientIP string `json:"client_ip"` Host string `json:"host"`
Method string `json:"method"` Path string `json:"path"`
Scheme string `json:"scheme"` Query string `json:"query"`
Host string `json:"host"` Protocol string `json:"protocol"`
Path string `json:"path"` Status int `json:"status"`
Query string `json:"query"` UpstreamStatus int `json:"upstream_status,omitempty"`
Protocol string `json:"protocol"` RequestBytes int64 `json:"request_bytes"`
Status int `json:"status"` ResponseBytes int64 `json:"response_bytes"`
RequestBytes int64 `json:"request_bytes"` Referer string `json:"referer"`
ResponseBytes int64 `json:"response_bytes"` UserAgent string `json:"user_agent"`
Referer string `json:"referer"` Action string `json:"action"`
UserAgent string `json:"user_agent"`
// Request detail. RequestID is the X-Request-ID a trusted proxy sent,
// or a new one, and is sent on to the app. ForwardedFor is the
// X-Forwarded-For header as received. ClientGroup is the netblock the
// client is counted as.
RequestID string `json:"request_id"`
PeerIP string `json:"peer_ip"`
ForwardedFor string `json:"forwarded_for,omitempty"`
ClientGroup string `json:"client_group"`
Country string `json:"country"`
ContentType string `json:"content_type,omitempty"`
// ContentLength is the length of its body the request announced.
ContentLength int64 `json:"content_length,omitempty"`
// RequestHeaders are the headers SWWAF_LOG_REQUEST_HEADERS names that
// the request carried, by name in lower case.
RequestHeaders map[string]string `json:"request_headers,omitempty"`
HasAuthorization bool `json:"has_authorization,omitempty"`
HasCookie bool `json:"has_cookie,omitempty"`
// Websocket is true when the connection was upgraded, as for a
// WebSocket.
Websocket bool `json:"websocket,omitempty"`
// Response detail, from the headers of the answer: the app's, as
// passed on, or those of smallwebwaf's own. Aborted is true when the
// client went away early.
ResponseContentType string `json:"response_content_type,omitempty"`
UpstreamStatus int `json:"upstream_status,omitempty"`
CacheControl string `json:"cache_control,omitempty"`
Location string `json:"location,omitempty"`
Aborted bool `json:"aborted,omitempty"`
// The decision.
Action string `json:"action"`
// WouldAction is, in observe mode, the action enforce mode would have // WouldAction is, in observe mode, the action enforce mode would have
// taken with a request it would have refused: ActionDenied, // taken with a request it would have refused: ActionDenied,
// ActionBanned, ActionCountryDenied, ActionRateLimited or // ActionBanned, ActionCountryDenied or ActionRateLimited.
// ActionRuleBlocked.
WouldAction string `json:"would_action,omitempty"` WouldAction string `json:"would_action,omitempty"`
// Counts are the client's requests as the rate limits counted them
// with this one, for a request they counted.
Counts ratelimit.Counts `json:"counts,omitzero"`
// RuleIDs are the ids of the rule file rules the request matched.
RuleIDs []string `json:"rule_ids,omitempty"`
// LimitHit is the window whose rate limit the request went over: // LimitHit is the window whose rate limit the request went over:
// minute, hour or day. // minute, hour or day.
LimitHit string `json:"limit_hit,omitempty"` LimitHit string `json:"limit_hit,omitempty"`
@@ -127,18 +79,11 @@ type Line struct {
// BanExpires is when the ban the request made, or was refused under, // BanExpires is when the ban the request made, or was refused under,
// ends: a time, or "permanent". // ends: a time, or "permanent".
BanExpires string `json:"ban_expires,omitempty"` BanExpires string `json:"ban_expires,omitempty"`
// Aborted is true when the client went away early.
// The timings, in milliseconds. DurationChecks is the time until the Aborted bool `json:"aborted,omitempty"`
// checks were done. DurationUpstreamConnect, DurationUpstreamFirstByte // DurationTotal and DurationUpstreamTotal are in milliseconds.
// and DurationUpstreamTotal run from when the request was handed to the DurationTotal float64 `json:"duration_total"`
// app: until there was a connection to it, until the first byte of its DurationUpstreamTotal float64 `json:"duration_upstream_total,omitempty"`
// answer arrived, and until the end. Each but DurationTotal is nil for
// a request that did not get that far.
DurationTotal float64 `json:"duration_total"`
DurationChecks *float64 `json:"duration_checks,omitempty"`
DurationUpstreamConnect *float64 `json:"duration_upstream_connect,omitempty"`
DurationUpstreamFirstByte *float64 `json:"duration_upstream_first_byte,omitempty"`
DurationUpstreamTotal *float64 `json:"duration_upstream_total,omitempty"`
} }
// Write writes line to w as one JSON line marked "type":"request". // Write writes line to w as one JSON line marked "type":"request".
+1 -5
View File
@@ -50,11 +50,7 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
} }
unset := []string{ unset := []string{
"forwarded_for", "content_type", "content_length", "request_headers", "upstream_status", "limit_hit", "offence", "ban_expires", "aborted",
"has_authorization", "has_cookie", "websocket", "response_content_type",
"upstream_status", "cache_control", "location", "aborted", "counts",
"limit_hit", "offence", "ban_expires", "duration_checks",
"duration_upstream_connect", "duration_upstream_first_byte",
"duration_upstream_total", "duration_upstream_total",
} }
for _, name := range unset { for _, name := range unset {
-469
View File
@@ -1,469 +0,0 @@
// Package rules reads the rule files: the plain text files in
// SWWAF_RULES_DIR, one rule to a line, that each request is checked
// against, as the "Rule files" section of SPEC.md describes. They are read
// at start, and again once the directory has had no change for a short
// time after one is edited, added or removed.
package rules
import (
"context"
"encoding/hex"
"errors"
"fmt"
"log/slog"
"net/http"
"os"
"path/filepath"
"regexp"
"slices"
"strings"
"sync/atomic"
"time"
"github.com/fsnotify/fsnotify"
"sneak.berlin/go/smallwebwaf/internal/config"
)
// The actions a rule takes when it matches.
const (
// ActionLog notes the match in the request log, and does nothing else.
ActionLog = "log"
// ActionBlock refuses the request with 403.
ActionBlock = "block"
// ActionBan refuses the request and bans the client's netblock: the
// request is a clear sign of attack.
ActionBan = "ban"
)
// extension ends the name of every rule file.
const extension = ".rules"
// quietTime is how long SWWAF_RULES_DIR must go without a change before
// the rule files are read again, so that a file still being written, such
// as one saved in place, appended to or copied in with scp, is read only
// once whole.
const quietTime = 2 * time.Second
// headerTarget starts the target that is one request header,
// header:<Name>.
const headerTarget = "header:"
// escapeLength is the length of a percent escape, such as %2e.
const escapeLength = 3
var (
// ruleLine is a rule: four fields separated by spaces or tabs, of
// which the fourth, the regex, runs to the end of the line.
ruleLine = regexp.MustCompile(`^([^ \t]+)[ \t]+([^ \t]+)[ \t]+([^ \t]+)[ \t]+(.+)$`)
// idChars are the characters of a rule's id.
idChars = regexp.MustCompile(`^[A-Za-z0-9_-]+$`)
)
var (
errNotRule = errors.New(
"is not a rule: an id, a target, an action and a regex, " +
"separated by spaces or tabs")
errNotID = errors.New("is not an id of letters, digits, - and _")
errNotTarget = errors.New(
"is not path, query, uri, method, host, user_agent, referer or header:<Name>")
errNotHeaderName = errors.New(
"has a character after header: that no header name can have")
errHeaderTakenOut = errors.New(
"names a header that Go's HTTP server takes out of every request, " +
"so a rule never sees it")
errNotAction = errors.New("is not log, block or ban")
errNotRegex = errors.New("does not compile")
errUsedTwice = errors.New("is already the id of the rule at")
)
// Rule is one rule of a rule file.
type Rule struct {
// ID names the rule in the request log, the metrics and ban notes.
ID string
// Target is what the regex is matched against, such as path or
// header:Accept.
Target string
// Action is ActionLog, ActionBlock or ActionBan.
Action string
regex *regexp.Regexp
}
// Params are what Load needs.
type Params struct {
// Dir is the directory of the rule files (SWWAF_RULES_DIR).
Dir string
// Enabled is SWWAF_RULES_ENABLED: while it is false, no file is read
// and no rule loaded.
Enabled bool
// ProcessLog receives how many rules were read, and the error in a
// rule file edited while smallwebwaf runs.
ProcessLog *slog.Logger
}
// Files are the rule files of a running smallwebwaf, and the rules read
// from them. They are safe for concurrent use.
type Files struct {
params Params
// rules are the rules loaded, in the order of their files' names, and
// then of their lines.
rules atomic.Pointer[[]Rule]
}
// Load reads the rules of every *.rules file in Dir, in the order of the
// files' names, unless Enabled is false. A Dir that cannot be read is an
// error, and so is a line that is not a rule, a header name with a
// character no header name can have, a rule for the Host or the
// Transfer-Encoding header, which Go's HTTP server takes out of every
// request, a regex that does not compile and an id used twice, each named
// with its file and line.
func Load(params Params) (*Files, error) {
f := &Files{params: params}
f.rules.Store(&[]Rule{})
if !params.Enabled {
return f, nil
}
rules, err := read(params.Dir)
if err != nil {
return nil, err
}
f.rules.Store(&rules)
f.logRead(len(rules))
return f, nil
}
// Match checks r against the rules, in order, and returns those it
// matches, up to the first whose action refuses it, block or ban, which
// is then the last one returned.
func (f *Files) Match(r *http.Request) []Rule {
var matched []Rule
for _, rule := range *f.rules.Load() {
if !rule.matches(r) {
continue
}
matched = append(matched, rule)
if rule.Action != ActionLog {
break
}
}
return matched
}
// Len returns how many rules are loaded.
func (f *Files) Len() int {
return len(*f.rules.Load())
}
// Watch watches Dir until ctx is done, and reads the rule files again
// once Dir has had no change for quietTime, after one is edited, added or
// removed, and after Watch starts watching. If they then hold an error,
// the rules stay as they were, the error is logged with its file and
// line, and the files are read again after the next change. If Dir cannot
// be watched, that is logged, and the rules stay as they were loaded.
// While Enabled is false, Watch returns at once.
func (f *Files) Watch(ctx context.Context) {
if !f.params.Enabled {
return
}
watcher, err := fsnotify.NewWatcher()
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(f.params.Dir)
}
if err != nil {
f.params.ProcessLog.Error("cannot watch the rule files for edits",
"error", err.Error())
return
}
f.params.ProcessLog.Info("watching the rule files for edits",
"directory", f.params.Dir)
f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
}
// readAfterChanges reads the rule files again once quietTime has passed
// without a change from events, until ctx is done, and logs the errors
// from errs. The wait starts at once, as if for a change, so that an edit
// saved after Load read the files, and before Dir was watched, is taken
// in too.
func (f *Files) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case <-events:
quiet.Reset(quietTime)
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the rule files failed",
"error", err.Error())
}
}
}
// readAgain reads the rule files again, in place of the rules loaded, or
// logs the error that keeps the rules as they were.
func (f *Files) readAgain() {
rules, err := read(f.params.Dir)
if err != nil {
f.params.ProcessLog.Error(
"a rule file has an error, and the rules stay as they were",
"error", err.Error())
return
}
f.rules.Store(&rules)
f.logRead(len(rules))
}
// logRead logs that the rule files were read, and how many rules they
// hold, which can be none.
func (f *Files) logRead(count int) {
f.params.ProcessLog.Info("read the rule files",
"directory", f.params.Dir, "rules", count)
}
// read returns the rules of every rule file in dir, in the order of the
// files' names, and then of their lines. A file whose name starts with a
// dot, such as an editor's lock file .#50-app.rules, is not a rule file,
// as a shell's *.rules would not match it.
func read(dir string) ([]Rule, error) {
entries, err := os.ReadDir(dir)
if err != nil {
return nil, fmt.Errorf("SWWAF_RULES_DIR cannot be read: %w", err)
}
var rules []Rule
// places are where each id is, as "<file>, line <n>".
places := map[string]string{}
for _, entry := range entries {
name := entry.Name()
if entry.IsDir() || strings.HasPrefix(name, ".") || filepath.Ext(name) != extension {
continue
}
rules, err = readFile(filepath.Join(dir, name), rules, places)
if err != nil {
return nil, err
}
}
return rules, nil
}
// readFile appends the rules of the rule file at path to rules. places
// are where each id read so far is, and gain those of the file.
func readFile(path string, rules []Rule, places map[string]string) ([]Rule, error) {
data, err := os.ReadFile(path) //nolint:gosec // a rule file, in SWWAF_RULES_DIR
if err != nil {
return nil, err
}
number := 0
for line := range strings.Lines(string(data)) {
number++
place := fmt.Sprintf("%s, line %d", path, number)
text := strings.TrimSuffix(strings.TrimSuffix(line, "\n"), "\r")
rule, isRule, err := parse(text)
if err != nil {
return nil, fmt.Errorf("%s: %w", place, err)
}
if !isRule {
continue
}
first, used := places[rule.ID]
if used {
return nil, fmt.Errorf("%s: the id %q %w %s", place, rule.ID, errUsedTwice, first)
}
places[rule.ID] = place
rules = append(rules, rule)
}
return rules, nil
}
// parse reads a line of a rule file. It returns false for a blank line
// and for a comment, a line that starts with #. Spaces and tabs at the
// end of the line are not part of its regex, so a line with only those
// after its action has no regex, and is not a rule.
func parse(line string) (Rule, bool, error) {
line = strings.Trim(line, " \t")
if line == "" || strings.HasPrefix(line, "#") {
return Rule{}, false, nil
}
fields := ruleLine.FindStringSubmatch(line)
if fields == nil {
return Rule{}, false, errNotRule
}
rule := Rule{ID: fields[1], Target: fields[2], Action: fields[3]}
headerName, isHeader := strings.CutPrefix(rule.Target, headerTarget)
switch {
case !idChars.MatchString(rule.ID):
return Rule{}, false, fmt.Errorf("the id %q %w", rule.ID, errNotID)
case !isTarget(rule.Target):
return Rule{}, false, fmt.Errorf("the target %q %w", rule.Target, errNotTarget)
case isHeader && !config.IsHeaderName(headerName):
return Rule{}, false, fmt.Errorf("the target %q %w", rule.Target, errNotHeaderName)
case strings.EqualFold(rule.Target, headerTarget+"Host"):
return Rule{}, false, fmt.Errorf(
"the target %q %w; the request's host is the target host",
rule.Target, errHeaderTakenOut)
case strings.EqualFold(rule.Target, headerTarget+"Transfer-Encoding"):
return Rule{}, false, fmt.Errorf("the target %q %w", rule.Target, errHeaderTakenOut)
case !slices.Contains([]string{ActionLog, ActionBlock, ActionBan}, rule.Action):
return Rule{}, false, fmt.Errorf("the action %q %w", rule.Action, errNotAction)
}
regex, err := regexp.Compile(fields[4])
if err != nil {
return Rule{}, false, fmt.Errorf("the regex %w: %w", errNotRegex, err)
}
rule.regex = regex
return rule, true, nil
}
// isTarget reports whether target is one a rule may have.
func isTarget(target string) bool {
switch target {
case "path", "query", "uri", "method", "host", "user_agent", "referer":
return true
}
name, isHeader := strings.CutPrefix(target, headerTarget)
return isHeader && name != ""
}
// matches reports whether the rule's regex matches its target in r. For
// uri it is matched against the path and query as received, and against
// them once percent-decoded, so that an encoded probe cannot slip past.
func (rule Rule) matches(r *http.Request) bool {
if rule.Target == "uri" {
uri := pathAndQuery(r)
return rule.regex.MatchString(uri) || rule.regex.MatchString(decodeOnce(uri))
}
return rule.regex.MatchString(value(rule.Target, r))
}
// value returns what a rule with target, other than uri, is matched
// against in r: the path and the query as the client sent them, before
// any decoding or re-encoding, split at the first ?, and a header's values
// joined by ", ", as HTTP joins those of a header sent more than once.
func value(target string, r *http.Request) string {
switch target {
case "path":
path, _, _ := strings.Cut(pathAndQuery(r), "?")
return path
case "query":
_, query, _ := strings.Cut(pathAndQuery(r), "?")
return query
case "method":
return r.Method
case "host":
return r.Host
case "user_agent":
return header(r, "User-Agent")
case "referer":
return header(r, "Referer")
default:
return header(r, strings.TrimPrefix(target, headerTarget))
}
}
// pathAndQuery returns the target of r's request line, r.RequestURI, as
// the client sent it, less any scheme and host: a target with a scheme
// gives what follows the scheme and its :, and the host when // follows.
// So http://host/path, as a client sends it to a proxy, gives /path, and
// so does http:/path, which Go reads as a target with a scheme and no
// host. r.URL is not used: when the path holds a character it escapes,
// such as \ or a non-ASCII byte, it decodes the whole path and escapes it
// again, so that \ becomes %5C and %2e a dot.
func pathAndQuery(r *http.Request) string {
if !r.URL.IsAbs() {
return r.RequestURI
}
_, afterScheme, _ := strings.Cut(r.RequestURI, ":")
hostAndRest, hasHost := strings.CutPrefix(afterScheme, "//")
if !hasHost {
return afterScheme
}
start := strings.IndexAny(hostAndRest, "/?")
if start < 0 {
return ""
}
return hostAndRest[start:]
}
// header returns the values of r's header name joined by ", ", or "" if
// r has no such header.
func header(r *http.Request, name string) string {
return strings.Join(r.Header.Values(name), ", ")
}
// decodeOnce returns s with each percent escape, such as %2e, replaced by
// the byte it stands for. A % that is not followed by two hex digits is
// left as it is, so that a malformed escape cannot keep the rest of s
// from being decoded.
func decodeOnce(s string) string {
var decoded strings.Builder
for i := 0; i < len(s); i++ {
if s[i] == '%' && i+escapeLength <= len(s) {
b, err := hex.DecodeString(s[i+1 : i+escapeLength])
if err == nil {
decoded.Write(b)
i += escapeLength - 1
continue
}
}
decoded.WriteByte(s[i])
}
return decoded.String()
}
-662
View File
@@ -1,662 +0,0 @@
package rules_test
import (
"context"
"encoding/json"
"log/slog"
"maps"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"strconv"
"testing"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
const (
// What the process log says once Watch watches the directory, after
// each reading of the rule files, and for one that has an error.
watching = "watching the rule files for edits"
read = "read the rule files"
hasError = "a rule file has an error, and the rules stay as they were"
// maxLogLines is how many lines of the process log wait for a test to
// read them.
maxLogLines = 64
// browser is the user agent of an ordinary visitor.
browser = "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0"
// testFile is the rule file of a test that needs only one, and
// firstFile the first of a test's rule files.
testFile = "test.rules"
firstFile = "00-a.rules"
// userAgent is the header that carries the user agent.
userAgent = "User-Agent"
)
func TestEachTargetMatchesWhatItNames(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
rule string // its target, action and regex
uri string // the request's path and query
header http.Header
want bool
}{
{"path as received", `path log ^/%2eenv$`, "/%2eenv", nil, true},
{"path not decoded", `path log ^/\.env$`, "/%2eenv", nil, false},
{"path without the query", `path log ^/a$`, "/a?b=c", nil, true},
{"query as received", `query log ^b=%2e$`, "/a?b=%2e", nil, true},
{"uri as received", `uri log ^/a\?b=%2e$`, "/a?b=%2e", nil, true},
{"uri decoded", `uri log (\.\./){2}`, "/a?f=%2e%2e%2f%2e%2e%2f", nil, true},
{
"uri decoded past malformed escapes", `uri log (\.\./){2}&h=%$`,
"/a?g=%zz&f=%2e%2e%2f%2e%2e%2f&h=%", nil, true,
},
{"uri decoded only once", `uri log ^/a\.b$`, "/a%252eb", nil, false},
{"method", `method log ^PUT$`, "/", nil, true},
{"host", `host log ^app\.example$`, "/", nil, true},
{
"user_agent", `user_agent log ^sqlmap/`, "/",
http.Header{userAgent: {"sqlmap/1.8"}}, true,
},
{
"user_agent sent twice", `user_agent log ^curl/8, sqlmap/`, "/",
http.Header{userAgent: {"curl/8", "sqlmap/1.8"}}, true,
},
{"user_agent missing", `user_agent log ^$`, "/", nil, true},
{
"referer", `referer log ^https://spam\.example/`, "/",
http.Header{"Referer": {"https://spam.example/buy"}}, true,
},
{
"a header sent twice", `header:x-api-version log ^2, 3$`, "/",
http.Header{"X-Api-Version": {"2", "3"}}, true,
},
{"a header missing", `header:X-Api-Version log ^$`, "/", nil, true},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
files := load(t, ruleFiles{testFile: "a-rule " + tc.rule + "\n"})
// Every request is a PUT, which the method rule looks for.
r := httptest.NewRequestWithContext(t.Context(), http.MethodPut,
"http://app.example"+tc.uri, nil)
maps.Copy(r.Header, tc.header)
got := len(files.Match(r)) == 1
if got != tc.want {
t.Errorf("%s matches %s: %t, want %t", tc.rule, tc.uri, got, tc.want)
}
})
}
}
func TestPathMatchedAsTheClientSentIt(t *testing.T) {
t.Parallel()
// Each path holds a character Go's URL type would escape again, \ or
// a non-ASCII byte, and each rule is written for the path as sent.
for _, tc := range []struct {
rule string // its target, action and regex
sent string // the path and query the client sent
}{
{`path log ^/\.\.\\\.\.\\windows\\win\.ini$`, `/..\..\windows\win.ini`},
{`path log ^/%2e%2e\\%2e%2e\\windows\\win\.ini$`, `/%2e%2e\%2e%2e\windows\win.ini`},
{`path log ^/café$`, "/café?x=1"},
{`uri log ^/%2e%2e\\%2e%2e\\boot\.ini\?x=1$`, `/%2e%2e\%2e%2e\boot.ini?x=1`},
} {
files := load(t, ruleFiles{testFile: "as-sent " + tc.rule + "\n"})
// The target in origin form, as traefik sends it, in absolute form,
// as a client sends it to a proxy, and with a scheme but no host,
// which Go reads as absolute form with no host, sending the app
// the path.
for _, target := range []string{
tc.sent, "http://app.example" + tc.sent, "http:" + tc.sent, "foo:" + tc.sent,
} {
r := httptest.NewRequestWithContext(t.Context(), http.MethodGet, target, nil)
wantMatched(t, files, r, "as-sent")
}
}
}
func TestMatchingStopsAtTheFirstRuleThatRefuses(t *testing.T) {
t.Parallel()
files := load(t, ruleFiles{testFile: `
every-path path log ^/
no-path path log ^$
first-refusal path block ^/probe
later-ban path ban ^/probe
after path log ^/
`})
// Every log rule that matches is noted, and the block rule ends the
// matching.
wantMatched(t, files, get(t, "/probe"), "every-path", "first-refusal")
wantMatched(t, files, get(t, "/page"), "every-path", "after")
// A ban rule ends it too.
files = load(t, ruleFiles{testFile: "ban path ban ^/\nlater path block ^/\n"})
wantMatched(t, files, get(t, "/"), "ban")
}
func TestSpacesAndTabsEndingALineAreNotPartOfItsRegex(t *testing.T) {
t.Parallel()
files := load(t, ruleFiles{testFile: "env-file path block ^/\\.env$ \t \n"})
wantMatched(t, files, get(t, "/.env"), "env-file")
}
func TestFilesReadInNameOrderThenLineOrder(t *testing.T) {
t.Parallel()
files := load(t, ruleFiles{
"50-b.rules": "b1 path log ^/\n\n# a comment\n # an indented one\nb2 path log ^/\n",
firstFile: "a1 path log ^/\r\n",
// None is a rule file.
"notes.txt": "notes, not rules\n",
"10-c.rules.bak": "an old copy\n",
"20-d.rules/keep": "a file in a directory\n",
})
wantMatched(t, files, get(t, "/"), "a1", "b1", "b2")
if files.Len() != 3 {
t.Errorf("%d rules loaded, want 3", files.Len())
}
}
func TestFileWhoseNameStartsWithADotIsNotARuleFile(t *testing.T) {
t.Parallel()
dir := writeFiles(t, ruleFiles{firstFile: "probe path block ^/probe\n"})
// The lock file Emacs makes beside a file while it is edited: a link to
// nothing, which cannot be read.
err := os.Symlink("user@host.1234:1700000000", filepath.Join(dir, ".#"+firstFile))
if err != nil {
t.Fatalf("symlink: %v", err)
}
params, _ := newParams(dir)
files, err := rules.Load(params)
if err != nil {
t.Fatalf("load: %v", err)
}
wantMatched(t, files, get(t, "/probe"), "probe")
}
func TestFaultStopsTheStartNamingTheFileAndLine(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
content string
line int
want string
}{
{
"too few fields", "env-file path ban\n", 1,
"is not a rule: an id, a target, an action and a regex, " +
"separated by spaces or tabs",
},
{
// Else its regex would be a space, found in nearly every user agent.
"a regex of only spaces and tabs", "scanner user_agent ban\t \n", 1,
"is not a rule: an id, a target, an action and a regex, " +
"separated by spaces or tabs",
},
{
"an id of other characters", "# ids\n\nenv.file path ban ^/\n", 3,
`the id "env.file" is not an id of letters, digits, - and _`,
},
{
"an unknown target", "env-file paths ban ^/\n", 1,
`the target "paths" is not path, query, uri, method, host, ` +
"user_agent, referer or header:<Name>",
},
{
"a header without a name", "env-file header: ban ^/\n", 1,
`the target "header:" is not path, query, uri, method, host, ` +
"user_agent, referer or header:<Name>",
},
{
"a header name written with its colon", "sqlmap header:User-Agent: ban sqlmap\n", 1,
`the target "header:User-Agent:" has a character after header: ` +
"that no header name can have",
},
{
"a header name with a semicolon", "accept header:Accept;q log ^$\n", 1,
`the target "header:Accept;q" has a character after header: ` +
"that no header name can have",
},
{
"a header name with brackets", "x-header header:X(y) log ^$\n", 1,
`the target "header:X(y)" has a character after header: ` +
"that no header name can have",
},
{
"the Host header", "host-header header:host block ^$\n", 1,
`the target "header:host" names a header that Go's HTTP server ` +
"takes out of every request, so a rule never sees it; " +
"the request's host is the target host",
},
{
"the Transfer-Encoding header",
"# bodies sent in chunks\nchunked header:Transfer-Encoding block ^chunked$\n", 2,
`the target "header:Transfer-Encoding" names a header that Go's ` +
"HTTP server takes out of every request, so a rule never sees it",
},
{
"an unknown action", "env-file path deny ^/\n", 1,
`the action "deny" is not log, block or ban`,
},
{
"a regex that does not compile", "env-file path ban ^/(\n", 1,
"the regex does not compile: error parsing regexp: " +
"missing closing ): `^/(`",
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
dir := writeFiles(t, ruleFiles{"00-default.rules": tc.content})
path := filepath.Join(dir, "00-default.rules")
wantRefused(t, dir, path+", line "+strconv.Itoa(tc.line)+": "+tc.want)
})
}
}
func TestIDUsedTwiceStopsTheStartNamingBothPlaces(t *testing.T) {
t.Parallel()
dir := writeFiles(t, ruleFiles{
"00-a.rules": "probe path log ^/a\n",
"50-b.rules": "other path log ^/b\nprobe path ban ^/c\n",
})
wantRefused(t, dir, filepath.Join(dir, "50-b.rules")+`, line 2: the id "probe" `+
"is already the id of the rule at "+filepath.Join(dir, "00-a.rules")+", line 1")
}
func TestDirectoryThatDoesNotExistStopsTheStart(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "rules.d")
wantRefused(t, dir, "SWWAF_RULES_DIR cannot be read: open "+dir+
": no such file or directory")
}
func TestEmptyDirectoryLoadsNoRulesAndSaysSo(t *testing.T) {
t.Parallel()
params, lines := newParams(writeFiles(t, ruleFiles{"00-default.rules": "# none\n"}))
files, err := rules.Load(params)
if err != nil {
t.Fatalf("load: %v", err)
}
line := lines.waitFor(t, read)
if files.Len() != 0 || line["rules"] != 0.0 {
t.Errorf("%d rules loaded, and the log says %v, want none", files.Len(), line)
}
}
func TestRuleFilesOffReadNothing(t *testing.T) {
t.Parallel()
// SWWAF_RULES_DIR does not exist, which would stop the start.
params, _ := newParams(filepath.Join(t.TempDir(), "rules.d"))
params.Enabled = false
files, err := rules.Load(params)
if err != nil {
t.Fatalf("load: %v", err)
}
if files.Len() != 0 || files.Match(get(t, "/")) != nil {
t.Errorf("%d rules loaded with the rule files off", files.Len())
}
// It would watch until the test ends.
files.Watch(t.Context())
}
func TestEditsTakenInWhileRunning(t *testing.T) {
t.Parallel()
dir := writeFiles(t, ruleFiles{firstFile: "first path block ^/first\n"})
files, lines := watch(t, dir)
// matches reports whether path matches a rule.
matches := func(path string) bool { return len(files.Match(get(t, path))) == 1 }
// A file added.
save(t, dir, "50-b.rules", "second path block ^/second\n")
lines.waitUntil(t, func() bool { return matches("/second") })
wantMatched(t, files, get(t, "/first"), "first")
// A file edited.
save(t, dir, firstFile, "first path block ^/edited\n")
lines.waitUntil(t, func() bool { return !matches("/first") })
wantMatched(t, files, get(t, "/edited"), "first")
// A file removed.
err := os.Remove(filepath.Join(dir, "50-b.rules"))
if err != nil {
t.Fatalf("remove: %v", err)
}
lines.waitUntil(t, func() bool { return !matches("/second") })
wantMatched(t, files, get(t, "/edited"), "first")
}
func TestBrokenEditKeepsTheRulesAsTheyWere(t *testing.T) {
t.Parallel()
dir := writeFiles(t, ruleFiles{firstFile: "first path block ^/first\n"})
files, lines := watch(t, dir)
// The edit's second line has an unknown action, so the rules stay as
// they were, the first line's earlier version included.
save(t, dir, firstFile, "first path block ^/edited\nsecond path bann ^/second\n")
line := lines.waitFor(t, hasError)
want := filepath.Join(dir, firstFile) +
`, line 2: the action "bann" is not log, block or ban`
if line["error"] != want || line["level"] != "ERROR" {
t.Errorf("logged %v, want an error %q", line, want)
}
wantMatched(t, files, get(t, "/first"), "first")
wantMatched(t, files, get(t, "/second"))
// Once mended, the file is read again.
save(t, dir, firstFile, "first path block ^/edited\nsecond path ban ^/second\n")
lines.waitUntil(t, func() bool { return len(files.Match(get(t, "/second"))) == 1 })
wantMatched(t, files, get(t, "/edited"), "first")
}
func TestDefaultFileBansProbesAtTheSiteRootAlone(t *testing.T) {
t.Parallel()
params, _ := newParams(filepath.Join("..", "..", "share", "rules.d"))
files, err := rules.Load(params)
if err != nil {
t.Fatalf("load the default file: %v", err)
}
// Probes sent by a browser, by the rule that refuses them.
for rule, targets := range map[string][]string{
"env-file": {"/.env", "/.env.production", "/.ENV"},
"vcs-dir": {"/.git/config", "/.git", "/.svn/entries"},
"secrets-dir": {"/.aws/credentials", "/.ssh/id_rsa"},
"secret-file": {"/.htpasswd", "/.DS_Store", "/.git-credentials"},
"editor-dir": {"/.vscode/sftp.json"},
"backup-file": {
"/wp-config.php.bak", "/index.php~", "/dump.sql", "/backup.sql.gz",
},
"log-file": {"/debug.log"},
"compose-file": {"/docker-compose.yml", "/compose.yaml"},
"php-shell": {"/shell.php"},
"path-traversal": {
"/static/../../etc/passwd", "/f?f=%2e%2e%2f%2e%2e%2fetc%2fpasswd",
},
} {
for _, target := range targets {
wantRefusedBy(t, files, target, browser, rule)
}
}
// Scanners, by their user agents.
for _, scanner := range []string{
"sqlmap/1.8.4#stable (https://sqlmap.org)",
"Mozilla/5.0 (compatible; Nuclei - Open-source project)",
} {
wantRefusedBy(t, files, "/", scanner, "scanner-agent")
}
// Ordinary requests to a code forge for files of those names deeper
// in its paths, and for other files at its root.
for _, target := range []string{
"/owner/repo/src/branch/main/.env.example",
"/owner/repo/src/branch/main/.env",
"/owner/repo/src/branch/main/.github/workflows/ci.yml",
"/owner/repo/src/branch/main/.vscode/settings.json",
"/owner/repo/src/branch/main/.htaccess",
"/owner/repo/src/branch/main/docker-compose.yml",
"/owner/repo/src/branch/main/db/schema.sql",
"/owner/repo/raw/branch/main/debug.log",
"/owner/repo.git/info/refs?service=git-upload-pack",
"/owner/repo/src/branch/main/docs/../README.md",
"/user/login?redirect_to=%2fowner%2frepo",
"/index.php",
"/.well-known/security.txt",
} {
r := get(t, target)
r.Header.Set(userAgent, browser)
matched := files.Match(r)
if len(matched) != 0 {
t.Errorf("%s matched %v, want no rule", target, ids(matched))
}
}
// A request without a user agent is only noted.
wantMatched(t, files, get(t, "/"), "empty-agent")
}
// ruleFiles are files to write into a directory of rule files, by name.
type ruleFiles map[string]string
// writeFiles writes files into a new directory, and returns it.
func writeFiles(t *testing.T, files ruleFiles) string {
t.Helper()
dir := t.TempDir()
for name, content := range files {
path := filepath.Join(dir, name)
err := os.MkdirAll(filepath.Dir(path), 0o700)
if err != nil {
t.Fatalf("mkdir: %v", err)
}
err = os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("write %s: %v", name, err)
}
}
return dir
}
// save writes content to the rule file name in dir as an editor that
// saves by renaming does, so that the file is never seen half written.
func save(t *testing.T, dir, name, content string) {
t.Helper()
path := filepath.Join(dir, name)
err := os.WriteFile(path+".tmp", []byte(content), 0o600)
if err != nil {
t.Fatalf("write %s: %v", name, err)
}
err = os.Rename(path+".tmp", path)
if err != nil {
t.Fatalf("rename: %v", err)
}
}
// newParams returns Params for the rule files in dir, switched on, with
// the process log in the processLog returned.
func newParams(dir string) (rules.Params, processLog) {
lines := make(processLog, maxLogLines)
return rules.Params{
Dir: dir,
Enabled: true,
ProcessLog: slog.New(slog.NewJSONHandler(lines, nil)),
}, lines
}
// load writes files into a new directory and loads the rules in it.
func load(t *testing.T, files ruleFiles) *rules.Files {
t.Helper()
params, _ := newParams(writeFiles(t, files))
params.ProcessLog = slog.New(slog.DiscardHandler)
loaded, err := rules.Load(params)
if err != nil {
t.Fatalf("load: %v", err)
}
return loaded
}
// watch loads the rules in dir, runs their Watch until the test ends, and
// waits until it watches the directory.
func watch(t *testing.T, dir string) (*rules.Files, processLog) {
t.Helper()
params, lines := newParams(dir)
files, err := rules.Load(params)
if err != nil {
t.Fatalf("load: %v", err)
}
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
files.Watch(ctx)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
lines.waitFor(t, watching)
return files, lines
}
// wantRefused checks that loading the rule files in dir fails with the
// error want.
func wantRefused(t *testing.T, dir, want string) {
t.Helper()
params, _ := newParams(dir)
_, err := rules.Load(params)
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
}
// get returns a GET request for target, a path and an optional query, as
// smallwebwaf's server reads it, without a user agent.
func get(t *testing.T, target string) *http.Request {
t.Helper()
return httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"http://app.example"+target, nil)
}
// wantRefusedBy checks that a GET request for target with the user agent
// sent matches rule alone, and that rule refuses it.
func wantRefusedBy(t *testing.T, files *rules.Files, target, sent, rule string) {
t.Helper()
r := get(t, target)
r.Header.Set(userAgent, sent)
matched := files.Match(r)
if len(matched) != 1 || matched[0].ID != rule || matched[0].Action == rules.ActionLog {
t.Errorf("%s from %q matched %v, want %s alone, refusing it", target,
sent, ids(matched), rule)
}
}
// wantMatched checks the ids of the rules r matches, in order.
func wantMatched(t *testing.T, files *rules.Files, r *http.Request, want ...string) {
t.Helper()
got := ids(files.Match(r))
if !slices.Equal(got, want) {
t.Errorf("%s matched %v, want %v", r.URL, got, want)
}
}
// ids returns the ids of matched.
func ids(matched []rules.Rule) []string {
got := make([]string, 0, len(matched))
for _, rule := range matched {
got = append(got, rule.ID)
}
return got
}
// processLog receives the lines of a process log, each a JSON object, for
// a test to wait for.
type processLog chan string
// Write receives a line of the process log.
func (l processLog) Write(line []byte) (int, error) {
l <- string(line)
return len(line), nil
}
// waitFor returns the next line of the process log whose message is msg,
// passing over the lines before it. It waits as long as that takes, so
// that a slow test process cannot fail the test.
func (l processLog) waitFor(t *testing.T, msg string) map[string]any {
t.Helper()
for line := range l {
var fields map[string]any
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("process log line %q is not JSON: %v", line, err)
}
if fields["msg"] == msg {
return fields
}
}
return nil
}
// waitUntil waits for the rule files to be read until done reports true,
// as it does once they have been read after the test's last change. They
// can be read before then too, as they are once Watch starts watching.
func (l processLog) waitUntil(t *testing.T, done func() bool) {
t.Helper()
for !done() {
l.waitFor(t, read)
}
}
-162
View File
@@ -1,162 +0,0 @@
package rules
import (
"context"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
)
// The tests below run readAfterChanges in a synctest bubble, where time is
// a clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
func TestFileWrittenInTwoPartsTakenInOnlyWhole(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
changes := run(t, files)
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create: %v", err)
}
defer func() {
_ = file.Close()
}()
// The first part ends in the middle of a ban rule's regex, which,
// read then, would ban every request.
write(t, file, "first path block ^/first\nprobe path ban ^/")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/anything")
// The second part starts the wait again.
write(t, file, `\.env$`+"\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env", "probe")
wantMatched(t, files, "/anything")
})
}
func TestEditSavedBeforeTheWatchStartsTakenIn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
// Saved after Load read the files, and before the directory was
// watched, so that no change is seen for it.
writeFile(t, path, "first path block ^/edited\n")
run(t, files)
time.Sleep(quietTime)
synctest.Wait()
wantMatched(t, files, "/edited", "first")
})
}
// load loads the rules in dir.
func load(t *testing.T, dir string) *Files {
t.Helper()
files, err := Load(Params{
Dir: dir, Enabled: true, ProcessLog: slog.New(slog.DiscardHandler),
})
if err != nil {
t.Fatalf("load: %v", err)
}
return files
}
// run runs files' readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func run(t *testing.T, files *Files) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
files.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// writeFile writes content to the file at path.
func writeFile(t *testing.T, path, content string) {
t.Helper()
err := os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("write %s: %v", path, err)
}
}
// write writes text to the end of file.
func write(t *testing.T, file *os.File, text string) {
t.Helper()
_, err := file.WriteString(text)
if err != nil {
t.Fatalf("write: %v", err)
}
}
// wantMatched checks the ids of the rules that a GET request for path
// matches, in order.
func wantMatched(t *testing.T, files *Files, path string, want ...string) {
t.Helper()
r := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"http://app.example"+path, nil)
matched := files.Match(r)
got := make([]string, 0, len(matched))
for _, rule := range matched {
got = append(got, rule.ID)
}
if !slices.Equal(got, want) {
t.Errorf("%s matched %v, want %v", path, got, want)
}
}
+3 -5
View File
@@ -23,8 +23,6 @@ var errHealthEndpoint = errors.New("smallwebwaf's health endpoint answered")
// smallwebwaf answers its health endpoint on 127.0.0.1, at the port in // smallwebwaf answers its health endpoint on 127.0.0.1, at the port in
// SWWAF_LISTEN_ADDR, and the app accepts connections at the address in // SWWAF_LISTEN_ADDR, and the app accepts connections at the address in
// SWWAF_UPSTREAM_URL. Otherwise it writes why to stderr and returns 1. // SWWAF_UPSTREAM_URL. Otherwise it writes why to stderr and returns 1.
// It reads no other setting, nor a file that another names, so neither
// can fail it.
// args are the arguments after `healthcheck`; it takes none, and given // args are the arguments after `healthcheck`; it takes none, and given
// one it names it on stderr and returns 1 without checking anything. // one it names it on stderr and returns 1 without checking anything.
func HealthCheck( func HealthCheck(
@@ -52,13 +50,13 @@ func healthCheck(ctx context.Context, lookupEnv func(string) (string, bool)) err
ctx, cancel := context.WithTimeout(ctx, healthCheckTimeout) ctx, cancel := context.WithTimeout(ctx, healthCheckTimeout)
defer cancel() defer cancel()
listenAddr, upstreamURL, err := config.ListenAddrAndUpstreamURL(lookupEnv) cfg, err := config.FromEnvironment(lookupEnv)
if err != nil { if err != nil {
return fmt.Errorf("invalid setting: %w", err) return fmt.Errorf("invalid setting: %w", err)
} }
// The settings have checked that the address has a port. // The settings have checked that the address has a port.
_, port, _ := net.SplitHostPort(listenAddr) _, port, _ := net.SplitHostPort(cfg.ListenAddr)
health := "http://" + net.JoinHostPort("127.0.0.1", port) + proxy.HealthPath health := "http://" + net.JoinHostPort("127.0.0.1", port) + proxy.HealthPath
req, err := http.NewRequestWithContext(ctx, http.MethodGet, health, http.NoBody) req, err := http.NewRequestWithContext(ctx, http.MethodGet, health, http.NoBody)
@@ -77,7 +75,7 @@ func healthCheck(ctx context.Context, lookupEnv func(string) (string, bool)) err
return fmt.Errorf("%w %s", errHealthEndpoint, res.Status) return fmt.Errorf("%w %s", errHealthEndpoint, res.Status)
} }
conn, err := (&net.Dialer{}).DialContext(ctx, "tcp", appAddress(upstreamURL)) conn, err := (&net.Dialer{}).DialContext(ctx, "tcp", appAddress(cfg.UpstreamURL))
if err != nil { if err != nil {
return fmt.Errorf("connect to the app: %w", err) return fmt.Errorf("connect to the app: %w", err)
} }
-33
View File
@@ -6,8 +6,6 @@ import (
"net" "net"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os"
"path/filepath"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -30,7 +28,6 @@ func TestHealthCheck(t *testing.T) {
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
upstreamURL: app.URL, upstreamURL: app.URL,
stateDir: t.TempDir(), stateDir: t.TempDir(),
rulesDir: t.TempDir(),
} }
go func() { go func() {
@@ -45,21 +42,6 @@ func TestHealthCheck(t *testing.T) {
wantHealthCheck(t, env, 0, "") wantHealthCheck(t, env, 0, "")
// The health check reads those two settings alone, here given as
// files: a removed or invalid token file, or an invalid value of
// another setting, does not fail it.
for _, other := range []struct{ name, value string }{
{"SWWAF_METRICS_TOKEN_FILE", filepath.Join(t.TempDir(), "removed")},
{"SWWAF_METRICS_TOKEN_FILE", writeFile(t, "too short\n")},
{"SWWAF_MODE", "neither"},
} {
wantHealthCheck(t, map[string]string{
listenAddr + "_FILE": writeFile(t, ":"+port+"\n"),
upstreamURL + "_FILE": writeFile(t, app.URL+"\n"),
other.name: other.value,
}, 0, "")
}
app.Close() app.Close()
wantHealthCheck(t, env, 1, "unhealthy: connect to the app: ") wantHealthCheck(t, env, 1, "unhealthy: connect to the app: ")
@@ -115,18 +97,3 @@ func wantHealthCheck(t *testing.T, env map[string]string, status int, message st
got, wrote, status, message) got, wrote, status, message)
} }
} }
// writeFile writes contents to a file in a directory of its own, removed
// when the test ends, and returns the file's path.
func writeFile(t *testing.T, contents string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "setting")
err := os.WriteFile(path, []byte(contents), 0o600)
if err != nil {
t.Fatalf("write %s: %v", path, err)
}
return path
}
+12 -93
View File
@@ -1,6 +1,6 @@
// Package smallwebwaf runs the smallwebwaf process: it reads the settings, // Package smallwebwaf runs the smallwebwaf process: it reads the settings
// the rule files and the state files, serves requests until it is told to // and the state files, serves requests until it is told to stop, and then
// stop, and then stops in an orderly way, writing the state files. // stops in an orderly way, writing the state files.
package smallwebwaf package smallwebwaf
import ( import (
@@ -18,9 +18,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/proxy" "sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/state" "sneak.berlin/go/smallwebwaf/internal/state"
) )
@@ -29,11 +27,6 @@ import (
// runit and docker wait a little longer before they kill the process. // runit and docker wait a little longer before they kill the process.
const shutdownTimeout = 5 * time.Second const shutdownTimeout = 5 * time.Second
// remoteLogStopTimeout is how long, as smallwebwaf stops, the log lines
// still waiting are sent to SWWAF_LOG_REMOTE_URL before they are given
// up. stdout has carried them.
const remoteLogStopTimeout = 2 * time.Second
// Params are what Run needs from the process. // Params are what Run needs from the process.
type Params struct { type Params struct {
// Version is the version of the binary, set when it is built. // Version is the version of the binary, set when it is built.
@@ -63,9 +56,9 @@ func Main(version string) int {
}) })
} }
// Run reads the settings, the rule files and the state files, then serves // Run reads the settings and the state files, then serves requests until
// requests until ctx is done. It returns the process's exit status, 1 // ctx is done. It returns the process's exit status, 1 when smallwebwaf
// when smallwebwaf cannot start. // cannot start.
func Run(ctx context.Context, params Params) int { func Run(ctx context.Context, params Params) int {
processLog := requestlog.NewProcessLogger(params.Stdout) processLog := requestlog.NewProcessLogger(params.Stdout)
@@ -76,46 +69,16 @@ func Run(ctx context.Context, params Params) int {
return 1 return 1
} }
// While SWWAF_LOG_REMOTE_URL is set, every line on stdout from here on
// is sent there too.
stdout := params.Stdout
var remote *remotelog.Sender
if cfg.LogRemoteURL != nil {
remote = newRemoteLogSender(cfg)
stdout = io.MultiWriter(params.Stdout, remote)
processLog = requestlog.NewProcessLogger(stdout)
stopSending := startSending(ctx, remote, processLog)
defer stopSending()
}
ruleFiles, err := rules.Load(rules.Params{
Dir: cfg.RulesDir,
Enabled: cfg.RulesEnabled,
ProcessLog: processLog,
})
if err != nil {
processLog.Error("cannot use the rule files", "error", err.Error())
return 1
}
// The state files give times in UTC. // The state files give times in UTC.
now := func() time.Time { return time.Now().UTC() } now := func() time.Time { return time.Now().UTC() }
server := proxy.New(proxy.Params{ server := proxy.New(proxy.Params{
Config: cfg, Config: cfg,
RequestLog: stdout, RequestLog: params.Stdout,
ProcessLog: processLog, ProcessLog: processLog,
GeoJSURL: lookup.URL, GeoJSURL: lookup.URL,
Now: now, Now: now,
Rules: ruleFiles,
}) })
if remote != nil {
server.Metrics.AddRemoteLog(remote)
}
files, err := state.Load(state.Params{ files, err := state.Load(state.Params{
Dir: cfg.StateDir, Dir: cfg.StateDir,
@@ -147,53 +110,16 @@ func Run(ctx context.Context, params Params) int {
"address", listener.Addr().String(), "address", listener.Addr().String(),
"settings", cfg) "settings", cfg)
return serve(ctx, server.Server, listener, files, ruleFiles, processLog) return serve(ctx, server.Server, listener, files, processLog)
}
// newRemoteLogSender returns a sender of the log lines to
// SWWAF_LOG_REMOTE_URL, with the settings for it.
func newRemoteLogSender(cfg *config.Config) *remotelog.Sender {
return remotelog.New(remotelog.Params{
URL: cfg.LogRemoteURL,
RootCAs: cfg.LogRemoteTLSCAs,
Buffer: cfg.LogRemoteBuffer,
Facility: cfg.LogRemoteFacility,
AppName: cfg.LogRemoteAppName,
})
}
// startSending runs remote until the function it returns is called, which
// then waits at most remoteLogStopTimeout for the lines still waiting to
// be sent. Sending goes on after ctx is done, so that the lines written
// while smallwebwaf stops are sent too.
func startSending(
ctx context.Context, remote *remotelog.Sender, processLog *slog.Logger,
) func() {
sending, stop := context.WithCancel(context.WithoutCancel(ctx))
sent := make(chan struct{})
go func() {
remote.Run(sending, processLog)
close(sent)
}()
return func() {
stop()
select {
case <-sent:
case <-time.After(remoteLogStopTimeout):
}
}
} }
// serve serves requests on listener, writes the state files as they are // serve serves requests on listener, writes the state files as they are
// due, takes in an admin's edits of them, and reads the rule files again // due, and takes in an admin's edits of them, until ctx is done. Then it
// as they change, until ctx is done. Then it gives the requests in // gives the requests in progress shutdownTimeout to finish, and writes
// progress shutdownTimeout to finish, and writes every state file. // every state file.
func serve( func serve(
ctx context.Context, server *http.Server, listener net.Listener, ctx context.Context, server *http.Server, listener net.Listener,
files *state.Files, ruleFiles *rules.Files, processLog *slog.Logger, files *state.Files, processLog *slog.Logger,
) int { ) int {
served := make(chan error, 1) served := make(chan error, 1)
@@ -206,7 +132,6 @@ func serve(
written := make(chan struct{}) written := make(chan struct{})
watched := make(chan struct{}) watched := make(chan struct{})
rulesWatched := make(chan struct{})
go func() { go func() {
files.Run(writing) files.Run(writing)
@@ -218,11 +143,6 @@ func serve(
close(watched) close(watched)
}() }()
go func() {
ruleFiles.Watch(writing)
close(rulesWatched)
}()
select { select {
case err := <-served: case err := <-served:
processLog.Error("serving failed", "error", err.Error()) processLog.Error("serving failed", "error", err.Error())
@@ -261,7 +181,6 @@ func serve(
// missing from clients.json. // missing from clients.json.
<-written <-written
<-watched <-watched
<-rulesWatched
err = files.WriteAll() err = files.WriteAll()
if err != nil { if err != nil {
@@ -1,82 +0,0 @@
package smallwebwaf
import (
"log/slog"
"net/url"
"testing"
"testing/synctest"
"time"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
)
// The stop's tests run in a synctest bubble, where the time package runs
// on a clock of the test's own, so that how long the stop takes can be
// told exactly. The sender is held up by its process log, not by the
// network: a goroutine of the bubble that waits on the network keeps that
// clock from moving on.
func TestStopWaitsForTheSenderToFinish(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
took := stopHeldSender(t, time.Second)
if took != time.Second {
t.Errorf("the stop took %s, want the second the sender took", took)
}
})
}
func TestStopWaitsForTheSenderAtMostTwoSeconds(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
took := stopHeldSender(t, time.Minute)
if took != 2*time.Second {
t.Errorf("the stop took %s, want 2s", took)
}
})
}
// heldLog holds each line written to it until it is closed.
type heldLog chan struct{}
// Write waits until the log is closed.
func (l heldLog) Write(p []byte) (int, error) {
<-l
return len(p), nil
}
// stopHeldSender starts sending to an endpoint the sender cannot connect
// to, holds the sender as it logs that failure until release has passed,
// stops the sending, and returns how long the stop took. It returns once
// the sender has ended, as a bubble must.
func stopHeldSender(t *testing.T, release time.Duration) time.Duration {
t.Helper()
log := make(heldLog)
sender := remotelog.New(remotelog.Params{
// No port is 65536, so each attempt to connect fails at once,
// before it reaches the network.
URL: &url.URL{Scheme: remotelog.SchemeTCP, Host: "127.0.0.1:65536"},
Buffer: 1,
})
stopSending := startSending(t.Context(), sender,
slog.New(slog.NewJSONHandler(log, nil)))
synctest.Wait()
time.AfterFunc(release, func() { close(log) })
stopped := time.Now()
stopSending()
took := time.Since(stopped)
time.Sleep(release)
synctest.Wait()
return took
}
+1 -269
View File
@@ -10,8 +10,6 @@ import (
"net/http/httptest" "net/http/httptest"
"os" "os"
"path/filepath" "path/filepath"
"slices"
"strconv"
"strings" "strings"
"sync" "sync"
"testing" "testing"
@@ -36,7 +34,6 @@ const (
stateWriteDelay = "SWWAF_STATE_WRITE_DELAY" stateWriteDelay = "SWWAF_STATE_WRITE_DELAY"
stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL" stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL"
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY" rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
rulesDir = "SWWAF_RULES_DIR"
// greeting is what the tests' app answers. // greeting is what the tests' app answers.
greeting = "hello from the app" greeting = "hello from the app"
) )
@@ -166,7 +163,6 @@ func TestAddressInUseStopsTheStart(t *testing.T) {
status := run(t.Context(), map[string]string{ status := run(t.Context(), map[string]string{
listenAddr: taken.Addr().String(), listenAddr: taken.Addr().String(),
stateDir: t.TempDir(), stateDir: t.TempDir(),
rulesDir: t.TempDir(),
}, out) }, out)
if status != 1 { if status != 1 {
t.Errorf("exit status %d, want 1", status) t.Errorf("exit status %d, want 1", status)
@@ -190,15 +186,9 @@ func TestServesUntilToldToStop(t *testing.T) {
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
upstreamURL: appURL, upstreamURL: appURL,
stateDir: dir, stateDir: dir,
rulesDir: filepath.Join("..", "..", "share", "rules.d"),
}, out) }, out)
}() }()
// The default rule file is read.
if rules := out.line(t, "msg", "read the rule files")["rules"]; rules != 12.0 {
t.Errorf("read %v rules from the default rule file, want 12", rules)
}
starting := out.line(t, "msg", "starting") starting := out.line(t, "msg", "starting")
wantStartingLine(t, starting, appURL, dir) wantStartingLine(t, starting, appURL, dir)
@@ -227,7 +217,6 @@ func TestStateKeptAcrossRestarts(t *testing.T) {
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
upstreamURL: startApp(t), upstreamURL: startApp(t),
stateDir: t.TempDir(), stateDir: t.TempDir(),
rulesDir: t.TempDir(),
rateLimitPerDay: "2", rateLimitPerDay: "2",
// Neither comes due in the test: the files are written as // Neither comes due in the test: the files are written as
// smallwebwaf stops. // smallwebwaf stops.
@@ -264,7 +253,6 @@ func TestBanRefusesItsNetblockAfterARestartWithAnotherScope(t *testing.T) {
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
upstreamURL: startApp(t), upstreamURL: startApp(t),
stateDir: t.TempDir(), stateDir: t.TempDir(),
rulesDir: t.TempDir(),
trustedProxies: localhost + "/32", trustedProxies: localhost + "/32",
rateLimitPerDay: "1", rateLimitPerDay: "1",
scope: "24", scope: "24",
@@ -312,7 +300,6 @@ func TestBanAddedAndLiftedByEditingBansJSON(t *testing.T) {
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
upstreamURL: startApp(t), upstreamURL: startApp(t),
stateDir: dir, stateDir: dir,
rulesDir: t.TempDir(),
trustedProxies: localhost + "/32", trustedProxies: localhost + "/32",
// No write comes due in the test, so only the watch on the // No write comes due in the test, so only the watch on the
// directory can take the edits in. // directory can take the edits in.
@@ -329,170 +316,6 @@ func TestBanAddedAndLiftedByEditingBansJSON(t *testing.T) {
}) })
} }
func TestRuleFileAddedWhileRunningTakesEffect(t *testing.T) {
t.Parallel()
dir := t.TempDir()
env := map[string]string{
listenAddr: localhost + ":0",
upstreamURL: startApp(t),
stateDir: t.TempDir(),
rulesDir: dir,
// The requests sent until the rule takes effect must not break a
// rate limit, whose ban would refuse them too.
"SWWAF_RATE_LIMIT_PER_MINUTE": "off",
}
out := runUntilStopped(t, env, func(url string) {
wantGreeting(t, url)
// Written once: each change would start the rule files' wait
// again. A file written before smallwebwaf watches the directory is
// read once it does.
err := os.WriteFile(filepath.Join(dir, "50-app.rules"),
[]byte("everything path block ^/\n"), 0o600)
if err != nil {
t.Fatalf("write the rule file: %v", err)
}
// As long as that takes, so that a slow test process cannot fail
// the test.
for statusFrom(t, url, "203.0.113.9") != http.StatusForbidden {
time.Sleep(pollInterval)
}
})
out.line(t, "action", "rule_blocked")
}
func TestRuleFileThatDoesNotParseStopsTheStart(t *testing.T) {
t.Parallel()
dir := t.TempDir()
path := filepath.Join(dir, "00-default.rules")
err := os.WriteFile(path, []byte("# probes\nenv-file path bann ^/\\.env$\n"), 0o600)
if err != nil {
t.Fatalf("write the rule file: %v", err)
}
wantRulesRefused(t, dir, path+`, line 2: the action "bann" is not log, block or ban`)
}
func TestRulesDirThatDoesNotExistStopsTheStart(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "rules.d")
wantRulesRefused(t, dir, "SWWAF_RULES_DIR cannot be read: open "+dir+
": no such file or directory")
}
func TestEveryLineIsAlsoSentToTheRemoteLogEndpoint(t *testing.T) {
t.Parallel()
endpoint, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
if err != nil {
t.Fatalf("listen: %v", err)
}
defer func() {
_ = endpoint.Close()
}()
env := map[string]string{
listenAddr: localhost + ":0",
upstreamURL: startApp(t),
stateDir: t.TempDir(),
rulesDir: t.TempDir(),
"SWWAF_LOG_REMOTE_URL": "syslog+tcp://" + endpoint.Addr().String(),
}
out := runUntilStopped(t, env, func(url string) {
wantGreeting(t, url)
})
out.line(t, "type", "request")
// smallwebwaf connected as it started, and closes the connection once
// it has sent the lines written as it stopped.
conn, err := endpoint.Accept()
if err != nil {
t.Fatalf("accept: %v", err)
}
received, err := io.ReadAll(conn)
_ = conn.Close()
if err != nil {
t.Fatalf("read: %v", err)
}
// Lines written at once by several goroutines may reach stdout and
// the endpoint in different orders.
sent := messages(t, string(received))
written := slices.Collect(strings.Lines(out.text()))
slices.Sort(sent)
slices.Sort(written)
if !slices.Equal(sent, written) {
t.Errorf("sent\n%v\nwrote\n%v", sent, written)
}
}
func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) {
t.Parallel()
const token = "0123456789abcdef0123456789abcdef"
// The endpoint takes connections and never answers, so the TLS
// handshake of each waits on it, and no line is ever sent.
endpoint, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
if err != nil {
t.Fatalf("listen: %v", err)
}
defer func() {
_ = endpoint.Close()
}()
env := map[string]string{
listenAddr: localhost + ":0",
upstreamURL: startApp(t),
stateDir: t.TempDir(),
rulesDir: t.TempDir(),
"SWWAF_LOG_REMOTE_URL": "syslog+tls://" + endpoint.Addr().String(),
"SWWAF_LOG_REMOTE_BUFFER": "1",
"SWWAF_METRICS_TOKEN": token,
}
out := runUntilStopped(t, env, func(url string) {
wantGreeting(t, url)
// More than one line has been written, and the buffer holds the
// last.
metrics := metricsText(t, url+"_smallwebwaf/metrics", token)
for _, series := range []string{
"smallwebwaf_remote_log_lines_sent_total 0",
"smallwebwaf_remote_log_buffer_depth 1",
} {
if !strings.Contains(metrics, "\n"+series+"\n") {
t.Errorf("no %q in the metrics:\n%s", series, metrics)
}
}
if strings.Contains(metrics, "\nsmallwebwaf_remote_log_lines_dropped_total 0\n") ||
!strings.Contains(metrics, "\nsmallwebwaf_remote_log_lines_dropped_total ") {
t.Errorf("no line dropped in the metrics:\n%s", metrics)
}
// Closed, the endpoint refuses the connection made to send the
// lines still waiting at the stop, which then does not wait.
_ = endpoint.Close()
})
out.line(t, "type", "request")
}
func TestStateFileThatDoesNotParseStopsTheStart(t *testing.T) { func TestStateFileThatDoesNotParseStopsTheStart(t *testing.T) {
t.Parallel() t.Parallel()
@@ -526,9 +349,7 @@ func wantStartRefused(t *testing.T, dir, want string) {
out := &output{} out := &output{}
status := run(ctx, map[string]string{ status := run(ctx, map[string]string{listenAddr: localhost + ":0", stateDir: dir}, out)
listenAddr: localhost + ":0", stateDir: dir, rulesDir: t.TempDir(),
}, out)
if status != 1 { if status != 1 {
t.Fatalf("exit status %d, want 1", status) t.Fatalf("exit status %d, want 1", status)
} }
@@ -541,30 +362,6 @@ func wantStartRefused(t *testing.T, dir, want string) {
} }
} }
// wantRulesRefused runs smallwebwaf with its rule files in dir, and
// checks that it stops at start, with the error want. If it starts
// instead, it is stopped after waitLimit.
func wantRulesRefused(t *testing.T, dir, want string) {
t.Helper()
ctx, stop := context.WithTimeout(t.Context(), waitLimit)
defer stop()
out := &output{}
status := run(ctx, map[string]string{
listenAddr: localhost + ":0", stateDir: t.TempDir(), rulesDir: dir,
}, out)
if status != 1 {
t.Fatalf("exit status %d, want 1", status)
}
line := out.line(t, "msg", "cannot use the rule files")
if line["error"] != want || line["level"] != "ERROR" {
t.Errorf("start refused with %v, want the error %q", line, want)
}
}
// startApp starts an app that answers every request with greeting, and // startApp starts an app that answers every request with greeting, and
// returns its URL. // returns its URL.
func startApp(t *testing.T) string { func startApp(t *testing.T) string {
@@ -646,10 +443,8 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
"SWWAF_LIMIT_BAN_DURATION": "1h", "SWWAF_LIMIT_BAN_DURATION": "1h",
"SWWAF_LIMIT_BAN_REPEAT_WINDOW": "24h", "SWWAF_LIMIT_BAN_REPEAT_WINDOW": "24h",
"SWWAF_MAX_BAN_DURATION": "7d", "SWWAF_MAX_BAN_DURATION": "7d",
"SWWAF_ATTACK_BAN_DURATION": "7d",
"SWWAF_MAX_BANS": "5000", "SWWAF_MAX_BANS": "5000",
"SWWAF_BAN_SCOPE_V4_PREFIX": "32", "SWWAF_BAN_SCOPE_V4_PREFIX": "32",
"SWWAF_RULES_ENABLED": "true",
} }
for name, value := range want { for name, value := range want {
@@ -689,69 +484,6 @@ func wantGreeting(t *testing.T, url string) {
} }
} }
// messages returns the message of each record in received, octet-counted
// frames of RFC 5424 records with the default facility and app name, each
// with the newline that ends a line on stdout.
func messages(t *testing.T, received string) []string {
t.Helper()
hostname, _ := os.Hostname()
header := " " + hostname + " " + hostname + " - - - "
var found []string
for received != "" {
count, rest, _ := strings.Cut(received, " ")
length, err := strconv.Atoi(count)
if err != nil || length > len(rest) {
t.Fatalf("no frame at %q", received)
}
record := rest[:length]
received = rest[length:]
_, message, ok := strings.Cut(record, header)
if !ok || !strings.HasPrefix(record, "<134>1 ") {
t.Fatalf("record %q, want priority <134> and header %q", record, header)
}
found = append(found, message+"\n")
}
return found
}
// metricsText asks for the metrics at url with token, and returns them.
func metricsText(t *testing.T, url, token string) string {
t.Helper()
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, url,
http.NoBody)
if err != nil {
t.Fatalf("new request: %v", err)
}
req.Header.Set("Authorization", "Bearer "+token)
transport := &http.Transport{}
defer transport.CloseIdleConnections()
res, err := (&http.Client{Transport: transport}).Do(req)
if err != nil {
t.Fatalf("request: %v", err)
}
body, err := io.ReadAll(res.Body)
_ = res.Body.Close()
if err != nil || res.StatusCode != http.StatusOK {
t.Fatalf("metrics answered %d (%v)", res.StatusCode, err)
}
return string(body)
}
// wantRefused checks that a request to url is refused with 403, the // wantRefused checks that a request to url is refused with 403, the
// default SWWAF_BAN_RESPONSE. // default SWWAF_BAN_RESPONSE.
func wantRefused(t *testing.T, url string) { func wantRefused(t *testing.T, url string) {
+10 -40
View File
@@ -48,7 +48,6 @@ var (
errVersion = errors.New("unknown version") errVersion = errors.New("unknown version")
// errMissing is for an entry without a field it needs. // errMissing is for an entry without a field it needs.
errMissing = errors.New("has no") errMissing = errors.New("has no")
errCause = errors.New("is not limit, attack or admin")
) )
// Params are what Load needs. // Params are what Load needs.
@@ -96,15 +95,11 @@ type bansFile struct {
} }
// banEntry is a ban as bans.json holds it: a permanent ban's expires is // banEntry is a ban as bans.json holds it: a permanent ban's expires is
// null, a ban an admin added may have no cause, which makes it an // null.
// admin's, and lifted is left out until an admin lifts the ban.
type banEntry struct { type banEntry struct {
Netblock netip.Prefix `json:"netblock"` Netblock netip.Prefix `json:"netblock"`
Start time.Time `json:"start"` Start time.Time `json:"start"`
Expires *time.Time `json:"expires"` Expires *time.Time `json:"expires"`
Cause string `json:"cause"`
Reason string `json:"reason,omitempty"`
Lifted *time.Time `json:"lifted,omitempty"`
Notes bans.Notes `json:"notes"` Notes bans.Notes `json:"notes"`
} }
@@ -265,7 +260,7 @@ func (f *Files) fileChanged(name string) {
// runs, by Watch or by a write, is taken in here. An edit that does not // runs, by Watch or by a write, is taken in here. An edit that does not
// parse is neither counted nor logged, and takeIn's error returned. // parse is neither counted nor logged, and takeIn's error returned.
func (f *Files) takeInEdit(name string, data []byte) error { func (f *Files) takeInEdit(name string, data []byte) error {
_, err := f.takeIn(name, data, true) _, err := f.takeIn(name, data)
if err != nil { if err != nil {
return err return err
} }
@@ -287,7 +282,7 @@ func (f *Files) read(name string) (int, error) {
return 0, err return 0, err
} }
return f.takeIn(name, data, false) return f.takeIn(name, data)
} }
// readChanged returns what the state file name holds, and whether that // readChanged returns what the state file name holds, and whether that
@@ -311,11 +306,9 @@ func (f *Files) readChanged(name string) ([]byte, bool, error) {
// takeIn parses data, what the state file name holds, puts it into the // takeIn parses data, what the state file name holds, puts it into the
// part that keeps that state, in place of what the part held, and returns // part that keeps that state, in place of what the part held, and returns
// how many entries the file holds. edit is whether data is an admin's // how many entries the file holds. An error names the file and, where the
// edit taken in while smallwebwaf runs, rather than the file read at the // JSON decoder tells it, the line and column, or else the entry.
// start. An error names the file and, where the JSON decoder tells it, func (f *Files) takeIn(name string, data []byte) (int, error) {
// the line and column, or else the entry.
func (f *Files) takeIn(name string, data []byte, edit bool) (int, error) {
path := filepath.Join(f.params.Dir, name) path := filepath.Join(f.params.Dir, name)
var entries int var entries int
@@ -334,12 +327,7 @@ func (f *Files) takeIn(name string, data []byte, edit bool) (int, error) {
held = append(held, entry.ban()) held = append(held, entry.ban())
} }
if edit { f.params.Ledger.Load(held)
f.params.Ledger.LoadEdit(held)
} else {
f.params.Ledger.Load(held)
}
entries = len(held) entries = len(held)
case clientsJSON: case clientsJSON:
var file clientsFile var file clientsFile
@@ -456,35 +444,21 @@ func (f *Files) encode(name string) ([]byte, error) {
// newBanEntry returns ban as bans.json holds it. // newBanEntry returns ban as bans.json holds it.
func newBanEntry(ban bans.Ban) banEntry { func newBanEntry(ban bans.Ban) banEntry {
entry := banEntry{ entry := banEntry{Netblock: ban.Netblock, Start: ban.Start, Notes: ban.Notes}
Netblock: ban.Netblock, Start: ban.Start, Cause: ban.Cause, Reason: ban.Reason,
Notes: ban.Notes,
}
if !ban.Permanent() { if !ban.Permanent() {
entry.Expires = &ban.Expires entry.Expires = &ban.Expires
} }
if !ban.Lifted.IsZero() {
entry.Lifted = &ban.Lifted
}
return entry return entry
} }
// ban returns the ban an entry of bans.json holds. // ban returns the ban an entry of bans.json holds.
func (e banEntry) ban() bans.Ban { func (e banEntry) ban() bans.Ban {
ban := bans.Ban{ ban := bans.Ban{Netblock: e.Netblock, Start: e.Start, Notes: e.Notes}
Netblock: e.Netblock, Start: e.Start, Cause: e.Cause, Reason: e.Reason,
Notes: e.Notes,
}
if e.Expires != nil { if e.Expires != nil {
ban.Expires = *e.Expires ban.Expires = *e.Expires
} }
if e.Lifted != nil {
ban.Lifted = *e.Lifted
}
return ban return ban
} }
@@ -492,8 +466,7 @@ func (e banEntry) ban() bans.Ban {
// client, a start, from which the length of the netblock's next ban is // client, a start, from which the length of the netblock's next ban is
// worked out, or an expires, which would make it permanent. A permanent // worked out, or an expires, which would make it permanent. A permanent
// ban's expires is null, which Bans cannot tell from a missing one, so // ban's expires is null, which Bans cannot tell from a missing one, so
// each expires is read again as written. A cause other than limit, // each expires is read again as written.
// attack or admin, most likely misspelt, is refused too.
func (f *bansFile) check(data []byte) error { func (f *bansFile) check(data []byte) error {
var written struct { var written struct {
Bans []struct { Bans []struct {
@@ -514,9 +487,6 @@ func (f *bansFile) check(data []byte) error {
return missing(i, "start") return missing(i, "start")
case written.Bans[i].Expires == nil: case written.Bans[i].Expires == nil:
return missing(i, "expires") return missing(i, "expires")
case entry.Cause != "" && entry.Cause != bans.CauseLimit &&
entry.Cause != bans.CauseAttack && entry.Cause != bans.CauseAdmin:
return fmt.Errorf("entry %d's cause %q %w", i+1, entry.Cause, errCause)
} }
} }
+5 -186
View File
@@ -48,8 +48,6 @@ const permanentBansJSON = `{
"netblock": "2001:db8::/64", "netblock": "2001:db8::/64",
"start": "2026-10-06T00:00:00Z", "start": "2026-10-06T00:00:00Z",
"expires": null, "expires": null,
"cause": "admin",
"reason": "scrapes every commit",
"notes": { "notes": {
"country": "DE", "country": "DE",
"limit": 1000, "limit": 1000,
@@ -65,26 +63,13 @@ const permanentBansJSON = `{
}, },
"requests": 1500, "requests": 1500,
"refused": 3, "refused": 3,
"earlier_bans": { "earlier_bans": 5
"limit": 3,
"attack": 1,
"admin": 1
}
} }
} }
] ]
} }
` `
// liftedClient is the client whose ban liftedBansJSON holds.
const liftedClient = "203.0.113.9"
// liftedBansJSON is bans.json holding an hour's ban for a broken limit on
// liftedClient, from midnight, that an admin lifted ten minutes in.
const liftedBansJSON = `{"version": 1, "bans": [{"netblock": "203.0.113.9/32", ` +
`"start": "2026-10-06T00:00:00Z", "expires": "2026-10-06T01:00:00Z", ` +
`"cause": "limit", "lifted": "2026-10-06T00:10:00Z"}]}`
func TestFilesWrittenAndReadBack(t *testing.T) { func TestFilesWrittenAndReadBack(t *testing.T) {
t.Parallel() t.Parallel()
@@ -278,19 +263,6 @@ func TestEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
} }
} }
func TestBanWithAnotherCauseStopsTheStart(t *testing.T) {
t.Parallel()
wantRefused(t, bansJSON, `{"version": 1, "bans": [`+
`{"netblock": "203.0.113.9/32", "start": "2026-10-06T00:00:00Z", `+
`"expires": null, "cause": "attack"}, `+
`{"netblock": "203.0.113.10/32", "start": "2026-10-06T00:00:00Z", `+
`"expires": null, "cause": "admin"}, `+
`{"netblock": "203.0.113.11/32", "start": "2026-10-06T00:00:00Z", `+
`"expires": null, "cause": "atack"}]}`,
`: entry 3's cause "atack" is not limit, attack or admin`)
}
func TestUnknownVersionStopsTheStart(t *testing.T) { func TestUnknownVersionStopsTheStart(t *testing.T) {
t.Parallel() t.Parallel()
@@ -624,7 +596,7 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
`"start": "2026-10-06T00:00:00Z", "expires": null}]}`) `"start": "2026-10-06T00:00:00Z", "expires": null}]}`)
wantTakenIn(t, lines, dir, bansJSON) wantTakenIn(t, lines, dir, bansJSON)
wantEqual(t, bansJSON, params.Ledger.Snapshot(), wantEqual(t, bansJSON, params.Ledger.Snapshot(),
[]bans.Ban{{Netblock: client, Start: midnight(), Cause: bans.CauseAdmin}}) []bans.Ban{{Netblock: client, Start: midnight()}})
edit(t, dir, clientsJSON, `{"version": 1, "clients": [`+ edit(t, dir, clientsJSON, `{"version": 1, "clients": [`+
`{"client": "198.51.100.7/32", "history": {"requests": 7}}]}`) `{"client": "198.51.100.7/32", "history": {"requests": 7}}]}`)
@@ -723,100 +695,6 @@ func TestBanAddedAndLiftedThroughBansJSON(t *testing.T) {
} }
} }
func TestBanWithoutACauseTakenInAsAnAdmins(t *testing.T) {
t.Parallel()
const reason = "probes for logins"
// adminsBansJSON is bans.json as an admin writes it, with a ban on
// netblock without a cause, and adminsBans the bans it holds.
adminsBansJSON := func(netblock string) string {
return `{"version": 1, "bans": [{"netblock": "` + netblock + `", ` +
`"start": "2026-10-06T00:00:00Z", "expires": null, "reason": "` +
reason + `"}]}`
}
adminsBans := func(netblock string) []bans.Ban {
return []bans.Ban{{
Netblock: netip.MustParsePrefix(netblock),
Start: midnight(),
Cause: bans.CauseAdmin,
Reason: reason,
}}
}
// Read at the start, the ban is taken in as an admin's, though not
// counted among the bans made since the start, and written back with
// that cause and the admin's reason.
dir := t.TempDir()
edit(t, dir, bansJSON, adminsBansJSON("203.0.113.0/24"))
params := newParams(dir)
lines := logInto(&params)
files := load(t, params)
wantEqual(t, bansJSON, params.Ledger.Snapshot(), adminsBans("203.0.113.0/24"))
wantMadeByAnAdmin(t, params.Ledger, 0)
err := files.WriteAll()
if err != nil {
t.Fatalf("write: %v", err)
}
var written struct {
Bans []struct {
Cause string `json:"cause"`
Reason string `json:"reason"`
} `json:"bans"`
}
err = json.Unmarshal([]byte(readFile(t, filepath.Join(dir, bansJSON))), &written)
if err != nil || len(written.Bans) != 1 || written.Bans[0].Cause != bans.CauseAdmin ||
written.Bans[0].Reason != reason {
t.Errorf("bans.json holds %+v (%v), want the ban with the cause admin "+
"and the reason %q", written, err, reason)
}
// Taken in while smallwebwaf runs, a ban on another netblock is an
// admin's too, and one made since the start.
watch(t, files, lines)
edit(t, dir, bansJSON, adminsBansJSON("198.51.100.0/24"))
wantTakenIn(t, lines, dir, bansJSON)
wantEqual(t, bansJSON, params.Ledger.Snapshot(), adminsBans("198.51.100.0/24"))
wantMadeByAnAdmin(t, params.Ledger, 1)
}
func TestLiftedBanReadAtTheStart(t *testing.T) {
t.Parallel()
dir := t.TempDir()
edit(t, dir, bansJSON, liftedBansJSON)
params := newParams(dir)
wantLiftedBanKept(t, load(t, params), dir, params.Ledger)
}
func TestBanLiftedByAnEditWhileRunning(t *testing.T) {
t.Parallel()
dir := t.TempDir()
params := newParams(dir)
lines := logInto(&params)
files := load(t, params)
watch(t, files, lines)
// The ban that liftedBansJSON lifts, before it is lifted.
netblock := netip.MustParsePrefix(liftedClient + "/32")
params.Ledger.BanForLimit(netblock, midnight(), bans.Notes{})
_, banned := params.Ledger.Find(netblock.Addr(), afterLifting())
if !banned {
t.Fatal("the ban does not refuse before it is lifted")
}
edit(t, dir, bansJSON, liftedBansJSON)
wantTakenIn(t, lines, dir, bansJSON)
wantLiftedBanKept(t, files, dir, params.Ledger)
}
func TestBrokenEditSetAsideAtTheNextWrite(t *testing.T) { func TestBrokenEditSetAsideAtTheNextWrite(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1003,7 +881,6 @@ func newParams(dir string) state.Params {
LimitBanDuration: time.Hour, LimitBanDuration: time.Hour,
LimitBanRepeatWindow: 24 * time.Hour, LimitBanRepeatWindow: 24 * time.Hour,
MaxBanDuration: 7 * 24 * time.Hour, MaxBanDuration: 7 * 24 * time.Hour,
AttackBanDuration: 7 * 24 * time.Hour,
MaxBans: 5000, MaxBans: 5000,
}), }),
Limiter: ratelimit.New(ratelimit.Limits{}), Limiter: ratelimit.New(ratelimit.Limits{}),
@@ -1016,17 +893,14 @@ func newParams(dir string) state.Params {
} }
} }
// fill puts a permanent ban an admin made, a ban for a broken limit and // fill puts a ban that ends and one that does not, clients with counts
// one for a clear sign of attack, clients with counts and histories, and // and histories, and GeoJS answers into the parts of params.
// GeoJS answers into the parts of params.
func fill(params state.Params) { func fill(params state.Params) {
now := midnight() now := midnight()
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
params.Ledger.Load([]bans.Ban{permanentBan()}) params.Ledger.Load([]bans.Ban{permanentBan()})
params.Ledger.BanForLimit(client, now, bans.Notes{Country: "DE", Limit: 1}) params.Ledger.BanForLimit(client, now, bans.Notes{Country: "DE", Limit: 1})
params.Ledger.BanForAttack(netip.MustParsePrefix("192.0.2.1/32"), now,
bans.Notes{RuleID: "env-file", Target: "path"})
for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} { for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} {
params.Limiter.Count(netip.MustParsePrefix(c), now) params.Limiter.Count(netip.MustParsePrefix(c), now)
@@ -1050,8 +924,6 @@ func permanentBan() bans.Ban {
return bans.Ban{ return bans.Ban{
Netblock: netip.MustParsePrefix("2001:db8::/64"), Netblock: netip.MustParsePrefix("2001:db8::/64"),
Start: midnight(), Start: midnight(),
Cause: bans.CauseAdmin,
Reason: "scrapes every commit",
Notes: bans.Notes{ Notes: bans.Notes{
Country: "DE", Country: "DE",
Limit: 1000, Limit: 1000,
@@ -1067,64 +939,11 @@ func permanentBan() bans.Ban {
}, },
Requests: 1500, Requests: 1500,
Refused: 3, Refused: 3,
EarlierBans: bans.EarlierBans{Limit: 3, Attack: 1, Admin: 1}, EarlierBans: 5,
}, },
} }
} }
// afterLifting is a time after the ban liftedBansJSON holds was lifted,
// while it would still last.
func afterLifting() time.Time {
return midnight().Add(30 * time.Minute)
}
// wantLiftedBanKept checks that ledger holds the ban liftedBansJSON holds,
// which refuses nothing and does not make the next ban for a broken limit
// longer, and that files write it to bans.json, in dir, still lifted.
func wantLiftedBanKept(
t *testing.T, files *state.Files, dir string, ledger *bans.Ledger,
) {
t.Helper()
err := files.WriteAll()
if err != nil {
t.Fatalf("write: %v", err)
}
const lifted = `"lifted": "2026-10-06T00:10:00Z"`
if got := readFile(t, filepath.Join(dir, bansJSON)); !strings.Contains(got, lifted) {
t.Errorf("bans.json holds\n%s\nwant the ban with %s", got, lifted)
}
netblock := netip.MustParsePrefix(liftedClient + "/32")
_, banned := ledger.Check(netblock.Addr(), afterLifting())
if banned {
t.Error("the lifted ban refuses")
}
// Were the lifted ban counted, the next would last three hours.
ban := ledger.BanForLimit(netblock, afterLifting(), bans.Notes{})
if ban.Expires.Sub(ban.Start) != time.Hour {
t.Errorf("the next ban lasts %s, want 1h", ban.Expires.Sub(ban.Start))
}
held := ledger.Bans(netblock)
if len(held) != 2 || !held[0].Lifted.Equal(midnight().Add(10*time.Minute)) {
t.Errorf("the ledger holds %+v, want the lifted ban and the new one", held)
}
}
// wantMadeByAnAdmin checks how many bans ledger counts as made by an
// admin since the start.
func wantMadeByAnAdmin(t *testing.T, ledger *bans.Ledger, want int) {
t.Helper()
if got := ledger.Made(bans.CauseAdmin); got != want {
t.Errorf("%d bans made by an admin, want %d", got, want)
}
}
// load reads the state files into the parts of params. // load reads the state files into the parts of params.
func load(t *testing.T, params state.Params) *state.Files { func load(t *testing.T, params state.Params) *state.Files {
t.Helper() t.Helper()
+5 -27
View File
@@ -3,10 +3,9 @@
# deploy/example-app, then run the app's container with a volume for the # deploy/example-app, then run the app's container with a volume for the
# state files and check that the health check passes, that a request is # state files and check that the health check passes, that a request is
# served through smallwebwaf, that a second one in a minute bans the # served through smallwebwaf, that a second one in a minute bans the
# client, that a probe for /.env bans another client, which its next # client, that `sv stop` stops smallwebwaf in order, that `docker stop`
# request bans for good, that `sv stop` stops smallwebwaf in order, that # stops the container without having to kill it, and that a new
# `docker stop` stops the container without having to kill it, and that # container on the same volume still refuses the banned client. The
# a new container on the same volume still refuses the banned client. The
# containers, the volume and both images are removed however the script # containers, the volume and both images are removed however the script
# ends. Building the app needs network access, for nixpkgs' binary cache. # ends. Building the app needs network access, for nixpkgs' binary cache.
# script/check does not run this. # script/check does not run this.
@@ -53,13 +52,9 @@ healthy() {
[ "$status" = healthy ] [ "$status" = healthy ]
} }
# logged <text>...: a line of the container's output holds every text, # logged <text>: the container's output holds text.
# in any order.
logged() { logged() {
lines="$(docker logs "$CONTAINER" 2>&1)" docker logs "$CONTAINER" 2>&1 | grep -qF "$1"
for text in "$@"; do
lines="$(printf '%s\n' "$lines" | grep -F "$text")" || return 1
done
} }
# start_container: run the app's container, with the state files on the # start_container: run the app's container, with the state files on the
@@ -82,15 +77,6 @@ refused() {
[ "$code" = 403 ] [ "$code" = 403 ]
} }
# refused_from <client> <path>: a request for path from client, as
# X-Forwarded-For names it, gets 403. smallwebwaf believes the header
# from docker's gateway, a private address.
refused_from() {
code="$(curl --silent --output /dev/null --write-out '%{http_code}' \
--max-time 10 --header "X-Forwarded-For: $1" "http://$address$2")" || true
[ "$code" = 403 ]
}
main() { main() {
cd "$ROOT" cd "$ROOT"
trap cleanup EXIT trap cleanup EXIT
@@ -114,14 +100,6 @@ main() {
wait_for "smallwebwaf logged no ban" logged '"action":"rate_limited"' wait_for "smallwebwaf logged no ban" logged '"action":"rate_limited"'
echo "example-app: a second request in a minute bans the client" echo "example-app: a second request in a minute bans the client"
refused_from 203.0.113.9 /.env || fail "a probe for /.env was not refused"
wait_for "smallwebwaf logged no ban for the probe" \
logged '"action":"banned"' '"rule_ids":["env-file"]'
refused_from 203.0.113.9 / || fail "the client of the probe was let through"
wait_for "the client's next request did not make its ban permanent" \
logged '"ban_expires":"permanent"'
echo "example-app: a probe for /.env bans the client, its next request for good"
docker exec "$CONTAINER" sv stop smallwebwaf >/dev/null || docker exec "$CONTAINER" sv stop smallwebwaf >/dev/null ||
fail "sv stop smallwebwaf failed" fail "sv stop smallwebwaf failed"
wait_for "smallwebwaf did not stop in order" logged '"msg":"stopped"' wait_for "smallwebwaf did not stop in order" logged '"msg":"stopped"'
+1 -7
View File
@@ -1,9 +1,7 @@
#!/bin/sh #!/bin/sh
# script/run: build bin/smallwebwaf with script/build and run it, with # script/run: build bin/smallwebwaf with script/build and run it, with
# the settings in the environment. Unless SWWAF_STATE_DIR is set, the # the settings in the environment. Unless SWWAF_STATE_DIR is set, the
# state files go in bin/state, beside the binary, and unless # state files go in bin/state, beside the binary.
# SWWAF_RULES_DIR is set, the rule files are those of share/rules.d,
# which the image ships.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -16,10 +14,6 @@ main() {
export SWWAF_STATE_DIR export SWWAF_STATE_DIR
mkdir -p "$SWWAF_STATE_DIR" mkdir -p "$SWWAF_STATE_DIR"
fi fi
if [ -z "${SWWAF_RULES_DIR+set}" ]; then
SWWAF_RULES_DIR="$ROOT/share/rules.d"
export SWWAF_RULES_DIR
fi
exec "$ROOT/bin/smallwebwaf" exec "$ROOT/bin/smallwebwaf"
} }
-15
View File
@@ -1,15 +0,0 @@
# 00-default.rules: probes no real visitor sends, anchored at the site root
# id target action regex
env-file path ban (?i)^/\.env(\.[a-z]+)?$
vcs-dir path ban (?i)^/\.(git|svn|hg|bzr)(/|$)
secrets-dir path ban (?i)^/\.(aws|ssh|docker|kube)/
secret-file path ban (?i)^/\.(htpasswd|htaccess|npmrc|netrc|pgpass|git-credentials|bash_history|DS_Store)$
editor-dir path ban (?i)^/\.(vscode|idea)/
backup-file path ban (?i)^/[^/]+\.(php(\.[a-z0-9]+|~)|sql(\.[a-z0-9]+)?)$
log-file path ban (?i)^/(debug|error|access)\.log$
compose-file path ban (?i)^/(docker-)?compose\.ya?ml$
php-shell path ban (?i)^/(shell|c99|r57|wso|alfa)\.php$
scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b
path-traversal uri block (\.\./){2,}
empty-agent user_agent log ^$