Compare commits
1
Commits
next
..
fc469d96a9
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fc469d96a9 |
@@ -167,10 +167,6 @@ RUN groupadd --system --gid 65532 smallwebwaf \
|
|||||||
# smallwebwaf user at each start.
|
# smallwebwaf user at each start.
|
||||||
RUN mkdir /var/lib/smallwebwaf
|
RUN mkdir /var/lib/smallwebwaf
|
||||||
|
|
||||||
# The default rule file, in SWWAF_RULES_DIR by default, where an app's
|
|
||||||
# Dockerfile can copy rule files of its own beside it.
|
|
||||||
COPY share/rules.d/00-default.rules /etc/smallwebwaf/rules.d/00-default.rules
|
|
||||||
|
|
||||||
# runsvinit starts runit's runsvdir on /etc/service, where Ubuntu's sv
|
# runsvinit starts runit's runsvdir on /etc/service, where Ubuntu's sv
|
||||||
# looks too.
|
# looks too.
|
||||||
COPY --chmod=755 share/smallwebwaf.run /etc/service/smallwebwaf/run
|
COPY --chmod=755 share/smallwebwaf.run /etc/service/smallwebwaf/run
|
||||||
|
|||||||
@@ -13,30 +13,24 @@ JSON log line for every request.
|
|||||||
|
|
||||||
Status: the first two milestones are built
|
Status: the first two milestones are built
|
||||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
|
(https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
|
||||||
https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are nine parts of
|
https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are seven parts of
|
||||||
milestone 3: the static lists, the bans that broken rate limits lead to, the ban
|
milestone 3: the static lists, the bans that broken rate limits lead to, the
|
||||||
ledger with the bans you make, keep and lift, the JSON state files with your
|
JSON state files with your edits taken in while it runs and the paths the rate
|
||||||
edits taken in while it runs and the paths the rate limits do not count, which
|
limits do not count, which come next in the build order, `observe` mode, which
|
||||||
come next in the build order, `observe` mode and the rest of the request log's
|
comes a little later, and the metrics endpoint and the header size and the idle
|
||||||
fields, which come a little later, and the metrics endpoint and the header size
|
time as settings, which come last in it. `smallwebwaf` passes each request to
|
||||||
and the idle time as settings, which come last in it. So are two parts of the
|
the app and the app's answer back, unchanged, within its timeouts and size
|
||||||
stage after it: the rule files, the first part, with the bans for a clear sign
|
limits, works out each client's address, bans a client that sends too many
|
||||||
of attack, and remote log sending. `smallwebwaf` passes each request to the app
|
requests, not counting those for the paths you choose, refuses a client that
|
||||||
and the app's answer back, unchanged, within its timeouts and size limits, works
|
comes from a country you refuse or from a network you refuse, lets the networks
|
||||||
out each client's address, bans a client that sends too many requests, not
|
you choose through, keeps its bans, each client's counters and history, and
|
||||||
counting those for the paths you choose, refuses a client that comes from a
|
GeoJS's answers in JSON files across restarts, takes in your edits of those
|
||||||
country you refuse or from a network you refuse, lets the networks you choose
|
files while it runs, writes a JSON log line for every request, serves Prometheus
|
||||||
through, checks each request against the rule files and bans a client whose
|
metrics to a scraper that holds the metrics token, and in `observe` mode passes
|
||||||
request is a clear sign of attack, keeps its bans, each client's counters and
|
on the requests it would refuse, logging what it would have done with them. It
|
||||||
history, and GeoJS's answers in JSON files across restarts, takes in your edits
|
comes as the image the app's own image is built on. The rest of the design comes
|
||||||
of those files, such as a ban you make, keep or lift, and of the rule files
|
after that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey
|
||||||
while it runs, writes a JSON log line for every request, sends its log lines to
|
of existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
|
||||||
a syslog server too if you name one, serves Prometheus metrics to a scraper that
|
|
||||||
holds the metrics token, and in `observe` mode passes on the requests it would
|
|
||||||
refuse, logging what it would have done with them. It comes as the image the
|
|
||||||
app's own image is built on. The rest of the design comes after that, in the
|
|
||||||
order of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools
|
|
||||||
that led to the design is in [`EVALUATION.md`](EVALUATION.md).
|
|
||||||
|
|
||||||
## Getting started
|
## Getting started
|
||||||
|
|
||||||
@@ -59,8 +53,7 @@ works.
|
|||||||
To work on the code, `make build` builds the binary alone, with Go installed,
|
To work on the code, `make build` builds the binary alone, with Go installed,
|
||||||
and `make run` builds and runs it, listening on port 8080 in front of an app at
|
and `make run` builds and runs it, listening on port 8080 in front of an app at
|
||||||
`SWWAF_UPSTREAM_URL`, by default `http://127.0.0.1:8081`, with its state files
|
`SWWAF_UPSTREAM_URL`, by default `http://127.0.0.1:8081`, with its state files
|
||||||
in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
in `bin/state` unless `SWWAF_STATE_DIR` is set.
|
||||||
`share/rules.d` unless `SWWAF_RULES_DIR` is set.
|
|
||||||
|
|
||||||
## What it does so far
|
## What it does so far
|
||||||
|
|
||||||
@@ -74,9 +67,7 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
|||||||
address outside `SWWAF_TRUSTED_PROXIES` is the client; if every address in it
|
address outside `SWWAF_TRUSTED_PROXIES` is the client; if every address in it
|
||||||
is inside, the leftmost is, and with no header the peer is. The app sees what
|
is inside, the leftmost is, and with no header the peer is. The app sees what
|
||||||
it would see from traefik directly: the same `Host`, the same
|
it would see from traefik directly: the same `Host`, the same
|
||||||
`X-Forwarded-Proto`, and `X-Forwarded-For` with the peer added at the end. It
|
`X-Forwarded-Proto`, and `X-Forwarded-For` with the peer added at the end.
|
||||||
also gets the request's id in `X-Request-ID`, the same id as in the request's
|
|
||||||
log line (see `request_id` in "Request log" below).
|
|
||||||
- Enforces the timeouts and the size limits below. A limit passed before the
|
- Enforces the timeouts and the size limits below. A limit passed before the
|
||||||
response has started gets `smallwebwaf`'s own answer: `408` for a client too
|
response has started gets `smallwebwaf`'s own answer: `408` for a client too
|
||||||
slow to send its request, `413` for a request body that is too large, `504`
|
slow to send its request, `413` for a request body that is too large, `504`
|
||||||
@@ -92,13 +83,13 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
|||||||
`SWWAF_BAN_RESPONSE`, `403` by default, before anything reaches the app, and
|
`SWWAF_BAN_RESPONSE`, `403` by default, before anything reaches the app, and
|
||||||
bans the client. A request whose path starts with one of
|
bans the client. A request whose path starts with one of
|
||||||
`SWWAF_RATE_LIMIT_EXEMPT_PATHS`, as that setting below describes, is neither
|
`SWWAF_RATE_LIMIT_EXEMPT_PATHS`, as that setting below describes, is neither
|
||||||
counted nor refused by the rate limits; the static lists, bans, the country
|
counted nor refused by the rate limits; the static lists, bans and the country
|
||||||
lists and the rule files still apply to it. A client is one IPv4 address, or
|
lists still apply to it. A client is one IPv4 address, or one IPv6 /64, since
|
||||||
one IPv6 /64, since one abuser usually holds a whole /64. Each window is
|
one abuser usually holds a whole /64. Each window is counted in two fixed
|
||||||
counted in two fixed buckets, the earlier one weighted by how much of it the
|
buckets, the earlier one weighted by how much of it the window still covers.
|
||||||
window still covers. At most 20,000 clients are kept, the least recently seen
|
At most 20,000 clients are kept, the least recently seen dropped first, with
|
||||||
dropped first, with their history, and a restart gives no client a fresh
|
their history, and a restart gives no client a fresh allowance (see "State
|
||||||
allowance (see "State files" below).
|
files" below).
|
||||||
- Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md)
|
- Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md)
|
||||||
describes: the first ban lasts an hour, and a limit broken again within a day
|
describes: the first ban lasts an hour, and a limit broken again within a day
|
||||||
of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and
|
of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and
|
||||||
@@ -112,28 +103,11 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
|||||||
window and the requests counted in it, the request that broke it, the client's
|
window and the requests counted in it, the request that broke it, the client's
|
||||||
country when it was looked up, the netblock's requests since it was first
|
country when it was looked up, the netblock's requests since it was first
|
||||||
seen, how many of them the ban has refused, and how many bans the netblock had
|
seen, how many of them the ban has refused, and how many bans the netblock had
|
||||||
before, for a broken limit, for a clear sign of attack and by an admin. At
|
before. At most `SWWAF_MAX_BANS` bans are kept, past, active and permanent;
|
||||||
most `SWWAF_MAX_BANS` bans `smallwebwaf` made are kept, past, active and
|
past that, the earliest ban of the netblock that has gone longest without a
|
||||||
permanent; past that, the earliest such ban of the netblock that has gone
|
request is dropped first. `bans.json` shows the bans and their notes, a
|
||||||
longest without a request is dropped first. The bans whose cause is `admin`,
|
restart lifts none, and you add or lift a ban by editing it (see "State files"
|
||||||
those you make or keep, are kept besides, and never dropped. `bans.json` shows
|
below).
|
||||||
the bans and their notes, a restart lifts none, and you make, keep or lift a
|
|
||||||
ban by editing it (see "State files" below).
|
|
||||||
- Checks each request against the rules of the rule files (see "Rule files"
|
|
||||||
below) after the rate limits, and before its body is read. A `log` rule that
|
|
||||||
matches is noted in the log line; a `block` rule refuses the request with
|
|
||||||
`403`, and bans no one; a `ban` rule refuses it with `SWWAF_BAN_RESPONSE` and
|
|
||||||
bans the client's netblock for a clear sign of attack. Matching stops at the
|
|
||||||
first rule that refuses. A client in `SWWAF_ALLOW_NETS` is not checked.
|
|
||||||
- Bans a client for a clear sign of attack, as "Bans" in [`SPEC.md`](SPEC.md)
|
|
||||||
describes: the first such ban lasts `SWWAF_ATTACK_BAN_DURATION`, seven days by
|
|
||||||
default, and any request from the netblock while it lasts makes it permanent.
|
|
||||||
Once it has run out, the netblock is served like any other, but its next clear
|
|
||||||
sign of attack bans it permanently at once. Such a ban covers the same
|
|
||||||
netblock as a ban for a broken rate limit, does not set the client's counters
|
|
||||||
back to zero, and does not make the netblock's next ban for a broken limit
|
|
||||||
longer. Its notes give the id and the target of the rule that matched in place
|
|
||||||
of the limit.
|
|
||||||
- Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
|
- Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
|
||||||
as the client's country is known and before its body is read; such a request
|
as the client's country is known and before its body is read; such a request
|
||||||
is not counted for the rate limits. While one of the country lists below is
|
is not counted for the rate limits. While one of the country lists below is
|
||||||
@@ -144,26 +118,25 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
|||||||
in `SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it.
|
in `SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it.
|
||||||
- Checks the client's own address against the static lists, the three netblock
|
- Checks the client's own address against the static lists, the three netblock
|
||||||
settings below, before anything else, its country included. A client in
|
settings below, before anything else, its country included. A client in
|
||||||
`SWWAF_ALLOW_NETS` skips bans, the country lists, the rate limits and the rule
|
`SWWAF_ALLOW_NETS` skips bans, the country lists and the rate limits, and is
|
||||||
files, and is not looked up; the timeouts and size limits still apply. A
|
not looked up; the timeouts and size limits still apply. A client in
|
||||||
client in `SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE` before its
|
`SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE` before its body is
|
||||||
body is read, and the request is not counted for the rate limits; an address
|
read, and the request is not counted for the rate limits; an address in
|
||||||
in `SWWAF_ALLOW_NETS` too is let through. A client in
|
`SWWAF_ALLOW_NETS` too is let through. A client in
|
||||||
`SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
|
`SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
|
||||||
limits; the country lists, the rule files and bans still apply to it.
|
limits; the country lists and bans still apply to it.
|
||||||
- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
|
- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
|
||||||
that `SWWAF_DENY_NETS`, a ban, the country lists, a rate limit or a rule would
|
that `SWWAF_DENY_NETS`, a ban, the country lists or a rate limit would refuse:
|
||||||
refuse: it passes them to the app, and their log lines name what `enforce`
|
it passes them to the app, and their log lines name what `enforce` mode would
|
||||||
mode would have done (see `would_action` in "Request log" below). The checks
|
have done (see `would_action` in "Request log" below). The checks run, and
|
||||||
run, and requests are counted, as in `enforce` mode, with three differences:
|
requests are counted, as in `enforce` mode, but a broken rate limit makes no
|
||||||
neither a broken rate limit nor a `ban` rule makes a ban; a broken rate limit
|
ban and does not set the client's counters back to zero, so each request over
|
||||||
does not set the client's counters back to zero, so each request over the
|
the limit is logged as one that would be refused. The bans in `bans.json` are
|
||||||
limit is logged as one that would be refused; and a request under a ban does
|
kept, and refuse requests again when `smallwebwaf` next runs in `enforce`
|
||||||
not make it permanent. The bans in `bans.json` are kept, and refuse requests
|
mode, as long as they last. The timeouts and size limits still apply, since
|
||||||
again when `smallwebwaf` next runs in `enforce` mode, as long as they last.
|
they protect `smallwebwaf` and the app themselves, and a request for the
|
||||||
The timeouts and size limits still apply, since they protect `smallwebwaf` and
|
metrics without the token is still answered `401`. It is for trying a
|
||||||
the app themselves, and a request for the metrics without the token is still
|
configuration before enforcing it.
|
||||||
answered `401`. It is for trying a configuration before enforcing it.
|
|
||||||
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
|
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
|
||||||
check and without asking the app, for the image's health check.
|
check and without asking the app, for the image's health check.
|
||||||
- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
|
- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
|
||||||
@@ -175,25 +148,16 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
|||||||
passed to the app: a banned client stays refused, and each counts toward the
|
passed to the app: a banned client stays refused, and each counts toward the
|
||||||
client's rate limits. None of them reaches the app.
|
client's rate limits. None of them reaches the app.
|
||||||
- Writes a line in the request log for each request (see "Request log" below).
|
- Writes a line in the request log for each request (see "Request log" below).
|
||||||
- Sends every line it writes on stdout to a syslog server as well, while
|
|
||||||
`SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
|
|
||||||
below).
|
|
||||||
|
|
||||||
## Settings
|
## Settings
|
||||||
|
|
||||||
Each setting is an environment variable, or a file one names (see "Settings
|
Each setting is an environment variable, and each has a default, so none has to
|
||||||
given as files" below), and each has a default, so none has to be set. A setting
|
be set. A setting that is set but invalid stops the start with a message naming
|
||||||
that is set but invalid stops the start with a message naming it, and the
|
it, and the effective settings are logged at start.
|
||||||
effective settings are logged at start.
|
|
||||||
|
|
||||||
- `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens.
|
- `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens.
|
||||||
- `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app, as `http` or
|
- `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app, as `http` or
|
||||||
`https`, a host and an optional port, and nothing more.
|
`https`, a host and an optional port, and nothing more.
|
||||||
- `SWWAF_INSTANCE_NAME` (default: the host's name, which docker sets to the
|
|
||||||
first 12 characters of the container's id unless the deployment names one):
|
|
||||||
the name each request log line gives as `instance`. Set it, for example to
|
|
||||||
`fsn1app1/gitea`, for a name that stays the same when a deploy replaces the
|
|
||||||
container, and that tells instances apart when several log to one place.
|
|
||||||
- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to pass on the
|
- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to pass on the
|
||||||
requests `smallwebwaf` would refuse and log what it would have done (see "What
|
requests `smallwebwaf` would refuse and log what it would have done (see "What
|
||||||
it does so far" above).
|
it does so far" above).
|
||||||
@@ -222,8 +186,8 @@ effective settings are logged at start.
|
|||||||
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
|
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
|
||||||
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
|
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
|
||||||
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
|
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
|
||||||
country lists, the rate limits and the rule files, such as your monitoring or
|
country lists and the rate limits, such as your monitoring or your own
|
||||||
your own networks.
|
networks.
|
||||||
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
|
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
|
||||||
rate limits do not apply to, such as a machine that talks to the app all day.
|
rate limits do not apply to, such as a machine that talks to the app all day.
|
||||||
- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
|
- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
|
||||||
@@ -238,13 +202,11 @@ effective settings are logged at start.
|
|||||||
`..` anywhere or a backslash, or whose path as sent holds an encoded slash
|
`..` anywhere or a backslash, or whose path as sent holds an encoded slash
|
||||||
(`%2F` or `%2f`), is never exempt, since the app may act on it as a path
|
(`%2F` or `%2f`), is never exempt, since the app may act on it as a path
|
||||||
outside every prefix: `/assets/..%2Flogin` as `/login`. Any other request is
|
outside every prefix: `/assets/..%2Flogin` as `/login`. Any other request is
|
||||||
exempt when its path as sent, the path the app receives, before any query
|
exempt when its path, percent-decoded and before any query string, starts with
|
||||||
string and not percent-decoded, starts with a prefix, character for character.
|
a prefix, character for character. `/assets/` matches `/assets/app.js` and
|
||||||
`/assets/` matches `/assets/app.js` and `/assets/`, but not `/assets`,
|
`/assets/`, but not `/assets`, `/Assets/app.js`, `/static/assets/app.js`,
|
||||||
`/Assets/app.js`, `/%61ssets/app.js`, `/static/assets/app.js`,
|
`/static/../assets/app.js` or `/assets%2Fapp.js`. A prefix is written without
|
||||||
`/static/../assets/app.js` or `/assets%2Fapp.js`. A character the client sends
|
percent-encoding, and there are no wildcards: `*` is a character like any
|
||||||
percent-encoded, such as a space, is written percent-encoded in a prefix, as
|
|
||||||
in `/my%20files/`, and there are no wildcards: `*` is a character like any
|
|
||||||
other.
|
other.
|
||||||
- `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused,
|
- `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused,
|
||||||
for example `cn,ru,kp`.
|
for example `cn,ru,kp`.
|
||||||
@@ -253,22 +215,18 @@ effective settings are logged at start.
|
|||||||
country cannot be found is refused too, so that new clients are not let in
|
country cannot be found is refused too, so that new clients are not let in
|
||||||
whenever GeoJS stops answering.
|
whenever GeoJS stops answering.
|
||||||
- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one
|
- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one
|
||||||
that is banned, breaks a rate limit, matches a `ban` rule, is in
|
that is banned, breaks a rate limit, is in `SWWAF_DENY_NETS` or comes from a
|
||||||
`SWWAF_DENY_NETS` or comes from a refused country: `403`, `429`, or `close` to
|
refused country: `403`, `429`, or `close` to close the connection without an
|
||||||
close the connection without an answer. Behind traefik, `close` does not leave
|
answer. Behind traefik, `close` does not leave the client unanswered: traefik
|
||||||
the client unanswered: traefik answers `502`, as it does whenever its backend
|
answers `502`, as it does whenever its backend drops a connection.
|
||||||
drops a connection. A `block` rule always answers `403`.
|
|
||||||
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
|
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
|
||||||
limit.
|
limit.
|
||||||
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit broken again
|
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit broken again
|
||||||
within this time after a ban ended, other than one for a clear sign of attack,
|
within this time after a ban ended bans for three times as long as that ban.
|
||||||
bans for three times as long as that ban.
|
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban that would be longer is
|
||||||
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken rate limit that
|
permanent instead.
|
||||||
would be longer is permanent instead.
|
- `SWWAF_MAX_BANS` (default `5000`): the most bans kept, past, active and
|
||||||
- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
|
permanent.
|
||||||
attack.
|
|
||||||
- `SWWAF_MAX_BANS` (default `5000`): the most bans `smallwebwaf` made that are
|
|
||||||
kept, past, active and permanent. The bans you make or keep are kept besides.
|
|
||||||
- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around
|
- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around
|
||||||
an IPv4 client that a ban covers, such as `24` to ban the surrounding /24. An
|
an IPv4 client that a ban covers, such as `24` to ban the surrounding /24. An
|
||||||
IPv6 ban covers the client's /64.
|
IPv6 ban covers the client's /64.
|
||||||
@@ -279,41 +237,12 @@ effective settings are logged at start.
|
|||||||
`bans.json` is written, with every ban made in between.
|
`bans.json` is written, with every ban made in between.
|
||||||
- `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is
|
- `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is
|
||||||
written.
|
written.
|
||||||
- `SWWAF_LOG_REQUEST_HEADERS` (default
|
|
||||||
`accept,accept-language,accept-encoding,content-type,origin,range`): the
|
|
||||||
request headers whose values the request log gives, in either case.
|
|
||||||
`Authorization`, `Cookie` and `Set-Cookie` are never logged, even when listed
|
|
||||||
(see "Request log" below). An entry naming `Host` or `Transfer-Encoding` stops
|
|
||||||
the start, since Go's HTTP server takes both out of the request; the request's
|
|
||||||
host is the field `host`.
|
|
||||||
- `SWWAF_METRICS_TOKEN` (default unset): the token a scraper sends for the
|
- `SWWAF_METRICS_TOKEN` (default unset): the token a scraper sends for the
|
||||||
metrics, a long random value. While it is unset the metrics are off; one
|
metrics, a long random value. While it is unset the metrics are off; one
|
||||||
shorter than 32 characters stops the start. The settings logged at start show
|
shorter than 32 characters stops the start. The settings logged at start show
|
||||||
`********` in its place. Given as a file, it can be kept out of the app's
|
`********` in its place.
|
||||||
reach (see "Settings given as files" below).
|
|
||||||
- `SWWAF_METRICS_TOP_N` (default `50`): how many countries get series of their
|
- `SWWAF_METRICS_TOP_N` (default `50`): how many countries get series of their
|
||||||
own in the metrics by country; the others are counted as `other`.
|
own in the metrics by country; the others are counted as `other`.
|
||||||
- `SWWAF_RULES_DIR` (default `/etc/smallwebwaf/rules.d`): the directory of the
|
|
||||||
rule files. A directory that does not exist stops the start.
|
|
||||||
- `SWWAF_RULES_ENABLED` (default `true`): `false` reads no rule file, and checks
|
|
||||||
no request against one.
|
|
||||||
- `SWWAF_LOG_REMOTE_URL` (default unset): a syslog server that every line on
|
|
||||||
stdout is also sent to, as `syslog+udp://`, `syslog+tcp://` or `syslog+tls://`
|
|
||||||
with a host and a port, such as `syslog+tls://logs.example:6514`. Unset or
|
|
||||||
empty, nothing is sent.
|
|
||||||
- `SWWAF_LOG_REMOTE_TLS_CA_FILE` (default unset): a file of PEM certificates,
|
|
||||||
which the certificate of a `syslog+tls` server must chain to instead of the
|
|
||||||
host's own. A file that cannot be read or holds no certificate stops the
|
|
||||||
start.
|
|
||||||
- `SWWAF_LOG_REMOTE_BUFFER` (default `10000`): the most lines held while they
|
|
||||||
wait to be sent.
|
|
||||||
- `SWWAF_LOG_REMOTE_FACILITY` (default `local0`): the syslog facility the lines
|
|
||||||
are sent with: `kern`, `user`, `mail`, `daemon`, `auth`, `syslog`, `lpr`,
|
|
||||||
`news`, `uucp`, `cron`, `authpriv`, `ftp`, or `local0` to `local7`.
|
|
||||||
- `SWWAF_LOG_REMOTE_APP_NAME` (default `SWWAF_INSTANCE_NAME`): the app name the
|
|
||||||
lines are sent with, 1 to 48 printable ASCII characters without a space. While
|
|
||||||
`SWWAF_LOG_REMOTE_URL` is set, an `SWWAF_INSTANCE_NAME` that is not such a
|
|
||||||
name stops the start too, unless this setting gives one that is.
|
|
||||||
|
|
||||||
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
|
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
|
||||||
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
|
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
|
||||||
@@ -323,116 +252,47 @@ ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
|
|||||||
the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
|
the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
|
||||||
`su`, stops the start, and so does a code on both country lists. `off` switches
|
`su`, stops the start, and so does a code on both country lists. `off` switches
|
||||||
a timeout, a size limit or a rate limit off;
|
a timeout, a size limit or a rate limit off;
|
||||||
`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings, the state settings,
|
`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings, the state settings
|
||||||
`SWWAF_METRICS_TOP_N` and `SWWAF_LOG_REMOTE_BUFFER` cannot be off.
|
and `SWWAF_METRICS_TOP_N` cannot be off.
|
||||||
|
|
||||||
Several limits are fixed rather than settings. At most 20,000 clients are kept,
|
Several limits are fixed rather than settings. At most 20,000 clients are kept,
|
||||||
with their counters and history, and an IPv6 client is counted by its /64. A new
|
with their counters and history, and an IPv6 client is counted by its /64. A new
|
||||||
client waits at most a second for its country, and at most 100,000 answers from
|
client waits at most a second for its country, and at most 100,000 answers from
|
||||||
GeoJS are kept, for 7 days each.
|
GeoJS are kept, for 7 days each.
|
||||||
|
|
||||||
### Settings given as files
|
|
||||||
|
|
||||||
Any setting may instead be given as a file that holds its value: the variable
|
|
||||||
named like the setting with `_FILE` added, such as `SWWAF_METRICS_TOKEN_FILE`,
|
|
||||||
names the file. `smallwebwaf` reads the file once, at start, and its health
|
|
||||||
check reads only the files of `SWWAF_LISTEN_ADDR` and `SWWAF_UPSTREAM_URL`, each
|
|
||||||
time it runs. The file's contents are the value, less one newline at their end
|
|
||||||
so that a file written with `echo` or an editor works, and are checked as the
|
|
||||||
setting's own value would be. Setting both the setting and its `_FILE` form, or
|
|
||||||
naming a file that cannot be read, stops the start with a message naming the
|
|
||||||
variable. The settings logged at start name the file, and show a token given in
|
|
||||||
one as `********`, as they show one given directly.
|
|
||||||
`SWWAF_LOG_REMOTE_TLS_CA_FILE`, whose value names a file already, has no `_FILE`
|
|
||||||
form.
|
|
||||||
|
|
||||||
The app starts with the same environment variables as `smallwebwaf`, so it can
|
|
||||||
read a token given as one. A token given as a file is out of the app's reach
|
|
||||||
only while the `smallwebwaf` user alone can read the file: make it on the host,
|
|
||||||
owned by uid 65532, the `smallwebwaf` user, with mode `0400`, and mount the
|
|
||||||
directory that holds it into the container read-only; the container sees the
|
|
||||||
same owner and mode. For example, on the host:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
mkdir -p /srv/app/tokens
|
|
||||||
openssl rand -hex 32 > /srv/app/tokens/metrics
|
|
||||||
chown 65532:65532 /srv/app/tokens/metrics
|
|
||||||
chmod 0400 /srv/app/tokens/metrics
|
|
||||||
```
|
|
||||||
|
|
||||||
and for the container, `-v /srv/app/tokens:/etc/smallwebwaf/tokens:ro` and
|
|
||||||
`-e SWWAF_METRICS_TOKEN_FILE=/etc/smallwebwaf/tokens/metrics`.
|
|
||||||
|
|
||||||
## Request log
|
## Request log
|
||||||
|
|
||||||
`smallwebwaf` writes one JSON object per line on stdout for every request,
|
`smallwebwaf` writes one JSON object per line on stdout for every request,
|
||||||
refused ones included:
|
refused ones included:
|
||||||
|
|
||||||
```
|
```
|
||||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104}
|
{"type":"request","time":"2026-10-03T12:00:00.123Z","client_ip":"203.0.113.9","peer_ip":"172.18.0.2","country":"DE","method":"GET","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"upstream_status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","action":"forward","duration_total":3.217,"duration_upstream_total":3.104}
|
||||||
```
|
```
|
||||||
|
|
||||||
A field that does not apply to a request is left out of its line, apart from
|
- `time` is when the request arrived, in UTC. `peer_ip` is the TCP peer,
|
||||||
`type`, the fields from `time` to `user_agent`, `request_id`, `peer_ip`,
|
normally traefik. `path` and `query` are as the client sent them.
|
||||||
`client_group`, `country`, `action` and `duration_total`, which every line has.
|
|
||||||
|
|
||||||
- `time` is when the request arrived, in UTC. `instance` is
|
|
||||||
`SWWAF_INSTANCE_NAME`. `scheme` is the `X-Forwarded-Proto` a trusted proxy
|
|
||||||
sent, and otherwise `http`. `path` and `query` are as the client sent them.
|
|
||||||
- `request_id` is the `X-Request-ID` a trusted proxy sent, or a new random one
|
|
||||||
of 26 letters and digits when it sent none, or when the peer is not a trusted
|
|
||||||
proxy. A request passed to the app takes it there in `X-Request-ID`.
|
|
||||||
- `peer_ip` is the TCP peer, normally traefik. `forwarded_for` is the
|
|
||||||
`X-Forwarded-For` header as received, several lines of it joined with `, `.
|
|
||||||
`client_group` is the client as the rate limits count it: its IPv4 address as
|
|
||||||
a /32, or the /64 of its IPv6 address.
|
|
||||||
- `country` is the client's country as GeoJS places it. It is empty with neither
|
- `country` is the client's country as GeoJS places it. It is empty with neither
|
||||||
country list set, for a client in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, for
|
country list set, for a client in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, for
|
||||||
a client on a private, loopback or link-local address, when GeoJS cannot place
|
a client on a private, loopback or link-local address, when GeoJS cannot place
|
||||||
the client or has not answered in time, and for a request whose client a ban
|
the client or has not answered in time, and for a request whose client a ban
|
||||||
covers, even when the client's country is known.
|
covers, even when the client's country is known.
|
||||||
- `content_type` is the request's `Content-Type`, and `content_length` the
|
|
||||||
length the request announced for its body, which is left out for none or zero.
|
|
||||||
- `request_headers` are the request's headers that `SWWAF_LOG_REQUEST_HEADERS`
|
|
||||||
names, by name in lower case, several lines of one joined with `, `.
|
|
||||||
`Authorization`, `Cookie` and `Set-Cookie` are never among them, whatever the
|
|
||||||
setting says: `has_authorization` and `has_cookie` are there instead, and
|
|
||||||
true, when the request has an `Authorization` or a `Cookie` header.
|
|
||||||
- `websocket` is there, and true, when the app switched the connection to
|
|
||||||
another protocol, as it does for a WebSocket.
|
|
||||||
- `status` is what the client was sent, `0` if nothing was; `upstream_status` is
|
- `status` is what the client was sent, `0` if nothing was; `upstream_status` is
|
||||||
what the app answered, and is left out when the app did not answer.
|
what the app answered, and is left out when the app did not answer.
|
||||||
- `response_content_type`, `cache_control` and `location` are the
|
|
||||||
`Content-Type`, `Cache-Control` and `Location` headers of the answer: the
|
|
||||||
app's, as passed on, or those of `smallwebwaf`'s own answer.
|
|
||||||
- `request_bytes` and `response_bytes` count body bytes.
|
- `request_bytes` and `response_bytes` count body bytes.
|
||||||
- `action` is `forward` for a request passed to the app, `denied` for one
|
- `action` is `forward` for a request passed to the app, `denied` for one
|
||||||
refused because its client is in `SWWAF_DENY_NETS`, `banned` for one refused
|
refused because its client is in `SWWAF_DENY_NETS`, `banned` for one refused
|
||||||
because a ban covers its client or because it matched a `ban` rule, which bans
|
because a ban covers its client, `country_denied` for one refused for its
|
||||||
its client, `country_denied` for one refused for its client's country,
|
client's country, `rate_limited` for one that broke a rate limit and banned
|
||||||
`rate_limited` for one that broke a rate limit and banned its client,
|
its client, `too_large` for a request or response over its size limit,
|
||||||
`rule_blocked` for one a `block` rule refused, `too_large` for a request or
|
`timed_out` for one that ran out of time, `upstream_error` when the app could
|
||||||
response over its size limit, `timed_out` for one that ran out of time,
|
not be reached or its answer broke off, and `admin` for one `smallwebwaf`
|
||||||
`upstream_error` when the app could not be reached or its answer broke off,
|
answered at its own endpoint.
|
||||||
and `admin` for one `smallwebwaf` answered at its own endpoint.
|
|
||||||
- `would_action` is there in `observe` mode for a request that
|
- `would_action` is there in `observe` mode for a request that
|
||||||
`SWWAF_DENY_NETS`, a ban, the country lists, a rate limit or a rule would have
|
`SWWAF_DENY_NETS`, a ban, the country lists or a rate limit would have refused
|
||||||
refused in `enforce` mode, and names the action that refusal would have had:
|
in `enforce` mode, and names the action that refusal would have had: `denied`,
|
||||||
`denied`, `banned`, `country_denied`, `rate_limited` or `rule_blocked`.
|
`banned`, `country_denied` or `rate_limited`. `action` then names what was
|
||||||
`action` then names what was done: `forward` for a request passed to the app,
|
done: `forward` for a request passed to the app, and another action, such as
|
||||||
and another action, such as `too_large`, for one a size or time limit refused.
|
`too_large`, for one a size or time limit refused.
|
||||||
- `counts` gives the client's requests in the minute, the hour and the day as
|
|
||||||
the rate limits count them, this request included: in each window, those in
|
|
||||||
the bucket under way and a share of those in the bucket before, so a count can
|
|
||||||
have a fraction. For a request that broke a limit, they are the counts that
|
|
||||||
broke it. It is left out for a request the rate limits do not count: the
|
|
||||||
health check, one from a client in `SWWAF_ALLOW_NETS` or
|
|
||||||
`SWWAF_RATE_LIMIT_EXEMPT_NETS`, one for a path that
|
|
||||||
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a ban
|
|
||||||
or the country lists refuse, or would refuse in `observe` mode. The byte
|
|
||||||
totals come with the byte limits.
|
|
||||||
- `rule_ids` is there for a request that matched rules of the rule files, and
|
|
||||||
lists their ids in the order they matched, up to the one that refused it.
|
|
||||||
- `limit_hit` is there for a request that broke a rate limit, and names the
|
- `limit_hit` is there for a request that broke a rate limit, and names the
|
||||||
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
|
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
|
||||||
went over several. `offence` is then `limit`.
|
went over several. `offence` is then `limit`.
|
||||||
@@ -440,19 +300,10 @@ A field that does not apply to a request is left out of its line, apart from
|
|||||||
or in `observe` mode would have been refused under one, and gives when the ban
|
or in `observe` mode would have been refused under one, and gives when the ban
|
||||||
ends, in the same form as `time`, or `permanent`.
|
ends, in the same form as `time`, or `permanent`.
|
||||||
- `aborted` is there, and true, when the client went away early.
|
- `aborted` is there, and true, when the client went away early.
|
||||||
- The timings are in milliseconds, to the microsecond. `duration_total` runs
|
- `duration_total` and `duration_upstream_total` are in milliseconds.
|
||||||
from when the request's headers had been read to when its line is written, and
|
|
||||||
`duration_checks` over the same start to when the checks were done; the health
|
|
||||||
check runs none, and its line has no `duration_checks`.
|
|
||||||
`duration_upstream_connect`, `duration_upstream_first_byte` and
|
|
||||||
`duration_upstream_total` are there for a request passed to the app, and run
|
|
||||||
from when it was handed to the app: until there was a connection to it, new or
|
|
||||||
kept open from an earlier request, until the first byte of its answer arrived,
|
|
||||||
and until the end. The first two are left out when that never happened, as for
|
|
||||||
an app that cannot be reached.
|
|
||||||
|
|
||||||
No body is logged, and no header but those above. `smallwebwaf`'s own messages
|
No body and no other header is logged. `smallwebwaf`'s own messages (start, the
|
||||||
(start, the settings, stop, errors) share the stream as JSON lines marked
|
settings, stop, errors) share the stream as JSON lines marked
|
||||||
`"type":"process"`.
|
`"type":"process"`.
|
||||||
|
|
||||||
Go's HTTP server, on which `smallwebwaf` is built, reads a request's line and
|
Go's HTTP server, on which `smallwebwaf` is built, reads a request's line and
|
||||||
@@ -462,32 +313,6 @@ which it answers `431`, headers slower than `SWWAF_CLIENT_REQUEST_TIMEOUT`,
|
|||||||
whose connection it closes without an answer, and requests it cannot read at
|
whose connection it closes without an answer, and requests it cannot read at
|
||||||
all, which it answers itself, mostly with `400`.
|
all, which it answers itself, mostly with `400`.
|
||||||
|
|
||||||
### Sending the log to a syslog server
|
|
||||||
|
|
||||||
While `SWWAF_LOG_REMOTE_URL` is set, every line `smallwebwaf` writes on stdout,
|
|
||||||
request lines and its own, is also sent to that syslog server, as the message of
|
|
||||||
an RFC 5424 record: one record to a datagram over UDP, and over TCP and TLS each
|
|
||||||
record after its length in bytes and a space. A record gives the facility
|
|
||||||
`SWWAF_LOG_REMOTE_FACILITY` names, the severity informational, the time the line
|
|
||||||
was written, in the same form as a request line's `time`, the host's name, and
|
|
||||||
the app name `SWWAF_LOG_REMOTE_APP_NAME` gives. stdout is unchanged.
|
|
||||||
|
|
||||||
The lines wait in a buffer of `SWWAF_LOG_REMOTE_BUFFER` lines and are sent from
|
|
||||||
there, so a server that is slow or cannot be reached never holds up a request or
|
|
||||||
stdout. When the buffer is full, its oldest line is dropped to make room. A line
|
|
||||||
whose sending fails is dropped too, and the connection closed. That failure,
|
|
||||||
like a failed attempt to connect, is logged and followed by the next attempt to
|
|
||||||
connect a second later, twice as long after each further failure up to a minute,
|
|
||||||
and a second again after a connection that stayed up for a minute before it
|
|
||||||
failed. A line too long for one UDP datagram is dropped alone, with no wait and
|
|
||||||
nothing logged. UDP gives no sign of what arrives, and over TCP and TLS a line
|
|
||||||
sent on a connection the server has just closed can be lost before a failure
|
|
||||||
shows; such a loss is not counted.
|
|
||||||
|
|
||||||
As `smallwebwaf` stops, it sends the lines still waiting, on the connection open
|
|
||||||
or a new one, for at most two seconds, and gives up the rest; stdout has carried
|
|
||||||
them.
|
|
||||||
|
|
||||||
## State files
|
## State files
|
||||||
|
|
||||||
`smallwebwaf` keeps its state in memory and a copy of it in three JSON files in
|
`smallwebwaf` keeps its state in memory and a copy of it in three JSON files in
|
||||||
@@ -495,14 +320,8 @@ them.
|
|||||||
[`SPEC.md`](SPEC.md) describes. Each has a top-level `version`, 1, and lists its
|
[`SPEC.md`](SPEC.md) describes. Each has a top-level `version`, 1, and lists its
|
||||||
entries by client address, with times in UTC.
|
entries by client address, with times in UTC.
|
||||||
|
|
||||||
- `bans.json`: every ban with its notes, indented to be read. A permanent ban's
|
- `bans.json`: every ban with its notes, indented to be read; a permanent ban's
|
||||||
`expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or
|
`expires` is `null`.
|
||||||
`attack` for a clear sign of attack, for a ban `smallwebwaf` made, and `admin`
|
|
||||||
for one you made or keep. Its `reason` is a short text: for a ban
|
|
||||||
`smallwebwaf` made, the limit broken, such as
|
|
||||||
`requests per minute over the limit of 1000`, or the rule that matched, such
|
|
||||||
as `matched the rule env-file`; for yours, what you wrote. Its `lifted` is
|
|
||||||
when you lifted it, and is left out until you do.
|
|
||||||
- `clients.json`: each client's two buckets in the minute, the hour and the day,
|
- `clients.json`: each client's two buckets in the minute, the hour and the day,
|
||||||
and its history: when it was first and last seen, its country as last looked
|
and its history: when it was first and last seen, its country as last looked
|
||||||
up and when, its requests, how many were forwarded and how many refused (one
|
up and when, its requests, how many were forwarded and how many refused (one
|
||||||
@@ -513,13 +332,12 @@ entries by client address, with times in UTC.
|
|||||||
- `lookups.json`: GeoJS's answers, one to a line, with when GeoJS gave each and
|
- `lookups.json`: GeoJS's answers, one to a line, with when GeoJS gave each and
|
||||||
when it was last used.
|
when it was last used.
|
||||||
|
|
||||||
`bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made or made
|
`bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, with every
|
||||||
permanent, with every such change in between, and every file every
|
ban made in between, and every file every `SWWAF_STATE_COUNTER_INTERVAL` and
|
||||||
`SWWAF_STATE_COUNTER_INTERVAL` and when `smallwebwaf` stops. Each write goes to
|
when `smallwebwaf` stops. Each write goes to a temporary file in the same
|
||||||
a temporary file in the same directory, which then replaces the file, so a crash
|
directory, which then replaces the file, so a crash leaves the old file or the
|
||||||
leaves the old file or the new one, whole. A write that fails is logged, and
|
new one, whole. A write that fails is logged, and tried again at the next write.
|
||||||
tried again at the next write. A hard kill loses what changed since the last
|
A hard kill loses what changed since the last write.
|
||||||
write.
|
|
||||||
|
|
||||||
At start the files are read back: each client keeps its counts, so a restart
|
At start the files are read back: each client keeps its counts, so a restart
|
||||||
gives it no fresh allowance, and each ban keeps refusing every client in its
|
gives it no fresh allowance, and each ban keeps refusing every client in its
|
||||||
@@ -534,8 +352,7 @@ without a field it needs, named with the entry's place in the file: a ban's
|
|||||||
`netblock`, `start` or `expires`, which is `null` for a permanent ban; a
|
`netblock`, `start` or `expires`, which is `null` for a permanent ban; a
|
||||||
client's `client`, or the `start` of a window in which it has requests; an
|
client's `client`, or the `start` of a window in which it has requests; an
|
||||||
answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
|
answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
|
||||||
`answered`. So does a ban whose `cause` is not `limit`, `attack` or `admin`. The
|
`answered`. The AS number and AS name come with their lookup.
|
||||||
AS number and AS name come with their lookup.
|
|
||||||
|
|
||||||
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
|
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
|
||||||
a state file as soon as you save it: what the file then holds replaces what
|
a state file as soon as you save it: what the file then holds replaces what
|
||||||
@@ -544,21 +361,17 @@ yours by comparing the file with what it last read or wrote, and before it
|
|||||||
writes a file it takes in any edit made since, so your edit is not overwritten;
|
writes a file it takes in any edit made since, so your edit is not overwritten;
|
||||||
a change `smallwebwaf` made after you opened the file, such as a new ban, is
|
a change `smallwebwaf` made after you opened the file, such as a new ban, is
|
||||||
lost when you save over it. An edit that would stop the start, because it does
|
lost when you save over it. An edit that would stop the start, because it does
|
||||||
not parse, has another `version`, leaves out a field an entry needs or gives a
|
not parse, has another `version` or leaves out a field an entry needs, does not
|
||||||
ban another `cause`, does not stop the running `smallwebwaf`: it keeps what it
|
stop the running `smallwebwaf`: it keeps what it holds, and at the file's next
|
||||||
holds, and at the file's next write renames your file to `<name>.bad`, such as
|
write renames your file to `<name>.bad`, such as `bans.json.bad`, writes the
|
||||||
`bans.json.bad`, writes the file again from memory, and logs the file and where
|
file again from memory, and logs the file and where the error is. It waits for
|
||||||
the error is. It waits for that write because an editor's file can be read
|
that write because an editor's file can be read before the editor has finished
|
||||||
before the editor has finished writing it. Mend the `.bad` file and move it
|
writing it. Mend the `.bad` file and move it back. A file you remove is written
|
||||||
back. A file you remove is written again at its next write.
|
again at its next write.
|
||||||
|
|
||||||
To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
|
To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
|
||||||
and its `expires`, `null` for a ban that never ends; its `reason` and its
|
and its `expires`, `null` for a ban that never ends; its `notes` may be left
|
||||||
`notes` may be left out, and so may its `cause`, which is then `admin`, and is
|
out. This `bans.json` bans `203.0.113.0/24` for good:
|
||||||
written so at the file's next write. A ban whose `cause` is `admin` is never
|
|
||||||
dropped and does not count toward `SWWAF_MAX_BANS`. A ban whose `cause` is
|
|
||||||
`attack` becomes permanent at the first request it refuses; one whose `cause` is
|
|
||||||
`admin` does not. This `bans.json` bans `203.0.113.0/24` for good:
|
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
@@ -567,97 +380,14 @@ dropped and does not count toward `SWWAF_MAX_BANS`. A ban whose `cause` is
|
|||||||
{
|
{
|
||||||
"netblock": "203.0.113.0/24",
|
"netblock": "203.0.113.0/24",
|
||||||
"start": "2026-10-06T12:00:00Z",
|
"start": "2026-10-06T12:00:00Z",
|
||||||
"expires": null,
|
"expires": null
|
||||||
"reason": "probes for logins"
|
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
To keep a ban `smallwebwaf` made, so that it is never dropped, set its `cause`
|
To lift a ban, delete its entry. `smallwebwaf` then forgets the ban, so it does
|
||||||
to `admin`: `"cause": "admin"`.
|
not make the netblock's next ban longer.
|
||||||
|
|
||||||
To lift a ban, add `lifted` to its entry, with the time you lift it, such as
|
|
||||||
`"lifted": "2026-10-06T13:00:00Z"`. From when the edit is taken in, the ban
|
|
||||||
refuses nothing, whatever time `lifted` gives, and does not make the netblock's
|
|
||||||
next ban longer; it is kept in `bans.json` with its notes, as any other ban is.
|
|
||||||
To forget a ban altogether, delete its entry: it then refuses nothing either,
|
|
||||||
and does not make the netblock's next ban longer.
|
|
||||||
|
|
||||||
## Rule files
|
|
||||||
|
|
||||||
`smallwebwaf` reads every `*.rules` file in `SWWAF_RULES_DIR`,
|
|
||||||
`/etc/smallwebwaf/rules.d` by default, in the order of their names, and checks
|
|
||||||
each request against their rules in that order, as "Rule files" in
|
|
||||||
[`SPEC.md`](SPEC.md) describes. A file whose name starts with `.`, such as an
|
|
||||||
editor's lock file `.#50-app.rules`, is not a rule file, as a shell's `*.rules`
|
|
||||||
would not match it. A rule is a line of four fields separated by spaces or tabs:
|
|
||||||
an id, a target, an action and a regex, which runs to the end of the line.
|
|
||||||
Spaces and tabs at the end of a line are not part of its regex, so a line with
|
|
||||||
only those after its action has no regex, and is not a rule. Blank lines and
|
|
||||||
lines that start with `#` are ignored.
|
|
||||||
|
|
||||||
```
|
|
||||||
# id target action regex
|
|
||||||
env-file path ban (?i)^/\.env(\.[a-z]+)?$
|
|
||||||
scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b
|
|
||||||
```
|
|
||||||
|
|
||||||
- The id is letters, digits, `-` and `_`, and no two rules share one. The
|
|
||||||
request log, the metrics and a ban's notes name the rule by it.
|
|
||||||
- The target is what the regex is matched against: `path` or `query`, as the
|
|
||||||
client sent it, before any decoding; `uri`, the path and the query together,
|
|
||||||
both as sent and once percent-decoded, so that an encoded probe does not slip
|
|
||||||
past; `method`; `host`; `user_agent`; `referer`; or `header:<Name>`, any one
|
|
||||||
request header but `Host` and `Transfer-Encoding`, which Go's HTTP server
|
|
||||||
takes out of every request; the request's host is the target `host`. A header
|
|
||||||
sent more than once is matched with its values joined by `, `, and one not
|
|
||||||
sent as empty text. No body is read.
|
|
||||||
- The action is `log`, `block` or `ban` (see "What it does so far" above). Keep
|
|
||||||
`ban` for requests no real visitor sends, and anchor a path at the site root
|
|
||||||
with `^/`: a file of the same name deeper in a site can be ordinary content,
|
|
||||||
such as a file in a repository on a code forge.
|
|
||||||
- The regex is in Go's syntax, RE2, which has no backreferences or lookaround,
|
|
||||||
and takes time linear in the text it reads. It matches anywhere in the target
|
|
||||||
unless anchored with `^` and `$`; `(?i)` at its front makes it ignore case.
|
|
||||||
|
|
||||||
A line that is not a rule, a `header:` name with a character no header name can
|
|
||||||
have, such as `header:User-Agent:`, a rule for the `Host` or the
|
|
||||||
`Transfer-Encoding` header, a regex that does not compile or an id used twice
|
|
||||||
stops the start with a message naming the file and the line, and so does a
|
|
||||||
`SWWAF_RULES_DIR` that does not exist. An empty directory is no error, and the
|
|
||||||
log says that it holds no rules. While it runs, `smallwebwaf` watches the
|
|
||||||
directory, and reads the rule files again once the directory has had no change
|
|
||||||
for 2 seconds after one is edited, added or removed, so that a file saved in
|
|
||||||
place, appended to or copied in with `scp` is read only once whole, unless its
|
|
||||||
writing stops for longer. It also reads them 2 seconds after it starts watching,
|
|
||||||
so that an edit saved while it started is not missed. If they then hold one of
|
|
||||||
those errors, the rules stay as they were, the earlier version of the edited
|
|
||||||
file included, the log names the file and the line, and the files are read again
|
|
||||||
after the next change.
|
|
||||||
|
|
||||||
The image ships one rule file, `share/rules.d/00-default.rules` here: rules that
|
|
||||||
ban probes no real visitor sends, for secrets, version control directories,
|
|
||||||
backups, logs and web shells at the site root, and the user agents of common
|
|
||||||
scanners; one that blocks `../` twice in a row in the path or the query; and one
|
|
||||||
that only notes a request without a user agent. An app's Dockerfile adds rules
|
|
||||||
of its own in a file beside it, named to sort after it, such as this
|
|
||||||
`50-gitea.rules` for an app that serves no WordPress:
|
|
||||||
|
|
||||||
```
|
|
||||||
wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/)
|
|
||||||
```
|
|
||||||
|
|
||||||
```dockerfile
|
|
||||||
COPY 50-gitea.rules /etc/smallwebwaf/rules.d/50-gitea.rules
|
|
||||||
```
|
|
||||||
|
|
||||||
A directory mounted over `/etc/smallwebwaf/rules.d` replaces the default file,
|
|
||||||
and single files mounted into it add to it. Docker does not show a single
|
|
||||||
mounted file being replaced, which is how many editors save, so rules to be
|
|
||||||
edited while `smallwebwaf` runs belong in a mounted directory, with a copy of
|
|
||||||
`00-default.rules` if its rules are to stay. To run without rules, mount an
|
|
||||||
empty directory or set `SWWAF_RULES_ENABLED=false`.
|
|
||||||
|
|
||||||
## Metrics
|
## Metrics
|
||||||
|
|
||||||
@@ -676,13 +406,8 @@ other request. No metric carries a client's address.
|
|||||||
- `smallwebwaf_rate_limit_hits_total` by `window`,
|
- `smallwebwaf_rate_limit_hits_total` by `window`,
|
||||||
`smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose
|
`smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose
|
||||||
limit was passed, `smallwebwaf_offences_total` by `kind`, and
|
limit was passed, `smallwebwaf_offences_total` by `kind`, and
|
||||||
`smallwebwaf_bans_made_total` by `cause`, `limit`, `attack` or `admin`, the
|
`smallwebwaf_bans_made_total` by `cause`; `smallwebwaf_active_bans` and
|
||||||
last for the bans whose `cause` is `admin` that you add to `bans.json` while
|
`smallwebwaf_permanent_bans`.
|
||||||
`smallwebwaf` runs; `smallwebwaf_active_bans` and
|
|
||||||
`smallwebwaf_permanent_bans`, neither of which counts a lifted ban.
|
|
||||||
- `smallwebwaf_rule_matches_total`: the requests that matched each rule, by
|
|
||||||
`rule_id` and `action`, the rule's own; and `smallwebwaf_rules_loaded`: the
|
|
||||||
rules read from the rule files.
|
|
||||||
- `smallwebwaf_country_requests_total`,
|
- `smallwebwaf_country_requests_total`,
|
||||||
`smallwebwaf_country_request_bytes_total`,
|
`smallwebwaf_country_request_bytes_total`,
|
||||||
`smallwebwaf_country_response_bytes_total`, and
|
`smallwebwaf_country_response_bytes_total`, and
|
||||||
@@ -705,16 +430,11 @@ other request. No metric carries a client's address.
|
|||||||
`smallwebwaf_state_file_edits_taken_in_total`: your edits taken in, and
|
`smallwebwaf_state_file_edits_taken_in_total`: your edits taken in, and
|
||||||
`smallwebwaf_state_file_edits_set_aside_total`: those renamed to `<name>.bad`
|
`smallwebwaf_state_file_edits_set_aside_total`: those renamed to `<name>.bad`
|
||||||
because they would stop the start.
|
because they would stop the start.
|
||||||
- While `SWWAF_LOG_REMOTE_URL` is set,
|
|
||||||
`smallwebwaf_remote_log_lines_sent_total`: the lines sent to it;
|
|
||||||
`smallwebwaf_remote_log_lines_dropped_total`: those dropped, from a full
|
|
||||||
buffer or because their sending failed; and
|
|
||||||
`smallwebwaf_remote_log_buffer_depth`: those waiting in the buffer.
|
|
||||||
- Go's own `go_` metrics and the process's `process_` metrics.
|
- Go's own `go_` metrics and the process's `process_` metrics.
|
||||||
|
|
||||||
The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
|
The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
|
||||||
log") are not counted. The metrics of the features still to come, such as the
|
log") are not counted. The metrics of the features still to come, such as the
|
||||||
Core Rule Set, come with them.
|
rule files, come with them.
|
||||||
|
|
||||||
## Why
|
## Why
|
||||||
|
|
||||||
@@ -988,29 +708,25 @@ addresses are never sent to GeoJS.
|
|||||||
## How the code is laid out
|
## How the code is laid out
|
||||||
|
|
||||||
- `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`.
|
- `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`.
|
||||||
- `internal/smallwebwaf`: the process: it reads the settings, the rule files and
|
- `internal/smallwebwaf`: the process: it reads the settings and the state
|
||||||
the state files, listens, serves requests until `SIGTERM` or `SIGINT`, and
|
files, listens, serves requests until `SIGTERM` or `SIGINT`, and stops,
|
||||||
stops, writing the state files. Run as `smallwebwaf healthcheck`, it is the
|
writing the state files. Run as `smallwebwaf healthcheck`, it is the image's
|
||||||
image's health check instead.
|
health check instead.
|
||||||
- `internal/config`: reads the settings, the one place they are read.
|
- `internal/config`: reads the settings, the one place they are read.
|
||||||
- `internal/proxy`: what happens to each request: it works out the client, runs
|
- `internal/proxy`: what happens to each request: it works out the client, runs
|
||||||
the checks, passes the request to the app and the answer back with the
|
the checks, passes the request to the app and the answer back with the
|
||||||
standard library's `httputil.ReverseProxy` within the timeouts and size
|
standard library's `httputil.ReverseProxy` within the timeouts and size
|
||||||
limits, and writes the request's log line. Its `check` method is where a
|
limits, and writes the request's log line. Its `check` method is where a
|
||||||
request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
|
request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
|
||||||
a ban, for the country lists, for a rate limit, which bans the client, for a
|
a ban, for the country lists, for a rate limit, which bans the client, and for
|
||||||
`block` or `ban` rule, the latter banning the client, and for an announced
|
an announced body over the size limit; in `observe` mode, only for the size
|
||||||
body over the size limit; in `observe` mode, only for the size limit, with
|
limit, with what it would have refused for noted in the log line. A request
|
||||||
what it would have refused for noted in the log line. A request under
|
under `/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
|
||||||
`/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
|
|
||||||
instead of reaching the app.
|
instead of reaching the app.
|
||||||
- `internal/metrics`: the metrics, counted as the other parts tell it what
|
- `internal/metrics`: the metrics, counted as the other parts tell it what
|
||||||
happened, and served in the Prometheus text format.
|
happened, and served in the Prometheus text format.
|
||||||
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how
|
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how
|
||||||
long a new ban lasts, when a ban for a clear sign of attack becomes permanent,
|
long a new ban lasts, and which ban is dropped when `SWWAF_MAX_BANS` are held.
|
||||||
and which ban `smallwebwaf` made is dropped when `SWWAF_MAX_BANS` are held.
|
|
||||||
- `internal/rules`: reads the rule files at start and again as they change, and
|
|
||||||
tells which of their rules a request matches.
|
|
||||||
- `internal/lookup`: looks up each client's country through GeoJS, and keeps the
|
- `internal/lookup`: looks up each client's country through GeoJS, and keeps the
|
||||||
answers.
|
answers.
|
||||||
- `internal/ratelimit`: the table of clients: counts each client's requests,
|
- `internal/ratelimit`: the table of clients: counts each client's requests,
|
||||||
@@ -1019,24 +735,18 @@ addresses are never sent to GeoJS.
|
|||||||
one while running, and writes them when they are due and at the stop.
|
one while running, and writes them when they are due and at the stop.
|
||||||
- `internal/requestlog`: the lines on stdout: the request log line and the
|
- `internal/requestlog`: the lines on stdout: the request log line and the
|
||||||
process's own messages.
|
process's own messages.
|
||||||
- `internal/remotelog`: sends the lines on stdout to `SWWAF_LOG_REMOTE_URL`,
|
|
||||||
each as a syslog record, from a buffer of its own. It is written with the
|
|
||||||
standard library alone, whose `log/syslog` writes only the older syslog
|
|
||||||
format.
|
|
||||||
- `Dockerfile`: the lint and test phases, then the image, whose last stage
|
- `Dockerfile`: the lint and test phases, then the image, whose last stage
|
||||||
installs Ubuntu's packages, nixpkgs, `runsvinit` and `smallwebwaf`, with
|
installs Ubuntu's packages, nixpkgs, `runsvinit` and `smallwebwaf`, with
|
||||||
`share/smallwebwaf.run` as runit's `run` script for `smallwebwaf` and
|
`share/smallwebwaf.run` as runit's `run` script for `smallwebwaf`.
|
||||||
`share/rules.d/00-default.rules` as its default rule file.
|
|
||||||
- `deploy/example-app`: an app built on the image, which `script/example-app`
|
- `deploy/example-app`: an app built on the image, which `script/example-app`
|
||||||
checks.
|
checks.
|
||||||
|
|
||||||
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
|
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
|
||||||
table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least
|
table of clients to 20,000, the GeoJS answers to 100,000 and the banned
|
||||||
recently seen, and the banned netblocks in the order they were last seen, from
|
netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen, and
|
||||||
which the ledger picks the ban to drop past `SWWAF_MAX_BANS`, and
|
|
||||||
`github.com/prometheus/client_golang` keeps the metrics and serves them, and
|
`github.com/prometheus/client_golang` keeps the metrics and serves them, and
|
||||||
`github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file or a rule
|
`github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file is saved.
|
||||||
file is saved. The country codes are the list in `internal/config/config.go`.
|
The country codes are the list in `internal/config/config.go`.
|
||||||
|
|
||||||
## Entrypoints
|
## Entrypoints
|
||||||
|
|
||||||
@@ -1066,22 +776,21 @@ so that they run in minimal containers.
|
|||||||
- `script/build`: builds `bin/smallwebwaf` on the host, with Go installed, for
|
- `script/build`: builds `bin/smallwebwaf` on the host, with Go installed, for
|
||||||
working on the code by hand; `make build` runs it.
|
working on the code by hand; `make build` runs it.
|
||||||
- `script/run`: builds `bin/smallwebwaf` with `script/build` and runs it, with
|
- `script/run`: builds `bin/smallwebwaf` with `script/build` and runs it, with
|
||||||
its state files in `bin/state` unless `SWWAF_STATE_DIR` is set, and the rule
|
its state files in `bin/state` unless `SWWAF_STATE_DIR` is set; `make run`
|
||||||
files of `share/rules.d` unless `SWWAF_RULES_DIR` is set; `make run` runs it.
|
runs it.
|
||||||
- `script/example-app`: builds the image and, on it, the example app in
|
- `script/example-app`: builds the image and, on it, the example app in
|
||||||
`deploy/example-app`, runs it with a volume for the state files, and checks
|
`deploy/example-app`, runs it with a volume for the state files, and checks
|
||||||
that the health check passes, that a request reaches the app through
|
that the health check passes, that a request reaches the app through
|
||||||
`smallwebwaf`, that a second request in a minute bans the client, that a probe
|
`smallwebwaf`, that a second request in a minute bans the client, that
|
||||||
for `/.env` bans another client, whose next request makes the ban permanent,
|
`sv stop` and `docker stop` stop it in order, and that a new container on the
|
||||||
that `sv stop` and `docker stop` stop it in order, and that a new container on
|
same volume still refuses the banned client; then removes the containers, the
|
||||||
the same volume still refuses the banned client; then removes the containers,
|
volume and both images. It needs network access, for nixpkgs' binary cache,
|
||||||
the volume and both images. It needs network access, for nixpkgs' binary
|
and `script/check` does not run it; `make example-app` does.
|
||||||
cache, and `script/check` does not run it; `make example-app` does.
|
|
||||||
|
|
||||||
## TODO
|
## TODO
|
||||||
|
|
||||||
- The rest of the design, in the order of the build order in
|
- The rest of milestone 3: the rest of the request log's fields; then the rest
|
||||||
[`SPEC.md`](SPEC.md).
|
of the design, in the order of the build order in [`SPEC.md`](SPEC.md).
|
||||||
|
|
||||||
## Documents
|
## Documents
|
||||||
|
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ from a directory of hand-editable text files.
|
|||||||
- Defence against traffic floods that saturate the host's network link. That
|
- Defence against traffic floods that saturate the host's network link. That
|
||||||
needs help upstream of the host.
|
needs help upstream of the host.
|
||||||
- A web UI or a configuration file. Settings are environment variables. Apart
|
- A web UI or a configuration file. Settings are environment variables. Apart
|
||||||
from settings given as files (the `_FILE` form of a setting, such as
|
from settings given as files (the `_FILE` form of any setting, such as
|
||||||
`SWWAF_ADMIN_TOKEN_FILE`, and `SWWAF_LOG_REMOTE_TLS_CA_FILE`), its own state
|
`SWWAF_ADMIN_TOKEN_FILE`, and `SWWAF_LOG_REMOTE_TLS_CA_FILE`), its own state
|
||||||
files and the lookup database, the only files read are the rule files, which
|
files and the lookup database, the only files read are the rule files, which
|
||||||
hold one regex per line and nothing more elaborate.
|
hold one regex per line and nothing more elaborate.
|
||||||
@@ -298,8 +298,7 @@ it.
|
|||||||
- A list set to an empty value is an empty list, and replaces the default.
|
- A list set to an empty value is an empty list, and replaces the default.
|
||||||
- Every setting may instead be given as a file holding the value, named by the
|
- Every setting may instead be given as a file holding the value, named by the
|
||||||
setting's name with `_FILE` added, such as `SWWAF_ADMIN_TOKEN_FILE`, for
|
setting's name with `_FILE` added, such as `SWWAF_ADMIN_TOKEN_FILE`, for
|
||||||
secrets and long lists. `SWWAF_LOG_REMOTE_TLS_CA_FILE`, whose value names a
|
secrets and long lists.
|
||||||
file already, has no `_FILE` form.
|
|
||||||
- Settings, including those given as files, are read once at start; changing one
|
- Settings, including those given as files, are read once at start; changing one
|
||||||
means restarting the container. The files `smallwebwaf` watches while it runs
|
means restarting the container. The files `smallwebwaf` watches while it runs
|
||||||
are its state files, its rule files and the lookup database.
|
are its state files, its rule files and the lookup database.
|
||||||
|
|||||||
@@ -1,186 +0,0 @@
|
|||||||
package bans_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/netip"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestBanWithoutACauseIsAnAdmins(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
netblock := netip.MustParsePrefix("203.0.113.0/24")
|
|
||||||
ledger := bans.New(defaultRules())
|
|
||||||
ledger.Load([]bans.Ban{{Netblock: netblock, Start: midnight()}})
|
|
||||||
|
|
||||||
if got := ledger.Bans(netblock)[0].Cause; got != bans.CauseAdmin {
|
|
||||||
t.Errorf("the ban's cause is %q, want admin", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
rules := defaultRules()
|
|
||||||
rules.MaxBans = 1
|
|
||||||
ledger := bans.New(rules)
|
|
||||||
adminsOnly := netip.MustParsePrefix("198.51.100.0/24")
|
|
||||||
both := netip.MustParsePrefix("203.0.113.1/32")
|
|
||||||
second := netip.MustParsePrefix("203.0.113.2/32")
|
|
||||||
third := netip.MustParsePrefix("203.0.113.3/32")
|
|
||||||
|
|
||||||
// Seen longest ago, a netblock with two of an admin's bans alone, and
|
|
||||||
// then one with an admin's ban before a ban smallwebwaf made: the one
|
|
||||||
// ban counted toward MaxBans.
|
|
||||||
ledger.Load([]bans.Ban{
|
|
||||||
{Netblock: adminsOnly, Start: midnight().Add(-3 * time.Hour), Cause: bans.CauseAdmin},
|
|
||||||
{Netblock: adminsOnly, Start: midnight().Add(-2 * time.Hour), Cause: bans.CauseAdmin},
|
|
||||||
{Netblock: both, Start: midnight().Add(-time.Hour), Cause: bans.CauseAdmin},
|
|
||||||
{
|
|
||||||
Netblock: both,
|
|
||||||
Start: midnight(),
|
|
||||||
Expires: midnight().Add(time.Hour),
|
|
||||||
Cause: bans.CauseLimit,
|
|
||||||
},
|
|
||||||
})
|
|
||||||
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 2})
|
|
||||||
|
|
||||||
// A new ban drops the ban smallwebwaf made, and only that one.
|
|
||||||
ledger.BanForLimit(second, midnight(), bans.Notes{})
|
|
||||||
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 1, second: 1})
|
|
||||||
|
|
||||||
if ledger.Bans(both)[0].Cause != bans.CauseAdmin {
|
|
||||||
t.Errorf("%s kept %+v, want the admin's ban", both, ledger.Bans(both))
|
|
||||||
}
|
|
||||||
|
|
||||||
// And the next drops that one.
|
|
||||||
ledger.BanForLimit(third, midnight(), bans.Notes{})
|
|
||||||
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 1, second: 0, third: 1})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestReasonOfTheBansSmallwebwafMakes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ledger := bans.New(defaultRules())
|
|
||||||
|
|
||||||
limit := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.1/32"), midnight(),
|
|
||||||
bans.Notes{Limit: 1000, Window: "minute"})
|
|
||||||
attack := ledger.BanForAttack(netip.MustParsePrefix("203.0.113.2/32"), midnight(),
|
|
||||||
bans.Notes{RuleID: "git-dir", Target: "path"})
|
|
||||||
|
|
||||||
for _, tc := range []struct{ got, want string }{
|
|
||||||
{limit.Reason, "requests per minute over the limit of 1000"},
|
|
||||||
{attack.Reason, "matched the rule git-dir"},
|
|
||||||
} {
|
|
||||||
if tc.got != tc.want {
|
|
||||||
t.Errorf("the reason is %q, want %q", tc.got, tc.want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLiftedBanForALimitRefusesNothingAndMakesNoBanLonger(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// An hour's ban lifted ten minutes after it started.
|
|
||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
|
||||||
lifted := bans.Ban{
|
|
||||||
Netblock: netblock,
|
|
||||||
Start: midnight(),
|
|
||||||
Expires: midnight().Add(time.Hour),
|
|
||||||
Cause: bans.CauseLimit,
|
|
||||||
Lifted: midnight().Add(10 * time.Minute),
|
|
||||||
}
|
|
||||||
|
|
||||||
ledger := bans.New(defaultRules())
|
|
||||||
ledger.Load([]bans.Ban{lifted})
|
|
||||||
|
|
||||||
// While it would still last, it refuses nothing, and a limit broken
|
|
||||||
// bans for an hour, as a first broken limit does; the lifted ban is
|
|
||||||
// kept, and counted among the earlier bans.
|
|
||||||
now := midnight().Add(30 * time.Minute)
|
|
||||||
|
|
||||||
_, banned := ledger.Check(netblock.Addr(), now)
|
|
||||||
if banned {
|
|
||||||
t.Error("the lifted ban refuses")
|
|
||||||
}
|
|
||||||
|
|
||||||
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
|
|
||||||
if ban.Expires.Sub(ban.Start) != time.Hour ||
|
|
||||||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
|
|
||||||
t.Errorf("the next ban lasts %s with earlier bans %+v, want 1h and 1 for a limit",
|
|
||||||
ban.Expires.Sub(ban.Start), ban.Notes.EarlierBans)
|
|
||||||
}
|
|
||||||
|
|
||||||
held := ledger.Bans(netblock)
|
|
||||||
if len(held) != 2 || held[0] != lifted {
|
|
||||||
t.Errorf("the ledger holds %+v, want the lifted ban and the new one", held)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLiftedBanForAnAttackRefusesNothingAndMakesNoBanLonger(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// A permanent ban for a clear sign of attack, lifted.
|
|
||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
|
||||||
ledger := bans.New(defaultRules())
|
|
||||||
ledger.Load([]bans.Ban{{
|
|
||||||
Netblock: netblock,
|
|
||||||
Start: midnight(),
|
|
||||||
Cause: bans.CauseAttack,
|
|
||||||
Lifted: midnight().Add(time.Hour),
|
|
||||||
}})
|
|
||||||
|
|
||||||
now := midnight().Add(2 * time.Hour)
|
|
||||||
|
|
||||||
_, banned := ledger.Find(netblock.Addr(), now)
|
|
||||||
if banned {
|
|
||||||
t.Error("the lifted ban refuses")
|
|
||||||
}
|
|
||||||
|
|
||||||
active, permanent := ledger.Count(now)
|
|
||||||
if active != 0 || permanent != 0 {
|
|
||||||
t.Errorf("%d bans are active and %d permanent, want none", active, permanent)
|
|
||||||
}
|
|
||||||
|
|
||||||
// The next clear sign of attack bans for seven days, as a first does.
|
|
||||||
ban := ledger.BanForAttack(netblock, now, bans.Notes{})
|
|
||||||
if ban.Expires.Sub(ban.Start) != 7*day {
|
|
||||||
t.Errorf("the next ban for an attack ends at %s, want seven days on", ban.Expires)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLoadEditCountsTheBansAnAdminMade(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ledger := bans.New(defaultRules())
|
|
||||||
made := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.1/32"), midnight(),
|
|
||||||
bans.Notes{})
|
|
||||||
atStart := bans.Ban{
|
|
||||||
Netblock: netip.MustParsePrefix("203.0.113.2/32"),
|
|
||||||
Start: midnight(),
|
|
||||||
}
|
|
||||||
|
|
||||||
// The bans read at the start were made before it.
|
|
||||||
ledger.Load([]bans.Ban{made, atStart})
|
|
||||||
|
|
||||||
if got := ledger.Made(bans.CauseAdmin); got != 0 {
|
|
||||||
t.Fatalf("%d bans made by an admin after the start's, want none", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
// The admin keeps the ban smallwebwaf made, keeps the one read at the
|
|
||||||
// start, and adds one without a cause: that one alone is made.
|
|
||||||
kept := made
|
|
||||||
kept.Cause = bans.CauseAdmin
|
|
||||||
added := bans.Ban{
|
|
||||||
Netblock: netip.MustParsePrefix("203.0.113.3/32"),
|
|
||||||
Start: midnight(),
|
|
||||||
}
|
|
||||||
ledger.LoadEdit([]bans.Ban{kept, atStart, added})
|
|
||||||
|
|
||||||
if ledger.Made(bans.CauseAdmin) != 1 || ledger.Made(bans.CauseLimit) != 1 {
|
|
||||||
t.Errorf("%d bans made by an admin and %d for a limit, want 1 of each",
|
|
||||||
ledger.Made(bans.CauseAdmin), ledger.Made(bans.CauseLimit))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+115
-309
@@ -1,13 +1,10 @@
|
|||||||
// Package bans is the ban ledger: the bans smallwebwaf makes on the
|
// Package bans is the ban ledger: the bans smallwebwaf makes on the
|
||||||
// netblocks of clients that break a rate limit or show a clear sign of
|
// netblocks of clients that break a rate limit, with their notes, as the
|
||||||
// attack, and those an admin makes, with their notes, as the "Bans"
|
// "Bans" section of SPEC.md describes. The bans are kept in memory, and
|
||||||
// section of SPEC.md describes. The bans are kept in memory, and written
|
// written to bans.json and read from it by the state package.
|
||||||
// to bans.json and read from it by the state package.
|
|
||||||
package bans
|
package bans
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
|
||||||
"math"
|
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -17,17 +14,6 @@ import (
|
|||||||
"github.com/hashicorp/golang-lru/v2/simplelru"
|
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The causes of bans.
|
|
||||||
const (
|
|
||||||
// CauseLimit is a ban smallwebwaf made for a broken limit.
|
|
||||||
CauseLimit = "limit"
|
|
||||||
// CauseAttack is a ban smallwebwaf made for a clear sign of attack.
|
|
||||||
CauseAttack = "attack"
|
|
||||||
// CauseAdmin is a ban an admin made, or one smallwebwaf made that an
|
|
||||||
// admin keeps. It is never dropped.
|
|
||||||
CauseAdmin = "admin"
|
|
||||||
)
|
|
||||||
|
|
||||||
// repeatFactor is how many times as long as the netblock's last ban a ban
|
// repeatFactor is how many times as long as the netblock's last ban a ban
|
||||||
// for a limit broken again within the repeat window lasts.
|
// for a limit broken again within the repeat window lasts.
|
||||||
const repeatFactor = 3
|
const repeatFactor = 3
|
||||||
@@ -35,44 +21,32 @@ const repeatFactor = 3
|
|||||||
// maxTextBytes is how much of each text in a ban's notes is kept.
|
// maxTextBytes is how much of each text in a ban's notes is kept.
|
||||||
const maxTextBytes = 256
|
const maxTextBytes = 256
|
||||||
|
|
||||||
// Rules are how long a ban lasts, and how many bans are held.
|
// Rules are how long a ban for a broken limit lasts, and how many bans
|
||||||
|
// are held.
|
||||||
type Rules struct {
|
type Rules struct {
|
||||||
// LimitBanDuration is how long a first ban for a broken limit lasts.
|
// LimitBanDuration is how long a first ban lasts.
|
||||||
LimitBanDuration time.Duration
|
LimitBanDuration time.Duration
|
||||||
// LimitBanRepeatWindow is how soon after the end of the netblock's
|
// LimitBanRepeatWindow is how soon after the end of the netblock's
|
||||||
// ban that ended last, other than one for a clear sign of attack, a
|
// ban that ended last a broken limit counts as a repeat, which bans
|
||||||
// broken limit counts as a repeat, which bans for repeatFactor times as
|
// for repeatFactor times as long as that ban.
|
||||||
// long as that ban.
|
|
||||||
LimitBanRepeatWindow time.Duration
|
LimitBanRepeatWindow time.Duration
|
||||||
// MaxBanDuration is the longest ban for a broken limit; one that would
|
// MaxBanDuration is the longest ban; a ban that would be longer is
|
||||||
// be longer is permanent instead.
|
// permanent instead.
|
||||||
MaxBanDuration time.Duration
|
MaxBanDuration time.Duration
|
||||||
// AttackBanDuration is how long a first ban for a clear sign of attack
|
// MaxBans is the most bans held, at least one. Past it, the earliest
|
||||||
// lasts.
|
// ban of the netblock that has gone longest without a request is
|
||||||
AttackBanDuration time.Duration
|
// dropped.
|
||||||
// MaxBans is the most bans held whose cause is not CauseAdmin, at
|
|
||||||
// least one. Past it, the earliest such ban of the netblock that has
|
|
||||||
// gone longest without a request is dropped. Bans whose cause is
|
|
||||||
// CauseAdmin are held besides, and never dropped.
|
|
||||||
MaxBans int
|
MaxBans int
|
||||||
}
|
}
|
||||||
|
|
||||||
// Ban is a ban on a netblock.
|
// Ban is a ban on a netblock for a broken limit, the only kind of ban
|
||||||
|
// smallwebwaf makes so far.
|
||||||
type Ban struct {
|
type Ban struct {
|
||||||
Netblock netip.Prefix
|
Netblock netip.Prefix
|
||||||
Start time.Time
|
Start time.Time
|
||||||
// Expires is when the ban ends, zero for a permanent ban.
|
// Expires is when the ban ends, zero for a permanent ban.
|
||||||
Expires time.Time
|
Expires time.Time
|
||||||
// Cause is CauseLimit, CauseAttack or CauseAdmin.
|
Notes Notes
|
||||||
Cause string
|
|
||||||
// Reason is a short text: for a ban smallwebwaf made, the limit broken
|
|
||||||
// or the rule that matched; for an admin's, what the admin wrote.
|
|
||||||
Reason string
|
|
||||||
// Lifted is when an admin lifted the ban, zero while no admin has. A
|
|
||||||
// lifted ban refuses nothing, and does not make the netblock's next
|
|
||||||
// ban longer.
|
|
||||||
Lifted time.Time
|
|
||||||
Notes Notes
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Permanent reports whether the ban never runs out.
|
// Permanent reports whether the ban never runs out.
|
||||||
@@ -80,10 +54,9 @@ func (b Ban) Permanent() bool {
|
|||||||
return b.Expires.IsZero()
|
return b.Expires.IsZero()
|
||||||
}
|
}
|
||||||
|
|
||||||
// ActiveAt reports whether the ban refuses requests at now: it has not
|
// ActiveAt reports whether the ban refuses requests at now.
|
||||||
// been lifted, and has not run out.
|
|
||||||
func (b Ban) ActiveAt(now time.Time) bool {
|
func (b Ban) ActiveAt(now time.Time) bool {
|
||||||
return b.Lifted.IsZero() && (b.Permanent() || now.Before(b.Expires))
|
return b.Permanent() || now.Before(b.Expires)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Notes are what an admin needs to decide whether to lift a ban. The
|
// Notes are what an admin needs to decide whether to lift a ban. The
|
||||||
@@ -93,36 +66,23 @@ func (b Ban) ActiveAt(now time.Time) bool {
|
|||||||
type Notes struct {
|
type Notes struct {
|
||||||
// Country is the client's country, when it was looked up.
|
// Country is the client's country, when it was looked up.
|
||||||
Country string `json:"country"`
|
Country string `json:"country"`
|
||||||
// Limit, Window and Count are, for a ban for a broken limit, the limit
|
// Limit, Window and Count are the limit that was broken, its window,
|
||||||
// that was broken, its window, "minute", "hour" or "day", and the
|
// "minute", "hour" or "day", and the count reached: the client's
|
||||||
// count reached: the client's requests in the window, the one that
|
// requests in the window, the one that broke the limit included.
|
||||||
// broke the limit included. These are the requests that counted
|
// These are the requests that counted toward the ban, and the window
|
||||||
// toward the ban, and the window is the time over which they came.
|
// is the time over which they came.
|
||||||
Limit int64 `json:"limit,omitempty"`
|
Limit int64 `json:"limit"`
|
||||||
Window string `json:"window,omitempty"`
|
Window string `json:"window"`
|
||||||
Count float64 `json:"count,omitempty"`
|
Count float64 `json:"count"`
|
||||||
// RuleID and Target are, for a ban for a clear sign of attack, the id
|
// Request is the request that broke the limit.
|
||||||
// of the rule file rule that matched, and its target.
|
|
||||||
RuleID string `json:"rule_id,omitempty"`
|
|
||||||
Target string `json:"target,omitempty"`
|
|
||||||
// Request is the request that broke the limit, or that was the clear
|
|
||||||
// sign of attack.
|
|
||||||
Request Request `json:"request"`
|
Request Request `json:"request"`
|
||||||
// Requests is how many requests the netblock has sent since it was
|
// Requests is how many requests the netblock has sent since it was
|
||||||
// first seen, and Refused how many of them the ban has refused so
|
// first seen, and Refused how many of them the ban has refused so
|
||||||
// far. Both go up with each request the ban refuses.
|
// far. Both go up with each request the ban refuses.
|
||||||
Requests int64 `json:"requests"`
|
Requests int64 `json:"requests"`
|
||||||
Refused int64 `json:"refused"`
|
Refused int64 `json:"refused"`
|
||||||
// EarlierBans is how many bans the netblock had before this one, by
|
// EarlierBans is how many bans the netblock had before this one.
|
||||||
// cause.
|
EarlierBans int `json:"earlier_bans"`
|
||||||
EarlierBans EarlierBans `json:"earlier_bans"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// EarlierBans counts a netblock's bans before a ban, by cause.
|
|
||||||
type EarlierBans struct {
|
|
||||||
Limit int `json:"limit"`
|
|
||||||
Attack int `json:"attack"`
|
|
||||||
Admin int `json:"admin"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Request is a request in a ban's notes. Each text is cut to 256 bytes.
|
// Request is a request in a ban's notes. Each text is cut to 256 bytes.
|
||||||
@@ -150,12 +110,10 @@ type Ledger struct {
|
|||||||
// netblocks holds each banned netblock's bans, oldest first. Check and
|
// netblocks holds each banned netblock's bans, oldest first. Check and
|
||||||
// Find make each netblock they find the most recently seen.
|
// Find make each netblock they find the most recently seen.
|
||||||
netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
|
netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
|
||||||
// held is how many bans netblocks holds whose cause is not CauseAdmin,
|
// held is how many bans netblocks holds, at most rules.MaxBans.
|
||||||
// at most rules.MaxBans.
|
|
||||||
held int
|
held int
|
||||||
// made is how many bans have been made since the start, by cause: by
|
// made is how many bans BanForLimit has made since the start.
|
||||||
// the ledger, and by an admin in an edit of bans.json.
|
made int
|
||||||
made map[string]int
|
|
||||||
// v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6
|
// v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6
|
||||||
// netblocks that have been banned. Check looks for a ban at each of
|
// netblocks that have been banned. Check looks for a ban at each of
|
||||||
// them, so that a ban read from bans.json refuses every client in its
|
// them, so that a ban read from bans.json refuses every client in its
|
||||||
@@ -166,10 +124,9 @@ type Ledger struct {
|
|||||||
|
|
||||||
// New returns a Ledger with no ban yet.
|
// New returns a Ledger with no ban yet.
|
||||||
func New(rules Rules) *Ledger {
|
func New(rules Rules) *Ledger {
|
||||||
// The ledger drops bans itself, and never those whose cause is
|
// Every netblock held has a ban, so there are never more netblocks
|
||||||
// CauseAdmin, however many there are, so the LRU has no limit of its
|
// than rules.MaxBans, and the LRU never drops one itself.
|
||||||
// own: it keeps the netblocks in the order they were last seen.
|
netblocks, err := simplelru.NewLRU[netip.Prefix, *[]Ban](rules.MaxBans, nil)
|
||||||
netblocks, err := simplelru.NewLRU[netip.Prefix, *[]Ban](math.MaxInt, nil)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
panic(err) // NewLRU fails only for a size below one
|
panic(err) // NewLRU fails only for a size below one
|
||||||
}
|
}
|
||||||
@@ -178,21 +135,18 @@ func New(rules Rules) *Ledger {
|
|||||||
rules: rules,
|
rules: rules,
|
||||||
changed: make(chan struct{}, 1),
|
changed: make(chan struct{}, 1),
|
||||||
netblocks: netblocks,
|
netblocks: netblocks,
|
||||||
made: map[string]int{},
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Changed receives a value after a ban is made or made permanent, so that
|
// Changed receives a value after a ban is made, so that bans.json can be
|
||||||
// bans.json can be written. Several changes before it is read leave one
|
// written. Several bans made before it is read leave one value.
|
||||||
// value.
|
|
||||||
func (l *Ledger) Changed() <-chan struct{} {
|
func (l *Ledger) Changed() <-chan struct{} {
|
||||||
return l.changed
|
return l.changed
|
||||||
}
|
}
|
||||||
|
|
||||||
// Check is called for a request from client, at now. It reports whether
|
// Check is called for a request from client, at now. It reports whether
|
||||||
// a ban on a netblock client is in is active, and returns that ban, with
|
// a ban on a netblock client is in is active, and returns that ban, with
|
||||||
// the request counted among those it refused. A ban for a clear sign of
|
// the request counted among those it refused.
|
||||||
// attack is made permanent by the request: the netblock is malicious.
|
|
||||||
func (l *Ledger) Check(client netip.Addr, now time.Time) (Ban, bool) {
|
func (l *Ledger) Check(client netip.Addr, now time.Time) (Ban, bool) {
|
||||||
l.mu.Lock()
|
l.mu.Lock()
|
||||||
defer l.mu.Unlock()
|
defer l.mu.Unlock()
|
||||||
@@ -205,12 +159,6 @@ func (l *Ledger) Check(client netip.Addr, now time.Time) (Ban, bool) {
|
|||||||
ban.Notes.Requests++
|
ban.Notes.Requests++
|
||||||
ban.Notes.Refused++
|
ban.Notes.Refused++
|
||||||
|
|
||||||
if ban.Cause == CauseAttack && !ban.Permanent() {
|
|
||||||
ban.Expires = time.Time{}
|
|
||||||
|
|
||||||
l.markChanged()
|
|
||||||
}
|
|
||||||
|
|
||||||
return *ban, true
|
return *ban, true
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -244,27 +192,54 @@ func activeBan(bans []Ban, now time.Time) *Ban {
|
|||||||
|
|
||||||
// BanForLimit bans netblock at now for a broken limit, with notes, and
|
// BanForLimit bans netblock at now for a broken limit, with notes, and
|
||||||
// returns the ban. A first ban lasts LimitBanDuration. A ban made within
|
// returns the ban. A first ban lasts LimitBanDuration. A ban made within
|
||||||
// LimitBanRepeatWindow after the netblock's ban that ended last, other
|
// LimitBanRepeatWindow after the netblock's ban that ended last lasts
|
||||||
// than one for a clear sign of attack or a lifted one, lasts repeatFactor
|
// repeatFactor times as long as that one. A ban that would be longer
|
||||||
// times as long as that one. A ban that would be longer than
|
// than MaxBanDuration is permanent instead. If a ban on netblock is still
|
||||||
// MaxBanDuration is permanent instead. If a ban on netblock is still
|
|
||||||
// active, as when two of its requests break a limit at once, that ban is
|
// active, as when two of its requests break a limit at once, that ban is
|
||||||
// returned and no other is made. The ledger fills in the notes' Refused
|
// returned and no other is made. The ledger fills in the notes' Refused
|
||||||
// and EarlierBans itself, and gives the ban the reason "requests per
|
// and EarlierBans itself.
|
||||||
// <Window> over the limit of <Limit>", from the notes.
|
|
||||||
func (l *Ledger) BanForLimit(netblock netip.Prefix, now time.Time, notes Notes) Ban {
|
func (l *Ledger) BanForLimit(netblock netip.Prefix, now time.Time, notes Notes) Ban {
|
||||||
reason := fmt.Sprintf("requests per %s over the limit of %d",
|
l.mu.Lock()
|
||||||
notes.Window, notes.Limit)
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
return l.ban(netblock, now, CauseLimit, reason, notes)
|
var last *Ban
|
||||||
}
|
|
||||||
|
|
||||||
// BanForAttack bans netblock at now for a clear sign of attack, with
|
bans, found := l.netblocks.Get(netblock)
|
||||||
// notes, and returns the ban, as BanForLimit does. A first ban lasts
|
if found {
|
||||||
// AttackBanDuration; once the netblock has had one that was not lifted,
|
active := activeBan(*bans, now)
|
||||||
// the next is permanent. Its reason is "matched the rule <RuleID>".
|
if active != nil {
|
||||||
func (l *Ledger) BanForAttack(netblock netip.Prefix, now time.Time, notes Notes) Ban {
|
return *active
|
||||||
return l.ban(netblock, now, CauseAttack, "matched the rule "+notes.RuleID, notes)
|
}
|
||||||
|
|
||||||
|
// No ban is active, so each has an end. A ban an admin adds to
|
||||||
|
// bans.json can start after another and end before it, so the
|
||||||
|
// ban that ended last is looked for among them all.
|
||||||
|
ended := slices.MaxFunc(*bans, func(a, b Ban) int {
|
||||||
|
return a.Expires.Compare(b.Expires)
|
||||||
|
})
|
||||||
|
last = &ended
|
||||||
|
|
||||||
|
// The netblock's first ban held counts the bans it had before that
|
||||||
|
// one, since dropped to make room, and each ban held adds one.
|
||||||
|
notes.EarlierBans = (*bans)[0].Notes.EarlierBans + len(*bans)
|
||||||
|
}
|
||||||
|
|
||||||
|
notes.Request = notes.Request.cut()
|
||||||
|
ban := Ban{
|
||||||
|
Netblock: netblock,
|
||||||
|
Start: now,
|
||||||
|
Expires: l.expiry(last, now),
|
||||||
|
Notes: notes,
|
||||||
|
}
|
||||||
|
l.add(ban)
|
||||||
|
l.made++
|
||||||
|
|
||||||
|
select {
|
||||||
|
case l.changed <- struct{}{}:
|
||||||
|
default: // a value is waiting already
|
||||||
|
}
|
||||||
|
|
||||||
|
return ban
|
||||||
}
|
}
|
||||||
|
|
||||||
// Bans returns the bans held on netblock, oldest first. It is not a
|
// Bans returns the bans held on netblock, oldest first. It is not a
|
||||||
@@ -281,19 +256,17 @@ func (l *Ledger) Bans(netblock netip.Prefix) []Ban {
|
|||||||
return slices.Clone(*bans)
|
return slices.Clone(*bans)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Made returns how many bans for cause have been made since the start:
|
// Made returns how many bans the ledger has made since the start; bans
|
||||||
// for CauseLimit and CauseAttack, by the ledger; for CauseAdmin, by an
|
// read from bans.json are not among them.
|
||||||
// admin in an edit of bans.json, as LoadEdit counts them. The bans read
|
func (l *Ledger) Made() int {
|
||||||
// from bans.json at the start are not among them.
|
|
||||||
func (l *Ledger) Made(cause string) int {
|
|
||||||
l.mu.Lock()
|
l.mu.Lock()
|
||||||
defer l.mu.Unlock()
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
return l.made[cause]
|
return l.made
|
||||||
}
|
}
|
||||||
|
|
||||||
// Count returns how many of the bans held are active at now, and how many
|
// Count returns how many of the bans held are active at now, and how many
|
||||||
// of those are permanent. A lifted ban is neither.
|
// are permanent.
|
||||||
func (l *Ledger) Count(now time.Time) (int, int) {
|
func (l *Ledger) Count(now time.Time) (int, int) {
|
||||||
l.mu.Lock()
|
l.mu.Lock()
|
||||||
defer l.mu.Unlock()
|
defer l.mu.Unlock()
|
||||||
@@ -302,12 +275,10 @@ func (l *Ledger) Count(now time.Time) (int, int) {
|
|||||||
|
|
||||||
for _, bans := range l.netblocks.Values() {
|
for _, bans := range l.netblocks.Values() {
|
||||||
for _, ban := range *bans {
|
for _, ban := range *bans {
|
||||||
if !ban.ActiveAt(now) {
|
if ban.ActiveAt(now) {
|
||||||
continue
|
active++
|
||||||
}
|
}
|
||||||
|
|
||||||
active++
|
|
||||||
|
|
||||||
if ban.Permanent() {
|
if ban.Permanent() {
|
||||||
permanent++
|
permanent++
|
||||||
}
|
}
|
||||||
@@ -335,142 +306,30 @@ func (l *Ledger) Snapshot() []Ban {
|
|||||||
return held
|
return held
|
||||||
}
|
}
|
||||||
|
|
||||||
// Load puts bans read from bans.json at the start into the ledger, in
|
// Load puts bans read from bans.json into the ledger, in place of the
|
||||||
// place of the bans it holds, in the order they started, so that a
|
// bans it holds, in the order they started, so that a netblock whose last
|
||||||
// netblock whose last ban started latest counts as the most recently
|
// ban started latest counts as the most recently seen. Each netblock is
|
||||||
// seen. A ban without a cause is an admin's, and gets CauseAdmin. Each
|
// masked to its length, so that 203.0.113.9/24 is 203.0.113.0/24, and
|
||||||
// netblock is masked to its length, so that 203.0.113.9/24 is
|
// each text in the notes is cut to 256 bytes. Past MaxBans the earliest
|
||||||
// 203.0.113.0/24, and each text in the notes is cut to 256 bytes. Past
|
// bans are dropped, as when they are made.
|
||||||
// MaxBans the earliest bans whose cause is not CauseAdmin are dropped, as
|
|
||||||
// when they are made.
|
|
||||||
func (l *Ledger) Load(bans []Ban) {
|
func (l *Ledger) Load(bans []Ban) {
|
||||||
l.mu.Lock()
|
|
||||||
defer l.mu.Unlock()
|
|
||||||
|
|
||||||
l.load(bans)
|
|
||||||
}
|
|
||||||
|
|
||||||
// LoadEdit is Load for an admin's edit of bans.json, taken in while
|
|
||||||
// smallwebwaf runs. Each ban in it whose cause is CauseAdmin, and which
|
|
||||||
// the ledger did not hold, with the same netblock and start, is one the
|
|
||||||
// admin made, and is counted among the bans made.
|
|
||||||
func (l *Ledger) LoadEdit(bans []Ban) {
|
|
||||||
l.mu.Lock()
|
|
||||||
defer l.mu.Unlock()
|
|
||||||
|
|
||||||
l.made[CauseAdmin] += l.load(bans)
|
|
||||||
}
|
|
||||||
|
|
||||||
// load does what Load describes, and returns how many of bans are bans
|
|
||||||
// whose cause is CauseAdmin that the ledger did not hold before.
|
|
||||||
func (l *Ledger) load(bans []Ban) int {
|
|
||||||
bans = slices.Clone(bans)
|
bans = slices.Clone(bans)
|
||||||
added := 0
|
|
||||||
|
|
||||||
for i := range bans {
|
|
||||||
ban := &bans[i]
|
|
||||||
ban.Netblock = ban.Netblock.Masked()
|
|
||||||
ban.Notes.Request = ban.Notes.Request.cut()
|
|
||||||
|
|
||||||
if ban.Cause == "" {
|
|
||||||
ban.Cause = CauseAdmin
|
|
||||||
}
|
|
||||||
|
|
||||||
if ban.Cause == CauseAdmin && !l.holds(ban.Netblock, ban.Start) {
|
|
||||||
added++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
slices.SortStableFunc(bans, func(a, b Ban) int {
|
slices.SortStableFunc(bans, func(a, b Ban) int {
|
||||||
return a.Start.Compare(b.Start)
|
return a.Start.Compare(b.Start)
|
||||||
})
|
})
|
||||||
|
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
l.netblocks.Purge()
|
l.netblocks.Purge()
|
||||||
l.held = 0
|
l.held = 0
|
||||||
l.v4Lengths, l.v6Lengths = nil, nil
|
l.v4Lengths, l.v6Lengths = nil, nil
|
||||||
|
|
||||||
for _, ban := range bans {
|
for _, ban := range bans {
|
||||||
|
ban.Netblock = ban.Netblock.Masked()
|
||||||
|
ban.Notes.Request = ban.Notes.Request.cut()
|
||||||
l.add(ban)
|
l.add(ban)
|
||||||
}
|
}
|
||||||
|
|
||||||
return added
|
|
||||||
}
|
|
||||||
|
|
||||||
// holds reports whether the ledger holds a ban on netblock that started
|
|
||||||
// at start.
|
|
||||||
func (l *Ledger) holds(netblock netip.Prefix, start time.Time) bool {
|
|
||||||
bans, found := l.netblocks.Peek(netblock)
|
|
||||||
|
|
||||||
return found && slices.ContainsFunc(*bans, func(ban Ban) bool {
|
|
||||||
return ban.Start.Equal(start)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// ban bans netblock at now for cause, with reason and notes, as
|
|
||||||
// BanForLimit and BanForAttack describe, and returns the ban.
|
|
||||||
func (l *Ledger) ban(
|
|
||||||
netblock netip.Prefix, now time.Time, cause, reason string, notes Notes,
|
|
||||||
) Ban {
|
|
||||||
l.mu.Lock()
|
|
||||||
defer l.mu.Unlock()
|
|
||||||
|
|
||||||
// held are the netblock's bans, none of them active.
|
|
||||||
var held []Ban
|
|
||||||
|
|
||||||
bans, found := l.netblocks.Get(netblock)
|
|
||||||
if found {
|
|
||||||
active := activeBan(*bans, now)
|
|
||||||
if active != nil {
|
|
||||||
return *active
|
|
||||||
}
|
|
||||||
|
|
||||||
held = *bans
|
|
||||||
notes.EarlierBans = earlierBans(held)
|
|
||||||
}
|
|
||||||
|
|
||||||
notes.Request = notes.Request.cut()
|
|
||||||
ban := Ban{Netblock: netblock, Start: now, Cause: cause, Reason: reason, Notes: notes}
|
|
||||||
|
|
||||||
if cause == CauseAttack {
|
|
||||||
ban.Expires = l.attackExpiry(held, now)
|
|
||||||
} else {
|
|
||||||
ban.Expires = l.limitExpiry(held, now)
|
|
||||||
}
|
|
||||||
|
|
||||||
l.add(ban)
|
|
||||||
l.made[cause]++
|
|
||||||
l.markChanged()
|
|
||||||
|
|
||||||
return ban
|
|
||||||
}
|
|
||||||
|
|
||||||
// earlierBans returns how many bans a netblock with the bans held, oldest
|
|
||||||
// first, has had, by cause: the first ban held counts the bans the
|
|
||||||
// netblock had before that one, since dropped to make room, and each ban
|
|
||||||
// held adds one.
|
|
||||||
func earlierBans(held []Ban) EarlierBans {
|
|
||||||
earlier := held[0].Notes.EarlierBans
|
|
||||||
|
|
||||||
for _, ban := range held {
|
|
||||||
switch ban.Cause {
|
|
||||||
case CauseLimit:
|
|
||||||
earlier.Limit++
|
|
||||||
case CauseAttack:
|
|
||||||
earlier.Attack++
|
|
||||||
case CauseAdmin:
|
|
||||||
earlier.Admin++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return earlier
|
|
||||||
}
|
|
||||||
|
|
||||||
// markChanged has Changed receive a value, unless one is waiting already.
|
|
||||||
func (l *Ledger) markChanged() {
|
|
||||||
select {
|
|
||||||
case l.changed <- struct{}{}:
|
|
||||||
default: // a value is waiting already
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// active returns the ban active at now on a netblock client is in, or
|
// active returns the ban active at now on a netblock client is in, or
|
||||||
@@ -497,11 +356,9 @@ func (l *Ledger) active(client netip.Addr, now time.Time) *Ban {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// add adds ban to its netblock's bans, after the last, and makes its
|
// add adds ban to its netblock's bans, after the last, and makes its
|
||||||
// netblock the most recently seen. With MaxBans held, it drops one first,
|
// netblock the most recently seen. With MaxBans held, it drops one first.
|
||||||
// unless ban's cause is CauseAdmin, which does not count toward MaxBans.
|
|
||||||
func (l *Ledger) add(ban Ban) {
|
func (l *Ledger) add(ban Ban) {
|
||||||
counted := ban.Cause != CauseAdmin
|
if l.held == l.rules.MaxBans {
|
||||||
if counted && l.held == l.rules.MaxBans {
|
|
||||||
l.dropOne()
|
l.dropOne()
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -514,10 +371,7 @@ func (l *Ledger) add(ban Ban) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
*bans = append(*bans, ban)
|
*bans = append(*bans, ban)
|
||||||
|
l.held++
|
||||||
if counted {
|
|
||||||
l.held++
|
|
||||||
}
|
|
||||||
|
|
||||||
lengths := &l.v6Lengths
|
lengths := &l.v6Lengths
|
||||||
if ban.Netblock.Addr().Is4() {
|
if ban.Netblock.Addr().Is4() {
|
||||||
@@ -529,24 +383,12 @@ func (l *Ledger) add(ban Ban) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// limitExpiry returns when a ban for a broken limit made at now ends, or
|
// expiry returns when a ban for a broken limit made at now ends, or zero
|
||||||
// zero when it is permanent. held are the netblock's bans, none of them
|
// when it is permanent. last is the netblock's ban that ended last, or nil
|
||||||
// active, of which the one that ended last, other than a ban for a clear
|
// when it has none.
|
||||||
// sign of attack or a lifted one, can make the new ban longer. A ban an
|
func (l *Ledger) expiry(last *Ban, now time.Time) time.Time {
|
||||||
// admin adds to bans.json can start after another and end before it, so
|
|
||||||
// that one is looked for among them all.
|
|
||||||
func (l *Ledger) limitExpiry(held []Ban, now time.Time) time.Time {
|
|
||||||
length := l.rules.LimitBanDuration
|
length := l.rules.LimitBanDuration
|
||||||
|
|
||||||
var last *Ban
|
|
||||||
|
|
||||||
for i, ban := range held {
|
|
||||||
if ban.Cause != CauseAttack && ban.Lifted.IsZero() &&
|
|
||||||
(last == nil || ban.Expires.After(last.Expires)) {
|
|
||||||
last = &held[i]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if last != nil && now.Sub(last.Expires) <= l.rules.LimitBanRepeatWindow {
|
if last != nil && now.Sub(last.Expires) <= l.rules.LimitBanRepeatWindow {
|
||||||
lastLength := last.Expires.Sub(last.Start)
|
lastLength := last.Expires.Sub(last.Start)
|
||||||
// This is repeatFactor * lastLength > MaxBanDuration, written so
|
// This is repeatFactor * lastLength > MaxBanDuration, written so
|
||||||
@@ -565,53 +407,17 @@ func (l *Ledger) limitExpiry(held []Ban, now time.Time) time.Time {
|
|||||||
return now.Add(length)
|
return now.Add(length)
|
||||||
}
|
}
|
||||||
|
|
||||||
// attackExpiry returns when a ban for a clear sign of attack made at now
|
// dropOne drops the earliest ban of the netblock that has gone longest
|
||||||
// ends. held are the netblock's bans, none of them active: if one of them
|
// without a request, and the netblock with it if that was its only ban.
|
||||||
// is for a clear sign of attack too, and was not lifted, the new ban is
|
|
||||||
// permanent, and its end zero; otherwise it ends AttackBanDuration later.
|
|
||||||
func (l *Ledger) attackExpiry(held []Ban, now time.Time) time.Time {
|
|
||||||
for _, ban := range held {
|
|
||||||
if ban.Cause == CauseAttack && ban.Lifted.IsZero() {
|
|
||||||
return time.Time{}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return now.Add(l.rules.AttackBanDuration)
|
|
||||||
}
|
|
||||||
|
|
||||||
// dropOne drops the earliest ban whose cause is not CauseAdmin of the
|
|
||||||
// netblock that has gone longest without a request, of those that hold
|
|
||||||
// such a ban, and the netblock with it if that was its only ban. It is
|
|
||||||
// called with at least one such ban held. It looks at each netblock once
|
|
||||||
// at most, and drops nothing when none holds such a ban.
|
|
||||||
func (l *Ledger) dropOne() {
|
func (l *Ledger) dropOne() {
|
||||||
for range l.netblocks.Len() {
|
netblock, bans, _ := l.netblocks.GetOldest()
|
||||||
netblock, bans, _ := l.netblocks.GetOldest()
|
if len(*bans) == 1 {
|
||||||
|
l.netblocks.Remove(netblock)
|
||||||
i := slices.IndexFunc(*bans, func(ban Ban) bool {
|
} else {
|
||||||
return ban.Cause != CauseAdmin
|
*bans = slices.Delete(*bans, 0, 1)
|
||||||
})
|
|
||||||
if i < 0 {
|
|
||||||
// Its bans are all an admin's, and never dropped. Get makes
|
|
||||||
// it the most recently seen, so that the next netblock is
|
|
||||||
// looked at; when it was seen matters only for dropping a
|
|
||||||
// ban, and a ban added to it makes it the most recently seen
|
|
||||||
// anyway.
|
|
||||||
l.netblocks.Get(netblock)
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(*bans) == 1 {
|
|
||||||
l.netblocks.Remove(netblock)
|
|
||||||
} else {
|
|
||||||
*bans = slices.Delete(*bans, i, i+1)
|
|
||||||
}
|
|
||||||
|
|
||||||
l.held--
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
l.held--
|
||||||
}
|
}
|
||||||
|
|
||||||
// cut returns r with each text cut to maxTextBytes and copied, so that
|
// cut returns r with each text cut to maxTextBytes and copied, so that
|
||||||
|
|||||||
+7
-103
@@ -24,9 +24,8 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
|
|||||||
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
|
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
|
||||||
|
|
||||||
length := time.Duration(hours) * time.Hour
|
length := time.Duration(hours) * time.Hour
|
||||||
if !ban.Expires.Equal(now.Add(length)) ||
|
if !ban.Expires.Equal(now.Add(length)) || ban.Notes.EarlierBans != i {
|
||||||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: i}) {
|
t.Fatalf("ban %d lasts %s with %d earlier bans, want %d hours and %d",
|
||||||
t.Fatalf("ban %d lasts %s with earlier bans %+v, want %d hours and %d for a limit",
|
|
||||||
i+1, ban.Expires.Sub(now), ban.Notes.EarlierBans, hours, i)
|
i+1, ban.Expires.Sub(now), ban.Notes.EarlierBans, hours, i)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -67,9 +66,8 @@ func TestRepeatWindowRunsOut(t *testing.T) {
|
|||||||
first := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
first := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
||||||
second := ledger.BanForLimit(netblock, first.Expires.Add(tc.gap), bans.Notes{})
|
second := ledger.BanForLimit(netblock, first.Expires.Add(tc.gap), bans.Notes{})
|
||||||
|
|
||||||
if second.Expires.Sub(second.Start) != tc.want ||
|
if second.Expires.Sub(second.Start) != tc.want || second.Notes.EarlierBans != 1 {
|
||||||
second.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
|
t.Errorf("second ban lasts %s with %d earlier bans, want %s and 1",
|
||||||
t.Errorf("second ban lasts %s with earlier bans %+v, want %s and 1 for a limit",
|
|
||||||
second.Expires.Sub(second.Start), second.Notes.EarlierBans, tc.want)
|
second.Expires.Sub(second.Start), second.Notes.EarlierBans, tc.want)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
@@ -237,102 +235,9 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
|
|||||||
second := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
|
second := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
|
||||||
|
|
||||||
held := ledger.Bans(netblock)
|
held := ledger.Bans(netblock)
|
||||||
if len(held) != 1 || held[0] != second ||
|
if len(held) != 1 || held[0] != second || held[0].Notes.EarlierBans != 1 {
|
||||||
held[0].Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
|
t.Errorf("the ledger holds %+v, want only the second ban, with 1 earlier ban",
|
||||||
t.Errorf("the ledger holds %+v, want only the second ban, "+
|
held)
|
||||||
"with 1 earlier ban for a limit", held)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRequestDuringAnAttackBanMakesItPermanent(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ledger := bans.New(defaultRules())
|
|
||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
|
||||||
notes := bans.Notes{RuleID: "env-file", Target: "path"}
|
|
||||||
|
|
||||||
ban := ledger.BanForAttack(netblock, midnight(), notes)
|
|
||||||
if !ban.Expires.Equal(midnight().Add(7*day)) || ban.Cause != bans.CauseAttack ||
|
|
||||||
ban.Notes.RuleID != "env-file" || ledger.Made(bans.CauseAttack) != 1 ||
|
|
||||||
ledger.Made(bans.CauseLimit) != 0 {
|
|
||||||
t.Fatalf("the ban is %+v, with %d made for an attack and %d for a limit, "+
|
|
||||||
"want one for an attack, of seven days", ban,
|
|
||||||
ledger.Made(bans.CauseAttack), ledger.Made(bans.CauseLimit))
|
|
||||||
}
|
|
||||||
|
|
||||||
wantChanged(t, ledger, true)
|
|
||||||
|
|
||||||
// In observe mode the ban refuses nothing, and stays as it is.
|
|
||||||
got, _ := ledger.Find(netblock.Addr(), midnight().Add(time.Hour))
|
|
||||||
if got.Permanent() {
|
|
||||||
t.Fatal("a request found under the ban made it permanent")
|
|
||||||
}
|
|
||||||
|
|
||||||
// A request it refuses makes it permanent, and bans.json due.
|
|
||||||
got, _ = ledger.Check(netblock.Addr(), midnight().Add(time.Hour))
|
|
||||||
if !got.Permanent() || !ledger.Bans(netblock)[0].Permanent() {
|
|
||||||
t.Fatalf("after a request during the ban, it is %+v, want it permanent", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantChanged(t, ledger, true)
|
|
||||||
|
|
||||||
_, banned := ledger.Check(netblock.Addr(), midnight().Add(100*365*day))
|
|
||||||
if !banned {
|
|
||||||
t.Error("the permanent ban ended")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ledger := bans.New(defaultRules())
|
|
||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
|
||||||
|
|
||||||
// A ban for a broken limit before does not count.
|
|
||||||
first := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
|
||||||
second := ledger.BanForAttack(netblock, first.Expires, bans.Notes{})
|
|
||||||
|
|
||||||
if second.Expires.Sub(second.Start) != 7*day {
|
|
||||||
t.Fatalf("the first ban for an attack lasts %s, want 7 days",
|
|
||||||
second.Expires.Sub(second.Start))
|
|
||||||
}
|
|
||||||
|
|
||||||
// Once that has run out without a request, the netblock is served, and
|
|
||||||
// its next clear sign of attack bans it for good.
|
|
||||||
_, banned := ledger.Check(netblock.Addr(), second.Expires)
|
|
||||||
if banned {
|
|
||||||
t.Fatal("the ban did not end")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Its notes show the earlier ban for an attack that makes it permanent,
|
|
||||||
// beside the one for a limit.
|
|
||||||
third := ledger.BanForAttack(netblock, second.Expires.Add(30*day), bans.Notes{})
|
|
||||||
if !third.Permanent() ||
|
|
||||||
third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Attack: 1}) {
|
|
||||||
t.Errorf("the next ban for an attack is %+v, want a permanent one, "+
|
|
||||||
"with 1 earlier ban for a limit and 1 for an attack", third)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAttackBanDoesNotLengthenTheNextBanForALimit(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ledger := bans.New(defaultRules())
|
|
||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
|
||||||
|
|
||||||
// Three times the seven days would be permanent; a limit broken as the
|
|
||||||
// ban for an attack ends bans for an hour, as a first broken limit does.
|
|
||||||
attack := ledger.BanForAttack(netblock, midnight(), bans.Notes{})
|
|
||||||
limit := ledger.BanForLimit(netblock, attack.Expires, bans.Notes{})
|
|
||||||
|
|
||||||
if limit.Expires.Sub(limit.Start) != time.Hour || limit.Cause != bans.CauseLimit {
|
|
||||||
t.Errorf("the ban for a limit is %+v, want one of an hour", limit)
|
|
||||||
}
|
|
||||||
|
|
||||||
// And a request during the ban for a limit leaves it as it is.
|
|
||||||
got, _ := ledger.Check(netblock.Addr(), limit.Start)
|
|
||||||
if got.Permanent() {
|
|
||||||
t.Error("a request during a ban for a limit made it permanent")
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -364,7 +269,6 @@ func defaultRules() bans.Rules {
|
|||||||
LimitBanDuration: time.Hour,
|
LimitBanDuration: time.Hour,
|
||||||
LimitBanRepeatWindow: day,
|
LimitBanRepeatWindow: day,
|
||||||
MaxBanDuration: 7 * day,
|
MaxBanDuration: 7 * day,
|
||||||
AttackBanDuration: 7 * day,
|
|
||||||
MaxBans: 5000,
|
MaxBans: 5000,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -82,9 +82,8 @@ func TestLoadedBansCarryOn(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
again := after.BanForLimit(netblock, ban.Expires, bans.Notes{})
|
again := after.BanForLimit(netblock, ban.Expires, bans.Notes{})
|
||||||
if again.Expires.Sub(again.Start) != 3*time.Hour ||
|
if again.Expires.Sub(again.Start) != 3*time.Hour || again.Notes.EarlierBans != 1 {
|
||||||
again.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
|
t.Errorf("the next ban lasts %s with %d earlier bans, want 3h and 1",
|
||||||
t.Errorf("the next ban lasts %s with earlier bans %+v, want 3h and 1 for a limit",
|
|
||||||
again.Expires.Sub(again.Start), again.Notes.EarlierBans)
|
again.Expires.Sub(again.Start), again.Notes.EarlierBans)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -173,14 +172,13 @@ func TestNextBanWorkedOutFromTheBanThatEndedLast(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
// A 9-hour ban smallwebwaf made, the third in a row, and an admin's
|
// A 9-hour ban smallwebwaf made, the third in a row, and an admin's
|
||||||
// 1-hour ban added to bans.json over it, with no cause and no notes.
|
// 1-hour ban added to bans.json over it, with no notes.
|
||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
nineHours := bans.Ban{
|
nineHours := bans.Ban{
|
||||||
Netblock: netblock,
|
Netblock: netblock,
|
||||||
Start: midnight(),
|
Start: midnight(),
|
||||||
Expires: midnight().Add(9 * time.Hour),
|
Expires: midnight().Add(9 * time.Hour),
|
||||||
Cause: bans.CauseLimit,
|
Notes: bans.Notes{EarlierBans: 2},
|
||||||
Notes: bans.Notes{EarlierBans: bans.EarlierBans{Limit: 2}},
|
|
||||||
}
|
}
|
||||||
admins := bans.Ban{
|
admins := bans.Ban{
|
||||||
Netblock: netblock,
|
Netblock: netblock,
|
||||||
@@ -193,12 +191,10 @@ func TestNextBanWorkedOutFromTheBanThatEndedLast(t *testing.T) {
|
|||||||
|
|
||||||
// Once both have ended, a limit broken within the repeat window bans
|
// Once both have ended, a limit broken within the repeat window bans
|
||||||
// for three times the 9 hours, and the notes count the two bans
|
// for three times the 9 hours, and the notes count the two bans
|
||||||
// before the 9-hour one and it, for a limit, and the admin's.
|
// before the 9-hour one, it, and the admin's.
|
||||||
ban := ledger.BanForLimit(netblock, nineHours.Expires.Add(time.Hour), bans.Notes{})
|
ban := ledger.BanForLimit(netblock, nineHours.Expires.Add(time.Hour), bans.Notes{})
|
||||||
if ban.Expires.Sub(ban.Start) != 27*time.Hour ||
|
if ban.Expires.Sub(ban.Start) != 27*time.Hour || ban.Notes.EarlierBans != 4 {
|
||||||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 3, Admin: 1}) {
|
t.Errorf("the next ban lasts %s with %d earlier bans, want 27h and 4",
|
||||||
t.Errorf("the next ban lasts %s with earlier bans %+v, "+
|
|
||||||
"want 27h, 3 for a limit and 1 an admin's",
|
|
||||||
ban.Expires.Sub(ban.Start), ban.Notes.EarlierBans)
|
ban.Expires.Sub(ban.Start), ban.Notes.EarlierBans)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -207,15 +203,10 @@ func TestLoadKeepsAtMostMaxBansDroppingTheEarliest(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
// bans.json lists the bans by netblock, not in the order they began.
|
// bans.json lists the bans by netblock, not in the order they began.
|
||||||
later := bans.Ban{
|
later := bans.Ban{Netblock: netip.MustParsePrefix("203.0.113.1/32"), Start: midnight()}
|
||||||
Netblock: netip.MustParsePrefix("203.0.113.1/32"),
|
|
||||||
Start: midnight(),
|
|
||||||
Cause: bans.CauseLimit,
|
|
||||||
}
|
|
||||||
earlier := bans.Ban{
|
earlier := bans.Ban{
|
||||||
Netblock: netip.MustParsePrefix("203.0.113.2/32"),
|
Netblock: netip.MustParsePrefix("203.0.113.2/32"),
|
||||||
Start: midnight().Add(-time.Hour),
|
Start: midnight().Add(-time.Hour),
|
||||||
Cause: bans.CauseLimit,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
rules := defaultRules()
|
rules := defaultRules()
|
||||||
@@ -237,17 +228,9 @@ func TestLoadReplacesTheBansHeld(t *testing.T) {
|
|||||||
rules := defaultRules()
|
rules := defaultRules()
|
||||||
rules.MaxBans = 3
|
rules.MaxBans = 3
|
||||||
ledger := bans.New(rules)
|
ledger := bans.New(rules)
|
||||||
kept := bans.Ban{
|
kept := bans.Ban{Netblock: netip.MustParsePrefix("2001:db8::/64"), Start: midnight()}
|
||||||
Netblock: netip.MustParsePrefix("2001:db8::/64"),
|
|
||||||
Start: midnight(),
|
|
||||||
Cause: bans.CauseLimit,
|
|
||||||
}
|
|
||||||
ledger.Load([]bans.Ban{
|
ledger.Load([]bans.Ban{
|
||||||
{
|
{Netblock: netip.MustParsePrefix("203.0.113.0/24"), Start: midnight()},
|
||||||
Netblock: netip.MustParsePrefix("203.0.113.0/24"),
|
|
||||||
Start: midnight(),
|
|
||||||
Cause: bans.CauseLimit,
|
|
||||||
},
|
|
||||||
kept,
|
kept,
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|||||||
+16
-340
@@ -1,11 +1,9 @@
|
|||||||
// Package config reads smallwebwaf's settings. Every setting is an
|
// Package config reads smallwebwaf's settings. Every setting is an
|
||||||
// environment variable whose name starts with SWWAF_, or a file such a
|
// environment variable whose name starts with SWWAF_, every setting has a
|
||||||
// variable names, every setting has a default, and this package is the
|
// default, and this package is the one place they are read.
|
||||||
// one place they are read.
|
|
||||||
package config
|
package config
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"crypto/x509"
|
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
@@ -14,15 +12,12 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"net/url"
|
"net/url"
|
||||||
"os"
|
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
"unicode/utf8"
|
"unicode/utf8"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// Config is smallwebwaf's settings. A timeout, size or rate limit of zero
|
// Config is smallwebwaf's settings. A timeout, size or rate limit of zero
|
||||||
@@ -32,14 +27,10 @@ type Config struct {
|
|||||||
ListenAddr string
|
ListenAddr string
|
||||||
// UpstreamURL is the app (SWWAF_UPSTREAM_URL).
|
// UpstreamURL is the app (SWWAF_UPSTREAM_URL).
|
||||||
UpstreamURL *url.URL
|
UpstreamURL *url.URL
|
||||||
// InstanceName is the name each request log line gives as instance
|
|
||||||
// (SWWAF_INSTANCE_NAME), by default the host's name, which docker sets
|
|
||||||
// to the first 12 characters of the container's id.
|
|
||||||
InstanceName string
|
|
||||||
// Observe is true in observe mode, when SWWAF_MODE is observe rather
|
// Observe is true in observe mode, when SWWAF_MODE is observe rather
|
||||||
// than enforce: a request that SWWAF_DENY_NETS, a ban, the country
|
// than enforce: a request that SWWAF_DENY_NETS, a ban, the country
|
||||||
// lists, a rate limit or a rule would refuse is passed to the app
|
// lists or a rate limit would refuse is passed to the app instead, and
|
||||||
// instead, and no ban is made.
|
// no ban is made.
|
||||||
Observe bool
|
Observe bool
|
||||||
// TrustedProxies are the netblocks whose X-Forwarded-For is
|
// TrustedProxies are the netblocks whose X-Forwarded-For is
|
||||||
// believed (SWWAF_TRUSTED_PROXIES).
|
// believed (SWWAF_TRUSTED_PROXIES).
|
||||||
@@ -98,8 +89,7 @@ type Config struct {
|
|||||||
// BanResponse is the status a refused client is answered with, 403
|
// BanResponse is the status a refused client is answered with, 403
|
||||||
// or 429, or 0 to close the connection without an answer
|
// or 429, or 0 to close the connection without an answer
|
||||||
// (SWWAF_BAN_RESPONSE). It answers a banned client, a request that
|
// (SWWAF_BAN_RESPONSE). It answers a banned client, a request that
|
||||||
// breaks a rate limit or matches a ban rule, SWWAF_DENY_NETS and the
|
// breaks a rate limit, SWWAF_DENY_NETS and the country lists.
|
||||||
// country lists.
|
|
||||||
BanResponse int
|
BanResponse int
|
||||||
// LimitBanDuration is the ban for a first broken rate limit
|
// LimitBanDuration is the ban for a first broken rate limit
|
||||||
// (SWWAF_LIMIT_BAN_DURATION). A limit broken again within
|
// (SWWAF_LIMIT_BAN_DURATION). A limit broken again within
|
||||||
@@ -110,9 +100,6 @@ type Config struct {
|
|||||||
LimitBanDuration time.Duration
|
LimitBanDuration time.Duration
|
||||||
LimitBanRepeatWindow time.Duration
|
LimitBanRepeatWindow time.Duration
|
||||||
MaxBanDuration time.Duration
|
MaxBanDuration time.Duration
|
||||||
// AttackBanDuration is the ban for a first clear sign of attack
|
|
||||||
// (SWWAF_ATTACK_BAN_DURATION). It cannot be off.
|
|
||||||
AttackBanDuration time.Duration
|
|
||||||
// MaxBans is the most bans held (SWWAF_MAX_BANS).
|
// MaxBans is the most bans held (SWWAF_MAX_BANS).
|
||||||
MaxBans int
|
MaxBans int
|
||||||
// BanScopeV4Prefix is the length of the netblock around an IPv4
|
// BanScopeV4Prefix is the length of the netblock around an IPv4
|
||||||
@@ -126,37 +113,15 @@ type Config struct {
|
|||||||
StateDir string
|
StateDir string
|
||||||
StateWriteDelay time.Duration
|
StateWriteDelay time.Duration
|
||||||
StateCounterInterval time.Duration
|
StateCounterInterval time.Duration
|
||||||
// LogRequestHeaders are the request headers whose values the request
|
|
||||||
// log gives, in lower case (SWWAF_LOG_REQUEST_HEADERS).
|
|
||||||
LogRequestHeaders []string
|
|
||||||
// MetricsToken is the bearer token a scraper sends for the metrics
|
// MetricsToken is the bearer token a scraper sends for the metrics
|
||||||
// (SWWAF_METRICS_TOKEN), "" while it is unset and the metrics are off.
|
// (SWWAF_METRICS_TOKEN), "" while it is unset and the metrics are off.
|
||||||
// MetricsTopN is how many countries get series of their own in the
|
// MetricsTopN is how many countries get series of their own in the
|
||||||
// metrics (SWWAF_METRICS_TOP_N).
|
// metrics (SWWAF_METRICS_TOP_N).
|
||||||
MetricsToken string
|
MetricsToken string
|
||||||
MetricsTopN int
|
MetricsTopN int
|
||||||
// RulesDir is the directory of the rule files (SWWAF_RULES_DIR), read
|
|
||||||
// unless RulesEnabled is false (SWWAF_RULES_ENABLED).
|
|
||||||
RulesDir string
|
|
||||||
RulesEnabled bool
|
|
||||||
// LogRemoteURL is where every line on stdout is also sent
|
|
||||||
// (SWWAF_LOG_REMOTE_URL), nil while it is unset and nothing is sent.
|
|
||||||
// LogRemoteTLSCAs are the certificates a syslog+tls endpoint's
|
|
||||||
// certificate must chain to (SWWAF_LOG_REMOTE_TLS_CA_FILE), nil while
|
|
||||||
// it is unset and the host's own are used. LogRemoteBuffer is the most
|
|
||||||
// lines held while they wait to be sent (SWWAF_LOG_REMOTE_BUFFER).
|
|
||||||
// LogRemoteFacility is the number of the syslog facility
|
|
||||||
// (SWWAF_LOG_REMOTE_FACILITY), and LogRemoteAppName the APP-NAME
|
|
||||||
// (SWWAF_LOG_REMOTE_APP_NAME, by default InstanceName), of the records
|
|
||||||
// the lines are sent in.
|
|
||||||
LogRemoteURL *url.URL
|
|
||||||
LogRemoteTLSCAs *x509.CertPool
|
|
||||||
LogRemoteBuffer int
|
|
||||||
LogRemoteFacility int
|
|
||||||
LogRemoteAppName string
|
|
||||||
|
|
||||||
// settings are the values read, as given or by default, and the
|
// settings are the values read, as given or by default, for the
|
||||||
// files they were read from, for the log line at start.
|
// log line at start.
|
||||||
settings []slog.Attr
|
settings []slog.Attr
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -174,10 +139,6 @@ const (
|
|||||||
minTokenLength = 32
|
minTokenLength = 32
|
||||||
// masked is what the log shows for a token that is set.
|
// masked is what the log shows for a token that is set.
|
||||||
masked = "********"
|
masked = "********"
|
||||||
// defaultListenAddr and defaultUpstreamURL are the defaults of
|
|
||||||
// SWWAF_LISTEN_ADDR and SWWAF_UPSTREAM_URL.
|
|
||||||
defaultListenAddr = ":8080"
|
|
||||||
defaultUpstreamURL = "http://127.0.0.1:8081"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
@@ -198,11 +159,6 @@ var (
|
|||||||
"such as http://127.0.0.1:8081")
|
"such as http://127.0.0.1:8081")
|
||||||
errNotCountry = errors.New(
|
errNotCountry = errors.New(
|
||||||
"is not a two-letter country code such as de or kp")
|
"is not a two-letter country code such as de or kp")
|
||||||
errNotHeaderName = errors.New(
|
|
||||||
"is not a header name such as accept-language")
|
|
||||||
errHeaderTakenOut = errors.New(
|
|
||||||
"is taken out of every request by Go's HTTP server, so it can never " +
|
|
||||||
"be logged")
|
|
||||||
errOnBothLists = errors.New("is in SWWAF_DENIED_COUNTRIES too")
|
errOnBothLists = errors.New("is in SWWAF_DENIED_COUNTRIES too")
|
||||||
errNotOver4K = errors.New("is not a size of more than 4K, such as 32K")
|
errNotOver4K = errors.New("is not a size of more than 4K, such as 32K")
|
||||||
errNotDurationAboveZero = errors.New(
|
errNotDurationAboveZero = errors.New(
|
||||||
@@ -218,29 +174,16 @@ var (
|
|||||||
errNotMode = errors.New("is not enforce or observe")
|
errNotMode = errors.New("is not enforce or observe")
|
||||||
errNotPathPrefix = errors.New(
|
errNotPathPrefix = errors.New(
|
||||||
"is not a path prefix starting with /, such as /assets/")
|
"is not a path prefix starting with /, such as /assets/")
|
||||||
errNotBoolean = errors.New("is not true or false")
|
|
||||||
errNotLogRemoteURL = errors.New(
|
|
||||||
"is not syslog+udp, syslog+tcp or syslog+tls with a host and a port, " +
|
|
||||||
"and nothing more, such as syslog+tls://logs.example:6514")
|
|
||||||
errNoCertificate = errors.New("holds no PEM certificate")
|
|
||||||
errNotFacility = errors.New("is not a syslog facility such as local0 or daemon")
|
|
||||||
errNotAppName = errors.New(
|
|
||||||
"is not 1 to 48 printable ASCII characters without a space, such as gitea")
|
|
||||||
errSetTwice = errors.New("set only one of them")
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// FromEnvironment reads the settings with lookupEnv, normally
|
// FromEnvironment reads the settings with lookupEnv, normally
|
||||||
// os.LookupEnv. A setting may instead be given as a file: the variable
|
// os.LookupEnv. A setting that is not set takes its default. A setting
|
||||||
// named by the setting's name with _FILE added names the file, which is
|
// that is set but invalid is an error that names it.
|
||||||
// read now (see lookup). A setting that is not set takes its default. A
|
|
||||||
// setting that is set but invalid is an error that names it.
|
|
||||||
func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
||||||
env := &environment{lookupEnv: lookupEnv}
|
env := &environment{lookupEnv: lookupEnv}
|
||||||
hostname, _ := os.Hostname() // "" when the host has no name to give
|
|
||||||
cfg := &Config{
|
cfg := &Config{
|
||||||
ListenAddr: env.address("SWWAF_LISTEN_ADDR", defaultListenAddr),
|
ListenAddr: env.address("SWWAF_LISTEN_ADDR", ":8080"),
|
||||||
UpstreamURL: env.appURL("SWWAF_UPSTREAM_URL", defaultUpstreamURL),
|
UpstreamURL: env.appURL("SWWAF_UPSTREAM_URL", "http://127.0.0.1:8081"),
|
||||||
InstanceName: env.value("SWWAF_INSTANCE_NAME", hostname),
|
|
||||||
Observe: env.observe("SWWAF_MODE", "enforce"),
|
Observe: env.observe("SWWAF_MODE", "enforce"),
|
||||||
TrustedProxies: env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges),
|
TrustedProxies: env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges),
|
||||||
ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"),
|
ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"),
|
||||||
@@ -266,27 +209,15 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
|||||||
LimitBanDuration: env.durationNotOff("SWWAF_LIMIT_BAN_DURATION", "1h"),
|
LimitBanDuration: env.durationNotOff("SWWAF_LIMIT_BAN_DURATION", "1h"),
|
||||||
LimitBanRepeatWindow: env.durationNotOff("SWWAF_LIMIT_BAN_REPEAT_WINDOW", "24h"),
|
LimitBanRepeatWindow: env.durationNotOff("SWWAF_LIMIT_BAN_REPEAT_WINDOW", "24h"),
|
||||||
MaxBanDuration: env.durationNotOff("SWWAF_MAX_BAN_DURATION", "7d"),
|
MaxBanDuration: env.durationNotOff("SWWAF_MAX_BAN_DURATION", "7d"),
|
||||||
AttackBanDuration: env.durationNotOff("SWWAF_ATTACK_BAN_DURATION", "7d"),
|
|
||||||
MaxBans: env.numberNotOff("SWWAF_MAX_BANS", "5000"),
|
MaxBans: env.numberNotOff("SWWAF_MAX_BANS", "5000"),
|
||||||
BanScopeV4Prefix: env.v4Prefix("SWWAF_BAN_SCOPE_V4_PREFIX", "32"),
|
BanScopeV4Prefix: env.v4Prefix("SWWAF_BAN_SCOPE_V4_PREFIX", "32"),
|
||||||
StateDir: env.absolutePath("SWWAF_STATE_DIR", "/var/lib/smallwebwaf"),
|
StateDir: env.absolutePath("SWWAF_STATE_DIR", "/var/lib/smallwebwaf"),
|
||||||
StateWriteDelay: env.durationNotOff("SWWAF_STATE_WRITE_DELAY", "10s"),
|
StateWriteDelay: env.durationNotOff("SWWAF_STATE_WRITE_DELAY", "10s"),
|
||||||
StateCounterInterval: env.durationNotOff("SWWAF_STATE_COUNTER_INTERVAL", "15m"),
|
StateCounterInterval: env.durationNotOff("SWWAF_STATE_COUNTER_INTERVAL", "15m"),
|
||||||
LogRequestHeaders: env.headerNames("SWWAF_LOG_REQUEST_HEADERS",
|
MetricsToken: env.token("SWWAF_METRICS_TOKEN"),
|
||||||
"accept,accept-language,accept-encoding,content-type,origin,range"),
|
MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
|
||||||
MetricsToken: env.token("SWWAF_METRICS_TOKEN"),
|
|
||||||
MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
|
|
||||||
RulesDir: env.value("SWWAF_RULES_DIR", "/etc/smallwebwaf/rules.d"),
|
|
||||||
RulesEnabled: env.boolean("SWWAF_RULES_ENABLED", "true"),
|
|
||||||
LogRemoteURL: env.logRemoteURL("SWWAF_LOG_REMOTE_URL"),
|
|
||||||
LogRemoteTLSCAs: env.certificates("SWWAF_LOG_REMOTE_TLS_CA_FILE"),
|
|
||||||
LogRemoteBuffer: env.numberNotOff("SWWAF_LOG_REMOTE_BUFFER", "10000"),
|
|
||||||
LogRemoteFacility: env.facility("SWWAF_LOG_REMOTE_FACILITY", "local0"),
|
|
||||||
}
|
}
|
||||||
|
|
||||||
cfg.LogRemoteAppName = env.appName("SWWAF_LOG_REMOTE_APP_NAME",
|
|
||||||
cfg.InstanceName, cfg.LogRemoteURL != nil)
|
|
||||||
|
|
||||||
for _, country := range cfg.ExclusivelyAllowedCountries {
|
for _, country := range cfg.ExclusivelyAllowedCountries {
|
||||||
if slices.Contains(cfg.DeniedCountries, country) {
|
if slices.Contains(cfg.DeniedCountries, country) {
|
||||||
env.check("SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES",
|
env.check("SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES",
|
||||||
@@ -303,24 +234,6 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
|||||||
return cfg, nil
|
return cfg, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// ListenAddrAndUpstreamURL reads only SWWAF_LISTEN_ADDR and
|
|
||||||
// SWWAF_UPSTREAM_URL, either of which may be given as a file, as
|
|
||||||
// FromEnvironment does. The health check needs no other setting, so it
|
|
||||||
// reads no other, nor a file that another names.
|
|
||||||
func ListenAddrAndUpstreamURL(
|
|
||||||
lookupEnv func(string) (string, bool),
|
|
||||||
) (string, *url.URL, error) {
|
|
||||||
env := &environment{lookupEnv: lookupEnv}
|
|
||||||
listenAddr := env.address("SWWAF_LISTEN_ADDR", defaultListenAddr)
|
|
||||||
upstreamURL := env.appURL("SWWAF_UPSTREAM_URL", defaultUpstreamURL)
|
|
||||||
|
|
||||||
if env.err != nil {
|
|
||||||
return "", nil, env.err
|
|
||||||
}
|
|
||||||
|
|
||||||
return listenAddr, upstreamURL, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// privateRanges are the private address ranges, the default trusted
|
// privateRanges are the private address ranges, the default trusted
|
||||||
// proxies.
|
// proxies.
|
||||||
const privateRanges = "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
|
const privateRanges = "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
|
||||||
@@ -342,8 +255,8 @@ type environment struct {
|
|||||||
// value returns a setting's value, or its default when it is not set,
|
// value returns a setting's value, or its default when it is not set,
|
||||||
// and notes it for the log.
|
// and notes it for the log.
|
||||||
func (e *environment) value(name, defaultValue string) string {
|
func (e *environment) value(name, defaultValue string) string {
|
||||||
value, set := e.lookup(name)
|
value, ok := e.lookupEnv(name)
|
||||||
if !set {
|
if !ok {
|
||||||
value = defaultValue
|
value = defaultValue
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -352,37 +265,6 @@ func (e *environment) value(name, defaultValue string) string {
|
|||||||
return value
|
return value
|
||||||
}
|
}
|
||||||
|
|
||||||
// lookup returns a setting's value and whether it is set: the value of the
|
|
||||||
// variable name, or the contents of the file that the variable name_FILE
|
|
||||||
// names, less one newline at their end. It notes that file's path for the
|
|
||||||
// log. Both variables set, or a file that cannot be read, is an error.
|
|
||||||
func (e *environment) lookup(name string) (string, bool) {
|
|
||||||
value, set := e.lookupEnv(name)
|
|
||||||
fileName := name + "_FILE"
|
|
||||||
|
|
||||||
path, inFile := e.lookupEnv(fileName)
|
|
||||||
if !inFile {
|
|
||||||
return value, set
|
|
||||||
}
|
|
||||||
|
|
||||||
if set {
|
|
||||||
e.check(name, fmt.Errorf("is set, and so is %s; %w", fileName, errSetTwice))
|
|
||||||
|
|
||||||
return value, set
|
|
||||||
}
|
|
||||||
|
|
||||||
e.settings = append(e.settings, slog.String(fileName, path))
|
|
||||||
|
|
||||||
contents, err := os.ReadFile(path) //nolint:gosec // a file the admin names
|
|
||||||
if err != nil {
|
|
||||||
e.check(fileName, fmt.Errorf("cannot be read: %w", err))
|
|
||||||
|
|
||||||
return "", false
|
|
||||||
}
|
|
||||||
|
|
||||||
return strings.TrimSuffix(string(contents), "\n"), true
|
|
||||||
}
|
|
||||||
|
|
||||||
// check keeps the first error, naming the setting it is about.
|
// check keeps the first error, naming the setting it is about.
|
||||||
func (e *environment) check(name string, err error) {
|
func (e *environment) check(name string, err error) {
|
||||||
if err != nil && e.err == nil {
|
if err != nil && e.err == nil {
|
||||||
@@ -417,16 +299,6 @@ func (e *environment) observe(name, defaultValue string) bool {
|
|||||||
return mode == "observe"
|
return mode == "observe"
|
||||||
}
|
}
|
||||||
|
|
||||||
// boolean reads a setting that is true or false.
|
|
||||||
func (e *environment) boolean(name, defaultValue string) bool {
|
|
||||||
value := e.value(name, defaultValue)
|
|
||||||
if value != "true" && value != "false" {
|
|
||||||
e.check(name, fmt.Errorf("%q %w", value, errNotBoolean))
|
|
||||||
}
|
|
||||||
|
|
||||||
return value == "true"
|
|
||||||
}
|
|
||||||
|
|
||||||
// netblocks reads a setting that is a list of netblocks.
|
// netblocks reads a setting that is a list of netblocks.
|
||||||
func (e *environment) netblocks(name, defaultValue string) []netip.Prefix {
|
func (e *environment) netblocks(name, defaultValue string) []netip.Prefix {
|
||||||
netblocks, err := parseNetblocks(e.value(name, defaultValue))
|
netblocks, err := parseNetblocks(e.value(name, defaultValue))
|
||||||
@@ -484,15 +356,6 @@ func (e *environment) countries(name, defaultValue string) []string {
|
|||||||
return countries
|
return countries
|
||||||
}
|
}
|
||||||
|
|
||||||
// headerNames reads a setting that is a list of header names, and
|
|
||||||
// returns them in lower case.
|
|
||||||
func (e *environment) headerNames(name, defaultValue string) []string {
|
|
||||||
headers, err := parseHeaderNames(e.value(name, defaultValue))
|
|
||||||
e.check(name, err)
|
|
||||||
|
|
||||||
return headers
|
|
||||||
}
|
|
||||||
|
|
||||||
// durationNotOff reads a setting that is a duration and, unlike a
|
// durationNotOff reads a setting that is a duration and, unlike a
|
||||||
// timeout, cannot be off.
|
// timeout, cannot be off.
|
||||||
func (e *environment) durationNotOff(name, defaultValue string) time.Duration {
|
func (e *environment) durationNotOff(name, defaultValue string) time.Duration {
|
||||||
@@ -541,7 +404,7 @@ func (e *environment) absolutePath(name, defaultValue string) string {
|
|||||||
// switches off what it guards; set, it must be at least minTokenLength
|
// switches off what it guards; set, it must be at least minTokenLength
|
||||||
// characters. Neither the log nor an error shows its value.
|
// characters. Neither the log nor an error shows its value.
|
||||||
func (e *environment) token(name string) string {
|
func (e *environment) token(name string) string {
|
||||||
value, set := e.lookup(name)
|
value, set := e.lookupEnv(name)
|
||||||
if !set {
|
if !set {
|
||||||
e.settings = append(e.settings, slog.String(name, ""))
|
e.settings = append(e.settings, slog.String(name, ""))
|
||||||
|
|
||||||
@@ -557,80 +420,6 @@ func (e *environment) token(name string) string {
|
|||||||
return value
|
return value
|
||||||
}
|
}
|
||||||
|
|
||||||
// logRemoteURL reads the setting that is where every log line is also
|
|
||||||
// sent. Unset or empty, it is nil, and nothing is sent.
|
|
||||||
func (e *environment) logRemoteURL(name string) *url.URL {
|
|
||||||
value := e.value(name, "")
|
|
||||||
if value == "" {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
remote, err := parseLogRemoteURL(value)
|
|
||||||
e.check(name, err)
|
|
||||||
|
|
||||||
return remote
|
|
||||||
}
|
|
||||||
|
|
||||||
// certificates reads a setting that is the path of a file of PEM
|
|
||||||
// certificates. Unset or empty, it is nil. Its value names a file
|
|
||||||
// already, so, unlike the other settings, it has no _FILE form.
|
|
||||||
func (e *environment) certificates(name string) *x509.CertPool {
|
|
||||||
path, _ := e.lookupEnv(name)
|
|
||||||
e.settings = append(e.settings, slog.String(name, path))
|
|
||||||
|
|
||||||
if path == "" {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
pem, err := os.ReadFile(path) //nolint:gosec // a file the admin names
|
|
||||||
if err != nil {
|
|
||||||
e.check(name, fmt.Errorf("cannot be read: %w", err))
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
pool := x509.NewCertPool()
|
|
||||||
if !pool.AppendCertsFromPEM(pem) {
|
|
||||||
e.check(name, fmt.Errorf("%q %w", path, errNoCertificate))
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return pool
|
|
||||||
}
|
|
||||||
|
|
||||||
// facility reads a setting that is a syslog facility, and returns its
|
|
||||||
// number.
|
|
||||||
func (e *environment) facility(name, defaultValue string) int {
|
|
||||||
number, err := parseFacility(e.value(name, defaultValue))
|
|
||||||
e.check(name, err)
|
|
||||||
|
|
||||||
return number
|
|
||||||
}
|
|
||||||
|
|
||||||
// appName reads the setting that is the APP-NAME of the records the log
|
|
||||||
// lines are sent in, by default the instance name. Its value is checked
|
|
||||||
// when it is set, and, while lines are sent, when it is the instance name.
|
|
||||||
func (e *environment) appName(name, instanceName string, sending bool) string {
|
|
||||||
value, set := e.lookup(name)
|
|
||||||
if !set {
|
|
||||||
value = instanceName
|
|
||||||
}
|
|
||||||
|
|
||||||
e.settings = append(e.settings, slog.String(name, value))
|
|
||||||
|
|
||||||
switch {
|
|
||||||
case isAppName(value):
|
|
||||||
case set:
|
|
||||||
e.check(name, fmt.Errorf("%q %w", value, errNotAppName))
|
|
||||||
case sending:
|
|
||||||
e.check(name, fmt.Errorf("is unset, and SWWAF_INSTANCE_NAME %q, its default, %w",
|
|
||||||
value, errNotAppName))
|
|
||||||
}
|
|
||||||
|
|
||||||
return value
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseDuration reads a duration in Go's syntax, such as 90s or 15m, a
|
// parseDuration reads a duration in Go's syntax, such as 90s or 15m, a
|
||||||
// whole number of days such as 7d, or off.
|
// whole number of days such as 7d, or off.
|
||||||
func parseDuration(value string) (time.Duration, error) {
|
func parseDuration(value string) (time.Duration, error) {
|
||||||
@@ -910,58 +699,6 @@ func parseCountries(value string) ([]string, error) {
|
|||||||
return countries, nil
|
return countries, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// headerNameChars are the characters RFC 9110 allows in a header name:
|
|
||||||
// letters, digits and these marks.
|
|
||||||
const headerNameChars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" +
|
|
||||||
"0123456789!#$%&'*+-.^_`|~"
|
|
||||||
|
|
||||||
// IsHeaderName reports whether name can be a header name: one or more of
|
|
||||||
// the characters RFC 9110 allows in one.
|
|
||||||
func IsHeaderName(name string) bool {
|
|
||||||
if name == "" {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, char := range name {
|
|
||||||
if !strings.ContainsRune(headerNameChars, char) {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseHeaderNames reads a comma-separated list of header names in either
|
|
||||||
// case, and returns them in lower case. Host and Transfer-Encoding are
|
|
||||||
// refused: Go's HTTP server takes them out of the request's headers.
|
|
||||||
func parseHeaderNames(value string) ([]string, error) {
|
|
||||||
items, err := parseList(value)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
headers := make([]string, 0, len(items))
|
|
||||||
|
|
||||||
for _, item := range items {
|
|
||||||
if !IsHeaderName(item) {
|
|
||||||
return nil, fmt.Errorf("%q %w", item, errNotHeaderName)
|
|
||||||
}
|
|
||||||
|
|
||||||
header := strings.ToLower(item)
|
|
||||||
switch header {
|
|
||||||
case "host":
|
|
||||||
return nil, fmt.Errorf("%q %w; the request's host is the field host",
|
|
||||||
item, errHeaderTakenOut)
|
|
||||||
case "transfer-encoding":
|
|
||||||
return nil, fmt.Errorf("%q %w", item, errHeaderTakenOut)
|
|
||||||
}
|
|
||||||
|
|
||||||
headers = append(headers, header)
|
|
||||||
}
|
|
||||||
|
|
||||||
return headers, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseListenAddr checks an address to listen on: an optional host and a
|
// parseListenAddr checks an address to listen on: an optional host and a
|
||||||
// port number.
|
// port number.
|
||||||
func parseListenAddr(value string) (string, error) {
|
func parseListenAddr(value string) (string, error) {
|
||||||
@@ -1004,64 +741,3 @@ func parseUpstreamURL(value string) (*url.URL, error) {
|
|||||||
|
|
||||||
return upstream, nil
|
return upstream, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// parseLogRemoteURL reads where every log line is also sent:
|
|
||||||
// syslog+udp, syslog+tcp or syslog+tls, a host and a port from 1 to
|
|
||||||
// 65535, and nothing else.
|
|
||||||
func parseLogRemoteURL(value string) (*url.URL, error) {
|
|
||||||
remote, err := url.Parse(value)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("%q %w", value, errNotLogRemoteURL)
|
|
||||||
}
|
|
||||||
|
|
||||||
schemes := []string{remotelog.SchemeUDP, remotelog.SchemeTCP, remotelog.SchemeTLS}
|
|
||||||
port, err := strconv.ParseUint(remote.Port(), 10, 16)
|
|
||||||
|
|
||||||
onlySchemeHostAndPort := slices.Contains(schemes, remote.Scheme) &&
|
|
||||||
remote.Hostname() != "" && err == nil && port != 0 &&
|
|
||||||
remote.User == nil && remote.Opaque == "" &&
|
|
||||||
(remote.Path == "" || remote.Path == "/") &&
|
|
||||||
remote.RawQuery == "" && remote.Fragment == ""
|
|
||||||
if !onlySchemeHostAndPort {
|
|
||||||
return nil, fmt.Errorf("%q %w", value, errNotLogRemoteURL)
|
|
||||||
}
|
|
||||||
|
|
||||||
return remote, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseFacility reads the name of a syslog facility, and returns its
|
|
||||||
// number, as RFC 5424 numbers them.
|
|
||||||
func parseFacility(value string) (int, error) {
|
|
||||||
//nolint:mnd // the facilities' numbers in RFC 5424
|
|
||||||
number, known := map[string]int{
|
|
||||||
"kern": 0, "user": 1, "mail": 2, "daemon": 3, "auth": 4, "syslog": 5,
|
|
||||||
"lpr": 6, "news": 7, "uucp": 8, "cron": 9, "authpriv": 10, "ftp": 11,
|
|
||||||
"local0": 16, "local1": 17, "local2": 18, "local3": 19,
|
|
||||||
"local4": 20, "local5": 21, "local6": 22, "local7": 23,
|
|
||||||
}[value]
|
|
||||||
if !known {
|
|
||||||
return 0, fmt.Errorf("%q %w", value, errNotFacility)
|
|
||||||
}
|
|
||||||
|
|
||||||
return number, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// appNameMaxLength is the most characters RFC 5424 allows in an
|
|
||||||
// APP-NAME.
|
|
||||||
const appNameMaxLength = 48
|
|
||||||
|
|
||||||
// isAppName reports whether value can be an APP-NAME: 1 to
|
|
||||||
// appNameMaxLength printable ASCII characters, none of them a space.
|
|
||||||
func isAppName(value string) bool {
|
|
||||||
if value == "" || len(value) > appNameMaxLength {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, char := range []byte(value) {
|
|
||||||
if char < '!' || char > '~' {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|||||||
+14
-446
@@ -2,13 +2,10 @@ package config_test
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"crypto/x509"
|
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"maps"
|
"maps"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -44,7 +41,6 @@ const (
|
|||||||
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
|
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
|
||||||
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
|
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
|
||||||
maxBanDuration = "SWWAF_MAX_BAN_DURATION"
|
maxBanDuration = "SWWAF_MAX_BAN_DURATION"
|
||||||
attackBanDuration = "SWWAF_ATTACK_BAN_DURATION"
|
|
||||||
maxBans = "SWWAF_MAX_BANS"
|
maxBans = "SWWAF_MAX_BANS"
|
||||||
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
|
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
|
||||||
stateDir = "SWWAF_STATE_DIR"
|
stateDir = "SWWAF_STATE_DIR"
|
||||||
@@ -52,45 +48,11 @@ const (
|
|||||||
stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL"
|
stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL"
|
||||||
metricsToken = "SWWAF_METRICS_TOKEN" //nolint:gosec // the setting's name
|
metricsToken = "SWWAF_METRICS_TOKEN" //nolint:gosec // the setting's name
|
||||||
metricsTopN = "SWWAF_METRICS_TOP_N"
|
metricsTopN = "SWWAF_METRICS_TOP_N"
|
||||||
instanceName = "SWWAF_INSTANCE_NAME"
|
|
||||||
logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS"
|
|
||||||
rulesDir = "SWWAF_RULES_DIR"
|
|
||||||
rulesEnabled = "SWWAF_RULES_ENABLED"
|
|
||||||
logRemoteURL = "SWWAF_LOG_REMOTE_URL"
|
|
||||||
logRemoteTLSCAFile = "SWWAF_LOG_REMOTE_TLS_CA_FILE"
|
|
||||||
logRemoteBuffer = "SWWAF_LOG_REMOTE_BUFFER"
|
|
||||||
logRemoteFacility = "SWWAF_LOG_REMOTE_FACILITY"
|
|
||||||
logRemoteAppName = "SWWAF_LOG_REMOTE_APP_NAME"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// defaultLogRequestHeaders is the default of SWWAF_LOG_REQUEST_HEADERS.
|
|
||||||
const defaultLogRequestHeaders = "accept,accept-language,accept-encoding," +
|
|
||||||
"content-type,origin,range"
|
|
||||||
|
|
||||||
// testCA is a CA certificate, of which only that it reads matters here.
|
|
||||||
const testCA = `-----BEGIN CERTIFICATE-----
|
|
||||||
MIIBkzCCATmgAwIBAgIUeySaE27dnr6A2HijrMB13gTLUKIwCgYIKoZIzj0EAwIw
|
|
||||||
HjEcMBoGA1UEAwwTc21hbGx3ZWJ3YWYgdGVzdCBDQTAgFw0yNjEwMDYxNDI2MTNa
|
|
||||||
GA8yMTI2MDkxMjE0MjYxM1owHjEcMBoGA1UEAwwTc21hbGx3ZWJ3YWYgdGVzdCBD
|
|
||||||
QTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABKDEhcWKKhet2KgSdME+iEPxyEyn
|
|
||||||
2sd9IdElbt8DM2SfCdB2JsXo0C07UNZaywMPMfn/n8LNI/PKwu+N2uX7gfSjUzBR
|
|
||||||
MB0GA1UdDgQWBBRZo3BPLv0KbV4drw6JI1JIUKmoRzAfBgNVHSMEGDAWgBRZo3BP
|
|
||||||
Lv0KbV4drw6JI1JIUKmoRzAPBgNVHRMBAf8EBTADAQH/MAoGCCqGSM49BAMCA0gA
|
|
||||||
MEUCICC5k+76UpWoSwVbZA+atu5WcALEOJGqwOUWua3zemhcAiEA8Hfdxgwp0z2v
|
|
||||||
rlG9y/jrJb6ORy3kTLWo2EA0BA67vuI=
|
|
||||||
-----END CERTIFICATE-----
|
|
||||||
`
|
|
||||||
|
|
||||||
// token is a token of 32 characters, the shortest allowed.
|
// token is a token of 32 characters, the shortest allowed.
|
||||||
const token = "0123456789abcdef0123456789abcdef"
|
const token = "0123456789abcdef0123456789abcdef"
|
||||||
|
|
||||||
// instance is an SWWAF_INSTANCE_NAME that is a valid app name too, and
|
|
||||||
// remoteURL an SWWAF_LOG_REMOTE_URL, for the tests that send the lines.
|
|
||||||
const (
|
|
||||||
instance = "fsn1app1/gitea"
|
|
||||||
remoteURL = "syslog+udp://192.0.2.1:514"
|
|
||||||
)
|
|
||||||
|
|
||||||
// off switches a timeout, a size limit or a rate limit off.
|
// off switches a timeout, a size limit or a rate limit off.
|
||||||
const off = "off"
|
const off = "off"
|
||||||
|
|
||||||
@@ -139,7 +101,6 @@ func TestDefaults(t *testing.T) {
|
|||||||
LimitBanDuration: time.Hour,
|
LimitBanDuration: time.Hour,
|
||||||
LimitBanRepeatWindow: 24 * time.Hour,
|
LimitBanRepeatWindow: 24 * time.Hour,
|
||||||
MaxBanDuration: 7 * 24 * time.Hour,
|
MaxBanDuration: 7 * 24 * time.Hour,
|
||||||
AttackBanDuration: 7 * 24 * time.Hour,
|
|
||||||
MaxBans: 5000,
|
MaxBans: 5000,
|
||||||
BanScopeV4Prefix: 32,
|
BanScopeV4Prefix: 32,
|
||||||
StateDir: "/var/lib/smallwebwaf",
|
StateDir: "/var/lib/smallwebwaf",
|
||||||
@@ -147,8 +108,6 @@ func TestDefaults(t *testing.T) {
|
|||||||
StateCounterInterval: 15 * time.Minute,
|
StateCounterInterval: 15 * time.Minute,
|
||||||
MetricsToken: "",
|
MetricsToken: "",
|
||||||
MetricsTopN: 50,
|
MetricsTopN: 50,
|
||||||
RulesDir: "/etc/smallwebwaf/rules.d",
|
|
||||||
RulesEnabled: true,
|
|
||||||
})
|
})
|
||||||
|
|
||||||
if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" {
|
if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" {
|
||||||
@@ -163,18 +122,6 @@ func TestDefaults(t *testing.T) {
|
|||||||
wantCountries(t, deniedCountries, cfg.DeniedCountries)
|
wantCountries(t, deniedCountries, cfg.DeniedCountries)
|
||||||
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries)
|
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries)
|
||||||
|
|
||||||
hostname, err := os.Hostname()
|
|
||||||
if err != nil || hostname == "" || cfg.InstanceName != hostname {
|
|
||||||
t.Errorf("%s is %q, want the host's name %q (%v)", instanceName,
|
|
||||||
cfg.InstanceName, hostname, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantHeaders := strings.Split(defaultLogRequestHeaders, ",")
|
|
||||||
if !slices.Equal(cfg.LogRequestHeaders, wantHeaders) {
|
|
||||||
t.Errorf("%s gave %v, want %v", logRequestHeaders, cfg.LogRequestHeaders,
|
|
||||||
wantHeaders)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(cfg.RateLimitExemptPaths) != 0 {
|
if len(cfg.RateLimitExemptPaths) != 0 {
|
||||||
t.Errorf("%s gave %v, want none", rateLimitExemptPaths, cfg.RateLimitExemptPaths)
|
t.Errorf("%s gave %v, want none", rateLimitExemptPaths, cfg.RateLimitExemptPaths)
|
||||||
}
|
}
|
||||||
@@ -202,13 +149,13 @@ func TestValuesAsSet(t *testing.T) {
|
|||||||
rateLimitPerMinute: "60",
|
rateLimitPerMinute: "60",
|
||||||
rateLimitPerHour: "600",
|
rateLimitPerHour: "600",
|
||||||
rateLimitPerDay: "6000",
|
rateLimitPerDay: "6000",
|
||||||
|
rateLimitExemptPaths: "/assets/, /favicon.ico",
|
||||||
deniedCountries: "cn, RU,kp,Xk",
|
deniedCountries: "cn, RU,kp,Xk",
|
||||||
allowedCountries: "de",
|
allowedCountries: "de",
|
||||||
banResponse: "429",
|
banResponse: "429",
|
||||||
limitBanDuration: "15m",
|
limitBanDuration: "15m",
|
||||||
limitBanRepeatWindow: "2d",
|
limitBanRepeatWindow: "2d",
|
||||||
maxBanDuration: "30d",
|
maxBanDuration: "30d",
|
||||||
attackBanDuration: "1d",
|
|
||||||
maxBans: "100",
|
maxBans: "100",
|
||||||
banScopeV4Prefix: "24",
|
banScopeV4Prefix: "24",
|
||||||
stateDir: "/srv/waf-state",
|
stateDir: "/srv/waf-state",
|
||||||
@@ -216,8 +163,6 @@ func TestValuesAsSet(t *testing.T) {
|
|||||||
stateCounterInterval: "1h",
|
stateCounterInterval: "1h",
|
||||||
metricsToken: token,
|
metricsToken: token,
|
||||||
metricsTopN: "10",
|
metricsTopN: "10",
|
||||||
rulesDir: "/srv/waf-rules",
|
|
||||||
rulesEnabled: "false",
|
|
||||||
})
|
})
|
||||||
|
|
||||||
wantSettings(t, cfg, config.Config{
|
wantSettings(t, cfg, config.Config{
|
||||||
@@ -238,7 +183,6 @@ func TestValuesAsSet(t *testing.T) {
|
|||||||
LimitBanDuration: 15 * time.Minute,
|
LimitBanDuration: 15 * time.Minute,
|
||||||
LimitBanRepeatWindow: 48 * time.Hour,
|
LimitBanRepeatWindow: 48 * time.Hour,
|
||||||
MaxBanDuration: 30 * 24 * time.Hour,
|
MaxBanDuration: 30 * 24 * time.Hour,
|
||||||
AttackBanDuration: 24 * time.Hour,
|
|
||||||
MaxBans: 100,
|
MaxBans: 100,
|
||||||
BanScopeV4Prefix: 24,
|
BanScopeV4Prefix: 24,
|
||||||
StateDir: "/srv/waf-state",
|
StateDir: "/srv/waf-state",
|
||||||
@@ -246,8 +190,6 @@ func TestValuesAsSet(t *testing.T) {
|
|||||||
StateCounterInterval: time.Hour,
|
StateCounterInterval: time.Hour,
|
||||||
MetricsToken: token,
|
MetricsToken: token,
|
||||||
MetricsTopN: 10,
|
MetricsTopN: 10,
|
||||||
RulesDir: "/srv/waf-rules",
|
|
||||||
RulesEnabled: false,
|
|
||||||
})
|
})
|
||||||
|
|
||||||
if cfg.UpstreamURL.String() != "https://app.internal:8443/" {
|
if cfg.UpstreamURL.String() != "https://app.internal:8443/" {
|
||||||
@@ -260,12 +202,6 @@ func TestValuesAsSet(t *testing.T) {
|
|||||||
wantNetblocks(t, cfg.DenyNets, "198.51.100.0/24")
|
wantNetblocks(t, cfg.DenyNets, "198.51.100.0/24")
|
||||||
wantCountries(t, deniedCountries, cfg.DeniedCountries, "CN", "RU", "KP", "XK")
|
wantCountries(t, deniedCountries, cfg.DeniedCountries, "CN", "RU", "KP", "XK")
|
||||||
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
|
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
|
||||||
}
|
|
||||||
|
|
||||||
func TestRateLimitExemptPathsAsSet(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cfg := fromEnvironment(t, environment{rateLimitExemptPaths: "/assets/, /favicon.ico"})
|
|
||||||
|
|
||||||
if !slices.Equal(cfg.RateLimitExemptPaths, []string{"/assets/", "/favicon.ico"}) {
|
if !slices.Equal(cfg.RateLimitExemptPaths, []string{"/assets/", "/favicon.ico"}) {
|
||||||
t.Errorf("%s gave %v, want /assets/ and /favicon.ico",
|
t.Errorf("%s gave %v, want /assets/ and /favicon.ico",
|
||||||
@@ -286,192 +222,6 @@ func TestPathPrefixNotStartingWithSlashStopsTheStart(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cfg := fromEnvironment(t, environment{
|
|
||||||
instanceName: "fsn1app1/gitea",
|
|
||||||
logRequestHeaders: " Accept , X-Custom",
|
|
||||||
})
|
|
||||||
|
|
||||||
if cfg.InstanceName != "fsn1app1/gitea" ||
|
|
||||||
!slices.Equal(cfg.LogRequestHeaders, []string{"accept", "x-custom"}) {
|
|
||||||
t.Errorf("%s is %q and %s gives %v", instanceName, cfg.InstanceName,
|
|
||||||
logRequestHeaders, cfg.LogRequestHeaders)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRemoteLogSettingsDefaults(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cfg := fromEnvironment(t, environment{instanceName: instance})
|
|
||||||
|
|
||||||
if cfg.LogRemoteURL != nil || cfg.LogRemoteTLSCAs != nil ||
|
|
||||||
cfg.LogRemoteBuffer != 10000 || cfg.LogRemoteFacility != 16 ||
|
|
||||||
cfg.LogRemoteAppName != instance {
|
|
||||||
t.Errorf("remote log settings %v, %v, %d, %d and %q, want no URL, no "+
|
|
||||||
"certificates, 10000, 16 and %s's %s", cfg.LogRemoteURL,
|
|
||||||
cfg.LogRemoteTLSCAs, cfg.LogRemoteBuffer, cfg.LogRemoteFacility,
|
|
||||||
cfg.LogRemoteAppName, instanceName, instance)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRemoteLogSettingsAsSet(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
caFile := filepath.Join(t.TempDir(), "ca.pem")
|
|
||||||
|
|
||||||
err := os.WriteFile(caFile, []byte(testCA), 0o600)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write %s: %v", caFile, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
cfg := fromEnvironment(t, environment{
|
|
||||||
logRemoteURL: "syslog+tls://logs.example:6514",
|
|
||||||
logRemoteTLSCAFile: caFile,
|
|
||||||
logRemoteBuffer: "500",
|
|
||||||
logRemoteFacility: "daemon",
|
|
||||||
logRemoteAppName: instance,
|
|
||||||
})
|
|
||||||
|
|
||||||
roots := x509.NewCertPool()
|
|
||||||
roots.AppendCertsFromPEM([]byte(testCA))
|
|
||||||
|
|
||||||
if cfg.LogRemoteURL.String() != "syslog+tls://logs.example:6514" ||
|
|
||||||
!roots.Equal(cfg.LogRemoteTLSCAs) || cfg.LogRemoteBuffer != 500 ||
|
|
||||||
cfg.LogRemoteFacility != 3 || cfg.LogRemoteAppName != instance {
|
|
||||||
t.Errorf("remote log settings %v, %v, %d, %d and %q", cfg.LogRemoteURL,
|
|
||||||
cfg.LogRemoteTLSCAs, cfg.LogRemoteBuffer, cfg.LogRemoteFacility,
|
|
||||||
cfg.LogRemoteAppName)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRemoteLogURLForms(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, value := range []string{
|
|
||||||
"syslog+udp://192.0.2.1:514",
|
|
||||||
"syslog+tcp://[2001:db8::1]:514",
|
|
||||||
"syslog+tls://logs.example:6514/",
|
|
||||||
} {
|
|
||||||
cfg := fromEnvironment(t, environment{logRemoteURL: value})
|
|
||||||
if cfg.LogRemoteURL.String() != value {
|
|
||||||
t.Errorf("%s read as %v", value, cfg.LogRemoteURL)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
cfg := fromEnvironment(t, environment{logRemoteURL: ""})
|
|
||||||
if cfg.LogRemoteURL != nil {
|
|
||||||
t.Errorf("set but empty, %s read as %v", logRemoteURL, cfg.LogRemoteURL)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRemoteLogFacilitiesByNumber(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for name, number := range map[string]int{
|
|
||||||
"kern": 0, "user": 1, "auth": 4, "authpriv": 10, "ftp": 11,
|
|
||||||
"local0": 16, "local5": 21, "local7": 23,
|
|
||||||
} {
|
|
||||||
cfg := fromEnvironment(t, environment{logRemoteFacility: name})
|
|
||||||
if cfg.LogRemoteFacility != number {
|
|
||||||
t.Errorf("%s read as %d, want %d", name, cfg.LogRemoteFacility, number)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestInvalidRemoteLogSettingStopsTheStart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, tc := range []struct{ name, value string }{
|
|
||||||
{logRemoteURL, "logs.example:514"},
|
|
||||||
{logRemoteURL, "syslog://logs.example:514"},
|
|
||||||
{logRemoteURL, "http://logs.example:514"},
|
|
||||||
{logRemoteURL, "syslog+udp://logs.example"},
|
|
||||||
{logRemoteURL, "syslog+tcp://:514"},
|
|
||||||
{logRemoteURL, "syslog+tcp://logs.example:0"},
|
|
||||||
{logRemoteURL, "syslog+tls://logs.example:65536"},
|
|
||||||
{logRemoteURL, "syslog+tls://user@logs.example:6514"},
|
|
||||||
{logRemoteURL, "syslog+tcp://logs.example:514/app"},
|
|
||||||
{logRemoteURL, "syslog+tcp://logs.example:514?tls=1"},
|
|
||||||
{logRemoteTLSCAFile, "/nonexistent/ca.pem"},
|
|
||||||
{logRemoteBuffer, off}, {logRemoteBuffer, "0"}, {logRemoteBuffer, "10K"},
|
|
||||||
{logRemoteFacility, "local8"}, {logRemoteFacility, "LOCAL0"},
|
|
||||||
{logRemoteFacility, "16"}, {logRemoteFacility, ""},
|
|
||||||
{logRemoteAppName, ""}, {logRemoteAppName, "my app"},
|
|
||||||
{logRemoteAppName, "gitéa"}, {logRemoteAppName, strings.Repeat("a", 49)},
|
|
||||||
} {
|
|
||||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
|
|
||||||
if err == nil || !strings.HasPrefix(err.Error(), tc.name+": ") {
|
|
||||||
t.Errorf("%s=%q: error %v, want one naming it", tc.name, tc.value, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRemoteLogCAFileWithoutCertificateStopsTheStart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
caFile := filepath.Join(t.TempDir(), "ca.pem")
|
|
||||||
|
|
||||||
err := os.WriteFile(caFile, []byte("not a certificate\n"), 0o600)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write %s: %v", caFile, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = config.FromEnvironment(environment{logRemoteTLSCAFile: caFile}.lookupEnv)
|
|
||||||
|
|
||||||
want := logRemoteTLSCAFile + `: "` + caFile + `" holds no PEM certificate`
|
|
||||||
if err == nil || err.Error() != want {
|
|
||||||
t.Errorf("error %v, want %s", err, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestInstanceNameNotAnAppNameStopsTheStartOnlyWhileSending(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const spaced = "fsn1 app1"
|
|
||||||
|
|
||||||
sending := environment{logRemoteURL: remoteURL, instanceName: spaced}
|
|
||||||
|
|
||||||
_, err := config.FromEnvironment(sending.lookupEnv)
|
|
||||||
|
|
||||||
want := logRemoteAppName + `: is unset, and ` + instanceName +
|
|
||||||
` "fsn1 app1", its default, is not 1 to 48 printable ASCII characters ` +
|
|
||||||
`without a space, such as gitea`
|
|
||||||
if err == nil || err.Error() != want {
|
|
||||||
t.Errorf("error %v, want %s", err, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
cfg := fromEnvironment(t, environment{instanceName: spaced})
|
|
||||||
if cfg.LogRemoteAppName != spaced {
|
|
||||||
t.Errorf("not sending, %s is %q", logRemoteAppName, cfg.LogRemoteAppName)
|
|
||||||
}
|
|
||||||
|
|
||||||
sending[logRemoteAppName] = instance
|
|
||||||
|
|
||||||
cfg = fromEnvironment(t, sending)
|
|
||||||
if cfg.LogRemoteAppName != instance {
|
|
||||||
t.Errorf("set to %s, %s is %q", instance, logRemoteAppName,
|
|
||||||
cfg.LogRemoteAppName)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAppNameSetStopsTheStartWhileSending(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
_, err := config.FromEnvironment(environment{
|
|
||||||
logRemoteURL: remoteURL,
|
|
||||||
instanceName: instance,
|
|
||||||
logRemoteAppName: "my app",
|
|
||||||
}.lookupEnv)
|
|
||||||
|
|
||||||
want := logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` +
|
|
||||||
`characters without a space, such as gitea`
|
|
||||||
if err == nil || err.Error() != want {
|
|
||||||
t.Errorf("error %v, want %s", err, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
|
func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -572,8 +322,10 @@ func TestTrustedProxiesSetButEmptyTrustNothing(t *testing.T) {
|
|||||||
func TestInvalidValueStopsTheStart(t *testing.T) {
|
func TestInvalidValueStopsTheStart(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
wantStartStopped(t, []struct{ name, value string }{
|
for _, tc := range []struct{ name, value string }{
|
||||||
{listenAddr, "8080"}, {listenAddr, ":http"}, {listenAddr, ":65536"},
|
{listenAddr, "8080"},
|
||||||
|
{listenAddr, ":http"},
|
||||||
|
{listenAddr, ":65536"},
|
||||||
{upstreamURL, "127.0.0.1:8081"},
|
{upstreamURL, "127.0.0.1:8081"},
|
||||||
{upstreamURL, "ftp://127.0.0.1:8081"},
|
{upstreamURL, "ftp://127.0.0.1:8081"},
|
||||||
{upstreamURL, "http://"},
|
{upstreamURL, "http://"},
|
||||||
@@ -591,7 +343,8 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
|
|||||||
{allowNets, "192.0.2.0/24,monitoring"},
|
{allowNets, "192.0.2.0/24,monitoring"},
|
||||||
{rateLimitExemptNets, "2001:db8::/129"},
|
{rateLimitExemptNets, "2001:db8::/129"},
|
||||||
{denyNets, "198.51.100.0/24,"},
|
{denyNets, "198.51.100.0/24,"},
|
||||||
{clientRequestTimeout, "60"}, {clientRequestTimeout, ""},
|
{clientRequestTimeout, "60"},
|
||||||
|
{clientRequestTimeout, ""},
|
||||||
{clientIdleTimeout, "0s"},
|
{clientIdleTimeout, "0s"},
|
||||||
{clientIdleTimeout, "2 minutes"},
|
{clientIdleTimeout, "2 minutes"},
|
||||||
{clientResponseTimeout, "1y"},
|
{clientResponseTimeout, "1y"},
|
||||||
@@ -608,7 +361,8 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
|
|||||||
{rateLimitPerMinute, "1K"},
|
{rateLimitPerMinute, "1K"},
|
||||||
{rateLimitPerHour, "0"},
|
{rateLimitPerHour, "0"},
|
||||||
{rateLimitPerHour, "1.5"},
|
{rateLimitPerHour, "1.5"},
|
||||||
{rateLimitPerDay, "-1"}, {rateLimitPerDay, "lots"},
|
{rateLimitPerDay, "-1"},
|
||||||
|
{rateLimitPerDay, "lots"},
|
||||||
{rateLimitExemptPaths, "/assets/,,/static/"},
|
{rateLimitExemptPaths, "/assets/,,/static/"},
|
||||||
{deniedCountries, "nk"},
|
{deniedCountries, "nk"},
|
||||||
{deniedCountries, "kp,,ir"},
|
{deniedCountries, "kp,,ir"},
|
||||||
@@ -622,37 +376,17 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
|
|||||||
{allowedCountries, "uk"},
|
{allowedCountries, "uk"},
|
||||||
{allowedCountries, "zz"},
|
{allowedCountries, "zz"},
|
||||||
{allowedCountries, "de,germany"},
|
{allowedCountries, "de,germany"},
|
||||||
{metricsTopN, off}, {metricsTopN, "0"}, {metricsTopN, "-1"},
|
|
||||||
{logRequestHeaders, "accept,,origin"}, {logRequestHeaders, "accept;origin"},
|
|
||||||
{logRequestHeaders, "accept language"}, {logRequestHeaders, "x-foo:"},
|
|
||||||
{logRequestHeaders, "host"}, {logRequestHeaders, "accept,Host"},
|
|
||||||
{logRequestHeaders, "transfer-encoding"}, {logRequestHeaders, "TRANSFER-ENCODING"},
|
|
||||||
{rulesEnabled, "yes"}, {rulesEnabled, "True"},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestInvalidBanOrStateValueStopsTheStart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
wantStartStopped(t, []struct{ name, value string }{
|
|
||||||
{banResponse, "404"}, {banResponse, "drop"}, {banResponse, ""},
|
{banResponse, "404"}, {banResponse, "drop"}, {banResponse, ""},
|
||||||
{limitBanDuration, off}, {limitBanDuration, "0s"}, {limitBanDuration, "1"},
|
{limitBanDuration, off}, {limitBanDuration, "0s"}, {limitBanDuration, "1"},
|
||||||
{limitBanRepeatWindow, off}, {limitBanRepeatWindow, "-1h"},
|
{limitBanRepeatWindow, off}, {limitBanRepeatWindow, "-1h"},
|
||||||
{maxBanDuration, off}, {maxBanDuration, "1w"}, {attackBanDuration, off},
|
{maxBanDuration, off}, {maxBanDuration, "1w"},
|
||||||
{maxBans, off}, {maxBans, "0"}, {maxBans, "5K"},
|
{maxBans, off}, {maxBans, "0"}, {maxBans, "5K"},
|
||||||
{banScopeV4Prefix, "33"}, {banScopeV4Prefix, "-1"}, {banScopeV4Prefix, "/24"},
|
{banScopeV4Prefix, "33"}, {banScopeV4Prefix, "-1"}, {banScopeV4Prefix, "/24"},
|
||||||
{stateDir, ""}, {stateDir, "state"}, {stateDir, "./var/lib/smallwebwaf"},
|
{stateDir, ""}, {stateDir, "state"}, {stateDir, "./var/lib/smallwebwaf"},
|
||||||
{stateWriteDelay, off}, {stateWriteDelay, "0s"},
|
{stateWriteDelay, off}, {stateWriteDelay, "0s"},
|
||||||
{stateCounterInterval, off}, {stateCounterInterval, "15"},
|
{stateCounterInterval, off}, {stateCounterInterval, "15"},
|
||||||
})
|
{metricsTopN, off}, {metricsTopN, "0"}, {metricsTopN, "-1"},
|
||||||
}
|
} {
|
||||||
|
|
||||||
// wantStartStopped checks that each setting, set to its value, stops the
|
|
||||||
// start with an error that names the setting.
|
|
||||||
func wantStartStopped(t *testing.T, invalid []struct{ name, value string }) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for _, tc := range invalid {
|
|
||||||
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -668,27 +402,6 @@ func wantStartStopped(t *testing.T, invalid []struct{ name, value string }) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestHostOrTransferEncodingStopsTheStart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// Only Host's message points to the field host.
|
|
||||||
for value, want := range map[string]string{
|
|
||||||
"Host": `"Host" is taken out of every request by Go's HTTP server, ` +
|
|
||||||
"so it can never be logged; the request's host is the field host",
|
|
||||||
"transfer-encoding": `"transfer-encoding" is taken out of every ` +
|
|
||||||
"request by Go's HTTP server, so it can never be logged",
|
|
||||||
} {
|
|
||||||
t.Run(value, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
_, err := config.FromEnvironment(environment{logRequestHeaders: value}.lookupEnv)
|
|
||||||
if err == nil || err.Error() != logRequestHeaders+": "+want {
|
|
||||||
t.Errorf("error %v, want %s: %s", err, logRequestHeaders, want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestShortTokenStopsTheStartWithoutShowingIt(t *testing.T) {
|
func TestShortTokenStopsTheStartWithoutShowingIt(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -722,133 +435,6 @@ func TestTokenIsLoggedMasked(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestSettingFromFileLosesOneNewlineAndNoMore(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for contents, want := range map[string]string{
|
|
||||||
token: token,
|
|
||||||
token + "\n": token,
|
|
||||||
token + "\n\n": token + "\n",
|
|
||||||
token + " \n": token + " ",
|
|
||||||
} {
|
|
||||||
cfg := fromEnvironment(t, environment{
|
|
||||||
metricsToken + "_FILE": writeFile(t, contents),
|
|
||||||
})
|
|
||||||
if cfg.MetricsToken != want {
|
|
||||||
t.Errorf("file holding %q gave %s %q, want %q", contents, metricsToken,
|
|
||||||
cfg.MetricsToken, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestSettingFromFileIsCheckedAsTheSettingItself(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
_, err := config.FromEnvironment(environment{
|
|
||||||
requestMaxBytes + "_FILE": writeFile(t, "lots\n"),
|
|
||||||
}.lookupEnv)
|
|
||||||
|
|
||||||
want := requestMaxBytes + `: "lots" is not a size such as 512K, 100M or 5G, or off`
|
|
||||||
if err == nil || err.Error() != want {
|
|
||||||
t.Errorf("error %v, want %s", err, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = config.FromEnvironment(environment{
|
|
||||||
logRemoteURL: remoteURL,
|
|
||||||
instanceName: instance,
|
|
||||||
logRemoteAppName + "_FILE": writeFile(t, "my app\n"),
|
|
||||||
}.lookupEnv)
|
|
||||||
|
|
||||||
want = logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` +
|
|
||||||
`characters without a space, such as gitea`
|
|
||||||
if err == nil || err.Error() != want {
|
|
||||||
t.Errorf("error %v, want %s", err, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestSettingAndItsFileBothSetStopsTheStart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
_, err := config.FromEnvironment(environment{
|
|
||||||
metricsToken: token,
|
|
||||||
metricsToken + "_FILE": writeFile(t, token),
|
|
||||||
}.lookupEnv)
|
|
||||||
|
|
||||||
want := metricsToken + ": is set, and so is " + metricsToken +
|
|
||||||
"_FILE; set only one of them"
|
|
||||||
if err == nil || err.Error() != want {
|
|
||||||
t.Errorf("error %v, want %s", err, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUnreadableSettingFileStopsTheStart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
|
|
||||||
for _, path := range []string{filepath.Join(dir, "missing"), dir} {
|
|
||||||
_, err := config.FromEnvironment(environment{metricsToken + "_FILE": path}.lookupEnv)
|
|
||||||
|
|
||||||
want := metricsToken + "_FILE: cannot be read: "
|
|
||||||
if err == nil || !strings.HasPrefix(err.Error(), want) {
|
|
||||||
t.Errorf("%s: error %v, want one starting %s", path, err, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTokenFromFileIsLoggedMaskedWithTheFile(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
path := writeFile(t, token+"\n")
|
|
||||||
cfg := fromEnvironment(t, environment{metricsToken + "_FILE": path})
|
|
||||||
|
|
||||||
var out bytes.Buffer
|
|
||||||
|
|
||||||
slog.New(slog.NewJSONHandler(&out, nil)).Info("starting", "settings", cfg)
|
|
||||||
|
|
||||||
var line struct {
|
|
||||||
Settings map[string]string `json:"settings"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := json.Unmarshal(out.Bytes(), &line)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("decode %s: %v", out.Bytes(), err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.Contains(out.String(), token) ||
|
|
||||||
line.Settings[metricsToken] != "********" ||
|
|
||||||
line.Settings[metricsToken+"_FILE"] != path {
|
|
||||||
t.Errorf("the token is not logged masked, with its file %s: %s", path,
|
|
||||||
out.String())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRemoteLogCAFileIsNotReadAsAFileInItsTurn(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cfg := fromEnvironment(t, environment{
|
|
||||||
logRemoteTLSCAFile + "_FILE": writeFile(t, "/nonexistent/ca.pem\n"),
|
|
||||||
})
|
|
||||||
if cfg.LogRemoteTLSCAs != nil {
|
|
||||||
t.Errorf("%s_FILE gave certificates", logRemoteTLSCAFile)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeFile writes contents to a file in a directory of its own, removed
|
|
||||||
// when the test ends, and returns the file's path.
|
|
||||||
func writeFile(t *testing.T, contents string) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
path := filepath.Join(t.TempDir(), "setting")
|
|
||||||
|
|
||||||
err := os.WriteFile(path, []byte(contents), 0o600)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write %s: %v", path, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return path
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLogsEachSettingWithItsValue(t *testing.T) {
|
func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -867,8 +453,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
|||||||
t.Fatalf("decode %s: %v", out.Bytes(), err)
|
t.Fatalf("decode %s: %v", out.Bytes(), err)
|
||||||
}
|
}
|
||||||
|
|
||||||
hostname, _ := os.Hostname()
|
|
||||||
|
|
||||||
want := map[string]string{
|
want := map[string]string{
|
||||||
listenAddr: ":8080",
|
listenAddr: ":8080",
|
||||||
upstreamURL: "http://127.0.0.1:8081",
|
upstreamURL: "http://127.0.0.1:8081",
|
||||||
@@ -895,7 +479,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
|||||||
limitBanDuration: "1h",
|
limitBanDuration: "1h",
|
||||||
limitBanRepeatWindow: "24h",
|
limitBanRepeatWindow: "24h",
|
||||||
maxBanDuration: "7d",
|
maxBanDuration: "7d",
|
||||||
attackBanDuration: "7d",
|
|
||||||
maxBans: "5000",
|
maxBans: "5000",
|
||||||
banScopeV4Prefix: "32",
|
banScopeV4Prefix: "32",
|
||||||
stateDir: "/var/lib/smallwebwaf",
|
stateDir: "/var/lib/smallwebwaf",
|
||||||
@@ -903,15 +486,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
|||||||
stateCounterInterval: "15m",
|
stateCounterInterval: "15m",
|
||||||
metricsToken: "",
|
metricsToken: "",
|
||||||
metricsTopN: "50",
|
metricsTopN: "50",
|
||||||
instanceName: hostname,
|
|
||||||
logRequestHeaders: defaultLogRequestHeaders,
|
|
||||||
rulesDir: "/etc/smallwebwaf/rules.d",
|
|
||||||
rulesEnabled: "true",
|
|
||||||
logRemoteURL: "",
|
|
||||||
logRemoteTLSCAFile: "",
|
|
||||||
logRemoteBuffer: "10000",
|
|
||||||
logRemoteFacility: "local0",
|
|
||||||
logRemoteAppName: hostname,
|
|
||||||
}
|
}
|
||||||
if !maps.Equal(line.Settings, want) {
|
if !maps.Equal(line.Settings, want) {
|
||||||
t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
|
t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
|
||||||
@@ -941,8 +515,8 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) {
|
|||||||
wantBanSettings(t, got, want)
|
wantBanSettings(t, got, want)
|
||||||
}
|
}
|
||||||
|
|
||||||
// wantBanSettings checks the settings for bans, the state files, the
|
// wantBanSettings checks the settings for bans, the state files and the
|
||||||
// metrics and the rule files.
|
// metrics.
|
||||||
func wantBanSettings(t *testing.T, got *config.Config, want config.Config) {
|
func wantBanSettings(t *testing.T, got *config.Config, want config.Config) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -950,17 +524,11 @@ func wantBanSettings(t *testing.T, got *config.Config, want config.Config) {
|
|||||||
got.LimitBanDuration != want.LimitBanDuration ||
|
got.LimitBanDuration != want.LimitBanDuration ||
|
||||||
got.LimitBanRepeatWindow != want.LimitBanRepeatWindow ||
|
got.LimitBanRepeatWindow != want.LimitBanRepeatWindow ||
|
||||||
got.MaxBanDuration != want.MaxBanDuration ||
|
got.MaxBanDuration != want.MaxBanDuration ||
|
||||||
got.AttackBanDuration != want.AttackBanDuration ||
|
|
||||||
got.MaxBans != want.MaxBans ||
|
got.MaxBans != want.MaxBans ||
|
||||||
got.BanScopeV4Prefix != want.BanScopeV4Prefix {
|
got.BanScopeV4Prefix != want.BanScopeV4Prefix {
|
||||||
t.Errorf("ban settings\n%+v\nwant\n%+v", got, want)
|
t.Errorf("ban settings\n%+v\nwant\n%+v", got, want)
|
||||||
}
|
}
|
||||||
|
|
||||||
if got.RulesDir != want.RulesDir || got.RulesEnabled != want.RulesEnabled {
|
|
||||||
t.Errorf("rule files in %q, on: %t, want %q, %t",
|
|
||||||
got.RulesDir, got.RulesEnabled, want.RulesDir, want.RulesEnabled)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got.StateDir != want.StateDir ||
|
if got.StateDir != want.StateDir ||
|
||||||
got.StateWriteDelay != want.StateWriteDelay ||
|
got.StateWriteDelay != want.StateWriteDelay ||
|
||||||
got.StateCounterInterval != want.StateCounterInterval {
|
got.StateCounterInterval != want.StateCounterInterval {
|
||||||
|
|||||||
@@ -13,9 +13,7 @@ import (
|
|||||||
"github.com/prometheus/client_golang/prometheus/promhttp"
|
"github.com/prometheus/client_golang/prometheus/promhttp"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// Metrics are smallwebwaf's metrics. They are safe for concurrent use.
|
// Metrics are smallwebwaf's metrics. They are safe for concurrent use.
|
||||||
@@ -32,9 +30,7 @@ type Metrics struct {
|
|||||||
rateLimitHits *prometheus.CounterVec
|
rateLimitHits *prometheus.CounterVec
|
||||||
sizeAndTimeLimitHits *prometheus.CounterVec
|
sizeAndTimeLimitHits *prometheus.CounterVec
|
||||||
offences *prometheus.CounterVec
|
offences *prometheus.CounterVec
|
||||||
// ruleMatches are made by AddRules.
|
countries *countries
|
||||||
ruleMatches *prometheus.CounterVec
|
|
||||||
countries *countries
|
|
||||||
|
|
||||||
// GeoJSRequests are the requests to GeoJS, and GeoJSFailures those
|
// GeoJSRequests are the requests to GeoJS, and GeoJSFailures those
|
||||||
// that failed. GeoJSUnanswered are the requests whose client counted
|
// that failed. GeoJSUnanswered are the requests whose client counted
|
||||||
@@ -139,22 +135,19 @@ func New(topN int) *Metrics {
|
|||||||
|
|
||||||
// AddBansAndClients adds the metrics read from the ledger and the table
|
// AddBansAndClients adds the metrics read from the ledger and the table
|
||||||
// of clients as the metrics are asked for: the bans made since the start,
|
// of clients as the metrics are asked for: the bans made since the start,
|
||||||
// by cause, the bans active and permanent at now, and the clients in the
|
// the bans active and permanent at now, and the clients in the table.
|
||||||
// table.
|
|
||||||
func (m *Metrics) AddBansAndClients(
|
func (m *Metrics) AddBansAndClients(
|
||||||
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
|
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
|
||||||
) {
|
) {
|
||||||
for _, cause := range []string{bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin} {
|
m.registry.MustRegister(
|
||||||
m.registry.MustRegister(prometheus.NewCounterFunc(prometheus.CounterOpts{
|
// Every ban smallwebwaf makes so far is for a broken limit.
|
||||||
|
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||||
Name: "smallwebwaf_bans_made_total",
|
Name: "smallwebwaf_bans_made_total",
|
||||||
Help: "Bans made, by cause.",
|
Help: "Bans made, by cause.",
|
||||||
ConstLabels: prometheus.Labels{"cause": cause},
|
ConstLabels: prometheus.Labels{"cause": "limit"},
|
||||||
}, func() float64 {
|
}, func() float64 {
|
||||||
return float64(ledger.Made(cause))
|
return float64(ledger.Made())
|
||||||
}))
|
}),
|
||||||
}
|
|
||||||
|
|
||||||
m.registry.MustRegister(
|
|
||||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||||
Name: "smallwebwaf_active_bans",
|
Name: "smallwebwaf_active_bans",
|
||||||
Help: "Bans active now, the permanent ones included.",
|
Help: "Bans active now, the permanent ones included.",
|
||||||
@@ -165,7 +158,7 @@ func (m *Metrics) AddBansAndClients(
|
|||||||
}),
|
}),
|
||||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||||
Name: "smallwebwaf_permanent_bans",
|
Name: "smallwebwaf_permanent_bans",
|
||||||
Help: "Permanent bans not lifted.",
|
Help: "Permanent bans.",
|
||||||
}, func() float64 {
|
}, func() float64 {
|
||||||
_, permanent := ledger.Count(now())
|
_, permanent := ledger.Count(now())
|
||||||
|
|
||||||
@@ -180,51 +173,6 @@ func (m *Metrics) AddBansAndClients(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// AddRules adds the metrics of the rule files: the requests that matched
|
|
||||||
// each rule, which RuleMatched counts, and the rules loaded from
|
|
||||||
// ruleFiles, read as the metrics are asked for. It is called once, before
|
|
||||||
// RuleMatched.
|
|
||||||
func (m *Metrics) AddRules(ruleFiles *rules.Files) {
|
|
||||||
m.ruleMatches = counterVec("smallwebwaf_rule_matches_total",
|
|
||||||
"Requests that matched a rule of the rule files, by its id and action.",
|
|
||||||
[]string{"rule_id", "action"})
|
|
||||||
|
|
||||||
m.registry.MustRegister(m.ruleMatches,
|
|
||||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
|
||||||
Name: "smallwebwaf_rules_loaded",
|
|
||||||
Help: "Rules loaded from the rule files.",
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(ruleFiles.Len())
|
|
||||||
}))
|
|
||||||
}
|
|
||||||
|
|
||||||
// AddRemoteLog adds the metrics of sending the log lines to
|
|
||||||
// SWWAF_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
|
|
||||||
// lines sent, those dropped, and those waiting in the buffer.
|
|
||||||
func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
|
|
||||||
m.registry.MustRegister(
|
|
||||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
|
||||||
Name: "smallwebwaf_remote_log_lines_sent_total",
|
|
||||||
Help: "Log lines sent to SWWAF_LOG_REMOTE_URL.",
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(remote.Sent())
|
|
||||||
}),
|
|
||||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
|
||||||
Name: "smallwebwaf_remote_log_lines_dropped_total",
|
|
||||||
Help: "Log lines dropped: the oldest in a full buffer, and those " +
|
|
||||||
"whose sending failed.",
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(remote.Dropped())
|
|
||||||
}),
|
|
||||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
|
||||||
Name: "smallwebwaf_remote_log_buffer_depth",
|
|
||||||
Help: "Log lines in the buffer, waiting to be sent.",
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(remote.Depth())
|
|
||||||
}),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ServeHTTP answers with the metrics in the Prometheus text format.
|
// ServeHTTP answers with the metrics in the Prometheus text format.
|
||||||
func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||||
m.handler.ServeHTTP(w, r)
|
m.handler.ServeHTTP(w, r)
|
||||||
@@ -271,12 +219,6 @@ func (m *Metrics) RequestEnded(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// RuleMatched counts a request that matched the rule id, whose action is
|
|
||||||
// action.
|
|
||||||
func (m *Metrics) RuleMatched(id, action string) {
|
|
||||||
m.ruleMatches.WithLabelValues(id, action).Inc()
|
|
||||||
}
|
|
||||||
|
|
||||||
// StateFileWritten counts a write of the state file name, of size bytes,
|
// StateFileWritten counts a write of the state file name, of size bytes,
|
||||||
// that ended with err.
|
// that ended with err.
|
||||||
func (m *Metrics) StateFileWritten(name string, size int, err error) {
|
func (m *Metrics) StateFileWritten(name string, size int, err error) {
|
||||||
|
|||||||
+19
-49
@@ -6,7 +6,6 @@ import (
|
|||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// banResponse is a refusal answered with SWWAF_BAN_RESPONSE, and logged
|
// banResponse is a refusal answered with SWWAF_BAN_RESPONSE, and logged
|
||||||
@@ -31,17 +30,14 @@ func (rq *request) banned(now time.Time) bool {
|
|||||||
return banned
|
return banned
|
||||||
}
|
}
|
||||||
|
|
||||||
// limitBroken counts the request for the rate limits at now, notes the
|
// limitBroken counts the request for the rate limits at now, and reports
|
||||||
// client's counts for the log line, and reports whether the request takes
|
// whether it takes the client over one. In enforce mode such a request
|
||||||
// the client over a limit. In enforce mode such a request bans the
|
// bans the client's netblock, and sets the client's counters back to
|
||||||
// client's netblock, and sets the client's counters back to zero; in
|
// zero; in observe mode it does neither.
|
||||||
// observe mode it does neither.
|
|
||||||
func (rq *request) limitBroken(now time.Time) bool {
|
func (rq *request) limitBroken(now time.Time) bool {
|
||||||
group := clientGroup(rq.client)
|
group := clientGroup(rq.client)
|
||||||
|
|
||||||
counts, hit, over := rq.h.limiter.Count(group, now)
|
hit, over := rq.h.limiter.Count(group, now)
|
||||||
rq.line.Counts = counts
|
|
||||||
|
|
||||||
if !over {
|
if !over {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
@@ -55,12 +51,20 @@ func (rq *request) limitBroken(now time.Time) bool {
|
|||||||
|
|
||||||
netblock := rq.netblock()
|
netblock := rq.netblock()
|
||||||
ban := rq.h.ledger.BanForLimit(netblock, now, bans.Notes{
|
ban := rq.h.ledger.BanForLimit(netblock, now, bans.Notes{
|
||||||
Country: rq.line.Country,
|
Country: rq.line.Country,
|
||||||
Limit: hit.Limit,
|
Limit: hit.Limit,
|
||||||
Window: hit.Window,
|
Window: hit.Window,
|
||||||
Count: hit.Requests,
|
Count: hit.Requests,
|
||||||
Request: rq.noted(now),
|
Request: bans.Request{
|
||||||
Requests: rq.netblockRequests(netblock),
|
Time: now,
|
||||||
|
Method: rq.in.Method,
|
||||||
|
Host: rq.in.Host,
|
||||||
|
Path: rq.in.URL.RequestURI(),
|
||||||
|
Status: rq.h.config.BanResponse,
|
||||||
|
UserAgent: rq.in.UserAgent(),
|
||||||
|
},
|
||||||
|
// The histories count this request only once it has ended.
|
||||||
|
Requests: rq.h.limiter.Requests(netblock) + 1,
|
||||||
})
|
})
|
||||||
rq.h.limiter.Reset(group)
|
rq.h.limiter.Reset(group)
|
||||||
rq.line.BanExpires = banExpires(ban)
|
rq.line.BanExpires = banExpires(ban)
|
||||||
@@ -68,40 +72,6 @@ func (rq *request) limitBroken(now time.Time) bool {
|
|||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
// banForAttack bans the client's netblock at now for a clear sign of
|
|
||||||
// attack, the match of rule, a ban rule.
|
|
||||||
func (rq *request) banForAttack(now time.Time, rule rules.Rule) {
|
|
||||||
netblock := rq.netblock()
|
|
||||||
ban := rq.h.ledger.BanForAttack(netblock, now, bans.Notes{
|
|
||||||
Country: rq.line.Country,
|
|
||||||
RuleID: rule.ID,
|
|
||||||
Target: rule.Target,
|
|
||||||
Request: rq.noted(now),
|
|
||||||
Requests: rq.netblockRequests(netblock),
|
|
||||||
})
|
|
||||||
rq.line.BanExpires = banExpires(ban)
|
|
||||||
}
|
|
||||||
|
|
||||||
// noted is the request, refused at now with SWWAF_BAN_RESPONSE, as the
|
|
||||||
// notes of the ban it makes keep it.
|
|
||||||
func (rq *request) noted(now time.Time) bans.Request {
|
|
||||||
return bans.Request{
|
|
||||||
Time: now,
|
|
||||||
Method: rq.in.Method,
|
|
||||||
Host: rq.in.Host,
|
|
||||||
Path: rq.in.URL.RequestURI(),
|
|
||||||
Status: rq.h.config.BanResponse,
|
|
||||||
UserAgent: rq.in.UserAgent(),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// netblockRequests is how many requests netblock has sent since it was
|
|
||||||
// first seen, this one included: the histories count it only once it has
|
|
||||||
// ended.
|
|
||||||
func (rq *request) netblockRequests(netblock netip.Prefix) int64 {
|
|
||||||
return rq.h.limiter.Requests(netblock) + 1
|
|
||||||
}
|
|
||||||
|
|
||||||
// netblock is the netblock a ban on the client covers: its IPv4 address,
|
// netblock is the netblock a ban on the client covers: its IPv4 address,
|
||||||
// widened to SWWAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
|
// widened to SWWAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
|
||||||
// counts it in.
|
// counts it in.
|
||||||
|
|||||||
@@ -278,8 +278,6 @@ func TestBanNotes(t *testing.T) {
|
|||||||
Netblock: netblock,
|
Netblock: netblock,
|
||||||
Start: start,
|
Start: start,
|
||||||
Expires: start.Add(time.Hour),
|
Expires: start.Add(time.Hour),
|
||||||
Cause: bans.CauseLimit,
|
|
||||||
Reason: "requests per minute over the limit of 1",
|
|
||||||
Notes: bans.Notes{
|
Notes: bans.Notes{
|
||||||
Country: "DE",
|
Country: "DE",
|
||||||
Limit: 1,
|
Limit: 1,
|
||||||
@@ -297,7 +295,7 @@ func TestBanNotes(t *testing.T) {
|
|||||||
// refused under the ban.
|
// refused under the ban.
|
||||||
Requests: 4,
|
Requests: 4,
|
||||||
Refused: 2,
|
Refused: 2,
|
||||||
EarlierBans: bans.EarlierBans{},
|
EarlierBans: 0,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -314,8 +312,8 @@ func TestBanNotes(t *testing.T) {
|
|||||||
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||||
|
|
||||||
got = ledger.Bans(netblock)
|
got = ledger.Bans(netblock)
|
||||||
if len(got) != 2 || got[1].Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
|
if len(got) != 2 || got[1].Notes.EarlierBans != 1 {
|
||||||
t.Errorf("bans %+v, want two, the second with one earlier ban for a limit", got)
|
t.Errorf("bans %+v, want two, the second with one earlier ban", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
package proxy
|
package proxy
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"crypto/rand"
|
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"slices"
|
"slices"
|
||||||
@@ -49,33 +48,6 @@ func clientAddress(
|
|||||||
return client
|
return client
|
||||||
}
|
}
|
||||||
|
|
||||||
// requestIDHeader carries the request's id, from traefik and to the app.
|
|
||||||
const requestIDHeader = "X-Request-ID"
|
|
||||||
|
|
||||||
// requestID is the request's id: the one a trusted proxy sent, or a new
|
|
||||||
// random one. A peer outside the trusted proxies did not come through
|
|
||||||
// traefik, so the id it sends is its own claim, and is replaced.
|
|
||||||
func requestID(r *http.Request, peerTrusted bool) string {
|
|
||||||
id := r.Header.Get(requestIDHeader)
|
|
||||||
if !peerTrusted || id == "" {
|
|
||||||
id = rand.Text()
|
|
||||||
}
|
|
||||||
|
|
||||||
return id
|
|
||||||
}
|
|
||||||
|
|
||||||
// scheme is how the client reached traefik, as a trusted proxy says in
|
|
||||||
// X-Forwarded-Proto, or otherwise http, the only scheme smallwebwaf
|
|
||||||
// serves.
|
|
||||||
func scheme(r *http.Request, peerTrusted bool) string {
|
|
||||||
proto := r.Header.Get("X-Forwarded-Proto")
|
|
||||||
if !peerTrusted || proto == "" {
|
|
||||||
return "http"
|
|
||||||
}
|
|
||||||
|
|
||||||
return proto
|
|
||||||
}
|
|
||||||
|
|
||||||
// ipv6GroupPrefix is the length of the IPv6 netblock that is one client.
|
// ipv6GroupPrefix is the length of the IPv6 netblock that is one client.
|
||||||
const ipv6GroupPrefix = 64
|
const ipv6GroupPrefix = 64
|
||||||
|
|
||||||
|
|||||||
@@ -14,14 +14,10 @@ const (
|
|||||||
appHost = "app.example"
|
appHost = "app.example"
|
||||||
// client is the client's address, as a proxy names it.
|
// client is the client's address, as a proxy names it.
|
||||||
client = "203.0.113.9"
|
client = "203.0.113.9"
|
||||||
// forwardedFor is the header that lists the client and its proxies,
|
// forwardedFor is the header that lists the client and its proxies.
|
||||||
// and forwardedProto the one that gives the scheme the client used.
|
forwardedFor = "X-Forwarded-For"
|
||||||
forwardedFor = "X-Forwarded-For"
|
// secure is the scheme a client reached traefik with.
|
||||||
forwardedProto = "X-Forwarded-Proto"
|
|
||||||
// secure is the scheme a client reached traefik with, and plain the
|
|
||||||
// one smallwebwaf serves.
|
|
||||||
secure = "https"
|
secure = "https"
|
||||||
plain = "http"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// appHeaders is what the app tells about the headers it received.
|
// appHeaders is what the app tells about the headers it received.
|
||||||
@@ -69,13 +65,13 @@ func TestClientAddressAndForwardedHeaders(t *testing.T) {
|
|||||||
func clientAddressCases() []clientAddressCase {
|
func clientAddressCases() []clientAddressCase {
|
||||||
trusted := map[string]string{trustedProxies: trustLocalhost}
|
trusted := map[string]string{trustedProxies: trustLocalhost}
|
||||||
forged := http.Header{
|
forged := http.Header{
|
||||||
forwardedFor: {client},
|
forwardedFor: {client},
|
||||||
"X-Forwarded-Host": {"forged.example"},
|
"X-Forwarded-Host": {"forged.example"},
|
||||||
forwardedProto: {secure},
|
"X-Forwarded-Proto": {secure},
|
||||||
"X-Real-Ip": {client},
|
"X-Real-Ip": {client},
|
||||||
}
|
}
|
||||||
replaced := appHeaders{
|
replaced := appHeaders{
|
||||||
ForwardedFor: localhost, ForwardedHost: appHost, ForwardedProto: plain,
|
ForwardedFor: localhost, ForwardedHost: appHost, ForwardedProto: "http",
|
||||||
}
|
}
|
||||||
|
|
||||||
return []clientAddressCase{{
|
return []clientAddressCase{{
|
||||||
@@ -91,10 +87,10 @@ func clientAddressCases() []clientAddressCase {
|
|||||||
"outside the trusted proxies from the right",
|
"outside the trusted proxies from the right",
|
||||||
env: trusted,
|
env: trusted,
|
||||||
header: http.Header{
|
header: http.Header{
|
||||||
forwardedFor: {"198.51.100.7, " + client + ", 10.0.0.2"},
|
forwardedFor: {"198.51.100.7, " + client + ", 10.0.0.2"},
|
||||||
"X-Forwarded-Host": {appHost},
|
"X-Forwarded-Host": {appHost},
|
||||||
forwardedProto: {secure},
|
"X-Forwarded-Proto": {secure},
|
||||||
"X-Real-Ip": {client},
|
"X-Real-Ip": {client},
|
||||||
},
|
},
|
||||||
wantClient: client,
|
wantClient: client,
|
||||||
wantApp: appHeaders{
|
wantApp: appHeaders{
|
||||||
@@ -142,7 +138,7 @@ func requestWithHeaders(
|
|||||||
Host: r.Host,
|
Host: r.Host,
|
||||||
ForwardedFor: r.Header.Get(forwardedFor),
|
ForwardedFor: r.Header.Get(forwardedFor),
|
||||||
ForwardedHost: r.Header.Get("X-Forwarded-Host"),
|
ForwardedHost: r.Header.Get("X-Forwarded-Host"),
|
||||||
ForwardedProto: r.Header.Get(forwardedProto),
|
ForwardedProto: r.Header.Get("X-Forwarded-Proto"),
|
||||||
RealIP: r.Header.Get("X-Real-IP"),
|
RealIP: r.Header.Get("X-Real-IP"),
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -21,18 +21,14 @@ func TestHealthEndpointIsAnsweredBeforeAnyCheck(t *testing.T) {
|
|||||||
// the last one would have it refused.
|
// the last one would have it refused.
|
||||||
addr, out := startProxy(t, app.URL, map[string]string{rateLimitPerMinute: "1"})
|
addr, out := startProxy(t, app.URL, map[string]string{rateLimitPerMinute: "1"})
|
||||||
|
|
||||||
const (
|
const healthChecks = 3
|
||||||
healthChecks = 3
|
|
||||||
contentType = "text/plain; charset=utf-8"
|
|
||||||
)
|
|
||||||
|
|
||||||
for range healthChecks {
|
for range healthChecks {
|
||||||
got := get(t, addr, proxy.HealthPath)
|
got := get(t, addr, proxy.HealthPath)
|
||||||
wantStatus(t, got, http.StatusOK)
|
wantStatus(t, got, http.StatusOK)
|
||||||
|
|
||||||
if string(got.body) != "ok\n" || got.header.Get("Content-Type") != contentType {
|
if string(got.body) != "ok\n" {
|
||||||
t.Errorf("health endpoint answered %q with Content-Type %q, want ok "+
|
t.Errorf("health endpoint answered %q, want ok", got.body)
|
||||||
"with %q", got.body, got.header.Get("Content-Type"), contentType)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -41,11 +37,6 @@ func TestHealthEndpointIsAnsweredBeforeAnyCheck(t *testing.T) {
|
|||||||
lines := out.requestLines(t, healthChecks+1)
|
lines := out.requestLines(t, healthChecks+1)
|
||||||
for _, line := range lines[:healthChecks] {
|
for _, line := range lines[:healthChecks] {
|
||||||
wantLine(t, line, http.StatusOK, requestlog.ActionAdmin)
|
wantLine(t, line, http.StatusOK, requestlog.ActionAdmin)
|
||||||
|
|
||||||
if line.ResponseContentType != contentType {
|
|
||||||
t.Errorf("health check's log line has response_content_type %q, "+
|
|
||||||
"want %q", line.ResponseContentType, contentType)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
wantLine(t, lines[healthChecks], http.StatusOK, requestlog.ActionForward)
|
wantLine(t, lines[healthChecks], http.StatusOK, requestlog.ActionForward)
|
||||||
|
|||||||
@@ -12,7 +12,6 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
@@ -244,29 +243,6 @@ func TestMetricsCountLimitsAndBans(t *testing.T) {
|
|||||||
wantMetric(t, metrics, "smallwebwaf_tracked_clients", 3)
|
wantMetric(t, metrics, "smallwebwaf_tracked_clients", 3)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestMetricsCountTheBansAnAdminMakes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const scraper = "192.0.2.200" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
|
|
||||||
|
|
||||||
s, clk, server := startWithClock(t, "", map[string]string{
|
|
||||||
metricsToken: token,
|
|
||||||
rateLimitExemptNets: scraper,
|
|
||||||
})
|
|
||||||
|
|
||||||
const admins = `smallwebwaf_bans_made_total{cause="admin"}`
|
|
||||||
|
|
||||||
wantMetric(t, s.scrape(scraper), admins, 0)
|
|
||||||
|
|
||||||
// As an admin's edit of bans.json that adds a ban is taken in.
|
|
||||||
server.Ledger.LoadEdit([]bans.Ban{{
|
|
||||||
Netblock: netip.MustParsePrefix(client + "/32"),
|
|
||||||
Start: clk.Now(),
|
|
||||||
}})
|
|
||||||
|
|
||||||
wantMetric(t, s.scrape(scraper), admins, 1)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) {
|
func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -95,7 +95,6 @@ func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
|
|||||||
Netblock: netip.MustParsePrefix(otherClient + "/32"),
|
Netblock: netip.MustParsePrefix(otherClient + "/32"),
|
||||||
Start: clk.Now(),
|
Start: clk.Now(),
|
||||||
Expires: clk.Now().Add(time.Hour),
|
Expires: clk.Now().Add(time.Hour),
|
||||||
Cause: bans.CauseAdmin,
|
|
||||||
}
|
}
|
||||||
server.Ledger.Load([]bans.Ban{kept})
|
server.Ledger.Load([]bans.Ban{kept})
|
||||||
|
|
||||||
|
|||||||
@@ -6,8 +6,6 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"os"
|
|
||||||
"reflect"
|
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
@@ -16,7 +14,6 @@ import (
|
|||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -118,34 +115,27 @@ func wantAnswer(t *testing.T, got answer, body []byte) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// wantRequestFields checks the log line's fields about the request. Its
|
// wantRequestFields checks the log line's fields about the request.
|
||||||
// time, its id and its timings are checked only for being there.
|
|
||||||
func wantRequestFields(t *testing.T, line logLine, host string, sent, received int) {
|
func wantRequestFields(t *testing.T, line logLine, host string, sent, received int) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
hostname, _ := os.Hostname()
|
want := requestlog.Line{
|
||||||
|
Type: "request", Time: line.Time, ClientIP: localhost, PeerIP: localhost,
|
||||||
want := withTimings(line, requestlog.Line{
|
Method: http.MethodPatch, Host: host, Path: rawPath, Query: rawQuery,
|
||||||
Type: requestType, Time: line.Time, Instance: hostname,
|
Protocol: "HTTP/1.1", Status: http.StatusTeapot,
|
||||||
ClientIP: localhost, Method: http.MethodPatch, Scheme: plain, Host: host,
|
UpstreamStatus: http.StatusTeapot, RequestBytes: int64(sent),
|
||||||
Path: rawPath, Query: rawQuery, Protocol: protocol,
|
|
||||||
Status: http.StatusTeapot, RequestBytes: int64(sent),
|
|
||||||
ResponseBytes: int64(received), UserAgent: "test-agent",
|
ResponseBytes: int64(received), UserAgent: "test-agent",
|
||||||
RequestID: line.RequestID, PeerIP: localhost, ClientGroup: localhost + "/32",
|
Action: requestlog.ActionForward, DurationTotal: line.DurationTotal,
|
||||||
ContentLength: int64(sent), ResponseContentType: "text/plain; charset=utf-8",
|
DurationUpstreamTotal: line.DurationUpstreamTotal,
|
||||||
UpstreamStatus: http.StatusTeapot, Action: requestlog.ActionForward,
|
}
|
||||||
Counts: ratelimit.Counts{Minute: 1, Hour: 1, Day: 1},
|
if line.Line != want {
|
||||||
})
|
|
||||||
if !reflect.DeepEqual(line.Line, want) {
|
|
||||||
t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
|
t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err := time.Parse(time.RFC3339, line.Time)
|
_, err := time.Parse(time.RFC3339, line.Time)
|
||||||
if err != nil || line.RequestID == "" || line.DurationTotal <= 0 ||
|
if err != nil || line.DurationTotal <= 0 || line.DurationUpstreamTotal <= 0 {
|
||||||
line.DurationUpstreamTotal == nil || *line.DurationUpstreamTotal <= 0 {
|
t.Errorf("log line has time %q and durations %v and %v",
|
||||||
t.Errorf("log line has time %q, request_id %q and durations %v and %v",
|
line.Time, line.DurationTotal, line.DurationUpstreamTotal)
|
||||||
line.Time, line.RequestID, line.DurationTotal,
|
|
||||||
line.fields["duration_upstream_total"])
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -381,13 +371,8 @@ func TestAnswers502WhenTheAppCannotBeReached(t *testing.T) {
|
|||||||
addr, out := startProxy(t, "http://"+localhost+":1", nil)
|
addr, out := startProxy(t, "http://"+localhost+":1", nil)
|
||||||
|
|
||||||
wantStatus(t, get(t, addr, "/"), http.StatusBadGateway)
|
wantStatus(t, get(t, addr, "/"), http.StatusBadGateway)
|
||||||
|
wantLine(t, out.requestLine(t), http.StatusBadGateway,
|
||||||
line := out.requestLine(t)
|
requestlog.ActionUpstreamError)
|
||||||
wantLine(t, line, http.StatusBadGateway, requestlog.ActionUpstreamError)
|
|
||||||
|
|
||||||
// There never was a connection to the app, nor an answer from it.
|
|
||||||
wantTimings(t, line, "duration_total", "duration_checks",
|
|
||||||
"duration_upstream_total")
|
|
||||||
|
|
||||||
logged := slices.ContainsFunc(out.lines(t), func(line map[string]any) bool {
|
logged := slices.ContainsFunc(out.lines(t), func(line map[string]any) bool {
|
||||||
return line["type"] == "process" && line["msg"] == "request to the app failed"
|
return line["type"] == "process" && line["msg"] == "request to the app failed"
|
||||||
|
|||||||
@@ -17,7 +17,6 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// How smallwebwaf keeps connections to the app open between requests.
|
// How smallwebwaf keeps connections to the app open between requests.
|
||||||
@@ -52,9 +51,6 @@ type Params struct {
|
|||||||
// limits, bans are made and run out, and GeoJS's answers are kept,
|
// limits, bans are made and run out, and GeoJS's answers are kept,
|
||||||
// normally time.Now in UTC, the time the state files give.
|
// normally time.Now in UTC, the time the state files give.
|
||||||
Now func() time.Time
|
Now func() time.Time
|
||||||
// Rules are the rule files' rules, which each request is checked
|
|
||||||
// against.
|
|
||||||
Rules *rules.Files
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Server is the server smallwebwaf runs, with the parts of the proxy
|
// Server is the server smallwebwaf runs, with the parts of the proxy
|
||||||
@@ -94,7 +90,6 @@ func New(params Params) *Server {
|
|||||||
LimitBanDuration: params.Config.LimitBanDuration,
|
LimitBanDuration: params.Config.LimitBanDuration,
|
||||||
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
|
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
|
||||||
MaxBanDuration: params.Config.MaxBanDuration,
|
MaxBanDuration: params.Config.MaxBanDuration,
|
||||||
AttackBanDuration: params.Config.AttackBanDuration,
|
|
||||||
MaxBans: params.Config.MaxBans,
|
MaxBans: params.Config.MaxBans,
|
||||||
}),
|
}),
|
||||||
geojs: lookup.New(lookup.Params{
|
geojs: lookup.New(lookup.Params{
|
||||||
@@ -103,10 +98,8 @@ func New(params Params) *Server {
|
|||||||
ProcessLog: params.ProcessLog,
|
ProcessLog: params.ProcessLog,
|
||||||
Metrics: m,
|
Metrics: m,
|
||||||
}),
|
}),
|
||||||
rules: params.Rules,
|
|
||||||
}
|
}
|
||||||
m.AddBansAndClients(h.ledger, h.limiter, params.Now)
|
m.AddBansAndClients(h.ledger, h.limiter, params.Now)
|
||||||
m.AddRules(params.Rules)
|
|
||||||
|
|
||||||
return &Server{
|
return &Server{
|
||||||
Server: &http.Server{
|
Server: &http.Server{
|
||||||
@@ -141,7 +134,6 @@ type handler struct {
|
|||||||
limiter *ratelimit.Limiter
|
limiter *ratelimit.Limiter
|
||||||
ledger *bans.Ledger
|
ledger *bans.Ledger
|
||||||
geojs *lookup.GeoJS
|
geojs *lookup.GeoJS
|
||||||
rules *rules.Files
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// newTransport returns what carries requests to the app. It never goes
|
// newTransport returns what carries requests to the app. It never goes
|
||||||
@@ -168,9 +160,6 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
// a health checker is never refused. It does not ask the app.
|
// a health checker is never refused. It does not ask the app.
|
||||||
if r.Method == http.MethodGet && r.URL.Path == HealthPath {
|
if r.Method == http.MethodGet && r.URL.Path == HealthPath {
|
||||||
rq.line.Action = requestlog.ActionAdmin
|
rq.line.Action = requestlog.ActionAdmin
|
||||||
// Set here rather than left to Go's server, which would set it only
|
|
||||||
// after the log line has taken the response's headers.
|
|
||||||
rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
|
||||||
_, _ = io.WriteString(rq.out, "ok\n")
|
_, _ = io.WriteString(rq.out, "ok\n")
|
||||||
|
|
||||||
return
|
return
|
||||||
@@ -180,8 +169,6 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
defer rq.addToHistory()
|
defer rq.addToHistory()
|
||||||
|
|
||||||
refused := rq.check(r.Context())
|
refused := rq.check(r.Context())
|
||||||
rq.checked = time.Now()
|
|
||||||
|
|
||||||
if refused != nil {
|
if refused != nil {
|
||||||
rq.answer(*refused)
|
rq.answer(*refused)
|
||||||
|
|
||||||
|
|||||||
@@ -17,7 +17,6 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -36,10 +35,6 @@ const (
|
|||||||
// localhost is where every test server listens, and so the address
|
// localhost is where every test server listens, and so the address
|
||||||
// smallwebwaf sees each test's requests come from.
|
// smallwebwaf sees each test's requests come from.
|
||||||
localhost = "127.0.0.1"
|
localhost = "127.0.0.1"
|
||||||
// requestType is the type that marks a request log line.
|
|
||||||
requestType = "request"
|
|
||||||
// protocol is the protocol of every test's requests.
|
|
||||||
protocol = "HTTP/1.1"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// shortTimeoutSetting is shortTimeout as a setting's value.
|
// shortTimeoutSetting is shortTimeout as a setting's value.
|
||||||
@@ -73,10 +68,6 @@ const (
|
|||||||
maxBanDuration = "SWWAF_MAX_BAN_DURATION"
|
maxBanDuration = "SWWAF_MAX_BAN_DURATION"
|
||||||
maxBans = "SWWAF_MAX_BANS"
|
maxBans = "SWWAF_MAX_BANS"
|
||||||
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
|
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
|
||||||
instanceName = "SWWAF_INSTANCE_NAME"
|
|
||||||
logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS"
|
|
||||||
attackBanDuration = "SWWAF_ATTACK_BAN_DURATION"
|
|
||||||
rulesDir = "SWWAF_RULES_DIR"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// output collects what smallwebwaf writes on stdout.
|
// output collects what smallwebwaf writes on stdout.
|
||||||
@@ -93,14 +84,6 @@ func (o *output) Write(p []byte) (int, error) {
|
|||||||
return o.buf.Write(p)
|
return o.buf.Write(p)
|
||||||
}
|
}
|
||||||
|
|
||||||
// text returns everything written so far.
|
|
||||||
func (o *output) text() string {
|
|
||||||
o.mu.Lock()
|
|
||||||
defer o.mu.Unlock()
|
|
||||||
|
|
||||||
return o.buf.String()
|
|
||||||
}
|
|
||||||
|
|
||||||
// lines returns every line written so far, decoded.
|
// lines returns every line written so far, decoded.
|
||||||
func (o *output) lines(t *testing.T) []map[string]any {
|
func (o *output) lines(t *testing.T) []map[string]any {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
@@ -140,7 +123,7 @@ func (o *output) requestLines(t *testing.T, count int) []logLine {
|
|||||||
var found []logLine
|
var found []logLine
|
||||||
|
|
||||||
for _, fields := range o.lines(t) {
|
for _, fields := range o.lines(t) {
|
||||||
if fields["type"] == requestType {
|
if fields["type"] == "request" {
|
||||||
found = append(found, decodeLine(t, fields))
|
found = append(found, decodeLine(t, fields))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -215,15 +198,14 @@ func startProxyWithGeoJS(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// startProxyWithClock is startProxyWithGeoJS with requests counted and
|
// startProxyWithClock is startProxyWithGeoJS with requests counted and
|
||||||
// bans made by the time now tells, and returns the server as well. Unless
|
// bans made by the time now tells, and returns the server as well.
|
||||||
// env sets SWWAF_RULES_DIR, it is an empty directory, of no rules.
|
|
||||||
func startProxyWithClock(
|
func startProxyWithClock(
|
||||||
t *testing.T, appURL, geojsURL string, now func() time.Time,
|
t *testing.T, appURL, geojsURL string, now func() time.Time,
|
||||||
env map[string]string,
|
env map[string]string,
|
||||||
) (string, *output, *proxy.Server) {
|
) (string, *output, *proxy.Server) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
settings := map[string]string{"SWWAF_UPSTREAM_URL": appURL, rulesDir: t.TempDir()}
|
settings := map[string]string{"SWWAF_UPSTREAM_URL": appURL}
|
||||||
maps.Copy(settings, env)
|
maps.Copy(settings, env)
|
||||||
|
|
||||||
cfg, err := config.FromEnvironment(func(name string) (string, bool) {
|
cfg, err := config.FromEnvironment(func(name string) (string, bool) {
|
||||||
@@ -236,22 +218,12 @@ func startProxyWithClock(
|
|||||||
}
|
}
|
||||||
|
|
||||||
out := &output{}
|
out := &output{}
|
||||||
processLog := requestlog.NewProcessLogger(out)
|
|
||||||
|
|
||||||
ruleFiles, err := rules.Load(rules.Params{
|
|
||||||
Dir: cfg.RulesDir, Enabled: cfg.RulesEnabled, ProcessLog: processLog,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("rule files: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
server := proxy.New(proxy.Params{
|
server := proxy.New(proxy.Params{
|
||||||
Config: cfg,
|
Config: cfg,
|
||||||
RequestLog: out,
|
RequestLog: out,
|
||||||
ProcessLog: processLog,
|
ProcessLog: requestlog.NewProcessLogger(out),
|
||||||
GeoJSURL: geojsURL,
|
GeoJSURL: geojsURL,
|
||||||
Now: now,
|
Now: now,
|
||||||
Rules: ruleFiles,
|
|
||||||
})
|
})
|
||||||
|
|
||||||
listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
|
listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
|
||||||
|
|||||||
@@ -6,7 +6,6 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -98,9 +97,8 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
|
|||||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||||
|
|
||||||
line := s.request(client, "/assets/app.js", http.StatusOK, requestlog.ActionForward)
|
line := s.request(client, "/assets/app.js", http.StatusOK, requestlog.ActionForward)
|
||||||
if line.LimitHit != "" || line.Counts != (ratelimit.Counts{}) {
|
if line.LimitHit != "" {
|
||||||
t.Errorf("log line has limit_hit %q and counts %+v, want neither",
|
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
|
||||||
line.LimitHit, line.Counts)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// A path outside every prefix is counted: /assets is not under
|
// A path outside every prefix is counted: /assets is not under
|
||||||
@@ -121,10 +119,6 @@ func TestRateLimitCountsPathsThatAreNotExempt(t *testing.T) {
|
|||||||
for _, sent := range []string{
|
for _, sent := range []string{
|
||||||
// A prefix matches only at the start of the path.
|
// A prefix matches only at the start of the path.
|
||||||
"/static/assets/app.js",
|
"/static/assets/app.js",
|
||||||
// A prefix matches the path as sent: a router that matches the
|
|
||||||
// path as received does not take /%61ssets/x for a path under
|
|
||||||
// /assets/.
|
|
||||||
"/%61ssets/x",
|
|
||||||
// .. once percent-decoded: an app may act on these as /login, the
|
// .. once percent-decoded: an app may act on these as /login, the
|
||||||
// last as a path under /sneak/app/ or as /assets/x.
|
// last as a path under /sneak/app/ or as /assets/x.
|
||||||
"/assets/../login",
|
"/assets/../login",
|
||||||
@@ -132,14 +126,10 @@ func TestRateLimitCountsPathsThatAreNotExempt(t *testing.T) {
|
|||||||
"/assets/..%2Flogin",
|
"/assets/..%2Flogin",
|
||||||
"/assets/..;/login",
|
"/assets/..;/login",
|
||||||
"/sneak/app/src/branch/main/..%2F..%2F..%2F..%2F..%2F..%2Fassets/x",
|
"/sneak/app/src/branch/main/..%2F..%2F..%2F..%2F..%2F..%2Fassets/x",
|
||||||
// Not under /assets/ as sent: Go's router takes /assets%2Fx for one
|
// An encoded slash or a backslash: Go's router takes /assets%2Fx
|
||||||
// path segment, not a path under /assets/.
|
// for one path segment, not a path under /assets/.
|
||||||
"/assets%2Fx",
|
"/assets%2Fx",
|
||||||
"/assets%2fx",
|
"/assets%2fx",
|
||||||
// Under /assets/ as sent, but holding an encoded slash, in either
|
|
||||||
// case, or a backslash: never exempt, whatever the prefix.
|
|
||||||
"/assets/x%2Fy",
|
|
||||||
"/assets/x%2fy",
|
|
||||||
`/assets/x\y`,
|
`/assets/x\y`,
|
||||||
} {
|
} {
|
||||||
t.Run(sent, func(t *testing.T) {
|
t.Run(sent, func(t *testing.T) {
|
||||||
|
|||||||
+43
-138
@@ -49,9 +49,7 @@ type request struct {
|
|||||||
peer netip.Addr
|
peer netip.Addr
|
||||||
peerTrusted bool
|
peerTrusted bool
|
||||||
start time.Time
|
start time.Time
|
||||||
// checked is when the checks were done, and upstreamStart when the
|
// upstreamStart is when the request was handed to the app.
|
||||||
// request was handed to the app.
|
|
||||||
checked time.Time
|
|
||||||
upstreamStart time.Time
|
upstreamStart time.Time
|
||||||
// cancel ends the request to the app.
|
// cancel ends the request to the app.
|
||||||
cancel context.CancelFunc
|
cancel context.CancelFunc
|
||||||
@@ -61,34 +59,26 @@ type request struct {
|
|||||||
complete bool
|
complete bool
|
||||||
|
|
||||||
// mu guards what follows. The timeouts run on goroutines of their
|
// mu guards what follows. The timeouts run on goroutines of their
|
||||||
// own, and the transport starts and stops them, and notes the times
|
// own, and the transport starts and stops them from its own; once
|
||||||
// below, from its own; once timersStopped is set, none of the timeouts
|
// timersStopped is set, none of them acts any more.
|
||||||
// acts any more.
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
timersStopped bool
|
timersStopped bool
|
||||||
clientRequestTimer *time.Timer
|
clientRequestTimer *time.Timer
|
||||||
upstreamRequestTimer *time.Timer
|
upstreamRequestTimer *time.Timer
|
||||||
upstreamResponseTimer *time.Timer
|
upstreamResponseTimer *time.Timer
|
||||||
// connected is when there was a connection to the app, requestSent
|
// requestSent is when the app had been sent the whole request.
|
||||||
// when the app had been sent the whole request, and answerStarted
|
requestSent time.Time
|
||||||
// when the first byte of its answer arrived.
|
|
||||||
connected time.Time
|
|
||||||
requestSent time.Time
|
|
||||||
answerStarted time.Time
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// newRequest starts handling r: it notes the time, counts the request as
|
// newRequest starts handling r: it notes the time, counts the request as
|
||||||
// under way, works out the client, and starts the log line with what is
|
// under way, and works out the client.
|
||||||
// known of the request.
|
|
||||||
func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
|
func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
|
||||||
h.metrics.RequestStarted()
|
h.metrics.RequestStarted()
|
||||||
|
|
||||||
start := time.Now()
|
start := time.Now()
|
||||||
peer := peerAddress(r)
|
peer := peerAddress(r)
|
||||||
trusted := h.config.TrustedProxies
|
trusted := h.config.TrustedProxies
|
||||||
peerTrusted := isInside(peer, trusted)
|
client := clientAddress(peer, r.Header.Values("X-Forwarded-For"), trusted)
|
||||||
forwardedFor := r.Header.Values("X-Forwarded-For")
|
|
||||||
client := clientAddress(peer, forwardedFor, trusted)
|
|
||||||
|
|
||||||
rq := &request{
|
rq := &request{
|
||||||
h: h,
|
h: h,
|
||||||
@@ -97,37 +87,22 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
|
|||||||
out: &responseWriter{ResponseWriter: w},
|
out: &responseWriter{ResponseWriter: w},
|
||||||
client: client,
|
client: client,
|
||||||
peer: peer,
|
peer: peer,
|
||||||
peerTrusted: peerTrusted,
|
peerTrusted: isInside(peer, trusted),
|
||||||
start: start,
|
start: start,
|
||||||
line: requestlog.Line{
|
line: requestlog.Line{
|
||||||
Time: requestlog.FormatTime(start),
|
Time: requestlog.FormatTime(start),
|
||||||
Instance: h.config.InstanceName,
|
ClientIP: client.String(),
|
||||||
ClientIP: client.String(),
|
PeerIP: peer.String(),
|
||||||
Method: r.Method,
|
Method: r.Method,
|
||||||
Scheme: scheme(r, peerTrusted),
|
Host: r.Host,
|
||||||
Host: r.Host,
|
Path: r.URL.EscapedPath(),
|
||||||
Path: r.URL.EscapedPath(),
|
Query: r.URL.RawQuery,
|
||||||
Query: r.URL.RawQuery,
|
Protocol: r.Proto,
|
||||||
Protocol: r.Proto,
|
Referer: r.Referer(),
|
||||||
Referer: r.Referer(),
|
UserAgent: r.UserAgent(),
|
||||||
UserAgent: r.UserAgent(),
|
Action: requestlog.ActionForward,
|
||||||
RequestID: requestID(r, peerTrusted),
|
|
||||||
PeerIP: peer.String(),
|
|
||||||
ForwardedFor: strings.Join(forwardedFor, ", "),
|
|
||||||
ClientGroup: clientGroup(client).String(),
|
|
||||||
ContentType: r.Header.Get("Content-Type"),
|
|
||||||
RequestHeaders: requestHeaders(r, h.config.LogRequestHeaders),
|
|
||||||
HasAuthorization: len(r.Header.Values("Authorization")) > 0,
|
|
||||||
HasCookie: len(r.Header.Values("Cookie")) > 0,
|
|
||||||
Action: requestlog.ActionForward,
|
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
// A length of -1 is a body whose length was not announced.
|
|
||||||
if r.ContentLength > 0 {
|
|
||||||
rq.line.ContentLength = r.ContentLength
|
|
||||||
}
|
|
||||||
|
|
||||||
if r.Body != http.NoBody {
|
if r.Body != http.NoBody {
|
||||||
rq.body = &requestBody{body: limitBody(r.Body, h.config.RequestMaxBytes), rq: rq}
|
rq.body = &requestBody{body: limitBody(r.Body, h.config.RequestMaxBytes), rq: rq}
|
||||||
}
|
}
|
||||||
@@ -135,47 +110,23 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
|
|||||||
return rq
|
return rq
|
||||||
}
|
}
|
||||||
|
|
||||||
// requestHeaders returns the headers of r that names lists, by name in
|
|
||||||
// lower case, each with its values joined by ", ". Authorization, Cookie
|
|
||||||
// and Set-Cookie are never among them, whatever names says.
|
|
||||||
func requestHeaders(r *http.Request, names []string) map[string]string {
|
|
||||||
headers := map[string]string{}
|
|
||||||
|
|
||||||
for _, name := range names {
|
|
||||||
switch name {
|
|
||||||
case "authorization", "cookie", "set-cookie":
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
values := r.Header.Values(name)
|
|
||||||
if len(values) > 0 {
|
|
||||||
headers[name] = strings.Join(values, ", ")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return headers
|
|
||||||
}
|
|
||||||
|
|
||||||
// check is the one place where a request can be refused once its client
|
// check is the one place where a request can be refused once its client
|
||||||
// is known, before its body is read or anything reaches the app. It
|
// is known, before its body is read or anything reaches the app. It
|
||||||
// returns nil to let the request through. The checks of checkClient come
|
// returns nil to let the request through. The checks of checkClient come
|
||||||
// first, answered with SWWAF_BAN_RESPONSE, or 403 for a block rule, and
|
// first, answered with SWWAF_BAN_RESPONSE, and then the size limit, so
|
||||||
// then the size limit, so that a request the rate limits count is counted
|
// that a request the rate limits count is counted even when it is
|
||||||
// even when it is refused for its size. In observe mode a request
|
// refused for its size. In observe mode a request checkClient refuses
|
||||||
// checkClient refuses goes on to the size limit like any other. ctx is
|
// goes on to the size limit like any other. ctx is the request's own
|
||||||
// the request's own context.
|
// context.
|
||||||
func (rq *request) check(ctx context.Context) *refusal {
|
func (rq *request) check(ctx context.Context) *refusal {
|
||||||
action := rq.checkClient(ctx)
|
action := rq.checkClient(ctx)
|
||||||
|
if action != "" {
|
||||||
|
if !rq.h.config.Observe {
|
||||||
|
return rq.banResponse(action)
|
||||||
|
}
|
||||||
|
|
||||||
switch {
|
|
||||||
case action == "":
|
|
||||||
case rq.h.config.Observe:
|
|
||||||
// The log line names what enforce mode would have done.
|
// The log line names what enforce mode would have done.
|
||||||
rq.line.WouldAction = action
|
rq.line.WouldAction = action
|
||||||
case action == requestlog.ActionRuleBlocked:
|
|
||||||
return &refusal{status: http.StatusForbidden, action: action}
|
|
||||||
default:
|
|
||||||
return rq.banResponse(action)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
maxBytes := rq.h.config.RequestMaxBytes
|
maxBytes := rq.h.config.RequestMaxBytes
|
||||||
@@ -198,8 +149,8 @@ func (rq *request) check(ctx context.Context) *refusal {
|
|||||||
// request any of them refuses is not counted for the rate limits. Then
|
// request any of them refuses is not counted for the rate limits. Then
|
||||||
// come the rate limits, unless the client is in
|
// come the rate limits, unless the client is in
|
||||||
// SWWAF_RATE_LIMIT_EXEMPT_NETS or the request's path is exempt under
|
// SWWAF_RATE_LIMIT_EXEMPT_NETS or the request's path is exempt under
|
||||||
// SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request is counted,
|
// SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request is counted.
|
||||||
// and last the rule files. ctx is the request's own context.
|
// ctx is the request's own context.
|
||||||
func (rq *request) checkClient(ctx context.Context) string {
|
func (rq *request) checkClient(ctx context.Context) string {
|
||||||
cfg := rq.h.config
|
cfg := rq.h.config
|
||||||
if isInside(rq.client, cfg.AllowNets) {
|
if isInside(rq.client, cfg.AllowNets) {
|
||||||
@@ -226,30 +177,28 @@ func (rq *request) checkClient(ctx context.Context) string {
|
|||||||
return requestlog.ActionRateLimited
|
return requestlog.ActionRateLimited
|
||||||
}
|
}
|
||||||
|
|
||||||
return rq.checkRules(now)
|
return ""
|
||||||
}
|
}
|
||||||
|
|
||||||
// pathExempt reports whether the rate limits leave out a request for u
|
// pathExempt reports whether the rate limits leave out a request for u
|
||||||
// because of SWWAF_RATE_LIMIT_EXEMPT_PATHS: whether its path as sent, the
|
// because of SWWAF_RATE_LIMIT_EXEMPT_PATHS: whether its path,
|
||||||
// path the app receives, not percent-decoded, starts with one of
|
// percent-decoded, starts with one of prefixes. A request whose decoded
|
||||||
// prefixes, so that /%61ssets/x is not under /assets/ for an app whose
|
// path contains .. anywhere or a backslash, or whose path as sent holds
|
||||||
// router matches the path as received. A request whose decoded path
|
// an encoded slash (%2F or %2f), never is, since an app may act on it as
|
||||||
// contains .. anywhere or a backslash, or whose path as sent holds an
|
// a path outside every prefix: /assets/..%2Flogin as /login, or
|
||||||
// encoded slash (%2F or %2f), never is, since an app may act on it as a
|
// /assets%2Fx as one path segment, as Go's router does.
|
||||||
// path outside every prefix: /assets/..%2Flogin as /login, or /assets%2Fx
|
|
||||||
// as one path segment, as Go's router does.
|
|
||||||
func pathExempt(u *url.URL, prefixes []string) bool {
|
func pathExempt(u *url.URL, prefixes []string) bool {
|
||||||
decoded := u.Path
|
decoded := u.Path
|
||||||
// EscapedPath is the path as the app receives it, not decoded.
|
// EscapedPath is the path as the app receives it, not decoded.
|
||||||
sent := u.EscapedPath()
|
sent := strings.ToLower(u.EscapedPath())
|
||||||
|
|
||||||
if strings.Contains(decoded, "..") || strings.Contains(decoded, `\`) ||
|
if strings.Contains(decoded, "..") || strings.Contains(decoded, `\`) ||
|
||||||
strings.Contains(strings.ToLower(sent), "%2f") {
|
strings.Contains(sent, "%2f") {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
return slices.ContainsFunc(prefixes, func(prefix string) bool {
|
return slices.ContainsFunc(prefixes, func(prefix string) bool {
|
||||||
return strings.HasPrefix(sent, prefix)
|
return strings.HasPrefix(decoded, prefix)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -261,9 +210,7 @@ func (rq *request) forward(ctx context.Context) {
|
|||||||
|
|
||||||
rq.cancel = cancel
|
rq.cancel = cancel
|
||||||
ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{
|
ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{
|
||||||
GotConn: rq.gotConn,
|
WroteRequest: rq.wroteRequest,
|
||||||
WroteRequest: rq.wroteRequest,
|
|
||||||
GotFirstResponseByte: rq.gotFirstResponseByte,
|
|
||||||
})
|
})
|
||||||
|
|
||||||
out := rq.in.WithContext(ctx)
|
out := rq.in.WithContext(ctx)
|
||||||
@@ -286,8 +233,7 @@ func (rq *request) forward(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// rewrite makes the request the app receives: the client's request,
|
// rewrite makes the request the app receives: the client's request,
|
||||||
// unchanged, sent to SWWAF_UPSTREAM_URL, with the forwarded headers and
|
// unchanged, sent to SWWAF_UPSTREAM_URL, with the forwarded headers set.
|
||||||
// the request's id set.
|
|
||||||
func (rq *request) rewrite(pr *httputil.ProxyRequest) {
|
func (rq *request) rewrite(pr *httputil.ProxyRequest) {
|
||||||
upstream := rq.h.config.UpstreamURL
|
upstream := rq.h.config.UpstreamURL
|
||||||
pr.Out.URL.Scheme = upstream.Scheme
|
pr.Out.URL.Scheme = upstream.Scheme
|
||||||
@@ -296,7 +242,6 @@ func (rq *request) rewrite(pr *httputil.ProxyRequest) {
|
|||||||
// the query as the client sent it.
|
// the query as the client sent it.
|
||||||
pr.Out.URL.RawQuery = pr.In.URL.RawQuery
|
pr.Out.URL.RawQuery = pr.In.URL.RawQuery
|
||||||
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
|
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
|
||||||
pr.Out.Header.Set(requestIDHeader, rq.line.RequestID)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// modifyResponse looks at the app's answer before ReverseProxy passes it
|
// modifyResponse looks at the app's answer before ReverseProxy passes it
|
||||||
@@ -310,7 +255,6 @@ func (rq *request) modifyResponse(res *http.Response) error {
|
|||||||
// connection it takes over, not through rq.out.
|
// connection it takes over, not through rq.out.
|
||||||
rq.stopTimers()
|
rq.stopTimers()
|
||||||
rq.out.status = res.StatusCode
|
rq.out.status = res.StatusCode
|
||||||
rq.line.Websocket = true
|
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -406,10 +350,6 @@ func (rq *request) finish() {
|
|||||||
line := &rq.line
|
line := &rq.line
|
||||||
line.Status = rq.out.status
|
line.Status = rq.out.status
|
||||||
line.ResponseBytes = rq.out.bytes
|
line.ResponseBytes = rq.out.bytes
|
||||||
header := rq.out.Header()
|
|
||||||
line.ResponseContentType = header.Get("Content-Type")
|
|
||||||
line.CacheControl = header.Get("Cache-Control")
|
|
||||||
line.Location = header.Get("Location")
|
|
||||||
|
|
||||||
if rq.body != nil {
|
if rq.body != nil {
|
||||||
line.RequestBytes = rq.body.bytes.Load()
|
line.RequestBytes = rq.body.bytes.Load()
|
||||||
@@ -434,18 +374,12 @@ func (rq *request) finish() {
|
|||||||
now := time.Now()
|
now := time.Now()
|
||||||
duration := now.Sub(rq.start)
|
duration := now.Sub(rq.start)
|
||||||
line.DurationTotal = requestlog.Milliseconds(duration)
|
line.DurationTotal = requestlog.Milliseconds(duration)
|
||||||
line.DurationChecks = timing(rq.start, rq.checked)
|
|
||||||
|
|
||||||
var upstreamDuration time.Duration
|
var upstreamDuration time.Duration
|
||||||
|
|
||||||
if !rq.upstreamStart.IsZero() {
|
if !rq.upstreamStart.IsZero() {
|
||||||
upstreamDuration = now.Sub(rq.upstreamStart)
|
upstreamDuration = now.Sub(rq.upstreamStart)
|
||||||
line.DurationUpstreamTotal = new(requestlog.Milliseconds(upstreamDuration))
|
line.DurationUpstreamTotal = requestlog.Milliseconds(upstreamDuration)
|
||||||
|
|
||||||
rq.mu.Lock()
|
|
||||||
line.DurationUpstreamConnect = timing(rq.upstreamStart, rq.connected)
|
|
||||||
line.DurationUpstreamFirstByte = timing(rq.upstreamStart, rq.answerStarted)
|
|
||||||
rq.mu.Unlock()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Counted before the log line is written, so that the metrics count
|
// Counted before the log line is written, so that the metrics count
|
||||||
@@ -458,17 +392,6 @@ func (rq *request) finish() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// timing is the time from start to end in milliseconds, for one of the
|
|
||||||
// log line's timings, or nil when end is zero: what it times never
|
|
||||||
// happened.
|
|
||||||
func timing(start, end time.Time) *float64 {
|
|
||||||
if end.IsZero() {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return new(requestlog.Milliseconds(end.Sub(start)))
|
|
||||||
}
|
|
||||||
|
|
||||||
// addToHistory adds the request, which has ended, to its client's
|
// addToHistory adds the request, which has ended, to its client's
|
||||||
// history.
|
// history.
|
||||||
func (rq *request) addToHistory() {
|
func (rq *request) addToHistory() {
|
||||||
@@ -578,24 +501,6 @@ func (rq *request) bodyReceived() {
|
|||||||
stopTimer(rq.clientRequestTimer)
|
stopTimer(rq.clientRequestTimer)
|
||||||
}
|
}
|
||||||
|
|
||||||
// gotConn is called once there is a connection to the app, a new one or
|
|
||||||
// one kept open from an earlier request.
|
|
||||||
func (rq *request) gotConn(httptrace.GotConnInfo) {
|
|
||||||
rq.mu.Lock()
|
|
||||||
defer rq.mu.Unlock()
|
|
||||||
|
|
||||||
rq.connected = time.Now()
|
|
||||||
}
|
|
||||||
|
|
||||||
// gotFirstResponseByte is called once the first byte of the app's answer
|
|
||||||
// has arrived.
|
|
||||||
func (rq *request) gotFirstResponseByte() {
|
|
||||||
rq.mu.Lock()
|
|
||||||
defer rq.mu.Unlock()
|
|
||||||
|
|
||||||
rq.answerStarted = time.Now()
|
|
||||||
}
|
|
||||||
|
|
||||||
// wroteRequest is called once the app has been sent the whole request:
|
// wroteRequest is called once the app has been sent the whole request:
|
||||||
// the request timeouts end and SWWAF_UPSTREAM_RESPONSE_TIMEOUT starts.
|
// the request timeouts end and SWWAF_UPSTREAM_RESPONSE_TIMEOUT starts.
|
||||||
func (rq *request) wroteRequest(info httptrace.WroteRequestInfo) {
|
func (rq *request) wroteRequest(info httptrace.WroteRequestInfo) {
|
||||||
|
|||||||
@@ -1,368 +0,0 @@
|
|||||||
package proxy_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"io"
|
|
||||||
"maps"
|
|
||||||
"math"
|
|
||||||
"net/http"
|
|
||||||
"reflect"
|
|
||||||
"slices"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// requestIDHeader carries the request's id.
|
|
||||||
requestIDHeader = "X-Request-ID"
|
|
||||||
// instance is the SWWAF_INSTANCE_NAME a test sets.
|
|
||||||
instance = "fsn1app1/gitea"
|
|
||||||
// ipv6Client is a client on IPv6, and ipv6Group the netblock the rate
|
|
||||||
// limits count it as.
|
|
||||||
ipv6Client = "2001:db8::7"
|
|
||||||
ipv6Group = "2001:db8::/64"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestLogLineHasEachFieldWhereItApplies(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
received := make(chan string, 2) // the request ids the app received
|
|
||||||
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
received <- r.Header.Get(requestIDHeader)
|
|
||||||
|
|
||||||
_, _ = io.Copy(io.Discard, r.Body)
|
|
||||||
|
|
||||||
if r.URL.Path != "/full" {
|
|
||||||
w.WriteHeader(http.StatusNoContent)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
w.Header().Set("Content-Type", "text/html")
|
|
||||||
w.Header().Set("Cache-Control", "no-store")
|
|
||||||
w.Header().Set("Location", "/elsewhere")
|
|
||||||
w.WriteHeader(http.StatusFound)
|
|
||||||
_, _ = io.WriteString(w, "moved")
|
|
||||||
})
|
|
||||||
addr, out := startProxy(t, app.URL, map[string]string{
|
|
||||||
trustedProxies: trustLocalhost,
|
|
||||||
rateLimitExemptNets: localhost,
|
|
||||||
instanceName: instance,
|
|
||||||
logRequestHeaders: "Accept,x-custom,Authorization,cookie,SET-COOKIE",
|
|
||||||
})
|
|
||||||
|
|
||||||
// This request comes from ipv6Client through a trusted proxy, with a
|
|
||||||
// body and each header the log line looks at, and is answered with a
|
|
||||||
// redirect.
|
|
||||||
conn := dial(t, addr)
|
|
||||||
send(t, conn, "POST /full HTTP/1.1\r\nHost: "+appHost+"\r\n"+
|
|
||||||
forwardedFor+": 198.51.100.7, "+ipv6Client+"\r\n"+
|
|
||||||
forwardedProto+": "+secure+"\r\n"+requestIDHeader+": from-traefik\r\n"+
|
|
||||||
"Content-Type: application/x-www-form-urlencoded\r\nContent-Length: 3\r\n"+
|
|
||||||
"Accept: text/html\r\nX-Custom: one\r\nX-Custom: two\r\n"+
|
|
||||||
"Authorization: Bearer secret-token\r\nCookie: session=secret-cookie\r\n"+
|
|
||||||
"Set-Cookie: secret-set-cookie\r\n\r\na=b")
|
|
||||||
wantStatus(t, readResponse(t, conn), http.StatusFound)
|
|
||||||
|
|
||||||
// A request's log line can come after its answer: each is waited for
|
|
||||||
// before the next request, so that the lines are in order.
|
|
||||||
full := out.requestLines(t, 1)[0]
|
|
||||||
|
|
||||||
// This one comes from 127.0.0.1, which the rate limits do not count,
|
|
||||||
// with a body of 4 bytes whose length it does not announce, so that its
|
|
||||||
// request_bytes is not its content_length, and no header the log line
|
|
||||||
// looks at, and is answered with 204 and no header.
|
|
||||||
conn = dial(t, addr)
|
|
||||||
send(t, conn, "POST /bare HTTP/1.1\r\nHost: "+appHost+"\r\n"+
|
|
||||||
"Transfer-Encoding: chunked\r\n\r\n4\r\nbody\r\n0\r\n\r\n")
|
|
||||||
wantStatus(t, readResponse(t, conn), http.StatusNoContent)
|
|
||||||
|
|
||||||
bare := out.requestLines(t, 2)[1]
|
|
||||||
|
|
||||||
wantFullLine(t, full)
|
|
||||||
wantBareLine(t, bare)
|
|
||||||
|
|
||||||
for _, line := range []logLine{full, bare} {
|
|
||||||
got := <-received
|
|
||||||
if got != line.RequestID {
|
|
||||||
t.Errorf("the app received request id %q, the log line has %q",
|
|
||||||
got, line.RequestID)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.Contains(out.text(), "secret") {
|
|
||||||
t.Errorf("a value of Authorization, Cookie or Set-Cookie is logged:\n%s",
|
|
||||||
out.text())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantFullLine checks the log line of the request with every header the
|
|
||||||
// line looks at. Its timings are checked by TestTimingsAreInOrder.
|
|
||||||
func wantFullLine(t *testing.T, line logLine) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
headers := map[string]string{"accept": "text/html", "x-custom": "one, two"}
|
|
||||||
|
|
||||||
want := withTimings(line, requestlog.Line{
|
|
||||||
Type: requestType, Time: line.Time, Instance: instance,
|
|
||||||
ClientIP: ipv6Client, Method: http.MethodPost, Scheme: secure,
|
|
||||||
Host: appHost, Path: "/full", Protocol: protocol,
|
|
||||||
Status: http.StatusFound, RequestBytes: 3, ResponseBytes: 5,
|
|
||||||
RequestID: "from-traefik", PeerIP: localhost,
|
|
||||||
ForwardedFor: "198.51.100.7, " + ipv6Client, ClientGroup: ipv6Group,
|
|
||||||
ContentType: "application/x-www-form-urlencoded", ContentLength: 3,
|
|
||||||
RequestHeaders: headers, HasAuthorization: true, HasCookie: true,
|
|
||||||
ResponseContentType: "text/html", UpstreamStatus: http.StatusFound,
|
|
||||||
CacheControl: "no-store", Location: "/elsewhere",
|
|
||||||
Action: requestlog.ActionForward,
|
|
||||||
Counts: ratelimit.Counts{Minute: 1, Hour: 1, Day: 1},
|
|
||||||
})
|
|
||||||
if !reflect.DeepEqual(line.Line, want) {
|
|
||||||
t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantBareLine checks the log line of the request with none of them, and
|
|
||||||
// that the fields that do not apply to it are left out.
|
|
||||||
func wantBareLine(t *testing.T, line logLine) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
want := withTimings(line, requestlog.Line{
|
|
||||||
Type: requestType, Time: line.Time, Instance: instance,
|
|
||||||
ClientIP: localhost, Method: http.MethodPost, Scheme: plain,
|
|
||||||
Host: appHost, Path: "/bare", Protocol: protocol,
|
|
||||||
Status: http.StatusNoContent, RequestBytes: 4, RequestID: line.RequestID,
|
|
||||||
PeerIP: localhost, ClientGroup: localhost + "/32",
|
|
||||||
UpstreamStatus: http.StatusNoContent, Action: requestlog.ActionForward,
|
|
||||||
})
|
|
||||||
if !reflect.DeepEqual(line.Line, want) || line.RequestID == "" {
|
|
||||||
t.Errorf("log line\n%+v\nwant\n%+v, with a request id", line.Line, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, name := range []string{
|
|
||||||
"forwarded_for", "content_type", "content_length", "request_headers",
|
|
||||||
"has_authorization", "has_cookie", "websocket", "response_content_type",
|
|
||||||
"cache_control", "location", "counts",
|
|
||||||
} {
|
|
||||||
_, present := line.fields[name]
|
|
||||||
if present {
|
|
||||||
t.Errorf("log line has %s, which does not apply", name)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// withTimings returns want with the timings of line.
|
|
||||||
func withTimings(line logLine, want requestlog.Line) requestlog.Line {
|
|
||||||
want.DurationTotal = line.DurationTotal
|
|
||||||
want.DurationChecks = line.DurationChecks
|
|
||||||
want.DurationUpstreamConnect = line.DurationUpstreamConnect
|
|
||||||
want.DurationUpstreamFirstByte = line.DurationUpstreamFirstByte
|
|
||||||
want.DurationUpstreamTotal = line.DurationUpstreamTotal
|
|
||||||
|
|
||||||
return want
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestHasAuthorizationAndHasCookieEachComeFromTheirOwnHeader(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const hasAuthorization, hasCookie = "has_authorization", "has_cookie"
|
|
||||||
|
|
||||||
for _, tc := range []struct{ header, field, other string }{
|
|
||||||
{"Authorization", hasAuthorization, hasCookie},
|
|
||||||
{"Cookie", hasCookie, hasAuthorization},
|
|
||||||
} {
|
|
||||||
t.Run("only "+tc.header, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
|
||||||
addr, out := startProxy(t, app.URL, nil)
|
|
||||||
|
|
||||||
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
|
|
||||||
req.Header.Set(tc.header, "secret")
|
|
||||||
wantStatus(t, do(t, req), http.StatusOK)
|
|
||||||
|
|
||||||
line := out.requestLine(t)
|
|
||||||
|
|
||||||
_, otherPresent := line.fields[tc.other]
|
|
||||||
if line.fields[tc.field] != true || otherPresent {
|
|
||||||
t.Errorf("log line has %s %v and %s %v, want true and none",
|
|
||||||
tc.field, line.fields[tc.field], tc.other, line.fields[tc.other])
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRequestIDAndSchemeComeOnlyFromATrustedProxy(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const sentID = "from-traefik"
|
|
||||||
|
|
||||||
sent := http.Header{requestIDHeader: {sentID}, forwardedProto: {secure}}
|
|
||||||
trusted := map[string]string{trustedProxies: trustLocalhost}
|
|
||||||
|
|
||||||
for _, tc := range []struct {
|
|
||||||
name string
|
|
||||||
env map[string]string
|
|
||||||
header http.Header
|
|
||||||
// wantID is the request id logged, "" for a new one.
|
|
||||||
wantID, wantScheme string
|
|
||||||
}{
|
|
||||||
{"a trusted proxy's are kept", trusted, sent, sentID, secure},
|
|
||||||
{"without them, the id is new and the scheme http", trusted, nil, "", plain},
|
|
||||||
{"another peer's are replaced", nil, sent, "", plain},
|
|
||||||
} {
|
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
received := make(chan string, 2)
|
|
||||||
app := startApp(t, func(_ http.ResponseWriter, r *http.Request) {
|
|
||||||
received <- r.Header.Get(requestIDHeader)
|
|
||||||
})
|
|
||||||
addr, out := startProxy(t, app.URL, tc.env)
|
|
||||||
|
|
||||||
// Two requests, so that two new ids can be told apart.
|
|
||||||
ids := make([]string, 0, 2)
|
|
||||||
|
|
||||||
for i := range 2 {
|
|
||||||
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
|
|
||||||
maps.Copy(req.Header, tc.header)
|
|
||||||
wantStatus(t, do(t, req), http.StatusOK)
|
|
||||||
|
|
||||||
line := out.requestLines(t, i+1)[i]
|
|
||||||
ids = append(ids, line.RequestID)
|
|
||||||
|
|
||||||
got := <-received
|
|
||||||
if line.RequestID != got || line.Scheme != tc.wantScheme {
|
|
||||||
t.Errorf("log line has request_id %q and scheme %q, and the "+
|
|
||||||
"app received id %q; want the same id and scheme %q",
|
|
||||||
line.RequestID, line.Scheme, got, tc.wantScheme)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
switch {
|
|
||||||
case tc.wantID != "" && (ids[0] != tc.wantID || ids[1] != tc.wantID):
|
|
||||||
t.Errorf("request ids %q, want %q", ids, tc.wantID)
|
|
||||||
case tc.wantID == "" && (slices.Contains(ids, sentID) ||
|
|
||||||
slices.Contains(ids, "") || ids[0] == ids[1]):
|
|
||||||
t.Errorf("request ids %q, want two new ones", ids)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTimingsAreInOrder(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const denied = "192.0.2.50" // in SWWAF_DENY_NETS
|
|
||||||
|
|
||||||
app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
// The pauses set the times apart; a hold-up of the test only
|
|
||||||
// lengthens them.
|
|
||||||
time.Sleep(time.Millisecond)
|
|
||||||
w.WriteHeader(http.StatusOK)
|
|
||||||
_ = http.NewResponseController(w).Flush()
|
|
||||||
|
|
||||||
time.Sleep(time.Millisecond)
|
|
||||||
|
|
||||||
_, _ = io.WriteString(w, "done")
|
|
||||||
})
|
|
||||||
addr, out := startProxy(t, app.URL, map[string]string{
|
|
||||||
trustedProxies: trustLocalhost,
|
|
||||||
denyNets: denied,
|
|
||||||
})
|
|
||||||
|
|
||||||
// Each log line is waited for before the next request, so that the
|
|
||||||
// lines are in order.
|
|
||||||
wantStatus(t, get(t, addr, "/"), http.StatusOK)
|
|
||||||
forwarded := out.requestLines(t, 1)[0]
|
|
||||||
|
|
||||||
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
|
|
||||||
req.Header.Set(forwardedFor, denied)
|
|
||||||
wantStatus(t, do(t, req), http.StatusForbidden)
|
|
||||||
refused := out.requestLines(t, 2)[1]
|
|
||||||
|
|
||||||
wantStatus(t, get(t, addr, proxy.HealthPath), http.StatusOK)
|
|
||||||
health := out.requestLines(t, 3)[2]
|
|
||||||
|
|
||||||
// A request passed to the app has every timing; one refused, none of
|
|
||||||
// the app's; the health check, which runs no check, only the total.
|
|
||||||
wantTimings(t, forwarded, "duration_total", "duration_checks",
|
|
||||||
"duration_upstream_connect", "duration_upstream_first_byte",
|
|
||||||
"duration_upstream_total")
|
|
||||||
wantTimings(t, refused, "duration_total", "duration_checks")
|
|
||||||
wantTimings(t, health, "duration_total")
|
|
||||||
|
|
||||||
if t.Failed() {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// In whole microseconds, as they are logged, so that the sum below is
|
|
||||||
// exact.
|
|
||||||
total := microseconds(forwarded.DurationTotal)
|
|
||||||
checks := microseconds(*forwarded.DurationChecks)
|
|
||||||
connect := microseconds(*forwarded.DurationUpstreamConnect)
|
|
||||||
firstByte := microseconds(*forwarded.DurationUpstreamFirstByte)
|
|
||||||
upstream := microseconds(*forwarded.DurationUpstreamTotal)
|
|
||||||
|
|
||||||
// The checks end before the request is handed to the app, and the
|
|
||||||
// connection comes before the answer, which the app ends after a
|
|
||||||
// pause.
|
|
||||||
if checks+upstream > total || connect >= firstByte || firstByte >= upstream {
|
|
||||||
t.Errorf("timings in microseconds: total %d, checks %d, connect %d, "+
|
|
||||||
"first byte %d, upstream total %d", total, checks, connect, firstByte,
|
|
||||||
upstream)
|
|
||||||
}
|
|
||||||
|
|
||||||
if *refused.DurationChecks > refused.DurationTotal {
|
|
||||||
t.Errorf("refused request's checks took %v of %v milliseconds",
|
|
||||||
*refused.DurationChecks, refused.DurationTotal)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantTimings checks that the timings named are the only ones line has.
|
|
||||||
func wantTimings(t *testing.T, line logLine, want ...string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var got []string
|
|
||||||
|
|
||||||
for name := range line.fields {
|
|
||||||
if strings.HasPrefix(name, "duration_") {
|
|
||||||
got = append(got, name)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
slices.Sort(got)
|
|
||||||
slices.Sort(want)
|
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("log line of %s has timings %v, want %v", line.Path, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// microseconds is a timing in whole microseconds.
|
|
||||||
func microseconds(milliseconds float64) int64 {
|
|
||||||
return int64(math.Round(milliseconds * 1000))
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLogsAnUpgradedConnection(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
app := startApp(t, echoAfterUpgrade)
|
|
||||||
addr, out := startProxy(t, app.URL, nil)
|
|
||||||
|
|
||||||
conn := dial(t, addr)
|
|
||||||
send(t, conn, "GET /socket HTTP/1.1\r\nHost: app\r\n"+
|
|
||||||
"Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
|
|
||||||
wantStatus(t, readResponse(t, conn), http.StatusSwitchingProtocols)
|
|
||||||
|
|
||||||
_ = conn.Close()
|
|
||||||
|
|
||||||
line := out.requestLine(t)
|
|
||||||
if line.fields["websocket"] != true {
|
|
||||||
t.Errorf("log line has websocket %v, want true", line.fields["websocket"])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,233 +0,0 @@
|
|||||||
package proxy_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/http"
|
|
||||||
"net/netip"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"slices"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
|
||||||
)
|
|
||||||
|
|
||||||
// testRules are the rules most tests here load: a block rule for
|
|
||||||
// /blocked and a ban rule for /.env.
|
|
||||||
const testRules = `
|
|
||||||
blocked path block ^/blocked$
|
|
||||||
probe path ban ^/\.env$
|
|
||||||
`
|
|
||||||
|
|
||||||
func TestEachRuleAction(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s, clk, server := startWithClock(t, "", map[string]string{
|
|
||||||
rulesDir: writeRules(t, "noted path log ^/\n"+testRules),
|
|
||||||
banResponse: "429",
|
|
||||||
})
|
|
||||||
start := clk.Now()
|
|
||||||
|
|
||||||
// A log rule notes its match, and lets the request through.
|
|
||||||
line := s.get(client, http.StatusOK, requestlog.ActionForward)
|
|
||||||
wantRuleIDs(t, line, "noted")
|
|
||||||
|
|
||||||
// A block rule refuses with 403, whatever SWWAF_BAN_RESPONSE is, and
|
|
||||||
// bans no one.
|
|
||||||
line = s.request(client, "/blocked", http.StatusForbidden,
|
|
||||||
requestlog.ActionRuleBlocked)
|
|
||||||
wantRuleIDs(t, line, "noted", "blocked")
|
|
||||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
|
||||||
|
|
||||||
// A ban rule refuses with SWWAF_BAN_RESPONSE, and bans the client for
|
|
||||||
// seven days, the default.
|
|
||||||
line = s.request(client, "/.env", http.StatusTooManyRequests, requestlog.ActionBanned)
|
|
||||||
wantRuleIDs(t, line, "noted", "probe")
|
|
||||||
|
|
||||||
if line.BanExpires != requestlog.FormatTime(start.Add(7*24*time.Hour)) {
|
|
||||||
t.Errorf("log line has ban_expires %q, want seven days on", line.BanExpires)
|
|
||||||
}
|
|
||||||
|
|
||||||
netblock := netip.MustParsePrefix(client + "/32")
|
|
||||||
want := bans.Ban{
|
|
||||||
Netblock: netblock,
|
|
||||||
Start: start,
|
|
||||||
Expires: start.Add(7 * 24 * time.Hour),
|
|
||||||
Cause: bans.CauseAttack,
|
|
||||||
Reason: "matched the rule probe",
|
|
||||||
Notes: bans.Notes{
|
|
||||||
RuleID: "probe",
|
|
||||||
Target: "path",
|
|
||||||
Request: bans.Request{
|
|
||||||
Time: start,
|
|
||||||
Method: http.MethodGet,
|
|
||||||
Host: appHost,
|
|
||||||
Path: "/.env",
|
|
||||||
Status: http.StatusTooManyRequests,
|
|
||||||
UserAgent: userAgent,
|
|
||||||
},
|
|
||||||
// The four requests up to and including the probe.
|
|
||||||
Requests: 4,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
got := server.Ledger.Bans(netblock)
|
|
||||||
if len(got) != 1 || got[0] != want {
|
|
||||||
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
// The next request is refused under the ban, without being checked
|
|
||||||
// against the rules, and makes the ban permanent.
|
|
||||||
clk.advance(time.Hour)
|
|
||||||
|
|
||||||
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
|
|
||||||
wantRuleIDs(t, line)
|
|
||||||
|
|
||||||
if line.BanExpires != permanent {
|
|
||||||
t.Errorf("log line has ban_expires %q, want permanent", line.BanExpires)
|
|
||||||
}
|
|
||||||
|
|
||||||
clk.advance(365 * 24 * time.Hour)
|
|
||||||
s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestNextClearSignOfAttackAfterABanBansPermanently(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s, clk, _ := startWithClock(t, "", map[string]string{
|
|
||||||
rulesDir: writeRules(t, testRules),
|
|
||||||
attackBanDuration: "1h",
|
|
||||||
})
|
|
||||||
|
|
||||||
// The first probe bans for SWWAF_ATTACK_BAN_DURATION.
|
|
||||||
line := s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
|
||||||
if line.BanExpires != requestlog.FormatTime(clk.Now().Add(time.Hour)) {
|
|
||||||
t.Errorf("log line has ban_expires %q, want an hour on", line.BanExpires)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Once that ban has run out without a request, the client is served,
|
|
||||||
// and its next probe bans it for good.
|
|
||||||
clk.advance(time.Hour)
|
|
||||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
|
||||||
|
|
||||||
line = s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
|
||||||
if line.BanExpires != permanent {
|
|
||||||
t.Errorf("log line has ban_expires %q, want permanent", line.BanExpires)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRulesComeAfterTheOtherChecks(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
|
|
||||||
exempt = "192.0.2.50" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
|
|
||||||
)
|
|
||||||
|
|
||||||
s, _, server := startWithClock(t, "", map[string]string{
|
|
||||||
rulesDir: writeRules(t, testRules),
|
|
||||||
allowNets: allowed,
|
|
||||||
rateLimitExemptNets: exempt,
|
|
||||||
rateLimitPerMinute: "1",
|
|
||||||
})
|
|
||||||
|
|
||||||
// A client in SWWAF_ALLOW_NETS is not checked.
|
|
||||||
line := s.request(allowed, "/.env", http.StatusOK, requestlog.ActionForward)
|
|
||||||
wantRuleIDs(t, line)
|
|
||||||
|
|
||||||
// A probe over the rate limit breaks the limit before any rule sees
|
|
||||||
// it.
|
|
||||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
|
||||||
line = s.request(client, "/.env", http.StatusForbidden, requestlog.ActionRateLimited)
|
|
||||||
wantRuleIDs(t, line)
|
|
||||||
|
|
||||||
limitBan := server.Ledger.Bans(netip.MustParsePrefix(client + "/32"))
|
|
||||||
if len(limitBan) != 1 || limitBan[0].Cause != bans.CauseLimit {
|
|
||||||
t.Errorf("bans %+v, want one for a broken limit", limitBan)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A client the rate limits do not apply to is still checked.
|
|
||||||
s.get(exempt, http.StatusOK, requestlog.ActionForward)
|
|
||||||
s.request(exempt, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestObserveModeLogsWhatTheRulesWouldDo(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s, _, server := startWithClock(t, "", map[string]string{
|
|
||||||
rulesDir: writeRules(t, testRules),
|
|
||||||
mode: observe,
|
|
||||||
})
|
|
||||||
|
|
||||||
line := s.request(client, "/blocked", http.StatusOK, requestlog.ActionForward)
|
|
||||||
wantWouldAction(t, line, requestlog.ActionRuleBlocked)
|
|
||||||
wantRuleIDs(t, line, "blocked")
|
|
||||||
|
|
||||||
line = s.request(client, "/.env", http.StatusOK, requestlog.ActionForward)
|
|
||||||
wantWouldAction(t, line, requestlog.ActionBanned)
|
|
||||||
wantRuleIDs(t, line, "probe")
|
|
||||||
|
|
||||||
if line.BanExpires != "" {
|
|
||||||
t.Errorf("log line has ban_expires %q, want none", line.BanExpires)
|
|
||||||
}
|
|
||||||
|
|
||||||
// No ban was made.
|
|
||||||
line = s.get(client, http.StatusOK, requestlog.ActionForward)
|
|
||||||
wantWouldAction(t, line, "")
|
|
||||||
|
|
||||||
if got := server.Ledger.Snapshot(); len(got) != 0 {
|
|
||||||
t.Errorf("bans %+v, want none", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestMetricsCountRuleMatchesAndBansForAnAttack(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const scraper = "192.0.2.200"
|
|
||||||
|
|
||||||
s, _, _ := startWithClock(t, "", map[string]string{
|
|
||||||
rulesDir: writeRules(t, testRules),
|
|
||||||
metricsToken: token,
|
|
||||||
})
|
|
||||||
|
|
||||||
s.request(client, "/blocked", http.StatusForbidden, requestlog.ActionRuleBlocked)
|
|
||||||
s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
|
||||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
|
||||||
|
|
||||||
metrics := s.scrape(scraper)
|
|
||||||
wantMetric(t, metrics,
|
|
||||||
`smallwebwaf_rule_matches_total{action="block",rule_id="blocked"}`, 1)
|
|
||||||
wantMetric(t, metrics,
|
|
||||||
`smallwebwaf_rule_matches_total{action="ban",rule_id="probe"}`, 1)
|
|
||||||
wantMetric(t, metrics, "smallwebwaf_rules_loaded", 2)
|
|
||||||
wantMetric(t, metrics,
|
|
||||||
`smallwebwaf_requests_total{action="rule_blocked",status_class="4xx"}`, 1)
|
|
||||||
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="attack"}`, 1)
|
|
||||||
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit"}`, 0)
|
|
||||||
wantMetric(t, metrics, "smallwebwaf_permanent_bans", 1)
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeRules writes content as a rule file into a new directory, and
|
|
||||||
// returns the directory.
|
|
||||||
func writeRules(t *testing.T, content string) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
|
|
||||||
err := os.WriteFile(filepath.Join(dir, "test.rules"), []byte(content), 0o600)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write the rule file: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return dir
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantRuleIDs checks the request log line's rule_ids.
|
|
||||||
func wantRuleIDs(t *testing.T, line logLine, want ...string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
if !slices.Equal(line.RuleIDs, want) {
|
|
||||||
t.Errorf("log line has rule_ids %v, want %v", line.RuleIDs, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
package proxy
|
|
||||||
|
|
||||||
import (
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
|
||||||
)
|
|
||||||
|
|
||||||
// checkRules checks the request against the rules of the rule files at
|
|
||||||
// now, notes the ids of those it matches in the log line, and returns the
|
|
||||||
// action of the rule that refuses it, ActionRuleBlocked for a block rule
|
|
||||||
// and ActionBanned for a ban rule, or "" when none does. In enforce mode
|
|
||||||
// a ban rule bans the client's netblock for a clear sign of attack.
|
|
||||||
func (rq *request) checkRules(now time.Time) string {
|
|
||||||
matched := rq.h.rules.Match(rq.in)
|
|
||||||
|
|
||||||
for _, rule := range matched {
|
|
||||||
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
|
|
||||||
rq.h.metrics.RuleMatched(rule.ID, rule.Action)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(matched) == 0 {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
// Only the last rule matched can refuse the request.
|
|
||||||
switch last := matched[len(matched)-1]; last.Action {
|
|
||||||
case rules.ActionBlock:
|
|
||||||
return requestlog.ActionRuleBlocked
|
|
||||||
case rules.ActionBan:
|
|
||||||
if !rq.h.config.Observe {
|
|
||||||
rq.banForAttack(now, last)
|
|
||||||
}
|
|
||||||
|
|
||||||
return requestlog.ActionBanned
|
|
||||||
default:
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -149,39 +149,26 @@ type Hit struct {
|
|||||||
Requests float64
|
Requests float64
|
||||||
}
|
}
|
||||||
|
|
||||||
// Counts are a client's requests in the minute, the hour and the day that
|
|
||||||
// end at a request, that request included.
|
|
||||||
type Counts struct {
|
|
||||||
Minute float64 `json:"minute"`
|
|
||||||
Hour float64 `json:"hour"`
|
|
||||||
Day float64 `json:"day"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// Count counts a request from client at now, in every window, whether or
|
// Count counts a request from client at now, in every window, whether or
|
||||||
// not it is refused, and returns the client's requests in each window. It
|
// not it is refused. It reports whether the request takes the client over
|
||||||
// reports whether the request takes the client over a limit, and the
|
// a limit, and the window whose limit it goes over, the shortest if it is
|
||||||
// window whose limit it goes over, the shortest if it is over several.
|
// over several.
|
||||||
func (l *Limiter) Count(client netip.Prefix, now time.Time) (Counts, Hit, bool) {
|
func (l *Limiter) Count(client netip.Prefix, now time.Time) (Hit, bool) {
|
||||||
l.mu.Lock()
|
l.mu.Lock()
|
||||||
defer l.mu.Unlock()
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
var (
|
var hit Hit
|
||||||
requests [3]float64
|
|
||||||
hit Hit
|
|
||||||
)
|
|
||||||
|
|
||||||
for i, b := range l.get(client).buckets() {
|
for i, b := range l.get(client).buckets() {
|
||||||
w := l.windows[i]
|
w := l.windows[i]
|
||||||
|
|
||||||
requests[i] = b.add(now, w.length)
|
requests := b.add(now, w.length)
|
||||||
if hit.Window == "" && w.limit > 0 && requests[i] > float64(w.limit) {
|
if hit.Window == "" && w.limit > 0 && requests > float64(w.limit) {
|
||||||
hit = Hit{Window: w.name, Limit: w.limit, Requests: requests[i]}
|
hit = Hit{Window: w.name, Limit: w.limit, Requests: requests}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
counts := Counts{Minute: requests[0], Hour: requests[1], Day: requests[2]}
|
return hit, hit.Window != ""
|
||||||
|
|
||||||
return counts, hit, hit.Window != ""
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Reset sets client's counts in every window back to zero. Its history
|
// Reset sets client's counts in every window back to zero. Its history
|
||||||
|
|||||||
@@ -62,14 +62,14 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
|
|||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
for range limit {
|
for range limit {
|
||||||
_, _, over := limiter.Count(client, start)
|
_, over := limiter.Count(client, start)
|
||||||
if over {
|
if over {
|
||||||
t.Fatal("a request within the limit is over it")
|
t.Fatal("a request within the limit is over it")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Over both limits; the minute's is named, with the four requests.
|
// Over both limits; the minute's is named, with the four requests.
|
||||||
_, hit, over := limiter.Count(client, start)
|
hit, over := limiter.Count(client, start)
|
||||||
|
|
||||||
want := ratelimit.Hit{Window: minute, Limit: limit, Requests: limit + 1}
|
want := ratelimit.Hit{Window: minute, Limit: limit, Requests: limit + 1}
|
||||||
if !over || hit != want {
|
if !over || hit != want {
|
||||||
@@ -78,29 +78,6 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{})
|
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
|
||||||
start := midnight()
|
|
||||||
|
|
||||||
for range 3 {
|
|
||||||
limiter.Count(client, start)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A quarter into the next hour, the minute has only this request. The
|
|
||||||
// hour still covers three quarters of the bucket before, with its three
|
|
||||||
// requests, which count 2.25, and this one: 3.25. The day covers all
|
|
||||||
// four.
|
|
||||||
counts, _, _ := limiter.Count(client, start.Add(time.Hour+time.Hour/4))
|
|
||||||
|
|
||||||
want := ratelimit.Counts{Minute: 1, Hour: 3.25, Day: 4}
|
|
||||||
if counts != want {
|
|
||||||
t.Errorf("counts %+v, want %+v", counts, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestResetSetsTheCountsBackToZero(t *testing.T) {
|
func TestResetSetsTheCountsBackToZero(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -261,7 +238,7 @@ func wantCount(
|
|||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
_, hit, _ := limiter.Count(client, now)
|
hit, _ := limiter.Count(client, now)
|
||||||
if hit.Window != want {
|
if hit.Window != want {
|
||||||
t.Errorf("request from %s at %s is over %q, want %q",
|
t.Errorf("request from %s at %s is over %q, want %q",
|
||||||
client, now.Format(time.RFC3339), hit.Window, want)
|
client, now.Format(time.RFC3339), hit.Window, want)
|
||||||
|
|||||||
@@ -1,310 +0,0 @@
|
|||||||
// Package remotelog sends the lines smallwebwaf writes on stdout to the
|
|
||||||
// remote log endpoint, SWWAF_LOG_REMOTE_URL, as the "Request log" section
|
|
||||||
// of SPEC.md describes: each line as the message of an RFC 5424 syslog
|
|
||||||
// record, over UDP, TCP or TLS. Lines wait in a bounded buffer, so a slow
|
|
||||||
// or unreachable endpoint never holds up a request or stdout.
|
|
||||||
package remotelog
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"context"
|
|
||||||
"crypto/tls"
|
|
||||||
"crypto/x509"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"log/slog"
|
|
||||||
"net"
|
|
||||||
"net/url"
|
|
||||||
"os"
|
|
||||||
"strconv"
|
|
||||||
"sync/atomic"
|
|
||||||
"syscall"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The forms of SWWAF_LOG_REMOTE_URL, by its scheme.
|
|
||||||
const (
|
|
||||||
SchemeUDP = "syslog+udp"
|
|
||||||
SchemeTCP = "syslog+tcp"
|
|
||||||
SchemeTLS = "syslog+tls"
|
|
||||||
)
|
|
||||||
|
|
||||||
// A record's priority is the number of its facility times the number of
|
|
||||||
// severities there are, plus the number of its severity. Every record's
|
|
||||||
// severity is informational.
|
|
||||||
const (
|
|
||||||
severities = 8
|
|
||||||
informational = 6
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// dialTimeout bounds connecting to the endpoint, the TLS handshake
|
|
||||||
// included.
|
|
||||||
dialTimeout = 10 * time.Second
|
|
||||||
// After a failed attempt to connect, or a connection on which a record
|
|
||||||
// fails, the next attempt to connect is made a second later, and
|
|
||||||
// retryDelayFactor times as long after each further failure in a row,
|
|
||||||
// up to a minute. A connection that fails after it has stayed up for
|
|
||||||
// resetRetryDelayAfter ends the row.
|
|
||||||
firstRetryDelay = time.Second
|
|
||||||
retryDelayFactor = 2
|
|
||||||
maxRetryDelay = time.Minute
|
|
||||||
resetRetryDelayAfter = time.Minute
|
|
||||||
)
|
|
||||||
|
|
||||||
// Params are what New needs.
|
|
||||||
type Params struct {
|
|
||||||
// URL is the endpoint (SWWAF_LOG_REMOTE_URL): SchemeUDP, SchemeTCP or
|
|
||||||
// SchemeTLS, a host and a port.
|
|
||||||
URL *url.URL
|
|
||||||
// RootCAs are the certificates a SchemeTLS endpoint's certificate
|
|
||||||
// must chain to (SWWAF_LOG_REMOTE_TLS_CA_FILE), nil for the host's.
|
|
||||||
RootCAs *x509.CertPool
|
|
||||||
// Buffer is the most lines held while they wait to be sent
|
|
||||||
// (SWWAF_LOG_REMOTE_BUFFER).
|
|
||||||
Buffer int
|
|
||||||
// Facility is the number of the records' syslog facility
|
|
||||||
// (SWWAF_LOG_REMOTE_FACILITY), and AppName their APP-NAME
|
|
||||||
// (SWWAF_LOG_REMOTE_APP_NAME).
|
|
||||||
Facility int
|
|
||||||
AppName string
|
|
||||||
}
|
|
||||||
|
|
||||||
// Sender sends lines to the endpoint. Write puts them in its buffer, and
|
|
||||||
// Run sends them from there.
|
|
||||||
type Sender struct {
|
|
||||||
url *url.URL
|
|
||||||
tlsConfig *tls.Config
|
|
||||||
// beforeTime and afterTime are the parts of every record's header
|
|
||||||
// before and after its time, as RFC 5424 lays the header out.
|
|
||||||
beforeTime string
|
|
||||||
afterTime string
|
|
||||||
// records is the buffer: each line's record, framed to be sent.
|
|
||||||
records chan []byte
|
|
||||||
sent atomic.Int64
|
|
||||||
dropped atomic.Int64
|
|
||||||
}
|
|
||||||
|
|
||||||
// New returns a Sender for the endpoint params.URL.
|
|
||||||
func New(params Params) *Sender {
|
|
||||||
hostname, err := os.Hostname()
|
|
||||||
if err != nil || hostname == "" {
|
|
||||||
hostname = "-" // RFC 5424's value for a field that has none
|
|
||||||
}
|
|
||||||
|
|
||||||
priority := params.Facility*severities + informational
|
|
||||||
|
|
||||||
return &Sender{
|
|
||||||
url: params.URL,
|
|
||||||
tlsConfig: &tls.Config{
|
|
||||||
RootCAs: params.RootCAs,
|
|
||||||
MinVersion: tls.VersionTLS12,
|
|
||||||
},
|
|
||||||
// The 1 is the version of the format. The process id, the message
|
|
||||||
// id and the structured data have no value.
|
|
||||||
beforeTime: "<" + strconv.Itoa(priority) + ">1 ",
|
|
||||||
afterTime: " " + hostname + " " + params.AppName + " - - - ",
|
|
||||||
records: make(chan []byte, params.Buffer),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Write puts each line in p in the buffer, as the message of a record of
|
|
||||||
// its own, and never waits: when the buffer is full, the oldest record in
|
|
||||||
// it is dropped to make room. It is safe for concurrent use.
|
|
||||||
func (s *Sender) Write(p []byte) (int, error) {
|
|
||||||
at := requestlog.FormatTime(time.Now())
|
|
||||||
|
|
||||||
for line := range bytes.Lines(p) {
|
|
||||||
line = bytes.TrimSuffix(line, []byte("\n"))
|
|
||||||
if len(line) > 0 {
|
|
||||||
s.put(s.record(at, line))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return len(p), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Sent is how many records have been sent.
|
|
||||||
func (s *Sender) Sent() int64 {
|
|
||||||
return s.sent.Load()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Dropped is how many records were dropped: the oldest in a full buffer,
|
|
||||||
// and those whose sending failed.
|
|
||||||
func (s *Sender) Dropped() int64 {
|
|
||||||
return s.dropped.Load()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Depth is how many records are in the buffer.
|
|
||||||
func (s *Sender) Depth() int {
|
|
||||||
return len(s.records)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Run connects to the endpoint and sends each record as it comes into the
|
|
||||||
// buffer, until ctx is done. Then it sends the records still in the buffer,
|
|
||||||
// on the connection open at that time or, if there is none, on a new one,
|
|
||||||
// until none is left or one fails, and returns. How long it may take over
|
|
||||||
// that is for the caller to bound.
|
|
||||||
//
|
|
||||||
// A connection on which a record fails is closed and the record dropped.
|
|
||||||
// That failure, like a failed attempt to connect, is logged to processLog
|
|
||||||
// and followed by the next attempt after firstRetryDelay, retryDelayFactor
|
|
||||||
// times as long after each further failure in a row up to maxRetryDelay,
|
|
||||||
// and firstRetryDelay again after a connection that stayed up for
|
|
||||||
// resetRetryDelayAfter. Meanwhile the records wait in the buffer.
|
|
||||||
func (s *Sender) Run(ctx context.Context, processLog *slog.Logger) {
|
|
||||||
conn := s.send(ctx, processLog)
|
|
||||||
if conn == nil && len(s.records) > 0 {
|
|
||||||
conn, _ = s.dial(context.WithoutCancel(ctx))
|
|
||||||
}
|
|
||||||
|
|
||||||
if conn == nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() {
|
|
||||||
_ = conn.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case record := <-s.records:
|
|
||||||
if s.write(conn, record) != nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
default:
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// record returns line as an RFC 5424 record made at the time at, framed
|
|
||||||
// for the endpoint: on its own over UDP, since each datagram holds one,
|
|
||||||
// and over TCP and TLS after its length in bytes and a space, the
|
|
||||||
// octet-counted framing of RFC 6587 and RFC 5425.
|
|
||||||
func (s *Sender) record(at string, line []byte) []byte {
|
|
||||||
record := make([]byte, 0, len(s.beforeTime)+len(at)+len(s.afterTime)+len(line))
|
|
||||||
record = append(record, s.beforeTime...)
|
|
||||||
record = append(record, at...)
|
|
||||||
record = append(record, s.afterTime...)
|
|
||||||
record = append(record, line...)
|
|
||||||
|
|
||||||
if s.url.Scheme == SchemeUDP {
|
|
||||||
return record
|
|
||||||
}
|
|
||||||
|
|
||||||
return append([]byte(strconv.Itoa(len(record))+" "), record...)
|
|
||||||
}
|
|
||||||
|
|
||||||
// put adds record to the buffer, first dropping the oldest record in it
|
|
||||||
// while it is full.
|
|
||||||
func (s *Sender) put(record []byte) {
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case s.records <- record:
|
|
||||||
return
|
|
||||||
default:
|
|
||||||
}
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-s.records:
|
|
||||||
s.dropped.Add(1)
|
|
||||||
default:
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// send connects to the endpoint and sends each record as it comes into
|
|
||||||
// the buffer, until ctx is done, and returns the connection then open, or
|
|
||||||
// nil.
|
|
||||||
func (s *Sender) send(ctx context.Context, processLog *slog.Logger) net.Conn {
|
|
||||||
delay := firstRetryDelay
|
|
||||||
|
|
||||||
for {
|
|
||||||
conn, err := s.dial(ctx)
|
|
||||||
if ctx.Err() != nil {
|
|
||||||
return conn
|
|
||||||
}
|
|
||||||
|
|
||||||
if err == nil {
|
|
||||||
connected := time.Now()
|
|
||||||
|
|
||||||
err = s.sendOn(ctx, conn)
|
|
||||||
if err == nil {
|
|
||||||
return conn
|
|
||||||
}
|
|
||||||
|
|
||||||
_ = conn.Close()
|
|
||||||
|
|
||||||
if time.Since(connected) >= resetRetryDelayAfter {
|
|
||||||
delay = firstRetryDelay
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
processLog.Warn("sending to SWWAF_LOG_REMOTE_URL failed",
|
|
||||||
"error", err.Error(), "connecting_again_in", delay.String())
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-time.After(delay):
|
|
||||||
case <-ctx.Done():
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
delay = min(retryDelayFactor*delay, maxRetryDelay)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// sendOn sends each record on conn as it comes into the buffer, until one
|
|
||||||
// fails, whose error it returns, or ctx is done.
|
|
||||||
func (s *Sender) sendOn(ctx context.Context, conn net.Conn) error {
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case record := <-s.records:
|
|
||||||
err := s.write(conn, record)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
case <-ctx.Done():
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// write sends record on conn, and counts it as sent or, if that fails,
|
|
||||||
// as dropped. A record too long for one UDP datagram is dropped without
|
|
||||||
// an error, since the connection has not failed: a long request must not
|
|
||||||
// hold up the lines after it.
|
|
||||||
func (s *Sender) write(conn net.Conn, record []byte) error {
|
|
||||||
_, err := conn.Write(record)
|
|
||||||
if err != nil {
|
|
||||||
s.dropped.Add(1)
|
|
||||||
|
|
||||||
if errors.Is(err, syscall.EMSGSIZE) {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return fmt.Errorf("send a record: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
s.sent.Add(1)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// dial connects to the endpoint.
|
|
||||||
func (s *Sender) dial(ctx context.Context) (net.Conn, error) {
|
|
||||||
dialer := &net.Dialer{Timeout: dialTimeout}
|
|
||||||
|
|
||||||
switch s.url.Scheme {
|
|
||||||
case SchemeUDP:
|
|
||||||
return dialer.DialContext(ctx, "udp", s.url.Host)
|
|
||||||
case SchemeTLS:
|
|
||||||
tlsDialer := &tls.Dialer{NetDialer: dialer, Config: s.tlsConfig}
|
|
||||||
|
|
||||||
return tlsDialer.DialContext(ctx, "tcp", s.url.Host)
|
|
||||||
default:
|
|
||||||
return dialer.DialContext(ctx, "tcp", s.url.Host)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,640 +0,0 @@
|
|||||||
package remotelog_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bufio"
|
|
||||||
"bytes"
|
|
||||||
"context"
|
|
||||||
"crypto/ecdsa"
|
|
||||||
"crypto/elliptic"
|
|
||||||
"crypto/rand"
|
|
||||||
"crypto/tls"
|
|
||||||
"crypto/x509"
|
|
||||||
"crypto/x509/pkix"
|
|
||||||
"encoding/json"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"log/slog"
|
|
||||||
"math/big"
|
|
||||||
"net"
|
|
||||||
"net/url"
|
|
||||||
"os"
|
|
||||||
"slices"
|
|
||||||
"strconv"
|
|
||||||
"strings"
|
|
||||||
"sync"
|
|
||||||
"sync/atomic"
|
|
||||||
"testing"
|
|
||||||
"testing/synctest"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The tests run in a synctest bubble, where the time package runs on a
|
|
||||||
// clock of the test's own, which starts at 2000-01-01T00:00:00Z: a wait
|
|
||||||
// lasts exactly as long as it should, however slowly the test process
|
|
||||||
// runs, and synctest.Wait returns once the sender has done all it can
|
|
||||||
// before time passes. The endpoint is a listener on the loopback address.
|
|
||||||
// A test reads from it only once the records are on their way, and checks
|
|
||||||
// the sender's counts first, since a goroutine of the bubble that waits on
|
|
||||||
// the network keeps that clock from moving on. For the same reason the
|
|
||||||
// endpoint that refuses connections, a tlsEndpoint, runs outside the
|
|
||||||
// bubble: a sender connecting over TLS waits on the endpoint's answer.
|
|
||||||
|
|
||||||
const (
|
|
||||||
// started is the time a record made as a test starts gives.
|
|
||||||
started = "2000-01-01T00:00:00.000Z"
|
|
||||||
appName = "fsn1app1/gitea"
|
|
||||||
// local0 is the number of the default facility, and local0Info the
|
|
||||||
// priority of its records.
|
|
||||||
local0 = 16
|
|
||||||
local0Info = "<134>"
|
|
||||||
// loopback is where the endpoints listen.
|
|
||||||
loopback = "127.0.0.1:0"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestRecordsOverUDPGoOnePerDatagram(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
endpoint, err := (&net.ListenConfig{}).ListenPacket(t.Context(), "udp", loopback)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("listen: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = endpoint.Close() })
|
|
||||||
|
|
||||||
sender, _, _ := run(t, params(remotelog.SchemeUDP, endpoint.LocalAddr()))
|
|
||||||
|
|
||||||
_, _ = sender.Write([]byte(`{"type":"request"}` + "\n" + `{"type":"process"}` + "\n"))
|
|
||||||
|
|
||||||
synctest.Wait()
|
|
||||||
wantCounts(t, sender, 2, 0, 0)
|
|
||||||
|
|
||||||
for _, line := range []string{`{"type":"request"}`, `{"type":"process"}`} {
|
|
||||||
datagram := make([]byte, 1024)
|
|
||||||
|
|
||||||
n, _, err := endpoint.ReadFrom(datagram)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("read: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := record(t, local0Info, appName, line)
|
|
||||||
if string(datagram[:n]) != want {
|
|
||||||
t.Errorf("datagram %q, want %q", datagram[:n], want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRecordsOverTCPAreOctetCountedWithTheirFacility(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
endpoint := listen(t)
|
|
||||||
endpointParams := params(remotelog.SchemeTCP, endpoint.Addr())
|
|
||||||
endpointParams.Facility = 19 // local3
|
|
||||||
endpointParams.AppName = "gitea"
|
|
||||||
sender, _, _ := run(t, endpointParams)
|
|
||||||
|
|
||||||
_, _ = sender.Write([]byte("first\nsecond\n"))
|
|
||||||
|
|
||||||
synctest.Wait()
|
|
||||||
wantCounts(t, sender, 2, 0, 0)
|
|
||||||
|
|
||||||
frames := bufio.NewReader(accept(t, endpoint))
|
|
||||||
wantFrame(t, frames, record(t, "<158>", "gitea", "first"))
|
|
||||||
wantFrame(t, frames, record(t, "<158>", "gitea", "second"))
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestStalledEndpointHoldsUpNoWriteAndOldestRecordsAreDropped(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
certificate, roots := testCertificate(t)
|
|
||||||
endpoint := listen(t)
|
|
||||||
endpointParams := params(remotelog.SchemeTLS, endpoint.Addr())
|
|
||||||
endpointParams.RootCAs = roots
|
|
||||||
endpointParams.Buffer = 3
|
|
||||||
sender, _, _ := run(t, endpointParams)
|
|
||||||
|
|
||||||
// The sender connects, and its TLS handshake waits for an answer
|
|
||||||
// the endpoint does not give yet.
|
|
||||||
conn := accept(t, endpoint)
|
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
|
||||||
|
|
||||||
out := io.MultiWriter(&stdout, sender)
|
|
||||||
for i := range 5 {
|
|
||||||
_, _ = fmt.Fprintf(out, "line %d\n", i+1)
|
|
||||||
}
|
|
||||||
|
|
||||||
if stdout.String() != "line 1\nline 2\nline 3\nline 4\nline 5\n" {
|
|
||||||
t.Errorf("stdout has %q", stdout.String())
|
|
||||||
}
|
|
||||||
|
|
||||||
wantCounts(t, sender, 0, 2, 3)
|
|
||||||
|
|
||||||
// Once the endpoint answers, the three newest records are sent.
|
|
||||||
server := tls.Server(conn, &tls.Config{
|
|
||||||
Certificates: []tls.Certificate{certificate},
|
|
||||||
MinVersion: tls.VersionTLS12,
|
|
||||||
})
|
|
||||||
|
|
||||||
err := server.HandshakeContext(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("handshake: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
synctest.Wait()
|
|
||||||
wantCounts(t, sender, 3, 2, 0)
|
|
||||||
|
|
||||||
frames := bufio.NewReader(server)
|
|
||||||
for _, line := range []string{"line 3", "line 4", "line 5"} {
|
|
||||||
wantFrame(t, frames, record(t, local0Info, appName, line))
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestReconnectsWithBackoffAfterTheEndpointGoesAway(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
certificate, roots := testCertificate(t)
|
|
||||||
endpoint := startTLSEndpoint(t, certificate)
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
endpointParams := params(remotelog.SchemeTLS, endpoint.addr)
|
|
||||||
endpointParams.RootCAs = roots
|
|
||||||
sender, logged, _ := run(t, endpointParams)
|
|
||||||
|
|
||||||
_, _ = sender.Write([]byte("one\n"))
|
|
||||||
|
|
||||||
synctest.Wait()
|
|
||||||
wantCounts(t, sender, 1, 0, 0)
|
|
||||||
|
|
||||||
conn := endpoint.next(t)
|
|
||||||
wantFrame(t, bufio.NewReader(conn), record(t, local0Info, appName, "one"))
|
|
||||||
|
|
||||||
// The endpoint goes away: it closes the connection, and refuses the
|
|
||||||
// next ones. The sender notices when a record fails, and tries to
|
|
||||||
// connect again a second later, then two seconds after that.
|
|
||||||
endpoint.refusing.Store(true)
|
|
||||||
|
|
||||||
_ = conn.Close()
|
|
||||||
|
|
||||||
writeUntilDropped(t, sender, 1)
|
|
||||||
sent := sender.Sent()
|
|
||||||
|
|
||||||
_, _ = sender.Write([]byte("two\n"))
|
|
||||||
|
|
||||||
time.Sleep(time.Second)
|
|
||||||
synctest.Wait()
|
|
||||||
|
|
||||||
endpoint.refusing.Store(false)
|
|
||||||
|
|
||||||
time.Sleep(2*time.Second - time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCounts(t, sender, sent, 1, 1)
|
|
||||||
|
|
||||||
// The endpoint is back, and the record waiting is sent.
|
|
||||||
time.Sleep(time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCounts(t, sender, sent+1, 1, 0)
|
|
||||||
|
|
||||||
conn = endpoint.next(t)
|
|
||||||
wantFrame(t, bufio.NewReader(conn), record(t, local0Info, appName, "two"))
|
|
||||||
wantRetries(t, logged, "1s", "2s")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAConnectionClosedAtOnceIsMadeAgainAfterAGrowingDelay(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
endpoint := listen(t)
|
|
||||||
sender, logged, _ := run(t, params(remotelog.SchemeTCP, endpoint.Addr()))
|
|
||||||
|
|
||||||
// The endpoint closes each connection as soon as it takes it. The
|
|
||||||
// sender notices when a record fails, and connects again a second
|
|
||||||
// later, then two seconds after that, then four.
|
|
||||||
delays := []time.Duration{time.Second, 2 * time.Second, 4 * time.Second}
|
|
||||||
for i, delay := range delays {
|
|
||||||
_ = accept(t, endpoint).Close()
|
|
||||||
|
|
||||||
writeUntilDropped(t, sender, int64(i+1))
|
|
||||||
wantConnectedAgainAfter(t, sender, delay)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantRetries(t, logged, "1s", "2s", "4s")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTheDelayStartsAgainAfterAConnectionThatStayedUpAMinute(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
endpoint := listen(t)
|
|
||||||
sender, logged, _ := run(t, params(remotelog.SchemeTCP, endpoint.Addr()))
|
|
||||||
|
|
||||||
_ = accept(t, endpoint).Close()
|
|
||||||
|
|
||||||
writeUntilDropped(t, sender, 1)
|
|
||||||
wantConnectedAgainAfter(t, sender, time.Second)
|
|
||||||
|
|
||||||
// A connection that fails just short of a minute after it was made
|
|
||||||
// leaves the delay growing.
|
|
||||||
conn := accept(t, endpoint)
|
|
||||||
|
|
||||||
time.Sleep(time.Minute - time.Nanosecond)
|
|
||||||
|
|
||||||
_ = conn.Close()
|
|
||||||
|
|
||||||
writeUntilDropped(t, sender, 2)
|
|
||||||
wantConnectedAgainAfter(t, sender, 2*time.Second)
|
|
||||||
|
|
||||||
// One that fails a minute after it was made starts it again from a
|
|
||||||
// second.
|
|
||||||
conn = accept(t, endpoint)
|
|
||||||
|
|
||||||
time.Sleep(time.Minute)
|
|
||||||
|
|
||||||
_ = conn.Close()
|
|
||||||
|
|
||||||
writeUntilDropped(t, sender, 3)
|
|
||||||
wantConnectedAgainAfter(t, sender, time.Second)
|
|
||||||
wantRetries(t, logged, "1s", "2s", "1s")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestALineTooLongForADatagramIsDroppedAlone(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
endpoint, err := (&net.ListenConfig{}).ListenPacket(t.Context(), "udp", loopback)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("listen: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = endpoint.Close() })
|
|
||||||
|
|
||||||
sender, logged, _ := run(t, params(remotelog.SchemeUDP, endpoint.LocalAddr()))
|
|
||||||
|
|
||||||
// With its header, the first line's record is longer than the 65507
|
|
||||||
// bytes a UDP datagram over IPv4 holds. It is dropped, nothing is
|
|
||||||
// logged, and the next line is sent at once.
|
|
||||||
_, _ = sender.Write([]byte(strings.Repeat("x", 65507) + "\nnext\n"))
|
|
||||||
|
|
||||||
synctest.Wait()
|
|
||||||
wantCounts(t, sender, 1, 1, 0)
|
|
||||||
wantRetries(t, logged)
|
|
||||||
|
|
||||||
datagram := make([]byte, 1024)
|
|
||||||
|
|
||||||
n, _, err := endpoint.ReadFrom(datagram)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("read: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := record(t, local0Info, appName, "next")
|
|
||||||
if string(datagram[:n]) != want {
|
|
||||||
t.Errorf("datagram %q, want %q", datagram[:n], want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRecordsWaitingAtTheStopAreSent(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
certificate, roots := testCertificate(t)
|
|
||||||
endpoint := startTLSEndpoint(t, certificate)
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
// The endpoint refuses the sender's first connection: it fails to
|
|
||||||
// connect, and waits a second to try again.
|
|
||||||
endpoint.refusing.Store(true)
|
|
||||||
|
|
||||||
endpointParams := params(remotelog.SchemeTLS, endpoint.addr)
|
|
||||||
endpointParams.RootCAs = roots
|
|
||||||
sender, logged, stop := run(t, endpointParams)
|
|
||||||
|
|
||||||
synctest.Wait()
|
|
||||||
wantRetries(t, logged, "1s")
|
|
||||||
|
|
||||||
_, _ = sender.Write([]byte("one\ntwo\n"))
|
|
||||||
|
|
||||||
endpoint.refusing.Store(false)
|
|
||||||
|
|
||||||
// Stopped before that second is over, it connects to send them.
|
|
||||||
stop()
|
|
||||||
wantCounts(t, sender, 2, 0, 0)
|
|
||||||
|
|
||||||
frames := bufio.NewReader(endpoint.next(t))
|
|
||||||
wantFrame(t, frames, record(t, local0Info, appName, "one"))
|
|
||||||
wantFrame(t, frames, record(t, local0Info, appName, "two"))
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// output collects what the sender logs.
|
|
||||||
type output struct {
|
|
||||||
mu sync.Mutex
|
|
||||||
buf bytes.Buffer
|
|
||||||
}
|
|
||||||
|
|
||||||
// Write adds lines the sender logs.
|
|
||||||
func (o *output) Write(p []byte) (int, error) {
|
|
||||||
o.mu.Lock()
|
|
||||||
defer o.mu.Unlock()
|
|
||||||
|
|
||||||
return o.buf.Write(p)
|
|
||||||
}
|
|
||||||
|
|
||||||
// text returns everything logged so far.
|
|
||||||
func (o *output) text() string {
|
|
||||||
o.mu.Lock()
|
|
||||||
defer o.mu.Unlock()
|
|
||||||
|
|
||||||
return o.buf.String()
|
|
||||||
}
|
|
||||||
|
|
||||||
// params returns the settings of a Sender for the endpoint at addr, in
|
|
||||||
// the form scheme names: room for ten lines, the default facility, and
|
|
||||||
// appName.
|
|
||||||
func params(scheme string, addr net.Addr) remotelog.Params {
|
|
||||||
return remotelog.Params{
|
|
||||||
URL: &url.URL{Scheme: scheme, Host: addr.String()},
|
|
||||||
Buffer: 10,
|
|
||||||
Facility: local0,
|
|
||||||
AppName: appName,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// run runs a Sender with settings until the test ends or the function
|
|
||||||
// it returns is called, which waits for Run to return. It returns the
|
|
||||||
// Sender, and what it logs.
|
|
||||||
func run(t *testing.T, settings remotelog.Params) (*remotelog.Sender, *output, func()) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
sender := remotelog.New(settings)
|
|
||||||
logged := &output{}
|
|
||||||
ctx, cancel := context.WithCancel(t.Context())
|
|
||||||
ran := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
sender.Run(ctx, slog.New(slog.NewJSONHandler(logged, nil)))
|
|
||||||
close(ran)
|
|
||||||
}()
|
|
||||||
|
|
||||||
stop := func() {
|
|
||||||
cancel()
|
|
||||||
<-ran
|
|
||||||
}
|
|
||||||
t.Cleanup(stop)
|
|
||||||
|
|
||||||
return sender, logged, stop
|
|
||||||
}
|
|
||||||
|
|
||||||
// listen returns a TCP listener on the loopback address, closed when the
|
|
||||||
// test ends.
|
|
||||||
func listen(t *testing.T) net.Listener {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", loopback)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("listen: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = listener.Close() })
|
|
||||||
|
|
||||||
return listener
|
|
||||||
}
|
|
||||||
|
|
||||||
// accept returns the next connection to listener, closed when the test
|
|
||||||
// ends.
|
|
||||||
func accept(t *testing.T, listener net.Listener) net.Conn {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
conn, err := listener.Accept()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("accept: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = conn.Close() })
|
|
||||||
|
|
||||||
return conn
|
|
||||||
}
|
|
||||||
|
|
||||||
// tlsEndpoint is a syslog+tls endpoint on the loopback address, which a
|
|
||||||
// test starts outside its bubble. It keeps its listener until the test
|
|
||||||
// ends, and either takes each connection or refuses it.
|
|
||||||
type tlsEndpoint struct {
|
|
||||||
addr net.Addr
|
|
||||||
// refusing is set while the endpoint closes each connection before the
|
|
||||||
// TLS handshake, which fails the sender's attempt to connect.
|
|
||||||
refusing atomic.Bool
|
|
||||||
// conns are the connections it has taken, after the handshake.
|
|
||||||
conns chan net.Conn
|
|
||||||
}
|
|
||||||
|
|
||||||
// startTLSEndpoint starts a tlsEndpoint with certificate, which takes
|
|
||||||
// connections until it is told to refuse them.
|
|
||||||
func startTLSEndpoint(t *testing.T, certificate tls.Certificate) *tlsEndpoint {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
listener := listen(t)
|
|
||||||
endpoint := &tlsEndpoint{addr: listener.Addr(), conns: make(chan net.Conn, 10)}
|
|
||||||
config := &tls.Config{
|
|
||||||
Certificates: []tls.Certificate{certificate},
|
|
||||||
MinVersion: tls.VersionTLS12,
|
|
||||||
}
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
for {
|
|
||||||
conn, err := listener.Accept()
|
|
||||||
if err != nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
server := tls.Server(conn, config)
|
|
||||||
if endpoint.refusing.Load() || server.HandshakeContext(t.Context()) != nil {
|
|
||||||
_ = conn.Close()
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
endpoint.conns <- server
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
return endpoint
|
|
||||||
}
|
|
||||||
|
|
||||||
// next returns the next connection the endpoint has taken, closed when
|
|
||||||
// the test ends.
|
|
||||||
func (e *tlsEndpoint) next(t *testing.T) net.Conn {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
conn := <-e.conns
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = conn.Close() })
|
|
||||||
|
|
||||||
return conn
|
|
||||||
}
|
|
||||||
|
|
||||||
// record returns the record of line made as the test started, with the
|
|
||||||
// priority and the app name given.
|
|
||||||
func record(t *testing.T, priority, app, line string) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
hostname, err := os.Hostname()
|
|
||||||
if err != nil || hostname == "" {
|
|
||||||
hostname = "-"
|
|
||||||
}
|
|
||||||
|
|
||||||
return priority + "1 " + started + " " + hostname + " " + app + " - - - " + line
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantFrame reads the next octet-counted frame from frames, and checks
|
|
||||||
// that it holds want.
|
|
||||||
func wantFrame(t *testing.T, frames *bufio.Reader, want string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
count, err := frames.ReadString(' ')
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("read a frame's length: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
length, err := strconv.Atoi(strings.TrimSuffix(count, " "))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("frame starts %q, not with its length", count)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := make([]byte, length)
|
|
||||||
|
|
||||||
_, err = io.ReadFull(frames, got)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("read a frame: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if string(got) != want {
|
|
||||||
t.Errorf("frame %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantCounts checks the records sender has sent, dropped and holds in
|
|
||||||
// its buffer.
|
|
||||||
func wantCounts(
|
|
||||||
t *testing.T, sender *remotelog.Sender, sent, dropped int64, depth int,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
if sender.Sent() != sent || sender.Dropped() != dropped || sender.Depth() != depth {
|
|
||||||
t.Fatalf("sent %d, dropped %d, %d in the buffer; want %d, %d and %d",
|
|
||||||
sender.Sent(), sender.Dropped(), sender.Depth(), sent, dropped, depth)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeUntilDropped writes a line at a time until the count of records
|
|
||||||
// sender has dropped reaches dropped. The records it sends on a
|
|
||||||
// connection the endpoint has closed are lost before one fails; how many
|
|
||||||
// depends on when the endpoint's host answers that the connection is
|
|
||||||
// gone.
|
|
||||||
func writeUntilDropped(t *testing.T, sender *remotelog.Sender, dropped int64) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for sender.Dropped() < dropped {
|
|
||||||
_, _ = sender.Write([]byte("lost\n"))
|
|
||||||
|
|
||||||
synctest.Wait()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantConnectedAgainAfter writes a line while the sender waits to connect
|
|
||||||
// again, and checks that it connects, and takes the line from the buffer,
|
|
||||||
// only once delay is over.
|
|
||||||
func wantConnectedAgainAfter(
|
|
||||||
t *testing.T, sender *remotelog.Sender, delay time.Duration,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
_, _ = sender.Write([]byte("waiting\n"))
|
|
||||||
|
|
||||||
time.Sleep(delay - time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
|
|
||||||
if sender.Depth() != 1 {
|
|
||||||
t.Fatalf("connected again before %v", delay)
|
|
||||||
}
|
|
||||||
|
|
||||||
time.Sleep(time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
|
|
||||||
if sender.Depth() != 0 {
|
|
||||||
t.Fatalf("not connected again after %v", delay)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantRetries checks that the sender logged a failure, of an attempt to
|
|
||||||
// connect or of a connection, for each of delays, the time until the next
|
|
||||||
// attempt, in order, and logged nothing else.
|
|
||||||
func wantRetries(t *testing.T, logged *output, delays ...string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var got []string
|
|
||||||
|
|
||||||
for line := range strings.Lines(logged.text()) {
|
|
||||||
var fields map[string]any
|
|
||||||
|
|
||||||
err := json.Unmarshal([]byte(line), &fields)
|
|
||||||
if err != nil || fields["msg"] != "sending to SWWAF_LOG_REMOTE_URL failed" {
|
|
||||||
t.Fatalf("logged %q", line)
|
|
||||||
}
|
|
||||||
|
|
||||||
delay, _ := fields["connecting_again_in"].(string)
|
|
||||||
got = append(got, delay)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(got, delays) {
|
|
||||||
t.Errorf("logged failures to connect again in %v, want %v", got, delays)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// testCertificate returns a certificate for 127.0.0.1 that is its own
|
|
||||||
// CA, and a pool that holds it. It is valid on the bubble's clock, which
|
|
||||||
// starts at 2000-01-01T00:00:00Z.
|
|
||||||
func testCertificate(t *testing.T) (tls.Certificate, *x509.CertPool) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("generate a key: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
template := &x509.Certificate{
|
|
||||||
SerialNumber: big.NewInt(1),
|
|
||||||
Subject: pkix.Name{CommonName: "smallwebwaf test CA"},
|
|
||||||
NotBefore: time.Date(1999, 12, 31, 0, 0, 0, 0, time.UTC),
|
|
||||||
NotAfter: time.Date(2000, 1, 2, 0, 0, 0, 0, time.UTC),
|
|
||||||
IsCA: true,
|
|
||||||
BasicConstraintsValid: true,
|
|
||||||
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageDigitalSignature,
|
|
||||||
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
|
|
||||||
IPAddresses: []net.IP{net.IPv4(127, 0, 0, 1)},
|
|
||||||
}
|
|
||||||
|
|
||||||
der, err := x509.CreateCertificate(rand.Reader, template, template,
|
|
||||||
&key.PublicKey, key)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("create a certificate: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
certificate, err := x509.ParseCertificate(der)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("parse the certificate: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
roots := x509.NewCertPool()
|
|
||||||
roots.AddCert(certificate)
|
|
||||||
|
|
||||||
return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key}, roots
|
|
||||||
}
|
|
||||||
@@ -9,8 +9,6 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// The action a request line names: what smallwebwaf did with the
|
// The action a request line names: what smallwebwaf did with the
|
||||||
@@ -28,12 +26,8 @@ const (
|
|||||||
// ActionRateLimited is a request refused because it took its client
|
// ActionRateLimited is a request refused because it took its client
|
||||||
// over a rate limit, which bans the client.
|
// over a rate limit, which bans the client.
|
||||||
ActionRateLimited = "rate_limited"
|
ActionRateLimited = "rate_limited"
|
||||||
// ActionBanned is a request refused because a ban covers its client,
|
// ActionBanned is a request refused because a ban covers its client.
|
||||||
// or because it matched a ban rule, which bans the client.
|
|
||||||
ActionBanned = "banned"
|
ActionBanned = "banned"
|
||||||
// ActionRuleBlocked is a request refused because it matched a block
|
|
||||||
// rule.
|
|
||||||
ActionRuleBlocked = "rule_blocked"
|
|
||||||
// ActionDenied is a request refused because its client is in
|
// ActionDenied is a request refused because its client is in
|
||||||
// SWWAF_DENY_NETS.
|
// SWWAF_DENY_NETS.
|
||||||
ActionDenied = "denied"
|
ActionDenied = "denied"
|
||||||
@@ -51,74 +45,32 @@ const OffenceLimit = "limit"
|
|||||||
// timeLayout is RFC 3339 with milliseconds.
|
// timeLayout is RFC 3339 with milliseconds.
|
||||||
const timeLayout = "2006-01-02T15:04:05.000Z07:00"
|
const timeLayout = "2006-01-02T15:04:05.000Z07:00"
|
||||||
|
|
||||||
// Line is one request's line in the request log. The field names, and
|
// Line is one request's line in the request log. The field names are
|
||||||
// their order, are those of the "Request log" section of SPEC.md. A field
|
// those of the "Request log" section of SPEC.md.
|
||||||
// that may not apply to a request is left out of its line when it does
|
|
||||||
// not.
|
|
||||||
//
|
//
|
||||||
//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
|
//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
|
||||||
type Line struct {
|
type Line struct {
|
||||||
Type string `json:"type"`
|
Type string `json:"type"`
|
||||||
|
Time string `json:"time"`
|
||||||
// The standard web log fields. Scheme is how the client reached
|
ClientIP string `json:"client_ip"`
|
||||||
// smallwebwaf, or the trusted proxy in front of it.
|
PeerIP string `json:"peer_ip"`
|
||||||
Time string `json:"time"`
|
Country string `json:"country"`
|
||||||
Instance string `json:"instance"`
|
Method string `json:"method"`
|
||||||
ClientIP string `json:"client_ip"`
|
Host string `json:"host"`
|
||||||
Method string `json:"method"`
|
Path string `json:"path"`
|
||||||
Scheme string `json:"scheme"`
|
Query string `json:"query"`
|
||||||
Host string `json:"host"`
|
Protocol string `json:"protocol"`
|
||||||
Path string `json:"path"`
|
Status int `json:"status"`
|
||||||
Query string `json:"query"`
|
UpstreamStatus int `json:"upstream_status,omitempty"`
|
||||||
Protocol string `json:"protocol"`
|
RequestBytes int64 `json:"request_bytes"`
|
||||||
Status int `json:"status"`
|
ResponseBytes int64 `json:"response_bytes"`
|
||||||
RequestBytes int64 `json:"request_bytes"`
|
Referer string `json:"referer"`
|
||||||
ResponseBytes int64 `json:"response_bytes"`
|
UserAgent string `json:"user_agent"`
|
||||||
Referer string `json:"referer"`
|
Action string `json:"action"`
|
||||||
UserAgent string `json:"user_agent"`
|
|
||||||
|
|
||||||
// Request detail. RequestID is the X-Request-ID a trusted proxy sent,
|
|
||||||
// or a new one, and is sent on to the app. ForwardedFor is the
|
|
||||||
// X-Forwarded-For header as received. ClientGroup is the netblock the
|
|
||||||
// client is counted as.
|
|
||||||
RequestID string `json:"request_id"`
|
|
||||||
PeerIP string `json:"peer_ip"`
|
|
||||||
ForwardedFor string `json:"forwarded_for,omitempty"`
|
|
||||||
ClientGroup string `json:"client_group"`
|
|
||||||
Country string `json:"country"`
|
|
||||||
ContentType string `json:"content_type,omitempty"`
|
|
||||||
// ContentLength is the length of its body the request announced.
|
|
||||||
ContentLength int64 `json:"content_length,omitempty"`
|
|
||||||
// RequestHeaders are the headers SWWAF_LOG_REQUEST_HEADERS names that
|
|
||||||
// the request carried, by name in lower case.
|
|
||||||
RequestHeaders map[string]string `json:"request_headers,omitempty"`
|
|
||||||
HasAuthorization bool `json:"has_authorization,omitempty"`
|
|
||||||
HasCookie bool `json:"has_cookie,omitempty"`
|
|
||||||
// Websocket is true when the connection was upgraded, as for a
|
|
||||||
// WebSocket.
|
|
||||||
Websocket bool `json:"websocket,omitempty"`
|
|
||||||
|
|
||||||
// Response detail, from the headers of the answer: the app's, as
|
|
||||||
// passed on, or those of smallwebwaf's own. Aborted is true when the
|
|
||||||
// client went away early.
|
|
||||||
ResponseContentType string `json:"response_content_type,omitempty"`
|
|
||||||
UpstreamStatus int `json:"upstream_status,omitempty"`
|
|
||||||
CacheControl string `json:"cache_control,omitempty"`
|
|
||||||
Location string `json:"location,omitempty"`
|
|
||||||
Aborted bool `json:"aborted,omitempty"`
|
|
||||||
|
|
||||||
// The decision.
|
|
||||||
Action string `json:"action"`
|
|
||||||
// WouldAction is, in observe mode, the action enforce mode would have
|
// WouldAction is, in observe mode, the action enforce mode would have
|
||||||
// taken with a request it would have refused: ActionDenied,
|
// taken with a request it would have refused: ActionDenied,
|
||||||
// ActionBanned, ActionCountryDenied, ActionRateLimited or
|
// ActionBanned, ActionCountryDenied or ActionRateLimited.
|
||||||
// ActionRuleBlocked.
|
|
||||||
WouldAction string `json:"would_action,omitempty"`
|
WouldAction string `json:"would_action,omitempty"`
|
||||||
// Counts are the client's requests as the rate limits counted them
|
|
||||||
// with this one, for a request they counted.
|
|
||||||
Counts ratelimit.Counts `json:"counts,omitzero"`
|
|
||||||
// RuleIDs are the ids of the rule file rules the request matched.
|
|
||||||
RuleIDs []string `json:"rule_ids,omitempty"`
|
|
||||||
// LimitHit is the window whose rate limit the request went over:
|
// LimitHit is the window whose rate limit the request went over:
|
||||||
// minute, hour or day.
|
// minute, hour or day.
|
||||||
LimitHit string `json:"limit_hit,omitempty"`
|
LimitHit string `json:"limit_hit,omitempty"`
|
||||||
@@ -127,18 +79,11 @@ type Line struct {
|
|||||||
// BanExpires is when the ban the request made, or was refused under,
|
// BanExpires is when the ban the request made, or was refused under,
|
||||||
// ends: a time, or "permanent".
|
// ends: a time, or "permanent".
|
||||||
BanExpires string `json:"ban_expires,omitempty"`
|
BanExpires string `json:"ban_expires,omitempty"`
|
||||||
|
// Aborted is true when the client went away early.
|
||||||
// The timings, in milliseconds. DurationChecks is the time until the
|
Aborted bool `json:"aborted,omitempty"`
|
||||||
// checks were done. DurationUpstreamConnect, DurationUpstreamFirstByte
|
// DurationTotal and DurationUpstreamTotal are in milliseconds.
|
||||||
// and DurationUpstreamTotal run from when the request was handed to the
|
DurationTotal float64 `json:"duration_total"`
|
||||||
// app: until there was a connection to it, until the first byte of its
|
DurationUpstreamTotal float64 `json:"duration_upstream_total,omitempty"`
|
||||||
// answer arrived, and until the end. Each but DurationTotal is nil for
|
|
||||||
// a request that did not get that far.
|
|
||||||
DurationTotal float64 `json:"duration_total"`
|
|
||||||
DurationChecks *float64 `json:"duration_checks,omitempty"`
|
|
||||||
DurationUpstreamConnect *float64 `json:"duration_upstream_connect,omitempty"`
|
|
||||||
DurationUpstreamFirstByte *float64 `json:"duration_upstream_first_byte,omitempty"`
|
|
||||||
DurationUpstreamTotal *float64 `json:"duration_upstream_total,omitempty"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Write writes line to w as one JSON line marked "type":"request".
|
// Write writes line to w as one JSON line marked "type":"request".
|
||||||
|
|||||||
@@ -50,11 +50,7 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
unset := []string{
|
unset := []string{
|
||||||
"forwarded_for", "content_type", "content_length", "request_headers",
|
"upstream_status", "limit_hit", "offence", "ban_expires", "aborted",
|
||||||
"has_authorization", "has_cookie", "websocket", "response_content_type",
|
|
||||||
"upstream_status", "cache_control", "location", "aborted", "counts",
|
|
||||||
"limit_hit", "offence", "ban_expires", "duration_checks",
|
|
||||||
"duration_upstream_connect", "duration_upstream_first_byte",
|
|
||||||
"duration_upstream_total",
|
"duration_upstream_total",
|
||||||
}
|
}
|
||||||
for _, name := range unset {
|
for _, name := range unset {
|
||||||
|
|||||||
@@ -1,469 +0,0 @@
|
|||||||
// Package rules reads the rule files: the plain text files in
|
|
||||||
// SWWAF_RULES_DIR, one rule to a line, that each request is checked
|
|
||||||
// against, as the "Rule files" section of SPEC.md describes. They are read
|
|
||||||
// at start, and again once the directory has had no change for a short
|
|
||||||
// time after one is edited, added or removed.
|
|
||||||
package rules
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/hex"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"regexp"
|
|
||||||
"slices"
|
|
||||||
"strings"
|
|
||||||
"sync/atomic"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/fsnotify/fsnotify"
|
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The actions a rule takes when it matches.
|
|
||||||
const (
|
|
||||||
// ActionLog notes the match in the request log, and does nothing else.
|
|
||||||
ActionLog = "log"
|
|
||||||
// ActionBlock refuses the request with 403.
|
|
||||||
ActionBlock = "block"
|
|
||||||
// ActionBan refuses the request and bans the client's netblock: the
|
|
||||||
// request is a clear sign of attack.
|
|
||||||
ActionBan = "ban"
|
|
||||||
)
|
|
||||||
|
|
||||||
// extension ends the name of every rule file.
|
|
||||||
const extension = ".rules"
|
|
||||||
|
|
||||||
// quietTime is how long SWWAF_RULES_DIR must go without a change before
|
|
||||||
// the rule files are read again, so that a file still being written, such
|
|
||||||
// as one saved in place, appended to or copied in with scp, is read only
|
|
||||||
// once whole.
|
|
||||||
const quietTime = 2 * time.Second
|
|
||||||
|
|
||||||
// headerTarget starts the target that is one request header,
|
|
||||||
// header:<Name>.
|
|
||||||
const headerTarget = "header:"
|
|
||||||
|
|
||||||
// escapeLength is the length of a percent escape, such as %2e.
|
|
||||||
const escapeLength = 3
|
|
||||||
|
|
||||||
var (
|
|
||||||
// ruleLine is a rule: four fields separated by spaces or tabs, of
|
|
||||||
// which the fourth, the regex, runs to the end of the line.
|
|
||||||
ruleLine = regexp.MustCompile(`^([^ \t]+)[ \t]+([^ \t]+)[ \t]+([^ \t]+)[ \t]+(.+)$`)
|
|
||||||
// idChars are the characters of a rule's id.
|
|
||||||
idChars = regexp.MustCompile(`^[A-Za-z0-9_-]+$`)
|
|
||||||
)
|
|
||||||
|
|
||||||
var (
|
|
||||||
errNotRule = errors.New(
|
|
||||||
"is not a rule: an id, a target, an action and a regex, " +
|
|
||||||
"separated by spaces or tabs")
|
|
||||||
errNotID = errors.New("is not an id of letters, digits, - and _")
|
|
||||||
errNotTarget = errors.New(
|
|
||||||
"is not path, query, uri, method, host, user_agent, referer or header:<Name>")
|
|
||||||
errNotHeaderName = errors.New(
|
|
||||||
"has a character after header: that no header name can have")
|
|
||||||
errHeaderTakenOut = errors.New(
|
|
||||||
"names a header that Go's HTTP server takes out of every request, " +
|
|
||||||
"so a rule never sees it")
|
|
||||||
errNotAction = errors.New("is not log, block or ban")
|
|
||||||
errNotRegex = errors.New("does not compile")
|
|
||||||
errUsedTwice = errors.New("is already the id of the rule at")
|
|
||||||
)
|
|
||||||
|
|
||||||
// Rule is one rule of a rule file.
|
|
||||||
type Rule struct {
|
|
||||||
// ID names the rule in the request log, the metrics and ban notes.
|
|
||||||
ID string
|
|
||||||
// Target is what the regex is matched against, such as path or
|
|
||||||
// header:Accept.
|
|
||||||
Target string
|
|
||||||
// Action is ActionLog, ActionBlock or ActionBan.
|
|
||||||
Action string
|
|
||||||
|
|
||||||
regex *regexp.Regexp
|
|
||||||
}
|
|
||||||
|
|
||||||
// Params are what Load needs.
|
|
||||||
type Params struct {
|
|
||||||
// Dir is the directory of the rule files (SWWAF_RULES_DIR).
|
|
||||||
Dir string
|
|
||||||
// Enabled is SWWAF_RULES_ENABLED: while it is false, no file is read
|
|
||||||
// and no rule loaded.
|
|
||||||
Enabled bool
|
|
||||||
// ProcessLog receives how many rules were read, and the error in a
|
|
||||||
// rule file edited while smallwebwaf runs.
|
|
||||||
ProcessLog *slog.Logger
|
|
||||||
}
|
|
||||||
|
|
||||||
// Files are the rule files of a running smallwebwaf, and the rules read
|
|
||||||
// from them. They are safe for concurrent use.
|
|
||||||
type Files struct {
|
|
||||||
params Params
|
|
||||||
// rules are the rules loaded, in the order of their files' names, and
|
|
||||||
// then of their lines.
|
|
||||||
rules atomic.Pointer[[]Rule]
|
|
||||||
}
|
|
||||||
|
|
||||||
// Load reads the rules of every *.rules file in Dir, in the order of the
|
|
||||||
// files' names, unless Enabled is false. A Dir that cannot be read is an
|
|
||||||
// error, and so is a line that is not a rule, a header name with a
|
|
||||||
// character no header name can have, a rule for the Host or the
|
|
||||||
// Transfer-Encoding header, which Go's HTTP server takes out of every
|
|
||||||
// request, a regex that does not compile and an id used twice, each named
|
|
||||||
// with its file and line.
|
|
||||||
func Load(params Params) (*Files, error) {
|
|
||||||
f := &Files{params: params}
|
|
||||||
f.rules.Store(&[]Rule{})
|
|
||||||
|
|
||||||
if !params.Enabled {
|
|
||||||
return f, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
rules, err := read(params.Dir)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
f.rules.Store(&rules)
|
|
||||||
f.logRead(len(rules))
|
|
||||||
|
|
||||||
return f, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Match checks r against the rules, in order, and returns those it
|
|
||||||
// matches, up to the first whose action refuses it, block or ban, which
|
|
||||||
// is then the last one returned.
|
|
||||||
func (f *Files) Match(r *http.Request) []Rule {
|
|
||||||
var matched []Rule
|
|
||||||
|
|
||||||
for _, rule := range *f.rules.Load() {
|
|
||||||
if !rule.matches(r) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
matched = append(matched, rule)
|
|
||||||
if rule.Action != ActionLog {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return matched
|
|
||||||
}
|
|
||||||
|
|
||||||
// Len returns how many rules are loaded.
|
|
||||||
func (f *Files) Len() int {
|
|
||||||
return len(*f.rules.Load())
|
|
||||||
}
|
|
||||||
|
|
||||||
// Watch watches Dir until ctx is done, and reads the rule files again
|
|
||||||
// once Dir has had no change for quietTime, after one is edited, added or
|
|
||||||
// removed, and after Watch starts watching. If they then hold an error,
|
|
||||||
// the rules stay as they were, the error is logged with its file and
|
|
||||||
// line, and the files are read again after the next change. If Dir cannot
|
|
||||||
// be watched, that is logged, and the rules stay as they were loaded.
|
|
||||||
// While Enabled is false, Watch returns at once.
|
|
||||||
func (f *Files) Watch(ctx context.Context) {
|
|
||||||
if !f.params.Enabled {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
watcher, err := fsnotify.NewWatcher()
|
|
||||||
if err == nil {
|
|
||||||
defer func() {
|
|
||||||
_ = watcher.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
err = watcher.Add(f.params.Dir)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
f.params.ProcessLog.Error("cannot watch the rule files for edits",
|
|
||||||
"error", err.Error())
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
f.params.ProcessLog.Info("watching the rule files for edits",
|
|
||||||
"directory", f.params.Dir)
|
|
||||||
|
|
||||||
f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
|
|
||||||
}
|
|
||||||
|
|
||||||
// readAfterChanges reads the rule files again once quietTime has passed
|
|
||||||
// without a change from events, until ctx is done, and logs the errors
|
|
||||||
// from errs. The wait starts at once, as if for a change, so that an edit
|
|
||||||
// saved after Load read the files, and before Dir was watched, is taken
|
|
||||||
// in too.
|
|
||||||
func (f *Files) readAfterChanges(
|
|
||||||
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
|
|
||||||
) {
|
|
||||||
quiet := time.NewTimer(quietTime)
|
|
||||||
defer quiet.Stop()
|
|
||||||
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
case <-events:
|
|
||||||
quiet.Reset(quietTime)
|
|
||||||
case <-quiet.C:
|
|
||||||
f.readAgain()
|
|
||||||
case err := <-errs:
|
|
||||||
f.params.ProcessLog.Warn("watching the rule files failed",
|
|
||||||
"error", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// readAgain reads the rule files again, in place of the rules loaded, or
|
|
||||||
// logs the error that keeps the rules as they were.
|
|
||||||
func (f *Files) readAgain() {
|
|
||||||
rules, err := read(f.params.Dir)
|
|
||||||
if err != nil {
|
|
||||||
f.params.ProcessLog.Error(
|
|
||||||
"a rule file has an error, and the rules stay as they were",
|
|
||||||
"error", err.Error())
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
f.rules.Store(&rules)
|
|
||||||
f.logRead(len(rules))
|
|
||||||
}
|
|
||||||
|
|
||||||
// logRead logs that the rule files were read, and how many rules they
|
|
||||||
// hold, which can be none.
|
|
||||||
func (f *Files) logRead(count int) {
|
|
||||||
f.params.ProcessLog.Info("read the rule files",
|
|
||||||
"directory", f.params.Dir, "rules", count)
|
|
||||||
}
|
|
||||||
|
|
||||||
// read returns the rules of every rule file in dir, in the order of the
|
|
||||||
// files' names, and then of their lines. A file whose name starts with a
|
|
||||||
// dot, such as an editor's lock file .#50-app.rules, is not a rule file,
|
|
||||||
// as a shell's *.rules would not match it.
|
|
||||||
func read(dir string) ([]Rule, error) {
|
|
||||||
entries, err := os.ReadDir(dir)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("SWWAF_RULES_DIR cannot be read: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var rules []Rule
|
|
||||||
|
|
||||||
// places are where each id is, as "<file>, line <n>".
|
|
||||||
places := map[string]string{}
|
|
||||||
|
|
||||||
for _, entry := range entries {
|
|
||||||
name := entry.Name()
|
|
||||||
if entry.IsDir() || strings.HasPrefix(name, ".") || filepath.Ext(name) != extension {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
rules, err = readFile(filepath.Join(dir, name), rules, places)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return rules, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// readFile appends the rules of the rule file at path to rules. places
|
|
||||||
// are where each id read so far is, and gain those of the file.
|
|
||||||
func readFile(path string, rules []Rule, places map[string]string) ([]Rule, error) {
|
|
||||||
data, err := os.ReadFile(path) //nolint:gosec // a rule file, in SWWAF_RULES_DIR
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
number := 0
|
|
||||||
|
|
||||||
for line := range strings.Lines(string(data)) {
|
|
||||||
number++
|
|
||||||
place := fmt.Sprintf("%s, line %d", path, number)
|
|
||||||
|
|
||||||
text := strings.TrimSuffix(strings.TrimSuffix(line, "\n"), "\r")
|
|
||||||
|
|
||||||
rule, isRule, err := parse(text)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("%s: %w", place, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !isRule {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
first, used := places[rule.ID]
|
|
||||||
if used {
|
|
||||||
return nil, fmt.Errorf("%s: the id %q %w %s", place, rule.ID, errUsedTwice, first)
|
|
||||||
}
|
|
||||||
|
|
||||||
places[rule.ID] = place
|
|
||||||
rules = append(rules, rule)
|
|
||||||
}
|
|
||||||
|
|
||||||
return rules, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// parse reads a line of a rule file. It returns false for a blank line
|
|
||||||
// and for a comment, a line that starts with #. Spaces and tabs at the
|
|
||||||
// end of the line are not part of its regex, so a line with only those
|
|
||||||
// after its action has no regex, and is not a rule.
|
|
||||||
func parse(line string) (Rule, bool, error) {
|
|
||||||
line = strings.Trim(line, " \t")
|
|
||||||
if line == "" || strings.HasPrefix(line, "#") {
|
|
||||||
return Rule{}, false, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
fields := ruleLine.FindStringSubmatch(line)
|
|
||||||
if fields == nil {
|
|
||||||
return Rule{}, false, errNotRule
|
|
||||||
}
|
|
||||||
|
|
||||||
rule := Rule{ID: fields[1], Target: fields[2], Action: fields[3]}
|
|
||||||
headerName, isHeader := strings.CutPrefix(rule.Target, headerTarget)
|
|
||||||
|
|
||||||
switch {
|
|
||||||
case !idChars.MatchString(rule.ID):
|
|
||||||
return Rule{}, false, fmt.Errorf("the id %q %w", rule.ID, errNotID)
|
|
||||||
case !isTarget(rule.Target):
|
|
||||||
return Rule{}, false, fmt.Errorf("the target %q %w", rule.Target, errNotTarget)
|
|
||||||
case isHeader && !config.IsHeaderName(headerName):
|
|
||||||
return Rule{}, false, fmt.Errorf("the target %q %w", rule.Target, errNotHeaderName)
|
|
||||||
case strings.EqualFold(rule.Target, headerTarget+"Host"):
|
|
||||||
return Rule{}, false, fmt.Errorf(
|
|
||||||
"the target %q %w; the request's host is the target host",
|
|
||||||
rule.Target, errHeaderTakenOut)
|
|
||||||
case strings.EqualFold(rule.Target, headerTarget+"Transfer-Encoding"):
|
|
||||||
return Rule{}, false, fmt.Errorf("the target %q %w", rule.Target, errHeaderTakenOut)
|
|
||||||
case !slices.Contains([]string{ActionLog, ActionBlock, ActionBan}, rule.Action):
|
|
||||||
return Rule{}, false, fmt.Errorf("the action %q %w", rule.Action, errNotAction)
|
|
||||||
}
|
|
||||||
|
|
||||||
regex, err := regexp.Compile(fields[4])
|
|
||||||
if err != nil {
|
|
||||||
return Rule{}, false, fmt.Errorf("the regex %w: %w", errNotRegex, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
rule.regex = regex
|
|
||||||
|
|
||||||
return rule, true, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// isTarget reports whether target is one a rule may have.
|
|
||||||
func isTarget(target string) bool {
|
|
||||||
switch target {
|
|
||||||
case "path", "query", "uri", "method", "host", "user_agent", "referer":
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
name, isHeader := strings.CutPrefix(target, headerTarget)
|
|
||||||
|
|
||||||
return isHeader && name != ""
|
|
||||||
}
|
|
||||||
|
|
||||||
// matches reports whether the rule's regex matches its target in r. For
|
|
||||||
// uri it is matched against the path and query as received, and against
|
|
||||||
// them once percent-decoded, so that an encoded probe cannot slip past.
|
|
||||||
func (rule Rule) matches(r *http.Request) bool {
|
|
||||||
if rule.Target == "uri" {
|
|
||||||
uri := pathAndQuery(r)
|
|
||||||
|
|
||||||
return rule.regex.MatchString(uri) || rule.regex.MatchString(decodeOnce(uri))
|
|
||||||
}
|
|
||||||
|
|
||||||
return rule.regex.MatchString(value(rule.Target, r))
|
|
||||||
}
|
|
||||||
|
|
||||||
// value returns what a rule with target, other than uri, is matched
|
|
||||||
// against in r: the path and the query as the client sent them, before
|
|
||||||
// any decoding or re-encoding, split at the first ?, and a header's values
|
|
||||||
// joined by ", ", as HTTP joins those of a header sent more than once.
|
|
||||||
func value(target string, r *http.Request) string {
|
|
||||||
switch target {
|
|
||||||
case "path":
|
|
||||||
path, _, _ := strings.Cut(pathAndQuery(r), "?")
|
|
||||||
|
|
||||||
return path
|
|
||||||
case "query":
|
|
||||||
_, query, _ := strings.Cut(pathAndQuery(r), "?")
|
|
||||||
|
|
||||||
return query
|
|
||||||
case "method":
|
|
||||||
return r.Method
|
|
||||||
case "host":
|
|
||||||
return r.Host
|
|
||||||
case "user_agent":
|
|
||||||
return header(r, "User-Agent")
|
|
||||||
case "referer":
|
|
||||||
return header(r, "Referer")
|
|
||||||
default:
|
|
||||||
return header(r, strings.TrimPrefix(target, headerTarget))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// pathAndQuery returns the target of r's request line, r.RequestURI, as
|
|
||||||
// the client sent it, less any scheme and host: a target with a scheme
|
|
||||||
// gives what follows the scheme and its :, and the host when // follows.
|
|
||||||
// So http://host/path, as a client sends it to a proxy, gives /path, and
|
|
||||||
// so does http:/path, which Go reads as a target with a scheme and no
|
|
||||||
// host. r.URL is not used: when the path holds a character it escapes,
|
|
||||||
// such as \ or a non-ASCII byte, it decodes the whole path and escapes it
|
|
||||||
// again, so that \ becomes %5C and %2e a dot.
|
|
||||||
func pathAndQuery(r *http.Request) string {
|
|
||||||
if !r.URL.IsAbs() {
|
|
||||||
return r.RequestURI
|
|
||||||
}
|
|
||||||
|
|
||||||
_, afterScheme, _ := strings.Cut(r.RequestURI, ":")
|
|
||||||
|
|
||||||
hostAndRest, hasHost := strings.CutPrefix(afterScheme, "//")
|
|
||||||
if !hasHost {
|
|
||||||
return afterScheme
|
|
||||||
}
|
|
||||||
|
|
||||||
start := strings.IndexAny(hostAndRest, "/?")
|
|
||||||
if start < 0 {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
return hostAndRest[start:]
|
|
||||||
}
|
|
||||||
|
|
||||||
// header returns the values of r's header name joined by ", ", or "" if
|
|
||||||
// r has no such header.
|
|
||||||
func header(r *http.Request, name string) string {
|
|
||||||
return strings.Join(r.Header.Values(name), ", ")
|
|
||||||
}
|
|
||||||
|
|
||||||
// decodeOnce returns s with each percent escape, such as %2e, replaced by
|
|
||||||
// the byte it stands for. A % that is not followed by two hex digits is
|
|
||||||
// left as it is, so that a malformed escape cannot keep the rest of s
|
|
||||||
// from being decoded.
|
|
||||||
func decodeOnce(s string) string {
|
|
||||||
var decoded strings.Builder
|
|
||||||
|
|
||||||
for i := 0; i < len(s); i++ {
|
|
||||||
if s[i] == '%' && i+escapeLength <= len(s) {
|
|
||||||
b, err := hex.DecodeString(s[i+1 : i+escapeLength])
|
|
||||||
if err == nil {
|
|
||||||
decoded.Write(b)
|
|
||||||
|
|
||||||
i += escapeLength - 1
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
decoded.WriteByte(s[i])
|
|
||||||
}
|
|
||||||
|
|
||||||
return decoded.String()
|
|
||||||
}
|
|
||||||
@@ -1,662 +0,0 @@
|
|||||||
package rules_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"log/slog"
|
|
||||||
"maps"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"slices"
|
|
||||||
"strconv"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// What the process log says once Watch watches the directory, after
|
|
||||||
// each reading of the rule files, and for one that has an error.
|
|
||||||
watching = "watching the rule files for edits"
|
|
||||||
read = "read the rule files"
|
|
||||||
hasError = "a rule file has an error, and the rules stay as they were"
|
|
||||||
// maxLogLines is how many lines of the process log wait for a test to
|
|
||||||
// read them.
|
|
||||||
maxLogLines = 64
|
|
||||||
// browser is the user agent of an ordinary visitor.
|
|
||||||
browser = "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0"
|
|
||||||
// testFile is the rule file of a test that needs only one, and
|
|
||||||
// firstFile the first of a test's rule files.
|
|
||||||
testFile = "test.rules"
|
|
||||||
firstFile = "00-a.rules"
|
|
||||||
// userAgent is the header that carries the user agent.
|
|
||||||
userAgent = "User-Agent"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestEachTargetMatchesWhatItNames(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, tc := range []struct {
|
|
||||||
name string
|
|
||||||
rule string // its target, action and regex
|
|
||||||
uri string // the request's path and query
|
|
||||||
header http.Header
|
|
||||||
want bool
|
|
||||||
}{
|
|
||||||
{"path as received", `path log ^/%2eenv$`, "/%2eenv", nil, true},
|
|
||||||
{"path not decoded", `path log ^/\.env$`, "/%2eenv", nil, false},
|
|
||||||
{"path without the query", `path log ^/a$`, "/a?b=c", nil, true},
|
|
||||||
{"query as received", `query log ^b=%2e$`, "/a?b=%2e", nil, true},
|
|
||||||
{"uri as received", `uri log ^/a\?b=%2e$`, "/a?b=%2e", nil, true},
|
|
||||||
{"uri decoded", `uri log (\.\./){2}`, "/a?f=%2e%2e%2f%2e%2e%2f", nil, true},
|
|
||||||
{
|
|
||||||
"uri decoded past malformed escapes", `uri log (\.\./){2}&h=%$`,
|
|
||||||
"/a?g=%zz&f=%2e%2e%2f%2e%2e%2f&h=%", nil, true,
|
|
||||||
},
|
|
||||||
{"uri decoded only once", `uri log ^/a\.b$`, "/a%252eb", nil, false},
|
|
||||||
{"method", `method log ^PUT$`, "/", nil, true},
|
|
||||||
{"host", `host log ^app\.example$`, "/", nil, true},
|
|
||||||
{
|
|
||||||
"user_agent", `user_agent log ^sqlmap/`, "/",
|
|
||||||
http.Header{userAgent: {"sqlmap/1.8"}}, true,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"user_agent sent twice", `user_agent log ^curl/8, sqlmap/`, "/",
|
|
||||||
http.Header{userAgent: {"curl/8", "sqlmap/1.8"}}, true,
|
|
||||||
},
|
|
||||||
{"user_agent missing", `user_agent log ^$`, "/", nil, true},
|
|
||||||
{
|
|
||||||
"referer", `referer log ^https://spam\.example/`, "/",
|
|
||||||
http.Header{"Referer": {"https://spam.example/buy"}}, true,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"a header sent twice", `header:x-api-version log ^2, 3$`, "/",
|
|
||||||
http.Header{"X-Api-Version": {"2", "3"}}, true,
|
|
||||||
},
|
|
||||||
{"a header missing", `header:X-Api-Version log ^$`, "/", nil, true},
|
|
||||||
} {
|
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
files := load(t, ruleFiles{testFile: "a-rule " + tc.rule + "\n"})
|
|
||||||
|
|
||||||
// Every request is a PUT, which the method rule looks for.
|
|
||||||
r := httptest.NewRequestWithContext(t.Context(), http.MethodPut,
|
|
||||||
"http://app.example"+tc.uri, nil)
|
|
||||||
maps.Copy(r.Header, tc.header)
|
|
||||||
|
|
||||||
got := len(files.Match(r)) == 1
|
|
||||||
if got != tc.want {
|
|
||||||
t.Errorf("%s matches %s: %t, want %t", tc.rule, tc.uri, got, tc.want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestPathMatchedAsTheClientSentIt(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// Each path holds a character Go's URL type would escape again, \ or
|
|
||||||
// a non-ASCII byte, and each rule is written for the path as sent.
|
|
||||||
for _, tc := range []struct {
|
|
||||||
rule string // its target, action and regex
|
|
||||||
sent string // the path and query the client sent
|
|
||||||
}{
|
|
||||||
{`path log ^/\.\.\\\.\.\\windows\\win\.ini$`, `/..\..\windows\win.ini`},
|
|
||||||
{`path log ^/%2e%2e\\%2e%2e\\windows\\win\.ini$`, `/%2e%2e\%2e%2e\windows\win.ini`},
|
|
||||||
{`path log ^/café$`, "/café?x=1"},
|
|
||||||
{`uri log ^/%2e%2e\\%2e%2e\\boot\.ini\?x=1$`, `/%2e%2e\%2e%2e\boot.ini?x=1`},
|
|
||||||
} {
|
|
||||||
files := load(t, ruleFiles{testFile: "as-sent " + tc.rule + "\n"})
|
|
||||||
|
|
||||||
// The target in origin form, as traefik sends it, in absolute form,
|
|
||||||
// as a client sends it to a proxy, and with a scheme but no host,
|
|
||||||
// which Go reads as absolute form with no host, sending the app
|
|
||||||
// the path.
|
|
||||||
for _, target := range []string{
|
|
||||||
tc.sent, "http://app.example" + tc.sent, "http:" + tc.sent, "foo:" + tc.sent,
|
|
||||||
} {
|
|
||||||
r := httptest.NewRequestWithContext(t.Context(), http.MethodGet, target, nil)
|
|
||||||
wantMatched(t, files, r, "as-sent")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestMatchingStopsAtTheFirstRuleThatRefuses(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
files := load(t, ruleFiles{testFile: `
|
|
||||||
every-path path log ^/
|
|
||||||
no-path path log ^$
|
|
||||||
first-refusal path block ^/probe
|
|
||||||
later-ban path ban ^/probe
|
|
||||||
after path log ^/
|
|
||||||
`})
|
|
||||||
|
|
||||||
// Every log rule that matches is noted, and the block rule ends the
|
|
||||||
// matching.
|
|
||||||
wantMatched(t, files, get(t, "/probe"), "every-path", "first-refusal")
|
|
||||||
wantMatched(t, files, get(t, "/page"), "every-path", "after")
|
|
||||||
|
|
||||||
// A ban rule ends it too.
|
|
||||||
files = load(t, ruleFiles{testFile: "ban path ban ^/\nlater path block ^/\n"})
|
|
||||||
wantMatched(t, files, get(t, "/"), "ban")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestSpacesAndTabsEndingALineAreNotPartOfItsRegex(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
files := load(t, ruleFiles{testFile: "env-file path block ^/\\.env$ \t \n"})
|
|
||||||
|
|
||||||
wantMatched(t, files, get(t, "/.env"), "env-file")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFilesReadInNameOrderThenLineOrder(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
files := load(t, ruleFiles{
|
|
||||||
"50-b.rules": "b1 path log ^/\n\n# a comment\n # an indented one\nb2 path log ^/\n",
|
|
||||||
firstFile: "a1 path log ^/\r\n",
|
|
||||||
// None is a rule file.
|
|
||||||
"notes.txt": "notes, not rules\n",
|
|
||||||
"10-c.rules.bak": "an old copy\n",
|
|
||||||
"20-d.rules/keep": "a file in a directory\n",
|
|
||||||
})
|
|
||||||
|
|
||||||
wantMatched(t, files, get(t, "/"), "a1", "b1", "b2")
|
|
||||||
|
|
||||||
if files.Len() != 3 {
|
|
||||||
t.Errorf("%d rules loaded, want 3", files.Len())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFileWhoseNameStartsWithADotIsNotARuleFile(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := writeFiles(t, ruleFiles{firstFile: "probe path block ^/probe\n"})
|
|
||||||
|
|
||||||
// The lock file Emacs makes beside a file while it is edited: a link to
|
|
||||||
// nothing, which cannot be read.
|
|
||||||
err := os.Symlink("user@host.1234:1700000000", filepath.Join(dir, ".#"+firstFile))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("symlink: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
params, _ := newParams(dir)
|
|
||||||
|
|
||||||
files, err := rules.Load(params)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("load: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantMatched(t, files, get(t, "/probe"), "probe")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFaultStopsTheStartNamingTheFileAndLine(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, tc := range []struct {
|
|
||||||
name string
|
|
||||||
content string
|
|
||||||
line int
|
|
||||||
want string
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
"too few fields", "env-file path ban\n", 1,
|
|
||||||
"is not a rule: an id, a target, an action and a regex, " +
|
|
||||||
"separated by spaces or tabs",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
// Else its regex would be a space, found in nearly every user agent.
|
|
||||||
"a regex of only spaces and tabs", "scanner user_agent ban\t \n", 1,
|
|
||||||
"is not a rule: an id, a target, an action and a regex, " +
|
|
||||||
"separated by spaces or tabs",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"an id of other characters", "# ids\n\nenv.file path ban ^/\n", 3,
|
|
||||||
`the id "env.file" is not an id of letters, digits, - and _`,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"an unknown target", "env-file paths ban ^/\n", 1,
|
|
||||||
`the target "paths" is not path, query, uri, method, host, ` +
|
|
||||||
"user_agent, referer or header:<Name>",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"a header without a name", "env-file header: ban ^/\n", 1,
|
|
||||||
`the target "header:" is not path, query, uri, method, host, ` +
|
|
||||||
"user_agent, referer or header:<Name>",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"a header name written with its colon", "sqlmap header:User-Agent: ban sqlmap\n", 1,
|
|
||||||
`the target "header:User-Agent:" has a character after header: ` +
|
|
||||||
"that no header name can have",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"a header name with a semicolon", "accept header:Accept;q log ^$\n", 1,
|
|
||||||
`the target "header:Accept;q" has a character after header: ` +
|
|
||||||
"that no header name can have",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"a header name with brackets", "x-header header:X(y) log ^$\n", 1,
|
|
||||||
`the target "header:X(y)" has a character after header: ` +
|
|
||||||
"that no header name can have",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"the Host header", "host-header header:host block ^$\n", 1,
|
|
||||||
`the target "header:host" names a header that Go's HTTP server ` +
|
|
||||||
"takes out of every request, so a rule never sees it; " +
|
|
||||||
"the request's host is the target host",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"the Transfer-Encoding header",
|
|
||||||
"# bodies sent in chunks\nchunked header:Transfer-Encoding block ^chunked$\n", 2,
|
|
||||||
`the target "header:Transfer-Encoding" names a header that Go's ` +
|
|
||||||
"HTTP server takes out of every request, so a rule never sees it",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"an unknown action", "env-file path deny ^/\n", 1,
|
|
||||||
`the action "deny" is not log, block or ban`,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"a regex that does not compile", "env-file path ban ^/(\n", 1,
|
|
||||||
"the regex does not compile: error parsing regexp: " +
|
|
||||||
"missing closing ): `^/(`",
|
|
||||||
},
|
|
||||||
} {
|
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := writeFiles(t, ruleFiles{"00-default.rules": tc.content})
|
|
||||||
path := filepath.Join(dir, "00-default.rules")
|
|
||||||
|
|
||||||
wantRefused(t, dir, path+", line "+strconv.Itoa(tc.line)+": "+tc.want)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestIDUsedTwiceStopsTheStartNamingBothPlaces(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := writeFiles(t, ruleFiles{
|
|
||||||
"00-a.rules": "probe path log ^/a\n",
|
|
||||||
"50-b.rules": "other path log ^/b\nprobe path ban ^/c\n",
|
|
||||||
})
|
|
||||||
|
|
||||||
wantRefused(t, dir, filepath.Join(dir, "50-b.rules")+`, line 2: the id "probe" `+
|
|
||||||
"is already the id of the rule at "+filepath.Join(dir, "00-a.rules")+", line 1")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestDirectoryThatDoesNotExistStopsTheStart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := filepath.Join(t.TempDir(), "rules.d")
|
|
||||||
|
|
||||||
wantRefused(t, dir, "SWWAF_RULES_DIR cannot be read: open "+dir+
|
|
||||||
": no such file or directory")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestEmptyDirectoryLoadsNoRulesAndSaysSo(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
params, lines := newParams(writeFiles(t, ruleFiles{"00-default.rules": "# none\n"}))
|
|
||||||
|
|
||||||
files, err := rules.Load(params)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("load: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
line := lines.waitFor(t, read)
|
|
||||||
if files.Len() != 0 || line["rules"] != 0.0 {
|
|
||||||
t.Errorf("%d rules loaded, and the log says %v, want none", files.Len(), line)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRuleFilesOffReadNothing(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// SWWAF_RULES_DIR does not exist, which would stop the start.
|
|
||||||
params, _ := newParams(filepath.Join(t.TempDir(), "rules.d"))
|
|
||||||
params.Enabled = false
|
|
||||||
|
|
||||||
files, err := rules.Load(params)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("load: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if files.Len() != 0 || files.Match(get(t, "/")) != nil {
|
|
||||||
t.Errorf("%d rules loaded with the rule files off", files.Len())
|
|
||||||
}
|
|
||||||
|
|
||||||
// It would watch until the test ends.
|
|
||||||
files.Watch(t.Context())
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestEditsTakenInWhileRunning(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := writeFiles(t, ruleFiles{firstFile: "first path block ^/first\n"})
|
|
||||||
files, lines := watch(t, dir)
|
|
||||||
|
|
||||||
// matches reports whether path matches a rule.
|
|
||||||
matches := func(path string) bool { return len(files.Match(get(t, path))) == 1 }
|
|
||||||
|
|
||||||
// A file added.
|
|
||||||
save(t, dir, "50-b.rules", "second path block ^/second\n")
|
|
||||||
lines.waitUntil(t, func() bool { return matches("/second") })
|
|
||||||
wantMatched(t, files, get(t, "/first"), "first")
|
|
||||||
|
|
||||||
// A file edited.
|
|
||||||
save(t, dir, firstFile, "first path block ^/edited\n")
|
|
||||||
lines.waitUntil(t, func() bool { return !matches("/first") })
|
|
||||||
wantMatched(t, files, get(t, "/edited"), "first")
|
|
||||||
|
|
||||||
// A file removed.
|
|
||||||
err := os.Remove(filepath.Join(dir, "50-b.rules"))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("remove: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
lines.waitUntil(t, func() bool { return !matches("/second") })
|
|
||||||
wantMatched(t, files, get(t, "/edited"), "first")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestBrokenEditKeepsTheRulesAsTheyWere(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := writeFiles(t, ruleFiles{firstFile: "first path block ^/first\n"})
|
|
||||||
files, lines := watch(t, dir)
|
|
||||||
|
|
||||||
// The edit's second line has an unknown action, so the rules stay as
|
|
||||||
// they were, the first line's earlier version included.
|
|
||||||
save(t, dir, firstFile, "first path block ^/edited\nsecond path bann ^/second\n")
|
|
||||||
|
|
||||||
line := lines.waitFor(t, hasError)
|
|
||||||
want := filepath.Join(dir, firstFile) +
|
|
||||||
`, line 2: the action "bann" is not log, block or ban`
|
|
||||||
|
|
||||||
if line["error"] != want || line["level"] != "ERROR" {
|
|
||||||
t.Errorf("logged %v, want an error %q", line, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantMatched(t, files, get(t, "/first"), "first")
|
|
||||||
wantMatched(t, files, get(t, "/second"))
|
|
||||||
|
|
||||||
// Once mended, the file is read again.
|
|
||||||
save(t, dir, firstFile, "first path block ^/edited\nsecond path ban ^/second\n")
|
|
||||||
lines.waitUntil(t, func() bool { return len(files.Match(get(t, "/second"))) == 1 })
|
|
||||||
wantMatched(t, files, get(t, "/edited"), "first")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestDefaultFileBansProbesAtTheSiteRootAlone(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
params, _ := newParams(filepath.Join("..", "..", "share", "rules.d"))
|
|
||||||
|
|
||||||
files, err := rules.Load(params)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("load the default file: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Probes sent by a browser, by the rule that refuses them.
|
|
||||||
for rule, targets := range map[string][]string{
|
|
||||||
"env-file": {"/.env", "/.env.production", "/.ENV"},
|
|
||||||
"vcs-dir": {"/.git/config", "/.git", "/.svn/entries"},
|
|
||||||
"secrets-dir": {"/.aws/credentials", "/.ssh/id_rsa"},
|
|
||||||
"secret-file": {"/.htpasswd", "/.DS_Store", "/.git-credentials"},
|
|
||||||
"editor-dir": {"/.vscode/sftp.json"},
|
|
||||||
"backup-file": {
|
|
||||||
"/wp-config.php.bak", "/index.php~", "/dump.sql", "/backup.sql.gz",
|
|
||||||
},
|
|
||||||
"log-file": {"/debug.log"},
|
|
||||||
"compose-file": {"/docker-compose.yml", "/compose.yaml"},
|
|
||||||
"php-shell": {"/shell.php"},
|
|
||||||
"path-traversal": {
|
|
||||||
"/static/../../etc/passwd", "/f?f=%2e%2e%2f%2e%2e%2fetc%2fpasswd",
|
|
||||||
},
|
|
||||||
} {
|
|
||||||
for _, target := range targets {
|
|
||||||
wantRefusedBy(t, files, target, browser, rule)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Scanners, by their user agents.
|
|
||||||
for _, scanner := range []string{
|
|
||||||
"sqlmap/1.8.4#stable (https://sqlmap.org)",
|
|
||||||
"Mozilla/5.0 (compatible; Nuclei - Open-source project)",
|
|
||||||
} {
|
|
||||||
wantRefusedBy(t, files, "/", scanner, "scanner-agent")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Ordinary requests to a code forge for files of those names deeper
|
|
||||||
// in its paths, and for other files at its root.
|
|
||||||
for _, target := range []string{
|
|
||||||
"/owner/repo/src/branch/main/.env.example",
|
|
||||||
"/owner/repo/src/branch/main/.env",
|
|
||||||
"/owner/repo/src/branch/main/.github/workflows/ci.yml",
|
|
||||||
"/owner/repo/src/branch/main/.vscode/settings.json",
|
|
||||||
"/owner/repo/src/branch/main/.htaccess",
|
|
||||||
"/owner/repo/src/branch/main/docker-compose.yml",
|
|
||||||
"/owner/repo/src/branch/main/db/schema.sql",
|
|
||||||
"/owner/repo/raw/branch/main/debug.log",
|
|
||||||
"/owner/repo.git/info/refs?service=git-upload-pack",
|
|
||||||
"/owner/repo/src/branch/main/docs/../README.md",
|
|
||||||
"/user/login?redirect_to=%2fowner%2frepo",
|
|
||||||
"/index.php",
|
|
||||||
"/.well-known/security.txt",
|
|
||||||
} {
|
|
||||||
r := get(t, target)
|
|
||||||
r.Header.Set(userAgent, browser)
|
|
||||||
|
|
||||||
matched := files.Match(r)
|
|
||||||
if len(matched) != 0 {
|
|
||||||
t.Errorf("%s matched %v, want no rule", target, ids(matched))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// A request without a user agent is only noted.
|
|
||||||
wantMatched(t, files, get(t, "/"), "empty-agent")
|
|
||||||
}
|
|
||||||
|
|
||||||
// ruleFiles are files to write into a directory of rule files, by name.
|
|
||||||
type ruleFiles map[string]string
|
|
||||||
|
|
||||||
// writeFiles writes files into a new directory, and returns it.
|
|
||||||
func writeFiles(t *testing.T, files ruleFiles) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
|
|
||||||
for name, content := range files {
|
|
||||||
path := filepath.Join(dir, name)
|
|
||||||
|
|
||||||
err := os.MkdirAll(filepath.Dir(path), 0o700)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("mkdir: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = os.WriteFile(path, []byte(content), 0o600)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write %s: %v", name, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return dir
|
|
||||||
}
|
|
||||||
|
|
||||||
// save writes content to the rule file name in dir as an editor that
|
|
||||||
// saves by renaming does, so that the file is never seen half written.
|
|
||||||
func save(t *testing.T, dir, name, content string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
path := filepath.Join(dir, name)
|
|
||||||
|
|
||||||
err := os.WriteFile(path+".tmp", []byte(content), 0o600)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write %s: %v", name, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = os.Rename(path+".tmp", path)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("rename: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// newParams returns Params for the rule files in dir, switched on, with
|
|
||||||
// the process log in the processLog returned.
|
|
||||||
func newParams(dir string) (rules.Params, processLog) {
|
|
||||||
lines := make(processLog, maxLogLines)
|
|
||||||
|
|
||||||
return rules.Params{
|
|
||||||
Dir: dir,
|
|
||||||
Enabled: true,
|
|
||||||
ProcessLog: slog.New(slog.NewJSONHandler(lines, nil)),
|
|
||||||
}, lines
|
|
||||||
}
|
|
||||||
|
|
||||||
// load writes files into a new directory and loads the rules in it.
|
|
||||||
func load(t *testing.T, files ruleFiles) *rules.Files {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
params, _ := newParams(writeFiles(t, files))
|
|
||||||
params.ProcessLog = slog.New(slog.DiscardHandler)
|
|
||||||
|
|
||||||
loaded, err := rules.Load(params)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("load: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return loaded
|
|
||||||
}
|
|
||||||
|
|
||||||
// watch loads the rules in dir, runs their Watch until the test ends, and
|
|
||||||
// waits until it watches the directory.
|
|
||||||
func watch(t *testing.T, dir string) (*rules.Files, processLog) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
params, lines := newParams(dir)
|
|
||||||
|
|
||||||
files, err := rules.Load(params)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("load: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, stop := context.WithCancel(t.Context())
|
|
||||||
stopped := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
files.Watch(ctx)
|
|
||||||
close(stopped)
|
|
||||||
}()
|
|
||||||
|
|
||||||
t.Cleanup(func() {
|
|
||||||
stop()
|
|
||||||
<-stopped
|
|
||||||
})
|
|
||||||
|
|
||||||
lines.waitFor(t, watching)
|
|
||||||
|
|
||||||
return files, lines
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantRefused checks that loading the rule files in dir fails with the
|
|
||||||
// error want.
|
|
||||||
func wantRefused(t *testing.T, dir, want string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
params, _ := newParams(dir)
|
|
||||||
|
|
||||||
_, err := rules.Load(params)
|
|
||||||
if err == nil || err.Error() != want {
|
|
||||||
t.Errorf("error %v, want %s", err, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// get returns a GET request for target, a path and an optional query, as
|
|
||||||
// smallwebwaf's server reads it, without a user agent.
|
|
||||||
func get(t *testing.T, target string) *http.Request {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
return httptest.NewRequestWithContext(t.Context(), http.MethodGet,
|
|
||||||
"http://app.example"+target, nil)
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantRefusedBy checks that a GET request for target with the user agent
|
|
||||||
// sent matches rule alone, and that rule refuses it.
|
|
||||||
func wantRefusedBy(t *testing.T, files *rules.Files, target, sent, rule string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
r := get(t, target)
|
|
||||||
r.Header.Set(userAgent, sent)
|
|
||||||
|
|
||||||
matched := files.Match(r)
|
|
||||||
if len(matched) != 1 || matched[0].ID != rule || matched[0].Action == rules.ActionLog {
|
|
||||||
t.Errorf("%s from %q matched %v, want %s alone, refusing it", target,
|
|
||||||
sent, ids(matched), rule)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantMatched checks the ids of the rules r matches, in order.
|
|
||||||
func wantMatched(t *testing.T, files *rules.Files, r *http.Request, want ...string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
got := ids(files.Match(r))
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("%s matched %v, want %v", r.URL, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ids returns the ids of matched.
|
|
||||||
func ids(matched []rules.Rule) []string {
|
|
||||||
got := make([]string, 0, len(matched))
|
|
||||||
for _, rule := range matched {
|
|
||||||
got = append(got, rule.ID)
|
|
||||||
}
|
|
||||||
|
|
||||||
return got
|
|
||||||
}
|
|
||||||
|
|
||||||
// processLog receives the lines of a process log, each a JSON object, for
|
|
||||||
// a test to wait for.
|
|
||||||
type processLog chan string
|
|
||||||
|
|
||||||
// Write receives a line of the process log.
|
|
||||||
func (l processLog) Write(line []byte) (int, error) {
|
|
||||||
l <- string(line)
|
|
||||||
|
|
||||||
return len(line), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// waitFor returns the next line of the process log whose message is msg,
|
|
||||||
// passing over the lines before it. It waits as long as that takes, so
|
|
||||||
// that a slow test process cannot fail the test.
|
|
||||||
func (l processLog) waitFor(t *testing.T, msg string) map[string]any {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for line := range l {
|
|
||||||
var fields map[string]any
|
|
||||||
|
|
||||||
err := json.Unmarshal([]byte(line), &fields)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("process log line %q is not JSON: %v", line, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if fields["msg"] == msg {
|
|
||||||
return fields
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// waitUntil waits for the rule files to be read until done reports true,
|
|
||||||
// as it does once they have been read after the test's last change. They
|
|
||||||
// can be read before then too, as they are once Watch starts watching.
|
|
||||||
func (l processLog) waitUntil(t *testing.T, done func() bool) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for !done() {
|
|
||||||
l.waitFor(t, read)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,162 +0,0 @@
|
|||||||
package rules
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"slices"
|
|
||||||
"testing"
|
|
||||||
"testing/synctest"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/fsnotify/fsnotify"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The tests below run readAfterChanges in a synctest bubble, where time is
|
|
||||||
// a clock of the test's own: time.Sleep moves it on at once, and
|
|
||||||
// synctest.Wait returns once readAfterChanges waits again, so that every
|
|
||||||
// reading due by then is done. The test sends the changes itself, as the
|
|
||||||
// watch of a directory cannot run in a bubble.
|
|
||||||
|
|
||||||
func TestFileWrittenInTwoPartsTakenInOnlyWhole(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
path := filepath.Join(dir, "50-app.rules")
|
|
||||||
writeFile(t, path, "first path block ^/first\n")
|
|
||||||
files := load(t, dir)
|
|
||||||
changes := run(t, files)
|
|
||||||
|
|
||||||
file, err := os.Create(path) //nolint:gosec // a file the test wrote
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("create: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() {
|
|
||||||
_ = file.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
// The first part ends in the middle of a ban rule's regex, which,
|
|
||||||
// read then, would ban every request.
|
|
||||||
write(t, file, "first path block ^/first\nprobe path ban ^/")
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
time.Sleep(quietTime - time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantMatched(t, files, "/anything")
|
|
||||||
|
|
||||||
// The second part starts the wait again.
|
|
||||||
write(t, file, `\.env$`+"\n")
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
time.Sleep(quietTime - time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantMatched(t, files, "/.env")
|
|
||||||
|
|
||||||
time.Sleep(time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantMatched(t, files, "/.env", "probe")
|
|
||||||
wantMatched(t, files, "/anything")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestEditSavedBeforeTheWatchStartsTakenIn(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
path := filepath.Join(dir, "50-app.rules")
|
|
||||||
writeFile(t, path, "first path block ^/first\n")
|
|
||||||
files := load(t, dir)
|
|
||||||
|
|
||||||
// Saved after Load read the files, and before the directory was
|
|
||||||
// watched, so that no change is seen for it.
|
|
||||||
writeFile(t, path, "first path block ^/edited\n")
|
|
||||||
run(t, files)
|
|
||||||
time.Sleep(quietTime)
|
|
||||||
synctest.Wait()
|
|
||||||
wantMatched(t, files, "/edited", "first")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// load loads the rules in dir.
|
|
||||||
func load(t *testing.T, dir string) *Files {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
files, err := Load(Params{
|
|
||||||
Dir: dir, Enabled: true, ProcessLog: slog.New(slog.DiscardHandler),
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("load: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return files
|
|
||||||
}
|
|
||||||
|
|
||||||
// run runs files' readAfterChanges until the test ends, and returns the
|
|
||||||
// channel that sends it changes.
|
|
||||||
func run(t *testing.T, files *Files) chan<- fsnotify.Event {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
changes := make(chan fsnotify.Event)
|
|
||||||
ctx, stop := context.WithCancel(t.Context())
|
|
||||||
stopped := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
files.readAfterChanges(ctx, changes, nil)
|
|
||||||
close(stopped)
|
|
||||||
}()
|
|
||||||
|
|
||||||
t.Cleanup(func() {
|
|
||||||
stop()
|
|
||||||
<-stopped
|
|
||||||
})
|
|
||||||
|
|
||||||
return changes
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeFile writes content to the file at path.
|
|
||||||
func writeFile(t *testing.T, path, content string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
err := os.WriteFile(path, []byte(content), 0o600)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write %s: %v", path, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// write writes text to the end of file.
|
|
||||||
func write(t *testing.T, file *os.File, text string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
_, err := file.WriteString(text)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantMatched checks the ids of the rules that a GET request for path
|
|
||||||
// matches, in order.
|
|
||||||
func wantMatched(t *testing.T, files *Files, path string, want ...string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
r := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
|
|
||||||
"http://app.example"+path, nil)
|
|
||||||
|
|
||||||
matched := files.Match(r)
|
|
||||||
|
|
||||||
got := make([]string, 0, len(matched))
|
|
||||||
for _, rule := range matched {
|
|
||||||
got = append(got, rule.ID)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("%s matched %v, want %v", path, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -23,8 +23,6 @@ var errHealthEndpoint = errors.New("smallwebwaf's health endpoint answered")
|
|||||||
// smallwebwaf answers its health endpoint on 127.0.0.1, at the port in
|
// smallwebwaf answers its health endpoint on 127.0.0.1, at the port in
|
||||||
// SWWAF_LISTEN_ADDR, and the app accepts connections at the address in
|
// SWWAF_LISTEN_ADDR, and the app accepts connections at the address in
|
||||||
// SWWAF_UPSTREAM_URL. Otherwise it writes why to stderr and returns 1.
|
// SWWAF_UPSTREAM_URL. Otherwise it writes why to stderr and returns 1.
|
||||||
// It reads no other setting, nor a file that another names, so neither
|
|
||||||
// can fail it.
|
|
||||||
// args are the arguments after `healthcheck`; it takes none, and given
|
// args are the arguments after `healthcheck`; it takes none, and given
|
||||||
// one it names it on stderr and returns 1 without checking anything.
|
// one it names it on stderr and returns 1 without checking anything.
|
||||||
func HealthCheck(
|
func HealthCheck(
|
||||||
@@ -52,13 +50,13 @@ func healthCheck(ctx context.Context, lookupEnv func(string) (string, bool)) err
|
|||||||
ctx, cancel := context.WithTimeout(ctx, healthCheckTimeout)
|
ctx, cancel := context.WithTimeout(ctx, healthCheckTimeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
listenAddr, upstreamURL, err := config.ListenAddrAndUpstreamURL(lookupEnv)
|
cfg, err := config.FromEnvironment(lookupEnv)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("invalid setting: %w", err)
|
return fmt.Errorf("invalid setting: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The settings have checked that the address has a port.
|
// The settings have checked that the address has a port.
|
||||||
_, port, _ := net.SplitHostPort(listenAddr)
|
_, port, _ := net.SplitHostPort(cfg.ListenAddr)
|
||||||
health := "http://" + net.JoinHostPort("127.0.0.1", port) + proxy.HealthPath
|
health := "http://" + net.JoinHostPort("127.0.0.1", port) + proxy.HealthPath
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, health, http.NoBody)
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, health, http.NoBody)
|
||||||
@@ -77,7 +75,7 @@ func healthCheck(ctx context.Context, lookupEnv func(string) (string, bool)) err
|
|||||||
return fmt.Errorf("%w %s", errHealthEndpoint, res.Status)
|
return fmt.Errorf("%w %s", errHealthEndpoint, res.Status)
|
||||||
}
|
}
|
||||||
|
|
||||||
conn, err := (&net.Dialer{}).DialContext(ctx, "tcp", appAddress(upstreamURL))
|
conn, err := (&net.Dialer{}).DialContext(ctx, "tcp", appAddress(cfg.UpstreamURL))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("connect to the app: %w", err)
|
return fmt.Errorf("connect to the app: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -6,8 +6,6 @@ import (
|
|||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
@@ -30,7 +28,6 @@ func TestHealthCheck(t *testing.T) {
|
|||||||
listenAddr: localhost + ":0",
|
listenAddr: localhost + ":0",
|
||||||
upstreamURL: app.URL,
|
upstreamURL: app.URL,
|
||||||
stateDir: t.TempDir(),
|
stateDir: t.TempDir(),
|
||||||
rulesDir: t.TempDir(),
|
|
||||||
}
|
}
|
||||||
|
|
||||||
go func() {
|
go func() {
|
||||||
@@ -45,21 +42,6 @@ func TestHealthCheck(t *testing.T) {
|
|||||||
|
|
||||||
wantHealthCheck(t, env, 0, "")
|
wantHealthCheck(t, env, 0, "")
|
||||||
|
|
||||||
// The health check reads those two settings alone, here given as
|
|
||||||
// files: a removed or invalid token file, or an invalid value of
|
|
||||||
// another setting, does not fail it.
|
|
||||||
for _, other := range []struct{ name, value string }{
|
|
||||||
{"SWWAF_METRICS_TOKEN_FILE", filepath.Join(t.TempDir(), "removed")},
|
|
||||||
{"SWWAF_METRICS_TOKEN_FILE", writeFile(t, "too short\n")},
|
|
||||||
{"SWWAF_MODE", "neither"},
|
|
||||||
} {
|
|
||||||
wantHealthCheck(t, map[string]string{
|
|
||||||
listenAddr + "_FILE": writeFile(t, ":"+port+"\n"),
|
|
||||||
upstreamURL + "_FILE": writeFile(t, app.URL+"\n"),
|
|
||||||
other.name: other.value,
|
|
||||||
}, 0, "")
|
|
||||||
}
|
|
||||||
|
|
||||||
app.Close()
|
app.Close()
|
||||||
wantHealthCheck(t, env, 1, "unhealthy: connect to the app: ")
|
wantHealthCheck(t, env, 1, "unhealthy: connect to the app: ")
|
||||||
|
|
||||||
@@ -115,18 +97,3 @@ func wantHealthCheck(t *testing.T, env map[string]string, status int, message st
|
|||||||
got, wrote, status, message)
|
got, wrote, status, message)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// writeFile writes contents to a file in a directory of its own, removed
|
|
||||||
// when the test ends, and returns the file's path.
|
|
||||||
func writeFile(t *testing.T, contents string) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
path := filepath.Join(t.TempDir(), "setting")
|
|
||||||
|
|
||||||
err := os.WriteFile(path, []byte(contents), 0o600)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write %s: %v", path, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return path
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
// Package smallwebwaf runs the smallwebwaf process: it reads the settings,
|
// Package smallwebwaf runs the smallwebwaf process: it reads the settings
|
||||||
// the rule files and the state files, serves requests until it is told to
|
// and the state files, serves requests until it is told to stop, and then
|
||||||
// stop, and then stops in an orderly way, writing the state files.
|
// stops in an orderly way, writing the state files.
|
||||||
package smallwebwaf
|
package smallwebwaf
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -18,9 +18,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/state"
|
"sneak.berlin/go/smallwebwaf/internal/state"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -29,11 +27,6 @@ import (
|
|||||||
// runit and docker wait a little longer before they kill the process.
|
// runit and docker wait a little longer before they kill the process.
|
||||||
const shutdownTimeout = 5 * time.Second
|
const shutdownTimeout = 5 * time.Second
|
||||||
|
|
||||||
// remoteLogStopTimeout is how long, as smallwebwaf stops, the log lines
|
|
||||||
// still waiting are sent to SWWAF_LOG_REMOTE_URL before they are given
|
|
||||||
// up. stdout has carried them.
|
|
||||||
const remoteLogStopTimeout = 2 * time.Second
|
|
||||||
|
|
||||||
// Params are what Run needs from the process.
|
// Params are what Run needs from the process.
|
||||||
type Params struct {
|
type Params struct {
|
||||||
// Version is the version of the binary, set when it is built.
|
// Version is the version of the binary, set when it is built.
|
||||||
@@ -63,9 +56,9 @@ func Main(version string) int {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// Run reads the settings, the rule files and the state files, then serves
|
// Run reads the settings and the state files, then serves requests until
|
||||||
// requests until ctx is done. It returns the process's exit status, 1
|
// ctx is done. It returns the process's exit status, 1 when smallwebwaf
|
||||||
// when smallwebwaf cannot start.
|
// cannot start.
|
||||||
func Run(ctx context.Context, params Params) int {
|
func Run(ctx context.Context, params Params) int {
|
||||||
processLog := requestlog.NewProcessLogger(params.Stdout)
|
processLog := requestlog.NewProcessLogger(params.Stdout)
|
||||||
|
|
||||||
@@ -76,46 +69,16 @@ func Run(ctx context.Context, params Params) int {
|
|||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
|
|
||||||
// While SWWAF_LOG_REMOTE_URL is set, every line on stdout from here on
|
|
||||||
// is sent there too.
|
|
||||||
stdout := params.Stdout
|
|
||||||
|
|
||||||
var remote *remotelog.Sender
|
|
||||||
|
|
||||||
if cfg.LogRemoteURL != nil {
|
|
||||||
remote = newRemoteLogSender(cfg)
|
|
||||||
stdout = io.MultiWriter(params.Stdout, remote)
|
|
||||||
processLog = requestlog.NewProcessLogger(stdout)
|
|
||||||
|
|
||||||
stopSending := startSending(ctx, remote, processLog)
|
|
||||||
defer stopSending()
|
|
||||||
}
|
|
||||||
|
|
||||||
ruleFiles, err := rules.Load(rules.Params{
|
|
||||||
Dir: cfg.RulesDir,
|
|
||||||
Enabled: cfg.RulesEnabled,
|
|
||||||
ProcessLog: processLog,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
processLog.Error("cannot use the rule files", "error", err.Error())
|
|
||||||
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
// The state files give times in UTC.
|
// The state files give times in UTC.
|
||||||
now := func() time.Time { return time.Now().UTC() }
|
now := func() time.Time { return time.Now().UTC() }
|
||||||
|
|
||||||
server := proxy.New(proxy.Params{
|
server := proxy.New(proxy.Params{
|
||||||
Config: cfg,
|
Config: cfg,
|
||||||
RequestLog: stdout,
|
RequestLog: params.Stdout,
|
||||||
ProcessLog: processLog,
|
ProcessLog: processLog,
|
||||||
GeoJSURL: lookup.URL,
|
GeoJSURL: lookup.URL,
|
||||||
Now: now,
|
Now: now,
|
||||||
Rules: ruleFiles,
|
|
||||||
})
|
})
|
||||||
if remote != nil {
|
|
||||||
server.Metrics.AddRemoteLog(remote)
|
|
||||||
}
|
|
||||||
|
|
||||||
files, err := state.Load(state.Params{
|
files, err := state.Load(state.Params{
|
||||||
Dir: cfg.StateDir,
|
Dir: cfg.StateDir,
|
||||||
@@ -147,53 +110,16 @@ func Run(ctx context.Context, params Params) int {
|
|||||||
"address", listener.Addr().String(),
|
"address", listener.Addr().String(),
|
||||||
"settings", cfg)
|
"settings", cfg)
|
||||||
|
|
||||||
return serve(ctx, server.Server, listener, files, ruleFiles, processLog)
|
return serve(ctx, server.Server, listener, files, processLog)
|
||||||
}
|
|
||||||
|
|
||||||
// newRemoteLogSender returns a sender of the log lines to
|
|
||||||
// SWWAF_LOG_REMOTE_URL, with the settings for it.
|
|
||||||
func newRemoteLogSender(cfg *config.Config) *remotelog.Sender {
|
|
||||||
return remotelog.New(remotelog.Params{
|
|
||||||
URL: cfg.LogRemoteURL,
|
|
||||||
RootCAs: cfg.LogRemoteTLSCAs,
|
|
||||||
Buffer: cfg.LogRemoteBuffer,
|
|
||||||
Facility: cfg.LogRemoteFacility,
|
|
||||||
AppName: cfg.LogRemoteAppName,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// startSending runs remote until the function it returns is called, which
|
|
||||||
// then waits at most remoteLogStopTimeout for the lines still waiting to
|
|
||||||
// be sent. Sending goes on after ctx is done, so that the lines written
|
|
||||||
// while smallwebwaf stops are sent too.
|
|
||||||
func startSending(
|
|
||||||
ctx context.Context, remote *remotelog.Sender, processLog *slog.Logger,
|
|
||||||
) func() {
|
|
||||||
sending, stop := context.WithCancel(context.WithoutCancel(ctx))
|
|
||||||
sent := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
remote.Run(sending, processLog)
|
|
||||||
close(sent)
|
|
||||||
}()
|
|
||||||
|
|
||||||
return func() {
|
|
||||||
stop()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-sent:
|
|
||||||
case <-time.After(remoteLogStopTimeout):
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// serve serves requests on listener, writes the state files as they are
|
// serve serves requests on listener, writes the state files as they are
|
||||||
// due, takes in an admin's edits of them, and reads the rule files again
|
// due, and takes in an admin's edits of them, until ctx is done. Then it
|
||||||
// as they change, until ctx is done. Then it gives the requests in
|
// gives the requests in progress shutdownTimeout to finish, and writes
|
||||||
// progress shutdownTimeout to finish, and writes every state file.
|
// every state file.
|
||||||
func serve(
|
func serve(
|
||||||
ctx context.Context, server *http.Server, listener net.Listener,
|
ctx context.Context, server *http.Server, listener net.Listener,
|
||||||
files *state.Files, ruleFiles *rules.Files, processLog *slog.Logger,
|
files *state.Files, processLog *slog.Logger,
|
||||||
) int {
|
) int {
|
||||||
served := make(chan error, 1)
|
served := make(chan error, 1)
|
||||||
|
|
||||||
@@ -206,7 +132,6 @@ func serve(
|
|||||||
|
|
||||||
written := make(chan struct{})
|
written := make(chan struct{})
|
||||||
watched := make(chan struct{})
|
watched := make(chan struct{})
|
||||||
rulesWatched := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
go func() {
|
||||||
files.Run(writing)
|
files.Run(writing)
|
||||||
@@ -218,11 +143,6 @@ func serve(
|
|||||||
close(watched)
|
close(watched)
|
||||||
}()
|
}()
|
||||||
|
|
||||||
go func() {
|
|
||||||
ruleFiles.Watch(writing)
|
|
||||||
close(rulesWatched)
|
|
||||||
}()
|
|
||||||
|
|
||||||
select {
|
select {
|
||||||
case err := <-served:
|
case err := <-served:
|
||||||
processLog.Error("serving failed", "error", err.Error())
|
processLog.Error("serving failed", "error", err.Error())
|
||||||
@@ -261,7 +181,6 @@ func serve(
|
|||||||
// missing from clients.json.
|
// missing from clients.json.
|
||||||
<-written
|
<-written
|
||||||
<-watched
|
<-watched
|
||||||
<-rulesWatched
|
|
||||||
|
|
||||||
err = files.WriteAll()
|
err = files.WriteAll()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -1,82 +0,0 @@
|
|||||||
package smallwebwaf
|
|
||||||
|
|
||||||
import (
|
|
||||||
"log/slog"
|
|
||||||
"net/url"
|
|
||||||
"testing"
|
|
||||||
"testing/synctest"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The stop's tests run in a synctest bubble, where the time package runs
|
|
||||||
// on a clock of the test's own, so that how long the stop takes can be
|
|
||||||
// told exactly. The sender is held up by its process log, not by the
|
|
||||||
// network: a goroutine of the bubble that waits on the network keeps that
|
|
||||||
// clock from moving on.
|
|
||||||
|
|
||||||
func TestStopWaitsForTheSenderToFinish(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
took := stopHeldSender(t, time.Second)
|
|
||||||
if took != time.Second {
|
|
||||||
t.Errorf("the stop took %s, want the second the sender took", took)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestStopWaitsForTheSenderAtMostTwoSeconds(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
took := stopHeldSender(t, time.Minute)
|
|
||||||
if took != 2*time.Second {
|
|
||||||
t.Errorf("the stop took %s, want 2s", took)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// heldLog holds each line written to it until it is closed.
|
|
||||||
type heldLog chan struct{}
|
|
||||||
|
|
||||||
// Write waits until the log is closed.
|
|
||||||
func (l heldLog) Write(p []byte) (int, error) {
|
|
||||||
<-l
|
|
||||||
|
|
||||||
return len(p), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// stopHeldSender starts sending to an endpoint the sender cannot connect
|
|
||||||
// to, holds the sender as it logs that failure until release has passed,
|
|
||||||
// stops the sending, and returns how long the stop took. It returns once
|
|
||||||
// the sender has ended, as a bubble must.
|
|
||||||
func stopHeldSender(t *testing.T, release time.Duration) time.Duration {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
log := make(heldLog)
|
|
||||||
sender := remotelog.New(remotelog.Params{
|
|
||||||
// No port is 65536, so each attempt to connect fails at once,
|
|
||||||
// before it reaches the network.
|
|
||||||
URL: &url.URL{Scheme: remotelog.SchemeTCP, Host: "127.0.0.1:65536"},
|
|
||||||
Buffer: 1,
|
|
||||||
})
|
|
||||||
|
|
||||||
stopSending := startSending(t.Context(), sender,
|
|
||||||
slog.New(slog.NewJSONHandler(log, nil)))
|
|
||||||
|
|
||||||
synctest.Wait()
|
|
||||||
time.AfterFunc(release, func() { close(log) })
|
|
||||||
|
|
||||||
stopped := time.Now()
|
|
||||||
|
|
||||||
stopSending()
|
|
||||||
|
|
||||||
took := time.Since(stopped)
|
|
||||||
|
|
||||||
time.Sleep(release)
|
|
||||||
synctest.Wait()
|
|
||||||
|
|
||||||
return took
|
|
||||||
}
|
|
||||||
@@ -10,8 +10,6 @@ import (
|
|||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"slices"
|
|
||||||
"strconv"
|
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -36,7 +34,6 @@ const (
|
|||||||
stateWriteDelay = "SWWAF_STATE_WRITE_DELAY"
|
stateWriteDelay = "SWWAF_STATE_WRITE_DELAY"
|
||||||
stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL"
|
stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL"
|
||||||
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
|
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
|
||||||
rulesDir = "SWWAF_RULES_DIR"
|
|
||||||
// greeting is what the tests' app answers.
|
// greeting is what the tests' app answers.
|
||||||
greeting = "hello from the app"
|
greeting = "hello from the app"
|
||||||
)
|
)
|
||||||
@@ -166,7 +163,6 @@ func TestAddressInUseStopsTheStart(t *testing.T) {
|
|||||||
status := run(t.Context(), map[string]string{
|
status := run(t.Context(), map[string]string{
|
||||||
listenAddr: taken.Addr().String(),
|
listenAddr: taken.Addr().String(),
|
||||||
stateDir: t.TempDir(),
|
stateDir: t.TempDir(),
|
||||||
rulesDir: t.TempDir(),
|
|
||||||
}, out)
|
}, out)
|
||||||
if status != 1 {
|
if status != 1 {
|
||||||
t.Errorf("exit status %d, want 1", status)
|
t.Errorf("exit status %d, want 1", status)
|
||||||
@@ -190,15 +186,9 @@ func TestServesUntilToldToStop(t *testing.T) {
|
|||||||
listenAddr: localhost + ":0",
|
listenAddr: localhost + ":0",
|
||||||
upstreamURL: appURL,
|
upstreamURL: appURL,
|
||||||
stateDir: dir,
|
stateDir: dir,
|
||||||
rulesDir: filepath.Join("..", "..", "share", "rules.d"),
|
|
||||||
}, out)
|
}, out)
|
||||||
}()
|
}()
|
||||||
|
|
||||||
// The default rule file is read.
|
|
||||||
if rules := out.line(t, "msg", "read the rule files")["rules"]; rules != 12.0 {
|
|
||||||
t.Errorf("read %v rules from the default rule file, want 12", rules)
|
|
||||||
}
|
|
||||||
|
|
||||||
starting := out.line(t, "msg", "starting")
|
starting := out.line(t, "msg", "starting")
|
||||||
wantStartingLine(t, starting, appURL, dir)
|
wantStartingLine(t, starting, appURL, dir)
|
||||||
|
|
||||||
@@ -227,7 +217,6 @@ func TestStateKeptAcrossRestarts(t *testing.T) {
|
|||||||
listenAddr: localhost + ":0",
|
listenAddr: localhost + ":0",
|
||||||
upstreamURL: startApp(t),
|
upstreamURL: startApp(t),
|
||||||
stateDir: t.TempDir(),
|
stateDir: t.TempDir(),
|
||||||
rulesDir: t.TempDir(),
|
|
||||||
rateLimitPerDay: "2",
|
rateLimitPerDay: "2",
|
||||||
// Neither comes due in the test: the files are written as
|
// Neither comes due in the test: the files are written as
|
||||||
// smallwebwaf stops.
|
// smallwebwaf stops.
|
||||||
@@ -264,7 +253,6 @@ func TestBanRefusesItsNetblockAfterARestartWithAnotherScope(t *testing.T) {
|
|||||||
listenAddr: localhost + ":0",
|
listenAddr: localhost + ":0",
|
||||||
upstreamURL: startApp(t),
|
upstreamURL: startApp(t),
|
||||||
stateDir: t.TempDir(),
|
stateDir: t.TempDir(),
|
||||||
rulesDir: t.TempDir(),
|
|
||||||
trustedProxies: localhost + "/32",
|
trustedProxies: localhost + "/32",
|
||||||
rateLimitPerDay: "1",
|
rateLimitPerDay: "1",
|
||||||
scope: "24",
|
scope: "24",
|
||||||
@@ -312,7 +300,6 @@ func TestBanAddedAndLiftedByEditingBansJSON(t *testing.T) {
|
|||||||
listenAddr: localhost + ":0",
|
listenAddr: localhost + ":0",
|
||||||
upstreamURL: startApp(t),
|
upstreamURL: startApp(t),
|
||||||
stateDir: dir,
|
stateDir: dir,
|
||||||
rulesDir: t.TempDir(),
|
|
||||||
trustedProxies: localhost + "/32",
|
trustedProxies: localhost + "/32",
|
||||||
// No write comes due in the test, so only the watch on the
|
// No write comes due in the test, so only the watch on the
|
||||||
// directory can take the edits in.
|
// directory can take the edits in.
|
||||||
@@ -329,170 +316,6 @@ func TestBanAddedAndLiftedByEditingBansJSON(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRuleFileAddedWhileRunningTakesEffect(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
env := map[string]string{
|
|
||||||
listenAddr: localhost + ":0",
|
|
||||||
upstreamURL: startApp(t),
|
|
||||||
stateDir: t.TempDir(),
|
|
||||||
rulesDir: dir,
|
|
||||||
// The requests sent until the rule takes effect must not break a
|
|
||||||
// rate limit, whose ban would refuse them too.
|
|
||||||
"SWWAF_RATE_LIMIT_PER_MINUTE": "off",
|
|
||||||
}
|
|
||||||
|
|
||||||
out := runUntilStopped(t, env, func(url string) {
|
|
||||||
wantGreeting(t, url)
|
|
||||||
|
|
||||||
// Written once: each change would start the rule files' wait
|
|
||||||
// again. A file written before smallwebwaf watches the directory is
|
|
||||||
// read once it does.
|
|
||||||
err := os.WriteFile(filepath.Join(dir, "50-app.rules"),
|
|
||||||
[]byte("everything path block ^/\n"), 0o600)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write the rule file: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// As long as that takes, so that a slow test process cannot fail
|
|
||||||
// the test.
|
|
||||||
for statusFrom(t, url, "203.0.113.9") != http.StatusForbidden {
|
|
||||||
time.Sleep(pollInterval)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
out.line(t, "action", "rule_blocked")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRuleFileThatDoesNotParseStopsTheStart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
path := filepath.Join(dir, "00-default.rules")
|
|
||||||
|
|
||||||
err := os.WriteFile(path, []byte("# probes\nenv-file path bann ^/\\.env$\n"), 0o600)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write the rule file: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantRulesRefused(t, dir, path+`, line 2: the action "bann" is not log, block or ban`)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRulesDirThatDoesNotExistStopsTheStart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := filepath.Join(t.TempDir(), "rules.d")
|
|
||||||
|
|
||||||
wantRulesRefused(t, dir, "SWWAF_RULES_DIR cannot be read: open "+dir+
|
|
||||||
": no such file or directory")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestEveryLineIsAlsoSentToTheRemoteLogEndpoint(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
endpoint, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("listen: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() {
|
|
||||||
_ = endpoint.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
env := map[string]string{
|
|
||||||
listenAddr: localhost + ":0",
|
|
||||||
upstreamURL: startApp(t),
|
|
||||||
stateDir: t.TempDir(),
|
|
||||||
rulesDir: t.TempDir(),
|
|
||||||
"SWWAF_LOG_REMOTE_URL": "syslog+tcp://" + endpoint.Addr().String(),
|
|
||||||
}
|
|
||||||
|
|
||||||
out := runUntilStopped(t, env, func(url string) {
|
|
||||||
wantGreeting(t, url)
|
|
||||||
})
|
|
||||||
out.line(t, "type", "request")
|
|
||||||
|
|
||||||
// smallwebwaf connected as it started, and closes the connection once
|
|
||||||
// it has sent the lines written as it stopped.
|
|
||||||
conn, err := endpoint.Accept()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("accept: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
received, err := io.ReadAll(conn)
|
|
||||||
_ = conn.Close()
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("read: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Lines written at once by several goroutines may reach stdout and
|
|
||||||
// the endpoint in different orders.
|
|
||||||
sent := messages(t, string(received))
|
|
||||||
written := slices.Collect(strings.Lines(out.text()))
|
|
||||||
|
|
||||||
slices.Sort(sent)
|
|
||||||
slices.Sort(written)
|
|
||||||
|
|
||||||
if !slices.Equal(sent, written) {
|
|
||||||
t.Errorf("sent\n%v\nwrote\n%v", sent, written)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const token = "0123456789abcdef0123456789abcdef"
|
|
||||||
|
|
||||||
// The endpoint takes connections and never answers, so the TLS
|
|
||||||
// handshake of each waits on it, and no line is ever sent.
|
|
||||||
endpoint, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("listen: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() {
|
|
||||||
_ = endpoint.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
env := map[string]string{
|
|
||||||
listenAddr: localhost + ":0",
|
|
||||||
upstreamURL: startApp(t),
|
|
||||||
stateDir: t.TempDir(),
|
|
||||||
rulesDir: t.TempDir(),
|
|
||||||
"SWWAF_LOG_REMOTE_URL": "syslog+tls://" + endpoint.Addr().String(),
|
|
||||||
"SWWAF_LOG_REMOTE_BUFFER": "1",
|
|
||||||
"SWWAF_METRICS_TOKEN": token,
|
|
||||||
}
|
|
||||||
|
|
||||||
out := runUntilStopped(t, env, func(url string) {
|
|
||||||
wantGreeting(t, url)
|
|
||||||
|
|
||||||
// More than one line has been written, and the buffer holds the
|
|
||||||
// last.
|
|
||||||
metrics := metricsText(t, url+"_smallwebwaf/metrics", token)
|
|
||||||
for _, series := range []string{
|
|
||||||
"smallwebwaf_remote_log_lines_sent_total 0",
|
|
||||||
"smallwebwaf_remote_log_buffer_depth 1",
|
|
||||||
} {
|
|
||||||
if !strings.Contains(metrics, "\n"+series+"\n") {
|
|
||||||
t.Errorf("no %q in the metrics:\n%s", series, metrics)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.Contains(metrics, "\nsmallwebwaf_remote_log_lines_dropped_total 0\n") ||
|
|
||||||
!strings.Contains(metrics, "\nsmallwebwaf_remote_log_lines_dropped_total ") {
|
|
||||||
t.Errorf("no line dropped in the metrics:\n%s", metrics)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Closed, the endpoint refuses the connection made to send the
|
|
||||||
// lines still waiting at the stop, which then does not wait.
|
|
||||||
_ = endpoint.Close()
|
|
||||||
})
|
|
||||||
|
|
||||||
out.line(t, "type", "request")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestStateFileThatDoesNotParseStopsTheStart(t *testing.T) {
|
func TestStateFileThatDoesNotParseStopsTheStart(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -526,9 +349,7 @@ func wantStartRefused(t *testing.T, dir, want string) {
|
|||||||
|
|
||||||
out := &output{}
|
out := &output{}
|
||||||
|
|
||||||
status := run(ctx, map[string]string{
|
status := run(ctx, map[string]string{listenAddr: localhost + ":0", stateDir: dir}, out)
|
||||||
listenAddr: localhost + ":0", stateDir: dir, rulesDir: t.TempDir(),
|
|
||||||
}, out)
|
|
||||||
if status != 1 {
|
if status != 1 {
|
||||||
t.Fatalf("exit status %d, want 1", status)
|
t.Fatalf("exit status %d, want 1", status)
|
||||||
}
|
}
|
||||||
@@ -541,30 +362,6 @@ func wantStartRefused(t *testing.T, dir, want string) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// wantRulesRefused runs smallwebwaf with its rule files in dir, and
|
|
||||||
// checks that it stops at start, with the error want. If it starts
|
|
||||||
// instead, it is stopped after waitLimit.
|
|
||||||
func wantRulesRefused(t *testing.T, dir, want string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
ctx, stop := context.WithTimeout(t.Context(), waitLimit)
|
|
||||||
defer stop()
|
|
||||||
|
|
||||||
out := &output{}
|
|
||||||
|
|
||||||
status := run(ctx, map[string]string{
|
|
||||||
listenAddr: localhost + ":0", stateDir: t.TempDir(), rulesDir: dir,
|
|
||||||
}, out)
|
|
||||||
if status != 1 {
|
|
||||||
t.Fatalf("exit status %d, want 1", status)
|
|
||||||
}
|
|
||||||
|
|
||||||
line := out.line(t, "msg", "cannot use the rule files")
|
|
||||||
if line["error"] != want || line["level"] != "ERROR" {
|
|
||||||
t.Errorf("start refused with %v, want the error %q", line, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// startApp starts an app that answers every request with greeting, and
|
// startApp starts an app that answers every request with greeting, and
|
||||||
// returns its URL.
|
// returns its URL.
|
||||||
func startApp(t *testing.T) string {
|
func startApp(t *testing.T) string {
|
||||||
@@ -646,10 +443,8 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
|
|||||||
"SWWAF_LIMIT_BAN_DURATION": "1h",
|
"SWWAF_LIMIT_BAN_DURATION": "1h",
|
||||||
"SWWAF_LIMIT_BAN_REPEAT_WINDOW": "24h",
|
"SWWAF_LIMIT_BAN_REPEAT_WINDOW": "24h",
|
||||||
"SWWAF_MAX_BAN_DURATION": "7d",
|
"SWWAF_MAX_BAN_DURATION": "7d",
|
||||||
"SWWAF_ATTACK_BAN_DURATION": "7d",
|
|
||||||
"SWWAF_MAX_BANS": "5000",
|
"SWWAF_MAX_BANS": "5000",
|
||||||
"SWWAF_BAN_SCOPE_V4_PREFIX": "32",
|
"SWWAF_BAN_SCOPE_V4_PREFIX": "32",
|
||||||
"SWWAF_RULES_ENABLED": "true",
|
|
||||||
}
|
}
|
||||||
|
|
||||||
for name, value := range want {
|
for name, value := range want {
|
||||||
@@ -689,69 +484,6 @@ func wantGreeting(t *testing.T, url string) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// messages returns the message of each record in received, octet-counted
|
|
||||||
// frames of RFC 5424 records with the default facility and app name, each
|
|
||||||
// with the newline that ends a line on stdout.
|
|
||||||
func messages(t *testing.T, received string) []string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
hostname, _ := os.Hostname()
|
|
||||||
header := " " + hostname + " " + hostname + " - - - "
|
|
||||||
|
|
||||||
var found []string
|
|
||||||
|
|
||||||
for received != "" {
|
|
||||||
count, rest, _ := strings.Cut(received, " ")
|
|
||||||
|
|
||||||
length, err := strconv.Atoi(count)
|
|
||||||
if err != nil || length > len(rest) {
|
|
||||||
t.Fatalf("no frame at %q", received)
|
|
||||||
}
|
|
||||||
|
|
||||||
record := rest[:length]
|
|
||||||
received = rest[length:]
|
|
||||||
|
|
||||||
_, message, ok := strings.Cut(record, header)
|
|
||||||
if !ok || !strings.HasPrefix(record, "<134>1 ") {
|
|
||||||
t.Fatalf("record %q, want priority <134> and header %q", record, header)
|
|
||||||
}
|
|
||||||
|
|
||||||
found = append(found, message+"\n")
|
|
||||||
}
|
|
||||||
|
|
||||||
return found
|
|
||||||
}
|
|
||||||
|
|
||||||
// metricsText asks for the metrics at url with token, and returns them.
|
|
||||||
func metricsText(t *testing.T, url, token string) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, url,
|
|
||||||
http.NoBody)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("new request: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
req.Header.Set("Authorization", "Bearer "+token)
|
|
||||||
|
|
||||||
transport := &http.Transport{}
|
|
||||||
defer transport.CloseIdleConnections()
|
|
||||||
|
|
||||||
res, err := (&http.Client{Transport: transport}).Do(req)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("request: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
body, err := io.ReadAll(res.Body)
|
|
||||||
_ = res.Body.Close()
|
|
||||||
|
|
||||||
if err != nil || res.StatusCode != http.StatusOK {
|
|
||||||
t.Fatalf("metrics answered %d (%v)", res.StatusCode, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return string(body)
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantRefused checks that a request to url is refused with 403, the
|
// wantRefused checks that a request to url is refused with 403, the
|
||||||
// default SWWAF_BAN_RESPONSE.
|
// default SWWAF_BAN_RESPONSE.
|
||||||
func wantRefused(t *testing.T, url string) {
|
func wantRefused(t *testing.T, url string) {
|
||||||
|
|||||||
+10
-40
@@ -48,7 +48,6 @@ var (
|
|||||||
errVersion = errors.New("unknown version")
|
errVersion = errors.New("unknown version")
|
||||||
// errMissing is for an entry without a field it needs.
|
// errMissing is for an entry without a field it needs.
|
||||||
errMissing = errors.New("has no")
|
errMissing = errors.New("has no")
|
||||||
errCause = errors.New("is not limit, attack or admin")
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// Params are what Load needs.
|
// Params are what Load needs.
|
||||||
@@ -96,15 +95,11 @@ type bansFile struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// banEntry is a ban as bans.json holds it: a permanent ban's expires is
|
// banEntry is a ban as bans.json holds it: a permanent ban's expires is
|
||||||
// null, a ban an admin added may have no cause, which makes it an
|
// null.
|
||||||
// admin's, and lifted is left out until an admin lifts the ban.
|
|
||||||
type banEntry struct {
|
type banEntry struct {
|
||||||
Netblock netip.Prefix `json:"netblock"`
|
Netblock netip.Prefix `json:"netblock"`
|
||||||
Start time.Time `json:"start"`
|
Start time.Time `json:"start"`
|
||||||
Expires *time.Time `json:"expires"`
|
Expires *time.Time `json:"expires"`
|
||||||
Cause string `json:"cause"`
|
|
||||||
Reason string `json:"reason,omitempty"`
|
|
||||||
Lifted *time.Time `json:"lifted,omitempty"`
|
|
||||||
Notes bans.Notes `json:"notes"`
|
Notes bans.Notes `json:"notes"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -265,7 +260,7 @@ func (f *Files) fileChanged(name string) {
|
|||||||
// runs, by Watch or by a write, is taken in here. An edit that does not
|
// runs, by Watch or by a write, is taken in here. An edit that does not
|
||||||
// parse is neither counted nor logged, and takeIn's error returned.
|
// parse is neither counted nor logged, and takeIn's error returned.
|
||||||
func (f *Files) takeInEdit(name string, data []byte) error {
|
func (f *Files) takeInEdit(name string, data []byte) error {
|
||||||
_, err := f.takeIn(name, data, true)
|
_, err := f.takeIn(name, data)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -287,7 +282,7 @@ func (f *Files) read(name string) (int, error) {
|
|||||||
return 0, err
|
return 0, err
|
||||||
}
|
}
|
||||||
|
|
||||||
return f.takeIn(name, data, false)
|
return f.takeIn(name, data)
|
||||||
}
|
}
|
||||||
|
|
||||||
// readChanged returns what the state file name holds, and whether that
|
// readChanged returns what the state file name holds, and whether that
|
||||||
@@ -311,11 +306,9 @@ func (f *Files) readChanged(name string) ([]byte, bool, error) {
|
|||||||
|
|
||||||
// takeIn parses data, what the state file name holds, puts it into the
|
// takeIn parses data, what the state file name holds, puts it into the
|
||||||
// part that keeps that state, in place of what the part held, and returns
|
// part that keeps that state, in place of what the part held, and returns
|
||||||
// how many entries the file holds. edit is whether data is an admin's
|
// how many entries the file holds. An error names the file and, where the
|
||||||
// edit taken in while smallwebwaf runs, rather than the file read at the
|
// JSON decoder tells it, the line and column, or else the entry.
|
||||||
// start. An error names the file and, where the JSON decoder tells it,
|
func (f *Files) takeIn(name string, data []byte) (int, error) {
|
||||||
// the line and column, or else the entry.
|
|
||||||
func (f *Files) takeIn(name string, data []byte, edit bool) (int, error) {
|
|
||||||
path := filepath.Join(f.params.Dir, name)
|
path := filepath.Join(f.params.Dir, name)
|
||||||
|
|
||||||
var entries int
|
var entries int
|
||||||
@@ -334,12 +327,7 @@ func (f *Files) takeIn(name string, data []byte, edit bool) (int, error) {
|
|||||||
held = append(held, entry.ban())
|
held = append(held, entry.ban())
|
||||||
}
|
}
|
||||||
|
|
||||||
if edit {
|
f.params.Ledger.Load(held)
|
||||||
f.params.Ledger.LoadEdit(held)
|
|
||||||
} else {
|
|
||||||
f.params.Ledger.Load(held)
|
|
||||||
}
|
|
||||||
|
|
||||||
entries = len(held)
|
entries = len(held)
|
||||||
case clientsJSON:
|
case clientsJSON:
|
||||||
var file clientsFile
|
var file clientsFile
|
||||||
@@ -456,35 +444,21 @@ func (f *Files) encode(name string) ([]byte, error) {
|
|||||||
|
|
||||||
// newBanEntry returns ban as bans.json holds it.
|
// newBanEntry returns ban as bans.json holds it.
|
||||||
func newBanEntry(ban bans.Ban) banEntry {
|
func newBanEntry(ban bans.Ban) banEntry {
|
||||||
entry := banEntry{
|
entry := banEntry{Netblock: ban.Netblock, Start: ban.Start, Notes: ban.Notes}
|
||||||
Netblock: ban.Netblock, Start: ban.Start, Cause: ban.Cause, Reason: ban.Reason,
|
|
||||||
Notes: ban.Notes,
|
|
||||||
}
|
|
||||||
if !ban.Permanent() {
|
if !ban.Permanent() {
|
||||||
entry.Expires = &ban.Expires
|
entry.Expires = &ban.Expires
|
||||||
}
|
}
|
||||||
|
|
||||||
if !ban.Lifted.IsZero() {
|
|
||||||
entry.Lifted = &ban.Lifted
|
|
||||||
}
|
|
||||||
|
|
||||||
return entry
|
return entry
|
||||||
}
|
}
|
||||||
|
|
||||||
// ban returns the ban an entry of bans.json holds.
|
// ban returns the ban an entry of bans.json holds.
|
||||||
func (e banEntry) ban() bans.Ban {
|
func (e banEntry) ban() bans.Ban {
|
||||||
ban := bans.Ban{
|
ban := bans.Ban{Netblock: e.Netblock, Start: e.Start, Notes: e.Notes}
|
||||||
Netblock: e.Netblock, Start: e.Start, Cause: e.Cause, Reason: e.Reason,
|
|
||||||
Notes: e.Notes,
|
|
||||||
}
|
|
||||||
if e.Expires != nil {
|
if e.Expires != nil {
|
||||||
ban.Expires = *e.Expires
|
ban.Expires = *e.Expires
|
||||||
}
|
}
|
||||||
|
|
||||||
if e.Lifted != nil {
|
|
||||||
ban.Lifted = *e.Lifted
|
|
||||||
}
|
|
||||||
|
|
||||||
return ban
|
return ban
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -492,8 +466,7 @@ func (e banEntry) ban() bans.Ban {
|
|||||||
// client, a start, from which the length of the netblock's next ban is
|
// client, a start, from which the length of the netblock's next ban is
|
||||||
// worked out, or an expires, which would make it permanent. A permanent
|
// worked out, or an expires, which would make it permanent. A permanent
|
||||||
// ban's expires is null, which Bans cannot tell from a missing one, so
|
// ban's expires is null, which Bans cannot tell from a missing one, so
|
||||||
// each expires is read again as written. A cause other than limit,
|
// each expires is read again as written.
|
||||||
// attack or admin, most likely misspelt, is refused too.
|
|
||||||
func (f *bansFile) check(data []byte) error {
|
func (f *bansFile) check(data []byte) error {
|
||||||
var written struct {
|
var written struct {
|
||||||
Bans []struct {
|
Bans []struct {
|
||||||
@@ -514,9 +487,6 @@ func (f *bansFile) check(data []byte) error {
|
|||||||
return missing(i, "start")
|
return missing(i, "start")
|
||||||
case written.Bans[i].Expires == nil:
|
case written.Bans[i].Expires == nil:
|
||||||
return missing(i, "expires")
|
return missing(i, "expires")
|
||||||
case entry.Cause != "" && entry.Cause != bans.CauseLimit &&
|
|
||||||
entry.Cause != bans.CauseAttack && entry.Cause != bans.CauseAdmin:
|
|
||||||
return fmt.Errorf("entry %d's cause %q %w", i+1, entry.Cause, errCause)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -48,8 +48,6 @@ const permanentBansJSON = `{
|
|||||||
"netblock": "2001:db8::/64",
|
"netblock": "2001:db8::/64",
|
||||||
"start": "2026-10-06T00:00:00Z",
|
"start": "2026-10-06T00:00:00Z",
|
||||||
"expires": null,
|
"expires": null,
|
||||||
"cause": "admin",
|
|
||||||
"reason": "scrapes every commit",
|
|
||||||
"notes": {
|
"notes": {
|
||||||
"country": "DE",
|
"country": "DE",
|
||||||
"limit": 1000,
|
"limit": 1000,
|
||||||
@@ -65,26 +63,13 @@ const permanentBansJSON = `{
|
|||||||
},
|
},
|
||||||
"requests": 1500,
|
"requests": 1500,
|
||||||
"refused": 3,
|
"refused": 3,
|
||||||
"earlier_bans": {
|
"earlier_bans": 5
|
||||||
"limit": 3,
|
|
||||||
"attack": 1,
|
|
||||||
"admin": 1
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
`
|
`
|
||||||
|
|
||||||
// liftedClient is the client whose ban liftedBansJSON holds.
|
|
||||||
const liftedClient = "203.0.113.9"
|
|
||||||
|
|
||||||
// liftedBansJSON is bans.json holding an hour's ban for a broken limit on
|
|
||||||
// liftedClient, from midnight, that an admin lifted ten minutes in.
|
|
||||||
const liftedBansJSON = `{"version": 1, "bans": [{"netblock": "203.0.113.9/32", ` +
|
|
||||||
`"start": "2026-10-06T00:00:00Z", "expires": "2026-10-06T01:00:00Z", ` +
|
|
||||||
`"cause": "limit", "lifted": "2026-10-06T00:10:00Z"}]}`
|
|
||||||
|
|
||||||
func TestFilesWrittenAndReadBack(t *testing.T) {
|
func TestFilesWrittenAndReadBack(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -278,19 +263,6 @@ func TestEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestBanWithAnotherCauseStopsTheStart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
wantRefused(t, bansJSON, `{"version": 1, "bans": [`+
|
|
||||||
`{"netblock": "203.0.113.9/32", "start": "2026-10-06T00:00:00Z", `+
|
|
||||||
`"expires": null, "cause": "attack"}, `+
|
|
||||||
`{"netblock": "203.0.113.10/32", "start": "2026-10-06T00:00:00Z", `+
|
|
||||||
`"expires": null, "cause": "admin"}, `+
|
|
||||||
`{"netblock": "203.0.113.11/32", "start": "2026-10-06T00:00:00Z", `+
|
|
||||||
`"expires": null, "cause": "atack"}]}`,
|
|
||||||
`: entry 3's cause "atack" is not limit, attack or admin`)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUnknownVersionStopsTheStart(t *testing.T) {
|
func TestUnknownVersionStopsTheStart(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -624,7 +596,7 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
|
|||||||
`"start": "2026-10-06T00:00:00Z", "expires": null}]}`)
|
`"start": "2026-10-06T00:00:00Z", "expires": null}]}`)
|
||||||
wantTakenIn(t, lines, dir, bansJSON)
|
wantTakenIn(t, lines, dir, bansJSON)
|
||||||
wantEqual(t, bansJSON, params.Ledger.Snapshot(),
|
wantEqual(t, bansJSON, params.Ledger.Snapshot(),
|
||||||
[]bans.Ban{{Netblock: client, Start: midnight(), Cause: bans.CauseAdmin}})
|
[]bans.Ban{{Netblock: client, Start: midnight()}})
|
||||||
|
|
||||||
edit(t, dir, clientsJSON, `{"version": 1, "clients": [`+
|
edit(t, dir, clientsJSON, `{"version": 1, "clients": [`+
|
||||||
`{"client": "198.51.100.7/32", "history": {"requests": 7}}]}`)
|
`{"client": "198.51.100.7/32", "history": {"requests": 7}}]}`)
|
||||||
@@ -723,100 +695,6 @@ func TestBanAddedAndLiftedThroughBansJSON(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestBanWithoutACauseTakenInAsAnAdmins(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const reason = "probes for logins"
|
|
||||||
|
|
||||||
// adminsBansJSON is bans.json as an admin writes it, with a ban on
|
|
||||||
// netblock without a cause, and adminsBans the bans it holds.
|
|
||||||
adminsBansJSON := func(netblock string) string {
|
|
||||||
return `{"version": 1, "bans": [{"netblock": "` + netblock + `", ` +
|
|
||||||
`"start": "2026-10-06T00:00:00Z", "expires": null, "reason": "` +
|
|
||||||
reason + `"}]}`
|
|
||||||
}
|
|
||||||
adminsBans := func(netblock string) []bans.Ban {
|
|
||||||
return []bans.Ban{{
|
|
||||||
Netblock: netip.MustParsePrefix(netblock),
|
|
||||||
Start: midnight(),
|
|
||||||
Cause: bans.CauseAdmin,
|
|
||||||
Reason: reason,
|
|
||||||
}}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Read at the start, the ban is taken in as an admin's, though not
|
|
||||||
// counted among the bans made since the start, and written back with
|
|
||||||
// that cause and the admin's reason.
|
|
||||||
dir := t.TempDir()
|
|
||||||
edit(t, dir, bansJSON, adminsBansJSON("203.0.113.0/24"))
|
|
||||||
|
|
||||||
params := newParams(dir)
|
|
||||||
lines := logInto(¶ms)
|
|
||||||
files := load(t, params)
|
|
||||||
wantEqual(t, bansJSON, params.Ledger.Snapshot(), adminsBans("203.0.113.0/24"))
|
|
||||||
wantMadeByAnAdmin(t, params.Ledger, 0)
|
|
||||||
|
|
||||||
err := files.WriteAll()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var written struct {
|
|
||||||
Bans []struct {
|
|
||||||
Cause string `json:"cause"`
|
|
||||||
Reason string `json:"reason"`
|
|
||||||
} `json:"bans"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err = json.Unmarshal([]byte(readFile(t, filepath.Join(dir, bansJSON))), &written)
|
|
||||||
if err != nil || len(written.Bans) != 1 || written.Bans[0].Cause != bans.CauseAdmin ||
|
|
||||||
written.Bans[0].Reason != reason {
|
|
||||||
t.Errorf("bans.json holds %+v (%v), want the ban with the cause admin "+
|
|
||||||
"and the reason %q", written, err, reason)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Taken in while smallwebwaf runs, a ban on another netblock is an
|
|
||||||
// admin's too, and one made since the start.
|
|
||||||
watch(t, files, lines)
|
|
||||||
edit(t, dir, bansJSON, adminsBansJSON("198.51.100.0/24"))
|
|
||||||
wantTakenIn(t, lines, dir, bansJSON)
|
|
||||||
wantEqual(t, bansJSON, params.Ledger.Snapshot(), adminsBans("198.51.100.0/24"))
|
|
||||||
wantMadeByAnAdmin(t, params.Ledger, 1)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLiftedBanReadAtTheStart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
edit(t, dir, bansJSON, liftedBansJSON)
|
|
||||||
|
|
||||||
params := newParams(dir)
|
|
||||||
wantLiftedBanKept(t, load(t, params), dir, params.Ledger)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestBanLiftedByAnEditWhileRunning(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
params := newParams(dir)
|
|
||||||
lines := logInto(¶ms)
|
|
||||||
files := load(t, params)
|
|
||||||
watch(t, files, lines)
|
|
||||||
|
|
||||||
// The ban that liftedBansJSON lifts, before it is lifted.
|
|
||||||
netblock := netip.MustParsePrefix(liftedClient + "/32")
|
|
||||||
params.Ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
|
||||||
|
|
||||||
_, banned := params.Ledger.Find(netblock.Addr(), afterLifting())
|
|
||||||
if !banned {
|
|
||||||
t.Fatal("the ban does not refuse before it is lifted")
|
|
||||||
}
|
|
||||||
|
|
||||||
edit(t, dir, bansJSON, liftedBansJSON)
|
|
||||||
wantTakenIn(t, lines, dir, bansJSON)
|
|
||||||
wantLiftedBanKept(t, files, dir, params.Ledger)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestBrokenEditSetAsideAtTheNextWrite(t *testing.T) {
|
func TestBrokenEditSetAsideAtTheNextWrite(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -1003,7 +881,6 @@ func newParams(dir string) state.Params {
|
|||||||
LimitBanDuration: time.Hour,
|
LimitBanDuration: time.Hour,
|
||||||
LimitBanRepeatWindow: 24 * time.Hour,
|
LimitBanRepeatWindow: 24 * time.Hour,
|
||||||
MaxBanDuration: 7 * 24 * time.Hour,
|
MaxBanDuration: 7 * 24 * time.Hour,
|
||||||
AttackBanDuration: 7 * 24 * time.Hour,
|
|
||||||
MaxBans: 5000,
|
MaxBans: 5000,
|
||||||
}),
|
}),
|
||||||
Limiter: ratelimit.New(ratelimit.Limits{}),
|
Limiter: ratelimit.New(ratelimit.Limits{}),
|
||||||
@@ -1016,17 +893,14 @@ func newParams(dir string) state.Params {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// fill puts a permanent ban an admin made, a ban for a broken limit and
|
// fill puts a ban that ends and one that does not, clients with counts
|
||||||
// one for a clear sign of attack, clients with counts and histories, and
|
// and histories, and GeoJS answers into the parts of params.
|
||||||
// GeoJS answers into the parts of params.
|
|
||||||
func fill(params state.Params) {
|
func fill(params state.Params) {
|
||||||
now := midnight()
|
now := midnight()
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
|
||||||
params.Ledger.Load([]bans.Ban{permanentBan()})
|
params.Ledger.Load([]bans.Ban{permanentBan()})
|
||||||
params.Ledger.BanForLimit(client, now, bans.Notes{Country: "DE", Limit: 1})
|
params.Ledger.BanForLimit(client, now, bans.Notes{Country: "DE", Limit: 1})
|
||||||
params.Ledger.BanForAttack(netip.MustParsePrefix("192.0.2.1/32"), now,
|
|
||||||
bans.Notes{RuleID: "env-file", Target: "path"})
|
|
||||||
|
|
||||||
for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} {
|
for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} {
|
||||||
params.Limiter.Count(netip.MustParsePrefix(c), now)
|
params.Limiter.Count(netip.MustParsePrefix(c), now)
|
||||||
@@ -1050,8 +924,6 @@ func permanentBan() bans.Ban {
|
|||||||
return bans.Ban{
|
return bans.Ban{
|
||||||
Netblock: netip.MustParsePrefix("2001:db8::/64"),
|
Netblock: netip.MustParsePrefix("2001:db8::/64"),
|
||||||
Start: midnight(),
|
Start: midnight(),
|
||||||
Cause: bans.CauseAdmin,
|
|
||||||
Reason: "scrapes every commit",
|
|
||||||
Notes: bans.Notes{
|
Notes: bans.Notes{
|
||||||
Country: "DE",
|
Country: "DE",
|
||||||
Limit: 1000,
|
Limit: 1000,
|
||||||
@@ -1067,64 +939,11 @@ func permanentBan() bans.Ban {
|
|||||||
},
|
},
|
||||||
Requests: 1500,
|
Requests: 1500,
|
||||||
Refused: 3,
|
Refused: 3,
|
||||||
EarlierBans: bans.EarlierBans{Limit: 3, Attack: 1, Admin: 1},
|
EarlierBans: 5,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// afterLifting is a time after the ban liftedBansJSON holds was lifted,
|
|
||||||
// while it would still last.
|
|
||||||
func afterLifting() time.Time {
|
|
||||||
return midnight().Add(30 * time.Minute)
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantLiftedBanKept checks that ledger holds the ban liftedBansJSON holds,
|
|
||||||
// which refuses nothing and does not make the next ban for a broken limit
|
|
||||||
// longer, and that files write it to bans.json, in dir, still lifted.
|
|
||||||
func wantLiftedBanKept(
|
|
||||||
t *testing.T, files *state.Files, dir string, ledger *bans.Ledger,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
err := files.WriteAll()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
const lifted = `"lifted": "2026-10-06T00:10:00Z"`
|
|
||||||
if got := readFile(t, filepath.Join(dir, bansJSON)); !strings.Contains(got, lifted) {
|
|
||||||
t.Errorf("bans.json holds\n%s\nwant the ban with %s", got, lifted)
|
|
||||||
}
|
|
||||||
|
|
||||||
netblock := netip.MustParsePrefix(liftedClient + "/32")
|
|
||||||
|
|
||||||
_, banned := ledger.Check(netblock.Addr(), afterLifting())
|
|
||||||
if banned {
|
|
||||||
t.Error("the lifted ban refuses")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Were the lifted ban counted, the next would last three hours.
|
|
||||||
ban := ledger.BanForLimit(netblock, afterLifting(), bans.Notes{})
|
|
||||||
if ban.Expires.Sub(ban.Start) != time.Hour {
|
|
||||||
t.Errorf("the next ban lasts %s, want 1h", ban.Expires.Sub(ban.Start))
|
|
||||||
}
|
|
||||||
|
|
||||||
held := ledger.Bans(netblock)
|
|
||||||
if len(held) != 2 || !held[0].Lifted.Equal(midnight().Add(10*time.Minute)) {
|
|
||||||
t.Errorf("the ledger holds %+v, want the lifted ban and the new one", held)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantMadeByAnAdmin checks how many bans ledger counts as made by an
|
|
||||||
// admin since the start.
|
|
||||||
func wantMadeByAnAdmin(t *testing.T, ledger *bans.Ledger, want int) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
if got := ledger.Made(bans.CauseAdmin); got != want {
|
|
||||||
t.Errorf("%d bans made by an admin, want %d", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// load reads the state files into the parts of params.
|
// load reads the state files into the parts of params.
|
||||||
func load(t *testing.T, params state.Params) *state.Files {
|
func load(t *testing.T, params state.Params) *state.Files {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|||||||
+5
-27
@@ -3,10 +3,9 @@
|
|||||||
# deploy/example-app, then run the app's container with a volume for the
|
# deploy/example-app, then run the app's container with a volume for the
|
||||||
# state files and check that the health check passes, that a request is
|
# state files and check that the health check passes, that a request is
|
||||||
# served through smallwebwaf, that a second one in a minute bans the
|
# served through smallwebwaf, that a second one in a minute bans the
|
||||||
# client, that a probe for /.env bans another client, which its next
|
# client, that `sv stop` stops smallwebwaf in order, that `docker stop`
|
||||||
# request bans for good, that `sv stop` stops smallwebwaf in order, that
|
# stops the container without having to kill it, and that a new
|
||||||
# `docker stop` stops the container without having to kill it, and that
|
# container on the same volume still refuses the banned client. The
|
||||||
# a new container on the same volume still refuses the banned client. The
|
|
||||||
# containers, the volume and both images are removed however the script
|
# containers, the volume and both images are removed however the script
|
||||||
# ends. Building the app needs network access, for nixpkgs' binary cache.
|
# ends. Building the app needs network access, for nixpkgs' binary cache.
|
||||||
# script/check does not run this.
|
# script/check does not run this.
|
||||||
@@ -53,13 +52,9 @@ healthy() {
|
|||||||
[ "$status" = healthy ]
|
[ "$status" = healthy ]
|
||||||
}
|
}
|
||||||
|
|
||||||
# logged <text>...: a line of the container's output holds every text,
|
# logged <text>: the container's output holds text.
|
||||||
# in any order.
|
|
||||||
logged() {
|
logged() {
|
||||||
lines="$(docker logs "$CONTAINER" 2>&1)"
|
docker logs "$CONTAINER" 2>&1 | grep -qF "$1"
|
||||||
for text in "$@"; do
|
|
||||||
lines="$(printf '%s\n' "$lines" | grep -F "$text")" || return 1
|
|
||||||
done
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# start_container: run the app's container, with the state files on the
|
# start_container: run the app's container, with the state files on the
|
||||||
@@ -82,15 +77,6 @@ refused() {
|
|||||||
[ "$code" = 403 ]
|
[ "$code" = 403 ]
|
||||||
}
|
}
|
||||||
|
|
||||||
# refused_from <client> <path>: a request for path from client, as
|
|
||||||
# X-Forwarded-For names it, gets 403. smallwebwaf believes the header
|
|
||||||
# from docker's gateway, a private address.
|
|
||||||
refused_from() {
|
|
||||||
code="$(curl --silent --output /dev/null --write-out '%{http_code}' \
|
|
||||||
--max-time 10 --header "X-Forwarded-For: $1" "http://$address$2")" || true
|
|
||||||
[ "$code" = 403 ]
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
trap cleanup EXIT
|
trap cleanup EXIT
|
||||||
@@ -114,14 +100,6 @@ main() {
|
|||||||
wait_for "smallwebwaf logged no ban" logged '"action":"rate_limited"'
|
wait_for "smallwebwaf logged no ban" logged '"action":"rate_limited"'
|
||||||
echo "example-app: a second request in a minute bans the client"
|
echo "example-app: a second request in a minute bans the client"
|
||||||
|
|
||||||
refused_from 203.0.113.9 /.env || fail "a probe for /.env was not refused"
|
|
||||||
wait_for "smallwebwaf logged no ban for the probe" \
|
|
||||||
logged '"action":"banned"' '"rule_ids":["env-file"]'
|
|
||||||
refused_from 203.0.113.9 / || fail "the client of the probe was let through"
|
|
||||||
wait_for "the client's next request did not make its ban permanent" \
|
|
||||||
logged '"ban_expires":"permanent"'
|
|
||||||
echo "example-app: a probe for /.env bans the client, its next request for good"
|
|
||||||
|
|
||||||
docker exec "$CONTAINER" sv stop smallwebwaf >/dev/null ||
|
docker exec "$CONTAINER" sv stop smallwebwaf >/dev/null ||
|
||||||
fail "sv stop smallwebwaf failed"
|
fail "sv stop smallwebwaf failed"
|
||||||
wait_for "smallwebwaf did not stop in order" logged '"msg":"stopped"'
|
wait_for "smallwebwaf did not stop in order" logged '"msg":"stopped"'
|
||||||
|
|||||||
+1
-7
@@ -1,9 +1,7 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/run: build bin/smallwebwaf with script/build and run it, with
|
# script/run: build bin/smallwebwaf with script/build and run it, with
|
||||||
# the settings in the environment. Unless SWWAF_STATE_DIR is set, the
|
# the settings in the environment. Unless SWWAF_STATE_DIR is set, the
|
||||||
# state files go in bin/state, beside the binary, and unless
|
# state files go in bin/state, beside the binary.
|
||||||
# SWWAF_RULES_DIR is set, the rule files are those of share/rules.d,
|
|
||||||
# which the image ships.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -16,10 +14,6 @@ main() {
|
|||||||
export SWWAF_STATE_DIR
|
export SWWAF_STATE_DIR
|
||||||
mkdir -p "$SWWAF_STATE_DIR"
|
mkdir -p "$SWWAF_STATE_DIR"
|
||||||
fi
|
fi
|
||||||
if [ -z "${SWWAF_RULES_DIR+set}" ]; then
|
|
||||||
SWWAF_RULES_DIR="$ROOT/share/rules.d"
|
|
||||||
export SWWAF_RULES_DIR
|
|
||||||
fi
|
|
||||||
exec "$ROOT/bin/smallwebwaf"
|
exec "$ROOT/bin/smallwebwaf"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,15 +0,0 @@
|
|||||||
# 00-default.rules: probes no real visitor sends, anchored at the site root
|
|
||||||
|
|
||||||
# id target action regex
|
|
||||||
env-file path ban (?i)^/\.env(\.[a-z]+)?$
|
|
||||||
vcs-dir path ban (?i)^/\.(git|svn|hg|bzr)(/|$)
|
|
||||||
secrets-dir path ban (?i)^/\.(aws|ssh|docker|kube)/
|
|
||||||
secret-file path ban (?i)^/\.(htpasswd|htaccess|npmrc|netrc|pgpass|git-credentials|bash_history|DS_Store)$
|
|
||||||
editor-dir path ban (?i)^/\.(vscode|idea)/
|
|
||||||
backup-file path ban (?i)^/[^/]+\.(php(\.[a-z0-9]+|~)|sql(\.[a-z0-9]+)?)$
|
|
||||||
log-file path ban (?i)^/(debug|error|access)\.log$
|
|
||||||
compose-file path ban (?i)^/(docker-)?compose\.ya?ml$
|
|
||||||
php-shell path ban (?i)^/(shell|c99|r57|wso|alfa)\.php$
|
|
||||||
scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b
|
|
||||||
path-traversal uri block (\.\./){2,}
|
|
||||||
empty-agent user_agent log ^$
|
|
||||||
Reference in New Issue
Block a user