Compare commits
1
Commits
next
...
f2fcf11aed
| Author | SHA1 | Date | |
|---|---|---|---|
 clawbot
|
f2fcf11aed |
@@ -22,29 +22,30 @@ fields, which come a little later, and the metrics endpoint and the header size
|
||||
and the idle time as settings, which come last in it. So are the four parts of
|
||||
the stage after it: the rule files, with the bans for a clear sign of attack,
|
||||
the other admin endpoints, alerts to all three destinations, a JSON webhook,
|
||||
Slack and ntfy, and remote log sending. So is the first part of the stage after
|
||||
Slack and ntfy, and remote log sending. So are two parts of the stage after
|
||||
that: the AS number and country of every client, looked up through GeoJS or in
|
||||
the IPinfo Lite database file. `smallwebwaf` passes each request to the app and
|
||||
the app's answer back, unchanged, within its timeouts and size limits, works out
|
||||
each client's address, looks up its AS number and country unless you switch that
|
||||
off, bans a client that sends too many requests, not counting those for the
|
||||
paths you choose, refuses a client that comes from a country you refuse or from
|
||||
a network you refuse, lets the networks you choose through, checks each request
|
||||
against the rule files and bans a client whose request is a clear sign of
|
||||
attack, keeps its bans, each client's counters and history, and GeoJS's answers
|
||||
in JSON files across restarts, takes in your edits of those files, such as a ban
|
||||
you make, keep or lift, and of the rule files while it runs, writes a JSON log
|
||||
line for every request, sends its log lines to a syslog server too if you name
|
||||
one, sends an alert to a webhook, to Slack and to ntfy, each if you name one,
|
||||
for each ban it makes or makes permanent, for GeoJS failing, for a rule file or
|
||||
state file with an error and for a replacement of the lookup database it cannot
|
||||
read, serves Prometheus metrics to a scraper that holds the metrics token, lets
|
||||
an admin who holds the admin token list, add and lift bans and ask what it knows
|
||||
of a client, and in `observe` mode passes on the requests it would refuse,
|
||||
logging what it would have done with them. It comes as the image the app's own
|
||||
image is built on. The rest of the design comes after that, in the order of the
|
||||
build order in [`SPEC.md`](SPEC.md). The survey of existing tools that led to
|
||||
the design is in [`EVALUATION.md`](EVALUATION.md).
|
||||
the IPinfo Lite database file, and the byte limits. `smallwebwaf` passes each
|
||||
request to the app and the app's answer back, unchanged, within its timeouts and
|
||||
size limits, works out each client's address, looks up its AS number and country
|
||||
unless you switch that off, bans a client that sends too many requests or too
|
||||
many bytes, not counting those for the paths you choose, refuses a client that
|
||||
comes from a country you refuse or from a network you refuse, lets the networks
|
||||
you choose through, checks each request against the rule files and bans a client
|
||||
whose request is a clear sign of attack, keeps its bans, each client's counters
|
||||
and history, and GeoJS's answers in JSON files across restarts, takes in your
|
||||
edits of those files, such as a ban you make, keep or lift, and of the rule
|
||||
files while it runs, writes a JSON log line for every request, sends its log
|
||||
lines to a syslog server too if you name one, sends an alert to a webhook, to
|
||||
Slack and to ntfy, each if you name one, for each ban it makes or makes
|
||||
permanent, for GeoJS failing, for a rule file or state file with an error and
|
||||
for a replacement of the lookup database it cannot read, serves Prometheus
|
||||
metrics to a scraper that holds the metrics token, lets an admin who holds the
|
||||
admin token list, add and lift bans and ask what it knows of a client, and in
|
||||
`observe` mode passes on the requests it would refuse, logging what it would
|
||||
have done with them. It comes as the image the app's own image is built on. The
|
||||
rest of the design comes after that, in the order of the build order in
|
||||
[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
|
||||
[`EVALUATION.md`](EVALUATION.md).
|
||||
|
||||
## Getting started
|
||||
|
||||
@@ -107,26 +108,40 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
||||
window still covers. At most 20,000 clients are kept, the least recently seen
|
||||
dropped first, with their history, and a restart gives no client a fresh
|
||||
allowance (see "State files" below).
|
||||
- Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md)
|
||||
describes: the first ban lasts an hour, and a limit broken again within a day
|
||||
of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and
|
||||
81 hours; a ban that would last longer than seven days is permanent instead. A
|
||||
ban covers the client's netblock: its IPv4 address, or the netblock around it
|
||||
that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 /64. While it lasts, every
|
||||
request from the netblock is refused with `SWWAF_BAN_RESPONSE` after the
|
||||
static lists and before the country lists, so the client is not looked up, and
|
||||
is not counted for the rate limits. A ban sets the client's counters back to
|
||||
zero. Each ban carries notes for deciding whether to lift it: the limit, its
|
||||
window and the requests counted in it, the request that broke it, the client's
|
||||
AS number, AS name and country once they are looked up, the netblock's
|
||||
requests since it was first seen, how many of them the ban has refused, and
|
||||
how many bans the netblock had before, for a broken limit, for a clear sign of
|
||||
attack and by an admin. At most `SWWAF_MAX_BANS` bans `smallwebwaf` made are
|
||||
kept, past, active and permanent; past that, the earliest such ban of the
|
||||
netblock that has gone longest without a request is dropped first. The bans
|
||||
whose cause is `admin`, those you make or keep, are kept besides, and never
|
||||
dropped. `bans.json` shows the bans and their notes, a restart lifts none, and
|
||||
you make, keep or lift a ban by editing it (see "State files" below).
|
||||
- Counts each client's bytes over a minute, an hour and a day, in the same way:
|
||||
once a request passed to the app has ended, the body bytes of its answer, of
|
||||
the request, or of both, as `SWWAF_BYTES_COUNT` says. For a WebSocket, or any
|
||||
other upgraded connection, what it carried from the app counts with the
|
||||
answer, and what it carried from the client with the request, once it closes.
|
||||
Bytes that take the client over one of the byte limits below break that limit,
|
||||
and ban the client as a broken rate limit does, so that its next request is
|
||||
refused. The byte limits never cut an answer or an upgraded connection short:
|
||||
the one whose bytes break a limit has already been passed on, or has closed.
|
||||
They leave out what the rate limits leave out: a client in `SWWAF_ALLOW_NETS`
|
||||
or `SWWAF_RATE_LIMIT_EXEMPT_NETS`, and a request for a path
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts.
|
||||
- Bans a client that breaks a rate limit or a byte limit, as "Bans" in
|
||||
[`SPEC.md`](SPEC.md) describes: the first ban lasts an hour, and a limit
|
||||
broken again within a day of a ban ending bans for three times as long as that
|
||||
ban, so 1, 3, 9, 27 and 81 hours; a ban that would last longer than seven days
|
||||
is permanent instead. A ban covers the client's netblock: its IPv4 address, or
|
||||
the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 /64.
|
||||
While it lasts, every request from the netblock is refused with
|
||||
`SWWAF_BAN_RESPONSE` after the static lists and before the country lists, so
|
||||
the client is not looked up, and is not counted for the rate limits. A ban
|
||||
sets the client's counters back to zero. Each ban carries notes for deciding
|
||||
whether to lift it: the limit, whether it is on requests or bytes, its window
|
||||
and the requests or bytes counted in it, the request that broke it, the
|
||||
client's AS number, AS name and country once they are looked up, the
|
||||
netblock's requests since it was first seen, how many of them the ban has
|
||||
refused, and how many bans the netblock had before, for a broken limit, for a
|
||||
clear sign of attack and by an admin. At most `SWWAF_MAX_BANS` bans
|
||||
`smallwebwaf` made are kept, past, active and permanent; past that, the
|
||||
earliest such ban of the netblock that has gone longest without a request is
|
||||
dropped first. The bans whose cause is `admin`, those you make or keep, are
|
||||
kept besides, and never dropped. `bans.json` shows the bans and their notes, a
|
||||
restart lifts none, and you make, keep or lift a ban by editing it (see "State
|
||||
files" below).
|
||||
- Checks each request against the rules of the rule files (see "Rule files"
|
||||
below) after the rate limits, and before its body is read. A `log` rule that
|
||||
matches is noted in the log line; a `block` rule refuses the request with
|
||||
@@ -138,10 +153,10 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
||||
default, and any request from the netblock while it lasts makes it permanent.
|
||||
Once it has run out, the netblock is served like any other, but its next clear
|
||||
sign of attack bans it permanently at once. Such a ban covers the same
|
||||
netblock as a ban for a broken rate limit, does not set the client's counters
|
||||
back to zero, and does not make the netblock's next ban for a broken limit
|
||||
longer. Its notes give the id and the target of the rule that matched in place
|
||||
of the limit.
|
||||
netblock as a ban for a broken limit, does not set the client's counters back
|
||||
to zero, and does not make the netblock's next ban for a broken limit longer.
|
||||
Its notes give the id and the target of the rule that matched in place of the
|
||||
limit.
|
||||
- Looks up the AS number and country of every client through GeoJS, or in the
|
||||
IPinfo Lite database file while `SWWAF_LOOKUP_SOURCE` is `file`, after the
|
||||
static lists and bans, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Country and
|
||||
@@ -160,29 +175,33 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
||||
`SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it.
|
||||
- Checks the client's own address against the static lists, the three netblock
|
||||
settings below, before anything else, its lookup included. A client in
|
||||
`SWWAF_ALLOW_NETS` skips bans, the country lists, the rate limits and the rule
|
||||
files, and is not looked up; the timeouts and size limits still apply. A
|
||||
client in `SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE` before its
|
||||
body is read, and the request is not counted for the rate limits; an address
|
||||
in `SWWAF_ALLOW_NETS` too is let through. A client in
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
|
||||
limits; the country lists, the rule files and bans still apply to it.
|
||||
`SWWAF_ALLOW_NETS` skips bans, the country lists, the rate limits, the byte
|
||||
limits and the rule files, and is not looked up; the timeouts and size limits
|
||||
still apply. A client in `SWWAF_DENY_NETS` is refused with
|
||||
`SWWAF_BAN_RESPONSE` before its body is read, and the request is not counted
|
||||
for the rate limits; an address in `SWWAF_ALLOW_NETS` too is let through. A
|
||||
client in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the
|
||||
rate limits, and has no bytes counted by the byte limits; the country lists,
|
||||
the rule files and bans still apply to it.
|
||||
- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
|
||||
that `SWWAF_DENY_NETS`, a ban, the country lists, a rate limit or a rule would
|
||||
refuse: it passes them to the app, and their log lines name what `enforce`
|
||||
mode would have done (see `would_action` in "Request log" below). The checks
|
||||
run, and requests are counted, as in `enforce` mode, with three differences:
|
||||
neither a broken rate limit nor a `ban` rule makes a ban; a broken rate limit
|
||||
does not set the client's counters back to zero, so each request over the
|
||||
limit is logged as one that would be refused; and a request under a ban does
|
||||
not make it permanent. A ban it would have made, or made permanent, raises the
|
||||
alert `enforce` mode would have raised, marked as what would have happened
|
||||
(see "Alerts" below). The bans in `bans.json` are kept, and refuse requests
|
||||
again when `smallwebwaf` next runs in `enforce` mode, as long as they last.
|
||||
The timeouts and size limits still apply, since they protect `smallwebwaf` and
|
||||
the app themselves, and a request for one of `smallwebwaf`'s own endpoints
|
||||
without its token is still answered `401`. It is for trying a configuration
|
||||
before enforcing it.
|
||||
run, and requests and bytes are counted, as in `enforce` mode, with three
|
||||
differences: neither a broken rate limit or byte limit nor a `ban` rule makes
|
||||
a ban; a broken limit does not set the client's counters back to zero, so each
|
||||
request over a rate limit is logged as one that would be refused, and each
|
||||
whose bytes keep the client over a byte limit as breaking it; and a request
|
||||
under a ban does not make it permanent. As in `enforce` mode, the bytes
|
||||
counted are only those of the requests `enforce` mode would have passed to the
|
||||
app. A ban it would have made, or made permanent, raises the alert `enforce`
|
||||
mode would have raised, marked as what would have happened (see "Alerts"
|
||||
below). The bans in `bans.json` are kept, and refuse requests again when
|
||||
`smallwebwaf` next runs in `enforce` mode, as long as they last. The timeouts
|
||||
and size limits still apply, since they protect `smallwebwaf` and the app
|
||||
themselves, and a request for one of `smallwebwaf`'s own endpoints without its
|
||||
token is still answered `401`. It is for trying a configuration before
|
||||
enforcing it.
|
||||
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
|
||||
check and without asking the app, for the image's health check.
|
||||
- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
|
||||
@@ -254,10 +273,11 @@ effective settings are logged at start.
|
||||
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
|
||||
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
|
||||
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
|
||||
country lists, the rate limits and the rule files, such as your monitoring or
|
||||
your own networks.
|
||||
country lists, the rate limits, the byte limits and the rule files, such as
|
||||
your monitoring or your own networks.
|
||||
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
|
||||
rate limits do not apply to, such as a machine that talks to the app all day.
|
||||
rate limits and the byte limits do not apply to, such as a machine that talks
|
||||
to the app all day.
|
||||
- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
|
||||
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR`
|
||||
(default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`): the most
|
||||
@@ -265,19 +285,29 @@ effective settings are logged at start.
|
||||
several times what one busy person produces, since a browser loading a heavy
|
||||
page makes a few hundred requests and several people often share one address.
|
||||
- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes whose requests
|
||||
the rate limits neither count nor refuse, such as `/assets/` for static
|
||||
assets; each starts with `/`. A request whose path, percent-decoded, contains
|
||||
`..` anywhere or a backslash, or whose path as sent holds an encoded slash
|
||||
(`%2F` or `%2f`), is never exempt, since the app may act on it as a path
|
||||
outside every prefix: `/assets/..%2Flogin` as `/login`. Any other request is
|
||||
exempt when its path as sent, the path the app receives, before any query
|
||||
string and not percent-decoded, starts with a prefix, character for character.
|
||||
`/assets/` matches `/assets/app.js` and `/assets/`, but not `/assets`,
|
||||
`/Assets/app.js`, `/%61ssets/app.js`, `/static/assets/app.js`,
|
||||
`/static/../assets/app.js` or `/assets%2Fapp.js`. A character the client sends
|
||||
percent-encoded, such as a space, is written percent-encoded in a prefix, as
|
||||
in `/my%20files/`, and there are no wildcards: `*` is a character like any
|
||||
other.
|
||||
the rate limits neither count nor refuse, and whose bytes the byte limits do
|
||||
not count, such as `/assets/` for static assets; each starts with `/`. A
|
||||
request whose path, percent-decoded, contains `..` anywhere or a backslash, or
|
||||
whose path as sent holds an encoded slash (`%2F` or `%2f`), is never exempt,
|
||||
since the app may act on it as a path outside every prefix:
|
||||
`/assets/..%2Flogin` as `/login`. Any other request is exempt when its path as
|
||||
sent, the path the app receives, before any query string and not
|
||||
percent-decoded, starts with a prefix, character for character. `/assets/`
|
||||
matches `/assets/app.js` and `/assets/`, but not `/assets`, `/Assets/app.js`,
|
||||
`/%61ssets/app.js`, `/static/assets/app.js`, `/static/../assets/app.js` or
|
||||
`/assets%2Fapp.js`. A character the client sends percent-encoded, such as a
|
||||
space, is written percent-encoded in a prefix, as in `/my%20files/`, and there
|
||||
are no wildcards: `*` is a character like any other.
|
||||
- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`), `SWWAF_BYTES_LIMIT_PER_HOUR`
|
||||
(default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `50G`): the most
|
||||
bytes a client may have counted in a minute, an hour and a day. A request's
|
||||
bytes are counted once its answer has ended, so each default is above the
|
||||
largest request body and the largest response together,
|
||||
`SWWAF_REQUEST_MAX_BYTES` and `SWWAF_RESPONSE_MAX_BYTES`: at the defaults no
|
||||
download breaks a limit on its own.
|
||||
- `SWWAF_BYTES_COUNT` (default `both`): which body bytes the byte limits count:
|
||||
`response` for those of the answers, `request` for those of the requests, or
|
||||
`both`.
|
||||
- `SWWAF_LOOKUP_SOURCE` (default `geojs`): where each client's AS number and
|
||||
country are looked up: `geojs`, the GeoJS web service, which is then told the
|
||||
address of every new visitor, `file`, the IPinfo Lite database file
|
||||
@@ -311,12 +341,12 @@ effective settings are logged at start.
|
||||
the client unanswered: traefik answers `502`, as it does whenever its backend
|
||||
drops a connection. A `block` rule always answers `403`.
|
||||
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
|
||||
limit.
|
||||
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit broken again
|
||||
within this time after a ban ended, other than one for a clear sign of attack,
|
||||
bans for three times as long as that ban.
|
||||
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken rate limit that
|
||||
would be longer is permanent instead.
|
||||
limit or byte limit.
|
||||
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit or byte limit
|
||||
broken again within this time after a ban ended, other than one for a clear
|
||||
sign of attack, bans for three times as long as that ban.
|
||||
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken rate limit or byte
|
||||
limit that would be longer is permanent instead.
|
||||
- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
|
||||
attack.
|
||||
- `SWWAF_MAX_BANS` (default `5000`): the most bans `smallwebwaf` made that are
|
||||
@@ -410,15 +440,16 @@ effective settings are logged at start.
|
||||
|
||||
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
|
||||
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
|
||||
bytes). Rate limits are whole numbers of requests. Netblocks are in CIDR form,
|
||||
and a bare address stands for itself alone. Countries are the two-letter codes
|
||||
ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
|
||||
the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
|
||||
`su`, stops the start, and so does a code on both country lists. `off` switches
|
||||
a timeout, a size limit, a rate limit, `SWWAF_ALERT_COOLDOWN` or
|
||||
`SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
|
||||
`SWWAF_LOOKUP_TIMEOUT`, the ban settings, the state settings,
|
||||
`SWWAF_METRICS_TOP_N` and `SWWAF_LOG_REMOTE_BUFFER` cannot be off.
|
||||
bytes). Rate limits are whole numbers of requests, and byte limits are sizes.
|
||||
Netblocks are in CIDR form, and a bare address stands for itself alone.
|
||||
Countries are the two-letter codes ISO 3166-1 assigns today, and `xk` for
|
||||
Kosovo, in either case (`de` and `DE` are the same); any other code, such as
|
||||
`nk` (North Korea is `kp`) or the withdrawn `su`, stops the start, and so does a
|
||||
code on both country lists. `off` switches a timeout, a size limit, a rate
|
||||
limit, a byte limit, `SWWAF_ALERT_COOLDOWN` or `SWWAF_ALERT_MAX_PER_HOUR` off;
|
||||
`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, `SWWAF_LOOKUP_TIMEOUT`, the ban
|
||||
settings, the state settings, `SWWAF_METRICS_TOP_N` and
|
||||
`SWWAF_LOG_REMOTE_BUFFER` cannot be off.
|
||||
|
||||
Several limits are fixed rather than settings. At most 20,000 clients are kept,
|
||||
with their counters and history, and an IPv6 client is counted by its /64. At
|
||||
@@ -462,7 +493,7 @@ and for the container, `-v /srv/app/tokens:/etc/smallwebwaf/tokens:ro` and
|
||||
refused ones included:
|
||||
|
||||
```
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104}
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104}
|
||||
```
|
||||
|
||||
A field that does not apply to a request is left out of its line, apart from
|
||||
@@ -527,13 +558,20 @@ which every line has.
|
||||
health check, one from a client in `SWWAF_ALLOW_NETS` or
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_NETS`, one for a path that
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a ban
|
||||
or the country lists refuse, or would refuse in `observe` mode. The byte
|
||||
totals come with the byte limits.
|
||||
or the country lists refuse, or would refuse in `observe` mode. Its
|
||||
`minute_bytes`, `hour_bytes` and `day_bytes` give the client's bytes in each
|
||||
window as the byte limits count them, in the same way: for a request whose
|
||||
bytes they count, with its own, once it has ended; for any other, those
|
||||
counted before it.
|
||||
- `rule_ids` is there for a request that matched rules of the rule files, and
|
||||
lists their ids in the order they matched, up to the one that refused it.
|
||||
- `limit_hit` is there for a request that broke a rate limit, and names the
|
||||
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
|
||||
went over several. `offence` is then `limit`.
|
||||
- `limit_hit` is there for a request that broke a rate limit, or whose bytes
|
||||
broke a byte limit, and names the window whose limit it went over as `counts`
|
||||
names it: `minute`, `hour` or `day` for a rate limit, and `minute_bytes`,
|
||||
`hour_bytes` or `day_bytes` for a byte limit, the shortest if it went over
|
||||
several. `offence` is then `limit`. A request whose bytes broke a byte limit
|
||||
is not refused: its `action` is what it would have been otherwise, such as
|
||||
`forward`.
|
||||
- `ban_expires` is there for a request that made a ban or was refused under one,
|
||||
or in `observe` mode would have been refused under one, and gives when the ban
|
||||
ends, in the same form as `time`, or `permanent`.
|
||||
@@ -596,8 +634,8 @@ gives, as "Alert webhook schema" in [`SPEC.md`](SPEC.md) describes, and to
|
||||
it, as below. An alert is for one of these events, and is sent when
|
||||
`SWWAF_ALERT_EVENTS` names its event:
|
||||
|
||||
- `ban`: a ban `smallwebwaf` makes, for a broken rate limit or a clear sign of
|
||||
attack.
|
||||
- `ban`: a ban `smallwebwaf` makes, for a broken rate limit or byte limit or a
|
||||
clear sign of attack.
|
||||
- `permanent_ban`: a permanent ban it makes, or a ban for a clear sign of attack
|
||||
that a request made permanent.
|
||||
- `source_failure`: GeoJS failing or refusing `smallwebwaf`.
|
||||
@@ -633,6 +671,7 @@ is sent on one line:
|
||||
"asn": "",
|
||||
"as_name": "",
|
||||
"country": "",
|
||||
"kind": "requests",
|
||||
"limit": 1000,
|
||||
"window": "minute",
|
||||
"count": 1001,
|
||||
@@ -752,20 +791,23 @@ entries by client address, but for the alerts waiting, with times in UTC.
|
||||
|
||||
- `bans.json`: every ban with its notes, indented to be read. A permanent ban's
|
||||
`expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or
|
||||
`attack` for a clear sign of attack, for a ban `smallwebwaf` made, and `admin`
|
||||
for one you made or keep. Its `reason` is a short text: for a ban
|
||||
`smallwebwaf` made, the limit broken, such as
|
||||
`requests per minute over the limit of 1000`, or the rule that matched, such
|
||||
byte limit or `attack` for a clear sign of attack, for a ban `smallwebwaf`
|
||||
made, and `admin` for one you made or keep. Its `reason` is a short text: for
|
||||
a ban `smallwebwaf` made, the limit broken, such as
|
||||
`requests per minute over the limit of 1000` or
|
||||
`bytes per hour over the limit of 21474836480`, or the rule that matched, such
|
||||
as `matched the rule env-file`; for yours, what you wrote. Its `lifted` is
|
||||
when you lifted it, and is left out until you do.
|
||||
- `clients.json`: each client's two buckets in the minute, the hour and the day,
|
||||
and its history: when it was first and last seen, its AS number, AS name and
|
||||
country as last looked up and when the lookup gave them, its requests, how
|
||||
many were forwarded and how many refused (one `smallwebwaf` answered at its
|
||||
own endpoints is neither, unless it was refused with `401` for a missing or
|
||||
wrong token), the body bytes in each direction, its responses by status class
|
||||
and its offences by kind. Each client is on a line of its own, so `grep` shows
|
||||
everything about one.
|
||||
when you lifted it, and is left out until you do. The `kind` in the notes of a
|
||||
ban for a broken limit is `requests` or `bytes`, what the limit is on.
|
||||
- `clients.json`: each client's two buckets of requests in the minute, the hour
|
||||
and the day, its two buckets of bytes in each, `minute_bytes`, `hour_bytes`
|
||||
and `day_bytes`, and its history: when it was first and last seen, its AS
|
||||
number, AS name and country as last looked up and when the lookup gave them,
|
||||
its requests, how many were forwarded and how many refused (one `smallwebwaf`
|
||||
answered at its own endpoints is neither, unless it was refused with `401` for
|
||||
a missing or wrong token), the body bytes in each direction, its responses by
|
||||
status class and its offences by kind. Each client is on a line of its own, so
|
||||
`grep` shows everything about one.
|
||||
- `lookups.json`: GeoJS's answers, one to a line, each with the client's AS
|
||||
number, AS name and country, when GeoJS gave it and when it was last used.
|
||||
- `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
|
||||
@@ -801,9 +843,9 @@ message naming the file, and the line and column where Go's JSON decoder gives
|
||||
them; so does a state directory `smallwebwaf` cannot write. So does an entry
|
||||
without a field it needs, named with the entry's place in the file: a ban's
|
||||
`netblock`, `start` or `expires`, which is `null` for a permanent ban; a
|
||||
client's `client`, or the `start` of a window in which it has requests; an
|
||||
answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
|
||||
`answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
|
||||
client's `client`, or the `start` of a window in which it has requests or bytes;
|
||||
an answer's `client`, `country`, which is `""` for a client GeoJS cannot place,
|
||||
or `answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
|
||||
`time`. So does a ban whose `cause` is not `limit`, `attack` or `admin`, and
|
||||
alerts waiting for a destination that is not `webhook`, `slack` or `ntfy`. An
|
||||
answer's `asn` or `as_name` left out reads as empty.
|
||||
@@ -955,7 +997,8 @@ scraped, and keeps this one as `exported_instance` unless the scrape sets
|
||||
`smallwebwaf_upstream_duration_seconds`: how long those passed to the app took
|
||||
from then on, as histograms; `smallwebwaf_requests_in_flight`: the requests
|
||||
under way.
|
||||
- `smallwebwaf_rate_limit_hits_total` by `window`,
|
||||
- `smallwebwaf_rate_limit_hits_total` by `window`, `minute`, `hour` or `day`,
|
||||
and `kind`, `requests` for a rate limit or `bytes` for a byte limit,
|
||||
`smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose
|
||||
limit was passed, `smallwebwaf_offences_total` by `kind`, and
|
||||
`smallwebwaf_bans_made_total` by `cause`, `limit`, `attack` or `admin`, the
|
||||
@@ -1375,7 +1418,8 @@ addresses are never sent to GeoJS.
|
||||
body over the size limit; in `observe` mode, only for the size limit, with
|
||||
what it would have refused for noted in the log line. A request under
|
||||
`/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
|
||||
instead of reaching the app.
|
||||
instead of reaching the app. Once the answer to a request passed to the app
|
||||
has ended, `countBytes` counts its bytes for the byte limits.
|
||||
- `internal/metrics`: the metrics, counted as the other parts tell it what
|
||||
happened, and served in the Prometheus text format.
|
||||
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how
|
||||
@@ -1388,8 +1432,9 @@ addresses are never sent to GeoJS.
|
||||
client's history and to the notes of its bans; or in the lookup database,
|
||||
which it reads again when the file is replaced. `internal/lookup/lookuptest`
|
||||
writes lookup databases for the tests.
|
||||
- `internal/ratelimit`: the table of clients: counts each client's requests,
|
||||
tells when one takes it over a rate limit, and keeps each client's history.
|
||||
- `internal/ratelimit`: the table of clients: counts each client's requests and
|
||||
bytes, tells when they take it over a rate limit or a byte limit, and keeps
|
||||
each client's history.
|
||||
- `internal/state`: reads the state files at start, takes in an admin's edit of
|
||||
one while running, and writes them when they are due and at the stop.
|
||||
- `internal/requestlog`: the lines on stdout: the request log line and the
|
||||
|
||||
@@ -66,12 +66,15 @@ func TestReasonOfTheBansSmallwebwafMakes(t *testing.T) {
|
||||
ledger := bans.New(defaultRules())
|
||||
|
||||
limit, _ := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.1/32"), midnight(),
|
||||
bans.Notes{Limit: 1000, Window: "minute"})
|
||||
bans.Notes{Kind: "requests", Limit: 1000, Window: "minute"})
|
||||
byteLimit, _ := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.3/32"),
|
||||
midnight(), bans.Notes{Kind: "bytes", Limit: 10 << 30, Window: "hour"})
|
||||
attack, _ := ledger.BanForAttack(netip.MustParsePrefix("203.0.113.2/32"), midnight(),
|
||||
bans.Notes{RuleID: "git-dir", Target: "path"})
|
||||
|
||||
for _, tc := range []struct{ got, want string }{
|
||||
{limit.Reason, "requests per minute over the limit of 1000"},
|
||||
{byteLimit.Reason, "bytes per hour over the limit of 10737418240"},
|
||||
{attack.Reason, "matched the rule git-dir"},
|
||||
} {
|
||||
if tc.got != tc.want {
|
||||
|
||||
+18
-13
@@ -1,8 +1,8 @@
|
||||
// Package bans is the ban ledger: the bans smallwebwaf makes on the
|
||||
// netblocks of clients that break a rate limit or show a clear sign of
|
||||
// attack, and those an admin makes, with their notes, as the "Bans"
|
||||
// section of SPEC.md describes. The bans are kept in memory, and written
|
||||
// to bans.json and read from it by the state package.
|
||||
// netblocks of clients that break a rate limit or a byte limit or show a
|
||||
// clear sign of attack, and those an admin makes, with their notes, as
|
||||
// the "Bans" section of SPEC.md describes. The bans are kept in memory,
|
||||
// and written to bans.json and read from it by the state package.
|
||||
package bans
|
||||
|
||||
import (
|
||||
@@ -97,11 +97,14 @@ type Notes struct {
|
||||
ASN string `json:"asn"`
|
||||
ASName string `json:"as_name"`
|
||||
Country string `json:"country"`
|
||||
// Limit, Window and Count are, for a ban for a broken limit, the limit
|
||||
// that was broken, its window, "minute", "hour" or "day", and the
|
||||
// count reached: the client's requests in the window, the one that
|
||||
// broke the limit included. These are the requests that counted
|
||||
// toward the ban, and the window is the time over which they came.
|
||||
// Kind, Limit, Window and Count are, for a ban for a broken limit,
|
||||
// what the limit was on, "requests" for a rate limit or "bytes" for a
|
||||
// byte limit, the limit that was broken, its window, "minute", "hour"
|
||||
// or "day", and the count reached: the client's requests, or bytes, in
|
||||
// the window, those of the request that broke the limit included.
|
||||
// These are what counted toward the ban, and the window is the time
|
||||
// over which they came.
|
||||
Kind string `json:"kind,omitempty"`
|
||||
Limit int64 `json:"limit,omitempty"`
|
||||
Window string `json:"window,omitempty"`
|
||||
Count float64 `json:"count,omitempty"`
|
||||
@@ -109,8 +112,8 @@ type Notes struct {
|
||||
// of the rule file rule that matched, and its target.
|
||||
RuleID string `json:"rule_id,omitempty"`
|
||||
Target string `json:"target,omitempty"`
|
||||
// Request is the request that broke the limit, or that was the clear
|
||||
// sign of attack.
|
||||
// Request is the request that broke the limit, or whose bytes broke
|
||||
// it, or that was the clear sign of attack.
|
||||
Request Request `json:"request"`
|
||||
// Requests is how many requests the netblock has sent since it was
|
||||
// first seen, and Refused how many of them the ban has refused so
|
||||
@@ -260,7 +263,8 @@ func activeBan(bans []Ban, now time.Time) *Ban {
|
||||
// active, as when two of its requests break a limit at once, that ban is
|
||||
// returned with false, and no other is made. The ledger fills in the
|
||||
// notes' Refused and EarlierBans itself, and gives the ban the reason
|
||||
// "requests per <Window> over the limit of <Limit>", from the notes.
|
||||
// "<Kind> per <Window> over the limit of <Limit>", from the notes, such
|
||||
// as "requests per minute over the limit of 1000".
|
||||
func (l *Ledger) BanForLimit(
|
||||
netblock netip.Prefix, now time.Time, notes Notes,
|
||||
) (Ban, bool) {
|
||||
@@ -317,7 +321,8 @@ func (l *Ledger) WouldBePermanent(
|
||||
|
||||
// limitReason is the reason of a ban for a broken limit, with notes.
|
||||
func limitReason(notes Notes) string {
|
||||
return fmt.Sprintf("requests per %s over the limit of %d", notes.Window, notes.Limit)
|
||||
return fmt.Sprintf("%s per %s over the limit of %d",
|
||||
notes.Kind, notes.Window, notes.Limit)
|
||||
}
|
||||
|
||||
// attackReason is the reason of a ban for a clear sign of attack, with
|
||||
|
||||
@@ -349,7 +349,7 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
|
||||
|
||||
// As it ends, a clear sign of attack would ban for seven days, and a
|
||||
// limit broken again for three hours, but neither is made.
|
||||
limitNotes := bans.Notes{Limit: 1, Window: "minute"}
|
||||
limitNotes := bans.Notes{Kind: "requests", Limit: 1, Window: "minute"}
|
||||
attack, wouldAttack := ledger.WouldBanForAttack(netblock, first.Expires,
|
||||
bans.Notes{RuleID: "git-dir"})
|
||||
limit, wouldLimit := ledger.WouldBanForLimit(netblock, first.Expires, limitNotes)
|
||||
|
||||
@@ -91,6 +91,15 @@ type Config struct {
|
||||
// limits neither count nor refuse (SWWAF_RATE_LIMIT_EXEMPT_PATHS).
|
||||
// Each starts with /.
|
||||
RateLimitExemptPaths []string
|
||||
// BytesLimitPerMinute, BytesLimitPerHour and BytesLimitPerDay are the
|
||||
// most bytes a client's requests may carry in a minute, an hour and a
|
||||
// day (SWWAF_BYTES_LIMIT_PER_MINUTE, SWWAF_BYTES_LIMIT_PER_HOUR and
|
||||
// SWWAF_BYTES_LIMIT_PER_DAY). BytesCount is which body bytes count
|
||||
// toward them (SWWAF_BYTES_COUNT): response, request or both.
|
||||
BytesLimitPerMinute int64
|
||||
BytesLimitPerHour int64
|
||||
BytesLimitPerDay int64
|
||||
BytesCount string
|
||||
// LookupSource is where each client's AS number and country are
|
||||
// looked up (SWWAF_LOOKUP_SOURCE): geojs, file, or off for nowhere.
|
||||
// LookupDBPath is the lookup database, the IPinfo Lite file looked up
|
||||
@@ -266,6 +275,7 @@ var (
|
||||
"is not an absolute path, such as /var/lib/smallwebwaf")
|
||||
errShortToken = errors.New("is shorter than 32 characters")
|
||||
errNotMode = errors.New("is not enforce or observe")
|
||||
errNotBytesCount = errors.New("is not response, request or both")
|
||||
errNotPathPrefix = errors.New(
|
||||
"is not a path prefix starting with /, such as /assets/")
|
||||
errNotBoolean = errors.New("is not true or false")
|
||||
@@ -321,6 +331,10 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
||||
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "10000"),
|
||||
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "50000"),
|
||||
RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
|
||||
BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "10G"),
|
||||
BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"),
|
||||
BytesLimitPerDay: env.size("SWWAF_BYTES_LIMIT_PER_DAY", "50G"),
|
||||
BytesCount: env.bytesCount("SWWAF_BYTES_COUNT", "both"),
|
||||
LookupSource: env.lookupSource("SWWAF_LOOKUP_SOURCE", "geojs"),
|
||||
LookupDBPath: env.value("SWWAF_LOOKUP_DB_PATH", ""),
|
||||
LookupTimeout: env.durationNotOff("SWWAF_LOOKUP_TIMEOUT", "1s"),
|
||||
@@ -551,6 +565,17 @@ func (e *environment) count(name, defaultValue string) int64 {
|
||||
return count
|
||||
}
|
||||
|
||||
// bytesCount reads the setting that is which body bytes count toward the
|
||||
// byte limits: response, request or both.
|
||||
func (e *environment) bytesCount(name, defaultValue string) string {
|
||||
value := e.value(name, defaultValue)
|
||||
if value != "response" && value != "request" && value != "both" {
|
||||
e.check(name, fmt.Errorf("%q %w", value, errNotBytesCount))
|
||||
}
|
||||
|
||||
return value
|
||||
}
|
||||
|
||||
// pathPrefixes reads a setting that is a list of path prefixes.
|
||||
func (e *environment) pathPrefixes(name, defaultValue string) []string {
|
||||
prefixes, err := parsePathPrefixes(e.value(name, defaultValue))
|
||||
|
||||
@@ -40,6 +40,10 @@ const (
|
||||
rateLimitPerHour = "SWWAF_RATE_LIMIT_PER_HOUR"
|
||||
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
|
||||
rateLimitExemptPaths = "SWWAF_RATE_LIMIT_EXEMPT_PATHS"
|
||||
bytesLimitPerMinute = "SWWAF_BYTES_LIMIT_PER_MINUTE"
|
||||
bytesLimitPerHour = "SWWAF_BYTES_LIMIT_PER_HOUR"
|
||||
bytesLimitPerDay = "SWWAF_BYTES_LIMIT_PER_DAY"
|
||||
bytesCount = "SWWAF_BYTES_COUNT"
|
||||
lookupSource = "SWWAF_LOOKUP_SOURCE"
|
||||
lookupDBPath = "SWWAF_LOOKUP_DB_PATH"
|
||||
lookupTimeout = "SWWAF_LOOKUP_TIMEOUT"
|
||||
@@ -190,6 +194,10 @@ func TestDefaults(t *testing.T) {
|
||||
wantLookupSettings(t, cfg, config.Config{
|
||||
LookupSource: defaultLookupSource, LookupTimeout: time.Second,
|
||||
})
|
||||
wantByteLimitSettings(t, cfg, config.Config{
|
||||
BytesLimitPerMinute: 10 << 30, BytesLimitPerHour: 20 << 30,
|
||||
BytesLimitPerDay: 50 << 30, BytesCount: "both",
|
||||
})
|
||||
|
||||
if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" {
|
||||
t.Errorf("%s is %s", upstreamURL, cfg.UpstreamURL)
|
||||
@@ -220,6 +228,22 @@ func TestDefaults(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoSingleRequestBreaksAByteLimitAtTheDefaults(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := fromEnvironment(t, environment{})
|
||||
|
||||
// The largest request body and the largest response, both counted.
|
||||
largest := cfg.RequestMaxBytes + cfg.ResponseMaxBytes
|
||||
for _, limit := range []int64{
|
||||
cfg.BytesLimitPerMinute, cfg.BytesLimitPerHour, cfg.BytesLimitPerDay,
|
||||
} {
|
||||
if largest > limit {
|
||||
t.Errorf("a request of %d bytes breaks the byte limit of %d", largest, limit)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestValuesAsSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -302,6 +326,22 @@ func TestValuesAsSet(t *testing.T) {
|
||||
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
|
||||
}
|
||||
|
||||
func TestByteLimitSettingsAsSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := fromEnvironment(t, environment{
|
||||
bytesLimitPerMinute: "512M",
|
||||
bytesLimitPerHour: off,
|
||||
bytesLimitPerDay: "100000",
|
||||
bytesCount: "response",
|
||||
})
|
||||
|
||||
wantByteLimitSettings(t, cfg, config.Config{
|
||||
BytesLimitPerMinute: 512 << 20, BytesLimitPerHour: 0,
|
||||
BytesLimitPerDay: 100000, BytesCount: "response",
|
||||
})
|
||||
}
|
||||
|
||||
func TestRateLimitExemptPathsAsSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -996,6 +1036,9 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
|
||||
{rateLimitPerHour, "1.5"},
|
||||
{rateLimitPerDay, "-1"}, {rateLimitPerDay, "lots"},
|
||||
{rateLimitExemptPaths, "/assets/,,/static/"},
|
||||
{bytesLimitPerMinute, "10GB"}, {bytesLimitPerHour, "0"},
|
||||
{bytesLimitPerDay, "-1G"},
|
||||
{bytesCount, "all"}, {bytesCount, "Both"}, {bytesCount, ""},
|
||||
{lookupSource, "ipinfo"}, {lookupSource, "GeoJS"}, {lookupSource, ""},
|
||||
{lookupTimeout, off}, {lookupTimeout, "0s"}, {lookupTimeout, "1"},
|
||||
{addLookupHeaders, "yes"},
|
||||
@@ -1250,20 +1293,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := fromEnvironment(t, environment{clientRequestTimeout: "45s"})
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
slog.New(slog.NewJSONHandler(&out, nil)).Info("starting", "settings", cfg)
|
||||
|
||||
var line struct {
|
||||
Settings map[string]string `json:"settings"`
|
||||
}
|
||||
|
||||
err := json.Unmarshal(out.Bytes(), &line)
|
||||
if err != nil {
|
||||
t.Fatalf("decode %s: %v", out.Bytes(), err)
|
||||
}
|
||||
|
||||
hostname, _ := os.Hostname()
|
||||
|
||||
want := map[string]string{
|
||||
@@ -1286,6 +1315,10 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
rateLimitPerHour: "10000",
|
||||
rateLimitPerDay: "50000",
|
||||
rateLimitExemptPaths: "",
|
||||
bytesLimitPerMinute: "10G",
|
||||
bytesLimitPerHour: "20G",
|
||||
bytesLimitPerDay: "50G",
|
||||
bytesCount: "both",
|
||||
lookupSource: defaultLookupSource,
|
||||
lookupDBPath: "",
|
||||
lookupTimeout: "1s",
|
||||
@@ -1323,11 +1356,31 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
alertCooldown: defaultAlertCooldown,
|
||||
alertMaxPerHour: "60",
|
||||
}
|
||||
if !maps.Equal(line.Settings, want) {
|
||||
t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
|
||||
if got := loggedSettings(t, cfg); !maps.Equal(got, want) {
|
||||
t.Errorf("logged settings\n%v\nwant\n%v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// loggedSettings returns the settings as cfg logs them, each by its name.
|
||||
func loggedSettings(t *testing.T, cfg *config.Config) map[string]string {
|
||||
t.Helper()
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
slog.New(slog.NewJSONHandler(&out, nil)).Info("starting", "settings", cfg)
|
||||
|
||||
var line struct {
|
||||
Settings map[string]string `json:"settings"`
|
||||
}
|
||||
|
||||
err := json.Unmarshal(out.Bytes(), &line)
|
||||
if err != nil {
|
||||
t.Fatalf("decode %s: %v", out.Bytes(), err)
|
||||
}
|
||||
|
||||
return line.Settings
|
||||
}
|
||||
|
||||
// wantSettings checks the settings that are plain values.
|
||||
func wantSettings(t *testing.T, got *config.Config, want config.Config) {
|
||||
t.Helper()
|
||||
@@ -1351,6 +1404,21 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) {
|
||||
wantBanSettings(t, got, want)
|
||||
}
|
||||
|
||||
// wantByteLimitSettings checks the settings for the byte limits.
|
||||
func wantByteLimitSettings(t *testing.T, got *config.Config, want config.Config) {
|
||||
t.Helper()
|
||||
|
||||
if got.BytesLimitPerMinute != want.BytesLimitPerMinute ||
|
||||
got.BytesLimitPerHour != want.BytesLimitPerHour ||
|
||||
got.BytesLimitPerDay != want.BytesLimitPerDay ||
|
||||
got.BytesCount != want.BytesCount {
|
||||
t.Errorf("byte limits %d, %d and %d counting %s, want %d, %d and %d counting %s",
|
||||
got.BytesLimitPerMinute, got.BytesLimitPerHour, got.BytesLimitPerDay,
|
||||
got.BytesCount, want.BytesLimitPerMinute, want.BytesLimitPerHour,
|
||||
want.BytesLimitPerDay, want.BytesCount)
|
||||
}
|
||||
}
|
||||
|
||||
// wantLookupSettings checks the settings for lookups.
|
||||
func wantLookupSettings(t *testing.T, got *config.Config, want config.Config) {
|
||||
t.Helper()
|
||||
|
||||
@@ -6,6 +6,7 @@ package metrics
|
||||
import (
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
@@ -89,8 +90,9 @@ func New(topN int, instanceName string) *Metrics {
|
||||
Help: "How long requests passed to the app took, from then to their end.",
|
||||
}),
|
||||
rateLimitHits: counterVec("smallwebwaf_rate_limit_hits_total",
|
||||
"Requests that broke a rate limit, by its window.",
|
||||
[]string{"window"}),
|
||||
"Requests that broke a rate limit or a byte limit, by its window and "+
|
||||
"its kind, requests or bytes.",
|
||||
[]string{"window", "kind"}),
|
||||
sizeAndTimeLimitHits: counterVec("smallwebwaf_size_and_time_limit_hits_total",
|
||||
"Requests that passed a size or time limit, by its setting.",
|
||||
[]string{"limit"}),
|
||||
@@ -325,7 +327,15 @@ func (m *Metrics) RequestEnded(
|
||||
}
|
||||
|
||||
if line.LimitHit != "" {
|
||||
m.rateLimitHits.WithLabelValues(line.LimitHit).Inc()
|
||||
// The log line names a byte limit's window with _bytes after it.
|
||||
window, isBytes := strings.CutSuffix(line.LimitHit, "_bytes")
|
||||
|
||||
kind := ratelimit.KindRequests
|
||||
if isBytes {
|
||||
kind = ratelimit.KindBytes
|
||||
}
|
||||
|
||||
m.rateLimitHits.WithLabelValues(window, kind).Inc()
|
||||
}
|
||||
|
||||
if limit != "" {
|
||||
|
||||
@@ -203,7 +203,16 @@ func startWithAlerts(
|
||||
) (*sender, *clock, *proxy.Server, *alerts.Queue) {
|
||||
t.Helper()
|
||||
|
||||
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
||||
return startAppWithAlerts(t, func(http.ResponseWriter, *http.Request) {}, env)
|
||||
}
|
||||
|
||||
// startAppWithAlerts is startWithAlerts in front of the app handler.
|
||||
func startAppWithAlerts(
|
||||
t *testing.T, handler http.HandlerFunc, env map[string]string,
|
||||
) (*sender, *clock, *proxy.Server, *alerts.Queue) {
|
||||
t.Helper()
|
||||
|
||||
app := startApp(t, handler)
|
||||
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)}
|
||||
settings := map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
|
||||
+77
-22
@@ -6,6 +6,7 @@ import (
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
@@ -41,36 +42,92 @@ func (rq *request) banned(now time.Time) bool {
|
||||
|
||||
// limitBroken counts the request for the rate limits at now, notes the
|
||||
// client's counts for the log line, and reports whether the request takes
|
||||
// the client over a limit. In enforce mode such a request bans the
|
||||
// client's netblock, and sets the client's counters back to zero; in
|
||||
// observe mode it does neither, and raises the alert for the ban it would
|
||||
// have made, if that alert would be sent.
|
||||
// the client over a rate limit, which breaks it.
|
||||
func (rq *request) limitBroken(now time.Time) bool {
|
||||
group := clientGroup(rq.client)
|
||||
|
||||
counts, hit, over := rq.h.limiter.Count(group, now)
|
||||
counts, hit, over := rq.h.limiter.Count(clientGroup(rq.client), now)
|
||||
rq.line.Counts = counts
|
||||
|
||||
if !over {
|
||||
return false
|
||||
if over {
|
||||
rq.banForLimit(now, hit, rq.h.config.BanResponse)
|
||||
}
|
||||
|
||||
return over
|
||||
}
|
||||
|
||||
// countBytes counts the request's bytes for the byte limits, once its
|
||||
// response has ended, and notes the client's byte totals for the log line;
|
||||
// its requests stay there as the rate limits counted them. The bytes are
|
||||
// the response's body bytes, the request's, or both, as SWWAF_BYTES_COUNT
|
||||
// says; for an upgraded connection, such as a WebSocket, which has closed
|
||||
// by then, what it carried from the app counts with the response's and
|
||||
// what it carried from the client with the request's. Only a request
|
||||
// passed to the app has them counted, and only one the rate limits
|
||||
// counted; in observe mode, not one that enforce mode would have refused.
|
||||
// Bytes that take the client over a byte limit break it; the response was
|
||||
// passed on whole.
|
||||
func (rq *request) countBytes() {
|
||||
if !rq.counted || rq.line.WouldAction != "" {
|
||||
return
|
||||
}
|
||||
|
||||
response, request := rq.out.bytes, rq.requestBytes()
|
||||
if rq.upgraded != nil {
|
||||
response += rq.upgraded.fromApp.Load()
|
||||
request += rq.upgraded.toApp.Load()
|
||||
}
|
||||
|
||||
var bytes int64
|
||||
|
||||
switch rq.h.config.BytesCount {
|
||||
case "response":
|
||||
bytes = response
|
||||
case "request":
|
||||
bytes = request
|
||||
default: // both
|
||||
bytes = response + request
|
||||
}
|
||||
|
||||
now := rq.h.now()
|
||||
|
||||
counts, hit, over := rq.h.limiter.CountBytes(clientGroup(rq.client), now, bytes)
|
||||
rq.line.Counts.MinuteBytes = counts.MinuteBytes
|
||||
rq.line.Counts.HourBytes = counts.HourBytes
|
||||
rq.line.Counts.DayBytes = counts.DayBytes
|
||||
|
||||
if over {
|
||||
rq.banForLimit(now, hit, rq.out.status)
|
||||
}
|
||||
}
|
||||
|
||||
// banForLimit bans the client's netblock at now for a broken limit, the
|
||||
// one hit names, and notes the offence for the log line. status is what
|
||||
// the client was sent, or is sent: SWWAF_BAN_RESPONSE for a request over
|
||||
// a rate limit, the app's answer for one whose bytes broke a byte limit.
|
||||
// The ban sets the client's counters back to zero. In observe mode it
|
||||
// makes no ban and sets nothing back, and raises the alert for the ban it
|
||||
// would have made, if that alert would be sent.
|
||||
func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
|
||||
rq.line.LimitHit = hit.Window
|
||||
if hit.Kind == ratelimit.KindBytes {
|
||||
rq.line.LimitHit += "_bytes" // as counts names the byte totals
|
||||
}
|
||||
|
||||
rq.line.Offence = requestlog.OffenceLimit
|
||||
|
||||
netblock := rq.h.netblock(rq.client)
|
||||
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseLimit) {
|
||||
return true
|
||||
return
|
||||
}
|
||||
|
||||
notes := bans.Notes{
|
||||
ASN: rq.line.ASN,
|
||||
ASName: rq.line.ASName,
|
||||
Country: rq.line.Country,
|
||||
Kind: hit.Kind,
|
||||
Limit: hit.Limit,
|
||||
Window: hit.Window,
|
||||
Count: hit.Requests,
|
||||
Request: rq.noted(now),
|
||||
Count: hit.Count,
|
||||
Request: rq.noted(now, status),
|
||||
Requests: rq.netblockRequests(netblock),
|
||||
}
|
||||
|
||||
@@ -80,18 +137,16 @@ func (rq *request) limitBroken(now time.Time) bool {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return true
|
||||
return
|
||||
}
|
||||
|
||||
ban, made := rq.h.ledger.BanForLimit(netblock, now, notes)
|
||||
rq.h.limiter.Reset(group)
|
||||
rq.h.limiter.Reset(clientGroup(rq.client))
|
||||
rq.line.BanExpires = banExpires(ban)
|
||||
|
||||
if made {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// banForAttack bans the client's netblock at now for a clear sign of
|
||||
@@ -110,7 +165,7 @@ func (rq *request) banForAttack(now time.Time, rule rules.Rule) {
|
||||
Country: rq.line.Country,
|
||||
RuleID: rule.ID,
|
||||
Target: rule.Target,
|
||||
Request: rq.noted(now),
|
||||
Request: rq.noted(now, rq.h.config.BanResponse),
|
||||
Requests: rq.netblockRequests(netblock),
|
||||
}
|
||||
|
||||
@@ -177,16 +232,16 @@ func (rq *request) alertBan(ban bans.Ban) {
|
||||
})
|
||||
}
|
||||
|
||||
// noted is the request, refused at now with SWWAF_BAN_RESPONSE, or in
|
||||
// observe mode as it would have been, as the notes of the ban it makes
|
||||
// keep it.
|
||||
func (rq *request) noted(now time.Time) bans.Request {
|
||||
// noted is the request, at now, with status, what the client was sent, or
|
||||
// in observe mode would have been, as the notes of the ban it makes keep
|
||||
// it.
|
||||
func (rq *request) noted(now time.Time, status int) bans.Request {
|
||||
return bans.Request{
|
||||
Time: now,
|
||||
Method: rq.in.Method,
|
||||
Host: rq.in.Host,
|
||||
Path: rq.in.URL.RequestURI(),
|
||||
Status: rq.h.config.BanResponse,
|
||||
Status: status,
|
||||
UserAgent: rq.in.UserAgent(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -284,6 +284,7 @@ func TestBanNotes(t *testing.T) {
|
||||
ASN: asnDE,
|
||||
ASName: asNameDE,
|
||||
Country: "DE",
|
||||
Kind: "requests",
|
||||
Limit: 1,
|
||||
Window: minute,
|
||||
Count: 2,
|
||||
|
||||
@@ -103,6 +103,48 @@ func (b *responseBody) Close() error {
|
||||
return b.body.Close()
|
||||
}
|
||||
|
||||
// upgradedConn is the connection to the app once the app has switched
|
||||
// protocols, as for a WebSocket. ReverseProxy writes to it what the client
|
||||
// sends and reads from it what the app sends, on goroutines of its own,
|
||||
// until the connection closes; it counts the bytes each way, for the byte
|
||||
// limits.
|
||||
type upgradedConn struct {
|
||||
io.ReadWriteCloser
|
||||
|
||||
// fromApp is how many bytes the app has sent, and toApp how many the
|
||||
// client has.
|
||||
fromApp atomic.Int64
|
||||
toApp atomic.Int64
|
||||
}
|
||||
|
||||
// Read reads what the app sends.
|
||||
func (c *upgradedConn) Read(p []byte) (int, error) {
|
||||
n, err := c.ReadWriteCloser.Read(p)
|
||||
c.fromApp.Add(int64(n))
|
||||
|
||||
return n, err
|
||||
}
|
||||
|
||||
// Write sends the app what the client sent.
|
||||
func (c *upgradedConn) Write(p []byte) (int, error) {
|
||||
n, err := c.ReadWriteCloser.Write(p)
|
||||
c.toApp.Add(int64(n))
|
||||
|
||||
return n, err
|
||||
}
|
||||
|
||||
// CloseWrite tells the app that the client sends no more, while what the
|
||||
// app sends still passes. ReverseProxy calls it once the client has
|
||||
// stopped sending, and closes the connection there if it is not supported.
|
||||
func (c *upgradedConn) CloseWrite() error {
|
||||
conn, ok := c.ReadWriteCloser.(interface{ CloseWrite() error })
|
||||
if !ok {
|
||||
return http.ErrNotSupported
|
||||
}
|
||||
|
||||
return conn.CloseWrite()
|
||||
}
|
||||
|
||||
// limitBody returns body, cut off with an *http.MaxBytesError after
|
||||
// maxBytes, or unchanged if maxBytes is zero, which is off.
|
||||
func limitBody(body io.ReadCloser, maxBytes int64) io.ReadCloser {
|
||||
|
||||
@@ -0,0 +1,508 @@
|
||||
package proxy_test
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// The byte limit settings.
|
||||
const (
|
||||
bytesLimitPerMinute = "SWWAF_BYTES_LIMIT_PER_MINUTE"
|
||||
bytesLimitPerHour = "SWWAF_BYTES_LIMIT_PER_HOUR"
|
||||
bytesLimitPerDay = "SWWAF_BYTES_LIMIT_PER_DAY"
|
||||
bytesCount = "SWWAF_BYTES_COUNT"
|
||||
)
|
||||
|
||||
// The values of SWWAF_BYTES_COUNT.
|
||||
const (
|
||||
countResponse = "response"
|
||||
countRequest = "request"
|
||||
countBoth = "both"
|
||||
)
|
||||
|
||||
const (
|
||||
// bodyBytes is the size of the body of each request these tests send
|
||||
// with one, and answerBytes that of each answer of the app.
|
||||
bodyBytes = 30
|
||||
answerBytes = 70
|
||||
// byteLimit is the byte limit these tests set, as a setting: a request
|
||||
// with a body and its answer, 100 bytes, go over it.
|
||||
byteLimit = "99"
|
||||
// minuteBytes is limit_hit for SWWAF_BYTES_LIMIT_PER_MINUTE.
|
||||
minuteBytes = "minute_bytes"
|
||||
)
|
||||
|
||||
func TestEachByteLimitBansOnceTheResponseHasEnded(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const scraper = "192.0.2.200"
|
||||
|
||||
for _, tc := range []struct {
|
||||
setting, window string
|
||||
// apart is the time between the two requests, which the window
|
||||
// still covers.
|
||||
apart time.Duration
|
||||
}{
|
||||
{bytesLimitPerMinute, minute, 0},
|
||||
{bytesLimitPerHour, "hour", 2 * time.Minute},
|
||||
{bytesLimitPerDay, "day", 2 * time.Hour},
|
||||
} {
|
||||
t.Run(tc.setting, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk := startWithAnswers(t, map[string]string{
|
||||
tc.setting: byteLimit, metricsToken: token,
|
||||
})
|
||||
|
||||
// 70 bytes are within the limit of 99.
|
||||
line, _ := s.download()
|
||||
if line.LimitHit != "" || line.Offence != "" {
|
||||
t.Errorf("log line has limit_hit %q and offence %q, want neither",
|
||||
line.LimitHit, line.Offence)
|
||||
}
|
||||
|
||||
// 140 bytes are over it. The response is passed on whole, and
|
||||
// then bans the client for an hour.
|
||||
clk.advance(tc.apart)
|
||||
expires := requestlog.FormatTime(clk.Now().Add(time.Hour))
|
||||
|
||||
line, got := s.download()
|
||||
if got.err != nil || len(got.body) != answerBytes ||
|
||||
line.ResponseBytes != answerBytes {
|
||||
t.Errorf("got %d bytes (%v), and the log line has response_bytes %d, "+
|
||||
"want %d", len(got.body), got.err, line.ResponseBytes, answerBytes)
|
||||
}
|
||||
|
||||
if line.LimitHit != tc.window+"_bytes" || line.Offence != requestlog.OffenceLimit ||
|
||||
line.BanExpires != expires {
|
||||
t.Errorf("log line has limit_hit %q, offence %q and ban_expires %q, "+
|
||||
"want %s_bytes, limit and %s", line.LimitHit, line.Offence,
|
||||
line.BanExpires, tc.window, expires)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
|
||||
wantMetric(t, s.scrape(scraper), `smallwebwaf_rate_limit_hits_total{`+
|
||||
`instance="`+alertInstance+`",kind="bytes",window="`+tc.window+`"}`, 1)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResponseOverAByteLimitByItselfIsPassedOnWhole(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{bytesLimitPerMinute: "50"})
|
||||
|
||||
// The answer's 70 bytes are over the limit of 50 on their own.
|
||||
line, got := s.download()
|
||||
if got.err != nil || len(got.body) != answerBytes || line.LimitHit != minuteBytes {
|
||||
t.Errorf("got %d bytes (%v), and the log line has limit_hit %q, want %d and %s",
|
||||
len(got.body), got.err, line.LimitHit, answerBytes, minuteBytes)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
}
|
||||
|
||||
func TestBytesOfAnAnswerThatBreaksOffAreCounted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, _, _ := startAppWithAlerts(t, breakOff, map[string]string{
|
||||
bytesLimitPerMinute: "50",
|
||||
})
|
||||
expires := requestlog.FormatTime(clk.Now().Add(time.Hour))
|
||||
|
||||
// The 70 bytes passed on before the app broke off are over the limit of
|
||||
// 50, and ban the client for an hour.
|
||||
line, got := s.requestWithBody(http.MethodGet, client, "/", "", "", http.StatusOK,
|
||||
requestlog.ActionUpstreamError)
|
||||
if len(got.body) != answerBytes || line.LimitHit != minuteBytes ||
|
||||
line.BanExpires != expires {
|
||||
t.Errorf("got %d bytes, and the log line has limit_hit %q and ban_expires %q, "+
|
||||
"want %d, %s and %s", len(got.body), line.LimitHit, line.BanExpires,
|
||||
answerBytes, minuteBytes, expires)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
}
|
||||
|
||||
func TestWebSocketBytesAreCountedOnceItCloses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
setting string
|
||||
counted float64
|
||||
}{
|
||||
{countResponse, answerBytes},
|
||||
{countRequest, bodyBytes},
|
||||
{countBoth, bodyBytes + answerBytes},
|
||||
} {
|
||||
t.Run(tc.setting, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _, _ := startAppWithAlerts(t, answerAfterUpgrade, map[string]string{
|
||||
bytesLimitPerMinute: "29", bytesCount: tc.setting,
|
||||
})
|
||||
|
||||
// The client sends 30 bytes and the app 70, each over the limit
|
||||
// of 29, which bans the client once the WebSocket has closed.
|
||||
line := s.webSocket()
|
||||
if line.LimitHit != minuteBytes || line.Counts.MinuteBytes != tc.counted {
|
||||
t.Errorf("log line has limit_hit %q and minute_bytes %v, want %s and %v",
|
||||
line.LimitHit, line.Counts.MinuteBytes, minuteBytes, tc.counted)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestBytesCountSaysWhichBytesCount(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
setting string
|
||||
// each is the bytes each request counts, and breaking the request
|
||||
// that goes over the limit of 99.
|
||||
each float64
|
||||
breaking int
|
||||
}{
|
||||
{countResponse, answerBytes, 2},
|
||||
{countRequest, bodyBytes, 4},
|
||||
{countBoth, bodyBytes + answerBytes, 1},
|
||||
} {
|
||||
t.Run(tc.setting, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{
|
||||
bytesLimitPerMinute: byteLimit, bytesCount: tc.setting,
|
||||
})
|
||||
|
||||
for i := 1; i <= tc.breaking; i++ {
|
||||
line := s.upload()
|
||||
|
||||
want := ""
|
||||
if i == tc.breaking {
|
||||
want = minuteBytes
|
||||
}
|
||||
|
||||
counted := float64(i) * tc.each
|
||||
if line.LimitHit != want || line.Counts.MinuteBytes != counted {
|
||||
t.Errorf("request %d: log line has limit_hit %q and minute_bytes %v, "+
|
||||
"want %q and %v", i, line.LimitHit, line.Counts.MinuteBytes,
|
||||
want, counted)
|
||||
}
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestByteLimitsLeaveOutWhatTheRateLimitsLeaveOut(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
allowed = "192.0.2.7" // in SWWAF_ALLOW_NETS
|
||||
exempt = "192.0.2.10" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
|
||||
)
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{
|
||||
bytesLimitPerMinute: byteLimit,
|
||||
allowNets: allowed,
|
||||
rateLimitExemptNets: exempt,
|
||||
rateLimitExemptPaths: "/assets/",
|
||||
})
|
||||
|
||||
// Each sends 200 bytes, none of which is counted.
|
||||
for _, sent := range []struct{ from, path string }{
|
||||
{allowed, "/"}, {exempt, "/"}, {client, "/assets/app.js"},
|
||||
} {
|
||||
for range 2 {
|
||||
line, _ := s.requestWithBody(http.MethodPost, sent.from, sent.path,
|
||||
uploadHeader, uploadBody, http.StatusOK, requestlog.ActionForward)
|
||||
if _, counted := line.fields["counts"]; counted || line.LimitHit != "" {
|
||||
t.Errorf("%s %s: log line has counts %v and limit_hit %q, want neither",
|
||||
sent.from, sent.path, line.fields["counts"], line.LimitHit)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A path that is not exempt is counted, and breaks the limit.
|
||||
line := s.upload()
|
||||
if line.LimitHit != minuteBytes {
|
||||
t.Errorf("log line has limit_hit %q, want %s", line.LimitHit, minuteBytes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestByteLimitsOffCountTheBytesAndBanNoOne(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const off = "off"
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{
|
||||
bytesLimitPerMinute: off, bytesLimitPerHour: off, bytesLimitPerDay: off,
|
||||
})
|
||||
|
||||
for i := 1; i <= 3; i++ {
|
||||
line := s.upload()
|
||||
|
||||
counted := float64(i * (bodyBytes + answerBytes))
|
||||
if line.LimitHit != "" || line.Counts.MinuteBytes != counted ||
|
||||
line.Counts.HourBytes != counted || line.Counts.DayBytes != counted {
|
||||
t.Errorf("request %d: log line has limit_hit %q and counts %+v, "+
|
||||
"want none and %v bytes in each window", i, line.LimitHit,
|
||||
line.Counts, counted)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBanForABrokenByteLimitHasItsNotesAndItsAlert(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server, queue := startAppWithAlerts(t, readAndAnswer, map[string]string{
|
||||
bytesLimitPerMinute: byteLimit,
|
||||
})
|
||||
start := clk.Now()
|
||||
|
||||
s.requestWithBody(http.MethodPost, client, "/upload?part=1", uploadHeader,
|
||||
uploadBody, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
netblock := netip.MustParsePrefix(client + "/32")
|
||||
want := bans.Ban{
|
||||
Netblock: netblock,
|
||||
Start: start,
|
||||
Expires: start.Add(time.Hour),
|
||||
Cause: bans.CauseLimit,
|
||||
Reason: "bytes per minute over the limit of " + byteLimit,
|
||||
Notes: bans.Notes{
|
||||
Kind: "bytes",
|
||||
Limit: 99,
|
||||
Window: minute,
|
||||
Count: bodyBytes + answerBytes,
|
||||
// The request as it was answered, by the app.
|
||||
Request: bans.Request{
|
||||
Time: start,
|
||||
Method: http.MethodPost,
|
||||
Host: appHost,
|
||||
Path: "/upload?part=1",
|
||||
Status: http.StatusOK,
|
||||
UserAgent: userAgent,
|
||||
},
|
||||
Requests: 1,
|
||||
},
|
||||
}
|
||||
|
||||
got := server.Ledger.Bans(netblock)
|
||||
if len(got) != 1 || got[0] != want {
|
||||
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
|
||||
}
|
||||
|
||||
wantAlerts(t, queue, banAlert(alerts.EventBan, start, client, want,
|
||||
requestlog.FormatTime(want.Expires)))
|
||||
|
||||
if offences := historyOf(t, server, client).Offences.Limit; offences != 1 {
|
||||
t.Errorf("history counts %d offences for a limit, want 1", offences)
|
||||
}
|
||||
}
|
||||
|
||||
func TestObserveModeLogsAndAlertsAByteLimitAndBansNoOne(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server, queue := startAppWithAlerts(t, readAndAnswer, map[string]string{
|
||||
mode: observe,
|
||||
bytesLimitPerMinute: byteLimit,
|
||||
})
|
||||
start := clk.Now()
|
||||
|
||||
// No ban sets the client's counters back to zero, so each request
|
||||
// breaks the limit again. The answer is the app's either way, and the
|
||||
// alert for the ban is not sent twice within the cooldown.
|
||||
for range 2 {
|
||||
line := s.upload()
|
||||
wantWouldAction(t, line, "")
|
||||
|
||||
if line.LimitHit != minuteBytes || line.Offence != requestlog.OffenceLimit ||
|
||||
line.BanExpires != "" {
|
||||
t.Errorf("log line has limit_hit %q, offence %q and ban_expires %q, "+
|
||||
"want %s, limit and none", line.LimitHit, line.Offence, line.BanExpires,
|
||||
minuteBytes)
|
||||
}
|
||||
}
|
||||
|
||||
if held := server.Ledger.Snapshot(); len(held) != 0 {
|
||||
t.Errorf("the ledger holds %+v, want no ban", held)
|
||||
}
|
||||
|
||||
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
if len(waiting) != 1 {
|
||||
t.Fatalf("%d alerts wait, want 1: %+v", len(waiting), waiting)
|
||||
}
|
||||
|
||||
notes, _ := waiting[0].Detail["notes"].(bans.Notes)
|
||||
alert := banAlert(alerts.EventBan, start, client, bans.Ban{
|
||||
Netblock: netip.MustParsePrefix(client + "/32"), Cause: bans.CauseLimit,
|
||||
Reason: "bytes per minute over the limit of " + byteLimit, Notes: notes,
|
||||
}, requestlog.FormatTime(start.Add(time.Hour)))
|
||||
alert.Detail["mode"] = observe
|
||||
wantAlerts(t, queue, alert)
|
||||
}
|
||||
|
||||
func TestObserveModeLeavesOutTheBytesOfARequestEnforceModeRefuses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{
|
||||
mode: observe,
|
||||
rateLimitPerMinute: "1",
|
||||
bytesLimitPerMinute: "150",
|
||||
})
|
||||
|
||||
s.upload()
|
||||
|
||||
// The second request breaks the rate limit, which in enforce mode would
|
||||
// refuse it before the app sent anything, so its 100 bytes are not
|
||||
// counted, and the byte limit is not broken. Its line gives the bytes
|
||||
// counted before it.
|
||||
line := s.upload()
|
||||
wantWouldAction(t, line, requestlog.ActionRateLimited)
|
||||
|
||||
if line.LimitHit != minute || line.Counts.MinuteBytes != bodyBytes+answerBytes {
|
||||
t.Errorf("log line has limit_hit %q and minute_bytes %v, want minute and %d",
|
||||
line.LimitHit, line.Counts.MinuteBytes, bodyBytes+answerBytes)
|
||||
}
|
||||
}
|
||||
|
||||
// uploadHeader and uploadBody are the header and the body of a request
|
||||
// with a body of bodyBytes.
|
||||
//
|
||||
//nolint:gochecknoglobals // a constant cannot call strings.Repeat
|
||||
var (
|
||||
uploadHeader = "Content-Length: " + strconv.Itoa(bodyBytes)
|
||||
uploadBody = strings.Repeat("u", bodyBytes)
|
||||
)
|
||||
|
||||
// readAndAnswer is the app of these tests: it reads each request's whole
|
||||
// body and answers with answerBytes bytes.
|
||||
func readAndAnswer(w http.ResponseWriter, r *http.Request) {
|
||||
_, _ = io.Copy(io.Discard, r.Body)
|
||||
_, _ = io.WriteString(w, strings.Repeat("a", answerBytes))
|
||||
}
|
||||
|
||||
// breakOff is an app that announces an answer of twice answerBytes, and
|
||||
// breaks off after answerBytes.
|
||||
func breakOff(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Length", strconv.Itoa(2*answerBytes))
|
||||
_, _ = io.WriteString(w, strings.Repeat("a", answerBytes))
|
||||
}
|
||||
|
||||
// answerAfterUpgrade is an app that switches protocols, as for a
|
||||
// WebSocket, and then answers each line it receives with a line of
|
||||
// answerBytes.
|
||||
func answerAfterUpgrade(w http.ResponseWriter, _ *http.Request) {
|
||||
conn, buffered, err := http.NewResponseController(w).Hijack()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
defer func() {
|
||||
_ = conn.Close()
|
||||
}()
|
||||
|
||||
_, _ = buffered.WriteString("HTTP/1.1 101 Switching Protocols\r\n" +
|
||||
"Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
|
||||
_ = buffered.Flush()
|
||||
|
||||
for {
|
||||
_, err := buffered.ReadString('\n')
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
_, _ = buffered.WriteString(strings.Repeat("a", answerBytes-1) + "\n")
|
||||
_ = buffered.Flush()
|
||||
}
|
||||
}
|
||||
|
||||
// webSocket opens a WebSocket from client to answerAfterUpgrade, sends a
|
||||
// line of bodyBytes on it, reads the answer, and closes it. It checks the
|
||||
// answer, and the log line as request does, and returns the log line.
|
||||
func (s *sender) webSocket() logLine {
|
||||
s.t.Helper()
|
||||
|
||||
conn := dial(s.t, s.addr)
|
||||
send(s.t, conn, "GET /socket HTTP/1.1\r\nHost: "+appHost+"\r\n"+forwardedFor+
|
||||
": "+client+"\r\nConnection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
|
||||
|
||||
err := conn.SetReadDeadline(time.Now().Add(waitLimit))
|
||||
if err != nil {
|
||||
s.t.Fatalf("set read deadline: %v", err)
|
||||
}
|
||||
|
||||
reader := bufio.NewReader(conn)
|
||||
|
||||
res, err := http.ReadResponse(reader, nil)
|
||||
if err != nil {
|
||||
s.t.Fatalf("read the answer to the upgrade: %v", err)
|
||||
}
|
||||
|
||||
_ = res.Body.Close()
|
||||
|
||||
if res.StatusCode != http.StatusSwitchingProtocols {
|
||||
s.t.Fatalf("status %d, want %d", res.StatusCode, http.StatusSwitchingProtocols)
|
||||
}
|
||||
|
||||
send(s.t, conn, strings.Repeat("u", bodyBytes-1)+"\n")
|
||||
|
||||
got, err := reader.ReadString('\n')
|
||||
if err != nil || len(got) != answerBytes {
|
||||
s.t.Errorf("got %d bytes (%v), want %d", len(got), err, answerBytes)
|
||||
}
|
||||
|
||||
_ = conn.Close()
|
||||
|
||||
line := s.out.requestLines(s.t, s.sent+1)[s.sent]
|
||||
s.sent++
|
||||
wantLine(s.t, line, http.StatusSwitchingProtocols, requestlog.ActionForward)
|
||||
|
||||
return line
|
||||
}
|
||||
|
||||
// startWithAnswers is startAppWithAlerts in front of readAndAnswer, for a
|
||||
// test that looks at neither the server nor the alerts.
|
||||
func startWithAnswers(t *testing.T, env map[string]string) (*sender, *clock) {
|
||||
t.Helper()
|
||||
|
||||
s, clk, _, _ := startAppWithAlerts(t, readAndAnswer, env)
|
||||
|
||||
return s, clk
|
||||
}
|
||||
|
||||
// download sends a GET request for / from client, and checks that the
|
||||
// app's answer is passed on, as request does. It returns the log line and
|
||||
// the answer.
|
||||
func (s *sender) download() (logLine, answer) {
|
||||
s.t.Helper()
|
||||
|
||||
return s.requestWithBody(http.MethodGet, client, "/", "", "", http.StatusOK,
|
||||
requestlog.ActionForward)
|
||||
}
|
||||
|
||||
// upload is download for a POST request with a body of bodyBytes, and
|
||||
// returns the log line.
|
||||
func (s *sender) upload() logLine {
|
||||
s.t.Helper()
|
||||
|
||||
line, _ := s.requestWithBody(http.MethodPost, client, "/", uploadHeader,
|
||||
uploadBody, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
return line
|
||||
}
|
||||
@@ -222,8 +222,8 @@ func TestMetricsCountLimitsAndBans(t *testing.T) {
|
||||
metrics := s.scrape(scraper)
|
||||
wantMetric(t, metrics,
|
||||
`smallwebwaf_requests_total{action="denied",instance="app",status_class="none"}`, 1)
|
||||
wantMetric(t, metrics,
|
||||
`smallwebwaf_rate_limit_hits_total{instance="app",window="minute"}`, 1)
|
||||
wantMetric(t, metrics, `smallwebwaf_rate_limit_hits_total{instance="app",`+
|
||||
`kind="requests",window="minute"}`, 1)
|
||||
wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="limit"}`, 1)
|
||||
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 1)
|
||||
wantMetric(t, metrics, `smallwebwaf_active_bans{instance="app"}`, 1)
|
||||
@@ -238,8 +238,8 @@ func TestMetricsCountLimitsAndBans(t *testing.T) {
|
||||
s.get(client, 0, requestlog.ActionRateLimited)
|
||||
|
||||
metrics = s.scrape(scraper)
|
||||
wantMetric(t, metrics,
|
||||
`smallwebwaf_rate_limit_hits_total{instance="app",window="minute"}`, 2)
|
||||
wantMetric(t, metrics, `smallwebwaf_rate_limit_hits_total{instance="app",`+
|
||||
`kind="requests",window="minute"}`, 2)
|
||||
wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="limit"}`, 2)
|
||||
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 2)
|
||||
wantMetric(t, metrics, `smallwebwaf_active_bans{instance="app"}`, 1)
|
||||
|
||||
@@ -122,6 +122,8 @@ func wantAnswer(t *testing.T, got answer, body []byte) {
|
||||
func wantRequestFields(t *testing.T, line logLine, host string, sent, received int) {
|
||||
t.Helper()
|
||||
|
||||
bytes := float64(sent + received)
|
||||
|
||||
want := withTimings(line, requestlog.Line{
|
||||
Type: requestType, Time: line.Time, Instance: "app",
|
||||
ClientIP: localhost, Method: http.MethodPatch, Scheme: plain, Host: host,
|
||||
@@ -131,7 +133,10 @@ func wantRequestFields(t *testing.T, line logLine, host string, sent, received i
|
||||
RequestID: line.RequestID, PeerIP: localhost, ClientGroup: localhost + "/32",
|
||||
ContentLength: int64(sent), ResponseContentType: "text/plain; charset=utf-8",
|
||||
UpstreamStatus: http.StatusTeapot, Action: requestlog.ActionForward,
|
||||
Counts: ratelimit.Counts{Minute: 1, Hour: 1, Day: 1},
|
||||
Counts: ratelimit.Counts{
|
||||
Minute: 1, Hour: 1, Day: 1,
|
||||
MinuteBytes: bytes, HourBytes: bytes, DayBytes: bytes,
|
||||
},
|
||||
})
|
||||
if !reflect.DeepEqual(line.Line, want) {
|
||||
t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
|
||||
|
||||
+11
-3
@@ -103,9 +103,12 @@ func New(params Params) *Server {
|
||||
now: params.Now,
|
||||
metrics: m,
|
||||
limiter: ratelimit.New(ratelimit.Limits{
|
||||
PerMinute: params.Config.RateLimitPerMinute,
|
||||
PerHour: params.Config.RateLimitPerHour,
|
||||
PerDay: params.Config.RateLimitPerDay,
|
||||
PerMinute: params.Config.RateLimitPerMinute,
|
||||
PerHour: params.Config.RateLimitPerHour,
|
||||
PerDay: params.Config.RateLimitPerDay,
|
||||
BytesPerMinute: params.Config.BytesLimitPerMinute,
|
||||
BytesPerHour: params.Config.BytesLimitPerHour,
|
||||
BytesPerDay: params.Config.BytesLimitPerDay,
|
||||
}),
|
||||
ledger: bans.New(bans.Rules{
|
||||
LimitBanDuration: params.Config.LimitBanDuration,
|
||||
@@ -226,5 +229,10 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
// Once the response has ended, before the request is added to its
|
||||
// client's history. Deferred, since ReverseProxy panics to end a
|
||||
// response it cannot finish.
|
||||
defer rq.countBytes()
|
||||
|
||||
rq.forward(r.Context())
|
||||
}
|
||||
|
||||
+33
-17
@@ -3,6 +3,7 @@ package proxy
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptrace"
|
||||
"net/http/httputil"
|
||||
@@ -54,7 +55,10 @@ type request struct {
|
||||
// what the lookup gave then, the zero Answer while GeoJS had given none.
|
||||
lookedUp bool
|
||||
lookupAnswer lookup.Answer
|
||||
start time.Time
|
||||
// counted is true for a request the rate limits counted, whose bytes
|
||||
// the byte limits count once it has ended.
|
||||
counted bool
|
||||
start time.Time
|
||||
// checked is when the checks were done, and upstreamStart when the
|
||||
// request was handed to the app.
|
||||
checked time.Time
|
||||
@@ -65,6 +69,9 @@ type request struct {
|
||||
refused atomic.Pointer[refusal]
|
||||
// complete is true once the app's whole answer has been passed on.
|
||||
complete bool
|
||||
// upgraded is the connection to the app once the app has switched
|
||||
// protocols, as for a WebSocket, and nil otherwise.
|
||||
upgraded *upgradedConn
|
||||
|
||||
// mu guards what follows. The timeouts run on goroutines of their
|
||||
// own, and the transport starts and stops them, and notes the times
|
||||
@@ -205,8 +212,9 @@ func (rq *request) check(ctx context.Context) *refusal {
|
||||
// them refuses is not counted for the rate limits. Then come the rate
|
||||
// limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the
|
||||
// request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that
|
||||
// every other request is counted, and last the rule files. ctx is the
|
||||
// request's own context.
|
||||
// every other request is counted, and last the rule files. A request
|
||||
// exempt from the rate limits is exempt from the byte limits too. ctx is
|
||||
// the request's own context.
|
||||
func (rq *request) checkClient(ctx context.Context) string {
|
||||
cfg := rq.h.config
|
||||
if isInside(rq.client, cfg.AllowNets) {
|
||||
@@ -229,9 +237,9 @@ func (rq *request) checkClient(ctx context.Context) string {
|
||||
return requestlog.ActionCountryDenied
|
||||
}
|
||||
|
||||
exempt := isInside(rq.client, cfg.RateLimitExemptNets) ||
|
||||
pathExempt(rq.in.URL, cfg.RateLimitExemptPaths)
|
||||
if !exempt && rq.limitBroken(now) {
|
||||
rq.counted = !isInside(rq.client, cfg.RateLimitExemptNets) &&
|
||||
!pathExempt(rq.in.URL, cfg.RateLimitExemptPaths)
|
||||
if rq.counted && rq.limitBroken(now) {
|
||||
return requestlog.ActionRateLimited
|
||||
}
|
||||
|
||||
@@ -324,11 +332,18 @@ func (rq *request) modifyResponse(res *http.Response) error {
|
||||
if res.StatusCode == http.StatusSwitchingProtocols {
|
||||
// An upgraded connection, such as a WebSocket, is not cut by the
|
||||
// timeouts. ReverseProxy writes this answer straight to the
|
||||
// connection it takes over, not through rq.out.
|
||||
// connection it takes over, not through rq.out, and then copies
|
||||
// what passes each way through res.Body, the connection to the app.
|
||||
rq.stopTimers()
|
||||
rq.out.status = res.StatusCode
|
||||
rq.line.Websocket = true
|
||||
|
||||
conn, ok := res.Body.(io.ReadWriteCloser)
|
||||
if ok {
|
||||
rq.upgraded = &upgradedConn{ReadWriteCloser: conn}
|
||||
res.Body = rq.upgraded
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -431,10 +446,7 @@ func (rq *request) finish() {
|
||||
line.ResponseContentType = header.Get("Content-Type")
|
||||
line.CacheControl = header.Get("Cache-Control")
|
||||
line.Location = header.Get("Location")
|
||||
|
||||
if rq.body != nil {
|
||||
line.RequestBytes = rq.body.bytes.Load()
|
||||
}
|
||||
line.RequestBytes = rq.requestBytes()
|
||||
|
||||
// limit is the setting whose size or time limit the request passed.
|
||||
var limit string
|
||||
@@ -497,11 +509,6 @@ func timing(start, end time.Time) *float64 {
|
||||
// may have come before either was there, and one from GeoJS that comes
|
||||
// later is added when it comes.
|
||||
func (rq *request) addToHistory() {
|
||||
var requestBytes int64
|
||||
if rq.body != nil {
|
||||
requestBytes = rq.body.bytes.Load()
|
||||
}
|
||||
|
||||
forwarded := !rq.upstreamStart.IsZero()
|
||||
group := clientGroup(rq.client)
|
||||
|
||||
@@ -509,7 +516,7 @@ func (rq *request) addToHistory() {
|
||||
Forwarded: forwarded,
|
||||
Refused: !forwarded && rq.refused.Load() != nil,
|
||||
Status: rq.out.status,
|
||||
RequestBytes: requestBytes,
|
||||
RequestBytes: rq.requestBytes(),
|
||||
ResponseBytes: rq.out.bytes,
|
||||
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
|
||||
})
|
||||
@@ -531,6 +538,15 @@ func (rq *request) addToHistory() {
|
||||
}
|
||||
}
|
||||
|
||||
// requestBytes is how many bytes of the request's body have been read.
|
||||
func (rq *request) requestBytes() int64 {
|
||||
if rq.body == nil {
|
||||
return 0
|
||||
}
|
||||
|
||||
return rq.body.bytes.Load()
|
||||
}
|
||||
|
||||
// clientRequestDeadline is when the client must have sent its whole
|
||||
// request, or zero when SWWAF_CLIENT_REQUEST_TIMEOUT is off.
|
||||
func (rq *request) clientRequestDeadline() time.Time {
|
||||
|
||||
@@ -119,7 +119,10 @@ func wantFullLine(t *testing.T, line logLine) {
|
||||
ResponseContentType: "text/html", UpstreamStatus: http.StatusFound,
|
||||
CacheControl: "no-store", Location: "/elsewhere",
|
||||
Action: requestlog.ActionForward,
|
||||
Counts: ratelimit.Counts{Minute: 1, Hour: 1, Day: 1},
|
||||
// Its 3 bytes in and 5 out, each way counted by default.
|
||||
Counts: ratelimit.Counts{
|
||||
Minute: 1, Hour: 1, Day: 1, MinuteBytes: 8, HourBytes: 8, DayBytes: 8,
|
||||
},
|
||||
})
|
||||
if !reflect.DeepEqual(line.Line, want) {
|
||||
t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
|
||||
|
||||
+159
-78
@@ -1,9 +1,9 @@
|
||||
// Package ratelimit keeps the table of clients: each client's requests
|
||||
// counted over a minute, an hour and a day, as the "Counting method"
|
||||
// section of SPEC.md describes, which tell when a request takes the client
|
||||
// over a rate limit, and each client's history since it was first seen.
|
||||
// At most 20,000 clients are kept, in memory, and written to clients.json
|
||||
// and read from it by the state package.
|
||||
// and bytes counted over a minute, an hour and a day, as the "Counting
|
||||
// method" section of SPEC.md describes, which tell when a request takes
|
||||
// the client over a rate limit or a byte limit, and each client's history
|
||||
// since it was first seen. At most 20,000 clients are kept, in memory, and
|
||||
// written to clients.json and read from it by the state package.
|
||||
package ratelimit
|
||||
|
||||
import (
|
||||
@@ -23,19 +23,30 @@ const maxClients = 20000
|
||||
|
||||
const day = 24 * time.Hour
|
||||
|
||||
// The kinds of limits, as the metrics name them.
|
||||
const (
|
||||
// KindRequests is a rate limit, on a client's requests.
|
||||
KindRequests = "requests"
|
||||
// KindBytes is a byte limit, on a client's bytes.
|
||||
KindBytes = "bytes"
|
||||
)
|
||||
|
||||
// Limits are the most requests a client may make in a minute, an hour and
|
||||
// a day. Zero is no limit.
|
||||
// a day, and the most bytes. Zero is no limit.
|
||||
type Limits struct {
|
||||
PerMinute int64
|
||||
PerHour int64
|
||||
PerDay int64
|
||||
PerMinute int64
|
||||
PerHour int64
|
||||
PerDay int64
|
||||
BytesPerMinute int64
|
||||
BytesPerHour int64
|
||||
BytesPerDay int64
|
||||
}
|
||||
|
||||
// Limiter counts each client's requests against the limits, and keeps
|
||||
// its history. It is safe for concurrent use.
|
||||
// Limiter counts each client's requests and bytes against the limits, and
|
||||
// keeps its history. It is safe for concurrent use.
|
||||
type Limiter struct {
|
||||
// windows are the minute, the hour and the day, in the order of
|
||||
// Client.buckets.
|
||||
// Client.buckets and Client.byteBuckets.
|
||||
windows [3]window
|
||||
|
||||
mu sync.Mutex
|
||||
@@ -43,17 +54,23 @@ type Limiter struct {
|
||||
}
|
||||
|
||||
// Client is a client in the table, as clients.json holds it: its buckets
|
||||
// in each window, and its history.
|
||||
// of requests and of bytes in each window, and its history.
|
||||
//
|
||||
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
||||
type Client struct {
|
||||
Client netip.Prefix `json:"client"`
|
||||
Minute Buckets `json:"minute"`
|
||||
Hour Buckets `json:"hour"`
|
||||
Day Buckets `json:"day"`
|
||||
History History `json:"history"`
|
||||
Client netip.Prefix `json:"client"`
|
||||
Minute Buckets `json:"minute"`
|
||||
Hour Buckets `json:"hour"`
|
||||
Day Buckets `json:"day"`
|
||||
MinuteBytes Buckets `json:"minute_bytes"`
|
||||
HourBytes Buckets `json:"hour_bytes"`
|
||||
DayBytes Buckets `json:"day_bytes"`
|
||||
History History `json:"history"`
|
||||
}
|
||||
|
||||
// Buckets are a client's two buckets in one window: the requests in the
|
||||
// bucket under way, which began at Start, and in the bucket before it.
|
||||
// Buckets are a client's two buckets in one window: the requests, or the
|
||||
// bytes, in the bucket under way, which began at Start, and in the bucket
|
||||
// before it.
|
||||
type Buckets struct {
|
||||
Start time.Time `json:"start"`
|
||||
Current int64 `json:"current"`
|
||||
@@ -101,7 +118,7 @@ type Responses struct {
|
||||
|
||||
// Offences are a client's offences, by kind.
|
||||
type Offences struct {
|
||||
// Limit is its requests that broke a rate limit.
|
||||
// Limit is its requests that broke a rate limit or a byte limit.
|
||||
Limit int64 `json:"limit"`
|
||||
}
|
||||
|
||||
@@ -119,7 +136,8 @@ type Request struct {
|
||||
// and of its response.
|
||||
RequestBytes int64
|
||||
ResponseBytes int64
|
||||
// BrokeLimit is true for a request that broke a rate limit.
|
||||
// BrokeLimit is true for a request that broke a rate limit or a byte
|
||||
// limit.
|
||||
BrokeLimit bool
|
||||
}
|
||||
|
||||
@@ -132,62 +150,71 @@ func New(limits Limits) *Limiter {
|
||||
|
||||
return &Limiter{
|
||||
windows: [3]window{
|
||||
{name: "minute", length: time.Minute, limit: limits.PerMinute},
|
||||
{name: "hour", length: time.Hour, limit: limits.PerHour},
|
||||
{name: "day", length: day, limit: limits.PerDay},
|
||||
{
|
||||
name: "minute", length: time.Minute,
|
||||
limit: limits.PerMinute, byteLimit: limits.BytesPerMinute,
|
||||
},
|
||||
{
|
||||
name: "hour", length: time.Hour,
|
||||
limit: limits.PerHour, byteLimit: limits.BytesPerHour,
|
||||
},
|
||||
{
|
||||
name: "day", length: day,
|
||||
limit: limits.PerDay, byteLimit: limits.BytesPerDay,
|
||||
},
|
||||
},
|
||||
clients: clients,
|
||||
}
|
||||
}
|
||||
|
||||
// Hit is a request that takes a client over a rate limit.
|
||||
// Hit is a request that takes a client over a rate limit, or whose bytes
|
||||
// take it over a byte limit.
|
||||
type Hit struct {
|
||||
// Kind is KindRequests for a rate limit, KindBytes for a byte limit.
|
||||
Kind string
|
||||
// Window is "minute", "hour" or "day".
|
||||
Window string
|
||||
// Limit is the window's limit.
|
||||
Limit int64
|
||||
// Requests is the client's requests counted in the window, this one
|
||||
// included.
|
||||
Requests float64
|
||||
// Count is the client's requests, or bytes, counted in the window,
|
||||
// this request's included.
|
||||
Count float64
|
||||
}
|
||||
|
||||
// Counts are a client's requests in the minute, the hour and the day that
|
||||
// end at a request, that request included.
|
||||
// Counts are a client's requests and bytes in the minute, the hour and
|
||||
// the day that end at a request, that request's included.
|
||||
//
|
||||
//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
|
||||
type Counts struct {
|
||||
Minute float64 `json:"minute"`
|
||||
Hour float64 `json:"hour"`
|
||||
Day float64 `json:"day"`
|
||||
Minute float64 `json:"minute"`
|
||||
Hour float64 `json:"hour"`
|
||||
Day float64 `json:"day"`
|
||||
MinuteBytes float64 `json:"minute_bytes"`
|
||||
HourBytes float64 `json:"hour_bytes"`
|
||||
DayBytes float64 `json:"day_bytes"`
|
||||
}
|
||||
|
||||
// Count counts a request from client at now, in every window, whether or
|
||||
// not it is refused, and returns the client's requests in each window. It
|
||||
// reports whether the request takes the client over a limit, and the
|
||||
// window whose limit it goes over, the shortest if it is over several.
|
||||
// not it is refused, and returns the client's counts in each window. It
|
||||
// reports whether the request takes the client over a rate limit, and the
|
||||
// hit: the window whose limit it goes over, the shortest if it is over
|
||||
// several.
|
||||
func (l *Limiter) Count(client netip.Prefix, now time.Time) (Counts, Hit, bool) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
var (
|
||||
requests [3]float64
|
||||
hit Hit
|
||||
)
|
||||
|
||||
for i, b := range l.get(client).buckets() {
|
||||
w := l.windows[i]
|
||||
|
||||
requests[i] = b.add(now, w.length)
|
||||
if hit.Window == "" && w.limit > 0 && requests[i] > float64(w.limit) {
|
||||
hit = Hit{Window: w.name, Limit: w.limit, Requests: requests[i]}
|
||||
}
|
||||
}
|
||||
|
||||
counts := Counts{Minute: requests[0], Hour: requests[1], Day: requests[2]}
|
||||
|
||||
return counts, hit, hit.Window != ""
|
||||
return l.count(client, now, 1, 0)
|
||||
}
|
||||
|
||||
// Reset sets client's counts in every window back to zero. Its history
|
||||
// keeps its totals.
|
||||
// CountBytes counts bytes, those of a request from client that has ended,
|
||||
// at now, in every window, and returns the client's counts in each window.
|
||||
// It reports whether the bytes take the client over a byte limit, and the
|
||||
// hit, as Count does.
|
||||
func (l *Limiter) CountBytes(
|
||||
client netip.Prefix, now time.Time, bytes int64,
|
||||
) (Counts, Hit, bool) {
|
||||
return l.count(client, now, 0, bytes)
|
||||
}
|
||||
|
||||
// Reset sets client's counts of requests and of bytes in every window
|
||||
// back to zero. Its history keeps its totals.
|
||||
func (l *Limiter) Reset(client netip.Prefix) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
@@ -195,6 +222,7 @@ func (l *Limiter) Reset(client netip.Prefix) {
|
||||
c, seen := l.clients.Peek(client)
|
||||
if seen {
|
||||
c.Minute, c.Hour, c.Day = Buckets{}, Buckets{}, Buckets{}
|
||||
c.MinuteBytes, c.HourBytes, c.DayBytes = Buckets{}, Buckets{}, Buckets{}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -328,12 +356,13 @@ func (l *Limiter) Load(clients []Client, now time.Time) {
|
||||
l.clients.Purge()
|
||||
|
||||
for _, c := range clients {
|
||||
for i, b := range c.buckets() {
|
||||
// The window that ends at now covers neither bucket once it
|
||||
// begins after the bucket under way has ended.
|
||||
length := l.windows[i].length
|
||||
if !now.Add(-length).Before(b.Start.Add(length)) {
|
||||
*b = Buckets{}
|
||||
for i, w := range l.windows {
|
||||
for _, b := range []*Buckets{c.buckets()[i], c.byteBuckets()[i]} {
|
||||
// The window that ends at now covers neither bucket once it
|
||||
// begins after the bucket under way has ended.
|
||||
if !now.Add(-w.length).Before(b.Start.Add(w.length)) {
|
||||
*b = Buckets{}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -341,6 +370,49 @@ func (l *Limiter) Load(clients []Client, now time.Time) {
|
||||
}
|
||||
}
|
||||
|
||||
// count adds requests and bytes from client at now to its buckets in
|
||||
// every window, and returns its counts. A limit is broken only by what is
|
||||
// added to it, so that a request whose bytes are counted after another of
|
||||
// the client's requests broke a rate limit does not break it too.
|
||||
func (l *Limiter) count(
|
||||
client netip.Prefix, now time.Time, requests, bytes int64,
|
||||
) (Counts, Hit, bool) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
c := l.get(client)
|
||||
requestBuckets, byteBuckets := c.buckets(), c.byteBuckets()
|
||||
|
||||
var (
|
||||
requestCounts, byteCounts [3]float64
|
||||
hit Hit
|
||||
)
|
||||
|
||||
for i, w := range l.windows {
|
||||
requestCounts[i] = requestBuckets[i].add(now, w.length, requests)
|
||||
byteCounts[i] = byteBuckets[i].add(now, w.length, bytes)
|
||||
|
||||
switch {
|
||||
case hit.Window != "":
|
||||
case requests > 0 && w.limit > 0 && requestCounts[i] > float64(w.limit):
|
||||
hit = Hit{
|
||||
Kind: KindRequests, Window: w.name, Limit: w.limit, Count: requestCounts[i],
|
||||
}
|
||||
case bytes > 0 && w.byteLimit > 0 && byteCounts[i] > float64(w.byteLimit):
|
||||
hit = Hit{
|
||||
Kind: KindBytes, Window: w.name, Limit: w.byteLimit, Count: byteCounts[i],
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
counts := Counts{
|
||||
Minute: requestCounts[0], Hour: requestCounts[1], Day: requestCounts[2],
|
||||
MinuteBytes: byteCounts[0], HourBytes: byteCounts[1], DayBytes: byteCounts[2],
|
||||
}
|
||||
|
||||
return counts, hit, hit.Window != ""
|
||||
}
|
||||
|
||||
// get returns client's entry in the table, a new one if it has none, and
|
||||
// makes it the most recently seen.
|
||||
func (l *Limiter) get(client netip.Prefix) *Client {
|
||||
@@ -353,30 +425,39 @@ func (l *Limiter) get(client netip.Prefix) *Client {
|
||||
return c
|
||||
}
|
||||
|
||||
// buckets returns c's buckets in the minute, the hour and the day.
|
||||
// buckets returns c's buckets of requests in the minute, the hour and the
|
||||
// day.
|
||||
func (c *Client) buckets() [3]*Buckets {
|
||||
return [3]*Buckets{&c.Minute, &c.Hour, &c.Day}
|
||||
}
|
||||
|
||||
// window is a length of time over which requests are counted, and the
|
||||
// most requests a client may make in it.
|
||||
type window struct {
|
||||
name string
|
||||
length time.Duration
|
||||
limit int64
|
||||
// byteBuckets returns c's buckets of bytes in the minute, the hour and the
|
||||
// day.
|
||||
func (c *Client) byteBuckets() [3]*Buckets {
|
||||
return [3]*Buckets{&c.MinuteBytes, &c.HourBytes, &c.DayBytes}
|
||||
}
|
||||
|
||||
// add counts a request at now in a window of length, and returns the
|
||||
// client's requests in the window that ends at now: those in the bucket
|
||||
// under way, and those in the bucket before it weighted by how much of
|
||||
// that bucket the window still covers.
|
||||
// window is a length of time over which requests and bytes are counted,
|
||||
// and the most requests and the most bytes a client may have in it.
|
||||
type window struct {
|
||||
name string
|
||||
length time.Duration
|
||||
limit int64
|
||||
byteLimit int64
|
||||
}
|
||||
|
||||
// add counts n requests, or n bytes, at now in a window of length, and
|
||||
// returns the client's count in the window that ends at now: what is in
|
||||
// the bucket under way, and what is in the bucket before it weighted by
|
||||
// how much of that bucket the window still covers. With n zero it counts
|
||||
// nothing, and returns the count.
|
||||
//
|
||||
// Concurrent requests can be counted out of order, so now can be a moment
|
||||
// before the bucket under way began; such a request is counted in that
|
||||
// bucket. A request dated more than a second before it means the clock
|
||||
// was set back, and the buckets start afresh: otherwise the bucket before
|
||||
// would keep its full weight until the clock caught up.
|
||||
func (b *Buckets) add(now time.Time, length time.Duration) float64 {
|
||||
func (b *Buckets) add(now time.Time, length time.Duration, n int64) float64 {
|
||||
if now.Before(b.Start.Add(-time.Second)) {
|
||||
*b = Buckets{}
|
||||
}
|
||||
@@ -393,7 +474,7 @@ func (b *Buckets) add(now time.Time, length time.Duration) float64 {
|
||||
b.Current = 0
|
||||
}
|
||||
|
||||
b.Current++
|
||||
b.Current += n
|
||||
|
||||
elapsed := max(now.Sub(b.Start), 0)
|
||||
covered := 1 - float64(elapsed)/float64(length)
|
||||
|
||||
@@ -71,13 +71,116 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
|
||||
// Over both limits; the minute's is named, with the four requests.
|
||||
_, hit, over := limiter.Count(client, start)
|
||||
|
||||
want := ratelimit.Hit{Window: minute, Limit: limit, Requests: limit + 1}
|
||||
want := ratelimit.Hit{
|
||||
Kind: ratelimit.KindRequests, Window: minute, Limit: limit, Count: limit + 1,
|
||||
}
|
||||
if !over || hit != want {
|
||||
t.Errorf("request over the limit gives %+v and %t, want %+v and true",
|
||||
hit, over, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const byteLimit = 1000
|
||||
|
||||
for _, tc := range []struct {
|
||||
window string
|
||||
limits ratelimit.Limits
|
||||
}{
|
||||
{minute, ratelimit.Limits{BytesPerMinute: byteLimit}},
|
||||
{hour, ratelimit.Limits{BytesPerHour: byteLimit}},
|
||||
{"day", ratelimit.Limits{BytesPerDay: byteLimit}},
|
||||
} {
|
||||
t.Run(tc.window, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(tc.limits)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
// 600 bytes are within the limit, 600 more over it.
|
||||
_, _, over := limiter.CountBytes(client, midnight(), 600)
|
||||
if over {
|
||||
t.Fatal("600 bytes are over the limit of 1000")
|
||||
}
|
||||
|
||||
_, hit, over := limiter.CountBytes(client, midnight(), 600)
|
||||
|
||||
want := ratelimit.Hit{
|
||||
Kind: ratelimit.KindBytes, Window: tc.window, Limit: byteLimit, Count: 1200,
|
||||
}
|
||||
if !over || hit != want {
|
||||
t.Errorf("1200 bytes give %+v and %t, want %+v and true", hit, over, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000})
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
other := netip.MustParsePrefix("203.0.113.10/32")
|
||||
start := midnight()
|
||||
|
||||
// The third request breaks the rate limit. The bytes of a request
|
||||
// counted after it, within the byte limit, do not break it again.
|
||||
for range 2 {
|
||||
wantCount(t, limiter, client, start, "")
|
||||
}
|
||||
|
||||
wantCount(t, limiter, client, start, minute)
|
||||
wantBytesCount(t, limiter, client, start, 500, "")
|
||||
wantBytesCount(t, limiter, client, start, 600, ratelimit.KindBytes)
|
||||
|
||||
// Bytes over the byte limit do not have the next request break it, nor
|
||||
// the rate limit, which that request is within.
|
||||
wantBytesCount(t, limiter, other, start, 1200, ratelimit.KindBytes)
|
||||
wantCount(t, limiter, other, start, "")
|
||||
}
|
||||
|
||||
func TestCountGivesTheBytesInEachWindow(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{})
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
limiter.CountBytes(client, start, 300)
|
||||
|
||||
// A quarter into the next hour, the minute has only these 100 bytes.
|
||||
// The hour still covers three quarters of the bucket before, whose 300
|
||||
// bytes count 225, and these: 325. The day covers all 400.
|
||||
later := start.Add(time.Hour + time.Hour/4)
|
||||
limiter.CountBytes(client, later, 100)
|
||||
|
||||
// A request's counts give the bytes counted so far too.
|
||||
counts, _, _ := limiter.Count(client, later)
|
||||
|
||||
want := ratelimit.Counts{
|
||||
Minute: 1, Hour: 1, Day: 1, MinuteBytes: 100, HourBytes: 325, DayBytes: 400,
|
||||
}
|
||||
if counts != want {
|
||||
t.Errorf("counts %+v, want %+v", counts, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResetSetsTheBytesBackToZero(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000})
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
wantBytesCount(t, limiter, client, start, 1200, ratelimit.KindBytes)
|
||||
limiter.Reset(client)
|
||||
|
||||
// The client has its whole allowance of bytes again.
|
||||
wantBytesCount(t, limiter, client, start, 1000, "")
|
||||
}
|
||||
|
||||
func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -267,3 +370,18 @@ func wantCount(
|
||||
client, now.Format(time.RFC3339), hit.Window, want)
|
||||
}
|
||||
}
|
||||
|
||||
// wantBytesCount counts bytes from client at now, and checks the kind of
|
||||
// the limit they break, "" for none.
|
||||
func wantBytesCount(
|
||||
t *testing.T, limiter *ratelimit.Limiter, client netip.Prefix, now time.Time,
|
||||
bytes int64, want string,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
_, hit, _ := limiter.CountBytes(client, now, bytes)
|
||||
if hit.Kind != want {
|
||||
t.Errorf("%d bytes from %s at %s break a limit on %q, want %q",
|
||||
bytes, client, now.Format(time.RFC3339), hit.Kind, want)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -64,6 +64,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{})
|
||||
limiter.Count(client, start)
|
||||
limiter.CountBytes(client, start, 5)
|
||||
limiter.AddToHistory(client, start, ratelimit.Request{Forwarded: true})
|
||||
|
||||
loaded := func(now time.Time) ratelimit.Client {
|
||||
@@ -76,19 +77,25 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
|
||||
}
|
||||
|
||||
// Two minutes on, the window that ends then covers neither of the
|
||||
// minute's buckets, which are emptied; the hour's and the day's stay,
|
||||
// and so does the history.
|
||||
// minute's buckets, of requests and of bytes, which are emptied; the
|
||||
// hour's and the day's stay, and so does the history.
|
||||
got := loaded(start.Add(2 * time.Minute))
|
||||
if got.Minute != (ratelimit.Buckets{}) || got.Hour.Current != 1 ||
|
||||
got.Day.Current != 1 || got.History.Requests != 1 {
|
||||
t.Errorf("loaded two minutes on as %+v", got)
|
||||
}
|
||||
|
||||
if got.MinuteBytes != (ratelimit.Buckets{}) || got.HourBytes.Current != 5 ||
|
||||
got.DayBytes.Current != 5 {
|
||||
t.Errorf("loaded two minutes on with buckets of bytes %+v, %+v and %+v",
|
||||
got.MinuteBytes, got.HourBytes, got.DayBytes)
|
||||
}
|
||||
|
||||
// A moment before, the window still covers some of the earlier one.
|
||||
got = loaded(start.Add(2*time.Minute - time.Nanosecond))
|
||||
if got.Minute.Current != 1 {
|
||||
t.Errorf("loaded just under two minutes on with minute buckets %+v",
|
||||
got.Minute)
|
||||
if got.Minute.Current != 1 || got.MinuteBytes.Current != 5 {
|
||||
t.Errorf("loaded just under two minutes on with minute buckets %+v and %+v",
|
||||
got.Minute, got.MinuteBytes)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -45,7 +45,7 @@ const (
|
||||
)
|
||||
|
||||
// OffenceLimit is the offence a request line names for a request that
|
||||
// broke a rate limit.
|
||||
// broke a rate limit, or whose bytes broke a byte limit.
|
||||
const OffenceLimit = "limit"
|
||||
|
||||
// timeLayout is RFC 3339 with milliseconds.
|
||||
@@ -117,13 +117,16 @@ type Line struct {
|
||||
// ActionBanned, ActionCountryDenied, ActionRateLimited or
|
||||
// ActionRuleBlocked.
|
||||
WouldAction string `json:"would_action,omitempty"`
|
||||
// Counts are the client's requests as the rate limits counted them
|
||||
// with this one, for a request they counted.
|
||||
// Counts are, for a request the rate limits counted, the client's
|
||||
// requests as they counted them with this one, and its bytes as the
|
||||
// byte limits counted them, with this request's once it has ended if
|
||||
// they count them.
|
||||
Counts ratelimit.Counts `json:"counts,omitzero"`
|
||||
// RuleIDs are the ids of the rule file rules the request matched.
|
||||
RuleIDs []string `json:"rule_ids,omitempty"`
|
||||
// LimitHit is the window whose rate limit the request went over:
|
||||
// minute, hour or day.
|
||||
// LimitHit is the window whose limit the request went over, named as
|
||||
// Counts names its count: minute, hour or day for a rate limit, and
|
||||
// minute_bytes, hour_bytes or day_bytes for a byte limit.
|
||||
LimitHit string `json:"limit_hit,omitempty"`
|
||||
// Offence is the offence the request was held as, OffenceLimit.
|
||||
Offence string `json:"offence,omitempty"`
|
||||
|
||||
+10
-4
@@ -615,8 +615,8 @@ func (f *bansFile) check(data []byte) error {
|
||||
}
|
||||
|
||||
// check refuses a client without its address, which would count nobody's
|
||||
// requests, or with requests in a window but no start, which would drop
|
||||
// them and give the client a fresh allowance.
|
||||
// requests, or with requests or bytes in a window but no start, which
|
||||
// would drop them and give the client a fresh allowance.
|
||||
func (f *clientsFile) check([]byte) error {
|
||||
for i, client := range f.Clients {
|
||||
switch {
|
||||
@@ -628,6 +628,12 @@ func (f *clientsFile) check([]byte) error {
|
||||
return missing(i, "hour.start")
|
||||
case countsWithoutStart(client.Day):
|
||||
return missing(i, "day.start")
|
||||
case countsWithoutStart(client.MinuteBytes):
|
||||
return missing(i, "minute_bytes.start")
|
||||
case countsWithoutStart(client.HourBytes):
|
||||
return missing(i, "hour_bytes.start")
|
||||
case countsWithoutStart(client.DayBytes):
|
||||
return missing(i, "day_bytes.start")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -697,8 +703,8 @@ func (f *alertsFile) check([]byte) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// countsWithoutStart reports whether b holds requests but no start, which
|
||||
// places them in time.
|
||||
// countsWithoutStart reports whether b holds requests, or bytes, but no
|
||||
// start, which places them in time.
|
||||
func countsWithoutStart(b ratelimit.Buckets) bool {
|
||||
return b.Start.IsZero() && (b.Current != 0 || b.Previous != 0)
|
||||
}
|
||||
|
||||
@@ -392,6 +392,12 @@ func TestEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
|
||||
`"hour": {"current": 3}}]}`,
|
||||
`: entry 1 has no "hour.start"`,
|
||||
},
|
||||
{
|
||||
"a client with bytes in a window without its start", clientsJSON,
|
||||
`{"version": 1, "clients": [{"client": "203.0.113.9/32", ` +
|
||||
`"minute_bytes": {"previous": 5120}}]}`,
|
||||
`: entry 1 has no "minute_bytes.start"`,
|
||||
},
|
||||
{
|
||||
"an answer without a client", lookupsJSON,
|
||||
`{"version": 1, "lookups": [{"country": "DE", ` + answer + `}]}`,
|
||||
@@ -1317,6 +1323,7 @@ func fill(params state.Params) {
|
||||
params.Limiter.Count(netip.MustParsePrefix(c), now)
|
||||
}
|
||||
|
||||
params.Limiter.CountBytes(client, now, 8)
|
||||
params.Limiter.AddToHistory(client, now, ratelimit.Request{
|
||||
Forwarded: true, Status: 200, RequestBytes: 3, ResponseBytes: 5,
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user