Byte limits per client over a minute, an hour and a day (closes #20)
check / check (push) Waiting to run
check / check (push) Waiting to run
SWWAF_BYTES_LIMIT_PER_MINUTE, _PER_HOUR and _PER_DAY (10G, 20G, 50G) and SWWAF_BYTES_COUNT (both). A request's bytes are counted once its answer has ended, for a request passed to the app that the rate limits count; what a WebSocket carries each way, once it closes. Bytes over a limit ban the client as a broken rate limit does, and cut nothing short. clients.json keeps the byte buckets, the log line's counts carry the byte totals, ban notes say what the limit is on, and the limit hits metric is labelled by kind. Judgement call: limit_hit names a byte window minute_bytes, hour_bytes or day_bytes, as counts names the byte totals. Judgement call: in observe mode, the bytes of a request enforce mode would have refused are not counted. Model: opus-5-5
This commit is contained in:
@@ -22,29 +22,30 @@ fields, which come a little later, and the metrics endpoint and the header size
|
||||
and the idle time as settings, which come last in it. So are the four parts of
|
||||
the stage after it: the rule files, with the bans for a clear sign of attack,
|
||||
the other admin endpoints, alerts to all three destinations, a JSON webhook,
|
||||
Slack and ntfy, and remote log sending. So is the first part of the stage after
|
||||
Slack and ntfy, and remote log sending. So are two parts of the stage after
|
||||
that: the AS number and country of every client, looked up through GeoJS or in
|
||||
the IPinfo Lite database file. `smallwebwaf` passes each request to the app and
|
||||
the app's answer back, unchanged, within its timeouts and size limits, works out
|
||||
each client's address, looks up its AS number and country unless you switch that
|
||||
off, bans a client that sends too many requests, not counting those for the
|
||||
paths you choose, refuses a client that comes from a country you refuse or from
|
||||
a network you refuse, lets the networks you choose through, checks each request
|
||||
against the rule files and bans a client whose request is a clear sign of
|
||||
attack, keeps its bans, each client's counters and history, and GeoJS's answers
|
||||
in JSON files across restarts, takes in your edits of those files, such as a ban
|
||||
you make, keep or lift, and of the rule files while it runs, writes a JSON log
|
||||
line for every request, sends its log lines to a syslog server too if you name
|
||||
one, sends an alert to a webhook, to Slack and to ntfy, each if you name one,
|
||||
for each ban it makes or makes permanent, for GeoJS failing, for a rule file or
|
||||
state file with an error and for a replacement of the lookup database it cannot
|
||||
read, serves Prometheus metrics to a scraper that holds the metrics token, lets
|
||||
an admin who holds the admin token list, add and lift bans and ask what it knows
|
||||
of a client, and in `observe` mode passes on the requests it would refuse,
|
||||
logging what it would have done with them. It comes as the image the app's own
|
||||
image is built on. The rest of the design comes after that, in the order of the
|
||||
build order in [`SPEC.md`](SPEC.md). The survey of existing tools that led to
|
||||
the design is in [`EVALUATION.md`](EVALUATION.md).
|
||||
the IPinfo Lite database file, and the byte limits. `smallwebwaf` passes each
|
||||
request to the app and the app's answer back, unchanged, within its timeouts and
|
||||
size limits, works out each client's address, looks up its AS number and country
|
||||
unless you switch that off, bans a client that sends too many requests or too
|
||||
many bytes, not counting those for the paths you choose, refuses a client that
|
||||
comes from a country you refuse or from a network you refuse, lets the networks
|
||||
you choose through, checks each request against the rule files and bans a client
|
||||
whose request is a clear sign of attack, keeps its bans, each client's counters
|
||||
and history, and GeoJS's answers in JSON files across restarts, takes in your
|
||||
edits of those files, such as a ban you make, keep or lift, and of the rule
|
||||
files while it runs, writes a JSON log line for every request, sends its log
|
||||
lines to a syslog server too if you name one, sends an alert to a webhook, to
|
||||
Slack and to ntfy, each if you name one, for each ban it makes or makes
|
||||
permanent, for GeoJS failing, for a rule file or state file with an error and
|
||||
for a replacement of the lookup database it cannot read, serves Prometheus
|
||||
metrics to a scraper that holds the metrics token, lets an admin who holds the
|
||||
admin token list, add and lift bans and ask what it knows of a client, and in
|
||||
`observe` mode passes on the requests it would refuse, logging what it would
|
||||
have done with them. It comes as the image the app's own image is built on. The
|
||||
rest of the design comes after that, in the order of the build order in
|
||||
[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
|
||||
[`EVALUATION.md`](EVALUATION.md).
|
||||
|
||||
## Getting started
|
||||
|
||||
@@ -107,26 +108,40 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
||||
window still covers. At most 20,000 clients are kept, the least recently seen
|
||||
dropped first, with their history, and a restart gives no client a fresh
|
||||
allowance (see "State files" below).
|
||||
- Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md)
|
||||
describes: the first ban lasts an hour, and a limit broken again within a day
|
||||
of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and
|
||||
81 hours; a ban that would last longer than seven days is permanent instead. A
|
||||
ban covers the client's netblock: its IPv4 address, or the netblock around it
|
||||
that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 /64. While it lasts, every
|
||||
request from the netblock is refused with `SWWAF_BAN_RESPONSE` after the
|
||||
static lists and before the country lists, so the client is not looked up, and
|
||||
is not counted for the rate limits. A ban sets the client's counters back to
|
||||
zero. Each ban carries notes for deciding whether to lift it: the limit, its
|
||||
window and the requests counted in it, the request that broke it, the client's
|
||||
AS number, AS name and country once they are looked up, the netblock's
|
||||
requests since it was first seen, how many of them the ban has refused, and
|
||||
how many bans the netblock had before, for a broken limit, for a clear sign of
|
||||
attack and by an admin. At most `SWWAF_MAX_BANS` bans `smallwebwaf` made are
|
||||
kept, past, active and permanent; past that, the earliest such ban of the
|
||||
netblock that has gone longest without a request is dropped first. The bans
|
||||
whose cause is `admin`, those you make or keep, are kept besides, and never
|
||||
dropped. `bans.json` shows the bans and their notes, a restart lifts none, and
|
||||
you make, keep or lift a ban by editing it (see "State files" below).
|
||||
- Counts each client's bytes over a minute, an hour and a day, in the same way:
|
||||
once a request passed to the app has ended, the body bytes of its answer, of
|
||||
the request, or of both, as `SWWAF_BYTES_COUNT` says. For a WebSocket, or any
|
||||
other upgraded connection, what it carried from the app counts with the
|
||||
answer, and what it carried from the client with the request, once it closes.
|
||||
Bytes that take the client over one of the byte limits below break that limit,
|
||||
and ban the client as a broken rate limit does, so that its next request is
|
||||
refused. The byte limits never cut an answer or an upgraded connection short:
|
||||
the one whose bytes break a limit has already been passed on, or has closed.
|
||||
They leave out what the rate limits leave out: a client in `SWWAF_ALLOW_NETS`
|
||||
or `SWWAF_RATE_LIMIT_EXEMPT_NETS`, and a request for a path
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts.
|
||||
- Bans a client that breaks a rate limit or a byte limit, as "Bans" in
|
||||
[`SPEC.md`](SPEC.md) describes: the first ban lasts an hour, and a limit
|
||||
broken again within a day of a ban ending bans for three times as long as that
|
||||
ban, so 1, 3, 9, 27 and 81 hours; a ban that would last longer than seven days
|
||||
is permanent instead. A ban covers the client's netblock: its IPv4 address, or
|
||||
the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 /64.
|
||||
While it lasts, every request from the netblock is refused with
|
||||
`SWWAF_BAN_RESPONSE` after the static lists and before the country lists, so
|
||||
the client is not looked up, and is not counted for the rate limits. A ban
|
||||
sets the client's counters back to zero. Each ban carries notes for deciding
|
||||
whether to lift it: the limit, whether it is on requests or bytes, its window
|
||||
and the requests or bytes counted in it, the request that broke it, the
|
||||
client's AS number, AS name and country once they are looked up, the
|
||||
netblock's requests since it was first seen, how many of them the ban has
|
||||
refused, and how many bans the netblock had before, for a broken limit, for a
|
||||
clear sign of attack and by an admin. At most `SWWAF_MAX_BANS` bans
|
||||
`smallwebwaf` made are kept, past, active and permanent; past that, the
|
||||
earliest such ban of the netblock that has gone longest without a request is
|
||||
dropped first. The bans whose cause is `admin`, those you make or keep, are
|
||||
kept besides, and never dropped. `bans.json` shows the bans and their notes, a
|
||||
restart lifts none, and you make, keep or lift a ban by editing it (see "State
|
||||
files" below).
|
||||
- Checks each request against the rules of the rule files (see "Rule files"
|
||||
below) after the rate limits, and before its body is read. A `log` rule that
|
||||
matches is noted in the log line; a `block` rule refuses the request with
|
||||
@@ -138,10 +153,10 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
||||
default, and any request from the netblock while it lasts makes it permanent.
|
||||
Once it has run out, the netblock is served like any other, but its next clear
|
||||
sign of attack bans it permanently at once. Such a ban covers the same
|
||||
netblock as a ban for a broken rate limit, does not set the client's counters
|
||||
back to zero, and does not make the netblock's next ban for a broken limit
|
||||
longer. Its notes give the id and the target of the rule that matched in place
|
||||
of the limit.
|
||||
netblock as a ban for a broken limit, does not set the client's counters back
|
||||
to zero, and does not make the netblock's next ban for a broken limit longer.
|
||||
Its notes give the id and the target of the rule that matched in place of the
|
||||
limit.
|
||||
- Looks up the AS number and country of every client through GeoJS, or in the
|
||||
IPinfo Lite database file while `SWWAF_LOOKUP_SOURCE` is `file`, after the
|
||||
static lists and bans, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Country and
|
||||
@@ -160,29 +175,33 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
||||
`SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it.
|
||||
- Checks the client's own address against the static lists, the three netblock
|
||||
settings below, before anything else, its lookup included. A client in
|
||||
`SWWAF_ALLOW_NETS` skips bans, the country lists, the rate limits and the rule
|
||||
files, and is not looked up; the timeouts and size limits still apply. A
|
||||
client in `SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE` before its
|
||||
body is read, and the request is not counted for the rate limits; an address
|
||||
in `SWWAF_ALLOW_NETS` too is let through. A client in
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
|
||||
limits; the country lists, the rule files and bans still apply to it.
|
||||
`SWWAF_ALLOW_NETS` skips bans, the country lists, the rate limits, the byte
|
||||
limits and the rule files, and is not looked up; the timeouts and size limits
|
||||
still apply. A client in `SWWAF_DENY_NETS` is refused with
|
||||
`SWWAF_BAN_RESPONSE` before its body is read, and the request is not counted
|
||||
for the rate limits; an address in `SWWAF_ALLOW_NETS` too is let through. A
|
||||
client in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the
|
||||
rate limits, and has no bytes counted by the byte limits; the country lists,
|
||||
the rule files and bans still apply to it.
|
||||
- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
|
||||
that `SWWAF_DENY_NETS`, a ban, the country lists, a rate limit or a rule would
|
||||
refuse: it passes them to the app, and their log lines name what `enforce`
|
||||
mode would have done (see `would_action` in "Request log" below). The checks
|
||||
run, and requests are counted, as in `enforce` mode, with three differences:
|
||||
neither a broken rate limit nor a `ban` rule makes a ban; a broken rate limit
|
||||
does not set the client's counters back to zero, so each request over the
|
||||
limit is logged as one that would be refused; and a request under a ban does
|
||||
not make it permanent. A ban it would have made, or made permanent, raises the
|
||||
alert `enforce` mode would have raised, marked as what would have happened
|
||||
(see "Alerts" below). The bans in `bans.json` are kept, and refuse requests
|
||||
again when `smallwebwaf` next runs in `enforce` mode, as long as they last.
|
||||
The timeouts and size limits still apply, since they protect `smallwebwaf` and
|
||||
the app themselves, and a request for one of `smallwebwaf`'s own endpoints
|
||||
without its token is still answered `401`. It is for trying a configuration
|
||||
before enforcing it.
|
||||
run, and requests and bytes are counted, as in `enforce` mode, with three
|
||||
differences: neither a broken rate limit or byte limit nor a `ban` rule makes
|
||||
a ban; a broken limit does not set the client's counters back to zero, so each
|
||||
request over a rate limit is logged as one that would be refused, and each
|
||||
whose bytes keep the client over a byte limit as breaking it; and a request
|
||||
under a ban does not make it permanent. As in `enforce` mode, the bytes
|
||||
counted are only those of the requests `enforce` mode would have passed to the
|
||||
app. A ban it would have made, or made permanent, raises the alert `enforce`
|
||||
mode would have raised, marked as what would have happened (see "Alerts"
|
||||
below). The bans in `bans.json` are kept, and refuse requests again when
|
||||
`smallwebwaf` next runs in `enforce` mode, as long as they last. The timeouts
|
||||
and size limits still apply, since they protect `smallwebwaf` and the app
|
||||
themselves, and a request for one of `smallwebwaf`'s own endpoints without its
|
||||
token is still answered `401`. It is for trying a configuration before
|
||||
enforcing it.
|
||||
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
|
||||
check and without asking the app, for the image's health check.
|
||||
- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
|
||||
@@ -254,10 +273,11 @@ effective settings are logged at start.
|
||||
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
|
||||
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
|
||||
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
|
||||
country lists, the rate limits and the rule files, such as your monitoring or
|
||||
your own networks.
|
||||
country lists, the rate limits, the byte limits and the rule files, such as
|
||||
your monitoring or your own networks.
|
||||
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
|
||||
rate limits do not apply to, such as a machine that talks to the app all day.
|
||||
rate limits and the byte limits do not apply to, such as a machine that talks
|
||||
to the app all day.
|
||||
- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
|
||||
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR`
|
||||
(default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`): the most
|
||||
@@ -265,19 +285,29 @@ effective settings are logged at start.
|
||||
several times what one busy person produces, since a browser loading a heavy
|
||||
page makes a few hundred requests and several people often share one address.
|
||||
- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes whose requests
|
||||
the rate limits neither count nor refuse, such as `/assets/` for static
|
||||
assets; each starts with `/`. A request whose path, percent-decoded, contains
|
||||
`..` anywhere or a backslash, or whose path as sent holds an encoded slash
|
||||
(`%2F` or `%2f`), is never exempt, since the app may act on it as a path
|
||||
outside every prefix: `/assets/..%2Flogin` as `/login`. Any other request is
|
||||
exempt when its path as sent, the path the app receives, before any query
|
||||
string and not percent-decoded, starts with a prefix, character for character.
|
||||
`/assets/` matches `/assets/app.js` and `/assets/`, but not `/assets`,
|
||||
`/Assets/app.js`, `/%61ssets/app.js`, `/static/assets/app.js`,
|
||||
`/static/../assets/app.js` or `/assets%2Fapp.js`. A character the client sends
|
||||
percent-encoded, such as a space, is written percent-encoded in a prefix, as
|
||||
in `/my%20files/`, and there are no wildcards: `*` is a character like any
|
||||
other.
|
||||
the rate limits neither count nor refuse, and whose bytes the byte limits do
|
||||
not count, such as `/assets/` for static assets; each starts with `/`. A
|
||||
request whose path, percent-decoded, contains `..` anywhere or a backslash, or
|
||||
whose path as sent holds an encoded slash (`%2F` or `%2f`), is never exempt,
|
||||
since the app may act on it as a path outside every prefix:
|
||||
`/assets/..%2Flogin` as `/login`. Any other request is exempt when its path as
|
||||
sent, the path the app receives, before any query string and not
|
||||
percent-decoded, starts with a prefix, character for character. `/assets/`
|
||||
matches `/assets/app.js` and `/assets/`, but not `/assets`, `/Assets/app.js`,
|
||||
`/%61ssets/app.js`, `/static/assets/app.js`, `/static/../assets/app.js` or
|
||||
`/assets%2Fapp.js`. A character the client sends percent-encoded, such as a
|
||||
space, is written percent-encoded in a prefix, as in `/my%20files/`, and there
|
||||
are no wildcards: `*` is a character like any other.
|
||||
- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`), `SWWAF_BYTES_LIMIT_PER_HOUR`
|
||||
(default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `50G`): the most
|
||||
bytes a client may have counted in a minute, an hour and a day. A request's
|
||||
bytes are counted once its answer has ended, so each default is above the
|
||||
largest request body and the largest response together,
|
||||
`SWWAF_REQUEST_MAX_BYTES` and `SWWAF_RESPONSE_MAX_BYTES`: at the defaults no
|
||||
download breaks a limit on its own.
|
||||
- `SWWAF_BYTES_COUNT` (default `both`): which body bytes the byte limits count:
|
||||
`response` for those of the answers, `request` for those of the requests, or
|
||||
`both`.
|
||||
- `SWWAF_LOOKUP_SOURCE` (default `geojs`): where each client's AS number and
|
||||
country are looked up: `geojs`, the GeoJS web service, which is then told the
|
||||
address of every new visitor, `file`, the IPinfo Lite database file
|
||||
@@ -311,12 +341,12 @@ effective settings are logged at start.
|
||||
the client unanswered: traefik answers `502`, as it does whenever its backend
|
||||
drops a connection. A `block` rule always answers `403`.
|
||||
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
|
||||
limit.
|
||||
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit broken again
|
||||
within this time after a ban ended, other than one for a clear sign of attack,
|
||||
bans for three times as long as that ban.
|
||||
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken rate limit that
|
||||
would be longer is permanent instead.
|
||||
limit or byte limit.
|
||||
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit or byte limit
|
||||
broken again within this time after a ban ended, other than one for a clear
|
||||
sign of attack, bans for three times as long as that ban.
|
||||
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken rate limit or byte
|
||||
limit that would be longer is permanent instead.
|
||||
- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
|
||||
attack.
|
||||
- `SWWAF_MAX_BANS` (default `5000`): the most bans `smallwebwaf` made that are
|
||||
@@ -410,15 +440,16 @@ effective settings are logged at start.
|
||||
|
||||
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
|
||||
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
|
||||
bytes). Rate limits are whole numbers of requests. Netblocks are in CIDR form,
|
||||
and a bare address stands for itself alone. Countries are the two-letter codes
|
||||
ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
|
||||
the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
|
||||
`su`, stops the start, and so does a code on both country lists. `off` switches
|
||||
a timeout, a size limit, a rate limit, `SWWAF_ALERT_COOLDOWN` or
|
||||
`SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
|
||||
`SWWAF_LOOKUP_TIMEOUT`, the ban settings, the state settings,
|
||||
`SWWAF_METRICS_TOP_N` and `SWWAF_LOG_REMOTE_BUFFER` cannot be off.
|
||||
bytes). Rate limits are whole numbers of requests, and byte limits are sizes.
|
||||
Netblocks are in CIDR form, and a bare address stands for itself alone.
|
||||
Countries are the two-letter codes ISO 3166-1 assigns today, and `xk` for
|
||||
Kosovo, in either case (`de` and `DE` are the same); any other code, such as
|
||||
`nk` (North Korea is `kp`) or the withdrawn `su`, stops the start, and so does a
|
||||
code on both country lists. `off` switches a timeout, a size limit, a rate
|
||||
limit, a byte limit, `SWWAF_ALERT_COOLDOWN` or `SWWAF_ALERT_MAX_PER_HOUR` off;
|
||||
`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, `SWWAF_LOOKUP_TIMEOUT`, the ban
|
||||
settings, the state settings, `SWWAF_METRICS_TOP_N` and
|
||||
`SWWAF_LOG_REMOTE_BUFFER` cannot be off.
|
||||
|
||||
Several limits are fixed rather than settings. At most 20,000 clients are kept,
|
||||
with their counters and history, and an IPv6 client is counted by its /64. At
|
||||
@@ -462,7 +493,7 @@ and for the container, `-v /srv/app/tokens:/etc/smallwebwaf/tokens:ro` and
|
||||
refused ones included:
|
||||
|
||||
```
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104}
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104}
|
||||
```
|
||||
|
||||
A field that does not apply to a request is left out of its line, apart from
|
||||
@@ -527,13 +558,20 @@ which every line has.
|
||||
health check, one from a client in `SWWAF_ALLOW_NETS` or
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_NETS`, one for a path that
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a ban
|
||||
or the country lists refuse, or would refuse in `observe` mode. The byte
|
||||
totals come with the byte limits.
|
||||
or the country lists refuse, or would refuse in `observe` mode. Its
|
||||
`minute_bytes`, `hour_bytes` and `day_bytes` give the client's bytes in each
|
||||
window as the byte limits count them, in the same way: for a request whose
|
||||
bytes they count, with its own, once it has ended; for any other, those
|
||||
counted before it.
|
||||
- `rule_ids` is there for a request that matched rules of the rule files, and
|
||||
lists their ids in the order they matched, up to the one that refused it.
|
||||
- `limit_hit` is there for a request that broke a rate limit, and names the
|
||||
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
|
||||
went over several. `offence` is then `limit`.
|
||||
- `limit_hit` is there for a request that broke a rate limit, or whose bytes
|
||||
broke a byte limit, and names the window whose limit it went over as `counts`
|
||||
names it: `minute`, `hour` or `day` for a rate limit, and `minute_bytes`,
|
||||
`hour_bytes` or `day_bytes` for a byte limit, the shortest if it went over
|
||||
several. `offence` is then `limit`. A request whose bytes broke a byte limit
|
||||
is not refused: its `action` is what it would have been otherwise, such as
|
||||
`forward`.
|
||||
- `ban_expires` is there for a request that made a ban or was refused under one,
|
||||
or in `observe` mode would have been refused under one, and gives when the ban
|
||||
ends, in the same form as `time`, or `permanent`.
|
||||
@@ -596,8 +634,8 @@ gives, as "Alert webhook schema" in [`SPEC.md`](SPEC.md) describes, and to
|
||||
it, as below. An alert is for one of these events, and is sent when
|
||||
`SWWAF_ALERT_EVENTS` names its event:
|
||||
|
||||
- `ban`: a ban `smallwebwaf` makes, for a broken rate limit or a clear sign of
|
||||
attack.
|
||||
- `ban`: a ban `smallwebwaf` makes, for a broken rate limit or byte limit or a
|
||||
clear sign of attack.
|
||||
- `permanent_ban`: a permanent ban it makes, or a ban for a clear sign of attack
|
||||
that a request made permanent.
|
||||
- `source_failure`: GeoJS failing or refusing `smallwebwaf`.
|
||||
@@ -633,6 +671,7 @@ is sent on one line:
|
||||
"asn": "",
|
||||
"as_name": "",
|
||||
"country": "",
|
||||
"kind": "requests",
|
||||
"limit": 1000,
|
||||
"window": "minute",
|
||||
"count": 1001,
|
||||
@@ -752,20 +791,23 @@ entries by client address, but for the alerts waiting, with times in UTC.
|
||||
|
||||
- `bans.json`: every ban with its notes, indented to be read. A permanent ban's
|
||||
`expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or
|
||||
`attack` for a clear sign of attack, for a ban `smallwebwaf` made, and `admin`
|
||||
for one you made or keep. Its `reason` is a short text: for a ban
|
||||
`smallwebwaf` made, the limit broken, such as
|
||||
`requests per minute over the limit of 1000`, or the rule that matched, such
|
||||
byte limit or `attack` for a clear sign of attack, for a ban `smallwebwaf`
|
||||
made, and `admin` for one you made or keep. Its `reason` is a short text: for
|
||||
a ban `smallwebwaf` made, the limit broken, such as
|
||||
`requests per minute over the limit of 1000` or
|
||||
`bytes per hour over the limit of 21474836480`, or the rule that matched, such
|
||||
as `matched the rule env-file`; for yours, what you wrote. Its `lifted` is
|
||||
when you lifted it, and is left out until you do.
|
||||
- `clients.json`: each client's two buckets in the minute, the hour and the day,
|
||||
and its history: when it was first and last seen, its AS number, AS name and
|
||||
country as last looked up and when the lookup gave them, its requests, how
|
||||
many were forwarded and how many refused (one `smallwebwaf` answered at its
|
||||
own endpoints is neither, unless it was refused with `401` for a missing or
|
||||
wrong token), the body bytes in each direction, its responses by status class
|
||||
and its offences by kind. Each client is on a line of its own, so `grep` shows
|
||||
everything about one.
|
||||
when you lifted it, and is left out until you do. The `kind` in the notes of a
|
||||
ban for a broken limit is `requests` or `bytes`, what the limit is on.
|
||||
- `clients.json`: each client's two buckets of requests in the minute, the hour
|
||||
and the day, its two buckets of bytes in each, `minute_bytes`, `hour_bytes`
|
||||
and `day_bytes`, and its history: when it was first and last seen, its AS
|
||||
number, AS name and country as last looked up and when the lookup gave them,
|
||||
its requests, how many were forwarded and how many refused (one `smallwebwaf`
|
||||
answered at its own endpoints is neither, unless it was refused with `401` for
|
||||
a missing or wrong token), the body bytes in each direction, its responses by
|
||||
status class and its offences by kind. Each client is on a line of its own, so
|
||||
`grep` shows everything about one.
|
||||
- `lookups.json`: GeoJS's answers, one to a line, each with the client's AS
|
||||
number, AS name and country, when GeoJS gave it and when it was last used.
|
||||
- `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
|
||||
@@ -801,9 +843,9 @@ message naming the file, and the line and column where Go's JSON decoder gives
|
||||
them; so does a state directory `smallwebwaf` cannot write. So does an entry
|
||||
without a field it needs, named with the entry's place in the file: a ban's
|
||||
`netblock`, `start` or `expires`, which is `null` for a permanent ban; a
|
||||
client's `client`, or the `start` of a window in which it has requests; an
|
||||
answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
|
||||
`answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
|
||||
client's `client`, or the `start` of a window in which it has requests or bytes;
|
||||
an answer's `client`, `country`, which is `""` for a client GeoJS cannot place,
|
||||
or `answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
|
||||
`time`. So does a ban whose `cause` is not `limit`, `attack` or `admin`, and
|
||||
alerts waiting for a destination that is not `webhook`, `slack` or `ntfy`. An
|
||||
answer's `asn` or `as_name` left out reads as empty.
|
||||
@@ -955,7 +997,8 @@ scraped, and keeps this one as `exported_instance` unless the scrape sets
|
||||
`smallwebwaf_upstream_duration_seconds`: how long those passed to the app took
|
||||
from then on, as histograms; `smallwebwaf_requests_in_flight`: the requests
|
||||
under way.
|
||||
- `smallwebwaf_rate_limit_hits_total` by `window`,
|
||||
- `smallwebwaf_rate_limit_hits_total` by `window`, `minute`, `hour` or `day`,
|
||||
and `kind`, `requests` for a rate limit or `bytes` for a byte limit,
|
||||
`smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose
|
||||
limit was passed, `smallwebwaf_offences_total` by `kind`, and
|
||||
`smallwebwaf_bans_made_total` by `cause`, `limit`, `attack` or `admin`, the
|
||||
@@ -1375,7 +1418,8 @@ addresses are never sent to GeoJS.
|
||||
body over the size limit; in `observe` mode, only for the size limit, with
|
||||
what it would have refused for noted in the log line. A request under
|
||||
`/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
|
||||
instead of reaching the app.
|
||||
instead of reaching the app. Once the answer to a request passed to the app
|
||||
has ended, `countBytes` counts its bytes for the byte limits.
|
||||
- `internal/metrics`: the metrics, counted as the other parts tell it what
|
||||
happened, and served in the Prometheus text format.
|
||||
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how
|
||||
@@ -1388,8 +1432,9 @@ addresses are never sent to GeoJS.
|
||||
client's history and to the notes of its bans; or in the lookup database,
|
||||
which it reads again when the file is replaced. `internal/lookup/lookuptest`
|
||||
writes lookup databases for the tests.
|
||||
- `internal/ratelimit`: the table of clients: counts each client's requests,
|
||||
tells when one takes it over a rate limit, and keeps each client's history.
|
||||
- `internal/ratelimit`: the table of clients: counts each client's requests and
|
||||
bytes, tells when they take it over a rate limit or a byte limit, and keeps
|
||||
each client's history.
|
||||
- `internal/state`: reads the state files at start, takes in an admin's edit of
|
||||
one while running, and writes them when they are due and at the stop.
|
||||
- `internal/requestlog`: the lines on stdout: the request log line and the
|
||||
|
||||
@@ -66,12 +66,15 @@ func TestReasonOfTheBansSmallwebwafMakes(t *testing.T) {
|
||||
ledger := bans.New(defaultRules())
|
||||
|
||||
limit, _ := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.1/32"), midnight(),
|
||||
bans.Notes{Limit: 1000, Window: "minute"})
|
||||
bans.Notes{Kind: "requests", Limit: 1000, Window: "minute"})
|
||||
byteLimit, _ := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.3/32"),
|
||||
midnight(), bans.Notes{Kind: "bytes", Limit: 10 << 30, Window: "hour"})
|
||||
attack, _ := ledger.BanForAttack(netip.MustParsePrefix("203.0.113.2/32"), midnight(),
|
||||
bans.Notes{RuleID: "git-dir", Target: "path"})
|
||||
|
||||
for _, tc := range []struct{ got, want string }{
|
||||
{limit.Reason, "requests per minute over the limit of 1000"},
|
||||
{byteLimit.Reason, "bytes per hour over the limit of 10737418240"},
|
||||
{attack.Reason, "matched the rule git-dir"},
|
||||
} {
|
||||
if tc.got != tc.want {
|
||||
|
||||
+18
-13
@@ -1,8 +1,8 @@
|
||||
// Package bans is the ban ledger: the bans smallwebwaf makes on the
|
||||
// netblocks of clients that break a rate limit or show a clear sign of
|
||||
// attack, and those an admin makes, with their notes, as the "Bans"
|
||||
// section of SPEC.md describes. The bans are kept in memory, and written
|
||||
// to bans.json and read from it by the state package.
|
||||
// netblocks of clients that break a rate limit or a byte limit or show a
|
||||
// clear sign of attack, and those an admin makes, with their notes, as
|
||||
// the "Bans" section of SPEC.md describes. The bans are kept in memory,
|
||||
// and written to bans.json and read from it by the state package.
|
||||
package bans
|
||||
|
||||
import (
|
||||
@@ -97,11 +97,14 @@ type Notes struct {
|
||||
ASN string `json:"asn"`
|
||||
ASName string `json:"as_name"`
|
||||
Country string `json:"country"`
|
||||
// Limit, Window and Count are, for a ban for a broken limit, the limit
|
||||
// that was broken, its window, "minute", "hour" or "day", and the
|
||||
// count reached: the client's requests in the window, the one that
|
||||
// broke the limit included. These are the requests that counted
|
||||
// toward the ban, and the window is the time over which they came.
|
||||
// Kind, Limit, Window and Count are, for a ban for a broken limit,
|
||||
// what the limit was on, "requests" for a rate limit or "bytes" for a
|
||||
// byte limit, the limit that was broken, its window, "minute", "hour"
|
||||
// or "day", and the count reached: the client's requests, or bytes, in
|
||||
// the window, those of the request that broke the limit included.
|
||||
// These are what counted toward the ban, and the window is the time
|
||||
// over which they came.
|
||||
Kind string `json:"kind,omitempty"`
|
||||
Limit int64 `json:"limit,omitempty"`
|
||||
Window string `json:"window,omitempty"`
|
||||
Count float64 `json:"count,omitempty"`
|
||||
@@ -109,8 +112,8 @@ type Notes struct {
|
||||
// of the rule file rule that matched, and its target.
|
||||
RuleID string `json:"rule_id,omitempty"`
|
||||
Target string `json:"target,omitempty"`
|
||||
// Request is the request that broke the limit, or that was the clear
|
||||
// sign of attack.
|
||||
// Request is the request that broke the limit, or whose bytes broke
|
||||
// it, or that was the clear sign of attack.
|
||||
Request Request `json:"request"`
|
||||
// Requests is how many requests the netblock has sent since it was
|
||||
// first seen, and Refused how many of them the ban has refused so
|
||||
@@ -260,7 +263,8 @@ func activeBan(bans []Ban, now time.Time) *Ban {
|
||||
// active, as when two of its requests break a limit at once, that ban is
|
||||
// returned with false, and no other is made. The ledger fills in the
|
||||
// notes' Refused and EarlierBans itself, and gives the ban the reason
|
||||
// "requests per <Window> over the limit of <Limit>", from the notes.
|
||||
// "<Kind> per <Window> over the limit of <Limit>", from the notes, such
|
||||
// as "requests per minute over the limit of 1000".
|
||||
func (l *Ledger) BanForLimit(
|
||||
netblock netip.Prefix, now time.Time, notes Notes,
|
||||
) (Ban, bool) {
|
||||
@@ -317,7 +321,8 @@ func (l *Ledger) WouldBePermanent(
|
||||
|
||||
// limitReason is the reason of a ban for a broken limit, with notes.
|
||||
func limitReason(notes Notes) string {
|
||||
return fmt.Sprintf("requests per %s over the limit of %d", notes.Window, notes.Limit)
|
||||
return fmt.Sprintf("%s per %s over the limit of %d",
|
||||
notes.Kind, notes.Window, notes.Limit)
|
||||
}
|
||||
|
||||
// attackReason is the reason of a ban for a clear sign of attack, with
|
||||
|
||||
@@ -349,7 +349,7 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
|
||||
|
||||
// As it ends, a clear sign of attack would ban for seven days, and a
|
||||
// limit broken again for three hours, but neither is made.
|
||||
limitNotes := bans.Notes{Limit: 1, Window: "minute"}
|
||||
limitNotes := bans.Notes{Kind: "requests", Limit: 1, Window: "minute"}
|
||||
attack, wouldAttack := ledger.WouldBanForAttack(netblock, first.Expires,
|
||||
bans.Notes{RuleID: "git-dir"})
|
||||
limit, wouldLimit := ledger.WouldBanForLimit(netblock, first.Expires, limitNotes)
|
||||
|
||||
@@ -91,6 +91,15 @@ type Config struct {
|
||||
// limits neither count nor refuse (SWWAF_RATE_LIMIT_EXEMPT_PATHS).
|
||||
// Each starts with /.
|
||||
RateLimitExemptPaths []string
|
||||
// BytesLimitPerMinute, BytesLimitPerHour and BytesLimitPerDay are the
|
||||
// most bytes a client's requests may carry in a minute, an hour and a
|
||||
// day (SWWAF_BYTES_LIMIT_PER_MINUTE, SWWAF_BYTES_LIMIT_PER_HOUR and
|
||||
// SWWAF_BYTES_LIMIT_PER_DAY). BytesCount is which body bytes count
|
||||
// toward them (SWWAF_BYTES_COUNT): response, request or both.
|
||||
BytesLimitPerMinute int64
|
||||
BytesLimitPerHour int64
|
||||
BytesLimitPerDay int64
|
||||
BytesCount string
|
||||
// LookupSource is where each client's AS number and country are
|
||||
// looked up (SWWAF_LOOKUP_SOURCE): geojs, file, or off for nowhere.
|
||||
// LookupDBPath is the lookup database, the IPinfo Lite file looked up
|
||||
@@ -266,6 +275,7 @@ var (
|
||||
"is not an absolute path, such as /var/lib/smallwebwaf")
|
||||
errShortToken = errors.New("is shorter than 32 characters")
|
||||
errNotMode = errors.New("is not enforce or observe")
|
||||
errNotBytesCount = errors.New("is not response, request or both")
|
||||
errNotPathPrefix = errors.New(
|
||||
"is not a path prefix starting with /, such as /assets/")
|
||||
errNotBoolean = errors.New("is not true or false")
|
||||
@@ -321,6 +331,10 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
||||
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "10000"),
|
||||
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "50000"),
|
||||
RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
|
||||
BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "10G"),
|
||||
BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"),
|
||||
BytesLimitPerDay: env.size("SWWAF_BYTES_LIMIT_PER_DAY", "50G"),
|
||||
BytesCount: env.bytesCount("SWWAF_BYTES_COUNT", "both"),
|
||||
LookupSource: env.lookupSource("SWWAF_LOOKUP_SOURCE", "geojs"),
|
||||
LookupDBPath: env.value("SWWAF_LOOKUP_DB_PATH", ""),
|
||||
LookupTimeout: env.durationNotOff("SWWAF_LOOKUP_TIMEOUT", "1s"),
|
||||
@@ -551,6 +565,17 @@ func (e *environment) count(name, defaultValue string) int64 {
|
||||
return count
|
||||
}
|
||||
|
||||
// bytesCount reads the setting that is which body bytes count toward the
|
||||
// byte limits: response, request or both.
|
||||
func (e *environment) bytesCount(name, defaultValue string) string {
|
||||
value := e.value(name, defaultValue)
|
||||
if value != "response" && value != "request" && value != "both" {
|
||||
e.check(name, fmt.Errorf("%q %w", value, errNotBytesCount))
|
||||
}
|
||||
|
||||
return value
|
||||
}
|
||||
|
||||
// pathPrefixes reads a setting that is a list of path prefixes.
|
||||
func (e *environment) pathPrefixes(name, defaultValue string) []string {
|
||||
prefixes, err := parsePathPrefixes(e.value(name, defaultValue))
|
||||
|
||||
@@ -40,6 +40,10 @@ const (
|
||||
rateLimitPerHour = "SWWAF_RATE_LIMIT_PER_HOUR"
|
||||
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
|
||||
rateLimitExemptPaths = "SWWAF_RATE_LIMIT_EXEMPT_PATHS"
|
||||
bytesLimitPerMinute = "SWWAF_BYTES_LIMIT_PER_MINUTE"
|
||||
bytesLimitPerHour = "SWWAF_BYTES_LIMIT_PER_HOUR"
|
||||
bytesLimitPerDay = "SWWAF_BYTES_LIMIT_PER_DAY"
|
||||
bytesCount = "SWWAF_BYTES_COUNT"
|
||||
lookupSource = "SWWAF_LOOKUP_SOURCE"
|
||||
lookupDBPath = "SWWAF_LOOKUP_DB_PATH"
|
||||
lookupTimeout = "SWWAF_LOOKUP_TIMEOUT"
|
||||
@@ -190,6 +194,10 @@ func TestDefaults(t *testing.T) {
|
||||
wantLookupSettings(t, cfg, config.Config{
|
||||
LookupSource: defaultLookupSource, LookupTimeout: time.Second,
|
||||
})
|
||||
wantByteLimitSettings(t, cfg, config.Config{
|
||||
BytesLimitPerMinute: 10 << 30, BytesLimitPerHour: 20 << 30,
|
||||
BytesLimitPerDay: 50 << 30, BytesCount: "both",
|
||||
})
|
||||
|
||||
if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" {
|
||||
t.Errorf("%s is %s", upstreamURL, cfg.UpstreamURL)
|
||||
@@ -220,6 +228,22 @@ func TestDefaults(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoSingleRequestBreaksAByteLimitAtTheDefaults(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := fromEnvironment(t, environment{})
|
||||
|
||||
// The largest request body and the largest response, both counted.
|
||||
largest := cfg.RequestMaxBytes + cfg.ResponseMaxBytes
|
||||
for _, limit := range []int64{
|
||||
cfg.BytesLimitPerMinute, cfg.BytesLimitPerHour, cfg.BytesLimitPerDay,
|
||||
} {
|
||||
if largest > limit {
|
||||
t.Errorf("a request of %d bytes breaks the byte limit of %d", largest, limit)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestValuesAsSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -302,6 +326,22 @@ func TestValuesAsSet(t *testing.T) {
|
||||
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
|
||||
}
|
||||
|
||||
func TestByteLimitSettingsAsSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := fromEnvironment(t, environment{
|
||||
bytesLimitPerMinute: "512M",
|
||||
bytesLimitPerHour: off,
|
||||
bytesLimitPerDay: "100000",
|
||||
bytesCount: "response",
|
||||
})
|
||||
|
||||
wantByteLimitSettings(t, cfg, config.Config{
|
||||
BytesLimitPerMinute: 512 << 20, BytesLimitPerHour: 0,
|
||||
BytesLimitPerDay: 100000, BytesCount: "response",
|
||||
})
|
||||
}
|
||||
|
||||
func TestRateLimitExemptPathsAsSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -996,6 +1036,9 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
|
||||
{rateLimitPerHour, "1.5"},
|
||||
{rateLimitPerDay, "-1"}, {rateLimitPerDay, "lots"},
|
||||
{rateLimitExemptPaths, "/assets/,,/static/"},
|
||||
{bytesLimitPerMinute, "10GB"}, {bytesLimitPerHour, "0"},
|
||||
{bytesLimitPerDay, "-1G"},
|
||||
{bytesCount, "all"}, {bytesCount, "Both"}, {bytesCount, ""},
|
||||
{lookupSource, "ipinfo"}, {lookupSource, "GeoJS"}, {lookupSource, ""},
|
||||
{lookupTimeout, off}, {lookupTimeout, "0s"}, {lookupTimeout, "1"},
|
||||
{addLookupHeaders, "yes"},
|
||||
@@ -1250,20 +1293,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := fromEnvironment(t, environment{clientRequestTimeout: "45s"})
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
slog.New(slog.NewJSONHandler(&out, nil)).Info("starting", "settings", cfg)
|
||||
|
||||
var line struct {
|
||||
Settings map[string]string `json:"settings"`
|
||||
}
|
||||
|
||||
err := json.Unmarshal(out.Bytes(), &line)
|
||||
if err != nil {
|
||||
t.Fatalf("decode %s: %v", out.Bytes(), err)
|
||||
}
|
||||
|
||||
hostname, _ := os.Hostname()
|
||||
|
||||
want := map[string]string{
|
||||
@@ -1286,6 +1315,10 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
rateLimitPerHour: "10000",
|
||||
rateLimitPerDay: "50000",
|
||||
rateLimitExemptPaths: "",
|
||||
bytesLimitPerMinute: "10G",
|
||||
bytesLimitPerHour: "20G",
|
||||
bytesLimitPerDay: "50G",
|
||||
bytesCount: "both",
|
||||
lookupSource: defaultLookupSource,
|
||||
lookupDBPath: "",
|
||||
lookupTimeout: "1s",
|
||||
@@ -1323,11 +1356,31 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
alertCooldown: defaultAlertCooldown,
|
||||
alertMaxPerHour: "60",
|
||||
}
|
||||
if !maps.Equal(line.Settings, want) {
|
||||
t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
|
||||
if got := loggedSettings(t, cfg); !maps.Equal(got, want) {
|
||||
t.Errorf("logged settings\n%v\nwant\n%v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// loggedSettings returns the settings as cfg logs them, each by its name.
|
||||
func loggedSettings(t *testing.T, cfg *config.Config) map[string]string {
|
||||
t.Helper()
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
slog.New(slog.NewJSONHandler(&out, nil)).Info("starting", "settings", cfg)
|
||||
|
||||
var line struct {
|
||||
Settings map[string]string `json:"settings"`
|
||||
}
|
||||
|
||||
err := json.Unmarshal(out.Bytes(), &line)
|
||||
if err != nil {
|
||||
t.Fatalf("decode %s: %v", out.Bytes(), err)
|
||||
}
|
||||
|
||||
return line.Settings
|
||||
}
|
||||
|
||||
// wantSettings checks the settings that are plain values.
|
||||
func wantSettings(t *testing.T, got *config.Config, want config.Config) {
|
||||
t.Helper()
|
||||
@@ -1351,6 +1404,21 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) {
|
||||
wantBanSettings(t, got, want)
|
||||
}
|
||||
|
||||
// wantByteLimitSettings checks the settings for the byte limits.
|
||||
func wantByteLimitSettings(t *testing.T, got *config.Config, want config.Config) {
|
||||
t.Helper()
|
||||
|
||||
if got.BytesLimitPerMinute != want.BytesLimitPerMinute ||
|
||||
got.BytesLimitPerHour != want.BytesLimitPerHour ||
|
||||
got.BytesLimitPerDay != want.BytesLimitPerDay ||
|
||||
got.BytesCount != want.BytesCount {
|
||||
t.Errorf("byte limits %d, %d and %d counting %s, want %d, %d and %d counting %s",
|
||||
got.BytesLimitPerMinute, got.BytesLimitPerHour, got.BytesLimitPerDay,
|
||||
got.BytesCount, want.BytesLimitPerMinute, want.BytesLimitPerHour,
|
||||
want.BytesLimitPerDay, want.BytesCount)
|
||||
}
|
||||
}
|
||||
|
||||
// wantLookupSettings checks the settings for lookups.
|
||||
func wantLookupSettings(t *testing.T, got *config.Config, want config.Config) {
|
||||
t.Helper()
|
||||
|
||||
@@ -6,6 +6,7 @@ package metrics
|
||||
import (
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
@@ -89,8 +90,9 @@ func New(topN int, instanceName string) *Metrics {
|
||||
Help: "How long requests passed to the app took, from then to their end.",
|
||||
}),
|
||||
rateLimitHits: counterVec("smallwebwaf_rate_limit_hits_total",
|
||||
"Requests that broke a rate limit, by its window.",
|
||||
[]string{"window"}),
|
||||
"Requests that broke a rate limit or a byte limit, by its window and "+
|
||||
"its kind, requests or bytes.",
|
||||
[]string{"window", "kind"}),
|
||||
sizeAndTimeLimitHits: counterVec("smallwebwaf_size_and_time_limit_hits_total",
|
||||
"Requests that passed a size or time limit, by its setting.",
|
||||
[]string{"limit"}),
|
||||
@@ -325,7 +327,15 @@ func (m *Metrics) RequestEnded(
|
||||
}
|
||||
|
||||
if line.LimitHit != "" {
|
||||
m.rateLimitHits.WithLabelValues(line.LimitHit).Inc()
|
||||
// The log line names a byte limit's window with _bytes after it.
|
||||
window, isBytes := strings.CutSuffix(line.LimitHit, "_bytes")
|
||||
|
||||
kind := ratelimit.KindRequests
|
||||
if isBytes {
|
||||
kind = ratelimit.KindBytes
|
||||
}
|
||||
|
||||
m.rateLimitHits.WithLabelValues(window, kind).Inc()
|
||||
}
|
||||
|
||||
if limit != "" {
|
||||
|
||||
@@ -203,7 +203,16 @@ func startWithAlerts(
|
||||
) (*sender, *clock, *proxy.Server, *alerts.Queue) {
|
||||
t.Helper()
|
||||
|
||||
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
||||
return startAppWithAlerts(t, func(http.ResponseWriter, *http.Request) {}, env)
|
||||
}
|
||||
|
||||
// startAppWithAlerts is startWithAlerts in front of the app handler.
|
||||
func startAppWithAlerts(
|
||||
t *testing.T, handler http.HandlerFunc, env map[string]string,
|
||||
) (*sender, *clock, *proxy.Server, *alerts.Queue) {
|
||||
t.Helper()
|
||||
|
||||
app := startApp(t, handler)
|
||||
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)}
|
||||
settings := map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
|
||||
+77
-22
@@ -6,6 +6,7 @@ import (
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
@@ -41,36 +42,92 @@ func (rq *request) banned(now time.Time) bool {
|
||||
|
||||
// limitBroken counts the request for the rate limits at now, notes the
|
||||
// client's counts for the log line, and reports whether the request takes
|
||||
// the client over a limit. In enforce mode such a request bans the
|
||||
// client's netblock, and sets the client's counters back to zero; in
|
||||
// observe mode it does neither, and raises the alert for the ban it would
|
||||
// have made, if that alert would be sent.
|
||||
// the client over a rate limit, which breaks it.
|
||||
func (rq *request) limitBroken(now time.Time) bool {
|
||||
group := clientGroup(rq.client)
|
||||
|
||||
counts, hit, over := rq.h.limiter.Count(group, now)
|
||||
counts, hit, over := rq.h.limiter.Count(clientGroup(rq.client), now)
|
||||
rq.line.Counts = counts
|
||||
|
||||
if !over {
|
||||
return false
|
||||
if over {
|
||||
rq.banForLimit(now, hit, rq.h.config.BanResponse)
|
||||
}
|
||||
|
||||
return over
|
||||
}
|
||||
|
||||
// countBytes counts the request's bytes for the byte limits, once its
|
||||
// response has ended, and notes the client's byte totals for the log line;
|
||||
// its requests stay there as the rate limits counted them. The bytes are
|
||||
// the response's body bytes, the request's, or both, as SWWAF_BYTES_COUNT
|
||||
// says; for an upgraded connection, such as a WebSocket, which has closed
|
||||
// by then, what it carried from the app counts with the response's and
|
||||
// what it carried from the client with the request's. Only a request
|
||||
// passed to the app has them counted, and only one the rate limits
|
||||
// counted; in observe mode, not one that enforce mode would have refused.
|
||||
// Bytes that take the client over a byte limit break it; the response was
|
||||
// passed on whole.
|
||||
func (rq *request) countBytes() {
|
||||
if !rq.counted || rq.line.WouldAction != "" {
|
||||
return
|
||||
}
|
||||
|
||||
response, request := rq.out.bytes, rq.requestBytes()
|
||||
if rq.upgraded != nil {
|
||||
response += rq.upgraded.fromApp.Load()
|
||||
request += rq.upgraded.toApp.Load()
|
||||
}
|
||||
|
||||
var bytes int64
|
||||
|
||||
switch rq.h.config.BytesCount {
|
||||
case "response":
|
||||
bytes = response
|
||||
case "request":
|
||||
bytes = request
|
||||
default: // both
|
||||
bytes = response + request
|
||||
}
|
||||
|
||||
now := rq.h.now()
|
||||
|
||||
counts, hit, over := rq.h.limiter.CountBytes(clientGroup(rq.client), now, bytes)
|
||||
rq.line.Counts.MinuteBytes = counts.MinuteBytes
|
||||
rq.line.Counts.HourBytes = counts.HourBytes
|
||||
rq.line.Counts.DayBytes = counts.DayBytes
|
||||
|
||||
if over {
|
||||
rq.banForLimit(now, hit, rq.out.status)
|
||||
}
|
||||
}
|
||||
|
||||
// banForLimit bans the client's netblock at now for a broken limit, the
|
||||
// one hit names, and notes the offence for the log line. status is what
|
||||
// the client was sent, or is sent: SWWAF_BAN_RESPONSE for a request over
|
||||
// a rate limit, the app's answer for one whose bytes broke a byte limit.
|
||||
// The ban sets the client's counters back to zero. In observe mode it
|
||||
// makes no ban and sets nothing back, and raises the alert for the ban it
|
||||
// would have made, if that alert would be sent.
|
||||
func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
|
||||
rq.line.LimitHit = hit.Window
|
||||
if hit.Kind == ratelimit.KindBytes {
|
||||
rq.line.LimitHit += "_bytes" // as counts names the byte totals
|
||||
}
|
||||
|
||||
rq.line.Offence = requestlog.OffenceLimit
|
||||
|
||||
netblock := rq.h.netblock(rq.client)
|
||||
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseLimit) {
|
||||
return true
|
||||
return
|
||||
}
|
||||
|
||||
notes := bans.Notes{
|
||||
ASN: rq.line.ASN,
|
||||
ASName: rq.line.ASName,
|
||||
Country: rq.line.Country,
|
||||
Kind: hit.Kind,
|
||||
Limit: hit.Limit,
|
||||
Window: hit.Window,
|
||||
Count: hit.Requests,
|
||||
Request: rq.noted(now),
|
||||
Count: hit.Count,
|
||||
Request: rq.noted(now, status),
|
||||
Requests: rq.netblockRequests(netblock),
|
||||
}
|
||||
|
||||
@@ -80,18 +137,16 @@ func (rq *request) limitBroken(now time.Time) bool {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return true
|
||||
return
|
||||
}
|
||||
|
||||
ban, made := rq.h.ledger.BanForLimit(netblock, now, notes)
|
||||
rq.h.limiter.Reset(group)
|
||||
rq.h.limiter.Reset(clientGroup(rq.client))
|
||||
rq.line.BanExpires = banExpires(ban)
|
||||
|
||||
if made {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// banForAttack bans the client's netblock at now for a clear sign of
|
||||
@@ -110,7 +165,7 @@ func (rq *request) banForAttack(now time.Time, rule rules.Rule) {
|
||||
Country: rq.line.Country,
|
||||
RuleID: rule.ID,
|
||||
Target: rule.Target,
|
||||
Request: rq.noted(now),
|
||||
Request: rq.noted(now, rq.h.config.BanResponse),
|
||||
Requests: rq.netblockRequests(netblock),
|
||||
}
|
||||
|
||||
@@ -177,16 +232,16 @@ func (rq *request) alertBan(ban bans.Ban) {
|
||||
})
|
||||
}
|
||||
|
||||
// noted is the request, refused at now with SWWAF_BAN_RESPONSE, or in
|
||||
// observe mode as it would have been, as the notes of the ban it makes
|
||||
// keep it.
|
||||
func (rq *request) noted(now time.Time) bans.Request {
|
||||
// noted is the request, at now, with status, what the client was sent, or
|
||||
// in observe mode would have been, as the notes of the ban it makes keep
|
||||
// it.
|
||||
func (rq *request) noted(now time.Time, status int) bans.Request {
|
||||
return bans.Request{
|
||||
Time: now,
|
||||
Method: rq.in.Method,
|
||||
Host: rq.in.Host,
|
||||
Path: rq.in.URL.RequestURI(),
|
||||
Status: rq.h.config.BanResponse,
|
||||
Status: status,
|
||||
UserAgent: rq.in.UserAgent(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -284,6 +284,7 @@ func TestBanNotes(t *testing.T) {
|
||||
ASN: asnDE,
|
||||
ASName: asNameDE,
|
||||
Country: "DE",
|
||||
Kind: "requests",
|
||||
Limit: 1,
|
||||
Window: minute,
|
||||
Count: 2,
|
||||
|
||||
@@ -103,6 +103,48 @@ func (b *responseBody) Close() error {
|
||||
return b.body.Close()
|
||||
}
|
||||
|
||||
// upgradedConn is the connection to the app once the app has switched
|
||||
// protocols, as for a WebSocket. ReverseProxy writes to it what the client
|
||||
// sends and reads from it what the app sends, on goroutines of its own,
|
||||
// until the connection closes; it counts the bytes each way, for the byte
|
||||
// limits.
|
||||
type upgradedConn struct {
|
||||
io.ReadWriteCloser
|
||||
|
||||
// fromApp is how many bytes the app has sent, and toApp how many the
|
||||
// client has.
|
||||
fromApp atomic.Int64
|
||||
toApp atomic.Int64
|
||||
}
|
||||
|
||||
// Read reads what the app sends.
|
||||
func (c *upgradedConn) Read(p []byte) (int, error) {
|
||||
n, err := c.ReadWriteCloser.Read(p)
|
||||
c.fromApp.Add(int64(n))
|
||||
|
||||
return n, err
|
||||
}
|
||||
|
||||
// Write sends the app what the client sent.
|
||||
func (c *upgradedConn) Write(p []byte) (int, error) {
|
||||
n, err := c.ReadWriteCloser.Write(p)
|
||||
c.toApp.Add(int64(n))
|
||||
|
||||
return n, err
|
||||
}
|
||||
|
||||
// CloseWrite tells the app that the client sends no more, while what the
|
||||
// app sends still passes. ReverseProxy calls it once the client has
|
||||
// stopped sending, and closes the connection there if it is not supported.
|
||||
func (c *upgradedConn) CloseWrite() error {
|
||||
conn, ok := c.ReadWriteCloser.(interface{ CloseWrite() error })
|
||||
if !ok {
|
||||
return http.ErrNotSupported
|
||||
}
|
||||
|
||||
return conn.CloseWrite()
|
||||
}
|
||||
|
||||
// limitBody returns body, cut off with an *http.MaxBytesError after
|
||||
// maxBytes, or unchanged if maxBytes is zero, which is off.
|
||||
func limitBody(body io.ReadCloser, maxBytes int64) io.ReadCloser {
|
||||
|
||||
@@ -0,0 +1,583 @@
|
||||
package proxy_test
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// The byte limit settings.
|
||||
const (
|
||||
bytesLimitPerMinute = "SWWAF_BYTES_LIMIT_PER_MINUTE"
|
||||
bytesLimitPerHour = "SWWAF_BYTES_LIMIT_PER_HOUR"
|
||||
bytesLimitPerDay = "SWWAF_BYTES_LIMIT_PER_DAY"
|
||||
bytesCount = "SWWAF_BYTES_COUNT"
|
||||
)
|
||||
|
||||
// The values of SWWAF_BYTES_COUNT.
|
||||
const (
|
||||
countResponse = "response"
|
||||
countRequest = "request"
|
||||
countBoth = "both"
|
||||
)
|
||||
|
||||
const (
|
||||
// bodyBytes is the size of the body of each request these tests send
|
||||
// with one, and answerBytes that of each answer of the app.
|
||||
bodyBytes = 30
|
||||
answerBytes = 70
|
||||
// byteLimit is the byte limit these tests set, as a setting: a request
|
||||
// with a body and its answer, 100 bytes, go over it.
|
||||
byteLimit = "99"
|
||||
// minuteBytes is limit_hit for SWWAF_BYTES_LIMIT_PER_MINUTE.
|
||||
minuteBytes = "minute_bytes"
|
||||
)
|
||||
|
||||
func TestEachByteLimitBansOnceTheResponseHasEnded(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const scraper = "192.0.2.200"
|
||||
|
||||
for _, tc := range []struct {
|
||||
setting, window string
|
||||
// apart is the time between the two requests, which the window
|
||||
// still covers.
|
||||
apart time.Duration
|
||||
}{
|
||||
{bytesLimitPerMinute, minute, 0},
|
||||
{bytesLimitPerHour, "hour", 2 * time.Minute},
|
||||
{bytesLimitPerDay, "day", 2 * time.Hour},
|
||||
} {
|
||||
t.Run(tc.setting, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk := startWithAnswers(t, map[string]string{
|
||||
tc.setting: byteLimit, metricsToken: token,
|
||||
})
|
||||
|
||||
// 70 bytes are within the limit of 99.
|
||||
line, _ := s.download()
|
||||
if line.LimitHit != "" || line.Offence != "" {
|
||||
t.Errorf("log line has limit_hit %q and offence %q, want neither",
|
||||
line.LimitHit, line.Offence)
|
||||
}
|
||||
|
||||
// 140 bytes are over it. The response is passed on whole, and
|
||||
// then bans the client for an hour.
|
||||
clk.advance(tc.apart)
|
||||
expires := requestlog.FormatTime(clk.Now().Add(time.Hour))
|
||||
|
||||
line, got := s.download()
|
||||
if got.err != nil || len(got.body) != answerBytes ||
|
||||
line.ResponseBytes != answerBytes {
|
||||
t.Errorf("got %d bytes (%v), and the log line has response_bytes %d, "+
|
||||
"want %d", len(got.body), got.err, line.ResponseBytes, answerBytes)
|
||||
}
|
||||
|
||||
if line.LimitHit != tc.window+"_bytes" || line.Offence != requestlog.OffenceLimit ||
|
||||
line.BanExpires != expires {
|
||||
t.Errorf("log line has limit_hit %q, offence %q and ban_expires %q, "+
|
||||
"want %s_bytes, limit and %s", line.LimitHit, line.Offence,
|
||||
line.BanExpires, tc.window, expires)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
|
||||
wantMetric(t, s.scrape(scraper), `smallwebwaf_rate_limit_hits_total{`+
|
||||
`instance="`+alertInstance+`",kind="bytes",window="`+tc.window+`"}`, 1)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResponseOverAByteLimitByItselfIsPassedOnWhole(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{bytesLimitPerMinute: "50"})
|
||||
|
||||
// The answer's 70 bytes are over the limit of 50 on their own.
|
||||
line, got := s.download()
|
||||
if got.err != nil || len(got.body) != answerBytes || line.LimitHit != minuteBytes {
|
||||
t.Errorf("got %d bytes (%v), and the log line has limit_hit %q, want %d and %s",
|
||||
len(got.body), got.err, line.LimitHit, answerBytes, minuteBytes)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
}
|
||||
|
||||
func TestBytesOfAnAnswerThatBreaksOffAreCounted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, _, _ := startAppWithAlerts(t, breakOff, map[string]string{
|
||||
bytesLimitPerMinute: "50",
|
||||
})
|
||||
expires := requestlog.FormatTime(clk.Now().Add(time.Hour))
|
||||
|
||||
// The 70 bytes passed on before the app broke off are over the limit of
|
||||
// 50, and ban the client for an hour.
|
||||
line, got := s.requestWithBody(http.MethodGet, client, "/", "", "", http.StatusOK,
|
||||
requestlog.ActionUpstreamError)
|
||||
if len(got.body) != answerBytes || line.LimitHit != minuteBytes ||
|
||||
line.BanExpires != expires {
|
||||
t.Errorf("got %d bytes, and the log line has limit_hit %q and ban_expires %q, "+
|
||||
"want %d, %s and %s", len(got.body), line.LimitHit, line.BanExpires,
|
||||
answerBytes, minuteBytes, expires)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
}
|
||||
|
||||
func TestWebSocketBytesAreCountedOnceItCloses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
setting string
|
||||
counted float64
|
||||
}{
|
||||
{countResponse, answerBytes},
|
||||
{countRequest, bodyBytes},
|
||||
{countBoth, bodyBytes + answerBytes},
|
||||
} {
|
||||
t.Run(tc.setting, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _, _ := startAppWithAlerts(t, answerAfterUpgrade, map[string]string{
|
||||
bytesLimitPerMinute: "29", bytesCount: tc.setting,
|
||||
})
|
||||
|
||||
// The client sends 30 bytes and the app 70, each over the limit
|
||||
// of 29, which bans the client once the WebSocket has closed.
|
||||
line := s.webSocket()
|
||||
if line.LimitHit != minuteBytes || line.Counts.MinuteBytes != tc.counted {
|
||||
t.Errorf("log line has limit_hit %q and minute_bytes %v, want %s and %v",
|
||||
line.LimitHit, line.Counts.MinuteBytes, minuteBytes, tc.counted)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestWebSocketPassesTheAnswerAfterTheClientStopsSending(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
app := startApp(t, echoOnceTheClientStops)
|
||||
addr, out := startProxy(t, app.URL,
|
||||
map[string]string{trustedProxies: trustLocalhost})
|
||||
s := &sender{t: t, addr: addr, out: out}
|
||||
|
||||
conn, reader := s.openWebSocket()
|
||||
send(t, conn, uploadBody)
|
||||
|
||||
// The client closes its sending side and waits for the answer, which the
|
||||
// app sends only once it has seen the client stop. smallwebwaf passes the
|
||||
// close on to the app through CloseWrite on upgradedConn; without that,
|
||||
// it closes both connections, and the answer is lost.
|
||||
tcp, ok := conn.(*net.TCPConn)
|
||||
if !ok {
|
||||
t.Fatalf("connection is a %T, want a *net.TCPConn", conn)
|
||||
}
|
||||
|
||||
err := tcp.CloseWrite()
|
||||
if err != nil {
|
||||
t.Fatalf("close the sending side: %v", err)
|
||||
}
|
||||
|
||||
got, err := io.ReadAll(reader)
|
||||
if err != nil || string(got) != uploadBody {
|
||||
t.Errorf("got %q (%v), want %q", got, err, uploadBody)
|
||||
}
|
||||
|
||||
s.closeWebSocket(conn)
|
||||
}
|
||||
|
||||
func TestBytesCountSaysWhichBytesCount(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
setting string
|
||||
// each is the bytes each request counts, and breaking the request
|
||||
// that goes over the limit of 99.
|
||||
each float64
|
||||
breaking int
|
||||
}{
|
||||
{countResponse, answerBytes, 2},
|
||||
{countRequest, bodyBytes, 4},
|
||||
{countBoth, bodyBytes + answerBytes, 1},
|
||||
} {
|
||||
t.Run(tc.setting, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{
|
||||
bytesLimitPerMinute: byteLimit, bytesCount: tc.setting,
|
||||
})
|
||||
|
||||
for i := 1; i <= tc.breaking; i++ {
|
||||
line := s.upload()
|
||||
|
||||
want := ""
|
||||
if i == tc.breaking {
|
||||
want = minuteBytes
|
||||
}
|
||||
|
||||
counted := float64(i) * tc.each
|
||||
if line.LimitHit != want || line.Counts.MinuteBytes != counted {
|
||||
t.Errorf("request %d: log line has limit_hit %q and minute_bytes %v, "+
|
||||
"want %q and %v", i, line.LimitHit, line.Counts.MinuteBytes,
|
||||
want, counted)
|
||||
}
|
||||
}
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestByteLimitsLeaveOutWhatTheRateLimitsLeaveOut(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
allowed = "192.0.2.7" // in SWWAF_ALLOW_NETS
|
||||
exempt = "192.0.2.10" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
|
||||
)
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{
|
||||
bytesLimitPerMinute: byteLimit,
|
||||
allowNets: allowed,
|
||||
rateLimitExemptNets: exempt,
|
||||
rateLimitExemptPaths: "/assets/",
|
||||
})
|
||||
|
||||
// Each sends 200 bytes, none of which is counted.
|
||||
for _, sent := range []struct{ from, path string }{
|
||||
{allowed, "/"}, {exempt, "/"}, {client, "/assets/app.js"},
|
||||
} {
|
||||
for range 2 {
|
||||
line, _ := s.requestWithBody(http.MethodPost, sent.from, sent.path,
|
||||
uploadHeader, uploadBody, http.StatusOK, requestlog.ActionForward)
|
||||
if _, counted := line.fields["counts"]; counted || line.LimitHit != "" {
|
||||
t.Errorf("%s %s: log line has counts %v and limit_hit %q, want neither",
|
||||
sent.from, sent.path, line.fields["counts"], line.LimitHit)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A path that is not exempt is counted, and breaks the limit.
|
||||
line := s.upload()
|
||||
if line.LimitHit != minuteBytes {
|
||||
t.Errorf("log line has limit_hit %q, want %s", line.LimitHit, minuteBytes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestByteLimitsOffCountTheBytesAndBanNoOne(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const off = "off"
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{
|
||||
bytesLimitPerMinute: off, bytesLimitPerHour: off, bytesLimitPerDay: off,
|
||||
})
|
||||
|
||||
for i := 1; i <= 3; i++ {
|
||||
line := s.upload()
|
||||
|
||||
counted := float64(i * (bodyBytes + answerBytes))
|
||||
if line.LimitHit != "" || line.Counts.MinuteBytes != counted ||
|
||||
line.Counts.HourBytes != counted || line.Counts.DayBytes != counted {
|
||||
t.Errorf("request %d: log line has limit_hit %q and counts %+v, "+
|
||||
"want none and %v bytes in each window", i, line.LimitHit,
|
||||
line.Counts, counted)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBanForABrokenByteLimitHasItsNotesAndItsAlert(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server, queue := startAppWithAlerts(t, readAndAnswer, map[string]string{
|
||||
bytesLimitPerMinute: byteLimit,
|
||||
})
|
||||
start := clk.Now()
|
||||
|
||||
s.requestWithBody(http.MethodPost, client, "/upload?part=1", uploadHeader,
|
||||
uploadBody, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
netblock := netip.MustParsePrefix(client + "/32")
|
||||
want := bans.Ban{
|
||||
Netblock: netblock,
|
||||
Start: start,
|
||||
Expires: start.Add(time.Hour),
|
||||
Cause: bans.CauseLimit,
|
||||
Reason: "bytes per minute over the limit of " + byteLimit,
|
||||
Notes: bans.Notes{
|
||||
Kind: "bytes",
|
||||
Limit: 99,
|
||||
Window: minute,
|
||||
Count: bodyBytes + answerBytes,
|
||||
// The request as it was answered, by the app.
|
||||
Request: bans.Request{
|
||||
Time: start,
|
||||
Method: http.MethodPost,
|
||||
Host: appHost,
|
||||
Path: "/upload?part=1",
|
||||
Status: http.StatusOK,
|
||||
UserAgent: userAgent,
|
||||
},
|
||||
Requests: 1,
|
||||
},
|
||||
}
|
||||
|
||||
got := server.Ledger.Bans(netblock)
|
||||
if len(got) != 1 || got[0] != want {
|
||||
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
|
||||
}
|
||||
|
||||
wantAlerts(t, queue, banAlert(alerts.EventBan, start, client, want,
|
||||
requestlog.FormatTime(want.Expires)))
|
||||
|
||||
if offences := historyOf(t, server, client).Offences.Limit; offences != 1 {
|
||||
t.Errorf("history counts %d offences for a limit, want 1", offences)
|
||||
}
|
||||
}
|
||||
|
||||
func TestObserveModeLogsAndAlertsAByteLimitAndBansNoOne(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server, queue := startAppWithAlerts(t, readAndAnswer, map[string]string{
|
||||
mode: observe,
|
||||
bytesLimitPerMinute: byteLimit,
|
||||
})
|
||||
start := clk.Now()
|
||||
|
||||
// No ban sets the client's counters back to zero, so each request
|
||||
// breaks the limit again. The answer is the app's either way, and the
|
||||
// alert for the ban is not sent twice within the cooldown.
|
||||
for range 2 {
|
||||
line := s.upload()
|
||||
wantWouldAction(t, line, "")
|
||||
|
||||
if line.LimitHit != minuteBytes || line.Offence != requestlog.OffenceLimit ||
|
||||
line.BanExpires != "" {
|
||||
t.Errorf("log line has limit_hit %q, offence %q and ban_expires %q, "+
|
||||
"want %s, limit and none", line.LimitHit, line.Offence, line.BanExpires,
|
||||
minuteBytes)
|
||||
}
|
||||
}
|
||||
|
||||
if held := server.Ledger.Snapshot(); len(held) != 0 {
|
||||
t.Errorf("the ledger holds %+v, want no ban", held)
|
||||
}
|
||||
|
||||
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
if len(waiting) != 1 {
|
||||
t.Fatalf("%d alerts wait, want 1: %+v", len(waiting), waiting)
|
||||
}
|
||||
|
||||
notes, _ := waiting[0].Detail["notes"].(bans.Notes)
|
||||
alert := banAlert(alerts.EventBan, start, client, bans.Ban{
|
||||
Netblock: netip.MustParsePrefix(client + "/32"), Cause: bans.CauseLimit,
|
||||
Reason: "bytes per minute over the limit of " + byteLimit, Notes: notes,
|
||||
}, requestlog.FormatTime(start.Add(time.Hour)))
|
||||
alert.Detail["mode"] = observe
|
||||
wantAlerts(t, queue, alert)
|
||||
}
|
||||
|
||||
func TestObserveModeLeavesOutTheBytesOfARequestEnforceModeRefuses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{
|
||||
mode: observe,
|
||||
rateLimitPerMinute: "1",
|
||||
bytesLimitPerMinute: "150",
|
||||
})
|
||||
|
||||
s.upload()
|
||||
|
||||
// The second request breaks the rate limit, which in enforce mode would
|
||||
// refuse it before the app sent anything, so its 100 bytes are not
|
||||
// counted, and the byte limit is not broken. Its line gives the bytes
|
||||
// counted before it.
|
||||
line := s.upload()
|
||||
wantWouldAction(t, line, requestlog.ActionRateLimited)
|
||||
|
||||
if line.LimitHit != minute || line.Counts.MinuteBytes != bodyBytes+answerBytes {
|
||||
t.Errorf("log line has limit_hit %q and minute_bytes %v, want minute and %d",
|
||||
line.LimitHit, line.Counts.MinuteBytes, bodyBytes+answerBytes)
|
||||
}
|
||||
}
|
||||
|
||||
// uploadHeader and uploadBody are the header and the body of a request
|
||||
// with a body of bodyBytes.
|
||||
//
|
||||
//nolint:gochecknoglobals // a constant cannot call strings.Repeat
|
||||
var (
|
||||
uploadHeader = "Content-Length: " + strconv.Itoa(bodyBytes)
|
||||
uploadBody = strings.Repeat("u", bodyBytes)
|
||||
)
|
||||
|
||||
// readAndAnswer is the app of these tests: it reads each request's whole
|
||||
// body and answers with answerBytes bytes.
|
||||
func readAndAnswer(w http.ResponseWriter, r *http.Request) {
|
||||
_, _ = io.Copy(io.Discard, r.Body)
|
||||
_, _ = io.WriteString(w, strings.Repeat("a", answerBytes))
|
||||
}
|
||||
|
||||
// breakOff is an app that announces an answer of twice answerBytes, and
|
||||
// breaks off after answerBytes.
|
||||
func breakOff(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Length", strconv.Itoa(2*answerBytes))
|
||||
_, _ = io.WriteString(w, strings.Repeat("a", answerBytes))
|
||||
}
|
||||
|
||||
// answerAfterUpgrade is an app that switches protocols, as for a
|
||||
// WebSocket, and then answers each line it receives with a line of
|
||||
// answerBytes.
|
||||
func answerAfterUpgrade(w http.ResponseWriter, _ *http.Request) {
|
||||
conn, buffered, err := http.NewResponseController(w).Hijack()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
defer func() {
|
||||
_ = conn.Close()
|
||||
}()
|
||||
|
||||
_, _ = buffered.WriteString("HTTP/1.1 101 Switching Protocols\r\n" +
|
||||
"Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
|
||||
_ = buffered.Flush()
|
||||
|
||||
for {
|
||||
_, err := buffered.ReadString('\n')
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
_, _ = buffered.WriteString(strings.Repeat("a", answerBytes-1) + "\n")
|
||||
_ = buffered.Flush()
|
||||
}
|
||||
}
|
||||
|
||||
// echoOnceTheClientStops is an app that switches protocols, as for a
|
||||
// WebSocket, reads what the client sends until the client stops sending,
|
||||
// and then sends it all back.
|
||||
func echoOnceTheClientStops(w http.ResponseWriter, _ *http.Request) {
|
||||
conn, buffered, err := http.NewResponseController(w).Hijack()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
defer func() {
|
||||
_ = conn.Close()
|
||||
}()
|
||||
|
||||
_, _ = buffered.WriteString("HTTP/1.1 101 Switching Protocols\r\n" +
|
||||
"Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
|
||||
_ = buffered.Flush()
|
||||
|
||||
received, _ := io.ReadAll(buffered)
|
||||
_, _ = buffered.Write(received)
|
||||
_ = buffered.Flush()
|
||||
}
|
||||
|
||||
// webSocket opens a WebSocket from client to answerAfterUpgrade, sends a
|
||||
// line of bodyBytes on it, reads the answer, and closes it. It checks the
|
||||
// answer, and the log line as request does, and returns the log line.
|
||||
func (s *sender) webSocket() logLine {
|
||||
s.t.Helper()
|
||||
|
||||
conn, reader := s.openWebSocket()
|
||||
send(s.t, conn, strings.Repeat("u", bodyBytes-1)+"\n")
|
||||
|
||||
got, err := reader.ReadString('\n')
|
||||
if err != nil || len(got) != answerBytes {
|
||||
s.t.Errorf("got %d bytes (%v), want %d", len(got), err, answerBytes)
|
||||
}
|
||||
|
||||
return s.closeWebSocket(conn)
|
||||
}
|
||||
|
||||
// openWebSocket sends a request from client to switch protocols, as for a
|
||||
// WebSocket, and checks that the app switches. It returns the connection,
|
||||
// on which reading fails once waitLimit has passed, and a reader of what
|
||||
// the app sends on it.
|
||||
func (s *sender) openWebSocket() (net.Conn, *bufio.Reader) {
|
||||
s.t.Helper()
|
||||
|
||||
conn := dial(s.t, s.addr)
|
||||
send(s.t, conn, "GET /socket HTTP/1.1\r\nHost: "+appHost+"\r\n"+forwardedFor+
|
||||
": "+client+"\r\nConnection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
|
||||
|
||||
err := conn.SetReadDeadline(time.Now().Add(waitLimit))
|
||||
if err != nil {
|
||||
s.t.Fatalf("set read deadline: %v", err)
|
||||
}
|
||||
|
||||
reader := bufio.NewReader(conn)
|
||||
|
||||
res, err := http.ReadResponse(reader, nil)
|
||||
if err != nil {
|
||||
s.t.Fatalf("read the answer to the upgrade: %v", err)
|
||||
}
|
||||
|
||||
_ = res.Body.Close()
|
||||
|
||||
if res.StatusCode != http.StatusSwitchingProtocols {
|
||||
s.t.Fatalf("status %d, want %d", res.StatusCode, http.StatusSwitchingProtocols)
|
||||
}
|
||||
|
||||
return conn, reader
|
||||
}
|
||||
|
||||
// closeWebSocket closes conn, a WebSocket openWebSocket opened, checks its
|
||||
// log line as request does, and returns it.
|
||||
func (s *sender) closeWebSocket(conn net.Conn) logLine {
|
||||
s.t.Helper()
|
||||
|
||||
_ = conn.Close()
|
||||
|
||||
line := s.out.requestLines(s.t, s.sent+1)[s.sent]
|
||||
s.sent++
|
||||
wantLine(s.t, line, http.StatusSwitchingProtocols, requestlog.ActionForward)
|
||||
|
||||
return line
|
||||
}
|
||||
|
||||
// startWithAnswers is startAppWithAlerts in front of readAndAnswer, for a
|
||||
// test that looks at neither the server nor the alerts.
|
||||
func startWithAnswers(t *testing.T, env map[string]string) (*sender, *clock) {
|
||||
t.Helper()
|
||||
|
||||
s, clk, _, _ := startAppWithAlerts(t, readAndAnswer, env)
|
||||
|
||||
return s, clk
|
||||
}
|
||||
|
||||
// download sends a GET request for / from client, and checks that the
|
||||
// app's answer is passed on, as request does. It returns the log line and
|
||||
// the answer.
|
||||
func (s *sender) download() (logLine, answer) {
|
||||
s.t.Helper()
|
||||
|
||||
return s.requestWithBody(http.MethodGet, client, "/", "", "", http.StatusOK,
|
||||
requestlog.ActionForward)
|
||||
}
|
||||
|
||||
// upload is download for a POST request with a body of bodyBytes, and
|
||||
// returns the log line.
|
||||
func (s *sender) upload() logLine {
|
||||
s.t.Helper()
|
||||
|
||||
line, _ := s.requestWithBody(http.MethodPost, client, "/", uploadHeader,
|
||||
uploadBody, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
return line
|
||||
}
|
||||
@@ -222,8 +222,8 @@ func TestMetricsCountLimitsAndBans(t *testing.T) {
|
||||
metrics := s.scrape(scraper)
|
||||
wantMetric(t, metrics,
|
||||
`smallwebwaf_requests_total{action="denied",instance="app",status_class="none"}`, 1)
|
||||
wantMetric(t, metrics,
|
||||
`smallwebwaf_rate_limit_hits_total{instance="app",window="minute"}`, 1)
|
||||
wantMetric(t, metrics, `smallwebwaf_rate_limit_hits_total{instance="app",`+
|
||||
`kind="requests",window="minute"}`, 1)
|
||||
wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="limit"}`, 1)
|
||||
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 1)
|
||||
wantMetric(t, metrics, `smallwebwaf_active_bans{instance="app"}`, 1)
|
||||
@@ -238,8 +238,8 @@ func TestMetricsCountLimitsAndBans(t *testing.T) {
|
||||
s.get(client, 0, requestlog.ActionRateLimited)
|
||||
|
||||
metrics = s.scrape(scraper)
|
||||
wantMetric(t, metrics,
|
||||
`smallwebwaf_rate_limit_hits_total{instance="app",window="minute"}`, 2)
|
||||
wantMetric(t, metrics, `smallwebwaf_rate_limit_hits_total{instance="app",`+
|
||||
`kind="requests",window="minute"}`, 2)
|
||||
wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="limit"}`, 2)
|
||||
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 2)
|
||||
wantMetric(t, metrics, `smallwebwaf_active_bans{instance="app"}`, 1)
|
||||
|
||||
@@ -122,6 +122,8 @@ func wantAnswer(t *testing.T, got answer, body []byte) {
|
||||
func wantRequestFields(t *testing.T, line logLine, host string, sent, received int) {
|
||||
t.Helper()
|
||||
|
||||
bytes := float64(sent + received)
|
||||
|
||||
want := withTimings(line, requestlog.Line{
|
||||
Type: requestType, Time: line.Time, Instance: "app",
|
||||
ClientIP: localhost, Method: http.MethodPatch, Scheme: plain, Host: host,
|
||||
@@ -131,7 +133,10 @@ func wantRequestFields(t *testing.T, line logLine, host string, sent, received i
|
||||
RequestID: line.RequestID, PeerIP: localhost, ClientGroup: localhost + "/32",
|
||||
ContentLength: int64(sent), ResponseContentType: "text/plain; charset=utf-8",
|
||||
UpstreamStatus: http.StatusTeapot, Action: requestlog.ActionForward,
|
||||
Counts: ratelimit.Counts{Minute: 1, Hour: 1, Day: 1},
|
||||
Counts: ratelimit.Counts{
|
||||
Minute: 1, Hour: 1, Day: 1,
|
||||
MinuteBytes: bytes, HourBytes: bytes, DayBytes: bytes,
|
||||
},
|
||||
})
|
||||
if !reflect.DeepEqual(line.Line, want) {
|
||||
t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
|
||||
|
||||
+11
-3
@@ -103,9 +103,12 @@ func New(params Params) *Server {
|
||||
now: params.Now,
|
||||
metrics: m,
|
||||
limiter: ratelimit.New(ratelimit.Limits{
|
||||
PerMinute: params.Config.RateLimitPerMinute,
|
||||
PerHour: params.Config.RateLimitPerHour,
|
||||
PerDay: params.Config.RateLimitPerDay,
|
||||
PerMinute: params.Config.RateLimitPerMinute,
|
||||
PerHour: params.Config.RateLimitPerHour,
|
||||
PerDay: params.Config.RateLimitPerDay,
|
||||
BytesPerMinute: params.Config.BytesLimitPerMinute,
|
||||
BytesPerHour: params.Config.BytesLimitPerHour,
|
||||
BytesPerDay: params.Config.BytesLimitPerDay,
|
||||
}),
|
||||
ledger: bans.New(bans.Rules{
|
||||
LimitBanDuration: params.Config.LimitBanDuration,
|
||||
@@ -226,5 +229,10 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
// Once the response has ended, before the request is added to its
|
||||
// client's history. Deferred, since ReverseProxy panics to end a
|
||||
// response it cannot finish.
|
||||
defer rq.countBytes()
|
||||
|
||||
rq.forward(r.Context())
|
||||
}
|
||||
|
||||
+33
-17
@@ -3,6 +3,7 @@ package proxy
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptrace"
|
||||
"net/http/httputil"
|
||||
@@ -54,7 +55,10 @@ type request struct {
|
||||
// what the lookup gave then, the zero Answer while GeoJS had given none.
|
||||
lookedUp bool
|
||||
lookupAnswer lookup.Answer
|
||||
start time.Time
|
||||
// counted is true for a request the rate limits counted, whose bytes
|
||||
// the byte limits count once it has ended.
|
||||
counted bool
|
||||
start time.Time
|
||||
// checked is when the checks were done, and upstreamStart when the
|
||||
// request was handed to the app.
|
||||
checked time.Time
|
||||
@@ -65,6 +69,9 @@ type request struct {
|
||||
refused atomic.Pointer[refusal]
|
||||
// complete is true once the app's whole answer has been passed on.
|
||||
complete bool
|
||||
// upgraded is the connection to the app once the app has switched
|
||||
// protocols, as for a WebSocket, and nil otherwise.
|
||||
upgraded *upgradedConn
|
||||
|
||||
// mu guards what follows. The timeouts run on goroutines of their
|
||||
// own, and the transport starts and stops them, and notes the times
|
||||
@@ -205,8 +212,9 @@ func (rq *request) check(ctx context.Context) *refusal {
|
||||
// them refuses is not counted for the rate limits. Then come the rate
|
||||
// limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the
|
||||
// request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that
|
||||
// every other request is counted, and last the rule files. ctx is the
|
||||
// request's own context.
|
||||
// every other request is counted, and last the rule files. A request
|
||||
// exempt from the rate limits is exempt from the byte limits too. ctx is
|
||||
// the request's own context.
|
||||
func (rq *request) checkClient(ctx context.Context) string {
|
||||
cfg := rq.h.config
|
||||
if isInside(rq.client, cfg.AllowNets) {
|
||||
@@ -229,9 +237,9 @@ func (rq *request) checkClient(ctx context.Context) string {
|
||||
return requestlog.ActionCountryDenied
|
||||
}
|
||||
|
||||
exempt := isInside(rq.client, cfg.RateLimitExemptNets) ||
|
||||
pathExempt(rq.in.URL, cfg.RateLimitExemptPaths)
|
||||
if !exempt && rq.limitBroken(now) {
|
||||
rq.counted = !isInside(rq.client, cfg.RateLimitExemptNets) &&
|
||||
!pathExempt(rq.in.URL, cfg.RateLimitExemptPaths)
|
||||
if rq.counted && rq.limitBroken(now) {
|
||||
return requestlog.ActionRateLimited
|
||||
}
|
||||
|
||||
@@ -324,11 +332,18 @@ func (rq *request) modifyResponse(res *http.Response) error {
|
||||
if res.StatusCode == http.StatusSwitchingProtocols {
|
||||
// An upgraded connection, such as a WebSocket, is not cut by the
|
||||
// timeouts. ReverseProxy writes this answer straight to the
|
||||
// connection it takes over, not through rq.out.
|
||||
// connection it takes over, not through rq.out, and then copies
|
||||
// what passes each way through res.Body, the connection to the app.
|
||||
rq.stopTimers()
|
||||
rq.out.status = res.StatusCode
|
||||
rq.line.Websocket = true
|
||||
|
||||
conn, ok := res.Body.(io.ReadWriteCloser)
|
||||
if ok {
|
||||
rq.upgraded = &upgradedConn{ReadWriteCloser: conn}
|
||||
res.Body = rq.upgraded
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -431,10 +446,7 @@ func (rq *request) finish() {
|
||||
line.ResponseContentType = header.Get("Content-Type")
|
||||
line.CacheControl = header.Get("Cache-Control")
|
||||
line.Location = header.Get("Location")
|
||||
|
||||
if rq.body != nil {
|
||||
line.RequestBytes = rq.body.bytes.Load()
|
||||
}
|
||||
line.RequestBytes = rq.requestBytes()
|
||||
|
||||
// limit is the setting whose size or time limit the request passed.
|
||||
var limit string
|
||||
@@ -497,11 +509,6 @@ func timing(start, end time.Time) *float64 {
|
||||
// may have come before either was there, and one from GeoJS that comes
|
||||
// later is added when it comes.
|
||||
func (rq *request) addToHistory() {
|
||||
var requestBytes int64
|
||||
if rq.body != nil {
|
||||
requestBytes = rq.body.bytes.Load()
|
||||
}
|
||||
|
||||
forwarded := !rq.upstreamStart.IsZero()
|
||||
group := clientGroup(rq.client)
|
||||
|
||||
@@ -509,7 +516,7 @@ func (rq *request) addToHistory() {
|
||||
Forwarded: forwarded,
|
||||
Refused: !forwarded && rq.refused.Load() != nil,
|
||||
Status: rq.out.status,
|
||||
RequestBytes: requestBytes,
|
||||
RequestBytes: rq.requestBytes(),
|
||||
ResponseBytes: rq.out.bytes,
|
||||
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
|
||||
})
|
||||
@@ -531,6 +538,15 @@ func (rq *request) addToHistory() {
|
||||
}
|
||||
}
|
||||
|
||||
// requestBytes is how many bytes of the request's body have been read.
|
||||
func (rq *request) requestBytes() int64 {
|
||||
if rq.body == nil {
|
||||
return 0
|
||||
}
|
||||
|
||||
return rq.body.bytes.Load()
|
||||
}
|
||||
|
||||
// clientRequestDeadline is when the client must have sent its whole
|
||||
// request, or zero when SWWAF_CLIENT_REQUEST_TIMEOUT is off.
|
||||
func (rq *request) clientRequestDeadline() time.Time {
|
||||
|
||||
@@ -119,7 +119,10 @@ func wantFullLine(t *testing.T, line logLine) {
|
||||
ResponseContentType: "text/html", UpstreamStatus: http.StatusFound,
|
||||
CacheControl: "no-store", Location: "/elsewhere",
|
||||
Action: requestlog.ActionForward,
|
||||
Counts: ratelimit.Counts{Minute: 1, Hour: 1, Day: 1},
|
||||
// Its 3 bytes in and 5 out, each way counted by default.
|
||||
Counts: ratelimit.Counts{
|
||||
Minute: 1, Hour: 1, Day: 1, MinuteBytes: 8, HourBytes: 8, DayBytes: 8,
|
||||
},
|
||||
})
|
||||
if !reflect.DeepEqual(line.Line, want) {
|
||||
t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
|
||||
|
||||
+159
-78
@@ -1,9 +1,9 @@
|
||||
// Package ratelimit keeps the table of clients: each client's requests
|
||||
// counted over a minute, an hour and a day, as the "Counting method"
|
||||
// section of SPEC.md describes, which tell when a request takes the client
|
||||
// over a rate limit, and each client's history since it was first seen.
|
||||
// At most 20,000 clients are kept, in memory, and written to clients.json
|
||||
// and read from it by the state package.
|
||||
// and bytes counted over a minute, an hour and a day, as the "Counting
|
||||
// method" section of SPEC.md describes, which tell when a request takes
|
||||
// the client over a rate limit or a byte limit, and each client's history
|
||||
// since it was first seen. At most 20,000 clients are kept, in memory, and
|
||||
// written to clients.json and read from it by the state package.
|
||||
package ratelimit
|
||||
|
||||
import (
|
||||
@@ -23,19 +23,30 @@ const maxClients = 20000
|
||||
|
||||
const day = 24 * time.Hour
|
||||
|
||||
// The kinds of limits, as the metrics name them.
|
||||
const (
|
||||
// KindRequests is a rate limit, on a client's requests.
|
||||
KindRequests = "requests"
|
||||
// KindBytes is a byte limit, on a client's bytes.
|
||||
KindBytes = "bytes"
|
||||
)
|
||||
|
||||
// Limits are the most requests a client may make in a minute, an hour and
|
||||
// a day. Zero is no limit.
|
||||
// a day, and the most bytes. Zero is no limit.
|
||||
type Limits struct {
|
||||
PerMinute int64
|
||||
PerHour int64
|
||||
PerDay int64
|
||||
PerMinute int64
|
||||
PerHour int64
|
||||
PerDay int64
|
||||
BytesPerMinute int64
|
||||
BytesPerHour int64
|
||||
BytesPerDay int64
|
||||
}
|
||||
|
||||
// Limiter counts each client's requests against the limits, and keeps
|
||||
// its history. It is safe for concurrent use.
|
||||
// Limiter counts each client's requests and bytes against the limits, and
|
||||
// keeps its history. It is safe for concurrent use.
|
||||
type Limiter struct {
|
||||
// windows are the minute, the hour and the day, in the order of
|
||||
// Client.buckets.
|
||||
// Client.buckets and Client.byteBuckets.
|
||||
windows [3]window
|
||||
|
||||
mu sync.Mutex
|
||||
@@ -43,17 +54,23 @@ type Limiter struct {
|
||||
}
|
||||
|
||||
// Client is a client in the table, as clients.json holds it: its buckets
|
||||
// in each window, and its history.
|
||||
// of requests and of bytes in each window, and its history.
|
||||
//
|
||||
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
||||
type Client struct {
|
||||
Client netip.Prefix `json:"client"`
|
||||
Minute Buckets `json:"minute"`
|
||||
Hour Buckets `json:"hour"`
|
||||
Day Buckets `json:"day"`
|
||||
History History `json:"history"`
|
||||
Client netip.Prefix `json:"client"`
|
||||
Minute Buckets `json:"minute"`
|
||||
Hour Buckets `json:"hour"`
|
||||
Day Buckets `json:"day"`
|
||||
MinuteBytes Buckets `json:"minute_bytes"`
|
||||
HourBytes Buckets `json:"hour_bytes"`
|
||||
DayBytes Buckets `json:"day_bytes"`
|
||||
History History `json:"history"`
|
||||
}
|
||||
|
||||
// Buckets are a client's two buckets in one window: the requests in the
|
||||
// bucket under way, which began at Start, and in the bucket before it.
|
||||
// Buckets are a client's two buckets in one window: the requests, or the
|
||||
// bytes, in the bucket under way, which began at Start, and in the bucket
|
||||
// before it.
|
||||
type Buckets struct {
|
||||
Start time.Time `json:"start"`
|
||||
Current int64 `json:"current"`
|
||||
@@ -101,7 +118,7 @@ type Responses struct {
|
||||
|
||||
// Offences are a client's offences, by kind.
|
||||
type Offences struct {
|
||||
// Limit is its requests that broke a rate limit.
|
||||
// Limit is its requests that broke a rate limit or a byte limit.
|
||||
Limit int64 `json:"limit"`
|
||||
}
|
||||
|
||||
@@ -119,7 +136,8 @@ type Request struct {
|
||||
// and of its response.
|
||||
RequestBytes int64
|
||||
ResponseBytes int64
|
||||
// BrokeLimit is true for a request that broke a rate limit.
|
||||
// BrokeLimit is true for a request that broke a rate limit or a byte
|
||||
// limit.
|
||||
BrokeLimit bool
|
||||
}
|
||||
|
||||
@@ -132,62 +150,71 @@ func New(limits Limits) *Limiter {
|
||||
|
||||
return &Limiter{
|
||||
windows: [3]window{
|
||||
{name: "minute", length: time.Minute, limit: limits.PerMinute},
|
||||
{name: "hour", length: time.Hour, limit: limits.PerHour},
|
||||
{name: "day", length: day, limit: limits.PerDay},
|
||||
{
|
||||
name: "minute", length: time.Minute,
|
||||
limit: limits.PerMinute, byteLimit: limits.BytesPerMinute,
|
||||
},
|
||||
{
|
||||
name: "hour", length: time.Hour,
|
||||
limit: limits.PerHour, byteLimit: limits.BytesPerHour,
|
||||
},
|
||||
{
|
||||
name: "day", length: day,
|
||||
limit: limits.PerDay, byteLimit: limits.BytesPerDay,
|
||||
},
|
||||
},
|
||||
clients: clients,
|
||||
}
|
||||
}
|
||||
|
||||
// Hit is a request that takes a client over a rate limit.
|
||||
// Hit is a request that takes a client over a rate limit, or whose bytes
|
||||
// take it over a byte limit.
|
||||
type Hit struct {
|
||||
// Kind is KindRequests for a rate limit, KindBytes for a byte limit.
|
||||
Kind string
|
||||
// Window is "minute", "hour" or "day".
|
||||
Window string
|
||||
// Limit is the window's limit.
|
||||
Limit int64
|
||||
// Requests is the client's requests counted in the window, this one
|
||||
// included.
|
||||
Requests float64
|
||||
// Count is the client's requests, or bytes, counted in the window,
|
||||
// this request's included.
|
||||
Count float64
|
||||
}
|
||||
|
||||
// Counts are a client's requests in the minute, the hour and the day that
|
||||
// end at a request, that request included.
|
||||
// Counts are a client's requests and bytes in the minute, the hour and
|
||||
// the day that end at a request, that request's included.
|
||||
//
|
||||
//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
|
||||
type Counts struct {
|
||||
Minute float64 `json:"minute"`
|
||||
Hour float64 `json:"hour"`
|
||||
Day float64 `json:"day"`
|
||||
Minute float64 `json:"minute"`
|
||||
Hour float64 `json:"hour"`
|
||||
Day float64 `json:"day"`
|
||||
MinuteBytes float64 `json:"minute_bytes"`
|
||||
HourBytes float64 `json:"hour_bytes"`
|
||||
DayBytes float64 `json:"day_bytes"`
|
||||
}
|
||||
|
||||
// Count counts a request from client at now, in every window, whether or
|
||||
// not it is refused, and returns the client's requests in each window. It
|
||||
// reports whether the request takes the client over a limit, and the
|
||||
// window whose limit it goes over, the shortest if it is over several.
|
||||
// not it is refused, and returns the client's counts in each window. It
|
||||
// reports whether the request takes the client over a rate limit, and the
|
||||
// hit: the window whose limit it goes over, the shortest if it is over
|
||||
// several.
|
||||
func (l *Limiter) Count(client netip.Prefix, now time.Time) (Counts, Hit, bool) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
var (
|
||||
requests [3]float64
|
||||
hit Hit
|
||||
)
|
||||
|
||||
for i, b := range l.get(client).buckets() {
|
||||
w := l.windows[i]
|
||||
|
||||
requests[i] = b.add(now, w.length)
|
||||
if hit.Window == "" && w.limit > 0 && requests[i] > float64(w.limit) {
|
||||
hit = Hit{Window: w.name, Limit: w.limit, Requests: requests[i]}
|
||||
}
|
||||
}
|
||||
|
||||
counts := Counts{Minute: requests[0], Hour: requests[1], Day: requests[2]}
|
||||
|
||||
return counts, hit, hit.Window != ""
|
||||
return l.count(client, now, 1, 0)
|
||||
}
|
||||
|
||||
// Reset sets client's counts in every window back to zero. Its history
|
||||
// keeps its totals.
|
||||
// CountBytes counts bytes, those of a request from client that has ended,
|
||||
// at now, in every window, and returns the client's counts in each window.
|
||||
// It reports whether the bytes take the client over a byte limit, and the
|
||||
// hit, as Count does.
|
||||
func (l *Limiter) CountBytes(
|
||||
client netip.Prefix, now time.Time, bytes int64,
|
||||
) (Counts, Hit, bool) {
|
||||
return l.count(client, now, 0, bytes)
|
||||
}
|
||||
|
||||
// Reset sets client's counts of requests and of bytes in every window
|
||||
// back to zero. Its history keeps its totals.
|
||||
func (l *Limiter) Reset(client netip.Prefix) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
@@ -195,6 +222,7 @@ func (l *Limiter) Reset(client netip.Prefix) {
|
||||
c, seen := l.clients.Peek(client)
|
||||
if seen {
|
||||
c.Minute, c.Hour, c.Day = Buckets{}, Buckets{}, Buckets{}
|
||||
c.MinuteBytes, c.HourBytes, c.DayBytes = Buckets{}, Buckets{}, Buckets{}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -328,12 +356,13 @@ func (l *Limiter) Load(clients []Client, now time.Time) {
|
||||
l.clients.Purge()
|
||||
|
||||
for _, c := range clients {
|
||||
for i, b := range c.buckets() {
|
||||
// The window that ends at now covers neither bucket once it
|
||||
// begins after the bucket under way has ended.
|
||||
length := l.windows[i].length
|
||||
if !now.Add(-length).Before(b.Start.Add(length)) {
|
||||
*b = Buckets{}
|
||||
for i, w := range l.windows {
|
||||
for _, b := range []*Buckets{c.buckets()[i], c.byteBuckets()[i]} {
|
||||
// The window that ends at now covers neither bucket once it
|
||||
// begins after the bucket under way has ended.
|
||||
if !now.Add(-w.length).Before(b.Start.Add(w.length)) {
|
||||
*b = Buckets{}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -341,6 +370,49 @@ func (l *Limiter) Load(clients []Client, now time.Time) {
|
||||
}
|
||||
}
|
||||
|
||||
// count adds requests and bytes from client at now to its buckets in
|
||||
// every window, and returns its counts. A limit is broken only by what is
|
||||
// added to it, so that a request whose bytes are counted after another of
|
||||
// the client's requests broke a rate limit does not break it too.
|
||||
func (l *Limiter) count(
|
||||
client netip.Prefix, now time.Time, requests, bytes int64,
|
||||
) (Counts, Hit, bool) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
c := l.get(client)
|
||||
requestBuckets, byteBuckets := c.buckets(), c.byteBuckets()
|
||||
|
||||
var (
|
||||
requestCounts, byteCounts [3]float64
|
||||
hit Hit
|
||||
)
|
||||
|
||||
for i, w := range l.windows {
|
||||
requestCounts[i] = requestBuckets[i].add(now, w.length, requests)
|
||||
byteCounts[i] = byteBuckets[i].add(now, w.length, bytes)
|
||||
|
||||
switch {
|
||||
case hit.Window != "":
|
||||
case requests > 0 && w.limit > 0 && requestCounts[i] > float64(w.limit):
|
||||
hit = Hit{
|
||||
Kind: KindRequests, Window: w.name, Limit: w.limit, Count: requestCounts[i],
|
||||
}
|
||||
case bytes > 0 && w.byteLimit > 0 && byteCounts[i] > float64(w.byteLimit):
|
||||
hit = Hit{
|
||||
Kind: KindBytes, Window: w.name, Limit: w.byteLimit, Count: byteCounts[i],
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
counts := Counts{
|
||||
Minute: requestCounts[0], Hour: requestCounts[1], Day: requestCounts[2],
|
||||
MinuteBytes: byteCounts[0], HourBytes: byteCounts[1], DayBytes: byteCounts[2],
|
||||
}
|
||||
|
||||
return counts, hit, hit.Window != ""
|
||||
}
|
||||
|
||||
// get returns client's entry in the table, a new one if it has none, and
|
||||
// makes it the most recently seen.
|
||||
func (l *Limiter) get(client netip.Prefix) *Client {
|
||||
@@ -353,30 +425,39 @@ func (l *Limiter) get(client netip.Prefix) *Client {
|
||||
return c
|
||||
}
|
||||
|
||||
// buckets returns c's buckets in the minute, the hour and the day.
|
||||
// buckets returns c's buckets of requests in the minute, the hour and the
|
||||
// day.
|
||||
func (c *Client) buckets() [3]*Buckets {
|
||||
return [3]*Buckets{&c.Minute, &c.Hour, &c.Day}
|
||||
}
|
||||
|
||||
// window is a length of time over which requests are counted, and the
|
||||
// most requests a client may make in it.
|
||||
type window struct {
|
||||
name string
|
||||
length time.Duration
|
||||
limit int64
|
||||
// byteBuckets returns c's buckets of bytes in the minute, the hour and the
|
||||
// day.
|
||||
func (c *Client) byteBuckets() [3]*Buckets {
|
||||
return [3]*Buckets{&c.MinuteBytes, &c.HourBytes, &c.DayBytes}
|
||||
}
|
||||
|
||||
// add counts a request at now in a window of length, and returns the
|
||||
// client's requests in the window that ends at now: those in the bucket
|
||||
// under way, and those in the bucket before it weighted by how much of
|
||||
// that bucket the window still covers.
|
||||
// window is a length of time over which requests and bytes are counted,
|
||||
// and the most requests and the most bytes a client may have in it.
|
||||
type window struct {
|
||||
name string
|
||||
length time.Duration
|
||||
limit int64
|
||||
byteLimit int64
|
||||
}
|
||||
|
||||
// add counts n requests, or n bytes, at now in a window of length, and
|
||||
// returns the client's count in the window that ends at now: what is in
|
||||
// the bucket under way, and what is in the bucket before it weighted by
|
||||
// how much of that bucket the window still covers. With n zero it counts
|
||||
// nothing, and returns the count.
|
||||
//
|
||||
// Concurrent requests can be counted out of order, so now can be a moment
|
||||
// before the bucket under way began; such a request is counted in that
|
||||
// bucket. A request dated more than a second before it means the clock
|
||||
// was set back, and the buckets start afresh: otherwise the bucket before
|
||||
// would keep its full weight until the clock caught up.
|
||||
func (b *Buckets) add(now time.Time, length time.Duration) float64 {
|
||||
func (b *Buckets) add(now time.Time, length time.Duration, n int64) float64 {
|
||||
if now.Before(b.Start.Add(-time.Second)) {
|
||||
*b = Buckets{}
|
||||
}
|
||||
@@ -393,7 +474,7 @@ func (b *Buckets) add(now time.Time, length time.Duration) float64 {
|
||||
b.Current = 0
|
||||
}
|
||||
|
||||
b.Current++
|
||||
b.Current += n
|
||||
|
||||
elapsed := max(now.Sub(b.Start), 0)
|
||||
covered := 1 - float64(elapsed)/float64(length)
|
||||
|
||||
@@ -71,13 +71,116 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
|
||||
// Over both limits; the minute's is named, with the four requests.
|
||||
_, hit, over := limiter.Count(client, start)
|
||||
|
||||
want := ratelimit.Hit{Window: minute, Limit: limit, Requests: limit + 1}
|
||||
want := ratelimit.Hit{
|
||||
Kind: ratelimit.KindRequests, Window: minute, Limit: limit, Count: limit + 1,
|
||||
}
|
||||
if !over || hit != want {
|
||||
t.Errorf("request over the limit gives %+v and %t, want %+v and true",
|
||||
hit, over, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const byteLimit = 1000
|
||||
|
||||
for _, tc := range []struct {
|
||||
window string
|
||||
limits ratelimit.Limits
|
||||
}{
|
||||
{minute, ratelimit.Limits{BytesPerMinute: byteLimit}},
|
||||
{hour, ratelimit.Limits{BytesPerHour: byteLimit}},
|
||||
{"day", ratelimit.Limits{BytesPerDay: byteLimit}},
|
||||
} {
|
||||
t.Run(tc.window, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(tc.limits)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
// 600 bytes are within the limit, 600 more over it.
|
||||
_, _, over := limiter.CountBytes(client, midnight(), 600)
|
||||
if over {
|
||||
t.Fatal("600 bytes are over the limit of 1000")
|
||||
}
|
||||
|
||||
_, hit, over := limiter.CountBytes(client, midnight(), 600)
|
||||
|
||||
want := ratelimit.Hit{
|
||||
Kind: ratelimit.KindBytes, Window: tc.window, Limit: byteLimit, Count: 1200,
|
||||
}
|
||||
if !over || hit != want {
|
||||
t.Errorf("1200 bytes give %+v and %t, want %+v and true", hit, over, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000})
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
other := netip.MustParsePrefix("203.0.113.10/32")
|
||||
start := midnight()
|
||||
|
||||
// The third request breaks the rate limit. The bytes of a request
|
||||
// counted after it, within the byte limit, do not break it again.
|
||||
for range 2 {
|
||||
wantCount(t, limiter, client, start, "")
|
||||
}
|
||||
|
||||
wantCount(t, limiter, client, start, minute)
|
||||
wantBytesCount(t, limiter, client, start, 500, "")
|
||||
wantBytesCount(t, limiter, client, start, 600, ratelimit.KindBytes)
|
||||
|
||||
// Bytes over the byte limit do not have the next request break it, nor
|
||||
// the rate limit, which that request is within.
|
||||
wantBytesCount(t, limiter, other, start, 1200, ratelimit.KindBytes)
|
||||
wantCount(t, limiter, other, start, "")
|
||||
}
|
||||
|
||||
func TestCountGivesTheBytesInEachWindow(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{})
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
limiter.CountBytes(client, start, 300)
|
||||
|
||||
// A quarter into the next hour, the minute has only these 100 bytes.
|
||||
// The hour still covers three quarters of the bucket before, whose 300
|
||||
// bytes count 225, and these: 325. The day covers all 400.
|
||||
later := start.Add(time.Hour + time.Hour/4)
|
||||
limiter.CountBytes(client, later, 100)
|
||||
|
||||
// A request's counts give the bytes counted so far too.
|
||||
counts, _, _ := limiter.Count(client, later)
|
||||
|
||||
want := ratelimit.Counts{
|
||||
Minute: 1, Hour: 1, Day: 1, MinuteBytes: 100, HourBytes: 325, DayBytes: 400,
|
||||
}
|
||||
if counts != want {
|
||||
t.Errorf("counts %+v, want %+v", counts, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResetSetsTheBytesBackToZero(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000})
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
wantBytesCount(t, limiter, client, start, 1200, ratelimit.KindBytes)
|
||||
limiter.Reset(client)
|
||||
|
||||
// The client has its whole allowance of bytes again.
|
||||
wantBytesCount(t, limiter, client, start, 1000, "")
|
||||
}
|
||||
|
||||
func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -267,3 +370,18 @@ func wantCount(
|
||||
client, now.Format(time.RFC3339), hit.Window, want)
|
||||
}
|
||||
}
|
||||
|
||||
// wantBytesCount counts bytes from client at now, and checks the kind of
|
||||
// the limit they break, "" for none.
|
||||
func wantBytesCount(
|
||||
t *testing.T, limiter *ratelimit.Limiter, client netip.Prefix, now time.Time,
|
||||
bytes int64, want string,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
_, hit, _ := limiter.CountBytes(client, now, bytes)
|
||||
if hit.Kind != want {
|
||||
t.Errorf("%d bytes from %s at %s break a limit on %q, want %q",
|
||||
bytes, client, now.Format(time.RFC3339), hit.Kind, want)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -64,6 +64,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{})
|
||||
limiter.Count(client, start)
|
||||
limiter.CountBytes(client, start, 5)
|
||||
limiter.AddToHistory(client, start, ratelimit.Request{Forwarded: true})
|
||||
|
||||
loaded := func(now time.Time) ratelimit.Client {
|
||||
@@ -76,19 +77,25 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
|
||||
}
|
||||
|
||||
// Two minutes on, the window that ends then covers neither of the
|
||||
// minute's buckets, which are emptied; the hour's and the day's stay,
|
||||
// and so does the history.
|
||||
// minute's buckets, of requests and of bytes, which are emptied; the
|
||||
// hour's and the day's stay, and so does the history.
|
||||
got := loaded(start.Add(2 * time.Minute))
|
||||
if got.Minute != (ratelimit.Buckets{}) || got.Hour.Current != 1 ||
|
||||
got.Day.Current != 1 || got.History.Requests != 1 {
|
||||
t.Errorf("loaded two minutes on as %+v", got)
|
||||
}
|
||||
|
||||
if got.MinuteBytes != (ratelimit.Buckets{}) || got.HourBytes.Current != 5 ||
|
||||
got.DayBytes.Current != 5 {
|
||||
t.Errorf("loaded two minutes on with buckets of bytes %+v, %+v and %+v",
|
||||
got.MinuteBytes, got.HourBytes, got.DayBytes)
|
||||
}
|
||||
|
||||
// A moment before, the window still covers some of the earlier one.
|
||||
got = loaded(start.Add(2*time.Minute - time.Nanosecond))
|
||||
if got.Minute.Current != 1 {
|
||||
t.Errorf("loaded just under two minutes on with minute buckets %+v",
|
||||
got.Minute)
|
||||
if got.Minute.Current != 1 || got.MinuteBytes.Current != 5 {
|
||||
t.Errorf("loaded just under two minutes on with minute buckets %+v and %+v",
|
||||
got.Minute, got.MinuteBytes)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -45,7 +45,7 @@ const (
|
||||
)
|
||||
|
||||
// OffenceLimit is the offence a request line names for a request that
|
||||
// broke a rate limit.
|
||||
// broke a rate limit, or whose bytes broke a byte limit.
|
||||
const OffenceLimit = "limit"
|
||||
|
||||
// timeLayout is RFC 3339 with milliseconds.
|
||||
@@ -117,13 +117,16 @@ type Line struct {
|
||||
// ActionBanned, ActionCountryDenied, ActionRateLimited or
|
||||
// ActionRuleBlocked.
|
||||
WouldAction string `json:"would_action,omitempty"`
|
||||
// Counts are the client's requests as the rate limits counted them
|
||||
// with this one, for a request they counted.
|
||||
// Counts are, for a request the rate limits counted, the client's
|
||||
// requests as they counted them with this one, and its bytes as the
|
||||
// byte limits counted them, with this request's once it has ended if
|
||||
// they count them.
|
||||
Counts ratelimit.Counts `json:"counts,omitzero"`
|
||||
// RuleIDs are the ids of the rule file rules the request matched.
|
||||
RuleIDs []string `json:"rule_ids,omitempty"`
|
||||
// LimitHit is the window whose rate limit the request went over:
|
||||
// minute, hour or day.
|
||||
// LimitHit is the window whose limit the request went over, named as
|
||||
// Counts names its count: minute, hour or day for a rate limit, and
|
||||
// minute_bytes, hour_bytes or day_bytes for a byte limit.
|
||||
LimitHit string `json:"limit_hit,omitempty"`
|
||||
// Offence is the offence the request was held as, OffenceLimit.
|
||||
Offence string `json:"offence,omitempty"`
|
||||
|
||||
+10
-4
@@ -615,8 +615,8 @@ func (f *bansFile) check(data []byte) error {
|
||||
}
|
||||
|
||||
// check refuses a client without its address, which would count nobody's
|
||||
// requests, or with requests in a window but no start, which would drop
|
||||
// them and give the client a fresh allowance.
|
||||
// requests, or with requests or bytes in a window but no start, which
|
||||
// would drop them and give the client a fresh allowance.
|
||||
func (f *clientsFile) check([]byte) error {
|
||||
for i, client := range f.Clients {
|
||||
switch {
|
||||
@@ -628,6 +628,12 @@ func (f *clientsFile) check([]byte) error {
|
||||
return missing(i, "hour.start")
|
||||
case countsWithoutStart(client.Day):
|
||||
return missing(i, "day.start")
|
||||
case countsWithoutStart(client.MinuteBytes):
|
||||
return missing(i, "minute_bytes.start")
|
||||
case countsWithoutStart(client.HourBytes):
|
||||
return missing(i, "hour_bytes.start")
|
||||
case countsWithoutStart(client.DayBytes):
|
||||
return missing(i, "day_bytes.start")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -697,8 +703,8 @@ func (f *alertsFile) check([]byte) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// countsWithoutStart reports whether b holds requests but no start, which
|
||||
// places them in time.
|
||||
// countsWithoutStart reports whether b holds requests, or bytes, but no
|
||||
// start, which places them in time.
|
||||
func countsWithoutStart(b ratelimit.Buckets) bool {
|
||||
return b.Start.IsZero() && (b.Current != 0 || b.Previous != 0)
|
||||
}
|
||||
|
||||
@@ -392,6 +392,12 @@ func TestEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
|
||||
`"hour": {"current": 3}}]}`,
|
||||
`: entry 1 has no "hour.start"`,
|
||||
},
|
||||
{
|
||||
"a client with bytes in a window without its start", clientsJSON,
|
||||
`{"version": 1, "clients": [{"client": "203.0.113.9/32", ` +
|
||||
`"minute_bytes": {"previous": 5120}}]}`,
|
||||
`: entry 1 has no "minute_bytes.start"`,
|
||||
},
|
||||
{
|
||||
"an answer without a client", lookupsJSON,
|
||||
`{"version": 1, "lookups": [{"country": "DE", ` + answer + `}]}`,
|
||||
@@ -1317,6 +1323,7 @@ func fill(params state.Params) {
|
||||
params.Limiter.Count(netip.MustParsePrefix(c), now)
|
||||
}
|
||||
|
||||
params.Limiter.CountBytes(client, now, 8)
|
||||
params.Limiter.AddToHistory(client, now, ratelimit.Request{
|
||||
Forwarded: true, Status: 200, RequestBytes: 3, ResponseBytes: 5,
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user