Byte limits per client over a minute, an hour and a day (closes #20)
check / check (push) Waiting to run
check / check (push) Waiting to run
SWWAF_BYTES_LIMIT_PER_MINUTE, _PER_HOUR and _PER_DAY (10G, 20G, 50G) and SWWAF_BYTES_COUNT (both). A request's bytes are counted once its answer has ended, for a request passed to the app that the rate limits count; what a WebSocket carries each way, once it closes. Bytes over a limit ban the client as a broken rate limit does, and cut nothing short. clients.json keeps the byte buckets, the log line's counts carry the byte totals, ban notes say what the limit is on, and the limit hits metric is labelled by kind. Judgement call: limit_hit names a byte window minute_bytes, hour_bytes or day_bytes, as counts names the byte totals. Judgement call: in observe mode, the bytes of a request enforce mode would have refused are not counted. Model: opus-5-5
This commit was merged in pull request #102.
This commit is contained in:
@@ -22,29 +22,30 @@ fields, which come a little later, and the metrics endpoint and the header size
|
||||
and the idle time as settings, which come last in it. So are the four parts of
|
||||
the stage after it: the rule files, with the bans for a clear sign of attack,
|
||||
the other admin endpoints, alerts to all three destinations, a JSON webhook,
|
||||
Slack and ntfy, and remote log sending. So is the first part of the stage after
|
||||
Slack and ntfy, and remote log sending. So are two parts of the stage after
|
||||
that: the AS number and country of every client, looked up through GeoJS or in
|
||||
the IPinfo Lite database file. `smallwebwaf` passes each request to the app and
|
||||
the app's answer back, unchanged, within its timeouts and size limits, works out
|
||||
each client's address, looks up its AS number and country unless you switch that
|
||||
off, bans a client that sends too many requests, not counting those for the
|
||||
paths you choose, refuses a client that comes from a country you refuse or from
|
||||
a network you refuse, lets the networks you choose through, checks each request
|
||||
against the rule files and bans a client whose request is a clear sign of
|
||||
attack, keeps its bans, each client's counters and history, and GeoJS's answers
|
||||
in JSON files across restarts, takes in your edits of those files, such as a ban
|
||||
you make, keep or lift, and of the rule files while it runs, writes a JSON log
|
||||
line for every request, sends its log lines to a syslog server too if you name
|
||||
one, sends an alert to a webhook, to Slack and to ntfy, each if you name one,
|
||||
for each ban it makes or makes permanent, for GeoJS failing, for a rule file or
|
||||
state file with an error and for a replacement of the lookup database it cannot
|
||||
read, serves Prometheus metrics to a scraper that holds the metrics token, lets
|
||||
an admin who holds the admin token list, add and lift bans and ask what it knows
|
||||
of a client, and in `observe` mode passes on the requests it would refuse,
|
||||
logging what it would have done with them. It comes as the image the app's own
|
||||
image is built on. The rest of the design comes after that, in the order of the
|
||||
build order in [`SPEC.md`](SPEC.md). The survey of existing tools that led to
|
||||
the design is in [`EVALUATION.md`](EVALUATION.md).
|
||||
the IPinfo Lite database file, and the byte limits. `smallwebwaf` passes each
|
||||
request to the app and the app's answer back, unchanged, within its timeouts and
|
||||
size limits, works out each client's address, looks up its AS number and country
|
||||
unless you switch that off, bans a client that sends too many requests or too
|
||||
many bytes, not counting those for the paths you choose, refuses a client that
|
||||
comes from a country you refuse or from a network you refuse, lets the networks
|
||||
you choose through, checks each request against the rule files and bans a client
|
||||
whose request is a clear sign of attack, keeps its bans, each client's counters
|
||||
and history, and GeoJS's answers in JSON files across restarts, takes in your
|
||||
edits of those files, such as a ban you make, keep or lift, and of the rule
|
||||
files while it runs, writes a JSON log line for every request, sends its log
|
||||
lines to a syslog server too if you name one, sends an alert to a webhook, to
|
||||
Slack and to ntfy, each if you name one, for each ban it makes or makes
|
||||
permanent, for GeoJS failing, for a rule file or state file with an error and
|
||||
for a replacement of the lookup database it cannot read, serves Prometheus
|
||||
metrics to a scraper that holds the metrics token, lets an admin who holds the
|
||||
admin token list, add and lift bans and ask what it knows of a client, and in
|
||||
`observe` mode passes on the requests it would refuse, logging what it would
|
||||
have done with them. It comes as the image the app's own image is built on. The
|
||||
rest of the design comes after that, in the order of the build order in
|
||||
[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
|
||||
[`EVALUATION.md`](EVALUATION.md).
|
||||
|
||||
## Getting started
|
||||
|
||||
@@ -107,26 +108,40 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
||||
window still covers. At most 20,000 clients are kept, the least recently seen
|
||||
dropped first, with their history, and a restart gives no client a fresh
|
||||
allowance (see "State files" below).
|
||||
- Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md)
|
||||
describes: the first ban lasts an hour, and a limit broken again within a day
|
||||
of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and
|
||||
81 hours; a ban that would last longer than seven days is permanent instead. A
|
||||
ban covers the client's netblock: its IPv4 address, or the netblock around it
|
||||
that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 /64. While it lasts, every
|
||||
request from the netblock is refused with `SWWAF_BAN_RESPONSE` after the
|
||||
static lists and before the country lists, so the client is not looked up, and
|
||||
is not counted for the rate limits. A ban sets the client's counters back to
|
||||
zero. Each ban carries notes for deciding whether to lift it: the limit, its
|
||||
window and the requests counted in it, the request that broke it, the client's
|
||||
AS number, AS name and country once they are looked up, the netblock's
|
||||
requests since it was first seen, how many of them the ban has refused, and
|
||||
how many bans the netblock had before, for a broken limit, for a clear sign of
|
||||
attack and by an admin. At most `SWWAF_MAX_BANS` bans `smallwebwaf` made are
|
||||
kept, past, active and permanent; past that, the earliest such ban of the
|
||||
netblock that has gone longest without a request is dropped first. The bans
|
||||
whose cause is `admin`, those you make or keep, are kept besides, and never
|
||||
dropped. `bans.json` shows the bans and their notes, a restart lifts none, and
|
||||
you make, keep or lift a ban by editing it (see "State files" below).
|
||||
- Counts each client's bytes over a minute, an hour and a day, in the same way:
|
||||
once a request passed to the app has ended, the body bytes of its answer, of
|
||||
the request, or of both, as `SWWAF_BYTES_COUNT` says. For a WebSocket, or any
|
||||
other upgraded connection, what it carried from the app counts with the
|
||||
answer, and what it carried from the client with the request, once it closes.
|
||||
Bytes that take the client over one of the byte limits below break that limit,
|
||||
and ban the client as a broken rate limit does, so that its next request is
|
||||
refused. The byte limits never cut an answer or an upgraded connection short:
|
||||
the one whose bytes break a limit has already been passed on, or has closed.
|
||||
They leave out what the rate limits leave out: a client in `SWWAF_ALLOW_NETS`
|
||||
or `SWWAF_RATE_LIMIT_EXEMPT_NETS`, and a request for a path
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts.
|
||||
- Bans a client that breaks a rate limit or a byte limit, as "Bans" in
|
||||
[`SPEC.md`](SPEC.md) describes: the first ban lasts an hour, and a limit
|
||||
broken again within a day of a ban ending bans for three times as long as that
|
||||
ban, so 1, 3, 9, 27 and 81 hours; a ban that would last longer than seven days
|
||||
is permanent instead. A ban covers the client's netblock: its IPv4 address, or
|
||||
the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 /64.
|
||||
While it lasts, every request from the netblock is refused with
|
||||
`SWWAF_BAN_RESPONSE` after the static lists and before the country lists, so
|
||||
the client is not looked up, and is not counted for the rate limits. A ban
|
||||
sets the client's counters back to zero. Each ban carries notes for deciding
|
||||
whether to lift it: the limit, whether it is on requests or bytes, its window
|
||||
and the requests or bytes counted in it, the request that broke it, the
|
||||
client's AS number, AS name and country once they are looked up, the
|
||||
netblock's requests since it was first seen, how many of them the ban has
|
||||
refused, and how many bans the netblock had before, for a broken limit, for a
|
||||
clear sign of attack and by an admin. At most `SWWAF_MAX_BANS` bans
|
||||
`smallwebwaf` made are kept, past, active and permanent; past that, the
|
||||
earliest such ban of the netblock that has gone longest without a request is
|
||||
dropped first. The bans whose cause is `admin`, those you make or keep, are
|
||||
kept besides, and never dropped. `bans.json` shows the bans and their notes, a
|
||||
restart lifts none, and you make, keep or lift a ban by editing it (see "State
|
||||
files" below).
|
||||
- Checks each request against the rules of the rule files (see "Rule files"
|
||||
below) after the rate limits, and before its body is read. A `log` rule that
|
||||
matches is noted in the log line; a `block` rule refuses the request with
|
||||
@@ -138,10 +153,10 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
||||
default, and any request from the netblock while it lasts makes it permanent.
|
||||
Once it has run out, the netblock is served like any other, but its next clear
|
||||
sign of attack bans it permanently at once. Such a ban covers the same
|
||||
netblock as a ban for a broken rate limit, does not set the client's counters
|
||||
back to zero, and does not make the netblock's next ban for a broken limit
|
||||
longer. Its notes give the id and the target of the rule that matched in place
|
||||
of the limit.
|
||||
netblock as a ban for a broken limit, does not set the client's counters back
|
||||
to zero, and does not make the netblock's next ban for a broken limit longer.
|
||||
Its notes give the id and the target of the rule that matched in place of the
|
||||
limit.
|
||||
- Looks up the AS number and country of every client through GeoJS, or in the
|
||||
IPinfo Lite database file while `SWWAF_LOOKUP_SOURCE` is `file`, after the
|
||||
static lists and bans, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Country and
|
||||
@@ -160,29 +175,33 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
||||
`SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it.
|
||||
- Checks the client's own address against the static lists, the three netblock
|
||||
settings below, before anything else, its lookup included. A client in
|
||||
`SWWAF_ALLOW_NETS` skips bans, the country lists, the rate limits and the rule
|
||||
files, and is not looked up; the timeouts and size limits still apply. A
|
||||
client in `SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE` before its
|
||||
body is read, and the request is not counted for the rate limits; an address
|
||||
in `SWWAF_ALLOW_NETS` too is let through. A client in
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
|
||||
limits; the country lists, the rule files and bans still apply to it.
|
||||
`SWWAF_ALLOW_NETS` skips bans, the country lists, the rate limits, the byte
|
||||
limits and the rule files, and is not looked up; the timeouts and size limits
|
||||
still apply. A client in `SWWAF_DENY_NETS` is refused with
|
||||
`SWWAF_BAN_RESPONSE` before its body is read, and the request is not counted
|
||||
for the rate limits; an address in `SWWAF_ALLOW_NETS` too is let through. A
|
||||
client in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the
|
||||
rate limits, and has no bytes counted by the byte limits; the country lists,
|
||||
the rule files and bans still apply to it.
|
||||
- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
|
||||
that `SWWAF_DENY_NETS`, a ban, the country lists, a rate limit or a rule would
|
||||
refuse: it passes them to the app, and their log lines name what `enforce`
|
||||
mode would have done (see `would_action` in "Request log" below). The checks
|
||||
run, and requests are counted, as in `enforce` mode, with three differences:
|
||||
neither a broken rate limit nor a `ban` rule makes a ban; a broken rate limit
|
||||
does not set the client's counters back to zero, so each request over the
|
||||
limit is logged as one that would be refused; and a request under a ban does
|
||||
not make it permanent. A ban it would have made, or made permanent, raises the
|
||||
alert `enforce` mode would have raised, marked as what would have happened
|
||||
(see "Alerts" below). The bans in `bans.json` are kept, and refuse requests
|
||||
again when `smallwebwaf` next runs in `enforce` mode, as long as they last.
|
||||
The timeouts and size limits still apply, since they protect `smallwebwaf` and
|
||||
the app themselves, and a request for one of `smallwebwaf`'s own endpoints
|
||||
without its token is still answered `401`. It is for trying a configuration
|
||||
before enforcing it.
|
||||
run, and requests and bytes are counted, as in `enforce` mode, with three
|
||||
differences: neither a broken rate limit or byte limit nor a `ban` rule makes
|
||||
a ban; a broken limit does not set the client's counters back to zero, so each
|
||||
request over a rate limit is logged as one that would be refused, and each
|
||||
whose bytes keep the client over a byte limit as breaking it; and a request
|
||||
under a ban does not make it permanent. As in `enforce` mode, the bytes
|
||||
counted are only those of the requests `enforce` mode would have passed to the
|
||||
app. A ban it would have made, or made permanent, raises the alert `enforce`
|
||||
mode would have raised, marked as what would have happened (see "Alerts"
|
||||
below). The bans in `bans.json` are kept, and refuse requests again when
|
||||
`smallwebwaf` next runs in `enforce` mode, as long as they last. The timeouts
|
||||
and size limits still apply, since they protect `smallwebwaf` and the app
|
||||
themselves, and a request for one of `smallwebwaf`'s own endpoints without its
|
||||
token is still answered `401`. It is for trying a configuration before
|
||||
enforcing it.
|
||||
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
|
||||
check and without asking the app, for the image's health check.
|
||||
- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
|
||||
@@ -254,10 +273,11 @@ effective settings are logged at start.
|
||||
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
|
||||
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
|
||||
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
|
||||
country lists, the rate limits and the rule files, such as your monitoring or
|
||||
your own networks.
|
||||
country lists, the rate limits, the byte limits and the rule files, such as
|
||||
your monitoring or your own networks.
|
||||
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
|
||||
rate limits do not apply to, such as a machine that talks to the app all day.
|
||||
rate limits and the byte limits do not apply to, such as a machine that talks
|
||||
to the app all day.
|
||||
- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
|
||||
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR`
|
||||
(default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`): the most
|
||||
@@ -265,19 +285,29 @@ effective settings are logged at start.
|
||||
several times what one busy person produces, since a browser loading a heavy
|
||||
page makes a few hundred requests and several people often share one address.
|
||||
- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes whose requests
|
||||
the rate limits neither count nor refuse, such as `/assets/` for static
|
||||
assets; each starts with `/`. A request whose path, percent-decoded, contains
|
||||
`..` anywhere or a backslash, or whose path as sent holds an encoded slash
|
||||
(`%2F` or `%2f`), is never exempt, since the app may act on it as a path
|
||||
outside every prefix: `/assets/..%2Flogin` as `/login`. Any other request is
|
||||
exempt when its path as sent, the path the app receives, before any query
|
||||
string and not percent-decoded, starts with a prefix, character for character.
|
||||
`/assets/` matches `/assets/app.js` and `/assets/`, but not `/assets`,
|
||||
`/Assets/app.js`, `/%61ssets/app.js`, `/static/assets/app.js`,
|
||||
`/static/../assets/app.js` or `/assets%2Fapp.js`. A character the client sends
|
||||
percent-encoded, such as a space, is written percent-encoded in a prefix, as
|
||||
in `/my%20files/`, and there are no wildcards: `*` is a character like any
|
||||
other.
|
||||
the rate limits neither count nor refuse, and whose bytes the byte limits do
|
||||
not count, such as `/assets/` for static assets; each starts with `/`. A
|
||||
request whose path, percent-decoded, contains `..` anywhere or a backslash, or
|
||||
whose path as sent holds an encoded slash (`%2F` or `%2f`), is never exempt,
|
||||
since the app may act on it as a path outside every prefix:
|
||||
`/assets/..%2Flogin` as `/login`. Any other request is exempt when its path as
|
||||
sent, the path the app receives, before any query string and not
|
||||
percent-decoded, starts with a prefix, character for character. `/assets/`
|
||||
matches `/assets/app.js` and `/assets/`, but not `/assets`, `/Assets/app.js`,
|
||||
`/%61ssets/app.js`, `/static/assets/app.js`, `/static/../assets/app.js` or
|
||||
`/assets%2Fapp.js`. A character the client sends percent-encoded, such as a
|
||||
space, is written percent-encoded in a prefix, as in `/my%20files/`, and there
|
||||
are no wildcards: `*` is a character like any other.
|
||||
- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`), `SWWAF_BYTES_LIMIT_PER_HOUR`
|
||||
(default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `50G`): the most
|
||||
bytes a client may have counted in a minute, an hour and a day. A request's
|
||||
bytes are counted once its answer has ended, so each default is above the
|
||||
largest request body and the largest response together,
|
||||
`SWWAF_REQUEST_MAX_BYTES` and `SWWAF_RESPONSE_MAX_BYTES`: at the defaults no
|
||||
download breaks a limit on its own.
|
||||
- `SWWAF_BYTES_COUNT` (default `both`): which body bytes the byte limits count:
|
||||
`response` for those of the answers, `request` for those of the requests, or
|
||||
`both`.
|
||||
- `SWWAF_LOOKUP_SOURCE` (default `geojs`): where each client's AS number and
|
||||
country are looked up: `geojs`, the GeoJS web service, which is then told the
|
||||
address of every new visitor, `file`, the IPinfo Lite database file
|
||||
@@ -311,12 +341,12 @@ effective settings are logged at start.
|
||||
the client unanswered: traefik answers `502`, as it does whenever its backend
|
||||
drops a connection. A `block` rule always answers `403`.
|
||||
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
|
||||
limit.
|
||||
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit broken again
|
||||
within this time after a ban ended, other than one for a clear sign of attack,
|
||||
bans for three times as long as that ban.
|
||||
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken rate limit that
|
||||
would be longer is permanent instead.
|
||||
limit or byte limit.
|
||||
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit or byte limit
|
||||
broken again within this time after a ban ended, other than one for a clear
|
||||
sign of attack, bans for three times as long as that ban.
|
||||
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken rate limit or byte
|
||||
limit that would be longer is permanent instead.
|
||||
- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
|
||||
attack.
|
||||
- `SWWAF_MAX_BANS` (default `5000`): the most bans `smallwebwaf` made that are
|
||||
@@ -410,15 +440,16 @@ effective settings are logged at start.
|
||||
|
||||
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
|
||||
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
|
||||
bytes). Rate limits are whole numbers of requests. Netblocks are in CIDR form,
|
||||
and a bare address stands for itself alone. Countries are the two-letter codes
|
||||
ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
|
||||
the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
|
||||
`su`, stops the start, and so does a code on both country lists. `off` switches
|
||||
a timeout, a size limit, a rate limit, `SWWAF_ALERT_COOLDOWN` or
|
||||
`SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
|
||||
`SWWAF_LOOKUP_TIMEOUT`, the ban settings, the state settings,
|
||||
`SWWAF_METRICS_TOP_N` and `SWWAF_LOG_REMOTE_BUFFER` cannot be off.
|
||||
bytes). Rate limits are whole numbers of requests, and byte limits are sizes.
|
||||
Netblocks are in CIDR form, and a bare address stands for itself alone.
|
||||
Countries are the two-letter codes ISO 3166-1 assigns today, and `xk` for
|
||||
Kosovo, in either case (`de` and `DE` are the same); any other code, such as
|
||||
`nk` (North Korea is `kp`) or the withdrawn `su`, stops the start, and so does a
|
||||
code on both country lists. `off` switches a timeout, a size limit, a rate
|
||||
limit, a byte limit, `SWWAF_ALERT_COOLDOWN` or `SWWAF_ALERT_MAX_PER_HOUR` off;
|
||||
`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, `SWWAF_LOOKUP_TIMEOUT`, the ban
|
||||
settings, the state settings, `SWWAF_METRICS_TOP_N` and
|
||||
`SWWAF_LOG_REMOTE_BUFFER` cannot be off.
|
||||
|
||||
Several limits are fixed rather than settings. At most 20,000 clients are kept,
|
||||
with their counters and history, and an IPv6 client is counted by its /64. At
|
||||
@@ -462,7 +493,7 @@ and for the container, `-v /srv/app/tokens:/etc/smallwebwaf/tokens:ro` and
|
||||
refused ones included:
|
||||
|
||||
```
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104}
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104}
|
||||
```
|
||||
|
||||
A field that does not apply to a request is left out of its line, apart from
|
||||
@@ -527,13 +558,20 @@ which every line has.
|
||||
health check, one from a client in `SWWAF_ALLOW_NETS` or
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_NETS`, one for a path that
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a ban
|
||||
or the country lists refuse, or would refuse in `observe` mode. The byte
|
||||
totals come with the byte limits.
|
||||
or the country lists refuse, or would refuse in `observe` mode. Its
|
||||
`minute_bytes`, `hour_bytes` and `day_bytes` give the client's bytes in each
|
||||
window as the byte limits count them, in the same way: for a request whose
|
||||
bytes they count, with its own, once it has ended; for any other, those
|
||||
counted before it.
|
||||
- `rule_ids` is there for a request that matched rules of the rule files, and
|
||||
lists their ids in the order they matched, up to the one that refused it.
|
||||
- `limit_hit` is there for a request that broke a rate limit, and names the
|
||||
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
|
||||
went over several. `offence` is then `limit`.
|
||||
- `limit_hit` is there for a request that broke a rate limit, or whose bytes
|
||||
broke a byte limit, and names the window whose limit it went over as `counts`
|
||||
names it: `minute`, `hour` or `day` for a rate limit, and `minute_bytes`,
|
||||
`hour_bytes` or `day_bytes` for a byte limit, the shortest if it went over
|
||||
several. `offence` is then `limit`. A request whose bytes broke a byte limit
|
||||
is not refused: its `action` is what it would have been otherwise, such as
|
||||
`forward`.
|
||||
- `ban_expires` is there for a request that made a ban or was refused under one,
|
||||
or in `observe` mode would have been refused under one, and gives when the ban
|
||||
ends, in the same form as `time`, or `permanent`.
|
||||
@@ -596,8 +634,8 @@ gives, as "Alert webhook schema" in [`SPEC.md`](SPEC.md) describes, and to
|
||||
it, as below. An alert is for one of these events, and is sent when
|
||||
`SWWAF_ALERT_EVENTS` names its event:
|
||||
|
||||
- `ban`: a ban `smallwebwaf` makes, for a broken rate limit or a clear sign of
|
||||
attack.
|
||||
- `ban`: a ban `smallwebwaf` makes, for a broken rate limit or byte limit or a
|
||||
clear sign of attack.
|
||||
- `permanent_ban`: a permanent ban it makes, or a ban for a clear sign of attack
|
||||
that a request made permanent.
|
||||
- `source_failure`: GeoJS failing or refusing `smallwebwaf`.
|
||||
@@ -633,6 +671,7 @@ is sent on one line:
|
||||
"asn": "",
|
||||
"as_name": "",
|
||||
"country": "",
|
||||
"kind": "requests",
|
||||
"limit": 1000,
|
||||
"window": "minute",
|
||||
"count": 1001,
|
||||
@@ -752,20 +791,23 @@ entries by client address, but for the alerts waiting, with times in UTC.
|
||||
|
||||
- `bans.json`: every ban with its notes, indented to be read. A permanent ban's
|
||||
`expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or
|
||||
`attack` for a clear sign of attack, for a ban `smallwebwaf` made, and `admin`
|
||||
for one you made or keep. Its `reason` is a short text: for a ban
|
||||
`smallwebwaf` made, the limit broken, such as
|
||||
`requests per minute over the limit of 1000`, or the rule that matched, such
|
||||
byte limit or `attack` for a clear sign of attack, for a ban `smallwebwaf`
|
||||
made, and `admin` for one you made or keep. Its `reason` is a short text: for
|
||||
a ban `smallwebwaf` made, the limit broken, such as
|
||||
`requests per minute over the limit of 1000` or
|
||||
`bytes per hour over the limit of 21474836480`, or the rule that matched, such
|
||||
as `matched the rule env-file`; for yours, what you wrote. Its `lifted` is
|
||||
when you lifted it, and is left out until you do.
|
||||
- `clients.json`: each client's two buckets in the minute, the hour and the day,
|
||||
and its history: when it was first and last seen, its AS number, AS name and
|
||||
country as last looked up and when the lookup gave them, its requests, how
|
||||
many were forwarded and how many refused (one `smallwebwaf` answered at its
|
||||
own endpoints is neither, unless it was refused with `401` for a missing or
|
||||
wrong token), the body bytes in each direction, its responses by status class
|
||||
and its offences by kind. Each client is on a line of its own, so `grep` shows
|
||||
everything about one.
|
||||
when you lifted it, and is left out until you do. The `kind` in the notes of a
|
||||
ban for a broken limit is `requests` or `bytes`, what the limit is on.
|
||||
- `clients.json`: each client's two buckets of requests in the minute, the hour
|
||||
and the day, its two buckets of bytes in each, `minute_bytes`, `hour_bytes`
|
||||
and `day_bytes`, and its history: when it was first and last seen, its AS
|
||||
number, AS name and country as last looked up and when the lookup gave them,
|
||||
its requests, how many were forwarded and how many refused (one `smallwebwaf`
|
||||
answered at its own endpoints is neither, unless it was refused with `401` for
|
||||
a missing or wrong token), the body bytes in each direction, its responses by
|
||||
status class and its offences by kind. Each client is on a line of its own, so
|
||||
`grep` shows everything about one.
|
||||
- `lookups.json`: GeoJS's answers, one to a line, each with the client's AS
|
||||
number, AS name and country, when GeoJS gave it and when it was last used.
|
||||
- `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
|
||||
@@ -801,9 +843,9 @@ message naming the file, and the line and column where Go's JSON decoder gives
|
||||
them; so does a state directory `smallwebwaf` cannot write. So does an entry
|
||||
without a field it needs, named with the entry's place in the file: a ban's
|
||||
`netblock`, `start` or `expires`, which is `null` for a permanent ban; a
|
||||
client's `client`, or the `start` of a window in which it has requests; an
|
||||
answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
|
||||
`answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
|
||||
client's `client`, or the `start` of a window in which it has requests or bytes;
|
||||
an answer's `client`, `country`, which is `""` for a client GeoJS cannot place,
|
||||
or `answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
|
||||
`time`. So does a ban whose `cause` is not `limit`, `attack` or `admin`, and
|
||||
alerts waiting for a destination that is not `webhook`, `slack` or `ntfy`. An
|
||||
answer's `asn` or `as_name` left out reads as empty.
|
||||
@@ -955,7 +997,8 @@ scraped, and keeps this one as `exported_instance` unless the scrape sets
|
||||
`smallwebwaf_upstream_duration_seconds`: how long those passed to the app took
|
||||
from then on, as histograms; `smallwebwaf_requests_in_flight`: the requests
|
||||
under way.
|
||||
- `smallwebwaf_rate_limit_hits_total` by `window`,
|
||||
- `smallwebwaf_rate_limit_hits_total` by `window`, `minute`, `hour` or `day`,
|
||||
and `kind`, `requests` for a rate limit or `bytes` for a byte limit,
|
||||
`smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose
|
||||
limit was passed, `smallwebwaf_offences_total` by `kind`, and
|
||||
`smallwebwaf_bans_made_total` by `cause`, `limit`, `attack` or `admin`, the
|
||||
@@ -1375,7 +1418,8 @@ addresses are never sent to GeoJS.
|
||||
body over the size limit; in `observe` mode, only for the size limit, with
|
||||
what it would have refused for noted in the log line. A request under
|
||||
`/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
|
||||
instead of reaching the app.
|
||||
instead of reaching the app. Once the answer to a request passed to the app
|
||||
has ended, `countBytes` counts its bytes for the byte limits.
|
||||
- `internal/metrics`: the metrics, counted as the other parts tell it what
|
||||
happened, and served in the Prometheus text format.
|
||||
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how
|
||||
@@ -1388,8 +1432,9 @@ addresses are never sent to GeoJS.
|
||||
client's history and to the notes of its bans; or in the lookup database,
|
||||
which it reads again when the file is replaced. `internal/lookup/lookuptest`
|
||||
writes lookup databases for the tests.
|
||||
- `internal/ratelimit`: the table of clients: counts each client's requests,
|
||||
tells when one takes it over a rate limit, and keeps each client's history.
|
||||
- `internal/ratelimit`: the table of clients: counts each client's requests and
|
||||
bytes, tells when they take it over a rate limit or a byte limit, and keeps
|
||||
each client's history.
|
||||
- `internal/state`: reads the state files at start, takes in an admin's edit of
|
||||
one while running, and writes them when they are due and at the stop.
|
||||
- `internal/requestlog`: the lines on stdout: the request log line and the
|
||||
|
||||
Reference in New Issue
Block a user