Compare commits
2
Commits
4bc8d9edb4
..
next
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a6634454cd | ||
|
|
ca787985f8 |
@@ -30,7 +30,7 @@ unusual traffic that refuse nothing. So are the first three parts of the stage
|
||||
after that: the blocklists you name by URL, which it fetches and keeps, with a
|
||||
file of AS numbers' percentages fetched the same way, the DNS blocklists (DNSBL
|
||||
zones), which it asks about each client in the background, and AbuseIPDB, which
|
||||
it asks in the background about each client that has broken a limit.
|
||||
it asks in the background about each client that has committed an offence.
|
||||
`smallwebwaf` passes each request to the app and the app's answer back,
|
||||
unchanged, within its timeouts and size limits, works out each client's address,
|
||||
looks up its AS number and country unless you switch that off, bans a client
|
||||
@@ -116,11 +116,12 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_PATHS`, as that setting below describes, is neither
|
||||
counted nor refused by the rate limits; the static lists, bans, the country
|
||||
lists and the rule files still apply to it. A client is one IPv4 address, or
|
||||
one IPv6 /64, since one abuser usually holds a whole /64. Each window is
|
||||
one IPv6 group, the netblock of `SWWAF_IPV6_GROUP_PREFIX` its address is in, a
|
||||
/64 by default, since one abuser usually holds a whole /64. Each window is
|
||||
counted in two fixed buckets, the earlier one weighted by how much of it the
|
||||
window still covers. At most 20,000 clients are kept, the least recently seen
|
||||
dropped first, with their history, and a restart gives no client a fresh
|
||||
allowance (see "State files" below).
|
||||
window still covers. At most `SWWAF_MAX_TRACKED_CLIENTS` clients are kept,
|
||||
20,000 by default, the least recently seen dropped first, with their history,
|
||||
and a restart gives no client a fresh allowance (see "State files" below).
|
||||
- Counts each client's bytes over a minute, an hour and a day, in the same way:
|
||||
once a request passed to the app has ended, the body bytes of its answer, of
|
||||
the request, or of both, as `SWWAF_BYTES_COUNT` says. For a WebSocket, or any
|
||||
@@ -153,8 +154,8 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
||||
broken again within a day of a ban ending bans for three times as long as that
|
||||
ban, so 1, 3, 9, 27 and 81 hours; a ban that would last longer than seven days
|
||||
is permanent instead. A ban covers the client's netblock: its IPv4 address, or
|
||||
the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 /64.
|
||||
While it lasts, every request from the netblock is refused with
|
||||
the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6
|
||||
group. While it lasts, every request from the netblock is refused with
|
||||
`SWWAF_BAN_RESPONSE` after the static lists and before the country lists, so
|
||||
the client is not looked up, and is not counted for the rate limits. A ban
|
||||
sets the client's counters back to zero. Each ban carries notes for deciding
|
||||
@@ -225,15 +226,16 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
||||
make no ban. With `log`, nothing more is done. Whatever the action, the
|
||||
request's log line names the zones, and each raises an alert. A client a
|
||||
blocklist refuses is not checked.
|
||||
- Checks the client's own address with AbuseIPDB while `SWWAF_ABUSEIPDB_KEY` is
|
||||
set, after the DNSBL zones and before the rate limits (see "AbuseIPDB" below),
|
||||
by the scores it keeps. Only a client whose history counts an offence is
|
||||
checked, so far one that has broken a rate limit or a byte limit, and only in
|
||||
the background, so that no request waits for AbuseIPDB. A score at or over
|
||||
`SWWAF_ABUSEIPDB_MIN_SCORE` is a hit, and `SWWAF_REPUTATION_ACTION` does with
|
||||
its client what it does with one a DNSBL zone's verdict lists. The request's
|
||||
log line names AbuseIPDB, and it raises an alert. A client a blocklist or a
|
||||
DNSBL zone refuses is not checked.
|
||||
- Checks the client with AbuseIPDB while `SWWAF_ABUSEIPDB_KEY` is set, after the
|
||||
DNSBL zones and before the rate limits (see "AbuseIPDB" below), by the scores
|
||||
it keeps. Only a client whose history counts an offence is checked, so far one
|
||||
that has broken a rate limit or a byte limit, matched a ban rule, or had a
|
||||
request refused by a block rule, and only in the background, so that no
|
||||
request waits for AbuseIPDB. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is
|
||||
a hit, and `SWWAF_REPUTATION_ACTION` does with its client what it does with
|
||||
one a DNSBL zone's verdict lists. The request's log line names AbuseIPDB, and
|
||||
it raises an alert. A client a blocklist or a DNSBL zone refuses is not
|
||||
checked.
|
||||
- Checks the client's own address against the static lists, the three netblock
|
||||
settings below, before anything else, its lookup included. A client in
|
||||
`SWWAF_ALLOW_NETS` skips bans, the country lists, the blocklists, the DNSBL
|
||||
@@ -314,7 +316,8 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
||||
Each setting is an environment variable, or a file one names (see "Settings
|
||||
given as files" below), and each has a default, so none has to be set. A setting
|
||||
that is set but invalid stops the start with a message naming it, and the
|
||||
effective settings are logged at start.
|
||||
effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
|
||||
`error`, which hold that line back.
|
||||
|
||||
- `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens.
|
||||
- `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app, as `http` or
|
||||
@@ -332,6 +335,18 @@ effective settings are logged at start.
|
||||
- `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`,
|
||||
the private address ranges): the netblocks whose `X-Forwarded-For` is
|
||||
believed. A list given replaces the default; set but empty, it trusts nothing.
|
||||
- `SWWAF_IPV6_GROUP_PREFIX` (default `64`): the length of an IPv6 client's
|
||||
group, the netblock that is one client, from 32 to 128, since a shorter one
|
||||
would make one client of the customers of several providers. The rate limits,
|
||||
the byte limits, bans, the table of clients, the lookups, AbuseIPDB's scores
|
||||
and the anomaly thresholds per client all take an IPv6 client as its group,
|
||||
and `client_group` gives it. After it changes, each IPv6 client starts afresh:
|
||||
what was kept of it under its earlier group, its counts, history, GeoJS answer
|
||||
and AbuseIPDB score, is not used for it, while each ban keeps refusing its
|
||||
netblock until it ends.
|
||||
- `SWWAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept in memory
|
||||
and in `clients.json`, with their counters and history, a whole number above
|
||||
zero. Past it, the least recently seen is dropped first.
|
||||
- `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to
|
||||
send its request line and headers, and then, from the end of the headers, its
|
||||
body.
|
||||
@@ -509,7 +524,7 @@ effective settings are logged at start.
|
||||
kept, past, active and permanent. The bans you make or keep are kept besides.
|
||||
- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around
|
||||
an IPv4 client that a ban covers, such as `24` to ban the surrounding /24. An
|
||||
IPv6 ban covers the client's /64.
|
||||
IPv6 ban covers the client's group, as `SWWAF_IPV6_GROUP_PREFIX` sets it.
|
||||
- `SWWAF_STATE_DIR` (default `/var/lib/smallwebwaf`): the directory of the state
|
||||
files, an absolute path. A directory `smallwebwaf` cannot write stops the
|
||||
start.
|
||||
@@ -524,6 +539,10 @@ effective settings are logged at start.
|
||||
(see "Request log" below). An entry naming `Host` or `Transfer-Encoding` stops
|
||||
the start, since Go's HTTP server takes both out of the request; the request's
|
||||
host is the field `host`.
|
||||
- `SWWAF_LOG_LEVEL` (default `info`): the least severe of `smallwebwaf`'s own
|
||||
messages that are written, on stdout and to the syslog server: `debug`,
|
||||
`info`, `warn` or `error`. No message is at `debug` yet, so it writes what
|
||||
`info` does. It holds back no line of the request log.
|
||||
- `SWWAF_ADMIN_TOKEN` (default unset): the token an admin sends for the ban
|
||||
endpoints and `/_smallwebwaf/clients/<ip>` (see "Admin endpoints" below), a
|
||||
long random value. While it is unset they are off; one shorter than 32
|
||||
@@ -636,7 +655,8 @@ Percentages are whole numbers from 0 to 100, and an entry of a list of them is
|
||||
an AS number or a country, `:` and a percentage; an AS number or a country
|
||||
listed twice in one of them stops the start. `off` switches a timeout, a size
|
||||
limit, a rate limit, a byte limit, an anomaly threshold, `SWWAF_ALERT_COOLDOWN`
|
||||
or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
|
||||
or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_IPV6_GROUP_PREFIX`,
|
||||
`SWWAF_MAX_TRACKED_CLIENTS`, `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
|
||||
`SWWAF_LOOKUP_TIMEOUT`, `SWWAF_UNKNOWN_LIMIT_PERCENT`,
|
||||
`SWWAF_BLOCKLIST_REFRESH`, `SWWAF_ABUSEIPDB_MIN_SCORE`,
|
||||
`SWWAF_ABUSEIPDB_DAILY_BUDGET`, `SWWAF_REPUTATION_CACHE_TTL`,
|
||||
@@ -644,12 +664,10 @@ or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
|
||||
`SWWAF_METRICS_TOP_N`, `SWWAF_LOG_REMOTE_BUFFER`, `SWWAF_ANOMALY_NET_V4_PREFIX`
|
||||
and `SWWAF_ANOMALY_NET_V6_PREFIX` cannot be off.
|
||||
|
||||
Several limits are fixed rather than settings. At most 20,000 clients are kept,
|
||||
with their counters and history, and an IPv6 client is counted by its /64. At
|
||||
most 100,000 answers from GeoJS are kept, for 7 days each, at most 20,000
|
||||
anomaly counters, at most 100,000 verdicts of the DNSBL zones, with at most
|
||||
1,000 queries to them under way at once, and at most 100,000 scores of
|
||||
AbuseIPDB.
|
||||
Several limits are fixed rather than settings. At most 100,000 answers from
|
||||
GeoJS are kept, for 7 days each, at most 20,000 anomaly counters, at most
|
||||
100,000 verdicts of the DNSBL zones, with at most 1,000 queries to them under
|
||||
way at once, and at most 100,000 scores of AbuseIPDB.
|
||||
|
||||
### Settings given as files
|
||||
|
||||
@@ -706,7 +724,7 @@ which every line has.
|
||||
- `peer_ip` is the TCP peer, normally traefik. `forwarded_for` is the
|
||||
`X-Forwarded-For` header as received, several lines of it joined with `, `.
|
||||
`client_group` is the client as the rate limits count it: its IPv4 address as
|
||||
a /32, or the /64 of its IPv6 address.
|
||||
a /32, or its IPv6 group, as `SWWAF_IPV6_GROUP_PREFIX` sets it.
|
||||
- `asn`, `as_name` and `country` are the client's AS number, such as `AS64496`,
|
||||
the name of that AS, and its country, as GeoJS or the lookup database gives
|
||||
them. Each is empty when `SWWAF_LOOKUP_SOURCE` is `off`, for a client in
|
||||
@@ -809,7 +827,8 @@ which every line has.
|
||||
|
||||
No body is logged, and no header but those above. `smallwebwaf`'s own messages
|
||||
(start, the settings, stop, errors) share the stream as JSON lines marked
|
||||
`"type":"process"`, each with `instance` as a request's line has it.
|
||||
`"type":"process"`, each with `instance` as a request's line has it, and those
|
||||
less severe than `SWWAF_LOG_LEVEL` are not written.
|
||||
|
||||
Go's HTTP server, on which `smallwebwaf` is built, reads a request's line and
|
||||
headers before `smallwebwaf` sees the request, and some requests end there,
|
||||
@@ -1077,9 +1096,10 @@ with times in UTC.
|
||||
zone gave it, `fetched`; and under `abuseipdb` (see "AbuseIPDB" below), the
|
||||
`day`, in UTC, whose checks it counts, left out before the first, the checks
|
||||
`spent` that day, and under `scores`, each score of AbuseIPDB still in use:
|
||||
the `client`'s address, its `score`, and when AbuseIPDB gave it, `fetched`. As
|
||||
the file is read, the lists the settings no longer name, and the verdicts of
|
||||
the zones they no longer name, are dropped.
|
||||
the `client`, its IPv4 address as a /32 or its IPv6 group, its `score`, and
|
||||
when AbuseIPDB gave it, `fetched`. As the file is read, the lists the settings
|
||||
no longer name, and the verdicts of the zones they no longer name, are
|
||||
dropped.
|
||||
- `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
|
||||
read: under `cooldowns`, for each event and netblock, with the `source` too
|
||||
for a `reputation_hit`, or event and `file` or `source`, or for an `anomaly`,
|
||||
@@ -1378,7 +1398,7 @@ request that carries the token as `Authorization: Bearer <token>`:
|
||||
`reason`. `netblock` is a netblock such as `203.0.113.0/24`, or a client's
|
||||
address, which bans the netblock a ban on that client covers: its IPv4
|
||||
address, or the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or
|
||||
its IPv6 /64. `duration` is a duration such as `1h` or `7d`, or `permanent`.
|
||||
its IPv6 group. `duration` is a duration such as `1h` or `7d`, or `permanent`.
|
||||
The ban starts at once, its `cause` is `admin`, and it is made even while
|
||||
another ban on the netblock lasts. A body that is not such an object, has
|
||||
another field, has anything but whitespace after the object, or is longer than
|
||||
@@ -1659,7 +1679,7 @@ setting uses the answer, unless you set `SWWAF_LOOKUP_SOURCE=off`. The only
|
||||
visitors it is not told about are those in `SWWAF_ALLOW_NETS` or
|
||||
`SWWAF_DENY_NETS`, those whose netblock a ban covers, and those on a private,
|
||||
loopback or link-local address. An IPv6 visitor is asked about by the first
|
||||
address of its /64. Each answer is kept for seven days, in memory and in
|
||||
address of its IPv6 group. Each answer is kept for seven days, in memory and in
|
||||
`lookups.json`, so that it survives a restart, and a visitor whose answer is
|
||||
kept is not asked about again.
|
||||
|
||||
@@ -1774,7 +1794,7 @@ request from it until the zone has answered. The name asked about is the one RFC
|
||||
`192.0.2.99` is asked about in `dnsbl.dronebl.org` as
|
||||
`99.2.0.192.dnsbl.dronebl.org`, or the 32 hex digits of an IPv6 address in
|
||||
reverse order, each followed by a dot. An IPv6 client is asked about by its own
|
||||
address, not by its /64.
|
||||
address, not by its IPv6 group.
|
||||
|
||||
A zone that answers that the name does not exist, or has no address, does not
|
||||
list the client, and one that answers with an address in `127.0.0.0/8` lists it.
|
||||
@@ -1820,15 +1840,19 @@ While `SWWAF_ABUSEIPDB_KEY` holds the key of an AbuseIPDB account, `smallwebwaf`
|
||||
asks AbuseIPDB's check endpoint, `https://api.abuseipdb.com/api/v2/check`, for
|
||||
the abuse confidence score of a client's own address, from 0 to 100. It is unset
|
||||
by default, for the reason no blocklist is named, and since AbuseIPDB needs an
|
||||
account. An IPv6 client is checked by its own address, not by its /64.
|
||||
account. An IPv6 client, an IPv6 group, is checked by the address of the request
|
||||
that has it checked, and its score is used for the whole group, whichever of its
|
||||
addresses sends, so that one client costs at most one check every
|
||||
`SWWAF_REPUTATION_CACHE_TTL`.
|
||||
|
||||
Only a client whose history counts an offence is checked, so that the checks are
|
||||
spent on suspects: so far, one that has broken a rate limit or a byte limit. A
|
||||
client dropped from the table of clients loses its history, and with it its
|
||||
offences. A client is checked in the background, at its first request after its
|
||||
offence that reaches the check: no request waits, a request refused under its
|
||||
ban is not checked, and the request that has it checked, and any other from it
|
||||
before the answer comes, goes on as from a client without a score.
|
||||
spent on suspects: so far, one that has broken a rate limit or a byte limit,
|
||||
matched a ban rule, or had a request refused by a block rule. A client dropped
|
||||
from the table of clients loses its history, and with it its offences. A client
|
||||
is checked in the background, at its first request after its offence that
|
||||
reaches the check: no request waits, a request refused under its ban is not
|
||||
checked, and the request that has it checked, and any other from it before the
|
||||
answer comes, goes on as from a client without a score.
|
||||
|
||||
A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE`, 75 by default, is a hit, and
|
||||
`SWWAF_REPUTATION_ACTION` says what is done with its client, as for a DNSBL
|
||||
@@ -1901,8 +1925,8 @@ given as files" above).
|
||||
file gives an AS number; asks the DNSBL zones about clients in the background,
|
||||
through the standard library's resolver, keeps their verdicts, and tells which
|
||||
zones' verdicts list an address; and checks clients with AbuseIPDB in the
|
||||
background, keeps their scores and the checks spent today, and tells whether
|
||||
an address's score is a hit.
|
||||
background, keeps their scores and the checks spent today, and tells whether a
|
||||
client's score is a hit.
|
||||
- `internal/ratelimit`: the table of clients: counts each client's requests and
|
||||
bytes, tells when they take it over a rate limit or a byte limit, and keeps
|
||||
each client's history.
|
||||
@@ -1929,17 +1953,17 @@ given as files" above).
|
||||
checks.
|
||||
|
||||
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
|
||||
table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least
|
||||
recently seen, the DNSBL zones' verdicts and AbuseIPDB's scores to 100,000 each,
|
||||
dropping the one fetched longest ago, the anomaly counters to 20,000, dropping
|
||||
the one counted least recently, and the banned netblocks in the order they were
|
||||
last seen, from which the ledger picks the ban to drop past `SWWAF_MAX_BANS`,
|
||||
and `github.com/prometheus/client_golang` keeps the metrics and serves them, and
|
||||
`github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file or a rule
|
||||
file is saved, or the lookup database replaced, and
|
||||
`github.com/oschwald/maxminddb-golang/v2` reads the lookup database, which the
|
||||
tests write with `github.com/maxmind/mmdbwriter`. The country codes are the list
|
||||
in `internal/config/config.go`.
|
||||
table of clients to `SWWAF_MAX_TRACKED_CLIENTS` and the GeoJS answers to
|
||||
100,000, dropping the least recently seen, the DNSBL zones' verdicts and
|
||||
AbuseIPDB's scores to 100,000 each, dropping the one fetched longest ago, the
|
||||
anomaly counters to 20,000, dropping the one counted least recently, and the
|
||||
banned netblocks in the order they were last seen, from which the ledger picks
|
||||
the ban to drop past `SWWAF_MAX_BANS`, and `github.com/prometheus/client_golang`
|
||||
keeps the metrics and serves them, and `github.com/fsnotify/fsnotify` tells
|
||||
`smallwebwaf` when a state file or a rule file is saved, or the lookup database
|
||||
replaced, and `github.com/oschwald/maxminddb-golang/v2` reads the lookup
|
||||
database, which the tests write with `github.com/maxmind/mmdbwriter`. The
|
||||
country codes are the list in `internal/config/config.go`.
|
||||
|
||||
## Entrypoints
|
||||
|
||||
|
||||
@@ -27,7 +27,8 @@ const maxCounters = 20000
|
||||
// The scopes, what a counter counts, as the settings, alerts.json and the
|
||||
// alerts name them.
|
||||
const (
|
||||
// ScopeClient is one client: an IPv4 address, or an IPv6 /64.
|
||||
// ScopeClient is one client: an IPv4 address, or an IPv6 netblock of
|
||||
// SWWAF_IPV6_GROUP_PREFIX.
|
||||
ScopeClient = "client"
|
||||
// ScopeNet is the netblock around a client, SWWAF_ANOMALY_NET_V4_PREFIX
|
||||
// or SWWAF_ANOMALY_NET_V6_PREFIX long.
|
||||
@@ -103,7 +104,8 @@ type Counter struct {
|
||||
// Request is a request that has ended, as the counters count it.
|
||||
type Request struct {
|
||||
// Client is the client's address, and ClientGroup the client it is
|
||||
// counted as: its IPv4 address, or its IPv6 /64.
|
||||
// counted as: its IPv4 address, or the IPv6 netblock of
|
||||
// SWWAF_IPV6_GROUP_PREFIX its address is in.
|
||||
Client netip.Addr
|
||||
ClientGroup netip.Prefix
|
||||
// ASN, ASName and Country are the client's as looked up, each "" when
|
||||
|
||||
@@ -48,6 +48,12 @@ type Config struct {
|
||||
// TrustedProxies are the netblocks whose X-Forwarded-For is
|
||||
// believed (SWWAF_TRUSTED_PROXIES).
|
||||
TrustedProxies []netip.Prefix
|
||||
// IPv6GroupPrefix is the length of the IPv6 netblock that is one client
|
||||
// (SWWAF_IPV6_GROUP_PREFIX), from 32 to 128.
|
||||
IPv6GroupPrefix int
|
||||
// MaxTrackedClients is the most clients the table of clients holds, in
|
||||
// memory and in clients.json (SWWAF_MAX_TRACKED_CLIENTS).
|
||||
MaxTrackedClients int
|
||||
// ClientRequestTimeout bounds reading the whole request from the
|
||||
// client (SWWAF_CLIENT_REQUEST_TIMEOUT).
|
||||
ClientRequestTimeout time.Duration
|
||||
@@ -209,6 +215,9 @@ type Config struct {
|
||||
// LogRequestHeaders are the request headers whose values the request
|
||||
// log gives, in lower case (SWWAF_LOG_REQUEST_HEADERS).
|
||||
LogRequestHeaders []string
|
||||
// LogLevel is the least severe of the process's own messages that are
|
||||
// written (SWWAF_LOG_LEVEL). It holds back no request log line.
|
||||
LogLevel slog.Level
|
||||
// AdminToken is the bearer token an admin sends for the ban endpoints
|
||||
// and /_smallwebwaf/clients/<ip> (SWWAF_ADMIN_TOKEN), "" while it is
|
||||
// unset and they are off.
|
||||
@@ -297,6 +306,10 @@ const (
|
||||
gibibyte = 1 << 30
|
||||
ipv4Bits = 32
|
||||
ipv6Bits = 128
|
||||
// minIPv6GroupPrefix is the shortest SWWAF_IPV6_GROUP_PREFIX, the
|
||||
// netblock a provider is usually given: a shorter one would make one
|
||||
// client of the customers of several providers.
|
||||
minIPv6GroupPrefix = 32
|
||||
// minTokenLength is the fewest characters a token may have.
|
||||
minTokenLength = 32
|
||||
// masked is what the log shows for a token that is set, and in place of
|
||||
@@ -346,6 +359,9 @@ var (
|
||||
"is not the length of an IPv4 netblock, from 0 to 32, such as 24")
|
||||
errNotV6Prefix = errors.New(
|
||||
"is not the length of an IPv6 netblock, from 0 to 128, such as 48")
|
||||
errNotIPv6GroupPrefix = errors.New(
|
||||
"is not the length of an IPv6 netblock, from 32 to 128, such as 64")
|
||||
errNotLogLevel = errors.New("is not debug, info, warn or error")
|
||||
errNotNamedNetblock = errors.New(
|
||||
"is not a name, = and a netblock, such as office=203.0.113.0/24")
|
||||
errNotAbsolutePath = errors.New(
|
||||
@@ -411,6 +427,8 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
||||
InstanceName: env.instanceName(),
|
||||
Observe: env.observe("SWWAF_MODE", "enforce"),
|
||||
TrustedProxies: env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges),
|
||||
IPv6GroupPrefix: env.ipv6GroupPrefix("SWWAF_IPV6_GROUP_PREFIX", "64"),
|
||||
MaxTrackedClients: env.numberNotOff("SWWAF_MAX_TRACKED_CLIENTS", "20000"),
|
||||
ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"),
|
||||
ClientRequestHeaderMaxBytes: env.headerSize(
|
||||
"SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES", "32K"),
|
||||
@@ -465,6 +483,7 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
||||
StateCounterInterval: env.durationNotOff("SWWAF_STATE_COUNTER_INTERVAL", "15m"),
|
||||
LogRequestHeaders: env.headerNames("SWWAF_LOG_REQUEST_HEADERS",
|
||||
"accept,accept-language,accept-encoding,content-type,origin,range"),
|
||||
LogLevel: env.logLevel("SWWAF_LOG_LEVEL", "info"),
|
||||
AdminToken: env.token("SWWAF_ADMIN_TOKEN"),
|
||||
MetricsToken: env.token("SWWAF_METRICS_TOKEN"),
|
||||
MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
|
||||
@@ -944,6 +963,37 @@ func (e *environment) v6Prefix(name, defaultValue string) int {
|
||||
return length
|
||||
}
|
||||
|
||||
// ipv6GroupPrefix reads the setting that is the length of the IPv6
|
||||
// netblock that is one client, from minIPv6GroupPrefix to 128.
|
||||
func (e *environment) ipv6GroupPrefix(name, defaultValue string) int {
|
||||
value := e.value(name, defaultValue)
|
||||
|
||||
length, err := strconv.Atoi(value)
|
||||
if err != nil || length < minIPv6GroupPrefix || length > ipv6Bits {
|
||||
e.check(name, fmt.Errorf("%q %w", value, errNotIPv6GroupPrefix))
|
||||
}
|
||||
|
||||
return length
|
||||
}
|
||||
|
||||
// logLevel reads the setting that is the least severe of the process's
|
||||
// own messages that are written: debug, info, warn or error.
|
||||
func (e *environment) logLevel(name, defaultValue string) slog.Level {
|
||||
value := e.value(name, defaultValue)
|
||||
|
||||
level, known := map[string]slog.Level{
|
||||
"debug": slog.LevelDebug,
|
||||
"info": slog.LevelInfo,
|
||||
"warn": slog.LevelWarn,
|
||||
"error": slog.LevelError,
|
||||
}[value]
|
||||
if !known {
|
||||
e.check(name, fmt.Errorf("%q %w", value, errNotLogLevel))
|
||||
}
|
||||
|
||||
return level
|
||||
}
|
||||
|
||||
// thresholds reads the four anomaly thresholds whose settings' names
|
||||
// start with prefix: requests and bytes per minute and per hour. Each is
|
||||
// off by default.
|
||||
|
||||
@@ -27,6 +27,8 @@ const (
|
||||
upstreamURL = "SWWAF_UPSTREAM_URL"
|
||||
mode = "SWWAF_MODE"
|
||||
trustedProxies = "SWWAF_TRUSTED_PROXIES"
|
||||
ipv6GroupPrefix = "SWWAF_IPV6_GROUP_PREFIX"
|
||||
maxTrackedClients = "SWWAF_MAX_TRACKED_CLIENTS"
|
||||
clientRequestTimeout = "SWWAF_CLIENT_REQUEST_TIMEOUT"
|
||||
clientHeaderMaxBytes = "SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES"
|
||||
clientIdleTimeout = "SWWAF_CLIENT_IDLE_TIMEOUT"
|
||||
@@ -84,6 +86,7 @@ const (
|
||||
metricsTopN = "SWWAF_METRICS_TOP_N"
|
||||
instanceName = "SWWAF_INSTANCE_NAME"
|
||||
logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS"
|
||||
logLevel = "SWWAF_LOG_LEVEL"
|
||||
rulesDir = "SWWAF_RULES_DIR"
|
||||
rulesEnabled = "SWWAF_RULES_ENABLED"
|
||||
logRemoteURL = "SWWAF_LOG_REMOTE_URL"
|
||||
@@ -399,6 +402,57 @@ func TestValuesAsSet(t *testing.T) {
|
||||
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
|
||||
}
|
||||
|
||||
func TestIPv6GroupPrefixMaxTrackedClientsAndLogLevel(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
env environment
|
||||
prefix, clients int
|
||||
level slog.Level
|
||||
}{
|
||||
{environment{}, 64, 20000, slog.LevelInfo},
|
||||
{
|
||||
environment{ipv6GroupPrefix: "48", maxTrackedClients: "500", logLevel: "warn"},
|
||||
48, 500, slog.LevelWarn,
|
||||
},
|
||||
} {
|
||||
cfg := fromEnvironment(t, tc.env)
|
||||
if cfg.IPv6GroupPrefix != tc.prefix || cfg.MaxTrackedClients != tc.clients ||
|
||||
cfg.LogLevel != tc.level {
|
||||
t.Errorf("%v gave %d, %d and %v, want %d, %d and %v", tc.env,
|
||||
cfg.IPv6GroupPrefix, cfg.MaxTrackedClients, cfg.LogLevel,
|
||||
tc.prefix, tc.clients, tc.level)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIPv6GroupPrefixFrom32To128(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, length := range []int{32, 128} {
|
||||
cfg := fromEnvironment(t, environment{ipv6GroupPrefix: strconv.Itoa(length)})
|
||||
if cfg.IPv6GroupPrefix != length {
|
||||
t.Errorf("%s=%d gave %d", ipv6GroupPrefix, length, cfg.IPv6GroupPrefix)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachLogLevel(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for value, want := range map[string]slog.Level{
|
||||
"debug": slog.LevelDebug,
|
||||
"info": slog.LevelInfo,
|
||||
"warn": slog.LevelWarn,
|
||||
"error": slog.LevelError,
|
||||
} {
|
||||
cfg := fromEnvironment(t, environment{logLevel: value})
|
||||
if cfg.LogLevel != want {
|
||||
t.Errorf("%s=%s gave %v, want %v", logLevel, value, cfg.LogLevel, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestByteLimitSettingsAsSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -1698,6 +1752,10 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
|
||||
{trustedProxies, "traefik"},
|
||||
{trustedProxies, "10.0.0.0/8,,192.168.0.0/16"},
|
||||
{trustedProxies, "fe80::1%eth0"},
|
||||
{ipv6GroupPrefix, "31"}, {ipv6GroupPrefix, "129"}, {ipv6GroupPrefix, "/64"},
|
||||
{ipv6GroupPrefix, off}, {ipv6GroupPrefix, ""},
|
||||
{maxTrackedClients, "0"}, {maxTrackedClients, "-1"}, {maxTrackedClients, off},
|
||||
{maxTrackedClients, "20K"},
|
||||
{allowNets, "192.0.2.0/24,monitoring"},
|
||||
{rateLimitExemptNets, "2001:db8::/129"},
|
||||
{denyNets, "198.51.100.0/24,"},
|
||||
@@ -1751,6 +1809,7 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
|
||||
{logRequestHeaders, "accept language"}, {logRequestHeaders, "x-foo:"},
|
||||
{logRequestHeaders, "host"}, {logRequestHeaders, "accept,Host"},
|
||||
{logRequestHeaders, "transfer-encoding"}, {logRequestHeaders, "TRANSFER-ENCODING"},
|
||||
{logLevel, "INFO"}, {logLevel, "warning"}, {logLevel, "trace"}, {logLevel, ""},
|
||||
{rulesEnabled, "yes"}, {rulesEnabled, "True"},
|
||||
})
|
||||
}
|
||||
@@ -1993,6 +2052,8 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
upstreamURL: "http://127.0.0.1:8081",
|
||||
mode: "enforce",
|
||||
trustedProxies: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
|
||||
ipv6GroupPrefix: "64",
|
||||
maxTrackedClients: "20000",
|
||||
clientRequestTimeout: "45s",
|
||||
clientHeaderMaxBytes: "32K",
|
||||
clientIdleTimeout: "120s",
|
||||
@@ -2050,6 +2111,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
metricsTopN: "50",
|
||||
instanceName: hostname,
|
||||
logRequestHeaders: defaultLogRequestHeaders,
|
||||
logLevel: "info",
|
||||
rulesDir: "/etc/smallwebwaf/rules.d",
|
||||
rulesEnabled: "true",
|
||||
logRemoteURL: "",
|
||||
|
||||
@@ -176,7 +176,8 @@ func New(params Params) *GeoJS {
|
||||
// when it ends.
|
||||
//
|
||||
// GeoJS is asked about the client's first address, which is the client's
|
||||
// own address for IPv4, and an address in the same place for an IPv6 /64.
|
||||
// own address for IPv4, and an address in the same place for an IPv6
|
||||
// netblock.
|
||||
func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
|
||||
answer, asked := g.answerOrWait(ctx, client)
|
||||
if asked == nil {
|
||||
|
||||
@@ -282,7 +282,7 @@ func (rq *request) showClient() {
|
||||
|
||||
answer := clientAnswer{Bans: state.BanEntries(rq.h.ledger.Covering(addr))}
|
||||
|
||||
client, seen := rq.h.limiter.Client(clientGroup(addr))
|
||||
client, seen := rq.h.limiter.Client(rq.h.clientGroup(addr))
|
||||
if seen {
|
||||
answer.Client = &client
|
||||
}
|
||||
|
||||
@@ -45,7 +45,7 @@ func (rq *request) banned(now time.Time) bool {
|
||||
// the client over a rate limit, as its limit percentage lowers it, which
|
||||
// breaks it.
|
||||
func (rq *request) limitBroken(now time.Time) bool {
|
||||
counts, hit, over := rq.h.limiter.Count(clientGroup(rq.client), now,
|
||||
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now,
|
||||
rq.limitPercent.percent)
|
||||
rq.line.Counts = counts
|
||||
|
||||
@@ -71,7 +71,7 @@ func (rq *request) countBytes() {
|
||||
|
||||
now := rq.h.now()
|
||||
|
||||
counts, hit, over := rq.h.limiter.CountBytes(clientGroup(rq.client), now,
|
||||
counts, hit, over := rq.h.limiter.CountBytes(rq.h.clientGroup(rq.client), now,
|
||||
rq.countedBytes(), rq.bytesPercent.percent)
|
||||
rq.line.Counts.MinuteBytes = counts.MinuteBytes
|
||||
rq.line.Counts.HourBytes = counts.HourBytes
|
||||
@@ -155,7 +155,7 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
|
||||
}
|
||||
|
||||
ban, made := rq.h.ledger.BanForLimit(netblock, now, notes)
|
||||
rq.h.limiter.Reset(clientGroup(rq.client))
|
||||
rq.h.limiter.Reset(rq.h.clientGroup(rq.client))
|
||||
rq.line.BanExpires = banExpires(ban)
|
||||
|
||||
if made {
|
||||
@@ -276,7 +276,7 @@ func (h *handler) netblock(client netip.Addr) netip.Prefix {
|
||||
return netip.PrefixFrom(addr, h.config.BanScopeV4Prefix).Masked()
|
||||
}
|
||||
|
||||
return clientGroup(addr)
|
||||
return h.clientGroup(addr)
|
||||
}
|
||||
|
||||
// banExpires is when ban ends, as the log line gives it: a time, or
|
||||
|
||||
@@ -165,9 +165,14 @@ func TestBanCoversTheClientsNetblock(t *testing.T) {
|
||||
[]string{otherClient, exempt}, []string{"203.0.112.9", allowed},
|
||||
},
|
||||
{
|
||||
"an IPv6 /64", nil, "2001:db8:5::1",
|
||||
"an IPv6 /64, by default", nil, "2001:db8:5::1",
|
||||
[]string{"2001:db8:5::ffff:1"}, []string{"2001:db8:5:1::1"},
|
||||
},
|
||||
{
|
||||
"the IPv6 netblock SWWAF_IPV6_GROUP_PREFIX sets",
|
||||
map[string]string{ipv6GroupPrefix: "48"}, "2001:db8:7::1",
|
||||
[]string{"2001:db8:7:ffff::1"}, []string{"2001:db8:8::1"},
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -76,16 +76,14 @@ func scheme(r *http.Request, peerTrusted bool) string {
|
||||
return proto
|
||||
}
|
||||
|
||||
// ipv6GroupPrefix is the length of the IPv6 netblock that is one client.
|
||||
const ipv6GroupPrefix = 64
|
||||
|
||||
// clientGroup is the client a request is counted toward: its IPv4
|
||||
// address, or the /64 its IPv6 address is in, since one abuser usually
|
||||
// holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
|
||||
func clientGroup(addr netip.Addr) netip.Prefix {
|
||||
// address, or its IPv6 group, the netblock its IPv6 address is in of the
|
||||
// length SWWAF_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser
|
||||
// usually holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
|
||||
func (h *handler) clientGroup(addr netip.Addr) netip.Prefix {
|
||||
addr = addr.Unmap()
|
||||
if addr.Is6() {
|
||||
return netip.PrefixFrom(addr, ipv6GroupPrefix).Masked()
|
||||
return netip.PrefixFrom(addr, h.config.IPv6GroupPrefix).Masked()
|
||||
}
|
||||
|
||||
return netip.PrefixFrom(addr, addr.BitLen())
|
||||
|
||||
@@ -56,6 +56,24 @@ func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTableOfClientsHoldsAtMostMaxTrackedClients(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, server := startWithClock(t, "", map[string]string{maxTrackedClients: "2"})
|
||||
|
||||
// The third client drops the least recently seen, the first, with its
|
||||
// history.
|
||||
for _, from := range []string{"192.0.2.1", "192.0.2.2", "192.0.2.3"} {
|
||||
s.get(from, http.StatusOK, requestlog.ActionForward)
|
||||
}
|
||||
|
||||
_, held := server.Limiter.Client(netip.MustParsePrefix("192.0.2.1/32"))
|
||||
if server.Limiter.Len() != 2 || held {
|
||||
t.Errorf("the table holds %d clients, the first among them: %t; want 2, "+
|
||||
"without it", server.Limiter.Len(), held)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHistoryCountsTheBodiesEachWay(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -31,9 +31,9 @@ func (rq *request) lookUp(ctx context.Context) {
|
||||
}
|
||||
|
||||
if rq.h.config.LookupSource == "file" {
|
||||
rq.lookupAnswer = rq.h.lookupFile.LookUp(clientGroup(rq.client))
|
||||
rq.lookupAnswer = rq.h.lookupFile.LookUp(rq.h.clientGroup(rq.client))
|
||||
} else {
|
||||
rq.lookupAnswer = rq.h.geojs.LookUp(ctx, clientGroup(rq.client))
|
||||
rq.lookupAnswer = rq.h.geojs.LookUp(ctx, rq.h.clientGroup(rq.client))
|
||||
}
|
||||
|
||||
rq.lookedUp = true
|
||||
|
||||
@@ -320,7 +320,7 @@ func TestServerHasTheDefaultLimits(t *testing.T) {
|
||||
server := proxy.New(proxy.Params{
|
||||
Config: cfg,
|
||||
RequestLog: io.Discard,
|
||||
ProcessLog: requestlog.NewProcessLogger(io.Discard, cfg.InstanceName),
|
||||
ProcessLog: requestlog.NewProcessLogger(io.Discard, cfg.InstanceName, cfg.LogLevel),
|
||||
})
|
||||
|
||||
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
|
||||
@@ -399,6 +399,25 @@ func TestAnswers502WhenTheAppCannotBeReached(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogLevelHoldsBackNoRequestLine(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// At error the warning that the request to the app failed is held back,
|
||||
// and is written before the answer is.
|
||||
addr, out := startProxy(t, "http://"+localhost+":1", map[string]string{
|
||||
"SWWAF_LOG_LEVEL": "error",
|
||||
})
|
||||
|
||||
wantStatus(t, get(t, addr, "/"), http.StatusBadGateway)
|
||||
wantLine(t, out.requestLine(t), http.StatusBadGateway, requestlog.ActionUpstreamError)
|
||||
|
||||
for _, line := range out.lines(t) {
|
||||
if line["type"] == "process" {
|
||||
t.Errorf("process line %v, want none at error", line)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogsAnAnswerThatBrokeOff(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -124,7 +124,7 @@ func New(params Params) *Server {
|
||||
BytesPerMinute: params.Config.BytesLimitPerMinute,
|
||||
BytesPerHour: params.Config.BytesLimitPerHour,
|
||||
BytesPerDay: params.Config.BytesLimitPerDay,
|
||||
}),
|
||||
}, params.Config.MaxTrackedClients),
|
||||
ledger: bans.New(bans.Rules{
|
||||
LimitBanDuration: params.Config.LimitBanDuration,
|
||||
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
|
||||
|
||||
@@ -61,6 +61,8 @@ const (
|
||||
requestMaxBytes = "SWWAF_REQUEST_MAX_BYTES"
|
||||
responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES"
|
||||
trustedProxies = "SWWAF_TRUSTED_PROXIES"
|
||||
ipv6GroupPrefix = "SWWAF_IPV6_GROUP_PREFIX"
|
||||
maxTrackedClients = "SWWAF_MAX_TRACKED_CLIENTS"
|
||||
allowNets = "SWWAF_ALLOW_NETS"
|
||||
rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS"
|
||||
denyNets = "SWWAF_DENY_NETS"
|
||||
@@ -295,7 +297,7 @@ func newProxy(
|
||||
}
|
||||
|
||||
out := &output{}
|
||||
processLog := requestlog.NewProcessLogger(out, cfg.InstanceName)
|
||||
processLog := requestlog.NewProcessLogger(out, cfg.InstanceName, cfg.LogLevel)
|
||||
|
||||
ruleFiles, err := rules.Load(rules.Params{
|
||||
Dir: cfg.RulesDir, Enabled: cfg.RulesEnabled, ProcessLog: processLog,
|
||||
|
||||
@@ -72,6 +72,49 @@ func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// With SWWAF_IPV6_GROUP_PREFIX at 48, the first two addresses, in two
|
||||
// /64s of one /48, are one client, and the second's request breaks the
|
||||
// limit; the third, in the next /48, is another client.
|
||||
const (
|
||||
first = "2001:db8:9::1"
|
||||
second = "2001:db8:9:1::1"
|
||||
other = "2001:db8:a::1"
|
||||
)
|
||||
|
||||
for _, tc := range []struct {
|
||||
setting, value string
|
||||
// status and action are those of the request that breaks the
|
||||
// limit: a rate limit refuses it, a byte limit passes it on.
|
||||
status int
|
||||
action string
|
||||
}{
|
||||
{rateLimitPerMinute, "1", http.StatusForbidden, requestlog.ActionRateLimited},
|
||||
{bytesLimitPerMinute, byteLimit, http.StatusOK, requestlog.ActionForward},
|
||||
} {
|
||||
t.Run(tc.setting, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{
|
||||
ipv6GroupPrefix: "48", tc.setting: tc.value,
|
||||
})
|
||||
|
||||
s.get(first, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line := s.get(second, tc.status, tc.action)
|
||||
if line.ClientGroup != "2001:db8:9::/48" ||
|
||||
line.Offence != requestlog.OffenceLimit {
|
||||
t.Errorf("log line has client_group %q and offence %q, "+
|
||||
"want 2001:db8:9::/48 and limit", line.ClientGroup, line.Offence)
|
||||
}
|
||||
|
||||
s.get(other, http.StatusOK, requestlog.ActionForward)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -43,18 +43,20 @@ func (rq *request) dnsblDenied(ctx context.Context) bool {
|
||||
// its score of the client is a hit, and reports whether
|
||||
// SWWAF_REPUTATION_ACTION, being deny, refuses the request, as dnsblDenied
|
||||
// does for a zone. While SWWAF_ABUSEIPDB_KEY is unset it does nothing. A
|
||||
// client without a score is checked in the background if its history
|
||||
// counts an offence, and the request does not wait for the answer. ctx is
|
||||
// the request's own context.
|
||||
// client without a score is checked in the background, by the request's
|
||||
// address, if its history counts an offence, and the request does not
|
||||
// wait for the answer. The score is then used for each address of the
|
||||
// client. ctx is the request's own context.
|
||||
func (rq *request) abuseIPDBDenied(ctx context.Context) bool {
|
||||
if rq.h.config.AbuseIPDBKey == "" {
|
||||
return false
|
||||
}
|
||||
|
||||
held, _ := rq.h.limiter.Client(clientGroup(rq.client))
|
||||
client := rq.h.clientGroup(rq.client)
|
||||
held, _ := rq.h.limiter.Client(client)
|
||||
offender := held.History.Offences != ratelimit.Offences{}
|
||||
|
||||
score, hit := rq.h.abuseIPDB.Hit(ctx, rq.client, offender)
|
||||
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
|
||||
if !hit {
|
||||
return false
|
||||
}
|
||||
@@ -86,7 +88,7 @@ func (rq *request) noteHit(source, reason string, detail map[string]any) {
|
||||
rq.h.alerts.Raise(alerts.Alert{
|
||||
Event: alerts.EventReputationHit,
|
||||
Client: rq.client,
|
||||
Netblock: clientGroup(rq.client),
|
||||
Netblock: rq.h.clientGroup(rq.client),
|
||||
ASN: rq.line.ASN,
|
||||
ASName: rq.line.ASName,
|
||||
Country: rq.line.Country,
|
||||
|
||||
@@ -1,17 +1,20 @@
|
||||
package proxy_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"maps"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
@@ -661,6 +664,90 @@ func TestOnlyAClientThatHasCommittedAnOffenceIsCheckedWithAbuseIPDB(t *testing.T
|
||||
wantAbuseIPDBChecks(t, server, 1)
|
||||
}
|
||||
|
||||
func TestIPv6ClientCostsOneAbuseIPDBCheckWhicheverOfItsAddressesSends(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
forward := requestlog.ActionForward
|
||||
|
||||
// 15 addresses of 2001:db8:1:2::/64, one client, each in a part of it
|
||||
// of its own.
|
||||
var addresses []string
|
||||
for i := 1; i < 16; i++ {
|
||||
addresses = append(addresses, fmt.Sprintf("2001:db8:1:2:%x::9", i<<12))
|
||||
}
|
||||
|
||||
s, clk, server := startWithClock(t, "", map[string]string{
|
||||
abuseIPDBKey: accountKey, reputationAction: actionLog,
|
||||
rateLimitPerMinute: strconv.Itoa(len(addresses)),
|
||||
})
|
||||
|
||||
// The client breaks the rate limit from its first address, which bans
|
||||
// it for an hour.
|
||||
for range addresses {
|
||||
s.get(addresses[0], http.StatusOK, forward)
|
||||
}
|
||||
|
||||
s.get(addresses[0], http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
clk.advance(time.Hour)
|
||||
|
||||
// Once the ban has ended, which set its counters back to zero, a
|
||||
// request from each of its addresses has it checked once.
|
||||
for _, address := range addresses {
|
||||
s.get(address, http.StatusOK, forward)
|
||||
}
|
||||
|
||||
wantAbuseIPDBChecks(t, server, 1)
|
||||
}
|
||||
|
||||
// probePath is the path the ban rule of testRules, probe, matches.
|
||||
const probePath = "/.env"
|
||||
|
||||
func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
// path is what the client asks for, status and action what that
|
||||
// request is answered and logged with, and want the offences its
|
||||
// history then counts.
|
||||
path string
|
||||
status int
|
||||
action string
|
||||
want ratelimit.Offences
|
||||
}{
|
||||
{
|
||||
"a block rule", "/blocked", http.StatusForbidden, requestlog.ActionRuleBlocked,
|
||||
ratelimit.Offences{RuleBlocked: 1},
|
||||
},
|
||||
{
|
||||
"a ban rule", probePath, http.StatusForbidden, requestlog.ActionBanned,
|
||||
ratelimit.Offences{Attack: 1},
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server := startWithClock(t, "", map[string]string{
|
||||
abuseIPDBKey: accountKey, reputationAction: actionLog,
|
||||
rulesDir: writeRules(t, testRules), attackBanDuration: "1h",
|
||||
})
|
||||
|
||||
s.request(client, tc.path, tc.status, tc.action)
|
||||
wantAbuseIPDBChecks(t, server, 0)
|
||||
|
||||
if got := historyOf(t, server, client).Offences; got != tc.want {
|
||||
t.Errorf("history counts the offences %+v, want %+v", got, tc.want)
|
||||
}
|
||||
|
||||
// Its next request, once a ban rule's ban has ended, has it
|
||||
// checked.
|
||||
clk.advance(time.Hour)
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
wantAbuseIPDBChecks(t, server, 1)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachReputationActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -820,12 +907,14 @@ func loadVerdicts(server *proxy.Server, listedBy map[string][]string) {
|
||||
}
|
||||
|
||||
// loadScores puts into server's AbuseIPDB the score scores gives each
|
||||
// client, fetched at verdictsFetched, as reputation.json would at start.
|
||||
// client, an IPv4 address, fetched at verdictsFetched, as reputation.json
|
||||
// would at start.
|
||||
func loadScores(server *proxy.Server, scores map[string]int64) {
|
||||
kept := make([]reputation.Score, 0, len(scores))
|
||||
for client, score := range scores {
|
||||
kept = append(kept, reputation.Score{
|
||||
Client: netip.MustParseAddr(client), Score: score, Fetched: verdictsFetched(),
|
||||
Client: netip.MustParsePrefix(client + "/32"), Score: score,
|
||||
Fetched: verdictsFetched(),
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -63,6 +63,10 @@ type request struct {
|
||||
// limits and for the byte limits.
|
||||
counted bool
|
||||
limitPercent, bytesPercent percentage
|
||||
// attack is true for a request that matched a ban rule, and
|
||||
// ruleBlocked for one a block rule refused, each an offence its
|
||||
// client's history counts.
|
||||
attack, ruleBlocked bool
|
||||
// blocklisted is true once a blocklist is found to list the client,
|
||||
// dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once
|
||||
// AbuseIPDB's score of it is a hit.
|
||||
@@ -136,7 +140,7 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
|
||||
RequestID: requestID(r, peerTrusted),
|
||||
PeerIP: peer.String(),
|
||||
ForwardedFor: strings.Join(forwardedFor, ", "),
|
||||
ClientGroup: clientGroup(client).String(),
|
||||
ClientGroup: h.clientGroup(client).String(),
|
||||
ContentType: r.Header.Get("Content-Type"),
|
||||
RequestHeaders: requestHeaders(r, h.config.LogRequestHeaders),
|
||||
HasAuthorization: len(r.Header.Values("Authorization")) > 0,
|
||||
@@ -535,13 +539,15 @@ func timing(start, end time.Time) *float64 {
|
||||
func (rq *request) addToHistory() {
|
||||
forwarded := !rq.upstreamStart.IsZero()
|
||||
|
||||
rq.h.limiter.AddToHistory(clientGroup(rq.client), rq.h.now(), ratelimit.Request{
|
||||
rq.h.limiter.AddToHistory(rq.h.clientGroup(rq.client), rq.h.now(), ratelimit.Request{
|
||||
Forwarded: forwarded,
|
||||
Refused: !forwarded && rq.refused.Load() != nil,
|
||||
Status: rq.out.status,
|
||||
RequestBytes: rq.requestBytes(),
|
||||
ResponseBytes: rq.out.bytes,
|
||||
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
|
||||
Attack: rq.attack,
|
||||
RuleBlocked: rq.ruleBlocked,
|
||||
})
|
||||
|
||||
answer, found := rq.answerAtTheEnd()
|
||||
@@ -566,7 +572,7 @@ func (rq *request) countAnomalies() {
|
||||
|
||||
rq.h.anomalies.Count(rq.h.now(), anomaly.Request{
|
||||
Client: rq.client,
|
||||
ClientGroup: clientGroup(rq.client),
|
||||
ClientGroup: rq.h.clientGroup(rq.client),
|
||||
ASN: answer.ASN,
|
||||
ASName: answer.ASName,
|
||||
Country: answer.Country,
|
||||
@@ -595,7 +601,7 @@ func (rq *request) answerAtTheEnd() (lookup.Answer, bool) {
|
||||
return rq.lookupAnswer, true
|
||||
}
|
||||
|
||||
return rq.h.geojs.Kept(clientGroup(rq.client))
|
||||
return rq.h.geojs.Kept(rq.h.clientGroup(rq.client))
|
||||
}
|
||||
|
||||
// requestBytes is how many bytes of the request's body have been read.
|
||||
|
||||
@@ -12,7 +12,8 @@ import (
|
||||
// action of the rule that refuses it, ActionRuleBlocked for a block rule
|
||||
// and ActionBanned for a ban rule, or "" when none does. A ban rule bans
|
||||
// the client's netblock for a clear sign of attack, or in observe mode
|
||||
// raises the alert for the ban it would have made.
|
||||
// raises the alert for the ban it would have made. Either rule's match
|
||||
// is noted as an offence, for the client's history.
|
||||
func (rq *request) checkRules(now time.Time) string {
|
||||
matched := rq.h.rules.Match(rq.in)
|
||||
|
||||
@@ -28,8 +29,11 @@ func (rq *request) checkRules(now time.Time) string {
|
||||
// Only the last rule matched can refuse the request.
|
||||
switch last := matched[len(matched)-1]; last.Action {
|
||||
case rules.ActionBlock:
|
||||
rq.ruleBlocked = true
|
||||
|
||||
return requestlog.ActionRuleBlocked
|
||||
case rules.ActionBan:
|
||||
rq.attack = true
|
||||
rq.banForAttack(now, last)
|
||||
|
||||
return requestlog.ActionBanned
|
||||
|
||||
@@ -11,7 +11,7 @@ import (
|
||||
func TestHistoryKeepsEveryRequest(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{})
|
||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
@@ -53,7 +53,7 @@ func TestHistoryKeepsEveryRequest(t *testing.T) {
|
||||
func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{})
|
||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
other := netip.MustParsePrefix("198.51.100.7/32")
|
||||
start := midnight()
|
||||
@@ -90,7 +90,7 @@ func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
|
||||
func TestResetKeepsTheHistory(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit})
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
@@ -109,7 +109,7 @@ func TestResetKeepsTheHistory(t *testing.T) {
|
||||
func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{})
|
||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
||||
|
||||
for client, requests := range map[string]int{
|
||||
"198.51.100.9/32": 2,
|
||||
|
||||
@@ -2,8 +2,9 @@
|
||||
// and bytes counted over a minute, an hour and a day, as the "Counting
|
||||
// method" section of SPEC.md describes, which tell when a request takes
|
||||
// the client over a rate limit or a byte limit, and each client's history
|
||||
// since it was first seen. At most 20,000 clients are kept, in memory, and
|
||||
// written to clients.json and read from it by the state package.
|
||||
// since it was first seen. At most SWWAF_MAX_TRACKED_CLIENTS clients are
|
||||
// kept, in memory, and written to clients.json and read from it by the
|
||||
// state package.
|
||||
package ratelimit
|
||||
|
||||
import (
|
||||
@@ -16,11 +17,6 @@ import (
|
||||
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||
)
|
||||
|
||||
// maxClients is how many clients are kept. Past it, the least recently
|
||||
// seen client is dropped, with its history, and starts afresh if it comes
|
||||
// back.
|
||||
const maxClients = 20000
|
||||
|
||||
const day = 24 * time.Hour
|
||||
|
||||
// The kinds of limits, as the metrics name them.
|
||||
@@ -117,9 +113,15 @@ type Responses struct {
|
||||
}
|
||||
|
||||
// Offences are a client's offences, by kind.
|
||||
//
|
||||
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
||||
type Offences struct {
|
||||
// Limit is its requests that broke a rate limit or a byte limit.
|
||||
Limit int64 `json:"limit"`
|
||||
// Limit is its requests that broke a rate limit or a byte limit,
|
||||
// Attack those that matched a ban rule, a clear sign of attack, and
|
||||
// RuleBlocked those a block rule refused.
|
||||
Limit int64 `json:"limit"`
|
||||
Attack int64 `json:"attack"`
|
||||
RuleBlocked int64 `json:"rule_blocked"`
|
||||
}
|
||||
|
||||
// Request is what a client's history keeps of one of its requests.
|
||||
@@ -137,12 +139,18 @@ type Request struct {
|
||||
RequestBytes int64
|
||||
ResponseBytes int64
|
||||
// BrokeLimit is true for a request that broke a rate limit or a byte
|
||||
// limit.
|
||||
BrokeLimit bool
|
||||
// limit, Attack for one that matched a ban rule, and RuleBlocked for
|
||||
// one a block rule refused.
|
||||
BrokeLimit bool
|
||||
Attack bool
|
||||
RuleBlocked bool
|
||||
}
|
||||
|
||||
// New returns a Limiter for limits, with no client counted yet.
|
||||
func New(limits Limits) *Limiter {
|
||||
// New returns a Limiter for limits, with no client counted yet, whose
|
||||
// table holds at most maxClients clients (SWWAF_MAX_TRACKED_CLIENTS). Past
|
||||
// it, the least recently seen client is dropped, with its history, and
|
||||
// starts afresh if it comes back.
|
||||
func New(limits Limits, maxClients int) *Limiter {
|
||||
clients, err := simplelru.NewLRU[netip.Prefix, *Client](maxClients, nil)
|
||||
if err != nil {
|
||||
panic(err) // NewLRU fails only for a size below one
|
||||
@@ -258,6 +266,14 @@ func (l *Limiter) AddToHistory(client netip.Prefix, now time.Time, r Request) {
|
||||
if r.BrokeLimit {
|
||||
h.Offences.Limit++
|
||||
}
|
||||
|
||||
if r.Attack {
|
||||
h.Offences.Attack++
|
||||
}
|
||||
|
||||
if r.RuleBlocked {
|
||||
h.Offences.RuleBlocked++
|
||||
}
|
||||
}
|
||||
|
||||
// AddLookup gives client's history its AS number, AS name and country, as
|
||||
|
||||
@@ -12,6 +12,10 @@ import (
|
||||
// limit is the limit the tests set.
|
||||
const limit = 3
|
||||
|
||||
// tableSize is the most clients the tests' tables hold, the default of
|
||||
// SWWAF_MAX_TRACKED_CLIENTS.
|
||||
const tableSize = 20000
|
||||
|
||||
// whole is the percentage of each limit a client gets when nothing lowers
|
||||
// its limits.
|
||||
const whole = 100
|
||||
@@ -37,7 +41,7 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
|
||||
t.Run(tc.window, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(tc.limits)
|
||||
limiter := ratelimit.New(tc.limits, tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
quarter := tc.length / 4
|
||||
@@ -62,7 +66,7 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
|
||||
func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: limit})
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: limit}, tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
@@ -88,7 +92,8 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
|
||||
func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 5, BytesPerDay: math.MaxInt64})
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 5, BytesPerDay: math.MaxInt64},
|
||||
tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
@@ -119,7 +124,8 @@ func TestZeroPercentIsAZeroAllowanceAndALimitOffStaysOff(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Only the hour has limits: the minute's and the day's are off.
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit, BytesPerHour: 1000})
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit, BytesPerHour: 1000},
|
||||
tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
@@ -156,7 +162,7 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
|
||||
t.Run(tc.window, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(tc.limits)
|
||||
limiter := ratelimit.New(tc.limits, tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
|
||||
// 600 bytes are within the limit, 600 more over it.
|
||||
@@ -180,7 +186,8 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
|
||||
func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000})
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000},
|
||||
tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
other := netip.MustParsePrefix("203.0.113.10/32")
|
||||
start := midnight()
|
||||
@@ -204,7 +211,7 @@ func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
|
||||
func TestCountGivesTheBytesInEachWindow(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{})
|
||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
@@ -230,7 +237,7 @@ func TestCountGivesTheBytesInEachWindow(t *testing.T) {
|
||||
func TestResetSetsTheBytesBackToZero(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000})
|
||||
limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000}, tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
@@ -244,7 +251,7 @@ func TestResetSetsTheBytesBackToZero(t *testing.T) {
|
||||
func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{})
|
||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
@@ -267,7 +274,7 @@ func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
|
||||
func TestResetSetsTheCountsBackToZero(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerDay: limit})
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerDay: limit}, tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
@@ -289,7 +296,7 @@ func TestResetSetsTheCountsBackToZero(t *testing.T) {
|
||||
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit})
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
@@ -308,7 +315,8 @@ func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
|
||||
func TestRefusedRequestsCount(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit})
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit},
|
||||
tableSize)
|
||||
refused := netip.MustParsePrefix("203.0.113.9/32")
|
||||
within := netip.MustParsePrefix("203.0.113.10/32")
|
||||
start := midnight()
|
||||
@@ -341,7 +349,7 @@ func TestRefusedRequestsCount(t *testing.T) {
|
||||
func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit})
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
@@ -357,7 +365,7 @@ func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
|
||||
func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit})
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
@@ -380,12 +388,12 @@ func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
|
||||
wantCount(t, limiter, client, setBack, hour)
|
||||
}
|
||||
|
||||
func TestKeepsAtMost20000ClientsDroppingTheLeastRecentlySeen(t *testing.T) {
|
||||
func TestKeepsAtMostMaxClientsDroppingTheLeastRecentlySeen(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const maxClients = 20000
|
||||
const maxClients = 3
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1})
|
||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1}, maxClients)
|
||||
now := midnight()
|
||||
|
||||
clients := make([]netip.Prefix, maxClients+1)
|
||||
@@ -407,6 +415,11 @@ func TestKeepsAtMost20000ClientsDroppingTheLeastRecentlySeen(t *testing.T) {
|
||||
// One client more drops the least recently seen, the second, which
|
||||
// starts afresh, while the first is kept.
|
||||
wantCount(t, limiter, clients[maxClients], now, "")
|
||||
|
||||
if limiter.Len() != maxClients {
|
||||
t.Errorf("the table holds %d clients, want %d", limiter.Len(), maxClients)
|
||||
}
|
||||
|
||||
wantCount(t, limiter, clients[1], now, "")
|
||||
wantCount(t, limiter, clients[0], now, minute)
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@ func TestSnapshotListsTheClientsByAddress(t *testing.T) {
|
||||
|
||||
want := []string{"192.0.2.1/32", "203.0.113.9/32", "203.0.113.10/32", "2001:db8::/64"}
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{})
|
||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
||||
for _, i := range []int{2, 3, 0, 1} {
|
||||
limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole)
|
||||
}
|
||||
@@ -43,7 +43,7 @@ func TestLoadedCountsCarryOn(t *testing.T) {
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
before := ratelimit.New(ratelimit.Limits{PerHour: limit})
|
||||
before := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
|
||||
for range limit {
|
||||
wantCount(t, before, client, start, "")
|
||||
}
|
||||
@@ -51,7 +51,7 @@ func TestLoadedCountsCarryOn(t *testing.T) {
|
||||
// Loaded into a new limiter, as across a restart, the client has no
|
||||
// fresh allowance.
|
||||
later := start.Add(time.Minute)
|
||||
after := ratelimit.New(ratelimit.Limits{PerHour: limit})
|
||||
after := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
|
||||
after.Load(before.Snapshot(), later)
|
||||
wantCount(t, after, client, later, hour)
|
||||
}
|
||||
@@ -62,7 +62,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
|
||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||
start := midnight()
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{})
|
||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
||||
limiter.Count(client, start, whole)
|
||||
limiter.CountBytes(client, start, 5, whole)
|
||||
limiter.AddToHistory(client, start, ratelimit.Request{Forwarded: true})
|
||||
@@ -70,7 +70,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
|
||||
loaded := func(now time.Time) ratelimit.Client {
|
||||
t.Helper()
|
||||
|
||||
after := ratelimit.New(ratelimit.Limits{})
|
||||
after := ratelimit.New(ratelimit.Limits{}, tableSize)
|
||||
after.Load(limiter.Snapshot(), now)
|
||||
|
||||
return after.Snapshot()[0]
|
||||
@@ -102,7 +102,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
|
||||
func TestLoadDropsTheLeastRecentlySeenFirst(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const maxClients = 20000
|
||||
const maxClients = 3
|
||||
|
||||
// clients.json lists the clients by address. Here each was last seen
|
||||
// a second before the one listed before it, so the last listed is the
|
||||
@@ -116,7 +116,7 @@ func TestLoadDropsTheLeastRecentlySeenFirst(t *testing.T) {
|
||||
addr = addr.Next()
|
||||
}
|
||||
|
||||
limiter := ratelimit.New(ratelimit.Limits{})
|
||||
limiter := ratelimit.New(ratelimit.Limits{}, maxClients)
|
||||
limiter.Load(clients, midnight())
|
||||
|
||||
got := limiter.Snapshot()
|
||||
|
||||
@@ -38,12 +38,12 @@ var (
|
||||
)
|
||||
|
||||
// Score is what AbuseIPDB said about a client, as reputation.json holds
|
||||
// it: the client's address, its abuse confidence score, from 0 to 100,
|
||||
// and when AbuseIPDB answered.
|
||||
// it: the client, an IPv4 address or an IPv6 group, its abuse confidence
|
||||
// score, from 0 to 100, and when AbuseIPDB answered.
|
||||
type Score struct {
|
||||
Client netip.Addr `json:"client"`
|
||||
Score int64 `json:"score"`
|
||||
Fetched time.Time `json:"fetched"`
|
||||
Client netip.Prefix `json:"client"`
|
||||
Score int64 `json:"score"`
|
||||
Fetched time.Time `json:"fetched"`
|
||||
}
|
||||
|
||||
// Checks are what reputation.json keeps of the checks of clients with
|
||||
@@ -89,9 +89,9 @@ type AbuseIPDB struct {
|
||||
mu sync.Mutex
|
||||
// scores are by client. Each is added as it is fetched and never moved
|
||||
// up, so that the one fetched longest ago is the first dropped.
|
||||
scores *simplelru.LRU[netip.Addr, Score]
|
||||
scores *simplelru.LRU[netip.Prefix, Score]
|
||||
// checking are the clients whose check is under way.
|
||||
checking map[netip.Addr]bool
|
||||
checking map[netip.Prefix]bool
|
||||
// day is the day, in UTC, of the checks spent counts.
|
||||
day time.Time
|
||||
spent int
|
||||
@@ -105,7 +105,7 @@ type AbuseIPDB struct {
|
||||
|
||||
// NewAbuseIPDB returns an AbuseIPDB with no score yet, and no check spent.
|
||||
func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
|
||||
scores, err := simplelru.NewLRU[netip.Addr, Score](maxVerdicts, nil)
|
||||
scores, err := simplelru.NewLRU[netip.Prefix, Score](maxVerdicts, nil)
|
||||
if err != nil {
|
||||
panic(err) // NewLRU fails only for a size below one
|
||||
}
|
||||
@@ -114,27 +114,29 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
|
||||
params: params,
|
||||
httpClient: &http.Client{},
|
||||
scores: scores,
|
||||
checking: map[netip.Addr]bool{},
|
||||
checking: map[netip.Prefix]bool{},
|
||||
}
|
||||
}
|
||||
|
||||
// Hit returns AbuseIPDB's score of addr, a client's address, and whether
|
||||
// it is a hit: MinScore or more. A score is used until CacheTTL has passed
|
||||
// since it was fetched. A client without one is checked in the
|
||||
// background if offender, if it has committed an offence, unless its
|
||||
// check is under way, a check failed less than failureDelay ago, or the
|
||||
// day's checks have used up DailyBudget; Hit never waits for a check. The
|
||||
// check that uses the budget up is logged and raised as a source_failure
|
||||
// alert. ctx is the context of the client's request, and a check goes on
|
||||
// after the request ends.
|
||||
// Hit returns AbuseIPDB's score of client, an IPv4 address or an IPv6
|
||||
// group, and whether it is a hit: MinScore or more. A score is used until
|
||||
// CacheTTL has passed since it was fetched, whichever of the client's
|
||||
// addresses its request comes from. A client without one is checked in
|
||||
// the background, by addr, the address its request came from, if
|
||||
// offender, if it has committed an offence, unless its check is under
|
||||
// way, a check failed less than failureDelay ago, or the day's checks
|
||||
// have used up DailyBudget; Hit never waits for a check. The check that
|
||||
// uses the budget up is logged and raised as a source_failure alert. ctx
|
||||
// is the context of the client's request, and a check goes on after the
|
||||
// request ends.
|
||||
func (a *AbuseIPDB) Hit(
|
||||
ctx context.Context, addr netip.Addr, offender bool,
|
||||
ctx context.Context, client netip.Prefix, addr netip.Addr, offender bool,
|
||||
) (int64, bool) {
|
||||
a.mu.Lock()
|
||||
|
||||
now := a.params.Now()
|
||||
|
||||
kept, found := a.scores.Peek(addr)
|
||||
kept, found := a.scores.Peek(client)
|
||||
if found && now.Sub(kept.Fetched) < a.params.CacheTTL {
|
||||
a.mu.Unlock()
|
||||
|
||||
@@ -145,14 +147,14 @@ func (a *AbuseIPDB) Hit(
|
||||
a.day, a.spent = today, 0
|
||||
}
|
||||
|
||||
check := offender && !a.checking[addr] && !now.Before(a.retryAt) &&
|
||||
check := offender && !a.checking[client] && !now.Before(a.retryAt) &&
|
||||
a.spent < a.params.DailyBudget
|
||||
if check {
|
||||
a.checking[addr] = true
|
||||
a.checking[client] = true
|
||||
a.checks++
|
||||
a.spent++
|
||||
|
||||
go a.check(context.WithoutCancel(ctx), addr)
|
||||
go a.check(context.WithoutCancel(ctx), client, addr)
|
||||
}
|
||||
|
||||
usedUp := check && a.spent == a.params.DailyBudget
|
||||
@@ -239,20 +241,20 @@ func (a *AbuseIPDB) Load(checks Checks) {
|
||||
}
|
||||
}
|
||||
|
||||
// check checks addr with AbuseIPDB, keeps the score, and notes the check
|
||||
// as no longer under way. A check that fails gives no score: it is
|
||||
// counted, logged and raised as a source_failure alert, and no client is
|
||||
// checked for failureDelay.
|
||||
func (a *AbuseIPDB) check(ctx context.Context, addr netip.Addr) {
|
||||
// check checks client with AbuseIPDB by addr, one of its addresses, keeps
|
||||
// the score as client's, and notes the check as no longer under way. A
|
||||
// check that fails gives no score: it is counted, logged and raised as a
|
||||
// source_failure alert, and no client is checked for failureDelay.
|
||||
func (a *AbuseIPDB) check(ctx context.Context, client netip.Prefix, addr netip.Addr) {
|
||||
score, err := a.ask(ctx, addr)
|
||||
now := a.params.Now()
|
||||
|
||||
a.mu.Lock()
|
||||
|
||||
delete(a.checking, addr)
|
||||
delete(a.checking, client)
|
||||
|
||||
if err == nil {
|
||||
a.scores.Add(addr, Score{Client: addr, Score: score, Fetched: now})
|
||||
a.scores.Add(client, Score{Client: client, Score: score, Fetched: now})
|
||||
} else {
|
||||
a.failures++
|
||||
a.retryAt = now.Add(failureDelay)
|
||||
|
||||
@@ -59,6 +59,38 @@ func TestOnlyAnOffenderWithoutAScoreIsChecked(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
func TestIPv6ClientIsCheckedOnceAndItsScoreUsedForEachOfItsAddresses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
// 15 addresses of 2001:db8:1:2::/64, one client, each in a part of
|
||||
// it of its own.
|
||||
var addresses []string
|
||||
for i := 1; i < 16; i++ {
|
||||
addresses = append(addresses, fmt.Sprintf("2001:db8:1:2:%x::9", i<<12))
|
||||
}
|
||||
|
||||
abuseIPDB := &abuseIPDBStandIn{scores: map[string]int64{addresses[0]: 100}}
|
||||
checker := newAbuseIPDB(abuseIPDB, abuseIPDBParams())
|
||||
|
||||
// A request from each has the client checked once, by the first.
|
||||
for _, address := range addresses {
|
||||
hitFrom(t, checker, address, true)
|
||||
}
|
||||
|
||||
synctest.Wait()
|
||||
wantChecked(t, abuseIPDB, addresses[0])
|
||||
|
||||
// Its score is the whole client's.
|
||||
for _, address := range addresses {
|
||||
wantScore(t, checker, address, true, 100, true)
|
||||
}
|
||||
|
||||
synctest.Wait()
|
||||
wantChecked(t, abuseIPDB, addresses[0])
|
||||
})
|
||||
}
|
||||
|
||||
func TestScoreAtOrOverTheMinimumIsAHit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -69,7 +101,7 @@ func TestScoreAtOrOverTheMinimumIsAHit(t *testing.T) {
|
||||
checker := newAbuseIPDB(&abuseIPDBStandIn{scores: scores}, p)
|
||||
|
||||
for client := range scores {
|
||||
checker.Hit(t.Context(), netip.MustParseAddr(client), true)
|
||||
hitFrom(t, checker, client, true)
|
||||
}
|
||||
|
||||
synctest.Wait()
|
||||
@@ -87,7 +119,7 @@ func TestScoreUsedUntilTheCacheTTLHasPassedSinceItWasFetched(t *testing.T) {
|
||||
abuseIPDB := &abuseIPDBStandIn{scores: map[string]int64{suspect: 100}}
|
||||
checker := newAbuseIPDB(abuseIPDB, abuseIPDBParams())
|
||||
|
||||
checker.Hit(t.Context(), netip.MustParseAddr(suspect), true)
|
||||
hitFrom(t, checker, suspect, true)
|
||||
synctest.Wait()
|
||||
|
||||
// AbuseIPDB gives another score from now on, but the one kept is
|
||||
@@ -130,7 +162,7 @@ func TestDailyBudgetKeptAcrossARestartAndWholeAgainAsTheDayEnds(t *testing.T) {
|
||||
|
||||
clients := []string{suspect, "192.0.2.2", "192.0.2.3", unchecked}
|
||||
for _, client := range clients {
|
||||
checker.Hit(t.Context(), netip.MustParseAddr(client), true)
|
||||
hitFrom(t, checker, client, true)
|
||||
}
|
||||
|
||||
synctest.Wait()
|
||||
@@ -352,10 +384,10 @@ func TestMetricsCountTheChecksTheFailuresAndTheBudgetLeft(t *testing.T) {
|
||||
m.AddAbuseIPDB(checker)
|
||||
|
||||
// One check that AbuseIPDB answers, and one that fails.
|
||||
checker.Hit(t.Context(), netip.MustParseAddr(suspect), true)
|
||||
hitFrom(t, checker, suspect, true)
|
||||
synctest.Wait()
|
||||
abuseIPDB.answerWith(http.StatusInternalServerError, "")
|
||||
checker.Hit(t.Context(), netip.MustParseAddr(other), true)
|
||||
hitFrom(t, checker, other, true)
|
||||
synctest.Wait()
|
||||
|
||||
scraped := scrapeMetrics(t, m)
|
||||
@@ -382,13 +414,15 @@ func TestScoreFetchedATTLAgoIsNeitherUsedNorKept(t *testing.T) {
|
||||
p.Now = func() time.Time { return now }
|
||||
checker := reputation.NewAbuseIPDB(p)
|
||||
|
||||
// The last score still in use, and one fetched a TTL ago.
|
||||
// The last score still in use, and one, of other's /64, fetched a TTL
|
||||
// ago.
|
||||
inUse := reputation.Score{
|
||||
Client: netip.MustParseAddr(suspect), Score: 100,
|
||||
Client: netip.MustParsePrefix(suspect + "/32"), Score: 100,
|
||||
Fetched: now.Add(-cacheTTL + time.Nanosecond),
|
||||
}
|
||||
stale := reputation.Score{
|
||||
Client: netip.MustParseAddr(other), Score: 100, Fetched: now.Add(-cacheTTL),
|
||||
Client: netip.MustParsePrefix("2001:db8::/64"), Score: 100,
|
||||
Fetched: now.Add(-cacheTTL),
|
||||
}
|
||||
|
||||
checker.Load(reputation.Checks{Scores: []reputation.Score{stale, inUse}})
|
||||
@@ -417,12 +451,13 @@ func TestAtMost100000ScoresKeptTheOneFetchedLongestAgoDroppedFirst(t *testing.T)
|
||||
|
||||
scores := make([]reputation.Score, 0, count)
|
||||
|
||||
client := netip.MustParseAddr("198.18.0.0")
|
||||
addr := netip.MustParseAddr("198.18.0.0")
|
||||
for i := range count {
|
||||
scores = append(scores, reputation.Score{
|
||||
Client: client, Fetched: now.Add(-time.Duration(i) * time.Millisecond),
|
||||
Client: netip.PrefixFrom(addr, 32),
|
||||
Fetched: now.Add(-time.Duration(i) * time.Millisecond),
|
||||
})
|
||||
client = client.Next()
|
||||
addr = addr.Next()
|
||||
}
|
||||
|
||||
checker.Load(reputation.Checks{Scores: scores})
|
||||
@@ -537,13 +572,31 @@ func wantScore(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
gotScore, gotHit := checker.Hit(t.Context(), netip.MustParseAddr(client), offender)
|
||||
gotScore, gotHit := hitFrom(t, checker, client, offender)
|
||||
if gotScore != score || gotHit != hit {
|
||||
t.Errorf("%s has the score %d, a hit %t, want %d, %t", client, gotScore, gotHit,
|
||||
score, hit)
|
||||
}
|
||||
}
|
||||
|
||||
// hitFrom is checker's Hit for a request from address, offender or not.
|
||||
// Its client is address for an IPv4 address, and its /64 for an IPv6 one,
|
||||
// as smallwebwaf counts clients.
|
||||
func hitFrom(
|
||||
t *testing.T, checker *reputation.AbuseIPDB, address string, offender bool,
|
||||
) (int64, bool) {
|
||||
t.Helper()
|
||||
|
||||
addr := netip.MustParseAddr(address)
|
||||
|
||||
client := netip.PrefixFrom(addr, addr.BitLen())
|
||||
if addr.Is6() {
|
||||
client = netip.PrefixFrom(addr, 64).Masked()
|
||||
}
|
||||
|
||||
return checker.Hit(t.Context(), client, addr, offender)
|
||||
}
|
||||
|
||||
// wantChecked checks the clients the stand-in was asked about, in any
|
||||
// order.
|
||||
func wantChecked(t *testing.T, abuseIPDB *abuseIPDBStandIn, want ...string) {
|
||||
|
||||
@@ -193,8 +193,11 @@ func Milliseconds(d time.Duration) float64 {
|
||||
// NewProcessLogger returns the logger for the process's own messages:
|
||||
// JSON lines on w, marked "type":"process", with the time in the same form
|
||||
// as a request line's, and instanceName, SWWAF_INSTANCE_NAME, as instance.
|
||||
func NewProcessLogger(w io.Writer, instanceName string) *slog.Logger {
|
||||
// It writes only the messages at level, SWWAF_LOG_LEVEL, or more severe;
|
||||
// the request lines Write writes are never held back.
|
||||
func NewProcessLogger(w io.Writer, instanceName string, level slog.Level) *slog.Logger {
|
||||
handler := slog.NewJSONHandler(w, &slog.HandlerOptions{
|
||||
Level: level,
|
||||
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
|
||||
if attr.Key == slog.TimeKey && len(groups) == 0 {
|
||||
return slog.String(slog.TimeKey, FormatTime(attr.Value.Time()))
|
||||
|
||||
@@ -3,6 +3,8 @@ package requestlog_test
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -70,7 +72,8 @@ func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
requestlog.NewProcessLogger(&out, "fsn1app1/gitea").Info("starting", "version", "v1")
|
||||
requestlog.NewProcessLogger(&out, "fsn1app1/gitea", slog.LevelInfo).Info("starting",
|
||||
"version", "v1")
|
||||
|
||||
var fields map[string]any
|
||||
|
||||
@@ -94,3 +97,47 @@ func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
|
||||
t.Errorf("process line time %q, want now in UTC with milliseconds", timeText)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
levels := []slog.Level{
|
||||
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
|
||||
}
|
||||
|
||||
for i, level := range levels {
|
||||
t.Run(level.String(), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
processLog := requestlog.NewProcessLogger(&out, "fsn1app1/gitea", level)
|
||||
for _, at := range levels {
|
||||
processLog.Log(t.Context(), at, "message")
|
||||
}
|
||||
|
||||
var got, want []string
|
||||
|
||||
for line := range strings.Lines(out.String()) {
|
||||
var fields struct {
|
||||
Level string `json:"level"`
|
||||
}
|
||||
|
||||
err := json.Unmarshal([]byte(line), &fields)
|
||||
if err != nil {
|
||||
t.Fatalf("decode %q: %v", line, err)
|
||||
}
|
||||
|
||||
got = append(got, fields.Level)
|
||||
}
|
||||
|
||||
for _, written := range levels[i:] {
|
||||
want = append(want, written.String())
|
||||
}
|
||||
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("lines at %v, want %v", got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -70,8 +70,10 @@ func Main(version string) int {
|
||||
// state files, then serves requests until ctx is done. It returns the
|
||||
// process's exit status, 1 when smallwebwaf cannot start.
|
||||
func Run(ctx context.Context, params Params) int {
|
||||
// Until the settings are read, the one message is an invalid setting's
|
||||
// error, which every SWWAF_LOG_LEVEL lets through.
|
||||
processLog := requestlog.NewProcessLogger(params.Stdout,
|
||||
config.InstanceName(params.LookupEnv))
|
||||
config.InstanceName(params.LookupEnv), slog.LevelError)
|
||||
|
||||
cfg, err := config.FromEnvironment(params.LookupEnv)
|
||||
if err != nil {
|
||||
@@ -89,8 +91,11 @@ func Run(ctx context.Context, params Params) int {
|
||||
if cfg.LogRemoteURL != nil {
|
||||
remote = newRemoteLogSender(cfg)
|
||||
stdout = io.MultiWriter(params.Stdout, remote)
|
||||
processLog = requestlog.NewProcessLogger(stdout, cfg.InstanceName)
|
||||
}
|
||||
|
||||
processLog = requestlog.NewProcessLogger(stdout, cfg.InstanceName, cfg.LogLevel)
|
||||
|
||||
if remote != nil {
|
||||
stopSending := startSending(ctx, remote, processLog)
|
||||
defer stopSending()
|
||||
}
|
||||
|
||||
@@ -249,6 +249,54 @@ func TestServesUntilToldToStop(t *testing.T) {
|
||||
out.line(t, "msg", "stopped")
|
||||
}
|
||||
|
||||
func TestLogLevelHoldsBackTheLessSevereProcessLines(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// A list that cannot be fetched has a warning written once smallwebwaf
|
||||
// serves, after its starting line.
|
||||
lists := httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusServiceUnavailable)
|
||||
}))
|
||||
t.Cleanup(lists.Close)
|
||||
|
||||
ctx, stop := context.WithCancel(t.Context())
|
||||
out := &output{}
|
||||
exited := make(chan int, 1)
|
||||
|
||||
go func() {
|
||||
exited <- run(ctx, map[string]string{
|
||||
listenAddr: localhost + ":0",
|
||||
stateDir: t.TempDir(),
|
||||
rulesDir: t.TempDir(),
|
||||
"SWWAF_BLOCKLIST_URLS": lists.URL + "/tor.txt",
|
||||
"SWWAF_LOG_LEVEL": "warn",
|
||||
}, out)
|
||||
}()
|
||||
|
||||
out.line(t, "msg", "fetching a list failed")
|
||||
stop()
|
||||
|
||||
select {
|
||||
case status := <-exited:
|
||||
if status != 0 {
|
||||
t.Fatalf("exit status %d, want 0; output:\n%s", status, out.text())
|
||||
}
|
||||
case <-time.After(waitLimit):
|
||||
t.Fatal("still running after being told to stop")
|
||||
}
|
||||
|
||||
// Not one of the info lines from the start to the stop.
|
||||
for line := range strings.Lines(out.text()) {
|
||||
var fields map[string]any
|
||||
|
||||
err := json.Unmarshal([]byte(line), &fields)
|
||||
if err != nil || fields["level"] == "INFO" {
|
||||
t.Errorf("line %q (%v), want none at info", line, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestEveryLogLineAndMetricCarriesTheInstanceName(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -252,12 +252,12 @@ const filledReputationJSON = `{
|
||||
"spent": 3,
|
||||
"scores": [
|
||||
{
|
||||
"client": "203.0.113.9",
|
||||
"client": "203.0.113.9/32",
|
||||
"score": 100,
|
||||
"fetched": "2026-10-05T23:00:00Z"
|
||||
},
|
||||
{
|
||||
"client": "2001:db8::1",
|
||||
"client": "2001:db8::/64",
|
||||
"score": 0,
|
||||
"fetched": "2026-10-05T22:00:00Z"
|
||||
}
|
||||
@@ -710,7 +710,7 @@ func TestReputationJSONScoreWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
|
||||
// score and fetched make a score.
|
||||
const (
|
||||
scores = `{"version": 1, "abuseipdb": {"scores": [`
|
||||
client = `"client": "198.51.100.7", `
|
||||
client = `"client": "198.51.100.7/32", `
|
||||
score = `"score": 0, `
|
||||
fetched = `"fetched": "2026-10-06T00:00:00Z"`
|
||||
ends = `}]}}`
|
||||
@@ -1250,7 +1250,7 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
|
||||
`"lines": ["198.51.100.7"]}], "verdicts": [{"zone": "`+dnsblZone+`", `+
|
||||
`"client": "198.51.100.7", "listed": true, "fetched": "2026-10-06T00:00:00Z"}], `+
|
||||
`"abuseipdb": {"day": "2026-10-06T00:00:00Z", "spent": 9, "scores": [`+
|
||||
`{"client": "198.51.100.7", "score": 80, "fetched": "2026-10-06T00:00:00Z"}]}}`)
|
||||
`{"client": "198.51.100.7/32", "score": 80, "fetched": "2026-10-06T00:00:00Z"}]}}`)
|
||||
wantTakenIn(t, lines, dir, reputationJSON)
|
||||
|
||||
listedBy := params.Lists.ListedBy(client.Addr())
|
||||
@@ -1265,7 +1265,7 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
|
||||
|
||||
checks := reputation.Checks{
|
||||
Day: midnight(), Spent: 9,
|
||||
Scores: []reputation.Score{{Client: client.Addr(), Score: 80, Fetched: midnight()}},
|
||||
Scores: []reputation.Score{{Client: client, Score: 80, Fetched: midnight()}},
|
||||
}
|
||||
if got := params.AbuseIPDB.Snapshot(); !reflect.DeepEqual(got, checks) {
|
||||
t.Errorf("%s taken in as\n%+v\nwant\n%+v", reputationJSON, got, checks)
|
||||
@@ -1679,7 +1679,7 @@ func newParams(dir string) state.Params {
|
||||
AttackBanDuration: 7 * 24 * time.Hour,
|
||||
MaxBans: 5000,
|
||||
}),
|
||||
Limiter: ratelimit.New(ratelimit.Limits{}),
|
||||
Limiter: ratelimit.New(ratelimit.Limits{}, 20000),
|
||||
GeoJS: lookup.New(lookup.Params{
|
||||
Now: midnight, ProcessLog: discard, Metrics: m,
|
||||
}),
|
||||
@@ -1778,8 +1778,8 @@ func fill(params state.Params) {
|
||||
{Zone: dnsblZone, Client: client.Addr(), Listed: true, Fetched: now.Add(-time.Hour)},
|
||||
})
|
||||
params.AbuseIPDB.Load(reputation.Checks{Day: now, Spent: 3, Scores: []reputation.Score{
|
||||
{Client: netip.MustParseAddr("2001:db8::1"), Fetched: now.Add(-2 * time.Hour)},
|
||||
{Client: client.Addr(), Score: 100, Fetched: now.Add(-time.Hour)},
|
||||
{Client: netip.MustParsePrefix("2001:db8::/64"), Fetched: now.Add(-2 * time.Hour)},
|
||||
{Client: client, Score: 100, Fetched: now.Add(-time.Hour)},
|
||||
}})
|
||||
|
||||
// An alert waiting, a repeat of it the cooldown holds back, another
|
||||
|
||||
Reference in New Issue
Block a user