Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c3c546c65b | ||
|
|
f76308add3 | ||
|
|
937d956d3a |
No files matched your search
@@ -510,7 +510,8 @@ request reaches the app.
|
|||||||
|
|
||||||
`waf` takes the checks:
|
`waf` takes the checks:
|
||||||
|
|
||||||
3. The tag lists.
|
3. The header limits, then the tag lists. A client in `SWW_ALLOW_NETS` skips the
|
||||||
|
header limits.
|
||||||
4. The static lists, `SWW_ALLOW_NETS` and `SWW_DENY_NETS`.
|
4. The static lists, `SWW_ALLOW_NETS` and `SWW_DENY_NETS`.
|
||||||
5. The bans.
|
5. The bans.
|
||||||
6. The lookup of the client's AS number and country.
|
6. The lookup of the client's AS number and country.
|
||||||
@@ -522,7 +523,7 @@ request reaches the app.
|
|||||||
11. The rate limits.
|
11. The rate limits.
|
||||||
12. The trap paths.
|
12. The trap paths.
|
||||||
13. The rule files.
|
13. The rule files.
|
||||||
14. The header limits, and then the Core Rule Set.
|
14. The Core Rule Set.
|
||||||
15. The tag refusals again, so that `exploitsignature` can refuse too.
|
15. The tag refusals again, so that `exploitsignature` can refuse too.
|
||||||
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||||
prefix.
|
prefix.
|
||||||
@@ -538,7 +539,7 @@ request reaches the app.
|
|||||||
A refused visitor is answered with `SWW_BAN_RESPONSE`. A `block` rule or a Core
|
A refused visitor is answered with `SWW_BAN_RESPONSE`. A `block` rule or a Core
|
||||||
Rule Set match answers `403` and bans nobody, since the Core Rule Set has false
|
Rule Set match answers `403` and bans nobody, since the Core Rule Set has false
|
||||||
positives; such refusals count toward `SWW_ERROR_BURST_THRESHOLD` instead. A
|
positives; such refusals count toward `SWW_ERROR_BURST_THRESHOLD` instead. A
|
||||||
request over the header limits of step 14 is answered `431`, and bans nobody
|
request over the header limits of step 3 is answered `431`, and bans nobody
|
||||||
either. `rproxy` itself answers `405` to a method it does not pass on, and the
|
either. `rproxy` itself answers `405` to a method it does not pass on, and the
|
||||||
status for a size or time limit, such as `413` for a body too large.
|
status for a size or time limit, such as `413` for a body too large.
|
||||||
|
|
||||||
@@ -621,13 +622,14 @@ client's limits by tag. See "Request tags" in [`SPEC.md`](SPEC.md).
|
|||||||
- The state files, in `SWW_STATE_DIR`, hold what cannot be fetched again:
|
- The state files, in `SWW_STATE_DIR`, hold what cannot be fetched again:
|
||||||
- `bans.json`;
|
- `bans.json`;
|
||||||
- `clients.json`;
|
- `clients.json`;
|
||||||
- `abuseipdb-checks.json`;
|
- `abuseipdb-checks-spent.json`;
|
||||||
- `alerts.json`.
|
- `alerts.json`.
|
||||||
- The cache files, in `SWW_CACHE_DIR`, hold answers from outside services:
|
- The cache files, in `SWW_CACHE_DIR`, hold answers from outside services:
|
||||||
- `lookups.json`, from GeoJS;
|
- `lookups.json`, from GeoJS;
|
||||||
- `lists.json`, the lists fetched by URL;
|
- `lists.json`, the lists fetched by URL;
|
||||||
- `dnsbl.json`, from the DNS blocklists;
|
- `dnsbl.json`, from the DNS blocklists;
|
||||||
- `abuseipdb-scores.json`, from AbuseIPDB.
|
- `abuseipdb-scores.json`, from AbuseIPDB;
|
||||||
|
- `routewatch.json`, from routewatch.
|
||||||
|
|
||||||
Back up the state directory. The cache directory can be left out of backups: a
|
Back up the state directory. The cache directory can be left out of backups: a
|
||||||
lost cache file starts empty and fills again.
|
lost cache file starts empty and fills again.
|
||||||
|
|||||||
@@ -149,10 +149,12 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in
|
|||||||
- An IPv6 client is its group: the netblock of `SWW_IPV6_GROUP_PREFIX`
|
- An IPv6 client is its group: the netblock of `SWW_IPV6_GROUP_PREFIX`
|
||||||
around its address, a /64 by default, since one abuser usually controls a
|
around its address, a /64 by default, since one abuser usually controls a
|
||||||
whole /64. A client is therefore one IPv4 address or one IPv6 group.
|
whole /64. A client is therefore one IPv4 address or one IPv6 group.
|
||||||
3. The tag lists (`tag_lists`; see "Request tags"). Each list that holds the
|
3. The header limits (`header_limits`; see "Answering in time"). They come first
|
||||||
client adds its tag. Next, `routewatch_tags` adds the client's `asn=` and
|
after `SWW_ALLOW_NETS`, since they only count header fields and cookies.
|
||||||
|
Next, the tag lists (`tag_lists`; see "Request tags"). Each list that holds
|
||||||
|
the client adds its tag. Next, `routewatch_tags` adds the client's `asn=` and
|
||||||
`prefix=` tags while `SWW_ROUTEWATCH_URL` is set. A client in
|
`prefix=` tags while `SWW_ROUTEWATCH_URL` is set. A client in
|
||||||
`SWW_ALLOW_NETS` gets no tag.
|
`SWW_ALLOW_NETS` skips the header limits and gets no tag.
|
||||||
4. The static lists. A client in `SWW_ALLOW_NETS` skips steps 5 to 15, the byte
|
4. The static lists. A client in `SWW_ALLOW_NETS` skips steps 5 to 15, the byte
|
||||||
limits and the error burst. A client in `SWW_DENY_NETS` is refused
|
limits and the error burst. A client in `SWW_DENY_NETS` is refused
|
||||||
(`deny_nets`); a client in both lists is let through.
|
(`deny_nets`); a client in both lists is let through.
|
||||||
@@ -185,10 +187,9 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in
|
|||||||
refused and bans the client.
|
refused and bans the client.
|
||||||
13. The rule files (`rule_files`; see "Rule files"). Matching stops at the first
|
13. The rule files (`rule_files`; see "Rule files"). Matching stops at the first
|
||||||
rule that refuses.
|
rule that refuses.
|
||||||
14. The header limits (`header_limits`; see "Answering in time"), and then the
|
14. The Core Rule Set (`core_rule_set`; see "Attack detection"). A match is
|
||||||
Core Rule Set (`core_rule_set`; see "Attack detection"). A Core Rule Set
|
refused with `403` while `SWW_WAF_MODE` is `block`, and only logged and
|
||||||
match is refused with `403` while `SWW_WAF_MODE` is `block`, and only logged
|
alerted while it is `detect`.
|
||||||
and alerted while it is `detect`.
|
|
||||||
15. The tag refusals again (`tag_deny`), so that `exploitsignature` can refuse
|
15. The tag refusals again (`tag_deny`), so that `exploitsignature` can refuse
|
||||||
too.
|
too.
|
||||||
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||||
@@ -241,11 +242,16 @@ apply.
|
|||||||
## Bans
|
## Bans
|
||||||
|
|
||||||
Until a ban ends, every request from its netblock is refused with
|
Until a ban ends, every request from its netblock is refused with
|
||||||
`SWW_BAN_RESPONSE`. The netblock is the client: its IPv4 address, widened to
|
`SWW_BAN_RESPONSE`, unless it is over the header limits. The netblock is the
|
||||||
`SWW_BAN_SCOPE_V4_PREFIX`, or its IPv6 group. At the defaults a ban covers one
|
client: its IPv4 address, widened to `SWW_BAN_SCOPE_V4_PREFIX`, or its IPv6
|
||||||
address or one /64, so a permanent ban does not reach neighbours who did
|
group. At the defaults a ban covers one address or one /64, so a permanent ban
|
||||||
nothing. Requests refused under a ban are not counted against limits. They are
|
does not reach neighbours who did nothing. Requests refused under a ban are not
|
||||||
counted in the client's history and in the ban's notes.
|
counted against limits. They are counted in the client's history and in the
|
||||||
|
ban's notes.
|
||||||
|
|
||||||
|
A request over the header limits is refused with `431` before the bans are
|
||||||
|
checked (see "Answering in time"). It makes no ban, makes no ban permanent and
|
||||||
|
is not counted in a ban's notes.
|
||||||
|
|
||||||
An offence is a request `waf` holds against the client. It is one of:
|
An offence is a request `waf` holds against the client. It is one of:
|
||||||
|
|
||||||
@@ -577,7 +583,7 @@ Each tag and where it comes from:
|
|||||||
- They are added before `deny_nets`, so the log line of a request it refused
|
- They are added before `deny_nets`, so the log line of a request it refused
|
||||||
has them.
|
has them.
|
||||||
- `waf` asks routewatch in the background and never waits for it. An answer
|
- `waf` asks routewatch in the background and never waits for it. An answer
|
||||||
is kept for 1 hour, in memory only.
|
is kept for 1 hour, in memory and in `routewatch.json`.
|
||||||
- After a query fails, routewatch is not asked for 1 minute.
|
- After a query fails, routewatch is not asked for 1 minute.
|
||||||
- A request without these tags gives the reason in the log field
|
- A request without these tags gives the reason in the log field
|
||||||
`routewatch_tags_missing` (see
|
`routewatch_tags_missing` (see
|
||||||
@@ -1653,11 +1659,11 @@ together.
|
|||||||
client's later requests. For an IPv6 client, AbuseIPDB is asked about the
|
client's later requests. For an IPv6 client, AbuseIPDB is asked about the
|
||||||
address of the request that set off the check, and the score is used for the
|
address of the request that set off the check, and the score is used for the
|
||||||
whole group. A score at or over `SWW_ABUSEIPDB_MIN_SCORE` is a hit. The budget
|
whole group. A score at or over `SWW_ABUSEIPDB_MIN_SCORE` is a hit. The budget
|
||||||
is counted in UTC days and kept in `abuseipdb-checks.json`, so a restart does
|
is counted in UTC days and kept in `abuseipdb-checks-spent.json`, so a restart
|
||||||
not make it whole again. The check that spends the last of it is logged and
|
does not make it whole again. The check that spends the last of it is logged
|
||||||
alerted, and no client is checked until the day ends. The key is sent in the
|
and alerted, and no client is checked until the day ends. The key is sent in
|
||||||
`Key` header and nowhere else, and is never logged. A check fails, after which
|
the `Key` header and nowhere else, and is never logged. A check fails, after
|
||||||
no client is checked for a minute, when:
|
which no client is checked for a minute, when:
|
||||||
- AbuseIPDB answers other than `200`;
|
- AbuseIPDB answers other than `200`;
|
||||||
- the answer gives no score;
|
- the answer gives no score;
|
||||||
- no answer comes within `SWW_REPUTATION_TIMEOUT`.
|
- no answer comes within `SWW_REPUTATION_TIMEOUT`.
|
||||||
@@ -1776,22 +1782,37 @@ them at any time, and the running `waf` takes the edit in. `docs/state-files.md`
|
|||||||
lists the files, what each holds, and their sizes.
|
lists the files, what each holds, and their sizes.
|
||||||
|
|
||||||
- The state files, in `SWW_STATE_DIR`, hold what cannot be fetched again:
|
- The state files, in `SWW_STATE_DIR`, hold what cannot be fetched again:
|
||||||
`bans.json`, `clients.json`, `abuseipdb-checks.json` and `alerts.json`.
|
- `bans.json`;
|
||||||
|
- `clients.json`;
|
||||||
|
- `abuseipdb-checks-spent.json`;
|
||||||
|
- `alerts.json`.
|
||||||
- The cache files, in `SWW_CACHE_DIR`, hold the answers of outside services:
|
- The cache files, in `SWW_CACHE_DIR`, hold the answers of outside services:
|
||||||
`lookups.json`, `lists.json`, `dnsbl.json` and `abuseipdb-scores.json`. A lost
|
- `lookups.json`;
|
||||||
cache file costs only questions asked again, so the cache directory can be
|
- `lists.json`;
|
||||||
left out of backups.
|
- `dnsbl.json`;
|
||||||
|
- `abuseipdb-scores.json`;
|
||||||
|
- `routewatch.json`.
|
||||||
|
- A lost cache file costs only questions asked again, so the cache directory can
|
||||||
|
be left out of backups.
|
||||||
- Each file holds one kind of state, as JSON with:
|
- Each file holds one kind of state, as JSON with:
|
||||||
- a top-level `version` number;
|
- a top-level `version` number;
|
||||||
- entries sorted by client address;
|
- entries sorted by client address;
|
||||||
- times in RFC 3339 UTC;
|
- times in RFC 3339 UTC;
|
||||||
- days as `YYYY-MM-DD`;
|
- days as `YYYY-MM-DD`;
|
||||||
- durations and sizes as plain numbers, with the unit in the field name.
|
- durations and sizes as plain numbers, with the unit in the field name.
|
||||||
- `bans.json`, `abuseipdb-checks.json`, `alerts.json` and `lists.json` are
|
- These files are indented, so that a person can open `bans.json` in an editor,
|
||||||
indented, so that a person can open `bans.json` in an editor, find an address,
|
find an address, and remove or add an entry:
|
||||||
and remove or add an entry. `clients.json`, `lookups.json`, `dnsbl.json` and
|
- `bans.json`;
|
||||||
`abuseipdb-scores.json` put each entry on one line, which halves their size
|
- `abuseipdb-checks-spent.json`;
|
||||||
and lets `grep` show everything about one client.
|
- `alerts.json`;
|
||||||
|
- `lists.json`.
|
||||||
|
- The others put each entry on one line, which halves their size and lets `grep`
|
||||||
|
show everything about one client:
|
||||||
|
- `clients.json`;
|
||||||
|
- `lookups.json`;
|
||||||
|
- `dnsbl.json`;
|
||||||
|
- `abuseipdb-scores.json`;
|
||||||
|
- `routewatch.json`.
|
||||||
- Clients that were never banned are kept too, so every client's history
|
- Clients that were never banned are kept too, so every client's history
|
||||||
survives a restart.
|
survives a restart.
|
||||||
- Each file has a bound, so it can be rewritten whole. Writing one takes well
|
- Each file has a bound, so it can be rewritten whole. Writing one takes well
|
||||||
@@ -2037,7 +2058,6 @@ waits on these, which run in the background:
|
|||||||
|
|
||||||
In `enforce` mode, a question does not wait for GeoJS when it is a request:
|
In `enforce` mode, a question does not wait for GeoJS when it is a request:
|
||||||
|
|
||||||
- over the header limits;
|
|
||||||
- from a client on a fetched blocklist, while `SWW_BLOCKLIST_ACTION` is `deny`;
|
- from a client on a fetched blocklist, while `SWW_BLOCKLIST_ACTION` is `deny`;
|
||||||
- from a client on the CrowdSec decision list;
|
- from a client on the CrowdSec decision list;
|
||||||
- that would take its client over a rate limit at 100% of it, since a percentage
|
- that would take its client over a rate limit at 100% of it, since a percentage
|
||||||
@@ -2049,8 +2069,8 @@ number and country stay unknown until GeoJS has answered. So while
|
|||||||
`SWW_UNKNOWN_LIMIT_PERCENT` does not apply to it until then, so not waiting
|
`SWW_UNKNOWN_LIMIT_PERCENT` does not apply to it until then, so not waiting
|
||||||
never lowers its limit.
|
never lowers its limit.
|
||||||
|
|
||||||
The static lists and bans come before the lookup, so they never wait. These can
|
The header limits, the static lists and bans come before the lookup, so they
|
||||||
still wait for GeoJS:
|
never wait. These can still wait for GeoJS:
|
||||||
|
|
||||||
- a country list, which needs the answer;
|
- a country list, which needs the answer;
|
||||||
- a rate limit a percentage has lowered, which needs the answer;
|
- a rate limit a percentage has lowered, which needs the answer;
|
||||||
@@ -2069,13 +2089,17 @@ either one:
|
|||||||
|
|
||||||
Real clients never reach these limits. The answer is `deny` with `431`, and the
|
Real clients never reach these limits. The answer is `deny` with `431`, and the
|
||||||
log line's action is `request_header_too_large`. The refusal makes no ban and is
|
log line's action is `request_header_too_large`. The refusal makes no ban and is
|
||||||
no offence. The rate limits count the request as usual, so a client that keeps
|
no offence.
|
||||||
sending such requests breaks a limit.
|
|
||||||
|
The header limits are the first check after `SWW_ALLOW_NETS`. A request over
|
||||||
|
them is refused before any later check runs. So it causes no lookup, no query to
|
||||||
|
an outside source and no rule match, and the rate limits do not count it. A
|
||||||
|
client under a ban or in `SWW_DENY_NETS` also gets `431` for such a request.
|
||||||
|
|
||||||
The limits apply on every path and in every `SWW_WAF_MODE`. A client in
|
The limits apply on every path and in every `SWW_WAF_MODE`. A client in
|
||||||
`SWW_ALLOW_NETS` skips them, as it skips the Core Rule Set. In `observe` mode
|
`SWW_ALLOW_NETS` skips them, as it skips the Core Rule Set. In `observe` mode
|
||||||
such a request is answered `allow` with `would_action`, and the Core Rule Set
|
such a request is answered `allow` with `would_action`, and no other check runs
|
||||||
does not read it.
|
on it.
|
||||||
|
|
||||||
The Core Rule Set runs each of its rules on each cookie, so thousands of small
|
The Core Rule Set runs each of its rules on each cookie, so thousands of small
|
||||||
cookies would keep it busy for over a second. Within the limits, one long cookie
|
cookies would keep it busy for over a second. Within the limits, one long cookie
|
||||||
@@ -2578,8 +2602,8 @@ app image built on it would inherit it. `rproxy` keeps nothing, and needs no
|
|||||||
volume.
|
volume.
|
||||||
|
|
||||||
`waf` keeps its cache files in `SWW_CACHE_DIR`, `/var/cache/sww` by default,
|
`waf` keeps its cache files in `SWW_CACHE_DIR`, `/var/cache/sww` by default,
|
||||||
which the `run` script makes. It needs no volume and can be left out of backups:
|
which the `run` script makes. The cache directory needs no volume and can be
|
||||||
a new container starts with empty caches, which fill again.
|
left out of backups: a new container starts with empty caches, which fill again.
|
||||||
|
|
||||||
### Tokens
|
### Tokens
|
||||||
|
|
||||||
|
|||||||
+2
-1
@@ -23,7 +23,8 @@ back or dropped.
|
|||||||
- the day's AbuseIPDB checks used up.
|
- the day's AbuseIPDB checks used up.
|
||||||
- `file_error`: one of:
|
- `file_error`: one of:
|
||||||
- a rule file or state file edit with an error;
|
- a rule file or state file edit with an error;
|
||||||
- a state file that cannot be written;
|
- a cache file that does not parse, at start or after an edit;
|
||||||
|
- a state file or cache file that cannot be written;
|
||||||
- a replacement lookup database that cannot be read.
|
- a replacement lookup database that cannot be read.
|
||||||
- `summary`: the hourly summary of alerts held back.
|
- `summary`: the hourly summary of alerts held back.
|
||||||
|
|
||||||
|
|||||||
+16
-4
@@ -8,7 +8,7 @@ top-level `version` of 1.
|
|||||||
what cannot be fetched again, so back them up:
|
what cannot be fetched again, so back them up:
|
||||||
- `bans.json`;
|
- `bans.json`;
|
||||||
- `clients.json`;
|
- `clients.json`;
|
||||||
- `abuseipdb-checks.json`;
|
- `abuseipdb-checks-spent.json`;
|
||||||
- `alerts.json`.
|
- `alerts.json`.
|
||||||
- The cache files are in `SWW_CACHE_DIR`, `/var/cache/sww` by default. They hold
|
- The cache files are in `SWW_CACHE_DIR`, `/var/cache/sww` by default. They hold
|
||||||
answers from outside services, which are asked again when a file is lost, so
|
answers from outside services, which are asked again when a file is lost, so
|
||||||
@@ -16,7 +16,8 @@ top-level `version` of 1.
|
|||||||
- `lookups.json`;
|
- `lookups.json`;
|
||||||
- `lists.json`;
|
- `lists.json`;
|
||||||
- `dnsbl.json`;
|
- `dnsbl.json`;
|
||||||
- `abuseipdb-scores.json`.
|
- `abuseipdb-scores.json`;
|
||||||
|
- `routewatch.json`.
|
||||||
|
|
||||||
## `bans.json`
|
## `bans.json`
|
||||||
|
|
||||||
@@ -81,7 +82,7 @@ since it was first seen:
|
|||||||
`blocked_by_core_rule_set`. A file may also hold `token_refused`, which is
|
`blocked_by_core_rule_set`. A file may also hold `token_refused`, which is
|
||||||
kept, and which nothing adds to.
|
kept, and which nothing adds to.
|
||||||
|
|
||||||
## `abuseipdb-checks.json`
|
## `abuseipdb-checks-spent.json`
|
||||||
|
|
||||||
The AbuseIPDB checks spent today, so that a restart does not spend the daily
|
The AbuseIPDB checks spent today, so that a restart does not spend the daily
|
||||||
budget again:
|
budget again:
|
||||||
@@ -132,6 +133,16 @@ A cache file. AbuseIPDB's scores still in use, one per line:
|
|||||||
- `score`: from 0 to 100;
|
- `score`: from 0 to 100;
|
||||||
- `fetched`: when AbuseIPDB answered.
|
- `fetched`: when AbuseIPDB answered.
|
||||||
|
|
||||||
|
## `routewatch.json`
|
||||||
|
|
||||||
|
A cache file. routewatch's answers from the last hour, one per line:
|
||||||
|
|
||||||
|
- `client`;
|
||||||
|
- `netblock`: the netblock the client's address is announced in, empty for an
|
||||||
|
address without a live route;
|
||||||
|
- `asn`: the number of the AS that originates the netblock, 0 for none;
|
||||||
|
- `answered`: when routewatch answered.
|
||||||
|
|
||||||
## Sizes
|
## Sizes
|
||||||
|
|
||||||
- `clients.json`: about 1 KiB per client, about 20 MiB at the default
|
- `clients.json`: about 1 KiB per client, about 20 MiB at the default
|
||||||
@@ -143,6 +154,7 @@ A cache file. AbuseIPDB's scores still in use, one per line:
|
|||||||
more.
|
more.
|
||||||
- `lookups.json`: about 150 bytes per answer, under 1.5 GiB of writes a day when
|
- `lookups.json`: about 150 bytes per answer, under 1.5 GiB of writes a day when
|
||||||
full.
|
full.
|
||||||
- `abuseipdb-checks.json`: a few bytes.
|
- `routewatch.json`: about 120 bytes per answer, at most about 12 MiB.
|
||||||
|
- `abuseipdb-checks-spent.json`: a few bytes.
|
||||||
- `lists.json`, `dnsbl.json`, `abuseipdb-scores.json` and `alerts.json`: usually
|
- `lists.json`, `dnsbl.json`, `abuseipdb-scores.json` and `alerts.json`: usually
|
||||||
a few MiB or less.
|
a few MiB or less.
|
||||||
@@ -45,7 +45,7 @@ type Score struct {
|
|||||||
Fetched time.Time `json:"fetched"`
|
Fetched time.Time `json:"fetched"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ChecksSpent is what abuseipdb-checks.json keeps of the checks of
|
// ChecksSpent is what abuseipdb-checks-spent.json keeps of the checks of
|
||||||
// clients with AbuseIPDB: the day, in UTC, of the checks Spent counts,
|
// clients with AbuseIPDB: the day, in UTC, of the checks Spent counts,
|
||||||
// zero before the first.
|
// zero before the first.
|
||||||
type ChecksSpent struct {
|
type ChecksSpent struct {
|
||||||
@@ -238,8 +238,8 @@ func (a *AbuseIPDB) Load(scores []Score) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ChecksSpent returns the checks spent, as abuseipdb-checks.json keeps
|
// ChecksSpent returns the checks spent, as abuseipdb-checks-spent.json
|
||||||
// them.
|
// keeps them.
|
||||||
func (a *AbuseIPDB) ChecksSpent() ChecksSpent {
|
func (a *AbuseIPDB) ChecksSpent() ChecksSpent {
|
||||||
a.mu.Lock()
|
a.mu.Lock()
|
||||||
defer a.mu.Unlock()
|
defer a.mu.Unlock()
|
||||||
@@ -247,8 +247,8 @@ func (a *AbuseIPDB) ChecksSpent() ChecksSpent {
|
|||||||
return ChecksSpent{Day: a.day, Spent: a.spent}
|
return ChecksSpent{Day: a.day, Spent: a.spent}
|
||||||
}
|
}
|
||||||
|
|
||||||
// LoadChecksSpent keeps checks, read from abuseipdb-checks.json, in place
|
// LoadChecksSpent keeps checks, read from abuseipdb-checks-spent.json, in
|
||||||
// of the checks spent.
|
// place of the checks spent.
|
||||||
func (a *AbuseIPDB) LoadChecksSpent(checks ChecksSpent) {
|
func (a *AbuseIPDB) LoadChecksSpent(checks ChecksSpent) {
|
||||||
a.mu.Lock()
|
a.mu.Lock()
|
||||||
defer a.mu.Unlock()
|
defer a.mu.Unlock()
|
||||||
|
|||||||
@@ -181,8 +181,8 @@ func TestDailyBudgetKeptAcrossARestartAndWholeAgainAsTheDayEnds(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Restarted with what abuseipdb-scores.json and
|
// Restarted with what abuseipdb-scores.json and
|
||||||
// abuseipdb-checks.json keep, it uses the scores, and checks no
|
// abuseipdb-checks-spent.json keep, it uses the scores, and checks
|
||||||
// client until the day ends.
|
// no client until the day ends.
|
||||||
restarted := &abuseIPDBStandIn{}
|
restarted := &abuseIPDBStandIn{}
|
||||||
again := newAbuseIPDB(restarted, p)
|
again := newAbuseIPDB(restarted, p)
|
||||||
again.Load(checker.Snapshot())
|
again.Load(checker.Snapshot())
|
||||||
|
|||||||
@@ -1,6 +1,8 @@
|
|||||||
// Package routewatch asks a routewatch instance about client addresses, in the
|
// Package routewatch asks a routewatch instance about client addresses, in the
|
||||||
// background, and keeps its answers in memory. A routewatch instance knows, for
|
// background, and keeps its answers. A routewatch instance knows, for an
|
||||||
// an address, the netblock it is announced in and the AS that originates it.
|
// address, the netblock it is announced in and the AS that originates it. The
|
||||||
|
// answers are kept in memory, for at most 100,000 clients and for an hour each,
|
||||||
|
// and are written to routewatch.json and read from it by the state package.
|
||||||
package routewatch
|
package routewatch
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -13,6 +15,7 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"net/url"
|
"net/url"
|
||||||
|
"slices"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -65,6 +68,17 @@ type Route struct {
|
|||||||
ASN uint32
|
ASN uint32
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Answer is what routewatch answered about a client, as routewatch.json holds
|
||||||
|
// it: the netblock the client's address is announced in, none for an address
|
||||||
|
// without a live route, the AS that originates it, and when routewatch
|
||||||
|
// answered.
|
||||||
|
type Answer struct {
|
||||||
|
Client netip.Prefix `json:"client"`
|
||||||
|
Netblock netip.Prefix `json:"netblock"`
|
||||||
|
ASN uint32 `json:"asn"`
|
||||||
|
Answered time.Time `json:"answered"`
|
||||||
|
}
|
||||||
|
|
||||||
// Params are what New needs.
|
// Params are what New needs.
|
||||||
type Params struct {
|
type Params struct {
|
||||||
// URL is where routewatch is (SWW_ROUTEWATCH_URL).
|
// URL is where routewatch is (SWW_ROUTEWATCH_URL).
|
||||||
@@ -84,23 +98,16 @@ type Client struct {
|
|||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
// answers are by client. Each is added as it comes and never moved up,
|
// answers are by client. Each is added as it comes and never moved up,
|
||||||
// so that the one that came longest ago is the first dropped.
|
// so that the one that came longest ago is the first dropped.
|
||||||
answers *simplelru.LRU[netip.Prefix, answer]
|
answers *simplelru.LRU[netip.Prefix, Answer]
|
||||||
// asking are the clients whose query is under way.
|
// asking are the clients whose query is under way.
|
||||||
asking map[netip.Prefix]bool
|
asking map[netip.Prefix]bool
|
||||||
// retryAt is when routewatch may be asked again after a query failed.
|
// retryAt is when routewatch may be asked again after a query failed.
|
||||||
retryAt time.Time
|
retryAt time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
// answer is one kept answer: the route, or why there is none, and when it came.
|
|
||||||
type answer struct {
|
|
||||||
route Route
|
|
||||||
missing Missing
|
|
||||||
came time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
// New returns a Client with no answer yet.
|
// New returns a Client with no answer yet.
|
||||||
func New(params Params) *Client {
|
func New(params Params) *Client {
|
||||||
answers, err := simplelru.NewLRU[netip.Prefix, answer](maxAnswers, nil)
|
answers, err := simplelru.NewLRU[netip.Prefix, Answer](maxAnswers, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
panic(err) // NewLRU fails only for a size below one
|
panic(err) // NewLRU fails only for a size below one
|
||||||
}
|
}
|
||||||
@@ -128,8 +135,12 @@ func (c *Client) Route(
|
|||||||
now := c.params.Now()
|
now := c.params.Now()
|
||||||
|
|
||||||
kept, found := c.answers.Peek(client)
|
kept, found := c.answers.Peek(client)
|
||||||
if found && now.Sub(kept.came) < answerKept {
|
if found && now.Sub(kept.Answered) < answerKept {
|
||||||
return kept.route, kept.missing
|
if !kept.Netblock.IsValid() {
|
||||||
|
return Route{}, NoRoute
|
||||||
|
}
|
||||||
|
|
||||||
|
return Route{Netblock: kept.Netblock, ASN: kept.ASN}, ""
|
||||||
}
|
}
|
||||||
|
|
||||||
if now.Before(c.retryAt) {
|
if now.Before(c.retryAt) {
|
||||||
@@ -145,12 +156,58 @@ func (c *Client) Route(
|
|||||||
return Route{}, NoAnswerYet
|
return Route{}, NoAnswerYet
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Snapshot returns every answer still in use, sorted by client, as
|
||||||
|
// routewatch.json lists them.
|
||||||
|
func (c *Client) Snapshot() []Answer {
|
||||||
|
c.mu.Lock()
|
||||||
|
|
||||||
|
now := c.params.Now()
|
||||||
|
answers := make([]Answer, 0, c.answers.Len())
|
||||||
|
|
||||||
|
for _, kept := range c.answers.Values() {
|
||||||
|
if now.Sub(kept.Answered) < answerKept {
|
||||||
|
answers = append(answers, kept)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
c.mu.Unlock()
|
||||||
|
|
||||||
|
slices.SortFunc(answers, func(a, b Answer) int {
|
||||||
|
return a.Client.Compare(b.Client)
|
||||||
|
})
|
||||||
|
|
||||||
|
return answers
|
||||||
|
}
|
||||||
|
|
||||||
|
// Load keeps answers, read from routewatch.json, in place of those it keeps,
|
||||||
|
// but for those that came answerKept ago or more and, past maxAnswers, those
|
||||||
|
// that came longest ago.
|
||||||
|
func (c *Client) Load(answers []Answer) {
|
||||||
|
answers = slices.Clone(answers)
|
||||||
|
slices.SortStableFunc(answers, func(a, b Answer) int {
|
||||||
|
return a.Answered.Compare(b.Answered)
|
||||||
|
})
|
||||||
|
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
|
||||||
|
c.answers.Purge()
|
||||||
|
|
||||||
|
now := c.params.Now()
|
||||||
|
|
||||||
|
for _, kept := range answers {
|
||||||
|
if now.Sub(kept.Answered) < answerKept {
|
||||||
|
c.answers.Add(kept.Client, kept)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ask asks routewatch about addr, keeps the answer as client's, and notes the
|
// ask asks routewatch about addr, keeps the answer as client's, and notes the
|
||||||
// query as no longer under way. A query that fails gives no answer, and
|
// query as no longer under way. A query that fails gives no answer, and
|
||||||
// routewatch is not asked again for failureKept. Only the first failure of
|
// routewatch is not asked again for failureKept. Only the first failure of
|
||||||
// those under way is logged.
|
// those under way is logged.
|
||||||
func (c *Client) ask(ctx context.Context, client netip.Prefix, addr netip.Addr) {
|
func (c *Client) ask(ctx context.Context, client netip.Prefix, addr netip.Addr) {
|
||||||
route, missing, err := c.query(ctx, addr)
|
route, err := c.query(ctx, addr)
|
||||||
now := c.params.Now()
|
now := c.params.Now()
|
||||||
|
|
||||||
c.mu.Lock()
|
c.mu.Lock()
|
||||||
@@ -160,7 +217,9 @@ func (c *Client) ask(ctx context.Context, client netip.Prefix, addr netip.Addr)
|
|||||||
firstFailure := err != nil && !now.Before(c.retryAt)
|
firstFailure := err != nil && !now.Before(c.retryAt)
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
c.answers.Add(client, answer{route: route, missing: missing, came: now})
|
c.answers.Add(client, Answer{
|
||||||
|
Client: client, Netblock: route.Netblock, ASN: route.ASN, Answered: now,
|
||||||
|
})
|
||||||
} else {
|
} else {
|
||||||
c.retryAt = now.Add(failureKept)
|
c.retryAt = now.Add(failureKept)
|
||||||
}
|
}
|
||||||
@@ -174,19 +233,19 @@ func (c *Client) ask(ctx context.Context, client netip.Prefix, addr netip.Addr)
|
|||||||
|
|
||||||
// query asks routewatch's IP info API about addr. An answer other than 200, one
|
// query asks routewatch's IP info API about addr. An answer other than 200, one
|
||||||
// that cannot be read, and none within queryTimeout, fail. A result with no
|
// that cannot be read, and none within queryTimeout, fail. A result with no
|
||||||
// netblock is an address without a live route.
|
// netblock is an address without a live route, and gives a Route without one.
|
||||||
func (c *Client) query(ctx context.Context, addr netip.Addr) (Route, Missing, error) {
|
func (c *Client) query(ctx context.Context, addr netip.Addr) (Route, error) {
|
||||||
ctx, cancel := context.WithTimeout(ctx, queryTimeout)
|
ctx, cancel := context.WithTimeout(ctx, queryTimeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
endpoint, err := url.JoinPath(c.params.URL, "api/v1/ip", addr.String())
|
endpoint, err := url.JoinPath(c.params.URL, "api/v1/ip", addr.String())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Route{}, "", fmt.Errorf("make the URL: %w", err)
|
return Route{}, fmt.Errorf("make the URL: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, http.NoBody)
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, http.NoBody)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Route{}, "", fmt.Errorf("make the request: %w", err)
|
return Route{}, fmt.Errorf("make the request: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
req.Header.Set("Accept", "application/json")
|
req.Header.Set("Accept", "application/json")
|
||||||
@@ -195,7 +254,7 @@ func (c *Client) query(ctx context.Context, addr netip.Addr) (Route, Missing, er
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
// Do's error names the URL, which holds the client's address, which
|
// Do's error names the URL, which holds the client's address, which
|
||||||
// is not to be logged: only what went wrong is kept.
|
// is not to be logged: only what went wrong is kept.
|
||||||
return Route{}, "", fmt.Errorf("ask routewatch: %w", errors.Unwrap(err))
|
return Route{}, fmt.Errorf("ask routewatch: %w", errors.Unwrap(err))
|
||||||
}
|
}
|
||||||
|
|
||||||
defer func() {
|
defer func() {
|
||||||
@@ -203,14 +262,14 @@ func (c *Client) query(ctx context.Context, addr netip.Addr) (Route, Missing, er
|
|||||||
}()
|
}()
|
||||||
|
|
||||||
if res.StatusCode != http.StatusOK {
|
if res.StatusCode != http.StatusOK {
|
||||||
return Route{}, "", fmt.Errorf("%w %s", errStatus, res.Status)
|
return Route{}, fmt.Errorf("%w %s", errStatus, res.Status)
|
||||||
}
|
}
|
||||||
|
|
||||||
return readAnswer(io.LimitReader(res.Body, maxAnswerBytes))
|
return readAnswer(io.LimitReader(res.Body, maxAnswerBytes))
|
||||||
}
|
}
|
||||||
|
|
||||||
// readAnswer reads routewatch's answer about one address.
|
// readAnswer reads routewatch's answer about one address.
|
||||||
func readAnswer(body io.Reader) (Route, Missing, error) {
|
func readAnswer(body io.Reader) (Route, error) {
|
||||||
var decoded struct {
|
var decoded struct {
|
||||||
Data struct {
|
Data struct {
|
||||||
Results []struct {
|
Results []struct {
|
||||||
@@ -222,22 +281,22 @@ func readAnswer(body io.Reader) (Route, Missing, error) {
|
|||||||
|
|
||||||
err := json.NewDecoder(body).Decode(&decoded)
|
err := json.NewDecoder(body).Decode(&decoded)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Route{}, "", fmt.Errorf("read the answer: %w", err)
|
return Route{}, fmt.Errorf("read the answer: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(decoded.Data.Results) == 0 {
|
if len(decoded.Data.Results) == 0 {
|
||||||
return Route{}, "", errNoResult
|
return Route{}, errNoResult
|
||||||
}
|
}
|
||||||
|
|
||||||
result := decoded.Data.Results[0]
|
result := decoded.Data.Results[0]
|
||||||
if result.Netblock == "" {
|
if result.Netblock == "" {
|
||||||
return Route{}, NoRoute, nil
|
return Route{}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
netblock, err := netip.ParsePrefix(result.Netblock)
|
netblock, err := netip.ParsePrefix(result.Netblock)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Route{}, "", fmt.Errorf("%w: %q", errNotNetblock, result.Netblock)
|
return Route{}, fmt.Errorf("%w: %q", errNotNetblock, result.Netblock)
|
||||||
}
|
}
|
||||||
|
|
||||||
return Route{Netblock: netblock.Masked(), ASN: result.ASN}, "", nil
|
return Route{Netblock: netblock.Masked(), ASN: result.ASN}, nil
|
||||||
}
|
}
|
||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
@@ -336,3 +337,96 @@ func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) {
|
|||||||
t.Errorf("%d queries after the answer expired, want 2", got)
|
t.Errorf("%d queries after the answer expired, want 2", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestSnapshotHoldsTheAnswerUntilItExpires(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
f := newFixture(t, answering(routedAnswer))
|
||||||
|
|
||||||
|
f.route()
|
||||||
|
f.waitFor("")
|
||||||
|
|
||||||
|
want := []routewatch.Answer{{
|
||||||
|
Client: netip.MustParsePrefix(clientAddr + "/32"),
|
||||||
|
Netblock: netip.MustParsePrefix("192.0.2.0/24"), ASN: 64496,
|
||||||
|
Answered: f.clk.Now(),
|
||||||
|
}}
|
||||||
|
if got := f.client.Snapshot(); !slices.Equal(got, want) {
|
||||||
|
t.Errorf("snapshot %+v, want %+v", got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
f.clk.advance(time.Hour)
|
||||||
|
|
||||||
|
if got := f.client.Snapshot(); len(got) != 0 {
|
||||||
|
t.Errorf("snapshot %+v an hour after the answer, want none", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadedAnswersAreUsedForAnHourFromWhenTheyCame(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
f := newFixture(t, answering(routedAnswer))
|
||||||
|
now := f.clk.Now()
|
||||||
|
kept := routewatch.Answer{
|
||||||
|
Client: netip.MustParsePrefix(clientAddr + "/32"),
|
||||||
|
Netblock: netip.MustParsePrefix("198.51.100.0/24"), ASN: 64497,
|
||||||
|
Answered: now.Add(-time.Hour + time.Second),
|
||||||
|
}
|
||||||
|
stale := routewatch.Answer{
|
||||||
|
Client: netip.MustParsePrefix("192.0.2.2/32"),
|
||||||
|
Netblock: netip.MustParsePrefix("192.0.2.0/24"), ASN: 64496,
|
||||||
|
Answered: now.Add(-time.Hour),
|
||||||
|
}
|
||||||
|
|
||||||
|
f.client.Load([]routewatch.Answer{kept, stale})
|
||||||
|
|
||||||
|
if got := f.client.Snapshot(); !slices.Equal(got, []routewatch.Answer{kept}) {
|
||||||
|
t.Errorf("kept %+v, want only the answer that came less than an hour ago", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
route, missing := f.route()
|
||||||
|
|
||||||
|
want := routewatch.Route{Netblock: kept.Netblock, ASN: kept.ASN}
|
||||||
|
if route != want || missing != "" {
|
||||||
|
t.Errorf("route %+v (%q), want %+v", route, missing, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := f.queries.Load(); got != 0 {
|
||||||
|
t.Errorf("%d queries for a client with an answer loaded, want none", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadDropsTheAnswerThatCameLongestAgoFirst(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const maxAnswers = 100000
|
||||||
|
|
||||||
|
f := newFixture(t, answering(routedAnswer))
|
||||||
|
now := f.clk.Now()
|
||||||
|
|
||||||
|
// routewatch.json lists the answers by client. Here each came a
|
||||||
|
// millisecond before the one listed before it, so the last listed is
|
||||||
|
// the one that came longest ago, and the one dropped.
|
||||||
|
answers := make([]routewatch.Answer, maxAnswers+1)
|
||||||
|
addr := netip.MustParseAddr("10.0.0.0")
|
||||||
|
|
||||||
|
for i := range answers {
|
||||||
|
answers[i] = routewatch.Answer{
|
||||||
|
Client: netip.PrefixFrom(addr, addr.BitLen()),
|
||||||
|
Answered: now.Add(-time.Duration(i) * time.Millisecond),
|
||||||
|
}
|
||||||
|
addr = addr.Next()
|
||||||
|
}
|
||||||
|
|
||||||
|
f.client.Load(answers)
|
||||||
|
|
||||||
|
got := f.client.Snapshot()
|
||||||
|
if len(got) != maxAnswers {
|
||||||
|
t.Fatalf("%d answers kept, want %d", len(got), maxAnswers)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got[0] != answers[0] || got[maxAnswers-1] != answers[maxAnswers-1] {
|
||||||
|
t.Errorf("answers kept from %s to %s, want from %s to %s", got[0].Client,
|
||||||
|
got[maxAnswers-1].Client, answers[0].Client, answers[maxAnswers-1].Client)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1023,7 +1023,9 @@ func TestAnAnswerOf1MiBGoesToTheVisitor(t *testing.T) {
|
|||||||
w.WriteHeader(http.StatusInternalServerError)
|
w.WriteHeader(http.StatusInternalServerError)
|
||||||
_, _ = w.Write(make([]byte, size))
|
_, _ = w.Write(make([]byte, size))
|
||||||
})
|
})
|
||||||
addr, _ := startRProxy(t, standIn.URL, nowhere, nil)
|
addr, _ := startRProxy(t, standIn.URL, nowhere, map[string]string{
|
||||||
|
timeoutSetting: "60000",
|
||||||
|
})
|
||||||
|
|
||||||
got := get(t, addr, "/")
|
got := get(t, addr, "/")
|
||||||
if got.status != http.StatusInternalServerError || len(got.body) != size {
|
if got.status != http.StatusInternalServerError || len(got.body) != size {
|
||||||
|
|||||||
+130
-36
@@ -4,7 +4,7 @@
|
|||||||
// The state files are in SWW_STATE_DIR:
|
// The state files are in SWW_STATE_DIR:
|
||||||
// - bans.json holds the bans;
|
// - bans.json holds the bans;
|
||||||
// - clients.json holds each client's counters and history;
|
// - clients.json holds each client's counters and history;
|
||||||
// - abuseipdb-checks.json holds the AbuseIPDB checks spent today;
|
// - abuseipdb-checks-spent.json holds the AbuseIPDB checks spent today;
|
||||||
// - alerts.json holds the cooldowns, the hour under way, the alerts
|
// - alerts.json holds the cooldowns, the hour under way, the alerts
|
||||||
// waiting for each destination and the anomaly counters.
|
// waiting for each destination and the anomaly counters.
|
||||||
//
|
//
|
||||||
@@ -14,7 +14,8 @@
|
|||||||
// - lists.json holds the last try and last good copy of each list
|
// - lists.json holds the last try and last good copy of each list
|
||||||
// fetched from a URL;
|
// fetched from a URL;
|
||||||
// - dnsbl.json holds the DNSBL zones' verdicts;
|
// - dnsbl.json holds the DNSBL zones' verdicts;
|
||||||
// - abuseipdb-scores.json holds AbuseIPDB's scores.
|
// - abuseipdb-scores.json holds AbuseIPDB's scores;
|
||||||
|
// - routewatch.json holds routewatch's answers.
|
||||||
//
|
//
|
||||||
// Load reads them at start, Watch takes in an admin's edit of one while
|
// Load reads them at start, Watch takes in an admin's edit of one while
|
||||||
// sww runs, and Run and WriteAll write them. The disk is read and
|
// sww runs, and Run and WriteAll write them. The disk is read and
|
||||||
@@ -49,6 +50,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/routewatch"
|
||||||
)
|
)
|
||||||
|
|
||||||
// version is the version of the files' format, the only one read.
|
// version is the version of the files' format, the only one read.
|
||||||
@@ -62,7 +64,7 @@ const fileMode = 0o600
|
|||||||
const (
|
const (
|
||||||
bansJSON = "bans.json"
|
bansJSON = "bans.json"
|
||||||
clientsJSON = "clients.json"
|
clientsJSON = "clients.json"
|
||||||
abuseIPDBChecksJSON = "abuseipdb-checks.json"
|
abuseIPDBChecksSpentJSON = "abuseipdb-checks-spent.json"
|
||||||
alertsJSON = "alerts.json"
|
alertsJSON = "alerts.json"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -72,21 +74,22 @@ const (
|
|||||||
listsJSON = "lists.json"
|
listsJSON = "lists.json"
|
||||||
dnsblJSON = "dnsbl.json"
|
dnsblJSON = "dnsbl.json"
|
||||||
abuseIPDBScoresJSON = "abuseipdb-scores.json"
|
abuseIPDBScoresJSON = "abuseipdb-scores.json"
|
||||||
|
routeWatchJSON = "routewatch.json"
|
||||||
)
|
)
|
||||||
|
|
||||||
// fileNames returns the names of the state files, then those of the
|
// fileNames returns the names of the state files, then those of the
|
||||||
// cache files, in the order they are written.
|
// cache files, in the order they are written.
|
||||||
func fileNames() []string {
|
func fileNames() []string {
|
||||||
return []string{
|
return []string{
|
||||||
bansJSON, clientsJSON, abuseIPDBChecksJSON, alertsJSON,
|
bansJSON, clientsJSON, abuseIPDBChecksSpentJSON, alertsJSON,
|
||||||
lookupsJSON, listsJSON, dnsblJSON, abuseIPDBScoresJSON,
|
lookupsJSON, listsJSON, dnsblJSON, abuseIPDBScoresJSON, routeWatchJSON,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// isCacheFile reports whether name is the name of a cache file.
|
// isCacheFile reports whether name is the name of a cache file.
|
||||||
func isCacheFile(name string) bool {
|
func isCacheFile(name string) bool {
|
||||||
switch name {
|
switch name {
|
||||||
case lookupsJSON, listsJSON, dnsblJSON, abuseIPDBScoresJSON:
|
case lookupsJSON, listsJSON, dnsblJSON, abuseIPDBScoresJSON, routeWatchJSON:
|
||||||
return true
|
return true
|
||||||
default:
|
default:
|
||||||
return false
|
return false
|
||||||
@@ -117,15 +120,18 @@ type Params struct {
|
|||||||
// is (SWW_STATE_COUNTER_INTERVAL).
|
// is (SWW_STATE_COUNTER_INTERVAL).
|
||||||
WriteDelay time.Duration
|
WriteDelay time.Duration
|
||||||
CounterInterval time.Duration
|
CounterInterval time.Duration
|
||||||
// Ledger, Limiter, GeoJS, Lists, DNSBL, AbuseIPDB, Alerts and Anomalies
|
// Ledger, Limiter, GeoJS, Lists, DNSBL, AbuseIPDB, RouteWatch, Alerts and
|
||||||
// hold the state. Alerts also receive a file_error alert for an edit set
|
// Anomalies hold the state. RouteWatch is nil while SWW_ROUTEWATCH_URL is
|
||||||
// aside, and for a write that fails while sww runs.
|
// unset, and routewatch.json then holds no answers. Alerts also receive a
|
||||||
|
// file_error alert for an edit set aside, and for a write that fails while
|
||||||
|
// sww runs.
|
||||||
Ledger *bans.Ledger
|
Ledger *bans.Ledger
|
||||||
Limiter *ratelimit.Limiter
|
Limiter *ratelimit.Limiter
|
||||||
GeoJS *lookup.GeoJS
|
GeoJS *lookup.GeoJS
|
||||||
Lists *reputation.Lists
|
Lists *reputation.Lists
|
||||||
DNSBL *reputation.DNSBL
|
DNSBL *reputation.DNSBL
|
||||||
AbuseIPDB *reputation.AbuseIPDB
|
AbuseIPDB *reputation.AbuseIPDB
|
||||||
|
RouteWatch *routewatch.Client
|
||||||
Alerts *alerts.Queue
|
Alerts *alerts.Queue
|
||||||
Anomalies *anomaly.Counters
|
Anomalies *anomaly.Counters
|
||||||
// Now tells the time by which the counters' buckets run out, normally
|
// Now tells the time by which the counters' buckets run out, normally
|
||||||
@@ -185,9 +191,9 @@ type lookupsFile struct {
|
|||||||
Lookups []lookup.Answer `json:"lookups"`
|
Lookups []lookup.Answer `json:"lookups"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// abuseIPDBChecksFile is abuseipdb-checks.json, indented for an admin to
|
// abuseIPDBChecksSpentFile is abuseipdb-checks-spent.json, indented for an
|
||||||
// read and edit.
|
// admin to read and edit.
|
||||||
type abuseIPDBChecksFile struct {
|
type abuseIPDBChecksSpentFile struct {
|
||||||
Version int `json:"version"`
|
Version int `json:"version"`
|
||||||
Day civil.Date `json:"day,omitzero"`
|
Day civil.Date `json:"day,omitzero"`
|
||||||
Spent int `json:"spent"`
|
Spent int `json:"spent"`
|
||||||
@@ -213,6 +219,12 @@ type abuseIPDBScoresFile struct {
|
|||||||
Scores []reputation.Score `json:"scores"`
|
Scores []reputation.Score `json:"scores"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// routeWatchFile is routewatch.json, with each answer on a line of its own.
|
||||||
|
type routeWatchFile struct {
|
||||||
|
Version int `json:"version"`
|
||||||
|
Answers []routewatch.Answer `json:"answers"`
|
||||||
|
}
|
||||||
|
|
||||||
// alertsFile is alerts.json, indented for an admin to read and edit.
|
// alertsFile is alerts.json, indented for an admin to read and edit.
|
||||||
//
|
//
|
||||||
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
||||||
@@ -257,7 +269,7 @@ func Load(params Params) (*Files, error) {
|
|||||||
|
|
||||||
bansRead, bansErr := f.read(bansJSON)
|
bansRead, bansErr := f.read(bansJSON)
|
||||||
clientsRead, clientsErr := f.read(clientsJSON)
|
clientsRead, clientsErr := f.read(clientsJSON)
|
||||||
_, checksErr := f.read(abuseIPDBChecksJSON)
|
_, checksErr := f.read(abuseIPDBChecksSpentJSON)
|
||||||
alertsRead, alertsErr := f.read(alertsJSON)
|
alertsRead, alertsErr := f.read(alertsJSON)
|
||||||
|
|
||||||
err = errors.Join(bansErr, clientsErr, checksErr, alertsErr)
|
err = errors.Join(bansErr, clientsErr, checksErr, alertsErr)
|
||||||
@@ -269,6 +281,7 @@ func Load(params Params) (*Files, error) {
|
|||||||
listsRead := f.readCache(listsJSON)
|
listsRead := f.readCache(listsJSON)
|
||||||
f.readCache(dnsblJSON)
|
f.readCache(dnsblJSON)
|
||||||
f.readCache(abuseIPDBScoresJSON)
|
f.readCache(abuseIPDBScoresJSON)
|
||||||
|
f.readCache(routeWatchJSON)
|
||||||
|
|
||||||
params.ProcessLog.Info("read the state files", "directory", params.Dir,
|
params.ProcessLog.Info("read the state files", "directory", params.Dir,
|
||||||
"cache_directory", params.CacheDir, "bans", bansRead, "clients", clientsRead,
|
"cache_directory", params.CacheDir, "bans", bansRead, "clients", clientsRead,
|
||||||
@@ -325,19 +338,11 @@ func (f *Files) WriteAll() error {
|
|||||||
// admin's edit of a file as soon as it is saved: what the file holds
|
// admin's edit of a file as soon as it is saved: what the file holds
|
||||||
// replaces what sww held for it. An edit that does not parse is left for
|
// replaces what sww held for it. An edit that does not parse is left for
|
||||||
// the file's next write, which sets it aside, since a file can be read
|
// the file's next write, which sets it aside, since a file can be read
|
||||||
// while an editor is still writing it. If a directory cannot be watched,
|
// while an editor is still writing it. A directory that cannot be watched
|
||||||
// that is logged, and an edit is taken in only before its file is
|
// is logged, and the other is still watched. An edit in a directory not
|
||||||
// written.
|
// watched is taken in only before its file is written.
|
||||||
func (f *Files) Watch(ctx context.Context) {
|
func (f *Files) Watch(ctx context.Context) {
|
||||||
watcher, err := fsnotify.NewWatcher()
|
watcher, err := fsnotify.NewWatcher()
|
||||||
if err == nil {
|
|
||||||
defer func() {
|
|
||||||
_ = watcher.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
err = errors.Join(watcher.Add(f.params.Dir), watcher.Add(f.params.CacheDir))
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
f.params.ProcessLog.Error("cannot watch the state files for edits",
|
f.params.ProcessLog.Error("cannot watch the state files for edits",
|
||||||
"error", err.Error())
|
"error", err.Error())
|
||||||
@@ -345,8 +350,33 @@ func (f *Files) Watch(ctx context.Context) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
defer func() {
|
||||||
|
_ = watcher.Close()
|
||||||
|
}()
|
||||||
|
|
||||||
|
var watched []string
|
||||||
|
|
||||||
|
for _, dir := range []struct{ path, failed string }{
|
||||||
|
{f.params.Dir, "cannot watch the state files for edits"},
|
||||||
|
{f.params.CacheDir, "cannot watch the cache files for edits"},
|
||||||
|
} {
|
||||||
|
err = watcher.Add(dir.path)
|
||||||
|
if err != nil {
|
||||||
|
f.params.ProcessLog.Error(dir.failed, "directory", dir.path,
|
||||||
|
"error", err.Error())
|
||||||
|
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
watched = append(watched, dir.path)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(watched) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
f.params.ProcessLog.Info("watching the state files for edits",
|
f.params.ProcessLog.Info("watching the state files for edits",
|
||||||
"directory", f.params.Dir, "cache_directory", f.params.CacheDir)
|
"directories", watched)
|
||||||
|
|
||||||
for {
|
for {
|
||||||
select {
|
select {
|
||||||
@@ -369,7 +399,10 @@ func (f *Files) Watch(ctx context.Context) {
|
|||||||
// a file_error alert for it.
|
// a file_error alert for it.
|
||||||
func (f *Files) logFailure(name string, err error) {
|
func (f *Files) logFailure(name string, err error) {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
const failed = "writing the state files failed"
|
failed := "writing the state files failed"
|
||||||
|
if isCacheFile(name) {
|
||||||
|
failed = "writing the cache files failed"
|
||||||
|
}
|
||||||
|
|
||||||
// Raised before it is logged, so that the alert is there once the
|
// Raised before it is logged, so that the alert is there once the
|
||||||
// log line is.
|
// log line is.
|
||||||
@@ -494,8 +527,8 @@ func (f *Files) readChanged(name string) ([]byte, bool, error) {
|
|||||||
// how many entries the file holds. edit is whether data is an admin's
|
// how many entries the file holds. edit is whether data is an admin's
|
||||||
// edit taken in while sww runs, rather than the file read at the
|
// edit taken in while sww runs, rather than the file read at the
|
||||||
// start. An error names the file and, where the JSON decoder tells it,
|
// start. An error names the file and, where the JSON decoder tells it,
|
||||||
// the line and column, or else the entry. abuseipdb-checks.json holds
|
// the line and column, or else the entry. abuseipdb-checks-spent.json
|
||||||
// no entries.
|
// holds no entries.
|
||||||
func (f *Files) takeIn(name string, data []byte, edit bool) (int, error) {
|
func (f *Files) takeIn(name string, data []byte, edit bool) (int, error) {
|
||||||
path := filepath.Join(f.dir(name), name)
|
path := filepath.Join(f.dir(name), name)
|
||||||
|
|
||||||
@@ -555,8 +588,8 @@ func (f *Files) takeInState(name, path string, data []byte, edit bool) (int, err
|
|||||||
|
|
||||||
f.params.Limiter.Load(file.Clients, f.params.Now())
|
f.params.Limiter.Load(file.Clients, f.params.Now())
|
||||||
entries = len(file.Clients)
|
entries = len(file.Clients)
|
||||||
case abuseIPDBChecksJSON:
|
case abuseIPDBChecksSpentJSON:
|
||||||
var file abuseIPDBChecksFile
|
var file abuseIPDBChecksSpentFile
|
||||||
|
|
||||||
err := parse(path, data, &file)
|
err := parse(path, data, &file)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -618,6 +651,19 @@ func (f *Files) takeInCache(name, path string, data []byte) (int, error) {
|
|||||||
f.params.DNSBL.Load(file.Verdicts)
|
f.params.DNSBL.Load(file.Verdicts)
|
||||||
|
|
||||||
return len(file.Verdicts), nil
|
return len(file.Verdicts), nil
|
||||||
|
case routeWatchJSON:
|
||||||
|
var file routeWatchFile
|
||||||
|
|
||||||
|
err := parse(path, data, &file)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
|
||||||
|
if f.params.RouteWatch != nil {
|
||||||
|
f.params.RouteWatch.Load(file.Answers)
|
||||||
|
}
|
||||||
|
|
||||||
|
return len(file.Answers), nil
|
||||||
default: // abuseipdb-scores.json
|
default: // abuseipdb-scores.json
|
||||||
var file abuseIPDBScoresFile
|
var file abuseIPDBScoresFile
|
||||||
|
|
||||||
@@ -727,7 +773,10 @@ func (f *Files) setAside(name string, parseErr error) error {
|
|||||||
return errors.Join(parseErr, err)
|
return errors.Join(parseErr, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
const setAside = "set aside an edit of a state file that does not parse"
|
setAside := "set aside an edit of a state file that does not parse"
|
||||||
|
if isCacheFile(name) {
|
||||||
|
setAside = "set aside an edit of a cache file that does not parse"
|
||||||
|
}
|
||||||
|
|
||||||
// Raised before it is logged, so that the alert is there once the log
|
// Raised before it is logged, so that the alert is there once the log
|
||||||
// line is.
|
// line is.
|
||||||
@@ -760,10 +809,17 @@ func (f *Files) encode(name string) ([]byte, error) {
|
|||||||
return encodeOnePerLine("verdicts", f.params.DNSBL.Snapshot())
|
return encodeOnePerLine("verdicts", f.params.DNSBL.Snapshot())
|
||||||
case abuseIPDBScoresJSON:
|
case abuseIPDBScoresJSON:
|
||||||
return encodeOnePerLine("scores", f.params.AbuseIPDB.Snapshot())
|
return encodeOnePerLine("scores", f.params.AbuseIPDB.Snapshot())
|
||||||
case abuseIPDBChecksJSON:
|
case routeWatchJSON:
|
||||||
|
var answers []routewatch.Answer
|
||||||
|
if f.params.RouteWatch != nil {
|
||||||
|
answers = f.params.RouteWatch.Snapshot()
|
||||||
|
}
|
||||||
|
|
||||||
|
return encodeOnePerLine("answers", answers)
|
||||||
|
case abuseIPDBChecksSpentJSON:
|
||||||
checks := f.params.AbuseIPDB.ChecksSpent()
|
checks := f.params.AbuseIPDB.ChecksSpent()
|
||||||
|
|
||||||
return encodeIndented(abuseIPDBChecksFile{
|
return encodeIndented(abuseIPDBChecksSpentFile{
|
||||||
Version: version, Day: checks.Day, Spent: checks.Spent,
|
Version: version, Day: checks.Day, Spent: checks.Spent,
|
||||||
})
|
})
|
||||||
default: // alerts.json
|
default: // alerts.json
|
||||||
@@ -1014,9 +1070,40 @@ func (f *abuseIPDBScoresFile) check(data []byte) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// check refuses an answer without its client, which would answer for
|
||||||
|
// nobody, its netblock, or the time routewatch gave it, which would drop
|
||||||
|
// it. "" is the netblock of an address without a live route, which Answer
|
||||||
|
// cannot tell from a missing one, so each netblock is read again as
|
||||||
|
// written.
|
||||||
|
func (f *routeWatchFile) check(data []byte) error {
|
||||||
|
var written struct {
|
||||||
|
Answers []struct {
|
||||||
|
Netblock *string `json:"netblock"`
|
||||||
|
} `json:"answers"`
|
||||||
|
}
|
||||||
|
|
||||||
|
err := json.Unmarshal(data, &written)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, answer := range f.Answers {
|
||||||
|
switch {
|
||||||
|
case !answer.Client.IsValid():
|
||||||
|
return missing(i, "client")
|
||||||
|
case written.Answers[i].Netblock == nil:
|
||||||
|
return missing(i, "netblock")
|
||||||
|
case answer.Answered.IsZero():
|
||||||
|
return missing(i, "answered")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// check refuses nothing, since a missing day or count means no check
|
// check refuses nothing, since a missing day or count means no check
|
||||||
// spent.
|
// spent.
|
||||||
func (f *abuseIPDBChecksFile) check([]byte) error {
|
func (f *abuseIPDBChecksSpentFile) check([]byte) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1162,8 +1249,15 @@ func (f *abuseIPDBScoresFile) toUTC() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// toUTC does nothing: abuseipdb-checks.json holds a day, not a time.
|
// toUTC converts the time of each answer to UTC.
|
||||||
func (f *abuseIPDBChecksFile) toUTC() {}
|
func (f *routeWatchFile) toUTC() {
|
||||||
|
for i := range f.Answers {
|
||||||
|
convertToUTC(&f.Answers[i].Answered)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// toUTC does nothing: abuseipdb-checks-spent.json holds a day, not a time.
|
||||||
|
func (f *abuseIPDBChecksSpentFile) toUTC() {}
|
||||||
|
|
||||||
// toUTC converts every time of each cooldown, of the hour, of each alert
|
// toUTC converts every time of each cooldown, of the hour, of each alert
|
||||||
// waiting and of each anomaly counter to UTC.
|
// waiting and of each anomaly counter to UTC.
|
||||||
|
|||||||
+235
-51
@@ -30,6 +30,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/routewatch"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/state"
|
"sneak.berlin/go/smallwebwaf/internal/state"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -37,15 +38,18 @@ const (
|
|||||||
// The state files.
|
// The state files.
|
||||||
bansJSON = "bans.json"
|
bansJSON = "bans.json"
|
||||||
clientsJSON = "clients.json"
|
clientsJSON = "clients.json"
|
||||||
abuseIPDBChecksJSON = "abuseipdb-checks.json"
|
abuseIPDBChecksSpentJSON = "abuseipdb-checks-spent.json"
|
||||||
alertsJSON = "alerts.json"
|
alertsJSON = "alerts.json"
|
||||||
// The cache files.
|
// The cache files.
|
||||||
lookupsJSON = "lookups.json"
|
lookupsJSON = "lookups.json"
|
||||||
listsJSON = "lists.json"
|
listsJSON = "lists.json"
|
||||||
dnsblJSON = "dnsbl.json"
|
dnsblJSON = "dnsbl.json"
|
||||||
abuseIPDBScoresJSON = "abuseipdb-scores.json"
|
abuseIPDBScoresJSON = "abuseipdb-scores.json"
|
||||||
// What the process log says as a file that does not parse is set aside.
|
routeWatchJSON = "routewatch.json"
|
||||||
|
// What the process log says as a state file, and a cache file, that does
|
||||||
|
// not parse is set aside.
|
||||||
setAside = "set aside an edit of a state file that does not parse"
|
setAside = "set aside an edit of a state file that does not parse"
|
||||||
|
cacheSetAside = "set aside an edit of a cache file that does not parse"
|
||||||
// What Load says, after a file's path, of a first entry without its
|
// What Load says, after a file's path, of a first entry without its
|
||||||
// client or the time it was fetched.
|
// client or the time it was fetched.
|
||||||
noClient = `: entry 1 has no "client"`
|
noClient = `: entry 1 has no "client"`
|
||||||
@@ -280,21 +284,34 @@ const filledAbuseIPDBScoresJSON = `{
|
|||||||
}
|
}
|
||||||
`
|
`
|
||||||
|
|
||||||
// filledAbuseIPDBChecksJSON is abuseipdb-checks.json holding the
|
// filledAbuseIPDBChecksSpentJSON is abuseipdb-checks-spent.json holding
|
||||||
// AbuseIPDB checks spent today, as fill puts them in.
|
// the AbuseIPDB checks spent today, as fill puts them in.
|
||||||
const filledAbuseIPDBChecksJSON = `{
|
const filledAbuseIPDBChecksSpentJSON = `{
|
||||||
"version": 1,
|
"version": 1,
|
||||||
"day": "2026-10-06",
|
"day": "2026-10-06",
|
||||||
"spent": 3
|
"spent": 3
|
||||||
}
|
}
|
||||||
`
|
`
|
||||||
|
|
||||||
|
// filledRouteWatchJSON is routewatch.json holding the two answers of
|
||||||
|
// routewatch fill puts in: a route, and an address without a live route.
|
||||||
|
const filledRouteWatchJSON = `{
|
||||||
|
"version": 1,
|
||||||
|
"answers": [
|
||||||
|
{"client":"203.0.113.9/32","netblock":"203.0.113.0/24","asn":64496,` +
|
||||||
|
`"answered":"2026-10-05T23:59:00Z"},
|
||||||
|
{"client":"2001:db8::/64","netblock":"","asn":0,` +
|
||||||
|
`"answered":"2026-10-05T23:58:00Z"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
`
|
||||||
|
|
||||||
// everyFile returns the names of the state and cache files, sorted, as
|
// everyFile returns the names of the state and cache files, sorted, as
|
||||||
// wantFiles lists them in the one directory newParams gives both.
|
// wantFiles lists them in the one directory newParams gives both.
|
||||||
func everyFile() []string {
|
func everyFile() []string {
|
||||||
return []string{
|
return []string{
|
||||||
abuseIPDBChecksJSON, abuseIPDBScoresJSON, alertsJSON, bansJSON, clientsJSON,
|
abuseIPDBChecksSpentJSON, abuseIPDBScoresJSON, alertsJSON, bansJSON, clientsJSON,
|
||||||
dnsblJSON, listsJSON, lookupsJSON,
|
dnsblJSON, listsJSON, lookupsJSON, routeWatchJSON,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -332,10 +349,13 @@ func TestFilesWrittenAndReadBack(t *testing.T) {
|
|||||||
wantEqual(t, dnsblJSON, after.DNSBL.Snapshot(), before.DNSBL.Snapshot())
|
wantEqual(t, dnsblJSON, after.DNSBL.Snapshot(), before.DNSBL.Snapshot())
|
||||||
wantEqual(t, abuseIPDBScoresJSON, after.AbuseIPDB.Snapshot(),
|
wantEqual(t, abuseIPDBScoresJSON, after.AbuseIPDB.Snapshot(),
|
||||||
before.AbuseIPDB.Snapshot())
|
before.AbuseIPDB.Snapshot())
|
||||||
|
wantEqual(t, routeWatchJSON, after.RouteWatch.Snapshot(),
|
||||||
|
before.RouteWatch.Snapshot())
|
||||||
|
|
||||||
checks, wantChecks := after.AbuseIPDB.ChecksSpent(), before.AbuseIPDB.ChecksSpent()
|
checks, wantChecks := after.AbuseIPDB.ChecksSpent(), before.AbuseIPDB.ChecksSpent()
|
||||||
if checks != wantChecks {
|
if checks != wantChecks {
|
||||||
t.Errorf("%s read back %+v, want %+v", abuseIPDBChecksJSON, checks, wantChecks)
|
t.Errorf("%s read back %+v, want %+v", abuseIPDBChecksSpentJSON, checks,
|
||||||
|
wantChecks)
|
||||||
}
|
}
|
||||||
|
|
||||||
if got, want := after.Alerts.Snapshot(), before.Alerts.Snapshot(); !reflect.DeepEqual(
|
if got, want := after.Alerts.Snapshot(), before.Alerts.Snapshot(); !reflect.DeepEqual(
|
||||||
@@ -346,7 +366,7 @@ func TestFilesWrittenAndReadBack(t *testing.T) {
|
|||||||
wantEqual(t, alertsJSON, after.Anomalies.Snapshot(), before.Anomalies.Snapshot())
|
wantEqual(t, alertsJSON, after.Anomalies.Snapshot(), before.Anomalies.Snapshot())
|
||||||
|
|
||||||
// Each one-per-line file lists its entries by client, and nothing
|
// Each one-per-line file lists its entries by client, and nothing
|
||||||
// but the eight files is left in the directory.
|
// but the nine files is left in the directory.
|
||||||
wantEntries(t, filepath.Join(dir, clientsJSON), "clients",
|
wantEntries(t, filepath.Join(dir, clientsJSON), "clients",
|
||||||
"192.0.2.1/32", "203.0.113.9/32", "2001:db8::/64")
|
"192.0.2.1/32", "203.0.113.9/32", "2001:db8::/64")
|
||||||
wantEntries(t, filepath.Join(dir, lookupsJSON), "lookups",
|
wantEntries(t, filepath.Join(dir, lookupsJSON), "lookups",
|
||||||
@@ -367,8 +387,9 @@ func TestStateFilesInDirAndCacheFilesInCacheDir(t *testing.T) {
|
|||||||
t.Fatalf("write: %v", err)
|
t.Fatalf("write: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
wantFiles(t, stateDir, abuseIPDBChecksJSON, alertsJSON, bansJSON, clientsJSON)
|
wantFiles(t, stateDir, abuseIPDBChecksSpentJSON, alertsJSON, bansJSON, clientsJSON)
|
||||||
wantFiles(t, cacheDir, abuseIPDBScoresJSON, dnsblJSON, listsJSON, lookupsJSON)
|
wantFiles(t, cacheDir, abuseIPDBScoresJSON, dnsblJSON, listsJSON, lookupsJSON,
|
||||||
|
routeWatchJSON)
|
||||||
|
|
||||||
// Read back from both, as at the next start.
|
// Read back from both, as at the next start.
|
||||||
after := newParams(stateDir)
|
after := newParams(stateDir)
|
||||||
@@ -380,7 +401,8 @@ func TestStateFilesInDirAndCacheFilesInCacheDir(t *testing.T) {
|
|||||||
|
|
||||||
checks, wantChecks := after.AbuseIPDB.ChecksSpent(), before.AbuseIPDB.ChecksSpent()
|
checks, wantChecks := after.AbuseIPDB.ChecksSpent(), before.AbuseIPDB.ChecksSpent()
|
||||||
if checks != wantChecks {
|
if checks != wantChecks {
|
||||||
t.Errorf("%s read back %+v, want %+v", abuseIPDBChecksJSON, checks, wantChecks)
|
t.Errorf("%s read back %+v, want %+v", abuseIPDBChecksSpentJSON, checks,
|
||||||
|
wantChecks)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -403,7 +425,7 @@ func TestReputationFilesAsWritten(t *testing.T) {
|
|||||||
listsJSON: filledListsJSON,
|
listsJSON: filledListsJSON,
|
||||||
dnsblJSON: filledDNSBLJSON,
|
dnsblJSON: filledDNSBLJSON,
|
||||||
abuseIPDBScoresJSON: filledAbuseIPDBScoresJSON,
|
abuseIPDBScoresJSON: filledAbuseIPDBScoresJSON,
|
||||||
abuseIPDBChecksJSON: filledAbuseIPDBChecksJSON,
|
abuseIPDBChecksSpentJSON: filledAbuseIPDBChecksSpentJSON,
|
||||||
} {
|
} {
|
||||||
if got := readFile(t, filepath.Join(dir, name)); got != want {
|
if got := readFile(t, filepath.Join(dir, name)); got != want {
|
||||||
t.Errorf("%s\n%s\nwant\n%s", name, got, want)
|
t.Errorf("%s\n%s\nwant\n%s", name, got, want)
|
||||||
@@ -411,6 +433,41 @@ func TestReputationFilesAsWritten(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRouteWatchJSONAsWritten(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
params := newParams(dir)
|
||||||
|
fill(params)
|
||||||
|
|
||||||
|
err := load(t, params).WriteAll()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("write: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
got := readFile(t, filepath.Join(dir, routeWatchJSON))
|
||||||
|
if got != filledRouteWatchJSON {
|
||||||
|
t.Errorf("%s\n%s\nwant\n%s", routeWatchJSON, got, filledRouteWatchJSON)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRouteWatchJSONHoldsNoAnswersWhileRoutewatchIsUnset(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
edit(t, dir, routeWatchJSON, filledRouteWatchJSON)
|
||||||
|
|
||||||
|
params := newParams(dir)
|
||||||
|
params.RouteWatch = nil
|
||||||
|
|
||||||
|
err := load(t, params).WriteAll()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("write: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantEntries(t, filepath.Join(dir, routeWatchJSON), "answers")
|
||||||
|
}
|
||||||
|
|
||||||
func TestAlertsJSONIsIndentedWithTheCooldownsTheHourAndTheAlertsWaiting(t *testing.T) {
|
func TestAlertsJSONIsIndentedWithTheCooldownsTheHourAndTheAlertsWaiting(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -536,6 +593,7 @@ func TestMissingFilesAreEmptyState(t *testing.T) {
|
|||||||
if len(params.Ledger.Snapshot()) != 0 || len(params.Limiter.Snapshot()) != 0 ||
|
if len(params.Ledger.Snapshot()) != 0 || len(params.Limiter.Snapshot()) != 0 ||
|
||||||
len(params.GeoJS.Snapshot()) != 0 || len(params.Lists.Snapshot()) != 0 ||
|
len(params.GeoJS.Snapshot()) != 0 || len(params.Lists.Snapshot()) != 0 ||
|
||||||
len(params.DNSBL.Snapshot()) != 0 || len(params.AbuseIPDB.Snapshot()) != 0 ||
|
len(params.DNSBL.Snapshot()) != 0 || len(params.AbuseIPDB.Snapshot()) != 0 ||
|
||||||
|
len(params.RouteWatch.Snapshot()) != 0 ||
|
||||||
params.AbuseIPDB.ChecksSpent().Spent != 0 ||
|
params.AbuseIPDB.ChecksSpent().Spent != 0 ||
|
||||||
len(held.Cooldowns) != 0 || len(held.Waiting[alerts.DestinationWebhook]) != 0 ||
|
len(held.Cooldowns) != 0 || len(held.Waiting[alerts.DestinationWebhook]) != 0 ||
|
||||||
held.Hour.Sent != 0 {
|
held.Hour.Sent != 0 {
|
||||||
@@ -571,7 +629,7 @@ func TestFileThatDoesNotParseStopsTheStart(t *testing.T) {
|
|||||||
", line 3, column 19: unexpected end of JSON input",
|
", line 3, column 19: unexpected end of JSON input",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"an unknown field", abuseIPDBChecksJSON,
|
"an unknown field", abuseIPDBChecksSpentJSON,
|
||||||
`{"version": 1, "day": "2026-10-06", "spnet": 3}`,
|
`{"version": 1, "day": "2026-10-06", "spnet": 3}`,
|
||||||
`: json: unknown field "spnet"`,
|
`: json: unknown field "spnet"`,
|
||||||
},
|
},
|
||||||
@@ -642,6 +700,25 @@ func TestCacheFileThatDoesNotParseIsSetAsideAndTheCacheStartsEmpty(t *testing.T)
|
|||||||
"an unknown version", abuseIPDBScoresJSON, `{"version": 2}`,
|
"an unknown version", abuseIPDBScoresJSON, `{"version": 2}`,
|
||||||
": unknown version ",
|
": unknown version ",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"an answer without its client", routeWatchJSON,
|
||||||
|
`{"version": 1, "answers": [{"netblock": "", ` +
|
||||||
|
`"answered": "2026-10-06T00:00:00Z"}]}`,
|
||||||
|
noClient,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// A netblock of "" is an address without a live route.
|
||||||
|
"an answer without its netblock", routeWatchJSON,
|
||||||
|
`{"version": 1, "answers": [{"client": "192.0.2.1/32", "netblock": "", ` +
|
||||||
|
`"answered": "2026-10-06T00:00:00Z"}, {"client": "203.0.113.9/32", ` +
|
||||||
|
`"answered": "2026-10-06T00:00:00Z"}]}`,
|
||||||
|
`: entry 2 has no "netblock"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"an answer without the time routewatch gave it", routeWatchJSON,
|
||||||
|
`{"version": 1, "answers": [{"client": "203.0.113.9/32", "netblock": ""}]}`,
|
||||||
|
`: entry 1 has no "answered"`,
|
||||||
|
},
|
||||||
} {
|
} {
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -1055,7 +1132,9 @@ func TestBanThatDoesNotEndAfterItsStartStopsTheStart(t *testing.T) {
|
|||||||
func TestUnknownVersionStopsTheStart(t *testing.T) {
|
func TestUnknownVersionStopsTheStart(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
for _, file := range []string{bansJSON, clientsJSON, abuseIPDBChecksJSON, alertsJSON} {
|
for _, file := range []string{
|
||||||
|
bansJSON, clientsJSON, abuseIPDBChecksSpentJSON, alertsJSON,
|
||||||
|
} {
|
||||||
for _, content := range []string{`{"version": 2}`, `{}`} {
|
for _, content := range []string{`{"version": 2}`, `{}`} {
|
||||||
t.Run(file+" "+content, func(t *testing.T) {
|
t.Run(file+" "+content, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -1221,43 +1300,52 @@ func TestWriteThatFailsWhileRunningRaisesAFileErrorAlertOncePerCooldown(t *testi
|
|||||||
params.CounterInterval = time.Minute
|
params.CounterInterval = time.Minute
|
||||||
run(t, load(t, params).Run)
|
run(t, load(t, params).Run)
|
||||||
|
|
||||||
// A directory in the way of clients.json's temporary file fails each
|
// A directory in the way of the temporary file of clients.json, a
|
||||||
// of its writes, while the other files are written. It holds a file,
|
// state file, and of lookups.json, a cache file, fails each of their
|
||||||
// so that the write cannot remove it.
|
// writes, while the other files are written. Each holds a file, so
|
||||||
err := os.Mkdir(filepath.Join(dir, clientsJSON+".tmp"), 0o700)
|
// that the write cannot remove it.
|
||||||
|
failing := []struct{ name, reason string }{
|
||||||
|
{clientsJSON, "writing the state files failed"},
|
||||||
|
{lookupsJSON, "writing the cache files failed"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, file := range failing {
|
||||||
|
err := os.Mkdir(filepath.Join(dir, file.name+".tmp"), 0o700)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
err = os.WriteFile(filepath.Join(dir, clientsJSON+".tmp", "kept"), nil, 0o600)
|
err = os.WriteFile(filepath.Join(dir, file.name+".tmp", "kept"), nil, 0o600)
|
||||||
}
|
}
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("put a directory in the way: %v", err)
|
t.Fatalf("put a directory in the way: %v", err)
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
time.Sleep(time.Minute)
|
time.Sleep(time.Minute)
|
||||||
synctest.Wait()
|
synctest.Wait()
|
||||||
|
|
||||||
waiting := params.Alerts.Snapshot().Waiting[alerts.DestinationWebhook]
|
waiting := params.Alerts.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||||
if len(waiting) != 1 {
|
if len(waiting) != len(failing) {
|
||||||
t.Fatalf("%d alerts wait, want 1", len(waiting))
|
t.Fatalf("%d alerts wait, want %d", len(waiting), len(failing))
|
||||||
}
|
}
|
||||||
|
|
||||||
message, _ := waiting[0].Detail["error"].(string)
|
for i, file := range failing {
|
||||||
|
message, _ := waiting[i].Detail["error"].(string)
|
||||||
|
|
||||||
if waiting[0].Event != alerts.EventFileError ||
|
if waiting[i].Event != alerts.EventFileError || waiting[i].Reason != file.reason ||
|
||||||
waiting[0].Reason != "writing the state files failed" ||
|
waiting[i].Detail["file"] != filepath.Join(dir, file.name) ||
|
||||||
waiting[0].Detail["file"] != filepath.Join(dir, clientsJSON) ||
|
!strings.Contains(message, file.name+".tmp") {
|
||||||
!strings.Contains(message, clientsJSON+".tmp") {
|
t.Fatalf("alerts waiting %+v, want a file_error alert for %s, saying %q "+
|
||||||
t.Fatalf("alerts waiting %+v, want a file_error alert for clients.json, "+
|
"and naming its temporary file", waiting, file.name, file.reason)
|
||||||
"naming its temporary file", waiting)
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// The next write fails too, within the cooldown, which holds it back.
|
// The next writes fail too, within the cooldown, which holds them back.
|
||||||
time.Sleep(time.Minute)
|
time.Sleep(time.Minute)
|
||||||
synctest.Wait()
|
synctest.Wait()
|
||||||
|
|
||||||
waiting = params.Alerts.Snapshot().Waiting[alerts.DestinationWebhook]
|
waiting = params.Alerts.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||||
if len(waiting) != 1 || params.Alerts.Suppressed() != 1 {
|
if len(waiting) != 2 || params.Alerts.Suppressed() != 2 {
|
||||||
t.Errorf("%d alerts wait and %d are held back, want 1 and 1",
|
t.Errorf("%d alerts wait and %d are held back, want 2 and 2",
|
||||||
len(waiting), params.Alerts.Suppressed())
|
len(waiting), params.Alerts.Suppressed())
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
@@ -1461,6 +1549,15 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
|
|||||||
wantEqual(t, lookupsJSON, params.GeoJS.Snapshot(),
|
wantEqual(t, lookupsJSON, params.GeoJS.Snapshot(),
|
||||||
[]lookup.Answer{{Client: client, Country: "FR", Answered: midnight()}})
|
[]lookup.Answer{{Client: client, Country: "FR", Answered: midnight()}})
|
||||||
|
|
||||||
|
edit(t, dir, routeWatchJSON, `{"version": 1, "answers": [`+
|
||||||
|
`{"client": "198.51.100.7/32", "netblock": "198.51.100.0/24", "asn": 64497, `+
|
||||||
|
`"answered": "2026-10-06T00:00:00Z"}]}`)
|
||||||
|
wantTakenIn(t, lines, dir, routeWatchJSON)
|
||||||
|
wantEqual(t, routeWatchJSON, params.RouteWatch.Snapshot(), []routewatch.Answer{{
|
||||||
|
Client: client, Netblock: netip.MustParsePrefix("198.51.100.0/24"), ASN: 64497,
|
||||||
|
Answered: midnight(),
|
||||||
|
}})
|
||||||
|
|
||||||
// A netblock with bits past its length is read as the netblock it is
|
// A netblock with bits past its length is read as the netblock it is
|
||||||
// in.
|
// in.
|
||||||
edit(t, dir, alertsJSON, `{"version": 1, "cooldowns": [{"event": "ban", `+
|
edit(t, dir, alertsJSON, `{"version": 1, "cooldowns": [{"event": "ban", `+
|
||||||
@@ -1527,12 +1624,13 @@ func TestEditOfEachReputationFileTakenIn(t *testing.T) {
|
|||||||
wantEqual(t, abuseIPDBScoresJSON, params.AbuseIPDB.Snapshot(),
|
wantEqual(t, abuseIPDBScoresJSON, params.AbuseIPDB.Snapshot(),
|
||||||
[]reputation.Score{{Client: client, Score: 80, Fetched: midnight()}})
|
[]reputation.Score{{Client: client, Score: 80, Fetched: midnight()}})
|
||||||
|
|
||||||
edit(t, dir, abuseIPDBChecksJSON, `{"version": 1, "day": "2026-10-06", "spent": 9}`)
|
edit(t, dir, abuseIPDBChecksSpentJSON,
|
||||||
wantTakenIn(t, lines, dir, abuseIPDBChecksJSON)
|
`{"version": 1, "day": "2026-10-06", "spent": 9}`)
|
||||||
|
wantTakenIn(t, lines, dir, abuseIPDBChecksSpentJSON)
|
||||||
|
|
||||||
checks := reputation.ChecksSpent{Day: civil.DateOf(midnight()), Spent: 9}
|
checks := reputation.ChecksSpent{Day: civil.DateOf(midnight()), Spent: 9}
|
||||||
if got := params.AbuseIPDB.ChecksSpent(); got != checks {
|
if got := params.AbuseIPDB.ChecksSpent(); got != checks {
|
||||||
t.Errorf("%s taken in as %+v, want %+v", abuseIPDBChecksJSON, got, checks)
|
t.Errorf("%s taken in as %+v, want %+v", abuseIPDBChecksSpentJSON, got, checks)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1558,10 +1656,10 @@ func TestTimesEditedWithAnOffsetWrittenBackInUTC(t *testing.T) {
|
|||||||
ahead := time.FixedZone("UTC+2", 2*60*60)
|
ahead := time.FixedZone("UTC+2", 2*60*60)
|
||||||
written := map[string]string{}
|
written := map[string]string{}
|
||||||
|
|
||||||
// abuseipdb-checks.json holds a day, but no time.
|
// abuseipdb-checks-spent.json holds a day, but no time.
|
||||||
for _, name := range []string{
|
for _, name := range []string{
|
||||||
bansJSON, clientsJSON, alertsJSON, lookupsJSON, listsJSON, dnsblJSON,
|
bansJSON, clientsJSON, alertsJSON, lookupsJSON, listsJSON, dnsblJSON,
|
||||||
abuseIPDBScoresJSON,
|
abuseIPDBScoresJSON, routeWatchJSON,
|
||||||
} {
|
} {
|
||||||
written[name] = readFile(t, filepath.Join(dir, name))
|
written[name] = readFile(t, filepath.Join(dir, name))
|
||||||
edited := stamp.ReplaceAllStringFunc(written[name], func(quoted string) string {
|
edited := stamp.ReplaceAllStringFunc(written[name], func(quoted string) string {
|
||||||
@@ -1823,8 +1921,8 @@ func TestBrokenEditSetAsideAtTheNextWrite(t *testing.T) {
|
|||||||
t.Errorf("alerts waiting %+v, want a file_error alert for %s", waiting, path+".bad")
|
t.Errorf("alerts waiting %+v, want a file_error alert for %s", waiting, path+".bad")
|
||||||
}
|
}
|
||||||
|
|
||||||
wantFiles(t, dir, abuseIPDBChecksJSON, abuseIPDBScoresJSON, alertsJSON, bansJSON,
|
wantFiles(t, dir, abuseIPDBChecksSpentJSON, abuseIPDBScoresJSON, alertsJSON, bansJSON,
|
||||||
bansJSON+".bad", clientsJSON, dnsblJSON, listsJSON, lookupsJSON)
|
bansJSON+".bad", clientsJSON, dnsblJSON, listsJSON, lookupsJSON, routeWatchJSON)
|
||||||
|
|
||||||
if got := readFile(t, path+".bad"); got != broken {
|
if got := readFile(t, path+".bad"); got != broken {
|
||||||
t.Errorf("bans.json.bad holds\n%s\nwant the edit", got)
|
t.Errorf("bans.json.bad holds\n%s\nwant the edit", got)
|
||||||
@@ -1947,16 +2045,73 @@ func TestDirectoryThatCannotBeWatchedIsLogged(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestCacheFileEditedInACacheDirectoryApartTakenIn(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
stateDir, cacheDir := t.TempDir(), t.TempDir()
|
||||||
|
params := newParams(stateDir)
|
||||||
|
params.CacheDir = cacheDir
|
||||||
|
lines := logInto(¶ms)
|
||||||
|
watch(t, load(t, params), lines)
|
||||||
|
|
||||||
|
edit(t, cacheDir, lookupsJSON, `{"version": 1, "lookups": []}`)
|
||||||
|
wantTakenIn(t, lines, cacheDir, lookupsJSON)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDirectoryThatCannotBeWatchedLeavesTheOtherWatched(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
// gone is the directory removed before Watch runs, and failed what
|
||||||
|
// the log says of it. edited is a file in the other directory.
|
||||||
|
gone, failed, edited string
|
||||||
|
}{
|
||||||
|
{"state", "cannot watch the state files for edits", lookupsJSON},
|
||||||
|
{"cache", "cannot watch the cache files for edits", bansJSON},
|
||||||
|
} {
|
||||||
|
t.Run(tc.gone, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
stateDir, cacheDir := t.TempDir(), t.TempDir()
|
||||||
|
params := newParams(stateDir)
|
||||||
|
params.CacheDir = cacheDir
|
||||||
|
lines := logInto(¶ms)
|
||||||
|
files := load(t, params)
|
||||||
|
|
||||||
|
gone, other := stateDir, cacheDir
|
||||||
|
if tc.gone == "cache" {
|
||||||
|
gone, other = cacheDir, stateDir
|
||||||
|
}
|
||||||
|
|
||||||
|
err := os.Remove(gone)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("remove: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
run(t, files.Watch)
|
||||||
|
|
||||||
|
line := lines.waitFor(t, tc.failed)
|
||||||
|
if line["level"] != "ERROR" || line["directory"] != gone {
|
||||||
|
t.Errorf("logged %v, want an error naming %s", line, gone)
|
||||||
|
}
|
||||||
|
|
||||||
|
lines.waitFor(t, watching)
|
||||||
|
edit(t, other, tc.edited, `{"version": 1}`)
|
||||||
|
wantTakenIn(t, lines, other, tc.edited)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// midnight is the time of the tests' clock.
|
// midnight is the time of the tests' clock.
|
||||||
func midnight() time.Time {
|
func midnight() time.Time {
|
||||||
return time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
|
return time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
|
||||||
}
|
}
|
||||||
|
|
||||||
// newParams returns Params for the state and cache files, both in dir,
|
// newParams returns Params for the state and cache files, both in dir,
|
||||||
// with parts that hold nothing yet. GeoJS is never asked, the lists, two
|
// with parts that hold nothing yet. GeoJS and routewatch are never asked,
|
||||||
// blocklists, are never fetched, and the alerts, at most two an hour, are
|
// the lists, two blocklists, are never fetched, and the alerts, at most two
|
||||||
// never sent. The anomaly counters count the scopes fill counts, with
|
// an hour, are never sent. The anomaly counters count the scopes fill
|
||||||
// thresholds fill does not reach.
|
// counts, with thresholds fill does not reach.
|
||||||
func newParams(dir string) state.Params {
|
func newParams(dir string) state.Params {
|
||||||
discard := slog.New(slog.DiscardHandler)
|
discard := slog.New(slog.DiscardHandler)
|
||||||
m := metrics.New(1, "app")
|
m := metrics.New(1, "app")
|
||||||
@@ -1997,6 +2152,9 @@ func newParams(dir string) state.Params {
|
|||||||
MinScore: 75, DailyBudget: 900, CacheTTL: 24 * time.Hour, Timeout: time.Second,
|
MinScore: 75, DailyBudget: 900, CacheTTL: 24 * time.Hour, Timeout: time.Second,
|
||||||
Now: midnight, ProcessLog: discard, Alerts: queue,
|
Now: midnight, ProcessLog: discard, Alerts: queue,
|
||||||
}),
|
}),
|
||||||
|
RouteWatch: routewatch.New(routewatch.Params{
|
||||||
|
URL: "http://routewatch.example/", Now: midnight, ProcessLog: discard,
|
||||||
|
}),
|
||||||
Alerts: queue,
|
Alerts: queue,
|
||||||
Anomalies: anomaly.New(anomaly.Params{
|
Anomalies: anomaly.New(anomaly.Params{
|
||||||
Net: anomaly.Thresholds{RequestsPerMinute: 1000},
|
Net: anomaly.Thresholds{RequestsPerMinute: 1000},
|
||||||
@@ -2023,10 +2181,11 @@ func office() netip.Prefix {
|
|||||||
// for a clear sign of attack and one for CrowdSec's decision, clients
|
// for a clear sign of attack and one for CrowdSec's decision, clients
|
||||||
// with counts and histories,
|
// with counts and histories,
|
||||||
// GeoJS answers, the blocklists' last tries and the copy of one, two
|
// GeoJS answers, the blocklists' last tries and the copy of one, two
|
||||||
// verdicts of a DNSBL zone, and the AbuseIPDB checks spent today with two
|
// verdicts of a DNSBL zone, the AbuseIPDB checks spent today with two
|
||||||
// scores, as filledListsJSON, filledDNSBLJSON, filledAbuseIPDBScoresJSON
|
// scores, and two answers of routewatch, as filledListsJSON,
|
||||||
// and filledAbuseIPDBChecksJSON hold them, and alerts and anomaly
|
// filledDNSBLJSON, filledAbuseIPDBScoresJSON, filledAbuseIPDBChecksSpentJSON
|
||||||
// counters, as filledAlertsJSON holds them, into the parts of params.
|
// and filledRouteWatchJSON hold them, and alerts and anomaly counters, as
|
||||||
|
// filledAlertsJSON holds them, into the parts of params.
|
||||||
func fill(params state.Params) {
|
func fill(params state.Params) {
|
||||||
now := midnight()
|
now := midnight()
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
@@ -2089,6 +2248,7 @@ func fill(params state.Params) {
|
|||||||
params.AbuseIPDB.LoadChecksSpent(reputation.ChecksSpent{
|
params.AbuseIPDB.LoadChecksSpent(reputation.ChecksSpent{
|
||||||
Day: civil.DateOf(now), Spent: 3,
|
Day: civil.DateOf(now), Spent: 3,
|
||||||
})
|
})
|
||||||
|
fillRouteWatch(params.RouteWatch)
|
||||||
|
|
||||||
// An alert waiting, a repeat of it the cooldown holds back, another
|
// An alert waiting, a repeat of it the cooldown holds back, another
|
||||||
// alert waiting, and one past the two an hour, for the hour's summary.
|
// alert waiting, and one past the two an hour, for the hour's summary.
|
||||||
@@ -2114,6 +2274,22 @@ func fill(params state.Params) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// fillRouteWatch puts the two answers filledRouteWatchJSON holds into
|
||||||
|
// routeWatch: a route, and an address without a live route.
|
||||||
|
func fillRouteWatch(routeWatch *routewatch.Client) {
|
||||||
|
routeWatch.Load([]routewatch.Answer{
|
||||||
|
{
|
||||||
|
Client: netip.MustParsePrefix("203.0.113.9/32"),
|
||||||
|
Netblock: netip.MustParsePrefix("203.0.113.0/24"), ASN: 64496,
|
||||||
|
Answered: midnight().Add(-time.Minute),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Client: netip.MustParsePrefix("2001:db8::/64"),
|
||||||
|
Answered: midnight().Add(-2 * time.Minute),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
// permanentBan is the ban permanentBansJSON holds.
|
// permanentBan is the ban permanentBansJSON holds.
|
||||||
func permanentBan() bans.Ban {
|
func permanentBan() bans.Ban {
|
||||||
score := int64(100)
|
score := int64(100)
|
||||||
@@ -2350,8 +2526,8 @@ func wantRefused(t *testing.T, file, content, want string) {
|
|||||||
|
|
||||||
// wantSetAside writes content to the cache file named file in a new
|
// wantSetAside writes content to the cache file named file in a new
|
||||||
// directory, and checks that Load starts all the same, with the cache
|
// directory, and checks that Load starts all the same, with the cache
|
||||||
// empty, and sets the file aside, logging an error that is the file's
|
// empty, and sets the file aside, raising a file_error alert and logging
|
||||||
// path and then starts with want.
|
// an error that is the file's path and then starts with want.
|
||||||
func wantSetAside(t *testing.T, file, content, want string) {
|
func wantSetAside(t *testing.T, file, content, want string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -2367,15 +2543,23 @@ func wantSetAside(t *testing.T, file, content, want string) {
|
|||||||
lines := logInto(¶ms)
|
lines := logInto(¶ms)
|
||||||
load(t, params)
|
load(t, params)
|
||||||
|
|
||||||
line := lines.waitFor(t, setAside)
|
line := lines.waitFor(t, cacheSetAside)
|
||||||
if message, _ := line["error"].(string); line["file"] != path+".bad" ||
|
if message, _ := line["error"].(string); line["file"] != path+".bad" ||
|
||||||
!strings.HasPrefix(message, path+want) {
|
!strings.HasPrefix(message, path+want) {
|
||||||
t.Errorf("set aside with %v, want %s.bad and an error starting %s%s", line, path,
|
t.Errorf("set aside with %v, want %s.bad and an error starting %s%s", line, path,
|
||||||
path, want)
|
path, want)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
waiting := params.Alerts.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||||
|
if len(waiting) != 1 || waiting[0].Event != alerts.EventFileError ||
|
||||||
|
waiting[0].Reason != cacheSetAside {
|
||||||
|
t.Errorf("alerts waiting %+v, want a file_error alert saying %q", waiting,
|
||||||
|
cacheSetAside)
|
||||||
|
}
|
||||||
|
|
||||||
if len(params.GeoJS.Snapshot()) != 0 || len(params.Lists.Snapshot()) != 0 ||
|
if len(params.GeoJS.Snapshot()) != 0 || len(params.Lists.Snapshot()) != 0 ||
|
||||||
len(params.DNSBL.Snapshot()) != 0 || len(params.AbuseIPDB.Snapshot()) != 0 {
|
len(params.DNSBL.Snapshot()) != 0 || len(params.AbuseIPDB.Snapshot()) != 0 ||
|
||||||
|
len(params.RouteWatch.Snapshot()) != 0 {
|
||||||
t.Error("a cache holds what a file set aside held")
|
t.Error("a cache holds what a file set aside held")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -253,6 +253,7 @@ func loadStateFiles(
|
|||||||
Lists: server.Lists,
|
Lists: server.Lists,
|
||||||
DNSBL: server.DNSBL,
|
DNSBL: server.DNSBL,
|
||||||
AbuseIPDB: server.AbuseIPDB,
|
AbuseIPDB: server.AbuseIPDB,
|
||||||
|
RouteWatch: server.RouteWatch,
|
||||||
Alerts: alertQueue,
|
Alerts: alertQueue,
|
||||||
Anomalies: server.Anomalies,
|
Anomalies: server.Anomalies,
|
||||||
Now: now,
|
Now: now,
|
||||||
|
|||||||
+145
-38
@@ -7,6 +7,7 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
|
"path/filepath"
|
||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -16,6 +17,7 @@ import (
|
|||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
@@ -216,93 +218,199 @@ func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRequestsOverTheHeaderLimitsCountTowardTheRateLimits(t *testing.T) {
|
func TestTheRateLimitsDoNotCountRequestsOverTheHeaderLimits(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
s := startWAF(t, "", map[string]string{rateLimitPerMinute: "1"})
|
s := startWAF(t, "", map[string]string{rateLimitPerMinute: "1"})
|
||||||
ask := func() answer {
|
|
||||||
|
for range 2 {
|
||||||
req := s.question(siteA, client, "/")
|
req := s.question(siteA, client, "/")
|
||||||
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
||||||
|
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
|
||||||
return do(t, req)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
wantDisposition(t, ask(), tooLarge, waf.DispositionDeny)
|
wantDisposition(t, s.ask(client, "/"), http.StatusOK, waf.DispositionAllow)
|
||||||
wantDisposition(t, ask(), http.StatusForbidden, waf.DispositionDeny)
|
|
||||||
|
|
||||||
lines := s.out.requestLines(t, 2)
|
|
||||||
wantLine(t, lines[1], http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestTheHeaderLimitsComeAfterTheTrapPathsAndTheRuleFiles(t *testing.T) {
|
func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
// The lookup database places client in Germany.
|
||||||
|
lookupDB := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
|
||||||
|
lookuptest.Write(t, lookupDB, map[string]lookuptest.Network{
|
||||||
|
client + "/32": {ASN: asnDE, ASName: asNameDE, Country: "DE"},
|
||||||
|
})
|
||||||
|
|
||||||
for _, tc := range []struct {
|
for _, tc := range []struct {
|
||||||
name, path, action string
|
name string
|
||||||
|
env map[string]string
|
||||||
|
path string
|
||||||
|
// prepare gets the server ready, at now, before the question.
|
||||||
|
prepare func(t *testing.T, server *waf.Server, now time.Time)
|
||||||
}{
|
}{
|
||||||
{"a trap path", "/wp-login.php", requestlog.ActionTemporarilyBanned},
|
{"SWW_DENY_NETS", map[string]string{denyNets: client}, "/", leaveAsItIs},
|
||||||
{"a block rule", "/blocked", requestlog.ActionBlockedByBlockRule},
|
{"a ban", nil, "/", banClient},
|
||||||
|
{
|
||||||
|
"the country list", map[string]string{
|
||||||
|
lookupSource: fileSource, lookupDBPath: lookupDB, allowedCountries: "FR",
|
||||||
|
},
|
||||||
|
"/", leaveAsItIs,
|
||||||
|
},
|
||||||
|
{"a blocklist", map[string]string{blocklistURLs: dropURL}, "/", listClient},
|
||||||
|
{
|
||||||
|
"the CrowdSec decision list",
|
||||||
|
map[string]string{crowdSecURL: lapi, crowdSecKey: bouncerKey}, "/", decideOnClient,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"a DNS blocklist", map[string]string{
|
||||||
|
dnsblZones: dnsblZone, dnsblResolver: noResolver, reputationAction: actionDeny,
|
||||||
|
},
|
||||||
|
"/", func(_ *testing.T, server *waf.Server, _ time.Time) {
|
||||||
|
loadVerdicts(server, map[string][]string{client: {dnsblZone}})
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"AbuseIPDB",
|
||||||
|
map[string]string{abuseIPDBKey: accountKey, reputationAction: actionDeny},
|
||||||
|
"/", func(_ *testing.T, server *waf.Server, _ time.Time) {
|
||||||
|
// Over SWW_ABUSEIPDB_MIN_SCORE, 75 by default.
|
||||||
|
loadScores(server, map[string]int64{client: 80})
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"SWW_TAG_DENY", map[string]string{tagLists: "vpn=" + vpnList, tagDeny: "vpn"},
|
||||||
|
"/", func(t *testing.T, server *waf.Server, _ time.Time) {
|
||||||
|
t.Helper()
|
||||||
|
loadLists(t, server, map[string][]string{vpnList: {client}})
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{"the rate limits", map[string]string{rateLimitPerMinute: "1"}, "/", countClient},
|
||||||
|
{
|
||||||
|
"a trap path", map[string]string{trapPaths: trapPathList}, "/wp-login.php",
|
||||||
|
leaveAsItIs,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"a block rule", map[string]string{rulesDir: writeRules(t, testRules)}, "/blocked",
|
||||||
|
leaveAsItIs,
|
||||||
|
},
|
||||||
} {
|
} {
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
s := startWAF(t, "", map[string]string{
|
s := startWAF(t, "", tc.env)
|
||||||
trapPaths: trapPathList, rulesDir: writeRules(t, testRules),
|
tc.prepare(t, s.server, s.clock.Now())
|
||||||
})
|
|
||||||
req := s.question(siteA, client, tc.path)
|
req := s.question(siteA, client, tc.path)
|
||||||
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
||||||
|
|
||||||
wantDisposition(t, do(t, req), http.StatusForbidden, waf.DispositionDeny)
|
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
|
||||||
wantLine(t, s.out.requestLine(t), http.StatusForbidden, tc.action)
|
wantLine(t, s.out.requestLine(t), tooLarge, requestlog.ActionRequestHeaderTooLarge)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestARequestOverTheHeaderLimitsRunsNoOtherCheck(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// Each check after the header limits leaves a mark on the log line of a
|
||||||
|
// request it reads: a tag, routewatch_tags_missing, a rule id, the
|
||||||
|
// counts, or the Core Rule Set's time.
|
||||||
|
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
|
||||||
|
lookuptest.Write(t, path, map[string]lookuptest.Network{
|
||||||
|
client + "/32": {ASN: asnDE, ASName: asNameDE, Country: "DE"},
|
||||||
|
})
|
||||||
|
s := startWAF(t, "", map[string]string{
|
||||||
|
lookupSource: fileSource, lookupDBPath: path,
|
||||||
|
tagLists: "vpn=" + vpnList,
|
||||||
|
routeWatchURL: startRouteWatch(t, map[string]standInRoute{client: {prefixDE, 64496}}),
|
||||||
|
dnsblZones: dnsblZone, reputationAction: actionLog,
|
||||||
|
rulesDir: writeRules(t, "noted path log ^/\n"),
|
||||||
|
wafMode: block,
|
||||||
|
})
|
||||||
|
loadLists(t, s.server, map[string][]string{vpnList: {client}})
|
||||||
|
loadVerdicts(s.server, map[string][]string{client: {dnsblZone}})
|
||||||
|
|
||||||
|
over := s.question(siteA, client, "/")
|
||||||
|
over.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
||||||
|
wantDisposition(t, do(t, over), tooLarge, waf.DispositionDeny)
|
||||||
|
|
||||||
|
within := s.question(siteA, client, "/")
|
||||||
|
within.Header.Set(cookieHeader, cookies(cookieLimit))
|
||||||
|
wantDisposition(t, do(t, within), http.StatusOK, waf.DispositionAllow)
|
||||||
|
|
||||||
|
lines := s.out.requestLines(t, 2)
|
||||||
|
|
||||||
|
wantTags(t, lines[0])
|
||||||
|
wantTagsMissing(t, lines[0], "")
|
||||||
|
|
||||||
|
if lines[0].RuleIDs != nil || lines[0].Counts != (ratelimit.Counts{}) ||
|
||||||
|
lines[0].DurationCoreRuleSetMS != nil {
|
||||||
|
t.Errorf("rule ids %v, counts %+v and Core Rule Set time %v, want none",
|
||||||
|
lines[0].RuleIDs, lines[0].Counts, lines[0].DurationCoreRuleSetMS)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The request within the limits shows that each check leaves its mark.
|
||||||
|
wantTags(t, lines[1],
|
||||||
|
"asn=64496", "country=DE", "dnsbl="+dnsblZone, "exploitsignature", "vpn")
|
||||||
|
wantTagsMissing(t, lines[1], "no_answer_yet")
|
||||||
|
|
||||||
|
if lines[1].RuleIDs == nil || lines[1].Counts.Minute != 1 ||
|
||||||
|
lines[1].DurationCoreRuleSetMS == nil {
|
||||||
|
t.Errorf("rule ids %v, counts %+v and Core Rule Set time %v, want each",
|
||||||
|
lines[1].RuleIDs, lines[1].Counts, lines[1].DurationCoreRuleSetMS)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) {
|
func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
// The client's next request takes it over the whole limit, so it does
|
||||||
|
// not wait.
|
||||||
geojsURL, _ := startGeoJS(t)
|
geojsURL, _ := startGeoJS(t)
|
||||||
s := startWAF(t, geojsURL, map[string]string{
|
s := startWAF(t, geojsURL, map[string]string{
|
||||||
unknownLimitPercent: "0", lookupTimeout: "1h",
|
unknownLimitPercent: "0", lookupTimeout: "1h", rateLimitPerMinute: "1",
|
||||||
})
|
})
|
||||||
req := s.question(siteA, fromDE, "/")
|
countClient(t, s.server, s.clock.Now())
|
||||||
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
|
||||||
|
|
||||||
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
|
wantDisposition(t, s.ask(fromDE, "/"), http.StatusForbidden, waf.DispositionDeny)
|
||||||
|
|
||||||
line := s.out.requestLine(t)
|
line := s.out.requestLine(t)
|
||||||
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
|
wantLine(t, line, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||||
|
|
||||||
if line.RateLimitPercent != nil {
|
if line.RateLimitPercent != nil {
|
||||||
t.Errorf("limit percentage %d, want none", *line.RateLimitPercent)
|
t.Errorf("limit percentage %d, want none", *line.RateLimitPercent)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The client is under no ban, and GeoJS places it in a country.
|
|
||||||
wantDisposition(t, s.ask(fromDE, "/"), http.StatusOK, waf.DispositionAllow)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(t *testing.T) {
|
func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
// The client's next request takes it over the whole limit, so it does
|
||||||
|
// not wait.
|
||||||
geojsURL, _ := startGeoJS(t)
|
geojsURL, _ := startGeoJS(t)
|
||||||
s := startWAF(t, geojsURL, map[string]string{unknownLimitPercent: "0"})
|
s := startWAF(t, geojsURL, map[string]string{
|
||||||
|
unknownLimitPercent: "0", rateLimitPerMinute: "1",
|
||||||
|
})
|
||||||
s.server.GeoJS.Load([]lookup.Answer{{
|
s.server.GeoJS.Load([]lookup.Answer{{
|
||||||
Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512",
|
Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512",
|
||||||
Answered: s.clock.Now(), Used: s.clock.Now(),
|
Answered: s.clock.Now(), Used: s.clock.Now(),
|
||||||
}})
|
}})
|
||||||
|
countClient(t, s.server, s.clock.Now())
|
||||||
|
|
||||||
req := s.question(siteA, client, "/")
|
wantDisposition(t, s.ask(client, "/"), http.StatusForbidden, waf.DispositionDeny)
|
||||||
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
|
||||||
|
|
||||||
wantDisposition(t, do(t, req), http.StatusForbidden, waf.DispositionDeny)
|
line := s.out.requestLine(t)
|
||||||
wantLine(t, s.out.requestLine(t), http.StatusForbidden,
|
wantLine(t, line, http.StatusForbidden, requestlog.ActionClientIsRateLimited)
|
||||||
requestlog.ActionClientIsRateLimited)
|
|
||||||
|
if line.RateLimitPercent == nil || *line.RateLimitPercent != 0 {
|
||||||
|
t.Errorf("limit percentage %v, want 0", line.RateLimitPercent)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestObserveModeWaitsForGeoJSForARequestOverTheHeaderLimits(t *testing.T) {
|
func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
geojsURL, _ := startGeoJS(t)
|
geojsURL, asked := startGeoJS(t)
|
||||||
s := startWAF(t, geojsURL, map[string]string{
|
s := startWAF(t, geojsURL, map[string]string{
|
||||||
mode: observe, addLookupHeaders: on, lookupTimeout: "1h",
|
mode: observe, addLookupHeaders: on, lookupTimeout: "1h",
|
||||||
})
|
})
|
||||||
@@ -312,10 +420,9 @@ func TestObserveModeWaitsForGeoJSForARequestOverTheHeaderLimits(t *testing.T) {
|
|||||||
got := do(t, req)
|
got := do(t, req)
|
||||||
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
|
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
|
||||||
|
|
||||||
asn, country := got.header.Get("X-Client-Asn"), got.header.Get("X-Client-Country")
|
if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" {
|
||||||
if asn != asnDE || country != "DE" {
|
t.Errorf("GeoJS was asked about %v, and the answer gave the AS number %q, "+
|
||||||
t.Errorf("answered with the AS number %q and the country %q, want %q and DE",
|
"want neither", asked(), got.header.Get("X-Client-Asn"))
|
||||||
asn, country, asnDE)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ import (
|
|||||||
|
|
||||||
// The CrowdSec settings, and the tests' engine, which is never asked: each
|
// The CrowdSec settings, and the tests' engine, which is never asked: each
|
||||||
// test puts in the copy of its decision list, at decisionsURL, that it
|
// test puts in the copy of its decision list, at decisionsURL, that it
|
||||||
// needs, as reputation.json would at start.
|
// needs, as lists.json would at start.
|
||||||
const (
|
const (
|
||||||
crowdSecURL = "SWW_CROWDSEC_LAPI_URL"
|
crowdSecURL = "SWW_CROWDSEC_LAPI_URL"
|
||||||
crowdSecKey = "SWW_CROWDSEC_LAPI_KEY"
|
crowdSecKey = "SWW_CROWDSEC_LAPI_KEY"
|
||||||
@@ -182,7 +182,7 @@ func TestObserveModeForwardsAClientTheCrowdSecDecisionListListsAndAlertsTheBan(
|
|||||||
|
|
||||||
// loadDecisions puts into server's lists the copy of the decision list at
|
// loadDecisions puts into server's lists the copy of the decision list at
|
||||||
// decisionsURL, answer, the engine's answer, fetched at fetched, as
|
// decisionsURL, answer, the engine's answer, fetched at fetched, as
|
||||||
// reputation.json would at start.
|
// lists.json would at start.
|
||||||
func loadDecisions(
|
func loadDecisions(
|
||||||
t *testing.T, server *waf.Server, fetched time.Time, answer string,
|
t *testing.T, server *waf.Server, fetched time.Time, answer string,
|
||||||
) {
|
) {
|
||||||
|
|||||||
@@ -10,22 +10,23 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The header limits: the most cookies and header fields a request may have
|
// The header limits: the most cookies and header fields a request may have.
|
||||||
// when the checks reach the Core Rule Set. Real clients never reach them.
|
// Real clients never reach them.
|
||||||
const (
|
const (
|
||||||
maxCookies = 50
|
maxCookies = 50
|
||||||
maxHeaderFields = 100
|
maxHeaderFields = 100
|
||||||
)
|
)
|
||||||
|
|
||||||
// headerLimits refuses a request over the header limits, with 431. Its figures
|
// headerLimits refuses a request with more than maxHeaderFields header fields
|
||||||
// are the request's header fields and cookies, and the limit of each.
|
// or more than maxCookies cookies, with 431. Its figures are the request's
|
||||||
|
// header fields and cookies, and the limit of each.
|
||||||
type headerLimits struct{}
|
type headerLimits struct{}
|
||||||
|
|
||||||
func (headerLimits) name() string { return "header_limits" }
|
func (headerLimits) name() string { return "header_limits" }
|
||||||
|
|
||||||
func (headerLimits) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
func (headerLimits) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||||
fields, cookies := rq.headerCounts()
|
fields, cookies := rq.headerCounts()
|
||||||
if !overHeaderLimits(fields, cookies) {
|
if fields <= maxHeaderFields && cookies <= maxCookies {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -38,13 +39,6 @@ func (headerLimits) check(_ context.Context, rq *request, _ time.Time) *verdict
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// overHeaderLimits reports whether a request with these counts of header fields
|
|
||||||
// and cookies, as headerCounts counts them, is over the header limits: it has
|
|
||||||
// more than maxHeaderFields fields or more than maxCookies cookies.
|
|
||||||
func overHeaderLimits(fields, cookies int) bool {
|
|
||||||
return fields > maxHeaderFields || cookies > maxCookies
|
|
||||||
}
|
|
||||||
|
|
||||||
// headerCounts returns the request's header fields, each value of a header sent
|
// headerCounts returns the request's header fields, each value of a header sent
|
||||||
// twice being one, and its cookies, each part of a Cookie header between
|
// twice being one, and its cookies, each part of a Cookie header between
|
||||||
// semicolons being one unless it is empty or only spaces and tabs. The Core
|
// semicolons being one unless it is empty or only spaces and tabs. The Core
|
||||||
|
|||||||
@@ -41,8 +41,11 @@ const (
|
|||||||
|
|
||||||
// beforeLookup are the heuristics that run before the client's AS number and
|
// beforeLookup are the heuristics that run before the client's AS number and
|
||||||
// country are looked up, so that a client they refuse is not looked up.
|
// country are looked up, so that a client they refuse is not looked up.
|
||||||
|
// header_limits comes first: it only counts, and a request it refuses causes no
|
||||||
|
// other work.
|
||||||
func beforeLookup() []heuristic {
|
func beforeLookup() []heuristic {
|
||||||
return []heuristic{
|
return []heuristic{
|
||||||
|
headerLimits{},
|
||||||
tagLists{},
|
tagLists{},
|
||||||
routeWatchTags{},
|
routeWatchTags{},
|
||||||
denyNets{},
|
denyNets{},
|
||||||
@@ -64,7 +67,6 @@ func afterLookup() []heuristic {
|
|||||||
rateLimits{},
|
rateLimits{},
|
||||||
trapPaths{},
|
trapPaths{},
|
||||||
ruleFiles{},
|
ruleFiles{},
|
||||||
headerLimits{},
|
|
||||||
coreRuleSet{},
|
coreRuleSet{},
|
||||||
tagDeny{},
|
tagDeny{},
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -86,8 +86,8 @@ type Params struct {
|
|||||||
// Server is the server waf runs, with the parts whose state the state
|
// Server is the server waf runs, with the parts whose state the state
|
||||||
// files keep, the lookup database, nil unless SWW_LOOKUP_SOURCE is file, the
|
// files keep, the lookup database, nil unless SWW_LOOKUP_SOURCE is file, the
|
||||||
// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts,
|
// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts,
|
||||||
// AbuseIPDB's scores and checks spent, and the metrics. RouteWatch, nil while
|
// AbuseIPDB's scores and checks spent, routewatch's answers, and the metrics.
|
||||||
// SWW_ROUTEWATCH_URL is unset, keeps its answers in memory only.
|
// RouteWatch is nil while SWW_ROUTEWATCH_URL is unset.
|
||||||
type Server struct {
|
type Server struct {
|
||||||
*http.Server
|
*http.Server
|
||||||
|
|
||||||
|
|||||||
@@ -39,7 +39,7 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// The lists these tests name, which are never fetched: each test puts in
|
// The lists these tests name, which are never fetched: each test puts in
|
||||||
// the copies it needs, as reputation.json would at start.
|
// the copies it needs, as lists.json would at start.
|
||||||
const (
|
const (
|
||||||
dropURL = "https://lists.example/drop.txt"
|
dropURL = "https://lists.example/drop.txt"
|
||||||
torURL = "https://lists.example/tor.txt"
|
torURL = "https://lists.example/tor.txt"
|
||||||
@@ -374,7 +374,7 @@ const (
|
|||||||
|
|
||||||
// The DNSBL zones these tests name, which are never asked about the
|
// The DNSBL zones these tests name, which are never asked about the
|
||||||
// clients the tests send requests from: each test puts in the verdicts it
|
// clients the tests send requests from: each test puts in the verdicts it
|
||||||
// needs, as reputation.json would at start. A query a test does start is
|
// needs, as dnsbl.json would at start. A query a test does start is
|
||||||
// sent to noResolver, where nothing listens, so that none leaves the host.
|
// sent to noResolver, where nothing listens, so that none leaves the host.
|
||||||
const (
|
const (
|
||||||
dnsblZone = "dnsbl.example"
|
dnsblZone = "dnsbl.example"
|
||||||
@@ -1072,7 +1072,7 @@ func verdictsFetched() time.Time {
|
|||||||
|
|
||||||
// loadVerdicts puts into server's DNSBL, for each client listedBy names,
|
// loadVerdicts puts into server's DNSBL, for each client listedBy names,
|
||||||
// a verdict of each zone SWW_DNSBL_ZONES names, fetched at
|
// a verdict of each zone SWW_DNSBL_ZONES names, fetched at
|
||||||
// verdictsFetched, as reputation.json would at start: one that lists the
|
// verdictsFetched, as dnsbl.json would at start: one that lists the
|
||||||
// client from each zone listedBy gives for it, and one that does not from
|
// client from each zone listedBy gives for it, and one that does not from
|
||||||
// each other zone.
|
// each other zone.
|
||||||
func loadVerdicts(server *waf.Server, listedBy map[string][]string) {
|
func loadVerdicts(server *waf.Server, listedBy map[string][]string) {
|
||||||
@@ -1091,8 +1091,8 @@ func loadVerdicts(server *waf.Server, listedBy map[string][]string) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// loadScores puts into server's AbuseIPDB the score scores gives each
|
// loadScores puts into server's AbuseIPDB the score scores gives each
|
||||||
// client, an IPv4 address, fetched at verdictsFetched, as reputation.json
|
// client, an IPv4 address, fetched at verdictsFetched, as
|
||||||
// would at start.
|
// abuseipdb-scores.json would at start.
|
||||||
func loadScores(server *waf.Server, scores map[string]int64) {
|
func loadScores(server *waf.Server, scores map[string]int64) {
|
||||||
kept := make([]reputation.Score, 0, len(scores))
|
kept := make([]reputation.Score, 0, len(scores))
|
||||||
for client, score := range scores {
|
for client, score := range scores {
|
||||||
@@ -1116,7 +1116,7 @@ func wantAbuseIPDBChecks(t *testing.T, server *waf.Server, want int) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// loadLists puts copies of lists into server's lists, by URL, each with
|
// loadLists puts copies of lists into server's lists, by URL, each with
|
||||||
// its lines, fetched at listsFetched, as reputation.json would at start.
|
// its lines, fetched at listsFetched, as lists.json would at start.
|
||||||
func loadLists(t *testing.T, server *waf.Server, copies map[string][]string) {
|
func loadLists(t *testing.T, server *waf.Server, copies map[string][]string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
|
|||||||
+4
-6
@@ -143,10 +143,9 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
|
|
||||||
// refusedWhateverTheAnswer reports whether the request is one that does not wait
|
// refusedWhateverTheAnswer reports whether the request is one that does not wait
|
||||||
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
|
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
|
||||||
// its client: the heuristic crowdsec, header_limits, blocklists or rate_limits
|
// its client: the heuristic crowdsec, blocklists or rate_limits will refuse it,
|
||||||
// will refuse it, as a function in that heuristic's file says. In observe mode,
|
// as a function in that heuristic's file says. In observe mode, which refuses
|
||||||
// which refuses nothing, none is. rproxy lets a request through when
|
// nothing, none is. rproxy lets a request through when waf answers too late.
|
||||||
// waf answers too late.
|
|
||||||
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
|
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
|
||||||
if rq.h.config.Observe {
|
if rq.h.config.Observe {
|
||||||
return false
|
return false
|
||||||
@@ -154,8 +153,7 @@ func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
|
|||||||
|
|
||||||
_, decided := rq.crowdSecDecisionInForce(now)
|
_, decided := rq.crowdSecDecisionInForce(now)
|
||||||
|
|
||||||
return decided || overHeaderLimits(rq.headerCounts()) ||
|
return decided || rq.refusedByBlocklists(rq.h.lists.ListedBy(rq.client)) ||
|
||||||
rq.refusedByBlocklists(rq.h.lists.ListedBy(rq.client)) ||
|
|
||||||
rq.overAWholeRateLimit(now)
|
rq.overAWholeRateLimit(now)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user