Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c125375ac7 | ||
|
|
8c9f20d346 | ||
|
|
caa3079847 | ||
|
|
a2fcc00ded | ||
|
|
3c6b0d519f |
No files matched your search
@@ -329,7 +329,7 @@ Every setting the code reads, with its default:
|
|||||||
| `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
|
| `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
|
||||||
| `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
|
| `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
|
||||||
| `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
| `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
||||||
| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take, in milliseconds. |
|
| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, in milliseconds. The request to GeoJS goes on, and a later answer is kept. |
|
||||||
| `SWW_WAF_ADD_LOOKUP_HEADERS` | `true`, or `false` while the lookup source is `off` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
| `SWW_WAF_ADD_LOOKUP_HEADERS` | `true`, or `false` while the lookup source is `off` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
||||||
| `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
| `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
||||||
| `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
| `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
||||||
@@ -599,6 +599,11 @@ request reaches the app.
|
|||||||
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||||
prefix.
|
prefix.
|
||||||
|
|
||||||
|
Steps 12 to 14 only read the request. While the lookup waits for GeoJS, they run
|
||||||
|
at the same time as steps 6 to 11. The first of the two sides to refuse the
|
||||||
|
request ends the checks, and `waf` answers at once. Steps 12 to 14 have 500 ms
|
||||||
|
together.
|
||||||
|
|
||||||
`rproxy` takes the rest, with `waf` counting at the end:
|
`rproxy` takes the rest, with `waf` counting at the end:
|
||||||
|
|
||||||
17. The request is passed to the app, streaming.
|
17. The request is passed to the app, streaming.
|
||||||
@@ -611,9 +616,10 @@ A refused visitor is answered with `SWW_WAF_BAN_RESPONSE`. A `block` rule or a
|
|||||||
Core Rule Set match answers `403` and bans nobody, since the Core Rule Set has
|
Core Rule Set match answers `403` and bans nobody, since the Core Rule Set has
|
||||||
false positives; such refusals count toward `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE`
|
false positives; such refusals count toward `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE`
|
||||||
instead. A request over the header limits of step 3 is answered `414` for a
|
instead. A request over the header limits of step 3 is answered `414` for a
|
||||||
request line over 8 KiB and `431` otherwise. It bans nobody either. `rproxy`
|
request line over 8 KiB and `431` otherwise. It bans nobody either. Nor does a
|
||||||
itself answers `405` to a method it does not pass on, and the status for a size
|
request that steps 12 to 14 have not finished within 500 ms, which is answered
|
||||||
or time limit, such as `413` for a body too large.
|
`503`. `rproxy` itself answers `405` to a method it does not pass on, and the
|
||||||
|
status for a size or time limit, such as `413` for a body too large.
|
||||||
|
|
||||||
### Clients and bans
|
### Clients and bans
|
||||||
|
|
||||||
@@ -749,9 +755,9 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
|
|||||||
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
|
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
|
||||||
endpoints.
|
endpoints.
|
||||||
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
|
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
|
||||||
`internal/coreruleset`, `internal/lookup`, `internal/reputation`,
|
`internal/coreruleset`, `internal/geojs`, `internal/lookup`,
|
||||||
`internal/routewatch`, `internal/anomaly` and `internal/alerts`: one check or
|
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and
|
||||||
source each.
|
`internal/alerts`: one check or source each.
|
||||||
- `internal/state`, `internal/requestlog`, `internal/remotelog` and
|
- `internal/state`, `internal/requestlog`, `internal/remotelog` and
|
||||||
`internal/metrics`: the state and cache files, the log lines, the syslog
|
`internal/metrics`: the state and cache files, the log lines, the syslog
|
||||||
sender and the metrics.
|
sender and the metrics.
|
||||||
|
|||||||
@@ -87,8 +87,10 @@ never as a time.
|
|||||||
change, and tells which rules a request matches.
|
change, and tells which rules a request matches.
|
||||||
- `internal/coreruleset`: the Core Rule Set, with the five changes and the
|
- `internal/coreruleset`: the Core Rule Set, with the five changes and the
|
||||||
apps' fixes listed under "Attack detection", as Coraza directives.
|
apps' fixes listed under "Attack detection", as Coraza directives.
|
||||||
- `internal/lookup`: looks clients up through GeoJS or in the lookup
|
- `internal/geojs`: looks clients up through GeoJS and keeps the answers. It
|
||||||
database.
|
imports no other package of this repository: `waf` gives it what it needs
|
||||||
|
to count and report failures.
|
||||||
|
- `internal/lookup`: looks clients up in the lookup database.
|
||||||
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
|
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
|
||||||
AbuseIPDB in the background, and keeps their answers.
|
AbuseIPDB in the background, and keeps their answers.
|
||||||
- `internal/routewatch`: asks a routewatch instance about clients in the
|
- `internal/routewatch`: asks a routewatch instance about clients in the
|
||||||
@@ -171,7 +173,8 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in
|
|||||||
`country=` tag.
|
`country=` tag.
|
||||||
|
|
||||||
Otherwise the request goes on at once. A request that ends before the answer
|
Otherwise the request goes on at once. A request that ends before the answer
|
||||||
comes is logged without it.
|
comes is logged without it. While the request waits, steps 12 to 14 run at
|
||||||
|
the same time as steps 6 to 11 (see "Answering in time").
|
||||||
|
|
||||||
7. The country list (`countries`; see "Country list").
|
7. The country list (`countries`; see "Country list").
|
||||||
8. The reputation sources (see "Reputation"), in this order:
|
8. The reputation sources (see "Reputation"), in this order:
|
||||||
@@ -507,19 +510,25 @@ service that keeps its visitors' addresses on its own host.
|
|||||||
dropped from that table. The answer reaches the client's history and ban notes
|
dropped from that table. The answer reaches the client's history and ban notes
|
||||||
when it comes.
|
when it comes.
|
||||||
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is how long a visitor's request
|
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is how long a visitor's request
|
||||||
waits for its client's first answer, when a setting needs it (see step 6). It
|
waits for its client's first answer, when a setting needs it (see step 6). A
|
||||||
also bounds each request to GeoJS. A client whose answer has not come in time
|
client whose answer has not come in time counts as unknown until it comes, and
|
||||||
counts as unknown until it comes, and its later requests do not wait. Its
|
its later requests do not wait.
|
||||||
address is asked about again in the background.
|
- A request to GeoJS may take up to 10 seconds, however long visitors wait. It
|
||||||
|
goes on after the visitor's request ends, and its answer is kept. Say GeoJS
|
||||||
|
answers after 5 seconds. The visitor's request is decided after 800
|
||||||
|
milliseconds, without a country. A request from the same client 6 seconds
|
||||||
|
later gets the answer.
|
||||||
|
- Requests from one client that come together share one request to GeoJS.
|
||||||
- A client the lookup cannot place has an unknown country and AS number, which
|
- A client the lookup cannot place has an unknown country and AS number, which
|
||||||
the country list and the biased thresholds treat as described there. Nothing
|
the country list and the biased thresholds treat as described there. Nothing
|
||||||
else treats it differently.
|
else treats it differently.
|
||||||
- GeoJS publishes no rate limit, but its terms forbid "an excessive amount of
|
- GeoJS publishes no rate limit, but its terms forbid "an excessive amount of
|
||||||
API requests", judged by GeoJS alone, and let it block a caller. While GeoJS
|
API requests", judged by GeoJS alone, and let it block a caller. A request to
|
||||||
is slow, down or refusing `waf`, new clients count as unknown. GeoJS is then
|
GeoJS fails when it takes over 10 seconds or GeoJS answers with an error.
|
||||||
left alone for a second, and twice as long after each further failure, up to
|
While GeoJS is slow, down or refusing `waf`, new clients count as unknown.
|
||||||
five minutes. Each failure raises a `source_failure` alert (see "Alerting"). A
|
After a failure GeoJS is left alone for a second, and twice as long after each
|
||||||
service that cannot accept this uses the file.
|
further failure, up to five minutes. Each failure raises a `source_failure`
|
||||||
|
alert (see "Alerting"). A service that cannot accept this uses the file.
|
||||||
- The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`,
|
- The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`,
|
||||||
`as_name` and `country_code` fields are read. The operator downloads it with a
|
`as_name` and `country_code` fields are read. The operator downloads it with a
|
||||||
free IPinfo account and mounts it read-only; `waf` never fetches it and holds
|
free IPinfo account and mounts it read-only; `waf` never fetches it and holds
|
||||||
@@ -2371,8 +2380,9 @@ runs no check, writes no request log line and counts nothing in the metrics.
|
|||||||
|
|
||||||
### The answer
|
### The answer
|
||||||
|
|
||||||
`waf` runs steps 3 to 16 of "What happens to a request", in that order. Every
|
`waf` runs steps 3 to 16 of "What happens to a request", in that order, apart
|
||||||
answer but the last below carries `X-Sww-Disposition`.
|
from the steps that run at the same time (see "Answering in time"). Every answer
|
||||||
|
but the last below carries `X-Sww-Disposition`.
|
||||||
|
|
||||||
- `X-Sww-Disposition: allow`, status `200`, no body. `rproxy` passes the request
|
- `X-Sww-Disposition: allow`, status `200`, no body. `rproxy` passes the request
|
||||||
to the app. While `SWW_WAF_ADD_LOOKUP_HEADERS` is `true`, the answer also
|
to the app. While `SWW_WAF_ADD_LOOKUP_HEADERS` is `true`, the answer also
|
||||||
@@ -2384,6 +2394,8 @@ answer but the last below carries `X-Sww-Disposition`.
|
|||||||
- `SWW_WAF_BAN_RESPONSE`, for a refusal;
|
- `SWW_WAF_BAN_RESPONSE`, for a refusal;
|
||||||
- `403`, for a `block` rule or the Core Rule Set;
|
- `403`, for a `block` rule or the Core Rule Set;
|
||||||
- `414` or `431`, for a request over the header limits;
|
- `414` or `431`, for a request over the header limits;
|
||||||
|
- `503`, for a request that steps 12 to 14 have not finished within 500
|
||||||
|
milliseconds;
|
||||||
- `404`, for a path under `/_sww/` that passes the checks, so that the app
|
- `404`, for a path under `/_sww/` that passes the checks, so that the app
|
||||||
never sees that prefix.
|
never sees that prefix.
|
||||||
- `X-Sww-Disposition: close`, status `200`, no body. This takes the place of a
|
- `X-Sww-Disposition: close`, status `200`, no body. This takes the place of a
|
||||||
@@ -2404,15 +2416,22 @@ The Core Rule Set inspects the body a question carries only while
|
|||||||
|
|
||||||
`rproxy` stops waiting for `waf` after `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, by
|
`rproxy` stops waiting for `waf` after `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, by
|
||||||
default 2 seconds, and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` then
|
default 2 seconds, and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` then
|
||||||
decides. The longest a question waits on an outside source is
|
decides. Two things in a decision take time:
|
||||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds, well
|
|
||||||
within that time.
|
- the wait for GeoJS, at most `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by
|
||||||
|
default 800 milliseconds;
|
||||||
|
- the trap paths, the rule files and the Core Rule Set, which have 500
|
||||||
|
milliseconds.
|
||||||
|
|
||||||
|
They run at the same time, so `waf` answers within the longer of the two, well
|
||||||
|
within `rproxy`'s wait.
|
||||||
|
|
||||||
GeoJS is the only outside source a question can wait on. A question waits only
|
GeoJS is the only outside source a question can wait on. A question waits only
|
||||||
when step 6 says so, and for at most
|
when step 6 says so, and for at most
|
||||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds. A
|
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds. A
|
||||||
client without an answer by then counts as unknown. No question waits on these,
|
client without an answer by then counts as unknown. The setting bounds only the
|
||||||
which run in the background:
|
question's wait: the request to GeoJS goes on, and its answer is kept for later
|
||||||
|
questions. No question waits on these, which run in the background:
|
||||||
|
|
||||||
- the DNS blocklists and AbuseIPDB;
|
- the DNS blocklists and AbuseIPDB;
|
||||||
- fetching the blocklists, the tag lists and the file
|
- fetching the blocklists, the tag lists and the file
|
||||||
@@ -2437,10 +2456,38 @@ never wait. These can still wait for GeoJS:
|
|||||||
- a country list, which needs the answer;
|
- a country list, which needs the answer;
|
||||||
- a rate limit a percentage has lowered, which needs the answer;
|
- a rate limit a percentage has lowered, which needs the answer;
|
||||||
- a DNS blocklist or AbuseIPDB, while `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` is
|
- a DNS blocklist or AbuseIPDB, while `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` is
|
||||||
`deny`;
|
`deny`.
|
||||||
- a trap path;
|
|
||||||
- a rule file;
|
The trap paths, the rule files and the Core Rule Set, steps 12 to 14, run during
|
||||||
- the Core Rule Set.
|
that wait. Once a request has passed the bans, the lookup starts. While it waits
|
||||||
|
for GeoJS, steps 12 to 14 run at the same time as steps 6 to 11. A lookup that
|
||||||
|
needs no wait has its answer at once, and steps 7 to 11 then run before steps 12
|
||||||
|
to 14 start.
|
||||||
|
|
||||||
|
- The first of the two sides to refuse the request ends the checks. `waf`
|
||||||
|
answers at once, and does not wait for the other side. The lookup goes on, and
|
||||||
|
its answer is kept.
|
||||||
|
- What the other side finds leaves no trace. It counts nothing, bans nothing,
|
||||||
|
raises no alert, counts no metric and adds nothing to the log line.
|
||||||
|
- When both sides would refuse, the one that finishes first is answered and
|
||||||
|
logged.
|
||||||
|
- When neither refuses, step 15 runs.
|
||||||
|
|
||||||
|
Steps 12 to 14 only read the request. They have 500 milliseconds together, from
|
||||||
|
their start. The limit is fixed, with no setting. A request they have not
|
||||||
|
finished by then is refused:
|
||||||
|
|
||||||
|
- the answer is `deny`, with `503`;
|
||||||
|
- the log line's action is `checks_timed_out`, and its `heuristic` names the
|
||||||
|
step that had not finished;
|
||||||
|
- the refusal makes no ban, is no offence and does not count toward the refusal
|
||||||
|
limit;
|
||||||
|
- it holds in every `SWW_WAF_CORE_RULE_SET_MODE`;
|
||||||
|
- in `observe` mode the request is answered `allow` with `would_action`, like
|
||||||
|
every refusal.
|
||||||
|
|
||||||
|
The limit is measured on the clock. While `waf`'s host is so busy that an
|
||||||
|
ordinary request takes 500 milliseconds, that request is refused too.
|
||||||
|
|
||||||
The header limits are fixed, with no setting. A line's length is in bytes and
|
The header limits are fixed, with no setting. A line's length is in bytes and
|
||||||
does not count its line ending. `waf` refuses a request whose request line is
|
does not count its line ending. `waf` refuses a request whose request line is
|
||||||
@@ -2490,9 +2537,11 @@ body can add up to more than the body. From 1 KiB of JSON, Coraza makes at most
|
|||||||
|
|
||||||
A request can reach all of these at once. At paranoia level 4, with no rule
|
A request can reach all of these at once. At paranoia level 4, with no rule
|
||||||
switched off, the Core Rule Set takes under a second on one core of a current
|
switched off, the Core Rule Set takes under a second on one core of a current
|
||||||
server to inspect it. So `waf` answers it within the default 2 seconds, even
|
server to inspect it. `waf` refuses it if the inspection takes over 500
|
||||||
after the default 800 milliseconds of waiting for GeoJS. Enough such requests at
|
milliseconds. Coraza cannot stop an inspection part-way, so the inspection runs
|
||||||
once can still overload `waf` past `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
|
to its end with nobody waiting for it. These limits keep one request from
|
||||||
|
holding a processor for more than about a second. Enough such requests at once
|
||||||
|
can still overload `waf` past `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
|
||||||
|
|
||||||
### The report
|
### The report
|
||||||
|
|
||||||
|
|||||||
@@ -105,6 +105,8 @@ Every one of these lines has:
|
|||||||
`blocked_by_core_rule_set`: refused by that check;
|
`blocked_by_core_rule_set`: refused by that check;
|
||||||
- `request_line_too_long`: a request line over the header limits;
|
- `request_line_too_long`: a request line over the header limits;
|
||||||
- `request_header_too_large`: headers over the header limits;
|
- `request_header_too_large`: headers over the header limits;
|
||||||
|
- `checks_timed_out`: refused with `503` because the trap paths, the rule
|
||||||
|
files and the Core Rule Set had not finished within 500 ms;
|
||||||
- `own_endpoint`: answered at one of `waf`'s own endpoints, or `404` for a
|
- `own_endpoint`: answered at one of `waf`'s own endpoints, or `404` for a
|
||||||
path under `/_sww/`.
|
path under `/_sww/`.
|
||||||
- `would_action`: in `observe` mode, what `enforce` mode would have done with a
|
- `would_action`: in `observe` mode, what `enforce` mode would have done with a
|
||||||
@@ -134,6 +136,11 @@ Every one of these lines has:
|
|||||||
`SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`;
|
`SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`;
|
||||||
- `refusal_limit`: `window`, `count` and `limit`, of the refusal limit;
|
- `refusal_limit`: `window`, `count` and `limit`, of the refusal limit;
|
||||||
- `byte_limits`: `window`, `count` and `limit`, the count in bytes.
|
- `byte_limits`: `window`, `count` and `limit`, the count in bytes.
|
||||||
|
|
||||||
|
For a request refused as `checks_timed_out`, `heuristic` is the one of
|
||||||
|
`trap_paths`, `rule_files` and `core_rule_set` that had not finished, and
|
||||||
|
its figure is `time_limit_ms`, 500.
|
||||||
|
|
||||||
- `after_answer_heuristic`: the heuristic that banned the client after `waf`
|
- `after_answer_heuristic`: the heuristic that banned the client after `waf`
|
||||||
answered the request, in `observe` mode too: `refusal_limit` or `byte_limits`.
|
answered the request, in `observe` mode too: `refusal_limit` or `byte_limits`.
|
||||||
- `after_answer_heuristic_figures`: an object of the figures that heuristic
|
- `after_answer_heuristic_figures`: an object of the figures that heuristic
|
||||||
|
|||||||
+6
-3
@@ -159,8 +159,9 @@ cut off; the client is banned.
|
|||||||
- `SWW_WAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in
|
- `SWW_WAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in
|
||||||
its `.mmdb` form, for `SWW_WAF_LOOKUP_SOURCE=file`.
|
its `.mmdb` form, for `SWW_WAF_LOOKUP_SOURCE=file`.
|
||||||
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` (default `800`): how long a
|
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` (default `800`): how long a
|
||||||
request waits for its client's first answer from GeoJS, and how long a request
|
request waits for its client's first answer from GeoJS. A time in
|
||||||
to GeoJS may take. A time in milliseconds; never `off`.
|
milliseconds; never `off`. It does not cut off the request to GeoJS, whose
|
||||||
|
answer is kept when it comes later.
|
||||||
- `SWW_WAF_ADD_LOOKUP_HEADERS` (default `true`): `true` has `rproxy` pass the
|
- `SWW_WAF_ADD_LOOKUP_HEADERS` (default `true`): `true` has `rproxy` pass the
|
||||||
app the client's AS number in `X-Client-ASN` and its country in
|
app the client's AS number in `X-Client-ASN` and its country in
|
||||||
`X-Client-Country`. The default is `false` while `SWW_WAF_LOOKUP_SOURCE` is
|
`X-Client-Country`. The default is `false` while `SWW_WAF_LOOKUP_SOURCE` is
|
||||||
@@ -434,7 +435,9 @@ A few limits are fixed rather than settings:
|
|||||||
|
|
||||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is the longest a question waits on
|
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is the longest a question waits on
|
||||||
an outside source. Keep it well under `rproxy`'s
|
an outside source. Keep it well under `rproxy`'s
|
||||||
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, as the default is. No question waits for
|
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, as the default is. The trap paths, the rule
|
||||||
|
files and the Core Rule Set run during that wait and have 500 ms, so `waf`
|
||||||
|
answers within the longer of the two times. No question waits for
|
||||||
`SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`, since the DNS blocklists and AbuseIPDB are
|
`SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`, since the DNS blocklists and AbuseIPDB are
|
||||||
asked in the background (see "Answering in time" under "`waf`" in
|
asked in the background (see "Answering in time" under "`waf`" in
|
||||||
[`SPEC.md`](../SPEC.md)).
|
[`SPEC.md`](../SPEC.md)).
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ require (
|
|||||||
github.com/oschwald/maxminddb-golang/v2 v2.7.0
|
github.com/oschwald/maxminddb-golang/v2 v2.7.0
|
||||||
github.com/prometheus/client_golang v1.24.1
|
github.com/prometheus/client_golang v1.24.1
|
||||||
github.com/spf13/viper v1.21.0
|
github.com/spf13/viper v1.21.0
|
||||||
|
golang.org/x/sys v0.48.0
|
||||||
)
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
@@ -47,7 +48,6 @@ require (
|
|||||||
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
|
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
|
||||||
golang.org/x/net v0.60.0 // indirect
|
golang.org/x/net v0.60.0 // indirect
|
||||||
golang.org/x/sync v0.23.0 // indirect
|
golang.org/x/sync v0.23.0 // indirect
|
||||||
golang.org/x/sys v0.48.0 // indirect
|
|
||||||
golang.org/x/text v0.42.0 // indirect
|
golang.org/x/text v0.42.0 // indirect
|
||||||
google.golang.org/protobuf v1.36.11 // indirect
|
google.golang.org/protobuf v1.36.11 // indirect
|
||||||
rsc.io/binaryregexp v0.2.0 // indirect
|
rsc.io/binaryregexp v0.2.0 // indirect
|
||||||
|
|||||||
@@ -5,12 +5,16 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
|
"regexp"
|
||||||
|
"runtime"
|
||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"golang.org/x/sys/unix"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -31,12 +35,46 @@ const (
|
|||||||
mostJSONRead = 1 << 10
|
mostJSONRead = 1 << 10
|
||||||
)
|
)
|
||||||
|
|
||||||
// inspectionTimeLimit is the most time the Core Rule Set may take on one
|
// waf refuses a request the Core Rule Set has not inspected within 500
|
||||||
// request. It is the default of rproxy's SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, 2
|
// milliseconds, but Coraza cannot stop an inspection part-way, so it runs to
|
||||||
// seconds, less the default of SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, 800
|
// its end with nobody waiting for it. The test checks that an inspection ends
|
||||||
// milliseconds, which a question can also wait for GeoJS, and less 200
|
// within about 1 second, so that a visitor cannot keep a processor busy for
|
||||||
// milliseconds for waf's other checks.
|
// longer with one request.
|
||||||
const inspectionTimeLimit = time.Second
|
//
|
||||||
|
// A busy host slows all work, so the test does not compare an inspection with
|
||||||
|
// 1 second. It compares it with referenceTime, timed on the same thread just
|
||||||
|
// before and after. inspectionTimeLimitMultiple is how many times
|
||||||
|
// referenceTime an inspection may take.
|
||||||
|
//
|
||||||
|
// On the host that runs our checks, referenceTime took at least 78
|
||||||
|
// milliseconds, so 13 times it is about 1 second. With that host fully busy,
|
||||||
|
// no try of the slowest request took more than 9.5 times referenceTime.
|
||||||
|
const inspectionTimeLimitMultiple = 13
|
||||||
|
|
||||||
|
// referencePattern is what referenceTime matches. Like the Core Rule Set's
|
||||||
|
// rules, it is a regular expression with alternatives and repeats.
|
||||||
|
var referencePattern = regexp.MustCompile(
|
||||||
|
`(?i)(union|select|insert|delete)\s+\w+\s+(from|into)\s+\w+[0-9]+;`,
|
||||||
|
)
|
||||||
|
|
||||||
|
// referenceTime returns the processor time the calling thread takes to do a
|
||||||
|
// fixed amount of work unrelated to the Core Rule Set, so that a change to the
|
||||||
|
// rules does not change it. referencePattern matches nowhere in the text, so
|
||||||
|
// each match reads all of it.
|
||||||
|
func referenceTime(t *testing.T) time.Duration {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
referenceText := text(4 << 10)
|
||||||
|
began := threadCPUTime(t)
|
||||||
|
|
||||||
|
for range 200 {
|
||||||
|
if referencePattern.MatchString(referenceText) {
|
||||||
|
t.Fatal("the reference pattern matched")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return threadCPUTime(t) - began
|
||||||
|
}
|
||||||
|
|
||||||
// text returns n bytes of letters, digits and spaces, which the Core Rule Set
|
// text returns n bytes of letters, digits and spaces, which the Core Rule Set
|
||||||
// takes longer over than one letter repeated.
|
// takes longer over than one letter repeated.
|
||||||
@@ -302,11 +340,30 @@ func worstCaseBodies() []visitorRequest {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// inspectionTime returns how long crs takes to inspect tc, and fails t unless
|
// threadCPUTime returns the processor time the calling thread has used.
|
||||||
// it reads all of tc's body and Coraza parses it.
|
func threadCPUTime(t *testing.T) time.Duration {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var now unix.Timespec
|
||||||
|
|
||||||
|
err := unix.ClockGettime(unix.CLOCK_THREAD_CPUTIME_ID, &now)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read the thread's processor time: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return time.Duration(now.Nano())
|
||||||
|
}
|
||||||
|
|
||||||
|
// inspectionTime returns the processor time crs takes to inspect tc, and the
|
||||||
|
// mean of referenceTime just before and after. It fails t unless crs reads all
|
||||||
|
// of tc's body and Coraza parses it.
|
||||||
|
//
|
||||||
|
// All of it runs on this goroutine, kept on one thread, so the thread's
|
||||||
|
// processor time measures it. Time other work holds the processor does not
|
||||||
|
// count.
|
||||||
func inspectionTime(
|
func inspectionTime(
|
||||||
t *testing.T, crs *coreruleset.CoreRuleSet, tc visitorRequest,
|
t *testing.T, crs *coreruleset.CoreRuleSet, tc visitorRequest,
|
||||||
) time.Duration {
|
) (time.Duration, time.Duration) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
req := httptest.NewRequestWithContext(
|
req := httptest.NewRequestWithContext(
|
||||||
@@ -318,11 +375,19 @@ func inspectionTime(
|
|||||||
req.Header.Set("Content-Type", tc.contentType)
|
req.Header.Set("Content-Type", tc.contentType)
|
||||||
}
|
}
|
||||||
|
|
||||||
began := time.Now()
|
runtime.LockOSThread()
|
||||||
|
|
||||||
|
defer runtime.UnlockOSThread()
|
||||||
|
|
||||||
|
before := referenceTime(t)
|
||||||
|
|
||||||
|
began := threadCPUTime(t)
|
||||||
result, read, _ := crs.Inspect(
|
result, read, _ := crs.Inspect(
|
||||||
req, netip.MustParseAddr("192.0.2.1"), strings.NewReader(tc.body),
|
req, netip.MustParseAddr("192.0.2.1"), strings.NewReader(tc.body),
|
||||||
)
|
)
|
||||||
took := time.Since(began)
|
took := threadCPUTime(t) - began
|
||||||
|
|
||||||
|
reference := (before + referenceTime(t)) / 2
|
||||||
|
|
||||||
if string(read) != tc.body {
|
if string(read) != tc.body {
|
||||||
t.Errorf("read %d bytes of %d", len(read), len(tc.body))
|
t.Errorf("read %d bytes of %d", len(read), len(tc.body))
|
||||||
@@ -333,13 +398,13 @@ func inspectionTime(
|
|||||||
t.Errorf("Coraza could not parse the body")
|
t.Errorf("Coraza could not parse the body")
|
||||||
}
|
}
|
||||||
|
|
||||||
return took
|
return took, reference
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime runs one at a time, so
|
// TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime runs one at a time, so
|
||||||
// that the tests running in parallel do not slow it. Other work on the host
|
// that the tests running in parallel do not slow it. Other work on the host
|
||||||
// can still slow one try, so a request fails only when three tries in a row
|
// can still slow one try more than the reference work around it, so a request
|
||||||
// take too long.
|
// fails only when three tries in a row take too long.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // one at a time, as above
|
//nolint:paralleltest // one at a time, as above
|
||||||
func TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime(t *testing.T) {
|
func TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime(t *testing.T) {
|
||||||
@@ -353,18 +418,19 @@ func TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime(t *testing.T) {
|
|||||||
|
|
||||||
for _, tc := range worstCaseRequests() {
|
for _, tc := range worstCaseRequests() {
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
var took time.Duration
|
var took, reference time.Duration
|
||||||
|
|
||||||
for range tries {
|
for range tries {
|
||||||
took = inspectionTime(t, crs, tc)
|
took, reference = inspectionTime(t, crs, tc)
|
||||||
if took <= inspectionTimeLimit {
|
if took <= inspectionTimeLimitMultiple*reference {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
"took %s on the last of %d tries, over %s",
|
"took %s on the last of %d tries, over %d times the %s "+
|
||||||
took, tries, inspectionTimeLimit,
|
"of the reference work",
|
||||||
|
took, tries, inspectionTimeLimitMultiple, reference,
|
||||||
)
|
)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
package lookup
|
package geojs
|
||||||
|
|
||||||
import "net/http"
|
import "net/http"
|
||||||
|
|
||||||
@@ -1,9 +1,9 @@
|
|||||||
// Package lookup looks up each client's AS number and country, through
|
// Package geojs looks up clients' AS numbers and countries through the GeoJS
|
||||||
// the GeoJS web service or in the lookup database, the IPinfo Lite file
|
// web service. Its answers are kept in memory, for at most 100,000 clients and
|
||||||
// SWW_WAF_LOOKUP_DB_PATH names. GeoJS's answers are kept in memory, for at
|
// for 7 days each. A caller waits for an answer as long as it says. A request to
|
||||||
// most 100,000 clients and for 7 days each, and are written to
|
// GeoJS goes on after its callers stop waiting, and its answer is kept. The
|
||||||
// lookups.json and read from it by the state package.
|
// package imports no other package of this repository.
|
||||||
package lookup
|
package geojs
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
@@ -11,7 +11,6 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"slices"
|
"slices"
|
||||||
@@ -21,8 +20,6 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/hashicorp/golang-lru/v2/simplelru"
|
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// URL is GeoJS's endpoint for an address's place and network. Asked about
|
// URL is GeoJS's endpoint for an address's place and network. Asked about
|
||||||
@@ -43,6 +40,9 @@ const (
|
|||||||
maxWaiting = 10000
|
maxWaiting = 10000
|
||||||
// maxPerRequest is how many addresses one request to GeoJS asks about.
|
// maxPerRequest is how many addresses one request to GeoJS asks about.
|
||||||
maxPerRequest = 200
|
maxPerRequest = 200
|
||||||
|
// requestTimeout is how long a request to GeoJS may take before it is
|
||||||
|
// abandoned, which is a failure. It is apart from how long callers wait.
|
||||||
|
requestTimeout = 10 * time.Second
|
||||||
// unknownASN is the AS number GeoJS gives when it knows none.
|
// unknownASN is the AS number GeoJS gives when it knows none.
|
||||||
unknownASN = 64512
|
unknownASN = 64512
|
||||||
// After a failure GeoJS is not asked again for a second, and for
|
// After a failure GeoJS is not asked again for a second, and for
|
||||||
@@ -64,39 +64,32 @@ var (
|
|||||||
type Params struct {
|
type Params struct {
|
||||||
// URL is where GeoJS is asked, normally URL.
|
// URL is where GeoJS is asked, normally URL.
|
||||||
URL string
|
URL string
|
||||||
// Timeout is how long a request waits for its client's first answer,
|
|
||||||
// and how long a request to GeoJS may take before it is abandoned
|
|
||||||
// (SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS).
|
|
||||||
Timeout time.Duration
|
|
||||||
// Wait is true when a setting needs each request's answer before the
|
|
||||||
// request goes on. Otherwise no request waits for one.
|
|
||||||
Wait bool
|
|
||||||
// Answered, unless nil, is given each answer GeoJS gives, once it is
|
// Answered, unless nil, is given each answer GeoJS gives, once it is
|
||||||
// kept.
|
// kept.
|
||||||
Answered func(Answer)
|
Answered func(Answer)
|
||||||
// Now tells the time, normally time.Now in UTC.
|
// Now tells the time, normally time.Now in UTC.
|
||||||
Now func() time.Time
|
Now func() time.Time
|
||||||
// ProcessLog receives GeoJS's failures.
|
// Asked is called for each request to GeoJS.
|
||||||
ProcessLog *slog.Logger
|
Asked func()
|
||||||
// Metrics count the requests to GeoJS, those that failed, and the
|
// Failed is called for each failure of GeoJS, with what went wrong and
|
||||||
// clients that go without an answer.
|
// how long GeoJS is left alone after it. The error never names the
|
||||||
Metrics *metrics.Metrics
|
// addresses asked about.
|
||||||
// Alerts receive a source_failure alert each time GeoJS fails.
|
Failed func(err error, askingAgainIn time.Duration)
|
||||||
Alerts *alerts.Queue
|
// WentWithoutAnswer is called each time a caller that waits for an
|
||||||
|
// answer goes on without one.
|
||||||
|
WentWithoutAnswer func()
|
||||||
}
|
}
|
||||||
|
|
||||||
// GeoJS looks up clients' AS numbers and countries through GeoJS. At most
|
// GeoJS looks up clients' AS numbers and countries through GeoJS. At most
|
||||||
// one request to GeoJS is under way at a time, and it asks about every
|
// one request to GeoJS is under way at a time, and it asks about every
|
||||||
// client waiting, up to maxPerRequest. It is safe for concurrent use.
|
// client waiting, up to maxPerRequest. It is safe for concurrent use.
|
||||||
type GeoJS struct {
|
type GeoJS struct {
|
||||||
url string
|
url string
|
||||||
timeout time.Duration
|
answered func(Answer)
|
||||||
wait bool
|
now func() time.Time
|
||||||
answered func(Answer)
|
asked func()
|
||||||
now func() time.Time
|
failed func(err error, askingAgainIn time.Duration)
|
||||||
processLog *slog.Logger
|
wentWithoutAnswer func()
|
||||||
metrics *metrics.Metrics
|
|
||||||
alerts *alerts.Queue
|
|
||||||
// httpClient follows no redirect, so that visitors' addresses go to
|
// httpClient follows no redirect, so that visitors' addresses go to
|
||||||
// GeoJS alone: a redirect is a failure.
|
// GeoJS alone: a redirect is a failure.
|
||||||
httpClient *http.Client
|
httpClient *http.Client
|
||||||
@@ -114,12 +107,10 @@ type GeoJS struct {
|
|||||||
retryAt time.Time
|
retryAt time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
// Answer is what GeoJS or the lookup database said about a client: its AS
|
// Answer is what GeoJS said about a client: its AS number, such as AS64496,
|
||||||
// number, such as AS64496, and the AS's name, both "" when the source knows
|
// and the AS's name, both "" when GeoJS knows no AS number for it; its
|
||||||
// no AS number for it; its country, "" when the source cannot place it;
|
// country, "" when GeoJS cannot place it; when GeoJS said so; and when the
|
||||||
// when the source said so; and, for GeoJS's answers, which lookups.json
|
// answer was last used. The zero Answer is that of a client with no answer.
|
||||||
// holds, when the answer was last used. The zero Answer is that of a
|
|
||||||
// client with no answer.
|
|
||||||
//
|
//
|
||||||
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
||||||
type Answer struct {
|
type Answer struct {
|
||||||
@@ -137,7 +128,7 @@ type wait struct {
|
|||||||
// replaced each time GeoJS fails before then.
|
// replaced each time GeoJS fails before then.
|
||||||
asked chan struct{}
|
asked chan struct{}
|
||||||
// late is true once the client has gone without an answer, for a
|
// late is true once the client has gone without an answer, for a
|
||||||
// whole timeout or because GeoJS failed: its requests no longer wait.
|
// whole wait or because GeoJS failed: its callers no longer wait.
|
||||||
late bool
|
late bool
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -149,14 +140,12 @@ func New(params Params) *GeoJS {
|
|||||||
}
|
}
|
||||||
|
|
||||||
return &GeoJS{
|
return &GeoJS{
|
||||||
url: params.URL,
|
url: params.URL,
|
||||||
timeout: params.Timeout,
|
answered: params.Answered,
|
||||||
wait: params.Wait,
|
now: params.Now,
|
||||||
answered: params.Answered,
|
asked: params.Asked,
|
||||||
now: params.Now,
|
failed: params.Failed,
|
||||||
processLog: params.ProcessLog,
|
wentWithoutAnswer: params.WentWithoutAnswer,
|
||||||
metrics: params.Metrics,
|
|
||||||
alerts: params.Alerts,
|
|
||||||
httpClient: &http.Client{
|
httpClient: &http.Client{
|
||||||
CheckRedirect: func(*http.Request, []*http.Request) error {
|
CheckRedirect: func(*http.Request, []*http.Request) error {
|
||||||
return http.ErrUseLastResponse
|
return http.ErrUseLastResponse
|
||||||
@@ -170,21 +159,22 @@ func New(params Params) *GeoJS {
|
|||||||
// LookUp returns the answer GeoJS gave about client, with its country as
|
// LookUp returns the answer GeoJS gave about client, with its country as
|
||||||
// a two-letter code in capitals, or the zero Answer when there is none
|
// a two-letter code in capitals, or the zero Answer when there is none
|
||||||
// yet. An answer is kept for 7 days. Without one, the client is asked
|
// yet. An answer is kept for 7 days. Without one, the client is asked
|
||||||
// about in the background, and, while Wait is set, the request waits up
|
// about in the background, and LookUp waits up to wait for the answer,
|
||||||
// to Timeout for the answer, unless the client has gone without one
|
// unless the client has gone without one before. The wait ends early when
|
||||||
// before. ctx is the context of the client's request, and ends the wait
|
// ctx ends. Neither ends the request to GeoJS.
|
||||||
// when it ends.
|
|
||||||
//
|
//
|
||||||
// GeoJS is asked about the client's first address, which is the client's
|
// GeoJS is asked about the client's first address, which is the client's
|
||||||
// own address for IPv4, and an address in the same place for an IPv6
|
// own address for IPv4, and an address in the same place for an IPv6
|
||||||
// netblock.
|
// netblock.
|
||||||
func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
|
func (g *GeoJS) LookUp(
|
||||||
answer, asked := g.answerOrWait(ctx, client, g.wait)
|
ctx context.Context, client netip.Prefix, wait time.Duration,
|
||||||
|
) Answer {
|
||||||
|
answer, asked := g.answerOrWait(ctx, client, true)
|
||||||
if asked == nil {
|
if asked == nil {
|
||||||
return answer
|
return answer
|
||||||
}
|
}
|
||||||
|
|
||||||
timer := time.NewTimer(g.timeout)
|
timer := time.NewTimer(wait)
|
||||||
defer timer.Stop()
|
defer timer.Stop()
|
||||||
|
|
||||||
select {
|
select {
|
||||||
@@ -198,7 +188,7 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
|
|||||||
|
|
||||||
answer, found := g.kept(client)
|
answer, found := g.kept(client)
|
||||||
if !found {
|
if !found {
|
||||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
g.wentWithoutAnswer()
|
||||||
}
|
}
|
||||||
|
|
||||||
w, waiting := g.waiting[client]
|
w, waiting := g.waiting[client]
|
||||||
@@ -209,8 +199,8 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
|
|||||||
return answer
|
return answer
|
||||||
}
|
}
|
||||||
|
|
||||||
// LookUpWithoutWaiting is LookUp for a request that does not need the answer,
|
// LookUpWithoutWaiting is LookUp for a caller that does not need the answer:
|
||||||
// whatever Wait says: it returns the zero Answer at once when none is kept.
|
// it returns the zero Answer at once when none is kept.
|
||||||
func (g *GeoJS) LookUpWithoutWaiting(
|
func (g *GeoJS) LookUpWithoutWaiting(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
client netip.Prefix,
|
client netip.Prefix,
|
||||||
@@ -228,8 +218,7 @@ func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) {
|
|||||||
return g.kept(client)
|
return g.kept(client)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Snapshot returns every answer kept, sorted by client, as lookups.json
|
// Snapshot returns every answer kept, sorted by client.
|
||||||
// lists them.
|
|
||||||
func (g *GeoJS) Snapshot() []Answer {
|
func (g *GeoJS) Snapshot() []Answer {
|
||||||
g.mu.Lock()
|
g.mu.Lock()
|
||||||
|
|
||||||
@@ -247,9 +236,9 @@ func (g *GeoJS) Snapshot() []Answer {
|
|||||||
return answers
|
return answers
|
||||||
}
|
}
|
||||||
|
|
||||||
// Load keeps answers read from lookups.json, in place of the answers it
|
// Load keeps answers, such as those of an earlier Snapshot, in place of the
|
||||||
// keeps, in the order they were last used, so that the one used longest
|
// answers it keeps, in the order they were last used, so that the one used
|
||||||
// ago is dropped first. Answers GeoJS gave keepFor ago or more are
|
// longest ago is dropped first. Answers GeoJS gave keepFor ago or more are
|
||||||
// dropped.
|
// dropped.
|
||||||
func (g *GeoJS) Load(answers []Answer) {
|
func (g *GeoJS) Load(answers []Answer) {
|
||||||
answers = slices.Clone(answers)
|
answers = slices.Clone(answers)
|
||||||
@@ -295,11 +284,11 @@ func (g *GeoJS) answerOrWait(
|
|||||||
g.ask(ctx)
|
g.ask(ctx)
|
||||||
|
|
||||||
if !mustWait {
|
if !mustWait {
|
||||||
return Answer{}, nil // the answer is not needed before the request goes on
|
return Answer{}, nil // the answer is not needed before the caller goes on
|
||||||
}
|
}
|
||||||
|
|
||||||
if w == nil {
|
if w == nil {
|
||||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
g.wentWithoutAnswer()
|
||||||
|
|
||||||
return Answer{}, nil // too many clients wait already
|
return Answer{}, nil // too many clients wait already
|
||||||
}
|
}
|
||||||
@@ -310,7 +299,7 @@ func (g *GeoJS) answerOrWait(
|
|||||||
}
|
}
|
||||||
|
|
||||||
if w.late {
|
if w.late {
|
||||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
g.wentWithoutAnswer()
|
||||||
|
|
||||||
return Answer{}, nil
|
return Answer{}, nil
|
||||||
}
|
}
|
||||||
@@ -335,8 +324,8 @@ func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
|
|||||||
|
|
||||||
// ask starts asking GeoJS about the waiting clients, unless a request to
|
// ask starts asking GeoJS about the waiting clients, unless a request to
|
||||||
// it is under way or it is left alone after a failure. The requests to
|
// it is under way or it is left alone after a failure. The requests to
|
||||||
// GeoJS are for every client waiting, so they go on when the client's
|
// GeoJS are for every client waiting, so they go on when the caller whose
|
||||||
// request whose ctx is given ends.
|
// ctx is given stops waiting.
|
||||||
func (g *GeoJS) ask(ctx context.Context) {
|
func (g *GeoJS) ask(ctx context.Context) {
|
||||||
if g.asking || g.now().Before(g.retryAt) {
|
if g.asking || g.now().Before(g.retryAt) {
|
||||||
return
|
return
|
||||||
@@ -441,8 +430,6 @@ func (g *GeoJS) keep(
|
|||||||
}
|
}
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
g.metrics.GeoJSFailures.Inc()
|
|
||||||
|
|
||||||
g.retryDelay = min(
|
g.retryDelay = min(
|
||||||
max(retryDelayFactor*g.retryDelay, firstRetryDelay),
|
max(retryDelayFactor*g.retryDelay, firstRetryDelay),
|
||||||
maxRetryDelay,
|
maxRetryDelay,
|
||||||
@@ -457,16 +444,7 @@ func (g *GeoJS) keep(
|
|||||||
w.late = true
|
w.late = true
|
||||||
}
|
}
|
||||||
|
|
||||||
g.processLog.Warn("asking GeoJS failed",
|
g.failed(err, g.retryDelay)
|
||||||
"error", err.Error(), "asking_again_in", g.retryDelay.String())
|
|
||||||
g.alerts.Raise(alerts.Alert{
|
|
||||||
Event: alerts.EventSourceFailure,
|
|
||||||
Reason: "asking GeoJS failed",
|
|
||||||
Detail: map[string]any{
|
|
||||||
"source": "geojs", "error": err.Error(),
|
|
||||||
"asking_again_in": g.retryDelay.String(),
|
|
||||||
},
|
|
||||||
})
|
|
||||||
|
|
||||||
return kept, false
|
return kept, false
|
||||||
}
|
}
|
||||||
@@ -479,7 +457,8 @@ func (g *GeoJS) keep(
|
|||||||
// request asks GeoJS about clients in one request, and returns the answer
|
// request asks GeoJS about clients in one request, and returns the answer
|
||||||
// for each address GeoJS's answer names: its AS number and the AS's name,
|
// for each address GeoJS's answer names: its AS number and the AS's name,
|
||||||
// both "" for the AS number 64512, which GeoJS gives when it knows none,
|
// both "" for the AS number 64512, which GeoJS gives when it knows none,
|
||||||
// and its country, in capitals.
|
// and its country, in capitals. The request is abandoned after
|
||||||
|
// requestTimeout.
|
||||||
func (g *GeoJS) request(
|
func (g *GeoJS) request(
|
||||||
ctx context.Context, clients []netip.Prefix,
|
ctx context.Context, clients []netip.Prefix,
|
||||||
) (map[netip.Addr]Answer, error) {
|
) (map[netip.Addr]Answer, error) {
|
||||||
@@ -489,7 +468,7 @@ func (g *GeoJS) request(
|
|||||||
addrs = append(addrs, client.Addr().String())
|
addrs = append(addrs, client.Addr().String())
|
||||||
}
|
}
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(ctx, g.timeout)
|
ctx, cancel := context.WithTimeout(ctx, requestTimeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext(
|
req, err := http.NewRequestWithContext(
|
||||||
@@ -504,7 +483,7 @@ func (g *GeoJS) request(
|
|||||||
|
|
||||||
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
|
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
|
||||||
|
|
||||||
g.metrics.GeoJSRequests.Inc()
|
g.asked()
|
||||||
|
|
||||||
res, err := g.httpClient.Do(req)
|
res, err := g.httpClient.Do(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -1,4 +1,4 @@
|
|||||||
package lookup_test
|
package geojs_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"net/netip"
|
"net/netip"
|
||||||
@@ -7,7 +7,7 @@ import (
|
|||||||
"testing/synctest"
|
"testing/synctest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
||||||
@@ -25,7 +25,7 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
|||||||
clock.advance(time.Hour)
|
clock.advance(time.Hour)
|
||||||
wantCountry(t, g, placed, germany)
|
wantCountry(t, g, placed, germany)
|
||||||
|
|
||||||
want := []lookup.Answer{
|
want := []geojs.Answer{
|
||||||
{Client: notPlaced, Answered: asked, Used: asked},
|
{Client: notPlaced, Answered: asked, Used: asked},
|
||||||
{
|
{
|
||||||
Client: placed, ASN: asn, ASName: asName, Country: germany,
|
Client: placed, ASN: asn, ASName: asName, Country: germany,
|
||||||
@@ -41,24 +41,24 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
|||||||
func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
|
func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
geojs, clock, g := start()
|
service, clock, g := start()
|
||||||
now := clock.Now()
|
now := clock.Now()
|
||||||
kept := lookup.Answer{
|
kept := geojs.Answer{
|
||||||
Client: netip.MustParsePrefix("203.0.113.9/32"),
|
Client: netip.MustParsePrefix("203.0.113.9/32"),
|
||||||
Country: "FR",
|
Country: "FR",
|
||||||
Answered: now.Add(-week + time.Second),
|
Answered: now.Add(-week + time.Second),
|
||||||
Used: now.Add(-time.Hour),
|
Used: now.Add(-time.Hour),
|
||||||
}
|
}
|
||||||
stale := lookup.Answer{
|
stale := geojs.Answer{
|
||||||
Client: netip.MustParsePrefix("203.0.113.10/32"),
|
Client: netip.MustParsePrefix("203.0.113.10/32"),
|
||||||
Country: "FR",
|
Country: "FR",
|
||||||
Answered: now.Add(-week),
|
Answered: now.Add(-week),
|
||||||
Used: now.Add(-time.Hour),
|
Used: now.Add(-time.Hour),
|
||||||
}
|
}
|
||||||
|
|
||||||
g.Load([]lookup.Answer{kept, stale})
|
g.Load([]geojs.Answer{kept, stale})
|
||||||
|
|
||||||
if got := g.Snapshot(); !slices.Equal(got, []lookup.Answer{kept}) {
|
if got := g.Snapshot(); !slices.Equal(got, []geojs.Answer{kept}) {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
"kept %+v, want only the answer GeoJS gave less than 7 days ago",
|
"kept %+v, want only the answer GeoJS gave less than 7 days ago",
|
||||||
got,
|
got,
|
||||||
@@ -66,7 +66,7 @@ func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
wantCountry(t, g, kept.Client, "FR")
|
wantCountry(t, g, kept.Client, "FR")
|
||||||
wantRequests(t, geojs, 0)
|
wantRequests(t, service, 0)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
|
func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
|
||||||
@@ -80,11 +80,11 @@ func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
|
|||||||
// lookups.json lists the answers by client. Here each was last used a
|
// lookups.json lists the answers by client. Here each was last used a
|
||||||
// second before the one listed before it, so the last listed is the
|
// second before the one listed before it, so the last listed is the
|
||||||
// one used longest ago, and the one dropped.
|
// one used longest ago, and the one dropped.
|
||||||
answers := make([]lookup.Answer, maxAnswers+1)
|
answers := make([]geojs.Answer, maxAnswers+1)
|
||||||
addr := netip.MustParseAddr("10.0.0.0")
|
addr := netip.MustParseAddr("10.0.0.0")
|
||||||
|
|
||||||
for i := range answers {
|
for i := range answers {
|
||||||
answers[i] = lookup.Answer{
|
answers[i] = geojs.Answer{
|
||||||
Client: netip.PrefixFrom(addr, addr.BitLen()),
|
Client: netip.PrefixFrom(addr, addr.BitLen()),
|
||||||
Country: germany,
|
Country: germany,
|
||||||
Answered: now,
|
Answered: now,
|
||||||
@@ -1,3 +1,6 @@
|
|||||||
|
// Package lookup looks up each client's AS number and country in the lookup
|
||||||
|
// database, the IPinfo Lite file SWW_WAF_LOOKUP_DB_PATH names. Its answers
|
||||||
|
// take the form of GeoJS's, so that waf handles both alike.
|
||||||
package lookup
|
package lookup
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -14,6 +17,7 @@ import (
|
|||||||
"github.com/oschwald/maxminddb-golang/v2"
|
"github.com/oschwald/maxminddb-golang/v2"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
)
|
)
|
||||||
|
|
||||||
// quietTime is how long the lookup database must go without a change
|
// quietTime is how long the lookup database must go without a change
|
||||||
@@ -77,7 +81,7 @@ func OpenFile(params FileParams) (*File, error) {
|
|||||||
// each "" when the database does not give it, as for an address missing
|
// each "" when the database does not give it, as for an address missing
|
||||||
// from it. The database is asked about the client's first address, as
|
// from it. The database is asked about the client's first address, as
|
||||||
// GeoJS is.
|
// GeoJS is.
|
||||||
func (f *File) LookUp(client netip.Prefix) Answer {
|
func (f *File) LookUp(client netip.Prefix) geojs.Answer {
|
||||||
f.mu.Lock()
|
f.mu.Lock()
|
||||||
reader := f.reader
|
reader := f.reader
|
||||||
f.mu.Unlock()
|
f.mu.Unlock()
|
||||||
@@ -91,7 +95,7 @@ func (f *File) LookUp(client netip.Prefix) Answer {
|
|||||||
found = record{}
|
found = record{}
|
||||||
}
|
}
|
||||||
|
|
||||||
return Answer{
|
return geojs.Answer{
|
||||||
Client: client,
|
Client: client,
|
||||||
ASN: found.ASN,
|
ASN: found.ASN,
|
||||||
ASName: found.ASName,
|
ASName: found.ASName,
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ import (
|
|||||||
"github.com/maxmind/mmdbwriter/mmdbtype"
|
"github.com/maxmind/mmdbwriter/mmdbtype"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -68,7 +69,7 @@ func TestFilePlacesClientsAndCountsAnAddressMissingFromItAsUnknown(
|
|||||||
prefix := netip.MustParsePrefix(client)
|
prefix := netip.MustParsePrefix(client)
|
||||||
|
|
||||||
got := f.LookUp(prefix)
|
got := f.LookUp(prefix)
|
||||||
if got != (Answer{
|
if got != (geojs.Answer{
|
||||||
Client: prefix, ASN: want.ASN, ASName: want.ASName, Country: want.Country,
|
Client: prefix, ASN: want.ASN, ASName: want.ASName, Country: want.Country,
|
||||||
Answered: now,
|
Answered: now,
|
||||||
}) {
|
}) {
|
||||||
|
|||||||
@@ -274,8 +274,7 @@ func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
|
|||||||
// AddLookupFile adds the metrics of the lookup database, read as the
|
// AddLookupFile adds the metrics of the lookup database, read as the
|
||||||
// metrics are asked for: when the file in use was read, which lastRead
|
// metrics are asked for: when the file in use was read, which lastRead
|
||||||
// returns, and the replacements of it that could not be read, which
|
// returns, and the replacements of it that could not be read, which
|
||||||
// readFailures returns. The lookup package's File, which has both, cannot
|
// readFailures returns.
|
||||||
// be named here: that package counts GeoJS's requests in these metrics.
|
|
||||||
func (m *Metrics) AddLookupFile(
|
func (m *Metrics) AddLookupFile(
|
||||||
lastRead func() time.Time,
|
lastRead func() time.Time,
|
||||||
readFailures func() int,
|
readFailures func() int,
|
||||||
|
|||||||
@@ -31,10 +31,12 @@ import (
|
|||||||
// this package: Go's resolver counts the queries under way in one
|
// this package: Go's resolver counts the queries under way in one
|
||||||
// sync.WaitGroup for the whole process, and the process fails when
|
// sync.WaitGroup for the whole process, and the process fails when
|
||||||
// queries from two bubbles, or from a bubble and from outside one, are
|
// queries from two bubbles, or from a bubble and from outside one, are
|
||||||
// under way at once. TestMain has the resolver make its configuration,
|
// under way at once. The one test outside a bubble is parallel, so it
|
||||||
// which it makes on its first query, outside every bubble, since the
|
// starts only once every test that is not parallel has ended; no other
|
||||||
// configuration holds a channel, which the bubble it was made in would
|
// parallel test of this package queries a resolver. TestMain has the
|
||||||
// keep to itself.
|
// resolver make its configuration, which it makes on its first query,
|
||||||
|
// outside every bubble, since the configuration holds a channel, which
|
||||||
|
// the bubble it was made in would keep to itself.
|
||||||
|
|
||||||
const (
|
const (
|
||||||
// zone and otherZone are the DNSBL zones the tests name.
|
// zone and otherZone are the DNSBL zones the tests name.
|
||||||
@@ -525,8 +527,9 @@ func TestAtMost100000VerdictsKeptTheOneFetchedLongestAgoDroppedFirst(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
|
||||||
func TestQueriesGoToTheResolverSWWDNSBLResolverNames(t *testing.T) {
|
func TestQueriesGoToTheResolverSWWDNSBLResolverNames(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
resolver := &resolverStandIn{answers: map[string]answer{
|
resolver := &resolverStandIn{answers: map[string]answer{
|
||||||
listedName: {addrs: []string{listing}},
|
listedName: {addrs: []string{listing}},
|
||||||
}}
|
}}
|
||||||
|
|||||||
@@ -61,6 +61,12 @@ const (
|
|||||||
// ActionCountryDenied is a request refused because its client's country
|
// ActionCountryDenied is a request refused because its client's country
|
||||||
// is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
// is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||||
ActionCountryDenied = "country_denied"
|
ActionCountryDenied = "country_denied"
|
||||||
|
|
||||||
|
// ActionChecksTimedOut is a request refused because the trap paths, the
|
||||||
|
// rule files and the Core Rule Set had not finished with it within 500
|
||||||
|
// milliseconds.
|
||||||
|
ActionChecksTimedOut = "checks_timed_out"
|
||||||
|
|
||||||
// ActionOwnEndpoint is a request waf answered at one of its own
|
// ActionOwnEndpoint is a request waf answered at one of its own
|
||||||
// endpoints, or a question about a path under /_sww/.
|
// endpoints, or a question about a path under /_sww/.
|
||||||
ActionOwnEndpoint = "own_endpoint"
|
ActionOwnEndpoint = "own_endpoint"
|
||||||
@@ -149,7 +155,8 @@ type Line struct {
|
|||||||
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
|
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
|
||||||
// ActionClientIsRateLimited, ActionBlockedByBlockRule,
|
// ActionClientIsRateLimited, ActionBlockedByBlockRule,
|
||||||
// ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or
|
// ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or
|
||||||
// ActionRequestHeaderTooLarge for the header limits.
|
// ActionRequestHeaderTooLarge for the header limits, or
|
||||||
|
// ActionChecksTimedOut.
|
||||||
WouldAction string `json:"would_action,omitempty"`
|
WouldAction string `json:"would_action,omitempty"`
|
||||||
// Heuristic is the name of the heuristic that refused the request, in
|
// Heuristic is the name of the heuristic that refused the request, in
|
||||||
// observe mode too, and HeuristicFigures are the figures it acted on, by
|
// observe mode too, and HeuristicFigures are the figures it acted on, by
|
||||||
|
|||||||
@@ -1424,7 +1424,14 @@ func TestFiveFailedAsksKeepTheNextRequestFromAskingWAF(t *testing.T) {
|
|||||||
failure(w, r)
|
failure(w, r)
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
app := startApp(t, greet)
|
nextID := make(chan string, 1)
|
||||||
|
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if answering.Load() {
|
||||||
|
nextID <- r.Header.Get(requestIDHeader)
|
||||||
|
}
|
||||||
|
|
||||||
|
greet(w, r)
|
||||||
|
})
|
||||||
addr, out := startRProxy(
|
addr, out := startRProxy(
|
||||||
t,
|
t,
|
||||||
standIn.URL,
|
standIn.URL,
|
||||||
@@ -1434,19 +1441,36 @@ func TestFiveFailedAsksKeepTheNextRequestFromAskingWAF(t *testing.T) {
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// An ask can time out before it reaches the stand-in, so these
|
||||||
|
// may leave fewer than five questions, or leave some late.
|
||||||
for range 5 {
|
for range 5 {
|
||||||
wantGreeting(t, get(t, addr, "/"))
|
wantGreeting(t, get(t, addr, "/"))
|
||||||
standIn.question(t)
|
}
|
||||||
|
|
||||||
|
// rproxy writes this line before it answers a request it kept
|
||||||
|
// from asking waf, so no wait is needed.
|
||||||
|
if out.has("the circuit breaker is open") {
|
||||||
|
t.Error("the circuit breaker opened before five asks failed")
|
||||||
}
|
}
|
||||||
|
|
||||||
answering.Store(true)
|
answering.Store(true)
|
||||||
wantGreeting(t, get(t, addr, "/"))
|
wantGreeting(t, get(t, addr, "/"))
|
||||||
out.waitFor(t, "the circuit breaker is open")
|
out.waitFor(t, "the circuit breaker is open")
|
||||||
|
|
||||||
|
// Had the next request asked waf, its question would have reached
|
||||||
|
// the stand-in before the request reached the app.
|
||||||
|
var id string
|
||||||
select {
|
select {
|
||||||
case <-standIn.questions:
|
case id = <-nextID:
|
||||||
t.Error("waf was asked, want it not to be")
|
default:
|
||||||
case <-time.After(10 * pollInterval):
|
t.Fatal("the app did not get the next request")
|
||||||
|
}
|
||||||
|
|
||||||
|
for len(standIn.questions) > 0 {
|
||||||
|
asked := <-standIn.questions
|
||||||
|
if asked.header.Get(requestIDHeader) == id {
|
||||||
|
t.Error("waf was asked, want it not to be")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -46,7 +46,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
@@ -135,7 +135,7 @@ type Params struct {
|
|||||||
// sww runs.
|
// sww runs.
|
||||||
Ledger *bans.Ledger
|
Ledger *bans.Ledger
|
||||||
Limiter *ratelimit.Limiter
|
Limiter *ratelimit.Limiter
|
||||||
GeoJS *lookup.GeoJS
|
GeoJS *geojs.GeoJS
|
||||||
Lists *reputation.Lists
|
Lists *reputation.Lists
|
||||||
DNSBL *reputation.DNSBL
|
DNSBL *reputation.DNSBL
|
||||||
AbuseIPDB *reputation.AbuseIPDB
|
AbuseIPDB *reputation.AbuseIPDB
|
||||||
@@ -195,8 +195,8 @@ type clientsFile struct {
|
|||||||
|
|
||||||
// lookupsFile is lookups.json, with each answer on a line of its own.
|
// lookupsFile is lookups.json, with each answer on a line of its own.
|
||||||
type lookupsFile struct {
|
type lookupsFile struct {
|
||||||
Version int `json:"version"`
|
Version int `json:"version"`
|
||||||
Lookups []lookup.Answer `json:"lookups"`
|
Lookups []geojs.Answer `json:"lookups"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// abuseIPDBChecksSpentFile is abuseipdb-checks-spent.json, indented for an
|
// abuseIPDBChecksSpentFile is abuseipdb-checks-spent.json, indented for an
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
@@ -1700,7 +1700,7 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
|
|||||||
t,
|
t,
|
||||||
lookupsJSON,
|
lookupsJSON,
|
||||||
params.GeoJS.Snapshot(),
|
params.GeoJS.Snapshot(),
|
||||||
[]lookup.Answer{
|
[]geojs.Answer{
|
||||||
{Client: client, Country: "FR", Answered: midnight()},
|
{Client: client, Country: "FR", Answered: midnight()},
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
@@ -2415,9 +2415,7 @@ func newParams(dir string) state.Params {
|
|||||||
MaxBans: 5000,
|
MaxBans: 5000,
|
||||||
}),
|
}),
|
||||||
Limiter: ratelimit.New(ratelimit.Limits{}, 20000),
|
Limiter: ratelimit.New(ratelimit.Limits{}, 20000),
|
||||||
GeoJS: lookup.New(lookup.Params{
|
GeoJS: geojs.New(geojs.Params{Now: midnight}),
|
||||||
Now: midnight, ProcessLog: discard, Metrics: m,
|
|
||||||
}),
|
|
||||||
Lists: reputation.New(reputation.Params{
|
Lists: reputation.New(reputation.Params{
|
||||||
BlocklistURLs: []string{
|
BlocklistURLs: []string{
|
||||||
blocklistURL,
|
blocklistURL,
|
||||||
@@ -2494,7 +2492,7 @@ func fill(params state.Params) {
|
|||||||
})
|
})
|
||||||
params.Limiter.AddLookup(client, now.Add(-time.Hour), asn, asName, "DE")
|
params.Limiter.AddLookup(client, now.Add(-time.Hour), asn, asName, "DE")
|
||||||
|
|
||||||
params.GeoJS.Load([]lookup.Answer{
|
params.GeoJS.Load([]geojs.Answer{
|
||||||
{
|
{
|
||||||
Client: client, ASN: asn, ASName: asName, Country: "DE",
|
Client: client, ASN: asn, ASName: asName, Country: "DE",
|
||||||
Answered: now.Add(-time.Hour), Used: now,
|
Answered: now.Add(-time.Hour), Used: now,
|
||||||
|
|||||||
+2
-1
@@ -25,6 +25,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
@@ -270,7 +271,7 @@ func newServer(
|
|||||||
Config: cfg,
|
Config: cfg,
|
||||||
RequestLog: stdout,
|
RequestLog: stdout,
|
||||||
ProcessLog: processLog,
|
ProcessLog: processLog,
|
||||||
GeoJSURL: lookup.URL,
|
GeoJSURL: geojs.URL,
|
||||||
AbuseIPDBURL: reputation.AbuseIPDBURL,
|
AbuseIPDBURL: reputation.AbuseIPDBURL,
|
||||||
LookupFile: lookupFile,
|
LookupFile: lookupFile,
|
||||||
CoreRuleSet: crs,
|
CoreRuleSet: crs,
|
||||||
|
|||||||
@@ -19,6 +19,8 @@ const (
|
|||||||
alertMaxPerHour = "SWW_WAF_ALERT_MAX_PER_HOUR"
|
alertMaxPerHour = "SWW_WAF_ALERT_MAX_PER_HOUR"
|
||||||
// alertInstance is the instance every alert of these tests gives.
|
// alertInstance is the instance every alert of these tests gives.
|
||||||
alertInstance = "host1/blog"
|
alertInstance = "host1/blog"
|
||||||
|
// alertWebhook is where the tests send alerts.
|
||||||
|
alertWebhook = "https://alerts.example/sww"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestBanForABrokenLimitRaisesABanAlertWithItsNotes(t *testing.T) {
|
func TestBanForABrokenLimitRaisesABanAlertWithItsNotes(t *testing.T) {
|
||||||
@@ -270,7 +272,7 @@ func startAppWithAlerts(
|
|||||||
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)}
|
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)}
|
||||||
settings := map[string]string{
|
settings := map[string]string{
|
||||||
trustedProxies: trustLocalhost,
|
trustedProxies: trustLocalhost,
|
||||||
alertWebhookURL: "https://alerts.example/sww",
|
alertWebhookURL: alertWebhook,
|
||||||
instanceName: alertInstance,
|
instanceName: alertInstance,
|
||||||
}
|
}
|
||||||
maps.Copy(settings, env)
|
maps.Copy(settings, env)
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ import (
|
|||||||
"github.com/prometheus/client_golang/prometheus/testutil"
|
"github.com/prometheus/client_golang/prometheus/testutil"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
@@ -591,7 +591,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
|
|||||||
s := startWAF(t, geojsURL, map[string]string{
|
s := startWAF(t, geojsURL, map[string]string{
|
||||||
unknownCountryLimitPercent: "0", rateLimitPerMinute: "1",
|
unknownCountryLimitPercent: "0", rateLimitPerMinute: "1",
|
||||||
})
|
})
|
||||||
s.server.GeoJS.Load([]lookup.Answer{{
|
s.server.GeoJS.Load([]geojs.Answer{{
|
||||||
Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512",
|
Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512",
|
||||||
Answered: s.clock.Now(), Used: s.clock.Now(),
|
Answered: s.clock.Now(), Used: s.clock.Now(),
|
||||||
}})
|
}})
|
||||||
@@ -770,6 +770,8 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
|
|||||||
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says,
|
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says,
|
||||||
// so that a request that does not wait takes no time at all.
|
// so that a request that does not wait takes no time at all.
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
server, out := newWAFInBubble(t, tc.env)
|
server, out := newWAFInBubble(t, tc.env)
|
||||||
tc.prepare(t, server, time.Now())
|
tc.prepare(t, server, time.Now())
|
||||||
|
|
||||||
@@ -838,6 +840,8 @@ func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays(
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
server, out := newWAFInBubble(t, tc.env)
|
server, out := newWAFInBubble(t, tc.env)
|
||||||
tc.prepare(t, server, time.Now())
|
tc.prepare(t, server, time.Now())
|
||||||
|
|
||||||
@@ -950,10 +954,11 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
|
|||||||
t.Errorf("%d checks with AbuseIPDB, want 1", checked)
|
t.Errorf("%d checks with AbuseIPDB, want 1", checked)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The query is answered, and the check with AbuseIPDB abandoned,
|
// The query is answered, and the check with AbuseIPDB and the request
|
||||||
// before the bubble ends.
|
// to GeoJS abandoned, before the bubble ends.
|
||||||
releaseResolver()
|
releaseResolver()
|
||||||
time.Sleep(defaultDNSBLAbuseIPDBWait)
|
time.Sleep(defaultDNSBLAbuseIPDBWait)
|
||||||
|
waitForGeoJSRequestsToBeAbandoned()
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,172 @@
|
|||||||
|
package waf
|
||||||
|
|
||||||
|
// The heuristics of besideLookup, and how they run beside the lookup.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"sync/atomic"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||||
|
)
|
||||||
|
|
||||||
|
// besideLookupTimeLimit is how long the heuristics of besideLookup have for a
|
||||||
|
// request, from when they start. They only read the request, and no request
|
||||||
|
// should take them that long. A request they have not finished by then is
|
||||||
|
// refused with 503, as ActionChecksTimedOut.
|
||||||
|
const besideLookupTimeLimit = 500 * time.Millisecond
|
||||||
|
|
||||||
|
// besideLookupFindings is what the heuristics of besideLookup found of a
|
||||||
|
// request, before any of it is applied to the request.
|
||||||
|
type besideLookupFindings struct {
|
||||||
|
// trapPath is the request's path when it is in SWW_WAF_TRAP_PATHS, and ""
|
||||||
|
// otherwise.
|
||||||
|
trapPath string
|
||||||
|
// rules are the rules of the rule files the request matched.
|
||||||
|
rules []rules.Rule
|
||||||
|
// inspected is true when the Core Rule Set inspected the request.
|
||||||
|
// inspection, read and inspectionTime are then what it found, what it read
|
||||||
|
// of the body, and how long it took.
|
||||||
|
inspected bool
|
||||||
|
inspection coreruleset.Result
|
||||||
|
read []byte
|
||||||
|
inspectionTime time.Duration
|
||||||
|
// refuses is true when one of the heuristics refuses the request, or would
|
||||||
|
// in observe mode. Those after it did no work.
|
||||||
|
refuses bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// runBesideLookup runs the heuristics of besideLookup while the lookup waits
|
||||||
|
// for GeoJS, and returns the first verdict, or nil when none gives one.
|
||||||
|
// lookedUp is the channel the lookup's result comes on, or nil once afterLookup
|
||||||
|
// has let the request through.
|
||||||
|
//
|
||||||
|
// The first of the two to refuse the request ends the checks: waf answers at
|
||||||
|
// once, and what the other found is dropped. besideLookup has
|
||||||
|
// besideLookupTimeLimit; past it, the request is refused as
|
||||||
|
// ActionChecksTimedOut. Once both have let the request through, the heuristics
|
||||||
|
// of afterBesideLookup run. ctx is the request's own context.
|
||||||
|
func (rq *request) runBesideLookup(
|
||||||
|
ctx context.Context, lookedUp <-chan lookupResult,
|
||||||
|
) *verdict {
|
||||||
|
var running atomic.Pointer[string]
|
||||||
|
|
||||||
|
running.Store(new(trapPaths{}.name()))
|
||||||
|
|
||||||
|
found := make(chan *besideLookupFindings, 1)
|
||||||
|
|
||||||
|
go func() { found <- rq.findBesideLookup(&running) }()
|
||||||
|
|
||||||
|
timeUp := time.NewTimer(rq.h.besideLookupLimit)
|
||||||
|
defer timeUp.Stop()
|
||||||
|
|
||||||
|
var findings *besideLookupFindings
|
||||||
|
|
||||||
|
for findings == nil || lookedUp != nil {
|
||||||
|
select {
|
||||||
|
case result := <-lookedUp:
|
||||||
|
lookedUp = nil
|
||||||
|
|
||||||
|
v := rq.runAfterLookup(ctx, result)
|
||||||
|
if v != nil {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
case findings = <-found:
|
||||||
|
found = nil
|
||||||
|
|
||||||
|
timeUp.Stop()
|
||||||
|
|
||||||
|
if findings.refuses {
|
||||||
|
return rq.applyFindings(ctx, findings)
|
||||||
|
}
|
||||||
|
case <-timeUp.C:
|
||||||
|
return rq.timedOut(*running.Load())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
v := rq.applyFindings(ctx, findings)
|
||||||
|
if v != nil {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
|
||||||
|
return rq.runHeuristics(ctx, afterBesideLookup(), rq.h.now())
|
||||||
|
}
|
||||||
|
|
||||||
|
// findBesideLookup does the work of the heuristics of besideLookup on the
|
||||||
|
// request, in their order, until one refuses it, and returns what they found.
|
||||||
|
// It changes nothing of the request or of what waf keeps, so that it can run
|
||||||
|
// while the lookup and afterLookup do. running holds the name of the heuristic
|
||||||
|
// at work, the first one's when findBesideLookup is called.
|
||||||
|
func (rq *request) findBesideLookup(
|
||||||
|
running *atomic.Pointer[string],
|
||||||
|
) *besideLookupFindings {
|
||||||
|
findings := &besideLookupFindings{}
|
||||||
|
|
||||||
|
findings.trapPath = rq.trapPath()
|
||||||
|
if findings.trapPath != "" {
|
||||||
|
findings.refuses = true
|
||||||
|
|
||||||
|
return findings
|
||||||
|
}
|
||||||
|
|
||||||
|
running.Store(new(ruleFiles{}.name()))
|
||||||
|
|
||||||
|
findings.rules = rq.h.rules.Match(rq.in)
|
||||||
|
if refusedByRule(findings.rules) {
|
||||||
|
findings.refuses = true
|
||||||
|
|
||||||
|
return findings
|
||||||
|
}
|
||||||
|
|
||||||
|
running.Store(new(coreRuleSet{}.name()))
|
||||||
|
|
||||||
|
cfg := rq.h.config
|
||||||
|
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
|
||||||
|
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
|
||||||
|
return findings
|
||||||
|
}
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
// A question's body has been read into memory, so that reading it here
|
||||||
|
// cannot fail.
|
||||||
|
findings.inspection, findings.read, _ = rq.h.coreRuleSet.Inspect(
|
||||||
|
rq.in, rq.client, rq.in.Body,
|
||||||
|
)
|
||||||
|
findings.inspectionTime = time.Since(start)
|
||||||
|
findings.inspected = true
|
||||||
|
findings.refuses = rq.refusedByCoreRuleSet(findings.inspection)
|
||||||
|
|
||||||
|
return findings
|
||||||
|
}
|
||||||
|
|
||||||
|
// applyFindings applies findings, what findBesideLookup found, to the
|
||||||
|
// request: it runs the heuristics of besideLookup, which read them, at the
|
||||||
|
// time it is called, and returns the first verdict, or nil when none gives
|
||||||
|
// one.
|
||||||
|
func (rq *request) applyFindings(
|
||||||
|
ctx context.Context, findings *besideLookupFindings,
|
||||||
|
) *verdict {
|
||||||
|
rq.findings = findings
|
||||||
|
|
||||||
|
return rq.runHeuristics(ctx, besideLookup(), rq.h.now())
|
||||||
|
}
|
||||||
|
|
||||||
|
// timedOut refuses the request for the heuristics of besideLookup not having
|
||||||
|
// finished with it within besideLookupTimeLimit. running is the name of the
|
||||||
|
// one at work, which the log line's heuristic gives. The refusal makes no ban
|
||||||
|
// and is no offence.
|
||||||
|
func (rq *request) timedOut(running string) *verdict {
|
||||||
|
v := &verdict{
|
||||||
|
action: requestlog.ActionChecksTimedOut,
|
||||||
|
figures: map[string]any{
|
||||||
|
"time_limit_ms": rq.h.besideLookupLimit.Milliseconds(),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
rq.line.Heuristic = running
|
||||||
|
rq.line.HeuristicFigures = v.figures
|
||||||
|
|
||||||
|
return v
|
||||||
|
}
|
||||||
@@ -0,0 +1,426 @@
|
|||||||
|
package waf_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io"
|
||||||
|
"maps"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/netip"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"testing/synctest"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/prometheus/client_golang/prometheus/testutil"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
|
)
|
||||||
|
|
||||||
|
// checksTimeLimit is how long the trap paths, the rule files and the Core Rule
|
||||||
|
// Set have for a request, as SPEC.md gives it, and checksTimedOut the status of
|
||||||
|
// a request they have not finished by then.
|
||||||
|
const (
|
||||||
|
checksTimeLimit = 500 * time.Millisecond
|
||||||
|
checksTimedOut = http.StatusServiceUnavailable
|
||||||
|
)
|
||||||
|
|
||||||
|
// heldCoreRuleSet is a stand-in for the Core Rule Set: each inspection takes
|
||||||
|
// took, then finds result. asked counts the inspections.
|
||||||
|
type heldCoreRuleSet struct {
|
||||||
|
took time.Duration
|
||||||
|
result coreruleset.Result
|
||||||
|
asked atomic.Int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// Inspect inspects a request as the stand-in does, and reads all of body.
|
||||||
|
func (c *heldCoreRuleSet) Inspect(
|
||||||
|
_ *http.Request, _ netip.Addr, body io.Reader,
|
||||||
|
) (coreruleset.Result, []byte, error) {
|
||||||
|
c.asked.Add(1)
|
||||||
|
time.Sleep(c.took)
|
||||||
|
|
||||||
|
read, err := io.ReadAll(body)
|
||||||
|
|
||||||
|
return c.result, read, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// coreRuleSetMatch returns what the stand-in finds of an attack: rule 942100,
|
||||||
|
// with a score of 5, the default SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD.
|
||||||
|
func coreRuleSetMatch() coreruleset.Result {
|
||||||
|
return coreruleset.Result{RuleIDs: []int{942100}, Score: 5}
|
||||||
|
}
|
||||||
|
|
||||||
|
// newWAFHoldingTheCoreRuleSet makes a waf, to be asked in a synctest bubble,
|
||||||
|
// with crs in place of the Core Rule Set, in block mode. GeoJS never answers,
|
||||||
|
// a setting needs its answer, alerts go to a webhook, and everything else is
|
||||||
|
// at its default but for env. No request to it may use the network.
|
||||||
|
func newWAFHoldingTheCoreRuleSet(
|
||||||
|
t *testing.T, crs *heldCoreRuleSet, env map[string]string,
|
||||||
|
) (*waf.Server, *output, *alerts.Queue) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
settings := map[string]string{
|
||||||
|
tagDeny: denyKP, coreRuleSetMode: block,
|
||||||
|
alertWebhookURL: alertWebhook,
|
||||||
|
}
|
||||||
|
maps.Copy(settings, env)
|
||||||
|
|
||||||
|
server, out, queue := newWAFWith(t, func(params waf.Params) *waf.Server {
|
||||||
|
params.CoreRuleSet = crs
|
||||||
|
|
||||||
|
return waf.New(params)
|
||||||
|
}, silentGeoJSURL, time.Now, settings)
|
||||||
|
server.SetBesideLookupTimeLimit(waf.BesideLookupTimeLimit)
|
||||||
|
|
||||||
|
return server, out, queue
|
||||||
|
}
|
||||||
|
|
||||||
|
// lookupWait is SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS for wait.
|
||||||
|
func lookupWait(wait time.Duration) string {
|
||||||
|
return strconv.FormatInt(wait.Milliseconds(), 10)
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantAnswer checks the status of an answer and how long it took.
|
||||||
|
func wantAnswer(
|
||||||
|
t *testing.T, status int, took time.Duration,
|
||||||
|
wantStatus int, wantTook time.Duration, out *output,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if status != wantStatus || took != wantTook {
|
||||||
|
t.Errorf(
|
||||||
|
"answered %d after %s, want %d after %s; output:\n%s",
|
||||||
|
status, took, wantStatus, wantTook, out.text(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantNoCoreRuleSetTrace checks that what the Core Rule Set found of a request
|
||||||
|
// left no trace: line has none of its fields, and server has counted no match
|
||||||
|
// of it and raised no alert for one.
|
||||||
|
func wantNoCoreRuleSetTrace(
|
||||||
|
t *testing.T, server *waf.Server, queue *alerts.Queue, line logLine,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if line.DurationCoreRuleSetMS != nil || line.CoreRuleSetScore != nil ||
|
||||||
|
line.CoreRuleSetRuleIDs != nil {
|
||||||
|
t.Errorf(
|
||||||
|
"log line has Core Rule Set time %v, score %v and rule ids %v, want none",
|
||||||
|
line.DurationCoreRuleSetMS,
|
||||||
|
line.CoreRuleSetScore,
|
||||||
|
line.CoreRuleSetRuleIDs,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
recorder := httptest.NewRecorder()
|
||||||
|
server.Metrics.ServeHTTP(
|
||||||
|
recorder,
|
||||||
|
httptest.NewRequestWithContext(
|
||||||
|
t.Context(),
|
||||||
|
http.MethodGet,
|
||||||
|
waf.MetricsPath,
|
||||||
|
http.NoBody,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
if strings.Contains(
|
||||||
|
recorder.Body.String(),
|
||||||
|
"sww_core_rule_set_matches_total{",
|
||||||
|
) {
|
||||||
|
t.Errorf("a Core Rule Set match is counted")
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, alert := range queue.Snapshot().Waiting[alerts.DestinationWebhook] {
|
||||||
|
if alert.Event == alerts.EventCoreRuleSetMatch {
|
||||||
|
t.Errorf("alert raised: %+v", alert)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheLookupAndTheCoreRuleSetTakeTheLongerOfTheirTimesNotTheirSum(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
inspection := checksTimeLimit - time.Millisecond
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
// wait is how long the request waits for GeoJS.
|
||||||
|
wait time.Duration
|
||||||
|
}{
|
||||||
|
{"the lookup waits longer", defaultLookupWait},
|
||||||
|
{"the inspection takes longer", inspection / 2},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
crs := &heldCoreRuleSet{took: inspection}
|
||||||
|
server, out, _ := newWAFHoldingTheCoreRuleSet(
|
||||||
|
t,
|
||||||
|
crs,
|
||||||
|
map[string]string{
|
||||||
|
asnAndCountryLookupTimeout: lookupWait(tc.wait),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
|
status, took := askDirectly(
|
||||||
|
t,
|
||||||
|
server,
|
||||||
|
func(*http.Request) {},
|
||||||
|
)
|
||||||
|
|
||||||
|
wantAnswer(t, status, took, http.StatusOK,
|
||||||
|
max(tc.wait, inspection), out)
|
||||||
|
wantLine(t, out.requestLine(t), http.StatusOK,
|
||||||
|
requestlog.ActionForward)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// firstToRefuse is a case of
|
||||||
|
// TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered: a request
|
||||||
|
// under env, whose inspection by the Core Rule Set takes inspection, refused as
|
||||||
|
// action after took.
|
||||||
|
type firstToRefuse struct {
|
||||||
|
name string
|
||||||
|
env map[string]string
|
||||||
|
inspection time.Duration
|
||||||
|
took time.Duration
|
||||||
|
action string
|
||||||
|
}
|
||||||
|
|
||||||
|
// firstToRefuseCases are the cases of
|
||||||
|
// TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered.
|
||||||
|
func firstToRefuseCases() []firstToRefuse {
|
||||||
|
return []firstToRefuse{
|
||||||
|
{
|
||||||
|
name: "a trap path, before GeoJS answers",
|
||||||
|
env: map[string]string{trapPaths: "/", allowedCountries: "DE"},
|
||||||
|
took: 0, action: requestlog.ActionTemporarilyBanned,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "the Core Rule Set, before GeoJS answers",
|
||||||
|
env: map[string]string{allowedCountries: "DE"},
|
||||||
|
inspection: 100 * time.Millisecond,
|
||||||
|
took: 100 * time.Millisecond, action: requestlog.ActionBlockedByCoreRuleSet,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "the country list, while the Core Rule Set inspects",
|
||||||
|
env: map[string]string{
|
||||||
|
allowedCountries: "DE", asnAndCountryLookupTimeout: "100",
|
||||||
|
},
|
||||||
|
inspection: 400 * time.Millisecond,
|
||||||
|
took: 100 * time.Millisecond, action: requestlog.ActionCountryDenied,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "the rate limits, while the Core Rule Set inspects",
|
||||||
|
env: map[string]string{
|
||||||
|
rateLimitPerMinute: "2", unknownCountryLimitPercent: "50",
|
||||||
|
asnAndCountryLookupTimeout: "100",
|
||||||
|
},
|
||||||
|
inspection: 400 * time.Millisecond,
|
||||||
|
took: 100 * time.Millisecond, action: requestlog.ActionClientIsRateLimited,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const forbidden = http.StatusForbidden
|
||||||
|
|
||||||
|
for _, tc := range firstToRefuseCases() {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
crs := &heldCoreRuleSet{
|
||||||
|
took: tc.inspection,
|
||||||
|
result: coreRuleSetMatch(),
|
||||||
|
}
|
||||||
|
server, out, queue := newWAFHoldingTheCoreRuleSet(t, crs, tc.env)
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
|
// Under the rate limits of the last case, the client's next
|
||||||
|
// request takes it over a limit only at the percentage of an
|
||||||
|
// unknown country, so it waits for GeoJS.
|
||||||
|
countClient(t, server, time.Now())
|
||||||
|
|
||||||
|
status, took := askDirectly(
|
||||||
|
t,
|
||||||
|
server,
|
||||||
|
func(*http.Request) {},
|
||||||
|
)
|
||||||
|
|
||||||
|
wantAnswer(t, status, took, forbidden, tc.took, out)
|
||||||
|
|
||||||
|
line := out.requestLine(t)
|
||||||
|
wantLine(t, line, forbidden, tc.action)
|
||||||
|
|
||||||
|
if tc.action == requestlog.ActionBlockedByCoreRuleSet {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if tc.action != requestlog.ActionClientIsRateLimited &&
|
||||||
|
line.Counts != (ratelimit.Counts{}) {
|
||||||
|
t.Errorf(
|
||||||
|
"the rate limits counted the request: %+v",
|
||||||
|
line.Counts,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// What the Core Rule Set finds once the request is answered is
|
||||||
|
// dropped.
|
||||||
|
time.Sleep(tc.inspection)
|
||||||
|
wantNoCoreRuleSetTrace(t, server, queue, line)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestARequestRefusedBeforeTheLookupStartsNoInspection(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
crs := &heldCoreRuleSet{took: time.Second}
|
||||||
|
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs, nil)
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
banClient(t, server, time.Now())
|
||||||
|
|
||||||
|
status, took := askDirectly(t, server, func(*http.Request) {})
|
||||||
|
|
||||||
|
wantAnswer(t, status, took, http.StatusForbidden, 0, out)
|
||||||
|
|
||||||
|
requests := testutil.ToFloat64(server.Metrics.GeoJSRequests)
|
||||||
|
if asked := crs.asked.Load(); asked != 0 || requests != 0 {
|
||||||
|
t.Errorf(
|
||||||
|
"%d inspections and %v requests to GeoJS, want none",
|
||||||
|
asked,
|
||||||
|
requests,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAClientInTheAllowNetsIsAllowedAtOnceAndNothingElseStarts(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
crs := &heldCoreRuleSet{took: time.Second, result: coreRuleSetMatch()}
|
||||||
|
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
|
||||||
|
map[string]string{allowNets: client, allowedCountries: "DE"})
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
status, took := askDirectly(t, server, func(*http.Request) {})
|
||||||
|
|
||||||
|
wantAnswer(t, status, took, http.StatusOK, 0, out)
|
||||||
|
|
||||||
|
requests := testutil.ToFloat64(server.Metrics.GeoJSRequests)
|
||||||
|
if asked := crs.asked.Load(); asked != 0 || requests != 0 {
|
||||||
|
t.Errorf(
|
||||||
|
"%d inspections and %v requests to GeoJS, want none",
|
||||||
|
asked,
|
||||||
|
requests,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestARequestTheCoreRuleSetHasNotFinishedWithIn500MillisecondsIsRefused(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
crs := &heldCoreRuleSet{took: checksTimeLimit + time.Second}
|
||||||
|
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
|
||||||
|
map[string]string{refusalsLimitPerMinute: "1"})
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
|
// Were it counted toward the refusal limit, the second refusal would
|
||||||
|
// ban the client, and the third request would be refused for the ban.
|
||||||
|
for i := range 3 {
|
||||||
|
status, took := askDirectly(t, server, func(req *http.Request) {
|
||||||
|
req.Header.Set(requestIDHeader, "id-"+strconv.Itoa(i))
|
||||||
|
})
|
||||||
|
|
||||||
|
wantAnswer(t, status, took, checksTimedOut, checksTimeLimit, out)
|
||||||
|
|
||||||
|
line := out.requestLines(t, i+1)[i]
|
||||||
|
wantLine(
|
||||||
|
t,
|
||||||
|
line,
|
||||||
|
checksTimedOut,
|
||||||
|
requestlog.ActionChecksTimedOut,
|
||||||
|
)
|
||||||
|
wantHeuristic(t, line, "core_rule_set", `{"time_limit_ms": 500}`)
|
||||||
|
|
||||||
|
if line.Offence != "" || line.BanExpires != "" {
|
||||||
|
t.Errorf(
|
||||||
|
"log line has offence %q and ban_expires %q, want neither",
|
||||||
|
line.Offence,
|
||||||
|
line.BanExpires,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if bans := server.Ledger.Snapshot(); len(bans) != 0 {
|
||||||
|
t.Errorf("bans %+v, want none", bans)
|
||||||
|
}
|
||||||
|
|
||||||
|
offences := historyOf(t, server, client).Offences
|
||||||
|
if offences != (ratelimit.Offences{}) {
|
||||||
|
t.Errorf("history counts the offences %+v, want none", offences)
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(crs.took)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestObserveModeAllowsARequestTheCoreRuleSetHasNotFinishedWith(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
crs := &heldCoreRuleSet{took: checksTimeLimit + time.Second}
|
||||||
|
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
|
||||||
|
map[string]string{mode: observe})
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
|
status, took := askDirectly(t, server, func(*http.Request) {})
|
||||||
|
|
||||||
|
wantAnswer(t, status, took, http.StatusOK, checksTimeLimit, out)
|
||||||
|
|
||||||
|
line := out.requestLine(t)
|
||||||
|
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||||
|
|
||||||
|
if line.WouldAction != requestlog.ActionChecksTimedOut {
|
||||||
|
t.Errorf(
|
||||||
|
"log line has would_action %q, want %q",
|
||||||
|
line.WouldAction,
|
||||||
|
requestlog.ActionChecksTimedOut,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(crs.took)
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -339,6 +339,8 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownCountryLimitPercent(
|
|||||||
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, with
|
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, with
|
||||||
// a GeoJS that never answers.
|
// a GeoJS that never answers.
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
||||||
map[string]string{unknownCountryLimitPercent: "0"})
|
map[string]string{unknownCountryLimitPercent: "0"})
|
||||||
|
|
||||||
@@ -390,6 +392,8 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
|
|||||||
// with a GeoJS that never answers. The headers, which need the
|
// with a GeoJS that never answers. The headers, which need the
|
||||||
// answer too, are off.
|
// answer too, are off.
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
||||||
map[string]string{
|
map[string]string{
|
||||||
asnAndCountryLookupTimeout: strconv.FormatInt(
|
asnAndCountryLookupTimeout: strconv.FormatInt(
|
||||||
|
|||||||
+23
-15
@@ -13,10 +13,11 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
// coreRuleSet inspects the request with the Core Rule Set, unless
|
// coreRuleSet takes what the Core Rule Set found of the request, which
|
||||||
// SWW_WAF_CORE_RULE_SET_MODE is off or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS
|
// findBesideLookup inspected with it unless SWW_WAF_CORE_RULE_SET_MODE is off
|
||||||
// exempts its path, as pathExempt decides, and notes the rules it matched and
|
// or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS exempts its path, as pathExempt
|
||||||
// its score in the log line, and the rules in the metrics. A score at or over
|
// decides. It notes the rules the request matched and its score in the log
|
||||||
|
// line, and the rules in the metrics. A score at or over
|
||||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD is a match: it adds the tag
|
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD is a match: it adds the tag
|
||||||
// exploitsignature, raises the core_rule_set_match alert, and in block mode
|
// exploitsignature, raises the core_rule_set_match alert, and in block mode
|
||||||
// refuses the request with 403, which is an offence its client's history
|
// refuses the request with 403, which is an offence its client's history
|
||||||
@@ -30,21 +31,17 @@ func (coreRuleSet) check(
|
|||||||
rq *request,
|
rq *request,
|
||||||
_ time.Time,
|
_ time.Time,
|
||||||
) *verdict {
|
) *verdict {
|
||||||
cfg := rq.h.config
|
findings := rq.findings
|
||||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
|
if !findings.inspected {
|
||||||
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
start := time.Now()
|
cfg := rq.h.config
|
||||||
|
result := findings.inspection
|
||||||
// A question's body has been read into memory, so that reading it here
|
rq.requestBytes = int64(len(findings.read))
|
||||||
// cannot fail.
|
|
||||||
result, read, _ := rq.h.coreRuleSet.Inspect(rq.in, rq.client, rq.in.Body)
|
|
||||||
rq.requestBytes = int64(len(read))
|
|
||||||
|
|
||||||
rq.line.DurationCoreRuleSetMS = new(
|
rq.line.DurationCoreRuleSetMS = new(
|
||||||
requestlog.Milliseconds(time.Since(start)),
|
requestlog.Milliseconds(findings.inspectionTime),
|
||||||
)
|
)
|
||||||
rq.line.CoreRuleSetRuleIDs = result.RuleIDs
|
rq.line.CoreRuleSetRuleIDs = result.RuleIDs
|
||||||
rq.line.CoreRuleSetScore = &result.Score
|
rq.line.CoreRuleSetScore = &result.Score
|
||||||
@@ -61,7 +58,7 @@ func (coreRuleSet) check(
|
|||||||
rq.addTag(config.ExploitSignatureTag)
|
rq.addTag(config.ExploitSignatureTag)
|
||||||
rq.alertCoreRuleSetMatch(result)
|
rq.alertCoreRuleSetMatch(result)
|
||||||
|
|
||||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeDetect {
|
if !rq.refusedByCoreRuleSet(result) {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -76,6 +73,17 @@ func (coreRuleSet) check(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core
|
||||||
|
// Rule Set found result of: whether the score is at or over
|
||||||
|
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode.
|
||||||
|
func (rq *request) refusedByCoreRuleSet(result coreruleset.Result) bool {
|
||||||
|
cfg := rq.h.config
|
||||||
|
threshold := cfg.CoreRuleSetScoreThreshold
|
||||||
|
|
||||||
|
return cfg.CoreRuleSetMode == config.CoreRuleSetModeBlock &&
|
||||||
|
threshold != 0 && result.Score >= threshold
|
||||||
|
}
|
||||||
|
|
||||||
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
|
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
|
||||||
// which the Core Rule Set scored at result, at or over
|
// which the Core Rule Set scored at result, at or over
|
||||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
|
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
|
||||||
|
|||||||
@@ -0,0 +1,13 @@
|
|||||||
|
package waf
|
||||||
|
|
||||||
|
import "time"
|
||||||
|
|
||||||
|
// BesideLookupTimeLimit is besideLookupTimeLimit.
|
||||||
|
const BesideLookupTimeLimit = besideLookupTimeLimit
|
||||||
|
|
||||||
|
// SetBesideLookupTimeLimit gives the heuristics of besideLookup limit on s in
|
||||||
|
// place of besideLookupTimeLimit.
|
||||||
|
func (s *Server) SetBesideLookupTimeLimit(limit time.Duration) {
|
||||||
|
h := s.Handler.(*handler) //nolint:forcetypeassert // New makes it one
|
||||||
|
h.besideLookupLimit = limit
|
||||||
|
}
|
||||||
@@ -54,8 +54,6 @@ func beforeLookup() []heuristic {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// afterLookup are the heuristics that run once the lookup has been made.
|
// afterLookup are the heuristics that run once the lookup has been made.
|
||||||
// tag_deny runs twice: before the rate limits, on the tags known by then, and
|
|
||||||
// last, on the tag rule_files and core_rule_set can add.
|
|
||||||
func afterLookup() []heuristic {
|
func afterLookup() []heuristic {
|
||||||
return []heuristic{
|
return []heuristic{
|
||||||
countries{},
|
countries{},
|
||||||
@@ -64,9 +62,26 @@ func afterLookup() []heuristic {
|
|||||||
abuseIPDB{},
|
abuseIPDB{},
|
||||||
tagDeny{},
|
tagDeny{},
|
||||||
rateLimits{},
|
rateLimits{},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// besideLookup are the heuristics that only read the request. Their work runs
|
||||||
|
// beside the lookup and afterLookup, in findBesideLookup, which changes nothing
|
||||||
|
// of the request. Their check then applies what that work found, and runs only
|
||||||
|
// once waf takes it.
|
||||||
|
func besideLookup() []heuristic {
|
||||||
|
return []heuristic{
|
||||||
trapPaths{},
|
trapPaths{},
|
||||||
ruleFiles{},
|
ruleFiles{},
|
||||||
coreRuleSet{},
|
coreRuleSet{},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// afterBesideLookup are the heuristics that run once afterLookup and
|
||||||
|
// besideLookup have both let the request through. tag_deny runs a second time,
|
||||||
|
// on the tag rule_files and core_rule_set can add.
|
||||||
|
func afterBesideLookup() []heuristic {
|
||||||
|
return []heuristic{
|
||||||
tagDeny{},
|
tagDeny{},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -95,7 +110,7 @@ func (rq *request) runAfterAnswer(ctx context.Context) {
|
|||||||
rq.runBanningHeuristics(ctx, afterAnswer())
|
rq.runBanningHeuristics(ctx, afterAnswer())
|
||||||
}
|
}
|
||||||
|
|
||||||
// runHeuristics runs list, beforeLookup or afterLookup, on the request at now,
|
// runHeuristics runs list, one of those checkClient runs, on the request at now,
|
||||||
// and returns the first verdict, or nil when none gives one. The log line's
|
// and returns the first verdict, or nil when none gives one. The log line's
|
||||||
// heuristic and heuristic_figures name the heuristic that gave it, and give its
|
// heuristic and heuristic_figures name the heuristic that gave it, and give its
|
||||||
// figures. ctx is the request's own context.
|
// figures. ctx is the request's own context.
|
||||||
|
|||||||
+85
-21
@@ -7,8 +7,9 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The headers in which an answer of allow gives rproxy the client's AS number
|
// The headers in which an answer of allow gives rproxy the client's AS number
|
||||||
@@ -20,33 +21,80 @@ const (
|
|||||||
countryHeader = "X-Client-Country"
|
countryHeader = "X-Client-Country"
|
||||||
)
|
)
|
||||||
|
|
||||||
// lookUp looks up the client's AS number and country, in the lookup
|
// lookupResult is what the lookup gave for a request: whether its client was
|
||||||
// database or through GeoJS, and notes them for the log line and as the
|
// looked up, the answer, and whether the request skipped the wait for GeoJS.
|
||||||
// request's asn= and country= tags, unless
|
type lookupResult struct {
|
||||||
// SWW_WAF_LOOKUP_SOURCE is off or the client is on a private, loopback or
|
lookedUp, skippedWait bool
|
||||||
// link-local address, which no lookup can place. The lookup database
|
answer geojs.Answer
|
||||||
// answers at once. With GeoJS, while a setting needs the answer, such as a
|
}
|
||||||
// country list or a biased threshold, a new client's request waits for it,
|
|
||||||
// unless refusedWhateverTheAnswer reports true. ctx is the request's own
|
// startLookUp starts looking up the client's AS number and country, in the
|
||||||
// context.
|
// lookup database or through GeoJS, unless SWW_WAF_LOOKUP_SOURCE is off or the
|
||||||
func (rq *request) lookUp(ctx context.Context, now time.Time) {
|
// client is on a private, loopback or link-local address, which no lookup can
|
||||||
|
// place. It returns the channel the result comes on. A result that needs no
|
||||||
|
// wait is on it when startLookUp returns: the lookup database's, an answer
|
||||||
|
// GeoJS gave earlier, or none. With GeoJS, while a setting needs the answer,
|
||||||
|
// such as a country list or a biased threshold, a new client's request waits
|
||||||
|
// for it up to SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, unless
|
||||||
|
// refusedWhateverTheAnswer reports true. The wait goes on to its end even once
|
||||||
|
// the request has been answered, so that it ends as it would have for a
|
||||||
|
// request that waited. startLookUp changes nothing of the request. ctx is the
|
||||||
|
// request's own context.
|
||||||
|
func (rq *request) startLookUp(
|
||||||
|
ctx context.Context, now time.Time,
|
||||||
|
) <-chan lookupResult {
|
||||||
|
result := make(chan lookupResult, 1)
|
||||||
|
|
||||||
if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) {
|
if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) {
|
||||||
return
|
result <- lookupResult{}
|
||||||
|
|
||||||
|
return result
|
||||||
}
|
}
|
||||||
|
|
||||||
client := rq.h.clientGroup(rq.client)
|
client := rq.h.clientGroup(rq.client)
|
||||||
|
|
||||||
switch {
|
if rq.h.config.LookupSource == "file" {
|
||||||
case rq.h.config.LookupSource == "file":
|
result <- lookupResult{
|
||||||
rq.lookupAnswer = rq.h.lookupFile.LookUp(client)
|
lookedUp: true, answer: rq.h.lookupFile.LookUp(client),
|
||||||
case rq.refusedWhateverTheAnswer(now):
|
}
|
||||||
rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client)
|
|
||||||
rq.skippedLookupWait = rq.lookupAnswer.Answered.IsZero()
|
return result
|
||||||
default:
|
}
|
||||||
rq.lookupAnswer = rq.h.geojs.LookUp(ctx, client)
|
|
||||||
|
answer := rq.h.geojs.LookUpWithoutWaiting(ctx, client)
|
||||||
|
refused := rq.refusedWhateverTheAnswer(now)
|
||||||
|
|
||||||
|
if !answer.Answered.IsZero() || refused || !rq.h.waitForGeoJS {
|
||||||
|
result <- lookupResult{
|
||||||
|
lookedUp: true,
|
||||||
|
skippedWait: refused && answer.Answered.IsZero(),
|
||||||
|
answer: answer,
|
||||||
|
}
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
result <- lookupResult{
|
||||||
|
lookedUp: true,
|
||||||
|
answer: rq.h.geojs.LookUp(context.WithoutCancel(ctx), client,
|
||||||
|
rq.h.config.ASNAndCountryLookupTimeout),
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// noteLookup notes result for the log line and as the request's asn= and
|
||||||
|
// country= tags.
|
||||||
|
func (rq *request) noteLookup(result lookupResult) {
|
||||||
|
if !result.lookedUp {
|
||||||
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.lookedUp = true
|
rq.lookedUp = true
|
||||||
|
rq.lookupAnswer = result.answer
|
||||||
|
rq.skippedLookupWait = result.skippedWait
|
||||||
rq.line.ASN = rq.lookupAnswer.ASN
|
rq.line.ASN = rq.lookupAnswer.ASN
|
||||||
rq.line.ASName = rq.lookupAnswer.ASName
|
rq.line.ASName = rq.lookupAnswer.ASName
|
||||||
rq.line.Country = rq.lookupAnswer.Country
|
rq.line.Country = rq.lookupAnswer.Country
|
||||||
@@ -83,13 +131,29 @@ func countryTag(country string) string {
|
|||||||
// addLookup adds answer, an answer about a client from the lookup
|
// addLookup adds answer, an answer about a client from the lookup
|
||||||
// database or GeoJS, to the client's history, and to the notes of the bans
|
// database or GeoJS, to the client's history, and to the notes of the bans
|
||||||
// on its netblock that have no AS number, AS name or country yet.
|
// on its netblock that have no AS number, AS name or country yet.
|
||||||
func (h *handler) addLookup(answer lookup.Answer) {
|
func (h *handler) addLookup(answer geojs.Answer) {
|
||||||
h.limiter.AddLookup(answer.Client, answer.Answered,
|
h.limiter.AddLookup(answer.Client, answer.Answered,
|
||||||
answer.ASN, answer.ASName, answer.Country)
|
answer.ASN, answer.ASName, answer.Country)
|
||||||
h.ledger.AddLookup(h.netblock(answer.Client.Addr()),
|
h.ledger.AddLookup(h.netblock(answer.Client.Addr()),
|
||||||
answer.ASN, answer.ASName, answer.Country)
|
answer.ASN, answer.ASName, answer.Country)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// geojsFailed logs a failure of GeoJS, raises a source_failure alert for it
|
||||||
|
// and counts it. askingAgainIn is how long GeoJS is left alone after it.
|
||||||
|
func (h *handler) geojsFailed(err error, askingAgainIn time.Duration) {
|
||||||
|
h.processLog.Warn("asking GeoJS failed",
|
||||||
|
"error", err.Error(), "asking_again_in", askingAgainIn.String())
|
||||||
|
h.alerts.Raise(alerts.Alert{
|
||||||
|
Event: alerts.EventSourceFailure,
|
||||||
|
Reason: "asking GeoJS failed",
|
||||||
|
Detail: map[string]any{
|
||||||
|
"source": "geojs", "error": err.Error(),
|
||||||
|
"asking_again_in": askingAgainIn.String(),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
h.metrics.GeoJSFailures.Inc()
|
||||||
|
}
|
||||||
|
|
||||||
// setLookupHeaders sets the headers that give the client's AS number and
|
// setLookupHeaders sets the headers that give the client's AS number and
|
||||||
// country, leaving out one that is unknown.
|
// country, leaving out one that is unknown.
|
||||||
func setLookupHeaders(header http.Header, asn, country string) {
|
func setLookupHeaders(header http.Header, asn, country string) {
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
@@ -138,7 +138,7 @@ func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) {
|
|||||||
// of lookups.json would keep it, while the first request still waits
|
// of lookups.json would keep it, while the first request still waits
|
||||||
// for GeoJS. The second request goes on at once, and is counted.
|
// for GeoJS. The second request goes on at once, and is counted.
|
||||||
clk.advance(2500 * time.Millisecond)
|
clk.advance(2500 * time.Millisecond)
|
||||||
server.GeoJS.Load([]lookup.Answer{{
|
server.GeoJS.Load([]geojs.Answer{{
|
||||||
Client: netip.MustParsePrefix(
|
Client: netip.MustParsePrefix(
|
||||||
fromDE + "/32",
|
fromDE + "/32",
|
||||||
), ASN: asnDE, ASName: asNameDE,
|
), ASN: asnDE, ASName: asNameDE,
|
||||||
@@ -190,7 +190,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(
|
|||||||
clk.Now,
|
clk.Now,
|
||||||
map[string]string{
|
map[string]string{
|
||||||
trustedProxies: trustLocalhost,
|
trustedProxies: trustLocalhost,
|
||||||
alertWebhookURL: "https://alerts.example/sww",
|
alertWebhookURL: alertWebhook,
|
||||||
rateLimitPerMinute: "1",
|
rateLimitPerMinute: "1",
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
@@ -199,7 +199,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(
|
|||||||
// The answer is kept before the requests, so GeoJS is not asked, and
|
// The answer is kept before the requests, so GeoJS is not asked, and
|
||||||
// gives no answer of its own.
|
// gives no answer of its own.
|
||||||
netblock := netip.MustParsePrefix(fromDE + "/32")
|
netblock := netip.MustParsePrefix(fromDE + "/32")
|
||||||
server.GeoJS.Load([]lookup.Answer{{
|
server.GeoJS.Load([]geojs.Answer{{
|
||||||
Client: netblock, ASN: asnDE, ASName: asNameDE, Country: "DE",
|
Client: netblock, ASN: asnDE, ASName: asNameDE, Country: "DE",
|
||||||
Answered: clk.Now(), Used: clk.Now(),
|
Answered: clk.Now(), Used: clk.Now(),
|
||||||
}})
|
}})
|
||||||
@@ -249,7 +249,7 @@ func TestLookupSourceOffLooksNoClientUp(t *testing.T) {
|
|||||||
)
|
)
|
||||||
|
|
||||||
// Even an answer kept from before is not used.
|
// Even an answer kept from before is not used.
|
||||||
server.GeoJS.Load([]lookup.Answer{{
|
server.GeoJS.Load([]geojs.Answer{{
|
||||||
Client: netip.MustParsePrefix(
|
Client: netip.MustParsePrefix(
|
||||||
fromDE + "/32",
|
fromDE + "/32",
|
||||||
), ASN: asnDE, ASName: asNameDE,
|
), ASN: asnDE, ASName: asNameDE,
|
||||||
@@ -450,6 +450,8 @@ func TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays(
|
|||||||
// network, which would keep that clock from moving on: the request is
|
// network, which would keep that clock from moving on: the request is
|
||||||
// handed to waf's handler, and GeoJS is one that never answers.
|
// handed to waf's handler, and GeoJS is one that never answers.
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
// Not the default. The exclusive list needs the answer, and the app
|
// Not the default. The exclusive list needs the answer, and the app
|
||||||
// is never reached.
|
// is never reached.
|
||||||
const timeout = 3 * time.Second
|
const timeout = 3 * time.Second
|
||||||
@@ -495,6 +497,17 @@ func TestMain(m *testing.M) {
|
|||||||
m.Run()
|
m.Run()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// geojsRequestTimeout is how long a request to GeoJS may take before it is
|
||||||
|
// abandoned.
|
||||||
|
const geojsRequestTimeout = 10 * time.Second
|
||||||
|
|
||||||
|
// waitForGeoJSRequestsToBeAbandoned sleeps until every request to the GeoJS at
|
||||||
|
// silentGeoJSURL made so far has been abandoned. A synctest bubble that asks
|
||||||
|
// it defers this: the bubble cannot end while a request waits on it.
|
||||||
|
func waitForGeoJSRequestsToBeAbandoned() {
|
||||||
|
time.Sleep(geojsRequestTimeout)
|
||||||
|
}
|
||||||
|
|
||||||
// silentGeoJS is the GeoJS at silentGeoJSURL.
|
// silentGeoJS is the GeoJS at silentGeoJSURL.
|
||||||
type silentGeoJS struct{}
|
type silentGeoJS struct{}
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package waf_test
|
package waf_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
@@ -10,8 +11,9 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
)
|
)
|
||||||
@@ -248,7 +250,7 @@ func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) {
|
|||||||
|
|
||||||
// The answers are kept before the requests, so that none waits for
|
// The answers are kept before the requests, so that none waits for
|
||||||
// GeoJS.
|
// GeoJS.
|
||||||
server.GeoJS.Load([]lookup.Answer{
|
server.GeoJS.Load([]geojs.Answer{
|
||||||
keptAnswer(fromKP, "KP"), keptAnswer(fromDE, "DE"),
|
keptAnswer(fromKP, "KP"), keptAnswer(fromDE, "DE"),
|
||||||
keptAnswer(fromFR, "FR"),
|
keptAnswer(fromFR, "FR"),
|
||||||
})
|
})
|
||||||
@@ -329,13 +331,13 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther(
|
|||||||
|
|
||||||
// The answers are kept before the requests, so that GeoJS gives none
|
// The answers are kept before the requests, so that GeoJS gives none
|
||||||
// of its own. Each client is in an AS of its own.
|
// of its own. Each client is in an AS of its own.
|
||||||
answer := func(addr, asn string) lookup.Answer {
|
answer := func(addr, asn string) geojs.Answer {
|
||||||
return lookup.Answer{
|
return geojs.Answer{
|
||||||
Client: netip.MustParsePrefix(addr + "/32"), ASN: asn,
|
Client: netip.MustParsePrefix(addr + "/32"), ASN: asn,
|
||||||
Answered: clk.Now(), Used: clk.Now(),
|
Answered: clk.Now(), Used: clk.Now(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
server.GeoJS.Load([]lookup.Answer{
|
server.GeoJS.Load([]geojs.Answer{
|
||||||
answer(fromDE, "AS64501"), answer(fromKP, "AS64502"),
|
answer(fromDE, "AS64501"), answer(fromKP, "AS64502"),
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -357,21 +359,23 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther(
|
|||||||
`sww_asn_requests_total{asn="AS64502",instance="app"}`)
|
`sww_asn_requests_total{asn="AS64502",instance="app"}`)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
|
func TestGeoJSFailureIsCountedLoggedAndAlerted(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
geojs := httptest.NewServer(http.HandlerFunc(
|
failingGeoJS := httptest.NewServer(http.HandlerFunc(
|
||||||
func(w http.ResponseWriter, _ *http.Request) {
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
w.WriteHeader(http.StatusServiceUnavailable)
|
w.WriteHeader(http.StatusServiceUnavailable)
|
||||||
}))
|
}))
|
||||||
t.Cleanup(geojs.Close)
|
t.Cleanup(failingGeoJS.Close)
|
||||||
|
|
||||||
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
||||||
addr, _, server := startProxyWithClock(t, app.URL, geojs.URL, time.Now,
|
addr, out, server, queue := startProxyWithAlerts(t, app.URL,
|
||||||
|
failingGeoJS.URL, time.Now,
|
||||||
map[string]string{
|
map[string]string{
|
||||||
trustedProxies: trustLocalhost,
|
trustedProxies: trustLocalhost,
|
||||||
metricsToken: token,
|
metricsToken: token,
|
||||||
tagDeny: denyKP,
|
tagDeny: denyKP,
|
||||||
|
alertWebhookURL: "https://alerts.example/waf",
|
||||||
})
|
})
|
||||||
|
|
||||||
// GeoJS fails, so the client has no country= tag, and SWW_WAF_TAG_DENY does
|
// GeoJS fails, so the client has no country= tag, and SWW_WAF_TAG_DENY does
|
||||||
@@ -401,14 +405,46 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
|
|||||||
`sww_requests_without_geojs_answer_total{instance="app"}`,
|
`sww_requests_without_geojs_answer_total{instance="app"}`,
|
||||||
1,
|
1,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// The process log's line names the failure but not the address asked
|
||||||
|
// about.
|
||||||
|
logged := ""
|
||||||
|
|
||||||
|
for line := range strings.Lines(out.text()) {
|
||||||
|
if strings.Contains(line, "asking GeoJS failed") {
|
||||||
|
logged = line
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if logged == "" || strings.Contains(logged, fromDE) {
|
||||||
|
t.Errorf(
|
||||||
|
"logged %q, want the failure without the address asked about",
|
||||||
|
logged,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// fmt prints a map sorted by its keys.
|
||||||
|
const want = "map[asking_again_in:1s " +
|
||||||
|
"error:GeoJS answered 503 Service Unavailable source:geojs]"
|
||||||
|
|
||||||
|
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||||
|
if len(waiting) != 1 || waiting[0].Event != alerts.EventSourceFailure ||
|
||||||
|
waiting[0].Reason != "asking GeoJS failed" ||
|
||||||
|
fmt.Sprint(waiting[0].Detail) != want {
|
||||||
|
t.Errorf(
|
||||||
|
"alerts waiting %+v, want one source_failure alert with %v",
|
||||||
|
waiting,
|
||||||
|
want,
|
||||||
|
)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// keptAnswer returns GeoJS's answer that the client at addr is in
|
// keptAnswer returns GeoJS's answer that the client at addr is in
|
||||||
// country, given now.
|
// country, given now.
|
||||||
func keptAnswer(addr, country string) lookup.Answer {
|
func keptAnswer(addr, country string) geojs.Answer {
|
||||||
now := time.Now()
|
now := time.Now()
|
||||||
|
|
||||||
return lookup.Answer{
|
return geojs.Answer{
|
||||||
Client: netip.MustParsePrefix(addr + "/32"), Country: country,
|
Client: netip.MustParsePrefix(addr + "/32"), Country: country,
|
||||||
Answered: now, Used: now,
|
Answered: now, Used: now,
|
||||||
}
|
}
|
||||||
|
|||||||
+35
-19
@@ -8,6 +8,7 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"net/netip"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
lru "github.com/hashicorp/golang-lru/v2"
|
lru "github.com/hashicorp/golang-lru/v2"
|
||||||
@@ -17,6 +18,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
@@ -55,7 +57,7 @@ type Params struct {
|
|||||||
// ProcessLog receives the process's own messages.
|
// ProcessLog receives the process's own messages.
|
||||||
ProcessLog *slog.Logger
|
ProcessLog *slog.Logger
|
||||||
// GeoJSURL is where clients' AS numbers and countries are looked up
|
// GeoJSURL is where clients' AS numbers and countries are looked up
|
||||||
// while SWW_WAF_LOOKUP_SOURCE is geojs, normally lookup.URL.
|
// while SWW_WAF_LOOKUP_SOURCE is geojs, normally geojs.URL.
|
||||||
GeoJSURL string
|
GeoJSURL string
|
||||||
// AbuseIPDBURL is where clients are checked with AbuseIPDB while
|
// AbuseIPDBURL is where clients are checked with AbuseIPDB while
|
||||||
// SWW_WAF_ABUSEIPDB_KEY is set, normally reputation.AbuseIPDBURL.
|
// SWW_WAF_ABUSEIPDB_KEY is set, normally reputation.AbuseIPDBURL.
|
||||||
@@ -65,7 +67,7 @@ type Params struct {
|
|||||||
LookupFile *lookup.File
|
LookupFile *lookup.File
|
||||||
// CoreRuleSet is what NewCoreRuleSet returns: what inspects requests
|
// CoreRuleSet is what NewCoreRuleSet returns: what inspects requests
|
||||||
// while SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
|
// while SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
|
||||||
CoreRuleSet *coreruleset.CoreRuleSet
|
CoreRuleSet CoreRuleSetInspector
|
||||||
// Now tells the time by which requests are counted for the rate
|
// Now tells the time by which requests are counted for the rate
|
||||||
// limits, bans are made and run out, and GeoJS's answers are kept,
|
// limits, bans are made and run out, and GeoJS's answers are kept,
|
||||||
// normally time.Now in UTC, the time the state files give.
|
// normally time.Now in UTC, the time the state files give.
|
||||||
@@ -83,6 +85,14 @@ type Params struct {
|
|||||||
Alerts *alerts.Queue
|
Alerts *alerts.Queue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// CoreRuleSetInspector inspects a request with the Core Rule Set, as
|
||||||
|
// coreruleset.CoreRuleSet's Inspect says. Tests give waf a stand-in for it.
|
||||||
|
type CoreRuleSetInspector interface {
|
||||||
|
Inspect(
|
||||||
|
r *http.Request, client netip.Addr, body io.Reader,
|
||||||
|
) (coreruleset.Result, []byte, error)
|
||||||
|
}
|
||||||
|
|
||||||
// Server is the server waf runs, with the parts whose state the state
|
// Server is the server waf runs, with the parts whose state the state
|
||||||
// files keep, the lookup database, nil unless SWW_WAF_LOOKUP_SOURCE is file, the
|
// files keep, the lookup database, nil unless SWW_WAF_LOOKUP_SOURCE is file, the
|
||||||
// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts,
|
// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts,
|
||||||
@@ -93,7 +103,7 @@ type Server struct {
|
|||||||
|
|
||||||
Ledger *bans.Ledger
|
Ledger *bans.Ledger
|
||||||
Limiter *ratelimit.Limiter
|
Limiter *ratelimit.Limiter
|
||||||
GeoJS *lookup.GeoJS
|
GeoJS *geojs.GeoJS
|
||||||
Anomalies *anomaly.Counters
|
Anomalies *anomaly.Counters
|
||||||
LookupFile *lookup.File
|
LookupFile *lookup.File
|
||||||
Lists *reputation.Lists
|
Lists *reputation.Lists
|
||||||
@@ -157,6 +167,8 @@ func newHandler(params Params) *handler {
|
|||||||
rules: params.Rules,
|
rules: params.Rules,
|
||||||
coreRuleSet: params.CoreRuleSet,
|
coreRuleSet: params.CoreRuleSet,
|
||||||
alerts: params.Alerts,
|
alerts: params.Alerts,
|
||||||
|
|
||||||
|
besideLookupLimit: besideLookupTimeLimit,
|
||||||
}
|
}
|
||||||
if params.Config.RouteWatchURL != "" {
|
if params.Config.RouteWatchURL != "" {
|
||||||
h.routeWatch = routewatch.New(routewatch.Params{
|
h.routeWatch = routewatch.New(routewatch.Params{
|
||||||
@@ -164,20 +176,19 @@ func newHandler(params Params) *handler {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
h.geojs = lookup.New(lookup.Params{
|
// The country list, the headers, the tag settings that name an asn= or
|
||||||
URL: params.GeoJSURL,
|
// country= tag and the biased thresholds act on the answer before the
|
||||||
Timeout: params.Config.ASNAndCountryLookupTimeout,
|
// request goes on.
|
||||||
// The country list, the headers, the tag settings that name an asn=
|
h.waitForGeoJS = len(params.Config.ExclusivelyAllowedCountries) > 0 ||
|
||||||
// or country= tag and the biased thresholds act on the answer before
|
params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() ||
|
||||||
// the request goes on.
|
biasedThresholdsSet(params.Config)
|
||||||
Wait: len(params.Config.ExclusivelyAllowedCountries) > 0 ||
|
h.geojs = geojs.New(geojs.Params{
|
||||||
params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() ||
|
URL: params.GeoJSURL,
|
||||||
biasedThresholdsSet(params.Config),
|
Answered: h.addLookup,
|
||||||
Answered: h.addLookup,
|
Now: params.Now,
|
||||||
Now: params.Now,
|
Asked: m.GeoJSRequests.Inc,
|
||||||
ProcessLog: params.ProcessLog,
|
Failed: h.geojsFailed,
|
||||||
Metrics: m,
|
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
|
||||||
Alerts: params.Alerts,
|
|
||||||
})
|
})
|
||||||
m.AddBansAndClients(h.ledger, h.limiter, params.Now)
|
m.AddBansAndClients(h.ledger, h.limiter, params.Now)
|
||||||
m.AddRules(params.Rules)
|
m.AddRules(params.Rules)
|
||||||
@@ -268,7 +279,7 @@ type handler struct {
|
|||||||
metrics *metrics.Metrics
|
metrics *metrics.Metrics
|
||||||
limiter *ratelimit.Limiter
|
limiter *ratelimit.Limiter
|
||||||
ledger *bans.Ledger
|
ledger *bans.Ledger
|
||||||
geojs *lookup.GeoJS
|
geojs *geojs.GeoJS
|
||||||
anomalies *anomaly.Counters
|
anomalies *anomaly.Counters
|
||||||
lookupFile *lookup.File
|
lookupFile *lookup.File
|
||||||
lists *reputation.Lists
|
lists *reputation.Lists
|
||||||
@@ -276,8 +287,13 @@ type handler struct {
|
|||||||
abuseIPDB *reputation.AbuseIPDB
|
abuseIPDB *reputation.AbuseIPDB
|
||||||
routeWatch *routewatch.Client
|
routeWatch *routewatch.Client
|
||||||
rules *rules.Files
|
rules *rules.Files
|
||||||
coreRuleSet *coreruleset.CoreRuleSet
|
coreRuleSet CoreRuleSetInspector
|
||||||
alerts *alerts.Queue
|
alerts *alerts.Queue
|
||||||
|
// waitForGeoJS is true when a setting needs each request's answer from
|
||||||
|
// GeoJS before the request goes on.
|
||||||
|
waitForGeoJS bool
|
||||||
|
// besideLookupLimit is besideLookupTimeLimit. Only tests change it.
|
||||||
|
besideLookupLimit time.Duration
|
||||||
// awaitingReport holds the decision on each answered question by its id
|
// awaitingReport holds the decision on each answered question by its id
|
||||||
// until its report comes, and nil for a question still being handled.
|
// until its report comes, and nil for a question still being handled.
|
||||||
awaitingReport *lru.Cache[string, *decision]
|
awaitingReport *lru.Cache[string, *decision]
|
||||||
|
|||||||
@@ -512,6 +512,11 @@ func newWAFWith(
|
|||||||
Alerts: alertQueue,
|
Alerts: alertQueue,
|
||||||
})
|
})
|
||||||
|
|
||||||
|
// The race detector and a busy host can slow the Core Rule Set's
|
||||||
|
// inspections past the time limit. The tests of the limit set it back, and
|
||||||
|
// hold the Core Rule Set with a stand-in.
|
||||||
|
server.SetBesideLookupTimeLimit(time.Hour)
|
||||||
|
|
||||||
return server, out, alertQueue
|
return server, out, alertQueue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import (
|
|||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
@@ -162,7 +162,7 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
|
|||||||
|
|
||||||
// The answers are kept before the requests, so that none waits for
|
// The answers are kept before the requests, so that none waits for
|
||||||
// GeoJS.
|
// GeoJS.
|
||||||
server.GeoJS.Load([]lookup.Answer{
|
server.GeoJS.Load([]geojs.Answer{
|
||||||
keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"),
|
keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"),
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|||||||
+47
-13
@@ -11,7 +11,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||||
@@ -42,7 +42,7 @@ type decision struct {
|
|||||||
// looked up, whether or not an answer was there, and lookupAnswer is
|
// looked up, whether or not an answer was there, and lookupAnswer is
|
||||||
// what the lookup gave then, the zero Answer while GeoJS had given none.
|
// what the lookup gave then, the zero Answer while GeoJS had given none.
|
||||||
lookedUp bool
|
lookedUp bool
|
||||||
lookupAnswer lookup.Answer
|
lookupAnswer geojs.Answer
|
||||||
// skippedLookupWait is true for a request that did not wait for GeoJS and
|
// skippedLookupWait is true for a request that did not wait for GeoJS and
|
||||||
// found no answer kept. SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT does not apply to it,
|
// found no answer kept. SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT does not apply to it,
|
||||||
// so that not waiting never lowers its client's limits.
|
// so that not waiting never lowers its client's limits.
|
||||||
@@ -81,6 +81,9 @@ type request struct {
|
|||||||
checked time.Time
|
checked time.Time
|
||||||
// refused is what the request was refused with, nil if it was not.
|
// refused is what the request was refused with, nil if it was not.
|
||||||
refused *refusal
|
refused *refusal
|
||||||
|
// findings is what the heuristics of besideLookup found of the request,
|
||||||
|
// once waf has taken it, and nil before.
|
||||||
|
findings *besideLookupFindings
|
||||||
// requestBytes is how many bytes of the request's body have been read.
|
// requestBytes is how many bytes of the request's body have been read.
|
||||||
requestBytes int64
|
requestBytes int64
|
||||||
}
|
}
|
||||||
@@ -177,9 +180,9 @@ func requestHeaders(r *http.Request, names []string) map[string]string {
|
|||||||
// check is the one place where a request can be refused. It returns nil to let
|
// check is the one place where a request can be refused. It returns nil to let
|
||||||
// the request through. The heuristics of checkClient decide: a request they
|
// the request through. The heuristics of checkClient decide: a request they
|
||||||
// refuse is answered with SWW_WAF_BAN_RESPONSE, 403 for a block rule or the Core
|
// refuse is answered with SWW_WAF_BAN_RESPONSE, 403 for a block rule or the Core
|
||||||
// Rule Set, or 414 or 431 for the header limits. In observe mode nothing is refused,
|
// Rule Set, 503 for heuristics that ran out of time, or 414 or 431 for the
|
||||||
// and the log line names what enforce mode would have done. ctx is the
|
// header limits. In observe mode nothing is refused, and the log line names
|
||||||
// request's own context.
|
// what enforce mode would have done. ctx is the request's own context.
|
||||||
func (rq *request) check(ctx context.Context) *refusal {
|
func (rq *request) check(ctx context.Context) *refusal {
|
||||||
v := rq.checkClient(ctx)
|
v := rq.checkClient(ctx)
|
||||||
if v == nil {
|
if v == nil {
|
||||||
@@ -196,6 +199,11 @@ func (rq *request) check(ctx context.Context) *refusal {
|
|||||||
case action == requestlog.ActionBlockedByBlockRule ||
|
case action == requestlog.ActionBlockedByBlockRule ||
|
||||||
action == requestlog.ActionBlockedByCoreRuleSet:
|
action == requestlog.ActionBlockedByCoreRuleSet:
|
||||||
return &refusal{status: http.StatusForbidden, action: action}
|
return &refusal{status: http.StatusForbidden, action: action}
|
||||||
|
case action == requestlog.ActionChecksTimedOut:
|
||||||
|
return &refusal{
|
||||||
|
status: http.StatusServiceUnavailable,
|
||||||
|
action: action,
|
||||||
|
}
|
||||||
case action == requestlog.ActionRequestLineTooLong:
|
case action == requestlog.ActionRequestLineTooLong:
|
||||||
return &refusal{status: http.StatusRequestURITooLong, action: action}
|
return &refusal{status: http.StatusRequestURITooLong, action: action}
|
||||||
case action == requestlog.ActionRequestHeaderTooLarge:
|
case action == requestlog.ActionRequestHeaderTooLarge:
|
||||||
@@ -208,11 +216,16 @@ func (rq *request) check(ctx context.Context) *refusal {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkClient runs the heuristics of beforeLookup on the request, then looks
|
// checkClient runs the heuristics on the request, and returns the first
|
||||||
// up the client's AS number and country, then runs those of afterLookup. It
|
// verdict, or nil when none gives one. A client in SWW_WAF_ALLOW_NETS skips them
|
||||||
// returns the first verdict, or nil when none gives one. A client in
|
// all, and is not looked up. ctx is the request's own context.
|
||||||
// SWW_WAF_ALLOW_NETS skips them all, and is not looked up. ctx is the request's
|
//
|
||||||
// own context.
|
// The heuristics of beforeLookup run first. Then the client's AS number and
|
||||||
|
// country are looked up, and afterLookup runs once the lookup has its result.
|
||||||
|
// While the lookup waits for GeoJS, besideLookup runs too, as runBesideLookup
|
||||||
|
// says. A lookup that needs no wait is done, and afterLookup with it, before
|
||||||
|
// besideLookup starts, so that besideLookup does no work for a request
|
||||||
|
// afterLookup refuses.
|
||||||
func (rq *request) checkClient(ctx context.Context) *verdict {
|
func (rq *request) checkClient(ctx context.Context) *verdict {
|
||||||
if isInside(rq.client, rq.h.config.AllowNets) {
|
if isInside(rq.client, rq.h.config.AllowNets) {
|
||||||
return nil
|
return nil
|
||||||
@@ -225,7 +238,28 @@ func (rq *request) checkClient(ctx context.Context) *verdict {
|
|||||||
return v
|
return v
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.lookUp(ctx, now)
|
lookedUp := rq.startLookUp(ctx, now)
|
||||||
|
|
||||||
|
select {
|
||||||
|
case result := <-lookedUp:
|
||||||
|
v := rq.runAfterLookup(ctx, result)
|
||||||
|
if v != nil {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
|
||||||
|
return rq.runBesideLookup(ctx, nil)
|
||||||
|
default:
|
||||||
|
return rq.runBesideLookup(ctx, lookedUp)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// runAfterLookup notes result, the lookup's, and runs the heuristics of
|
||||||
|
// afterLookup. It returns the first verdict, or nil when none gives one. ctx is
|
||||||
|
// the request's own context.
|
||||||
|
func (rq *request) runAfterLookup(
|
||||||
|
ctx context.Context, result lookupResult,
|
||||||
|
) *verdict {
|
||||||
|
rq.noteLookup(result)
|
||||||
|
|
||||||
// The lookup can wait for GeoJS, so the checks after it read the time
|
// The lookup can wait for GeoJS, so the checks after it read the time
|
||||||
// again: a request counted at the time it arrived could be counted
|
// again: a request counted at the time it arrived could be counted
|
||||||
@@ -382,9 +416,9 @@ func anomalyThresholdsSet(cfg *config.WAF) bool {
|
|||||||
// answer about it as the request ends, and whether there is one: the
|
// answer about it as the request ends, and whether there is one: the
|
||||||
// lookup database's, which was there at once, or the one GeoJS has given
|
// lookup database's, which was there at once, or the one GeoJS has given
|
||||||
// by then, which a request does not wait for unless a setting needs it.
|
// by then, which a request does not wait for unless a setting needs it.
|
||||||
func (rq *request) answerAtTheEnd() (lookup.Answer, bool) {
|
func (rq *request) answerAtTheEnd() (geojs.Answer, bool) {
|
||||||
if !rq.lookedUp {
|
if !rq.lookedUp {
|
||||||
return lookup.Answer{}, false
|
return geojs.Answer{}, false
|
||||||
}
|
}
|
||||||
|
|
||||||
if rq.h.config.LookupSource == "file" {
|
if rq.h.config.LookupSource == "file" {
|
||||||
|
|||||||
@@ -13,8 +13,8 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ruleFiles checks the request against the rules of the rule files at now,
|
// ruleFiles takes the rules of the rule files the request matched, which
|
||||||
// notes the ids of those it matches in the log line, adds the tag
|
// findBesideLookup found, notes their ids in the log line, adds the tag
|
||||||
// exploitsignature when it matches any, and refuses the request
|
// exploitsignature when it matches any, and refuses the request
|
||||||
// when the last rule it matches refuses it: with ActionBlockedByBlockRule for a
|
// when the last rule it matches refuses it: with ActionBlockedByBlockRule for a
|
||||||
// block rule, and for a ban rule with the action for the ban it makes. A ban
|
// block rule, and for a ban rule with the action for the ban it makes. A ban
|
||||||
@@ -31,7 +31,7 @@ func (ruleFiles) check(
|
|||||||
rq *request,
|
rq *request,
|
||||||
now time.Time,
|
now time.Time,
|
||||||
) *verdict {
|
) *verdict {
|
||||||
matched := rq.h.rules.Match(rq.in)
|
matched := rq.findings.rules
|
||||||
|
|
||||||
for _, rule := range matched {
|
for _, rule := range matched {
|
||||||
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
|
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
|
||||||
@@ -70,3 +70,15 @@ func (ruleFiles) check(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// refusedByRule reports whether ruleFiles refuses a request that matched
|
||||||
|
// matched: whether the last of them is a block rule or a ban rule.
|
||||||
|
func refusedByRule(matched []rules.Rule) bool {
|
||||||
|
if len(matched) == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
action := matched[len(matched)-1].Action
|
||||||
|
|
||||||
|
return action == rules.ActionBlock || action == rules.ActionBan
|
||||||
|
}
|
||||||
@@ -11,9 +11,8 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||||
)
|
)
|
||||||
|
|
||||||
// trapPaths refuses a request for a path in SWW_WAF_TRAP_PATHS: its path as a
|
// trapPaths refuses a request for a path in SWW_WAF_TRAP_PATHS, as trapPath
|
||||||
// path rule sees it, without the query, as sent or once percent-decoded, is one
|
// finds it. Such a request is a clear sign of attack, as a ban rule's match is:
|
||||||
// of them. Such a request is a clear sign of attack, as a ban rule's match is:
|
|
||||||
// it bans the client's netblock, or in observe mode raises the alert for the
|
// it bans the client's netblock, or in observe mode raises the alert for the
|
||||||
// ban it would have made. The ban's notes give the path as sent, and so does
|
// ban it would have made. The ban's notes give the path as sent, and so does
|
||||||
// its figure.
|
// its figure.
|
||||||
@@ -26,9 +25,8 @@ func (trapPaths) check(
|
|||||||
rq *request,
|
rq *request,
|
||||||
now time.Time,
|
now time.Time,
|
||||||
) *verdict {
|
) *verdict {
|
||||||
path := rules.Path(rq.in)
|
path := rq.findings.trapPath
|
||||||
if !slices.Contains(rq.h.config.TrapPaths, path) &&
|
if path == "" {
|
||||||
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -39,3 +37,16 @@ func (trapPaths) check(
|
|||||||
figures: map[string]any{"path": path},
|
figures: map[string]any{"path": path},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// trapPath returns the request's path as sent, when its path as a path rule
|
||||||
|
// sees it, without the query, as sent or once percent-decoded, is in
|
||||||
|
// SWW_WAF_TRAP_PATHS, and "" otherwise.
|
||||||
|
func (rq *request) trapPath() string {
|
||||||
|
path := rules.Path(rq.in)
|
||||||
|
if !slices.Contains(rq.h.config.TrapPaths, path) &&
|
||||||
|
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
return path
|
||||||
|
}
|
||||||
Reference in new issue
Block a user