Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
01a8a7aa08 | ||
|
|
988f117ef3 |
No files matched your search
@@ -329,7 +329,7 @@ Every setting the code reads, with its default:
|
|||||||
| `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
|
| `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
|
||||||
| `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
|
| `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
|
||||||
| `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
| `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
||||||
| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, in milliseconds. The request to GeoJS goes on, and a later answer is kept. |
|
| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `500` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take, in milliseconds. |
|
||||||
| `SWW_WAF_ADD_LOOKUP_HEADERS` | `true`, or `false` while the lookup source is `off` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
| `SWW_WAF_ADD_LOOKUP_HEADERS` | `true`, or `false` while the lookup source is `off` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
||||||
| `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
| `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
||||||
| `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
| `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
||||||
@@ -749,9 +749,9 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
|
|||||||
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
|
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
|
||||||
endpoints.
|
endpoints.
|
||||||
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
|
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
|
||||||
`internal/coreruleset`, `internal/geojs`, `internal/lookup`,
|
`internal/coreruleset`, `internal/lookup`, `internal/reputation`,
|
||||||
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and
|
`internal/routewatch`, `internal/anomaly` and `internal/alerts`: one check or
|
||||||
`internal/alerts`: one check or source each.
|
source each.
|
||||||
- `internal/state`, `internal/requestlog`, `internal/remotelog` and
|
- `internal/state`, `internal/requestlog`, `internal/remotelog` and
|
||||||
`internal/metrics`: the state and cache files, the log lines, the syslog
|
`internal/metrics`: the state and cache files, the log lines, the syslog
|
||||||
sender and the metrics.
|
sender and the metrics.
|
||||||
|
|||||||
@@ -87,10 +87,8 @@ never as a time.
|
|||||||
change, and tells which rules a request matches.
|
change, and tells which rules a request matches.
|
||||||
- `internal/coreruleset`: the Core Rule Set, with the five changes and the
|
- `internal/coreruleset`: the Core Rule Set, with the five changes and the
|
||||||
apps' fixes listed under "Attack detection", as Coraza directives.
|
apps' fixes listed under "Attack detection", as Coraza directives.
|
||||||
- `internal/geojs`: looks clients up through GeoJS and keeps the answers. It
|
- `internal/lookup`: looks clients up through GeoJS or in the lookup
|
||||||
imports no other package of this repository: `waf` gives it what it needs
|
database.
|
||||||
to count and report failures.
|
|
||||||
- `internal/lookup`: looks clients up in the lookup database.
|
|
||||||
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
|
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
|
||||||
AbuseIPDB in the background, and keeps their answers.
|
AbuseIPDB in the background, and keeps their answers.
|
||||||
- `internal/routewatch`: asks a routewatch instance about clients in the
|
- `internal/routewatch`: asks a routewatch instance about clients in the
|
||||||
@@ -509,25 +507,19 @@ service that keeps its visitors' addresses on its own host.
|
|||||||
dropped from that table. The answer reaches the client's history and ban notes
|
dropped from that table. The answer reaches the client's history and ban notes
|
||||||
when it comes.
|
when it comes.
|
||||||
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is how long a visitor's request
|
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is how long a visitor's request
|
||||||
waits for its client's first answer, when a setting needs it (see step 6). A
|
waits for its client's first answer, when a setting needs it (see step 6). It
|
||||||
client whose answer has not come in time counts as unknown until it comes, and
|
also bounds each request to GeoJS. A client whose answer has not come in time
|
||||||
its later requests do not wait.
|
counts as unknown until it comes, and its later requests do not wait. Its
|
||||||
- A request to GeoJS may take up to 10 seconds, however long visitors wait. It
|
address is asked about again in the background.
|
||||||
goes on after the visitor's request ends, and its answer is kept. Say GeoJS
|
|
||||||
answers after 5 seconds. The visitor's request is decided after 800
|
|
||||||
milliseconds, without a country. A request from the same client 6 seconds
|
|
||||||
later gets the answer.
|
|
||||||
- Requests from one client that come together share one request to GeoJS.
|
|
||||||
- A client the lookup cannot place has an unknown country and AS number, which
|
- A client the lookup cannot place has an unknown country and AS number, which
|
||||||
the country list and the biased thresholds treat as described there. Nothing
|
the country list and the biased thresholds treat as described there. Nothing
|
||||||
else treats it differently.
|
else treats it differently.
|
||||||
- GeoJS publishes no rate limit, but its terms forbid "an excessive amount of
|
- GeoJS publishes no rate limit, but its terms forbid "an excessive amount of
|
||||||
API requests", judged by GeoJS alone, and let it block a caller. A request to
|
API requests", judged by GeoJS alone, and let it block a caller. While GeoJS
|
||||||
GeoJS fails when it takes over 10 seconds or GeoJS answers with an error.
|
is slow, down or refusing `waf`, new clients count as unknown. GeoJS is then
|
||||||
While GeoJS is slow, down or refusing `waf`, new clients count as unknown.
|
left alone for a second, and twice as long after each further failure, up to
|
||||||
After a failure GeoJS is left alone for a second, and twice as long after each
|
five minutes. Each failure raises a `source_failure` alert (see "Alerting"). A
|
||||||
further failure, up to five minutes. Each failure raises a `source_failure`
|
service that cannot accept this uses the file.
|
||||||
alert (see "Alerting"). A service that cannot accept this uses the file.
|
|
||||||
- The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`,
|
- The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`,
|
||||||
`as_name` and `country_code` fields are read. The operator downloads it with a
|
`as_name` and `country_code` fields are read. The operator downloads it with a
|
||||||
free IPinfo account and mounts it read-only; `waf` never fetches it and holds
|
free IPinfo account and mounts it read-only; `waf` never fetches it and holds
|
||||||
@@ -2413,15 +2405,14 @@ The Core Rule Set inspects the body a question carries only while
|
|||||||
`rproxy` stops waiting for `waf` after `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, by
|
`rproxy` stops waiting for `waf` after `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, by
|
||||||
default 2 seconds, and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` then
|
default 2 seconds, and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` then
|
||||||
decides. The longest a question waits on an outside source is
|
decides. The longest a question waits on an outside source is
|
||||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds, well
|
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 500 milliseconds, well
|
||||||
within that time.
|
within that time.
|
||||||
|
|
||||||
GeoJS is the only outside source a question can wait on. A question waits only
|
GeoJS is the only outside source a question can wait on. A question waits only
|
||||||
when step 6 says so, and for at most
|
when step 6 says so, and for at most
|
||||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds. A
|
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 500 milliseconds. A
|
||||||
client without an answer by then counts as unknown. The setting bounds only the
|
client without an answer by then counts as unknown. No question waits on these,
|
||||||
question's wait: the request to GeoJS goes on, and its answer is kept for later
|
which run in the background:
|
||||||
questions. No question waits on these, which run in the background:
|
|
||||||
|
|
||||||
- the DNS blocklists and AbuseIPDB;
|
- the DNS blocklists and AbuseIPDB;
|
||||||
- fetching the blocklists, the tag lists and the file
|
- fetching the blocklists, the tag lists and the file
|
||||||
@@ -2460,8 +2451,11 @@ It refuses a request with `431` for any of:
|
|||||||
|
|
||||||
- a header line longer than 8 KiB. Each value of a header sent twice is a line
|
- a header line longer than 8 KiB. Each value of a header sent twice is a line
|
||||||
of its own. `Host` counts as a line.
|
of its own. `Host` counts as a line.
|
||||||
- more than 50 cookies. Each part of its `Cookie` headers between semicolons
|
- more than one `Cookie` line. A browser sends all its cookies on one line, as
|
||||||
counts as one, unless it is empty or only spaces and tabs.
|
RFC 6265 requires. Go's server joins the `Cookie` lines of an HTTP/2 request
|
||||||
|
into one before `rproxy` sees them.
|
||||||
|
- more than 50 cookies. Each part of its `Cookie` line between semicolons counts
|
||||||
|
as one, unless it is empty or only spaces and tabs.
|
||||||
- more than 100 header fields. A header sent twice counts twice. `Host` and
|
- more than 100 header fields. A header sent twice counts twice. `Host` and
|
||||||
`Transfer-Encoding` do not count. `X-Request-ID` and `X-Forwarded-Proto` do.
|
`Transfer-Encoding` do not count. `X-Request-ID` and `X-Forwarded-Proto` do.
|
||||||
|
|
||||||
@@ -2499,9 +2493,11 @@ body can add up to more than the body. From 1 KiB of JSON, Coraza makes at most
|
|||||||
|
|
||||||
A request can reach all of these at once. At paranoia level 4, with no rule
|
A request can reach all of these at once. At paranoia level 4, with no rule
|
||||||
switched off, the Core Rule Set takes under a second on one core of a current
|
switched off, the Core Rule Set takes under a second on one core of a current
|
||||||
server to inspect it. So `waf` answers it within the default 2 seconds, even
|
server to inspect it, even with every core busy. Its share of the default 2
|
||||||
after the default 800 milliseconds of waiting for GeoJS. Enough such requests at
|
seconds is 1.3 seconds: the rest is the default 500 milliseconds of waiting for
|
||||||
once can still overload `waf` past `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
|
GeoJS and 200 milliseconds for the other checks. So `waf` answers it in time.
|
||||||
|
Enough such requests at once can still overload `waf` past
|
||||||
|
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
|
||||||
|
|
||||||
### The report
|
### The report
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -128,8 +128,8 @@ Every one of these lines has:
|
|||||||
- `rule_files`: `rule_id` and `rule_action`, of the rule that refused it;
|
- `rule_files`: `rule_id` and `rule_action`, of the rule that refused it;
|
||||||
- `header_limits`: for a request line, `request_line_bytes` and
|
- `header_limits`: for a request line, `request_line_bytes` and
|
||||||
`request_line_bytes_limit`; for headers, `longest_header_line_bytes`,
|
`request_line_bytes_limit`; for headers, `longest_header_line_bytes`,
|
||||||
`header_line_bytes_limit`, `fields`, `fields_limit`, `cookies` and
|
`header_line_bytes_limit`, `fields`, `fields_limit`, `cookie_lines`,
|
||||||
`cookies_limit`;
|
`cookie_lines_limit`, `cookies` and `cookies_limit`;
|
||||||
- `core_rule_set`: `score`, and `threshold`,
|
- `core_rule_set`: `score`, and `threshold`,
|
||||||
`SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`;
|
`SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`;
|
||||||
- `refusal_limit`: `window`, `count` and `limit`, of the refusal limit;
|
- `refusal_limit`: `window`, `count` and `limit`, of the refusal limit;
|
||||||
|
|||||||
+9
-7
@@ -158,10 +158,9 @@ cut off; the client is banned.
|
|||||||
- an anomaly threshold per AS number.
|
- an anomaly threshold per AS number.
|
||||||
- `SWW_WAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in
|
- `SWW_WAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in
|
||||||
its `.mmdb` form, for `SWW_WAF_LOOKUP_SOURCE=file`.
|
its `.mmdb` form, for `SWW_WAF_LOOKUP_SOURCE=file`.
|
||||||
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` (default `800`): how long a
|
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` (default `500`): how long a
|
||||||
request waits for its client's first answer from GeoJS. A time in
|
request waits for its client's first answer from GeoJS, and how long a request
|
||||||
milliseconds; never `off`. It does not cut off the request to GeoJS, whose
|
to GeoJS may take. A time in milliseconds; never `off`.
|
||||||
answer is kept when it comes later.
|
|
||||||
- `SWW_WAF_ADD_LOOKUP_HEADERS` (default `true`): `true` has `rproxy` pass the
|
- `SWW_WAF_ADD_LOOKUP_HEADERS` (default `true`): `true` has `rproxy` pass the
|
||||||
app the client's AS number in `X-Client-ASN` and its country in
|
app the client's AS number in `X-Client-ASN` and its country in
|
||||||
`X-Client-Country`. The default is `false` while `SWW_WAF_LOOKUP_SOURCE` is
|
`X-Client-Country`. The default is `false` while `SWW_WAF_LOOKUP_SOURCE` is
|
||||||
@@ -423,9 +422,12 @@ A few limits are fixed rather than settings:
|
|||||||
- at most 20,000 anomaly counters are kept;
|
- at most 20,000 anomaly counters are kept;
|
||||||
- each alert destination's queue holds at most 1,000 alerts;
|
- each alert destination's queue holds at most 1,000 alerts;
|
||||||
- `waf` holds at most 10,000 answered requests awaiting their reports;
|
- `waf` holds at most 10,000 answered requests awaiting their reports;
|
||||||
- `waf` refuses a request with a request line or header line over 8 KiB, more
|
- `waf` refuses a request with any of these (see "Answering in time" under
|
||||||
than 50 cookies or more than 100 header fields (see "Answering in time" under
|
"`waf`" in [`SPEC.md`](../SPEC.md)):
|
||||||
"`waf`" in [`SPEC.md`](../SPEC.md));
|
- a request line or header line over 8 KiB;
|
||||||
|
- more than one `Cookie` line;
|
||||||
|
- more than 50 cookies;
|
||||||
|
- more than 100 header fields;
|
||||||
- `rproxy` takes a request line and headers of at most 32 KiB, and closes a
|
- `rproxy` takes a request line and headers of at most 32 KiB, and closes a
|
||||||
kept-open connection idle for 120 seconds;
|
kept-open connection idle for 120 seconds;
|
||||||
- `rproxy` reads at most 1 MiB of the body of `waf`'s answer, and answers `502`
|
- `rproxy` reads at most 1 MiB of the body of `waf`'s answer, and answers `502`
|
||||||
|
|||||||
@@ -548,7 +548,7 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
|
|||||||
LookupDBPath: env.value("SWW_WAF_LOOKUP_DB_PATH", ""),
|
LookupDBPath: env.value("SWW_WAF_LOOKUP_DB_PATH", ""),
|
||||||
ASNAndCountryLookupTimeout: env.milliseconds(
|
ASNAndCountryLookupTimeout: env.milliseconds(
|
||||||
"SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS",
|
"SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS",
|
||||||
"800",
|
"500",
|
||||||
),
|
),
|
||||||
ExclusivelyAllowedCountries: env.countries(
|
ExclusivelyAllowedCountries: env.countries(
|
||||||
"SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
|
"SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
|
||||||
|
|||||||
@@ -286,7 +286,7 @@ func TestDefaults(t *testing.T) {
|
|||||||
RuleFilesEnabled: true,
|
RuleFilesEnabled: true,
|
||||||
})
|
})
|
||||||
wantLookupSettings(t, cfg, config.WAF{
|
wantLookupSettings(t, cfg, config.WAF{
|
||||||
LookupSource: defaultLookupSource, ASNAndCountryLookupTimeout: 800 * time.Millisecond,
|
LookupSource: defaultLookupSource, ASNAndCountryLookupTimeout: 500 * time.Millisecond,
|
||||||
AddLookupHeaders: true,
|
AddLookupHeaders: true,
|
||||||
})
|
})
|
||||||
wantByteLimitSettings(t, cfg, config.WAF{
|
wantByteLimitSettings(t, cfg, config.WAF{
|
||||||
@@ -1407,7 +1407,7 @@ func TestLookupSettingsAsSet(t *testing.T) {
|
|||||||
// With no client looked up, the headers are off unless set.
|
// With no client looked up, the headers are off unless set.
|
||||||
cfg = wafFromEnvironment(t, environment{lookupSource: off})
|
cfg = wafFromEnvironment(t, environment{lookupSource: off})
|
||||||
wantLookupSettings(t, cfg, config.WAF{
|
wantLookupSettings(t, cfg, config.WAF{
|
||||||
LookupSource: off, ASNAndCountryLookupTimeout: 800 * time.Millisecond,
|
LookupSource: off, ASNAndCountryLookupTimeout: 500 * time.Millisecond,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -2933,7 +2933,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
|||||||
bytesCounted: "both",
|
bytesCounted: "both",
|
||||||
lookupSource: defaultLookupSource,
|
lookupSource: defaultLookupSource,
|
||||||
lookupDBPath: "",
|
lookupDBPath: "",
|
||||||
asnAndCountryLookupTimeout: "800",
|
asnAndCountryLookupTimeout: "500",
|
||||||
addLookupHeaders: enabled,
|
addLookupHeaders: enabled,
|
||||||
allowedCountries: "",
|
allowedCountries: "",
|
||||||
tagLists: "",
|
tagLists: "",
|
||||||
|
|||||||
@@ -35,20 +35,20 @@ const (
|
|||||||
mostJSONRead = 1 << 10
|
mostJSONRead = 1 << 10
|
||||||
)
|
)
|
||||||
|
|
||||||
// The Core Rule Set has 1 second for one request. That is the default of
|
// The Core Rule Set has 1.3 seconds for one request. That is the default of
|
||||||
// rproxy's SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, 2 seconds, less the default of
|
// rproxy's SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, 2 seconds, less the default of
|
||||||
// SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, 800 milliseconds, and less 200
|
// SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, 500 milliseconds, and less 200
|
||||||
// milliseconds for waf's other checks.
|
// milliseconds for waf's other checks.
|
||||||
//
|
//
|
||||||
// A busy host slows all work, so the test does not compare an inspection with
|
// A busy host slows all work, so the test does not compare an inspection with
|
||||||
// 1 second. It compares it with referenceTime, timed on the same thread just
|
// 1.3 seconds. It compares it with referenceTime, timed on the same thread
|
||||||
// before and after. inspectionTimeLimitMultiple is how many times
|
// just before and after. inspectionTimeLimitMultiple is how many times
|
||||||
// referenceTime an inspection may take.
|
// referenceTime an inspection may take.
|
||||||
//
|
//
|
||||||
// On the host that runs our checks, referenceTime took at least 78
|
// On the host that runs our checks, referenceTime took at least 78
|
||||||
// milliseconds, so 13 times it is about 1 second. With that host fully busy,
|
// milliseconds, so 16 times it is about 1.25 seconds. With that host fully
|
||||||
// no try of the slowest request took more than 9.5 times referenceTime.
|
// busy, no try of the slowest request took more than 6.7 times referenceTime.
|
||||||
const inspectionTimeLimitMultiple = 13
|
const inspectionTimeLimitMultiple = 16
|
||||||
|
|
||||||
// referencePattern is what referenceTime matches. Like the Core Rule Set's
|
// referencePattern is what referenceTime matches. Like the Core Rule Set's
|
||||||
// rules, it is a regular expression with alternatives and repeats.
|
// rules, it is a regular expression with alternatives and repeats.
|
||||||
@@ -221,20 +221,22 @@ const fullLine = lineLimit - 64
|
|||||||
|
|
||||||
// worstCaseRequests are, for each kind of thing a visitor sends, the requests
|
// worstCaseRequests are, for each kind of thing a visitor sends, the requests
|
||||||
// that keep the Core Rule Set busiest, within the bounds above. The last
|
// that keep the Core Rule Set busiest, within the bounds above. The last
|
||||||
// holds the most of everything at once: Cookie lines keep it busier than
|
// holds the most of everything at once. waf refuses a second Cookie line, so
|
||||||
// other header lines, and the JSON body of the most elements under a key
|
// it has one, and User-Agent and Referer lines, which keep the Core Rule Set
|
||||||
// keeps it busier than the other bodies.
|
// busier than other header lines. The JSON body of the most elements under a
|
||||||
|
// key keeps it busier than the other bodies.
|
||||||
func worstCaseRequests() []visitorRequest {
|
func worstCaseRequests() []visitorRequest {
|
||||||
fullCookieLine := cookieLine(1)[:fullLine-len("Cookie: ")]
|
|
||||||
|
|
||||||
return append(
|
return append(
|
||||||
append(worstCaseHeaders(), worstCaseBodies()...),
|
append(worstCaseHeaders(), worstCaseBodies()...),
|
||||||
visitorRequest{
|
visitorRequest{
|
||||||
name: "the longest request line, three Cookie lines and body",
|
name: "the longest request line, Cookie, User-Agent and " +
|
||||||
|
"Referer lines and body",
|
||||||
target: longestTarget()[:fullLine-len("POST HTTP/1.1")],
|
target: longestTarget()[:fullLine-len("POST HTTP/1.1")],
|
||||||
header: http.Header{cookieHeader: {
|
header: http.Header{
|
||||||
fullCookieLine, fullCookieLine, fullCookieLine,
|
cookieHeader: {cookieLine(1)[:fullLine-len("Cookie: ")]},
|
||||||
}},
|
"User-Agent": {headerValue("User-Agent", fullLine)},
|
||||||
|
"Referer": {headerValue("Referer", fullLine)},
|
||||||
|
},
|
||||||
contentType: jsonType, body: jsonOfMostElementsUnderAKey(),
|
contentType: jsonType, body: jsonOfMostElementsUnderAKey(),
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
package geojs
|
package lookup
|
||||||
|
|
||||||
import "net/http"
|
import "net/http"
|
||||||
|
|
||||||
@@ -1,6 +1,3 @@
|
|||||||
// Package lookup looks up each client's AS number and country in the lookup
|
|
||||||
// database, the IPinfo Lite file SWW_WAF_LOOKUP_DB_PATH names. Its answers
|
|
||||||
// take the form of GeoJS's, so that waf handles both alike.
|
|
||||||
package lookup
|
package lookup
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -17,7 +14,6 @@ import (
|
|||||||
"github.com/oschwald/maxminddb-golang/v2"
|
"github.com/oschwald/maxminddb-golang/v2"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// quietTime is how long the lookup database must go without a change
|
// quietTime is how long the lookup database must go without a change
|
||||||
@@ -81,7 +77,7 @@ func OpenFile(params FileParams) (*File, error) {
|
|||||||
// each "" when the database does not give it, as for an address missing
|
// each "" when the database does not give it, as for an address missing
|
||||||
// from it. The database is asked about the client's first address, as
|
// from it. The database is asked about the client's first address, as
|
||||||
// GeoJS is.
|
// GeoJS is.
|
||||||
func (f *File) LookUp(client netip.Prefix) geojs.Answer {
|
func (f *File) LookUp(client netip.Prefix) Answer {
|
||||||
f.mu.Lock()
|
f.mu.Lock()
|
||||||
reader := f.reader
|
reader := f.reader
|
||||||
f.mu.Unlock()
|
f.mu.Unlock()
|
||||||
@@ -95,7 +91,7 @@ func (f *File) LookUp(client netip.Prefix) geojs.Answer {
|
|||||||
found = record{}
|
found = record{}
|
||||||
}
|
}
|
||||||
|
|
||||||
return geojs.Answer{
|
return Answer{
|
||||||
Client: client,
|
Client: client,
|
||||||
ASN: found.ASN,
|
ASN: found.ASN,
|
||||||
ASName: found.ASName,
|
ASName: found.ASName,
|
||||||
|
|||||||
@@ -16,7 +16,6 @@ import (
|
|||||||
"github.com/maxmind/mmdbwriter/mmdbtype"
|
"github.com/maxmind/mmdbwriter/mmdbtype"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -69,7 +68,7 @@ func TestFilePlacesClientsAndCountsAnAddressMissingFromItAsUnknown(
|
|||||||
prefix := netip.MustParsePrefix(client)
|
prefix := netip.MustParsePrefix(client)
|
||||||
|
|
||||||
got := f.LookUp(prefix)
|
got := f.LookUp(prefix)
|
||||||
if got != (geojs.Answer{
|
if got != (Answer{
|
||||||
Client: prefix, ASN: want.ASN, ASName: want.ASName, Country: want.Country,
|
Client: prefix, ASN: want.ASN, ASName: want.ASName, Country: want.Country,
|
||||||
Answered: now,
|
Answered: now,
|
||||||
}) {
|
}) {
|
||||||
|
|||||||
@@ -1,9 +1,9 @@
|
|||||||
// Package geojs looks up clients' AS numbers and countries through the GeoJS
|
// Package lookup looks up each client's AS number and country, through
|
||||||
// web service. Its answers are kept in memory, for at most 100,000 clients and
|
// the GeoJS web service or in the lookup database, the IPinfo Lite file
|
||||||
// for 7 days each. A caller waits for an answer as long as it says. A request to
|
// SWW_WAF_LOOKUP_DB_PATH names. GeoJS's answers are kept in memory, for at
|
||||||
// GeoJS goes on after its callers stop waiting, and its answer is kept. The
|
// most 100,000 clients and for 7 days each, and are written to
|
||||||
// package imports no other package of this repository.
|
// lookups.json and read from it by the state package.
|
||||||
package geojs
|
package lookup
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"slices"
|
"slices"
|
||||||
@@ -20,6 +21,8 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/hashicorp/golang-lru/v2/simplelru"
|
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
)
|
)
|
||||||
|
|
||||||
// URL is GeoJS's endpoint for an address's place and network. Asked about
|
// URL is GeoJS's endpoint for an address's place and network. Asked about
|
||||||
@@ -40,9 +43,6 @@ const (
|
|||||||
maxWaiting = 10000
|
maxWaiting = 10000
|
||||||
// maxPerRequest is how many addresses one request to GeoJS asks about.
|
// maxPerRequest is how many addresses one request to GeoJS asks about.
|
||||||
maxPerRequest = 200
|
maxPerRequest = 200
|
||||||
// requestTimeout is how long a request to GeoJS may take before it is
|
|
||||||
// abandoned, which is a failure. It is apart from how long callers wait.
|
|
||||||
requestTimeout = 10 * time.Second
|
|
||||||
// unknownASN is the AS number GeoJS gives when it knows none.
|
// unknownASN is the AS number GeoJS gives when it knows none.
|
||||||
unknownASN = 64512
|
unknownASN = 64512
|
||||||
// After a failure GeoJS is not asked again for a second, and for
|
// After a failure GeoJS is not asked again for a second, and for
|
||||||
@@ -64,32 +64,39 @@ var (
|
|||||||
type Params struct {
|
type Params struct {
|
||||||
// URL is where GeoJS is asked, normally URL.
|
// URL is where GeoJS is asked, normally URL.
|
||||||
URL string
|
URL string
|
||||||
|
// Timeout is how long a request waits for its client's first answer,
|
||||||
|
// and how long a request to GeoJS may take before it is abandoned
|
||||||
|
// (SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS).
|
||||||
|
Timeout time.Duration
|
||||||
|
// Wait is true when a setting needs each request's answer before the
|
||||||
|
// request goes on. Otherwise no request waits for one.
|
||||||
|
Wait bool
|
||||||
// Answered, unless nil, is given each answer GeoJS gives, once it is
|
// Answered, unless nil, is given each answer GeoJS gives, once it is
|
||||||
// kept.
|
// kept.
|
||||||
Answered func(Answer)
|
Answered func(Answer)
|
||||||
// Now tells the time, normally time.Now in UTC.
|
// Now tells the time, normally time.Now in UTC.
|
||||||
Now func() time.Time
|
Now func() time.Time
|
||||||
// Asked is called for each request to GeoJS.
|
// ProcessLog receives GeoJS's failures.
|
||||||
Asked func()
|
ProcessLog *slog.Logger
|
||||||
// Failed is called for each failure of GeoJS, with what went wrong and
|
// Metrics count the requests to GeoJS, those that failed, and the
|
||||||
// how long GeoJS is left alone after it. The error never names the
|
// clients that go without an answer.
|
||||||
// addresses asked about.
|
Metrics *metrics.Metrics
|
||||||
Failed func(err error, askingAgainIn time.Duration)
|
// Alerts receive a source_failure alert each time GeoJS fails.
|
||||||
// WentWithoutAnswer is called each time a caller that waits for an
|
Alerts *alerts.Queue
|
||||||
// answer goes on without one.
|
|
||||||
WentWithoutAnswer func()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// GeoJS looks up clients' AS numbers and countries through GeoJS. At most
|
// GeoJS looks up clients' AS numbers and countries through GeoJS. At most
|
||||||
// one request to GeoJS is under way at a time, and it asks about every
|
// one request to GeoJS is under way at a time, and it asks about every
|
||||||
// client waiting, up to maxPerRequest. It is safe for concurrent use.
|
// client waiting, up to maxPerRequest. It is safe for concurrent use.
|
||||||
type GeoJS struct {
|
type GeoJS struct {
|
||||||
url string
|
url string
|
||||||
answered func(Answer)
|
timeout time.Duration
|
||||||
now func() time.Time
|
wait bool
|
||||||
asked func()
|
answered func(Answer)
|
||||||
failed func(err error, askingAgainIn time.Duration)
|
now func() time.Time
|
||||||
wentWithoutAnswer func()
|
processLog *slog.Logger
|
||||||
|
metrics *metrics.Metrics
|
||||||
|
alerts *alerts.Queue
|
||||||
// httpClient follows no redirect, so that visitors' addresses go to
|
// httpClient follows no redirect, so that visitors' addresses go to
|
||||||
// GeoJS alone: a redirect is a failure.
|
// GeoJS alone: a redirect is a failure.
|
||||||
httpClient *http.Client
|
httpClient *http.Client
|
||||||
@@ -107,10 +114,12 @@ type GeoJS struct {
|
|||||||
retryAt time.Time
|
retryAt time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
// Answer is what GeoJS said about a client: its AS number, such as AS64496,
|
// Answer is what GeoJS or the lookup database said about a client: its AS
|
||||||
// and the AS's name, both "" when GeoJS knows no AS number for it; its
|
// number, such as AS64496, and the AS's name, both "" when the source knows
|
||||||
// country, "" when GeoJS cannot place it; when GeoJS said so; and when the
|
// no AS number for it; its country, "" when the source cannot place it;
|
||||||
// answer was last used. The zero Answer is that of a client with no answer.
|
// when the source said so; and, for GeoJS's answers, which lookups.json
|
||||||
|
// holds, when the answer was last used. The zero Answer is that of a
|
||||||
|
// client with no answer.
|
||||||
//
|
//
|
||||||
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
||||||
type Answer struct {
|
type Answer struct {
|
||||||
@@ -128,7 +137,7 @@ type wait struct {
|
|||||||
// replaced each time GeoJS fails before then.
|
// replaced each time GeoJS fails before then.
|
||||||
asked chan struct{}
|
asked chan struct{}
|
||||||
// late is true once the client has gone without an answer, for a
|
// late is true once the client has gone without an answer, for a
|
||||||
// whole wait or because GeoJS failed: its callers no longer wait.
|
// whole timeout or because GeoJS failed: its requests no longer wait.
|
||||||
late bool
|
late bool
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -140,12 +149,14 @@ func New(params Params) *GeoJS {
|
|||||||
}
|
}
|
||||||
|
|
||||||
return &GeoJS{
|
return &GeoJS{
|
||||||
url: params.URL,
|
url: params.URL,
|
||||||
answered: params.Answered,
|
timeout: params.Timeout,
|
||||||
now: params.Now,
|
wait: params.Wait,
|
||||||
asked: params.Asked,
|
answered: params.Answered,
|
||||||
failed: params.Failed,
|
now: params.Now,
|
||||||
wentWithoutAnswer: params.WentWithoutAnswer,
|
processLog: params.ProcessLog,
|
||||||
|
metrics: params.Metrics,
|
||||||
|
alerts: params.Alerts,
|
||||||
httpClient: &http.Client{
|
httpClient: &http.Client{
|
||||||
CheckRedirect: func(*http.Request, []*http.Request) error {
|
CheckRedirect: func(*http.Request, []*http.Request) error {
|
||||||
return http.ErrUseLastResponse
|
return http.ErrUseLastResponse
|
||||||
@@ -159,22 +170,21 @@ func New(params Params) *GeoJS {
|
|||||||
// LookUp returns the answer GeoJS gave about client, with its country as
|
// LookUp returns the answer GeoJS gave about client, with its country as
|
||||||
// a two-letter code in capitals, or the zero Answer when there is none
|
// a two-letter code in capitals, or the zero Answer when there is none
|
||||||
// yet. An answer is kept for 7 days. Without one, the client is asked
|
// yet. An answer is kept for 7 days. Without one, the client is asked
|
||||||
// about in the background, and LookUp waits up to wait for the answer,
|
// about in the background, and, while Wait is set, the request waits up
|
||||||
// unless the client has gone without one before. The wait ends early when
|
// to Timeout for the answer, unless the client has gone without one
|
||||||
// ctx ends. Neither ends the request to GeoJS.
|
// before. ctx is the context of the client's request, and ends the wait
|
||||||
|
// when it ends.
|
||||||
//
|
//
|
||||||
// GeoJS is asked about the client's first address, which is the client's
|
// GeoJS is asked about the client's first address, which is the client's
|
||||||
// own address for IPv4, and an address in the same place for an IPv6
|
// own address for IPv4, and an address in the same place for an IPv6
|
||||||
// netblock.
|
// netblock.
|
||||||
func (g *GeoJS) LookUp(
|
func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
|
||||||
ctx context.Context, client netip.Prefix, wait time.Duration,
|
answer, asked := g.answerOrWait(ctx, client, g.wait)
|
||||||
) Answer {
|
|
||||||
answer, asked := g.answerOrWait(ctx, client, true)
|
|
||||||
if asked == nil {
|
if asked == nil {
|
||||||
return answer
|
return answer
|
||||||
}
|
}
|
||||||
|
|
||||||
timer := time.NewTimer(wait)
|
timer := time.NewTimer(g.timeout)
|
||||||
defer timer.Stop()
|
defer timer.Stop()
|
||||||
|
|
||||||
select {
|
select {
|
||||||
@@ -188,7 +198,7 @@ func (g *GeoJS) LookUp(
|
|||||||
|
|
||||||
answer, found := g.kept(client)
|
answer, found := g.kept(client)
|
||||||
if !found {
|
if !found {
|
||||||
g.wentWithoutAnswer()
|
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
||||||
}
|
}
|
||||||
|
|
||||||
w, waiting := g.waiting[client]
|
w, waiting := g.waiting[client]
|
||||||
@@ -199,8 +209,8 @@ func (g *GeoJS) LookUp(
|
|||||||
return answer
|
return answer
|
||||||
}
|
}
|
||||||
|
|
||||||
// LookUpWithoutWaiting is LookUp for a caller that does not need the answer:
|
// LookUpWithoutWaiting is LookUp for a request that does not need the answer,
|
||||||
// it returns the zero Answer at once when none is kept.
|
// whatever Wait says: it returns the zero Answer at once when none is kept.
|
||||||
func (g *GeoJS) LookUpWithoutWaiting(
|
func (g *GeoJS) LookUpWithoutWaiting(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
client netip.Prefix,
|
client netip.Prefix,
|
||||||
@@ -218,7 +228,8 @@ func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) {
|
|||||||
return g.kept(client)
|
return g.kept(client)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Snapshot returns every answer kept, sorted by client.
|
// Snapshot returns every answer kept, sorted by client, as lookups.json
|
||||||
|
// lists them.
|
||||||
func (g *GeoJS) Snapshot() []Answer {
|
func (g *GeoJS) Snapshot() []Answer {
|
||||||
g.mu.Lock()
|
g.mu.Lock()
|
||||||
|
|
||||||
@@ -236,9 +247,9 @@ func (g *GeoJS) Snapshot() []Answer {
|
|||||||
return answers
|
return answers
|
||||||
}
|
}
|
||||||
|
|
||||||
// Load keeps answers, such as those of an earlier Snapshot, in place of the
|
// Load keeps answers read from lookups.json, in place of the answers it
|
||||||
// answers it keeps, in the order they were last used, so that the one used
|
// keeps, in the order they were last used, so that the one used longest
|
||||||
// longest ago is dropped first. Answers GeoJS gave keepFor ago or more are
|
// ago is dropped first. Answers GeoJS gave keepFor ago or more are
|
||||||
// dropped.
|
// dropped.
|
||||||
func (g *GeoJS) Load(answers []Answer) {
|
func (g *GeoJS) Load(answers []Answer) {
|
||||||
answers = slices.Clone(answers)
|
answers = slices.Clone(answers)
|
||||||
@@ -284,11 +295,11 @@ func (g *GeoJS) answerOrWait(
|
|||||||
g.ask(ctx)
|
g.ask(ctx)
|
||||||
|
|
||||||
if !mustWait {
|
if !mustWait {
|
||||||
return Answer{}, nil // the answer is not needed before the caller goes on
|
return Answer{}, nil // the answer is not needed before the request goes on
|
||||||
}
|
}
|
||||||
|
|
||||||
if w == nil {
|
if w == nil {
|
||||||
g.wentWithoutAnswer()
|
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
||||||
|
|
||||||
return Answer{}, nil // too many clients wait already
|
return Answer{}, nil // too many clients wait already
|
||||||
}
|
}
|
||||||
@@ -299,7 +310,7 @@ func (g *GeoJS) answerOrWait(
|
|||||||
}
|
}
|
||||||
|
|
||||||
if w.late {
|
if w.late {
|
||||||
g.wentWithoutAnswer()
|
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
||||||
|
|
||||||
return Answer{}, nil
|
return Answer{}, nil
|
||||||
}
|
}
|
||||||
@@ -324,8 +335,8 @@ func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
|
|||||||
|
|
||||||
// ask starts asking GeoJS about the waiting clients, unless a request to
|
// ask starts asking GeoJS about the waiting clients, unless a request to
|
||||||
// it is under way or it is left alone after a failure. The requests to
|
// it is under way or it is left alone after a failure. The requests to
|
||||||
// GeoJS are for every client waiting, so they go on when the caller whose
|
// GeoJS are for every client waiting, so they go on when the client's
|
||||||
// ctx is given stops waiting.
|
// request whose ctx is given ends.
|
||||||
func (g *GeoJS) ask(ctx context.Context) {
|
func (g *GeoJS) ask(ctx context.Context) {
|
||||||
if g.asking || g.now().Before(g.retryAt) {
|
if g.asking || g.now().Before(g.retryAt) {
|
||||||
return
|
return
|
||||||
@@ -430,6 +441,8 @@ func (g *GeoJS) keep(
|
|||||||
}
|
}
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
g.metrics.GeoJSFailures.Inc()
|
||||||
|
|
||||||
g.retryDelay = min(
|
g.retryDelay = min(
|
||||||
max(retryDelayFactor*g.retryDelay, firstRetryDelay),
|
max(retryDelayFactor*g.retryDelay, firstRetryDelay),
|
||||||
maxRetryDelay,
|
maxRetryDelay,
|
||||||
@@ -444,7 +457,16 @@ func (g *GeoJS) keep(
|
|||||||
w.late = true
|
w.late = true
|
||||||
}
|
}
|
||||||
|
|
||||||
g.failed(err, g.retryDelay)
|
g.processLog.Warn("asking GeoJS failed",
|
||||||
|
"error", err.Error(), "asking_again_in", g.retryDelay.String())
|
||||||
|
g.alerts.Raise(alerts.Alert{
|
||||||
|
Event: alerts.EventSourceFailure,
|
||||||
|
Reason: "asking GeoJS failed",
|
||||||
|
Detail: map[string]any{
|
||||||
|
"source": "geojs", "error": err.Error(),
|
||||||
|
"asking_again_in": g.retryDelay.String(),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
|
||||||
return kept, false
|
return kept, false
|
||||||
}
|
}
|
||||||
@@ -457,8 +479,7 @@ func (g *GeoJS) keep(
|
|||||||
// request asks GeoJS about clients in one request, and returns the answer
|
// request asks GeoJS about clients in one request, and returns the answer
|
||||||
// for each address GeoJS's answer names: its AS number and the AS's name,
|
// for each address GeoJS's answer names: its AS number and the AS's name,
|
||||||
// both "" for the AS number 64512, which GeoJS gives when it knows none,
|
// both "" for the AS number 64512, which GeoJS gives when it knows none,
|
||||||
// and its country, in capitals. The request is abandoned after
|
// and its country, in capitals.
|
||||||
// requestTimeout.
|
|
||||||
func (g *GeoJS) request(
|
func (g *GeoJS) request(
|
||||||
ctx context.Context, clients []netip.Prefix,
|
ctx context.Context, clients []netip.Prefix,
|
||||||
) (map[netip.Addr]Answer, error) {
|
) (map[netip.Addr]Answer, error) {
|
||||||
@@ -468,7 +489,7 @@ func (g *GeoJS) request(
|
|||||||
addrs = append(addrs, client.Addr().String())
|
addrs = append(addrs, client.Addr().String())
|
||||||
}
|
}
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(ctx, requestTimeout)
|
ctx, cancel := context.WithTimeout(ctx, g.timeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext(
|
req, err := http.NewRequestWithContext(
|
||||||
@@ -483,7 +504,7 @@ func (g *GeoJS) request(
|
|||||||
|
|
||||||
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
|
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
|
||||||
|
|
||||||
g.asked()
|
g.metrics.GeoJSRequests.Inc()
|
||||||
|
|
||||||
res, err := g.httpClient.Do(req)
|
res, err := g.httpClient.Do(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -1,4 +1,4 @@
|
|||||||
package geojs_test
|
package lookup_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"net/netip"
|
"net/netip"
|
||||||
@@ -7,7 +7,7 @@ import (
|
|||||||
"testing/synctest"
|
"testing/synctest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
||||||
@@ -25,7 +25,7 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
|||||||
clock.advance(time.Hour)
|
clock.advance(time.Hour)
|
||||||
wantCountry(t, g, placed, germany)
|
wantCountry(t, g, placed, germany)
|
||||||
|
|
||||||
want := []geojs.Answer{
|
want := []lookup.Answer{
|
||||||
{Client: notPlaced, Answered: asked, Used: asked},
|
{Client: notPlaced, Answered: asked, Used: asked},
|
||||||
{
|
{
|
||||||
Client: placed, ASN: asn, ASName: asName, Country: germany,
|
Client: placed, ASN: asn, ASName: asName, Country: germany,
|
||||||
@@ -41,24 +41,24 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
|||||||
func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
|
func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
service, clock, g := start()
|
geojs, clock, g := start()
|
||||||
now := clock.Now()
|
now := clock.Now()
|
||||||
kept := geojs.Answer{
|
kept := lookup.Answer{
|
||||||
Client: netip.MustParsePrefix("203.0.113.9/32"),
|
Client: netip.MustParsePrefix("203.0.113.9/32"),
|
||||||
Country: "FR",
|
Country: "FR",
|
||||||
Answered: now.Add(-week + time.Second),
|
Answered: now.Add(-week + time.Second),
|
||||||
Used: now.Add(-time.Hour),
|
Used: now.Add(-time.Hour),
|
||||||
}
|
}
|
||||||
stale := geojs.Answer{
|
stale := lookup.Answer{
|
||||||
Client: netip.MustParsePrefix("203.0.113.10/32"),
|
Client: netip.MustParsePrefix("203.0.113.10/32"),
|
||||||
Country: "FR",
|
Country: "FR",
|
||||||
Answered: now.Add(-week),
|
Answered: now.Add(-week),
|
||||||
Used: now.Add(-time.Hour),
|
Used: now.Add(-time.Hour),
|
||||||
}
|
}
|
||||||
|
|
||||||
g.Load([]geojs.Answer{kept, stale})
|
g.Load([]lookup.Answer{kept, stale})
|
||||||
|
|
||||||
if got := g.Snapshot(); !slices.Equal(got, []geojs.Answer{kept}) {
|
if got := g.Snapshot(); !slices.Equal(got, []lookup.Answer{kept}) {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
"kept %+v, want only the answer GeoJS gave less than 7 days ago",
|
"kept %+v, want only the answer GeoJS gave less than 7 days ago",
|
||||||
got,
|
got,
|
||||||
@@ -66,7 +66,7 @@ func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
wantCountry(t, g, kept.Client, "FR")
|
wantCountry(t, g, kept.Client, "FR")
|
||||||
wantRequests(t, service, 0)
|
wantRequests(t, geojs, 0)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
|
func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
|
||||||
@@ -80,11 +80,11 @@ func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
|
|||||||
// lookups.json lists the answers by client. Here each was last used a
|
// lookups.json lists the answers by client. Here each was last used a
|
||||||
// second before the one listed before it, so the last listed is the
|
// second before the one listed before it, so the last listed is the
|
||||||
// one used longest ago, and the one dropped.
|
// one used longest ago, and the one dropped.
|
||||||
answers := make([]geojs.Answer, maxAnswers+1)
|
answers := make([]lookup.Answer, maxAnswers+1)
|
||||||
addr := netip.MustParseAddr("10.0.0.0")
|
addr := netip.MustParseAddr("10.0.0.0")
|
||||||
|
|
||||||
for i := range answers {
|
for i := range answers {
|
||||||
answers[i] = geojs.Answer{
|
answers[i] = lookup.Answer{
|
||||||
Client: netip.PrefixFrom(addr, addr.BitLen()),
|
Client: netip.PrefixFrom(addr, addr.BitLen()),
|
||||||
Country: germany,
|
Country: germany,
|
||||||
Answered: now,
|
Answered: now,
|
||||||
@@ -274,7 +274,8 @@ func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
|
|||||||
// AddLookupFile adds the metrics of the lookup database, read as the
|
// AddLookupFile adds the metrics of the lookup database, read as the
|
||||||
// metrics are asked for: when the file in use was read, which lastRead
|
// metrics are asked for: when the file in use was read, which lastRead
|
||||||
// returns, and the replacements of it that could not be read, which
|
// returns, and the replacements of it that could not be read, which
|
||||||
// readFailures returns.
|
// readFailures returns. The lookup package's File, which has both, cannot
|
||||||
|
// be named here: that package counts GeoJS's requests in these metrics.
|
||||||
func (m *Metrics) AddLookupFile(
|
func (m *Metrics) AddLookupFile(
|
||||||
lastRead func() time.Time,
|
lastRead func() time.Time,
|
||||||
readFailures func() int,
|
readFailures func() int,
|
||||||
|
|||||||
@@ -46,7 +46,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
@@ -135,7 +135,7 @@ type Params struct {
|
|||||||
// sww runs.
|
// sww runs.
|
||||||
Ledger *bans.Ledger
|
Ledger *bans.Ledger
|
||||||
Limiter *ratelimit.Limiter
|
Limiter *ratelimit.Limiter
|
||||||
GeoJS *geojs.GeoJS
|
GeoJS *lookup.GeoJS
|
||||||
Lists *reputation.Lists
|
Lists *reputation.Lists
|
||||||
DNSBL *reputation.DNSBL
|
DNSBL *reputation.DNSBL
|
||||||
AbuseIPDB *reputation.AbuseIPDB
|
AbuseIPDB *reputation.AbuseIPDB
|
||||||
@@ -195,8 +195,8 @@ type clientsFile struct {
|
|||||||
|
|
||||||
// lookupsFile is lookups.json, with each answer on a line of its own.
|
// lookupsFile is lookups.json, with each answer on a line of its own.
|
||||||
type lookupsFile struct {
|
type lookupsFile struct {
|
||||||
Version int `json:"version"`
|
Version int `json:"version"`
|
||||||
Lookups []geojs.Answer `json:"lookups"`
|
Lookups []lookup.Answer `json:"lookups"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// abuseIPDBChecksSpentFile is abuseipdb-checks-spent.json, indented for an
|
// abuseIPDBChecksSpentFile is abuseipdb-checks-spent.json, indented for an
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
@@ -1700,7 +1700,7 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
|
|||||||
t,
|
t,
|
||||||
lookupsJSON,
|
lookupsJSON,
|
||||||
params.GeoJS.Snapshot(),
|
params.GeoJS.Snapshot(),
|
||||||
[]geojs.Answer{
|
[]lookup.Answer{
|
||||||
{Client: client, Country: "FR", Answered: midnight()},
|
{Client: client, Country: "FR", Answered: midnight()},
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
@@ -2415,7 +2415,9 @@ func newParams(dir string) state.Params {
|
|||||||
MaxBans: 5000,
|
MaxBans: 5000,
|
||||||
}),
|
}),
|
||||||
Limiter: ratelimit.New(ratelimit.Limits{}, 20000),
|
Limiter: ratelimit.New(ratelimit.Limits{}, 20000),
|
||||||
GeoJS: geojs.New(geojs.Params{Now: midnight}),
|
GeoJS: lookup.New(lookup.Params{
|
||||||
|
Now: midnight, ProcessLog: discard, Metrics: m,
|
||||||
|
}),
|
||||||
Lists: reputation.New(reputation.Params{
|
Lists: reputation.New(reputation.Params{
|
||||||
BlocklistURLs: []string{
|
BlocklistURLs: []string{
|
||||||
blocklistURL,
|
blocklistURL,
|
||||||
@@ -2492,7 +2494,7 @@ func fill(params state.Params) {
|
|||||||
})
|
})
|
||||||
params.Limiter.AddLookup(client, now.Add(-time.Hour), asn, asName, "DE")
|
params.Limiter.AddLookup(client, now.Add(-time.Hour), asn, asName, "DE")
|
||||||
|
|
||||||
params.GeoJS.Load([]geojs.Answer{
|
params.GeoJS.Load([]lookup.Answer{
|
||||||
{
|
{
|
||||||
Client: client, ASN: asn, ASName: asName, Country: "DE",
|
Client: client, ASN: asn, ASName: asName, Country: "DE",
|
||||||
Answered: now.Add(-time.Hour), Used: now,
|
Answered: now.Add(-time.Hour), Used: now,
|
||||||
|
|||||||
+1
-2
@@ -25,7 +25,6 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
@@ -271,7 +270,7 @@ func newServer(
|
|||||||
Config: cfg,
|
Config: cfg,
|
||||||
RequestLog: stdout,
|
RequestLog: stdout,
|
||||||
ProcessLog: processLog,
|
ProcessLog: processLog,
|
||||||
GeoJSURL: geojs.URL,
|
GeoJSURL: lookup.URL,
|
||||||
AbuseIPDBURL: reputation.AbuseIPDBURL,
|
AbuseIPDBURL: reputation.AbuseIPDBURL,
|
||||||
LookupFile: lookupFile,
|
LookupFile: lookupFile,
|
||||||
CoreRuleSet: crs,
|
CoreRuleSet: crs,
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ import (
|
|||||||
"github.com/prometheus/client_golang/prometheus/testutil"
|
"github.com/prometheus/client_golang/prometheus/testutil"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
@@ -33,7 +33,7 @@ const (
|
|||||||
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, the time waf has to answer.
|
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, the time waf has to answer.
|
||||||
answerTimeLimit = 2 * time.Second
|
answerTimeLimit = 2 * time.Second
|
||||||
// defaultLookupWait is the default of SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS.
|
// defaultLookupWait is the default of SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS.
|
||||||
defaultLookupWait = 800 * time.Millisecond
|
defaultLookupWait = 500 * time.Millisecond
|
||||||
// defaultDNSBLAbuseIPDBWait is the default of SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS.
|
// defaultDNSBLAbuseIPDBWait is the default of SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS.
|
||||||
defaultDNSBLAbuseIPDBWait = 2 * time.Second
|
defaultDNSBLAbuseIPDBWait = 2 * time.Second
|
||||||
// The header limits SPEC.md gives.
|
// The header limits SPEC.md gives.
|
||||||
@@ -111,7 +111,8 @@ func TestHeaderLimitRefusalNamesItsHeuristicCountsAndLimits(t *testing.T) {
|
|||||||
s.out.requestLine(t),
|
s.out.requestLine(t),
|
||||||
"header_limits",
|
"header_limits",
|
||||||
`{"longest_header_line_bytes": 40, "header_line_bytes_limit": 8192,
|
`{"longest_header_line_bytes": 40, "header_line_bytes_limit": 8192,
|
||||||
"fields": 101, "fields_limit": 100, "cookies": 0, "cookies_limit": 50}`,
|
"fields": 101, "fields_limit": 100, "cookie_lines": 0,
|
||||||
|
"cookie_lines_limit": 1, "cookies": 0, "cookies_limit": 50}`,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -198,6 +199,9 @@ func TestAHeaderLineOverTheLimitIsRefusedWith431(t *testing.T) {
|
|||||||
{"a header line over the limit", func(req *http.Request) {
|
{"a header line over the limit", func(req *http.Request) {
|
||||||
req.Header.Set("X-Long", strings.Repeat("a", lineLimit+1-len("X-Long: ")))
|
req.Header.Set("X-Long", strings.Repeat("a", lineLimit+1-len("X-Long: ")))
|
||||||
}, true},
|
}, true},
|
||||||
|
{"a Cookie line at the limit", func(req *http.Request) {
|
||||||
|
req.Header.Set(cookieHeader, "a="+strings.Repeat("a", lineLimit-len("Cookie: a=")))
|
||||||
|
}, false},
|
||||||
{"one cookie over the limit", func(req *http.Request) {
|
{"one cookie over the limit", func(req *http.Request) {
|
||||||
req.Header.Set(cookieHeader, "a="+strings.Repeat("a", lineLimit))
|
req.Header.Set(cookieHeader, "a="+strings.Repeat("a", lineLimit))
|
||||||
}, true},
|
}, true},
|
||||||
@@ -267,8 +271,8 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
|
|||||||
{"parts of no-break spaces count", func(req *http.Request) {
|
{"parts of no-break spaces count", func(req *http.Request) {
|
||||||
req.Header.Set(cookieHeader, strings.Repeat(" ;", cookieLimit+1))
|
req.Header.Set(cookieHeader, strings.Repeat(" ;", cookieLimit+1))
|
||||||
}, true},
|
}, true},
|
||||||
{"cookies over the limit across two headers", func(req *http.Request) {
|
{"two Cookie lines", func(req *http.Request) {
|
||||||
req.Header[cookieHeader] = []string{cookies(cookieLimit), "last=1"}
|
req.Header[cookieHeader] = []string{"a=1", "b=1"}
|
||||||
}, true},
|
}, true},
|
||||||
{"cookies without a value count", func(req *http.Request) {
|
{"cookies without a value count", func(req *http.Request) {
|
||||||
req.Header.Set(cookieHeader, strings.Repeat("a;", cookieLimit+1))
|
req.Header.Set(cookieHeader, strings.Repeat("a;", cookieLimit+1))
|
||||||
@@ -591,7 +595,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
|
|||||||
s := startWAF(t, geojsURL, map[string]string{
|
s := startWAF(t, geojsURL, map[string]string{
|
||||||
unknownCountryLimitPercent: "0", rateLimitPerMinute: "1",
|
unknownCountryLimitPercent: "0", rateLimitPerMinute: "1",
|
||||||
})
|
})
|
||||||
s.server.GeoJS.Load([]geojs.Answer{{
|
s.server.GeoJS.Load([]lookup.Answer{{
|
||||||
Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512",
|
Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512",
|
||||||
Answered: s.clock.Now(), Used: s.clock.Now(),
|
Answered: s.clock.Now(), Used: s.clock.Now(),
|
||||||
}})
|
}})
|
||||||
@@ -770,8 +774,6 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
|
|||||||
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says,
|
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says,
|
||||||
// so that a request that does not wait takes no time at all.
|
// so that a request that does not wait takes no time at all.
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
defer waitForGeoJSRequestsToBeAbandoned()
|
|
||||||
|
|
||||||
server, out := newWAFInBubble(t, tc.env)
|
server, out := newWAFInBubble(t, tc.env)
|
||||||
tc.prepare(t, server, time.Now())
|
tc.prepare(t, server, time.Now())
|
||||||
|
|
||||||
@@ -840,8 +842,6 @@ func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays(
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
defer waitForGeoJSRequestsToBeAbandoned()
|
|
||||||
|
|
||||||
server, out := newWAFInBubble(t, tc.env)
|
server, out := newWAFInBubble(t, tc.env)
|
||||||
tc.prepare(t, server, time.Now())
|
tc.prepare(t, server, time.Now())
|
||||||
|
|
||||||
@@ -954,11 +954,10 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
|
|||||||
t.Errorf("%d checks with AbuseIPDB, want 1", checked)
|
t.Errorf("%d checks with AbuseIPDB, want 1", checked)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The query is answered, and the check with AbuseIPDB and the request
|
// The query is answered, and the check with AbuseIPDB abandoned,
|
||||||
// to GeoJS abandoned, before the bubble ends.
|
// before the bubble ends.
|
||||||
releaseResolver()
|
releaseResolver()
|
||||||
time.Sleep(defaultDNSBLAbuseIPDBWait)
|
time.Sleep(defaultDNSBLAbuseIPDBWait)
|
||||||
waitForGeoJSRequestsToBeAbandoned()
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -339,8 +339,6 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownCountryLimitPercent(
|
|||||||
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, with
|
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, with
|
||||||
// a GeoJS that never answers.
|
// a GeoJS that never answers.
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
defer waitForGeoJSRequestsToBeAbandoned()
|
|
||||||
|
|
||||||
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
||||||
map[string]string{unknownCountryLimitPercent: "0"})
|
map[string]string{unknownCountryLimitPercent: "0"})
|
||||||
|
|
||||||
@@ -392,8 +390,6 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
|
|||||||
// with a GeoJS that never answers. The headers, which need the
|
// with a GeoJS that never answers. The headers, which need the
|
||||||
// answer too, are off.
|
// answer too, are off.
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
defer waitForGeoJSRequestsToBeAbandoned()
|
|
||||||
|
|
||||||
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
||||||
map[string]string{
|
map[string]string{
|
||||||
asnAndCountryLookupTimeout: strconv.FormatInt(
|
asnAndCountryLookupTimeout: strconv.FormatInt(
|
||||||
|
|||||||
@@ -10,11 +10,15 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The header limits: the most cookies and header fields a request may have,
|
// The header limits: the most Cookie lines, cookies and header fields a request
|
||||||
// and the longest its request line and each header line may be, in bytes, not
|
// may have, and the longest its request line and each header line may be, in
|
||||||
// counting the line ending. Real clients rarely reach them. maxLineBytes is
|
// bytes, not counting the line ending. Real clients rarely reach them.
|
||||||
// what nginx allows by default.
|
// maxLineBytes is what nginx allows by default. A browser sends all its cookies
|
||||||
|
// on one Cookie line, as RFC 6265 requires, and Go's server joins the Cookie
|
||||||
|
// lines of an HTTP/2 request into one. A second line would only give a visitor
|
||||||
|
// another 8 KiB for the Core Rule Set to read.
|
||||||
const (
|
const (
|
||||||
|
maxCookieLines = 1
|
||||||
maxCookies = 50
|
maxCookies = 50
|
||||||
maxHeaderFields = 100
|
maxHeaderFields = 100
|
||||||
maxLineBytes = 8 << 10
|
maxLineBytes = 8 << 10
|
||||||
@@ -22,10 +26,11 @@ const (
|
|||||||
|
|
||||||
// headerLimits refuses a request with a request line longer than maxLineBytes,
|
// headerLimits refuses a request with a request line longer than maxLineBytes,
|
||||||
// with 414. It refuses a request with a header line longer than maxLineBytes,
|
// with 414. It refuses a request with a header line longer than maxLineBytes,
|
||||||
// more than maxHeaderFields header fields or more than maxCookies cookies, with
|
// more than maxHeaderFields header fields, more than maxCookieLines Cookie
|
||||||
// 431. Its figures are the request line's length and its limit for 414. For
|
// lines or more than maxCookies cookies, with 431. Its figures are the request
|
||||||
// 431 they are the longest header line, the header fields and the cookies,
|
// line's length and its limit for 414. For 431 they are the longest header
|
||||||
// each with its limit.
|
// line, the header fields, the Cookie lines and the cookies, each with its
|
||||||
|
// limit.
|
||||||
type headerLimits struct{}
|
type headerLimits struct{}
|
||||||
|
|
||||||
func (headerLimits) name() string { return "header_limits" }
|
func (headerLimits) name() string { return "header_limits" }
|
||||||
@@ -48,9 +53,10 @@ func (headerLimits) check(
|
|||||||
|
|
||||||
longest := rq.longestHeaderLineBytes()
|
longest := rq.longestHeaderLineBytes()
|
||||||
fields, cookies := rq.headerCounts()
|
fields, cookies := rq.headerCounts()
|
||||||
|
cookieLines := len(rq.in.Header.Values("Cookie"))
|
||||||
|
|
||||||
if longest <= maxLineBytes && fields <= maxHeaderFields &&
|
if longest <= maxLineBytes && fields <= maxHeaderFields &&
|
||||||
cookies <= maxCookies {
|
cookieLines <= maxCookieLines && cookies <= maxCookies {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -60,6 +66,7 @@ func (headerLimits) check(
|
|||||||
"longest_header_line_bytes": longest,
|
"longest_header_line_bytes": longest,
|
||||||
"header_line_bytes_limit": maxLineBytes,
|
"header_line_bytes_limit": maxLineBytes,
|
||||||
"fields": fields, "fields_limit": maxHeaderFields,
|
"fields": fields, "fields_limit": maxHeaderFields,
|
||||||
|
"cookie_lines": cookieLines, "cookie_lines_limit": maxCookieLines,
|
||||||
"cookies": cookies, "cookies_limit": maxCookies,
|
"cookies": cookies, "cookies_limit": maxCookies,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|||||||
+6
-26
@@ -7,9 +7,8 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The headers in which an answer of allow gives rproxy the client's AS number
|
// The headers in which an answer of allow gives rproxy the client's AS number
|
||||||
@@ -27,9 +26,9 @@ const (
|
|||||||
// SWW_WAF_LOOKUP_SOURCE is off or the client is on a private, loopback or
|
// SWW_WAF_LOOKUP_SOURCE is off or the client is on a private, loopback or
|
||||||
// link-local address, which no lookup can place. The lookup database
|
// link-local address, which no lookup can place. The lookup database
|
||||||
// answers at once. With GeoJS, while a setting needs the answer, such as a
|
// answers at once. With GeoJS, while a setting needs the answer, such as a
|
||||||
// country list or a biased threshold, a new client's request waits for it up
|
// country list or a biased threshold, a new client's request waits for it,
|
||||||
// to SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, unless refusedWhateverTheAnswer
|
// unless refusedWhateverTheAnswer reports true. ctx is the request's own
|
||||||
// reports true. ctx is the request's own context.
|
// context.
|
||||||
func (rq *request) lookUp(ctx context.Context, now time.Time) {
|
func (rq *request) lookUp(ctx context.Context, now time.Time) {
|
||||||
if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) {
|
if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) {
|
||||||
return
|
return
|
||||||
@@ -43,11 +42,8 @@ func (rq *request) lookUp(ctx context.Context, now time.Time) {
|
|||||||
case rq.refusedWhateverTheAnswer(now):
|
case rq.refusedWhateverTheAnswer(now):
|
||||||
rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client)
|
rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client)
|
||||||
rq.skippedLookupWait = rq.lookupAnswer.Answered.IsZero()
|
rq.skippedLookupWait = rq.lookupAnswer.Answered.IsZero()
|
||||||
case !rq.h.waitForGeoJS:
|
|
||||||
rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client)
|
|
||||||
default:
|
default:
|
||||||
rq.lookupAnswer = rq.h.geojs.LookUp(ctx, client,
|
rq.lookupAnswer = rq.h.geojs.LookUp(ctx, client)
|
||||||
rq.h.config.ASNAndCountryLookupTimeout)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.lookedUp = true
|
rq.lookedUp = true
|
||||||
@@ -87,29 +83,13 @@ func countryTag(country string) string {
|
|||||||
// addLookup adds answer, an answer about a client from the lookup
|
// addLookup adds answer, an answer about a client from the lookup
|
||||||
// database or GeoJS, to the client's history, and to the notes of the bans
|
// database or GeoJS, to the client's history, and to the notes of the bans
|
||||||
// on its netblock that have no AS number, AS name or country yet.
|
// on its netblock that have no AS number, AS name or country yet.
|
||||||
func (h *handler) addLookup(answer geojs.Answer) {
|
func (h *handler) addLookup(answer lookup.Answer) {
|
||||||
h.limiter.AddLookup(answer.Client, answer.Answered,
|
h.limiter.AddLookup(answer.Client, answer.Answered,
|
||||||
answer.ASN, answer.ASName, answer.Country)
|
answer.ASN, answer.ASName, answer.Country)
|
||||||
h.ledger.AddLookup(h.netblock(answer.Client.Addr()),
|
h.ledger.AddLookup(h.netblock(answer.Client.Addr()),
|
||||||
answer.ASN, answer.ASName, answer.Country)
|
answer.ASN, answer.ASName, answer.Country)
|
||||||
}
|
}
|
||||||
|
|
||||||
// geojsFailed logs a failure of GeoJS, raises a source_failure alert for it
|
|
||||||
// and counts it. askingAgainIn is how long GeoJS is left alone after it.
|
|
||||||
func (h *handler) geojsFailed(err error, askingAgainIn time.Duration) {
|
|
||||||
h.processLog.Warn("asking GeoJS failed",
|
|
||||||
"error", err.Error(), "asking_again_in", askingAgainIn.String())
|
|
||||||
h.alerts.Raise(alerts.Alert{
|
|
||||||
Event: alerts.EventSourceFailure,
|
|
||||||
Reason: "asking GeoJS failed",
|
|
||||||
Detail: map[string]any{
|
|
||||||
"source": "geojs", "error": err.Error(),
|
|
||||||
"asking_again_in": askingAgainIn.String(),
|
|
||||||
},
|
|
||||||
})
|
|
||||||
h.metrics.GeoJSFailures.Inc()
|
|
||||||
}
|
|
||||||
|
|
||||||
// setLookupHeaders sets the headers that give the client's AS number and
|
// setLookupHeaders sets the headers that give the client's AS number and
|
||||||
// country, leaving out one that is unknown.
|
// country, leaving out one that is unknown.
|
||||||
func setLookupHeaders(header http.Header, asn, country string) {
|
func setLookupHeaders(header http.Header, asn, country string) {
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
@@ -138,7 +138,7 @@ func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) {
|
|||||||
// of lookups.json would keep it, while the first request still waits
|
// of lookups.json would keep it, while the first request still waits
|
||||||
// for GeoJS. The second request goes on at once, and is counted.
|
// for GeoJS. The second request goes on at once, and is counted.
|
||||||
clk.advance(2500 * time.Millisecond)
|
clk.advance(2500 * time.Millisecond)
|
||||||
server.GeoJS.Load([]geojs.Answer{{
|
server.GeoJS.Load([]lookup.Answer{{
|
||||||
Client: netip.MustParsePrefix(
|
Client: netip.MustParsePrefix(
|
||||||
fromDE + "/32",
|
fromDE + "/32",
|
||||||
), ASN: asnDE, ASName: asNameDE,
|
), ASN: asnDE, ASName: asNameDE,
|
||||||
@@ -199,7 +199,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(
|
|||||||
// The answer is kept before the requests, so GeoJS is not asked, and
|
// The answer is kept before the requests, so GeoJS is not asked, and
|
||||||
// gives no answer of its own.
|
// gives no answer of its own.
|
||||||
netblock := netip.MustParsePrefix(fromDE + "/32")
|
netblock := netip.MustParsePrefix(fromDE + "/32")
|
||||||
server.GeoJS.Load([]geojs.Answer{{
|
server.GeoJS.Load([]lookup.Answer{{
|
||||||
Client: netblock, ASN: asnDE, ASName: asNameDE, Country: "DE",
|
Client: netblock, ASN: asnDE, ASName: asNameDE, Country: "DE",
|
||||||
Answered: clk.Now(), Used: clk.Now(),
|
Answered: clk.Now(), Used: clk.Now(),
|
||||||
}})
|
}})
|
||||||
@@ -249,7 +249,7 @@ func TestLookupSourceOffLooksNoClientUp(t *testing.T) {
|
|||||||
)
|
)
|
||||||
|
|
||||||
// Even an answer kept from before is not used.
|
// Even an answer kept from before is not used.
|
||||||
server.GeoJS.Load([]geojs.Answer{{
|
server.GeoJS.Load([]lookup.Answer{{
|
||||||
Client: netip.MustParsePrefix(
|
Client: netip.MustParsePrefix(
|
||||||
fromDE + "/32",
|
fromDE + "/32",
|
||||||
), ASN: asnDE, ASName: asNameDE,
|
), ASN: asnDE, ASName: asNameDE,
|
||||||
@@ -450,8 +450,6 @@ func TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays(
|
|||||||
// network, which would keep that clock from moving on: the request is
|
// network, which would keep that clock from moving on: the request is
|
||||||
// handed to waf's handler, and GeoJS is one that never answers.
|
// handed to waf's handler, and GeoJS is one that never answers.
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
defer waitForGeoJSRequestsToBeAbandoned()
|
|
||||||
|
|
||||||
// Not the default. The exclusive list needs the answer, and the app
|
// Not the default. The exclusive list needs the answer, and the app
|
||||||
// is never reached.
|
// is never reached.
|
||||||
const timeout = 3 * time.Second
|
const timeout = 3 * time.Second
|
||||||
@@ -497,17 +495,6 @@ func TestMain(m *testing.M) {
|
|||||||
m.Run()
|
m.Run()
|
||||||
}
|
}
|
||||||
|
|
||||||
// geojsRequestTimeout is how long a request to GeoJS may take before it is
|
|
||||||
// abandoned.
|
|
||||||
const geojsRequestTimeout = 10 * time.Second
|
|
||||||
|
|
||||||
// waitForGeoJSRequestsToBeAbandoned sleeps until every request to the GeoJS at
|
|
||||||
// silentGeoJSURL made so far has been abandoned. A synctest bubble that asks
|
|
||||||
// it defers this: the bubble cannot end while a request waits on it.
|
|
||||||
func waitForGeoJSRequestsToBeAbandoned() {
|
|
||||||
time.Sleep(geojsRequestTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
// silentGeoJS is the GeoJS at silentGeoJSURL.
|
// silentGeoJS is the GeoJS at silentGeoJSURL.
|
||||||
type silentGeoJS struct{}
|
type silentGeoJS struct{}
|
||||||
|
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
package waf_test
|
package waf_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
|
||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
@@ -11,9 +10,8 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
)
|
)
|
||||||
@@ -250,7 +248,7 @@ func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) {
|
|||||||
|
|
||||||
// The answers are kept before the requests, so that none waits for
|
// The answers are kept before the requests, so that none waits for
|
||||||
// GeoJS.
|
// GeoJS.
|
||||||
server.GeoJS.Load([]geojs.Answer{
|
server.GeoJS.Load([]lookup.Answer{
|
||||||
keptAnswer(fromKP, "KP"), keptAnswer(fromDE, "DE"),
|
keptAnswer(fromKP, "KP"), keptAnswer(fromDE, "DE"),
|
||||||
keptAnswer(fromFR, "FR"),
|
keptAnswer(fromFR, "FR"),
|
||||||
})
|
})
|
||||||
@@ -331,13 +329,13 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther(
|
|||||||
|
|
||||||
// The answers are kept before the requests, so that GeoJS gives none
|
// The answers are kept before the requests, so that GeoJS gives none
|
||||||
// of its own. Each client is in an AS of its own.
|
// of its own. Each client is in an AS of its own.
|
||||||
answer := func(addr, asn string) geojs.Answer {
|
answer := func(addr, asn string) lookup.Answer {
|
||||||
return geojs.Answer{
|
return lookup.Answer{
|
||||||
Client: netip.MustParsePrefix(addr + "/32"), ASN: asn,
|
Client: netip.MustParsePrefix(addr + "/32"), ASN: asn,
|
||||||
Answered: clk.Now(), Used: clk.Now(),
|
Answered: clk.Now(), Used: clk.Now(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
server.GeoJS.Load([]geojs.Answer{
|
server.GeoJS.Load([]lookup.Answer{
|
||||||
answer(fromDE, "AS64501"), answer(fromKP, "AS64502"),
|
answer(fromDE, "AS64501"), answer(fromKP, "AS64502"),
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -359,23 +357,21 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther(
|
|||||||
`sww_asn_requests_total{asn="AS64502",instance="app"}`)
|
`sww_asn_requests_total{asn="AS64502",instance="app"}`)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestGeoJSFailureIsCountedLoggedAndAlerted(t *testing.T) {
|
func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
failingGeoJS := httptest.NewServer(http.HandlerFunc(
|
geojs := httptest.NewServer(http.HandlerFunc(
|
||||||
func(w http.ResponseWriter, _ *http.Request) {
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
w.WriteHeader(http.StatusServiceUnavailable)
|
w.WriteHeader(http.StatusServiceUnavailable)
|
||||||
}))
|
}))
|
||||||
t.Cleanup(failingGeoJS.Close)
|
t.Cleanup(geojs.Close)
|
||||||
|
|
||||||
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
||||||
addr, out, server, queue := startProxyWithAlerts(t, app.URL,
|
addr, _, server := startProxyWithClock(t, app.URL, geojs.URL, time.Now,
|
||||||
failingGeoJS.URL, time.Now,
|
|
||||||
map[string]string{
|
map[string]string{
|
||||||
trustedProxies: trustLocalhost,
|
trustedProxies: trustLocalhost,
|
||||||
metricsToken: token,
|
metricsToken: token,
|
||||||
tagDeny: denyKP,
|
tagDeny: denyKP,
|
||||||
alertWebhookURL: "https://alerts.example/waf",
|
|
||||||
})
|
})
|
||||||
|
|
||||||
// GeoJS fails, so the client has no country= tag, and SWW_WAF_TAG_DENY does
|
// GeoJS fails, so the client has no country= tag, and SWW_WAF_TAG_DENY does
|
||||||
@@ -405,46 +401,14 @@ func TestGeoJSFailureIsCountedLoggedAndAlerted(t *testing.T) {
|
|||||||
`sww_requests_without_geojs_answer_total{instance="app"}`,
|
`sww_requests_without_geojs_answer_total{instance="app"}`,
|
||||||
1,
|
1,
|
||||||
)
|
)
|
||||||
|
|
||||||
// The process log's line names the failure but not the address asked
|
|
||||||
// about.
|
|
||||||
logged := ""
|
|
||||||
|
|
||||||
for line := range strings.Lines(out.text()) {
|
|
||||||
if strings.Contains(line, "asking GeoJS failed") {
|
|
||||||
logged = line
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if logged == "" || strings.Contains(logged, fromDE) {
|
|
||||||
t.Errorf(
|
|
||||||
"logged %q, want the failure without the address asked about",
|
|
||||||
logged,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// fmt prints a map sorted by its keys.
|
|
||||||
const want = "map[asking_again_in:1s " +
|
|
||||||
"error:GeoJS answered 503 Service Unavailable source:geojs]"
|
|
||||||
|
|
||||||
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
|
||||||
if len(waiting) != 1 || waiting[0].Event != alerts.EventSourceFailure ||
|
|
||||||
waiting[0].Reason != "asking GeoJS failed" ||
|
|
||||||
fmt.Sprint(waiting[0].Detail) != want {
|
|
||||||
t.Errorf(
|
|
||||||
"alerts waiting %+v, want one source_failure alert with %v",
|
|
||||||
waiting,
|
|
||||||
want,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// keptAnswer returns GeoJS's answer that the client at addr is in
|
// keptAnswer returns GeoJS's answer that the client at addr is in
|
||||||
// country, given now.
|
// country, given now.
|
||||||
func keptAnswer(addr, country string) geojs.Answer {
|
func keptAnswer(addr, country string) lookup.Answer {
|
||||||
now := time.Now()
|
now := time.Now()
|
||||||
|
|
||||||
return geojs.Answer{
|
return lookup.Answer{
|
||||||
Client: netip.MustParsePrefix(addr + "/32"), Country: country,
|
Client: netip.MustParsePrefix(addr + "/32"), Country: country,
|
||||||
Answered: now, Used: now,
|
Answered: now, Used: now,
|
||||||
}
|
}
|
||||||
|
|||||||
+17
-20
@@ -17,7 +17,6 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
@@ -56,7 +55,7 @@ type Params struct {
|
|||||||
// ProcessLog receives the process's own messages.
|
// ProcessLog receives the process's own messages.
|
||||||
ProcessLog *slog.Logger
|
ProcessLog *slog.Logger
|
||||||
// GeoJSURL is where clients' AS numbers and countries are looked up
|
// GeoJSURL is where clients' AS numbers and countries are looked up
|
||||||
// while SWW_WAF_LOOKUP_SOURCE is geojs, normally geojs.URL.
|
// while SWW_WAF_LOOKUP_SOURCE is geojs, normally lookup.URL.
|
||||||
GeoJSURL string
|
GeoJSURL string
|
||||||
// AbuseIPDBURL is where clients are checked with AbuseIPDB while
|
// AbuseIPDBURL is where clients are checked with AbuseIPDB while
|
||||||
// SWW_WAF_ABUSEIPDB_KEY is set, normally reputation.AbuseIPDBURL.
|
// SWW_WAF_ABUSEIPDB_KEY is set, normally reputation.AbuseIPDBURL.
|
||||||
@@ -94,7 +93,7 @@ type Server struct {
|
|||||||
|
|
||||||
Ledger *bans.Ledger
|
Ledger *bans.Ledger
|
||||||
Limiter *ratelimit.Limiter
|
Limiter *ratelimit.Limiter
|
||||||
GeoJS *geojs.GeoJS
|
GeoJS *lookup.GeoJS
|
||||||
Anomalies *anomaly.Counters
|
Anomalies *anomaly.Counters
|
||||||
LookupFile *lookup.File
|
LookupFile *lookup.File
|
||||||
Lists *reputation.Lists
|
Lists *reputation.Lists
|
||||||
@@ -165,19 +164,20 @@ func newHandler(params Params) *handler {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// The country list, the headers, the tag settings that name an asn= or
|
h.geojs = lookup.New(lookup.Params{
|
||||||
// country= tag and the biased thresholds act on the answer before the
|
URL: params.GeoJSURL,
|
||||||
// request goes on.
|
Timeout: params.Config.ASNAndCountryLookupTimeout,
|
||||||
h.waitForGeoJS = len(params.Config.ExclusivelyAllowedCountries) > 0 ||
|
// The country list, the headers, the tag settings that name an asn=
|
||||||
params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() ||
|
// or country= tag and the biased thresholds act on the answer before
|
||||||
biasedThresholdsSet(params.Config)
|
// the request goes on.
|
||||||
h.geojs = geojs.New(geojs.Params{
|
Wait: len(params.Config.ExclusivelyAllowedCountries) > 0 ||
|
||||||
URL: params.GeoJSURL,
|
params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() ||
|
||||||
Answered: h.addLookup,
|
biasedThresholdsSet(params.Config),
|
||||||
Now: params.Now,
|
Answered: h.addLookup,
|
||||||
Asked: m.GeoJSRequests.Inc,
|
Now: params.Now,
|
||||||
Failed: h.geojsFailed,
|
ProcessLog: params.ProcessLog,
|
||||||
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
|
Metrics: m,
|
||||||
|
Alerts: params.Alerts,
|
||||||
})
|
})
|
||||||
m.AddBansAndClients(h.ledger, h.limiter, params.Now)
|
m.AddBansAndClients(h.ledger, h.limiter, params.Now)
|
||||||
m.AddRules(params.Rules)
|
m.AddRules(params.Rules)
|
||||||
@@ -268,7 +268,7 @@ type handler struct {
|
|||||||
metrics *metrics.Metrics
|
metrics *metrics.Metrics
|
||||||
limiter *ratelimit.Limiter
|
limiter *ratelimit.Limiter
|
||||||
ledger *bans.Ledger
|
ledger *bans.Ledger
|
||||||
geojs *geojs.GeoJS
|
geojs *lookup.GeoJS
|
||||||
anomalies *anomaly.Counters
|
anomalies *anomaly.Counters
|
||||||
lookupFile *lookup.File
|
lookupFile *lookup.File
|
||||||
lists *reputation.Lists
|
lists *reputation.Lists
|
||||||
@@ -278,9 +278,6 @@ type handler struct {
|
|||||||
rules *rules.Files
|
rules *rules.Files
|
||||||
coreRuleSet *coreruleset.CoreRuleSet
|
coreRuleSet *coreruleset.CoreRuleSet
|
||||||
alerts *alerts.Queue
|
alerts *alerts.Queue
|
||||||
// waitForGeoJS is true when a setting needs each request's answer from
|
|
||||||
// GeoJS before the request goes on.
|
|
||||||
waitForGeoJS bool
|
|
||||||
// awaitingReport holds the decision on each answered question by its id
|
// awaitingReport holds the decision on each answered question by its id
|
||||||
// until its report comes, and nil for a question still being handled.
|
// until its report comes, and nil for a question still being handled.
|
||||||
awaitingReport *lru.Cache[string, *decision]
|
awaitingReport *lru.Cache[string, *decision]
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import (
|
|||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
@@ -162,7 +162,7 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
|
|||||||
|
|
||||||
// The answers are kept before the requests, so that none waits for
|
// The answers are kept before the requests, so that none waits for
|
||||||
// GeoJS.
|
// GeoJS.
|
||||||
server.GeoJS.Load([]geojs.Answer{
|
server.GeoJS.Load([]lookup.Answer{
|
||||||
keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"),
|
keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"),
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||||
@@ -42,7 +42,7 @@ type decision struct {
|
|||||||
// looked up, whether or not an answer was there, and lookupAnswer is
|
// looked up, whether or not an answer was there, and lookupAnswer is
|
||||||
// what the lookup gave then, the zero Answer while GeoJS had given none.
|
// what the lookup gave then, the zero Answer while GeoJS had given none.
|
||||||
lookedUp bool
|
lookedUp bool
|
||||||
lookupAnswer geojs.Answer
|
lookupAnswer lookup.Answer
|
||||||
// skippedLookupWait is true for a request that did not wait for GeoJS and
|
// skippedLookupWait is true for a request that did not wait for GeoJS and
|
||||||
// found no answer kept. SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT does not apply to it,
|
// found no answer kept. SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT does not apply to it,
|
||||||
// so that not waiting never lowers its client's limits.
|
// so that not waiting never lowers its client's limits.
|
||||||
@@ -382,9 +382,9 @@ func anomalyThresholdsSet(cfg *config.WAF) bool {
|
|||||||
// answer about it as the request ends, and whether there is one: the
|
// answer about it as the request ends, and whether there is one: the
|
||||||
// lookup database's, which was there at once, or the one GeoJS has given
|
// lookup database's, which was there at once, or the one GeoJS has given
|
||||||
// by then, which a request does not wait for unless a setting needs it.
|
// by then, which a request does not wait for unless a setting needs it.
|
||||||
func (rq *request) answerAtTheEnd() (geojs.Answer, bool) {
|
func (rq *request) answerAtTheEnd() (lookup.Answer, bool) {
|
||||||
if !rq.lookedUp {
|
if !rq.lookedUp {
|
||||||
return geojs.Answer{}, false
|
return lookup.Answer{}, false
|
||||||
}
|
}
|
||||||
|
|
||||||
if rq.h.config.LookupSource == "file" {
|
if rq.h.config.LookupSource == "file" {
|
||||||
|
|||||||
+1
-4
@@ -2,9 +2,6 @@
|
|||||||
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
||||||
# checks out and runs this and nothing else, and script/fmt-check runs
|
# checks out and runs this and nothing else, and script/fmt-check runs
|
||||||
# the formatter on the host, which a pristine checkout cannot do.
|
# the formatter on the host, which a pristine checkout cannot do.
|
||||||
# --no-cache for the same reason as script/docker: the gate phases the
|
|
||||||
# final stage depends on are RUN steps, and a cached one is a check that
|
|
||||||
# did not run.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -20,7 +17,7 @@ main() {
|
|||||||
# the version a build stage derives from the .git in the context.
|
# the version a build stage derives from the .git in the context.
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
docker build --no-cache \
|
docker build \
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$("$SCRIPT_DIR/projectname")" .
|
-t "$("$SCRIPT_DIR/projectname")" .
|
||||||
}
|
}
|
||||||
|
|||||||
+1
-3
@@ -1,8 +1,6 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/docker: build the Docker image tagged with the project name.
|
# script/docker: build the Docker image tagged with the project name.
|
||||||
# Identical in all repos; the tag comes from script/projectname.
|
# Identical in all repos; the tag comes from script/projectname.
|
||||||
# --no-cache because the gate phases the final stage depends on are RUN
|
|
||||||
# steps, and a cached one is a check that did not run.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -16,7 +14,7 @@ main() {
|
|||||||
# the version a build stage derives from the .git in the context.
|
# the version a build stage derives from the .git in the context.
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
docker build --no-cache \
|
docker build \
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$("$SCRIPT_DIR/projectname")" .
|
-t "$("$SCRIPT_DIR/projectname")" .
|
||||||
}
|
}
|
||||||
|
|||||||
+2
-2
@@ -170,8 +170,8 @@ main() {
|
|||||||
trap cleanup EXIT
|
trap cleanup EXIT
|
||||||
trap 'exit 1' HUP INT TERM
|
trap 'exit 1' HUP INT TERM
|
||||||
|
|
||||||
docker build --no-cache -t "$IMAGE" .
|
docker build -t "$IMAGE" .
|
||||||
docker build --no-cache --build-arg SWW_IMAGE="$IMAGE" \
|
docker build --build-arg SWW_IMAGE="$IMAGE" \
|
||||||
-t "$APP_IMAGE" deploy/example-app
|
-t "$APP_IMAGE" deploy/example-app
|
||||||
|
|
||||||
docker network create "$NETWORK" >/dev/null
|
docker network create "$NETWORK" >/dev/null
|
||||||
|
|||||||
+2
-3
@@ -5,8 +5,7 @@
|
|||||||
# make its answer untrustworthy.
|
# make its answer untrustworthy.
|
||||||
#
|
#
|
||||||
# The phase is not the last stage in the file, so it is built only when
|
# The phase is not the last stage in the file, so it is built only when
|
||||||
# --target names it. --no-cache because a cached lint layer is a lint
|
# --target names it. The tag makes each build replace the previous image
|
||||||
# that did not run. The tag makes each build replace the previous image
|
|
||||||
# instead of leaving a dangling one behind.
|
# instead of leaving a dangling one behind.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
@@ -15,7 +14,7 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
docker build --no-cache \
|
docker build \
|
||||||
--target lint \
|
--target lint \
|
||||||
-t "$("$SCRIPT_DIR/projectname")-lint" .
|
-t "$("$SCRIPT_DIR/projectname")-lint" .
|
||||||
}
|
}
|
||||||
|
|||||||
+2
-3
@@ -2,8 +2,7 @@
|
|||||||
# script/test: run the test suite. Testing is a phase of the Dockerfile
|
# script/test: run the test suite. Testing is a phase of the Dockerfile
|
||||||
# and this builds that phase alone, on the same terms as script/lint:
|
# and this builds that phase alone, on the same terms as script/lint:
|
||||||
# --target because a phase that is not the last stage is built only when
|
# --target because a phase that is not the last stage is built only when
|
||||||
# named, --no-cache because a cached test layer is a test that did not
|
# named, and a tag so each build replaces the previous image.
|
||||||
# run, and a tag so each build replaces the previous image.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -11,7 +10,7 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
docker build --no-cache \
|
docker build \
|
||||||
--target test \
|
--target test \
|
||||||
-t "$("$SCRIPT_DIR/projectname")-test" .
|
-t "$("$SCRIPT_DIR/projectname")-test" .
|
||||||
}
|
}
|
||||||
|
|||||||
+1
-3
@@ -3,15 +3,13 @@
|
|||||||
# the test phase of the Dockerfile checks. This builds the Dockerfile's
|
# the test phase of the Dockerfile checks. This builds the Dockerfile's
|
||||||
# tidy-files stage, which holds the two files alone, and --output writes
|
# tidy-files stage, which holds the two files alone, and --output writes
|
||||||
# them into the working tree. The build makes no image, so it has no tag.
|
# them into the working tree. The build makes no image, so it has no tag.
|
||||||
# --no-cache because `go mod tidy` asks the module proxy, whose answers a
|
|
||||||
# cached layer would repeat.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
docker build --no-cache \
|
docker build \
|
||||||
--target tidy-files \
|
--target tidy-files \
|
||||||
--output "type=local,dest=$ROOT" .
|
--output "type=local,dest=$ROOT" .
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in new issue
Block a user