Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
b0ac6fd9ce | ||
 clawbot
|
3550956cbb |
No files matched your search
@@ -260,7 +260,7 @@ Every setting the code reads, with its default:
|
||||
| `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
|
||||
| `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
||||
| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take, in milliseconds. |
|
||||
| `SWW_WAF_ADD_LOOKUP_HEADERS` | `false` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
||||
| `SWW_WAF_ADD_LOOKUP_HEADERS` | `true`, or `false` while the lookup source is `off` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
||||
| `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
||||
| `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
||||
| `SWW_WAF_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. |
|
||||
@@ -540,9 +540,10 @@ request reaches the app.
|
||||
A refused visitor is answered with `SWW_WAF_BAN_RESPONSE`. A `block` rule or a
|
||||
Core Rule Set match answers `403` and bans nobody, since the Core Rule Set has
|
||||
false positives; such refusals count toward `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE`
|
||||
instead. A request over the header limits of step 3 is answered `431`, and bans
|
||||
nobody either. `rproxy` itself answers `405` to a method it does not pass on,
|
||||
and the status for a size or time limit, such as `413` for a body too large.
|
||||
instead. A request over the header limits of step 3 is answered `414` for a
|
||||
request line over 8 KiB and `431` otherwise. It bans nobody either. `rproxy`
|
||||
itself answers `405` to a method it does not pass on, and the status for a size
|
||||
or time limit, such as `413` for a body too large.
|
||||
|
||||
### Clients and bans
|
||||
|
||||
|
||||
@@ -151,26 +151,26 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in
|
||||
around its address, a /64 by default, since one abuser usually controls a
|
||||
whole /64. A client is therefore one IPv4 address or one IPv6 group.
|
||||
3. The header limits (`header_limits`; see "Answering in time"). They come first
|
||||
after `SWW_WAF_ALLOW_NETS`, since they only count header fields and cookies.
|
||||
Next, the tag lists (`tag_lists`; see "Request tags"). Each list that holds
|
||||
the client adds its tag. Next, `routewatch_tags` adds the client's `asn=` and
|
||||
`prefix=` tags while `SWW_WAF_ROUTEWATCH_URL` is set. A client in
|
||||
`SWW_WAF_ALLOW_NETS` skips the header limits and gets no tag.
|
||||
after `SWW_WAF_ALLOW_NETS`, since they only measure lines and count header
|
||||
fields and cookies. Next, the tag lists (`tag_lists`; see "Request tags").
|
||||
Each list that holds the client adds its tag. Next, `routewatch_tags` adds
|
||||
the client's `asn=` and `prefix=` tags while `SWW_WAF_ROUTEWATCH_URL` is set.
|
||||
A client in `SWW_WAF_ALLOW_NETS` skips the header limits and gets no tag.
|
||||
4. The static lists. A client in `SWW_WAF_ALLOW_NETS` skips steps 5 to 15, the
|
||||
byte limits and the refusal limit. A client in `SWW_WAF_DENY_NETS` is refused
|
||||
(`deny_nets`); a client in both lists is let through.
|
||||
5. Bans. A client under a ban is refused (`active_ban`; see "Bans").
|
||||
6. The lookup, unless `SWW_WAF_LOOKUP_SOURCE` is `off` (see "Lookup of AS number
|
||||
and country"). The request goes on at once. It waits for the first GeoJS
|
||||
answer, up to `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, only when one of
|
||||
these needs it:
|
||||
and country"). The request waits for the first GeoJS answer, up to
|
||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, when one of these needs it:
|
||||
- `SWW_WAF_ADD_LOOKUP_HEADERS`, which is `true` by default;
|
||||
- the country list;
|
||||
- a biased threshold;
|
||||
- `SWW_WAF_TAG_DENY` or `SWW_WAF_TAG_LIMIT_PERCENT` naming an `asn=` or
|
||||
`country=` tag;
|
||||
- `SWW_WAF_ADD_LOOKUP_HEADERS`.
|
||||
`country=` tag.
|
||||
|
||||
A request that ends before the answer comes is logged without it.
|
||||
Otherwise the request goes on at once. A request that ends before the answer
|
||||
comes is logged without it.
|
||||
|
||||
7. The country list (`countries`; see "Country list").
|
||||
8. The reputation sources (see "Reputation"), in this order:
|
||||
@@ -197,10 +197,10 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in
|
||||
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||
prefix.
|
||||
17. `rproxy` passes the request to `SWW_RPROXY_UPSTREAM_URL`, streaming (see
|
||||
"Passing a request to the app"). With `SWW_WAF_ADD_LOOKUP_HEADERS`,
|
||||
`X-Client-ASN` and `X-Client-Country` are added for the app's own logs; a
|
||||
header whose value is unknown is left out. Whatever that setting says, any
|
||||
such header the client sent is removed.
|
||||
"Passing a request to the app"). While `SWW_WAF_ADD_LOOKUP_HEADERS` is
|
||||
`true`, as by default, `X-Client-ASN` and `X-Client-Country` are added for
|
||||
the app's own logs; a header whose value is unknown is left out. Whatever
|
||||
that setting says, any such header the client sent is removed.
|
||||
18. After the visitor's answer has ended, `rproxy` sends the report (see "The
|
||||
report"). `waf` then counts:
|
||||
- For a request it allowed, the body bytes of the response, of the request
|
||||
@@ -252,9 +252,9 @@ does not reach neighbours who did nothing. Requests refused under a ban are not
|
||||
counted against limits. They are counted in the client's history and in the
|
||||
ban's notes.
|
||||
|
||||
A request over the header limits is refused with `431` before the bans are
|
||||
checked (see "Answering in time"). It makes no ban, makes no ban permanent and
|
||||
is not counted in a ban's notes.
|
||||
A request over the header limits is refused with `414` or `431` before the bans
|
||||
are checked (see "Answering in time"). It makes no ban, makes no ban permanent
|
||||
and is not counted in a ban's notes.
|
||||
|
||||
An offence is a request `waf` holds against the client. It is one of:
|
||||
|
||||
@@ -2373,15 +2373,15 @@ runs no check, writes no request log line and counts nothing in the metrics.
|
||||
answer but the last below carries `X-Sww-Disposition`.
|
||||
|
||||
- `X-Sww-Disposition: allow`, status `200`, no body. `rproxy` passes the request
|
||||
to the app. With `SWW_WAF_ADD_LOOKUP_HEADERS` the answer also carries
|
||||
`X-Client-ASN` and `X-Client-Country`, which `rproxy` adds to the request it
|
||||
passes on. A header whose value is unknown is left out.
|
||||
to the app. While `SWW_WAF_ADD_LOOKUP_HEADERS` is `true`, the answer also
|
||||
carries `X-Client-ASN` and `X-Client-Country`, which `rproxy` adds to the
|
||||
request it passes on. A header whose value is unknown is left out.
|
||||
- `X-Sww-Disposition: deny`, with the status, headers and body the visitor is to
|
||||
get. `rproxy` gives them to the visitor, less this header. The status is one
|
||||
of:
|
||||
- `SWW_WAF_BAN_RESPONSE`, for a refusal;
|
||||
- `403`, for a `block` rule or the Core Rule Set;
|
||||
- `431`, for a request over the header limits;
|
||||
- `414` or `431`, for a request over the header limits;
|
||||
- `404`, for a path under `/_sww/` that passes the checks, so that the app
|
||||
never sees that prefix.
|
||||
- `X-Sww-Disposition: close`, status `200`, no body. This takes the place of a
|
||||
@@ -2442,22 +2442,31 @@ never wait. These can still wait for GeoJS:
|
||||
- a rule file;
|
||||
- the Core Rule Set.
|
||||
|
||||
The header limits are fixed, with no setting. `waf` refuses a request over
|
||||
either one:
|
||||
The header limits are fixed, with no setting. A line's length is in bytes and
|
||||
does not count its line ending. `waf` refuses a request whose request line is
|
||||
longer than 8 KiB with `414`. An example is `GET /search?q=` followed by 30,000
|
||||
characters.
|
||||
|
||||
It refuses a request with `431` for any of:
|
||||
|
||||
- a header line longer than 8 KiB. Each value of a header sent twice is a line
|
||||
of its own. `Host` counts as a line.
|
||||
- more than 50 cookies. Each part of its `Cookie` headers between semicolons
|
||||
counts as one, unless it is empty or only spaces and tabs.
|
||||
- more than 100 header fields. A header sent twice counts twice. `Host` and
|
||||
`Transfer-Encoding` do not count. `X-Request-ID` and `X-Forwarded-Proto` do.
|
||||
|
||||
Real clients never reach these limits. The answer is `deny` with `431`, and the
|
||||
log line's action is `request_header_too_large`. The refusal makes no ban and is
|
||||
no offence.
|
||||
Real clients rarely reach these limits. nginx refuses lines over 8 KiB by
|
||||
default, and Apache's default limit is about the same. The answer is `deny`. The
|
||||
log line's action is `request_line_too_long` for `414` and
|
||||
`request_header_too_large` for `431`. The refusal makes no ban and is no
|
||||
offence.
|
||||
|
||||
The header limits are the first check after `SWW_WAF_ALLOW_NETS`. A request over
|
||||
them is refused before any later check runs. So it causes no lookup, no query to
|
||||
an outside source and no rule match, and the rate limits do not count it. A
|
||||
client under a ban or in `SWW_WAF_DENY_NETS` also gets `431` for such a request.
|
||||
client under a ban or in `SWW_WAF_DENY_NETS` also gets `414` or `431` for such a
|
||||
request.
|
||||
|
||||
The limits apply on every path and in every `SWW_WAF_CORE_RULE_SET_MODE`. A
|
||||
client in `SWW_WAF_ALLOW_NETS` skips them, as it skips the Core Rule Set. In
|
||||
@@ -2465,9 +2474,10 @@ client in `SWW_WAF_ALLOW_NETS` skips them, as it skips the Core Rule Set. In
|
||||
other check runs on it.
|
||||
|
||||
The Core Rule Set runs each of its rules on each cookie, so thousands of small
|
||||
cookies would keep it busy for over a second. Within the limits, one long cookie
|
||||
or query string can still keep it busy for a fraction of a second. So enough
|
||||
such requests at once can overload `waf` past
|
||||
cookies would keep it busy for over a second. One 30 KB cookie or query string
|
||||
would keep it busy for a third of a second or more. A request with several long
|
||||
`Cookie` lines, each within the limits, can keep it as busy. Enough such
|
||||
requests at once can still overload `waf` past
|
||||
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
|
||||
|
||||
### The report
|
||||
|
||||
+6
-2
@@ -102,7 +102,8 @@ Every one of these lines has:
|
||||
ban this request made or made permanent;
|
||||
- `country_denied`, `client_is_rate_limited`, `blocked_by_block_rule` and
|
||||
`blocked_by_core_rule_set`: refused by that check;
|
||||
- `request_header_too_large`: over a header limit;
|
||||
- `request_line_too_long`: a request line over the header limits;
|
||||
- `request_header_too_large`: headers over the header limits;
|
||||
- `own_endpoint`: answered at one of `waf`'s own endpoints, or `404` for a
|
||||
path under `/_sww/`.
|
||||
- `would_action`: in `observe` mode, what `enforce` mode would have done with a
|
||||
@@ -124,7 +125,10 @@ Every one of these lines has:
|
||||
- `rate_limits`: `window`, `count` and `limit`;
|
||||
- `trap_paths`: `path`, the trap path asked for;
|
||||
- `rule_files`: `rule_id` and `rule_action`, of the rule that refused it;
|
||||
- `header_limits`: `fields`, `fields_limit`, `cookies` and `cookies_limit`;
|
||||
- `header_limits`: for a request line, `request_line_bytes` and
|
||||
`request_line_bytes_limit`; for headers, `longest_header_line_bytes`,
|
||||
`header_line_bytes_limit`, `fields`, `fields_limit`, `cookies` and
|
||||
`cookies_limit`;
|
||||
- `core_rule_set`: `score`, and `threshold`,
|
||||
`SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`;
|
||||
- `refusal_limit`: `window`, `count` and `limit`, of the refusal limit;
|
||||
|
||||
+8
-5
@@ -150,7 +150,7 @@ cut off; the client is banned.
|
||||
- `SWW_WAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `off`
|
||||
stops the start while a setting that needs a lookup is set:
|
||||
- `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES`;
|
||||
- `SWW_WAF_ADD_LOOKUP_HEADERS`;
|
||||
- `SWW_WAF_ADD_LOOKUP_HEADERS` set to `true`;
|
||||
- `SWW_WAF_TAG_DENY` or `SWW_WAF_TAG_LIMIT_PERCENT` naming a `country=` tag,
|
||||
or an `asn=` tag while `SWW_WAF_ROUTEWATCH_URL` is unset;
|
||||
- the two byte percentage lists and `SWW_WAF_ASN_LIMIT_PERCENT_URL`;
|
||||
@@ -161,9 +161,11 @@ cut off; the client is banned.
|
||||
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` (default `800`): how long a
|
||||
request waits for its client's first answer from GeoJS, and how long a request
|
||||
to GeoJS may take. A time in milliseconds; never `off`.
|
||||
- `SWW_WAF_ADD_LOOKUP_HEADERS` (default `false`): `true` has `rproxy` pass the
|
||||
- `SWW_WAF_ADD_LOOKUP_HEADERS` (default `true`): `true` has `rproxy` pass the
|
||||
app the client's AS number in `X-Client-ASN` and its country in
|
||||
`X-Client-Country`.
|
||||
`X-Client-Country`. The default is `false` while `SWW_WAF_LOOKUP_SOURCE` is
|
||||
`off`. With GeoJS, a new client's first request waits for its answer, up to
|
||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`.
|
||||
|
||||
## Country list
|
||||
|
||||
@@ -421,8 +423,9 @@ A few limits are fixed rather than settings:
|
||||
- at most 20,000 anomaly counters are kept;
|
||||
- each alert destination's queue holds at most 1,000 alerts;
|
||||
- `waf` holds at most 10,000 answered requests awaiting their reports;
|
||||
- `waf` refuses a request with more than 50 cookies or more than 100 header
|
||||
fields (see "Answering in time" under "`waf`" in [`SPEC.md`](../SPEC.md));
|
||||
- `waf` refuses a request with a request line or header line over 8 KiB, more
|
||||
than 50 cookies or more than 100 header fields (see "Answering in time" under
|
||||
"`waf`" in [`SPEC.md`](../SPEC.md));
|
||||
- `rproxy` takes a request line and headers of at most 32 KiB, and closes a
|
||||
kept-open connection idle for 120 seconds;
|
||||
- `rproxy` reads at most 1 MiB of the body of `waf`'s answer, and answers `502`
|
||||
|
||||
@@ -84,7 +84,8 @@ type WAF struct {
|
||||
// client's first answer from GeoJS while a setting needs it
|
||||
// (SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS), which cannot be off.
|
||||
// AddLookupHeaders is true when the app is passed the client's AS number
|
||||
// and country in headers (SWW_WAF_ADD_LOOKUP_HEADERS).
|
||||
// and country in headers (SWW_WAF_ADD_LOOKUP_HEADERS), by default unless
|
||||
// LookupSource is off.
|
||||
LookupSource string
|
||||
LookupDBPath string
|
||||
ASNAndCountryLookupTimeout time.Duration
|
||||
@@ -549,7 +550,6 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
|
||||
"SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS",
|
||||
"800",
|
||||
),
|
||||
AddLookupHeaders: env.boolean("SWW_WAF_ADD_LOOKUP_HEADERS", "false"),
|
||||
ExclusivelyAllowedCountries: env.countries(
|
||||
"SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
|
||||
TagLists: env.tagLists("SWW_WAF_TAG_LISTS"),
|
||||
@@ -724,6 +724,9 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
|
||||
|
||||
cfg.LogRemoteAppName = env.appName("SWW_WAF_LOG_REMOTE_APP_NAME",
|
||||
cfg.InstanceName, cfg.LogRemoteURL != nil)
|
||||
// The app gets the headers by default, unless no client is looked up.
|
||||
cfg.AddLookupHeaders = env.boolean("SWW_WAF_ADD_LOOKUP_HEADERS",
|
||||
strconv.FormatBool(cfg.LookupSource != off))
|
||||
cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action(
|
||||
"SWW_WAF_BLOCKLIST_ACTION", "deny")
|
||||
cfg.DNSBLAbuseIPDBAction, cfg.DNSBLAbuseIPDBLimitPercent = env.action(
|
||||
|
||||
@@ -287,6 +287,7 @@ func TestDefaults(t *testing.T) {
|
||||
})
|
||||
wantLookupSettings(t, cfg, config.WAF{
|
||||
LookupSource: defaultLookupSource, ASNAndCountryLookupTimeout: 800 * time.Millisecond,
|
||||
AddLookupHeaders: true,
|
||||
})
|
||||
wantByteLimitSettings(t, cfg, config.WAF{
|
||||
BytesLimitPerMinute: 0, BytesLimitPerHour: 50 << 30,
|
||||
@@ -1395,13 +1396,13 @@ func TestLookupSettingsAsSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := wafFromEnvironment(t, environment{
|
||||
asnAndCountryLookupTimeout: "500", addLookupHeaders: enabled,
|
||||
asnAndCountryLookupTimeout: "500", addLookupHeaders: disabled,
|
||||
})
|
||||
wantLookupSettings(t, cfg, config.WAF{
|
||||
LookupSource: defaultLookupSource, ASNAndCountryLookupTimeout: 500 * time.Millisecond,
|
||||
AddLookupHeaders: true,
|
||||
})
|
||||
|
||||
// With no client looked up, the headers are off unless set.
|
||||
cfg = wafFromEnvironment(t, environment{lookupSource: off})
|
||||
wantLookupSettings(t, cfg, config.WAF{
|
||||
LookupSource: off, ASNAndCountryLookupTimeout: 800 * time.Millisecond,
|
||||
@@ -1422,6 +1423,10 @@ func TestLookupDBPathGoesWithTheFileSourceAlone(t *testing.T) {
|
||||
cfg.LookupSource, cfg.LookupDBPath, path)
|
||||
}
|
||||
|
||||
if !cfg.AddLookupHeaders {
|
||||
t.Error("headers off by default with lookups from file, want on")
|
||||
}
|
||||
|
||||
for _, tc := range []struct {
|
||||
env environment
|
||||
want string
|
||||
@@ -2927,7 +2932,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
lookupSource: defaultLookupSource,
|
||||
lookupDBPath: "",
|
||||
asnAndCountryLookupTimeout: "800",
|
||||
addLookupHeaders: disabled,
|
||||
addLookupHeaders: enabled,
|
||||
allowedCountries: "",
|
||||
tagLists: "",
|
||||
tagDeny: "",
|
||||
|
||||
@@ -21,7 +21,12 @@ const (
|
||||
// the app.
|
||||
ActionForward = "forward"
|
||||
|
||||
// ActionRequestHeaderTooLarge is a request over the header limits.
|
||||
// ActionRequestLineTooLong is a request whose request line is too long for
|
||||
// the header limits.
|
||||
ActionRequestLineTooLong = "request_line_too_long"
|
||||
|
||||
// ActionRequestHeaderTooLarge is a request whose headers are over the
|
||||
// header limits.
|
||||
ActionRequestHeaderTooLarge = "request_header_too_large"
|
||||
|
||||
// ActionClientIsRateLimited is a request refused because it took its
|
||||
@@ -143,7 +148,7 @@ type Line struct {
|
||||
// taken with a request it would have refused: ActionClientDenied,
|
||||
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
|
||||
// ActionClientIsRateLimited, ActionBlockedByBlockRule,
|
||||
// ActionBlockedByCoreRuleSet, or, in waf,
|
||||
// ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or
|
||||
// ActionRequestHeaderTooLarge for the header limits.
|
||||
WouldAction string `json:"would_action,omitempty"`
|
||||
// Heuristic is the name of the heuristic that refused the request, in
|
||||
|
||||
@@ -349,6 +349,7 @@ func TestRequestCountsForTheASNumberGeoJSGivesBeforeItEnds(t *testing.T) {
|
||||
map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
asnAndCountryLookupTimeout: "3600000",
|
||||
addLookupHeaders: falseValue,
|
||||
anomalyASN + requestsPerMinute: aLot,
|
||||
})
|
||||
server.Store(started)
|
||||
|
||||
@@ -39,6 +39,7 @@ const (
|
||||
// The header limits SPEC.md gives.
|
||||
cookieLimit = 50
|
||||
headerFieldLimit = 100
|
||||
lineLimit = 8192
|
||||
// cookieHeader is the header the cookie limit counts.
|
||||
cookieHeader = "Cookie"
|
||||
// tooLarge is how waf refuses a request over the header limits.
|
||||
@@ -104,11 +105,139 @@ func TestHeaderLimitRefusalNamesItsHeuristicCountsAndLimits(t *testing.T) {
|
||||
fillHeaders(req, headerFieldLimit+1)
|
||||
do(t, req)
|
||||
|
||||
// The longest header line is X-Request-ID's, with its 26-character id.
|
||||
wantHeuristic(
|
||||
t,
|
||||
s.out.requestLine(t),
|
||||
"header_limits",
|
||||
`{"fields": 101, "fields_limit": 100, "cookies": 0, "cookies_limit": 50}`,
|
||||
`{"longest_header_line_bytes": 40, "header_line_bytes_limit": 8192,
|
||||
"fields": 101, "fields_limit": 100, "cookies": 0, "cookies_limit": 50}`,
|
||||
)
|
||||
}
|
||||
|
||||
// longQuery returns a path whose request line, sent by Go's client as
|
||||
// "GET <path> HTTP/1.1", is lineBytes long.
|
||||
func longQuery(lineBytes int) string {
|
||||
const around = len("GET /?q= HTTP/1.1")
|
||||
|
||||
return "/?q=" + strings.Repeat("a", lineBytes-around)
|
||||
}
|
||||
|
||||
func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
lineBytes int
|
||||
// cookies is how many cookies the request carries.
|
||||
cookies int
|
||||
status int
|
||||
action string
|
||||
}{
|
||||
{
|
||||
"at the limit", lineLimit, 0, http.StatusOK,
|
||||
requestlog.ActionForward,
|
||||
},
|
||||
{
|
||||
"over the limit", lineLimit + 1, 0, http.StatusRequestURITooLong,
|
||||
requestlog.ActionRequestLineTooLong,
|
||||
},
|
||||
{
|
||||
"over the limit and the cookie limit", lineLimit + 1,
|
||||
cookieLimit + 1, http.StatusRequestURITooLong,
|
||||
requestlog.ActionRequestLineTooLong,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := startWAF(t, "", map[string]string{coreRuleSetMode: block})
|
||||
req := s.question(siteA, client, longQuery(tc.lineBytes))
|
||||
|
||||
if tc.cookies > 0 {
|
||||
req.Header.Set(cookieHeader, cookies(tc.cookies))
|
||||
}
|
||||
|
||||
got := do(t, req)
|
||||
line := s.out.requestLine(t)
|
||||
|
||||
if tc.status == http.StatusOK {
|
||||
wantDisposition(t, got, tc.status, waf.DispositionAllow)
|
||||
} else {
|
||||
wantDisposition(t, got, tc.status, waf.DispositionDeny)
|
||||
}
|
||||
|
||||
wantLine(t, line, tc.status, tc.action)
|
||||
|
||||
refused := tc.status != http.StatusOK
|
||||
if refused != (line.DurationCoreRuleSetMS == nil) {
|
||||
t.Errorf(
|
||||
"Core Rule Set time %v for a request refused: %t",
|
||||
line.DurationCoreRuleSetMS,
|
||||
refused,
|
||||
)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestAHeaderLineOverTheLimitIsRefusedWith431(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
make func(*http.Request)
|
||||
// refused is true for a request over the limit, and false for one at it.
|
||||
refused bool
|
||||
}{
|
||||
{"a header line at the limit", func(req *http.Request) {
|
||||
req.Header.Set("X-Long", strings.Repeat("a", lineLimit-len("X-Long: ")))
|
||||
}, false},
|
||||
{"a header line over the limit", func(req *http.Request) {
|
||||
req.Header.Set("X-Long", strings.Repeat("a", lineLimit+1-len("X-Long: ")))
|
||||
}, true},
|
||||
{"one cookie over the limit", func(req *http.Request) {
|
||||
req.Header.Set(cookieHeader, "a="+strings.Repeat("a", lineLimit))
|
||||
}, true},
|
||||
{"a Host line over the limit", func(req *http.Request) {
|
||||
req.Host = strings.Repeat("a", lineLimit+1-len("Host: "))
|
||||
}, true},
|
||||
{"a header sent twice, each line within the limit", func(req *http.Request) {
|
||||
req.Header.Add("X-Long", strings.Repeat("a", 8000))
|
||||
req.Header.Add("X-Long", strings.Repeat("a", 8000))
|
||||
}, false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := startWAF(t, "", nil)
|
||||
req := s.question(siteA, client, "/")
|
||||
tc.make(req)
|
||||
|
||||
got := do(t, req)
|
||||
|
||||
if tc.refused {
|
||||
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
|
||||
} else {
|
||||
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRequestLineRefusalNamesItsHeuristicLengthAndLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := startWAF(t, "", nil)
|
||||
do(t, s.question(siteA, client, longQuery(lineLimit+1)))
|
||||
|
||||
wantHeuristic(
|
||||
t,
|
||||
s.out.requestLine(t),
|
||||
"header_limits",
|
||||
`{"request_line_bytes": 8193, "request_line_bytes_limit": 8192}`,
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -387,14 +387,16 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
|
||||
|
||||
// In a synctest bubble, as
|
||||
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says,
|
||||
// with a GeoJS that never answers.
|
||||
// with a GeoJS that never answers. The headers, which need the
|
||||
// answer too, are off.
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
||||
map[string]string{
|
||||
asnAndCountryLookupTimeout: strconv.FormatInt(
|
||||
timeout.Milliseconds(), 10),
|
||||
tagLists: "vpn=" + vpnList,
|
||||
tc.setting: tc.value,
|
||||
addLookupHeaders: falseValue,
|
||||
tagLists: "vpn=" + vpnList,
|
||||
tc.setting: tc.value,
|
||||
})
|
||||
began := time.Now()
|
||||
|
||||
|
||||
@@ -108,8 +108,11 @@ func TestCountryRefusalNamesItsHeuristicCountryAndSetting(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
geojsURL, _ := startGeoJS(t)
|
||||
// The headers are off, so that only the country list makes the request
|
||||
// wait for its answer.
|
||||
s, _, _ := startWithClock(t, geojsURL, map[string]string{
|
||||
asnAndCountryLookupTimeout: "3600000", allowedCountries: "de",
|
||||
asnAndCountryLookupTimeout: "3600000", addLookupHeaders: falseValue,
|
||||
allowedCountries: "de",
|
||||
})
|
||||
|
||||
line := s.get(
|
||||
@@ -239,7 +242,10 @@ func TestPrivateAddressIsNeverLookedUp(t *testing.T) {
|
||||
name string
|
||||
env map[string]string
|
||||
}{
|
||||
{"no setting needs the lookup", nil},
|
||||
{
|
||||
"no setting needs the lookup",
|
||||
map[string]string{addLookupHeaders: falseValue},
|
||||
},
|
||||
{"a tag setting needs it", map[string]string{tagDeny: denyKP}},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
|
||||
@@ -10,16 +10,22 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// The header limits: the most cookies and header fields a request may have.
|
||||
// Real clients never reach them.
|
||||
// The header limits: the most cookies and header fields a request may have,
|
||||
// and the longest its request line and each header line may be, in bytes, not
|
||||
// counting the line ending. Real clients rarely reach them. maxLineBytes is
|
||||
// what nginx allows by default.
|
||||
const (
|
||||
maxCookies = 50
|
||||
maxHeaderFields = 100
|
||||
maxLineBytes = 8 << 10
|
||||
)
|
||||
|
||||
// headerLimits refuses a request with more than maxHeaderFields header fields
|
||||
// or more than maxCookies cookies, with 431. Its figures are the request's
|
||||
// header fields and cookies, and the limit of each.
|
||||
// headerLimits refuses a request with a request line longer than maxLineBytes,
|
||||
// with 414. It refuses a request with a header line longer than maxLineBytes,
|
||||
// more than maxHeaderFields header fields or more than maxCookies cookies, with
|
||||
// 431. Its figures are the request line's length and its limit for 414. For
|
||||
// 431 they are the longest header line, the header fields and the cookies,
|
||||
// each with its limit.
|
||||
type headerLimits struct{}
|
||||
|
||||
func (headerLimits) name() string { return "header_limits" }
|
||||
@@ -29,20 +35,59 @@ func (headerLimits) check(
|
||||
rq *request,
|
||||
_ time.Time,
|
||||
) *verdict {
|
||||
requestLine := rq.requestLineBytes()
|
||||
if requestLine > maxLineBytes {
|
||||
return &verdict{
|
||||
action: requestlog.ActionRequestLineTooLong,
|
||||
figures: map[string]any{
|
||||
"request_line_bytes": requestLine,
|
||||
"request_line_bytes_limit": maxLineBytes,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
longest := rq.longestHeaderLineBytes()
|
||||
fields, cookies := rq.headerCounts()
|
||||
if fields <= maxHeaderFields && cookies <= maxCookies {
|
||||
|
||||
if longest <= maxLineBytes && fields <= maxHeaderFields &&
|
||||
cookies <= maxCookies {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionRequestHeaderTooLarge,
|
||||
figures: map[string]any{
|
||||
"fields": fields, "fields_limit": maxHeaderFields,
|
||||
"longest_header_line_bytes": longest,
|
||||
"header_line_bytes_limit": maxLineBytes,
|
||||
"fields": fields, "fields_limit": maxHeaderFields,
|
||||
"cookies": cookies, "cookies_limit": maxCookies,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// requestLineBytes returns the length of the request's request line, such as
|
||||
// "GET /search?q=x HTTP/1.1", not counting its line ending. The target is the
|
||||
// one the visitor sent, which rproxy's question carries unchanged.
|
||||
func (rq *request) requestLineBytes() int {
|
||||
return len(rq.in.Method) + len(" ") + len(rq.in.RequestURI) +
|
||||
len(" ") + len(rq.in.Proto)
|
||||
}
|
||||
|
||||
// longestHeaderLineBytes returns the length of the request's longest header
|
||||
// line, such as "Cookie: a=1", not counting its line ending. Each value of a
|
||||
// header sent twice is a line of its own, and Host is a line too.
|
||||
func (rq *request) longestHeaderLineBytes() int {
|
||||
longest := len("Host: ") + len(rq.in.Host)
|
||||
|
||||
for name, values := range rq.in.Header {
|
||||
for _, value := range values {
|
||||
longest = max(longest, len(name)+len(": ")+len(value))
|
||||
}
|
||||
}
|
||||
|
||||
return longest
|
||||
}
|
||||
|
||||
// headerCounts returns the request's header fields, each value of a header sent
|
||||
// twice being one, and its cookies, each part of a Cookie header between
|
||||
// semicolons being one unless it is empty or only spaces and tabs. The Core
|
||||
|
||||
@@ -41,8 +41,8 @@ const (
|
||||
|
||||
// beforeLookup are the heuristics that run before the client's AS number and
|
||||
// country are looked up, so that a client they refuse is not looked up.
|
||||
// header_limits comes first: it only counts, and a request it refuses causes no
|
||||
// other work.
|
||||
// header_limits comes first: it only measures and counts, and a request it
|
||||
// refuses causes no other work.
|
||||
func beforeLookup() []heuristic {
|
||||
return []heuristic{
|
||||
headerLimits{},
|
||||
|
||||
@@ -12,7 +12,7 @@ import (
|
||||
)
|
||||
|
||||
// The headers in which an answer of allow gives rproxy the client's AS number
|
||||
// and country, for the app, while SWW_WAF_ADD_LOOKUP_HEADERS is set. Go writes
|
||||
// and country, for the app, while SWW_WAF_ADD_LOOKUP_HEADERS is true. Go writes
|
||||
// every header name in this form, as it sends it and as it receives it, so
|
||||
// X-Client-ASN arrives as X-Client-Asn; header names are not case-sensitive.
|
||||
const (
|
||||
|
||||
@@ -36,6 +36,7 @@ func TestEveryClientIsLookedUpWithoutWaitingWhileNoSettingNeedsIt(
|
||||
geojsURL, asked, release := startHeldGeoJS(t)
|
||||
s, _, server := startWithClock(t, geojsURL, map[string]string{
|
||||
asnAndCountryLookupTimeout: "3600000",
|
||||
addLookupHeaders: falseValue,
|
||||
rateLimitPerMinute: "1",
|
||||
})
|
||||
|
||||
@@ -418,7 +419,7 @@ func TestClientsOwnLookupHeadersAreRemovedWhileTheSettingIsOff(
|
||||
geojsURL,
|
||||
time.Now,
|
||||
map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
trustedProxies: trustLocalhost, addLookupHeaders: falseValue,
|
||||
},
|
||||
)
|
||||
s := &sender{t: t, addr: addr, out: out}
|
||||
@@ -457,6 +458,8 @@ func TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays(
|
||||
map[string]string{
|
||||
asnAndCountryLookupTimeout: strconv.FormatInt(
|
||||
timeout.Milliseconds(), 10),
|
||||
// Off, so that only the country list makes the request wait.
|
||||
addLookupHeaders: falseValue,
|
||||
allowedCountries: "DE",
|
||||
})
|
||||
|
||||
|
||||
@@ -93,6 +93,10 @@ const off = "off"
|
||||
// on is the value that switches a setting that takes true or false on.
|
||||
const on = "true"
|
||||
|
||||
// falseValue is the value that switches a setting that takes true or false
|
||||
// off.
|
||||
const falseValue = "false"
|
||||
|
||||
// output collects what sww writes on stdout.
|
||||
type output struct {
|
||||
mu sync.Mutex
|
||||
|
||||
@@ -177,7 +177,7 @@ func requestHeaders(r *http.Request, names []string) map[string]string {
|
||||
// check is the one place where a request can be refused. It returns nil to let
|
||||
// the request through. The heuristics of checkClient decide: a request they
|
||||
// refuse is answered with SWW_WAF_BAN_RESPONSE, 403 for a block rule or the Core
|
||||
// Rule Set, or 431 for the header limits. In observe mode nothing is refused,
|
||||
// Rule Set, or 414 or 431 for the header limits. In observe mode nothing is refused,
|
||||
// and the log line names what enforce mode would have done. ctx is the
|
||||
// request's own context.
|
||||
func (rq *request) check(ctx context.Context) *refusal {
|
||||
@@ -196,6 +196,8 @@ func (rq *request) check(ctx context.Context) *refusal {
|
||||
case action == requestlog.ActionBlockedByBlockRule ||
|
||||
action == requestlog.ActionBlockedByCoreRuleSet:
|
||||
return &refusal{status: http.StatusForbidden, action: action}
|
||||
case action == requestlog.ActionRequestLineTooLong:
|
||||
return &refusal{status: http.StatusRequestURITooLong, action: action}
|
||||
case action == requestlog.ActionRequestHeaderTooLarge:
|
||||
return &refusal{
|
||||
status: http.StatusRequestHeaderFieldsTooLarge,
|
||||
|
||||
+1
-1
@@ -320,7 +320,7 @@ func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request {
|
||||
}
|
||||
|
||||
// allow answers a question with DispositionAllow and, while
|
||||
// SWW_WAF_ADD_LOOKUP_HEADERS is set, the client's AS number and country.
|
||||
// SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country.
|
||||
func (rq *request) allow() {
|
||||
rq.allowed = true
|
||||
|
||||
|
||||
@@ -869,7 +869,7 @@ func TestAllowCarriesTheLookupHeadersWhileTheSettingIsSet(t *testing.T) {
|
||||
wantASN, wantCountry string
|
||||
}{
|
||||
{on, "AS64496", "DE"},
|
||||
{"false", "", ""},
|
||||
{falseValue, "", ""},
|
||||
} {
|
||||
t.Run(tc.setting, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -1690,7 +1690,12 @@ func TestALargeBodyAnnouncedAndLongHeadersAreNoReasonToRefuse(t *testing.T) {
|
||||
strings.NewReader("a body"))
|
||||
req.Header.Set(waf.ContentLengthHeader, "1073741824")
|
||||
req.Header.Set(forwardedFor, otherClient)
|
||||
req.Header.Set("X-Long", strings.Repeat("x", 40<<10))
|
||||
|
||||
// 40,000 bytes of headers, over rproxy's 32 KiB, in lines within the
|
||||
// header limits.
|
||||
for i := range 5 {
|
||||
req.Header.Set("X-Long-"+strconv.Itoa(i), strings.Repeat("x", 8000))
|
||||
}
|
||||
|
||||
got := do(t, req)
|
||||
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
|
||||
|
||||
Reference in new issue
Block a user