Per-client request rate limits over a minute, an hour and a day (closes #43)
check / check (push) Successful in 2m50s
check / check (push) Successful in 2m50s
Each client, one IPv4 address or one IPv6 /64, is counted in two buckets per window, the earlier weighted by how much of it the window covers; at most 20,000 clients are kept, least recently seen dropped first. A request over SWWAF_RATE_LIMIT_PER_MINUTE, _HOUR or _DAY (1000, 10000, 50000, or off) gets 429 before reaching the app. Refused requests count, 413s included. A clock set back over a second behind a bucket's start restarts that window. The log line gains limit_hit and the action rate_limited. Deviation from SPEC.md, per the issue: the 20,000 bound and /64 are fixed. Judgement call: golang-lru/v2 holds the table; httprate does not count refused requests. Deviation: go.mod and go.sum hand-written; no make target tidies them. Model: opus-5-5
This commit was merged in pull request #48.
This commit is contained in:
@@ -12,14 +12,15 @@ state in memory and in JSON files you can read and edit, and writes a detailed
|
|||||||
JSON log line for every request.
|
JSON log line for every request.
|
||||||
|
|
||||||
Status: the first milestone is built
|
Status: the first milestone is built
|
||||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/13). `smallwebwaf` passes each
|
(https://git.eeqj.de/sneak/smallwebwaf/issues/13), and the rate limits of the
|
||||||
request to the app and the app's answer back, unchanged, within its timeouts and
|
second (https://git.eeqj.de/sneak/smallwebwaf/issues/14). `smallwebwaf` passes
|
||||||
size limits, works out each client's address, and writes a JSON log line for
|
each request to the app and the app's answer back, unchanged, within its
|
||||||
every request. Rate limits per client, the country lists and the image an app
|
timeouts and size limits, works out each client's address, refuses a client that
|
||||||
builds on come with milestone 2
|
sends too many requests, and writes a JSON log line for every request. The
|
||||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/14), and the rest of the design
|
country lists and the image an app builds on come with the rest of milestone 2,
|
||||||
after that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey
|
and the rest of the design after that, in the order of the build order in
|
||||||
of existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
|
[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
|
||||||
|
[`EVALUATION.md`](EVALUATION.md).
|
||||||
|
|
||||||
## Getting started
|
## Getting started
|
||||||
|
|
||||||
@@ -38,7 +39,7 @@ It then listens on port 8080 and passes every request to the app at
|
|||||||
app on `127.0.0.1:8081`. On `SIGTERM` or `SIGINT` it stops taking requests and
|
app on `127.0.0.1:8081`. On `SIGTERM` or `SIGINT` it stops taking requests and
|
||||||
gives those in progress five seconds to finish.
|
gives those in progress five seconds to finish.
|
||||||
|
|
||||||
## What milestone 1 does
|
## What it does so far
|
||||||
|
|
||||||
- Passes each request to the app and the app's answer back unchanged: method,
|
- Passes each request to the app and the app's answer back unchanged: method,
|
||||||
path, query, headers, body and status. Bodies stream through in both
|
path, query, headers, body and status. Bodies stream through in both
|
||||||
@@ -61,6 +62,16 @@ gives those in progress five seconds to finish.
|
|||||||
`smallwebwaf` was waiting for the client to send more, and `504` if it was
|
`smallwebwaf` was waiting for the client to send more, and `504` if it was
|
||||||
waiting for the app to take what it had. Once the response has started, a
|
waiting for the app to take what it had. Once the response has started, a
|
||||||
limit can only cut the connection.
|
limit can only cut the connection.
|
||||||
|
- Counts each client's requests over a minute, an hour and a day. A request that
|
||||||
|
takes the client over one of the rate limits below is refused with `429`
|
||||||
|
before anything reaches the app, and so is each request after it until the
|
||||||
|
client is back under every limit. A client is one IPv4 address, or one IPv6
|
||||||
|
/64, since one abuser usually holds a whole /64. Refused requests count too,
|
||||||
|
so a client that keeps sending too fast stays refused until it slows down.
|
||||||
|
Each window is counted in two fixed buckets, the earlier one weighted by how
|
||||||
|
much of it the window still covers. At most 20,000 clients are kept, the least
|
||||||
|
recently seen dropped first, and only in memory: a restart starts every client
|
||||||
|
afresh.
|
||||||
- Writes a line in the request log for each request (see "Request log" below).
|
- Writes a line in the request log for each request (see "Request log" below).
|
||||||
|
|
||||||
## Settings
|
## Settings
|
||||||
@@ -86,18 +97,25 @@ it, and the effective settings are logged at start.
|
|||||||
to send its whole answer, from the end of the request to the last byte.
|
to send its whole answer, from the end of the request to the last byte.
|
||||||
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
|
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
|
||||||
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
|
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
|
||||||
|
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR`
|
||||||
|
(default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`): the most
|
||||||
|
requests a client may make in a minute, an hour and a day. The defaults are
|
||||||
|
several times what one busy person produces, since a browser loading a heavy
|
||||||
|
page makes a few hundred requests and several people often share one address.
|
||||||
|
|
||||||
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
|
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
|
||||||
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
|
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
|
||||||
bytes). Netblocks are in CIDR form, and a bare address stands for itself alone.
|
bytes). Rate limits are whole numbers of requests. Netblocks are in CIDR form,
|
||||||
`off` switches a timeout or a size limit off.
|
and a bare address stands for itself alone. `off` switches a timeout, a size
|
||||||
|
limit or a rate limit off.
|
||||||
|
|
||||||
Two limits are fixed rather than settings: the request line and headers may take
|
Four limits are fixed rather than settings. The request line and headers may
|
||||||
up to 32 KiB, above which the answer is `431` and nothing reaches the app, and a
|
take up to 32 KiB, above which the answer is `431` and nothing reaches the app.
|
||||||
kept-open connection that sends nothing for 120 seconds is closed. That is
|
A kept-open connection that sends nothing for 120 seconds is closed. That is
|
||||||
longer than the 90 seconds after which traefik closes a connection it is not
|
longer than the 90 seconds after which traefik closes a connection it is not
|
||||||
using, so traefik never sends a request on a connection `smallwebwaf` is
|
using, so traefik never sends a request on a connection `smallwebwaf` is
|
||||||
closing.
|
closing. At most 20,000 clients are kept for the rate limits, and an IPv6 client
|
||||||
|
is counted by its /64.
|
||||||
|
|
||||||
## Request log
|
## Request log
|
||||||
|
|
||||||
@@ -113,10 +131,13 @@ refused ones included:
|
|||||||
- `status` is what the client was sent, `0` if nothing was; `upstream_status` is
|
- `status` is what the client was sent, `0` if nothing was; `upstream_status` is
|
||||||
what the app answered, and is left out when the app did not answer.
|
what the app answered, and is left out when the app did not answer.
|
||||||
- `request_bytes` and `response_bytes` count body bytes.
|
- `request_bytes` and `response_bytes` count body bytes.
|
||||||
- `action` is `forward` for a request passed to the app, `too_large` for a
|
- `action` is `forward` for a request passed to the app, `rate_limited` for one
|
||||||
request or response over its size limit, `timed_out` for one that ran out of
|
refused for a rate limit, `too_large` for a request or response over its size
|
||||||
time, and `upstream_error` when the app could not be reached or its answer
|
limit, `timed_out` for one that ran out of time, and `upstream_error` when the
|
||||||
broke off.
|
app could not be reached or its answer broke off.
|
||||||
|
- `limit_hit` is there for a request refused for a rate limit, and names the
|
||||||
|
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
|
||||||
|
went over several.
|
||||||
- `aborted` is there, and true, when the client went away early.
|
- `aborted` is there, and true, when the client went away early.
|
||||||
- `duration_total` and `duration_upstream_total` are in milliseconds.
|
- `duration_total` and `duration_upstream_total` are in milliseconds.
|
||||||
|
|
||||||
@@ -391,12 +412,17 @@ refusal comes with `SWWAF_ALLOW_NETS` in milestone 3 or later.
|
|||||||
- `internal/proxy`: what happens to each request: it works out the client, runs
|
- `internal/proxy`: what happens to each request: it works out the client, runs
|
||||||
the checks, passes the request to the app and the answer back with the
|
the checks, passes the request to the app and the answer back with the
|
||||||
standard library's `httputil.ReverseProxy` within the timeouts and size
|
standard library's `httputil.ReverseProxy` within the timeouts and size
|
||||||
limits, and writes the request's log line. Its `check` method is where
|
limits, and writes the request's log line. Its `check` method is where a
|
||||||
milestone 2's rate limits and country lists refuse a request.
|
request is refused before anything reaches the app: for a rate limit, for an
|
||||||
|
announced body over the size limit, and, with the rest of milestone 2, for the
|
||||||
|
country lists.
|
||||||
|
- `internal/ratelimit`: counts each client's requests and tells when one takes
|
||||||
|
it over a rate limit.
|
||||||
- `internal/requestlog`: the lines on stdout: the request log line and the
|
- `internal/requestlog`: the lines on stdout: the request log line and the
|
||||||
process's own messages.
|
process's own messages.
|
||||||
|
|
||||||
Only the Go standard library is used.
|
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
|
||||||
|
table of clients to 20,000, dropping the least recently seen.
|
||||||
|
|
||||||
## Entrypoints
|
## Entrypoints
|
||||||
|
|
||||||
@@ -430,8 +456,8 @@ so that they run in minimal containers.
|
|||||||
|
|
||||||
## TODO
|
## TODO
|
||||||
|
|
||||||
- Milestone 2: rate limits per client, the country lists and the image an app
|
- Milestone 2: the country lists and the image an app builds on
|
||||||
builds on (https://git.eeqj.de/sneak/smallwebwaf/issues/14).
|
(https://git.eeqj.de/sneak/smallwebwaf/issues/14); its rate limits are built.
|
||||||
- The rest of the design, in the order of the build order in
|
- The rest of the design, in the order of the build order in
|
||||||
[`SPEC.md`](SPEC.md).
|
[`SPEC.md`](SPEC.md).
|
||||||
|
|
||||||
|
|||||||
@@ -1,3 +1,5 @@
|
|||||||
module sneak.berlin/go/smallwebwaf
|
module sneak.berlin/go/smallwebwaf
|
||||||
|
|
||||||
go 1.26.0
|
go 1.26.0
|
||||||
|
|
||||||
|
require github.com/hashicorp/golang-lru/v2 v2.0.7
|
||||||
|
|||||||
@@ -0,0 +1,2 @@
|
|||||||
|
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||||
|
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||||
|
|||||||
@@ -16,7 +16,8 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Config is smallwebwaf's settings. A timeout or size of zero is off.
|
// Config is smallwebwaf's settings. A timeout, size or rate limit of zero
|
||||||
|
// is off.
|
||||||
type Config struct {
|
type Config struct {
|
||||||
// ListenAddr is where smallwebwaf listens (SWWAF_LISTEN_ADDR).
|
// ListenAddr is where smallwebwaf listens (SWWAF_LISTEN_ADDR).
|
||||||
ListenAddr string
|
ListenAddr string
|
||||||
@@ -43,13 +44,21 @@ type Config struct {
|
|||||||
// ResponseMaxBytes is the largest response body
|
// ResponseMaxBytes is the largest response body
|
||||||
// (SWWAF_RESPONSE_MAX_BYTES).
|
// (SWWAF_RESPONSE_MAX_BYTES).
|
||||||
ResponseMaxBytes int64
|
ResponseMaxBytes int64
|
||||||
|
// RateLimitPerMinute, RateLimitPerHour and RateLimitPerDay are the
|
||||||
|
// most requests a client may make in a minute, an hour and a day
|
||||||
|
// (SWWAF_RATE_LIMIT_PER_MINUTE, SWWAF_RATE_LIMIT_PER_HOUR and
|
||||||
|
// SWWAF_RATE_LIMIT_PER_DAY).
|
||||||
|
RateLimitPerMinute int64
|
||||||
|
RateLimitPerHour int64
|
||||||
|
RateLimitPerDay int64
|
||||||
|
|
||||||
// settings are the values read, as given or by default, for the
|
// settings are the values read, as given or by default, for the
|
||||||
// log line at start.
|
// log line at start.
|
||||||
settings []slog.Attr
|
settings []slog.Attr
|
||||||
}
|
}
|
||||||
|
|
||||||
// off is the value that switches a timeout or a size limit off.
|
// off is the value that switches a timeout, a size limit or a rate limit
|
||||||
|
// off.
|
||||||
const off = "off"
|
const off = "off"
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -64,6 +73,8 @@ var (
|
|||||||
"is not a duration such as 90s, 15m or 7d, or off")
|
"is not a duration such as 90s, 15m or 7d, or off")
|
||||||
errNotSize = errors.New(
|
errNotSize = errors.New(
|
||||||
"is not a size such as 512K, 100M or 5G, or off")
|
"is not a size such as 512K, 100M or 5G, or off")
|
||||||
|
errNotCount = errors.New(
|
||||||
|
"is not a whole number of requests such as 1000, or off")
|
||||||
errNotPositive = errors.New("must be more than zero, or off")
|
errNotPositive = errors.New("must be more than zero, or off")
|
||||||
errEmptyItem = errors.New("has an empty item in its list")
|
errEmptyItem = errors.New("has an empty item in its list")
|
||||||
errNotNetblock = errors.New(
|
errNotNetblock = errors.New(
|
||||||
@@ -90,6 +101,9 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
|||||||
UpstreamResponseTimeout: env.duration("SWWAF_UPSTREAM_RESPONSE_TIMEOUT", "30m"),
|
UpstreamResponseTimeout: env.duration("SWWAF_UPSTREAM_RESPONSE_TIMEOUT", "30m"),
|
||||||
RequestMaxBytes: env.size("SWWAF_REQUEST_MAX_BYTES", "100M"),
|
RequestMaxBytes: env.size("SWWAF_REQUEST_MAX_BYTES", "100M"),
|
||||||
ResponseMaxBytes: env.size("SWWAF_RESPONSE_MAX_BYTES", "5G"),
|
ResponseMaxBytes: env.size("SWWAF_RESPONSE_MAX_BYTES", "5G"),
|
||||||
|
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "1000"),
|
||||||
|
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "10000"),
|
||||||
|
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "50000"),
|
||||||
}
|
}
|
||||||
|
|
||||||
if env.err != nil {
|
if env.err != nil {
|
||||||
@@ -179,6 +193,14 @@ func (e *environment) size(name, defaultValue string) int64 {
|
|||||||
return size
|
return size
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// count reads a setting that is a number of requests.
|
||||||
|
func (e *environment) count(name, defaultValue string) int64 {
|
||||||
|
count, err := parseCount(e.value(name, defaultValue))
|
||||||
|
e.check(name, err)
|
||||||
|
|
||||||
|
return count
|
||||||
|
}
|
||||||
|
|
||||||
// parseDuration reads a duration in Go's syntax, such as 90s or 15m, a
|
// parseDuration reads a duration in Go's syntax, such as 90s or 15m, a
|
||||||
// whole number of days such as 7d, or off.
|
// whole number of days such as 7d, or off.
|
||||||
func parseDuration(value string) (time.Duration, error) {
|
func parseDuration(value string) (time.Duration, error) {
|
||||||
@@ -249,6 +271,24 @@ func splitUnit(value string) (string, int64) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// parseCount reads a whole number of requests, or off.
|
||||||
|
func parseCount(value string) (int64, error) {
|
||||||
|
if value == off {
|
||||||
|
return 0, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
n, err := strconv.ParseInt(value, 10, 64)
|
||||||
|
if err != nil {
|
||||||
|
return 0, fmt.Errorf("%q %w", value, errNotCount)
|
||||||
|
}
|
||||||
|
|
||||||
|
if n <= 0 {
|
||||||
|
return 0, fmt.Errorf("%q %w", value, errNotPositive)
|
||||||
|
}
|
||||||
|
|
||||||
|
return n, nil
|
||||||
|
}
|
||||||
|
|
||||||
// parseList splits a comma-separated list and trims the spaces around
|
// parseList splits a comma-separated list and trims the spaces around
|
||||||
// each item. An empty value is an empty list.
|
// each item. An empty value is an empty list.
|
||||||
func parseList(value string) ([]string, error) {
|
func parseList(value string) ([]string, error) {
|
||||||
|
|||||||
@@ -25,9 +25,12 @@ const (
|
|||||||
upstreamResponseTimeout = "SWWAF_UPSTREAM_RESPONSE_TIMEOUT"
|
upstreamResponseTimeout = "SWWAF_UPSTREAM_RESPONSE_TIMEOUT"
|
||||||
requestMaxBytes = "SWWAF_REQUEST_MAX_BYTES"
|
requestMaxBytes = "SWWAF_REQUEST_MAX_BYTES"
|
||||||
responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES"
|
responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES"
|
||||||
|
rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE"
|
||||||
|
rateLimitPerHour = "SWWAF_RATE_LIMIT_PER_HOUR"
|
||||||
|
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
|
||||||
)
|
)
|
||||||
|
|
||||||
// off switches a timeout or a size limit off.
|
// off switches a timeout, a size limit or a rate limit off.
|
||||||
const off = "off"
|
const off = "off"
|
||||||
|
|
||||||
// environment is a set of environment variables, for FromEnvironment.
|
// environment is a set of environment variables, for FromEnvironment.
|
||||||
@@ -65,6 +68,9 @@ func TestDefaults(t *testing.T) {
|
|||||||
UpstreamResponseTimeout: 30 * time.Minute,
|
UpstreamResponseTimeout: 30 * time.Minute,
|
||||||
RequestMaxBytes: 100 << 20,
|
RequestMaxBytes: 100 << 20,
|
||||||
ResponseMaxBytes: 5 << 30,
|
ResponseMaxBytes: 5 << 30,
|
||||||
|
RateLimitPerMinute: 1000,
|
||||||
|
RateLimitPerHour: 10000,
|
||||||
|
RateLimitPerDay: 50000,
|
||||||
})
|
})
|
||||||
|
|
||||||
if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" {
|
if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" {
|
||||||
@@ -88,6 +94,9 @@ func TestValuesAsSet(t *testing.T) {
|
|||||||
upstreamResponseTimeout: off,
|
upstreamResponseTimeout: off,
|
||||||
requestMaxBytes: "512K",
|
requestMaxBytes: "512K",
|
||||||
responseMaxBytes: "1234",
|
responseMaxBytes: "1234",
|
||||||
|
rateLimitPerMinute: "60",
|
||||||
|
rateLimitPerHour: "600",
|
||||||
|
rateLimitPerDay: "6000",
|
||||||
})
|
})
|
||||||
|
|
||||||
wantSettings(t, cfg, config.Config{
|
wantSettings(t, cfg, config.Config{
|
||||||
@@ -98,6 +107,9 @@ func TestValuesAsSet(t *testing.T) {
|
|||||||
UpstreamResponseTimeout: 0,
|
UpstreamResponseTimeout: 0,
|
||||||
RequestMaxBytes: 512 << 10,
|
RequestMaxBytes: 512 << 10,
|
||||||
ResponseMaxBytes: 1234,
|
ResponseMaxBytes: 1234,
|
||||||
|
RateLimitPerMinute: 60,
|
||||||
|
RateLimitPerHour: 600,
|
||||||
|
RateLimitPerDay: 6000,
|
||||||
})
|
})
|
||||||
|
|
||||||
if cfg.UpstreamURL.String() != "https://app.internal:8443/" {
|
if cfg.UpstreamURL.String() != "https://app.internal:8443/" {
|
||||||
@@ -123,6 +135,22 @@ func TestSizesAndOff(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRateLimitsOff(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cfg := fromEnvironment(t, environment{
|
||||||
|
rateLimitPerMinute: off,
|
||||||
|
rateLimitPerHour: off,
|
||||||
|
rateLimitPerDay: off,
|
||||||
|
})
|
||||||
|
|
||||||
|
if cfg.RateLimitPerMinute != 0 || cfg.RateLimitPerHour != 0 ||
|
||||||
|
cfg.RateLimitPerDay != 0 {
|
||||||
|
t.Errorf("off read as %d, %d and %d",
|
||||||
|
cfg.RateLimitPerMinute, cfg.RateLimitPerHour, cfg.RateLimitPerDay)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestTrustedProxiesSetButEmptyTrustNothing(t *testing.T) {
|
func TestTrustedProxiesSetButEmptyTrustNothing(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -164,6 +192,12 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
|
|||||||
{responseMaxBytes, "0"},
|
{responseMaxBytes, "0"},
|
||||||
{responseMaxBytes, "-5"},
|
{responseMaxBytes, "-5"},
|
||||||
{responseMaxBytes, "99999999999G"},
|
{responseMaxBytes, "99999999999G"},
|
||||||
|
{rateLimitPerMinute, ""},
|
||||||
|
{rateLimitPerMinute, "1K"},
|
||||||
|
{rateLimitPerHour, "0"},
|
||||||
|
{rateLimitPerHour, "1.5"},
|
||||||
|
{rateLimitPerDay, "-1"},
|
||||||
|
{rateLimitPerDay, "lots"},
|
||||||
} {
|
} {
|
||||||
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -208,6 +242,9 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
|||||||
upstreamResponseTimeout: "30m",
|
upstreamResponseTimeout: "30m",
|
||||||
requestMaxBytes: "100M",
|
requestMaxBytes: "100M",
|
||||||
responseMaxBytes: "5G",
|
responseMaxBytes: "5G",
|
||||||
|
rateLimitPerMinute: "1000",
|
||||||
|
rateLimitPerHour: "10000",
|
||||||
|
rateLimitPerDay: "50000",
|
||||||
}
|
}
|
||||||
if !maps.Equal(line.Settings, want) {
|
if !maps.Equal(line.Settings, want) {
|
||||||
t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
|
t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
|
||||||
@@ -224,7 +261,10 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) {
|
|||||||
got.UpstreamRequestTimeout != want.UpstreamRequestTimeout ||
|
got.UpstreamRequestTimeout != want.UpstreamRequestTimeout ||
|
||||||
got.UpstreamResponseTimeout != want.UpstreamResponseTimeout ||
|
got.UpstreamResponseTimeout != want.UpstreamResponseTimeout ||
|
||||||
got.RequestMaxBytes != want.RequestMaxBytes ||
|
got.RequestMaxBytes != want.RequestMaxBytes ||
|
||||||
got.ResponseMaxBytes != want.ResponseMaxBytes {
|
got.ResponseMaxBytes != want.ResponseMaxBytes ||
|
||||||
|
got.RateLimitPerMinute != want.RateLimitPerMinute ||
|
||||||
|
got.RateLimitPerHour != want.RateLimitPerHour ||
|
||||||
|
got.RateLimitPerDay != want.RateLimitPerDay {
|
||||||
t.Errorf("settings\n%+v\nwant\n%+v", got, want)
|
t.Errorf("settings\n%+v\nwant\n%+v", got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -48,6 +48,21 @@ func clientAddress(
|
|||||||
return client
|
return client
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ipv6GroupPrefix is the length of the IPv6 netblock that is one client.
|
||||||
|
const ipv6GroupPrefix = 64
|
||||||
|
|
||||||
|
// clientGroup is the client a request is counted toward: its IPv4
|
||||||
|
// address, or the /64 its IPv6 address is in, since one abuser usually
|
||||||
|
// holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
|
||||||
|
func clientGroup(addr netip.Addr) netip.Prefix {
|
||||||
|
addr = addr.Unmap()
|
||||||
|
if addr.Is6() {
|
||||||
|
return netip.PrefixFrom(addr, ipv6GroupPrefix).Masked()
|
||||||
|
}
|
||||||
|
|
||||||
|
return netip.PrefixFrom(addr, addr.BitLen())
|
||||||
|
}
|
||||||
|
|
||||||
// isInside reports whether addr is in one of the netblocks.
|
// isInside reports whether addr is in one of the netblocks.
|
||||||
func isInside(addr netip.Addr, netblocks []netip.Prefix) bool {
|
func isInside(addr netip.Addr, netblocks []netip.Prefix) bool {
|
||||||
return slices.ContainsFunc(netblocks, func(netblock netip.Prefix) bool {
|
return slices.ContainsFunc(netblocks, func(netblock netip.Prefix) bool {
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The request line and headers a client may send, and how long a
|
// The request line and headers a client may send, and how long a
|
||||||
@@ -57,6 +58,11 @@ func New(params Params) *http.Server {
|
|||||||
processLog: params.ProcessLog,
|
processLog: params.ProcessLog,
|
||||||
errorLog: errorLog,
|
errorLog: errorLog,
|
||||||
transport: newTransport(),
|
transport: newTransport(),
|
||||||
|
limiter: ratelimit.New(ratelimit.Limits{
|
||||||
|
PerMinute: params.Config.RateLimitPerMinute,
|
||||||
|
PerHour: params.Config.RateLimitPerHour,
|
||||||
|
PerDay: params.Config.RateLimitPerDay,
|
||||||
|
}),
|
||||||
},
|
},
|
||||||
ReadHeaderTimeout: params.Config.ClientRequestTimeout,
|
ReadHeaderTimeout: params.Config.ClientRequestTimeout,
|
||||||
IdleTimeout: clientIdleTimeout,
|
IdleTimeout: clientIdleTimeout,
|
||||||
@@ -73,6 +79,7 @@ type handler struct {
|
|||||||
processLog *slog.Logger
|
processLog *slog.Logger
|
||||||
errorLog *log.Logger
|
errorLog *log.Logger
|
||||||
transport http.RoundTripper
|
transport http.RoundTripper
|
||||||
|
limiter *ratelimit.Limiter
|
||||||
}
|
}
|
||||||
|
|
||||||
// newTransport returns what carries requests to the app. It never goes
|
// newTransport returns what carries requests to the app. It never goes
|
||||||
|
|||||||
@@ -44,6 +44,7 @@ const (
|
|||||||
requestMaxBytes = "SWWAF_REQUEST_MAX_BYTES"
|
requestMaxBytes = "SWWAF_REQUEST_MAX_BYTES"
|
||||||
responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES"
|
responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES"
|
||||||
trustedProxies = "SWWAF_TRUSTED_PROXIES"
|
trustedProxies = "SWWAF_TRUSTED_PROXIES"
|
||||||
|
rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE"
|
||||||
)
|
)
|
||||||
|
|
||||||
// output collects what smallwebwaf writes on stdout.
|
// output collects what smallwebwaf writes on stdout.
|
||||||
|
|||||||
@@ -0,0 +1,67 @@
|
|||||||
|
package proxy_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRateLimitRefusesWith429BeforeTheApp(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var calls atomic.Int32
|
||||||
|
|
||||||
|
app := startApp(t, func(http.ResponseWriter, *http.Request) {
|
||||||
|
calls.Add(1)
|
||||||
|
})
|
||||||
|
addr, out := startProxy(t, app.URL, map[string]string{
|
||||||
|
trustedProxies: trustLocalhost,
|
||||||
|
rateLimitPerMinute: "1",
|
||||||
|
})
|
||||||
|
|
||||||
|
const otherClient = "203.0.113.10"
|
||||||
|
|
||||||
|
// With a limit of one request a minute, a client's second request is
|
||||||
|
// refused. A client is one IPv4 address, or one IPv6 /64; an IPv4
|
||||||
|
// address in IPv6 form is that IPv4 address.
|
||||||
|
requests := []struct {
|
||||||
|
client string // as X-Forwarded-For names it
|
||||||
|
logged string // as the log line's client_ip names it
|
||||||
|
want int
|
||||||
|
}{
|
||||||
|
{client, client, http.StatusOK},
|
||||||
|
{client, client, http.StatusTooManyRequests},
|
||||||
|
{otherClient, otherClient, http.StatusOK},
|
||||||
|
{"::ffff:" + otherClient, otherClient, http.StatusTooManyRequests},
|
||||||
|
{"2001:db8::1", "2001:db8::1", http.StatusOK},
|
||||||
|
{"2001:db8::8000:0:0:1", "2001:db8::8000:0:0:1", http.StatusTooManyRequests},
|
||||||
|
{"2001:db8:0:1::1", "2001:db8:0:1::1", http.StatusOK},
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, sent := range requests {
|
||||||
|
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
|
||||||
|
req.Header.Set(forwardedFor, sent.client)
|
||||||
|
wantStatus(t, do(t, req), sent.want)
|
||||||
|
|
||||||
|
line := out.requestLines(t, i+1)[i]
|
||||||
|
if line.ClientIP != sent.logged {
|
||||||
|
t.Errorf("log line has client_ip %q, want %q", line.ClientIP, sent.logged)
|
||||||
|
}
|
||||||
|
|
||||||
|
if sent.want == http.StatusOK {
|
||||||
|
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||||
|
} else {
|
||||||
|
wantLine(t, line, http.StatusTooManyRequests, requestlog.ActionRateLimited)
|
||||||
|
|
||||||
|
if line.LimitHit != "minute" {
|
||||||
|
t.Errorf("log line has limit_hit %q, want minute", line.LimitHit)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if calls.Load() != 4 {
|
||||||
|
t.Errorf("the app was called %d times, want 4", calls.Load())
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -38,6 +38,7 @@ type request struct {
|
|||||||
body *requestBody // nil for a request without a body
|
body *requestBody // nil for a request without a body
|
||||||
line requestlog.Line
|
line requestlog.Line
|
||||||
|
|
||||||
|
client netip.Addr
|
||||||
peer netip.Addr
|
peer netip.Addr
|
||||||
peerTrusted bool
|
peerTrusted bool
|
||||||
start time.Time
|
start time.Time
|
||||||
@@ -75,6 +76,7 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
|
|||||||
in: r,
|
in: r,
|
||||||
rc: http.NewResponseController(w),
|
rc: http.NewResponseController(w),
|
||||||
out: &responseWriter{ResponseWriter: w},
|
out: &responseWriter{ResponseWriter: w},
|
||||||
|
client: client,
|
||||||
peer: peer,
|
peer: peer,
|
||||||
peerTrusted: isInside(peer, trusted),
|
peerTrusted: isInside(peer, trusted),
|
||||||
start: start,
|
start: start,
|
||||||
@@ -100,10 +102,20 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// check is the one place where a request can be refused once its client
|
// check is the one place where a request can be refused once its client
|
||||||
// is known, before its body is read or anything reaches the app; the rate
|
// is known, before its body is read or anything reaches the app. It
|
||||||
// limits and country lists of milestone 2 go here. It returns nil to let
|
// returns nil to let the request through. The rate limits come first, so
|
||||||
// the request through.
|
// that every request is counted, one refused for its size too.
|
||||||
func (rq *request) check() *refusal {
|
func (rq *request) check() *refusal {
|
||||||
|
limitHit := rq.h.limiter.Count(clientGroup(rq.client), rq.start)
|
||||||
|
if limitHit != "" {
|
||||||
|
rq.line.LimitHit = limitHit
|
||||||
|
|
||||||
|
return &refusal{
|
||||||
|
status: http.StatusTooManyRequests,
|
||||||
|
action: requestlog.ActionRateLimited,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
maxBytes := rq.h.config.RequestMaxBytes
|
maxBytes := rq.h.config.RequestMaxBytes
|
||||||
if maxBytes > 0 && rq.in.ContentLength > maxBytes {
|
if maxBytes > 0 && rq.in.ContentLength > maxBytes {
|
||||||
return &refusal{
|
return &refusal{
|
||||||
|
|||||||
@@ -0,0 +1,132 @@
|
|||||||
|
// Package ratelimit counts each client's requests over a minute, an hour
|
||||||
|
// and a day, as the "Counting method" section of SPEC.md describes, and
|
||||||
|
// tells when a request takes a client over a rate limit. The counts are
|
||||||
|
// kept in memory only, for at most 20,000 clients.
|
||||||
|
package ratelimit
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/netip"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||||
|
)
|
||||||
|
|
||||||
|
// maxClients is how many clients are kept. Past it, the least recently
|
||||||
|
// seen client is dropped, and starts afresh if it comes back.
|
||||||
|
const maxClients = 20000
|
||||||
|
|
||||||
|
const day = 24 * time.Hour
|
||||||
|
|
||||||
|
// Limits are the most requests a client may make in a minute, an hour and
|
||||||
|
// a day. Zero is no limit.
|
||||||
|
type Limits struct {
|
||||||
|
PerMinute int64
|
||||||
|
PerHour int64
|
||||||
|
PerDay int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// Limiter counts each client's requests against the limits. It is safe
|
||||||
|
// for concurrent use.
|
||||||
|
type Limiter struct {
|
||||||
|
windows [3]window
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
// clients holds each client's buckets, one pair for each of windows,
|
||||||
|
// in the same order.
|
||||||
|
clients *simplelru.LRU[netip.Prefix, *[3]buckets]
|
||||||
|
}
|
||||||
|
|
||||||
|
// New returns a Limiter for limits, with no client counted yet.
|
||||||
|
func New(limits Limits) *Limiter {
|
||||||
|
clients, err := simplelru.NewLRU[netip.Prefix, *[3]buckets](maxClients, nil)
|
||||||
|
if err != nil {
|
||||||
|
panic(err) // NewLRU fails only for a size below one
|
||||||
|
}
|
||||||
|
|
||||||
|
return &Limiter{
|
||||||
|
windows: [3]window{
|
||||||
|
{name: "minute", length: time.Minute, limit: limits.PerMinute},
|
||||||
|
{name: "hour", length: time.Hour, limit: limits.PerHour},
|
||||||
|
{name: "day", length: day, limit: limits.PerDay},
|
||||||
|
},
|
||||||
|
clients: clients,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Count counts a request from client at now, in every window, whether or
|
||||||
|
// not it is refused. It returns the window whose limit the request takes
|
||||||
|
// the client over, "minute", "hour" or "day", the shortest if it is over
|
||||||
|
// several, or "" if it is within every limit.
|
||||||
|
func (l *Limiter) Count(client netip.Prefix, now time.Time) string {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
|
counts, seen := l.clients.Get(client)
|
||||||
|
if !seen {
|
||||||
|
counts = &[3]buckets{}
|
||||||
|
l.clients.Add(client, counts)
|
||||||
|
}
|
||||||
|
|
||||||
|
limitHit := ""
|
||||||
|
|
||||||
|
for i, w := range l.windows {
|
||||||
|
requests := counts[i].add(now, w.length)
|
||||||
|
if limitHit == "" && w.limit > 0 && requests > float64(w.limit) {
|
||||||
|
limitHit = w.name
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return limitHit
|
||||||
|
}
|
||||||
|
|
||||||
|
// window is a length of time over which requests are counted, and the
|
||||||
|
// most requests a client may make in it.
|
||||||
|
type window struct {
|
||||||
|
name string
|
||||||
|
length time.Duration
|
||||||
|
limit int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// buckets are a client's two buckets in one window: the requests in the
|
||||||
|
// bucket under way, which began at start, and in the bucket before it.
|
||||||
|
type buckets struct {
|
||||||
|
start time.Time
|
||||||
|
current int64
|
||||||
|
previous int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// add counts a request at now in a window of length, and returns the
|
||||||
|
// client's requests in the window that ends at now: those in the bucket
|
||||||
|
// under way, and those in the bucket before it weighted by how much of
|
||||||
|
// that bucket the window still covers.
|
||||||
|
//
|
||||||
|
// Concurrent requests can be counted out of order, so now can be a moment
|
||||||
|
// before the bucket under way began; such a request is counted in that
|
||||||
|
// bucket. A request dated more than a second before it means the clock
|
||||||
|
// was set back, and the buckets start afresh: otherwise the bucket before
|
||||||
|
// would keep its full weight until the clock caught up.
|
||||||
|
func (b *buckets) add(now time.Time, length time.Duration) float64 {
|
||||||
|
if now.Before(b.start.Add(-time.Second)) {
|
||||||
|
*b = buckets{}
|
||||||
|
}
|
||||||
|
|
||||||
|
start := now.Truncate(length)
|
||||||
|
if start.After(b.start) {
|
||||||
|
if start.Equal(b.start.Add(length)) {
|
||||||
|
b.previous = b.current
|
||||||
|
} else {
|
||||||
|
b.previous = 0
|
||||||
|
}
|
||||||
|
|
||||||
|
b.start = start
|
||||||
|
b.current = 0
|
||||||
|
}
|
||||||
|
|
||||||
|
b.current++
|
||||||
|
|
||||||
|
elapsed := max(now.Sub(b.start), 0)
|
||||||
|
covered := 1 - float64(elapsed)/float64(length)
|
||||||
|
|
||||||
|
return float64(b.previous)*covered + float64(b.current)
|
||||||
|
}
|
||||||
@@ -0,0 +1,200 @@
|
|||||||
|
package ratelimit_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/netip"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
|
)
|
||||||
|
|
||||||
|
// limit is the limit the tests set.
|
||||||
|
const limit = 3
|
||||||
|
|
||||||
|
// The windows, as Count names them.
|
||||||
|
const (
|
||||||
|
minute = "minute"
|
||||||
|
hour = "hour"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
window string
|
||||||
|
limits ratelimit.Limits
|
||||||
|
length time.Duration
|
||||||
|
}{
|
||||||
|
{minute, ratelimit.Limits{PerMinute: limit}, time.Minute},
|
||||||
|
{hour, ratelimit.Limits{PerHour: limit}, time.Hour},
|
||||||
|
{"day", ratelimit.Limits{PerDay: limit}, 24 * time.Hour},
|
||||||
|
} {
|
||||||
|
t.Run(tc.window, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
limiter := ratelimit.New(tc.limits)
|
||||||
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
start := midnight()
|
||||||
|
quarter := tc.length / 4
|
||||||
|
|
||||||
|
for range limit {
|
||||||
|
wantCount(t, limiter, client, start, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
wantCount(t, limiter, client, start, tc.window)
|
||||||
|
|
||||||
|
// A quarter into the next bucket, the window still covers three
|
||||||
|
// quarters of the bucket before, with its four requests: 3 + 1
|
||||||
|
// is over the limit.
|
||||||
|
wantCount(t, limiter, client, start.Add(tc.length+quarter), tc.window)
|
||||||
|
|
||||||
|
// Three quarters into it, a quarter: 1 + 2 is within.
|
||||||
|
wantCount(t, limiter, client, start.Add(tc.length+3*quarter), "")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit})
|
||||||
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
start := midnight()
|
||||||
|
|
||||||
|
for range limit {
|
||||||
|
wantCount(t, limiter, client, start, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
wantCount(t, limiter, client, start, hour)
|
||||||
|
|
||||||
|
// No request in the whole next bucket, so a quarter into the one after
|
||||||
|
// it the window covers none of the four requests: 1 is within the
|
||||||
|
// limit. Were they counted as the bucket before, 3 + 1 would be over.
|
||||||
|
wantCount(t, limiter, client, start.Add(2*time.Hour+time.Hour/4), "")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRefusedRequestsCount(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit})
|
||||||
|
refused := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
within := netip.MustParsePrefix("203.0.113.10/32")
|
||||||
|
start := midnight()
|
||||||
|
|
||||||
|
for range limit {
|
||||||
|
wantCount(t, limiter, refused, start, "")
|
||||||
|
wantCount(t, limiter, within, start, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
for range limit {
|
||||||
|
wantCount(t, limiter, refused, start, minute)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Half a minute into the next bucket the window covers half of the
|
||||||
|
// bucket before: 3 + 1 is over the minute's limit for the client
|
||||||
|
// whose three refused requests count, and 1.5 + 1 within it for the
|
||||||
|
// other. The first is over the hour's limit too, and the shorter
|
||||||
|
// window is named.
|
||||||
|
halfway := start.Add(time.Minute + time.Minute/2)
|
||||||
|
wantCount(t, limiter, refused, halfway, minute)
|
||||||
|
wantCount(t, limiter, within, halfway, "")
|
||||||
|
|
||||||
|
// The refused requests count in the hour as well: 6 + 1 + 1 is over
|
||||||
|
// its limit, and 3 + 1 + 1 within it.
|
||||||
|
later := start.Add(10 * time.Minute)
|
||||||
|
wantCount(t, limiter, refused, later, hour)
|
||||||
|
wantCount(t, limiter, within, later, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit})
|
||||||
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
start := midnight()
|
||||||
|
|
||||||
|
for range limit {
|
||||||
|
wantCount(t, limiter, client, start, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
// A concurrent request dated a moment before the bucket under way, but
|
||||||
|
// counted after it began, is counted in it: 3 + 1 is over the limit.
|
||||||
|
wantCount(t, limiter, client, start.Add(-time.Millisecond), minute)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit})
|
||||||
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
start := midnight()
|
||||||
|
|
||||||
|
for range limit {
|
||||||
|
wantCount(t, limiter, client, start, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Half an hour into the next bucket: 3 / 2 + 1 is within the limit.
|
||||||
|
wantCount(t, limiter, client, start.Add(time.Hour+time.Hour/2), "")
|
||||||
|
|
||||||
|
// The clock is set back an hour. Counted in the bucket under way, the
|
||||||
|
// next request would find the bucket before it at full weight, 3 + 2,
|
||||||
|
// over the limit until the clock caught up. The buckets start afresh
|
||||||
|
// instead, and the client is refused only past the limit again.
|
||||||
|
setBack := start.Add(time.Hour / 2)
|
||||||
|
for range limit {
|
||||||
|
wantCount(t, limiter, client, setBack, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
wantCount(t, limiter, client, setBack, hour)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestKeepsAtMost20000ClientsDroppingTheLeastRecentlySeen(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const maxClients = 20000
|
||||||
|
|
||||||
|
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1})
|
||||||
|
now := midnight()
|
||||||
|
|
||||||
|
clients := make([]netip.Prefix, maxClients+1)
|
||||||
|
addr := netip.MustParseAddr("10.0.0.0")
|
||||||
|
|
||||||
|
for i := range clients {
|
||||||
|
clients[i] = netip.PrefixFrom(addr, addr.BitLen())
|
||||||
|
addr = addr.Next()
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, client := range clients[:maxClients] {
|
||||||
|
wantCount(t, limiter, client, now, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
// The first client is seen again: its second request is over the
|
||||||
|
// limit of one, so it is still counted.
|
||||||
|
wantCount(t, limiter, clients[0], now, minute)
|
||||||
|
|
||||||
|
// One client more drops the least recently seen, the second, which
|
||||||
|
// starts afresh, while the first is kept.
|
||||||
|
wantCount(t, limiter, clients[maxClients], now, "")
|
||||||
|
wantCount(t, limiter, clients[1], now, "")
|
||||||
|
wantCount(t, limiter, clients[0], now, minute)
|
||||||
|
}
|
||||||
|
|
||||||
|
// midnight is the start of a bucket in every window.
|
||||||
|
func midnight() time.Time {
|
||||||
|
return time.Date(2026, 10, 4, 0, 0, 0, 0, time.UTC)
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantCount counts a request from client at now, and checks the window
|
||||||
|
// whose limit it goes over, "" for none.
|
||||||
|
func wantCount(
|
||||||
|
t *testing.T, limiter *ratelimit.Limiter, client netip.Prefix, now time.Time,
|
||||||
|
want string,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
got := limiter.Count(client, now)
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("request from %s at %s is over %q, want %q",
|
||||||
|
client, now.Format(time.RFC3339), got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -23,6 +23,9 @@ const (
|
|||||||
// ActionUpstreamError is a request the app could not be reached
|
// ActionUpstreamError is a request the app could not be reached
|
||||||
// for, or whose answer could not be passed on.
|
// for, or whose answer could not be passed on.
|
||||||
ActionUpstreamError = "upstream_error"
|
ActionUpstreamError = "upstream_error"
|
||||||
|
// ActionRateLimited is a request refused because it took its client
|
||||||
|
// over a rate limit, or came while the client was over one.
|
||||||
|
ActionRateLimited = "rate_limited"
|
||||||
)
|
)
|
||||||
|
|
||||||
// timeLayout is RFC 3339 with milliseconds.
|
// timeLayout is RFC 3339 with milliseconds.
|
||||||
@@ -49,6 +52,9 @@ type Line struct {
|
|||||||
Referer string `json:"referer"`
|
Referer string `json:"referer"`
|
||||||
UserAgent string `json:"user_agent"`
|
UserAgent string `json:"user_agent"`
|
||||||
Action string `json:"action"`
|
Action string `json:"action"`
|
||||||
|
// LimitHit is the window whose rate limit the request went over:
|
||||||
|
// minute, hour or day.
|
||||||
|
LimitHit string `json:"limit_hit,omitempty"`
|
||||||
// Aborted is true when the client went away early.
|
// Aborted is true when the client went away early.
|
||||||
Aborted bool `json:"aborted,omitempty"`
|
Aborted bool `json:"aborted,omitempty"`
|
||||||
// DurationTotal and DurationUpstreamTotal are in milliseconds.
|
// DurationTotal and DurationUpstreamTotal are in milliseconds.
|
||||||
|
|||||||
@@ -49,7 +49,9 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
unset := []string{"upstream_status", "aborted", "duration_upstream_total"}
|
unset := []string{
|
||||||
|
"upstream_status", "limit_hit", "aborted", "duration_upstream_total",
|
||||||
|
}
|
||||||
for _, name := range unset {
|
for _, name := range unset {
|
||||||
_, present := fields[name]
|
_, present := fields[name]
|
||||||
if present {
|
if present {
|
||||||
|
|||||||
@@ -185,6 +185,9 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL string) {
|
|||||||
"SWWAF_UPSTREAM_RESPONSE_TIMEOUT": "30m",
|
"SWWAF_UPSTREAM_RESPONSE_TIMEOUT": "30m",
|
||||||
"SWWAF_REQUEST_MAX_BYTES": "100M",
|
"SWWAF_REQUEST_MAX_BYTES": "100M",
|
||||||
"SWWAF_RESPONSE_MAX_BYTES": "5G",
|
"SWWAF_RESPONSE_MAX_BYTES": "5G",
|
||||||
|
"SWWAF_RATE_LIMIT_PER_MINUTE": "1000",
|
||||||
|
"SWWAF_RATE_LIMIT_PER_HOUR": "10000",
|
||||||
|
"SWWAF_RATE_LIMIT_PER_DAY": "50000",
|
||||||
}
|
}
|
||||||
|
|
||||||
for name, value := range want {
|
for name, value := range want {
|
||||||
|
|||||||
Reference in New Issue
Block a user