From a9dc59546ba782593b74de5da315ad314411133b Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Tue, 27 Oct 2026 19:28:09 +0100 Subject: [PATCH 1/2] New defaults for the request and byte limits (closes #161) The request limits and the hour and day byte limits now sit about twice above what 40 people busy with heavy web apps make together on one address: - requests: 6000 a minute, 100,000 an hour, 1,000,000 a day; - bytes: 20 GiB an hour, 200 GiB a day. The largest response drops from 5 GiB to 1 GiB. The minute byte limit of 4 GiB is sized to hold two of the largest responses with their requests. SPEC.md gives the new reasoning; README.md and the reference documents give the new figures. Model: opus-5-5 --- README.md | 8 +++---- SPEC.md | 30 ++++++++++++++++-------- docs/alerts.md | 10 ++++---- docs/gitea.md | 19 ++++++++------- docs/settings.md | 12 +++++----- internal/config/config.go | 12 +++++----- internal/config/config_test.go | 24 +++++++++---------- internal/smallwebwaf/smallwebwaf_test.go | 8 +++---- 8 files changed, 68 insertions(+), 55 deletions(-) diff --git a/README.md b/README.md index 4aecf31..712ad5a 100644 --- a/README.md +++ b/README.md @@ -17,10 +17,10 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below. ## What it protects against - Scrapers that fetch every page of a site, all day. Rate limiting gives each - client 1000 requests a minute, 10,000 an hour and 50,000 a day. Byte limits - give it 10 GiB a minute, 20 GiB an hour and 50 GiB a day. A client over a - limit is refused and banned. The first ban lasts an hour. Each repeat within a - day makes the next ban three times as long. A ban that would last more than + client 6000 requests a minute, 100,000 an hour and 1,000,000 a day. Byte + limits give it 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over + a limit is refused and banned. The first ban lasts an hour. Each repeat within + a day makes the next ban three times as long. A ban that would last more than seven days is permanent. - Scanners that probe for secrets, such as `/.env` or `/.git/config`. The default rule file bans such a client for seven days at its first probe. Its diff --git a/SPEC.md b/SPEC.md index 90b6c95..0b3d7b6 100644 --- a/SPEC.md +++ b/SPEC.md @@ -368,16 +368,23 @@ Conventions: - `SWWAF_ALLOW_NETS` skips the checks step 3 names, for monitoring or the owner's own networks. `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and byte limits only; the error burst, attack detection and bans still apply. -- Many clients share one address (mobile carriers, offices, Tor), and so share - its limits and its bans. `SWWAF_ALLOW_NETS` and `SWWAF_RATE_LIMIT_EXEMPT_NETS` - take known shared addresses out. +- Many people can share one address, as in an office, on a mobile carrier or + through Tor. They are one client, and share its limits and its bans. The + default limits are sized for about 40 busy people on one address. A known + shared address busier than that belongs in `SWWAF_ALLOW_NETS` or + `SWWAF_RATE_LIMIT_EXEMPT_NETS`. ### Request rate limits, per client -Breaking one bans the client, so the defaults sit several times above what one -busy person produces. A browser loading a heavy page makes a few hundred -requests, and a git clone a handful. A machine that talks to the app all day can -still pass the day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. +Breaking one bans the client, so the defaults sit about twice above what 40 +people busy with heavy web apps make together on one address. Each such person +makes about 75 requests in a busy minute, and 1,250 an hour for ten hours a day. + +A client slower than those 40 people cannot be told from them by counting. Only +the other checks can catch an abuser that slow. They include the trap paths, the +rule files, the Core Rule Set and the blocklists. An abuser that trips none of +them is not caught. A machine that talks to the app all day can still pass the +day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. `SWWAF_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as static assets or health checks; their bytes are not counted either. A prefix is @@ -390,9 +397,12 @@ matched the same way. ### Byte limits, per client -A response's bytes are counted when it ends (see step 15). So every default sits -above the largest request and answer together, and no single download breaks a -limit. +Each of the same 40 people moves about 256 MiB an hour, for ten hours a day. The +hour and day limits sit about twice above what they move together. A response's +bytes are counted when it ends (see step 15). So the minute limit holds two of +the largest responses, with their requests, ending in the same minute. No single +download then breaks a limit. An operator who raises `SWWAF_RESPONSE_MAX_BYTES` +raises the byte limits with it. ### Size and time limits diff --git a/docs/alerts.md b/docs/alerts.md index cbc4fe2..916912d 100644 --- a/docs/alerts.md +++ b/docs/alerts.md @@ -42,7 +42,7 @@ on one line: "asn": "", "as_name": "", "country": "", - "reason": "requests per minute over the limit of 1000", + "reason": "requests per minute over the limit of 6000", "detail": { "ban_expires": "2026-10-06T13:00:00.123Z", "cause": "limit", @@ -51,9 +51,9 @@ on one line: "as_name": "", "country": "", "kind": "requests", - "limit": 1000, + "limit": 6000, "window": "minute", - "count": 1001, + "count": 6001, "request": { "time": "2026-10-06T12:00:00.123456789Z", "method": "GET", @@ -62,7 +62,7 @@ on one line: "status": 403, "user_agent": "scraper/1.0" }, - "requests": 5210, + "requests": 8210, "refused": 0, "earlier_bans": { "limit": 0, @@ -89,7 +89,7 @@ on one line: - `asn`, `as_name` and `country`: the client's, as known when the alert was raised. They are empty when GeoJS had not answered yet. - `reason`: a short sentence, such as - `requests per minute over the limit of 1000` or `listed by a blocklist`. + `requests per minute over the limit of 6000` or `listed by a blocklist`. - `detail`: what is particular to the event: - for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed under `bans.json` in [`state-files.md`](state-files.md)); diff --git a/docs/gitea.md b/docs/gitea.md index 312d778..df40899 100644 --- a/docs/gitea.md +++ b/docs/gitea.md @@ -11,8 +11,10 @@ checked again when gitea or the Core Rule Set changes version. - SSH access to gitea does not pass through `smallwebwaf` and is not protected by it. -- A clone is a response. It fits the defaults of 30 minutes and 5 GiB for all - but the largest repositories and slowest links. +- A clone is a response. It fits the defaults of 30 minutes and 1 GiB unless the + repository is large or the link slow. A repository whose clone is larger than + 1 GiB needs `SWWAF_RESPONSE_MAX_BYTES` raised, and the byte limits with it + ("Byte limits, per client" in `SPEC.md`). - A push is a request, and so is every other upload: - a git push; - an LFS object; @@ -30,12 +32,13 @@ checked again when gitea or the Core Rule Set changes version. request limits do most of the work there. - An Actions runner that reaches gitea through `smallwebwaf` sends requests all day. One older than version 0.4 (April 2026) asks for work every 2 seconds, - 43,200 requests a day, and reports a running job's log and state every second. - So it passes the day limit of 50,000 after about an hour and a quarter of - jobs. It is then banned, the job it is running fails, and each repeat within a - day makes the ban longer. Newer runners ask less often, but a busy one, or - several on one address, can still pass it. Put the runners' addresses in - `SWWAF_RATE_LIMIT_EXEMPT_NETS`. + 43,200 requests a day. While it runs a job, it reports the job's log and state + every second instead, two requests a second. So one runner busy all day sends + 172,800 requests, under the request limits. Six such runners on one address, + each busy all day, pass the day limit of 1,000,000 within two days; five stay + under it. Their address is then banned, their running jobs fail, and each + repeat within a day makes the ban longer. Newer runners ask for work less + often. Put the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. ## What the Core Rule Set lets through diff --git a/docs/settings.md b/docs/settings.md index 3ed2db3..ad41aee 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -99,16 +99,16 @@ that take `off`, the `_FILE` form, and what each group does and why. ## Request rate limits, per client -- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR` - (default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`): the most - requests a client may make in each window. A count or `off`. +- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `6000`), `SWWAF_RATE_LIMIT_PER_HOUR` + (default `100000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `1000000`): the + most requests a client may make in each window. A count or `off`. - `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. ## Byte limits, per client -- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`), `SWWAF_BYTES_LIMIT_PER_HOUR` - (default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `50G`): the most +- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `4G`), `SWWAF_BYTES_LIMIT_PER_HOUR` + (default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `200G`): the most bytes counted for a client in each window. A size or `off`. - `SWWAF_BYTES_COUNT` (default `both`): which body bytes the byte limits count: `response`, `request` or `both`. @@ -131,7 +131,7 @@ that take `off`, the `_FILE` form, and what each group does and why. to send its whole answer. A duration or `off`. - `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. A size or `off`. -- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. A size +- `SWWAF_RESPONSE_MAX_BYTES` (default `1G`): the largest response body. A size or `off`. ## Lookup of AS number and country diff --git a/internal/config/config.go b/internal/config/config.go index c86de80..0cc5def 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -502,17 +502,17 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) { UpstreamRequestTimeout: env.duration("SWWAF_UPSTREAM_REQUEST_TIMEOUT", "60s"), UpstreamResponseTimeout: env.duration("SWWAF_UPSTREAM_RESPONSE_TIMEOUT", "30m"), RequestMaxBytes: env.size("SWWAF_REQUEST_MAX_BYTES", "100M"), - ResponseMaxBytes: env.size("SWWAF_RESPONSE_MAX_BYTES", "5G"), + ResponseMaxBytes: env.size("SWWAF_RESPONSE_MAX_BYTES", "1G"), AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""), RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""), DenyNets: env.netblocks("SWWAF_DENY_NETS", ""), - RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "1000"), - RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "10000"), - RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "50000"), + RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "6000"), + RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "100000"), + RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "1000000"), RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""), - BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "10G"), + BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "4G"), BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"), - BytesLimitPerDay: env.size("SWWAF_BYTES_LIMIT_PER_DAY", "50G"), + BytesLimitPerDay: env.size("SWWAF_BYTES_LIMIT_PER_DAY", "200G"), BytesCount: env.bytesCount("SWWAF_BYTES_COUNT", "both"), LookupSource: env.lookupSource("SWWAF_LOOKUP_SOURCE", "geojs"), LookupDBPath: env.value("SWWAF_LOOKUP_DB_PATH", ""), diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 9234df4..ff61910 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -261,10 +261,10 @@ func TestDefaults(t *testing.T) { UpstreamRequestTimeout: time.Minute, UpstreamResponseTimeout: 30 * time.Minute, RequestMaxBytes: 100 << 20, - ResponseMaxBytes: 5 << 30, - RateLimitPerMinute: 1000, - RateLimitPerHour: 10000, - RateLimitPerDay: 50000, + ResponseMaxBytes: 1 << 30, + RateLimitPerMinute: 6000, + RateLimitPerHour: 100000, + RateLimitPerDay: 1000000, BanResponse: 403, LimitBanDuration: time.Hour, LimitBanRepeatWindow: 24 * time.Hour, @@ -284,8 +284,8 @@ func TestDefaults(t *testing.T) { LookupSource: defaultLookupSource, LookupTimeout: time.Second, }) wantByteLimitSettings(t, cfg, config.Config{ - BytesLimitPerMinute: 10 << 30, BytesLimitPerHour: 20 << 30, - BytesLimitPerDay: 50 << 30, BytesCount: "both", + BytesLimitPerMinute: 4 << 30, BytesLimitPerHour: 20 << 30, + BytesLimitPerDay: 200 << 30, BytesCount: "both", }) if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" { @@ -2371,17 +2371,17 @@ func TestLogsEachSettingWithItsValue(t *testing.T) { upstreamRequestTimeout: "60s", upstreamResponseTimeout: "30m", requestMaxBytes: "100M", - responseMaxBytes: "5G", + responseMaxBytes: "1G", allowNets: "", rateLimitExemptNets: "", denyNets: "", - rateLimitPerMinute: "1000", - rateLimitPerHour: "10000", - rateLimitPerDay: "50000", + rateLimitPerMinute: "6000", + rateLimitPerHour: "100000", + rateLimitPerDay: "1000000", rateLimitExemptPaths: "", - bytesLimitPerMinute: "10G", + bytesLimitPerMinute: "4G", bytesLimitPerHour: "20G", - bytesLimitPerDay: "50G", + bytesLimitPerDay: "200G", bytesCount: "both", lookupSource: defaultLookupSource, lookupDBPath: "", diff --git a/internal/smallwebwaf/smallwebwaf_test.go b/internal/smallwebwaf/smallwebwaf_test.go index 3fdf2a3..0aee935 100644 --- a/internal/smallwebwaf/smallwebwaf_test.go +++ b/internal/smallwebwaf/smallwebwaf_test.go @@ -1240,13 +1240,13 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) { "SWWAF_UPSTREAM_REQUEST_TIMEOUT": "60s", "SWWAF_UPSTREAM_RESPONSE_TIMEOUT": "30m", "SWWAF_REQUEST_MAX_BYTES": "100M", - "SWWAF_RESPONSE_MAX_BYTES": "5G", + "SWWAF_RESPONSE_MAX_BYTES": "1G", "SWWAF_ALLOW_NETS": "", rateLimitExemptNets: "", "SWWAF_DENY_NETS": "", - "SWWAF_RATE_LIMIT_PER_MINUTE": "1000", - "SWWAF_RATE_LIMIT_PER_HOUR": "10000", - rateLimitPerDay: "50000", + "SWWAF_RATE_LIMIT_PER_MINUTE": "6000", + "SWWAF_RATE_LIMIT_PER_HOUR": "100000", + rateLimitPerDay: "1000000", "SWWAF_RATE_LIMIT_EXEMPT_PATHS": "", "SWWAF_DENIED_COUNTRIES": "", "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "", -- 2.54.0 From fdabf4974bccc11671e49924d4c02d3bb85d0101 Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Tue, 27 Oct 2026 21:00:16 +0100 Subject: [PATCH 2/2] Health check test uses addresses nothing can connect to (closes #158) TestHealthCheck closed the app's server, and later smallwebwaf, and expected connections to the freed ports to fail. Another test could take such a port in between, so the connection succeeded and the test failed. The failing app check now uses the multicast address 224.0.0.1, to which TCP cannot connect. The failing smallwebwaf check uses port 0, on which nothing listens. Neither result depends on other processes. The test now stops smallwebwaf after its last check, so that it exits before the test's temporary directories are removed. The code under test is unchanged. Model: opus-5-5 --- internal/smallwebwaf/healthcheck_test.go | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/internal/smallwebwaf/healthcheck_test.go b/internal/smallwebwaf/healthcheck_test.go index 3089f22..a5ffd60 100644 --- a/internal/smallwebwaf/healthcheck_test.go +++ b/internal/smallwebwaf/healthcheck_test.go @@ -60,8 +60,19 @@ func TestHealthCheck(t *testing.T) { }, 0, "") } - app.Close() - wantHealthCheck(t, env, 1, "unhealthy: connect to the app: ") + // Another test can take a closed server's port at once, so the app's + // address here is a multicast address, to which TCP cannot connect. + wantHealthCheck(t, map[string]string{ + listenAddr: ":" + port, + upstreamURL: "http://224.0.0.1", + }, 1, "unhealthy: connect to the app: ") + + // Listening on port 0 takes any free port, so nothing listens on port 0 + // itself. + wantHealthCheck(t, map[string]string{listenAddr: ":0"}, 1, + "unhealthy: ask smallwebwaf: ") + wantHealthCheck(t, map[string]string{listenAddr: "8080"}, 1, + "unhealthy: invalid setting: SWWAF_LISTEN_ADDR: ") stop() @@ -70,10 +81,6 @@ func TestHealthCheck(t *testing.T) { case <-time.After(waitLimit): t.Fatal("still running after being told to stop") } - - wantHealthCheck(t, env, 1, "unhealthy: ask smallwebwaf: ") - wantHealthCheck(t, map[string]string{listenAddr: "8080"}, 1, - "unhealthy: invalid setting: SWWAF_LISTEN_ADDR: ") } func TestHealthCheckRefusesAnArgument(t *testing.T) { -- 2.54.0