From a9dc59546ba782593b74de5da315ad314411133b Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Tue, 27 Oct 2026 19:28:09 +0100 Subject: [PATCH] New defaults for the request and byte limits (closes #161) The request limits and the hour and day byte limits now sit about twice above what 40 people busy with heavy web apps make together on one address: - requests: 6000 a minute, 100,000 an hour, 1,000,000 a day; - bytes: 20 GiB an hour, 200 GiB a day. The largest response drops from 5 GiB to 1 GiB. The minute byte limit of 4 GiB is sized to hold two of the largest responses with their requests. SPEC.md gives the new reasoning; README.md and the reference documents give the new figures. Model: opus-5-5 --- README.md | 8 +++---- SPEC.md | 30 ++++++++++++++++-------- docs/alerts.md | 10 ++++---- docs/gitea.md | 19 ++++++++------- docs/settings.md | 12 +++++----- internal/config/config.go | 12 +++++----- internal/config/config_test.go | 24 +++++++++---------- internal/smallwebwaf/smallwebwaf_test.go | 8 +++---- 8 files changed, 68 insertions(+), 55 deletions(-) diff --git a/README.md b/README.md index 4aecf31..712ad5a 100644 --- a/README.md +++ b/README.md @@ -17,10 +17,10 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below. ## What it protects against - Scrapers that fetch every page of a site, all day. Rate limiting gives each - client 1000 requests a minute, 10,000 an hour and 50,000 a day. Byte limits - give it 10 GiB a minute, 20 GiB an hour and 50 GiB a day. A client over a - limit is refused and banned. The first ban lasts an hour. Each repeat within a - day makes the next ban three times as long. A ban that would last more than + client 6000 requests a minute, 100,000 an hour and 1,000,000 a day. Byte + limits give it 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over + a limit is refused and banned. The first ban lasts an hour. Each repeat within + a day makes the next ban three times as long. A ban that would last more than seven days is permanent. - Scanners that probe for secrets, such as `/.env` or `/.git/config`. The default rule file bans such a client for seven days at its first probe. Its diff --git a/SPEC.md b/SPEC.md index 90b6c95..0b3d7b6 100644 --- a/SPEC.md +++ b/SPEC.md @@ -368,16 +368,23 @@ Conventions: - `SWWAF_ALLOW_NETS` skips the checks step 3 names, for monitoring or the owner's own networks. `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and byte limits only; the error burst, attack detection and bans still apply. -- Many clients share one address (mobile carriers, offices, Tor), and so share - its limits and its bans. `SWWAF_ALLOW_NETS` and `SWWAF_RATE_LIMIT_EXEMPT_NETS` - take known shared addresses out. +- Many people can share one address, as in an office, on a mobile carrier or + through Tor. They are one client, and share its limits and its bans. The + default limits are sized for about 40 busy people on one address. A known + shared address busier than that belongs in `SWWAF_ALLOW_NETS` or + `SWWAF_RATE_LIMIT_EXEMPT_NETS`. ### Request rate limits, per client -Breaking one bans the client, so the defaults sit several times above what one -busy person produces. A browser loading a heavy page makes a few hundred -requests, and a git clone a handful. A machine that talks to the app all day can -still pass the day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. +Breaking one bans the client, so the defaults sit about twice above what 40 +people busy with heavy web apps make together on one address. Each such person +makes about 75 requests in a busy minute, and 1,250 an hour for ten hours a day. + +A client slower than those 40 people cannot be told from them by counting. Only +the other checks can catch an abuser that slow. They include the trap paths, the +rule files, the Core Rule Set and the blocklists. An abuser that trips none of +them is not caught. A machine that talks to the app all day can still pass the +day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. `SWWAF_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as static assets or health checks; their bytes are not counted either. A prefix is @@ -390,9 +397,12 @@ matched the same way. ### Byte limits, per client -A response's bytes are counted when it ends (see step 15). So every default sits -above the largest request and answer together, and no single download breaks a -limit. +Each of the same 40 people moves about 256 MiB an hour, for ten hours a day. The +hour and day limits sit about twice above what they move together. A response's +bytes are counted when it ends (see step 15). So the minute limit holds two of +the largest responses, with their requests, ending in the same minute. No single +download then breaks a limit. An operator who raises `SWWAF_RESPONSE_MAX_BYTES` +raises the byte limits with it. ### Size and time limits diff --git a/docs/alerts.md b/docs/alerts.md index cbc4fe2..916912d 100644 --- a/docs/alerts.md +++ b/docs/alerts.md @@ -42,7 +42,7 @@ on one line: "asn": "", "as_name": "", "country": "", - "reason": "requests per minute over the limit of 1000", + "reason": "requests per minute over the limit of 6000", "detail": { "ban_expires": "2026-10-06T13:00:00.123Z", "cause": "limit", @@ -51,9 +51,9 @@ on one line: "as_name": "", "country": "", "kind": "requests", - "limit": 1000, + "limit": 6000, "window": "minute", - "count": 1001, + "count": 6001, "request": { "time": "2026-10-06T12:00:00.123456789Z", "method": "GET", @@ -62,7 +62,7 @@ on one line: "status": 403, "user_agent": "scraper/1.0" }, - "requests": 5210, + "requests": 8210, "refused": 0, "earlier_bans": { "limit": 0, @@ -89,7 +89,7 @@ on one line: - `asn`, `as_name` and `country`: the client's, as known when the alert was raised. They are empty when GeoJS had not answered yet. - `reason`: a short sentence, such as - `requests per minute over the limit of 1000` or `listed by a blocklist`. + `requests per minute over the limit of 6000` or `listed by a blocklist`. - `detail`: what is particular to the event: - for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed under `bans.json` in [`state-files.md`](state-files.md)); diff --git a/docs/gitea.md b/docs/gitea.md index 312d778..df40899 100644 --- a/docs/gitea.md +++ b/docs/gitea.md @@ -11,8 +11,10 @@ checked again when gitea or the Core Rule Set changes version. - SSH access to gitea does not pass through `smallwebwaf` and is not protected by it. -- A clone is a response. It fits the defaults of 30 minutes and 5 GiB for all - but the largest repositories and slowest links. +- A clone is a response. It fits the defaults of 30 minutes and 1 GiB unless the + repository is large or the link slow. A repository whose clone is larger than + 1 GiB needs `SWWAF_RESPONSE_MAX_BYTES` raised, and the byte limits with it + ("Byte limits, per client" in `SPEC.md`). - A push is a request, and so is every other upload: - a git push; - an LFS object; @@ -30,12 +32,13 @@ checked again when gitea or the Core Rule Set changes version. request limits do most of the work there. - An Actions runner that reaches gitea through `smallwebwaf` sends requests all day. One older than version 0.4 (April 2026) asks for work every 2 seconds, - 43,200 requests a day, and reports a running job's log and state every second. - So it passes the day limit of 50,000 after about an hour and a quarter of - jobs. It is then banned, the job it is running fails, and each repeat within a - day makes the ban longer. Newer runners ask less often, but a busy one, or - several on one address, can still pass it. Put the runners' addresses in - `SWWAF_RATE_LIMIT_EXEMPT_NETS`. + 43,200 requests a day. While it runs a job, it reports the job's log and state + every second instead, two requests a second. So one runner busy all day sends + 172,800 requests, under the request limits. Six such runners on one address, + each busy all day, pass the day limit of 1,000,000 within two days; five stay + under it. Their address is then banned, their running jobs fail, and each + repeat within a day makes the ban longer. Newer runners ask for work less + often. Put the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. ## What the Core Rule Set lets through diff --git a/docs/settings.md b/docs/settings.md index 3ed2db3..ad41aee 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -99,16 +99,16 @@ that take `off`, the `_FILE` form, and what each group does and why. ## Request rate limits, per client -- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR` - (default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`): the most - requests a client may make in each window. A count or `off`. +- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `6000`), `SWWAF_RATE_LIMIT_PER_HOUR` + (default `100000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `1000000`): the + most requests a client may make in each window. A count or `off`. - `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. ## Byte limits, per client -- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`), `SWWAF_BYTES_LIMIT_PER_HOUR` - (default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `50G`): the most +- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `4G`), `SWWAF_BYTES_LIMIT_PER_HOUR` + (default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `200G`): the most bytes counted for a client in each window. A size or `off`. - `SWWAF_BYTES_COUNT` (default `both`): which body bytes the byte limits count: `response`, `request` or `both`. @@ -131,7 +131,7 @@ that take `off`, the `_FILE` form, and what each group does and why. to send its whole answer. A duration or `off`. - `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. A size or `off`. -- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. A size +- `SWWAF_RESPONSE_MAX_BYTES` (default `1G`): the largest response body. A size or `off`. ## Lookup of AS number and country diff --git a/internal/config/config.go b/internal/config/config.go index c86de80..0cc5def 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -502,17 +502,17 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) { UpstreamRequestTimeout: env.duration("SWWAF_UPSTREAM_REQUEST_TIMEOUT", "60s"), UpstreamResponseTimeout: env.duration("SWWAF_UPSTREAM_RESPONSE_TIMEOUT", "30m"), RequestMaxBytes: env.size("SWWAF_REQUEST_MAX_BYTES", "100M"), - ResponseMaxBytes: env.size("SWWAF_RESPONSE_MAX_BYTES", "5G"), + ResponseMaxBytes: env.size("SWWAF_RESPONSE_MAX_BYTES", "1G"), AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""), RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""), DenyNets: env.netblocks("SWWAF_DENY_NETS", ""), - RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "1000"), - RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "10000"), - RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "50000"), + RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "6000"), + RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "100000"), + RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "1000000"), RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""), - BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "10G"), + BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "4G"), BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"), - BytesLimitPerDay: env.size("SWWAF_BYTES_LIMIT_PER_DAY", "50G"), + BytesLimitPerDay: env.size("SWWAF_BYTES_LIMIT_PER_DAY", "200G"), BytesCount: env.bytesCount("SWWAF_BYTES_COUNT", "both"), LookupSource: env.lookupSource("SWWAF_LOOKUP_SOURCE", "geojs"), LookupDBPath: env.value("SWWAF_LOOKUP_DB_PATH", ""), diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 9234df4..ff61910 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -261,10 +261,10 @@ func TestDefaults(t *testing.T) { UpstreamRequestTimeout: time.Minute, UpstreamResponseTimeout: 30 * time.Minute, RequestMaxBytes: 100 << 20, - ResponseMaxBytes: 5 << 30, - RateLimitPerMinute: 1000, - RateLimitPerHour: 10000, - RateLimitPerDay: 50000, + ResponseMaxBytes: 1 << 30, + RateLimitPerMinute: 6000, + RateLimitPerHour: 100000, + RateLimitPerDay: 1000000, BanResponse: 403, LimitBanDuration: time.Hour, LimitBanRepeatWindow: 24 * time.Hour, @@ -284,8 +284,8 @@ func TestDefaults(t *testing.T) { LookupSource: defaultLookupSource, LookupTimeout: time.Second, }) wantByteLimitSettings(t, cfg, config.Config{ - BytesLimitPerMinute: 10 << 30, BytesLimitPerHour: 20 << 30, - BytesLimitPerDay: 50 << 30, BytesCount: "both", + BytesLimitPerMinute: 4 << 30, BytesLimitPerHour: 20 << 30, + BytesLimitPerDay: 200 << 30, BytesCount: "both", }) if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" { @@ -2371,17 +2371,17 @@ func TestLogsEachSettingWithItsValue(t *testing.T) { upstreamRequestTimeout: "60s", upstreamResponseTimeout: "30m", requestMaxBytes: "100M", - responseMaxBytes: "5G", + responseMaxBytes: "1G", allowNets: "", rateLimitExemptNets: "", denyNets: "", - rateLimitPerMinute: "1000", - rateLimitPerHour: "10000", - rateLimitPerDay: "50000", + rateLimitPerMinute: "6000", + rateLimitPerHour: "100000", + rateLimitPerDay: "1000000", rateLimitExemptPaths: "", - bytesLimitPerMinute: "10G", + bytesLimitPerMinute: "4G", bytesLimitPerHour: "20G", - bytesLimitPerDay: "50G", + bytesLimitPerDay: "200G", bytesCount: "both", lookupSource: defaultLookupSource, lookupDBPath: "", diff --git a/internal/smallwebwaf/smallwebwaf_test.go b/internal/smallwebwaf/smallwebwaf_test.go index 3fdf2a3..0aee935 100644 --- a/internal/smallwebwaf/smallwebwaf_test.go +++ b/internal/smallwebwaf/smallwebwaf_test.go @@ -1240,13 +1240,13 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) { "SWWAF_UPSTREAM_REQUEST_TIMEOUT": "60s", "SWWAF_UPSTREAM_RESPONSE_TIMEOUT": "30m", "SWWAF_REQUEST_MAX_BYTES": "100M", - "SWWAF_RESPONSE_MAX_BYTES": "5G", + "SWWAF_RESPONSE_MAX_BYTES": "1G", "SWWAF_ALLOW_NETS": "", rateLimitExemptNets: "", "SWWAF_DENY_NETS": "", - "SWWAF_RATE_LIMIT_PER_MINUTE": "1000", - "SWWAF_RATE_LIMIT_PER_HOUR": "10000", - rateLimitPerDay: "50000", + "SWWAF_RATE_LIMIT_PER_MINUTE": "6000", + "SWWAF_RATE_LIMIT_PER_HOUR": "100000", + rateLimitPerDay: "1000000", "SWWAF_RATE_LIMIT_EXEMPT_PATHS": "", "SWWAF_DENIED_COUNTRIES": "", "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "",