From 8bdffbd4188185eb75648475d465de62db2bc115 Mon Sep 17 00:00:00 2001 From: sneak Date: Wed, 4 Nov 2026 19:16:45 +0000 Subject: [PATCH] Default request limits per minute, hour and day, sized for API use (closes #176) The default request limits now suit programmatic use of an API, such as an automation fleet on another host, and still catch abusive bots: - per minute: 500, room for a fleet's busiest minute and a few first visits with an empty cache, and passed within seconds by a client firing as fast as it can; - per hour: 20000, passed by a client that keeps up 6 a second; - per day: 300000, passed by a scraper that keeps up 4 a second. The byte limits and ban lengths are unchanged. SPEC.md, README.md and docs/settings.md give the reasons; docs/gitea.md redoes the Actions runner sums. Judgement call: the docs/alerts.md ban example shows the minute limit. Model: opus-5-5 --- README.md | 19 ++++++--- SPEC.md | 50 ++++++++++++++++-------- docs/alerts.md | 18 ++++----- docs/gitea.md | 12 +++--- docs/settings.md | 18 +++++++-- internal/config/config.go | 6 +-- internal/config/config_test.go | 12 +++--- internal/smallwebwaf/smallwebwaf_test.go | 6 +-- 8 files changed, 90 insertions(+), 51 deletions(-) diff --git a/README.md b/README.md index 712ad5a..3caa55a 100644 --- a/README.md +++ b/README.md @@ -17,11 +17,20 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below. ## What it protects against - Scrapers that fetch every page of a site, all day. Rate limiting gives each - client 6000 requests a minute, 100,000 an hour and 1,000,000 a day. Byte - limits give it 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over - a limit is refused and banned. The first ban lasts an hour. Each repeat within - a day makes the next ban three times as long. A ban that would last more than - seven days is permanent. + client 500 requests a minute, 20,000 an hour and 300,000 a day. That leaves + room for programmatic use of an API, such as an automation fleet on another + host. A fleet of about 20 agents makes about 120 requests a minute at its + busiest. A gitea page opened with an empty cache loads about 40 files, so a + few such first visits fit in the minute limit too. The minute limit catches, + within seconds, a client firing requests as fast as it can. The hour limit + catches a client that keeps up more than about 5.5 a second for an hour. The + day limit catches a client that keeps up 4 a second around the clock. Counting + cannot catch a client that stays under all three limits, such as one that + keeps up 3 a second; only the protections below can. Byte limits give each + client 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over a limit + is refused and banned. The first ban lasts an hour. Each repeat within a day + makes the next ban three times as long. A ban that would last more than seven + days is permanent. - Scanners that probe for secrets, such as `/.env` or `/.git/config`. The default rule file bans such a client for seven days at its first probe. Its next request during the ban makes the ban permanent. The default rule file diff --git a/SPEC.md b/SPEC.md index 0b3d7b6..127a35f 100644 --- a/SPEC.md +++ b/SPEC.md @@ -369,22 +369,38 @@ Conventions: owner's own networks. `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and byte limits only; the error burst, attack detection and bans still apply. - Many people can share one address, as in an office, on a mobile carrier or - through Tor. They are one client, and share its limits and its bans. The - default limits are sized for about 40 busy people on one address. A known - shared address busier than that belongs in `SWWAF_ALLOW_NETS` or + through Tor. They are one client, and share its limits and its bans. A known + shared address busier than the limits allow belongs in `SWWAF_ALLOW_NETS` or `SWWAF_RATE_LIMIT_EXEMPT_NETS`. ### Request rate limits, per client -Breaking one bans the client, so the defaults sit about twice above what 40 -people busy with heavy web apps make together on one address. Each such person -makes about 75 requests in a busy minute, and 1,250 an hour for ten hours a day. +Breaking one bans the client. The defaults suit programmatic use of an API, such +as an automation fleet on another host, and still stop abusive bots. A fleet of +20 agents, each making a request every 10 seconds at its busiest, makes 120 +requests a minute and 7,200 an hour. Working around the clock, it makes 172,800 +a day. It stays under all three limits. -A client slower than those 40 people cannot be told from them by counting. Only -the other checks can catch an abuser that slow. They include the trap paths, the -rule files, the Core Rule Set and the blocklists. An abuser that trips none of -them is not caught. A machine that talks to the app all day can still pass the -day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. +- Per minute: 500, about 8 requests a second. It leaves room for the fleet's 120 + a minute and a few quick first visits with an empty cache. A gitea page opened + with an empty cache loads about 40 files. A client firing requests as fast as + it can passes the limit within seconds. +- Per hour: 20,000, about 5.5 requests a second for a whole hour. A client that + keeps up 6 a second makes 21,600 in an hour, and passes it, at 360 a minute, + under the minute limit. The fleet's 7,200 an hour is about a third of it. +- Per day: 300,000, about 3.5 requests a second around the clock. A scraper that + keeps up 4 a second makes 345,600 a day, and passes it. At 240 a minute and + 14,400 an hour, it stays under the other two limits. + +Counting cannot catch a client that stays under all three limits, such as one +that keeps up 3 requests a second. Only the other checks can catch an abuser +that slow. They include the trap paths, the rule files, the Core Rule Set and +the blocklists. An abuser that trips none of them is not caught. A machine that +keeps up more than 500 requests a minute passes the minute limit. One that keeps +up more than about 5.5 a second for an hour passes the hour limit. One that +keeps up more than about 3.5 a second all day passes the day limit, as two busy +Actions runners on one address do. Such a machine's address belongs in +`SWWAF_RATE_LIMIT_EXEMPT_NETS`. `SWWAF_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as static assets or health checks; their bytes are not counted either. A prefix is @@ -397,12 +413,12 @@ matched the same way. ### Byte limits, per client -Each of the same 40 people moves about 256 MiB an hour, for ten hours a day. The -hour and day limits sit about twice above what they move together. A response's -bytes are counted when it ends (see step 15). So the minute limit holds two of -the largest responses, with their requests, ending in the same minute. No single -download then breaks a limit. An operator who raises `SWWAF_RESPONSE_MAX_BYTES` -raises the byte limits with it. +The hour and day limits sit about twice above what 40 people busy with heavy web +apps move together on one address. Each moves about 256 MiB an hour, for ten +hours a day. A response's bytes are counted when it ends (see step 15). So the +minute limit holds two of the largest responses, with their requests, ending in +the same minute. No single download then breaks a limit. An operator who raises +`SWWAF_RESPONSE_MAX_BYTES` raises the byte limits with it. ### Size and time limits diff --git a/docs/alerts.md b/docs/alerts.md index 916912d..4d2bbfd 100644 --- a/docs/alerts.md +++ b/docs/alerts.md @@ -35,34 +35,34 @@ on one line: ```json { "instance": "fsn1app1/gitea", - "time": "2026-10-06T12:00:00.123461Z", + "time": "2026-10-06T12:00:50.123461Z", "event": "ban", "client": "203.0.113.9", "netblock": "203.0.113.9/32", "asn": "", "as_name": "", "country": "", - "reason": "requests per minute over the limit of 6000", + "reason": "requests per minute over the limit of 500", "detail": { - "ban_expires": "2026-10-06T13:00:00.123Z", + "ban_expires": "2026-10-06T13:00:50.123Z", "cause": "limit", "notes": { "asn": "", "as_name": "", "country": "", "kind": "requests", - "limit": 6000, + "limit": 500, "window": "minute", - "count": 6001, + "count": 501, "request": { - "time": "2026-10-06T12:00:00.123456789Z", + "time": "2026-10-06T12:00:50.123456789Z", "method": "GET", "host": "app.example", - "path": "/owner/repo/commits/branch/main?page=812", + "path": "/owner/repo/commits/branch/main?page=501", "status": 403, "user_agent": "scraper/1.0" }, - "requests": 8210, + "requests": 501, "refused": 0, "earlier_bans": { "limit": 0, @@ -89,7 +89,7 @@ on one line: - `asn`, `as_name` and `country`: the client's, as known when the alert was raised. They are empty when GeoJS had not answered yet. - `reason`: a short sentence, such as - `requests per minute over the limit of 6000` or `listed by a blocklist`. + `requests per minute over the limit of 500` or `listed by a blocklist`. - `detail`: what is particular to the event: - for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed under `bans.json` in [`state-files.md`](state-files.md)); diff --git a/docs/gitea.md b/docs/gitea.md index df40899..1edf488 100644 --- a/docs/gitea.md +++ b/docs/gitea.md @@ -34,11 +34,13 @@ checked again when gitea or the Core Rule Set changes version. day. One older than version 0.4 (April 2026) asks for work every 2 seconds, 43,200 requests a day. While it runs a job, it reports the job's log and state every second instead, two requests a second. So one runner busy all day sends - 172,800 requests, under the request limits. Six such runners on one address, - each busy all day, pass the day limit of 1,000,000 within two days; five stay - under it. Their address is then banned, their running jobs fail, and each - repeat within a day makes the ban longer. Newer runners ask for work less - often. Put the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. + 120 requests a minute and 172,800 a day, under the request limits. Two such + runners on one address, each busy all day, make 345,600 and pass the day limit + of 300,000. Three busy at once make 21,600 an hour and pass the hour limit of + 20,000. Five busy at once make 600 a minute and pass the minute limit of 500. + Their address is then banned, their running jobs fail, and each repeat within + a day makes the ban longer. Newer runners ask for work less often. Put the + runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. ## What the Core Rule Set lets through diff --git a/docs/settings.md b/docs/settings.md index ad41aee..83d995c 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -99,9 +99,21 @@ that take `off`, the `_FILE` form, and what each group does and why. ## Request rate limits, per client -- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `6000`), `SWWAF_RATE_LIMIT_PER_HOUR` - (default `100000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `1000000`): the - most requests a client may make in each window. A count or `off`. +- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `500`, about 8 a second): the most + requests a client may make in a minute. A count or `off`. The default catches, + within seconds, a client firing requests as fast as it can. It leaves room for + a fleet of about 20 agents at its busiest, about 120 requests a minute. It + also leaves room for a few quick first visits to gitea with an empty cache, + about 40 files each. +- `SWWAF_RATE_LIMIT_PER_HOUR` (default `20000`, about 5.5 a second): the most + requests a client may make in an hour. A count or `off`. The default leaves + room for an automation fleet at work. It catches a client that keeps up more + than about 5.5 a second for an hour. +- `SWWAF_RATE_LIMIT_PER_DAY` (default `300000`, about 3.5 a second): the most + requests a client may make in a day. A count or `off`. The default catches a + scraper that keeps up 4 a second around the clock. +- Counting cannot catch a client that stays under all three limits (see "Request + rate limits, per client" in [`SPEC.md`](../SPEC.md)). - `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. diff --git a/internal/config/config.go b/internal/config/config.go index 0cc5def..5f01c82 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -506,9 +506,9 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) { AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""), RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""), DenyNets: env.netblocks("SWWAF_DENY_NETS", ""), - RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "6000"), - RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "100000"), - RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "1000000"), + RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "500"), + RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "20000"), + RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"), RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""), BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "4G"), BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"), diff --git a/internal/config/config_test.go b/internal/config/config_test.go index ff61910..032a3e0 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -262,9 +262,9 @@ func TestDefaults(t *testing.T) { UpstreamResponseTimeout: 30 * time.Minute, RequestMaxBytes: 100 << 20, ResponseMaxBytes: 1 << 30, - RateLimitPerMinute: 6000, - RateLimitPerHour: 100000, - RateLimitPerDay: 1000000, + RateLimitPerMinute: 500, + RateLimitPerHour: 20000, + RateLimitPerDay: 300000, BanResponse: 403, LimitBanDuration: time.Hour, LimitBanRepeatWindow: 24 * time.Hour, @@ -2375,9 +2375,9 @@ func TestLogsEachSettingWithItsValue(t *testing.T) { allowNets: "", rateLimitExemptNets: "", denyNets: "", - rateLimitPerMinute: "6000", - rateLimitPerHour: "100000", - rateLimitPerDay: "1000000", + rateLimitPerMinute: "500", + rateLimitPerHour: "20000", + rateLimitPerDay: "300000", rateLimitExemptPaths: "", bytesLimitPerMinute: "4G", bytesLimitPerHour: "20G", diff --git a/internal/smallwebwaf/smallwebwaf_test.go b/internal/smallwebwaf/smallwebwaf_test.go index 0aee935..1d30bf4 100644 --- a/internal/smallwebwaf/smallwebwaf_test.go +++ b/internal/smallwebwaf/smallwebwaf_test.go @@ -1244,9 +1244,9 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) { "SWWAF_ALLOW_NETS": "", rateLimitExemptNets: "", "SWWAF_DENY_NETS": "", - "SWWAF_RATE_LIMIT_PER_MINUTE": "6000", - "SWWAF_RATE_LIMIT_PER_HOUR": "100000", - rateLimitPerDay: "1000000", + "SWWAF_RATE_LIMIT_PER_MINUTE": "500", + "SWWAF_RATE_LIMIT_PER_HOUR": "20000", + rateLimitPerDay: "300000", "SWWAF_RATE_LIMIT_EXEMPT_PATHS": "", "SWWAF_DENIED_COUNTRIES": "", "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "",