Default request limits per minute, hour and day, sized for API use (closes #176) #179

Merged
clawbot merged 1 commits from issue-176-api-request-limits into next 2026-11-04 23:02:40 +01:00
8 changed files with 90 additions and 51 deletions
+14 -5
View File
@@ -17,11 +17,20 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below.
## What it protects against ## What it protects against
- Scrapers that fetch every page of a site, all day. Rate limiting gives each - Scrapers that fetch every page of a site, all day. Rate limiting gives each
client 6000 requests a minute, 100,000 an hour and 1,000,000 a day. Byte client 500 requests a minute, 20,000 an hour and 300,000 a day. That leaves
limits give it 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over room for programmatic use of an API, such as an automation fleet on another
a limit is refused and banned. The first ban lasts an hour. Each repeat within host. A fleet of about 20 agents makes about 120 requests a minute at its
a day makes the next ban three times as long. A ban that would last more than busiest. A gitea page opened with an empty cache loads about 40 files, so a
seven days is permanent. few such first visits fit in the minute limit too. The minute limit catches,
within seconds, a client firing requests as fast as it can. The hour limit
catches a client that keeps up more than about 5.5 a second for an hour. The
day limit catches a client that keeps up 4 a second around the clock. Counting
cannot catch a client that stays under all three limits, such as one that
keeps up 3 a second; only the protections below can. Byte limits give each
client 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over a limit
is refused and banned. The first ban lasts an hour. Each repeat within a day
makes the next ban three times as long. A ban that would last more than seven
days is permanent.
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The - Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
default rule file bans such a client for seven days at its first probe. Its default rule file bans such a client for seven days at its first probe. Its
next request during the ban makes the ban permanent. The default rule file next request during the ban makes the ban permanent. The default rule file
+33 -17
View File
@@ -369,22 +369,38 @@ Conventions:
owner's own networks. `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and owner's own networks. `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and
byte limits only; the error burst, attack detection and bans still apply. byte limits only; the error burst, attack detection and bans still apply.
- Many people can share one address, as in an office, on a mobile carrier or - Many people can share one address, as in an office, on a mobile carrier or
through Tor. They are one client, and share its limits and its bans. The through Tor. They are one client, and share its limits and its bans. A known
default limits are sized for about 40 busy people on one address. A known shared address busier than the limits allow belongs in `SWWAF_ALLOW_NETS` or
shared address busier than that belongs in `SWWAF_ALLOW_NETS` or
`SWWAF_RATE_LIMIT_EXEMPT_NETS`. `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
### Request rate limits, per client ### Request rate limits, per client
Breaking one bans the client, so the defaults sit about twice above what 40 Breaking one bans the client. The defaults suit programmatic use of an API, such
people busy with heavy web apps make together on one address. Each such person as an automation fleet on another host, and still stop abusive bots. A fleet of
makes about 75 requests in a busy minute, and 1,250 an hour for ten hours a day. 20 agents, each making a request every 10 seconds at its busiest, makes 120
requests a minute and 7,200 an hour. Working around the clock, it makes 172,800
a day. It stays under all three limits.
A client slower than those 40 people cannot be told from them by counting. Only - Per minute: 500, about 8 requests a second. It leaves room for the fleet's 120
the other checks can catch an abuser that slow. They include the trap paths, the a minute and a few quick first visits with an empty cache. A gitea page opened
rule files, the Core Rule Set and the blocklists. An abuser that trips none of with an empty cache loads about 40 files. A client firing requests as fast as
them is not caught. A machine that talks to the app all day can still pass the it can passes the limit within seconds.
day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. - Per hour: 20,000, about 5.5 requests a second for a whole hour. A client that
keeps up 6 a second makes 21,600 in an hour, and passes it, at 360 a minute,
under the minute limit. The fleet's 7,200 an hour is about a third of it.
- Per day: 300,000, about 3.5 requests a second around the clock. A scraper that
keeps up 4 a second makes 345,600 a day, and passes it. At 240 a minute and
14,400 an hour, it stays under the other two limits.
Counting cannot catch a client that stays under all three limits, such as one
that keeps up 3 requests a second. Only the other checks can catch an abuser
that slow. They include the trap paths, the rule files, the Core Rule Set and
the blocklists. An abuser that trips none of them is not caught. A machine that
keeps up more than 500 requests a minute passes the minute limit. One that keeps
up more than about 5.5 a second for an hour passes the hour limit. One that
keeps up more than about 3.5 a second all day passes the day limit, as two busy
Actions runners on one address do. Such a machine's address belongs in
`SWWAF_RATE_LIMIT_EXEMPT_NETS`.
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as `SWWAF_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as
static assets or health checks; their bytes are not counted either. A prefix is static assets or health checks; their bytes are not counted either. A prefix is
@@ -397,12 +413,12 @@ matched the same way.
### Byte limits, per client ### Byte limits, per client
Each of the same 40 people moves about 256 MiB an hour, for ten hours a day. The The hour and day limits sit about twice above what 40 people busy with heavy web
hour and day limits sit about twice above what they move together. A response's apps move together on one address. Each moves about 256 MiB an hour, for ten
bytes are counted when it ends (see step 15). So the minute limit holds two of hours a day. A response's bytes are counted when it ends (see step 15). So the
the largest responses, with their requests, ending in the same minute. No single minute limit holds two of the largest responses, with their requests, ending in
download then breaks a limit. An operator who raises `SWWAF_RESPONSE_MAX_BYTES` the same minute. No single download then breaks a limit. An operator who raises
raises the byte limits with it. `SWWAF_RESPONSE_MAX_BYTES` raises the byte limits with it.
### Size and time limits ### Size and time limits
+9 -9
View File
@@ -35,34 +35,34 @@ on one line:
```json ```json
{ {
"instance": "fsn1app1/gitea", "instance": "fsn1app1/gitea",
"time": "2026-10-06T12:00:00.123461Z", "time": "2026-10-06T12:00:50.123461Z",
"event": "ban", "event": "ban",
"client": "203.0.113.9", "client": "203.0.113.9",
"netblock": "203.0.113.9/32", "netblock": "203.0.113.9/32",
"asn": "", "asn": "",
"as_name": "", "as_name": "",
"country": "", "country": "",
"reason": "requests per minute over the limit of 6000", "reason": "requests per minute over the limit of 500",
"detail": { "detail": {
"ban_expires": "2026-10-06T13:00:00.123Z", "ban_expires": "2026-10-06T13:00:50.123Z",
"cause": "limit", "cause": "limit",
"notes": { "notes": {
"asn": "", "asn": "",
"as_name": "", "as_name": "",
"country": "", "country": "",
"kind": "requests", "kind": "requests",
"limit": 6000, "limit": 500,
"window": "minute", "window": "minute",
"count": 6001, "count": 501,
"request": { "request": {
"time": "2026-10-06T12:00:00.123456789Z", "time": "2026-10-06T12:00:50.123456789Z",
"method": "GET", "method": "GET",
"host": "app.example", "host": "app.example",
"path": "/owner/repo/commits/branch/main?page=812", "path": "/owner/repo/commits/branch/main?page=501",
"status": 403, "status": 403,
"user_agent": "scraper/1.0" "user_agent": "scraper/1.0"
}, },
"requests": 8210, "requests": 501,
"refused": 0, "refused": 0,
"earlier_bans": { "earlier_bans": {
"limit": 0, "limit": 0,
@@ -89,7 +89,7 @@ on one line:
- `asn`, `as_name` and `country`: the client's, as known when the alert was - `asn`, `as_name` and `country`: the client's, as known when the alert was
raised. They are empty when GeoJS had not answered yet. raised. They are empty when GeoJS had not answered yet.
- `reason`: a short sentence, such as - `reason`: a short sentence, such as
`requests per minute over the limit of 6000` or `listed by a blocklist`. `requests per minute over the limit of 500` or `listed by a blocklist`.
- `detail`: what is particular to the event: - `detail`: what is particular to the event:
- for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed - for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed
under `bans.json` in [`state-files.md`](state-files.md)); under `bans.json` in [`state-files.md`](state-files.md));
+7 -5
View File
@@ -34,11 +34,13 @@ checked again when gitea or the Core Rule Set changes version.
day. One older than version 0.4 (April 2026) asks for work every 2 seconds, day. One older than version 0.4 (April 2026) asks for work every 2 seconds,
43,200 requests a day. While it runs a job, it reports the job's log and state 43,200 requests a day. While it runs a job, it reports the job's log and state
every second instead, two requests a second. So one runner busy all day sends every second instead, two requests a second. So one runner busy all day sends
172,800 requests, under the request limits. Six such runners on one address, 120 requests a minute and 172,800 a day, under the request limits. Two such
each busy all day, pass the day limit of 1,000,000 within two days; five stay runners on one address, each busy all day, make 345,600 and pass the day limit
under it. Their address is then banned, their running jobs fail, and each of 300,000. Three busy at once make 21,600 an hour and pass the hour limit of
repeat within a day makes the ban longer. Newer runners ask for work less 20,000. Five busy at once make 600 a minute and pass the minute limit of 500.
often. Put the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. Their address is then banned, their running jobs fail, and each repeat within
a day makes the ban longer. Newer runners ask for work less often. Put the
runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
## What the Core Rule Set lets through ## What the Core Rule Set lets through
+15 -3
View File
@@ -99,9 +99,21 @@ that take `off`, the `_FILE` form, and what each group does and why.
## Request rate limits, per client ## Request rate limits, per client
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `6000`), `SWWAF_RATE_LIMIT_PER_HOUR` - `SWWAF_RATE_LIMIT_PER_MINUTE` (default `500`, about 8 a second): the most
(default `100000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `1000000`): the requests a client may make in a minute. A count or `off`. The default catches,
most requests a client may make in each window. A count or `off`. within seconds, a client firing requests as fast as it can. It leaves room for
a fleet of about 20 agents at its busiest, about 120 requests a minute. It
also leaves room for a few quick first visits to gitea with an empty cache,
about 40 files each.
- `SWWAF_RATE_LIMIT_PER_HOUR` (default `20000`, about 5.5 a second): the most
requests a client may make in an hour. A count or `off`. The default leaves
room for an automation fleet at work. It catches a client that keeps up more
than about 5.5 a second for an hour.
- `SWWAF_RATE_LIMIT_PER_DAY` (default `300000`, about 3.5 a second): the most
requests a client may make in a day. A count or `off`. The default catches a
scraper that keeps up 4 a second around the clock.
- Counting cannot catch a client that stays under all three limits (see "Request
rate limits, per client" in [`SPEC.md`](../SPEC.md)).
- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from - `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from
rate limiting and the byte limits, such as `/assets/`. rate limiting and the byte limits, such as `/assets/`.
+3 -3
View File
@@ -506,9 +506,9 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""), AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""),
RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""), RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""),
DenyNets: env.netblocks("SWWAF_DENY_NETS", ""), DenyNets: env.netblocks("SWWAF_DENY_NETS", ""),
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "6000"), RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "500"),
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "100000"), RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "20000"),
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "1000000"), RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"),
RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""), RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "4G"), BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "4G"),
BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"), BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"),
+6 -6
View File
@@ -262,9 +262,9 @@ func TestDefaults(t *testing.T) {
UpstreamResponseTimeout: 30 * time.Minute, UpstreamResponseTimeout: 30 * time.Minute,
RequestMaxBytes: 100 << 20, RequestMaxBytes: 100 << 20,
ResponseMaxBytes: 1 << 30, ResponseMaxBytes: 1 << 30,
RateLimitPerMinute: 6000, RateLimitPerMinute: 500,
RateLimitPerHour: 100000, RateLimitPerHour: 20000,
RateLimitPerDay: 1000000, RateLimitPerDay: 300000,
BanResponse: 403, BanResponse: 403,
LimitBanDuration: time.Hour, LimitBanDuration: time.Hour,
LimitBanRepeatWindow: 24 * time.Hour, LimitBanRepeatWindow: 24 * time.Hour,
@@ -2375,9 +2375,9 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
allowNets: "", allowNets: "",
rateLimitExemptNets: "", rateLimitExemptNets: "",
denyNets: "", denyNets: "",
rateLimitPerMinute: "6000", rateLimitPerMinute: "500",
rateLimitPerHour: "100000", rateLimitPerHour: "20000",
rateLimitPerDay: "1000000", rateLimitPerDay: "300000",
rateLimitExemptPaths: "", rateLimitExemptPaths: "",
bytesLimitPerMinute: "4G", bytesLimitPerMinute: "4G",
bytesLimitPerHour: "20G", bytesLimitPerHour: "20G",
+3 -3
View File
@@ -1244,9 +1244,9 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
"SWWAF_ALLOW_NETS": "", "SWWAF_ALLOW_NETS": "",
rateLimitExemptNets: "", rateLimitExemptNets: "",
"SWWAF_DENY_NETS": "", "SWWAF_DENY_NETS": "",
"SWWAF_RATE_LIMIT_PER_MINUTE": "6000", "SWWAF_RATE_LIMIT_PER_MINUTE": "500",
"SWWAF_RATE_LIMIT_PER_HOUR": "100000", "SWWAF_RATE_LIMIT_PER_HOUR": "20000",
rateLimitPerDay: "1000000", rateLimitPerDay: "300000",
"SWWAF_RATE_LIMIT_EXEMPT_PATHS": "", "SWWAF_RATE_LIMIT_EXEMPT_PATHS": "",
"SWWAF_DENIED_COUNTRIES": "", "SWWAF_DENIED_COUNTRIES": "",
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "", "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "",