Default request limits per minute, hour and day, sized for API use (closes #176) #179

Merged
clawbot merged 1 commits from issue-176-api-request-limits into next 2026-11-04 23:02:40 +01:00
8 changed files with 90 additions and 51 deletions
Showing only changes of commit 8bdffbd418 - Show all commits
+14 -5
View File
@@ -17,11 +17,20 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below.
## What it protects against
- Scrapers that fetch every page of a site, all day. Rate limiting gives each
client 6000 requests a minute, 100,000 an hour and 1,000,000 a day. Byte
limits give it 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over
a limit is refused and banned. The first ban lasts an hour. Each repeat within
a day makes the next ban three times as long. A ban that would last more than
seven days is permanent.
client 500 requests a minute, 20,000 an hour and 300,000 a day. That leaves
room for programmatic use of an API, such as an automation fleet on another
host. A fleet of about 20 agents makes about 120 requests a minute at its
busiest. A gitea page opened with an empty cache loads about 40 files, so a
few such first visits fit in the minute limit too. The minute limit catches,
within seconds, a client firing requests as fast as it can. The hour limit
catches a client that keeps up more than about 5.5 a second for an hour. The
day limit catches a client that keeps up 4 a second around the clock. Counting
cannot catch a client that stays under all three limits, such as one that
keeps up 3 a second; only the protections below can. Byte limits give each
client 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over a limit
is refused and banned. The first ban lasts an hour. Each repeat within a day
makes the next ban three times as long. A ban that would last more than seven
days is permanent.
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
default rule file bans such a client for seven days at its first probe. Its
next request during the ban makes the ban permanent. The default rule file
+33 -17
View File
@@ -369,22 +369,38 @@ Conventions:
owner's own networks. `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and
byte limits only; the error burst, attack detection and bans still apply.
- Many people can share one address, as in an office, on a mobile carrier or
through Tor. They are one client, and share its limits and its bans. The
default limits are sized for about 40 busy people on one address. A known
shared address busier than that belongs in `SWWAF_ALLOW_NETS` or
through Tor. They are one client, and share its limits and its bans. A known
shared address busier than the limits allow belongs in `SWWAF_ALLOW_NETS` or
`SWWAF_RATE_LIMIT_EXEMPT_NETS`.
### Request rate limits, per client
Breaking one bans the client, so the defaults sit about twice above what 40
people busy with heavy web apps make together on one address. Each such person
makes about 75 requests in a busy minute, and 1,250 an hour for ten hours a day.
Breaking one bans the client. The defaults suit programmatic use of an API, such
as an automation fleet on another host, and still stop abusive bots. A fleet of
20 agents, each making a request every 10 seconds at its busiest, makes 120
requests a minute and 7,200 an hour. Working around the clock, it makes 172,800
a day. It stays under all three limits.
A client slower than those 40 people cannot be told from them by counting. Only
the other checks can catch an abuser that slow. They include the trap paths, the
rule files, the Core Rule Set and the blocklists. An abuser that trips none of
them is not caught. A machine that talks to the app all day can still pass the
day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
- Per minute: 500, about 8 requests a second. It leaves room for the fleet's 120
a minute and a few quick first visits with an empty cache. A gitea page opened
with an empty cache loads about 40 files. A client firing requests as fast as
it can passes the limit within seconds.
- Per hour: 20,000, about 5.5 requests a second for a whole hour. A client that
keeps up 6 a second makes 21,600 in an hour, and passes it, at 360 a minute,
under the minute limit. The fleet's 7,200 an hour is about a third of it.
- Per day: 300,000, about 3.5 requests a second around the clock. A scraper that
keeps up 4 a second makes 345,600 a day, and passes it. At 240 a minute and
14,400 an hour, it stays under the other two limits.
Counting cannot catch a client that stays under all three limits, such as one
that keeps up 3 requests a second. Only the other checks can catch an abuser
that slow. They include the trap paths, the rule files, the Core Rule Set and
the blocklists. An abuser that trips none of them is not caught. A machine that
keeps up more than 500 requests a minute passes the minute limit. One that keeps
up more than about 5.5 a second for an hour passes the hour limit. One that
keeps up more than about 3.5 a second all day passes the day limit, as two busy
Actions runners on one address do. Such a machine's address belongs in
`SWWAF_RATE_LIMIT_EXEMPT_NETS`.
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as
static assets or health checks; their bytes are not counted either. A prefix is
@@ -397,12 +413,12 @@ matched the same way.
### Byte limits, per client
Each of the same 40 people moves about 256 MiB an hour, for ten hours a day. The
hour and day limits sit about twice above what they move together. A response's
bytes are counted when it ends (see step 15). So the minute limit holds two of
the largest responses, with their requests, ending in the same minute. No single
download then breaks a limit. An operator who raises `SWWAF_RESPONSE_MAX_BYTES`
raises the byte limits with it.
The hour and day limits sit about twice above what 40 people busy with heavy web
apps move together on one address. Each moves about 256 MiB an hour, for ten
hours a day. A response's bytes are counted when it ends (see step 15). So the
minute limit holds two of the largest responses, with their requests, ending in
the same minute. No single download then breaks a limit. An operator who raises
`SWWAF_RESPONSE_MAX_BYTES` raises the byte limits with it.
### Size and time limits
+9 -9
View File
@@ -35,34 +35,34 @@ on one line:
```json
{
"instance": "fsn1app1/gitea",
"time": "2026-10-06T12:00:00.123461Z",
"time": "2026-10-06T12:00:50.123461Z",
"event": "ban",
"client": "203.0.113.9",
"netblock": "203.0.113.9/32",
"asn": "",
"as_name": "",
"country": "",
"reason": "requests per minute over the limit of 6000",
"reason": "requests per minute over the limit of 500",
"detail": {
"ban_expires": "2026-10-06T13:00:00.123Z",
"ban_expires": "2026-10-06T13:00:50.123Z",
"cause": "limit",
"notes": {
"asn": "",
"as_name": "",
"country": "",
"kind": "requests",
"limit": 6000,
"limit": 500,
"window": "minute",
"count": 6001,
"count": 501,
"request": {
"time": "2026-10-06T12:00:00.123456789Z",
"time": "2026-10-06T12:00:50.123456789Z",
"method": "GET",
"host": "app.example",
"path": "/owner/repo/commits/branch/main?page=812",
"path": "/owner/repo/commits/branch/main?page=501",
"status": 403,
"user_agent": "scraper/1.0"
},
"requests": 8210,
"requests": 501,
"refused": 0,
"earlier_bans": {
"limit": 0,
@@ -89,7 +89,7 @@ on one line:
- `asn`, `as_name` and `country`: the client's, as known when the alert was
raised. They are empty when GeoJS had not answered yet.
- `reason`: a short sentence, such as
`requests per minute over the limit of 6000` or `listed by a blocklist`.
`requests per minute over the limit of 500` or `listed by a blocklist`.
- `detail`: what is particular to the event:
- for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed
under `bans.json` in [`state-files.md`](state-files.md));
+7 -5
View File
@@ -34,11 +34,13 @@ checked again when gitea or the Core Rule Set changes version.
day. One older than version 0.4 (April 2026) asks for work every 2 seconds,
43,200 requests a day. While it runs a job, it reports the job's log and state
every second instead, two requests a second. So one runner busy all day sends
172,800 requests, under the request limits. Six such runners on one address,
each busy all day, pass the day limit of 1,000,000 within two days; five stay
under it. Their address is then banned, their running jobs fail, and each
repeat within a day makes the ban longer. Newer runners ask for work less
often. Put the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
120 requests a minute and 172,800 a day, under the request limits. Two such
runners on one address, each busy all day, make 345,600 and pass the day limit
of 300,000. Three busy at once make 21,600 an hour and pass the hour limit of
20,000. Five busy at once make 600 a minute and pass the minute limit of 500.
Their address is then banned, their running jobs fail, and each repeat within
a day makes the ban longer. Newer runners ask for work less often. Put the
runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
## What the Core Rule Set lets through
+15 -3
View File
@@ -99,9 +99,21 @@ that take `off`, the `_FILE` form, and what each group does and why.
## Request rate limits, per client
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `6000`), `SWWAF_RATE_LIMIT_PER_HOUR`
(default `100000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `1000000`): the
most requests a client may make in each window. A count or `off`.
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `500`, about 8 a second): the most
requests a client may make in a minute. A count or `off`. The default catches,
within seconds, a client firing requests as fast as it can. It leaves room for
a fleet of about 20 agents at its busiest, about 120 requests a minute. It
also leaves room for a few quick first visits to gitea with an empty cache,
about 40 files each.
- `SWWAF_RATE_LIMIT_PER_HOUR` (default `20000`, about 5.5 a second): the most
requests a client may make in an hour. A count or `off`. The default leaves
room for an automation fleet at work. It catches a client that keeps up more
than about 5.5 a second for an hour.
- `SWWAF_RATE_LIMIT_PER_DAY` (default `300000`, about 3.5 a second): the most
requests a client may make in a day. A count or `off`. The default catches a
scraper that keeps up 4 a second around the clock.
- Counting cannot catch a client that stays under all three limits (see "Request
rate limits, per client" in [`SPEC.md`](../SPEC.md)).
- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from
rate limiting and the byte limits, such as `/assets/`.
+3 -3
View File
@@ -506,9 +506,9 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""),
RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""),
DenyNets: env.netblocks("SWWAF_DENY_NETS", ""),
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "6000"),
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "100000"),
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "1000000"),
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "500"),
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "20000"),
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"),
RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "4G"),
BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"),
+6 -6
View File
@@ -262,9 +262,9 @@ func TestDefaults(t *testing.T) {
UpstreamResponseTimeout: 30 * time.Minute,
RequestMaxBytes: 100 << 20,
ResponseMaxBytes: 1 << 30,
RateLimitPerMinute: 6000,
RateLimitPerHour: 100000,
RateLimitPerDay: 1000000,
RateLimitPerMinute: 500,
RateLimitPerHour: 20000,
RateLimitPerDay: 300000,
BanResponse: 403,
LimitBanDuration: time.Hour,
LimitBanRepeatWindow: 24 * time.Hour,
@@ -2375,9 +2375,9 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
allowNets: "",
rateLimitExemptNets: "",
denyNets: "",
rateLimitPerMinute: "6000",
rateLimitPerHour: "100000",
rateLimitPerDay: "1000000",
rateLimitPerMinute: "500",
rateLimitPerHour: "20000",
rateLimitPerDay: "300000",
rateLimitExemptPaths: "",
bytesLimitPerMinute: "4G",
bytesLimitPerHour: "20G",
+3 -3
View File
@@ -1244,9 +1244,9 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
"SWWAF_ALLOW_NETS": "",
rateLimitExemptNets: "",
"SWWAF_DENY_NETS": "",
"SWWAF_RATE_LIMIT_PER_MINUTE": "6000",
"SWWAF_RATE_LIMIT_PER_HOUR": "100000",
rateLimitPerDay: "1000000",
"SWWAF_RATE_LIMIT_PER_MINUTE": "500",
"SWWAF_RATE_LIMIT_PER_HOUR": "20000",
rateLimitPerDay: "300000",
"SWWAF_RATE_LIMIT_EXEMPT_PATHS": "",
"SWWAF_DENIED_COUNTRIES": "",
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "",