1 Commits
Author SHA1 Message Date
sneak 3b9afd5ec4 Default request limits per minute, hour and day, sized for API use (closes #176)
check / check (push) Successful in 5m15s
The default request limits now suit programmatic use of an API, such
as an automation fleet on another host, and still catch abusive bots:

- per minute: 300, passed within seconds by a client firing as fast
  as it can;
- per hour: 30000;
- per day: 300000, passed by a crawler that keeps up 4 a second.

At the default minute limit no client reaches the hour limit: 300 a
minute is at most 18,000 an hour.

The byte limits and ban lengths are unchanged. SPEC.md, README.md and
docs/settings.md give the reasons; docs/gitea.md redoes the Actions
runner sums.

Judgement call: the docs/alerts.md ban example shows the minute limit.

Model: opus-5-5
2026-11-04 20:16:50 +00:00
8 changed files with 41 additions and 44 deletions

No files matched your search

+10 -13
View File
@@ -17,20 +17,17 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below.
## What it protects against ## What it protects against
- Scrapers that fetch every page of a site, all day. Rate limiting gives each - Scrapers that fetch every page of a site, all day. Rate limiting gives each
client 500 requests a minute, 30,000 an hour and 300,000 a day. That leaves client 300 requests a minute, 30,000 an hour and 300,000 a day. That leaves
room for programmatic use of an API, such as an automation fleet on another room for programmatic use of an API, such as an automation fleet on another
host. A fleet of about 20 agents makes about 120 requests a minute at its host. The minute limit catches, within seconds, a client firing requests as
busiest. A Gitea page opened with an empty cache loads about 40 files, so a fast as it can. With the minute limit raised or off, the hour limit catches
few such first visits fit in the minute limit too. The minute limit catches, one that keeps up 10 a second. The day limit catches one that keeps up 4 a
within seconds, a client firing requests as fast as it can. A client at the second around the clock. Counting cannot catch a client that stays under all
minute limit for a whole hour reaches the hour limit exactly. The day limit three limits, such as one that keeps up 3 a second; only the protections below
catches a client that keeps up 4 a second around the clock. Counting cannot can. Byte limits give each client 4 GiB a minute, 20 GiB an hour and 200 GiB a
catch a client that stays under all three limits, such as one that keeps up 3 day. A client over a limit is refused and banned. The first ban lasts an hour.
a second; only the protections below can. Byte limits give each client 4 GiB a Each repeat within a day makes the next ban three times as long. A ban that
minute, 20 GiB an hour and 200 GiB a day. A client over a limit is refused and would last more than seven days is permanent.
banned. The first ban lasts an hour. Each repeat within a day makes the next
ban three times as long. A ban that would last more than seven days is
permanent.
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The - Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
default rule file bans such a client for seven days at its first probe. Its default rule file bans such a client for seven days at its first probe. Its
next request during the ban makes the ban permanent. The default rule file next request during the ban makes the ban permanent. The default rule file
+9 -8
View File
@@ -381,13 +381,14 @@ as an automation fleet on another host, and still stop abusive bots. A fleet of
requests a minute and 7,200 an hour. Working around the clock, it makes 172,800 requests a minute and 7,200 an hour. Working around the clock, it makes 172,800
a day. It stays under all three limits. a day. It stays under all three limits.
- Per minute: 500, about 8 requests a second. It leaves room for the fleet's 120 - Per minute: 300, 5 requests a second. A client firing requests as fast as it
a minute and a few quick first visits with an empty cache. A Gitea page opened can passes it within seconds. It leaves room for the fleet and a few quick
with an empty cache loads about 40 files. A client firing requests as fast as first visits with an empty cache. A Gitea page opened with an empty cache
it can passes the limit within seconds. loads about 40 files.
- Per hour: 30,000, about 8 requests a second for a whole hour. A client at the - Per hour: 30,000, about 8 requests a second for a whole hour. At the default
minute limit for a whole hour reaches it exactly. The fleet's 7,200 an hour is minute limit no client reaches it, since 300 a minute is at most 18,000 an
about a quarter of it. hour. With the minute limit raised or off, a crawler that keeps up 10 a second
makes 36,000 an hour, and passes it.
- Per day: 300,000, about 3.5 requests a second around the clock. A crawler that - Per day: 300,000, about 3.5 requests a second around the clock. A crawler that
keeps up 4 a second makes 345,600 a day, and passes it. At 240 a minute and keeps up 4 a second makes 345,600 a day, and passes it. At 240 a minute and
14,400 an hour, it stays under the other two limits. 14,400 an hour, it stays under the other two limits.
@@ -396,7 +397,7 @@ Counting cannot catch a client that stays under all three limits, such as one
that keeps up 3 requests a second. Only the other checks can catch an abuser that keeps up 3 requests a second. Only the other checks can catch an abuser
that slow. They include the trap paths, the rule files, the Core Rule Set and that slow. They include the trap paths, the rule files, the Core Rule Set and
the blocklists. An abuser that trips none of them is not caught. A machine that the blocklists. An abuser that trips none of them is not caught. A machine that
keeps up more than 500 requests a minute passes the minute limit. One that keeps keeps up more than 5 requests a second passes the minute limit. One that keeps
up more than about 3.5 a second all day passes the day limit, as two busy up more than about 3.5 a second all day passes the day limit, as two busy
Actions runners on one address do. Such a machine's address belongs in Actions runners on one address do. Such a machine's address belongs in
`SWWAF_RATE_LIMIT_EXEMPT_NETS`. `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
+9 -9
View File
@@ -35,34 +35,34 @@ on one line:
```json ```json
{ {
"instance": "fsn1app1/gitea", "instance": "fsn1app1/gitea",
"time": "2026-10-06T12:00:50.123461Z", "time": "2026-10-06T12:00:30.123461Z",
"event": "ban", "event": "ban",
"client": "203.0.113.9", "client": "203.0.113.9",
"netblock": "203.0.113.9/32", "netblock": "203.0.113.9/32",
"asn": "", "asn": "",
"as_name": "", "as_name": "",
"country": "", "country": "",
"reason": "requests per minute over the limit of 500", "reason": "requests per minute over the limit of 300",
"detail": { "detail": {
"ban_expires": "2026-10-06T13:00:50.123Z", "ban_expires": "2026-10-06T13:00:30.123Z",
"cause": "limit", "cause": "limit",
"notes": { "notes": {
"asn": "", "asn": "",
"as_name": "", "as_name": "",
"country": "", "country": "",
"kind": "requests", "kind": "requests",
"limit": 500, "limit": 300,
"window": "minute", "window": "minute",
"count": 501, "count": 301,
"request": { "request": {
"time": "2026-10-06T12:00:50.123456789Z", "time": "2026-10-06T12:00:30.123456789Z",
"method": "GET", "method": "GET",
"host": "app.example", "host": "app.example",
"path": "/owner/repo/commits/branch/main?page=501", "path": "/owner/repo/commits/branch/main?page=301",
"status": 403, "status": 403,
"user_agent": "scraper/1.0" "user_agent": "scraper/1.0"
}, },
"requests": 501, "requests": 301,
"refused": 0, "refused": 0,
"earlier_bans": { "earlier_bans": {
"limit": 0, "limit": 0,
@@ -89,7 +89,7 @@ on one line:
- `asn`, `as_name` and `country`: the client's, as known when the alert was - `asn`, `as_name` and `country`: the client's, as known when the alert was
raised. They are empty when GeoJS had not answered yet. raised. They are empty when GeoJS had not answered yet.
- `reason`: a short sentence, such as - `reason`: a short sentence, such as
`requests per minute over the limit of 500` or `listed by a blocklist`. `requests per minute over the limit of 300` or `listed by a blocklist`.
- `detail`: what is particular to the event: - `detail`: what is particular to the event:
- for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed - for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed
under `bans.json` in [`state-files.md`](state-files.md)); under `bans.json` in [`state-files.md`](state-files.md));
+2 -2
View File
@@ -36,8 +36,8 @@ checked again when gitea or the Core Rule Set changes version.
every second instead, two requests a second. So one runner busy all day sends every second instead, two requests a second. So one runner busy all day sends
120 requests a minute and 172,800 a day, under the request limits. Two such 120 requests a minute and 172,800 a day, under the request limits. Two such
runners on one address, each busy all day, make 345,600 and pass the day limit runners on one address, each busy all day, make 345,600 and pass the day limit
of 300,000. Five busy at once make 600 a minute and pass the minute limit of 300,000. Three busy at once make 360 a minute and pass the minute limit
of 500. Their address is then banned, their running jobs fail, and each repeat of 300. Their address is then banned, their running jobs fail, and each repeat
within a day makes the ban longer. Newer runners ask for work less often. Put within a day makes the ban longer. Newer runners ask for work less often. Put
the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
+7 -8
View File
@@ -99,16 +99,15 @@ that take `off`, the `_FILE` form, and what each group does and why.
## Request rate limits, per client ## Request rate limits, per client
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `500`, about 8 a second): the most - `SWWAF_RATE_LIMIT_PER_MINUTE` (default `300`, 5 a second): the most requests a
requests a client may make in a minute. A count or `off`. The default catches, client may make in a minute. A count or `off`. The default catches, within
within seconds, a client firing requests as fast as it can. It leaves room for seconds, a client firing requests as fast as it can. It leaves room for an
a fleet of about 20 agents at its busiest, about 120 requests a minute. It automation fleet's busiest minute and a few quick first visits with an empty
also leaves room for a few quick first visits to Gitea with an empty cache, cache.
about 40 files each.
- `SWWAF_RATE_LIMIT_PER_HOUR` (default `30000`, about 8 a second): the most - `SWWAF_RATE_LIMIT_PER_HOUR` (default `30000`, about 8 a second): the most
requests a client may make in an hour. A count or `off`. The default leaves requests a client may make in an hour. A count or `off`. The default leaves
room for an automation fleet at work. A client at the minute limit for a whole room for an automation fleet at work. With the minute limit raised or off, it
hour reaches it exactly. catches a crawler that keeps up 10 a second.
- `SWWAF_RATE_LIMIT_PER_DAY` (default `300000`, about 3.5 a second): the most - `SWWAF_RATE_LIMIT_PER_DAY` (default `300000`, about 3.5 a second): the most
requests a client may make in a day. A count or `off`. The default catches a requests a client may make in a day. A count or `off`. The default catches a
crawler that keeps up 4 a second around the clock. crawler that keeps up 4 a second around the clock.
+1 -1
View File
@@ -506,7 +506,7 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""), AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""),
RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""), RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""),
DenyNets: env.netblocks("SWWAF_DENY_NETS", ""), DenyNets: env.netblocks("SWWAF_DENY_NETS", ""),
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "500"), RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "300"),
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "30000"), RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "30000"),
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"), RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"),
RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""), RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
+2 -2
View File
@@ -262,7 +262,7 @@ func TestDefaults(t *testing.T) {
UpstreamResponseTimeout: 30 * time.Minute, UpstreamResponseTimeout: 30 * time.Minute,
RequestMaxBytes: 100 << 20, RequestMaxBytes: 100 << 20,
ResponseMaxBytes: 1 << 30, ResponseMaxBytes: 1 << 30,
RateLimitPerMinute: 500, RateLimitPerMinute: 300,
RateLimitPerHour: 30000, RateLimitPerHour: 30000,
RateLimitPerDay: 300000, RateLimitPerDay: 300000,
BanResponse: 403, BanResponse: 403,
@@ -2375,7 +2375,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
allowNets: "", allowNets: "",
rateLimitExemptNets: "", rateLimitExemptNets: "",
denyNets: "", denyNets: "",
rateLimitPerMinute: "500", rateLimitPerMinute: "300",
rateLimitPerHour: "30000", rateLimitPerHour: "30000",
rateLimitPerDay: "300000", rateLimitPerDay: "300000",
rateLimitExemptPaths: "", rateLimitExemptPaths: "",
+1 -1
View File
@@ -1244,7 +1244,7 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
"SWWAF_ALLOW_NETS": "", "SWWAF_ALLOW_NETS": "",
rateLimitExemptNets: "", rateLimitExemptNets: "",
"SWWAF_DENY_NETS": "", "SWWAF_DENY_NETS": "",
"SWWAF_RATE_LIMIT_PER_MINUTE": "500", "SWWAF_RATE_LIMIT_PER_MINUTE": "300",
"SWWAF_RATE_LIMIT_PER_HOUR": "30000", "SWWAF_RATE_LIMIT_PER_HOUR": "30000",
rateLimitPerDay: "300000", rateLimitPerDay: "300000",
"SWWAF_RATE_LIMIT_EXEMPT_PATHS": "", "SWWAF_RATE_LIMIT_EXEMPT_PATHS": "",