1 Commits
Author SHA1 Message Date
sneak a55b4c789c Default request limits per minute, hour and day, sized for API use (closes #176)
check / check (push) Successful in 4m45s
The default request limits now suit programmatic use of an API, such
as an automation fleet on another host, and still catch abusive bots:

- per minute: 500, room for a fleet's busiest minute and a few first
  visits with an empty cache, and passed within seconds by a client
  firing as fast as it can;
- per hour: 30000, which a client at the minute limit for a whole
  hour reaches exactly;
- per day: 300000, passed by a crawler that keeps up 4 a second.

The byte limits and ban lengths are unchanged. SPEC.md, README.md and
docs/settings.md give the reasons; docs/gitea.md redoes the Actions
runner sums.

Judgement call: the docs/alerts.md ban example shows the minute limit.

Model: opus-5-5
2026-11-04 20:30:11 +00:00
8 changed files with 44 additions and 41 deletions
+13 -10
View File
@@ -17,17 +17,20 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below.
## What it protects against
- Scrapers that fetch every page of a site, all day. Rate limiting gives each
client 300 requests a minute, 30,000 an hour and 300,000 a day. That leaves
client 500 requests a minute, 30,000 an hour and 300,000 a day. That leaves
room for programmatic use of an API, such as an automation fleet on another
host. The minute limit catches, within seconds, a client firing requests as
fast as it can. With the minute limit raised or off, the hour limit catches
one that keeps up 10 a second. The day limit catches one that keeps up 4 a
second around the clock. Counting cannot catch a client that stays under all
three limits, such as one that keeps up 3 a second; only the protections below
can. Byte limits give each client 4 GiB a minute, 20 GiB an hour and 200 GiB a
day. A client over a limit is refused and banned. The first ban lasts an hour.
Each repeat within a day makes the next ban three times as long. A ban that
would last more than seven days is permanent.
host. A fleet of about 20 agents makes about 120 requests a minute at its
busiest. A Gitea page opened with an empty cache loads about 40 files, so a
few such first visits fit in the minute limit too. The minute limit catches,
within seconds, a client firing requests as fast as it can. A client at the
minute limit for a whole hour reaches the hour limit exactly. The day limit
catches a client that keeps up 4 a second around the clock. Counting cannot
catch a client that stays under all three limits, such as one that keeps up 3
a second; only the protections below can. Byte limits give each client 4 GiB a
minute, 20 GiB an hour and 200 GiB a day. A client over a limit is refused and
banned. The first ban lasts an hour. Each repeat within a day makes the next
ban three times as long. A ban that would last more than seven days is
permanent.
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
default rule file bans such a client for seven days at its first probe. Its
next request during the ban makes the ban permanent. The default rule file
+8 -9
View File
@@ -381,14 +381,13 @@ as an automation fleet on another host, and still stop abusive bots. A fleet of
requests a minute and 7,200 an hour. Working around the clock, it makes 172,800
a day. It stays under all three limits.
- Per minute: 300, 5 requests a second. A client firing requests as fast as it
can passes it within seconds. It leaves room for the fleet and a few quick
first visits with an empty cache. A Gitea page opened with an empty cache
loads about 40 files.
- Per hour: 30,000, about 8 requests a second for a whole hour. At the default
minute limit no client reaches it, since 300 a minute is at most 18,000 an
hour. With the minute limit raised or off, a crawler that keeps up 10 a second
makes 36,000 an hour, and passes it.
- Per minute: 500, about 8 requests a second. It leaves room for the fleet's 120
a minute and a few quick first visits with an empty cache. A Gitea page opened
with an empty cache loads about 40 files. A client firing requests as fast as
it can passes the limit within seconds.
- Per hour: 30,000, about 8 requests a second for a whole hour. A client at the
minute limit for a whole hour reaches it exactly. The fleet's 7,200 an hour is
about a quarter of it.
- Per day: 300,000, about 3.5 requests a second around the clock. A crawler that
keeps up 4 a second makes 345,600 a day, and passes it. At 240 a minute and
14,400 an hour, it stays under the other two limits.
@@ -397,7 +396,7 @@ Counting cannot catch a client that stays under all three limits, such as one
that keeps up 3 requests a second. Only the other checks can catch an abuser
that slow. They include the trap paths, the rule files, the Core Rule Set and
the blocklists. An abuser that trips none of them is not caught. A machine that
keeps up more than 5 requests a second passes the minute limit. One that keeps
keeps up more than 500 requests a minute passes the minute limit. One that keeps
up more than about 3.5 a second all day passes the day limit, as two busy
Actions runners on one address do. Such a machine's address belongs in
`SWWAF_RATE_LIMIT_EXEMPT_NETS`.
+9 -9
View File
@@ -35,34 +35,34 @@ on one line:
```json
{
"instance": "fsn1app1/gitea",
"time": "2026-10-06T12:00:30.123461Z",
"time": "2026-10-06T12:00:50.123461Z",
"event": "ban",
"client": "203.0.113.9",
"netblock": "203.0.113.9/32",
"asn": "",
"as_name": "",
"country": "",
"reason": "requests per minute over the limit of 300",
"reason": "requests per minute over the limit of 500",
"detail": {
"ban_expires": "2026-10-06T13:00:30.123Z",
"ban_expires": "2026-10-06T13:00:50.123Z",
"cause": "limit",
"notes": {
"asn": "",
"as_name": "",
"country": "",
"kind": "requests",
"limit": 300,
"limit": 500,
"window": "minute",
"count": 301,
"count": 501,
"request": {
"time": "2026-10-06T12:00:30.123456789Z",
"time": "2026-10-06T12:00:50.123456789Z",
"method": "GET",
"host": "app.example",
"path": "/owner/repo/commits/branch/main?page=301",
"path": "/owner/repo/commits/branch/main?page=501",
"status": 403,
"user_agent": "scraper/1.0"
},
"requests": 301,
"requests": 501,
"refused": 0,
"earlier_bans": {
"limit": 0,
@@ -89,7 +89,7 @@ on one line:
- `asn`, `as_name` and `country`: the client's, as known when the alert was
raised. They are empty when GeoJS had not answered yet.
- `reason`: a short sentence, such as
`requests per minute over the limit of 300` or `listed by a blocklist`.
`requests per minute over the limit of 500` or `listed by a blocklist`.
- `detail`: what is particular to the event:
- for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed
under `bans.json` in [`state-files.md`](state-files.md));
+2 -2
View File
@@ -36,8 +36,8 @@ checked again when gitea or the Core Rule Set changes version.
every second instead, two requests a second. So one runner busy all day sends
120 requests a minute and 172,800 a day, under the request limits. Two such
runners on one address, each busy all day, make 345,600 and pass the day limit
of 300,000. Three busy at once make 360 a minute and pass the minute limit
of 300. Their address is then banned, their running jobs fail, and each repeat
of 300,000. Five busy at once make 600 a minute and pass the minute limit
of 500. Their address is then banned, their running jobs fail, and each repeat
within a day makes the ban longer. Newer runners ask for work less often. Put
the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
+8 -7
View File
@@ -99,15 +99,16 @@ that take `off`, the `_FILE` form, and what each group does and why.
## Request rate limits, per client
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `300`, 5 a second): the most requests a
client may make in a minute. A count or `off`. The default catches, within
seconds, a client firing requests as fast as it can. It leaves room for an
automation fleet's busiest minute and a few quick first visits with an empty
cache.
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `500`, about 8 a second): the most
requests a client may make in a minute. A count or `off`. The default catches,
within seconds, a client firing requests as fast as it can. It leaves room for
a fleet of about 20 agents at its busiest, about 120 requests a minute. It
also leaves room for a few quick first visits to Gitea with an empty cache,
about 40 files each.
- `SWWAF_RATE_LIMIT_PER_HOUR` (default `30000`, about 8 a second): the most
requests a client may make in an hour. A count or `off`. The default leaves
room for an automation fleet at work. With the minute limit raised or off, it
catches a crawler that keeps up 10 a second.
room for an automation fleet at work. A client at the minute limit for a whole
hour reaches it exactly.
- `SWWAF_RATE_LIMIT_PER_DAY` (default `300000`, about 3.5 a second): the most
requests a client may make in a day. A count or `off`. The default catches a
crawler that keeps up 4 a second around the clock.
+1 -1
View File
@@ -506,7 +506,7 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""),
RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""),
DenyNets: env.netblocks("SWWAF_DENY_NETS", ""),
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "300"),
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "500"),
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "30000"),
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"),
RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
+2 -2
View File
@@ -262,7 +262,7 @@ func TestDefaults(t *testing.T) {
UpstreamResponseTimeout: 30 * time.Minute,
RequestMaxBytes: 100 << 20,
ResponseMaxBytes: 1 << 30,
RateLimitPerMinute: 300,
RateLimitPerMinute: 500,
RateLimitPerHour: 30000,
RateLimitPerDay: 300000,
BanResponse: 403,
@@ -2375,7 +2375,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
allowNets: "",
rateLimitExemptNets: "",
denyNets: "",
rateLimitPerMinute: "300",
rateLimitPerMinute: "500",
rateLimitPerHour: "30000",
rateLimitPerDay: "300000",
rateLimitExemptPaths: "",
+1 -1
View File
@@ -1244,7 +1244,7 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
"SWWAF_ALLOW_NETS": "",
rateLimitExemptNets: "",
"SWWAF_DENY_NETS": "",
"SWWAF_RATE_LIMIT_PER_MINUTE": "300",
"SWWAF_RATE_LIMIT_PER_MINUTE": "500",
"SWWAF_RATE_LIMIT_PER_HOUR": "30000",
rateLimitPerDay: "300000",
"SWWAF_RATE_LIMIT_EXEMPT_PATHS": "",