Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a55b4c789c |
@@ -17,17 +17,20 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below.
|
||||
## What it protects against
|
||||
|
||||
- Scrapers that fetch every page of a site, all day. Rate limiting gives each
|
||||
client 300 requests a minute, 30,000 an hour and 300,000 a day. That leaves
|
||||
client 500 requests a minute, 30,000 an hour and 300,000 a day. That leaves
|
||||
room for programmatic use of an API, such as an automation fleet on another
|
||||
host. The minute limit catches, within seconds, a client firing requests as
|
||||
fast as it can. With the minute limit raised or off, the hour limit catches
|
||||
one that keeps up 10 a second. The day limit catches one that keeps up 4 a
|
||||
second around the clock. Counting cannot catch a client that stays under all
|
||||
three limits, such as one that keeps up 3 a second; only the protections below
|
||||
can. Byte limits give each client 4 GiB a minute, 20 GiB an hour and 200 GiB a
|
||||
day. A client over a limit is refused and banned. The first ban lasts an hour.
|
||||
Each repeat within a day makes the next ban three times as long. A ban that
|
||||
would last more than seven days is permanent.
|
||||
host. A fleet of about 20 agents makes about 120 requests a minute at its
|
||||
busiest. A Gitea page opened with an empty cache loads about 40 files, so a
|
||||
few such first visits fit in the minute limit too. The minute limit catches,
|
||||
within seconds, a client firing requests as fast as it can. A client at the
|
||||
minute limit for a whole hour reaches the hour limit exactly. The day limit
|
||||
catches a client that keeps up 4 a second around the clock. Counting cannot
|
||||
catch a client that stays under all three limits, such as one that keeps up 3
|
||||
a second; only the protections below can. Byte limits give each client 4 GiB a
|
||||
minute, 20 GiB an hour and 200 GiB a day. A client over a limit is refused and
|
||||
banned. The first ban lasts an hour. Each repeat within a day makes the next
|
||||
ban three times as long. A ban that would last more than seven days is
|
||||
permanent.
|
||||
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
|
||||
default rule file bans such a client for seven days at its first probe. Its
|
||||
next request during the ban makes the ban permanent. The default rule file
|
||||
|
||||
@@ -381,14 +381,13 @@ as an automation fleet on another host, and still stop abusive bots. A fleet of
|
||||
requests a minute and 7,200 an hour. Working around the clock, it makes 172,800
|
||||
a day. It stays under all three limits.
|
||||
|
||||
- Per minute: 300, 5 requests a second. A client firing requests as fast as it
|
||||
can passes it within seconds. It leaves room for the fleet and a few quick
|
||||
first visits with an empty cache. A Gitea page opened with an empty cache
|
||||
loads about 40 files.
|
||||
- Per hour: 30,000, about 8 requests a second for a whole hour. At the default
|
||||
minute limit no client reaches it, since 300 a minute is at most 18,000 an
|
||||
hour. With the minute limit raised or off, a crawler that keeps up 10 a second
|
||||
makes 36,000 an hour, and passes it.
|
||||
- Per minute: 500, about 8 requests a second. It leaves room for the fleet's 120
|
||||
a minute and a few quick first visits with an empty cache. A Gitea page opened
|
||||
with an empty cache loads about 40 files. A client firing requests as fast as
|
||||
it can passes the limit within seconds.
|
||||
- Per hour: 30,000, about 8 requests a second for a whole hour. A client at the
|
||||
minute limit for a whole hour reaches it exactly. The fleet's 7,200 an hour is
|
||||
about a quarter of it.
|
||||
- Per day: 300,000, about 3.5 requests a second around the clock. A crawler that
|
||||
keeps up 4 a second makes 345,600 a day, and passes it. At 240 a minute and
|
||||
14,400 an hour, it stays under the other two limits.
|
||||
@@ -397,7 +396,7 @@ Counting cannot catch a client that stays under all three limits, such as one
|
||||
that keeps up 3 requests a second. Only the other checks can catch an abuser
|
||||
that slow. They include the trap paths, the rule files, the Core Rule Set and
|
||||
the blocklists. An abuser that trips none of them is not caught. A machine that
|
||||
keeps up more than 5 requests a second passes the minute limit. One that keeps
|
||||
keeps up more than 500 requests a minute passes the minute limit. One that keeps
|
||||
up more than about 3.5 a second all day passes the day limit, as two busy
|
||||
Actions runners on one address do. Such a machine's address belongs in
|
||||
`SWWAF_RATE_LIMIT_EXEMPT_NETS`.
|
||||
|
||||
+9
-9
@@ -35,34 +35,34 @@ on one line:
|
||||
```json
|
||||
{
|
||||
"instance": "fsn1app1/gitea",
|
||||
"time": "2026-10-06T12:00:30.123461Z",
|
||||
"time": "2026-10-06T12:00:50.123461Z",
|
||||
"event": "ban",
|
||||
"client": "203.0.113.9",
|
||||
"netblock": "203.0.113.9/32",
|
||||
"asn": "",
|
||||
"as_name": "",
|
||||
"country": "",
|
||||
"reason": "requests per minute over the limit of 300",
|
||||
"reason": "requests per minute over the limit of 500",
|
||||
"detail": {
|
||||
"ban_expires": "2026-10-06T13:00:30.123Z",
|
||||
"ban_expires": "2026-10-06T13:00:50.123Z",
|
||||
"cause": "limit",
|
||||
"notes": {
|
||||
"asn": "",
|
||||
"as_name": "",
|
||||
"country": "",
|
||||
"kind": "requests",
|
||||
"limit": 300,
|
||||
"limit": 500,
|
||||
"window": "minute",
|
||||
"count": 301,
|
||||
"count": 501,
|
||||
"request": {
|
||||
"time": "2026-10-06T12:00:30.123456789Z",
|
||||
"time": "2026-10-06T12:00:50.123456789Z",
|
||||
"method": "GET",
|
||||
"host": "app.example",
|
||||
"path": "/owner/repo/commits/branch/main?page=301",
|
||||
"path": "/owner/repo/commits/branch/main?page=501",
|
||||
"status": 403,
|
||||
"user_agent": "scraper/1.0"
|
||||
},
|
||||
"requests": 301,
|
||||
"requests": 501,
|
||||
"refused": 0,
|
||||
"earlier_bans": {
|
||||
"limit": 0,
|
||||
@@ -89,7 +89,7 @@ on one line:
|
||||
- `asn`, `as_name` and `country`: the client's, as known when the alert was
|
||||
raised. They are empty when GeoJS had not answered yet.
|
||||
- `reason`: a short sentence, such as
|
||||
`requests per minute over the limit of 300` or `listed by a blocklist`.
|
||||
`requests per minute over the limit of 500` or `listed by a blocklist`.
|
||||
- `detail`: what is particular to the event:
|
||||
- for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed
|
||||
under `bans.json` in [`state-files.md`](state-files.md));
|
||||
|
||||
+2
-2
@@ -36,8 +36,8 @@ checked again when gitea or the Core Rule Set changes version.
|
||||
every second instead, two requests a second. So one runner busy all day sends
|
||||
120 requests a minute and 172,800 a day, under the request limits. Two such
|
||||
runners on one address, each busy all day, make 345,600 and pass the day limit
|
||||
of 300,000. Three busy at once make 360 a minute and pass the minute limit
|
||||
of 300. Their address is then banned, their running jobs fail, and each repeat
|
||||
of 300,000. Five busy at once make 600 a minute and pass the minute limit
|
||||
of 500. Their address is then banned, their running jobs fail, and each repeat
|
||||
within a day makes the ban longer. Newer runners ask for work less often. Put
|
||||
the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
|
||||
|
||||
|
||||
+8
-7
@@ -99,15 +99,16 @@ that take `off`, the `_FILE` form, and what each group does and why.
|
||||
|
||||
## Request rate limits, per client
|
||||
|
||||
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `300`, 5 a second): the most requests a
|
||||
client may make in a minute. A count or `off`. The default catches, within
|
||||
seconds, a client firing requests as fast as it can. It leaves room for an
|
||||
automation fleet's busiest minute and a few quick first visits with an empty
|
||||
cache.
|
||||
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `500`, about 8 a second): the most
|
||||
requests a client may make in a minute. A count or `off`. The default catches,
|
||||
within seconds, a client firing requests as fast as it can. It leaves room for
|
||||
a fleet of about 20 agents at its busiest, about 120 requests a minute. It
|
||||
also leaves room for a few quick first visits to Gitea with an empty cache,
|
||||
about 40 files each.
|
||||
- `SWWAF_RATE_LIMIT_PER_HOUR` (default `30000`, about 8 a second): the most
|
||||
requests a client may make in an hour. A count or `off`. The default leaves
|
||||
room for an automation fleet at work. With the minute limit raised or off, it
|
||||
catches a crawler that keeps up 10 a second.
|
||||
room for an automation fleet at work. A client at the minute limit for a whole
|
||||
hour reaches it exactly.
|
||||
- `SWWAF_RATE_LIMIT_PER_DAY` (default `300000`, about 3.5 a second): the most
|
||||
requests a client may make in a day. A count or `off`. The default catches a
|
||||
crawler that keeps up 4 a second around the clock.
|
||||
|
||||
@@ -506,7 +506,7 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
||||
AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""),
|
||||
RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""),
|
||||
DenyNets: env.netblocks("SWWAF_DENY_NETS", ""),
|
||||
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "300"),
|
||||
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "500"),
|
||||
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "30000"),
|
||||
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"),
|
||||
RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
|
||||
|
||||
@@ -262,7 +262,7 @@ func TestDefaults(t *testing.T) {
|
||||
UpstreamResponseTimeout: 30 * time.Minute,
|
||||
RequestMaxBytes: 100 << 20,
|
||||
ResponseMaxBytes: 1 << 30,
|
||||
RateLimitPerMinute: 300,
|
||||
RateLimitPerMinute: 500,
|
||||
RateLimitPerHour: 30000,
|
||||
RateLimitPerDay: 300000,
|
||||
BanResponse: 403,
|
||||
@@ -2375,7 +2375,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
allowNets: "",
|
||||
rateLimitExemptNets: "",
|
||||
denyNets: "",
|
||||
rateLimitPerMinute: "300",
|
||||
rateLimitPerMinute: "500",
|
||||
rateLimitPerHour: "30000",
|
||||
rateLimitPerDay: "300000",
|
||||
rateLimitExemptPaths: "",
|
||||
|
||||
@@ -1244,7 +1244,7 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
|
||||
"SWWAF_ALLOW_NETS": "",
|
||||
rateLimitExemptNets: "",
|
||||
"SWWAF_DENY_NETS": "",
|
||||
"SWWAF_RATE_LIMIT_PER_MINUTE": "300",
|
||||
"SWWAF_RATE_LIMIT_PER_MINUTE": "500",
|
||||
"SWWAF_RATE_LIMIT_PER_HOUR": "30000",
|
||||
rateLimitPerDay: "300000",
|
||||
"SWWAF_RATE_LIMIT_EXEMPT_PATHS": "",
|
||||
|
||||
Reference in New Issue
Block a user