Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3b9afd5ec4 |
No files matched your search
@@ -17,20 +17,17 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below.
|
|||||||
## What it protects against
|
## What it protects against
|
||||||
|
|
||||||
- Scrapers that fetch every page of a site, all day. Rate limiting gives each
|
- Scrapers that fetch every page of a site, all day. Rate limiting gives each
|
||||||
client 500 requests a minute, 30,000 an hour and 300,000 a day. That leaves
|
client 300 requests a minute, 30,000 an hour and 300,000 a day. That leaves
|
||||||
room for programmatic use of an API, such as an automation fleet on another
|
room for programmatic use of an API, such as an automation fleet on another
|
||||||
host. A fleet of about 20 agents makes about 120 requests a minute at its
|
host. The minute limit catches, within seconds, a client firing requests as
|
||||||
busiest. A Gitea page opened with an empty cache loads about 40 files, so a
|
fast as it can. With the minute limit raised or off, the hour limit catches
|
||||||
few such first visits fit in the minute limit too. The minute limit catches,
|
one that keeps up 10 a second. The day limit catches one that keeps up 4 a
|
||||||
within seconds, a client firing requests as fast as it can. A client at the
|
second around the clock. Counting cannot catch a client that stays under all
|
||||||
minute limit for a whole hour reaches the hour limit exactly. The day limit
|
three limits, such as one that keeps up 3 a second; only the protections below
|
||||||
catches a client that keeps up 4 a second around the clock. Counting cannot
|
can. Byte limits give each client 4 GiB a minute, 20 GiB an hour and 200 GiB a
|
||||||
catch a client that stays under all three limits, such as one that keeps up 3
|
day. A client over a limit is refused and banned. The first ban lasts an hour.
|
||||||
a second; only the protections below can. Byte limits give each client 4 GiB a
|
Each repeat within a day makes the next ban three times as long. A ban that
|
||||||
minute, 20 GiB an hour and 200 GiB a day. A client over a limit is refused and
|
would last more than seven days is permanent.
|
||||||
banned. The first ban lasts an hour. Each repeat within a day makes the next
|
|
||||||
ban three times as long. A ban that would last more than seven days is
|
|
||||||
permanent.
|
|
||||||
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
|
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
|
||||||
default rule file bans such a client for seven days at its first probe. Its
|
default rule file bans such a client for seven days at its first probe. Its
|
||||||
next request during the ban makes the ban permanent. The default rule file
|
next request during the ban makes the ban permanent. The default rule file
|
||||||
|
|||||||
@@ -381,13 +381,14 @@ as an automation fleet on another host, and still stop abusive bots. A fleet of
|
|||||||
requests a minute and 7,200 an hour. Working around the clock, it makes 172,800
|
requests a minute and 7,200 an hour. Working around the clock, it makes 172,800
|
||||||
a day. It stays under all three limits.
|
a day. It stays under all three limits.
|
||||||
|
|
||||||
- Per minute: 500, about 8 requests a second. It leaves room for the fleet's 120
|
- Per minute: 300, 5 requests a second. A client firing requests as fast as it
|
||||||
a minute and a few quick first visits with an empty cache. A Gitea page opened
|
can passes it within seconds. It leaves room for the fleet and a few quick
|
||||||
with an empty cache loads about 40 files. A client firing requests as fast as
|
first visits with an empty cache. A Gitea page opened with an empty cache
|
||||||
it can passes the limit within seconds.
|
loads about 40 files.
|
||||||
- Per hour: 30,000, about 8 requests a second for a whole hour. A client at the
|
- Per hour: 30,000, about 8 requests a second for a whole hour. At the default
|
||||||
minute limit for a whole hour reaches it exactly. The fleet's 7,200 an hour is
|
minute limit no client reaches it, since 300 a minute is at most 18,000 an
|
||||||
about a quarter of it.
|
hour. With the minute limit raised or off, a crawler that keeps up 10 a second
|
||||||
|
makes 36,000 an hour, and passes it.
|
||||||
- Per day: 300,000, about 3.5 requests a second around the clock. A crawler that
|
- Per day: 300,000, about 3.5 requests a second around the clock. A crawler that
|
||||||
keeps up 4 a second makes 345,600 a day, and passes it. At 240 a minute and
|
keeps up 4 a second makes 345,600 a day, and passes it. At 240 a minute and
|
||||||
14,400 an hour, it stays under the other two limits.
|
14,400 an hour, it stays under the other two limits.
|
||||||
@@ -396,7 +397,7 @@ Counting cannot catch a client that stays under all three limits, such as one
|
|||||||
that keeps up 3 requests a second. Only the other checks can catch an abuser
|
that keeps up 3 requests a second. Only the other checks can catch an abuser
|
||||||
that slow. They include the trap paths, the rule files, the Core Rule Set and
|
that slow. They include the trap paths, the rule files, the Core Rule Set and
|
||||||
the blocklists. An abuser that trips none of them is not caught. A machine that
|
the blocklists. An abuser that trips none of them is not caught. A machine that
|
||||||
keeps up more than 500 requests a minute passes the minute limit. One that keeps
|
keeps up more than 5 requests a second passes the minute limit. One that keeps
|
||||||
up more than about 3.5 a second all day passes the day limit, as two busy
|
up more than about 3.5 a second all day passes the day limit, as two busy
|
||||||
Actions runners on one address do. Such a machine's address belongs in
|
Actions runners on one address do. Such a machine's address belongs in
|
||||||
`SWWAF_RATE_LIMIT_EXEMPT_NETS`.
|
`SWWAF_RATE_LIMIT_EXEMPT_NETS`.
|
||||||
|
|||||||
+9
-9
@@ -35,34 +35,34 @@ on one line:
|
|||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"instance": "fsn1app1/gitea",
|
"instance": "fsn1app1/gitea",
|
||||||
"time": "2026-10-06T12:00:50.123461Z",
|
"time": "2026-10-06T12:00:30.123461Z",
|
||||||
"event": "ban",
|
"event": "ban",
|
||||||
"client": "203.0.113.9",
|
"client": "203.0.113.9",
|
||||||
"netblock": "203.0.113.9/32",
|
"netblock": "203.0.113.9/32",
|
||||||
"asn": "",
|
"asn": "",
|
||||||
"as_name": "",
|
"as_name": "",
|
||||||
"country": "",
|
"country": "",
|
||||||
"reason": "requests per minute over the limit of 500",
|
"reason": "requests per minute over the limit of 300",
|
||||||
"detail": {
|
"detail": {
|
||||||
"ban_expires": "2026-10-06T13:00:50.123Z",
|
"ban_expires": "2026-10-06T13:00:30.123Z",
|
||||||
"cause": "limit",
|
"cause": "limit",
|
||||||
"notes": {
|
"notes": {
|
||||||
"asn": "",
|
"asn": "",
|
||||||
"as_name": "",
|
"as_name": "",
|
||||||
"country": "",
|
"country": "",
|
||||||
"kind": "requests",
|
"kind": "requests",
|
||||||
"limit": 500,
|
"limit": 300,
|
||||||
"window": "minute",
|
"window": "minute",
|
||||||
"count": 501,
|
"count": 301,
|
||||||
"request": {
|
"request": {
|
||||||
"time": "2026-10-06T12:00:50.123456789Z",
|
"time": "2026-10-06T12:00:30.123456789Z",
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
"host": "app.example",
|
"host": "app.example",
|
||||||
"path": "/owner/repo/commits/branch/main?page=501",
|
"path": "/owner/repo/commits/branch/main?page=301",
|
||||||
"status": 403,
|
"status": 403,
|
||||||
"user_agent": "scraper/1.0"
|
"user_agent": "scraper/1.0"
|
||||||
},
|
},
|
||||||
"requests": 501,
|
"requests": 301,
|
||||||
"refused": 0,
|
"refused": 0,
|
||||||
"earlier_bans": {
|
"earlier_bans": {
|
||||||
"limit": 0,
|
"limit": 0,
|
||||||
@@ -89,7 +89,7 @@ on one line:
|
|||||||
- `asn`, `as_name` and `country`: the client's, as known when the alert was
|
- `asn`, `as_name` and `country`: the client's, as known when the alert was
|
||||||
raised. They are empty when GeoJS had not answered yet.
|
raised. They are empty when GeoJS had not answered yet.
|
||||||
- `reason`: a short sentence, such as
|
- `reason`: a short sentence, such as
|
||||||
`requests per minute over the limit of 500` or `listed by a blocklist`.
|
`requests per minute over the limit of 300` or `listed by a blocklist`.
|
||||||
- `detail`: what is particular to the event:
|
- `detail`: what is particular to the event:
|
||||||
- for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed
|
- for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed
|
||||||
under `bans.json` in [`state-files.md`](state-files.md));
|
under `bans.json` in [`state-files.md`](state-files.md));
|
||||||
|
|||||||
+2
-2
@@ -36,8 +36,8 @@ checked again when gitea or the Core Rule Set changes version.
|
|||||||
every second instead, two requests a second. So one runner busy all day sends
|
every second instead, two requests a second. So one runner busy all day sends
|
||||||
120 requests a minute and 172,800 a day, under the request limits. Two such
|
120 requests a minute and 172,800 a day, under the request limits. Two such
|
||||||
runners on one address, each busy all day, make 345,600 and pass the day limit
|
runners on one address, each busy all day, make 345,600 and pass the day limit
|
||||||
of 300,000. Five busy at once make 600 a minute and pass the minute limit
|
of 300,000. Three busy at once make 360 a minute and pass the minute limit
|
||||||
of 500. Their address is then banned, their running jobs fail, and each repeat
|
of 300. Their address is then banned, their running jobs fail, and each repeat
|
||||||
within a day makes the ban longer. Newer runners ask for work less often. Put
|
within a day makes the ban longer. Newer runners ask for work less often. Put
|
||||||
the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
|
the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
|
||||||
|
|
||||||
|
|||||||
+7
-8
@@ -99,16 +99,15 @@ that take `off`, the `_FILE` form, and what each group does and why.
|
|||||||
|
|
||||||
## Request rate limits, per client
|
## Request rate limits, per client
|
||||||
|
|
||||||
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `500`, about 8 a second): the most
|
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `300`, 5 a second): the most requests a
|
||||||
requests a client may make in a minute. A count or `off`. The default catches,
|
client may make in a minute. A count or `off`. The default catches, within
|
||||||
within seconds, a client firing requests as fast as it can. It leaves room for
|
seconds, a client firing requests as fast as it can. It leaves room for an
|
||||||
a fleet of about 20 agents at its busiest, about 120 requests a minute. It
|
automation fleet's busiest minute and a few quick first visits with an empty
|
||||||
also leaves room for a few quick first visits to Gitea with an empty cache,
|
cache.
|
||||||
about 40 files each.
|
|
||||||
- `SWWAF_RATE_LIMIT_PER_HOUR` (default `30000`, about 8 a second): the most
|
- `SWWAF_RATE_LIMIT_PER_HOUR` (default `30000`, about 8 a second): the most
|
||||||
requests a client may make in an hour. A count or `off`. The default leaves
|
requests a client may make in an hour. A count or `off`. The default leaves
|
||||||
room for an automation fleet at work. A client at the minute limit for a whole
|
room for an automation fleet at work. With the minute limit raised or off, it
|
||||||
hour reaches it exactly.
|
catches a crawler that keeps up 10 a second.
|
||||||
- `SWWAF_RATE_LIMIT_PER_DAY` (default `300000`, about 3.5 a second): the most
|
- `SWWAF_RATE_LIMIT_PER_DAY` (default `300000`, about 3.5 a second): the most
|
||||||
requests a client may make in a day. A count or `off`. The default catches a
|
requests a client may make in a day. A count or `off`. The default catches a
|
||||||
crawler that keeps up 4 a second around the clock.
|
crawler that keeps up 4 a second around the clock.
|
||||||
|
|||||||
@@ -506,7 +506,7 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
|||||||
AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""),
|
AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""),
|
||||||
RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""),
|
RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""),
|
||||||
DenyNets: env.netblocks("SWWAF_DENY_NETS", ""),
|
DenyNets: env.netblocks("SWWAF_DENY_NETS", ""),
|
||||||
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "500"),
|
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "300"),
|
||||||
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "30000"),
|
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "30000"),
|
||||||
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"),
|
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"),
|
||||||
RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
|
RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
|
||||||
|
|||||||
@@ -262,7 +262,7 @@ func TestDefaults(t *testing.T) {
|
|||||||
UpstreamResponseTimeout: 30 * time.Minute,
|
UpstreamResponseTimeout: 30 * time.Minute,
|
||||||
RequestMaxBytes: 100 << 20,
|
RequestMaxBytes: 100 << 20,
|
||||||
ResponseMaxBytes: 1 << 30,
|
ResponseMaxBytes: 1 << 30,
|
||||||
RateLimitPerMinute: 500,
|
RateLimitPerMinute: 300,
|
||||||
RateLimitPerHour: 30000,
|
RateLimitPerHour: 30000,
|
||||||
RateLimitPerDay: 300000,
|
RateLimitPerDay: 300000,
|
||||||
BanResponse: 403,
|
BanResponse: 403,
|
||||||
@@ -2375,7 +2375,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
|||||||
allowNets: "",
|
allowNets: "",
|
||||||
rateLimitExemptNets: "",
|
rateLimitExemptNets: "",
|
||||||
denyNets: "",
|
denyNets: "",
|
||||||
rateLimitPerMinute: "500",
|
rateLimitPerMinute: "300",
|
||||||
rateLimitPerHour: "30000",
|
rateLimitPerHour: "30000",
|
||||||
rateLimitPerDay: "300000",
|
rateLimitPerDay: "300000",
|
||||||
rateLimitExemptPaths: "",
|
rateLimitExemptPaths: "",
|
||||||
|
|||||||
@@ -1244,7 +1244,7 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
|
|||||||
"SWWAF_ALLOW_NETS": "",
|
"SWWAF_ALLOW_NETS": "",
|
||||||
rateLimitExemptNets: "",
|
rateLimitExemptNets: "",
|
||||||
"SWWAF_DENY_NETS": "",
|
"SWWAF_DENY_NETS": "",
|
||||||
"SWWAF_RATE_LIMIT_PER_MINUTE": "500",
|
"SWWAF_RATE_LIMIT_PER_MINUTE": "300",
|
||||||
"SWWAF_RATE_LIMIT_PER_HOUR": "30000",
|
"SWWAF_RATE_LIMIT_PER_HOUR": "30000",
|
||||||
rateLimitPerDay: "300000",
|
rateLimitPerDay: "300000",
|
||||||
"SWWAF_RATE_LIMIT_EXEMPT_PATHS": "",
|
"SWWAF_RATE_LIMIT_EXEMPT_PATHS": "",
|
||||||
|
|||||||
Reference in new issue
Block a user