Default request limits per minute, hour and day, sized for API use (closes #176) #179
@@ -17,11 +17,20 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below.
|
|||||||
## What it protects against
|
## What it protects against
|
||||||
|
|
||||||
- Scrapers that fetch every page of a site, all day. Rate limiting gives each
|
- Scrapers that fetch every page of a site, all day. Rate limiting gives each
|
||||||
client 6000 requests a minute, 100,000 an hour and 1,000,000 a day. Byte
|
client 500 requests a minute, 20,000 an hour and 300,000 a day. That leaves
|
||||||
limits give it 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over
|
room for programmatic use of an API, such as an automation fleet on another
|
||||||
a limit is refused and banned. The first ban lasts an hour. Each repeat within
|
host. A fleet of about 20 agents makes about 120 requests a minute at its
|
||||||
a day makes the next ban three times as long. A ban that would last more than
|
busiest. A gitea page opened with an empty cache loads about 40 files, so a
|
||||||
seven days is permanent.
|
few such first visits fit in the minute limit too. The minute limit catches,
|
||||||
|
within seconds, a client firing requests as fast as it can. The hour limit
|
||||||
|
catches a client that keeps up more than about 5.5 a second for an hour. The
|
||||||
|
day limit catches a client that keeps up 4 a second around the clock. Counting
|
||||||
|
cannot catch a client that stays under all three limits, such as one that
|
||||||
|
keeps up 3 a second; only the protections below can. Byte limits give each
|
||||||
|
client 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over a limit
|
||||||
|
is refused and banned. The first ban lasts an hour. Each repeat within a day
|
||||||
|
makes the next ban three times as long. A ban that would last more than seven
|
||||||
|
days is permanent.
|
||||||
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
|
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
|
||||||
default rule file bans such a client for seven days at its first probe. Its
|
default rule file bans such a client for seven days at its first probe. Its
|
||||||
next request during the ban makes the ban permanent. The default rule file
|
next request during the ban makes the ban permanent. The default rule file
|
||||||
|
|||||||
@@ -369,22 +369,38 @@ Conventions:
|
|||||||
owner's own networks. `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and
|
owner's own networks. `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and
|
||||||
byte limits only; the error burst, attack detection and bans still apply.
|
byte limits only; the error burst, attack detection and bans still apply.
|
||||||
- Many people can share one address, as in an office, on a mobile carrier or
|
- Many people can share one address, as in an office, on a mobile carrier or
|
||||||
through Tor. They are one client, and share its limits and its bans. The
|
through Tor. They are one client, and share its limits and its bans. A known
|
||||||
default limits are sized for about 40 busy people on one address. A known
|
shared address busier than the limits allow belongs in `SWWAF_ALLOW_NETS` or
|
||||||
shared address busier than that belongs in `SWWAF_ALLOW_NETS` or
|
|
||||||
`SWWAF_RATE_LIMIT_EXEMPT_NETS`.
|
`SWWAF_RATE_LIMIT_EXEMPT_NETS`.
|
||||||
|
|
||||||
### Request rate limits, per client
|
### Request rate limits, per client
|
||||||
|
|
||||||
Breaking one bans the client, so the defaults sit about twice above what 40
|
Breaking one bans the client. The defaults suit programmatic use of an API, such
|
||||||
people busy with heavy web apps make together on one address. Each such person
|
as an automation fleet on another host, and still stop abusive bots. A fleet of
|
||||||
makes about 75 requests in a busy minute, and 1,250 an hour for ten hours a day.
|
20 agents, each making a request every 10 seconds at its busiest, makes 120
|
||||||
|
requests a minute and 7,200 an hour. Working around the clock, it makes 172,800
|
||||||
|
a day. It stays under all three limits.
|
||||||
|
|
||||||
A client slower than those 40 people cannot be told from them by counting. Only
|
- Per minute: 500, about 8 requests a second. It leaves room for the fleet's 120
|
||||||
the other checks can catch an abuser that slow. They include the trap paths, the
|
a minute and a few quick first visits with an empty cache. A gitea page opened
|
||||||
rule files, the Core Rule Set and the blocklists. An abuser that trips none of
|
with an empty cache loads about 40 files. A client firing requests as fast as
|
||||||
them is not caught. A machine that talks to the app all day can still pass the
|
it can passes the limit within seconds.
|
||||||
day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
|
- Per hour: 20,000, about 5.5 requests a second for a whole hour. A client that
|
||||||
|
keeps up 6 a second makes 21,600 in an hour, and passes it, at 360 a minute,
|
||||||
|
under the minute limit. The fleet's 7,200 an hour is about a third of it.
|
||||||
|
- Per day: 300,000, about 3.5 requests a second around the clock. A scraper that
|
||||||
|
keeps up 4 a second makes 345,600 a day, and passes it. At 240 a minute and
|
||||||
|
14,400 an hour, it stays under the other two limits.
|
||||||
|
|
||||||
|
Counting cannot catch a client that stays under all three limits, such as one
|
||||||
|
that keeps up 3 requests a second. Only the other checks can catch an abuser
|
||||||
|
that slow. They include the trap paths, the rule files, the Core Rule Set and
|
||||||
|
the blocklists. An abuser that trips none of them is not caught. A machine that
|
||||||
|
keeps up more than 500 requests a minute passes the minute limit. One that keeps
|
||||||
|
up more than about 5.5 a second for an hour passes the hour limit. One that
|
||||||
|
keeps up more than about 3.5 a second all day passes the day limit, as two busy
|
||||||
|
Actions runners on one address do. Such a machine's address belongs in
|
||||||
|
`SWWAF_RATE_LIMIT_EXEMPT_NETS`.
|
||||||
|
|
||||||
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as
|
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as
|
||||||
static assets or health checks; their bytes are not counted either. A prefix is
|
static assets or health checks; their bytes are not counted either. A prefix is
|
||||||
@@ -397,12 +413,12 @@ matched the same way.
|
|||||||
|
|
||||||
### Byte limits, per client
|
### Byte limits, per client
|
||||||
|
|
||||||
Each of the same 40 people moves about 256 MiB an hour, for ten hours a day. The
|
The hour and day limits sit about twice above what 40 people busy with heavy web
|
||||||
hour and day limits sit about twice above what they move together. A response's
|
apps move together on one address. Each moves about 256 MiB an hour, for ten
|
||||||
bytes are counted when it ends (see step 15). So the minute limit holds two of
|
hours a day. A response's bytes are counted when it ends (see step 15). So the
|
||||||
the largest responses, with their requests, ending in the same minute. No single
|
minute limit holds two of the largest responses, with their requests, ending in
|
||||||
download then breaks a limit. An operator who raises `SWWAF_RESPONSE_MAX_BYTES`
|
the same minute. No single download then breaks a limit. An operator who raises
|
||||||
raises the byte limits with it.
|
`SWWAF_RESPONSE_MAX_BYTES` raises the byte limits with it.
|
||||||
|
|
||||||
### Size and time limits
|
### Size and time limits
|
||||||
|
|
||||||
|
|||||||
+9
-9
@@ -35,34 +35,34 @@ on one line:
|
|||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"instance": "fsn1app1/gitea",
|
"instance": "fsn1app1/gitea",
|
||||||
"time": "2026-10-06T12:00:00.123461Z",
|
"time": "2026-10-06T12:00:50.123461Z",
|
||||||
"event": "ban",
|
"event": "ban",
|
||||||
"client": "203.0.113.9",
|
"client": "203.0.113.9",
|
||||||
"netblock": "203.0.113.9/32",
|
"netblock": "203.0.113.9/32",
|
||||||
"asn": "",
|
"asn": "",
|
||||||
"as_name": "",
|
"as_name": "",
|
||||||
"country": "",
|
"country": "",
|
||||||
"reason": "requests per minute over the limit of 6000",
|
"reason": "requests per minute over the limit of 500",
|
||||||
"detail": {
|
"detail": {
|
||||||
"ban_expires": "2026-10-06T13:00:00.123Z",
|
"ban_expires": "2026-10-06T13:00:50.123Z",
|
||||||
"cause": "limit",
|
"cause": "limit",
|
||||||
"notes": {
|
"notes": {
|
||||||
"asn": "",
|
"asn": "",
|
||||||
"as_name": "",
|
"as_name": "",
|
||||||
"country": "",
|
"country": "",
|
||||||
"kind": "requests",
|
"kind": "requests",
|
||||||
"limit": 6000,
|
"limit": 500,
|
||||||
"window": "minute",
|
"window": "minute",
|
||||||
"count": 6001,
|
"count": 501,
|
||||||
"request": {
|
"request": {
|
||||||
"time": "2026-10-06T12:00:00.123456789Z",
|
"time": "2026-10-06T12:00:50.123456789Z",
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
"host": "app.example",
|
"host": "app.example",
|
||||||
"path": "/owner/repo/commits/branch/main?page=812",
|
"path": "/owner/repo/commits/branch/main?page=501",
|
||||||
"status": 403,
|
"status": 403,
|
||||||
"user_agent": "scraper/1.0"
|
"user_agent": "scraper/1.0"
|
||||||
},
|
},
|
||||||
"requests": 8210,
|
"requests": 501,
|
||||||
"refused": 0,
|
"refused": 0,
|
||||||
"earlier_bans": {
|
"earlier_bans": {
|
||||||
"limit": 0,
|
"limit": 0,
|
||||||
@@ -89,7 +89,7 @@ on one line:
|
|||||||
- `asn`, `as_name` and `country`: the client's, as known when the alert was
|
- `asn`, `as_name` and `country`: the client's, as known when the alert was
|
||||||
raised. They are empty when GeoJS had not answered yet.
|
raised. They are empty when GeoJS had not answered yet.
|
||||||
- `reason`: a short sentence, such as
|
- `reason`: a short sentence, such as
|
||||||
`requests per minute over the limit of 6000` or `listed by a blocklist`.
|
`requests per minute over the limit of 500` or `listed by a blocklist`.
|
||||||
- `detail`: what is particular to the event:
|
- `detail`: what is particular to the event:
|
||||||
- for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed
|
- for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed
|
||||||
under `bans.json` in [`state-files.md`](state-files.md));
|
under `bans.json` in [`state-files.md`](state-files.md));
|
||||||
|
|||||||
+7
-5
@@ -34,11 +34,13 @@ checked again when gitea or the Core Rule Set changes version.
|
|||||||
day. One older than version 0.4 (April 2026) asks for work every 2 seconds,
|
day. One older than version 0.4 (April 2026) asks for work every 2 seconds,
|
||||||
43,200 requests a day. While it runs a job, it reports the job's log and state
|
43,200 requests a day. While it runs a job, it reports the job's log and state
|
||||||
every second instead, two requests a second. So one runner busy all day sends
|
every second instead, two requests a second. So one runner busy all day sends
|
||||||
172,800 requests, under the request limits. Six such runners on one address,
|
120 requests a minute and 172,800 a day, under the request limits. Two such
|
||||||
each busy all day, pass the day limit of 1,000,000 within two days; five stay
|
runners on one address, each busy all day, make 345,600 and pass the day limit
|
||||||
under it. Their address is then banned, their running jobs fail, and each
|
of 300,000. Three busy at once make 21,600 an hour and pass the hour limit of
|
||||||
repeat within a day makes the ban longer. Newer runners ask for work less
|
20,000. Five busy at once make 600 a minute and pass the minute limit of 500.
|
||||||
often. Put the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
|
Their address is then banned, their running jobs fail, and each repeat within
|
||||||
|
a day makes the ban longer. Newer runners ask for work less often. Put the
|
||||||
|
runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
|
||||||
|
|
||||||
## What the Core Rule Set lets through
|
## What the Core Rule Set lets through
|
||||||
|
|
||||||
|
|||||||
+15
-3
@@ -99,9 +99,21 @@ that take `off`, the `_FILE` form, and what each group does and why.
|
|||||||
|
|
||||||
## Request rate limits, per client
|
## Request rate limits, per client
|
||||||
|
|
||||||
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `6000`), `SWWAF_RATE_LIMIT_PER_HOUR`
|
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `500`, about 8 a second): the most
|
||||||
(default `100000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `1000000`): the
|
requests a client may make in a minute. A count or `off`. The default catches,
|
||||||
most requests a client may make in each window. A count or `off`.
|
within seconds, a client firing requests as fast as it can. It leaves room for
|
||||||
|
a fleet of about 20 agents at its busiest, about 120 requests a minute. It
|
||||||
|
also leaves room for a few quick first visits to gitea with an empty cache,
|
||||||
|
about 40 files each.
|
||||||
|
- `SWWAF_RATE_LIMIT_PER_HOUR` (default `20000`, about 5.5 a second): the most
|
||||||
|
requests a client may make in an hour. A count or `off`. The default leaves
|
||||||
|
room for an automation fleet at work. It catches a client that keeps up more
|
||||||
|
than about 5.5 a second for an hour.
|
||||||
|
- `SWWAF_RATE_LIMIT_PER_DAY` (default `300000`, about 3.5 a second): the most
|
||||||
|
requests a client may make in a day. A count or `off`. The default catches a
|
||||||
|
scraper that keeps up 4 a second around the clock.
|
||||||
|
- Counting cannot catch a client that stays under all three limits (see "Request
|
||||||
|
rate limits, per client" in [`SPEC.md`](../SPEC.md)).
|
||||||
- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from
|
- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from
|
||||||
rate limiting and the byte limits, such as `/assets/`.
|
rate limiting and the byte limits, such as `/assets/`.
|
||||||
|
|
||||||
|
|||||||
@@ -506,9 +506,9 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
|||||||
AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""),
|
AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""),
|
||||||
RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""),
|
RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""),
|
||||||
DenyNets: env.netblocks("SWWAF_DENY_NETS", ""),
|
DenyNets: env.netblocks("SWWAF_DENY_NETS", ""),
|
||||||
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "6000"),
|
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "500"),
|
||||||
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "100000"),
|
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "20000"),
|
||||||
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "1000000"),
|
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"),
|
||||||
RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
|
RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
|
||||||
BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "4G"),
|
BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "4G"),
|
||||||
BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"),
|
BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"),
|
||||||
|
|||||||
@@ -262,9 +262,9 @@ func TestDefaults(t *testing.T) {
|
|||||||
UpstreamResponseTimeout: 30 * time.Minute,
|
UpstreamResponseTimeout: 30 * time.Minute,
|
||||||
RequestMaxBytes: 100 << 20,
|
RequestMaxBytes: 100 << 20,
|
||||||
ResponseMaxBytes: 1 << 30,
|
ResponseMaxBytes: 1 << 30,
|
||||||
RateLimitPerMinute: 6000,
|
RateLimitPerMinute: 500,
|
||||||
RateLimitPerHour: 100000,
|
RateLimitPerHour: 20000,
|
||||||
RateLimitPerDay: 1000000,
|
RateLimitPerDay: 300000,
|
||||||
BanResponse: 403,
|
BanResponse: 403,
|
||||||
LimitBanDuration: time.Hour,
|
LimitBanDuration: time.Hour,
|
||||||
LimitBanRepeatWindow: 24 * time.Hour,
|
LimitBanRepeatWindow: 24 * time.Hour,
|
||||||
@@ -2375,9 +2375,9 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
|||||||
allowNets: "",
|
allowNets: "",
|
||||||
rateLimitExemptNets: "",
|
rateLimitExemptNets: "",
|
||||||
denyNets: "",
|
denyNets: "",
|
||||||
rateLimitPerMinute: "6000",
|
rateLimitPerMinute: "500",
|
||||||
rateLimitPerHour: "100000",
|
rateLimitPerHour: "20000",
|
||||||
rateLimitPerDay: "1000000",
|
rateLimitPerDay: "300000",
|
||||||
rateLimitExemptPaths: "",
|
rateLimitExemptPaths: "",
|
||||||
bytesLimitPerMinute: "4G",
|
bytesLimitPerMinute: "4G",
|
||||||
bytesLimitPerHour: "20G",
|
bytesLimitPerHour: "20G",
|
||||||
|
|||||||
@@ -1244,9 +1244,9 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
|
|||||||
"SWWAF_ALLOW_NETS": "",
|
"SWWAF_ALLOW_NETS": "",
|
||||||
rateLimitExemptNets: "",
|
rateLimitExemptNets: "",
|
||||||
"SWWAF_DENY_NETS": "",
|
"SWWAF_DENY_NETS": "",
|
||||||
"SWWAF_RATE_LIMIT_PER_MINUTE": "6000",
|
"SWWAF_RATE_LIMIT_PER_MINUTE": "500",
|
||||||
"SWWAF_RATE_LIMIT_PER_HOUR": "100000",
|
"SWWAF_RATE_LIMIT_PER_HOUR": "20000",
|
||||||
rateLimitPerDay: "1000000",
|
rateLimitPerDay: "300000",
|
||||||
"SWWAF_RATE_LIMIT_EXEMPT_PATHS": "",
|
"SWWAF_RATE_LIMIT_EXEMPT_PATHS": "",
|
||||||
"SWWAF_DENIED_COUNTRIES": "",
|
"SWWAF_DENIED_COUNTRIES": "",
|
||||||
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "",
|
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "",
|
||||||
|
|||||||
Reference in New Issue
Block a user