Merge pull request 'next into main' (#160 ) from next into main

Reviewed-on: #160
2026-10-26 13:11:04 +01:00
9 changed files with 66 additions and 120 deletions
@@ -17,17 +17,11 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below.
 ## What it protects against

 - Scrapers that fetch every page of a site, all day. Rate limiting gives each
-  client 300 requests a minute, 30,000 an hour and 300,000 a day. That leaves
-  room for programmatic use of an API, such as an automation fleet on another
-  host. The minute limit catches, within seconds, a client firing requests as
-  fast as it can. With the minute limit raised or off, the hour limit catches
-  one that keeps up 10 a second. The day limit catches one that keeps up 4 a
-  second around the clock. Counting cannot catch a client that stays under all
-  three limits, such as one that keeps up 3 a second; only the protections below
-  can. Byte limits give each client 4 GiB a minute, 20 GiB an hour and 200 GiB a
-  day. A client over a limit is refused and banned. The first ban lasts an hour.
-  Each repeat within a day makes the next ban three times as long. A ban that
-  would last more than seven days is permanent.
+  client 1000 requests a minute, 10,000 an hour and 50,000 a day. Byte limits
+  give it 10 GiB a minute, 20 GiB an hour and 50 GiB a day. A client over a
+  limit is refused and banned. The first ban lasts an hour. Each repeat within a
+  day makes the next ban three times as long. A ban that would last more than
+  seven days is permanent.
 - Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
  default rule file bans such a client for seven days at its first probe. Its
  next request during the ban makes the ban permanent. The default rule file
@@ -368,39 +368,16 @@ Conventions:
 - `SWWAF_ALLOW_NETS` skips the checks step 3 names, for monitoring or the
  owner's own networks. `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and
  byte limits only; the error burst, attack detection and bans still apply.
- Many people can share one address, as in an office, on a mobile carrier or
-  through Tor. They are one client, and share its limits and its bans. A known
-  shared address busier than the limits allow belongs in `SWWAF_ALLOW_NETS` or
-  `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
+- Many clients share one address (mobile carriers, offices, Tor), and so share
+  its limits and its bans. `SWWAF_ALLOW_NETS` and `SWWAF_RATE_LIMIT_EXEMPT_NETS`
+  take known shared addresses out.

 ### Request rate limits, per client

-Breaking one bans the client. The defaults suit programmatic use of an API, such
-as an automation fleet on another host, and still stop abusive bots. A fleet of
-20 agents, each making a request every 10 seconds at its busiest, makes 120
-requests a minute and 7,200 an hour. Working around the clock, it makes 172,800
-a day. It stays under all three limits.
-
- Per minute: 300, 5 requests a second. A client firing requests as fast as it
-  can passes it within seconds. It leaves room for the fleet and a few quick
-  first visits with an empty cache. A Gitea page opened with an empty cache
-  loads about 40 files.
- Per hour: 30,000, about 8 requests a second for a whole hour. At the default
-  minute limit no client reaches it, since 300 a minute is at most 18,000 an
-  hour. With the minute limit raised or off, a crawler that keeps up 10 a second
-  makes 36,000 an hour, and passes it.
- Per day: 300,000, about 3.5 requests a second around the clock. A crawler that
-  keeps up 4 a second makes 345,600 a day, and passes it. At 240 a minute and
-  14,400 an hour, it stays under the other two limits.
-
-Counting cannot catch a client that stays under all three limits, such as one
-that keeps up 3 requests a second. Only the other checks can catch an abuser
-that slow. They include the trap paths, the rule files, the Core Rule Set and
-the blocklists. An abuser that trips none of them is not caught. A machine that
-keeps up more than 5 requests a second passes the minute limit. One that keeps
-up more than about 3.5 a second all day passes the day limit, as two busy
-Actions runners on one address do. Such a machine's address belongs in
-`SWWAF_RATE_LIMIT_EXEMPT_NETS`.
+Breaking one bans the client, so the defaults sit several times above what one
+busy person produces. A browser loading a heavy page makes a few hundred
+requests, and a git clone a handful. A machine that talks to the app all day can
+still pass the day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.

 `SWWAF_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as
 static assets or health checks; their bytes are not counted either. A prefix is
@@ -413,12 +390,9 @@ matched the same way.

 ### Byte limits, per client

-The hour and day limits sit about twice above what 40 people busy with heavy web
-apps move together on one address. Each moves about 256 MiB an hour, for ten
-hours a day. A response's bytes are counted when it ends (see step 15). So the
-minute limit holds two of the largest responses, with their requests, ending in
-the same minute. No single download then breaks a limit. An operator who raises
-`SWWAF_RESPONSE_MAX_BYTES` raises the byte limits with it.
+A response's bytes are counted when it ends (see step 15). So every default sits
+above the largest request and answer together, and no single download breaks a
+limit.

 ### Size and time limits

@@ -35,34 +35,34 @@ on one line:
 ```json
 {
    "instance": "fsn1app1/gitea",
-    "time": "2026-10-06T12:00:30.123461Z",
+    "time": "2026-10-06T12:00:00.123461Z",
    "event": "ban",
    "client": "203.0.113.9",
    "netblock": "203.0.113.9/32",
    "asn": "",
    "as_name": "",
    "country": "",
-    "reason": "requests per minute over the limit of 300",
+    "reason": "requests per minute over the limit of 1000",
    "detail": {
-        "ban_expires": "2026-10-06T13:00:30.123Z",
+        "ban_expires": "2026-10-06T13:00:00.123Z",
        "cause": "limit",
        "notes": {
            "asn": "",
            "as_name": "",
            "country": "",
            "kind": "requests",
-            "limit": 300,
+            "limit": 1000,
            "window": "minute",
-            "count": 301,
+            "count": 1001,
            "request": {
-                "time": "2026-10-06T12:00:30.123456789Z",
+                "time": "2026-10-06T12:00:00.123456789Z",
                "method": "GET",
                "host": "app.example",
-                "path": "/owner/repo/commits/branch/main?page=301",
+                "path": "/owner/repo/commits/branch/main?page=812",
                "status": 403,
                "user_agent": "scraper/1.0"
            },
-            "requests": 301,
+            "requests": 5210,
            "refused": 0,
            "earlier_bans": {
                "limit": 0,
@@ -89,7 +89,7 @@ on one line:
 - `asn`, `as_name` and `country`: the client's, as known when the alert was
  raised. They are empty when GeoJS had not answered yet.
 - `reason`: a short sentence, such as
-  `requests per minute over the limit of 300` or `listed by a blocklist`.
+  `requests per minute over the limit of 1000` or `listed by a blocklist`.
 - `detail`: what is particular to the event:
    - for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed
      under `bans.json` in [`state-files.md`](state-files.md));
@@ -11,10 +11,8 @@ checked again when gitea or the Core Rule Set changes version.

 - SSH access to gitea does not pass through `smallwebwaf` and is not protected
  by it.
- A clone is a response. It fits the defaults of 30 minutes and 1 GiB unless the
-  repository is large or the link slow. A repository whose clone is larger than
-  1 GiB needs `SWWAF_RESPONSE_MAX_BYTES` raised, and the byte limits with it
-  ("Byte limits, per client" in `SPEC.md`).
+- A clone is a response. It fits the defaults of 30 minutes and 5 GiB for all
+  but the largest repositories and slowest links.
 - A push is a request, and so is every other upload:
    - a git push;
    - an LFS object;
@@ -32,14 +30,12 @@ checked again when gitea or the Core Rule Set changes version.
  request limits do most of the work there.
 - An Actions runner that reaches gitea through `smallwebwaf` sends requests all
  day. One older than version 0.4 (April 2026) asks for work every 2 seconds,
-  43,200 requests a day. While it runs a job, it reports the job's log and state
-  every second instead, two requests a second. So one runner busy all day sends
-  120 requests a minute and 172,800 a day, under the request limits. Two such
-  runners on one address, each busy all day, make 345,600 and pass the day limit
-  of 300,000. Three busy at once make 360 a minute and pass the minute limit
-  of 300. Their address is then banned, their running jobs fail, and each repeat
-  within a day makes the ban longer. Newer runners ask for work less often. Put
-  the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
+  43,200 requests a day, and reports a running job's log and state every second.
+  So it passes the day limit of 50,000 after about an hour and a quarter of
+  jobs. It is then banned, the job it is running fails, and each repeat within a
+  day makes the ban longer. Newer runners ask less often, but a busy one, or
+  several on one address, can still pass it. Put the runners' addresses in
+  `SWWAF_RATE_LIMIT_EXEMPT_NETS`.

 ## What the Core Rule Set lets through

@@ -99,27 +99,16 @@ that take `off`, the `_FILE` form, and what each group does and why.

 ## Request rate limits, per client

- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `300`, 5 a second): the most requests a
-  client may make in a minute. A count or `off`. The default catches, within
-  seconds, a client firing requests as fast as it can. It leaves room for an
-  automation fleet's busiest minute and a few quick first visits with an empty
-  cache.
- `SWWAF_RATE_LIMIT_PER_HOUR` (default `30000`, about 8 a second): the most
-  requests a client may make in an hour. A count or `off`. The default leaves
-  room for an automation fleet at work. With the minute limit raised or off, it
-  catches a crawler that keeps up 10 a second.
- `SWWAF_RATE_LIMIT_PER_DAY` (default `300000`, about 3.5 a second): the most
-  requests a client may make in a day. A count or `off`. The default catches a
-  crawler that keeps up 4 a second around the clock.
- Counting cannot catch a client that stays under all three limits (see "Request
-  rate limits, per client" in [`SPEC.md`](../SPEC.md)).
+- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR`
+  (default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`): the most
+  requests a client may make in each window. A count or `off`.
 - `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from
  rate limiting and the byte limits, such as `/assets/`.

 ## Byte limits, per client

- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `4G`), `SWWAF_BYTES_LIMIT_PER_HOUR`
-  (default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `200G`): the most
+- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`), `SWWAF_BYTES_LIMIT_PER_HOUR`
+  (default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `50G`): the most
  bytes counted for a client in each window. A size or `off`.
 - `SWWAF_BYTES_COUNT` (default `both`): which body bytes the byte limits count:
  `response`, `request` or `both`.
@@ -142,7 +131,7 @@ that take `off`, the `_FILE` form, and what each group does and why.
  to send its whole answer. A duration or `off`.
 - `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. A size
  or `off`.
- `SWWAF_RESPONSE_MAX_BYTES` (default `1G`): the largest response body. A size
+- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. A size
  or `off`.

 ## Lookup of AS number and country
@@ -502,17 +502,17 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
 		UpstreamRequestTimeout:  env.duration("SWWAF_UPSTREAM_REQUEST_TIMEOUT", "60s"),
 		UpstreamResponseTimeout: env.duration("SWWAF_UPSTREAM_RESPONSE_TIMEOUT", "30m"),
 		RequestMaxBytes:         env.size("SWWAF_REQUEST_MAX_BYTES", "100M"),
-		ResponseMaxBytes:        env.size("SWWAF_RESPONSE_MAX_BYTES", "1G"),
+		ResponseMaxBytes:        env.size("SWWAF_RESPONSE_MAX_BYTES", "5G"),
 		AllowNets:               env.netblocks("SWWAF_ALLOW_NETS", ""),
 		RateLimitExemptNets:     env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""),
 		DenyNets:                env.netblocks("SWWAF_DENY_NETS", ""),
-		RateLimitPerMinute:      env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "300"),
-		RateLimitPerHour:        env.count("SWWAF_RATE_LIMIT_PER_HOUR", "30000"),
-		RateLimitPerDay:         env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"),
+		RateLimitPerMinute:      env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "1000"),
+		RateLimitPerHour:        env.count("SWWAF_RATE_LIMIT_PER_HOUR", "10000"),
+		RateLimitPerDay:         env.count("SWWAF_RATE_LIMIT_PER_DAY", "50000"),
 		RateLimitExemptPaths:    env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
-		BytesLimitPerMinute:     env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "4G"),
+		BytesLimitPerMinute:     env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "10G"),
 		BytesLimitPerHour:       env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"),
-		BytesLimitPerDay:        env.size("SWWAF_BYTES_LIMIT_PER_DAY", "200G"),
+		BytesLimitPerDay:        env.size("SWWAF_BYTES_LIMIT_PER_DAY", "50G"),
 		BytesCount:              env.bytesCount("SWWAF_BYTES_COUNT", "both"),
 		LookupSource:            env.lookupSource("SWWAF_LOOKUP_SOURCE", "geojs"),
 		LookupDBPath:            env.value("SWWAF_LOOKUP_DB_PATH", ""),
@@ -261,10 +261,10 @@ func TestDefaults(t *testing.T) {
 		UpstreamRequestTimeout:      time.Minute,
 		UpstreamResponseTimeout:     30 * time.Minute,
 		RequestMaxBytes:             100 << 20,
-		ResponseMaxBytes:            1 << 30,
-		RateLimitPerMinute:          300,
-		RateLimitPerHour:            30000,
-		RateLimitPerDay:             300000,
+		ResponseMaxBytes:            5 << 30,
+		RateLimitPerMinute:          1000,
+		RateLimitPerHour:            10000,
+		RateLimitPerDay:             50000,
 		BanResponse:                 403,
 		LimitBanDuration:            time.Hour,
 		LimitBanRepeatWindow:        24 * time.Hour,
@@ -284,8 +284,8 @@ func TestDefaults(t *testing.T) {
 		LookupSource: defaultLookupSource, LookupTimeout: time.Second,
 	})
 	wantByteLimitSettings(t, cfg, config.Config{
-		BytesLimitPerMinute: 4 << 30, BytesLimitPerHour: 20 << 30,
-		BytesLimitPerDay: 200 << 30, BytesCount: "both",
+		BytesLimitPerMinute: 10 << 30, BytesLimitPerHour: 20 << 30,
+		BytesLimitPerDay: 50 << 30, BytesCount: "both",
 	})

 	if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" {
@@ -2371,17 +2371,17 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
 		upstreamRequestTimeout:  "60s",
 		upstreamResponseTimeout: "30m",
 		requestMaxBytes:         "100M",
-		responseMaxBytes:        "1G",
+		responseMaxBytes:        "5G",
 		allowNets:               "",
 		rateLimitExemptNets:     "",
 		denyNets:                "",
-		rateLimitPerMinute:      "300",
-		rateLimitPerHour:        "30000",
-		rateLimitPerDay:         "300000",
+		rateLimitPerMinute:      "1000",
+		rateLimitPerHour:        "10000",
+		rateLimitPerDay:         "50000",
 		rateLimitExemptPaths:    "",
-		bytesLimitPerMinute:     "4G",
+		bytesLimitPerMinute:     "10G",
 		bytesLimitPerHour:       "20G",
-		bytesLimitPerDay:        "200G",
+		bytesLimitPerDay:        "50G",
 		bytesCount:              "both",
 		lookupSource:            defaultLookupSource,
 		lookupDBPath:            "",
@@ -60,19 +60,8 @@ func TestHealthCheck(t *testing.T) {
 		}, 0, "")
 	}

-	// Another test can take a closed server's port at once, so the app's
-	// address here is a multicast address, to which TCP cannot connect.
-	wantHealthCheck(t, map[string]string{
-		listenAddr:  ":" + port,
-		upstreamURL: "http://224.0.0.1",
-	}, 1, "unhealthy: connect to the app: ")
-
-	// Listening on port 0 takes any free port, so nothing listens on port 0
-	// itself.
-	wantHealthCheck(t, map[string]string{listenAddr: ":0"}, 1,
-		"unhealthy: ask smallwebwaf: ")
-	wantHealthCheck(t, map[string]string{listenAddr: "8080"}, 1,
-		"unhealthy: invalid setting: SWWAF_LISTEN_ADDR: ")
+	app.Close()
+	wantHealthCheck(t, env, 1, "unhealthy: connect to the app: ")

 	stop()

@@ -81,6 +70,10 @@ func TestHealthCheck(t *testing.T) {
 	case <-time.After(waitLimit):
 		t.Fatal("still running after being told to stop")
 	}
+
+	wantHealthCheck(t, env, 1, "unhealthy: ask smallwebwaf: ")
+	wantHealthCheck(t, map[string]string{listenAddr: "8080"}, 1,
+		"unhealthy: invalid setting: SWWAF_LISTEN_ADDR: ")
 }

 func TestHealthCheckRefusesAnArgument(t *testing.T) {
@@ -1240,13 +1240,13 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
 		"SWWAF_UPSTREAM_REQUEST_TIMEOUT":        "60s",
 		"SWWAF_UPSTREAM_RESPONSE_TIMEOUT":       "30m",
 		"SWWAF_REQUEST_MAX_BYTES":               "100M",
-		"SWWAF_RESPONSE_MAX_BYTES":              "1G",
+		"SWWAF_RESPONSE_MAX_BYTES":              "5G",
 		"SWWAF_ALLOW_NETS":                      "",
 		rateLimitExemptNets:                     "",
 		"SWWAF_DENY_NETS":                       "",
-		"SWWAF_RATE_LIMIT_PER_MINUTE":           "300",
-		"SWWAF_RATE_LIMIT_PER_HOUR":             "30000",
-		rateLimitPerDay:                         "300000",
+		"SWWAF_RATE_LIMIT_PER_MINUTE":           "1000",
+		"SWWAF_RATE_LIMIT_PER_HOUR":             "10000",
+		rateLimitPerDay:                         "50000",
 		"SWWAF_RATE_LIMIT_EXEMPT_PATHS":         "",
 		"SWWAF_DENIED_COUNTRIES":                "",
 		"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES":   "",