Scrapers that fetch every page of a site, all day. Rate limiting gives each client 1000 requests a minute, 10,000 an hour and 50,000 a day. Byte limits give it 10 GiB a minute, 20 GiB an hour and 50 GiB a day. A client over a limit is refused and banned. The first ban lasts an hour. Each repeat within a day makes the next ban three times as long. A ban that would last more than seven days is permanent.
This is not acceptable for a single machine running selfhosted apps. Where did these figures come from? 16 requests per second? 10GiB a minute? 1300Mbit/sec? WTAF?
> Scrapers that fetch every page of a site, all day. Rate limiting gives each client 1000 requests a minute, 10,000 an hour and 50,000 a day. Byte limits give it 10 GiB a minute, 20 GiB an hour and 50 GiB a day. A client over a limit is refused and banned. The first ban lasts an hour. Each repeat within a day makes the next ban three times as long. A ban that would last more than seven days is permanent.
This is not acceptable for a single machine running selfhosted apps. Where did these figures come from? 16 requests per second? 10GiB a minute? 1300Mbit/sec? WTAF?
clawbot
was assigned by sneak2026-10-26 13:13:39 +01:00
Nobody chose them for a single machine. The fleet picked them when the spec
was written, and you never ruled on them. The old spec (SPEC.md on main,
"Request rate limits" and "Byte limits") gives two reasons:
Breaking a limit bans the client, so the request limits were set "several
times above what one busy person produces": a heavy page in a browser makes a
few hundred requests.
The byte limits were set above the largest allowed response
(SWWAF_RESPONSE_MAX_BYTES, default 5 GiB) plus the largest request, so that
no single download breaks a limit. That is where 10 GiB a minute comes from.
Both reasons assume a busy public site, not a few self-hosted apps.
Question. Should the defaults be sized for one person using a few
self-hosted apps, with a scraper caught within minutes?
Recommendation. Yes:
requests: 300 a minute (5 a second), 2,000 an hour and 10,000 a day;
bytes: 1 GiB a minute, 5 GiB an hour and 20 GiB a day;
the largest response (SWWAF_RESPONSE_MAX_BYTES): 1 GiB, so a single
download still never breaks the minute limit.
The bans stay as they are. Each figure stays a setting, so a bigger site raises
it. Reply with other figures if you want them; I build whichever you choose as
one reviewed change on next.
Model: opus-5-5
Nobody chose them for a single machine. The fleet picked them when the spec
was written, and you never ruled on them. The old spec (`SPEC.md` on `main`,
"Request rate limits" and "Byte limits") gives two reasons:
- Breaking a limit bans the client, so the request limits were set "several
times above what one busy person produces": a heavy page in a browser makes a
few hundred requests.
- The byte limits were set above the largest allowed response
(`SWWAF_RESPONSE_MAX_BYTES`, default 5 GiB) plus the largest request, so that
no single download breaks a limit. That is where 10 GiB a minute comes from.
Both reasons assume a busy public site, not a few self-hosted apps.
**Question.** Should the defaults be sized for one person using a few
self-hosted apps, with a scraper caught within minutes?
**Recommendation.** Yes:
- requests: 300 a minute (5 a second), 2,000 an hour and 10,000 a day;
- bytes: 1 GiB a minute, 5 GiB an hour and 20 GiB a day;
- the largest response (`SWWAF_RESPONSE_MAX_BYTES`): 1 GiB, so a single
download still never breaks the minute limit.
The bans stay as they are. Each figure stays a setting, so a bigger site raises
it. Reply with other figures if you want them; I build whichever you choose as
one reviewed change on `next`.
Model: opus-5-5
clawbot
removed their assignment 2026-10-26 13:21:12 +01:00
sneak
was assigned by clawbot2026-10-26 13:21:12 +01:00
Not one person, no. The services being protected might be legitimately serving thousands of requests per second in total. Remember, we're setting up per-source (IPv4 or /64) limits.
Please choose sane and reasonable defaults. Remember that some services will be used by multiple people behind the same nat simultaneously.
Not one person, no. The services being protected might be legitimately serving thousands of requests per second in total. Remember, we're setting up per-source (IPv4 or /64) limits.
Please choose sane and reasonable defaults. Remember that some services will be used by multiple people behind the same nat simultaneously.
sneak
was unassigned by clawbot2026-10-27 18:25:21 +01:00
clawbot
self-assigned this 2026-10-27 18:25:21 +01:00
Every limit counts one client: one IPv4 address or one IPv6 /64, which can be a whole office behind one NAT. So the new defaults are sized for 40 people on one address at once, each busy with heavy web apps: about 75 requests and 25 MiB a minute, 1,250 requests and 256 MiB an hour, ten hours a day. Each limit sits about twice above what they make together.
SWWAF_RATE_LIMIT_PER_MINUTE: 1000 → 6000 (100 a second). The 40 people make about 3,000 in a busy minute, and a flood faster than 100 a second is banned within the minute.
SWWAF_RATE_LIMIT_PER_HOUR: 10000 → 100000 (28 a second on average). They make about 50,000 in an hour, and a scraper fetching 50 pages a second is banned within 35 minutes.
SWWAF_RATE_LIMIT_PER_DAY: 50000 → 1000000 (12 a second on average). They make about 500,000 in a ten-hour day, and a scraper fetching 20 pages a second is banned within 14 hours.
SWWAF_RESPONSE_MAX_BYTES: 5G → 1G. A web app's pages and most of its files are far smaller, and a lower largest response lets the minute's byte limit come down while one download still never breaks a limit.
SWWAF_BYTES_LIMIT_PER_MINUTE: 10G → 4G (570 Mbit/s). It holds the 40 people's busy minute (1 GiB) plus two of the largest responses finishing in it, and a client pulling faster than 570 Mbit/s is banned within the minute.
SWWAF_BYTES_LIMIT_PER_HOUR: 20G, unchanged (48 Mbit/s on average). They move about 10 GiB in an hour, and a client pulling 100 Mbit/s is banned within half an hour.
SWWAF_BYTES_LIMIT_PER_DAY: 50G → 200G (20 Mbit/s on average). They move about 100 GiB in a ten-hour day, twice the old limit.
SWWAF_REQUEST_MAX_BYTES stays 100M, and the bans stay as they are. A client slower than those 40 people cannot be told from them by counting; the trap paths, rule files and blocklists are what catch it. Each figure stays a setting. I am building these now as one reviewed change on next.
Model: opus-5-5
Every limit counts one client: one IPv4 address or one IPv6 /64, which can be a whole office behind one NAT. So the new defaults are sized for 40 people on one address at once, each busy with heavy web apps: about 75 requests and 25 MiB a minute, 1,250 requests and 256 MiB an hour, ten hours a day. Each limit sits about twice above what they make together.
- `SWWAF_RATE_LIMIT_PER_MINUTE`: `1000` → `6000` (100 a second). The 40 people make about 3,000 in a busy minute, and a flood faster than 100 a second is banned within the minute.
- `SWWAF_RATE_LIMIT_PER_HOUR`: `10000` → `100000` (28 a second on average). They make about 50,000 in an hour, and a scraper fetching 50 pages a second is banned within 35 minutes.
- `SWWAF_RATE_LIMIT_PER_DAY`: `50000` → `1000000` (12 a second on average). They make about 500,000 in a ten-hour day, and a scraper fetching 20 pages a second is banned within 14 hours.
- `SWWAF_RESPONSE_MAX_BYTES`: `5G` → `1G`. A web app's pages and most of its files are far smaller, and a lower largest response lets the minute's byte limit come down while one download still never breaks a limit.
- `SWWAF_BYTES_LIMIT_PER_MINUTE`: `10G` → `4G` (570 Mbit/s). It holds the 40 people's busy minute (1 GiB) plus two of the largest responses finishing in it, and a client pulling faster than 570 Mbit/s is banned within the minute.
- `SWWAF_BYTES_LIMIT_PER_HOUR`: `20G`, unchanged (48 Mbit/s on average). They move about 10 GiB in an hour, and a client pulling 100 Mbit/s is banned within half an hour.
- `SWWAF_BYTES_LIMIT_PER_DAY`: `50G` → `200G` (20 Mbit/s on average). They move about 100 GiB in a ten-hour day, twice the old limit.
`SWWAF_REQUEST_MAX_BYTES` stays `100M`, and the bans stay as they are. A client slower than those 40 people cannot be told from them by counting; the trap paths, rule files and blocklists are what catch it. Each figure stays a setting. I am building these now as one reviewed change on `next`.
Model: opus-5-5
Built as #162: the new defaults, the tests that pin them, and the reasoning in SPEC.md with the figures in README.md and docs/.
Model: opus-5-5
Built as https://git.eeqj.de/sneak/smallwebwaf/pulls/162: the new defaults, the tests that pin them, and the reasoning in `SPEC.md` with the figures in `README.md` and `docs/`.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
This is not acceptable for a single machine running selfhosted apps. Where did these figures come from? 16 requests per second? 10GiB a minute? 1300Mbit/sec? WTAF?
Nobody chose them for a single machine. The fleet picked them when the spec
was written, and you never ruled on them. The old spec (
SPEC.mdonmain,"Request rate limits" and "Byte limits") gives two reasons:
times above what one busy person produces": a heavy page in a browser makes a
few hundred requests.
(
SWWAF_RESPONSE_MAX_BYTES, default 5 GiB) plus the largest request, so thatno single download breaks a limit. That is where 10 GiB a minute comes from.
Both reasons assume a busy public site, not a few self-hosted apps.
Question. Should the defaults be sized for one person using a few
self-hosted apps, with a scraper caught within minutes?
Recommendation. Yes:
SWWAF_RESPONSE_MAX_BYTES): 1 GiB, so a singledownload still never breaks the minute limit.
The bans stay as they are. Each figure stays a setting, so a bigger site raises
it. Reply with other figures if you want them; I build whichever you choose as
one reviewed change on
next.Model: opus-5-5
Not one person, no. The services being protected might be legitimately serving thousands of requests per second in total. Remember, we're setting up per-source (IPv4 or /64) limits.
Please choose sane and reasonable defaults. Remember that some services will be used by multiple people behind the same nat simultaneously.
Every limit counts one client: one IPv4 address or one IPv6 /64, which can be a whole office behind one NAT. So the new defaults are sized for 40 people on one address at once, each busy with heavy web apps: about 75 requests and 25 MiB a minute, 1,250 requests and 256 MiB an hour, ten hours a day. Each limit sits about twice above what they make together.
SWWAF_RATE_LIMIT_PER_MINUTE:1000→6000(100 a second). The 40 people make about 3,000 in a busy minute, and a flood faster than 100 a second is banned within the minute.SWWAF_RATE_LIMIT_PER_HOUR:10000→100000(28 a second on average). They make about 50,000 in an hour, and a scraper fetching 50 pages a second is banned within 35 minutes.SWWAF_RATE_LIMIT_PER_DAY:50000→1000000(12 a second on average). They make about 500,000 in a ten-hour day, and a scraper fetching 20 pages a second is banned within 14 hours.SWWAF_RESPONSE_MAX_BYTES:5G→1G. A web app's pages and most of its files are far smaller, and a lower largest response lets the minute's byte limit come down while one download still never breaks a limit.SWWAF_BYTES_LIMIT_PER_MINUTE:10G→4G(570 Mbit/s). It holds the 40 people's busy minute (1 GiB) plus two of the largest responses finishing in it, and a client pulling faster than 570 Mbit/s is banned within the minute.SWWAF_BYTES_LIMIT_PER_HOUR:20G, unchanged (48 Mbit/s on average). They move about 10 GiB in an hour, and a client pulling 100 Mbit/s is banned within half an hour.SWWAF_BYTES_LIMIT_PER_DAY:50G→200G(20 Mbit/s on average). They move about 100 GiB in a ten-hour day, twice the old limit.SWWAF_REQUEST_MAX_BYTESstays100M, and the bans stay as they are. A client slower than those 40 people cannot be told from them by counting; the trap paths, rule files and blocklists are what catch it. Each figure stays a setting. I am building these now as one reviewed change onnext.Model: opus-5-5
Built as #162: the new defaults, the tests that pin them, and the reasoning in
SPEC.mdwith the figures inREADME.mdanddocs/.Model: opus-5-5