sneak, in chat on 6 November, on the byte limits proposed in #193:
wait no. 1500MB per hour means someone can’t download a single 3GB file. i think we need to add some decider logic that looks at both byte traffic AND recent request count. a spider that has downloaded 5000 urls today and then starts downloading multiple 2GB files sequentially should be banned
Definition of done (settles once sneak approves the design on this issue):
A client that has made many requests recently gets a much lower byte limit than one that has not. The thresholds are settings, with defaults sneak approves.
The case above is tested: a client that has fetched 5,000 URLs today and then starts downloading 2 GB files is banned. A client that has made a few requests and then downloads a 3 GB file is not.
The README's settings table lists the new settings and their defaults.
make check passes.
Model: opus-5-5
sneak, in chat on 6 November, on the byte limits proposed in https://git.eeqj.de/sneak/smallwebwaf/issues/193:
> wait no. 1500MB per hour means someone can’t download a single 3GB file. i think we need to add some decider logic that looks at both byte traffic AND recent request count. a spider that has downloaded 5000 urls today and then starts downloading multiple 2GB files sequentially should be banned
**Definition of done** (settles once sneak approves the design on this issue):
- A client that has made many requests recently gets a much lower byte limit than one that has not. The thresholds are settings, with defaults sneak approves.
- The case above is tested: a client that has fetched 5,000 URLs today and then starts downloading 2 GB files is banned. A client that has made a few requests and then downloads a 3 GB file is not.
- The README's settings table lists the new settings and their defaults.
- `make check` passes.
Model: opus-5-5
The new rule: a client that has made more than 2,000 requests in the last 24 hours may move at most 2 GB in those 24 hours. The first byte past that bans it, like any other limit.
The spider in the example is banned partway through its first 2 GB file: 5,000 pages plus that file is well over 2 GB.
Someone who browsed a little and then downloads a 3 GB file is not caught: they are far below 2,000 requests.
2,000 is a third of the proposed daily request limit (6,000). A person rarely makes that many requests in a day, and almost never also downloads gigabytes the same day.
The plain byte limits become loose, for clients that make few requests: none a minute, 10 GB an hour, 30 GB a day. Someone can fetch a 3 GB file, or three, in an hour. The daily figure sits inside your 20-40 GB.
Both thresholds are settings, so they can be tuned per site.
Question: build it this way, with these figures? Recommended: yes.
Model: opus-5-5
**Proposal, not yet a ruling:**
- **The new rule:** a client that has made more than 2,000 requests in the last 24 hours may move at most 2 GB in those 24 hours. The first byte past that bans it, like any other limit.
- The spider in the example is banned partway through its first 2 GB file: 5,000 pages plus that file is well over 2 GB.
- Someone who browsed a little and then downloads a 3 GB file is not caught: they are far below 2,000 requests.
- 2,000 is a third of the proposed daily request limit (6,000). A person rarely makes that many requests in a day, and almost never also downloads gigabytes the same day.
- **The plain byte limits** become loose, for clients that make few requests: none a minute, 10 GB an hour, 30 GB a day. Someone can fetch a 3 GB file, or three, in an hour. The daily figure sits inside your 20-40 GB.
Both thresholds are settings, so they can be tuned per site.
**Question:** build it this way, with these figures? Recommended: yes.
Model: opus-5-5
sneak
was assigned by clawbot2026-11-06 18:33:48 +01:00
Amended proposal: the plain byte limits become none a minute, 50 GB an hour and 100 GB a day. They bind only on clients that make few requests, since the new rule (over 2,000 requests in 24 hours means at most 2 GB in that time) catches spiders long before.
One thing those figures allow: a client that only fetches large files, with few requests, can take up to 100 GB a day per address before anything stops it. That is about 9 Mbps averaged over the day, or 50 GB in one hour at about 110 Mbps.
Question: build the new rule with 2,000 requests and 2 GB, and set the plain byte limits to none / 50 GB / 100 GB? Recommended: yes.
Model: opus-5-5
sneak, in chat on 6 November:
> 50GB per hour is a safe bet, and 100GB per day?
**Amended proposal:** the plain byte limits become none a minute, 50 GB an hour and 100 GB a day. They bind only on clients that make few requests, since the new rule (over 2,000 requests in 24 hours means at most 2 GB in that time) catches spiders long before.
One thing those figures allow: a client that only fetches large files, with few requests, can take up to 100 GB a day per address before anything stops it. That is about 9 Mbps averaged over the day, or 50 GB in one hour at about 110 Mbps.
**Question:** build the new rule with 2,000 requests and 2 GB, and set the plain byte limits to none / 50 GB / 100 GB? Recommended: yes.
Model: opus-5-5
Approved by sneak in chat on 6 November ("yes and yes"):
A client that has made more than 2,000 requests in the last 24 hours may move at most 2 GB in that time. The first byte past that bans it. Both figures are settings, with these as defaults.
The plain byte limits default to none a minute, 50 GB an hour and 100 GB a day.
He added, in chat the same day:
make sure the decider’s heuristics are modular in our code. we are going to be analyzing the logs from the real world tomorrow and will be adding many heuristics and tuning others.
That is filed as its own unit, #195. This rule is built after it, as one of its modules.
This is clawbot's now, for smallwebwaf's manager. Its setting names must say what they do (#192).
Model: opus-5-5
Approved by sneak in chat on 6 November ("yes and yes"):
- A client that has made more than 2,000 requests in the last 24 hours may move at most 2 GB in that time. The first byte past that bans it. Both figures are settings, with these as defaults.
- The plain byte limits default to none a minute, 50 GB an hour and 100 GB a day.
He added, in chat the same day:
> make sure the decider’s heuristics are modular in our code. we are going to be analyzing the logs from the real world tomorrow and will be adding many heuristics and tuning others.
That is filed as its own unit, https://git.eeqj.de/sneak/smallwebwaf/issues/195. This rule is built after it, as one of its modules.
This is `clawbot`'s now, for smallwebwaf's manager. Its setting names must say what they do (https://git.eeqj.de/sneak/smallwebwaf/issues/192).
Model: opus-5-5
sneak
was unassigned by clawbot2026-11-06 18:37:54 +01:00
clawbot
self-assigned this 2026-11-06 18:37:54 +01:00
The plain byte defaults (none a minute, 50 GB an hour, 100 GB a day) land with the other defaults in #193. This issue adds only the new rule, as a module, after #195.
Model: opus-5-5
The plain byte defaults (none a minute, 50 GB an hour, 100 GB a day) land with the other defaults in https://git.eeqj.de/sneak/smallwebwaf/issues/193. This issue adds only the new rule, as a module, after https://git.eeqj.de/sneak/smallwebwaf/issues/195.
Model: opus-5-5
Plan for the implementer. Start from next after #195 has landed, and build the rule as one heuristic in its module structure: one new file, internal/servewaf/manyrequestsbytes.go, with its tests in internal/servewaf/manyrequestsbytes_test.go, registered in the list that runs once rproxy's report has come, right after byte_limits. Heuristic name: many_requests_bytes_limit.
Rule: a client that has made more than SWWAF_MANY_REQUESTS_PER_DAY requests in the last day may move at most SWWAF_MANY_REQUESTS_BYTES_LIMIT_PER_DAY bytes in that day. The report whose bytes take it past that bans it, as a byte limit does: same ban, same lengths, same SWWAF_BYTES_COUNT.
Settings (parsed in internal/config, failing at startup on a bad value): SWWAF_MANY_REQUESTS_PER_DAY, default 2000, a count or off (off turns the rule off); SWWAF_MANY_REQUESTS_BYTES_LIMIT_PER_DAY, default 2G, a size. The client's limit percentages do not apply to either.
Counting: use the day request count and day byte count the rate limiter already keeps for the client. Make sure they are kept while this rule is on, even with SWWAF_RATE_LIMIT_PER_DAY or SWWAF_BYTES_LIMIT_PER_DAY set to off. Requests the rate limits do not count (exempt nets and paths) are not counted here either.
Log line:heuristic is many_requests_bytes_limit; heuristic_figures gives the day's requests, the request threshold, the day's bytes and the byte limit.
Tests: the two cases in the definition of done. A client that has made 5,000 requests today and then fetches a 2 GB file is banned when that file's report comes, and its next request is refused. A client that has made 20 requests and then fetches a 3 GB file is not banned. Also: off turns the rule off, and a bad value of either setting fails startup.
Docs: the README settings table and docs/settings.md ("Byte limits, per client") list both settings with their defaults and the rule in one or two plain sentences; SPEC.md's byte-limit section says the same; docs/request-log.md adds the heuristic and its figures.
Disclosure for the PR body: the ban comes when rproxy's report of the response arrives, as for every byte limit, so the response that crosses the limit is delivered whole.
Model: opus-5-5
Plan for the implementer. Start from `next` after https://git.eeqj.de/sneak/smallwebwaf/issues/195 has landed, and build the rule as one heuristic in its module structure: one new file, `internal/servewaf/manyrequestsbytes.go`, with its tests in `internal/servewaf/manyrequestsbytes_test.go`, registered in the list that runs once rproxy's report has come, right after `byte_limits`. Heuristic name: `many_requests_bytes_limit`.
- **Rule:** a client that has made more than `SWWAF_MANY_REQUESTS_PER_DAY` requests in the last day may move at most `SWWAF_MANY_REQUESTS_BYTES_LIMIT_PER_DAY` bytes in that day. The report whose bytes take it past that bans it, as a byte limit does: same ban, same lengths, same `SWWAF_BYTES_COUNT`.
- **Settings** (parsed in `internal/config`, failing at startup on a bad value): `SWWAF_MANY_REQUESTS_PER_DAY`, default `2000`, a count or `off` (`off` turns the rule off); `SWWAF_MANY_REQUESTS_BYTES_LIMIT_PER_DAY`, default `2G`, a size. The client's limit percentages do not apply to either.
- **Counting:** use the day request count and day byte count the rate limiter already keeps for the client. Make sure they are kept while this rule is on, even with `SWWAF_RATE_LIMIT_PER_DAY` or `SWWAF_BYTES_LIMIT_PER_DAY` set to `off`. Requests the rate limits do not count (exempt nets and paths) are not counted here either.
- **Log line:** `heuristic` is `many_requests_bytes_limit`; `heuristic_figures` gives the day's requests, the request threshold, the day's bytes and the byte limit.
- **Tests:** the two cases in the definition of done. A client that has made 5,000 requests today and then fetches a 2 GB file is banned when that file's report comes, and its next request is refused. A client that has made 20 requests and then fetches a 3 GB file is not banned. Also: `off` turns the rule off, and a bad value of either setting fails startup.
- **Docs:** the README settings table and `docs/settings.md` ("Byte limits, per client") list both settings with their defaults and the rule in one or two plain sentences; `SPEC.md`'s byte-limit section says the same; `docs/request-log.md` adds the heuristic and its figures.
Disclosure for the PR body: the ban comes when rproxy's report of the response arrives, as for every byte limit, so the response that crosses the limit is delivered whole.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
sneak, in chat on 6 November, on the byte limits proposed in #193:
Definition of done (settles once sneak approves the design on this issue):
make checkpasses.Model: opus-5-5
Proposal, not yet a ruling:
Both thresholds are settings, so they can be tuned per site.
Question: build it this way, with these figures? Recommended: yes.
Model: opus-5-5
sneak, in chat on 6 November:
Amended proposal: the plain byte limits become none a minute, 50 GB an hour and 100 GB a day. They bind only on clients that make few requests, since the new rule (over 2,000 requests in 24 hours means at most 2 GB in that time) catches spiders long before.
One thing those figures allow: a client that only fetches large files, with few requests, can take up to 100 GB a day per address before anything stops it. That is about 9 Mbps averaged over the day, or 50 GB in one hour at about 110 Mbps.
Question: build the new rule with 2,000 requests and 2 GB, and set the plain byte limits to none / 50 GB / 100 GB? Recommended: yes.
Model: opus-5-5
Approved by sneak in chat on 6 November ("yes and yes"):
He added, in chat the same day:
That is filed as its own unit, #195. This rule is built after it, as one of its modules.
This is
clawbot's now, for smallwebwaf's manager. Its setting names must say what they do (#192).Model: opus-5-5
The plain byte defaults (none a minute, 50 GB an hour, 100 GB a day) land with the other defaults in #193. This issue adds only the new rule, as a module, after #195.
Model: opus-5-5
Plan for the implementer. Start from
nextafter #195 has landed, and build the rule as one heuristic in its module structure: one new file,internal/servewaf/manyrequestsbytes.go, with its tests ininternal/servewaf/manyrequestsbytes_test.go, registered in the list that runs once rproxy's report has come, right afterbyte_limits. Heuristic name:many_requests_bytes_limit.SWWAF_MANY_REQUESTS_PER_DAYrequests in the last day may move at mostSWWAF_MANY_REQUESTS_BYTES_LIMIT_PER_DAYbytes in that day. The report whose bytes take it past that bans it, as a byte limit does: same ban, same lengths, sameSWWAF_BYTES_COUNT.internal/config, failing at startup on a bad value):SWWAF_MANY_REQUESTS_PER_DAY, default2000, a count oroff(offturns the rule off);SWWAF_MANY_REQUESTS_BYTES_LIMIT_PER_DAY, default2G, a size. The client's limit percentages do not apply to either.SWWAF_RATE_LIMIT_PER_DAYorSWWAF_BYTES_LIMIT_PER_DAYset tooff. Requests the rate limits do not count (exempt nets and paths) are not counted here either.heuristicismany_requests_bytes_limit;heuristic_figuresgives the day's requests, the request threshold, the day's bytes and the byte limit.offturns the rule off, and a bad value of either setting fails startup.docs/settings.md("Byte limits, per client") list both settings with their defaults and the rule in one or two plain sentences;SPEC.md's byte-limit section says the same;docs/request-log.mdadds the heuristic and its figures.Disclosure for the PR body: the ban comes when rproxy's report of the response arrives, as for every byte limit, so the response that crosses the limit is delivered whole.
Model: opus-5-5