sneak, 4 November, in chat. Asked whether to resume smallwebwaf to build the rproxy and servewaf split (#165) and then the setup for running it in front of Gitea, after a read-only review found today's single-process release candidate usable there only with changed settings:
Why would gitea be down when small web WAF is down? The whole point of the reverse proxy is so that if the decider goes down, the reverse proxy can continue reverse proxying and fail open.
Yes. Also, the default rate limits need to be changed to support API usage. So those will need to be updated. I will need to update the request size limit and the request response time. One thing we can do for something like this is we can reduce the response time from 30 minutes. We don't need to support anything downloading anything larger than that. Also, the Sandcastle container that makes all these API calls is on the same host as the Git container, so it would naturally be whitelisted. It's in the same Docker network, and it's accessing traefik on the same box. Regardless, we need to make the rate limits support something like this. If Sandcastle were run on a different host, for example. So they need to be low enough that they will still stop abusive bots, but high enough that they won't interfere with normal programmatic usage. I think perhaps whole day request counts or per hour request counts, rather rate limits, are going to be the best way to do things like this.
What the review found at today's defaults, for reference: uploads over 100 MiB and uploads taking over 60 s are refused; responses over 1 GiB or 30 minutes are cut; the request-rule set refuses ordinary Gitea searches and file names; an address making 6,000 requests a minute, 100,000 an hour or 1,000,000 a day is banned, with longer bans on repeats and then a permanent one; query strings, Gitea's ?token= included, are written to the log whole.
Definition of done
The split in #165 is built: when servewaf is down, slow or overloaded, rproxy keeps proxying and fails open.
The default request limits suit programmatic API use, such as an automation fleet on another host, while still stopping abusive bots. They count requests per minute, hour and day, at 500, 20,000 and 300,000 (sneak, #169 (comment) and #179 (comment)).
The request size limit and the response time limit are set for Gitea. The 30-minute response limit is not raised, since nothing larger needs to download.
Secrets in query strings, such as Gitea's ?token=, never reach the log or ban notes. Struck: query strings are logged verbatim; secrets go in POST data (sneak, #177 (comment)).
The repository carries a setup for running it in front of Gitea behind traefik, with an allow-list for addresses on the same Docker network as Gitea, such as sandcastle. A push, a clone and a web page load pass through it in a test against stock Gitea.
Model: opus-5-5
sneak, 4 November, in chat. Asked whether to resume smallwebwaf to build the rproxy and servewaf split (https://git.eeqj.de/sneak/smallwebwaf/issues/165) and then the setup for running it in front of Gitea, after a read-only review found today's single-process release candidate usable there only with changed settings:
> Why would gitea be down when small web WAF is down? The whole point of the reverse proxy is so that if the decider goes down, the reverse proxy can continue reverse proxying and fail open.
> Yes. Also, the default rate limits need to be changed to support API usage. So those will need to be updated. I will need to update the request size limit and the request response time. One thing we can do for something like this is we can reduce the response time from 30 minutes. We don't need to support anything downloading anything larger than that. Also, the Sandcastle container that makes all these API calls is on the same host as the Git container, so it would naturally be whitelisted. It's in the same Docker network, and it's accessing traefik on the same box. Regardless, we need to make the rate limits support something like this. If Sandcastle were run on a different host, for example. So they need to be low enough that they will still stop abusive bots, but high enough that they won't interfere with normal programmatic usage. I think perhaps whole day request counts or per hour request counts, rather rate limits, are going to be the best way to do things like this.
What the review found at today's defaults, for reference: uploads over 100 MiB and uploads taking over 60 s are refused; responses over 1 GiB or 30 minutes are cut; the request-rule set refuses ordinary Gitea searches and file names; an address making 6,000 requests a minute, 100,000 an hour or 1,000,000 a day is banned, with longer bans on repeats and then a permanent one; query strings, Gitea's ?token= included, are written to the log whole.
## Definition of done
- The split in https://git.eeqj.de/sneak/smallwebwaf/issues/165 is built: when servewaf is down, slow or overloaded, rproxy keeps proxying and fails open.
- The default request limits suit programmatic API use, such as an automation fleet on another host, while still stopping abusive bots. They count requests per minute, hour and day, at 500, 20,000 and 300,000 (sneak, https://git.eeqj.de/sneak/smallwebwaf/issues/169#issuecomment-167897 and https://git.eeqj.de/sneak/smallwebwaf/pulls/179#issuecomment-168078).
- The request size limit and the response time limit are set for Gitea. The 30-minute response limit is not raised, since nothing larger needs to download.
- ~~Secrets in query strings, such as Gitea's ?token=, never reach the log or ban notes.~~ Struck: query strings are logged verbatim; secrets go in POST data (sneak, https://git.eeqj.de/sneak/smallwebwaf/issues/177#issuecomment-167904).
- The repository carries a setup for running it in front of Gitea behind traefik, with an allow-list for addresses on the same Docker network as Gitea, such as sandcastle. A push, a clone and a web page load pass through it in a test against stock Gitea.
Model: opus-5-5
clawbot
self-assigned this 2026-11-04 19:58:49 +01:00
Default request limits, per client (one IPv4 address or one IPv6 /64), as built in #176. Updated for sneak's rulings: a per-minute limit there must be (#179), of 300 to 500 (#169 (comment)), and 20,000 an hour (#179 (comment)). A change to the numbers is a small follow-up.
Per minute: 500 (today 6,000), about 8 a second. A scraper or scanner firing requests as fast as it can passes it within seconds. A fleet of about 20 agents makes about 120 a minute at its busiest, and a person opening a gitea page with an empty cache loads about 40 files, so a few quick first visits fit too.
Per hour: 20,000 (today 100,000), about 5.5 a second for a whole hour. A scraper that keeps up 6 a second passes it within the hour, while staying under the minute limit. The same fleet, each agent making a request every ten seconds while it works, makes about 7,200 an hour, about a third of the limit.
Per day: 300,000 (today 1,000,000), about 3.5 a second around the clock. A scraper that keeps up 4 a second all day passes it within the day, while staying under the other two. The same fleet busy around the clock makes about 172,800, under 60% of it.
What counting cannot catch: a client slower than these, which the trap paths, rule files, Core Rule Set and blocklists are left to catch. The byte limits and ban lengths stay as they are. Actions runners on another host stay best in SWWAF_RATE_LIMIT_EXEMPT_NETS: three busy at once pass the hour limit, and two busy all day pass the day limit.
Model: opus-5-5
Default request limits, per client (one IPv4 address or one IPv6 /64), as built in https://git.eeqj.de/sneak/smallwebwaf/issues/176. Updated for sneak's rulings: a per-minute limit there must be (https://git.eeqj.de/sneak/smallwebwaf/pulls/179), of 300 to 500 (https://git.eeqj.de/sneak/smallwebwaf/issues/169#issuecomment-167897), and 20,000 an hour (https://git.eeqj.de/sneak/smallwebwaf/pulls/179#issuecomment-168078). A change to the numbers is a small follow-up.
- **Per minute: 500** (today 6,000), about 8 a second. A scraper or scanner firing requests as fast as it can passes it within seconds. A fleet of about 20 agents makes about 120 a minute at its busiest, and a person opening a gitea page with an empty cache loads about 40 files, so a few quick first visits fit too.
- **Per hour: 20,000** (today 100,000), about 5.5 a second for a whole hour. A scraper that keeps up 6 a second passes it within the hour, while staying under the minute limit. The same fleet, each agent making a request every ten seconds while it works, makes about 7,200 an hour, about a third of the limit.
- **Per day: 300,000** (today 1,000,000), about 3.5 a second around the clock. A scraper that keeps up 4 a second all day passes it within the day, while staying under the other two. The same fleet busy around the clock makes about 172,800, under 60% of it.
What counting cannot catch: a client slower than these, which the trap paths, rule files, Core Rule Set and blocklists are left to catch. The byte limits and ban lengths stay as they are. Actions runners on another host stay best in `SWWAF_RATE_LIMIT_EXEMPT_NETS`: three busy at once pass the hour limit, and two busy all day pass the day limit.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
sneak, 4 November, in chat. Asked whether to resume smallwebwaf to build the rproxy and servewaf split (#165) and then the setup for running it in front of Gitea, after a read-only review found today's single-process release candidate usable there only with changed settings:
What the review found at today's defaults, for reference: uploads over 100 MiB and uploads taking over 60 s are refused; responses over 1 GiB or 30 minutes are cut; the request-rule set refuses ordinary Gitea searches and file names; an address making 6,000 requests a minute, 100,000 an hour or 1,000,000 a day is banned, with longer bans on repeats and then a permanent one; query strings, Gitea's ?token= included, are written to the log whole.
Definition of done
Secrets in query strings, such as Gitea's ?token=, never reach the log or ban notes.Struck: query strings are logged verbatim; secrets go in POST data (sneak, #177 (comment)).Model: opus-5-5
clawbot referenced this issue2026-11-04 20:09:56 +01:00
Default request limits, per client (one IPv4 address or one IPv6 /64), as built in #176. Updated for sneak's rulings: a per-minute limit there must be (#179), of 300 to 500 (#169 (comment)), and 20,000 an hour (#179 (comment)). A change to the numbers is a small follow-up.
What counting cannot catch: a client slower than these, which the trap paths, rule files, Core Rule Set and blocklists are left to catch. The byte limits and ban lengths stay as they are. Actions runners on another host stay best in
SWWAF_RATE_LIMIT_EXEMPT_NETS: three busy at once pass the hour limit, and two busy all day pass the day limit.Model: opus-5-5
per-minute should be 300-500, no?
the full query strings should be logged, that's correct.
clawbot referenced this issue2026-11-05 02:49:15 +01:00