Compare commits

...
Author SHA1 Message Date
sneak 8d9bea7c43 README: blocks misbehaving scrapers, not normal ones (closes #387)
check / check (push) Failing after 1m41s
The opening now says a scraper within the default limits is left alone
whatever its user agent, and lists what bans a client: the rate and byte
limits, a probe for a secret path, and the refusal limit.

Model: opus-5-5
2026-11-14 16:40:41 +00:00
+17 -1
View File
@@ -2,7 +2,23 @@
`smallwebwaf` is an MIT-licensed web application firewall, written in Go by
[@sneak](https://sneak.berlin), for self-hosted web apps. It refuses the
requests of scrapers, scanners and other abusers before they reach an app.
requests of misbehaving scrapers, scanners and other abusers before they reach
an app.
It does not block scrapers as such. A scraper that stays within the limits below
is left alone, whatever its user agent. With the default settings, a client is
banned when it does any of these:
- makes more than 120 requests in a minute, 1500 in an hour or 6000 in a day;
- moves more than `50G` of request and response bodies in an hour, or `100G` in
a day;
- asks for a path no real visitor asks for, such as `/.env` or `/.git/config`;
- has more than 30 requests in a minute refused as probes or attacks.
A scraper that waits 15 seconds between requests stays under every one of these
limits. The user agent alone gets a client refused only when it names a security
scanner, such as `sqlmap` or `nikto`. Settings you add, such as a refused
country or a blocklist, can refuse any client.
## What it does