README: blocks misbehaving scrapers, not normal ones (closes #387 )

The opening now says a scraper within the default limits is left alone whatever its user agent, and lists what bans a client: the rate and byte limits, a probe for a secret path, and the refusal limit. Model: opus-5-5
waf: refuse request lines and header lines over 8 KiB first (closes #339 )
2026-11-14 16:40:41 +00:00 · 2026-11-13 21:56:09 +01:00
1 changed files with 17 additions and 1 deletions
@@ -2,7 +2,23 @@
 `smallwebwaf` is an MIT-licensed web application firewall, written in Go by
 [@sneak](https://sneak.berlin), for self-hosted web apps. It refuses the
-requests of scrapers, scanners and other abusers before they reach an app.
+requests of misbehaving scrapers, scanners and other abusers before they reach
 an app.
 It does not block scrapers as such. A scraper that stays within the limits below
 is left alone, whatever its user agent. With the default settings, a client is
 banned when it does any of these:
 - makes more than 120 requests in a minute, 1500 in an hour or 6000 in a day;
 - moves more than `50G` of request and response bodies in an hour, or `100G` in
  a day;
 - asks for a path no real visitor asks for, such as `/.env` or `/.git/config`;
 - has more than 30 requests in a minute refused as probes or attacks.
 A scraper that waits 15 seconds between requests stays under every one of these
 limits. The user agent alone gets a client refused only when it names a security
 scanner, such as `sqlmap` or `nikto`. Settings you add, such as a refused
 country or a blocklist, can refuse any client.
 ## What it does