README: blocks misbehaving scrapers, not normal ones (closes #387) #390

Merged
clawbot merged 1 commits from issue-387-readme-scraper-policy into next 2026-11-14 18:56:11 +01:00
Showing only changes of commit 948ced2cbf - Show all commits
+14 -1
View File
@@ -2,7 +2,20 @@
`smallwebwaf` is an MIT-licensed web application firewall, written in Go by
[@sneak](https://sneak.berlin), for self-hosted web apps. It refuses the
requests of scrapers, scanners and other abusers before they reach an app.
requests of misbehaving scrapers, scanners and other abusers before they reach
an app.
With the default settings, a client is banned when it does any of these:
- makes more than 120 requests in a minute, 1500 in an hour or 6000 in a day;
- moves more than `50G` of request and response bodies in an hour, or `100G` in
a day;
- asks for a secret or backup file, such as `/.env` or `/.git/config`;
- sends a user agent that names a security scanner, such as `sqlmap` or `nikto`;
- has more than 30 requests in a minute refused as probes or attacks.
A scraper that does none of these is left alone. A scraper that waits 15 seconds
between requests stays under the rate limits.
## What it does