Compare commits

..
1 Commits
Author SHA1 Message Date
sneak 8d9bea7c43 README: blocks misbehaving scrapers, not normal ones (closes #387)
check / check (push) Failing after 1m41s
The opening now says a scraper within the default limits is left alone
whatever its user agent, and lists what bans a client: the rate and byte
limits, a probe for a secret path, and the refusal limit.

Model: opus-5-5
2026-11-14 16:40:41 +00:00
+8 -5
View File
@@ -5,17 +5,20 @@
requests of misbehaving scrapers, scanners and other abusers before they reach
an app.
With the default settings, a client is banned when it does any of these:
It does not block scrapers as such. A scraper that stays within the limits below
is left alone, whatever its user agent. With the default settings, a client is
banned when it does any of these:
- makes more than 120 requests in a minute, 1500 in an hour or 6000 in a day;
- moves more than `50G` of request and response bodies in an hour, or `100G` in
a day;
- asks for a secret or backup file, such as `/.env` or `/.git/config`;
- sends a user agent that names a security scanner, such as `sqlmap` or `nikto`;
- asks for a path no real visitor asks for, such as `/.env` or `/.git/config`;
- has more than 30 requests in a minute refused as probes or attacks.
A scraper that does none of these is left alone. A scraper that waits 15 seconds
between requests stays under the rate limits.
A scraper that waits 15 seconds between requests stays under every one of these
limits. The user agent alone gets a client refused only when it names a security
scanner, such as `sqlmap` or `nikto`. Settings you add, such as a refused
country or a blocklist, can refuse any client.
## What it does