Compare commits

..
Author SHA1 Message Date
sneak 948ced2cbf README: blocks misbehaving scrapers, not normal ones (closes #387)
check / check (push) Failing after 2m12s
The opening now lists what bans a client with the default settings: the
rate and byte limits, a probe for a secret path, a security scanner's
user agent, and the refusal limit. A scraper that does none of these is
left alone.

Model: opus-5-5
2026-11-14 17:10:54 +00:00
+5 -8
View File
@@ -5,20 +5,17 @@
requests of misbehaving scrapers, scanners and other abusers before they reach requests of misbehaving scrapers, scanners and other abusers before they reach
an app. an app.
It does not block scrapers as such. A scraper that stays within the limits below With the default settings, a client is banned when it does any of these:
is left alone, whatever its user agent. With the default settings, a client is
banned when it does any of these:
- makes more than 120 requests in a minute, 1500 in an hour or 6000 in a day; - makes more than 120 requests in a minute, 1500 in an hour or 6000 in a day;
- moves more than `50G` of request and response bodies in an hour, or `100G` in - moves more than `50G` of request and response bodies in an hour, or `100G` in
a day; a day;
- asks for a path no real visitor asks for, such as `/.env` or `/.git/config`; - asks for a secret or backup file, such as `/.env` or `/.git/config`;
- sends a user agent that names a security scanner, such as `sqlmap` or `nikto`;
- has more than 30 requests in a minute refused as probes or attacks. - has more than 30 requests in a minute refused as probes or attacks.
A scraper that waits 15 seconds between requests stays under every one of these A scraper that does none of these is left alone. A scraper that waits 15 seconds
limits. The user agent alone gets a client refused only when it names a security between requests stays under the rate limits.
scanner, such as `sqlmap` or `nikto`. Settings you add, such as a refused
country or a blocklist, can refuse any client.
## What it does ## What it does