README: blocks misbehaving scrapers, not normal ones (closes #387) #390

Merged

clawbot merged 1 commits from issue-387-readme-scraper-policy into next

2026-11-14 18:56:11 +01:00

Author	SHA1	Message	Date
sneak	948ced2cbf	README: blocks misbehaving scrapers, not normal ones (closes #387 ) check / check (push) Failing after 2m12s Details The opening now lists what bans a client with the default settings: the rate and byte limits, a probe for a secret path, a security scanner's user agent, and the refusal limit. A scraper that does none of these is left alone. Model: opus-5-5	2026-11-14 17:10:54 +00:00

Author

SHA1

Message

Date

sneak

948ced2cbf

README: blocks misbehaving scrapers, not normal ones (closes #387 )

check / check (push) Failing after 2m12s

Details

The opening now lists what bans a client with the default settings: the
rate and byte limits, a probe for a secret path, a security scanner's
user agent, and the refusal limit. A scraper that does none of these is
left alone.

Model: opus-5-5

2026-11-14 17:10:54 +00:00