sneak/smallwebwaf
1
0 Fork 0
Code Issues 12 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

README: blocks misbehaving scrapers, not normal ones #387

New Issue
Closed
opened 2026-11-14 17:36:35 +01:00 by clawbot · 1 comment
clawbot commented 2026-11-14 17:36:35 +01:00
Collaborator
Copy link
Copy source

sneak, chat, 14 November 2026:

> the readme should also be clear that it blocks misbehaving scrapers but not normal ones that do not negatively affect site operation

Task

Make README.md say plainly, near its opening description, that smallwebwaf blocks scrapers that misbehave (that harm the site's operation) and leaves alone ordinary scrapers that do not. Say in plain words what counts as misbehaving, from what the code at the head of next actually checks (request rates, bans, the rules that apply), with the default thresholds.

If the code does not draw this line (for example it blocks a well-behaved scraper by its user agent alone, or never blocks a fast one), do not write the sentence as fact: post on this issue what the code does instead, and label the issue needs-decision.

Definition of done

  • README.md states the policy, and every claim matches the code on next; or the gap is reported here as above.
  • The prose meets the README rules: one idea per sentence, plain words.
  • make check passes. One PR into next.

Model: opus-5-5

sneak, chat, 14 November 2026: > the readme should also be clear that it blocks misbehaving scrapers but not normal ones that do not negatively affect site operation ## Task Make `README.md` say plainly, near its opening description, that smallwebwaf blocks scrapers that misbehave (that harm the site's operation) and leaves alone ordinary scrapers that do not. Say in plain words what counts as misbehaving, from what the code at the head of `next` actually checks (request rates, bans, the rules that apply), with the default thresholds. If the code does not draw this line (for example it blocks a well-behaved scraper by its user agent alone, or never blocks a fast one), do not write the sentence as fact: post on this issue what the code does instead, and label the issue `needs-decision`. ## Definition of done - `README.md` states the policy, and every claim matches the code on `next`; or the gap is reported here as above. - The prose meets the README rules: one idea per sentence, plain words. - `make check` passes. One PR into `next`. Model: opus-5-5
clawbot commented 2026-11-14 17:40:55 +01:00
Author
Collaborator
Copy link
Copy source

The code draws the line the issue describes. A crawler is never banned for its user agent alone, and a fast one hits the rate limits. The README's opening now states this, with the default limits.

PR: #390 (branch issue-387-readme-scraper-policy).

Model: opus-5-5

The code draws the line the issue describes. A crawler is never banned for its user agent alone, and a fast one hits the rate limits. The README's opening now states this, with the default limits. PR: https://git.eeqj.de/sneak/smallwebwaf/pulls/390 (branch `issue-387-readme-scraper-policy`). Model: opus-5-5
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
critical
Extremely important: data loss, a security hole, production down, or basic use impossible
 merge-ready
Reviewed and ready for the owner to merge
 needs-checks
Checks must be fixed before review
 needs-decision
Waits on an owner decision
 needs-decision
Waiting on a decision by the owner
 needs-decision
needs-rebase
Conflicts with next; rebase required
 needs-review
Review not yet done
 needs-rework
Review found issues to fix
 post-1.0
red-next
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
clawbot sneak (Jeffrey Paul)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/smallwebwaf#387