Back off WHOIS lookups more slowly and stop asking for personal data (closes #159) #167

Merged
clawbot merged 1 commits from issue-159-whois-backoff-and-ripe-query into next 2026-10-26 23:40:14 +01:00
Collaborator

Implements #159.

  • A failed lookup makes the fetcher's wait 1.5 times as long, up to one minute instead of five. A success still halves it, down to one second.
  • RIPE counts the person and role objects an answer holds against a daily limit per IP address. Role objects for abuse contacts do not count. RIPE refuses an address for good that passes the limit too often.
  • RIPE, APNIC and AFRINIC are now asked with --no-personal, not -r as the plan said. -r also leaves out the organisation object, which holds the organisation's name and country. APNIC and AFRINIC list --no-personal in their answers to the query help.
  • An answer holding %ERROR:201: or %ERROR:202: now fails as "RIR refused the query: <the RIR's line>". These are the access errors in RIPE's documentation, and APNIC and AFRINIC run the same software.

Disclosures:

  • No field RIPE filled before is now empty. The address no longer holds the addresses of person and role objects.
  • ARIN and LACNIC are unchanged: their documentation names no personal-data limit and no refusal line.
  • Judgement call: APNIC documents no daily limit, but gets --no-personal because it runs RIPE's software.
  • Unverified: RIPE refuses the build host, so the RIPE answer in the tests is built from RIPE's documentation and server source.
  • The README states none of the fetcher's intervals, so it is unchanged.

Model: opus-5-5

Implements https://git.eeqj.de/sneak/routewatch/issues/159. - A failed lookup makes the fetcher's wait 1.5 times as long, up to one minute instead of five. A success still halves it, down to one second. - RIPE counts the person and role objects an answer holds against a daily limit per IP address. Role objects for abuse contacts do not count. RIPE refuses an address for good that passes the limit too often. - RIPE, APNIC and AFRINIC are now asked with `--no-personal`, not `-r` as the plan said. `-r` also leaves out the organisation object, which holds the organisation's name and country. APNIC and AFRINIC list `--no-personal` in their answers to the query `help`. - An answer holding `%ERROR:201:` or `%ERROR:202:` now fails as "RIR refused the query: &lt;the RIR's line&gt;". These are the access errors in RIPE's documentation, and APNIC and AFRINIC run the same software. Disclosures: - No field RIPE filled before is now empty. The address no longer holds the addresses of person and role objects. - ARIN and LACNIC are unchanged: their documentation names no personal-data limit and no refusal line. - Judgement call: APNIC documents no daily limit, but gets `--no-personal` because it runs RIPE's software. - Unverified: RIPE refuses the build host, so the RIPE answer in the tests is built from RIPE's documentation and server source. - The README states none of the fetcher's intervals, so it is unchanged. Model: opus-5-5
clawbot added the needs-review label 2026-10-26 23:10:11 +01:00
clawbot self-assigned this 2026-10-26 23:10:11 +01:00
Author
Collaborator

FAIL: needs-rework

  1. internal/whois/whois.go, lines 62 to 68 and 168 to 181. The second query rests on a false premise. The comment says -r "is the only such flag APNIC and AFRINIC document", but both servers list --no-personal ("Filter PERSON and ROLE objects from results") in their answer to the query help. The cited AFRINIC page also sends readers to help for the full list of flags. Asked --no-personal AS&lt;n&gt;, APNIC and AFRINIC return the AS's object and its organisation object in one answer, as RIPE does. The second query therefore adds a query per AS at two RIRs, and a second way for each lookup to fail, for nothing. Acceptable: rirQuery sends --no-personal to RIPE, APNIC and AFRINIC, with a comment citing each server's help answer and the AFRINIC page. The organisation query, noReferencedFlag, their test and fixtures go, and the TODO entry, commit message and PR body are corrected to match.

  2. internal/whois/whois.go, line 176. A failed organisation query returns no result, so the AS's name, organisation ID, abuse address and dates, already read from the first answer, are thrown away. The lookup counts as a failure. At APNIC and AFRINIC this gives each AS two chances to fail, which works against the issue's goal of fewer failures. Acceptable: no second query (finding 1). If one stays, a failed one must not throw away the AS's fields, and a test must show it.

  3. internal/whois/whois.go, lines 71 to 74. The error lines are cited by the index page of RIPE's appendices, not by the page that lists them. Acceptable: cite https://docs.db.ripe.net/Appendices/Appendix-C--RIPE-Database-Query-Server-Response-Codes-and-Messages/ directly.

Judgement call: I take each server's answer to the query help as that RIR's own documentation. Neither RIR has a web page listing --no-personal.

Model: opus-5-5

FAIL: `needs-rework` 1. `internal/whois/whois.go`, lines 62 to 68 and 168 to 181. The second query rests on a false premise. The comment says `-r` "is the only such flag APNIC and AFRINIC document", but both servers list `--no-personal` ("Filter PERSON and ROLE objects from results") in their answer to the query `help`. The cited AFRINIC page also sends readers to `help` for the full list of flags. Asked `--no-personal AS&lt;n&gt;`, APNIC and AFRINIC return the AS's object and its organisation object in one answer, as RIPE does. The second query therefore adds a query per AS at two RIRs, and a second way for each lookup to fail, for nothing. Acceptable: `rirQuery` sends `--no-personal` to RIPE, APNIC and AFRINIC, with a comment citing each server's `help` answer and the AFRINIC page. The organisation query, `noReferencedFlag`, their test and fixtures go, and the TODO entry, commit message and PR body are corrected to match. 2. `internal/whois/whois.go`, line 176. A failed organisation query returns no result, so the AS's name, organisation ID, abuse address and dates, already read from the first answer, are thrown away. The lookup counts as a failure. At APNIC and AFRINIC this gives each AS two chances to fail, which works against the issue's goal of fewer failures. Acceptable: no second query (finding 1). If one stays, a failed one must not throw away the AS's fields, and a test must show it. 3. `internal/whois/whois.go`, lines 71 to 74. The error lines are cited by the index page of RIPE's appendices, not by the page that lists them. Acceptable: cite https://docs.db.ripe.net/Appendices/Appendix-C--RIPE-Database-Query-Server-Response-Codes-and-Messages/ directly. Judgement call: I take each server's answer to the query `help` as that RIR's own documentation. Neither RIR has a web page listing `--no-personal`. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-26 23:21:15 +01:00
clawbot added 1 commit 2026-10-26 23:26:38 +01:00
A failed lookup now makes the fetcher's wait 1.5 times as long, up to
one minute instead of five. A success still halves it.

RIPE counts the person and role objects an answer holds against a daily
limit per IP address, and refuses an address for good that passes it
too often. RIPE, APNIC and AFRINIC are now asked with --no-personal,
which leaves those objects out but keeps the organisation object.

An answer with RIPE's access-denied or limit-reached error line is now
reported as a refusal holding that line, not as an answer with no
fields.

Model: opus-5-5
clawbot force-pushed issue-159-whois-backoff-and-ripe-query from 56549d2a0c to 080b641ce6 2026-10-26 23:26:38 +01:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-26 23:27:04 +01:00
Author
Collaborator
  1. Fixed: RIPE, APNIC and AFRINIC all get --no-personal AS&lt;n&gt;, with a comment citing RIPE's page, both servers' help answers and the AFRINIC page. The organisation query, noReferencedFlag, their test and fixtures are gone, and a test now shows APNIC and AFRINIC get --no-personal. TODO.md, the commit message and the PR body match.
  2. Fixed by 1: LookupASN sends the RIR one query, so no second answer can throw away fields already read.
  3. Fixed: the comment cites the Appendix C page directly, its URL split over two comment lines to stay within the line-length rule.

Also fixed: the comment no longer says RIPE counts every person and role object. Role objects for abuse contacts do not count.

Model: opus-5-5

1. Fixed: RIPE, APNIC and AFRINIC all get `--no-personal AS&lt;n&gt;`, with a comment citing RIPE's page, both servers' `help` answers and the AFRINIC page. The organisation query, `noReferencedFlag`, their test and fixtures are gone, and a test now shows APNIC and AFRINIC get `--no-personal`. `TODO.md`, the commit message and the PR body match. 2. Fixed by 1: `LookupASN` sends the RIR one query, so no second answer can throw away fields already read. 3. Fixed: the comment cites the Appendix C page directly, its URL split over two comment lines to stay within the line-length rule. Also fixed: the comment no longer says RIPE counts every person and role object. Role objects for abuse contacts do not count. Model: opus-5-5
Author
Collaborator

PASS: the backoff, the --no-personal queries to RIPE, APNIC and AFRINIC, and the refusal error meet #159 and fix every finding of the first review.

  • Judgement call: APNIC has no web page listing --no-personal, so I take its answer to the query help as its documentation.
  • Judgement call: the Appendix C URL is split over two comment lines because the line-length rule forbids it on one; the comment says so.
  • Unverified: RIPE's live answer to --no-personal, because RIPE refuses this host. I checked it against RIPE's documentation and server source instead.

Model: opus-5-5

PASS: the backoff, the `--no-personal` queries to RIPE, APNIC and AFRINIC, and the refusal error meet https://git.eeqj.de/sneak/routewatch/issues/159 and fix every finding of the first review. - Judgement call: APNIC has no web page listing `--no-personal`, so I take its answer to the query `help` as its documentation. - Judgement call: the Appendix C URL is split over two comment lines because the line-length rule forbids it on one; the comment says so. - Unverified: RIPE's live answer to `--no-personal`, because RIPE refuses this host. I checked it against RIPE's documentation and server source instead. Model: opus-5-5
clawbot merged commit 087b8de9ae into next 2026-10-26 23:40:14 +01:00
clawbot deleted branch issue-159-whois-backoff-and-ripe-query 2026-10-26 23:40:15 +01:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/routewatch#167