Back off WHOIS lookups more slowly and stop asking for personal data (closes #159)
check / check (push) Canceled after 0s

A failed lookup now makes the fetcher's wait 1.5 times as long, up to
one minute instead of five. A success still halves it.

RIPE counts the person and role objects an answer holds against a daily
limit per IP address, and refuses an address for good that passes it
too often. RIPE, APNIC and AFRINIC are now asked with --no-personal,
which leaves those objects out but keeps the organisation object.

An answer with RIPE's access-denied or limit-reached error line is now
reported as a refusal holding that line, not as an answer with no
fields.

Model: opus-5-5
This commit was merged in pull request #167.
This commit is contained in:
clawbot committed 2026-10-26 23:40:13 +01:00
1 parent 99c7efc156
commit 087b8de9ae
5 files changed
+300 -42

No files matched your search

+13 -6
View File
@@ -21,10 +21,16 @@ const (
minInterval = 1 * time.Second
// maxInterval is the maximum interval after failures (backoff cap).
maxInterval = 5 * time.Minute
maxInterval = time.Minute
// backoffMultiplier is how much to multiply interval on failure.
backoffMultiplier = 2
// A failure multiplies the interval by backoffNumerator divided by
// backoffDenominator, 1.5, up to maxInterval.
backoffNumerator = 3
backoffDenominator = 2
// A success divides the interval by successDivisor, down to
// minInterval.
successDivisor = 2
// whoisStaleThreshold is how old WHOIS data can be before refresh.
whoisStaleThreshold = 30 * 24 * time.Hour // 30 days
@@ -160,7 +166,7 @@ func (f *ASNFetcher) recordSuccess() {
f.consecutiveFails = 0
// Decrease interval by half, but not below minimum
newInterval := max(f.currentInterval/backoffMultiplier, minInterval)
newInterval := max(f.currentInterval/successDivisor, minInterval)
if newInterval != f.currentInterval {
f.logger.Debug("Decreased fetch interval",
@@ -184,8 +190,9 @@ func (f *ASNFetcher) recordFailure() {
f.intervalMu.Lock()
f.consecutiveFails++
// Exponential backoff: multiply by 2, capped at max
newInterval := min(f.currentInterval*backoffMultiplier, maxInterval)
// Exponential backoff: multiply by 1.5, capped at max
newInterval := min(
f.currentInterval*backoffNumerator/backoffDenominator, maxInterval)
if newInterval != f.currentInterval {
f.logger.Debug("Increased fetch interval due to failure",