Back off WHOIS lookups more slowly and stop asking for personal data (closes #159)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
A failed lookup now makes the fetcher's wait 1.5 times as long, up to one minute instead of five. A success still halves it. RIPE counts the person and role objects an answer holds against a daily limit per IP address, and refuses an address for good that passes it too often. RIPE, APNIC and AFRINIC are now asked with --no-personal, which leaves those objects out but keeps the organisation object. An answer with RIPE's access-denied or limit-reached error line is now reported as a refusal holding that line, not as an answer with no fields. Model: opus-5-5
This commit was merged in pull request #167.
This commit is contained in:
1 parent
99c7efc156
commit
087b8de9ae
5 files changed
+300
-42
No files matched your search
@@ -21,10 +21,16 @@ const (
|
||||
minInterval = 1 * time.Second
|
||||
|
||||
// maxInterval is the maximum interval after failures (backoff cap).
|
||||
maxInterval = 5 * time.Minute
|
||||
maxInterval = time.Minute
|
||||
|
||||
// backoffMultiplier is how much to multiply interval on failure.
|
||||
backoffMultiplier = 2
|
||||
// A failure multiplies the interval by backoffNumerator divided by
|
||||
// backoffDenominator, 1.5, up to maxInterval.
|
||||
backoffNumerator = 3
|
||||
backoffDenominator = 2
|
||||
|
||||
// A success divides the interval by successDivisor, down to
|
||||
// minInterval.
|
||||
successDivisor = 2
|
||||
|
||||
// whoisStaleThreshold is how old WHOIS data can be before refresh.
|
||||
whoisStaleThreshold = 30 * 24 * time.Hour // 30 days
|
||||
@@ -160,7 +166,7 @@ func (f *ASNFetcher) recordSuccess() {
|
||||
f.consecutiveFails = 0
|
||||
|
||||
// Decrease interval by half, but not below minimum
|
||||
newInterval := max(f.currentInterval/backoffMultiplier, minInterval)
|
||||
newInterval := max(f.currentInterval/successDivisor, minInterval)
|
||||
|
||||
if newInterval != f.currentInterval {
|
||||
f.logger.Debug("Decreased fetch interval",
|
||||
@@ -184,8 +190,9 @@ func (f *ASNFetcher) recordFailure() {
|
||||
f.intervalMu.Lock()
|
||||
f.consecutiveFails++
|
||||
|
||||
// Exponential backoff: multiply by 2, capped at max
|
||||
newInterval := min(f.currentInterval*backoffMultiplier, maxInterval)
|
||||
// Exponential backoff: multiply by 1.5, capped at max
|
||||
newInterval := min(
|
||||
f.currentInterval*backoffNumerator/backoffDenominator, maxInterval)
|
||||
|
||||
if newInterval != f.currentInterval {
|
||||
f.logger.Debug("Increased fetch interval due to failure",
|
||||
|
||||
Reference in new issue
Block a user