Back off WHOIS lookups more slowly and stop asking for personal data (closes #159)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
A failed lookup now makes the fetcher's wait 1.5 times as long, up to one minute instead of five. A success still halves it. RIPE counts the person and role objects an answer holds against a daily limit per IP address, and refuses an address for good that passes it too often. RIPE, APNIC and AFRINIC are now asked with --no-personal, which leaves those objects out but keeps the organisation object. An answer with RIPE's access-denied or limit-reached error line is now reported as a refusal holding that line, not as an answer with no fields. Model: opus-5-5
This commit was merged in pull request #167.
This commit is contained in:
@@ -120,17 +120,58 @@ func TestFetchAndUpdateKeepsTheRowWhenTheLookupFails(t *testing.T) {
|
||||
"was stored", stats.NeverFetched)
|
||||
}
|
||||
|
||||
want := baseInterval * backoffNumerator / backoffDenominator
|
||||
|
||||
got := f.GetStats()
|
||||
if got.CurrentInterval != baseInterval*backoffMultiplier ||
|
||||
got.ConsecutiveFails != 1 {
|
||||
if got.CurrentInterval != want || got.ConsecutiveFails != 1 {
|
||||
t.Errorf("interval %v after %d failures, want %v after 1",
|
||||
got.CurrentInterval, got.ConsecutiveFails,
|
||||
baseInterval*backoffMultiplier)
|
||||
got.CurrentInterval, got.ConsecutiveFails, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecordStepsTheInterval has the fetcher fail from its base interval of
|
||||
// 15 seconds, then succeed. Each failure makes the interval 1.5 times as
|
||||
// long, up to one minute, and each success halves it, down to one second.
|
||||
func TestRecordStepsTheInterval(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := NewASNFetcher(newMockStore(), logger.New().Logger)
|
||||
|
||||
for i, want := range []time.Duration{
|
||||
22500 * time.Millisecond,
|
||||
33750 * time.Millisecond,
|
||||
50625 * time.Millisecond,
|
||||
time.Minute,
|
||||
time.Minute,
|
||||
} {
|
||||
f.recordFailure()
|
||||
|
||||
if got := f.getInterval(); got != want {
|
||||
t.Errorf("interval after %d failures = %v, want %v",
|
||||
i+1, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
for i, want := range []time.Duration{
|
||||
30 * time.Second,
|
||||
15 * time.Second,
|
||||
7500 * time.Millisecond,
|
||||
3750 * time.Millisecond,
|
||||
1875 * time.Millisecond,
|
||||
time.Second,
|
||||
time.Second,
|
||||
} {
|
||||
f.recordSuccess()
|
||||
|
||||
if got := f.getInterval(); got != want {
|
||||
t.Errorf("interval after %d successes = %v, want %v",
|
||||
i+1, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFetchAndUpdateMarksAnASWithoutReferral looks up a private-use AS, which
|
||||
// IANA refers to no RIR. The AS must not be looked up again until it is
|
||||
// stale, and the lookup counts as neither a success nor a failure.
|
||||
|
||||
Reference in New Issue
Block a user