clawbot
30f8f05fd5
check / check (push) Successful in 1m32s
ResolveIPAddresses, which a domain check runs for each nameserver, asked every nameserver of the name's zone for all eight record types and read only A, AAAA and CNAME. It now asks for those three, so a domain check of example.com sends about a third fewer queries. The per-attempt deadline in livednstest goes from 8 to 18 seconds. It was sized for one lookup, while a watcher check sends over a hundred queries in a row and on the CI runner ran past 8 seconds. Three attempts still end under the 60-second cap. A nameserver that answers none of the three queries now counts as not answering even if it would have answered another type; the watcher keeps the previous addresses either way. Model: opus-5-5
125 lines
4.0 KiB
Go
125 lines
4.0 KiB
Go
// Package livednstest runs the live DNS operations of tests. Tests that
|
|
// look something up in DNS query live DNS servers, never a stand-in —
|
|
// see TESTING.md. Nothing here mocks, fakes, stubs, records or replays
|
|
// DNS, and nothing here skips a test: it only changes *how* the live
|
|
// queries are issued, so that a single dropped UDP packet or one slow
|
|
// authoritative server does not turn correct code into a red build.
|
|
//
|
|
// Two mechanisms:
|
|
//
|
|
// 1. Bounded concurrency. Tests run in parallel and the build hosts
|
|
// have many cores, so without a limit every test starts its own
|
|
// iterative resolution at the same instant and they all send their
|
|
// first queries to the root servers within a few milliseconds of
|
|
// each other. Root servers rate-limit that, which shows up as a
|
|
// different arbitrary subset of tests failing on each run. Run caps
|
|
// how many live operations are in flight at once in one test binary.
|
|
//
|
|
// 2. Retry with exponential backoff. Each live operation gets several
|
|
// attempts with its own timeout. An attempt is retried when it
|
|
// obtained nothing to check, never because of what the test
|
|
// asserts about the result, so a wrong result still fails on the
|
|
// first attempt. A fault in the code under test that leaves
|
|
// nothing to check looks the same as live DNS not answering, and
|
|
// fails only after the last attempt.
|
|
package livednstest
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
const (
|
|
// attempts is how many times a live DNS operation is attempted
|
|
// before the test fails.
|
|
attempts = 3
|
|
|
|
// AttemptTimeout bounds one attempt. It must fit the longest
|
|
// operation, a watcher check, which sends over a hundred queries one
|
|
// after another and on a slow build host takes several times as long
|
|
// as the few seconds it takes on a fast one. Worst case for an
|
|
// operation is attempts * AttemptTimeout plus the backoff — about 56
|
|
// seconds, under the suite's 60-second cap and inside the 90-second
|
|
// `go test -timeout` backstop.
|
|
AttemptTimeout = 18 * time.Second
|
|
|
|
// backoffBase is the delay after the first failed attempt; it is
|
|
// multiplied by backoffFactor each time.
|
|
backoffBase = 500 * time.Millisecond
|
|
|
|
// backoffFactor is the exponential backoff multiplier.
|
|
backoffFactor = 2
|
|
|
|
// Concurrency caps how many live operations may be in flight
|
|
// across one test binary at once.
|
|
Concurrency = 6
|
|
)
|
|
|
|
// gate bounds concurrent live operations. It has to be package scoped:
|
|
// the whole point is that it is shared by every parallel test in the
|
|
// test binary.
|
|
//
|
|
//nolint:gochecknoglobals // package-wide live query rate limit
|
|
var gate = make(chan struct{}, Concurrency)
|
|
|
|
// ErrNoAnswer reports that a live operation produced no usable answer,
|
|
// which is retried rather than asserted on.
|
|
var ErrNoAnswer = errors.New("no answer from live DNS")
|
|
|
|
// Run executes one attempt of a live operation, holding a slot in gate
|
|
// for its duration and bounding it with its own timeout.
|
|
func Run(op func(ctx context.Context) error) error {
|
|
gate <- struct{}{}
|
|
defer func() { <-gate }()
|
|
|
|
ctx, cancel := context.WithTimeout(
|
|
context.Background(), AttemptTimeout,
|
|
)
|
|
defer cancel()
|
|
|
|
return op(ctx)
|
|
}
|
|
|
|
// Retry runs op until it reports success, retrying failures with
|
|
// exponential backoff, and fails the test if every attempt fails. op
|
|
// returns an error only for a failure to obtain an answer — never for
|
|
// an answer the test disagrees with, which belongs in an assertion so
|
|
// that it fails immediately. op stores whatever it obtained where its
|
|
// caller can find it.
|
|
func Retry(
|
|
t *testing.T,
|
|
what string,
|
|
op func(ctx context.Context) error,
|
|
) {
|
|
t.Helper()
|
|
|
|
var last error
|
|
|
|
backoff := backoffBase
|
|
|
|
for attempt := range attempts {
|
|
if attempt > 0 {
|
|
t.Logf(
|
|
"%s: attempt %d of %d failed (%v), "+
|
|
"retrying in %s",
|
|
what, attempt, attempts, last, backoff,
|
|
)
|
|
time.Sleep(backoff)
|
|
|
|
backoff *= backoffFactor
|
|
}
|
|
|
|
last = Run(op)
|
|
if last == nil {
|
|
return
|
|
}
|
|
}
|
|
|
|
t.Fatalf(
|
|
"%s: all %d live attempts failed: %v",
|
|
what, attempts, last,
|
|
)
|
|
}
|