clawbot
dba932c9e3
check / check (push) Successful in 1m11s
A domain check looked up each nameserver's addresses by asking every nameserver of that name's zone for all eight record types; it now asks only for A, AAAA and CNAME, the ones it reads. The watcher tests now check example.org instead of cloudflare.com (two nameservers instead of five) and desec.io instead of example.com (its nameservers are in zones with two, not cloudflare.com's five). The record change and NS failure tests start from saved state built on one NS lookup instead of a first full check, and the port change test runs only the port checks again. A live test attempt may take 18 seconds, not 8. A new live test checks that a nameserver's addresses include IPv4 and IPv6. Model: opus-5-5
129 lines
4.2 KiB
Go
129 lines
4.2 KiB
Go
// Package livednstest runs the live DNS operations of tests. Tests that
|
|
// look something up in DNS query live DNS servers, never a stand-in —
|
|
// see TESTING.md. Nothing here mocks, fakes, stubs, records or replays
|
|
// DNS, and nothing here skips a test: it only changes *how* the live
|
|
// queries are issued, so that a single dropped UDP packet or one slow
|
|
// authoritative server does not turn correct code into a red build.
|
|
//
|
|
// Two mechanisms:
|
|
//
|
|
// 1. Bounded concurrency. Tests run in parallel and the build hosts
|
|
// have many cores, so without a limit every test starts its own
|
|
// iterative resolution at the same instant and they all send their
|
|
// first queries to the root servers within a few milliseconds of
|
|
// each other. Root servers rate-limit that, which shows up as a
|
|
// different arbitrary subset of tests failing on each run. Run caps
|
|
// how many live operations are in flight at once in one test binary.
|
|
//
|
|
// 2. Retry with exponential backoff. Each live operation gets several
|
|
// attempts with its own timeout. An attempt is retried when it
|
|
// obtained nothing to check, never because of what the test
|
|
// asserts about the result, so a wrong result still fails on the
|
|
// first attempt. A fault in the code under test that leaves
|
|
// nothing to check looks the same as live DNS not answering, and
|
|
// fails only after the last attempt.
|
|
package livednstest
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
const (
|
|
// attempts is how many times a live DNS operation is attempted
|
|
// before the test fails.
|
|
attempts = 3
|
|
|
|
// AttemptTimeout bounds one attempt. It must fit the longest
|
|
// operation, a watcher check, which sends over a hundred queries one
|
|
// after another and on a slow build host takes several times as long
|
|
// as the few seconds it takes on a fast one. An operation whose
|
|
// every attempt fails takes attempts * AttemptTimeout plus the
|
|
// backoff, about 56 seconds, after it waits for one of the
|
|
// Concurrency slots that every live operation in the test binary
|
|
// shares. So when live DNS does not answer at all, a test binary
|
|
// with more live operations than slots runs into the 90-second
|
|
// `go test -timeout` backstop instead of each test failing on its
|
|
// own.
|
|
AttemptTimeout = 18 * time.Second
|
|
|
|
// backoffBase is the delay after the first failed attempt; it is
|
|
// multiplied by backoffFactor each time.
|
|
backoffBase = 500 * time.Millisecond
|
|
|
|
// backoffFactor is the exponential backoff multiplier.
|
|
backoffFactor = 2
|
|
|
|
// Concurrency caps how many live operations may be in flight
|
|
// across one test binary at once.
|
|
Concurrency = 6
|
|
)
|
|
|
|
// gate bounds concurrent live operations. It has to be package scoped:
|
|
// the whole point is that it is shared by every parallel test in the
|
|
// test binary.
|
|
//
|
|
//nolint:gochecknoglobals // package-wide live query rate limit
|
|
var gate = make(chan struct{}, Concurrency)
|
|
|
|
// ErrNoAnswer reports that a live operation produced no usable answer,
|
|
// which is retried rather than asserted on.
|
|
var ErrNoAnswer = errors.New("no answer from live DNS")
|
|
|
|
// Run executes one attempt of a live operation, holding a slot in gate
|
|
// for its duration and bounding it with its own timeout.
|
|
func Run(op func(ctx context.Context) error) error {
|
|
gate <- struct{}{}
|
|
defer func() { <-gate }()
|
|
|
|
ctx, cancel := context.WithTimeout(
|
|
context.Background(), AttemptTimeout,
|
|
)
|
|
defer cancel()
|
|
|
|
return op(ctx)
|
|
}
|
|
|
|
// Retry runs op until it reports success, retrying failures with
|
|
// exponential backoff, and fails the test if every attempt fails. op
|
|
// returns an error only for a failure to obtain an answer — never for
|
|
// an answer the test disagrees with, which belongs in an assertion so
|
|
// that it fails immediately. op stores whatever it obtained where its
|
|
// caller can find it.
|
|
func Retry(
|
|
t *testing.T,
|
|
what string,
|
|
op func(ctx context.Context) error,
|
|
) {
|
|
t.Helper()
|
|
|
|
var last error
|
|
|
|
backoff := backoffBase
|
|
|
|
for attempt := range attempts {
|
|
if attempt > 0 {
|
|
t.Logf(
|
|
"%s: attempt %d of %d failed (%v), "+
|
|
"retrying in %s",
|
|
what, attempt, attempts, last, backoff,
|
|
)
|
|
time.Sleep(backoff)
|
|
|
|
backoff *= backoffFactor
|
|
}
|
|
|
|
last = Run(op)
|
|
if last == nil {
|
|
return
|
|
}
|
|
}
|
|
|
|
t.Fatalf(
|
|
"%s: all %d live attempts failed: %v",
|
|
what, attempts, last,
|
|
)
|
|
}
|