Rebasing this branch onto next surfaced two failures that the branch
did not have in isolation. Both come from the change this PR makes:
the watcher package now queries live DNS, and it is the second package
to do so.
Burst fan-out in the watcher package. All 13 watcher tests are
parallel and each runs a full iterative resolution, so they hit the
same root servers in the same instant and get rate-limited. This is
exactly the pathology internal/resolver/livedns_test.go was written to
prevent (9cb2c2b); that gate is package-scoped and cannot reach into
this package's test binary, so liveWatcherGate is its counterpart
here, acquired in newTestWatcher and released when the test ends.
Cross-package oversubscription. Go runs package binaries in parallel,
so with both live-DNS packages in flight their gates sum rather than
hold. The excess is rate-limited and the resolver's 8s per-attempt
deadlines expire, failing ResolveIPAddresses tests this branch never
touched. script/test now passes -p 1 so each gate is authoritative
while its package runs. Serialising is also net faster here, because
the retries it removes cost more than the lost parallelism: resolver
16-22s (was 30-36s under contention), watcher 12-13s (was 27-37s),
whole suite 39-46s against the 60s cap and the 90s -timeout backstop.
TestQueryNameserverIP_UnreachableServer is dropped rather than fixed.
It asserted that a query to an RFC 5737 documentation address comes
back non-OK with no records, which does not hold in the build
environment: that network transparently intercepts all UDP/53 traffic
regardless of destination and answers it locally, so the query returns
StatusOK with 9 real records for example.com. Verified directly with
dig @192.0.2.1 inside the build network. The mock DNSClient that used
to force this classification is what this PR removes, and a live
substitute would only be testing the sandbox's network behaviour, so
the coverage gap is recorded in a comment where the test was.
Verified: three consecutive `docker build .` runs green, after three
consecutive failures without these changes.
`script/test` did not pass `-count=1`, so on an unchanged tree Go
served the whole suite from its test cache: exit 0 in ~0.2s with every
package marked `(cached)` and not one DNS query made. This repo's suite
exists to exercise live resolution on every run (`TESTING.md`), so that
green asserted nothing — and it is exactly the green used as evidence
that a flakiness fix works, since "run it a few times" stops being
runs after the first.
`-count=1` now disables caching on every invocation.
The conditional verbose rerun that `REPO_POLICIES.md` mandates was
missing at the same spot and is added here rather than left broken: the
primary run had been unconditionally `-v`, which is the failure mode
the policy exists to prevent (unreadable CI and `docker build` logs on
success). Tests now run quiet, and only a failure triggers the `-v`
rerun. The rerun carries `-count=1` too, so it cannot replay a cached
copy of the failure it is meant to diagnose, and its exit status is
discarded in favour of a forced 1: the first failure already proved the
suite broken, so a flake that passes the second time must not turn the
build green.
`-timeout 90s` is untouched. It is a deliberate backstop that must
strictly exceed the 60s hard cap on suite duration.
No special-casing for the Docker build, which also reaches this script
via `RUN make test`: a fresh container's test cache is empty, so
`-count=1` changes nothing there and carving out an exception would
only create a second code path that could drift.
Verified: three back-to-back `make test` runs on an unchanged tree,
zero `(cached)` markers, ~4.0-4.5s wall each (was ~0.2s cached),
comfortably inside the 20s target with `-race` and `-cover` both still
working and coverage percentages unchanged. The rerun-and-still-fail
path was exercised against a purpose-built flaky test that fails once
then passes: quiet failure, verbose rerun that genuinely re-executed,
exit 1 regardless. `make check` green.
The resolver's live-DNS tests failed nondeterministically, a different
subset each run. Three structural causes, all test-side:
- Burst fan-out. Every test in the package is parallel and the build
hosts have many cores, so all ~35 iterative resolutions started at
the same instant and, because queryServers walks rootServerList() in
fixed order, hit the same root server within milliseconds. Root
servers rate-limit that.
- No retry anywhere. One dropped UDP packet in a delegation chain
failed a test outright.
- Unanimity assertions. TestQueryAllNameservers_AllReturnOK and
_NXDomainFromAllNS required every one of a domain's nameservers to
answer, with no tolerance for one being slow.
New internal/resolver/livedns_test.go addresses each: a package-wide
gate bounds how many live resolutions are in flight at once, every
live operation gets three attempts with exponential backoff and its
own deadline, and multi-nameserver assertions now need a strict
majority rather than unanimity. The retry predicate is deliberately
transport-level -- "did a nameserver answer at all" -- never the
assertion under test, so a resolver that answers incorrectly still
fails on the first attempt. A nameserver that stays silent is
tolerated; one that answers wrongly is not.
livedns_harness_test.go tests that machinery directly: quorum
arithmetic, status counting, the gate's concurrency bound, per-attempt
deadlines, and recovery from a transient failure. It touches no DNS.
Nothing is mocked, faked, stubbed, recorded, skipped or build-tagged,
and production resolver behaviour is unchanged.
Test caps move to the new org-wide values ruled at prompts issue 41:
60s hard cap, 20s target, 90s -timeout backstop. REPO_POLICIES.md is
re-vendored byte-identical from sneak/prompts rather than hand-edited,
which also picks up the golangci-lint paragraph this copy had drifted
behind on. TESTING.md's stale 30-second target follows to 60.
#93
clawbot
sneak