clawbot
aa3da062a2
check / check (push) Failing after 1m58s
The DNS-mock removal deleted TestQueryNameserverIP_Timeout and left a comment in its place, so the resolver's StatusTimeout / StatusError classification branch went untested. The stated obstacle was that a query to a black-holed RFC 5737 address comes back StatusOK, because the build environment transparently intercepts UDP/53 and answers it locally. That is a property of that environment, not of the resolver, and it only rules out choosing a remote address. internal/resolver/transport_test.go binds real nameservers on 127.0.0.1 instead and aims the query at them: one silent on A queries and answering every other type (StatusTimeout), one answering SERVFAIL (StatusError), and one address with nothing listening, which is refused rather than dropped and so classifies as NoData. This is not a mock — no DNSClient is substituted. The resolver dials a real socket, writes a real query with the real miekg/dns client, and applies its real deadline and real classification logic to what comes back. Substituting the client is what TESTING.md bans; choosing the server is not, and the resolver is aimed at a caller-chosen nameserver in production too. queryDNS now dials a nameserver address that already carries a port as written, defaulting to 53 only for a bare address. That is what makes a nameserver on any other port reachable, on loopback or otherwise. Silence on one record type rather than all eight keeps the timeout test to two query timeouts (4s) instead of sixteen (32s), and it is asserted: the test fails if it ever costs more than 8s. The watcher's assertStatePopulated and TestDomainPortAndTLSChecks asserted only that hostname, port and certificate state were non-empty, plus non-zero checker call counts. Neither was vacuous, but neither would have caught the watcher resolving the wrong addresses. The port and TLS test doubles now record their arguments, and both tests assert that the state keys and the arguments the checkers were actually called with match the addresses live DNS returned — exactly those, no more and no fewer. Verified by mutation: making the watcher drop all but one resolved address fails both tests, and it passed both of them before. TESTING.md records why a loopback nameserver is not a mock, so the new tests are not mistaken for a violation of the rule they respect.
92 lines
4.3 KiB
Markdown
92 lines
4.3 KiB
Markdown
# Testing Policy
|
||
|
||
## DNS Resolution Tests
|
||
|
||
All tests that involve DNS resolution — in every package, including
|
||
consumers of the resolver such as the watcher — **MUST** use live
|
||
queries against real DNS servers. No mocking, faking, or stubbing of
|
||
DNS at any layer is permitted.
|
||
|
||
### Rationale
|
||
|
||
The resolver performs iterative resolution from root nameservers through
|
||
the full delegation chain. Mocked responses cannot faithfully represent
|
||
the variety of real-world DNS behavior (truncation, referrals, glue
|
||
records, DNSSEC, varied response times, EDNS, etc.). Testing against
|
||
real servers ensures the resolver works correctly in production.
|
||
Robustness comes from handling real-world DNS behavior with tolerant
|
||
assertions and sensible timeouts, not from mocks.
|
||
|
||
### Constraints
|
||
|
||
- Tests hit real DNS infrastructure and require network access
|
||
- Test duration depends on network conditions; timeout tuning keeps
|
||
the suite within the 60-second target
|
||
- Query timeout is calibrated to 3× maximum antipodal RTT (~300ms)
|
||
plus processing margin
|
||
- Root server fan-out is limited to reduce parallel query load
|
||
- Flaky failures from transient network issues are acceptable and
|
||
should be investigated as potential resolver bugs, not papered over
|
||
with mocks or skip flags
|
||
- Watcher change-detection tests seed a synthetic *previous state*
|
||
and compare it against fresh live lookups; the DNS side is never
|
||
faked
|
||
- Live query concurrency is bounded per package (`liveGate` in
|
||
`internal/resolver`, `liveWatcherGate` in `internal/watcher`) so
|
||
parallel tests do not burst at the root servers
|
||
- Those gates are package-scoped and therefore per test binary, so
|
||
`script/test` also passes `-p 1`: with both live-DNS packages
|
||
running at once the gates sum instead of holding, and the resolver's
|
||
per-attempt deadlines start expiring
|
||
|
||
### Transport failures: loopback nameservers, not mocks
|
||
|
||
The resolver classifies a nameserver that stays silent as
|
||
`StatusTimeout` and one that answers SERVFAIL as `StatusError`. The
|
||
public network cannot be made to produce either on demand — a
|
||
black-holed address is only black-holed on some networks, and build
|
||
environments that transparently intercept UDP/53 answer it locally —
|
||
so a test built on a chosen remote address asserts on the network it
|
||
happens to run on rather than on the resolver.
|
||
|
||
`internal/resolver/transport_test.go` binds a real nameserver on
|
||
`127.0.0.1` instead and points the query at it.
|
||
`nameserverAddr` dials an address that already carries a port as
|
||
written, so no production behaviour is bypassed to arrange this.
|
||
|
||
**This is permitted, and it is not a mock.** The rule above bans
|
||
substituting `DNSClient` or any other DNS abstraction, which lets the
|
||
code under test skip DNS and hands it a manufactured verdict. A
|
||
loopback nameserver does the opposite: the resolver dials a real
|
||
socket, writes a real query with the real `miekg/dns` client, and
|
||
applies its real deadline and its real classification logic to what
|
||
comes back. Choosing which nameserver a live query is sent to is not
|
||
faking DNS — the resolver is aimed at a nameserver of the caller's
|
||
choosing in production too.
|
||
|
||
The distinction to hold on to: **substituting the client is banned;
|
||
choosing the server is not.** A test that reaches for a fake
|
||
`DNSClient` to force a classification is still forbidden, no matter
|
||
how awkward the alternative looks.
|
||
|
||
Such a test must stay cheap. The resolver asks for eight record
|
||
types and retries each once, so a nameserver silent on every type
|
||
costs sixteen query timeouts. `TestQueryNameserverIP_Timeout` is
|
||
silent on `A` alone and answers the rest, which is all the resolver
|
||
needs to classify the response and keeps the test to two.
|
||
|
||
### What NOT to do
|
||
|
||
- **Do not mock `DNSClient`**, the watcher's `DNSResolver` interface,
|
||
or any other DNS abstraction — in any package, for any reason
|
||
- **Do not add `-short` flags** to skip slow tests
|
||
- **Do not increase `-timeout`** to hide hanging queries
|
||
- **Do not remove `-count=1` from `script/test`** — Go's test cache
|
||
replays a previous run's output without querying anything, so a
|
||
cached pass is not evidence that live resolution works
|
||
- **Do not remove `-p 1` from `script/test`** — running the live-DNS
|
||
packages in parallel oversubscribes live DNS past what their gates
|
||
bound, which surfaces as unrelated resolver tests failing on
|
||
expired deadlines
|
||
- **Do not modify linter configuration** to suppress findings
|