resolver: try servers in a random order on each resolution (closes #138)

Every resolution walked the root servers in a fixed order, so
a.root-servers.net got every first query and its timeouts were paid on
every lookup. Each list of servers the resolver walks is now walked in a
random order from rand.Shuffle, chosen anew each time; a server that does
not reply, refuses, or gives an error reply or a referral that leads no
closer is still passed over for the next. When a referral names a zone's
nameservers without their addresses, all of them are now looked up, not
only the first that resolves, so the zone is not given up because one
randomly picked nameserver failed. No test fails if the walk stops
shuffling: which server a live query reached is not observable.

Model: opus-5-5

README.md

@@ -40,6 +40,29 @@ Contributions that introduce mocked, faked, or stubbed DNS will be rejected.
 
 ---
 
+## Getting Started
+
+You need git and Docker. This builds the image and runs dnswatcher watching
+`example.com` and `www.example.com`:
+
+```sh
+git clone https://git.eeqj.de/sneak/dnswatcher.git
+cd dnswatcher
+docker build -t dnswatcher .
+docker run -d --name dnswatcher \
+  -p 8080:8080 \
+  -v dnswatcher-data:/var/lib/dnswatcher \
+  -e DNSWATCHER_TARGETS=example.com,www.example.com \
+  dnswatcher
+```
+
+The build also runs the linter and the test suite, which queries live DNS. Once
+the container is running, the dashboard is at <http://localhost:8080/>. With no
+notification endpoint set, changes show only on the dashboard; see
+[Configuration](#configuration) to add one.
+
+---
+
 ## Features
 
 ### DNS Domain Monitoring (Apex Domains)
@@ -272,48 +295,6 @@ navigation needs, and its URL may name internal hosts.
 
 ---
 
-## Architecture
-
-```
-cmd/dnswatcher/main.go         Entry point (uber/fx bootstrap)
-
-internal/
-  config/config.go             Viper-based configuration
-  globals/globals.go           Build-time variables (version)
-  logger/logger.go             slog structured logging (TTY detection)
-  healthcheck/healthcheck.go   Health check service
-  middleware/middleware.go      HTTP middleware (logging, CORS, security
-                               headers, metrics auth and rate limit)
-  handlers/handlers.go         HTTP request handlers
-  server/
-    server.go                  HTTP server lifecycle
-    routes.go                  Route definitions
-  state/state.go               JSON file state persistence
-  resolver/resolver.go         Iterative DNS resolution engine
-  portcheck/portcheck.go       TCP port connectivity checker
-  tlscheck/tlscheck.go        TLS certificate inspector
-  notify/notify.go             Notification service (Slack, Mattermost, ntfy)
-  watcher/watcher.go           Main monitoring orchestrator and scheduler
-  livednstest/livednstest.go   Retry and concurrency limit for tests
-                               against live DNS (imported only by tests)
-```
-
-### Design Principles
-
-- **No recursive resolvers**: All DNS resolution is performed iteratively,
-  tracing from root nameservers through the delegation chain to authoritative
-  servers.
-- **No external database**: State is persisted as a single JSON file.
-- **Dependency injection**: All components are wired via
-  [uber/fx](https://github.com/uber-go/fx).
-- **Structured logging**: All logs use `log/slog` with JSON output in production
-  (TTY detection for development).
-- **Graceful shutdown**: All background goroutines respect context cancellation
-  and the fx lifecycle. In-flight notification deliveries are drained on
-  shutdown, bounded by the shutdown timeout.
-
----
-
 ## Configuration
 
 Configuration is loaded via [Viper](https://github.com/spf13/viper) with the
@@ -401,6 +382,13 @@ performs full iterative resolution:
 4. **Authoritative query**: Queries all discovered authoritative nameservers
    directly for the requested records.
 
+In steps 2 and 3 the servers are asked one at a time in a random order, chosen
+anew each time, so no one root server gets every first query. A server that does
+not reply, refuses the query, or gives an error reply such as SERVFAIL or a
+referral that leads no closer to the name is passed over for the next one. When
+a referral names a zone's nameservers without their addresses, the addresses of
+all of them are looked up, so that each can be asked.
+
 This approach ensures:
 
 - Independence from any upstream resolver's cache or filtering.
@@ -667,6 +655,68 @@ configuration.
 
 ---
 
+## Rationale
+
+dnswatcher exists to report changes to the DNS records, TCP port availability
+and TLS certificates of its configured domains and hostnames, failures and
+recoveries included: it is designed as a real-time change feed. It queries the
+authoritative nameservers directly, tracing from the root, instead of a
+recursive resolver, so no resolver's cache or filtering hides a change and
+nameservers that disagree with each other are seen. Its state is a single JSON
+file, so it survives a restart without an external database.
+
+---
+
+## Design
+
+```
+cmd/dnswatcher/main.go         Entry point (uber/fx bootstrap)
+
+internal/
+  config/config.go             Viper-based configuration
+  globals/globals.go           Build-time variables (version)
+  logger/logger.go             slog structured logging (TTY detection)
+  healthcheck/healthcheck.go   Health check service
+  middleware/middleware.go      HTTP middleware (logging, CORS, security
+                               headers, metrics auth and rate limit)
+  handlers/handlers.go         HTTP request handlers
+  server/
+    server.go                  HTTP server lifecycle
+    routes.go                  Route definitions
+  state/state.go               JSON file state persistence
+  resolver/resolver.go         Iterative DNS resolution engine
+  portcheck/portcheck.go       TCP port connectivity checker
+  tlscheck/tlscheck.go        TLS certificate inspector
+  notify/notify.go             Notification service (Slack, Mattermost, ntfy)
+  watcher/watcher.go           Main monitoring orchestrator and scheduler
+  livednstest/livednstest.go   Retry and concurrency limit for tests
+                               against live DNS (imported only by tests)
+```
+
+### Design Principles
+
+- **No recursive resolvers**: All DNS resolution is performed iteratively,
+  tracing from root nameservers through the delegation chain to authoritative
+  servers.
+- **No external database**: State is persisted as a single JSON file.
+- **Dependency injection**: All components are wired via
+  [uber/fx](https://github.com/uber-go/fx).
+- **Structured logging**: All logs use `log/slog` with JSON output in production
+  (TTY detection for development).
+- **Graceful shutdown**: All background goroutines respect context cancellation
+  and the fx lifecycle. In-flight notification deliveries are drained on
+  shutdown, bounded by the shutdown timeout.
+
+---
+
+## TODO
+
+[`TODO.md`](./TODO.md) names the next step and the steps planned after it. The
+work for 1.0 is tracked as issues on the
+[1.0 milestone](https://git.eeqj.de/sneak/dnswatcher/milestone/7).
+
+---
+
 ## License
 
 dnswatcher is released under the MIT License, Copyright (c) 2026

TODO.md

@@ -19,6 +19,16 @@ trial run of the finished image: https://git.eeqj.de/sneak/dnswatcher/issues/149
 
 # Completed Steps
 
+- 2026-10-02: the resolver tries root servers, and every other server list it
+  walks, in a random order each time, not always from the top (closes #138).
+- 2026-10-01: a certificate within the expiry warning period is warned about on
+  every TLS check, where some checks used to skip it at random (closes #204).
+- 2026-10-01: a domain's NS set is its delegation from the parent zone's
+  servers, not whichever of its own servers answered first (closes #200).
+- 2026-10-01: README has Getting Started, Rationale and TODO sections, and its
+  Architecture section is now Design, in the order policy sets (closes #173).
+- 2026-10-01: a zone's server that answers SERVFAIL or a referral leading no
+  closer is passed over for the next, as one that times out is (closes #197).
 - 2026-10-01: when none of a configured name's nameservers answered, the port
   state saved for its addresses is kept, not removed (closes #193).
 - 2026-10-01: `ResolveIPAddresses` returns an error, not no addresses, when no
@@ -120,7 +130,4 @@ trial run of the finished image: https://git.eeqj.de/sneak/dnswatcher/issues/149
 - 1.0 readiness: run it with a real config and read the logs:
   https://git.eeqj.de/sneak/dnswatcher/issues/66
 - README accuracy sweep: https://git.eeqj.de/sneak/dnswatcher/issues/108
-- README sections required by policy:
-  https://git.eeqj.de/sneak/dnswatcher/issues/173
-- fixed root server order: https://git.eeqj.de/sneak/dnswatcher/issues/138
 - review toward 1.0: https://git.eeqj.de/sneak/dnswatcher/issues/144

internal/resolver/errors.go

@@ -15,6 +15,13 @@ var (
 	// so whether the name has addresses is unknown.
 	ErrNoNameserverAnswered = errors.New("no nameserver answered")
 
+	// ErrUnusableReply is returned when a server replied with an
+	// error such as SERVFAIL, or with a referral that leads no
+	// closer to the name asked about.
+	ErrUnusableReply = errors.New(
+		"reply is an error or a referral that leads no closer",
+	)
+
 	// ErrCNAMEDepthExceeded is returned when a CNAME chain
 	// exceeds MaxCNAMEDepth.
 	ErrCNAMEDepthExceeded = errors.New(

internal/resolver/export_test.go

@@ -11,6 +11,16 @@ func ExtractRecordValue(rr dns.RR) string {
 	return extractRecordValue(rr)
 }
 
+// UsableReply exports usableReply for testing.
+func UsableReply(resp *dns.Msg, zone string, name string) bool {
+	return usableReply(resp, zone, name)
+}
+
+// NSSetFrom exports nsSetFrom for testing.
+func NSSetFrom(resp *dns.Msg, domain string) []string {
+	return nsSetFrom(resp, domain)
+}
+
 // CollectIPs exports collectIPs for testing.
 func CollectIPs(
 	results map[string]*NameserverResponse,
@@ -26,3 +36,24 @@ func (r *Resolver) QueryEachNS(
 ) (map[string]*NameserverResponse, error) {
 	return r.queryEachNS(ctx, nameservers, hostname)
 }
+
+// ResolveNSIPs exports resolveNSIPs for testing.
+func (r *Resolver) ResolveNSIPs(
+	ctx context.Context,
+	nsNames []string,
+) []string {
+	return r.resolveNSIPs(ctx, nsNames)
+}
+
+// RootServerList exports rootServerList for testing.
+func RootServerList() []string {
+	return rootServerList()
+}
+
+// Shuffled exports shuffled for testing.
+func Shuffled(
+	servers []string,
+	shuffle func(n int, swap func(i, j int)),
+) []string {
+	return shuffled(servers, shuffle)
+}

internal/resolver/iterative.go

@@ -4,7 +4,9 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"math/rand/v2"
 	"net"
+	"slices"
 	"sort"
 	"strings"
 	"time"
@@ -207,21 +209,24 @@ func (r *Resolver) followDelegation(
 	domain string,
 	servers []string,
 ) ([]string, error) {
+	// servers are the root servers, the servers of zone ".".
+	zone := "."
+
 	for range maxDelegation {
 		if checkCtx(ctx) != nil {
 			return nil, ErrContextCanceled
 		}
 
 		resp, err := r.queryServers(
-			ctx, servers, domain, dns.TypeNS,
+			ctx, servers, zone, domain, dns.TypeNS,
 		)
 		if err != nil {
 			return nil, err
 		}
 
-		ansNS := extractNSSet(resp.Answer)
+		nsSet := nsSetFrom(resp, domain)
-		if len(ansNS) > 0 {
+		if len(nsSet) > 0 {
-			return ansNS, nil
+			return nsSet, nil
 		}
 
 		// An authoritative reply comes from the servers of the zone
@@ -250,25 +255,52 @@ func (r *Resolver) followDelegation(
 		}
 
 		servers = nextServers
+		zone = referralZone(resp)
 	}
 
 	return nil, ErrNoNameservers
 }
 
+// shuffled returns a copy of servers in the order shuffle puts them
+// in. The resolver passes rand.Shuffle, so each time it walks a list of
+// servers it starts at a random one, and no one server gets every
+// first query.
+func shuffled(
+	servers []string,
+	shuffle func(n int, swap func(i, j int)),
+) []string {
+	order := slices.Clone(servers)
+	shuffle(len(order), func(i, j int) {
+		order[i], order[j] = order[j], order[i]
+	})
+
+	return order
+}
+
+// queryServers asks servers, the servers of zone, about name in a random
+// order until one gives a usable reply. A server that times out, refuses
+// or gives a reply that is not usable is passed over for the next.
 func (r *Resolver) queryServers(
 	ctx context.Context,
 	servers []string,
+	zone string,
 	name string,
 	qtype uint16,
 ) (*dns.Msg, error) {
 	var lastErr error
 
-	for _, ip := range servers {
+	for _, ip := range shuffled(servers, rand.Shuffle) {
 		if checkCtx(ctx) != nil {
 			return nil, ErrContextCanceled
 		}
 
 		resp, err := r.queryDNS(ctx, ip, name, qtype)
+		if err == nil && !usableReply(resp, zone, name) {
+			err = fmt.Errorf(
+				"query %s @%s: %w", name, ip, ErrUnusableReply,
+			)
+		}
+
 		if err == nil {
 			return resp, nil
 		}
@@ -279,6 +311,57 @@ func (r *Resolver) queryServers(
 	return nil, fmt.Errorf("all servers failed: %w", lastErr)
 }
 
+// usableReply reports whether resp, a reply from one of the servers of
+// zone to a query about name, is usable. An error reply such as SERVFAIL
+// is not. Nor is a referral, unless it refers the query to a zone below
+// zone that name is in: a server that refers it back to zone, up or
+// sideways does not serve zone as it should.
+func usableReply(resp *dns.Msg, zone string, name string) bool {
+	if resp.Rcode != dns.RcodeSuccess && resp.Rcode != dns.RcodeNameError {
+		return false
+	}
+
+	child := referralZone(resp)
+	if resp.Authoritative || len(resp.Answer) > 0 || child == "" {
+		return true
+	}
+
+	return child != zone && dns.IsSubDomain(zone, child) &&
+		dns.IsSubDomain(child, name)
+}
+
+// referralZone returns the zone a referral refers the query to: the
+// owner name of the NS records in resp's authority section, or "" when
+// there are none.
+func referralZone(resp *dns.Msg) string {
+	for _, rr := range resp.Ns {
+		if ns, ok := rr.(*dns.NS); ok {
+			return strings.ToLower(ns.Hdr.Name)
+		}
+	}
+
+	return ""
+}
+
+// nsSetFrom returns the NS set of domain that resp, a reply to a query
+// for domain's NS records, gives: the delegation in a referral to domain
+// itself, or else the NS records in the answer; empty when it gives
+// neither. A referral to domain comes from its parent zone's servers,
+// which all hold the same delegation, so the set does not depend on
+// which of them answered. domain's own servers, which can disagree about
+// their NS records, are then not asked.
+func nsSetFrom(resp *dns.Msg, domain string) []string {
+	if referralZone(resp) == domain {
+		return extractNSSet(resp.Ns)
+	}
+
+	return extractNSSet(resp.Answer)
+}
+
+// resolveNSIPs returns the addresses of every nameserver in nsNames
+// whose name resolves, for a referral that carries none. The walk can
+// then go on to the zone's other nameservers when one gives no usable
+// reply.
 func (r *Resolver) resolveNSIPs(
 	ctx context.Context,
 	nsNames []string,
@@ -290,10 +373,6 @@ func (r *Resolver) resolveNSIPs(
 		if err == nil {
 			ips = append(ips, resolved...)
 		}
-
-		if len(ips) > 0 {
-			break
-		}
 	}
 
 	return ips
@@ -312,6 +391,7 @@ func (r *Resolver) resolveNSIterative(
 
 	domain = dns.Fqdn(domain)
 	servers := rootServerList()
+	zone := "."
 
 	for range maxDelegation {
 		if checkCtx(ctx) != nil {
@@ -319,13 +399,13 @@ func (r *Resolver) resolveNSIterative(
 		}
 
 		resp, err := r.queryServers(
-			ctx, servers, domain, dns.TypeNS,
+			ctx, servers, zone, domain, dns.TypeNS,
 		)
 		if err != nil {
 			return nil, err
 		}
 
-		nsNames := extractNSSet(resp.Answer)
+		nsNames := nsSetFrom(resp, domain)
 		if len(nsNames) > 0 {
 			return nsNames, nil
 		}
@@ -344,6 +424,7 @@ func (r *Resolver) resolveNSIterative(
 		}
 
 		servers = nextServers
+		zone = referralZone(resp)
 	}
 
 	return nil, ErrNoNameservers
@@ -361,6 +442,7 @@ func (r *Resolver) resolveARecord(
 
 	hostname = dns.Fqdn(hostname)
 	servers := rootServerList()
+	zone := "."
 
 	for range maxDelegation {
 		if checkCtx(ctx) != nil {
@@ -368,7 +450,7 @@ func (r *Resolver) resolveARecord(
 		}
 
 		resp, err := r.queryServers(
-			ctx, servers, hostname, dns.TypeA,
+			ctx, servers, zone, hostname, dns.TypeA,
 		)
 		if err != nil {
 			return nil, fmt.Errorf(
@@ -406,6 +488,7 @@ func (r *Resolver) resolveARecord(
 		}
 
 		servers = nextServers
+		zone = referralZone(resp)
 	}
 
 	return nil, fmt.Errorf(
@@ -415,7 +498,8 @@ func (r *Resolver) resolveARecord(
 
 // FindAuthoritativeNameservers traces the delegation chain from
 // root servers to discover all authoritative nameservers for the
-// given domain. For a name that is not a zone apex it tries each
+// given domain, as the delegation from its parent zone's servers lists
+// them. For a name that is not a zone apex it tries each
 // parent name in turn, so it returns the nameservers of the zone the
 // name is in.
 func (r *Resolver) FindAuthoritativeNameservers(
@@ -581,9 +665,10 @@ func (r *Resolver) querySingleType(
 
 	// A reply with no answer that lists other nameservers, from a server
 	// that does not hold the name's zone, is a referral and says nothing
-	// about the name's records. A parent zone's servers send one when
+	// about the name's records. A server named in the delegation that
-	// every server of the name's own zone failed and
+	// does not hold the zone may send one, as do a parent zone's servers
-	// FindAuthoritativeNameservers moved on to the parent name.
+	// when FindAuthoritativeNameservers found no delegation for the
+	// name's zone and moved on to a parent name.
 	if !msg.Authoritative && len(msg.Answer) == 0 &&
 		len(extractNSSet(msg.Ns)) > 0 {
 		state.gotReferral = true

internal/resolver/iterative_test.go

@@ -1,6 +1,8 @@
 package resolver_test
 
 import (
+	"math/rand/v2"
+	"slices"
 	"testing"
 
 	"github.com/miekg/dns"
@@ -41,6 +43,149 @@ func TestCollectIPs_FailedIsNoAnswer(t *testing.T) {
 	assert.Empty(t, ips)
 }
 
+const (
+	// exampleCom is the zone most cases of TestUsableReply and
+	// TestNSSetFrom are about, and wwwExampleCom a name in it.
+	exampleCom    = "example.com."
+	wwwExampleCom = "www.example.com."
+
+	// exampleNS is the server the NS records nsRecord builds name.
+	exampleNS = "ns1.example.net."
+)
+
+// nsRecord builds an NS record that names a server of zone.
+func nsRecord(zone string) *dns.NS {
+	return &dns.NS{
+		Hdr: dns.RR_Header{
+			Name: zone, Rrtype: dns.TypeNS, Class: dns.ClassINET,
+		},
+		Ns: exampleNS,
+	}
+}
+
+// referralTo builds a reply that refers the query to the servers of
+// zone.
+func referralTo(zone string) *dns.Msg {
+	msg := new(dns.Msg)
+	msg.Ns = []dns.RR{nsRecord(zone)}
+
+	return msg
+}
+
+// TestUsableReply checks which replies from one of a zone's servers are
+// used. A reply that is not usable moves the query on to the zone's
+// next server.
+func TestUsableReply(t *testing.T) {
+	t.Parallel()
+
+	servfail := new(dns.Msg)
+	servfail.Rcode = dns.RcodeServerFailure
+
+	answer := new(dns.Msg)
+	answer.Authoritative = true
+	answer.Answer = []dns.RR{nsRecord(exampleCom)}
+
+	nxdomain := new(dns.Msg)
+	nxdomain.Authoritative = true
+	nxdomain.Rcode = dns.RcodeNameError
+
+	tests := []struct {
+		name  string
+		resp  *dns.Msg
+		zone  string
+		query string
+		want  bool
+	}{
+		{
+			name: "SERVFAIL", resp: servfail,
+			zone: exampleCom, query: exampleCom, want: false,
+		},
+		{
+			name: "answer", resp: answer,
+			zone: exampleCom, query: exampleCom, want: true,
+		},
+		{
+			name: "NXDOMAIN", resp: nxdomain,
+			zone: ".", query: exampleCom, want: true,
+		},
+		{
+			name: "root refers to com", resp: referralTo("com."),
+			zone: ".", query: exampleCom, want: true,
+		},
+		{
+			name: "com refers to example.com", resp: referralTo(exampleCom),
+			zone: "com.", query: wwwExampleCom, want: true,
+		},
+		{
+			name: "referral back to the zone", resp: referralTo(exampleCom),
+			zone: exampleCom, query: exampleCom, want: false,
+		},
+		{
+			name: "referral up to the root", resp: referralTo("."),
+			zone: exampleCom, query: exampleCom, want: false,
+		},
+		{
+			name: "referral sideways", resp: referralTo("net."),
+			zone: ".", query: exampleCom, want: false,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			assert.Equal(t, tt.want,
+				resolver.UsableReply(tt.resp, tt.zone, tt.query),
+			)
+		})
+	}
+}
+
+// TestNSSetFrom checks which NS set a reply gives for a domain; a set
+// that is not empty ends the walk. The referral to example.com that
+// com's servers all send alike gives its delegation, so the set is the
+// same whichever of them answered, and example.com's own servers, which
+// can disagree, are not asked.
+func TestNSSetFrom(t *testing.T) {
+	t.Parallel()
+
+	answer := new(dns.Msg)
+	answer.Authoritative = true
+	answer.Answer = []dns.RR{nsRecord(exampleCom)}
+
+	tests := []struct {
+		name   string
+		resp   *dns.Msg
+		domain string
+		want   []string
+	}{
+		{
+			name: "com refers to example.com", resp: referralTo(exampleCom),
+			domain: exampleCom, want: []string{exampleNS},
+		},
+		{
+			name: "com refers on, for www.example.com",
+			resp: referralTo(exampleCom), domain: wwwExampleCom,
+			want: nil,
+		},
+		{
+			name: "answer from a server that holds example.com",
+			resp: answer, domain: exampleCom,
+			want: []string{exampleNS},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			assert.ElementsMatch(t, tt.want,
+				resolver.NSSetFrom(tt.resp, tt.domain),
+			)
+		})
+	}
+}
+
 func TestExtractRecordValue_LetterCase(t *testing.T) {
 	t.Parallel()
 
@@ -92,3 +237,30 @@ func TestExtractRecordValue_LetterCase(t *testing.T) {
 		})
 	}
 }
+
+// TestShuffled shuffles the root servers with many seeds. Every order
+// must hold each root server once, so each is tried before a
+// resolution fails; each root server must come first for some seed, so
+// no one root server gets every first query; and the list passed in
+// must be left as it was.
+func TestShuffled(t *testing.T) {
+	t.Parallel()
+
+	const seeds = 1000
+
+	roots := resolver.RootServerList()
+	before := slices.Clone(roots)
+	first := make(map[string]bool)
+
+	for seed := range uint64(seeds) {
+		rng := rand.New(rand.NewPCG(seed, 0)) //nolint:gosec // seeded on purpose
+		order := resolver.Shuffled(roots, rng.Shuffle)
+
+		assert.ElementsMatch(t, roots, order)
+
+		first[order[0]] = true
+	}
+
+	assert.Len(t, first, len(roots))
+	assert.Equal(t, before, roots)
+}

internal/resolver/livedns_test.go

@@ -383,3 +383,37 @@ func liveResolveIPsAllowingEmpty(
 
 	return out
 }
+
+// liveResolveNSIPs looks up the addresses of the nameservers named
+// names, retrying until there are at least atLeast of them: a name
+// whose lookup got no reply is left out of the result, not an error.
+func liveResolveNSIPs(
+	t *testing.T,
+	r *resolver.Resolver,
+	names []string,
+	atLeast int,
+) []string {
+	t.Helper()
+
+	var out []string
+
+	livednstest.Retry(
+		t,
+		"ResolveNSIPs("+strings.Join(names, ", ")+")",
+		func(ctx context.Context) error {
+			ips := r.ResolveNSIPs(ctx, names)
+			if len(ips) < atLeast {
+				return fmt.Errorf(
+					"%w: %d addresses, expected at least %d",
+					livednstest.ErrNoAnswer, len(ips), atLeast,
+				)
+			}
+
+			out = ips
+
+			return nil
+		},
+	)
+
+	return out
+}

internal/resolver/resolver_test.go

@@ -139,6 +139,28 @@ func TestFindAuthoritativeNameservers_CloudflareDomain(
 	}
 }
 
+// TestResolveNSIPs_EveryNameserver looks up the addresses of two of
+// google.com's nameservers together, as the walk does when a referral
+// names a zone's nameservers without their addresses, and compares them
+// with each looked up alone. Together they must give the addresses of
+// both, not only of the first that resolves, so that when one gives no
+// usable reply the walk goes on to the other.
+func TestResolveNSIPs_EveryNameserver(t *testing.T) {
+	t.Parallel()
+
+	r := newTestResolver(t)
+	names := []string{"ns3.google.com.", "ns4.google.com."}
+	want := make([]string, 0, len(names))
+
+	for _, name := range names {
+		want = append(want, liveResolveNSIPs(t, r, []string{name}, 1)...)
+	}
+
+	got := liveResolveNSIPs(t, r, names, len(want))
+
+	assert.ElementsMatch(t, want, got)
+}
+
 // ----------------------------------------------------------------
 // QueryNameserver tests
 // ----------------------------------------------------------------

internal/watcher/export_test.go

@@ -20,15 +20,14 @@ func NewForTest(
 	n Notifier,
 ) *Watcher {
 	return &Watcher{
-		log:            slog.Default(),
+		log:       slog.Default(),
-		config:         cfg,
+		config:    cfg,
-		state:          st,
+		state:     st,
-		resolver:       res,
+		resolver:  res,
-		portCheck:      pc,
+		portCheck: pc,
-		tlsCheck:       tc,
+		tlsCheck:  tc,
-		notify:         n,
+		notify:    n,
-		firstRun:       true,
+		firstRun:  true,
-		expiryNotified: make(map[string]time.Time),
 	}
 }
 
@@ -72,6 +71,11 @@ func (w *Watcher) CheckAllPorts(ctx context.Context) {
 	w.checkAllPorts(ctx)
 }
 
+// RunTLSChecks exports runTLSChecks for testing.
+func (w *Watcher) RunTLSChecks(ctx context.Context) {
+	w.runTLSChecks(ctx)
+}
+
 // BuildHostnameState exports buildHostnameState for testing.
 func BuildHostnameState(
 	results map[string]*resolver.NameserverResponse,

internal/watcher/watcher.go

@@ -7,7 +7,6 @@ import (
 	"slices"
 	"sort"
 	"strings"
-	"sync"
 	"time"
 
 	"go.uber.org/fx"
@@ -49,18 +48,16 @@ type Params struct {
 
 // Watcher orchestrates all monitoring checks on a schedule.
 type Watcher struct {
-	log              *slog.Logger
+	log       *slog.Logger
-	config           *config.Config
+	config    *config.Config
-	state            *state.State
+	state     *state.State
-	resolver         DNSResolver
+	resolver  DNSResolver
-	portCheck        PortChecker
+	portCheck PortChecker
-	tlsCheck         TLSChecker
+	tlsCheck  TLSChecker
-	notify           Notifier
+	notify    Notifier
-	cancel           context.CancelFunc
+	cancel    context.CancelFunc
-	done             chan struct{} // closed when Run returns
+	done      chan struct{} // closed when Run returns
-	firstRun         bool
+	firstRun  bool
-	expiryNotifiedMu sync.Mutex
-	expiryNotified   map[string]time.Time
 }
 
 // New creates a new Watcher instance wired into the fx lifecycle.
@@ -69,15 +66,14 @@ func New(
 	params Params,
 ) (*Watcher, error) {
 	w := &Watcher{
-		log:            params.Logger.Get(),
+		log:       params.Logger.Get(),
-		config:         params.Config,
+		config:    params.Config,
-		state:          params.State,
+		state:     params.State,
-		resolver:       params.Resolver,
+		resolver:  params.Resolver,
-		portCheck:      params.PortCheck,
+		portCheck: params.PortCheck,
-		tlsCheck:       params.TLSCheck,
+		tlsCheck:  params.TLSCheck,
-		notify:         params.Notify,
+		notify:    params.Notify,
-		firstRun:       true,
+		firstRun:  true,
-		expiryNotified: make(map[string]time.Time),
 	}
 
 	lifecycle.Append(fx.Hook{
@@ -1028,22 +1024,6 @@ func (w *Watcher) checkTLSExpiry(
 		return
 	}
 
-	// Deduplicate expiry warnings: don't re-notify for the same
-	// hostname within the TLS check interval.
-	dedupKey := fmt.Sprintf("expiry:%s:%s", hostname, ip)
-
-	w.expiryNotifiedMu.Lock()
-
-	lastNotified, seen := w.expiryNotified[dedupKey]
-	if seen && time.Since(lastNotified) < w.config.TLSInterval {
-		w.expiryNotifiedMu.Unlock()
-
-		return
-	}
-
-	w.expiryNotified[dedupKey] = time.Now()
-	w.expiryNotifiedMu.Unlock()
-
 	msg := fmt.Sprintf(
 		"Host: %s\nIP: %s\nCN: %s\n"+
 			"Expires: %s (%.0f days)",

internal/watcher/watcher_test.go

@@ -615,33 +615,54 @@ func TestTLSExpiryWarning(t *testing.T) {
 	assertNotified(t, deps, "TLS Expiry Warning: "+testHost, "warning")
 }
 
-func TestTLSExpiryWarningDedup(t *testing.T) {
+// TestTLSExpiryWarningEachCheck runs the TLS checks three times in a
+// row on hostname and port state built here, for a certificate that
+// expires within the warning period. Each check warns once, whether the
+// TLS interval is a nanosecond, shorter than the time between two
+// checks, or a day, longer than it.
+func TestTLSExpiryWarningEachCheck(t *testing.T) {
 	t.Parallel()
 
-	cfg := defaultTestConfig(t)
+	title := "TLS Expiry Warning: " + host
-	cfg.Hostnames = []string{testHost}
-	cfg.TLSInterval = 24 * time.Hour
 
-	title := "TLS Expiry Warning: " + testHost
+	for _, interval := range []time.Duration{time.Nanosecond, 24 * time.Hour} {
+		t.Run(interval.String(), func(t *testing.T) {
+			t.Parallel()
 
-	// The second check comes within the TLS interval of the first,
+			cfg := defaultTestConfig(t)
-	// so it must not warn again.
+			cfg.Hostnames = []string{host}
-	var warnings int
+			cfg.TLSInterval = interval
 
-	deps := runChecks(t, cfg, expiresInThreeDays, func(deps *testDeps) {
+			// The TLS checks read the saved hostname and port state and
-		warnings = countNotifications(deps, title)
+			// look nothing up, so the watcher has no resolver.
-	})
+			deps := newTestDeps(t, cfg)
+			w := watcher.NewForTest(
+				cfg, deps.state, nil,
+				deps.portChecker, deps.tlsChecker, deps.notifier,
+			)
 
-	if warnings == 0 {
+			expiresInThreeDays(deps)
-		t.Fatal("expected expiry warnings from the first check")
+			deps.state.SetHostnameState(host, saved(
-	}
+				map[string]*state.NameserverRecordState{
+					nsA: answered(map[string][]string{"A": {ip1}}),
+				},
+			))
+			deps.state.SetPortState(ip1+":443", &state.PortState{
+				Open: true, Hostnames: []string{host},
+			})
 
-	got := countNotifications(deps, title)
+			for check := 1; check <= 3; check++ {
-	if got != warnings {
+				w.RunTLSChecks(t.Context())
-		t.Errorf(
+
-			"expected %d expiry warnings (dedup), got %d",
+				got := countNotifications(deps, title)
-			warnings, got,
+				if got != check {
-		)
+					t.Fatalf(
+						"after check %d: %d expiry warnings, want %d",
+						check, got, check,
+					)
+				}
+			}
+		})
 	}
 }