watcher: a name removed from the targets leaves the state (closes #223)

At startup, before the first check, Run removes from the loaded state
the domain, hostname and certificate entries of names no longer in
DNSWATCHER_TARGETS, takes those names off each port entry's list of
names and removes a port entry left with none, so the dashboard,
/api/v1/status and the startup notification count only configured
names. A configured domain's own records, saved as a hostname entry
under its name, are kept. Nothing is notified. Each port check, next to
the removal of stale port entries, now also removes the certificate
entries for an address a name no longer resolves to, except while none
of its nameservers answered, as port entries already were.

Model: opus-5-5

.gitea/workflows/check.yml

@@ -1,9 +1,17 @@
 name: check
 on: [push]
+# A new push to a branch cancels that branch's older run, queued or running;
+# runs on other branches, `next` and `main` among them, are left alone.
+concurrency:
+    group: ${{ github.workflow }}-${{ github.ref }}
+    cancel-in-progress: true
 jobs:
     check:
         runs-on: ubuntu-latest
         steps:
             # actions/checkout v4.2.2, 2026-02-28
             - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+              # script/cibuild needs no token, so none is left in .git/config.
+              with:
+                  persist-credentials: false
             - run: script/cibuild

Dockerfile

@@ -24,6 +24,11 @@ FROM golang@sha256:f6751d823c26342f9506c03797d2527668d095b0a15f1862cddb4d927a7a4
 
 RUN apk add --no-cache git make gcc musl-dev binutils-gold
 
+# A build context sent as a tar archive keeps its files' owners, and git
+# refuses to read a checkout owned by another user. Trust this one
+# whoever owns it.
+RUN git config --system --add safe.directory /src
+
 # Force BuildKit to run the lint stage before proceeding
 COPY --from=lint /src/go.sum /dev/null
 

README.md

@@ -80,7 +80,14 @@ notification endpoint set, changes show only on the dashboard; see
       different addresses than on the previous check. A nameserver added or
       removed gets only the NS change notification. When the lookup of a
       nameserver's addresses fails or finds none, its previous addresses are
-      kept and nothing is sent.
+      kept and nothing is sent. The lookup fails when no nameserver it asks
+      answers every one of its queries, for A, AAAA and CNAME.
+- Also watches the domain's own records as a hostname's are watched (see DNS
+  Hostname Monitoring below): its A, AAAA, CNAME, MX, TXT, SRV, CAA and NS
+  records, stored per nameserver. Their changes are notified as a hostname's
+  are, as a record change, NS query failure, NS recovery, inconsistency or CNAME
+  address change, in a message that starts `Domain:` where a hostname's starts
+  `Hostname:`.
 
 ### DNS Hostname Monitoring (Subdomains)
 
@@ -91,6 +98,19 @@ notification endpoint set, changes show only on the dashboard; see
   its last two labels (a name under `co.uk`, or in a delegated subdomain).
 - Queries **each** authoritative nameserver independently for **all** record
   types: A, AAAA, CNAME, MX, TXT, SRV, CAA, NS.
+- Each record type is a query of its own. When a nameserver answers some types
+  but the query for another gets no usable reply (no reply after two tries, an
+  error reply such as SERVFAIL, a referral, or a reply too large for UDP whose
+  retry over TCP fails), the failure is logged with the reason, and the type is
+  listed in the nameserver's `failedTypes` and keeps the records saved for the
+  nameserver by the previous check. On that check those records are not compared
+  with the other nameservers', so no record change or inconsistency is reported
+  for the type; on the next check they are compared with the nameserver's answer
+  as usual. When the previous check did not know the type's records either,
+  because the nameserver was new or failing then or the type was already listed
+  in `unknownTypes`, the type is also listed in `unknownTypes` and left out of
+  every comparison until it answers. A nameserver none of whose queries got a
+  usable reply has failed (see NS query failure below).
 - Stores results **per nameserver**. The state for a hostname is not a merged
   view — it is a map from nameserver to record set.
 - DNS names inside record values (CNAME, MX, SRV and NS targets) are stored in
@@ -119,16 +139,29 @@ notification endpoint set, changes show only on the dashboard; see
       they keep disagreeing, including after a restart. A nameserver that was
       not in the previous check (newly added, or back after dropping out), or
       failed on it, and answers differently is reported on the check where it
-      answers. If a pair agrees again and later disagrees, the alert is sent
-      again.
+      answers. So is a pair that differs in a record type whose query to either
+      nameserver failed on the previous check. If a pair agrees again and later
+      disagrees, the alert is sent again.
+    - **CNAME address change**: The addresses at the end of a name's CNAME chain
+      differ from those of the previous check. They are found when its
+      nameservers answer with a CNAME and no address; a name that answers with
+      an address has none. A change from or to no addresses is sent too, as when
+      a name moves between A records and a CNAME. Nothing is sent when the
+      previous addresses were kept because a chain could not be followed or none
+      of the name's nameservers answered. The first check after loading a state
+      file without `cnameAddresses` sends nothing: it saves the addresses it
+      finds for the next check to compare.
 
 ### TCP Port Monitoring
 
 - For every configured domain and hostname, constructs a deduplicated list of
   the IPv4 and IPv6 addresses in the A and AAAA records its authoritative
-  nameservers returned. A CNAME is not followed: a name whose CNAME points into
-  another zone usually has no addresses here, so its ports and certificate are
-  not checked.
+  nameservers returned. When they returned a CNAME and no address, the CNAME
+  chain is followed and the addresses at its end are used, and a change in those
+  is notified as a CNAME address change. When the nameservers gave different
+  CNAME targets, each is followed and the addresses of all are used. When a
+  chain cannot be followed, or none of the name's nameservers answered, the
+  addresses the last check found at its end are used.
 - Checks TCP connectivity on ports **80** and **443** for each IP address.
 - Every **1 hour** by default, re-checks all ports.
 - Any change in port availability triggers a notification:
@@ -171,17 +204,21 @@ Supported notification backends:
 All configured endpoints receive every notification. Notification content
 includes:
 
-- **DNS record changes**: Which hostname, which nameserver, what record type,
-  old values, new values.
+- **DNS record changes**: Which hostname or domain, which nameserver, what
+  record type, old values, new values.
 - **DNS NS changes**: Which domain, which nameservers were added/removed.
 - **NS address changes**: Which domain, which nameserver, its old and new
   addresses.
+- **CNAME address changes**: Which hostname or domain, the old and new addresses
+  at the end of its CNAME chain.
 - **NS query failures**: Which nameserver failed, error type (timeout, SERVFAIL,
   REFUSED, network error), which hostname/domain affected.
 - **NS recoveries**: Which nameserver recovered, which hostname/domain.
 - **NS inconsistencies**: Which nameservers disagree, what each one returned,
-  which hostname affected.
-- **Port changes**: Which IP:port, its new state, all associated hostnames.
+  which hostname or domain affected.
+- **Port changes**: Which IP:port, its new state, and the domains and the
+  hostnames that resolve to it, on a `Domains:` line and a `Hostnames:` line. A
+  line that would name nothing is left out.
 - **TLS expiry warnings**: Expiry date and days remaining, CN, associated
   hostname and IP.
 - **TLS certificate changes**: Old and new CN and issuer, associated hostname
@@ -208,6 +245,14 @@ clears them.
   false-positive change notifications.
 - State is written atomically (write to temp file, then rename) to prevent
   corruption.
+- A name removed from `DNSWATCHER_TARGETS` is removed from the state at startup,
+  before the first check, without a notification: its domain, hostname and
+  certificate entries go, it is taken off each port entry's list of names, and a
+  port entry left with no name goes, so the dashboard and `/api/v1/status` no
+  longer list or count it. The first check's port checks remove the port entries
+  of addresses no configured name has.
+- Each port check also removes the certificate entries for an address their name
+  no longer resolves to, except while none of the name's nameservers answer.
 
 ### Web Dashboard
 
@@ -215,10 +260,14 @@ dnswatcher includes an unauthenticated, read-only web dashboard at the root URL
 (`/`). It displays:
 
 - **Summary counts** for monitored domains, hostnames, ports, and certificates.
-- **Domains** with their discovered nameservers.
-- **Hostnames** with per-nameserver DNS records and status.
-- **Ports** with open/closed state and associated hostnames.
-- **TLS certificates** with CN, issuer, expiry, and status.
+- **Domains** with their discovered nameservers, and each domain's own records
+  per nameserver and status, shown as a hostname's are.
+- **Hostnames** with per-nameserver DNS records and status. For a nameserver
+  whose query failed, the reason is shown in place of the records.
+- **Ports** with open/closed state and the domains and hostnames that resolve to
+  each address, in separate columns.
+- **TLS certificates** with CN, issuer, expiry, and status. For a failed check,
+  the reason is shown in place of CN, issuer and expiry.
 - **Recent alerts** (last 100 notifications sent since the process started),
   displayed in reverse chronological order.
 
@@ -245,6 +294,14 @@ dnswatcher exposes a lightweight HTTP API for operational visibility:
 | `GET /api/v1/status`           | Current monitoring state      |
 | `GET /metrics`                 | Prometheus metrics, see below |
 
+In `/api/v1/status`, each nameserver entry and certificate entry whose `status`
+is `error` also has `error`, the reason, as in the state file (see State File
+Format). A domain's own records are in its entry in `domains`, under
+`recordsByNameserver`, in the form a hostname's entry in `hostnames` has them
+under `nameservers`; `hostnames` and `counts.hostnames` hold no domain. A port
+entry lists the domains that resolve to its address in `domains`, and the
+hostnames in `hostnames`.
+
 `/metrics` is served only when `DNSWATCHER_METRICS_USERNAME` is set, behind
 Basic Auth. It has the Prometheus Go client's default metrics only (Go runtime,
 process, and counts of `/metrics` requests); dnswatcher records no metrics of
@@ -412,7 +469,12 @@ anew each time, so no one root server gets every first query. A server that does
 not reply, refuses the query, or gives an error reply such as SERVFAIL or a
 referral that leads no closer to the name is passed over for the next one. When
 a referral names a zone's nameservers without their addresses, the addresses of
-all of them are looked up, so that each can be asked.
+all of them are looked up, so that each can be asked. When it gives addresses
+for only some of them, those are asked first, and the others are looked up and
+asked only if none of those gives a usable reply. Both hold in the walk to a
+name's nameservers and in the lookup of a nameserver's own address. Such a
+lookup can need others in turn; lookups go at most three deep, one inside
+another, so delegations that point at each other still end.
 
 This approach ensures:
 
@@ -420,9 +482,11 @@ This approach ensures:
 - Ability to detect split-horizon or inconsistent responses across authoritative
   servers.
 
-CNAME chains are followed (with a depth limit to prevent loops) only to find the
-addresses of nameservers. A watched name's records are stored as its nameservers
-return them, CNAME included, without following it.
+A watched name's records are stored as its nameservers return them, CNAME
+included. When they return a CNAME and no address, the chain of every CNAME
+target they gave is followed (with a depth limit to prevent loops) to the A and
+AAAA records at its end, and the port and TLS checks use those addresses.
+Nameservers' addresses are also found by following CNAME chains.
 
 Sending a notification or a Sentry report is the one use of the system's
 resolver: the HTTP client looks up the webhook's or Sentry's host name with it.
@@ -433,7 +497,8 @@ resolver: the HTTP client looks up the webhook's or Sentry's host name with it.
 
 The state file (`DATA_DIR/state.json`) contains the complete monitoring
 snapshot. Hostname records are stored **per authoritative nameserver**, not as a
-merged view, to enable inconsistency detection.
+merged view, to enable inconsistency detection. `hostnames` also holds each
+domain's own records, under the domain's name.
 
 ```json
 {
@@ -469,6 +534,7 @@ merged view, to enable inconsistency detection.
                     "lastChecked": "2026-02-19T12:00:00Z"
                 }
             },
+            "cnameAddresses": [],
             "lastChecked": "2026-02-19T12:00:00Z"
         }
     },
@@ -502,7 +568,7 @@ reachability:
 
 | Status  | Meaning                                                  |
 | ------- | -------------------------------------------------------- |
-| `ok`    | Query succeeded, records are current                     |
+| `ok`    | Query succeeded, records are current except as below     |
 | `error` | Query failed (timeout, SERVFAIL, REFUSED, network error) |
 
 A nameserver that answers NXDOMAIN or with no records has status `ok` and empty
@@ -511,12 +577,28 @@ nameservers, has status `error`, empty `records`, and the reason in `error`. A
 certificate entry whose TLS connection or handshake failed likewise has status
 `error`, the reason in `error`, and the certificate fields left empty or zero.
 
+A nameserver with status `ok` whose query for one record type failed lists that
+type in `failedTypes` and holds its records from the previous check, which may
+not be current. When the previous check did not know the type's records either,
+because the nameserver was new or failing then or the type was already listed in
+`unknownTypes`, the type is also listed in `unknownTypes`, and `records` holds
+nothing for it. Both lists are left out when empty.
+
 `nameserverAddresses` lists, by nameserver, the sorted addresses its name
 resolves to. A state file without it loads, and the next check fills it in
 without a notification.
 
-A port entry in the older format, with one `hostname` instead of the `hostnames`
-list, loads as a list of that one name.
+`cnameAddresses` lists the sorted addresses at the end of the chain of every
+CNAME target a hostname's nameservers gave, found when they answered with a
+CNAME and no address; it is empty when they answered with an address. When a
+chain cannot be followed, or none of the name's nameservers answered its queries
+for A, AAAA and CNAME, the previous check's list is kept, or `null` when no
+earlier check saved one. A state file without it loads, and the first check
+after that saves it without a notification.
+
+A port entry's `hostnames` lists every name that resolves to its address,
+domains included. A port entry in the older format, with one `hostname` instead
+of the `hostnames` list, loads as a list of that one name.
 
 ---
 
@@ -592,9 +674,14 @@ appears in the startup log and in the health check response.
 
 The image takes it the same way, from the `.git` the build context carries, so a
 plain `docker build .` of a clone stamps the commit it was built from; a clone
-without tags, such as a shallow one, stamps the short commit. `.dockerignore`
-sends `.git` without `.git/config`, which `git describe` does not need and which
-can hold a credential. A non-empty `--build-arg VERSION=...` takes precedence;
+without tags stamps the short commit. A clone made with `--depth 1` carries at
+most a tag on its own commit, so such a clone of an untagged commit stamps the
+short commit. In a build from a directory, `.dockerignore` keeps out
+`.git/config`, which `git describe` does not need and which can hold a
+credential. Docker does not apply `.dockerignore` to a context sent as a tar
+archive, as upaas sends it, so that context carries `.git/config` into the
+build. It also keeps its files' owners, so git in the build trusts the checkout
+whoever owns it. A non-empty `--build-arg VERSION=...` takes precedence;
 `make docker` passes the version `git describe` gives on the host. The build
 fails when the context carries `.git`, as a directory or as a file, and the
 version comes out empty, `dev` or `unknown`. `.dockerignore` must list no
@@ -655,7 +742,8 @@ docker run -d \
 1. **Startup**: Check that the data directory can be written, and exit with an
    error naming it if not. Load state from disk. If no state file exists, start
    with empty state (first check will establish baseline without triggering
-   change notifications).
+   change notifications). Remove from the state the names no longer in
+   `DNSWATCHER_TARGETS` (see State Management).
 2. **Initial check**: Immediately perform all DNS, port, and TLS checks on
    startup.
 3. **Periodic checks** (DNS always runs first):
@@ -667,7 +755,9 @@ docker run -d \
     - Port and TLS checks use the IP addresses found by the DNS phase that
       immediately precedes them. When that phase cannot find a name's
       nameservers at all, the addresses an earlier check saved for the name are
-      used.
+      used. When it cannot follow a name's CNAME chain, or none of the name's
+      nameservers answered, the addresses an earlier check found at the end of
+      the chain are used.
 4. **On change detection**: Send notifications to all configured endpoints,
    update in-memory state, persist to disk.
 5. **Shutdown**: The watcher stops checking and saves the final state to disk,

TODO.md

@@ -19,8 +19,46 @@ trial run of the finished image: https://git.eeqj.de/sneak/dnswatcher/issues/149
 
 # Completed Steps
 
+- 2026-10-02: a name removed from `DNSWATCHER_TARGETS` leaves the state, and so
+  the dashboard and API, at startup, before the first check (closes #223).
+- 2026-10-02: a record type whose query to a nameserver fails keeps its previous
+  records and alerts nothing; the other types are still saved (closes #231).
+- 2026-10-02: a Port Change notification lists the port's domains on a
+  `Domains:` line and its hostnames on a `Hostnames:` line (closes #248).
+- 2026-10-02: the dashboard's Ports table and `/api/v1/status` port entries list
+  a port's domains apart from its hostnames (closes #245).
+- 2026-10-02: nameservers a referral names without addresses are looked up,
+  three deep at most; `pool.ntp.org`'s nameservers resolve (closes #221).
+- 2026-10-02: an apex domain is not counted or listed as a hostname; its records
+  show under Domains, and notifications about them say `Domain:` (closes #224).
+- 2026-10-02: the dashboard lists each nameserver's record types in one fixed
+  order, the README's, then any other type, not a random one (closes #226).
+- 2026-10-02: the dashboard and `/api/v1/status` show why a nameserver query or
+  a certificate check failed, which only the state file showed (closes #225).
+- 2026-10-02: a name's CNAME is stored once per nameserver, not once per record
+  type asked for; a state file with repeats loads each value once (closes #220).
+- 2026-10-02: a DNS lookup that shutdown cuts short logs no error; one that
+  fails otherwise, or runs out of time, still does (closes #229).
+- 2026-10-02: Record Change and Inconsistency notifications list only the record
+  types that differ, each with its values as plain text (closes #219).
+- 2026-10-02: the startup notification no longer says every notification
+  endpoint works; it says it is a test sent to each of them (closes #230).
+- 2026-10-02: a Mattermost webhook that answers an HTTP error is logged as
+  `mattermost notification failed`, not as a Slack failure (closes #227).
+- 2026-10-02: durations in the log are written as text such as `2m0s`, not as a
+  bare count of nanoseconds (closes #228).
+- 2026-10-02: a watched name whose nameservers answer with a CNAME and no
+  address gets port and TLS checks at the end of its CNAME chain (closes #203).
+- 2026-10-02: a resolver test that reads one record type from a nameserver's
+  answer asks again when that type is missing from it (closes #218).
 - 2026-10-02: a plain `docker build .` of a clone stamps its tag or short
   commit, not `dev`: the build context now carries `.git` (closes #210).
+- 2026-10-02: a query a server refuses is not resent asking for recursion, and
+  every root server refusing is reported as DNS interception (closes #206).
+- 2026-10-02: a push to a branch cancels that branch's older CI run, and the
+  checkout leaves no token in `.git/config` (closes #216).
+- 2026-10-02: watcher tests send far fewer queries and a live attempt may take
+  18s; nameserver addresses are asked only for A, AAAA, CNAME (closes #214).
 - 2026-10-02: the resolver tries root servers, and every other server list it
   walks, in a random order each time, not always from the top (closes #138).
 - 2026-10-02: a name listed more than once in `DNSWATCHER_TARGETS`, in any

internal/handlers/dashboard.go

@@ -1,11 +1,14 @@
 package handlers
 
 import (
+	"cmp"
 	"embed"
 	"fmt"
 	"html/template"
+	"maps"
 	"math"
 	"net/http"
+	"slices"
 	"strings"
 	"time"
 
@@ -40,12 +43,19 @@ func newDashboardTemplate() *template.Template {
 	)
 }
 
-// dashboardData is the data passed to the dashboard template.
+// dashboardData is the data passed to the dashboard template. Hostnames
+// and DomainRecords split the records in Snapshot.Hostnames, which also
+// holds the apex domains' own (see splitHostnames). Ports holds
+// Snapshot.Ports with each port's names split into domains and
+// hostnames, as /api/v1/status gives them (see buildPorts).
 type dashboardData struct {
-	Snapshot    state.Snapshot
-	Alerts      []notify.AlertEntry
-	StateAge    string
-	GeneratedAt string
+	Snapshot      state.Snapshot
+	Hostnames     map[string]*state.HostnameState
+	DomainRecords map[string]*state.HostnameState
+	Ports         map[string]*statusPortInfo
+	Alerts        []notify.AlertEntry
+	StateAge      string
+	GeneratedAt   string
 }
 
 // HandleDashboard returns the dashboard page handler.
@@ -58,12 +68,16 @@ func (h *Handlers) HandleDashboard() http.HandlerFunc {
 	) {
 		snap := h.state.GetSnapshot()
 		alerts := h.notifyHistory.Recent()
+		hostnames, domainRecords := splitHostnames(snap)
 
 		data := dashboardData{
-			Snapshot:    snap,
-			Alerts:      alerts,
-			StateAge:    relTime(snap.LastUpdated),
-			GeneratedAt: time.Now().UTC().Format("2006-01-02 15:04:05"),
+			Snapshot:      snap,
+			Hostnames:     hostnames,
+			DomainRecords: domainRecords,
+			Ports:         buildPorts(snap),
+			Alerts:        alerts,
+			StateAge:      relTime(snap.LastUpdated),
+			GeneratedAt:   time.Now().UTC().Format("2006-01-02 15:04:05"),
 		}
 
 		writer.Header().Set(
@@ -122,16 +136,37 @@ func joinStrings(items []string, sep string) string {
 }
 
 // formatRecords formats a map of record type → values into a
-// compact display string.
+// compact display string. Record types are listed in the order the
+// README lists them, any other type after them in alphabetical order,
+// so rows of nameservers with the same records read the same.
 func formatRecords(records map[string][]string) string {
 	if len(records) == 0 {
 		return "-"
 	}
 
+	order := []string{"A", "AAAA", "CNAME", "MX", "TXT", "SRV", "CAA", "NS"}
+
+	position := func(rtype string) int {
+		i := slices.Index(order, rtype)
+		if i < 0 {
+			return len(order)
+		}
+
+		return i
+	}
+
+	rtypes := slices.Collect(maps.Keys(records))
+	slices.SortFunc(rtypes, func(a, b string) int {
+		return cmp.Or(
+			cmp.Compare(position(a), position(b)),
+			strings.Compare(a, b),
+		)
+	})
+
 	var parts []string
 
-	for rtype, values := range records {
-		for _, v := range values {
+	for _, rtype := range rtypes {
+		for _, v := range records[rtype] {
 			parts = append(parts, rtype+": "+v)
 		}
 	}

internal/handlers/dashboard_test.go

@@ -1,6 +1,8 @@
 package handlers_test
 
 import (
+	"regexp"
+	"strings"
 	"testing"
 	"time"
 
@@ -78,3 +80,176 @@ func TestFormatRecords(t *testing.T) {
 		t.Errorf("unexpected format: %q", got)
 	}
 }
+
+// TestFormatRecordsTypeOrder checks that record types are listed in
+// the README's order (A, AAAA, CNAME, MX, TXT, SRV, CAA, NS), with
+// any other type after them in alphabetical order.
+func TestFormatRecordsTypeOrder(t *testing.T) {
+	t.Parallel()
+
+	got := handlers.FormatRecords(map[string][]string{
+		"SOA":   {"ns1.example.com. hostmaster.example.com. 1 2 3 4 5"},
+		"NS":    {"ns1.example.com.", "ns2.example.com."},
+		"CAA":   {`0 issue "letsencrypt.org"`},
+		"DNAME": {"example.net."},
+		"TXT":   {"v=spf1 -all"},
+		"SRV":   {"10 5 443 www.example.com."},
+		"MX":    {"10 mail.example.com."},
+		"CNAME": {"www.example.com."},
+		"AAAA":  {"2001:db8::1"},
+		"A":     {"192.0.2.1"},
+	})
+
+	want := strings.Join([]string{
+		"A: 192.0.2.1",
+		"AAAA: 2001:db8::1",
+		"CNAME: www.example.com.",
+		"MX: 10 mail.example.com.",
+		"TXT: v=spf1 -all",
+		"SRV: 10 5 443 www.example.com.",
+		`CAA: 0 issue "letsencrypt.org"`,
+		"NS: ns1.example.com.",
+		"NS: ns2.example.com.",
+		"DNAME: example.net.",
+		"SOA: ns1.example.com. hostmaster.example.com. 1 2 3 4 5",
+	}, ", ")
+
+	if got != want {
+		t.Errorf("FormatRecords lists types out of order:\n got %q\nwant %q",
+			got, want)
+	}
+}
+
+// dashboardRow returns the table row of page that contains name.
+func dashboardRow(t *testing.T, page string, name string) string {
+	t.Helper()
+
+	for row := range strings.SplitSeq(page, "<tr") {
+		if strings.Contains(row, name) {
+			return row
+		}
+	}
+
+	t.Fatalf("dashboard has no row containing %q", name)
+
+	return ""
+}
+
+// TestDashboardShowsFailureReasons checks that the dashboard shows the
+// reason in the row of a failed nameserver and of a failed certificate,
+// and not in the row of a nameserver that answered.
+func TestDashboardShowsFailureReasons(t *testing.T) {
+	t.Parallel()
+
+	page := get(t, newHandlersWithFailures(t).HandleDashboard())
+
+	if !strings.Contains(dashboardRow(t, page, failedNS), nsFailureReason) {
+		t.Errorf("row of %s does not show %q", failedNS, nsFailureReason)
+	}
+
+	if strings.Contains(dashboardRow(t, page, answeringNS), nsFailureReason) {
+		t.Errorf("row of %s shows %q", answeringNS, nsFailureReason)
+	}
+
+	if !strings.Contains(dashboardRow(t, page, certKey), certFailedReason) {
+		t.Errorf("row of %s does not show %q", certKey, certFailedReason)
+	}
+}
+
+// dashboardSection returns the section of page under heading.
+func dashboardSection(t *testing.T, page string, heading string) string {
+	t.Helper()
+
+	for section := range strings.SplitSeq(page, "<section") {
+		words := strings.Join(strings.Fields(section), " ")
+		if strings.Contains(words, "> "+heading+" </h2>") {
+			return section
+		}
+	}
+
+	t.Fatalf("dashboard has no section headed %q", heading)
+
+	return ""
+}
+
+// TestDashboardShowsDomainRecordsUnderDomains checks that the dashboard
+// shows an apex domain's own records in the Domains section, and
+// neither lists nor counts the domain as a hostname.
+func TestDashboardShowsDomainRecordsUnderDomains(t *testing.T) {
+	t.Parallel()
+
+	page := get(t, newHandlersWithFailures(t).HandleDashboard())
+
+	domains := dashboardSection(t, page, "Domains")
+	if !strings.Contains(dashboardRow(t, domains, domainAddress), testDomain) {
+		t.Errorf("row of %s does not name %s", domainAddress, testDomain)
+	}
+
+	if strings.Contains(dashboardSection(t, page, "Hostnames"), testDomain) {
+		t.Errorf("Hostnames section lists the domain %s", testDomain)
+	}
+
+	words := strings.Join(strings.Fields(page), " ")
+
+	footer := "monitoring 1 domains + 1 hostnames"
+	if !strings.Contains(words, footer) {
+		t.Errorf("dashboard does not say %q", footer)
+	}
+
+	// With the tags taken out, the summary bar starts "Domains 1
+	// Hostnames 1".
+	text := regexp.MustCompile(`<[^>]*>`).ReplaceAllString(page, " ")
+	summary := "Domains 1 Hostnames 1"
+
+	if !strings.Contains(strings.Join(strings.Fields(text), " "), summary) {
+		t.Errorf("summary bar does not say %q", summary)
+	}
+}
+
+// rowCells returns the text of each cell of a dashboard table row
+// whose cells start with tag, "<th" or "<td".
+func rowCells(row string, tag string) []string {
+	tags := regexp.MustCompile(`<[^>]*>`)
+	parts := strings.Split(row, tag)[1:]
+	cells := make([]string, 0, len(parts))
+
+	for _, cell := range parts {
+		text := tags.ReplaceAllString(tag+cell, " ")
+		cells = append(cells, strings.Join(strings.Fields(text), " "))
+	}
+
+	return cells
+}
+
+// TestDashboardPortsTellDomainsFromHostnames checks that the Ports
+// table lists an apex domain under Domains and a hostname under
+// Hostnames when both resolve to the port's address.
+func TestDashboardPortsTellDomainsFromHostnames(t *testing.T) {
+	t.Parallel()
+
+	page := get(t, newHandlersWithFailures(t).HandleDashboard())
+	ports := dashboardSection(t, page, "Ports")
+
+	headings := rowCells(dashboardRow(t, ports, "Address</th>"), "<th")
+	cells := rowCells(dashboardRow(t, ports, sharedPort), "<td")
+
+	if len(cells) != len(headings) {
+		t.Fatalf("row of %s has cells %q under headings %q",
+			sharedPort, cells, headings)
+	}
+
+	under := make(map[string]string)
+	for i, heading := range headings {
+		under[heading] = cells[i]
+	}
+
+	if under["Domains"] != testDomain {
+		t.Errorf("row of %s lists %q under Domains, want %q",
+			sharedPort, under["Domains"], testDomain)
+	}
+
+	if under["Hostnames"] != testHostname {
+		t.Errorf("row of %s lists %q under Hostnames, want %q",
+			sharedPort, under["Hostnames"], testHostname)
+	}
+}

internal/handlers/status.go

@@ -9,15 +9,19 @@ import (
 )
 
 // statusDomainInfo holds status information for a monitored domain.
+// RecordsByNameserver holds the domain's own records, in the form a
+// hostname's Nameservers holds the hostname's.
 type statusDomainInfo struct {
-	Nameservers []string  `json:"nameservers"`
-	LastChecked time.Time `json:"lastChecked"`
+	Nameservers         []string                         `json:"nameservers"`
+	RecordsByNameserver map[string]*statusHostnameNSInfo `json:"recordsByNameserver"`
+	LastChecked         time.Time                        `json:"lastChecked"`
 }
 
 // statusHostnameNSInfo holds per-nameserver status for a hostname.
 type statusHostnameNSInfo struct {
 	Records     map[string][]string `json:"records"`
 	Status      string              `json:"status"`
+	Error       string              `json:"error,omitempty"`
 	LastChecked time.Time           `json:"lastChecked"`
 }
 
@@ -28,8 +32,11 @@ type statusHostnameInfo struct {
 }
 
 // statusPortInfo holds status information for a monitored port.
+// Domains and Hostnames list the apex domains and the hostnames that
+// resolve to its address.
 type statusPortInfo struct {
 	Open        bool      `json:"open"`
+	Domains     []string  `json:"domains"`
 	Hostnames   []string  `json:"hostnames"`
 	LastChecked time.Time `json:"lastChecked"`
 }
@@ -41,6 +48,7 @@ type statusCertificateInfo struct {
 	NotAfter                time.Time `json:"notAfter"`
 	SubjectAlternativeNames []string  `json:"subjectAlternativeNames"`
 	Status                  string    `json:"status"`
+	Error                   string    `json:"error,omitempty"`
 	LastChecked             time.Time `json:"lastChecked"`
 }
 
@@ -94,21 +102,44 @@ func buildStatusResponse(
 		LastUpdated:  snap.LastUpdated,
 		Domains:      make(map[string]*statusDomainInfo),
 		Hostnames:    make(map[string]*statusHostnameInfo),
-		Ports:        make(map[string]*statusPortInfo),
 		Certificates: make(map[string]*statusCertificateInfo),
 	}
 
-	buildDomains(snap, resp)
-	buildHostnames(snap, resp)
-	buildPorts(snap, resp)
+	hostnames, domainRecords := splitHostnames(snap)
+
+	buildDomains(snap, domainRecords, resp)
+	buildHostnames(hostnames, resp)
+	resp.Ports = buildPorts(snap)
 	buildCertificates(snap, resp)
 	buildCounts(resp)
 
 	return resp
 }
 
+// splitHostnames returns the records saved in snap.Hostnames in two
+// maps: the hostnames' and the apex domains' own. The watcher saves a
+// domain's own records there under the domain's name, which has an
+// entry in snap.Domains too.
+func splitHostnames(
+	snap state.Snapshot,
+) (map[string]*state.HostnameState, map[string]*state.HostnameState) {
+	hostnames := make(map[string]*state.HostnameState)
+	domainRecords := make(map[string]*state.HostnameState)
+
+	for name, hs := range snap.Hostnames {
+		if _, isDomain := snap.Domains[name]; isDomain {
+			domainRecords[name] = hs
+		} else {
+			hostnames[name] = hs
+		}
+	}
+
+	return hostnames, domainRecords
+}
+
 func buildDomains(
 	snap state.Snapshot,
+	domainRecords map[string]*state.HostnameState,
 	resp *statusResponse,
 ) {
 	for name, ds := range snap.Domains {
@@ -116,57 +147,86 @@ func buildDomains(
 		copy(ns, ds.Nameservers)
 		sort.Strings(ns)
 
+		records := make(map[string]*statusHostnameNSInfo)
+		if hs, ok := domainRecords[name]; ok {
+			records = nameserverInfo(hs)
+		}
+
 		resp.Domains[name] = &statusDomainInfo{
-			Nameservers: ns,
-			LastChecked: ds.LastChecked,
+			Nameservers:         ns,
+			RecordsByNameserver: records,
+			LastChecked:         ds.LastChecked,
 		}
 	}
 }
 
 func buildHostnames(
-	snap state.Snapshot,
+	hostnames map[string]*state.HostnameState,
 	resp *statusResponse,
 ) {
-	for name, hs := range snap.Hostnames {
-		info := &statusHostnameInfo{
-			Nameservers: make(map[string]*statusHostnameNSInfo),
+	for name, hs := range hostnames {
+		resp.Hostnames[name] = &statusHostnameInfo{
+			Nameservers: nameserverInfo(hs),
 			LastChecked: hs.LastChecked,
 		}
-
-		for ns, nsState := range hs.RecordsByNameserver {
-			recs := make(map[string][]string, len(nsState.Records))
-			for rtype, vals := range nsState.Records {
-				copied := make([]string, len(vals))
-				copy(copied, vals)
-				recs[rtype] = copied
-			}
-
-			info.Nameservers[ns] = &statusHostnameNSInfo{
-				Records:     recs,
-				Status:      nsState.Status,
-				LastChecked: nsState.LastChecked,
-			}
-		}
-
-		resp.Hostnames[name] = info
 	}
 }
 
-func buildPorts(
-	snap state.Snapshot,
-	resp *statusResponse,
-) {
+// nameserverInfo copies each nameserver's answer saved in hs.
+func nameserverInfo(
+	hs *state.HostnameState,
+) map[string]*statusHostnameNSInfo {
+	info := make(map[string]*statusHostnameNSInfo)
+
+	for ns, nsState := range hs.RecordsByNameserver {
+		recs := make(map[string][]string, len(nsState.Records))
+		for rtype, vals := range nsState.Records {
+			copied := make([]string, len(vals))
+			copy(copied, vals)
+			recs[rtype] = copied
+		}
+
+		info[ns] = &statusHostnameNSInfo{
+			Records:     recs,
+			Status:      nsState.Status,
+			Error:       nsState.Error,
+			LastChecked: nsState.LastChecked,
+		}
+	}
+
+	return info
+}
+
+// buildPorts returns the port entries saved in snap. A port entry
+// saves apex domains with its hostnames; they are told apart as in
+// splitHostnames, by a domain entry in snap.Domains.
+func buildPorts(snap state.Snapshot) map[string]*statusPortInfo {
+	ports := make(map[string]*statusPortInfo, len(snap.Ports))
+
 	for key, ps := range snap.Ports {
-		hostnames := make([]string, len(ps.Hostnames))
-		copy(hostnames, ps.Hostnames)
+		domains := []string{}
+		hostnames := []string{}
+
+		for _, name := range ps.Hostnames {
+			if _, isDomain := snap.Domains[name]; isDomain {
+				domains = append(domains, name)
+			} else {
+				hostnames = append(hostnames, name)
+			}
+		}
+
+		sort.Strings(domains)
 		sort.Strings(hostnames)
 
-		resp.Ports[key] = &statusPortInfo{
+		ports[key] = &statusPortInfo{
 			Open:        ps.Open,
+			Domains:     domains,
 			Hostnames:   hostnames,
 			LastChecked: ps.LastChecked,
 		}
 	}
+
+	return ports
 }
 
 func buildCertificates(
@@ -183,6 +243,7 @@ func buildCertificates(
 			NotAfter:                cs.NotAfter,
 			SubjectAlternativeNames: sans,
 			Status:                  cs.Status,
+			Error:                   cs.Error,
 			LastChecked:             cs.LastChecked,
 		}
 	}

internal/handlers/status_test.go

@@ -0,0 +1,267 @@
+package handlers_test
+
+import (
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"slices"
+	"testing"
+	"time"
+
+	"go.uber.org/fx/fxtest"
+
+	"sneak.berlin/go/dnswatcher/internal/config"
+	"sneak.berlin/go/dnswatcher/internal/globals"
+	"sneak.berlin/go/dnswatcher/internal/handlers"
+	"sneak.berlin/go/dnswatcher/internal/logger"
+	"sneak.berlin/go/dnswatcher/internal/notify"
+	"sneak.berlin/go/dnswatcher/internal/state"
+)
+
+// The state the handler tests serve: www.example.com has one nameserver
+// that answered and one whose query failed, and its certificate check
+// failed. example.net is an apex domain, whose own records are saved
+// with the hostnames' records, as the watcher saves them. Both names
+// resolve to domainAddress, whose port 443 entry lists them.
+const (
+	testHostname     = "www.example.com"
+	answeringNS      = "ns1.example.com."
+	failedNS         = "ns2.example.com."
+	nsFailureReason  = "server returned a referral"
+	certKey          = "192.0.2.1:443:www.example.com"
+	certFailedReason = "x509: certificate has expired or is not yet valid"
+	testDomain       = "example.net"
+	domainNS         = "a.iana-servers.net."
+	domainAddress    = "192.0.2.2"
+	sharedPort       = domainAddress + ":443"
+)
+
+// newHandlersWithFailures builds real Handlers whose state holds the
+// entries described above.
+func newHandlersWithFailures(t *testing.T) *handlers.Handlers {
+	t.Helper()
+
+	glob, err := globals.New(nil)
+	if err != nil {
+		t.Fatalf("globals.New: %v", err)
+	}
+
+	log, err := logger.New(nil, logger.Params{Globals: glob})
+	if err != nil {
+		t.Fatalf("logger.New: %v", err)
+	}
+
+	notifier, err := notify.New(fxtest.NewLifecycle(t), notify.Params{
+		Logger: log,
+		Config: &config.Config{},
+	})
+	if err != nil {
+		t.Fatalf("notify.New: %v", err)
+	}
+
+	st, err := state.New(fxtest.NewLifecycle(t), state.Params{
+		Logger: log,
+		Config: &config.Config{DataDir: t.TempDir()},
+	})
+	if err != nil {
+		t.Fatalf("state.New: %v", err)
+	}
+
+	setTestState(st)
+
+	hnd, err := handlers.New(nil, handlers.Params{
+		Logger:  log,
+		Globals: glob,
+		State:   st,
+		Notify:  notifier,
+	})
+	if err != nil {
+		t.Fatalf("handlers.New: %v", err)
+	}
+
+	return hnd
+}
+
+// setTestState sets the entries described above in st.
+func setTestState(st *state.State) {
+	now := time.Now()
+
+	st.SetHostnameState(testHostname, &state.HostnameState{
+		RecordsByNameserver: map[string]*state.NameserverRecordState{
+			answeringNS: {
+				Records: map[string][]string{
+					"A": {"192.0.2.1", domainAddress},
+				},
+				Status:      "ok",
+				LastChecked: now,
+			},
+			failedNS: {
+				Records:     map[string][]string{},
+				Status:      "error",
+				Error:       nsFailureReason,
+				LastChecked: now,
+			},
+		},
+		LastChecked: now,
+	})
+
+	st.SetCertificateState(certKey, &state.CertificateState{
+		Status:      "error",
+		Error:       certFailedReason,
+		LastChecked: now,
+	})
+
+	st.SetDomainState(testDomain, &state.DomainState{
+		Nameservers: []string{domainNS},
+		LastChecked: now,
+	})
+
+	st.SetHostnameState(testDomain, &state.HostnameState{
+		RecordsByNameserver: map[string]*state.NameserverRecordState{
+			domainNS: {
+				Records:     map[string][]string{"A": {domainAddress}},
+				Status:      "ok",
+				LastChecked: now,
+			},
+		},
+		LastChecked: now,
+	})
+
+	st.SetPortState(sharedPort, &state.PortState{
+		Open:        true,
+		Hostnames:   []string{testDomain, testHostname},
+		LastChecked: now,
+	})
+}
+
+// get serves one GET request to handler and returns the response body.
+func get(t *testing.T, handler http.HandlerFunc) string {
+	t.Helper()
+
+	rec := httptest.NewRecorder()
+	req := httptest.NewRequestWithContext(
+		t.Context(), http.MethodGet, "/", nil,
+	)
+
+	handler(rec, req)
+
+	if rec.Code != http.StatusOK {
+		t.Fatalf("status = %d, want 200", rec.Code)
+	}
+
+	return rec.Body.String()
+}
+
+// TestStatusGivesFailureReasons checks that /api/v1/status gives the
+// reason for a failed nameserver entry and a failed certificate entry,
+// and no error for a nameserver that answered.
+func TestStatusGivesFailureReasons(t *testing.T) {
+	t.Parallel()
+
+	body := get(t, newHandlersWithFailures(t).HandleStatus())
+
+	var resp struct {
+		Hostnames map[string]struct {
+			Nameservers map[string]map[string]any `json:"nameservers"`
+		} `json:"hostnames"`
+		Certificates map[string]map[string]any `json:"certificates"`
+	}
+
+	err := json.Unmarshal([]byte(body), &resp)
+	if err != nil {
+		t.Fatalf("decoding response: %v", err)
+	}
+
+	nameservers := resp.Hostnames[testHostname].Nameservers
+
+	got := nameservers[failedNS]["error"]
+	if got != nsFailureReason {
+		t.Errorf("failed nameserver error = %v, want %q",
+			got, nsFailureReason)
+	}
+
+	_, has := nameservers[answeringNS]["error"]
+	if has {
+		t.Errorf("answering nameserver has an error field: %v",
+			nameservers[answeringNS])
+	}
+
+	got = resp.Certificates[certKey]["error"]
+	if got != certFailedReason {
+		t.Errorf("failed certificate error = %v, want %q",
+			got, certFailedReason)
+	}
+}
+
+// TestStatusGivesDomainRecordsUnderTheDomain checks that /api/v1/status
+// gives an apex domain's own records in its domain entry, and neither
+// lists nor counts the domain as a hostname.
+func TestStatusGivesDomainRecordsUnderTheDomain(t *testing.T) {
+	t.Parallel()
+
+	body := get(t, newHandlersWithFailures(t).HandleStatus())
+
+	var resp struct {
+		Counts struct {
+			Hostnames int `json:"hostnames"`
+		} `json:"counts"`
+		Domains map[string]struct {
+			RecordsByNameserver map[string]struct {
+				Records map[string][]string `json:"records"`
+			} `json:"recordsByNameserver"`
+		} `json:"domains"`
+		Hostnames map[string]any `json:"hostnames"`
+	}
+
+	err := json.Unmarshal([]byte(body), &resp)
+	if err != nil {
+		t.Fatalf("decoding response: %v", err)
+	}
+
+	if resp.Counts.Hostnames != 1 {
+		t.Errorf("counts.hostnames = %d, want 1", resp.Counts.Hostnames)
+	}
+
+	if _, listed := resp.Hostnames[testDomain]; listed {
+		t.Errorf("hostnames lists the domain %s", testDomain)
+	}
+
+	records := resp.Domains[testDomain].RecordsByNameserver[domainNS].Records
+	if !slices.Equal(records["A"], []string{domainAddress}) {
+		t.Errorf("domain %s records at %s = %v, want A %s",
+			testDomain, domainNS, records, domainAddress)
+	}
+}
+
+// TestStatusPortsTellDomainsFromHostnames checks that a port entry in
+// /api/v1/status lists an apex domain in domains and a hostname in
+// hostnames when both resolve to its address.
+func TestStatusPortsTellDomainsFromHostnames(t *testing.T) {
+	t.Parallel()
+
+	body := get(t, newHandlersWithFailures(t).HandleStatus())
+
+	var resp struct {
+		Ports map[string]struct {
+			Domains   []string `json:"domains"`
+			Hostnames []string `json:"hostnames"`
+		} `json:"ports"`
+	}
+
+	err := json.Unmarshal([]byte(body), &resp)
+	if err != nil {
+		t.Fatalf("decoding response: %v", err)
+	}
+
+	port := resp.Ports[sharedPort]
+
+	if !slices.Equal(port.Domains, []string{testDomain}) {
+		t.Errorf("port %s domains = %v, want [%s]",
+			sharedPort, port.Domains, testDomain)
+	}
+
+	if !slices.Equal(port.Hostnames, []string{testHostname}) {
+		t.Errorf("port %s hostnames = %v, want [%s]",
+			sharedPort, port.Hostnames, testHostname)
+	}
+}

internal/handlers/templates/dashboard.html

@@ -39,7 +39,7 @@
             Hostnames
           </div>
           <div class="text-2xl font-bold text-teal-400 mt-1">
-            {{ len .Snapshot.Hostnames }}
+            {{ len .Hostnames }}
           </div>
         </div>
         <div class="bg-surface-800 border border-slate-700/50 rounded-lg p-4">
@@ -94,6 +94,24 @@
             </tbody>
           </table>
         </div>
+        {{ if .DomainRecords }}
+        <div class="overflow-x-auto mt-4">
+          <table class="w-full text-left text-xs">
+            <thead>
+              <tr class="text-slate-500 uppercase tracking-wider">
+                <th class="py-2 px-3">Domain</th>
+                <th class="py-2 px-3">NS</th>
+                <th class="py-2 px-3">Status</th>
+                <th class="py-2 px-3">Records</th>
+                <th class="py-2 px-3">Checked</th>
+              </tr>
+            </thead>
+            <tbody class="divide-y divide-slate-800">
+              {{ template "records" .DomainRecords }}
+            </tbody>
+          </table>
+        </div>
+        {{ end }}
         {{ else }}
         <p class="text-slate-600 italic text-xs">
           No domains configured.
@@ -108,7 +126,7 @@
         >
           Hostnames
         </h2>
-        {{ if .Snapshot.Hostnames }}
+        {{ if .Hostnames }}
         <div class="overflow-x-auto">
           <table class="w-full text-left text-xs">
             <thead>
@@ -121,39 +139,7 @@
               </tr>
             </thead>
             <tbody class="divide-y divide-slate-800">
-              {{ range $name, $hs := .Snapshot.Hostnames }}
-              {{ range $ns, $nsr := $hs.RecordsByNameserver }}
-              <tr class="hover:bg-surface-800/50">
-                <td class="py-2 px-3 text-slate-200 font-medium">
-                  {{ $name }}
-                </td>
-                <td class="py-2 px-3 text-slate-400 break-all">
-                  {{ $ns }}
-                </td>
-                <td class="py-2 px-3">
-                  {{ if eq $nsr.Status "ok" }}
-                  <span
-                    class="inline-block px-1.5 py-0.5 rounded text-[10px] font-bold uppercase bg-teal-900/50 text-teal-400 border border-teal-700/30"
-                    >ok</span
-                  >
-                  {{ else }}
-                  <span
-                    class="inline-block px-1.5 py-0.5 rounded text-[10px] font-bold uppercase bg-red-900/50 text-red-400 border border-red-700/30"
-                    >{{ $nsr.Status }}</span
-                  >
-                  {{ end }}
-                </td>
-                <td
-                  class="py-2 px-3 text-slate-400 break-all max-w-xs"
-                >
-                  {{ formatRecords $nsr.Records }}
-                </td>
-                <td class="py-2 px-3 text-slate-500 whitespace-nowrap">
-                  {{ relTime $nsr.LastChecked }}
-                </td>
-              </tr>
-              {{ end }}
-              {{ end }}
+              {{ template "records" .Hostnames }}
             </tbody>
           </table>
         </div>
@@ -171,19 +157,20 @@
         >
           Ports
         </h2>
-        {{ if .Snapshot.Ports }}
+        {{ if .Ports }}
         <div class="overflow-x-auto">
           <table class="w-full text-left text-xs">
             <thead>
               <tr class="text-slate-500 uppercase tracking-wider">
                 <th class="py-2 px-3">Address</th>
                 <th class="py-2 px-3">State</th>
+                <th class="py-2 px-3">Domains</th>
                 <th class="py-2 px-3">Hostnames</th>
                 <th class="py-2 px-3">Checked</th>
               </tr>
             </thead>
             <tbody class="divide-y divide-slate-800">
-              {{ range $key, $ps := .Snapshot.Ports }}
+              {{ range $key, $ps := .Ports }}
               <tr class="hover:bg-surface-800/50">
                 <td class="py-2 px-3 text-slate-200 font-medium">
                   {{ $key }}
@@ -201,6 +188,9 @@
                   >
                   {{ end }}
                 </td>
+                <td class="py-2 px-3 text-slate-400 break-all">
+                  {{ joinStrings $ps.Domains ", " }}
+                </td>
                 <td class="py-2 px-3 text-slate-400 break-all">
                   {{ joinStrings $ps.Hostnames ", " }}
                 </td>
@@ -258,6 +248,11 @@
                   >
                   {{ end }}
                 </td>
+                {{ if $cs.Error }}
+                <td colspan="3" class="py-2 px-3 text-red-400 break-all">
+                  <div class="max-w-xs">{{ $cs.Error }}</div>
+                </td>
+                {{ else }}
                 <td class="py-2 px-3 text-slate-200">
                   {{ $cs.CommonName }}
                 </td>
@@ -285,6 +280,7 @@
                     {{ end }}
                   {{ end }}
                 </td>
+                {{ end }}
                 <td class="py-2 px-3 text-slate-500 whitespace-nowrap">
                   {{ relTime $cs.LastChecked }}
                 </td>
@@ -363,8 +359,48 @@
         class="text-[11px] text-slate-700 border-t border-slate-800 pt-4 mt-8"
       >
         dnswatcher &middot; monitoring {{ len .Snapshot.Domains }} domains +
-        {{ len .Snapshot.Hostnames }} hostnames
+        {{ len .Hostnames }} hostnames
       </div>
     </div>
   </body>
 </html>
+{{/* ---- One row per nameserver of each name in the map it is given ---- */}}
+{{ define "records" }}
+{{ range $name, $hs := . }}
+{{ range $ns, $nsr := $hs.RecordsByNameserver }}
+<tr class="hover:bg-surface-800/50">
+  <td class="py-2 px-3 text-slate-200 font-medium">
+    {{ $name }}
+  </td>
+  <td class="py-2 px-3 text-slate-400 break-all">
+    {{ $ns }}
+  </td>
+  <td class="py-2 px-3">
+    {{ if eq $nsr.Status "ok" }}
+    <span
+      class="inline-block px-1.5 py-0.5 rounded text-[10px] font-bold uppercase bg-teal-900/50 text-teal-400 border border-teal-700/30"
+      >ok</span
+    >
+    {{ else }}
+    <span
+      class="inline-block px-1.5 py-0.5 rounded text-[10px] font-bold uppercase bg-red-900/50 text-red-400 border border-red-700/30"
+      >{{ $nsr.Status }}</span
+    >
+    {{ end }}
+  </td>
+  <td
+    class="py-2 px-3 text-slate-400 break-all max-w-xs"
+  >
+    {{ if $nsr.Error }}
+    <span class="text-red-400">{{ $nsr.Error }}</span>
+    {{ else }}
+    {{ formatRecords $nsr.Records }}
+    {{ end }}
+  </td>
+  <td class="py-2 px-3 text-slate-500 whitespace-nowrap">
+    {{ relTime $nsr.LastChecked }}
+  </td>
+</tr>
+{{ end }}
+{{ end }}
+{{ end }}

internal/livednstest/livednstest.go

@@ -36,11 +36,18 @@ const (
 	// before the test fails.
 	attempts = 3
 
-	// AttemptTimeout bounds one attempt. Worst case for an operation
-	// is attempts * AttemptTimeout plus the backoff — about 26
-	// seconds, well inside the 90-second `go test -timeout` backstop
-	// even when several operations exhaust their attempts.
-	AttemptTimeout = 8 * time.Second
+	// AttemptTimeout bounds one attempt. It must fit the longest
+	// operation, a watcher check, which sends over a hundred queries one
+	// after another and on a slow build host takes several times as long
+	// as the few seconds it takes on a fast one. An operation whose
+	// every attempt fails takes attempts * AttemptTimeout plus the
+	// backoff, about 56 seconds, after it waits for one of the
+	// Concurrency slots that every live operation in the test binary
+	// shares. So when live DNS does not answer at all, a test binary
+	// with more live operations than slots runs into the 90-second
+	// `go test -timeout` backstop instead of each test failing on its
+	// own.
+	AttemptTimeout = 18 * time.Second
 
 	// backoffBase is the delay after the first failed attempt; it is
 	// multiplied by backoffFactor each time.

internal/notify/delivery_test.go

@@ -6,9 +6,11 @@ import (
 	"encoding/json"
 	"errors"
 	"io"
+	"maps"
 	"net/http"
 	"net/http/httptest"
 	"net/url"
+	"strings"
 	"sync"
 	"testing"
 	"time"
@@ -413,7 +415,8 @@ func sendSlackInfo(
 	svc *notify.Service, target *url.URL,
 ) error {
 	return svc.SendSlack(
-		context.Background(), target, "t", "m", prioInfo,
+		context.Background(), target, notify.ErrSlackFailed,
+		"t", "m", prioInfo,
 	)
 }
 
@@ -506,6 +509,7 @@ func TestSendSlackPayloadFields(t *testing.T) {
 	err := svc.SendSlack(
 		context.Background(),
 		webhookURL,
+		notify.ErrSlackFailed,
 		"Alert Title",
 		"Alert body text",
 		"warning",
@@ -608,7 +612,8 @@ func TestSendSlackAllColors(t *testing.T) {
 
 			err := svc.SendSlack(
 				context.Background(),
-				webhookURL, "t", "m", tc.priority,
+				webhookURL, notify.ErrSlackFailed,
+				"t", "m", tc.priority,
 			)
 			if err != nil {
 				t.Fatalf("SendSlack error: %v", err)
@@ -659,7 +664,8 @@ func TestSendSlackNetworkError(t *testing.T) {
 	)
 
 	err := svc.SendSlack(
-		context.Background(), webhookURL, "t", "m", "info",
+		context.Background(), webhookURL, notify.ErrSlackFailed,
+		"t", "m", "info",
 	)
 	if err == nil {
 		t.Fatal("expected error for network failure")
@@ -1028,6 +1034,66 @@ func TestSendNotificationMattermostError(t *testing.T) {
 	)
 }
 
+// TestSendNotificationErrorNamesEndpoint verifies that, with both
+// Slack and Mattermost set, a failed delivery's logged error names
+// the endpoint that failed. Both are sent by the Slack sender.
+func TestSendNotificationErrorNamesEndpoint(t *testing.T) {
+	t.Parallel()
+
+	srv := httptest.NewServer(
+		http.HandlerFunc(
+			func(w http.ResponseWriter, _ *http.Request) {
+				w.WriteHeader(http.StatusServiceUnavailable)
+			}),
+	)
+	defer srv.Close()
+
+	target, _ := url.Parse(srv.URL)
+
+	svc, logs := newLoggingService(http.DefaultTransport)
+	svc.SetSlackWebhookURL(target)
+	svc.SetMattermostWebhookURL(target)
+	svc.SetSleepFunc(instantSleep)
+	svc.SetRetryConfig(notify.RetryConfig{
+		MaxRetries: 1,
+		BaseDelay:  time.Millisecond,
+		MaxDelay:   time.Millisecond,
+	})
+
+	svc.SendNotification(
+		context.Background(), "t", "m", prioError,
+	)
+
+	waitForCondition(t, func() bool {
+		return svc.OutstandingDeliveries() == 0
+	})
+
+	got := map[string]string{}
+
+	for line := range strings.Lines(logs.String()) {
+		var record struct {
+			Msg      string `json:"msg"`
+			Endpoint string `json:"endpoint"`
+			Error    string `json:"error"`
+		}
+
+		_ = json.Unmarshal([]byte(line), &record)
+
+		if record.Msg == "failed to send notification after retries" {
+			got[record.Endpoint] = record.Error
+		}
+	}
+
+	want := map[string]string{
+		"slack":      "slack notification failed: status 503",
+		"mattermost": "mattermost notification failed: status 503",
+	}
+
+	if !maps.Equal(got, want) {
+		t.Errorf("logged errors = %v, want %v", got, want)
+	}
+}
+
 // ── SlackPayload JSON marshaling ──────────────────────────
 
 func TestSlackPayloadJSON(t *testing.T) {

internal/notify/export_test.go

@@ -85,10 +85,11 @@ func (svc *Service) SendNtfy(
 func (svc *Service) SendSlack(
 	ctx context.Context,
 	webhookURL *url.URL,
+	failed error,
 	title, message, priority string,
 ) error {
 	return svc.sendSlack(
-		ctx, webhookURL, title, message, priority,
+		ctx, webhookURL, failed, title, message, priority,
 	)
 }
 

internal/notify/notify.go

@@ -277,7 +277,8 @@ func (svc *Service) dispatchSlack(
 
 	svc.dispatch(ctx, "slack", func(c context.Context) error {
 		return svc.sendSlack(
-			c, svc.slackWebhookURL, title, message, priority,
+			c, svc.slackWebhookURL, ErrSlackFailed,
+			title, message, priority,
 		)
 	})
 }
@@ -294,7 +295,7 @@ func (svc *Service) dispatchMattermost(
 		ctx, "mattermost",
 		func(c context.Context) error {
 			return svc.sendSlack(
-				c, svc.mattermostWebhookURL,
+				c, svc.mattermostWebhookURL, ErrMattermostFailed,
 				title, message, priority,
 			)
 		},
@@ -370,9 +371,14 @@ type SlackAttachment struct {
 	Text  string `json:"text"`
 }
 
+// sendSlack posts to a Slack or Mattermost incoming webhook, which
+// take the same payload. An HTTP error status is returned wrapped in
+// failed, ErrSlackFailed or ErrMattermostFailed, so the error names
+// the endpoint.
 func (svc *Service) sendSlack(
 	ctx context.Context,
 	webhookURL *url.URL,
+	failed error,
 	title, message, priority string,
 ) error {
 	ctx, cancel := context.WithTimeout(
@@ -420,7 +426,7 @@ func (svc *Service) sendSlack(
 	if resp.StatusCode >= httpStatusClientError {
 		return fmt.Errorf(
 			"%w: status %d",
-			ErrSlackFailed, resp.StatusCode,
+			failed, resp.StatusCode,
 		)
 	}
 

internal/notify/retry.go

@@ -115,7 +115,9 @@ func (svc *Service) deliverWithRetry(
 			"endpoint", endpoint,
 			"attempt", attempt+1,
 			"maxAttempts", cfg.MaxRetries+1,
-			"retryIn", delay,
+			// As text: the JSON log writes a time.Duration as
+			// bare nanoseconds.
+			"retryIn", delay.String(),
 			"error", lastErr,
 		)
 

internal/notify/retry_test.go

@@ -2,6 +2,7 @@ package notify_test
 
 import (
 	"context"
+	"encoding/json"
 	"errors"
 	"net/http"
 	"net/http/httptest"
@@ -189,6 +190,50 @@ func TestDeliverWithRetryExhaustsAttempts(t *testing.T) {
 	}
 }
 
+// TestDeliverWithRetryLogsRetryInAsText checks that the wait
+// before a retry is logged as text such as "1.02s", not as a
+// count of nanoseconds.
+func TestDeliverWithRetryLogsRetryInAsText(t *testing.T) {
+	t.Parallel()
+
+	svc, logs := newLoggingService(http.DefaultTransport)
+	svc.SetRetryConfig(notify.RetryConfig{
+		MaxRetries: 1,
+		BaseDelay:  time.Second,
+		MaxDelay:   time.Second,
+	})
+
+	var waited time.Duration
+
+	svc.SetSleepFunc(func(d time.Duration) <-chan time.Time {
+		waited = d
+
+		return instantSleep(d)
+	})
+
+	_ = svc.DeliverWithRetry(
+		context.Background(), "test",
+		func(_ context.Context) error {
+			return errFail
+		},
+	)
+
+	// With one retry, only the first failure is logged.
+	var record map[string]any
+
+	err := json.Unmarshal([]byte(logs.String()), &record)
+	if err != nil {
+		t.Fatalf("log is not one JSON record: %v\n%s", err, logs)
+	}
+
+	if record["retryIn"] != waited.String() {
+		t.Errorf(
+			"retryIn logged as %v, want %q",
+			record["retryIn"], waited.String(),
+		)
+	}
+}
+
 func TestDeliverWithRetryRespectsContextCancellation(
 	t *testing.T,
 ) {

internal/portcheck/portcheck.go

@@ -193,7 +193,9 @@ func (c *Checker) checkConnection(
 	c.log.Debug(
 		"port check succeeded",
 		"target", target,
-		"latency", latency,
+		// As text: the JSON log writes a time.Duration as bare
+		// nanoseconds.
+		"latency", latency.String(),
 	)
 
 	return &PortResult{

internal/resolver/errors.go

@@ -22,12 +22,30 @@ var (
 		"reply is an error or a referral that leads no closer",
 	)
 
+	// ErrTruncated is the reason given for a reply too large for UDP
+	// whose retry over TCP failed.
+	ErrTruncated = errors.New(
+		"reply truncated and its retry over TCP failed",
+	)
+
+	// ErrIntercepted is returned when every root server refused a
+	// query. Root servers refuse no query, so the refusals came from
+	// something on the network answering in their place.
+	ErrIntercepted = errors.New("this network intercepts DNS queries")
+
 	// ErrCNAMEDepthExceeded is returned when a CNAME chain
 	// exceeds MaxCNAMEDepth.
 	ErrCNAMEDepthExceeded = errors.New(
 		"CNAME chain depth exceeded",
 	)
 
+	// ErrLookupDepthExceeded is returned when nameserver addresses
+	// were not looked up because lookups were already maxLookupDepth
+	// deep, one inside another.
+	ErrLookupDepthExceeded = errors.New(
+		"lookups of nameserver addresses go too deep",
+	)
+
 	// ErrContextCanceled wraps context cancellation for the
 	// resolver's iterative queries.
 	ErrContextCanceled = errors.New("context canceled")

internal/resolver/export_test.go

@@ -2,15 +2,33 @@ package resolver
 
 import (
 	"context"
+	"log/slog"
+	"time"
 
 	"github.com/miekg/dns"
 )
 
+// NewWithFailingTCP returns a Resolver whose TCP client gives up before
+// it can connect, so the retry over TCP of every truncated reply fails.
+func NewWithFailingTCP(log *slog.Logger) *Resolver {
+	r := NewFromLogger(log)
+	r.tcp = &tcpClient{timeout: time.Nanosecond}
+
+	return r
+}
+
 // ExtractRecordValue exports extractRecordValue for testing.
 func ExtractRecordValue(rr dns.RR) string {
 	return extractRecordValue(rr)
 }
 
+// CollectAnswerRecords exports collectAnswerRecords for testing.
+func CollectAnswerRecords(msg *dns.Msg, resp *NameserverResponse) {
+	var state queryState
+
+	collectAnswerRecords(msg, resp, &state)
+}
+
 // UsableReply exports usableReply for testing.
 func UsableReply(resp *dns.Msg, zone string, name string) bool {
 	return usableReply(resp, zone, name)
@@ -28,21 +46,53 @@ func CollectIPs(
 	return collectIPs(results)
 }
 
+// QueryServers exports queryServers for testing.
+func (r *Resolver) QueryServers(
+	ctx context.Context,
+	servers []string,
+	zone string,
+	name string,
+	qtype uint16,
+) (*dns.Msg, error) {
+	return r.queryServers(ctx, servers, zone, name, qtype)
+}
+
 // QueryEachNS exports queryEachNS for testing.
 func (r *Resolver) QueryEachNS(
 	ctx context.Context,
 	nameservers []string,
 	hostname string,
 ) (map[string]*NameserverResponse, error) {
-	return r.queryEachNS(ctx, nameservers, hostname)
+	return r.queryEachNS(ctx, nameservers, hostname, recordTypes())
 }
 
-// ResolveNSIPs exports resolveNSIPs for testing.
+// ResolveNSIPs exports resolveNSIPs for testing, looking each name up
+// as a lookup that no other lookup started.
 func (r *Resolver) ResolveNSIPs(
 	ctx context.Context,
 	nsNames []string,
 ) []string {
-	return r.resolveNSIPs(ctx, nsNames)
+	ips, _ := r.resolveNSIPs(ctx, nsNames, 1)
+
+	return ips
+}
+
+// MaxLookupDepth exports maxLookupDepth for testing.
+const MaxLookupDepth = maxLookupDepth
+
+// QueryZone exports queryZone for testing.
+func (r *Resolver) QueryZone(
+	ctx context.Context,
+	given []string,
+	withoutAddresses []string,
+	zone string,
+	name string,
+	qtype uint16,
+	depth int,
+) (*dns.Msg, error) {
+	return r.queryZone(
+		ctx, given, withoutAddresses, zone, name, qtype, depth,
+	)
 }
 
 // RootServerList exports rootServerList for testing.

internal/resolver/iterative.go

@@ -8,6 +8,7 @@ import (
 	"net"
 	"slices"
 	"sort"
+	"strconv"
 	"strings"
 	"time"
 
@@ -19,6 +20,16 @@ const (
 	maxRetries           = 2
 	maxDelegation        = 20
 	timeoutMultiplier    = 2
+
+	// maxLookupDepth is how many lookups of nameserver addresses may be
+	// under way one inside another. Looking up a nameserver's address
+	// can meet a referral that names nameservers without their
+	// addresses, which are then looked up in turn; without a limit,
+	// delegations that point at each other would never end. Each level
+	// multiplies the queries sent. pool.ntp.org needs three: the
+	// address of its nameserver g.ntpns.org can need a.ntpns.org's,
+	// which needs a bitnames.com nameserver's.
+	maxLookupDepth = 3
 )
 
 // ErrRefused is returned when a DNS server refuses a query.
@@ -89,6 +100,9 @@ func (r *Resolver) tryExchange(
 	return resp, err
 }
 
+// retryTCP returns the reply to msg over TCP when resp, its reply over
+// UDP, is truncated. When that fails it returns resp, still truncated,
+// which holds only the records that fit.
 func (r *Resolver) retryTCP(
 	ctx context.Context,
 	msg *dns.Msg,
@@ -107,9 +121,8 @@ func (r *Resolver) retryTCP(
 	return resp
 }
 
-// queryDNS sends a DNS query to a specific server IP.
-// Tries non-recursive first, falls back to recursive on
-// REFUSED (handles DNS interception environments).
+// queryDNS sends a DNS query to a specific server IP, never asking it
+// for recursion. A reply of REFUSED is returned as ErrRefused.
 func (r *Resolver) queryDNS(
 	ctx context.Context,
 	serverIP string,
@@ -133,25 +146,12 @@ func (r *Resolver) queryDNS(
 	}
 
 	if resp.Rcode == dns.RcodeRefused {
-		msg.RecursionDesired = true
-
-		resp, err = r.tryExchange(ctx, msg, addr)
-		if err != nil {
-			return nil, fmt.Errorf(
-				"query %s @%s: %w", name, serverIP, err,
-			)
-		}
-
-		if resp.Rcode == dns.RcodeRefused {
-			return nil, fmt.Errorf(
-				"query %s @%s: %w", name, serverIP, ErrRefused,
-			)
-		}
+		return nil, fmt.Errorf(
+			"query %s @%s: %w", name, serverIP, ErrRefused,
+		)
 	}
 
-	resp = r.retryTCP(ctx, msg, addr, resp)
-
-	return resp, nil
+	return r.retryTCP(ctx, msg, addr, resp), nil
 }
 
 func extractNSSet(rrs []dns.RR) []string {
@@ -212,13 +212,15 @@ func (r *Resolver) followDelegation(
 	// servers are the root servers, the servers of zone ".".
 	zone := "."
 
+	var withoutAddresses []string
+
 	for range maxDelegation {
 		if checkCtx(ctx) != nil {
 			return nil, ErrContextCanceled
 		}
 
-		resp, err := r.queryServers(
-			ctx, servers, zone, domain, dns.TypeNS,
+		resp, err := r.queryZone(
+			ctx, servers, withoutAddresses, zone, domain, dns.TypeNS, 0,
 		)
 		if err != nil {
 			return nil, err
@@ -243,18 +245,7 @@ func (r *Resolver) followDelegation(
 			return r.resolveNSIterative(ctx, domain)
 		}
 
-		glue := extractGlue(resp.Extra)
-		nextServers := glueIPs(authNS, glue)
-
-		if len(nextServers) == 0 {
-			nextServers = r.resolveNSIPs(ctx, authNS)
-		}
-
-		if len(nextServers) == 0 {
-			return nil, ErrNoNameservers
-		}
-
-		servers = nextServers
+		servers, withoutAddresses = referralNameservers(resp)
 		zone = referralZone(resp)
 	}
 
@@ -279,7 +270,9 @@ func shuffled(
 
 // queryServers asks servers, the servers of zone, about name in a random
 // order until one gives a usable reply. A server that times out, refuses
-// or gives a reply that is not usable is passed over for the next.
+// or gives a reply that is not usable is passed over for the next. When
+// every server refused, the error says so, and when they are the root
+// servers it is ErrIntercepted.
 func (r *Resolver) queryServers(
 	ctx context.Context,
 	servers []string,
@@ -289,6 +282,8 @@ func (r *Resolver) queryServers(
 ) (*dns.Msg, error) {
 	var lastErr error
 
+	refused := 0
+
 	for _, ip := range shuffled(servers, rand.Shuffle) {
 		if checkCtx(ctx) != nil {
 			return nil, ErrContextCanceled
@@ -305,19 +300,44 @@ func (r *Resolver) queryServers(
 			return resp, nil
 		}
 
+		if errors.Is(err, ErrRefused) {
+			refused++
+		}
+
 		lastErr = err
 	}
 
+	if refused == len(servers) && zone == "." {
+		return nil, fmt.Errorf(
+			"every root server refused a query for %s: %w",
+			name, ErrIntercepted,
+		)
+	}
+
+	if refused == len(servers) {
+		return nil, fmt.Errorf(
+			"every server of %s refused a query for %s: %w",
+			zone, name, ErrRefused,
+		)
+	}
+
 	return nil, fmt.Errorf("all servers failed: %w", lastErr)
 }
 
+// isErrorReply reports whether msg is an error reply: one with any code
+// but NOERROR and NXDOMAIN, such as SERVFAIL, NOTIMP or FORMERR. An error
+// reply says nothing about the name's records.
+func isErrorReply(msg *dns.Msg) bool {
+	return msg.Rcode != dns.RcodeSuccess && msg.Rcode != dns.RcodeNameError
+}
+
 // usableReply reports whether resp, a reply from one of the servers of
 // zone to a query about name, is usable. An error reply such as SERVFAIL
 // is not. Nor is a referral, unless it refers the query to a zone below
 // zone that name is in: a server that refers it back to zone, up or
 // sideways does not serve zone as it should.
 func usableReply(resp *dns.Msg, zone string, name string) bool {
-	if resp.Rcode != dns.RcodeSuccess && resp.Rcode != dns.RcodeNameError {
+	if isErrorReply(resp) {
 		return false
 	}
 
@@ -358,24 +378,110 @@ func nsSetFrom(resp *dns.Msg, domain string) []string {
 	return extractNSSet(resp.Answer)
 }
 
-// resolveNSIPs returns the addresses of every nameserver in nsNames
-// whose name resolves, for a referral that carries none. The walk can
-// then go on to the zone's other nameservers when one gives no usable
-// reply.
-func (r *Resolver) resolveNSIPs(
-	ctx context.Context,
-	nsNames []string,
-) []string {
-	var ips []string
+// referralNameservers returns the IPv4 addresses that resp, a referral,
+// gives for the nameservers it names, and the names of the nameservers
+// it gives no address for.
+func referralNameservers(resp *dns.Msg) ([]string, []string) {
+	glue := extractGlue(resp.Extra)
 
-	for _, ns := range nsNames {
-		resolved, err := r.resolveARecord(ctx, ns)
+	var given, withoutAddresses []string
+
+	for _, ns := range extractNSSet(resp.Ns) {
+		ips := glueIPs([]string{ns}, glue)
+		if len(ips) == 0 {
+			withoutAddresses = append(withoutAddresses, ns)
+		}
+
+		given = append(given, ips...)
+	}
+
+	return given, withoutAddresses
+}
+
+// queryZone asks the servers of zone about name as queryServers does:
+// first those at given, the addresses a referral gave, and only when
+// none of them gives a usable reply, the nameservers named
+// withoutAddresses, once their addresses are looked up. depth is how
+// many lookups of a nameserver's address are under way, 0 in the walk
+// to a domain's nameservers; at maxLookupDepth, no address is looked
+// up. When the limit is why none was found, here or in a lookup this
+// one started, the error is ErrLookupDepthExceeded.
+func (r *Resolver) queryZone(
+	ctx context.Context,
+	given []string,
+	withoutAddresses []string,
+	zone string,
+	name string,
+	qtype uint16,
+	depth int,
+) (*dns.Msg, error) {
+	err := fmt.Errorf(
+		"no address for any nameserver of %s: %w", zone, ErrNoNameservers,
+	)
+
+	if len(given) > 0 {
+		var resp *dns.Msg
+
+		resp, err = r.queryServers(ctx, given, zone, name, qtype)
 		if err == nil {
-			ips = append(ips, resolved...)
+			return resp, nil
 		}
 	}
 
-	return ips
+	if len(withoutAddresses) == 0 {
+		return nil, err
+	}
+
+	if depth >= maxLookupDepth {
+		return nil, fmt.Errorf(
+			"addresses of the nameservers of %s not looked up: %w",
+			zone, ErrLookupDepthExceeded,
+		)
+	}
+
+	lookedUp, limitErr := r.resolveNSIPs(ctx, withoutAddresses, depth+1)
+	if limitErr != nil {
+		return nil, limitErr
+	}
+
+	if len(lookedUp) == 0 {
+		return nil, err
+	}
+
+	return r.queryServers(ctx, lookedUp, zone, name, qtype)
+}
+
+// resolveNSIPs returns the addresses of every nameserver in nsNames
+// whose name resolves, each looked up at depth (see resolveARecord).
+// The walk can then go on to the zone's other nameservers when one
+// gives no usable reply. When none resolves and the depth limit
+// stopped one of the lookups, it returns that lookup's error.
+func (r *Resolver) resolveNSIPs(
+	ctx context.Context,
+	nsNames []string,
+	depth int,
+) ([]string, error) {
+	var (
+		ips      []string
+		limitErr error
+	)
+
+	for _, ns := range nsNames {
+		resolved, err := r.resolveARecord(ctx, ns, depth)
+
+		switch {
+		case err == nil:
+			ips = append(ips, resolved...)
+		case errors.Is(err, ErrLookupDepthExceeded):
+			limitErr = err
+		}
+	}
+
+	if len(ips) > 0 {
+		return ips, nil
+	}
+
+	return nil, limitErr
 }
 
 // resolveNSIterative queries for NS records using iterative
@@ -430,11 +536,14 @@ func (r *Resolver) resolveNSIterative(
 	return nil, ErrNoNameservers
 }
 
-// resolveARecord resolves a hostname to IPv4 addresses using
-// iterative resolution through the delegation chain.
+// resolveARecord resolves a hostname, a nameserver's name, to IPv4
+// addresses using iterative resolution through the delegation chain.
+// depth is how many lookups of a nameserver's address are under way,
+// this one included: 1 for a lookup that no other lookup started.
 func (r *Resolver) resolveARecord(
 	ctx context.Context,
 	hostname string,
+	depth int,
 ) ([]string, error) {
 	if checkCtx(ctx) != nil {
 		return nil, ErrContextCanceled
@@ -444,13 +553,16 @@ func (r *Resolver) resolveARecord(
 	servers := rootServerList()
 	zone := "."
 
+	var withoutAddresses []string
+
 	for range maxDelegation {
 		if checkCtx(ctx) != nil {
 			return nil, ErrContextCanceled
 		}
 
-		resp, err := r.queryServers(
-			ctx, servers, zone, hostname, dns.TypeA,
+		resp, err := r.queryZone(
+			ctx, servers, withoutAddresses, zone, hostname, dns.TypeA,
+			depth,
 		)
 		if err != nil {
 			return nil, fmt.Errorf(
@@ -477,17 +589,7 @@ func (r *Resolver) resolveARecord(
 			break
 		}
 
-		glue := extractGlue(resp.Extra)
-		nextServers := glueIPs(authNS, glue)
-
-		if len(nextServers) == 0 {
-			// Resolve NS IPs iteratively — but guard
-			// against infinite recursion by using only
-			// already-resolved servers.
-			break
-		}
-
-		servers = nextServers
+		servers, withoutAddresses = referralNameservers(resp)
 		zone = referralZone(resp)
 	}
 
@@ -528,30 +630,62 @@ func (r *Resolver) FindAuthoritativeNameservers(
 
 			return nsNames, nil
 		}
+
+		// The root servers would refuse every parent name too.
+		if errors.Is(err, ErrIntercepted) {
+			return nil, err
+		}
 	}
 
 	return nil, ErrNoNameservers
 }
 
+// recordTypes returns the record types a nameserver is asked for when a
+// name is checked.
+func recordTypes() []uint16 {
+	return []uint16{
+		dns.TypeA, dns.TypeAAAA, dns.TypeCNAME,
+		dns.TypeMX, dns.TypeTXT, dns.TypeSRV,
+		dns.TypeCAA, dns.TypeNS,
+	}
+}
+
+// addressTypes returns the record types ResolveIPAddresses asks for,
+// the only ones it reads.
+func addressTypes() []uint16 {
+	return []uint16{dns.TypeA, dns.TypeAAAA, dns.TypeCNAME}
+}
+
 // QueryNameserver queries a specific nameserver for all record
 // types and builds a NameserverResponse.
 func (r *Resolver) QueryNameserver(
 	ctx context.Context,
 	nsHostname string,
 	hostname string,
+) (*NameserverResponse, error) {
+	return r.queryNameserver(ctx, nsHostname, hostname, recordTypes())
+}
+
+// queryNameserver queries a specific nameserver for the record types
+// in qtypes and builds a NameserverResponse.
+func (r *Resolver) queryNameserver(
+	ctx context.Context,
+	nsHostname string,
+	hostname string,
+	qtypes []uint16,
 ) (*NameserverResponse, error) {
 	if checkCtx(ctx) != nil {
 		return nil, ErrContextCanceled
 	}
 
-	nsIPs, err := r.resolveARecord(ctx, nsHostname)
+	nsIPs, err := r.resolveARecord(ctx, nsHostname, 1)
 	if err != nil {
 		return nil, fmt.Errorf("resolving NS %s: %w", nsHostname, err)
 	}
 
 	hostname = dns.Fqdn(hostname)
 
-	return r.queryAllTypes(ctx, nsHostname, nsIPs[0], hostname)
+	return r.queryTypes(ctx, nsHostname, nsIPs[0], hostname, qtypes)
 }
 
 // QueryNameserverIP queries a nameserver by its IP address directly,
@@ -568,14 +702,15 @@ func (r *Resolver) QueryNameserverIP(
 
 	hostname = dns.Fqdn(hostname)
 
-	return r.queryAllTypes(ctx, nsHostname, nsIP, hostname)
+	return r.queryTypes(ctx, nsHostname, nsIP, hostname, recordTypes())
 }
 
-func (r *Resolver) queryAllTypes(
+func (r *Resolver) queryTypes(
 	ctx context.Context,
 	nsHostname string,
 	nsIP string,
 	hostname string,
+	qtypes []uint16,
 ) (*NameserverResponse, error) {
 	resp := &NameserverResponse{
 		Nameserver: nsHostname,
@@ -583,12 +718,6 @@ func (r *Resolver) queryAllTypes(
 		Status:     StatusOK,
 	}
 
-	qtypes := []uint16{
-		dns.TypeA, dns.TypeAAAA, dns.TypeCNAME,
-		dns.TypeMX, dns.TypeTXT, dns.TypeSRV,
-		dns.TypeCAA, dns.TypeNS,
-	}
-
 	state := r.queryEachType(ctx, nsIP, hostname, qtypes, resp)
 	classifyResponse(resp, state)
 
@@ -596,15 +725,21 @@ func (r *Resolver) queryAllTypes(
 }
 
 type queryState struct {
-	gotNXDomain bool
-	gotSERVFAIL bool
-	gotRefused  bool
-	gotTimeout  bool
-	gotReferral bool
-	netErr      error
-	hasRecords  bool
+	gotNXDomain   bool
+	gotErrorReply bool
+	errorReply    string // its code, such as SERVFAIL, or number if unnamed
+	gotRefused    bool
+	gotTimeout    bool
+	gotReferral   bool
+	netErr        error
+	hasRecords    bool
+	answered      bool
 }
 
+// queryEachType asks the nameserver at nsIP about hostname once for each
+// record type in qtypes, and lists in resp.FailedTypes the types whose
+// query got no usable reply, logging each with the reason unless ctx was
+// cancelled: shutdown cancels it, and a query it cut short did not fail.
 func (r *Resolver) queryEachType(
 	ctx context.Context,
 	nsIP string,
@@ -619,7 +754,34 @@ func (r *Resolver) queryEachType(
 			break
 		}
 
-		r.querySingleType(ctx, nsIP, hostname, qtype, resp, &state)
+		err := r.querySingleType(ctx, nsIP, hostname, qtype, resp, &state)
+		if err == nil {
+			state.answered = true
+
+			continue
+		}
+
+		rtype := dns.TypeToString[qtype]
+		resp.FailedTypes = append(resp.FailedTypes, rtype)
+
+		if errors.Is(ctx.Err(), context.Canceled) {
+			continue
+		}
+
+		r.log.Warn(
+			"record type query failed",
+			"hostname", hostname,
+			"nameserver", resp.Nameserver,
+			"type", rtype,
+			"error", err,
+		)
+	}
+
+	// The reply about another type can carry the name's CNAME. When the
+	// query for CNAME itself failed, that is left out too, so Records
+	// holds nothing for a failed type.
+	for _, rtype := range resp.FailedTypes {
+		delete(resp.Records, rtype)
 	}
 
 	for k := range resp.Records {
@@ -629,6 +791,9 @@ func (r *Resolver) queryEachType(
 	return state
 }
 
+// querySingleType asks the nameserver at nsIP about hostname's records
+// of type qtype. It returns nil when the nameserver answered: with
+// records, with none, or with NXDOMAIN; otherwise it returns why not.
 func (r *Resolver) querySingleType(
 	ctx context.Context,
 	nsIP string,
@@ -636,7 +801,7 @@ func (r *Resolver) querySingleType(
 	qtype uint16,
 	resp *NameserverResponse,
 	state *queryState,
-) {
+) error {
 	msg, err := r.queryDNS(ctx, nsIP, hostname, qtype)
 	if err != nil {
 		switch {
@@ -648,19 +813,40 @@ func (r *Resolver) querySingleType(
 			state.netErr = err
 		}
 
-		return
+		return err
 	}
 
+	return readReply(msg, resp, state)
+}
+
+// readReply adds to resp the records in msg, a nameserver's reply to a
+// query about one record type. It returns nil when the nameserver
+// answered: with records, with none, or with NXDOMAIN; otherwise it
+// returns why not.
+func readReply(
+	msg *dns.Msg,
+	resp *NameserverResponse,
+	state *queryState,
+) error {
 	if msg.Rcode == dns.RcodeNameError {
 		state.gotNXDomain = true
 
-		return
+		return nil
 	}
 
-	if msg.Rcode == dns.RcodeServerFailure {
-		state.gotSERVFAIL = true
+	if isErrorReply(msg) {
+		state.gotErrorReply = true
 
-		return
+		code, named := dns.RcodeToString[msg.Rcode]
+		if !named {
+			code = strconv.Itoa(msg.Rcode)
+		}
+
+		state.errorReply = code
+
+		return fmt.Errorf(
+			"server returned %s: %w", state.errorReply, ErrUnusableReply,
+		)
 	}
 
 	// A reply with no answer that lists other nameservers, from a server
@@ -673,12 +859,26 @@ func (r *Resolver) querySingleType(
 		len(extractNSSet(msg.Ns)) > 0 {
 		state.gotReferral = true
 
-		return
+		return fmt.Errorf("server returned a referral: %w", ErrUnusableReply)
+	}
+
+	// A reply still truncated is one whose TCP retry failed, and holds
+	// only the records that fit.
+	if msg.Truncated {
+		state.netErr = ErrTruncated
+
+		return ErrTruncated
 	}
 
 	collectAnswerRecords(msg, resp, state)
+
+	return nil
 }
 
+// collectAnswerRecords adds the records in msg's answer to resp, each
+// value once per record type. For a name with a CNAME, a nameserver
+// answers a query of any type with that CNAME, so the same value comes
+// in the answer to every type asked for.
 func collectAnswerRecords(
 	msg *dns.Msg,
 	resp *NameserverResponse,
@@ -691,9 +891,12 @@ func collectAnswerRecords(
 		}
 
 		typeName := dns.TypeToString[rr.Header().Rrtype]
-		resp.Records[typeName] = append(
-			resp.Records[typeName], val,
-		)
+		if !slices.Contains(resp.Records[typeName], val) {
+			resp.Records[typeName] = append(
+				resp.Records[typeName], val,
+			)
+		}
+
 		state.hasRecords = true
 	}
 }
@@ -708,23 +911,26 @@ func isTimeout(err error) bool {
 	return false
 }
 
+// classifyResponse sets the nameserver's status. One that answered no
+// record type has failed, and Error says why; one that answered some has
+// the status of those answers.
 func classifyResponse(resp *NameserverResponse, state queryState) {
 	switch {
 	case state.gotNXDomain && !state.hasRecords:
 		resp.Status = StatusNXDomain
-	case state.gotTimeout && !state.hasRecords:
+	case state.gotTimeout && !state.answered:
 		resp.Status = StatusTimeout
 		resp.Error = "all queries timed out"
-	case state.gotSERVFAIL && !state.hasRecords:
+	case state.gotErrorReply && !state.answered:
 		resp.Status = StatusError
-		resp.Error = "server returned SERVFAIL"
-	case state.gotRefused && !state.hasRecords:
+		resp.Error = "server returned " + state.errorReply
+	case state.gotRefused && !state.answered:
 		resp.Status = StatusError
 		resp.Error = "server returned REFUSED"
-	case state.netErr != nil && !state.hasRecords:
+	case state.netErr != nil && !state.answered:
 		resp.Status = StatusError
 		resp.Error = "network error: " + state.netErr.Error()
-	case state.gotReferral && !state.hasRecords:
+	case state.gotReferral && !state.answered:
 		resp.Status = StatusError
 		resp.Error = "server returned a referral"
 	case !state.hasRecords && !state.gotNXDomain:
@@ -779,18 +985,19 @@ func (r *Resolver) QueryAllNameservers(
 		return nil, err
 	}
 
-	return r.queryEachNS(ctx, nameservers, hostname)
+	return r.queryEachNS(ctx, nameservers, hostname, recordTypes())
 }
 
 func (r *Resolver) queryEachNS(
 	ctx context.Context,
 	nameservers []string,
 	hostname string,
+	qtypes []uint16,
 ) (map[string]*NameserverResponse, error) {
 	results := make(map[string]*NameserverResponse)
 
 	for _, ns := range nameservers {
-		resp, err := r.QueryNameserver(ctx, ns, hostname)
+		resp, err := r.queryNameserver(ctx, ns, hostname, qtypes)
 
 		// A query the context cut short says nothing about the
 		// nameserver, so it must not be returned as its failure.
@@ -835,9 +1042,10 @@ func (r *Resolver) LookupAllRecords(
 }
 
 // ResolveIPAddresses resolves a hostname to all IPv4 and IPv6
-// addresses, following CNAME chains up to MaxCNAMEDepth. When no
-// nameserver of the name's zone answered, it returns an error rather
-// than no addresses.
+// addresses, following CNAME chains up to MaxCNAMEDepth. It asks each
+// nameserver of the name's zone for its A, AAAA and CNAME records only.
+// When no nameserver of the name's zone answered, it returns an error
+// rather than no addresses.
 func (r *Resolver) ResolveIPAddresses(
 	ctx context.Context,
 	hostname string,
@@ -858,7 +1066,12 @@ func (r *Resolver) resolveIPWithCNAME(
 		return nil, ErrCNAMEDepthExceeded
 	}
 
-	results, err := r.QueryAllNameservers(ctx, hostname)
+	nameservers, err := r.FindAuthoritativeNameservers(ctx, hostname)
+	if err != nil {
+		return nil, err
+	}
+
+	results, err := r.queryEachNS(ctx, nameservers, hostname, addressTypes())
 	if err != nil {
 		return nil, err
 	}
@@ -878,9 +1091,11 @@ func (r *Resolver) resolveIPWithCNAME(
 }
 
 // collectIPs returns the addresses in the nameservers' answers and the
-// first CNAME target among them. It returns ErrNoNameserverAnswered when
-// every nameserver timed out, failed or returned a referral: that is not
-// a name with no addresses.
+// first CNAME target among them. A nameserver whose query for one of the
+// types failed gave only part of the addresses, and is left out. It
+// returns ErrNoNameserverAnswered when every nameserver timed out,
+// failed, returned a referral or was left out: that is not a name with
+// no addresses.
 func collectIPs(
 	results map[string]*NameserverResponse,
 ) ([]string, string, error) {
@@ -893,7 +1108,8 @@ func collectIPs(
 	answered := false
 
 	for _, resp := range results {
-		if resp.Status == StatusTimeout || resp.Status == StatusError {
+		if resp.Status == StatusTimeout || resp.Status == StatusError ||
+			len(resp.FailedTypes) > 0 {
 			continue
 		}
 

internal/resolver/iterative_internal_test.go

@@ -0,0 +1,131 @@
+package resolver
+
+import (
+	"strconv"
+	"syscall"
+	"testing"
+
+	"github.com/miekg/dns"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// TestClassifyResponse sets a nameserver's status from the results of
+// its queries, built here. One that answered some record types, even
+// with no records, has not failed when its query for another type got
+// no usable reply, whatever the reason; one whose every query got none
+// has.
+func TestClassifyResponse(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name       string
+		results    queryState
+		wantStatus string
+		wantError  string
+	}{
+		{
+			"some types answered with no records, another timed out",
+			queryState{answered: true, gotTimeout: true},
+			StatusNoData, "",
+		},
+		{
+			"some types answered with no records, another got SERVFAIL",
+			queryState{
+				answered: true, gotErrorReply: true, errorReply: "SERVFAIL",
+			},
+			StatusNoData, "",
+		},
+		{
+			"some types answered with no records, another was refused",
+			queryState{answered: true, gotRefused: true},
+			StatusNoData, "",
+		},
+		{
+			"some types answered with no records, another got a network error",
+			queryState{answered: true, netErr: syscall.ECONNREFUSED},
+			StatusNoData, "",
+		},
+		{
+			"some types answered with no records, another's reply was " +
+				"truncated and its retry over TCP failed",
+			queryState{answered: true, netErr: ErrTruncated},
+			StatusNoData, "",
+		},
+		{
+			"some types answered with no records, another got a referral",
+			queryState{answered: true, gotReferral: true},
+			StatusNoData, "",
+		},
+		{
+			"every query timed out",
+			queryState{gotTimeout: true},
+			StatusTimeout, "all queries timed out",
+		},
+		{
+			"every query got NOTIMP",
+			queryState{gotErrorReply: true, errorReply: "NOTIMP"},
+			StatusError, "server returned NOTIMP",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			resp := &NameserverResponse{Status: StatusOK}
+			classifyResponse(resp, tt.results)
+
+			assert.Equal(t, tt.wantStatus, resp.Status)
+			assert.Equal(t, tt.wantError, resp.Error)
+		})
+	}
+}
+
+// TestReadReply checks which replies to a query about one record type,
+// built here, are an answer: one with the code NOERROR or NXDOMAIN. A
+// reply with any other code is not, and the type's query has failed; a
+// nameserver whose only reply it is has failed, and Error gives the
+// code, or its number when the code has no name.
+func TestReadReply(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		rcode      int
+		wantStatus string
+		wantError  string
+	}{
+		{dns.RcodeSuccess, StatusNoData, ""},
+		{dns.RcodeNameError, StatusNXDomain, ""},
+		{dns.RcodeServerFailure, StatusError, "server returned SERVFAIL"},
+		{dns.RcodeNotImplemented, StatusError, "server returned NOTIMP"},
+		{dns.RcodeFormatError, StatusError, "server returned FORMERR"},
+		{12, StatusError, "server returned 12"}, // unassigned, no name
+	}
+
+	for _, tt := range tests {
+		t.Run(strconv.Itoa(tt.rcode), func(t *testing.T) {
+			t.Parallel()
+
+			msg := new(dns.Msg)
+			msg.Authoritative = true
+			msg.Rcode = tt.rcode
+
+			resp := &NameserverResponse{Records: map[string][]string{}}
+
+			var state queryState
+
+			err := readReply(msg, resp, &state)
+			classifyResponse(resp, state)
+
+			if tt.wantStatus == StatusError {
+				require.ErrorIs(t, err, ErrUnusableReply)
+			} else {
+				require.NoError(t, err)
+			}
+
+			assert.Equal(t, tt.wantStatus, resp.Status)
+			assert.Equal(t, tt.wantError, resp.Error)
+		})
+	}
+}

internal/resolver/iterative_test.go

@@ -43,6 +43,25 @@ func TestCollectIPs_FailedIsNoAnswer(t *testing.T) {
 	assert.Empty(t, ips)
 }
 
+// TestCollectIPs_FailedTypeIsNoAnswer checks that a nameserver whose
+// query for one of the types failed is no answer: its addresses are
+// only part of them.
+func TestCollectIPs_FailedTypeIsNoAnswer(t *testing.T) {
+	t.Parallel()
+
+	ips, _, err := resolver.CollectIPs(
+		map[string]*resolver.NameserverResponse{
+			nsExample1: {
+				Records:     map[string][]string{"A": {"192.0.2.1"}},
+				FailedTypes: []string{"AAAA"},
+				Status:      resolver.StatusOK,
+			},
+		},
+	)
+	require.ErrorIs(t, err, resolver.ErrNoNameserverAnswered)
+	assert.Empty(t, ips)
+}
+
 const (
 	// exampleCom is the zone most cases of TestUsableReply and
 	// TestNSSetFrom are about, and wwwExampleCom a name in it.
@@ -238,6 +257,38 @@ func TestExtractRecordValue_LetterCase(t *testing.T) {
 	}
 }
 
+// TestCollectAnswerRecords_CNAMEOnce collects the answers a nameserver
+// gives for a name with a CNAME, one for each record type a check asks
+// for. Each answer holds the CNAME, which must be stored once.
+func TestCollectAnswerRecords_CNAMEOnce(t *testing.T) {
+	t.Parallel()
+
+	cname := &dns.CNAME{
+		Hdr: dns.RR_Header{
+			Name: "git.eeqj.de.", Rrtype: dns.TypeCNAME, Class: dns.ClassINET,
+		},
+		Target: "fsn1app1.datavi.be.",
+	}
+
+	resp := &resolver.NameserverResponse{Records: map[string][]string{}}
+
+	for _, qtype := range []uint16{
+		dns.TypeA, dns.TypeAAAA, dns.TypeCNAME, dns.TypeMX,
+		dns.TypeTXT, dns.TypeSRV, dns.TypeCAA, dns.TypeNS,
+	} {
+		msg := new(dns.Msg)
+		msg.SetQuestion("git.eeqj.de.", qtype)
+		msg.Answer = []dns.RR{cname}
+
+		resolver.CollectAnswerRecords(msg, resp)
+	}
+
+	assert.Equal(t,
+		map[string][]string{"CNAME": {"fsn1app1.datavi.be."}},
+		resp.Records,
+	)
+}
+
 // TestShuffled shuffles the root servers with many seeds. Every order
 // must hold each root server once, so each is tried before a
 // resolution fails; each root server must come first for some seed, so

internal/resolver/livedns_test.go

@@ -226,11 +226,17 @@ func liveLookupNS(
 // liveQueryNameserver queries one nameserver, retrying while that
 // nameserver fails to answer. NXDOMAIN and NODATA are answers and
 // are returned to the caller to assert on.
+//
+// QueryNameserver sends one query per record type, so one lost query
+// leaves its type out of an answer that is otherwise fine. A test names
+// in types the record types it reads; an answer holding records of none
+// of them is retried too.
 func liveQueryNameserver(
 	t *testing.T,
 	r *resolver.Resolver,
 	nameserver string,
 	hostname string,
+	types ...string,
 ) *resolver.NameserverResponse {
 	t.Helper()
 
@@ -260,6 +266,18 @@ func liveQueryNameserver(
 				)
 			}
 
+			hasRecords := func(recordType string) bool {
+				return len(resp.Records[recordType]) > 0
+			}
+
+			if len(types) > 0 && !slices.ContainsFunc(types, hasRecords) {
+				return fmt.Errorf(
+					"%w: %s returned no %s records",
+					livednstest.ErrNoAnswer, nameserver,
+					strings.Join(types, " or "),
+				)
+			}
+
 			out = resp
 
 			return nil

internal/resolver/resolver.go

@@ -31,11 +31,15 @@ type Params struct {
 }
 
 // NameserverResponse holds one nameserver's response for a query.
+// FailedTypes lists the record types whose query got no usable reply,
+// and Records holds nothing for them: their records are not known. When
+// no record type got one, Status and Error say the nameserver failed.
 type NameserverResponse struct {
-	Nameserver string
-	Records    map[string][]string
-	Status     string
-	Error      string
+	Nameserver  string
+	Records     map[string][]string
+	FailedTypes []string
+	Status      string
+	Error       string
 }
 
 // Resolver performs iterative DNS resolution from root servers.

internal/resolver/resolver_test.go

@@ -1,7 +1,9 @@
 package resolver_test
 
 import (
+	"bytes"
 	"context"
+	"errors"
 	"fmt"
 	"log/slog"
 	"net"
@@ -11,6 +13,7 @@ import (
 	"testing"
 	"time"
 
+	"github.com/miekg/dns"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
@@ -161,6 +164,112 @@ func TestResolveNSIPs_EveryNameserver(t *testing.T) {
 	assert.ElementsMatch(t, want, got)
 }
 
+// TestResolveNSIPs_ZoneDelegatedWithoutAddresses looks up the address
+// of a.ntpns.org, a nameserver of pool.ntp.org. The org servers delegate
+// ntpns.org to nameservers in other zones and give none of their
+// addresses, so those are looked up on the way.
+func TestResolveNSIPs_ZoneDelegatedWithoutAddresses(t *testing.T) {
+	t.Parallel()
+
+	r := newTestResolver(t)
+	ips := liveResolveNSIPs(t, r, []string{"a.ntpns.org."}, 1)
+
+	for _, ip := range ips {
+		assert.NotNil(t, net.ParseIP(ip), "should be valid IP: %s", ip)
+	}
+}
+
+// TestQueryZone_GivenAddressesFail asks the servers of ntp.org about
+// pool.ntp.org, as the walk to a name under ntp.org does after the org
+// servers' referral. That referral names four nameservers and gives an
+// address for ns1.everett.org alone; here the given address is
+// 192.0.2.1, where nothing answers, so the other three must be looked
+// up and asked.
+func TestQueryZone_GivenAddressesFail(t *testing.T) {
+	t.Parallel()
+
+	r := newTestResolver(t)
+
+	var resp *dns.Msg
+
+	livednstest.Retry(
+		t,
+		"QueryZone(192.0.2.1 and three ntp.org nameservers, pool.ntp.org)",
+		func(ctx context.Context) error {
+			var err error
+
+			resp, err = r.QueryZone(
+				ctx, []string{"192.0.2.1"},
+				[]string{"anyns.pch.net.", "dns1.udel.edu.", "dns2.udel.edu."},
+				"ntp.org.", "pool.ntp.org.", dns.TypeNS, 0,
+			)
+
+			return err
+		},
+	)
+
+	assert.NotEmpty(t, resolver.NSSetFrom(resp, "pool.ntp.org."))
+}
+
+// TestQueryZone_LookupDepth asks the servers of g.ntpns.org, a
+// nameserver of pool.ntp.org, for its address, as looking that address
+// up does when anyns.pch.net, one of the servers of ntpns.org, gives the
+// referral to g.ntpns.org without addresses. Their addresses are looked
+// up (here only a.ntpns.org's), and that needs a bitnames.com
+// nameserver's address, as the org servers delegate ntpns.org without
+// addresses. From depth 1, where looking up g.ntpns.org's address
+// starts, that makes three lookups and the address is found. From one
+// below maxLookupDepth, the bitnames.com lookup would be past the limit,
+// so nothing can be asked, and the error says the limit is why.
+func TestQueryZone_LookupDepth(t *testing.T) {
+	t.Parallel()
+
+	r := newTestResolver(t)
+	withoutAddresses := []string{"a.ntpns.org."}
+
+	var resp *dns.Msg
+
+	livednstest.Retry(
+		t,
+		"QueryZone(a.ntpns.org without its address, g.ntpns.org)",
+		func(ctx context.Context) error {
+			var err error
+
+			resp, err = r.QueryZone(
+				ctx, nil, withoutAddresses, "g.ntpns.org.", "g.ntpns.org.",
+				dns.TypeA, 1,
+			)
+
+			return err
+		},
+	)
+
+	assert.NotEmpty(t, resp.Answer)
+
+	var limitErr error
+
+	// Any other error is live DNS not answering, and is retried.
+	livednstest.Retry(
+		t,
+		"QueryZone(a.ntpns.org without its address, g.ntpns.org, "+
+			"one below the limit)",
+		func(ctx context.Context) error {
+			_, limitErr = r.QueryZone(
+				ctx, nil, withoutAddresses, "g.ntpns.org.", "g.ntpns.org.",
+				dns.TypeA, resolver.MaxLookupDepth-1,
+			)
+			if limitErr == nil ||
+				errors.Is(limitErr, resolver.ErrLookupDepthExceeded) {
+				return nil
+			}
+
+			return limitErr
+		},
+	)
+
+	require.ErrorIs(t, limitErr, resolver.ErrLookupDepthExceeded)
+}
+
 // ----------------------------------------------------------------
 // QueryNameserver tests
 // ----------------------------------------------------------------
@@ -170,7 +279,7 @@ func TestQueryNameserver_BasicA(t *testing.T) {
 
 	r := newTestResolver(t)
 	ns := findOneNSForDomain(t, r, "google.com")
-	resp := liveQueryNameserver(t, r, ns, "www.google.com")
+	resp := liveQueryNameserver(t, r, ns, "www.google.com", "A", "CNAME")
 
 	require.NotNil(t, resp)
 
@@ -184,12 +293,26 @@ func TestQueryNameserver_BasicA(t *testing.T) {
 	)
 }
 
+// TestQueryNameserver_ZoneDelegatedWithoutAddresses asks a.ntpns.org, a
+// nameserver of pool.ntp.org, about pool.ntp.org, as the watcher does.
+// The org servers delegate ntpns.org without the addresses of its
+// nameservers, so finding a.ntpns.org's address needs a lookup inside
+// the one QueryNameserver starts.
+func TestQueryNameserver_ZoneDelegatedWithoutAddresses(t *testing.T) {
+	t.Parallel()
+
+	r := newTestResolver(t)
+	resp := liveQueryNameserver(t, r, "a.ntpns.org.", "pool.ntp.org", "A")
+
+	assert.Equal(t, resolver.StatusOK, resp.Status)
+}
+
 func TestQueryNameserver_AAAA(t *testing.T) {
 	t.Parallel()
 
 	r := newTestResolver(t)
 	ns := findOneNSForDomain(t, r, "cloudflare.com")
-	resp := liveQueryNameserver(t, r, ns, "cloudflare.com")
+	resp := liveQueryNameserver(t, r, ns, "cloudflare.com", "AAAA")
 
 	aaaaRecords := resp.Records["AAAA"]
 	require.NotEmpty(t, aaaaRecords,
@@ -209,7 +332,7 @@ func TestQueryNameserver_MX(t *testing.T) {
 
 	r := newTestResolver(t)
 	ns := findOneNSForDomain(t, r, "google.com")
-	resp := liveQueryNameserver(t, r, ns, "google.com")
+	resp := liveQueryNameserver(t, r, ns, "google.com", "MX")
 
 	mxRecords := resp.Records["MX"]
 	require.NotEmpty(t, mxRecords,
@@ -222,7 +345,7 @@ func TestQueryNameserver_TXT(t *testing.T) {
 
 	r := newTestResolver(t)
 	ns := findOneNSForDomain(t, r, "google.com")
-	resp := liveQueryNameserver(t, r, ns, "google.com")
+	resp := liveQueryNameserver(t, r, ns, "google.com", "TXT")
 
 	txtRecords := resp.Records["TXT"]
 	require.NotEmpty(t, txtRecords,
@@ -244,6 +367,30 @@ func TestQueryNameserver_TXT(t *testing.T) {
 	)
 }
 
+// TestQueryNameserver_TruncatedReplyWhoseTCPRetryFails asks a google.com
+// nameserver about google.com with a resolver whose retries over TCP
+// fail. google.com's TXT records do not fit in a reply over UDP, so TXT
+// is reported as failed, holding none of the records that fit, and
+// logged with the reason, while the nameserver, which answered the other
+// types, is ok.
+func TestQueryNameserver_TruncatedReplyWhoseTCPRetryFails(t *testing.T) {
+	t.Parallel()
+
+	ns := findOneNSForDomain(t, newTestResolver(t), "google.com")
+
+	var logs bytes.Buffer
+
+	r := resolver.NewWithFailingTCP(slog.New(slog.NewTextHandler(&logs, nil)))
+	resp := liveQueryNameserver(t, r, ns, "google.com")
+
+	assert.Equal(t, resolver.StatusOK, resp.Status)
+	assert.Contains(t, resp.FailedTypes, "TXT")
+	assert.NotContains(t, resp.Records, "TXT")
+	assert.Contains(t, logs.String(),
+		"hostname=google.com. nameserver="+ns+" type=TXT error=",
+	)
+}
+
 func TestQueryNameserver_NXDomain(t *testing.T) {
 	t.Parallel()
 
@@ -295,6 +442,187 @@ func TestQueryNameserver_Refused(t *testing.T) {
 	assert.Equal(t, "server returned REFUSED", resp.Error)
 }
 
+// TestQueryNameserverIP_RecursiveResolverRefused asks Quad9, a public
+// recursive resolver, about google.com at both of its addresses. Quad9
+// refuses a query that does not ask for recursion and answers one that
+// does. The resolver never asks for recursion, so it must be reported
+// as refusing, never as answering.
+func TestQueryNameserverIP_RecursiveResolverRefused(t *testing.T) {
+	t.Parallel()
+
+	r := newTestResolver(t)
+
+	for _, ip := range []string{"9.9.9.9", "149.112.112.112"} {
+		var resp *resolver.NameserverResponse
+
+		livednstest.Retry(
+			t,
+			"QueryNameserverIP("+ip+", google.com)",
+			func(ctx context.Context) error {
+				var err error
+
+				resp, err = r.QueryNameserverIP(
+					ctx, ip, ip, "google.com",
+				)
+				if err != nil {
+					return err
+				}
+
+				// A timeout or a network error is no reply at all.
+				if resp.Status == resolver.StatusTimeout ||
+					strings.HasPrefix(resp.Error, "network error") {
+					return fmt.Errorf(
+						"%w: %s: %s",
+						livednstest.ErrNoAnswer, ip, resp.Error,
+					)
+				}
+
+				return nil
+			},
+		)
+
+		assert.Equal(t, resolver.StatusError, resp.Status, ip)
+		assert.Equal(t, "server returned REFUSED", resp.Error, ip)
+	}
+}
+
+// googleNameserverIPv4s returns the IPv4 addresses of google.com's
+// nameservers, the only addresses the resolver asks servers at.
+func googleNameserverIPv4s(t *testing.T, r *resolver.Resolver) []string {
+	t.Helper()
+
+	names := liveFindAuthoritative(t, r, "google.com")
+
+	return liveResolveNSIPs(t, r, names, len(names))
+}
+
+// TestQueryServers_EveryServerRefused asks all of google.com's
+// nameservers about cloudflare.com, a zone they do not serve, which
+// they all refuse. The error says every server refused; it is not
+// ErrIntercepted, which only the root servers refusing shows.
+func TestQueryServers_EveryServerRefused(t *testing.T) {
+	t.Parallel()
+
+	r := newTestResolver(t)
+	servers := googleNameserverIPv4s(t, r)
+
+	var err error
+
+	livednstest.Retry(
+		t,
+		"QueryServers(google.com servers, cloudflare.com)",
+		func(ctx context.Context) error {
+			_, err = r.QueryServers(
+				ctx, servers, "google.com.", "cloudflare.com.",
+				dns.TypeNS,
+			)
+
+			// When not every server refused, one may have given no
+			// reply at all, so the attempt is tried again.
+			if err != nil &&
+				!strings.HasPrefix(err.Error(), "every server of") {
+				return fmt.Errorf(
+					"%w: %w", livednstest.ErrNoAnswer, err,
+				)
+			}
+
+			return nil
+		},
+	)
+
+	require.ErrorIs(t, err, resolver.ErrRefused)
+	require.NotErrorIs(t, err, resolver.ErrIntercepted)
+	require.EqualError(
+		t, err,
+		"every server of google.com. refused a query for "+
+			"cloudflare.com.: dns query refused",
+	)
+}
+
+// TestQueryServers_EveryRootServerRefused passes google.com's
+// nameservers to QueryServers as the servers of the root zone. They
+// refuse a query about cloudflare.com, as root servers would if
+// something on the network answered in their place, so the error is
+// ErrIntercepted.
+func TestQueryServers_EveryRootServerRefused(t *testing.T) {
+	t.Parallel()
+
+	r := newTestResolver(t)
+	servers := googleNameserverIPv4s(t, r)
+
+	var err error
+
+	livednstest.Retry(
+		t,
+		"QueryServers(google.com servers as root servers, cloudflare.com)",
+		func(ctx context.Context) error {
+			_, err = r.QueryServers(
+				ctx, servers, ".", "cloudflare.com.", dns.TypeNS,
+			)
+
+			// When not every server refused, one may have given no
+			// reply at all, so the attempt is tried again. Both errors
+			// for every server refusing say "refused a query for".
+			if err != nil &&
+				!strings.Contains(err.Error(), "refused a query for") {
+				return fmt.Errorf(
+					"%w: %w", livednstest.ErrNoAnswer, err,
+				)
+			}
+
+			return nil
+		},
+	)
+
+	require.ErrorIs(t, err, resolver.ErrIntercepted)
+	require.EqualError(
+		t, err,
+		"every root server refused a query for cloudflare.com.: "+
+			"this network intercepts DNS queries",
+	)
+}
+
+// TestQueryServers_NotEveryRootServerRefused passes google.com's
+// nameservers and 192.0.2.1 to QueryServers as the servers of the root
+// zone. The google.com nameservers refuse a query about cloudflare.com,
+// but nothing answers at 192.0.2.1, a documentation address, so not
+// every server refused, wherever 192.0.2.1 falls in the random order:
+// the error is not ErrIntercepted and does not say every server refused.
+func TestQueryServers_NotEveryRootServerRefused(t *testing.T) {
+	t.Parallel()
+
+	r := newTestResolver(t)
+	servers := googleNameserverIPv4s(t, r)
+	servers = append(servers, "192.0.2.1")
+
+	var err error
+
+	livednstest.Retry(
+		t,
+		"QueryServers(google.com servers and 192.0.2.1, cloudflare.com)",
+		func(ctx context.Context) error {
+			_, err = r.QueryServers(
+				ctx, servers, ".", "cloudflare.com.", dns.TypeNS,
+			)
+
+			// An attempt that ran out of time may not have asked every
+			// server, so it is tried again.
+			if ctx.Err() != nil {
+				return fmt.Errorf(
+					"%w: %w", livednstest.ErrNoAnswer, err,
+				)
+			}
+
+			return nil
+		},
+	)
+
+	require.Error(t, err)
+	require.NotErrorIs(t, err, resolver.ErrIntercepted)
+	// Both errors for every server refusing say "refused a query for".
+	require.NotContains(t, err.Error(), "refused a query for")
+}
+
 func TestQueryNameserver_RecordsSorted(t *testing.T) {
 	t.Parallel()
 
@@ -499,6 +827,21 @@ func TestLookupNS_MatchesFindAuthoritative(t *testing.T) {
 	assert.Equal(t, fromFind, fromLookup)
 }
 
+// TestLookupNS_ParentZoneDelegatedWithoutAddresses looks up the
+// nameservers of g.ntpns.org. The org servers delegate its parent zone,
+// ntpns.org, without the addresses of its nameservers, so the walk has
+// to look them up to ask them. If it did not, the walk for g.ntpns.org
+// would fail and LookupNS would return the nameservers of ntpns.org,
+// which a.ntpns.org is not one of.
+func TestLookupNS_ParentZoneDelegatedWithoutAddresses(t *testing.T) {
+	t.Parallel()
+
+	r := newTestResolver(t)
+	nameservers := liveLookupNS(t, r, "g.ntpns.org")
+
+	assert.Contains(t, nameservers, "a.ntpns.org.")
+}
+
 // ----------------------------------------------------------------
 // ResolveIPAddresses tests
 // ----------------------------------------------------------------
@@ -562,6 +905,34 @@ func TestResolveIPAddresses_CloudflareDomain(t *testing.T) {
 	assert.NotEmpty(t, ips)
 }
 
+// TestResolveIPAddresses_NameserverIPv4AndIPv6 looks up the addresses of
+// one of cloudflare.com's nameservers, as a domain check does for each
+// nameserver. That name has A and AAAA records, so both kinds of address
+// come back.
+func TestResolveIPAddresses_NameserverIPv4AndIPv6(t *testing.T) {
+	t.Parallel()
+
+	r := newTestResolver(t)
+	ns := findOneNSForDomain(t, r, "cloudflare.com")
+	ips := liveResolveIPs(t, r, ns)
+
+	var ipv4, ipv6 int
+
+	for _, ip := range ips {
+		parsed := net.ParseIP(ip)
+		require.NotNil(t, parsed, "should be valid IP: %s", ip)
+
+		if parsed.To4() != nil {
+			ipv4++
+		} else {
+			ipv6++
+		}
+	}
+
+	assert.Positive(t, ipv4, "no IPv4 address for %s: %v", ns, ips)
+	assert.Positive(t, ipv6, "no IPv6 address for %s: %v", ns, ips)
+}
+
 // ----------------------------------------------------------------
 // Context cancellation tests
 // ----------------------------------------------------------------
@@ -655,6 +1026,29 @@ func TestQueryNameserverIP_Timeout(t *testing.T) {
 	assert.NotEmpty(t, resp.Error)
 }
 
+// TestQueryNameserverIP_CancelledLogsNothing cancels the context while
+// a query to 192.0.2.1, where nothing answers, is waiting for a reply,
+// as shutdown does. The query was cut short, not failed, so nothing is
+// logged.
+func TestQueryNameserverIP_CancelledLogsNothing(t *testing.T) {
+	t.Parallel()
+
+	var logs bytes.Buffer
+
+	r := resolver.NewFromLogger(slog.New(slog.NewTextHandler(&logs, nil)))
+
+	ctx, cancel := context.WithCancel(context.Background())
+	t.Cleanup(cancel)
+	time.AfterFunc(100*time.Millisecond, cancel)
+
+	_, err := r.QueryNameserverIP(
+		ctx, "unreachable.test.", "192.0.2.1", "example.com",
+	)
+	require.NoError(t, err)
+
+	assert.Empty(t, logs.String())
+}
+
 // TestCollectIPs_NoNameserverAnswered takes the response of a
 // nameserver at 192.0.2.1, where nothing answers, as
 // TestQueryNameserverIP_Timeout does. Addresses collected from

internal/state/state.go

@@ -8,6 +8,7 @@ import (
 	"log/slog"
 	"os"
 	"path/filepath"
+	"slices"
 	"sync"
 	"time"
 
@@ -45,16 +46,28 @@ type DomainState struct {
 }
 
 // NameserverRecordState holds one NS's response for a hostname.
+// FailedTypes lists the record types whose query to the nameserver
+// failed on this check: Records holds for them the records saved by the
+// previous check, which are kept. UnknownTypes lists those of them whose
+// records the previous check did not know either, as when the
+// nameserver was new or failing then: Records holds nothing for them.
 type NameserverRecordState struct {
-	Records     map[string][]string `json:"records"`
-	Status      string              `json:"status"`
-	Error       string              `json:"error,omitempty"`
-	LastChecked time.Time           `json:"lastChecked"`
+	Records      map[string][]string `json:"records"`
+	FailedTypes  []string            `json:"failedTypes,omitempty"`
+	UnknownTypes []string            `json:"unknownTypes,omitempty"`
+	Status       string              `json:"status"`
+	Error        string              `json:"error,omitempty"`
+	LastChecked  time.Time           `json:"lastChecked"`
 }
 
 // HostnameState holds per-nameserver monitoring state for a hostname.
+// CNAMEAddresses holds the sorted addresses at the end of the name's
+// CNAME chain, found when its nameservers answered with a CNAME and no
+// address; it is empty otherwise. It is nil when they are not known: a
+// state file written before it existed loads with it nil.
 type HostnameState struct {
 	RecordsByNameserver map[string]*NameserverRecordState `json:"recordsByNameserver"`
+	CNAMEAddresses      []string                          `json:"cnameAddresses"`
 	LastChecked         time.Time                         `json:"lastChecked"`
 }
 
@@ -116,6 +129,8 @@ type CertificateState struct {
 }
 
 // Snapshot is the complete monitoring state persisted to disk.
+// Hostnames also holds each apex domain's own records, under the
+// domain's name, which has an entry in Domains too.
 type Snapshot struct {
 	Version      int                          `json:"version"`
 	LastUpdated  time.Time                    `json:"lastUpdated"`
@@ -196,6 +211,19 @@ func (s *State) Load() error {
 		return fmt.Errorf("parsing state file: %w", err)
 	}
 
+	// A state file saved before each record value was stored once can
+	// hold a hostname's CNAME once for every record type asked for.
+	// Each value is kept once, so the first check does not see a
+	// record change.
+	for _, hs := range snapshot.Hostnames {
+		for _, ns := range hs.RecordsByNameserver {
+			for recordType, values := range ns.Records {
+				slices.Sort(values)
+				ns.Records[recordType] = slices.Compact(values)
+			}
+		}
+	}
+
 	s.snapshot = &snapshot
 	s.log.Info("loaded state from disk", "path", path)
 
@@ -270,6 +298,27 @@ func (s *State) GetDomainState(
 	return ds, ok
 }
 
+// DeleteDomainState removes a domain state entry.
+func (s *State) DeleteDomainState(domain string) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	delete(s.snapshot.Domains, domain)
+}
+
+// GetAllDomainNames returns the names of all domain state entries.
+func (s *State) GetAllDomainNames() []string {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	names := make([]string, 0, len(s.snapshot.Domains))
+	for name := range s.snapshot.Domains {
+		names = append(names, name)
+	}
+
+	return names
+}
+
 // SetHostnameState updates the state for a hostname.
 func (s *State) SetHostnameState(
 	hostname string,
@@ -293,6 +342,28 @@ func (s *State) GetHostnameState(
 	return hs, ok
 }
 
+// DeleteHostnameState removes a hostname state entry.
+func (s *State) DeleteHostnameState(hostname string) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	delete(s.snapshot.Hostnames, hostname)
+}
+
+// GetAllHostnames returns the names of all hostname state entries,
+// which include each apex domain's own records.
+func (s *State) GetAllHostnames() []string {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	names := make([]string, 0, len(s.snapshot.Hostnames))
+	for name := range s.snapshot.Hostnames {
+		names = append(names, name)
+	}
+
+	return names
+}
+
 // SetPortState updates the state for a port.
 func (s *State) SetPortState(key string, ps *PortState) {
 	s.mu.Lock()
@@ -355,6 +426,27 @@ func (s *State) GetCertificateState(
 	return cs, ok
 }
 
+// DeleteCertificateState removes a certificate state entry.
+func (s *State) DeleteCertificateState(key string) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	delete(s.snapshot.Certificates, key)
+}
+
+// GetAllCertificateKeys returns all certificate state keys.
+func (s *State) GetAllCertificateKeys() []string {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	keys := make([]string, 0, len(s.snapshot.Certificates))
+	for k := range s.snapshot.Certificates {
+		keys = append(keys, k)
+	}
+
+	return keys
+}
+
 // checkDataDirWritable creates the data directory if needed, then writes
 // and removes the temp file that Save uses. It runs at startup so that an
 // unwritable directory stops the process, instead of the process running

internal/state/state_test.go

@@ -188,6 +188,203 @@ func TestLoadStateFromBeforeNameserverAddresses(t *testing.T) {
 	}
 }
 
+// TestSaveLoadRoundTrip_CNAMEAddresses checks that no addresses at the
+// end of a hostname's CNAME chain load as an empty list, and addresses
+// that are not known load as nil: the watcher tells the two apart.
+func TestSaveLoadRoundTrip_CNAMEAddresses(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	s := state.NewForTestWithDataDir(dir)
+
+	want := map[string][]string{
+		"cname.example.com":     {testIP},
+		"none.example.com":      {},
+		"not-known.example.com": nil,
+	}
+
+	for name, addresses := range want {
+		s.SetHostnameState(name, &state.HostnameState{
+			CNAMEAddresses: addresses,
+		})
+	}
+
+	err := s.Save()
+	if err != nil {
+		t.Fatalf("Save() error: %v", err)
+	}
+
+	loaded := state.NewForTestWithDataDir(dir)
+
+	err = loaded.Load()
+	if err != nil {
+		t.Fatalf("Load() error: %v", err)
+	}
+
+	for name, addresses := range want {
+		hs, ok := loaded.GetHostnameState(name)
+		if !ok {
+			t.Fatalf("missing hostname %s", name)
+		}
+
+		if !reflect.DeepEqual(hs.CNAMEAddresses, addresses) {
+			t.Errorf(
+				"%s: loaded %#v, want %#v",
+				name, hs.CNAMEAddresses, addresses,
+			)
+		}
+	}
+}
+
+// TestSaveLoadRoundTrip_FailedTypes checks that a nameserver's
+// failedTypes and unknownTypes survive a save and load. Without
+// unknownTypes, a type whose records were not known would load as one
+// with no records.
+func TestSaveLoadRoundTrip_FailedTypes(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	s := state.NewForTestWithDataDir(dir)
+
+	failed := []string{"TXT", "CAA"}
+	unknown := []string{"CAA"}
+
+	s.SetHostnameState(testHostname, &state.HostnameState{
+		RecordsByNameserver: map[string]*state.NameserverRecordState{
+			testNS1: {
+				Records:      map[string][]string{"TXT": {"v=spf1 -all"}},
+				FailedTypes:  failed,
+				UnknownTypes: unknown,
+				Status:       "ok",
+			},
+		},
+	})
+
+	err := s.Save()
+	if err != nil {
+		t.Fatalf("Save() error: %v", err)
+	}
+
+	loaded := state.NewForTestWithDataDir(dir)
+
+	err = loaded.Load()
+	if err != nil {
+		t.Fatalf("Load() error: %v", err)
+	}
+
+	hs, ok := loaded.GetHostnameState(testHostname)
+	if !ok {
+		t.Fatal("missing hostname " + testHostname)
+	}
+
+	ns1 := hs.RecordsByNameserver[testNS1]
+	if ns1 == nil {
+		t.Fatal("missing nameserver " + testNS1)
+	}
+
+	if !reflect.DeepEqual(ns1.FailedTypes, failed) {
+		t.Errorf("failedTypes: got %#v", ns1.FailedTypes)
+	}
+
+	if !reflect.DeepEqual(ns1.UnknownTypes, unknown) {
+		t.Errorf("unknownTypes: got %#v", ns1.UnknownTypes)
+	}
+}
+
+// TestLoadStateFromBeforeCNAMEAddresses loads a state file written
+// before the addresses at the end of a hostname's CNAME chain were
+// saved. They load as not known (nil), not as none.
+func TestLoadStateFromBeforeCNAMEAddresses(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+
+	data := []byte(`{
+		"version": 1,
+		"lastUpdated": "2026-02-19T12:00:00Z",
+		"hostnames": {
+			"www.example.com": {
+				"recordsByNameserver": {},
+				"lastChecked": "2026-02-19T12:00:00Z"
+			}
+		}
+	}`)
+
+	err := os.WriteFile(filepath.Join(dir, "state.json"), data, 0o600)
+	if err != nil {
+		t.Fatalf("writing state file: %v", err)
+	}
+
+	s := state.NewForTestWithDataDir(dir)
+
+	err = s.Load()
+	if err != nil {
+		t.Fatalf("Load() error: %v", err)
+	}
+
+	hs, ok := s.GetHostnameState(testHostname)
+	if !ok {
+		t.Fatal("missing hostname " + testHostname)
+	}
+
+	if hs.CNAMEAddresses != nil {
+		t.Errorf("CNAME addresses: got %#v, want nil", hs.CNAMEAddresses)
+	}
+}
+
+// TestLoadStateWithRepeatedValues loads a state file saved when a
+// hostname's CNAME was stored once for every record type asked for.
+// Each value must load once, and every different value must load.
+func TestLoadStateWithRepeatedValues(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+
+	data := []byte(`{
+		"version": 1,
+		"hostnames": {
+			"www.example.com": {
+				"recordsByNameserver": {
+					"ns1.example.com.": {
+						"records": {
+							"A": ["192.0.2.2", "192.0.2.1", "192.0.2.2", "192.0.2.1"],
+							"CNAME": ["a.example.net.", "a.example.net.", "a.example.net."]
+						},
+						"status": "ok"
+					}
+				}
+			}
+		}
+	}`)
+
+	err := os.WriteFile(filepath.Join(dir, "state.json"), data, 0o600)
+	if err != nil {
+		t.Fatalf("writing state file: %v", err)
+	}
+
+	s := state.NewForTestWithDataDir(dir)
+
+	err = s.Load()
+	if err != nil {
+		t.Fatalf("Load() error: %v", err)
+	}
+
+	hs, ok := s.GetHostnameState(testHostname)
+	if !ok {
+		t.Fatal("missing hostname " + testHostname)
+	}
+
+	want := map[string][]string{
+		"A":     {"192.0.2.1", "192.0.2.2"},
+		"CNAME": {"a.example.net."},
+	}
+
+	got := hs.RecordsByNameserver[testNS1].Records
+	if !reflect.DeepEqual(got, want) {
+		t.Errorf("records: got %v, want %v", got, want)
+	}
+}
+
 // TestSaveLoadRoundTrip_Hostnames verifies hostname data survives a save/load cycle.
 func TestSaveLoadRoundTrip_Hostnames(t *testing.T) {
 	t.Parallel()

internal/watcher/cancelled_test.go

@@ -1,10 +1,13 @@
 package watcher_test
 
 import (
+	"bytes"
 	"context"
 	"log/slog"
 	"reflect"
+	"strings"
 	"testing"
+	"time"
 
 	"sneak.berlin/go/dnswatcher/internal/portcheck"
 	"sneak.berlin/go/dnswatcher/internal/resolver"
@@ -79,3 +82,72 @@ func TestCancelledCheckSavesNothing(t *testing.T) {
 		t.Errorf("sent %v, want no notifications", notifications)
 	}
 }
+
+// newLoggingWatcher returns a watcher for a domain and a hostname, with
+// the real resolver, that writes what it logs at warning level or above
+// into the returned buffer.
+func newLoggingWatcher(t *testing.T) (*watcher.Watcher, *bytes.Buffer) {
+	t.Helper()
+
+	cfg := defaultTestConfig(t)
+	cfg.Domains = []string{testSmallDomain}
+	cfg.Hostnames = []string{host}
+
+	w, _ := newTestWatcher(t, cfg)
+
+	logs := &bytes.Buffer{}
+	w.SetLogger(slog.New(slog.NewJSONHandler(
+		logs, &slog.HandlerOptions{Level: slog.LevelWarn},
+	)))
+
+	return w, logs
+}
+
+// TestLookupCutShortIsNotLogged checks a domain and a hostname, looks
+// up a nameserver's addresses and follows a CNAME, with the context
+// cancelled, as shutdown leaves it. The real resolver fails each lookup
+// without sending a query. Shutdown cutting a lookup short is not a
+// failure, so nothing may be logged at warning level or above.
+func TestLookupCutShortIsNotLogged(t *testing.T) {
+	t.Parallel()
+
+	w, logs := newLoggingWatcher(t)
+
+	ctx, cancel := context.WithCancel(t.Context())
+	cancel()
+
+	w.RunOnce(ctx)
+	w.ResolveNameserverAddresses(ctx, []string{nsA}, nil)
+	w.ResolveCNAMEAddresses(ctx, host, cnameState(), nil)
+
+	if logs.Len() > 0 {
+		t.Errorf("logged at warning level or above:\n%s", logs)
+	}
+}
+
+// TestLookupOutOfTimeIsLoggedAsError does what
+// TestLookupCutShortIsNotLogged does, with the context's deadline passed
+// instead. A lookup that ran out of time did fail, so the domain's NS
+// lookup, the hostname's lookup, the nameserver's address lookup and the
+// CNAME's are each logged as an error.
+func TestLookupOutOfTimeIsLoggedAsError(t *testing.T) {
+	t.Parallel()
+
+	w, logs := newLoggingWatcher(t)
+
+	ctx, cancel := context.WithDeadline(t.Context(), time.Now())
+	t.Cleanup(cancel)
+
+	w.RunOnce(ctx)
+	w.ResolveNameserverAddresses(ctx, []string{nsA}, nil)
+	w.ResolveCNAMEAddresses(ctx, host, cnameState(), nil)
+
+	const want = 4
+
+	lines := strings.Count(logs.String(), "\n")
+	errorLines := strings.Count(logs.String(), `"level":"ERROR"`)
+
+	if lines != want || errorLines != want {
+		t.Errorf("logged:\n%s\nwant %d lines, each at error level", logs, want)
+	}
+}

internal/watcher/cname_test.go

@@ -0,0 +1,372 @@
+package watcher_test
+
+import (
+	"context"
+	"log/slog"
+	"slices"
+	"testing"
+
+	"sneak.berlin/go/dnswatcher/internal/livednstest"
+	"sneak.berlin/go/dnswatcher/internal/resolver"
+	"sneak.berlin/go/dnswatcher/internal/state"
+	"sneak.berlin/go/dnswatcher/internal/watcher"
+)
+
+// TestCNAMEIntoAnotherZonePortAndTLSChecks runs the port and TLS
+// checks on hostname state built here: the name's nameserver answered
+// with a CNAME into another zone, and following it found ip1. Both
+// checks must use ip1. They look nothing up, so the watcher has no
+// resolver.
+func TestCNAMEIntoAnotherZonePortAndTLSChecks(t *testing.T) {
+	t.Parallel()
+
+	cfg := defaultTestConfig(t)
+	cfg.Hostnames = []string{host}
+
+	deps := newTestDeps(t, cfg)
+	w := watcher.NewForTest(
+		cfg, deps.state, nil,
+		deps.portChecker, deps.tlsChecker, deps.notifier,
+	)
+
+	deps.state.SetHostnameState(host, cnameState(ip1))
+
+	w.CheckAllPorts(t.Context())
+	w.RunTLSChecks(t.Context())
+
+	snap := deps.state.GetSnapshot()
+
+	ps, ok := snap.Ports[ip1+":443"]
+	if !ok || !slices.Contains(ps.Hostnames, host) {
+		t.Errorf("no port state for %s at %s:443", host, ip1)
+	}
+
+	certKey := ip1 + ":443:" + host
+	if _, ok := snap.Certificates[certKey]; !ok {
+		t.Errorf("no certificate state %s", certKey)
+	}
+}
+
+// TestCNAMEThatCannotBeFollowedKeepsPrevious runs a check of a name, not
+// the watcher's first, from the point where its records have been looked
+// up: they hold a CNAME to a target under .invalid, whose lookup fails.
+// The previous check found the same records, and oldIP at the end of the
+// CNAME. The check must keep oldIP and send nothing.
+func TestCNAMEThatCannotBeFollowedKeepsPrevious(t *testing.T) {
+	t.Parallel()
+
+	w, deps := newTestWatcher(t, defaultTestConfig(t))
+	w.SetFirstRun(false)
+
+	records := map[string]map[string][]string{
+		nsA: cnameTo("target.example.invalid."),
+	}
+
+	prev := hostnameState(records)
+	prev.CNAMEAddresses = []string{oldIP}
+	deps.state.SetHostnameState(host, prev)
+
+	// The result is the same whether or not live DNS answers, so the
+	// lookup is not retried.
+	_ = livednstest.Run(func(ctx context.Context) error {
+		w.UpdateHostnameState(ctx, host, hostnameState(records))
+
+		return nil
+	})
+
+	hs, _ := deps.state.GetHostnameState(host)
+	if !slices.Equal(hs.CNAMEAddresses, prev.CNAMEAddresses) {
+		t.Errorf(
+			"saved %v, want %v",
+			hs.CNAMEAddresses, prev.CNAMEAddresses,
+		)
+	}
+
+	notifications := deps.notifier.getNotifications()
+	if len(notifications) != 0 {
+		t.Errorf("sent %v, want no notifications", notifications)
+	}
+}
+
+// followLive follows in live DNS the CNAMEs in a name's records, built
+// from records, and returns the addresses saved for the name. The
+// previous check saved oldIP, which is kept when a target cannot be
+// followed; that is retried. The tests point CNAMEs only at names in
+// zones with two nameservers, to keep queries few (see the top of
+// watcher_test.go).
+func followLive(
+	t *testing.T,
+	records map[string]map[string][]string,
+) []string {
+	t.Helper()
+
+	w := watcher.NewForTest(
+		nil, nil, resolver.NewFromLogger(slog.Default()), nil, nil, nil,
+	)
+	prev := cnameState(oldIP)
+
+	var current *state.HostnameState
+
+	livednstest.Retry(t, "following CNAMEs", func(ctx context.Context) error {
+		current = hostnameState(records)
+
+		w.ResolveCNAMEAddresses(ctx, host, current, prev)
+
+		if slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
+			return livednstest.ErrNoAnswer
+		}
+
+		return nil
+	})
+
+	return current.CNAMEAddresses
+}
+
+// TestCNAMEAddressesOfEveryTarget gives a name's two nameservers
+// different CNAME targets, as when a secondary still serves an old one.
+// The addresses at the end of both are saved, whichever answer is read
+// first: one.one.one.one has 1.1.1.1, and dns.adguard-dns.com has
+// 94.140.14.14.
+func TestCNAMEAddressesOfEveryTarget(t *testing.T) {
+	t.Parallel()
+
+	found := followLive(t, map[string]map[string][]string{
+		nsA: cnameTo("one.one.one.one."),
+		nsB: cnameTo("dns.adguard-dns.com."),
+	})
+
+	for _, ip := range []string{"1.1.1.1", "94.140.14.14"} {
+		if !slices.Contains(found, ip) {
+			t.Errorf("saved %v, want %s among them", found, ip)
+		}
+	}
+}
+
+// TestCNAMEChainEndingInNoAddressSavesEmptyList follows a CNAME to a
+// name live DNS answers with NXDOMAIN. An empty list is saved, not nil,
+// which would mean the addresses are not known.
+func TestCNAMEChainEndingInNoAddressSavesEmptyList(t *testing.T) {
+	t.Parallel()
+
+	found := followLive(t, map[string]map[string][]string{
+		nsA: cnameTo("this-surely-does-not-exist-xyz.example.org."),
+	})
+
+	if found == nil || len(found) != 0 {
+		t.Errorf("saved %#v, want an empty list", found)
+	}
+}
+
+// TestCNAMEBesideAnAddressNotFollowed gives one nameserver of a name an
+// address and another a CNAME. The CNAME is not followed: an empty list
+// is saved, not nil, and nothing is looked up, the watcher having no
+// resolver.
+func TestCNAMEBesideAnAddressNotFollowed(t *testing.T) {
+	t.Parallel()
+
+	w := watcher.NewForTest(nil, nil, nil, nil, nil, nil)
+
+	current := hostnameState(map[string]map[string][]string{
+		nsA: {"A": {ip1}},
+		nsB: cnameTo("target.example.org."),
+	})
+
+	w.ResolveCNAMEAddresses(t.Context(), host, current, nil)
+
+	if current.CNAMEAddresses == nil || len(current.CNAMEAddresses) != 0 {
+		t.Errorf("saved %#v, want an empty list", current.CNAMEAddresses)
+	}
+}
+
+// TestCNAMEWhoseNameserversAllFailedKeepsPrevious checks a name none of
+// whose nameservers answered. The addresses the previous check saved
+// from following its CNAME are kept, and nothing is looked up: the
+// watcher has no resolver.
+func TestCNAMEWhoseNameserversAllFailedKeepsPrevious(t *testing.T) {
+	t.Parallel()
+
+	w := watcher.NewForTest(nil, nil, nil, nil, nil, nil)
+
+	current := saved(map[string]*state.NameserverRecordState{
+		nsA: failed(), nsB: failed(),
+	})
+	prev := cnameState(oldIP)
+
+	w.ResolveCNAMEAddresses(t.Context(), host, current, prev)
+
+	if !slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
+		t.Errorf(
+			"saved %v, want %v",
+			current.CNAMEAddresses, prev.CNAMEAddresses,
+		)
+	}
+}
+
+// TestCNAMEWhoseAddressQueryFailedKeepsPrevious checks a name whose
+// nameserver answered, but whose query for A, AAAA or CNAME failed with
+// nothing kept for it. That is not an answer with no address: the
+// addresses the previous check saved from following its CNAME are kept,
+// and nothing is looked up, the watcher having no resolver.
+func TestCNAMEWhoseAddressQueryFailedKeepsPrevious(t *testing.T) {
+	t.Parallel()
+
+	for _, rtype := range []string{"A", "AAAA", "CNAME"} {
+		t.Run(rtype, func(t *testing.T) {
+			t.Parallel()
+
+			w := watcher.NewForTest(nil, nil, nil, nil, nil, nil)
+
+			current := saved(map[string]*state.NameserverRecordState{
+				nsA: {
+					Records:      map[string][]string{},
+					FailedTypes:  []string{rtype},
+					UnknownTypes: []string{rtype},
+					Status:       "ok",
+				},
+			})
+			prev := cnameState(oldIP)
+
+			w.ResolveCNAMEAddresses(t.Context(), host, current, prev)
+
+			if !slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
+				t.Errorf(
+					"saved %v, want %v",
+					current.CNAMEAddresses, prev.CNAMEAddresses,
+				)
+			}
+		})
+	}
+}
+
+// cnameTo builds the records of a nameserver that answered with a CNAME
+// to target and no address.
+func cnameTo(target string) map[string][]string {
+	return map[string][]string{"CNAME": {target}}
+}
+
+// cnameState builds the state a check leaves behind for a name whose
+// nameserver answered with a CNAME and no address, when following the
+// CNAME found these addresses, which may be none.
+func cnameState(addresses ...string) *state.HostnameState {
+	hs := hostnameState(map[string]map[string][]string{
+		nsA: cnameTo("target.example.org."),
+	})
+
+	hs.CNAMEAddresses = append([]string{}, addresses...)
+
+	return hs
+}
+
+func TestCNAMEAddressChangeAlerts(t *testing.T) {
+	t.Parallel()
+
+	// A state file written before the addresses were saved loads with
+	// them nil.
+	olderStateFile := cnameState()
+	olderStateFile.CNAMEAddresses = nil
+
+	// Each case is the state saved by the previous check and by the
+	// current one. The name's records are the same in both.
+	tests := []struct {
+		name          string
+		prev, current *state.HostnameState
+		want          int
+	}{
+		{
+			"same addresses",
+			cnameState(ip1, ip2), cnameState(ip1, ip2), 0,
+		},
+		{
+			"same addresses in another order",
+			cnameState(ip2, ip1), cnameState(ip1, ip2), 0,
+		},
+		{
+			"address replaced",
+			cnameState(ip1), cnameState(ip2), 1,
+		},
+		{
+			"address added",
+			cnameState(ip1), cnameState(ip1, ip2), 1,
+		},
+		{
+			"no address at the end of the chain now",
+			cnameState(ip1), cnameState(), 1,
+		},
+		{
+			"addresses at the end of the chain again",
+			cnameState(), cnameState(ip1), 1,
+		},
+		{
+			"state file from before addresses were saved",
+			olderStateFile, cnameState(ip1), 0,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			notifier := &mockNotifier{}
+			w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
+
+			w.DetectHostnameChanges(t.Context(), host, tt.prev, tt.current)
+
+			got := len(notifier.getNotifications())
+			if got != tt.want {
+				t.Errorf("sent %d notifications, want %d", got, tt.want)
+			}
+		})
+	}
+}
+
+func TestCNAMEAddressChangeAlertNamesHostnameAndAddresses(t *testing.T) {
+	t.Parallel()
+
+	notifier := &mockNotifier{}
+	w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
+
+	w.DetectHostnameChanges(
+		t.Context(), host, cnameState(ip1), cnameState(ip2, ip3),
+	)
+
+	want := notification{
+		Title: "CNAME Address Change: " + host,
+		Message: "Hostname: " + host +
+			"\nOld: " + ip1 + "\nNew: " + ip2 + ", " + ip3,
+		Priority: "warning",
+	}
+
+	got := notifier.getNotifications()
+	if len(got) != 1 || got[0] != want {
+		t.Errorf("sent %v, want %v", got, want)
+	}
+}
+
+// TestNameMovedFromARecordsToCNAMEAlerts checks a name that answers
+// with an A record and then with a CNAME whose chain ends in ip2. The
+// second check is notified as a CNAME address change from no addresses,
+// beside the record change. Nothing is looked up: the watcher has no
+// resolver.
+func TestNameMovedFromARecordsToCNAMEAlerts(t *testing.T) {
+	t.Parallel()
+
+	notifier := &mockNotifier{}
+	w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
+
+	prev := hostnameState(map[string]map[string][]string{
+		nsA: {"A": {ip1}},
+	})
+	w.ResolveCNAMEAddresses(t.Context(), host, prev, nil)
+
+	w.DetectHostnameChanges(t.Context(), host, prev, cnameState(ip2))
+
+	title := "CNAME Address Change: " + host
+	message := "Hostname: " + host + "\nOld: \nNew: " + ip2
+
+	got := notifier.getNotifications()
+	if !slices.ContainsFunc(got, func(n notification) bool {
+		return n.Title == title && n.Message == message
+	}) {
+		t.Errorf("sent %v, want %q with %q among them", got, title, message)
+	}
+}

internal/watcher/export_test.go

@@ -10,7 +10,8 @@ import (
 	"sneak.berlin/go/dnswatcher/internal/state"
 )
 
-// NewForTest creates a Watcher without fx for unit testing.
+// NewForTest creates a Watcher without fx for unit testing. A nil cfg
+// is an empty configuration.
 func NewForTest(
 	cfg *config.Config,
 	st *state.State,
@@ -19,6 +20,10 @@ func NewForTest(
 	tc TLSChecker,
 	n Notifier,
 ) *Watcher {
+	if cfg == nil {
+		cfg = &config.Config{}
+	}
+
 	return &Watcher{
 		log:       slog.Default(),
 		config:    cfg,
@@ -31,6 +36,12 @@ func NewForTest(
 	}
 }
 
+// SetLogger replaces the watcher's logger, so a test can read what it
+// logs.
+func (w *Watcher) SetLogger(log *slog.Logger) {
+	w.log = log
+}
+
 // NewlyDisagreeingPairs exports newlyDisagreeingPairs for testing.
 func NewlyDisagreeingPairs(
 	prev, current *state.HostnameState,
@@ -38,6 +49,21 @@ func NewlyDisagreeingPairs(
 	return newlyDisagreeingPairs(prev, current)
 }
 
+// SetFirstRun sets whether the watcher is on its first check, in which
+// nothing is compared with the previous check. NewForTest's watcher is.
+func (w *Watcher) SetFirstRun(firstRun bool) {
+	w.firstRun = firstRun
+}
+
+// UpdateHostnameState exports updateHostnameState for testing.
+func (w *Watcher) UpdateHostnameState(
+	ctx context.Context,
+	hostname string,
+	newState *state.HostnameState,
+) {
+	w.updateHostnameState(ctx, hostname, newState)
+}
+
 // DetectHostnameChanges exports detectHostnameChanges for testing.
 func (w *Watcher) DetectHostnameChanges(
 	ctx context.Context,
@@ -57,6 +83,15 @@ func (w *Watcher) ResolveNameserverAddresses(
 	return w.resolveNameserverAddresses(ctx, nameservers, prev)
 }
 
+// ResolveCNAMEAddresses exports resolveCNAMEAddresses for testing.
+func (w *Watcher) ResolveCNAMEAddresses(
+	ctx context.Context,
+	hostname string,
+	current, prev *state.HostnameState,
+) {
+	w.resolveCNAMEAddresses(ctx, hostname, current, prev)
+}
+
 // DetectNSAddressChanges exports detectNSAddressChanges for testing.
 func (w *Watcher) DetectNSAddressChanges(
 	ctx context.Context,
@@ -66,6 +101,17 @@ func (w *Watcher) DetectNSAddressChanges(
 	w.detectNSAddressChanges(ctx, domain, prev, current)
 }
 
+// MaybeSendTestNotification exports maybeSendTestNotification for
+// testing.
+func (w *Watcher) MaybeSendTestNotification(ctx context.Context) {
+	w.maybeSendTestNotification(ctx)
+}
+
+// CleanupRemovedTargets exports cleanupRemovedTargets for testing.
+func (w *Watcher) CleanupRemovedTargets() {
+	w.cleanupRemovedTargets()
+}
+
 // CheckAllPorts exports checkAllPorts for testing.
 func (w *Watcher) CheckAllPorts(ctx context.Context) {
 	w.checkAllPorts(ctx)
@@ -79,7 +125,8 @@ func (w *Watcher) RunTLSChecks(ctx context.Context) {
 // BuildHostnameState exports buildHostnameState for testing.
 func BuildHostnameState(
 	results map[string]*resolver.NameserverResponse,
+	prev *state.HostnameState,
 	now time.Time,
 ) *state.HostnameState {
-	return buildHostnameState(results, now)
+	return buildHostnameState(results, prev, now)
 }

internal/watcher/failedtype_test.go

@@ -0,0 +1,364 @@
+package watcher_test
+
+import (
+	"maps"
+	"slices"
+	"testing"
+	"time"
+
+	"sneak.berlin/go/dnswatcher/internal/resolver"
+	"sneak.berlin/go/dnswatcher/internal/state"
+	"sneak.berlin/go/dnswatcher/internal/watcher"
+)
+
+const (
+	// txt is the record type whose query fails in these tests.
+	txt  = "TXT"
+	spf1 = "v=spf1 -all"
+	spf2 = "v=spf1 include:example.net -all"
+)
+
+// response is a nameserver's response with these records, whose queries
+// for failedTypes failed.
+func response(
+	records map[string][]string,
+	failedTypes ...string,
+) *resolver.NameserverResponse {
+	return &resolver.NameserverResponse{
+		Records:     records,
+		FailedTypes: failedTypes,
+		Status:      resolver.StatusOK,
+	}
+}
+
+// savedChecks saves the state of each check in turn from the
+// nameservers' responses, each from the state the check before saved.
+func savedChecks(
+	checks ...map[string]*resolver.NameserverResponse,
+) []*state.HostnameState {
+	states := make([]*state.HostnameState, 0, len(checks))
+
+	var prev *state.HostnameState
+
+	for _, results := range checks {
+		prev = watcher.BuildHostnameState(results, prev, time.Now())
+		states = append(states, prev)
+	}
+
+	return states
+}
+
+// TestFailedTypeKeepsPreviousRecords saves a check in which nsA's query
+// for TXT failed, after previous checks of several kinds. TXT is always
+// saved in FailedTypes, and in UnknownTypes when there was nothing to
+// keep.
+func TestFailedTypeKeepsPreviousRecords(t *testing.T) {
+	t.Parallel()
+
+	aOnly := map[string][]string{"A": {ip1}}
+	withTXT := map[string][]string{"A": {ip1}, txt: {spf1}}
+	txtKept := &state.NameserverRecordState{
+		Records: withTXT, FailedTypes: []string{txt}, Status: "ok",
+	}
+	txtNotKnown := &state.NameserverRecordState{
+		Records:      aOnly,
+		FailedTypes:  []string{txt},
+		UnknownTypes: []string{txt},
+		Status:       "ok",
+	}
+
+	tests := []struct {
+		name        string
+		prev        *state.HostnameState
+		wantRecords map[string][]string
+		wantUnknown []string
+	}{
+		{
+			"previous TXT records are kept",
+			saved(map[string]*state.NameserverRecordState{nsA: answered(withTXT)}),
+			withTXT, nil,
+		},
+		{
+			"previous check had no TXT records",
+			saved(map[string]*state.NameserverRecordState{nsA: answered(aOnly)}),
+			aOnly, nil,
+		},
+		{
+			"TXT failed on the previous check, which kept its records",
+			saved(map[string]*state.NameserverRecordState{nsA: txtKept}),
+			withTXT, nil,
+		},
+		{"first check", nil, aOnly, []string{txt}},
+		{
+			"nameserver new on this check",
+			saved(map[string]*state.NameserverRecordState{nsB: answered(withTXT)}),
+			aOnly, []string{txt},
+		},
+		{
+			"nameserver failed on the previous check",
+			saved(map[string]*state.NameserverRecordState{nsA: failed()}),
+			aOnly, []string{txt},
+		},
+		{
+			"TXT failed on the previous check with nothing to keep",
+			saved(map[string]*state.NameserverRecordState{nsA: txtNotKnown}),
+			aOnly, []string{txt},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			hs := watcher.BuildHostnameState(
+				map[string]*resolver.NameserverResponse{
+					nsA: response(map[string][]string{"A": {ip1}}, txt),
+				},
+				tt.prev, time.Now(),
+			)
+
+			got := hs.RecordsByNameserver[nsA]
+			if got.Status != "ok" ||
+				!maps.EqualFunc(got.Records, tt.wantRecords, slices.Equal) ||
+				!slices.Equal(got.FailedTypes, []string{txt}) ||
+				!slices.Equal(got.UnknownTypes, tt.wantUnknown) {
+				t.Errorf(
+					"saved status %q, records %v, failed types %v, "+
+						"unknown types %v; want ok, %v, [%s], %v",
+					got.Status, got.Records, got.FailedTypes,
+					got.UnknownTypes, tt.wantRecords, txt, tt.wantUnknown,
+				)
+			}
+		})
+	}
+}
+
+// TestFailedTypeAlerts saves the checks of each case in turn from the
+// nameservers' responses, the first being the state loaded at startup,
+// and counts the alerts sent. nsB's TXT query fails on one check, and
+// nothing changes.
+func TestFailedTypeAlerts(t *testing.T) {
+	t.Parallel()
+
+	records := map[string][]string{"A": {ip1}, txt: {spf1}}
+	aOnly := map[string][]string{"A": {ip1}}
+
+	bothAnswer := map[string]*resolver.NameserverResponse{
+		nsA: response(records), nsB: response(records),
+	}
+	bTXTFails := map[string]*resolver.NameserverResponse{
+		nsA: response(records), nsB: response(aOnly, txt),
+	}
+	onlyA := map[string]*resolver.NameserverResponse{
+		nsA: response(records),
+	}
+	bFails := map[string]*resolver.NameserverResponse{
+		nsA: response(records),
+		nsB: {
+			Records: map[string][]string{},
+			Status:  resolver.StatusTimeout,
+			Error:   "all queries timed out",
+		},
+	}
+
+	tests := []struct {
+		name   string
+		checks []map[string]*resolver.NameserverResponse
+		want   alertCounts
+	}{
+		{
+			"type failing at one nameserver alerts nothing, nor its next answer",
+			[]map[string]*resolver.NameserverResponse{
+				bothAnswer, bTXTFails, bothAnswer,
+			},
+			alertCounts{},
+		},
+		{
+			"type failing on the first check alerts nothing on the next",
+			[]map[string]*resolver.NameserverResponse{bTXTFails, bothAnswer},
+			alertCounts{},
+		},
+		{
+			"type failing at a nameserver new on that check alerts nothing",
+			[]map[string]*resolver.NameserverResponse{
+				onlyA, bTXTFails, bothAnswer,
+			},
+			alertCounts{},
+		},
+		{
+			"type failing at a recovering nameserver alerts the recovery",
+			[]map[string]*resolver.NameserverResponse{
+				bFails, bTXTFails, bothAnswer,
+			},
+			alertCounts{recoveries: 1},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			states := savedChecks(tt.checks...)
+
+			got := countAlerts(t, states[0], states[1:])
+			if got != tt.want {
+				t.Errorf("sent %+v, want %+v", got, tt.want)
+			}
+		})
+	}
+}
+
+// TestFailedTypeComparedOnceItAnswers saves the checks of each case in
+// turn as TestFailedTypeAlerts does. nsB's TXT query fails on one check,
+// and the TXT record changes: the change is sent as a Record Change for
+// each nameserver on the check where it answers it, and an Inconsistency
+// only when nsB still answers the old record.
+func TestFailedTypeComparedOnceItAnswers(t *testing.T) {
+	t.Parallel()
+
+	records := map[string][]string{"A": {ip1}, txt: {spf1}}
+	changed := map[string][]string{"A": {ip1}, txt: {spf2}}
+	aOnly := map[string][]string{"A": {ip1}}
+
+	bothAnswer := map[string]*resolver.NameserverResponse{
+		nsA: response(records), nsB: response(records),
+	}
+	bTXTFails := map[string]*resolver.NameserverResponse{
+		nsA: response(records), nsB: response(aOnly, txt),
+	}
+	bothChange := map[string]*resolver.NameserverResponse{
+		nsA: response(changed), nsB: response(changed),
+	}
+	aChangesBTXTFails := map[string]*resolver.NameserverResponse{
+		nsA: response(changed), nsB: response(aOnly, txt),
+	}
+	bStillOld := map[string]*resolver.NameserverResponse{
+		nsA: response(changed), nsB: response(records),
+	}
+
+	tests := []struct {
+		name   string
+		checks []map[string]*resolver.NameserverResponse
+		want   alertCounts
+	}{
+		{
+			"change made while the type failed",
+			[]map[string]*resolver.NameserverResponse{
+				bothAnswer, bTXTFails, bothChange,
+			},
+			alertCounts{recordChanges: 2},
+		},
+		{
+			"change seen at one nameserver while the other's type failed",
+			[]map[string]*resolver.NameserverResponse{
+				bothAnswer, aChangesBTXTFails, bothChange,
+			},
+			alertCounts{recordChanges: 2},
+		},
+		{
+			"old record answered after the type failed",
+			[]map[string]*resolver.NameserverResponse{
+				bothAnswer, aChangesBTXTFails, bStillOld,
+			},
+			alertCounts{recordChanges: 1, inconsistencies: 1},
+		},
+		{
+			"change after the type failed on the first check and answered",
+			[]map[string]*resolver.NameserverResponse{
+				bTXTFails, bothAnswer, bothChange,
+			},
+			alertCounts{recordChanges: 2},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			states := savedChecks(tt.checks...)
+
+			got := countAlerts(t, states[0], states[1:])
+			if got != tt.want {
+				t.Errorf("sent %+v, want %+v", got, tt.want)
+			}
+		})
+	}
+}
+
+// TestFailedTypeLeftOutOfMessages checks that a Record Change and an
+// Inconsistency name only the record types they compared. nsB's TXT
+// records are not known on the first check, and on the second either
+// answered or still not known; nsB's A record changes, so both alerts
+// are sent and name the A record alone.
+func TestFailedTypeLeftOutOfMessages(t *testing.T) {
+	t.Parallel()
+
+	withTXT := map[string][]string{"A": {ip1}, txt: {spf1}}
+	txtNotKnown := func(address string) *state.NameserverRecordState {
+		return &state.NameserverRecordState{
+			Records:      map[string][]string{"A": {address}},
+			FailedTypes:  []string{txt},
+			UnknownTypes: []string{txt},
+			Status:       "ok",
+		}
+	}
+
+	before := saved(map[string]*state.NameserverRecordState{
+		nsA: answered(withTXT), nsB: txtNotKnown(ip1),
+	})
+
+	tests := []struct {
+		name  string
+		after *state.HostnameState
+	}{
+		{
+			"TXT answers",
+			saved(map[string]*state.NameserverRecordState{
+				nsA: answered(withTXT),
+				nsB: answered(map[string][]string{"A": {ip2}, txt: {spf1}}),
+			}),
+		},
+		{
+			"TXT still not known",
+			saved(map[string]*state.NameserverRecordState{
+				nsA: answered(withTXT), nsB: txtNotKnown(ip2),
+			}),
+		},
+	}
+
+	want := map[string]string{
+		"Record Change: " + host: "Hostname: " + host +
+			"\nNameserver: " + nsB + "\nType: A\nOld: " + ip1 + "\nNew: " + ip2,
+		"Inconsistency: " + host: "Hostname: " + host +
+			"\nType: A\n" + nsA + ": " + ip1 + "\n" + nsB + ": " + ip2,
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			// The hostname change detection uses only the notifier.
+			notifier := &mockNotifier{}
+			w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
+
+			w.DetectHostnameChanges(t.Context(), host, before, tt.after)
+
+			notifications := notifier.getNotifications()
+			if len(notifications) != len(want) {
+				t.Fatalf(
+					"sent %d notifications, want %d: %v",
+					len(notifications), len(want), notifications,
+				)
+			}
+
+			for _, n := range notifications {
+				if n.Message != want[n.Title] {
+					t.Errorf(
+						"%s message:\n%s\nwant:\n%s",
+						n.Title, n.Message, want[n.Title],
+					)
+				}
+			}
+		})
+	}
+}

internal/watcher/inconsistency_test.go

@@ -183,3 +183,58 @@ func TestInconsistencyAlert(t *testing.T) {
 		})
 	}
 }
+
+// TestFirstCheckAfterRepeatedValuesLoaded saves a state file holding a
+// hostname's CNAME once for every record type asked for, as checks did
+// before each value was stored once, and two addresses each repeated,
+// and loads it. A check that then finds each value once at each
+// nameserver must notify nothing.
+func TestFirstCheckAfterRepeatedValuesLoaded(t *testing.T) {
+	t.Parallel()
+
+	const (
+		cnameType = "CNAME"
+		cname     = "c.example.net."
+	)
+
+	cfg := defaultTestConfig(t)
+	repeated := map[string][]string{
+		"A":       {ip2, ip1, ip2, ip1},
+		cnameType: {cname, cname, cname, cname, cname, cname, cname, cname},
+	}
+	once := map[string][]string{"A": {ip1, ip2}, cnameType: {cname}}
+
+	saved := newTestDeps(t, cfg).state
+	saved.SetHostnameState(host, hostnameState(map[string]map[string][]string{
+		nsA: repeated, nsB: repeated,
+	}))
+
+	err := saved.Save()
+	if err != nil {
+		t.Fatalf("saving the state file: %v", err)
+	}
+
+	deps := newTestDeps(t, cfg)
+
+	err = deps.state.Load()
+	if err != nil {
+		t.Fatalf("loading the state file: %v", err)
+	}
+
+	prev, ok := deps.state.GetHostnameState(host)
+	if !ok {
+		t.Fatal("the state file has no state for " + host)
+	}
+
+	current := hostnameState(map[string]map[string][]string{
+		nsA: once, nsB: once,
+	})
+
+	// The hostname change detection uses only the notifier.
+	w := watcher.NewForTest(nil, nil, nil, nil, nil, deps.notifier)
+	w.DetectHostnameChanges(t.Context(), host, prev, current)
+
+	if got := deps.notifier.getNotifications(); len(got) != 0 {
+		t.Errorf("sent %v, want no notification", got)
+	}
+}

internal/watcher/message_test.go

@@ -0,0 +1,215 @@
+package watcher_test
+
+import (
+	"maps"
+	"strings"
+	"testing"
+
+	"sneak.berlin/go/dnswatcher/internal/config"
+	"sneak.berlin/go/dnswatcher/internal/state"
+	"sneak.berlin/go/dnswatcher/internal/watcher"
+)
+
+// When one nameserver's A record changes and its TXT record does not,
+// the record change and the inconsistency it starts name the A record
+// alone, with its values written as plain text.
+func TestChangeMessagesNameTheChangedType(t *testing.T) {
+	t.Parallel()
+
+	// A nameserver's records: this A address and the same TXT record.
+	records := func(address string) map[string][]string {
+		return map[string][]string{
+			"A":   {address},
+			"TXT": {"v=spf1 -all"},
+		}
+	}
+
+	before := hostnameState(map[string]map[string][]string{
+		nsA: records(ip1),
+		nsB: records(ip1),
+	})
+	after := hostnameState(map[string]map[string][]string{
+		nsA: records(ip1),
+		nsB: records(ip2),
+	})
+
+	// The hostname change detection uses only the notifier.
+	notifier := &mockNotifier{}
+	w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
+
+	w.DetectHostnameChanges(t.Context(), host, before, after)
+
+	want := map[string]string{
+		"Record Change: " + host: `Hostname: www.example.net
+Nameserver: b.ns.example.net.
+Type: A
+Old: 192.0.2.1
+New: 192.0.2.2`,
+		"Inconsistency: " + host: `Hostname: www.example.net
+Type: A
+a.ns.example.net.: 192.0.2.1
+b.ns.example.net.: 192.0.2.2`,
+	}
+
+	notifications := notifier.getNotifications()
+	if len(notifications) != len(want) {
+		t.Fatalf(
+			"sent %d notifications, want %d: %v",
+			len(notifications), len(want), notifications,
+		)
+	}
+
+	for _, n := range notifications {
+		if n.Message != want[n.Title] {
+			t.Errorf(
+				"%s message:\n%s\nwant:\n%s",
+				n.Title, n.Message, want[n.Title],
+			)
+		}
+	}
+}
+
+// Every kind of notification about a configured apex domain's own
+// records names it as a domain.
+func TestDomainRecordNotificationsNameTheDomain(t *testing.T) {
+	t.Parallel()
+
+	notifier := &mockNotifier{}
+	w := watcher.NewForTest(
+		&config.Config{Domains: []string{domain}},
+		nil, nil, nil, nil, notifier,
+	)
+
+	// nsA's address changes, which also makes it differ from nsC; nsB
+	// fails; nsC answers again; nsD is gone.
+	nsD := "d.ns.example.net."
+	w.DetectHostnameChanges(t.Context(), domain,
+		saved(map[string]*state.NameserverRecordState{
+			nsA: answered(map[string][]string{"A": {ip1}}),
+			nsB: answered(map[string][]string{"A": {ip1}}),
+			nsC: failed(),
+			nsD: answered(map[string][]string{"A": {ip1}}),
+		}),
+		saved(map[string]*state.NameserverRecordState{
+			nsA: answered(map[string][]string{"A": {ip2}}),
+			nsB: failed(),
+			nsC: answered(map[string][]string{"A": {ip1}}),
+		}),
+	)
+
+	// The address at the end of its CNAME chain changes.
+	w.DetectHostnameChanges(
+		t.Context(), domain, cnameState(ip1), cnameState(ip2),
+	)
+
+	// NS Failure is sent for nsB failing and for nsD being gone.
+	want := map[string]int{
+		"Record Change":        1,
+		"Inconsistency":        1,
+		"NS Failure":           2,
+		"NS Recovery":          1,
+		"CNAME Address Change": 1,
+	}
+
+	sent := make(map[string]int)
+
+	for _, n := range notifier.getNotifications() {
+		kind, _, _ := strings.Cut(n.Title, ":")
+		sent[kind]++
+
+		if !strings.HasPrefix(n.Message, "Domain: "+domain+"\n") {
+			t.Errorf("%s message does not name the domain:\n%s",
+				n.Title, n.Message)
+		}
+	}
+
+	if !maps.Equal(sent, want) {
+		t.Errorf("sent %v, want %v", sent, want)
+	}
+}
+
+// The startup notification counts the configured domains and hostnames,
+// although the state's hostnames also hold the apex domain's own
+// records. Nothing is looked up: the watcher has no resolver.
+func TestStartupNotificationCountsConfiguredNames(t *testing.T) {
+	t.Parallel()
+
+	cfg := defaultTestConfig(t)
+	cfg.SendTestNotification = true
+	cfg.Domains = []string{domain}
+	cfg.Hostnames = []string{host}
+
+	deps := newTestDeps(t, cfg)
+	w := watcher.NewForTest(cfg, deps.state, nil, nil, nil, deps.notifier)
+
+	// The state a check of both names saves.
+	deps.state.SetDomainState(domain, &state.DomainState{
+		Nameservers: []string{nsA},
+	})
+
+	for _, name := range []string{domain, host} {
+		deps.state.SetHostnameState(name, saved(
+			map[string]*state.NameserverRecordState{
+				nsA: answered(map[string][]string{"A": {ip1}}),
+			},
+		))
+	}
+
+	w.MaybeSendTestNotification(t.Context())
+
+	notifications := deps.notifier.getNotifications()
+
+	counts := "\nMonitoring 1 domain(s) and 1 hostname(s).\n"
+	if len(notifications) != 1 ||
+		!strings.Contains(notifications[0].Message, counts) {
+		t.Errorf("sent %v, want one message with %q", notifications, counts)
+	}
+}
+
+// A Port Change notification lists the configured apex domain and the
+// hostname that resolve to the port's address on separate lines. The
+// port checks read the saved hostname state and look nothing up, so the
+// watcher has no resolver.
+func TestPortChangeListsDomainsApartFromHostnames(t *testing.T) {
+	t.Parallel()
+
+	cfg := defaultTestConfig(t)
+	cfg.Domains = []string{domain}
+	cfg.Hostnames = []string{host}
+
+	deps := newTestDeps(t, cfg)
+	w := watcher.NewForTest(
+		cfg, deps.state, nil,
+		deps.portChecker, deps.tlsChecker, deps.notifier,
+	)
+	w.SetFirstRun(false)
+
+	// Both names resolve to ip1, whose port 443 the previous check
+	// found open. It is closed now.
+	for _, name := range []string{domain, host} {
+		deps.state.SetHostnameState(name, saved(
+			map[string]*state.NameserverRecordState{
+				nsA: answered(map[string][]string{"A": {ip1}}),
+			},
+		))
+	}
+
+	key := ip1 + ":443"
+	deps.state.SetPortState(key, &state.PortState{
+		Open: true, Hostnames: []string{domain, host},
+	})
+	deps.portChecker.closed = true
+
+	w.CheckAllPorts(t.Context())
+
+	title := "Port Change: " + key
+	want := `Domains: example.net
+Hostnames: www.example.net
+Address: 192.0.2.1:443
+Port now closed`
+
+	got := deps.notifier.getNotifications()
+	if len(got) != 1 || got[0].Title != title || got[0].Message != want {
+		t.Errorf("sent %v, want one %q with message:\n%s", got, title, want)
+	}
+}

internal/watcher/nsfailure_test.go

@@ -201,7 +201,7 @@ func TestNameserverThatNeverAnswers(t *testing.T) {
 	}
 
 	hs := watcher.BuildHostnameState(
-		map[string]*resolver.NameserverResponse{nsA: resp}, time.Now(),
+		map[string]*resolver.NameserverResponse{nsA: resp}, nil, time.Now(),
 	)
 
 	got := hs.RecordsByNameserver[nsA]
@@ -256,7 +256,7 @@ func TestNameserverThatAnswersNXDOMAIN(t *testing.T) {
 	}
 
 	hs := watcher.BuildHostnameState(
-		map[string]*resolver.NameserverResponse{ns: resp}, time.Now(),
+		map[string]*resolver.NameserverResponse{ns: resp}, nil, time.Now(),
 	)
 
 	got := hs.RecordsByNameserver[ns]
@@ -320,7 +320,7 @@ func TestNameserverThatRefuses(t *testing.T) {
 	}
 
 	hs := watcher.BuildHostnameState(
-		map[string]*resolver.NameserverResponse{ns: resp}, time.Now(),
+		map[string]*resolver.NameserverResponse{ns: resp}, nil, time.Now(),
 	)
 
 	got := hs.RecordsByNameserver[ns]

internal/watcher/removed_test.go

@@ -0,0 +1,224 @@
+package watcher_test
+
+import (
+	"maps"
+	"slices"
+	"testing"
+
+	"sneak.berlin/go/dnswatcher/internal/state"
+	"sneak.berlin/go/dnswatcher/internal/watcher"
+)
+
+// TestRemovedTargetsLeaveTheState loads a state saved while a domain
+// and a hostname now removed from the configuration were still in it,
+// and runs the removal that Run does before the first check. The
+// removed names' domain, hostname and certificate entries are gone,
+// the configured names' are kept, and nothing is notified. Nothing is
+// looked up: the watcher has no resolver.
+func TestRemovedTargetsLeaveTheState(t *testing.T) {
+	t.Parallel()
+
+	const (
+		removedDomain = "example.com"
+		removedHost   = "www.example.com"
+	)
+
+	cfg := defaultTestConfig(t)
+	cfg.Domains = []string{domain}
+	cfg.Hostnames = []string{host}
+
+	deps := newTestDeps(t, cfg)
+	w := watcher.NewForTest(
+		cfg, deps.state, nil,
+		deps.portChecker, deps.tlsChecker, deps.notifier,
+	)
+
+	// The state a check of all four names saves, each name at ip1.
+	for _, name := range []string{domain, removedDomain} {
+		deps.state.SetDomainState(name, &state.DomainState{
+			Nameservers: []string{nsA},
+		})
+	}
+
+	for _, name := range []string{domain, host, removedDomain, removedHost} {
+		deps.state.SetHostnameState(name, saved(
+			map[string]*state.NameserverRecordState{
+				nsA: answered(map[string][]string{"A": {ip1}}),
+			},
+		))
+		deps.state.SetCertificateState(
+			ip1+":443:"+name, &state.CertificateState{Status: "ok"},
+		)
+	}
+
+	err := deps.state.Save()
+	if err != nil {
+		t.Fatalf("saving the state: %v", err)
+	}
+
+	err = deps.state.Load()
+	if err != nil {
+		t.Fatalf("loading the state: %v", err)
+	}
+
+	w.CleanupRemovedTargets()
+
+	snap := deps.state.GetSnapshot()
+
+	got := slices.Sorted(maps.Keys(snap.Domains))
+	if want := []string{domain}; !slices.Equal(got, want) {
+		t.Errorf("domain entries %v, want %v", got, want)
+	}
+
+	got = slices.Sorted(maps.Keys(snap.Hostnames))
+	if want := []string{domain, host}; !slices.Equal(got, want) {
+		t.Errorf("hostname entries %v, want %v", got, want)
+	}
+
+	got = slices.Sorted(maps.Keys(snap.Certificates))
+	if want := []string{
+		ip1 + ":443:" + domain, ip1 + ":443:" + host,
+	}; !slices.Equal(got, want) {
+		t.Errorf("certificate entries %v, want %v", got, want)
+	}
+
+	if sent := deps.notifier.getNotifications(); len(sent) != 0 {
+		t.Errorf("sent %v, want nothing", sent)
+	}
+}
+
+// TestRemovedTargetsLeaveThePortEntries loads a state whose port
+// entries name a domain and a hostname now removed from the
+// configuration, and runs the removal that Run does before the first
+// check. The removed names are off each port entry's list of names, the
+// entry only they had is gone, the entry that also names configured
+// names is kept for the port checks, and nothing is notified.
+func TestRemovedTargetsLeaveThePortEntries(t *testing.T) {
+	t.Parallel()
+
+	const (
+		removedDomain = "example.com"
+		removedHost   = "www.example.com"
+	)
+
+	cfg := defaultTestConfig(t)
+	cfg.Domains = []string{domain}
+	cfg.Hostnames = []string{host}
+
+	deps := newTestDeps(t, cfg)
+	w := watcher.NewForTest(
+		cfg, deps.state, nil,
+		deps.portChecker, deps.tlsChecker, deps.notifier,
+	)
+
+	// The port 443 entries a check of all four names saves: each name
+	// at ip1, except the removed hostname, at ip2.
+	deps.state.SetPortState(ip1+":443", &state.PortState{
+		Open: true, Hostnames: []string{removedDomain, domain, host},
+	})
+	deps.state.SetPortState(ip2+":443", &state.PortState{
+		Open: true, Hostnames: []string{removedHost},
+	})
+
+	err := deps.state.Save()
+	if err != nil {
+		t.Fatalf("saving the state: %v", err)
+	}
+
+	err = deps.state.Load()
+	if err != nil {
+		t.Fatalf("loading the state: %v", err)
+	}
+
+	w.CleanupRemovedTargets()
+
+	if sent := deps.notifier.getNotifications(); len(sent) != 0 {
+		t.Errorf("sent %v, want nothing", sent)
+	}
+
+	snap := deps.state.GetSnapshot()
+
+	got := slices.Sorted(maps.Keys(snap.Ports))
+	if want := []string{ip1 + ":443"}; !slices.Equal(got, want) {
+		t.Fatalf("port entries %v, want %v", got, want)
+	}
+
+	got = snap.Ports[ip1+":443"].Hostnames
+	if want := []string{domain, host}; !slices.Equal(got, want) {
+		t.Errorf("names of port entry %s:443 %v, want %v", ip1, got, want)
+	}
+}
+
+// TestCertificateStateForAnAddressGone runs the port checks on hostname
+// state built here for a configured hostname, with certificate entries
+// saved for it at ip1, ip2 and an IPv6 address. When its nameservers
+// answered with ip1 and the IPv6 address, the entry for ip2 is removed.
+// When none of them answered, its addresses are not known, and every
+// entry is kept. Nothing is notified, and nothing is looked up.
+func TestCertificateStateForAnAddressGone(t *testing.T) {
+	t.Parallel()
+
+	const ip6 = "2001:db8::1"
+
+	tests := []struct {
+		name     string
+		hostname *state.HostnameState
+		want     []string
+	}{
+		{
+			"answered without ip2",
+			saved(map[string]*state.NameserverRecordState{
+				nsA: answered(map[string][]string{
+					"A": {ip1}, "AAAA": {ip6},
+				}),
+			}),
+			[]string{ip1 + ":443:" + host, ip6 + ":443:" + host},
+		},
+		{
+			"no nameserver answered",
+			saved(map[string]*state.NameserverRecordState{
+				nsA: failed(), nsB: failed(),
+			}),
+			[]string{
+				ip1 + ":443:" + host,
+				ip2 + ":443:" + host,
+				ip6 + ":443:" + host,
+			},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			cfg := defaultTestConfig(t)
+			cfg.Hostnames = []string{host}
+
+			deps := newTestDeps(t, cfg)
+			w := watcher.NewForTest(
+				cfg, deps.state, nil,
+				deps.portChecker, deps.tlsChecker, deps.notifier,
+			)
+			w.SetFirstRun(false)
+
+			deps.state.SetHostnameState(host, tt.hostname)
+
+			for _, ip := range []string{ip1, ip2, ip6} {
+				deps.state.SetCertificateState(
+					ip+":443:"+host, &state.CertificateState{Status: "ok"},
+				)
+			}
+
+			w.CheckAllPorts(t.Context())
+
+			got := slices.Sorted(maps.Keys(deps.state.GetSnapshot().Certificates))
+			if !slices.Equal(got, tt.want) {
+				t.Errorf("certificate entries %v, want %v", got, tt.want)
+			}
+
+			if sent := deps.notifier.getNotifications(); len(sent) != 0 {
+				t.Errorf("sent %v, want nothing", sent)
+			}
+		})
+	}
+}

internal/watcher/watcher.go

@@ -2,8 +2,10 @@ package watcher
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"log/slog"
+	"maps"
 	"slices"
 	"sort"
 	"strings"
@@ -123,10 +125,13 @@ func (w *Watcher) Run(ctx context.Context) {
 		"watcher starting",
 		"domains", len(w.config.Domains),
 		"hostnames", len(w.config.Hostnames),
-		"dnsInterval", w.config.DNSInterval,
-		"tlsInterval", w.config.TLSInterval,
+		// As text: the JSON log writes a time.Duration as bare
+		// nanoseconds.
+		"dnsInterval", w.config.DNSInterval.String(),
+		"tlsInterval", w.config.TLSInterval.String(),
 	)
 
+	w.cleanupRemovedTargets()
 	w.RunOnce(ctx)
 	w.maybeSendTestNotification(ctx)
 
@@ -197,6 +202,59 @@ func (w *Watcher) detectFirstRun() {
 	}
 }
 
+// cleanupRemovedTargets removes from the loaded state the domain,
+// hostname and certificate entries of names no longer in the
+// configuration, which changes only at a restart, and takes those names
+// off each port entry's list of names, removing a port entry left with
+// none. Nothing is notified. A configured domain's own records are
+// saved as a hostname entry under its name, which is kept.
+func (w *Watcher) cleanupRemovedTargets() {
+	for _, name := range w.state.GetAllDomainNames() {
+		if !w.isDomain(name) {
+			w.state.DeleteDomainState(name)
+		}
+	}
+
+	for _, name := range w.state.GetAllHostnames() {
+		if !w.isConfigured(name) {
+			w.state.DeleteHostnameState(name)
+		}
+	}
+
+	for _, key := range w.state.GetAllCertificateKeys() {
+		if _, hostname := parseCertKey(key); !w.isConfigured(hostname) {
+			w.state.DeleteCertificateState(key)
+		}
+	}
+
+	for _, key := range w.state.GetAllPortKeys() {
+		ps, ok := w.state.GetPortState(key)
+		if !ok {
+			continue
+		}
+
+		var names []string
+
+		for _, name := range ps.Hostnames {
+			if w.isConfigured(name) {
+				names = append(names, name)
+			}
+		}
+
+		if len(names) == 0 {
+			w.state.DeletePortState(key)
+
+			continue
+		}
+
+		w.state.SetPortState(key, &state.PortState{
+			Open:        ps.Open,
+			Hostnames:   names,
+			LastChecked: ps.LastChecked,
+		})
+	}
+}
+
 // runDNSChecks performs DNS resolution for all configured domains
 // and hostnames, updating state with freshly resolved records.
 // This must complete before port or TLS checks run so those
@@ -211,13 +269,29 @@ func (w *Watcher) runDNSChecks(ctx context.Context) {
 	}
 }
 
+// logFailedLookup logs a failed DNS lookup at error level, unless ctx
+// was cancelled: shutdown cancels it, and a lookup it cut short did not
+// fail. A lookup that ran out of time did fail, so it is logged.
+func (w *Watcher) logFailedLookup(
+	ctx context.Context,
+	msg string,
+	args ...any,
+) {
+	if errors.Is(ctx.Err(), context.Canceled) {
+		return
+	}
+
+	w.log.Error(msg, args...)
+}
+
 func (w *Watcher) checkDomain(
 	ctx context.Context,
 	domain string,
 ) {
 	nameservers, err := w.resolver.LookupNS(ctx, domain)
 	if err != nil {
-		w.log.Error(
+		w.logFailedLookup(
+			ctx,
 			"failed to lookup NS",
 			"domain", domain,
 			"error", err,
@@ -252,28 +326,10 @@ func (w *Watcher) checkDomain(
 		LastChecked:         now,
 	})
 
-	// Also look up A/AAAA records for the apex domain so that
-	// port and TLS checks (which read HostnameState) can find
-	// the domain's IP addresses.
-	results, err := w.resolver.LookupAllRecords(ctx, domain)
-	if err != nil {
-		w.log.Error(
-			"failed to lookup records for domain",
-			"domain", domain,
-			"error", err,
-		)
-
-		return
-	}
-
-	newState := buildHostnameState(results, now)
-
-	prevHS, hasPrevHS := w.state.GetHostnameState(domain)
-	if hasPrevHS && !w.firstRun {
-		w.detectHostnameChanges(ctx, domain, prevHS, newState)
-	}
-
-	w.state.SetHostnameState(domain, newState)
+	// The apex domain's records are also checked and saved as a
+	// hostname's, so that the port and TLS checks find its addresses.
+	// Notifications about them name it as a domain (see nameLine).
+	w.checkHostname(ctx, domain)
 }
 
 func (w *Watcher) detectNSChanges(
@@ -337,7 +393,8 @@ func (w *Watcher) resolveNameserverAddresses(
 			continue
 		}
 
-		w.log.Error(
+		w.logFailedLookup(
+			ctx,
 			"no addresses found for nameserver",
 			"nameserver", ns,
 			"error", err,
@@ -389,7 +446,8 @@ func (w *Watcher) checkHostname(
 ) {
 	results, err := w.resolver.LookupAllRecords(ctx, hostname)
 	if err != nil {
-		w.log.Error(
+		w.logFailedLookup(
+			ctx,
 			"failed to lookup records",
 			"hostname", hostname,
 			"error", err,
@@ -398,9 +456,25 @@ func (w *Watcher) checkHostname(
 		return
 	}
 
-	newState := buildHostnameState(results, time.Now().UTC())
+	prev, _ := w.state.GetHostnameState(hostname)
 
+	w.updateHostnameState(
+		ctx, hostname, buildHostnameState(results, prev, time.Now().UTC()),
+	)
+}
+
+// updateHostnameState finishes a check of hostname from newState, built
+// from its nameservers' answers: it follows the CNAME in them, notifies
+// what changed since the previous check, and saves newState.
+func (w *Watcher) updateHostnameState(
+	ctx context.Context,
+	hostname string,
+	newState *state.HostnameState,
+) {
 	prev, hasPrev := w.state.GetHostnameState(hostname)
+
+	w.resolveCNAMEAddresses(ctx, hostname, newState, prev)
+
 	if hasPrev && !w.firstRun {
 		w.detectHostnameChanges(ctx, hostname, prev, newState)
 	}
@@ -408,12 +482,89 @@ func (w *Watcher) checkHostname(
 	w.state.SetHostnameState(hostname, newState)
 }
 
+// resolveCNAMEAddresses saves in current the addresses at the end of
+// hostname's CNAME chain, when the nameservers' answers in current hold
+// a CNAME and no address, and an empty list otherwise. Every CNAME
+// target the nameservers gave is followed with ResolveIPAddresses and
+// the addresses found for all of them are saved, so nameservers that
+// disagree on the target do not change the result from check to check.
+// The addresses saved in prev, which may be nil, are kept when none of
+// the name's nameservers answered its queries for A, AAAA and CNAME,
+// and when a target cannot be followed, as when no nameserver of a zone
+// in its chain answers.
+func (w *Watcher) resolveCNAMEAddresses(
+	ctx context.Context,
+	hostname string,
+	current, prev *state.HostnameState,
+) {
+	var prevAddresses []string
+	if prev != nil {
+		prevAddresses = prev.CNAMEAddresses
+	}
+
+	// Empty, not nil: nil means the addresses are not known.
+	current.CNAMEAddresses = []string{}
+
+	answered := false
+	targets := make(map[string]bool)
+
+	for _, nsState := range current.RecordsByNameserver {
+		if nsState.Status != statusOK ||
+			slices.Contains(nsState.FailedTypes, "A") ||
+			slices.Contains(nsState.FailedTypes, "AAAA") ||
+			slices.Contains(nsState.FailedTypes, "CNAME") {
+			continue
+		}
+
+		answered = true
+
+		if len(nsState.Records["A"]) > 0 || len(nsState.Records["AAAA"]) > 0 {
+			return
+		}
+
+		for _, target := range nsState.Records["CNAME"] {
+			targets[target] = true
+		}
+	}
+
+	if !answered {
+		current.CNAMEAddresses = prevAddresses
+
+		return
+	}
+
+	for target := range targets {
+		ips, err := w.resolver.ResolveIPAddresses(ctx, target)
+		if err != nil {
+			w.logFailedLookup(
+				ctx,
+				"failed to follow CNAME",
+				"hostname", hostname,
+				"target", target,
+				"error", err,
+			)
+
+			current.CNAMEAddresses = prevAddresses
+
+			return
+		}
+
+		current.CNAMEAddresses = append(current.CNAMEAddresses, ips...)
+	}
+
+	// Still the empty list when every chain ends in no address.
+	slices.Sort(current.CNAMEAddresses)
+	current.CNAMEAddresses = slices.Compact(current.CNAMEAddresses)
+}
+
 // buildHostnameState saves each nameserver's response. A nameserver
-// that answered, even with NXDOMAIN or no records, is saved as ok; one
-// that timed out or failed is saved as error with the reason, and its
-// empty record set is not an answer.
+// that answered, even with NXDOMAIN or no records, is saved as ok, with
+// the record types whose query failed; one that timed out or failed is
+// saved as error with the reason, and its empty record set is not an
+// answer. prev is the hostname's state from the previous check, or nil.
 func buildHostnameState(
 	results map[string]*resolver.NameserverResponse,
+	prev *state.HostnameState,
 	now time.Time,
 ) *state.HostnameState {
 	hs := &state.HostnameState{
@@ -425,7 +576,7 @@ func buildHostnameState(
 
 	for ns, resp := range results {
 		nsState := &state.NameserverRecordState{
-			Records:     resp.Records,
+			Records:     maps.Clone(resp.Records),
 			Status:      statusOK,
 			LastChecked: now,
 		}
@@ -434,6 +585,15 @@ func buildHostnameState(
 			resp.Status == resolver.StatusError {
 			nsState.Status = statusError
 			nsState.Error = resp.Error
+		} else {
+			nsState.FailedTypes = resp.FailedTypes
+
+			var prevNS *state.NameserverRecordState
+			if prev != nil {
+				prevNS = prev.RecordsByNameserver[ns]
+			}
+
+			keepFailedTypes(nsState, prevNS)
 		}
 
 		hs.RecordsByNameserver[ns] = nsState
@@ -442,6 +602,27 @@ func buildHostnameState(
 	return hs
 }
 
+// keepFailedTypes copies into nsState, for each record type in its
+// FailedTypes, the records prevNS, the nameserver's state from the
+// previous check, holds for that type, which may be none. When prevNS
+// does not know them either, because the nameserver was new or failing
+// then or the type was in its UnknownTypes, the type goes in
+// nsState.UnknownTypes instead.
+func keepFailedTypes(nsState, prevNS *state.NameserverRecordState) {
+	for _, rtype := range nsState.FailedTypes {
+		if prevNS == nil || prevNS.Status != statusOK ||
+			slices.Contains(prevNS.UnknownTypes, rtype) {
+			nsState.UnknownTypes = append(nsState.UnknownTypes, rtype)
+
+			continue
+		}
+
+		if records, ok := prevNS.Records[rtype]; ok {
+			nsState.Records[rtype] = records
+		}
+	}
+}
+
 func (w *Watcher) detectHostnameChanges(
 	ctx context.Context,
 	hostname string,
@@ -451,11 +632,89 @@ func (w *Watcher) detectHostnameChanges(
 	w.detectNSDisappearances(ctx, hostname, prev, current)
 	w.detectNSFailures(ctx, hostname, prev, current)
 	w.detectInconsistencies(ctx, hostname, prev, current)
+	w.detectCNAMEAddressChanges(ctx, hostname, prev, current)
+}
+
+// isDomain reports whether name is a configured apex domain, whose own
+// records are checked and saved as a hostname's are.
+func (w *Watcher) isDomain(name string) bool {
+	return slices.Contains(w.config.Domains, name)
+}
+
+// nameLine is the line a notification about name's records starts with:
+// "Domain: " and the name for a configured apex domain, and
+// "Hostname: " otherwise.
+func (w *Watcher) nameLine(name string) string {
+	if w.isDomain(name) {
+		return "Domain: " + name
+	}
+
+	return "Hostname: " + name
+}
+
+// portNameLines lists the names that resolve to a port's address, the
+// configured apex domains on one line and the hostnames on the next,
+// leaving out a line that would name nothing.
+func (w *Watcher) portNameLines(names []string) string {
+	var domains, hostnames []string
+
+	for _, name := range names {
+		if w.isDomain(name) {
+			domains = append(domains, name)
+		} else {
+			hostnames = append(hostnames, name)
+		}
+	}
+
+	var lines []string
+
+	if len(domains) > 0 {
+		lines = append(lines, "Domains: "+strings.Join(domains, ", "))
+	}
+
+	if len(hostnames) > 0 {
+		lines = append(lines, "Hostnames: "+strings.Join(hostnames, ", "))
+	}
+
+	return strings.Join(lines, "\n")
+}
+
+// detectCNAMEAddressChanges notifies when the addresses at the end of
+// hostname's CNAME chain differ from those the previous check saved,
+// including a change from or to none. When the previous addresses are
+// not known (nil), as on the first check after loading a state file
+// written before they were saved, nothing is compared.
+func (w *Watcher) detectCNAMEAddressChanges(
+	ctx context.Context,
+	hostname string,
+	prev, current *state.HostnameState,
+) {
+	old, cur := prev.CNAMEAddresses, current.CNAMEAddresses
+	if old == nil || sliceEqual(old, cur) {
+		return
+	}
+
+	msg := fmt.Sprintf(
+		"%s\nOld: %s\nNew: %s",
+		w.nameLine(hostname),
+		strings.Join(old, ", "),
+		strings.Join(cur, ", "),
+	)
+
+	w.notify.SendNotification(
+		ctx,
+		"CNAME Address Change: "+hostname,
+		msg,
+		"warning",
+	)
 }
 
 // detectRecordChanges compares each nameserver's records with those of
 // the previous check. Only answers are compared: a nameserver that
-// failed on either check has no records to compare.
+// failed on either check has no records to compare. The records kept
+// for a record type whose query failed are compared too, but not those
+// of a type in UnknownTypes on either check, which the message leaves
+// out as well.
 func (w *Watcher) detectRecordChanges(
 	ctx context.Context,
 	hostname string,
@@ -467,15 +726,21 @@ func (w *Watcher) detectRecordChanges(
 			continue
 		}
 
-		if recordsEqual(prevNS.Records, cur.Records) {
+		unknown := slices.Concat(prevNS.UnknownTypes, cur.UnknownTypes)
+		oldRecords := withoutTypes(prevNS.Records, unknown)
+		newRecords := withoutTypes(cur.Records, unknown)
+
+		if recordsEqual(oldRecords, newRecords) {
 			continue
 		}
 
 		msg := fmt.Sprintf(
-			"Hostname: %s\nNameserver: %s\n"+
-				"Old: %v\nNew: %v",
-			hostname, ns,
-			prevNS.Records, cur.Records,
+			"%s\nNameserver: %s\n%s",
+			w.nameLine(hostname), ns,
+			recordDifferences(
+				"Old", oldRecords,
+				"New", newRecords,
+			),
 		)
 
 		w.notify.SendNotification(
@@ -498,8 +763,8 @@ func (w *Watcher) detectNSDisappearances(
 		}
 
 		msg := fmt.Sprintf(
-			"Hostname: %s\nNameserver: %s disappeared",
-			hostname, ns,
+			"%s\nNameserver: %s disappeared",
+			w.nameLine(hostname), ns,
 		)
 
 		w.notify.SendNotification(
@@ -528,8 +793,8 @@ func (w *Watcher) detectNSFailures(
 		switch {
 		case prevNS.Status == statusOK && cur.Status == statusError:
 			msg := fmt.Sprintf(
-				"Hostname: %s\nNameserver: %s\nError: %s",
-				hostname, ns, cur.Error,
+				"%s\nNameserver: %s\nError: %s",
+				w.nameLine(hostname), ns, cur.Error,
 			)
 
 			w.notify.SendNotification(
@@ -540,8 +805,8 @@ func (w *Watcher) detectNSFailures(
 			)
 		case prevNS.Status == statusError && cur.Status == statusOK:
 			msg := fmt.Sprintf(
-				"Hostname: %s\nNameserver: %s recovered",
-				hostname, ns,
+				"%s\nNameserver: %s recovered",
+				w.nameLine(hostname), ns,
 			)
 
 			w.notify.SendNotification(
@@ -561,12 +826,20 @@ func (w *Watcher) detectInconsistencies(
 ) {
 	for _, pair := range newlyDisagreeingPairs(prev, current) {
 		ns1, ns2 := pair[0], pair[1]
+		state1 := current.RecordsByNameserver[ns1]
+		state2 := current.RecordsByNameserver[ns2]
+
+		// The record types left out of the comparison are left out of
+		// the message too.
+		failed := slices.Concat(state1.FailedTypes, state2.FailedTypes)
 
 		msg := fmt.Sprintf(
-			"Hostname: %s\n%s: %v\n%s: %v",
-			hostname,
-			ns1, current.RecordsByNameserver[ns1].Records,
-			ns2, current.RecordsByNameserver[ns2].Records,
+			"%s\n%s",
+			w.nameLine(hostname),
+			recordDifferences(
+				ns1, withoutTypes(state1.Records, failed),
+				ns2, withoutTypes(state2.Records, failed),
+			),
 		)
 
 		w.notify.SendNotification(
@@ -583,7 +856,9 @@ func (w *Watcher) detectInconsistencies(
 // except pairs where both nameservers answered in prev and already
 // differed there. A nameserver missing from prev, or that failed there,
 // is paired with every nameserver it differs from. A nameserver that
-// failed in current has no records to compare and is in no pair.
+// failed in current has no records to compare and is in no pair. In
+// both checks, a record type whose query failed at either nameserver is
+// not compared.
 func newlyDisagreeingPairs(
 	prev, current *state.HostnameState,
 ) [][2]string {
@@ -600,9 +875,9 @@ func newlyDisagreeingPairs(
 
 	for i, ns1 := range nameservers {
 		for _, ns2 := range nameservers[i+1:] {
-			if recordsEqual(
-				current.RecordsByNameserver[ns1].Records,
-				current.RecordsByNameserver[ns2].Records,
+			if nameserversAgree(
+				current.RecordsByNameserver[ns1],
+				current.RecordsByNameserver[ns2],
 			) {
 				continue
 			}
@@ -612,7 +887,7 @@ func newlyDisagreeingPairs(
 
 			if ok1 && ok2 &&
 				prev1.Status == statusOK && prev2.Status == statusOK &&
-				!recordsEqual(prev1.Records, prev2.Records) {
+				!nameserversAgree(prev1, prev2) {
 				continue
 			}
 
@@ -640,8 +915,10 @@ func (w *Watcher) checkAllPorts(ctx context.Context) {
 	}
 
 	// Phase 3: Remove port state entries that no longer have
-	// any hostname referencing them.
+	// any hostname referencing them, and certificate entries for
+	// an address their name no longer has.
 	w.cleanupStalePorts(associations)
+	w.cleanupStaleCertificates()
 }
 
 // buildPortAssociations constructs a map from IP:port keys to
@@ -725,11 +1002,45 @@ func (w *Watcher) cleanupStalePorts(
 	}
 }
 
+// cleanupStaleCertificates removes the certificate entries for an
+// address their name no longer resolves to. An entry saved for a name
+// none of whose nameservers answered is kept: that name's addresses are
+// not known, not gone.
+func (w *Watcher) cleanupStaleCertificates() {
+	for _, key := range w.state.GetAllCertificateKeys() {
+		ip, hostname := parseCertKey(key)
+
+		if slices.Contains(w.collectIPs(hostname), ip) ||
+			w.noNameserverAnswered(hostname) {
+			continue
+		}
+
+		w.state.DeleteCertificateState(key)
+	}
+}
+
+// parseCertKey splits an "ip:port:hostname" certificate key into its
+// address and hostname.
+func parseCertKey(key string) (string, string) {
+	lastColon := strings.LastIndex(key, ":")
+	if lastColon < 0 {
+		return "", key
+	}
+
+	ip, _ := parsePortKey(key[:lastColon])
+
+	return ip, key[lastColon+1:]
+}
+
+// isConfigured reports whether name is a configured domain or hostname.
+func (w *Watcher) isConfigured(name string) bool {
+	return w.isDomain(name) || slices.Contains(w.config.Hostnames, name)
+}
+
 // noNameserverAnswered reports whether name is a configured domain or
 // hostname and none of its nameservers answered on its last check.
 func (w *Watcher) noNameserverAnswered(name string) bool {
-	if !slices.Contains(w.config.Hostnames, name) &&
-		!slices.Contains(w.config.Domains, name) {
+	if !w.isConfigured(name) {
 		return false
 	}
 
@@ -747,6 +1058,9 @@ func (w *Watcher) noNameserverAnswered(name string) bool {
 	return true
 }
 
+// collectIPs returns the addresses saved for hostname: those in its
+// nameservers' A and AAAA records, and those at the end of its CNAME
+// chain.
 func (w *Watcher) collectIPs(hostname string) []string {
 	hs, ok := w.state.GetHostnameState(hostname)
 	if !ok {
@@ -765,6 +1079,10 @@ func (w *Watcher) collectIPs(hostname string) []string {
 		}
 	}
 
+	for _, ip := range hs.CNAMEAddresses {
+		ipSet[ip] = true
+	}
+
 	result := make([]string, 0, len(ipSet))
 	for ip := range ipSet {
 		result = append(result, ip)
@@ -811,8 +1129,8 @@ func (w *Watcher) checkSinglePort(
 		}
 
 		msg := fmt.Sprintf(
-			"Hosts: %s\nAddress: %s\nPort now %s",
-			strings.Join(hostnames, ", "), key, stateStr,
+			"%s\nAddress: %s\nPort now %s",
+			w.portNameLines(hostnames), key, stateStr,
 		)
 
 		w.notify.SendNotification(
@@ -1049,8 +1367,11 @@ func (w *Watcher) saveState() {
 
 // maybeSendTestNotification sends a startup status notification
 // after the first full scan completes, if SEND_TEST_NOTIFICATION
-// is enabled. The message is clearly informational ("all ok")
-// and not an error or anomaly alert.
+// is enabled. The message is informational, not an error or anomaly
+// alert. It is written before it reaches any endpoint, so it claims
+// nothing about whether the endpoints work. Domains and hostnames are
+// counted from the configuration: the state's hostnames also hold each
+// apex domain's own records.
 func (w *Watcher) maybeSendTestNotification(ctx context.Context) {
 	if !w.config.SendTestNotification {
 		return
@@ -1062,9 +1383,10 @@ func (w *Watcher) maybeSendTestNotification(ctx context.Context) {
 		"dnswatcher has started and completed its initial scan.\n"+
 			"Monitoring %d domain(s) and %d hostname(s).\n"+
 			"Tracking %d port endpoint(s) and %d TLS certificate(s).\n"+
-			"All notification channels are working.",
-		len(snap.Domains),
-		len(snap.Hostnames),
+			"This is a test notification, sent to every configured "+
+			"notification endpoint.",
+		len(w.config.Domains),
+		len(w.config.Hostnames),
 		len(snap.Ports),
 		len(snap.Certificates),
 	)
@@ -1090,6 +1412,33 @@ func toSet(items []string) map[string]bool {
 	return set
 }
 
+// nameserversAgree reports whether two nameservers' states from the same
+// check hold the same records, leaving out the record types either lists
+// in FailedTypes: the records held for those are kept from an earlier
+// check, or not known.
+func nameserversAgree(a, b *state.NameserverRecordState) bool {
+	failed := slices.Concat(a.FailedTypes, b.FailedTypes)
+
+	return recordsEqual(
+		withoutTypes(a.Records, failed), withoutTypes(b.Records, failed),
+	)
+}
+
+// withoutTypes returns a copy of records without the record types in
+// types.
+func withoutTypes(
+	records map[string][]string,
+	types []string,
+) map[string][]string {
+	records = maps.Clone(records)
+
+	for _, rtype := range types {
+		delete(records, rtype)
+	}
+
+	return records
+}
+
 func recordsEqual(
 	a, b map[string][]string,
 ) bool {
@@ -1107,6 +1456,54 @@ func recordsEqual(
 	return true
 }
 
+// recordDifferences describes, in sorted order of type, each record
+// type whose values differ between a and b: a line naming the type,
+// then a line with a's values after labelA and one with b's after
+// labelB. Types with the same values in both are left out.
+func recordDifferences(
+	labelA string, a map[string][]string,
+	labelB string, b map[string][]string,
+) string {
+	types := make([]string, 0, len(a)+len(b))
+
+	for recordType := range a {
+		types = append(types, recordType)
+	}
+
+	for recordType := range b {
+		if _, ok := a[recordType]; !ok {
+			types = append(types, recordType)
+		}
+	}
+
+	sort.Strings(types)
+
+	var lines []string
+
+	for _, recordType := range types {
+		if sliceEqual(a[recordType], b[recordType]) {
+			continue
+		}
+
+		lines = append(lines,
+			"Type: "+recordType,
+			labelA+": "+joinValues(a[recordType]),
+			labelB+": "+joinValues(b[recordType]),
+		)
+	}
+
+	return strings.Join(lines, "\n")
+}
+
+// joinValues lists record values separated by commas, or says none.
+func joinValues(values []string) string {
+	if len(values) == 0 {
+		return "none"
+	}
+
+	return strings.Join(values, ", ")
+}
+
 func sliceEqual(a, b []string) bool {
 	if len(a) != len(b) {
 		return false

internal/watcher/watcher_test.go

@@ -26,18 +26,22 @@ import (
 
 // The watcher looks these names up in live DNS with the real resolver,
 // so tests assert on what the watcher does with the answers, never on
-// the records these zones publish. testHost's nameservers and addresses
-// stay the same from one check to the next, which the tests that check
-// it twice rely on, and testSmallDomain's nameservers stay the same
-// between a test looking them up and its check. A domain check looks up
-// each nameserver's addresses, about a second per nameserver, so the
-// tests that check a domain use testSmallDomain, which has two
-// nameservers, and check it once. The tests that query testDomain's
-// nameservers directly do no domain check.
+// the records these zones publish. The nameservers of testHost and
+// testSmallDomain stay the same between a test looking them up and its
+// check. Every query a check sends is one more that can be lost, so the
+// tests keep them few. A check asks each of a name's nameservers about
+// every record type, and both names have two. A domain check also looks
+// up each nameserver's addresses at every nameserver of the zone that
+// nameserver is in: testSmallDomain's nameservers are in zones with two
+// nameservers, while a domain whose nameservers are in, say,
+// cloudflare.com, which has five, makes each domain check much longer.
+// A test checks a domain only when it is about domains, and checks once,
+// from saved state it builds, rather than twice. The tests that query
+// testDomain's nameservers directly do no domain check.
 const (
 	testDomain      = "google.com"
-	testSmallDomain = "example.com"
-	testHost        = "cloudflare.com"
+	testSmallDomain = "desec.io"
+	testHost        = "example.org"
 	testIssuer      = "DigiCert"
 )
 
@@ -259,55 +263,48 @@ func checkOnce(
 
 // runChecks builds a watcher, lets prepare set up the saved state and
 // stand-ins it starts from, and runs its checks once against live DNS.
-// If change is not nil, change then alters the saved state or stand-ins
-// and the checks run a second time. When either check finds no fresh
-// address for a name (see checkOnce), the watcher is thrown away and
-// all of this runs again on a new one, so a failed attempt leaves
-// nothing behind in the saved state, the stand-ins or the notifications.
+// When the check finds no fresh address for a name (see checkOnce), the
+// watcher is thrown away and all of this runs again on a new one, so a
+// failed attempt leaves nothing behind in the saved state, the
+// stand-ins or the notifications.
 func runChecks(
 	t *testing.T,
 	cfg *config.Config,
-	prepare, change func(deps *testDeps),
-) *testDeps {
+	prepare func(deps *testDeps),
+) (*watcher.Watcher, *testDeps) {
 	t.Helper()
 
-	var deps *testDeps
+	var (
+		w    *watcher.Watcher
+		deps *testDeps
+	)
 
 	livednstest.Retry(t, "watcher checks", func(ctx context.Context) error {
-		var w *watcher.Watcher
-
 		w, deps = newTestWatcher(t, cfg)
 
 		if prepare != nil {
 			prepare(deps)
 		}
 
-		err := checkOnce(ctx, w, deps)
-		if err != nil || change == nil {
-			return err
-		}
-
-		change(deps)
-
 		return checkOnce(ctx, w, deps)
 	})
 
-	return deps
+	return w, deps
 }
 
-// lookupNameservers returns the nameservers live DNS lists for domain,
+// lookupNameservers returns the nameservers live DNS lists for name,
 // for a test to save in the state its check starts from.
-func lookupNameservers(t *testing.T, domain string) []string {
+func lookupNameservers(t *testing.T, name string) []string {
 	t.Helper()
 
 	res := resolver.NewFromLogger(slog.Default())
 
 	var nameservers []string
 
-	livednstest.Retry(t, "LookupNS("+domain+")", func(ctx context.Context) error {
+	livednstest.Retry(t, "LookupNS("+name+")", func(ctx context.Context) error {
 		var err error
 
-		nameservers, err = res.LookupNS(ctx, domain)
+		nameservers, err = res.LookupNS(ctx, name)
 
 		return err
 	})
@@ -315,7 +312,8 @@ func lookupNameservers(t *testing.T, domain string) []string {
 	return nameservers
 }
 
-// addresses returns the A and AAAA values saved for a hostname.
+// addresses returns the A and AAAA values saved for a hostname, and the
+// addresses saved at the end of its CNAME chain.
 func addresses(hs *state.HostnameState) []string {
 	var ips []string
 
@@ -324,7 +322,7 @@ func addresses(hs *state.HostnameState) []string {
 		ips = append(ips, nsState.Records["AAAA"]...)
 	}
 
-	return ips
+	return append(ips, hs.CNAMEAddresses...)
 }
 
 // assertNotified checks that a notification with this title and
@@ -370,10 +368,18 @@ func TestFirstRunBaseline(t *testing.T) {
 	cfg.Domains = []string{testSmallDomain}
 	cfg.Hostnames = []string{testHost}
 
-	deps := runChecks(t, cfg, nil, nil)
+	_, deps := runChecks(t, cfg, nil)
 
 	assertNoNotifications(t, deps)
 	assertStatePopulated(t, deps)
+
+	// testHost answers with an address, so the check saves an empty list
+	// of CNAME addresses for it; nil would mean the check did not look
+	// at whether to follow a CNAME.
+	hs, _ := deps.state.GetHostnameState(testHost)
+	if hs.CNAMEAddresses == nil || len(hs.CNAMEAddresses) != 0 {
+		t.Errorf("saved CNAME addresses %#v, want []", hs.CNAMEAddresses)
+	}
 }
 
 func assertNoNotifications(
@@ -422,7 +428,7 @@ func TestDomainPortAndTLSChecks(t *testing.T) {
 	cfg := defaultTestConfig(t)
 	cfg.Domains = []string{testSmallDomain}
 
-	deps := runChecks(t, cfg, nil, nil)
+	_, deps := runChecks(t, cfg, nil)
 
 	snap := deps.state.GetSnapshot()
 
@@ -462,11 +468,11 @@ func TestNSChangeDetection(t *testing.T) {
 	cfg.Domains = []string{testSmallDomain}
 
 	// The saved state lists nameservers that live DNS does not.
-	deps := runChecks(t, cfg, func(deps *testDeps) {
+	_, deps := runChecks(t, cfg, func(deps *testDeps) {
 		deps.state.SetDomainState(testSmallDomain, &state.DomainState{
 			Nameservers: []string{oldNS1, oldNS2},
 		})
-	}, nil)
+	})
 
 	assertNotified(t, deps, "NS Change: "+testSmallDomain, "warning")
 
@@ -486,7 +492,7 @@ func TestNSAddressChangeDetection(t *testing.T) {
 
 	// The saved state lists the nameservers live DNS lists, each at an
 	// address live DNS never returns.
-	deps := runChecks(t, cfg, func(deps *testDeps) {
+	_, deps := runChecks(t, cfg, func(deps *testDeps) {
 		nsAddresses := make(map[string][]string, len(nameservers))
 		for _, ns := range nameservers {
 			nsAddresses[ns] = []string{oldIP}
@@ -496,7 +502,7 @@ func TestNSAddressChangeDetection(t *testing.T) {
 			Nameservers:         nameservers,
 			NameserverAddresses: nsAddresses,
 		})
-	}, nil)
+	})
 
 	title := "NS Address Change: " + testSmallDomain
 	ds, _ := deps.state.GetDomainState(testSmallDomain)
@@ -542,12 +548,12 @@ func TestNSAddedAndRemovedIsNoAddressChange(t *testing.T) {
 	// The saved state lists oldNS1, which live DNS does not, in place of
 	// the first nameserver live DNS lists, so that the check finds that
 	// one added and oldNS1 removed. Only oldNS1 has addresses saved.
-	deps := runChecks(t, cfg, func(deps *testDeps) {
+	_, deps := runChecks(t, cfg, func(deps *testDeps) {
 		deps.state.SetDomainState(testSmallDomain, &state.DomainState{
 			Nameservers:         append([]string{oldNS1}, nameservers[1:]...),
 			NameserverAddresses: map[string][]string{oldNS1: {oldIP}},
 		})
-	}, nil)
+	})
 
 	if n := countNotifications(deps, "NS Change: "+testSmallDomain); n != 1 {
 		t.Errorf("sent %d NS changes, want 1", n)
@@ -565,15 +571,17 @@ func TestRecordChangeDetection(t *testing.T) {
 	cfg := defaultTestConfig(t)
 	cfg.Hostnames = []string{testHost}
 
-	// Between the checks, save for every nameserver an address live DNS
-	// never returns.
-	deps := runChecks(t, cfg, nil, func(deps *testDeps) {
-		hs, _ := deps.state.GetHostnameState(testHost)
-		for _, nsState := range hs.RecordsByNameserver {
-			nsState.Records = map[string][]string{"A": {oldIP}}
+	nameservers := lookupNameservers(t, testHost)
+
+	// The saved state has every nameserver live DNS lists answering
+	// with an address live DNS never returns.
+	_, deps := runChecks(t, cfg, func(deps *testDeps) {
+		byNameserver := make(map[string]*state.NameserverRecordState)
+		for _, ns := range nameservers {
+			byNameserver[ns] = answered(map[string][]string{"A": {oldIP}})
 		}
 
-		deps.state.SetHostnameState(testHost, hs)
+		deps.state.SetHostnameState(testHost, saved(byNameserver))
 	})
 
 	assertNotified(t, deps, "Record Change: "+testHost, "warning")
@@ -585,12 +593,15 @@ func TestPortStateChange(t *testing.T) {
 	cfg := defaultTestConfig(t)
 	cfg.Hostnames = []string{testHost}
 
-	// Between the checks, every port closes.
-	deps := runChecks(t, cfg, nil, func(deps *testDeps) {
-		deps.portChecker.mu.Lock()
-		deps.portChecker.closed = true
-		deps.portChecker.mu.Unlock()
-	})
+	w, deps := runChecks(t, cfg, nil)
+
+	// Every port closes, and the port checks run again. They look
+	// nothing up.
+	deps.portChecker.mu.Lock()
+	deps.portChecker.closed = true
+	deps.portChecker.mu.Unlock()
+
+	w.CheckAllPorts(t.Context())
 
 	hs, _ := deps.state.GetHostnameState(testHost)
 	assertNotified(
@@ -610,7 +621,7 @@ func TestTLSExpiryWarning(t *testing.T) {
 	cfg := defaultTestConfig(t)
 	cfg.Hostnames = []string{testHost}
 
-	deps := runChecks(t, cfg, expiresInThreeDays, nil)
+	_, deps := runChecks(t, cfg, expiresInThreeDays)
 
 	assertNotified(t, deps, "TLS Expiry Warning: "+testHost, "warning")
 }
@@ -782,7 +793,7 @@ func TestDNSRunsBeforePortAndTLSChecks(t *testing.T) {
 	cfg.Hostnames = []string{testHost}
 
 	// The saved state says the last check found testHost at oldIP.
-	deps := runChecks(t, cfg, func(deps *testDeps) {
+	_, deps := runChecks(t, cfg, func(deps *testDeps) {
 		deps.state.SetHostnameState(testHost, &state.HostnameState{
 			RecordsByNameserver: map[string]*state.NameserverRecordState{
 				oldNS1: {
@@ -791,7 +802,7 @@ func TestDNSRunsBeforePortAndTLSChecks(t *testing.T) {
 				},
 			},
 		})
-	}, nil)
+	})
 
 	snap := deps.state.GetSnapshot()
 
@@ -864,18 +875,27 @@ func TestSendTestNotification_ViaRun(t *testing.T) {
 
 	notifications := deps.notifier.getNotifications()
 
+	// No names are configured, so every count is 0.
+	wantMessage := "dnswatcher has started and completed its initial scan.\n" +
+		"Monitoring 0 domain(s) and 0 hostname(s).\n" +
+		"Tracking 0 port endpoint(s) and 0 TLS certificate(s).\n" +
+		"This is a test notification, sent to every configured " +
+		"notification endpoint."
+
 	found := false
 
 	for _, n := range notifications {
 		if n.Priority == "success" &&
-			n.Title == "✅ dnswatcher startup complete" {
+			n.Title == "✅ dnswatcher startup complete" &&
+			n.Message == wantMessage {
 			found = true
 		}
 	}
 
 	if !found {
 		t.Errorf(
-			"expected startup test notification, got: %v",
+			"expected startup test notification with message %q, got: %v",
+			wantMessage,
 			notifications,
 		)
 	}
@@ -922,21 +942,20 @@ func TestNSFailureAndRecovery(t *testing.T) {
 	cfg := defaultTestConfig(t)
 	cfg.Hostnames = []string{testHost}
 
-	// Between the checks, save every nameserver the first check found
-	// as one that did not answer, and add, as answering, one that live
-	// DNS does not list, which then disappears.
-	deps := runChecks(t, cfg, nil, func(deps *testDeps) {
-		hs, _ := deps.state.GetHostnameState(testHost)
-		for ns := range hs.RecordsByNameserver {
-			hs.RecordsByNameserver[ns] = failed()
+	nameservers := lookupNameservers(t, testHost)
+
+	// The saved state has every nameserver live DNS lists as one that
+	// did not answer, and, as answering, one that live DNS does not
+	// list, which then disappears.
+	_, deps := runChecks(t, cfg, func(deps *testDeps) {
+		byNameserver := map[string]*state.NameserverRecordState{
+			oldNS1: answered(map[string][]string{"A": {oldIP}}),
+		}
+		for _, ns := range nameservers {
+			byNameserver[ns] = failed()
 		}
 
-		hs.RecordsByNameserver[oldNS1] = &state.NameserverRecordState{
-			Records: map[string][]string{"A": {oldIP}},
-			Status:  "ok",
-		}
-
-		deps.state.SetHostnameState(testHost, hs)
+		deps.state.SetHostnameState(testHost, saved(byNameserver))
 	})
 
 	assertNotified(t, deps, "NS Failure: "+testHost, "error")