watcher: change messages name only the record types that differ (closes #219)

A Record Change notification printed the nameserver's whole old and new
record sets in Go map syntax, and an Inconsistency notification the two
nameservers' whole sets, so a one-address change had to be found by eye
among kilobytes of unchanged TXT, CAA, MX and NS values. Both now list,
in sorted order of type, only the record types whose values differ: a
line naming the type, then each side's values separated by commas, or
none when that side has no records of that type. The dashboard's Recent
alerts shows the same text.

Model: opus-5-5

README.md

@@ -121,14 +121,26 @@ notification endpoint set, changes show only on the dashboard; see
       failed on it, and answers differently is reported on the check where it
       answers. If a pair agrees again and later disagrees, the alert is sent
       again.
+    - **CNAME address change**: The addresses at the end of a name's CNAME chain
+      differ from those of the previous check. They are found when its
+      nameservers answer with a CNAME and no address; a name that answers with
+      an address has none. A change from or to no addresses is sent too, as when
+      a name moves between A records and a CNAME. Nothing is sent when the
+      previous addresses were kept because a chain could not be followed or none
+      of the name's nameservers answered. The first check after loading a state
+      file without `cnameAddresses` sends nothing: it saves the addresses it
+      finds for the next check to compare.
 
 ### TCP Port Monitoring
 
 - For every configured domain and hostname, constructs a deduplicated list of
   the IPv4 and IPv6 addresses in the A and AAAA records its authoritative
-  nameservers returned. A CNAME is not followed: a name whose CNAME points into
-  another zone usually has no addresses here, so its ports and certificate are
-  not checked.
+  nameservers returned. When they returned a CNAME and no address, the CNAME
+  chain is followed and the addresses at its end are used, and a change in those
+  is notified as a CNAME address change. When the nameservers gave different
+  CNAME targets, each is followed and the addresses of all are used. When a
+  chain cannot be followed, or none of the name's nameservers answered, the
+  addresses the last check found at its end are used.
 - Checks TCP connectivity on ports **80** and **443** for each IP address.
 - Every **1 hour** by default, re-checks all ports.
 - Any change in port availability triggers a notification:
@@ -176,6 +188,8 @@ includes:
 - **DNS NS changes**: Which domain, which nameservers were added/removed.
 - **NS address changes**: Which domain, which nameserver, its old and new
   addresses.
+- **CNAME address changes**: Which hostname, the old and new addresses at the
+  end of its CNAME chain.
 - **NS query failures**: Which nameserver failed, error type (timeout, SERVFAIL,
   REFUSED, network error), which hostname/domain affected.
 - **NS recoveries**: Which nameserver recovered, which hostname/domain.
@@ -420,9 +434,11 @@ This approach ensures:
 - Ability to detect split-horizon or inconsistent responses across authoritative
   servers.
 
-CNAME chains are followed (with a depth limit to prevent loops) only to find the
-addresses of nameservers. A watched name's records are stored as its nameservers
-return them, CNAME included, without following it.
+A watched name's records are stored as its nameservers return them, CNAME
+included. When they return a CNAME and no address, the chain of every CNAME
+target they gave is followed (with a depth limit to prevent loops) to the A and
+AAAA records at its end, and the port and TLS checks use those addresses.
+Nameservers' addresses are also found by following CNAME chains.
 
 Sending a notification or a Sentry report is the one use of the system's
 resolver: the HTTP client looks up the webhook's or Sentry's host name with it.
@@ -469,6 +485,7 @@ merged view, to enable inconsistency detection.
                     "lastChecked": "2026-02-19T12:00:00Z"
                 }
             },
+            "cnameAddresses": [],
             "lastChecked": "2026-02-19T12:00:00Z"
         }
     },
@@ -515,6 +532,14 @@ certificate entry whose TLS connection or handshake failed likewise has status
 resolves to. A state file without it loads, and the next check fills it in
 without a notification.
 
+`cnameAddresses` lists the sorted addresses at the end of the chain of every
+CNAME target a hostname's nameservers gave, found when they answered with a
+CNAME and no address; it is empty when they answered with an address. When a
+chain cannot be followed, or none of the name's nameservers answered, the
+previous check's list is kept, or `null` when no earlier check saved one. A
+state file without it loads, and the first check after that saves it without a
+notification.
+
 A port entry in the older format, with one `hostname` instead of the `hostnames`
 list, loads as a list of that one name.
 
@@ -672,7 +697,9 @@ docker run -d \
     - Port and TLS checks use the IP addresses found by the DNS phase that
       immediately precedes them. When that phase cannot find a name's
       nameservers at all, the addresses an earlier check saved for the name are
-      used.
+      used. When it cannot follow a name's CNAME chain, or none of the name's
+      nameservers answered, the addresses an earlier check found at the end of
+      the chain are used.
 4. **On change detection**: Send notifications to all configured endpoints,
    update in-memory state, persist to disk.
 5. **Shutdown**: The watcher stops checking and saves the final state to disk,

TODO.md

@@ -19,6 +19,16 @@ trial run of the finished image: https://git.eeqj.de/sneak/dnswatcher/issues/149
 
 # Completed Steps
 
+- 2026-10-02: Record Change and Inconsistency notifications list only the record
+  types that differ, each with its values as plain text (closes #219).
+- 2026-10-02: the startup notification no longer says every notification
+  endpoint works; it says it is a test sent to each of them (closes #230).
+- 2026-10-02: a Mattermost webhook that answers an HTTP error is logged as
+  `mattermost notification failed`, not as a Slack failure (closes #227).
+- 2026-10-02: durations in the log are written as text such as `2m0s`, not as a
+  bare count of nanoseconds (closes #228).
+- 2026-10-02: a watched name whose nameservers answer with a CNAME and no
+  address gets port and TLS checks at the end of its CNAME chain (closes #203).
 - 2026-10-02: a resolver test that reads one record type from a nameserver's
   answer asks again when that type is missing from it (closes #218).
 - 2026-10-02: a plain `docker build .` of a clone stamps its tag or short

internal/notify/delivery_test.go

@@ -6,9 +6,11 @@ import (
 	"encoding/json"
 	"errors"
 	"io"
+	"maps"
 	"net/http"
 	"net/http/httptest"
 	"net/url"
+	"strings"
 	"sync"
 	"testing"
 	"time"
@@ -413,7 +415,8 @@ func sendSlackInfo(
 	svc *notify.Service, target *url.URL,
 ) error {
 	return svc.SendSlack(
-		context.Background(), target, "t", "m", prioInfo,
+		context.Background(), target, notify.ErrSlackFailed,
+		"t", "m", prioInfo,
 	)
 }
 
@@ -506,6 +509,7 @@ func TestSendSlackPayloadFields(t *testing.T) {
 	err := svc.SendSlack(
 		context.Background(),
 		webhookURL,
+		notify.ErrSlackFailed,
 		"Alert Title",
 		"Alert body text",
 		"warning",
@@ -608,7 +612,8 @@ func TestSendSlackAllColors(t *testing.T) {
 
 			err := svc.SendSlack(
 				context.Background(),
-				webhookURL, "t", "m", tc.priority,
+				webhookURL, notify.ErrSlackFailed,
+				"t", "m", tc.priority,
 			)
 			if err != nil {
 				t.Fatalf("SendSlack error: %v", err)
@@ -659,7 +664,8 @@ func TestSendSlackNetworkError(t *testing.T) {
 	)
 
 	err := svc.SendSlack(
-		context.Background(), webhookURL, "t", "m", "info",
+		context.Background(), webhookURL, notify.ErrSlackFailed,
+		"t", "m", "info",
 	)
 	if err == nil {
 		t.Fatal("expected error for network failure")
@@ -1028,6 +1034,66 @@ func TestSendNotificationMattermostError(t *testing.T) {
 	)
 }
 
+// TestSendNotificationErrorNamesEndpoint verifies that, with both
+// Slack and Mattermost set, a failed delivery's logged error names
+// the endpoint that failed. Both are sent by the Slack sender.
+func TestSendNotificationErrorNamesEndpoint(t *testing.T) {
+	t.Parallel()
+
+	srv := httptest.NewServer(
+		http.HandlerFunc(
+			func(w http.ResponseWriter, _ *http.Request) {
+				w.WriteHeader(http.StatusServiceUnavailable)
+			}),
+	)
+	defer srv.Close()
+
+	target, _ := url.Parse(srv.URL)
+
+	svc, logs := newLoggingService(http.DefaultTransport)
+	svc.SetSlackWebhookURL(target)
+	svc.SetMattermostWebhookURL(target)
+	svc.SetSleepFunc(instantSleep)
+	svc.SetRetryConfig(notify.RetryConfig{
+		MaxRetries: 1,
+		BaseDelay:  time.Millisecond,
+		MaxDelay:   time.Millisecond,
+	})
+
+	svc.SendNotification(
+		context.Background(), "t", "m", prioError,
+	)
+
+	waitForCondition(t, func() bool {
+		return svc.OutstandingDeliveries() == 0
+	})
+
+	got := map[string]string{}
+
+	for line := range strings.Lines(logs.String()) {
+		var record struct {
+			Msg      string `json:"msg"`
+			Endpoint string `json:"endpoint"`
+			Error    string `json:"error"`
+		}
+
+		_ = json.Unmarshal([]byte(line), &record)
+
+		if record.Msg == "failed to send notification after retries" {
+			got[record.Endpoint] = record.Error
+		}
+	}
+
+	want := map[string]string{
+		"slack":      "slack notification failed: status 503",
+		"mattermost": "mattermost notification failed: status 503",
+	}
+
+	if !maps.Equal(got, want) {
+		t.Errorf("logged errors = %v, want %v", got, want)
+	}
+}
+
 // ── SlackPayload JSON marshaling ──────────────────────────
 
 func TestSlackPayloadJSON(t *testing.T) {

internal/notify/export_test.go

@@ -85,10 +85,11 @@ func (svc *Service) SendNtfy(
 func (svc *Service) SendSlack(
 	ctx context.Context,
 	webhookURL *url.URL,
+	failed error,
 	title, message, priority string,
 ) error {
 	return svc.sendSlack(
-		ctx, webhookURL, title, message, priority,
+		ctx, webhookURL, failed, title, message, priority,
 	)
 }
 

internal/notify/notify.go

@@ -277,7 +277,8 @@ func (svc *Service) dispatchSlack(
 
 	svc.dispatch(ctx, "slack", func(c context.Context) error {
 		return svc.sendSlack(
-			c, svc.slackWebhookURL, title, message, priority,
+			c, svc.slackWebhookURL, ErrSlackFailed,
+			title, message, priority,
 		)
 	})
 }
@@ -294,7 +295,7 @@ func (svc *Service) dispatchMattermost(
 		ctx, "mattermost",
 		func(c context.Context) error {
 			return svc.sendSlack(
-				c, svc.mattermostWebhookURL,
+				c, svc.mattermostWebhookURL, ErrMattermostFailed,
 				title, message, priority,
 			)
 		},
@@ -370,9 +371,14 @@ type SlackAttachment struct {
 	Text  string `json:"text"`
 }
 
+// sendSlack posts to a Slack or Mattermost incoming webhook, which
+// take the same payload. An HTTP error status is returned wrapped in
+// failed, ErrSlackFailed or ErrMattermostFailed, so the error names
+// the endpoint.
 func (svc *Service) sendSlack(
 	ctx context.Context,
 	webhookURL *url.URL,
+	failed error,
 	title, message, priority string,
 ) error {
 	ctx, cancel := context.WithTimeout(
@@ -420,7 +426,7 @@ func (svc *Service) sendSlack(
 	if resp.StatusCode >= httpStatusClientError {
 		return fmt.Errorf(
 			"%w: status %d",
-			ErrSlackFailed, resp.StatusCode,
+			failed, resp.StatusCode,
 		)
 	}
 

internal/notify/retry.go

@@ -115,7 +115,9 @@ func (svc *Service) deliverWithRetry(
 			"endpoint", endpoint,
 			"attempt", attempt+1,
 			"maxAttempts", cfg.MaxRetries+1,
-			"retryIn", delay,
+			// As text: the JSON log writes a time.Duration as
+			// bare nanoseconds.
+			"retryIn", delay.String(),
 			"error", lastErr,
 		)
 

internal/notify/retry_test.go

@@ -2,6 +2,7 @@ package notify_test
 
 import (
 	"context"
+	"encoding/json"
 	"errors"
 	"net/http"
 	"net/http/httptest"
@@ -189,6 +190,50 @@ func TestDeliverWithRetryExhaustsAttempts(t *testing.T) {
 	}
 }
 
+// TestDeliverWithRetryLogsRetryInAsText checks that the wait
+// before a retry is logged as text such as "1.02s", not as a
+// count of nanoseconds.
+func TestDeliverWithRetryLogsRetryInAsText(t *testing.T) {
+	t.Parallel()
+
+	svc, logs := newLoggingService(http.DefaultTransport)
+	svc.SetRetryConfig(notify.RetryConfig{
+		MaxRetries: 1,
+		BaseDelay:  time.Second,
+		MaxDelay:   time.Second,
+	})
+
+	var waited time.Duration
+
+	svc.SetSleepFunc(func(d time.Duration) <-chan time.Time {
+		waited = d
+
+		return instantSleep(d)
+	})
+
+	_ = svc.DeliverWithRetry(
+		context.Background(), "test",
+		func(_ context.Context) error {
+			return errFail
+		},
+	)
+
+	// With one retry, only the first failure is logged.
+	var record map[string]any
+
+	err := json.Unmarshal([]byte(logs.String()), &record)
+	if err != nil {
+		t.Fatalf("log is not one JSON record: %v\n%s", err, logs)
+	}
+
+	if record["retryIn"] != waited.String() {
+		t.Errorf(
+			"retryIn logged as %v, want %q",
+			record["retryIn"], waited.String(),
+		)
+	}
+}
+
 func TestDeliverWithRetryRespectsContextCancellation(
 	t *testing.T,
 ) {

internal/portcheck/portcheck.go

@@ -193,7 +193,9 @@ func (c *Checker) checkConnection(
 	c.log.Debug(
 		"port check succeeded",
 		"target", target,
-		"latency", latency,
+		// As text: the JSON log writes a time.Duration as bare
+		// nanoseconds.
+		"latency", latency.String(),
 	)
 
 	return &PortResult{

internal/state/state.go

@@ -53,8 +53,13 @@ type NameserverRecordState struct {
 }
 
 // HostnameState holds per-nameserver monitoring state for a hostname.
+// CNAMEAddresses holds the sorted addresses at the end of the name's
+// CNAME chain, found when its nameservers answered with a CNAME and no
+// address; it is empty otherwise. It is nil when they are not known: a
+// state file written before it existed loads with it nil.
 type HostnameState struct {
 	RecordsByNameserver map[string]*NameserverRecordState `json:"recordsByNameserver"`
+	CNAMEAddresses      []string                          `json:"cnameAddresses"`
 	LastChecked         time.Time                         `json:"lastChecked"`
 }
 

internal/state/state_test.go

@@ -188,6 +188,95 @@ func TestLoadStateFromBeforeNameserverAddresses(t *testing.T) {
 	}
 }
 
+// TestSaveLoadRoundTrip_CNAMEAddresses checks that no addresses at the
+// end of a hostname's CNAME chain load as an empty list, and addresses
+// that are not known load as nil: the watcher tells the two apart.
+func TestSaveLoadRoundTrip_CNAMEAddresses(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	s := state.NewForTestWithDataDir(dir)
+
+	want := map[string][]string{
+		"cname.example.com":     {testIP},
+		"none.example.com":      {},
+		"not-known.example.com": nil,
+	}
+
+	for name, addresses := range want {
+		s.SetHostnameState(name, &state.HostnameState{
+			CNAMEAddresses: addresses,
+		})
+	}
+
+	err := s.Save()
+	if err != nil {
+		t.Fatalf("Save() error: %v", err)
+	}
+
+	loaded := state.NewForTestWithDataDir(dir)
+
+	err = loaded.Load()
+	if err != nil {
+		t.Fatalf("Load() error: %v", err)
+	}
+
+	for name, addresses := range want {
+		hs, ok := loaded.GetHostnameState(name)
+		if !ok {
+			t.Fatalf("missing hostname %s", name)
+		}
+
+		if !reflect.DeepEqual(hs.CNAMEAddresses, addresses) {
+			t.Errorf(
+				"%s: loaded %#v, want %#v",
+				name, hs.CNAMEAddresses, addresses,
+			)
+		}
+	}
+}
+
+// TestLoadStateFromBeforeCNAMEAddresses loads a state file written
+// before the addresses at the end of a hostname's CNAME chain were
+// saved. They load as not known (nil), not as none.
+func TestLoadStateFromBeforeCNAMEAddresses(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+
+	data := []byte(`{
+		"version": 1,
+		"lastUpdated": "2026-02-19T12:00:00Z",
+		"hostnames": {
+			"www.example.com": {
+				"recordsByNameserver": {},
+				"lastChecked": "2026-02-19T12:00:00Z"
+			}
+		}
+	}`)
+
+	err := os.WriteFile(filepath.Join(dir, "state.json"), data, 0o600)
+	if err != nil {
+		t.Fatalf("writing state file: %v", err)
+	}
+
+	s := state.NewForTestWithDataDir(dir)
+
+	err = s.Load()
+	if err != nil {
+		t.Fatalf("Load() error: %v", err)
+	}
+
+	hs, ok := s.GetHostnameState(testHostname)
+	if !ok {
+		t.Fatal("missing hostname " + testHostname)
+	}
+
+	if hs.CNAMEAddresses != nil {
+		t.Errorf("CNAME addresses: got %#v, want nil", hs.CNAMEAddresses)
+	}
+}
+
 // TestSaveLoadRoundTrip_Hostnames verifies hostname data survives a save/load cycle.
 func TestSaveLoadRoundTrip_Hostnames(t *testing.T) {
 	t.Parallel()

internal/watcher/cname_test.go

@@ -0,0 +1,336 @@
+package watcher_test
+
+import (
+	"context"
+	"log/slog"
+	"slices"
+	"testing"
+
+	"sneak.berlin/go/dnswatcher/internal/livednstest"
+	"sneak.berlin/go/dnswatcher/internal/resolver"
+	"sneak.berlin/go/dnswatcher/internal/state"
+	"sneak.berlin/go/dnswatcher/internal/watcher"
+)
+
+// TestCNAMEIntoAnotherZonePortAndTLSChecks runs the port and TLS
+// checks on hostname state built here: the name's nameserver answered
+// with a CNAME into another zone, and following it found ip1. Both
+// checks must use ip1. They look nothing up, so the watcher has no
+// resolver.
+func TestCNAMEIntoAnotherZonePortAndTLSChecks(t *testing.T) {
+	t.Parallel()
+
+	cfg := defaultTestConfig(t)
+	cfg.Hostnames = []string{host}
+
+	deps := newTestDeps(t, cfg)
+	w := watcher.NewForTest(
+		cfg, deps.state, nil,
+		deps.portChecker, deps.tlsChecker, deps.notifier,
+	)
+
+	deps.state.SetHostnameState(host, cnameState(ip1))
+
+	w.CheckAllPorts(t.Context())
+	w.RunTLSChecks(t.Context())
+
+	snap := deps.state.GetSnapshot()
+
+	ps, ok := snap.Ports[ip1+":443"]
+	if !ok || !slices.Contains(ps.Hostnames, host) {
+		t.Errorf("no port state for %s at %s:443", host, ip1)
+	}
+
+	certKey := ip1 + ":443:" + host
+	if _, ok := snap.Certificates[certKey]; !ok {
+		t.Errorf("no certificate state %s", certKey)
+	}
+}
+
+// TestCNAMEThatCannotBeFollowedKeepsPrevious runs a check of a name, not
+// the watcher's first, from the point where its records have been looked
+// up: they hold a CNAME to a target under .invalid, whose lookup fails.
+// The previous check found the same records, and oldIP at the end of the
+// CNAME. The check must keep oldIP and send nothing.
+func TestCNAMEThatCannotBeFollowedKeepsPrevious(t *testing.T) {
+	t.Parallel()
+
+	w, deps := newTestWatcher(t, defaultTestConfig(t))
+	w.SetFirstRun(false)
+
+	records := map[string]map[string][]string{
+		nsA: cnameTo("target.example.invalid."),
+	}
+
+	prev := hostnameState(records)
+	prev.CNAMEAddresses = []string{oldIP}
+	deps.state.SetHostnameState(host, prev)
+
+	// The result is the same whether or not live DNS answers, so the
+	// lookup is not retried.
+	_ = livednstest.Run(func(ctx context.Context) error {
+		w.UpdateHostnameState(ctx, host, hostnameState(records))
+
+		return nil
+	})
+
+	hs, _ := deps.state.GetHostnameState(host)
+	if !slices.Equal(hs.CNAMEAddresses, prev.CNAMEAddresses) {
+		t.Errorf(
+			"saved %v, want %v",
+			hs.CNAMEAddresses, prev.CNAMEAddresses,
+		)
+	}
+
+	notifications := deps.notifier.getNotifications()
+	if len(notifications) != 0 {
+		t.Errorf("sent %v, want no notifications", notifications)
+	}
+}
+
+// followLive follows in live DNS the CNAMEs in a name's records, built
+// from records, and returns the addresses saved for the name. The
+// previous check saved oldIP, which is kept when a target cannot be
+// followed; that is retried. The tests point CNAMEs only at names in
+// zones with two nameservers, to keep queries few (see the top of
+// watcher_test.go).
+func followLive(
+	t *testing.T,
+	records map[string]map[string][]string,
+) []string {
+	t.Helper()
+
+	w := watcher.NewForTest(
+		nil, nil, resolver.NewFromLogger(slog.Default()), nil, nil, nil,
+	)
+	prev := cnameState(oldIP)
+
+	var current *state.HostnameState
+
+	livednstest.Retry(t, "following CNAMEs", func(ctx context.Context) error {
+		current = hostnameState(records)
+
+		w.ResolveCNAMEAddresses(ctx, host, current, prev)
+
+		if slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
+			return livednstest.ErrNoAnswer
+		}
+
+		return nil
+	})
+
+	return current.CNAMEAddresses
+}
+
+// TestCNAMEAddressesOfEveryTarget gives a name's two nameservers
+// different CNAME targets, as when a secondary still serves an old one.
+// The addresses at the end of both are saved, whichever answer is read
+// first: one.one.one.one has 1.1.1.1, and dns.adguard-dns.com has
+// 94.140.14.14.
+func TestCNAMEAddressesOfEveryTarget(t *testing.T) {
+	t.Parallel()
+
+	found := followLive(t, map[string]map[string][]string{
+		nsA: cnameTo("one.one.one.one."),
+		nsB: cnameTo("dns.adguard-dns.com."),
+	})
+
+	for _, ip := range []string{"1.1.1.1", "94.140.14.14"} {
+		if !slices.Contains(found, ip) {
+			t.Errorf("saved %v, want %s among them", found, ip)
+		}
+	}
+}
+
+// TestCNAMEChainEndingInNoAddressSavesEmptyList follows a CNAME to a
+// name live DNS answers with NXDOMAIN. An empty list is saved, not nil,
+// which would mean the addresses are not known.
+func TestCNAMEChainEndingInNoAddressSavesEmptyList(t *testing.T) {
+	t.Parallel()
+
+	found := followLive(t, map[string]map[string][]string{
+		nsA: cnameTo("this-surely-does-not-exist-xyz.example.org."),
+	})
+
+	if found == nil || len(found) != 0 {
+		t.Errorf("saved %#v, want an empty list", found)
+	}
+}
+
+// TestCNAMEBesideAnAddressNotFollowed gives one nameserver of a name an
+// address and another a CNAME. The CNAME is not followed: an empty list
+// is saved, not nil, and nothing is looked up, the watcher having no
+// resolver.
+func TestCNAMEBesideAnAddressNotFollowed(t *testing.T) {
+	t.Parallel()
+
+	w := watcher.NewForTest(nil, nil, nil, nil, nil, nil)
+
+	current := hostnameState(map[string]map[string][]string{
+		nsA: {"A": {ip1}},
+		nsB: cnameTo("target.example.org."),
+	})
+
+	w.ResolveCNAMEAddresses(t.Context(), host, current, nil)
+
+	if current.CNAMEAddresses == nil || len(current.CNAMEAddresses) != 0 {
+		t.Errorf("saved %#v, want an empty list", current.CNAMEAddresses)
+	}
+}
+
+// TestCNAMEWhoseNameserversAllFailedKeepsPrevious checks a name none of
+// whose nameservers answered. The addresses the previous check saved
+// from following its CNAME are kept, and nothing is looked up: the
+// watcher has no resolver.
+func TestCNAMEWhoseNameserversAllFailedKeepsPrevious(t *testing.T) {
+	t.Parallel()
+
+	w := watcher.NewForTest(nil, nil, nil, nil, nil, nil)
+
+	current := saved(map[string]*state.NameserverRecordState{
+		nsA: failed(), nsB: failed(),
+	})
+	prev := cnameState(oldIP)
+
+	w.ResolveCNAMEAddresses(t.Context(), host, current, prev)
+
+	if !slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
+		t.Errorf(
+			"saved %v, want %v",
+			current.CNAMEAddresses, prev.CNAMEAddresses,
+		)
+	}
+}
+
+// cnameTo builds the records of a nameserver that answered with a CNAME
+// to target and no address.
+func cnameTo(target string) map[string][]string {
+	return map[string][]string{"CNAME": {target}}
+}
+
+// cnameState builds the state a check leaves behind for a name whose
+// nameserver answered with a CNAME and no address, when following the
+// CNAME found these addresses, which may be none.
+func cnameState(addresses ...string) *state.HostnameState {
+	hs := hostnameState(map[string]map[string][]string{
+		nsA: cnameTo("target.example.org."),
+	})
+
+	hs.CNAMEAddresses = append([]string{}, addresses...)
+
+	return hs
+}
+
+func TestCNAMEAddressChangeAlerts(t *testing.T) {
+	t.Parallel()
+
+	// A state file written before the addresses were saved loads with
+	// them nil.
+	olderStateFile := cnameState()
+	olderStateFile.CNAMEAddresses = nil
+
+	// Each case is the state saved by the previous check and by the
+	// current one. The name's records are the same in both.
+	tests := []struct {
+		name          string
+		prev, current *state.HostnameState
+		want          int
+	}{
+		{
+			"same addresses",
+			cnameState(ip1, ip2), cnameState(ip1, ip2), 0,
+		},
+		{
+			"same addresses in another order",
+			cnameState(ip2, ip1), cnameState(ip1, ip2), 0,
+		},
+		{
+			"address replaced",
+			cnameState(ip1), cnameState(ip2), 1,
+		},
+		{
+			"address added",
+			cnameState(ip1), cnameState(ip1, ip2), 1,
+		},
+		{
+			"no address at the end of the chain now",
+			cnameState(ip1), cnameState(), 1,
+		},
+		{
+			"addresses at the end of the chain again",
+			cnameState(), cnameState(ip1), 1,
+		},
+		{
+			"state file from before addresses were saved",
+			olderStateFile, cnameState(ip1), 0,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			notifier := &mockNotifier{}
+			w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
+
+			w.DetectHostnameChanges(t.Context(), host, tt.prev, tt.current)
+
+			got := len(notifier.getNotifications())
+			if got != tt.want {
+				t.Errorf("sent %d notifications, want %d", got, tt.want)
+			}
+		})
+	}
+}
+
+func TestCNAMEAddressChangeAlertNamesHostnameAndAddresses(t *testing.T) {
+	t.Parallel()
+
+	notifier := &mockNotifier{}
+	w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
+
+	w.DetectHostnameChanges(
+		t.Context(), host, cnameState(ip1), cnameState(ip2, ip3),
+	)
+
+	want := notification{
+		Title: "CNAME Address Change: " + host,
+		Message: "Hostname: " + host +
+			"\nOld: " + ip1 + "\nNew: " + ip2 + ", " + ip3,
+		Priority: "warning",
+	}
+
+	got := notifier.getNotifications()
+	if len(got) != 1 || got[0] != want {
+		t.Errorf("sent %v, want %v", got, want)
+	}
+}
+
+// TestNameMovedFromARecordsToCNAMEAlerts checks a name that answers
+// with an A record and then with a CNAME whose chain ends in ip2. The
+// second check is notified as a CNAME address change from no addresses,
+// beside the record change. Nothing is looked up: the watcher has no
+// resolver.
+func TestNameMovedFromARecordsToCNAMEAlerts(t *testing.T) {
+	t.Parallel()
+
+	notifier := &mockNotifier{}
+	w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
+
+	prev := hostnameState(map[string]map[string][]string{
+		nsA: {"A": {ip1}},
+	})
+	w.ResolveCNAMEAddresses(t.Context(), host, prev, nil)
+
+	w.DetectHostnameChanges(t.Context(), host, prev, cnameState(ip2))
+
+	title := "CNAME Address Change: " + host
+	message := "Hostname: " + host + "\nOld: \nNew: " + ip2
+
+	got := notifier.getNotifications()
+	if !slices.ContainsFunc(got, func(n notification) bool {
+		return n.Title == title && n.Message == message
+	}) {
+		t.Errorf("sent %v, want %q with %q among them", got, title, message)
+	}
+}

internal/watcher/export_test.go

@@ -38,6 +38,21 @@ func NewlyDisagreeingPairs(
 	return newlyDisagreeingPairs(prev, current)
 }
 
+// SetFirstRun sets whether the watcher is on its first check, in which
+// nothing is compared with the previous check. NewForTest's watcher is.
+func (w *Watcher) SetFirstRun(firstRun bool) {
+	w.firstRun = firstRun
+}
+
+// UpdateHostnameState exports updateHostnameState for testing.
+func (w *Watcher) UpdateHostnameState(
+	ctx context.Context,
+	hostname string,
+	newState *state.HostnameState,
+) {
+	w.updateHostnameState(ctx, hostname, newState)
+}
+
 // DetectHostnameChanges exports detectHostnameChanges for testing.
 func (w *Watcher) DetectHostnameChanges(
 	ctx context.Context,
@@ -57,6 +72,15 @@ func (w *Watcher) ResolveNameserverAddresses(
 	return w.resolveNameserverAddresses(ctx, nameservers, prev)
 }
 
+// ResolveCNAMEAddresses exports resolveCNAMEAddresses for testing.
+func (w *Watcher) ResolveCNAMEAddresses(
+	ctx context.Context,
+	hostname string,
+	current, prev *state.HostnameState,
+) {
+	w.resolveCNAMEAddresses(ctx, hostname, current, prev)
+}
+
 // DetectNSAddressChanges exports detectNSAddressChanges for testing.
 func (w *Watcher) DetectNSAddressChanges(
 	ctx context.Context,

internal/watcher/message_test.go

@@ -0,0 +1,66 @@
+package watcher_test
+
+import (
+	"testing"
+
+	"sneak.berlin/go/dnswatcher/internal/watcher"
+)
+
+// When one nameserver's A record changes and its TXT record does not,
+// the record change and the inconsistency it starts name the A record
+// alone, with its values written as plain text.
+func TestChangeMessagesNameTheChangedType(t *testing.T) {
+	t.Parallel()
+
+	// A nameserver's records: this A address and the same TXT record.
+	records := func(address string) map[string][]string {
+		return map[string][]string{
+			"A":   {address},
+			"TXT": {"v=spf1 -all"},
+		}
+	}
+
+	before := hostnameState(map[string]map[string][]string{
+		nsA: records(ip1),
+		nsB: records(ip1),
+	})
+	after := hostnameState(map[string]map[string][]string{
+		nsA: records(ip1),
+		nsB: records(ip2),
+	})
+
+	// The hostname change detection uses only the notifier.
+	notifier := &mockNotifier{}
+	w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
+
+	w.DetectHostnameChanges(t.Context(), host, before, after)
+
+	want := map[string]string{
+		"Record Change: " + host: `Hostname: www.example.net
+Nameserver: b.ns.example.net.
+Type: A
+Old: 192.0.2.1
+New: 192.0.2.2`,
+		"Inconsistency: " + host: `Hostname: www.example.net
+Type: A
+a.ns.example.net.: 192.0.2.1
+b.ns.example.net.: 192.0.2.2`,
+	}
+
+	notifications := notifier.getNotifications()
+	if len(notifications) != len(want) {
+		t.Fatalf(
+			"sent %d notifications, want %d: %v",
+			len(notifications), len(want), notifications,
+		)
+	}
+
+	for _, n := range notifications {
+		if n.Message != want[n.Title] {
+			t.Errorf(
+				"%s message:\n%s\nwant:\n%s",
+				n.Title, n.Message, want[n.Title],
+			)
+		}
+	}
+}

internal/watcher/watcher.go

@@ -123,8 +123,10 @@ func (w *Watcher) Run(ctx context.Context) {
 		"watcher starting",
 		"domains", len(w.config.Domains),
 		"hostnames", len(w.config.Hostnames),
-		"dnsInterval", w.config.DNSInterval,
-		"tlsInterval", w.config.TLSInterval,
+		// As text: the JSON log writes a time.Duration as bare
+		// nanoseconds.
+		"dnsInterval", w.config.DNSInterval.String(),
+		"tlsInterval", w.config.TLSInterval.String(),
 	)
 
 	w.RunOnce(ctx)
@@ -252,28 +254,9 @@ func (w *Watcher) checkDomain(
 		LastChecked:         now,
 	})
 
-	// Also look up A/AAAA records for the apex domain so that
-	// port and TLS checks (which read HostnameState) can find
-	// the domain's IP addresses.
-	results, err := w.resolver.LookupAllRecords(ctx, domain)
-	if err != nil {
-		w.log.Error(
-			"failed to lookup records for domain",
-			"domain", domain,
-			"error", err,
-		)
-
-		return
-	}
-
-	newState := buildHostnameState(results, now)
-
-	prevHS, hasPrevHS := w.state.GetHostnameState(domain)
-	if hasPrevHS && !w.firstRun {
-		w.detectHostnameChanges(ctx, domain, prevHS, newState)
-	}
-
-	w.state.SetHostnameState(domain, newState)
+	// The apex domain's records are also checked as a hostname's, so
+	// that the port and TLS checks find its addresses.
+	w.checkHostname(ctx, domain)
 }
 
 func (w *Watcher) detectNSChanges(
@@ -398,9 +381,23 @@ func (w *Watcher) checkHostname(
 		return
 	}
 
-	newState := buildHostnameState(results, time.Now().UTC())
+	w.updateHostnameState(
+		ctx, hostname, buildHostnameState(results, time.Now().UTC()),
+	)
+}
 
+// updateHostnameState finishes a check of hostname from newState, built
+// from its nameservers' answers: it follows the CNAME in them, notifies
+// what changed since the previous check, and saves newState.
+func (w *Watcher) updateHostnameState(
+	ctx context.Context,
+	hostname string,
+	newState *state.HostnameState,
+) {
 	prev, hasPrev := w.state.GetHostnameState(hostname)
+
+	w.resolveCNAMEAddresses(ctx, hostname, newState, prev)
+
 	if hasPrev && !w.firstRun {
 		w.detectHostnameChanges(ctx, hostname, prev, newState)
 	}
@@ -408,6 +405,76 @@ func (w *Watcher) checkHostname(
 	w.state.SetHostnameState(hostname, newState)
 }
 
+// resolveCNAMEAddresses saves in current the addresses at the end of
+// hostname's CNAME chain, when the nameservers' answers in current hold
+// a CNAME and no address, and an empty list otherwise. Every CNAME
+// target the nameservers gave is followed with ResolveIPAddresses and
+// the addresses found for all of them are saved, so nameservers that
+// disagree on the target do not change the result from check to check.
+// The addresses saved in prev, which may be nil, are kept when none of
+// the name's nameservers answered, and when a target cannot be
+// followed, as when no nameserver of a zone in its chain answers.
+func (w *Watcher) resolveCNAMEAddresses(
+	ctx context.Context,
+	hostname string,
+	current, prev *state.HostnameState,
+) {
+	var prevAddresses []string
+	if prev != nil {
+		prevAddresses = prev.CNAMEAddresses
+	}
+
+	// Empty, not nil: nil means the addresses are not known.
+	current.CNAMEAddresses = []string{}
+
+	answered := false
+	targets := make(map[string]bool)
+
+	for _, nsState := range current.RecordsByNameserver {
+		if nsState.Status != statusOK {
+			continue
+		}
+
+		answered = true
+
+		if len(nsState.Records["A"]) > 0 || len(nsState.Records["AAAA"]) > 0 {
+			return
+		}
+
+		for _, target := range nsState.Records["CNAME"] {
+			targets[target] = true
+		}
+	}
+
+	if !answered {
+		current.CNAMEAddresses = prevAddresses
+
+		return
+	}
+
+	for target := range targets {
+		ips, err := w.resolver.ResolveIPAddresses(ctx, target)
+		if err != nil {
+			w.log.Error(
+				"failed to follow CNAME",
+				"hostname", hostname,
+				"target", target,
+				"error", err,
+			)
+
+			current.CNAMEAddresses = prevAddresses
+
+			return
+		}
+
+		current.CNAMEAddresses = append(current.CNAMEAddresses, ips...)
+	}
+
+	// Still the empty list when every chain ends in no address.
+	slices.Sort(current.CNAMEAddresses)
+	current.CNAMEAddresses = slices.Compact(current.CNAMEAddresses)
+}
+
 // buildHostnameState saves each nameserver's response. A nameserver
 // that answered, even with NXDOMAIN or no records, is saved as ok; one
 // that timed out or failed is saved as error with the reason, and its
@@ -451,6 +518,37 @@ func (w *Watcher) detectHostnameChanges(
 	w.detectNSDisappearances(ctx, hostname, prev, current)
 	w.detectNSFailures(ctx, hostname, prev, current)
 	w.detectInconsistencies(ctx, hostname, prev, current)
+	w.detectCNAMEAddressChanges(ctx, hostname, prev, current)
+}
+
+// detectCNAMEAddressChanges notifies when the addresses at the end of
+// hostname's CNAME chain differ from those the previous check saved,
+// including a change from or to none. When the previous addresses are
+// not known (nil), as on the first check after loading a state file
+// written before they were saved, nothing is compared.
+func (w *Watcher) detectCNAMEAddressChanges(
+	ctx context.Context,
+	hostname string,
+	prev, current *state.HostnameState,
+) {
+	old, cur := prev.CNAMEAddresses, current.CNAMEAddresses
+	if old == nil || sliceEqual(old, cur) {
+		return
+	}
+
+	msg := fmt.Sprintf(
+		"Hostname: %s\nOld: %s\nNew: %s",
+		hostname,
+		strings.Join(old, ", "),
+		strings.Join(cur, ", "),
+	)
+
+	w.notify.SendNotification(
+		ctx,
+		"CNAME Address Change: "+hostname,
+		msg,
+		"warning",
+	)
 }
 
 // detectRecordChanges compares each nameserver's records with those of
@@ -472,10 +570,12 @@ func (w *Watcher) detectRecordChanges(
 		}
 
 		msg := fmt.Sprintf(
-			"Hostname: %s\nNameserver: %s\n"+
-				"Old: %v\nNew: %v",
+			"Hostname: %s\nNameserver: %s\n%s",
 			hostname, ns,
-			prevNS.Records, cur.Records,
+			recordDifferences(
+				"Old", prevNS.Records,
+				"New", cur.Records,
+			),
 		)
 
 		w.notify.SendNotification(
@@ -563,10 +663,12 @@ func (w *Watcher) detectInconsistencies(
 		ns1, ns2 := pair[0], pair[1]
 
 		msg := fmt.Sprintf(
-			"Hostname: %s\n%s: %v\n%s: %v",
+			"Hostname: %s\n%s",
 			hostname,
-			ns1, current.RecordsByNameserver[ns1].Records,
-			ns2, current.RecordsByNameserver[ns2].Records,
+			recordDifferences(
+				ns1, current.RecordsByNameserver[ns1].Records,
+				ns2, current.RecordsByNameserver[ns2].Records,
+			),
 		)
 
 		w.notify.SendNotification(
@@ -747,6 +849,9 @@ func (w *Watcher) noNameserverAnswered(name string) bool {
 	return true
 }
 
+// collectIPs returns the addresses saved for hostname: those in its
+// nameservers' A and AAAA records, and those at the end of its CNAME
+// chain.
 func (w *Watcher) collectIPs(hostname string) []string {
 	hs, ok := w.state.GetHostnameState(hostname)
 	if !ok {
@@ -765,6 +870,10 @@ func (w *Watcher) collectIPs(hostname string) []string {
 		}
 	}
 
+	for _, ip := range hs.CNAMEAddresses {
+		ipSet[ip] = true
+	}
+
 	result := make([]string, 0, len(ipSet))
 	for ip := range ipSet {
 		result = append(result, ip)
@@ -1049,8 +1158,9 @@ func (w *Watcher) saveState() {
 
 // maybeSendTestNotification sends a startup status notification
 // after the first full scan completes, if SEND_TEST_NOTIFICATION
-// is enabled. The message is clearly informational ("all ok")
-// and not an error or anomaly alert.
+// is enabled. The message is informational, not an error or anomaly
+// alert. It is written before it reaches any endpoint, so it claims
+// nothing about whether the endpoints work.
 func (w *Watcher) maybeSendTestNotification(ctx context.Context) {
 	if !w.config.SendTestNotification {
 		return
@@ -1062,7 +1172,8 @@ func (w *Watcher) maybeSendTestNotification(ctx context.Context) {
 		"dnswatcher has started and completed its initial scan.\n"+
 			"Monitoring %d domain(s) and %d hostname(s).\n"+
 			"Tracking %d port endpoint(s) and %d TLS certificate(s).\n"+
-			"All notification channels are working.",
+			"This is a test notification, sent to every configured "+
+			"notification endpoint.",
 		len(snap.Domains),
 		len(snap.Hostnames),
 		len(snap.Ports),
@@ -1107,6 +1218,54 @@ func recordsEqual(
 	return true
 }
 
+// recordDifferences describes, in sorted order of type, each record
+// type whose values differ between a and b: a line naming the type,
+// then a line with a's values after labelA and one with b's after
+// labelB. Types with the same values in both are left out.
+func recordDifferences(
+	labelA string, a map[string][]string,
+	labelB string, b map[string][]string,
+) string {
+	types := make([]string, 0, len(a)+len(b))
+
+	for recordType := range a {
+		types = append(types, recordType)
+	}
+
+	for recordType := range b {
+		if _, ok := a[recordType]; !ok {
+			types = append(types, recordType)
+		}
+	}
+
+	sort.Strings(types)
+
+	var lines []string
+
+	for _, recordType := range types {
+		if sliceEqual(a[recordType], b[recordType]) {
+			continue
+		}
+
+		lines = append(lines,
+			"Type: "+recordType,
+			labelA+": "+joinValues(a[recordType]),
+			labelB+": "+joinValues(b[recordType]),
+		)
+	}
+
+	return strings.Join(lines, "\n")
+}
+
+// joinValues lists record values separated by commas, or says none.
+func joinValues(values []string) string {
+	if len(values) == 0 {
+		return "none"
+	}
+
+	return strings.Join(values, ", ")
+}
+
 func sliceEqual(a, b []string) bool {
 	if len(a) != len(b) {
 		return false

internal/watcher/watcher_test.go

@@ -312,7 +312,8 @@ func lookupNameservers(t *testing.T, name string) []string {
 	return nameservers
 }
 
-// addresses returns the A and AAAA values saved for a hostname.
+// addresses returns the A and AAAA values saved for a hostname, and the
+// addresses saved at the end of its CNAME chain.
 func addresses(hs *state.HostnameState) []string {
 	var ips []string
 
@@ -321,7 +322,7 @@ func addresses(hs *state.HostnameState) []string {
 		ips = append(ips, nsState.Records["AAAA"]...)
 	}
 
-	return ips
+	return append(ips, hs.CNAMEAddresses...)
 }
 
 // assertNotified checks that a notification with this title and
@@ -371,6 +372,14 @@ func TestFirstRunBaseline(t *testing.T) {
 
 	assertNoNotifications(t, deps)
 	assertStatePopulated(t, deps)
+
+	// testHost answers with an address, so the check saves an empty list
+	// of CNAME addresses for it; nil would mean the check did not look
+	// at whether to follow a CNAME.
+	hs, _ := deps.state.GetHostnameState(testHost)
+	if hs.CNAMEAddresses == nil || len(hs.CNAMEAddresses) != 0 {
+		t.Errorf("saved CNAME addresses %#v, want []", hs.CNAMEAddresses)
+	}
 }
 
 func assertNoNotifications(
@@ -866,18 +875,27 @@ func TestSendTestNotification_ViaRun(t *testing.T) {
 
 	notifications := deps.notifier.getNotifications()
 
+	// No names are configured, so every count is 0.
+	wantMessage := "dnswatcher has started and completed its initial scan.\n" +
+		"Monitoring 0 domain(s) and 0 hostname(s).\n" +
+		"Tracking 0 port endpoint(s) and 0 TLS certificate(s).\n" +
+		"This is a test notification, sent to every configured " +
+		"notification endpoint."
+
 	found := false
 
 	for _, n := range notifications {
 		if n.Priority == "success" &&
-			n.Title == "✅ dnswatcher startup complete" {
+			n.Title == "✅ dnswatcher startup complete" &&
+			n.Message == wantMessage {
 			found = true
 		}
 	}
 
 	if !found {
 		t.Errorf(
-			"expected startup test notification, got: %v",
+			"expected startup test notification with message %q, got: %v",
+			wantMessage,
 			notifications,
 		)
 	}