watcher: notify nameserver address changes (closes #105)

Each domain check now looks up the addresses every nameserver's name
resolves to, with the resolver's ResolveIPAddresses, and saves them
sorted in the domain's state. A nameserver that stays in the
delegation and resolves to different addresses sends one NS Address
Change notification naming the domain, the nameserver and the old and
new addresses. Added or removed nameservers get only the NS change
notification. A failed or empty lookup keeps the previous addresses,
because the resolver returns no address without an error when every
server it asks times out. State files without the field load, and the
next check fills it in silently. Watcher tests that run domain checks
use example.com, which has two nameservers, to stay within the
per-attempt limit.

Model: opus-5-5

README.md

@@ -46,10 +46,15 @@ rejected.
 - Every **1 hour**, performs a full iterative trace from root servers to
   discover all authoritative nameservers (NS records) for each domain.
 - Queries **every** discovered authoritative nameserver independently.
-- Stores the NS record set as observed by the delegation chain.
+- Stores the NS record set as observed by the delegation chain, and the
+  IPv4 and IPv6 addresses each nameserver's name resolves to.
 - Any change triggers a notification:
   - NS added to or removed from the delegation.
-  - NS IP address changed (glue record change).
+  - NS address change: a nameserver that stays in the delegation
+    resolves to different addresses than on the previous check. A
+    nameserver added or removed gets only the NS change notification.
+    When the lookup of a nameserver's addresses fails or finds none,
+    its previous addresses are kept and nothing is sent.
 
 ### DNS Hostname Monitoring (Subdomains)
 
@@ -139,6 +144,8 @@ includes:
 - **DNS record changes**: Which hostname, which nameserver, what record
   type, old values, new values.
 - **DNS NS changes**: Which domain, which nameservers were added/removed.
+- **NS address changes**: Which domain, which nameserver, its old and
+  new addresses.
 - **NS query failures**: Which nameserver failed, error type (timeout,
   SERVFAIL, REFUSED, network error), which hostname/domain affected.
 - **NS recoveries**: Which nameserver recovered, which hostname/domain.
@@ -359,6 +366,13 @@ either is set to anything else, including a bare number or a zero or negative
 duration, dnswatcher refuses to start with an error naming the variable and
 the value.
 
+**`DNSWATCHER_SENTRY_DSN` reports crashes in HTTP requests to Sentry.** When it
+is set, a panic in an HTTP request handler is sent to Sentry, and the request
+still gets a `500 Internal Server Error` answer. Nothing else is sent to Sentry:
+DNS, port and TLS problems are reported as notifications. A value Sentry cannot
+parse stops dnswatcher at startup. At shutdown, reports not yet sent are sent,
+waiting at most 2 seconds.
+
 ### Example `.env`
 
 ```sh
@@ -411,6 +425,10 @@ not as a merged view, to enable inconsistency detection.
   "domains": {
     "example.com": {
       "nameservers": ["ns1.example.com.", "ns2.example.com."],
+      "nameserverAddresses": {
+        "ns1.example.com.": ["192.0.2.53", "2001:db8::53"],
+        "ns2.example.com.": ["198.51.100.53"]
+      },
       "lastChecked": "2026-02-19T12:00:00Z"
     }
   },
@@ -474,6 +492,10 @@ A nameserver that answers NXDOMAIN or with no records has status `ok` and
 empty `records`. A nameserver whose query failed has status `error`, empty
 `records`, and the reason in `error`.
 
+`nameserverAddresses` lists, by nameserver, the sorted addresses its name
+resolves to. A state file without it loads, and the next check fills it in
+without a notification.
+
 ---
 
 ## Entrypoints
@@ -611,9 +633,10 @@ repository's `Dockerfile` and runs it. The app needs:
      from a previous cycle.
 4. **On change detection**: Send notifications to all configured
    endpoints, update in-memory state, persist to disk.
-5. **Shutdown**: Persist final state to disk, wait for in-flight
+5. **Shutdown**: The watcher stops checking and saves the final state
-   notification deliveries to complete, stop gracefully. The wait is
+   to disk, and shutdown waits for that save before it goes on. Then it
-   bounded by the fx shutdown timeout (15s by default): deliveries still
+   waits for in-flight notification deliveries to complete. Both waits
+   share the fx shutdown timeout (15s by default): deliveries still
    retrying against an unreachable endpoint when that expires are
    abandoned, and the number abandoned is logged at warn level rather
    than dropped silently. Notifications generated after shutdown has

TODO.md

@@ -15,10 +15,17 @@ on the 1.0 milestone: https://git.eeqj.de/sneak/dnswatcher/milestone/7
 
 # Next Step
 
-nameserver IP address changes: https://git.eeqj.de/sneak/dnswatcher/issues/105
+trial run of the finished image:
+https://git.eeqj.de/sneak/dnswatcher/issues/149
 
 # Completed Steps
 
+- 2026-10-01: each nameserver's addresses are saved with its domain, and a
+  change while it stays in the delegation is notified (closes #105).
+- 2026-10-01: the watcher saves state when it stops, and shutdown waits for that
+  save, so it no longer relies on the state's own stop hook (closes #114).
+- 2026-10-01: `DNSWATCHER_SENTRY_DSN` reports panics in HTTP handlers to Sentry,
+  and a DSN Sentry cannot parse stops startup (closes #107).
 - 2026-10-01: a port or TLS check that shutdown cuts short saves nothing and
   sends no notification, as a cut-short DNS lookup already did (closes #185).
 - 2026-10-01: the client address from `X-Forwarded-For` is the last entry that
@@ -101,15 +108,10 @@ nameserver IP address changes: https://git.eeqj.de/sneak/dnswatcher/issues/105
 
 # Future Steps
 
-- `DNSWATCHER_SENTRY_DSN` does nothing:
-  https://git.eeqj.de/sneak/dnswatcher/issues/107
-- trial run of the finished image:
-  https://git.eeqj.de/sneak/dnswatcher/issues/149
 - 1.0 readiness: run it with a real config and read the logs:
   https://git.eeqj.de/sneak/dnswatcher/issues/66
 - `goimports` in `make fmt-check`, Markdown formatting:
   https://git.eeqj.de/sneak/dnswatcher/issues/119
-- final state save at shutdown: https://git.eeqj.de/sneak/dnswatcher/issues/114
 - README accuracy sweep: https://git.eeqj.de/sneak/dnswatcher/issues/108
 - README sections required by policy:
   https://git.eeqj.de/sneak/dnswatcher/issues/173

go.mod

@@ -4,6 +4,7 @@ go 1.25.5
 
 require (
 	github.com/99designs/basicauth-go v0.0.0-20230316000542-bf6f9cbbf0f8
+	github.com/getsentry/sentry-go v0.49.0
 	github.com/go-chi/chi/v5 v5.2.5
 	github.com/go-chi/cors v1.2.2
 	github.com/go-chi/httprate v0.16.0
@@ -13,20 +14,20 @@ require (
 	github.com/spf13/viper v1.21.0
 	github.com/stretchr/testify v1.11.1
 	go.uber.org/fx v1.24.0
-	golang.org/x/net v0.50.0
+	golang.org/x/net v0.56.0
-	golang.org/x/sync v0.19.0
+	golang.org/x/sync v0.21.0
 )
 
 require (
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
-	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
 	github.com/fsnotify/fsnotify v1.9.0 // indirect
 	github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
 	github.com/klauspost/cpuid/v2 v2.2.10 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/pelletier/go-toml/v2 v2.2.4 // indirect
-	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
 	github.com/prometheus/client_model v0.6.2 // indirect
 	github.com/prometheus/common v0.66.1 // indirect
 	github.com/prometheus/procfs v0.16.1 // indirect
@@ -42,10 +43,10 @@ require (
 	go.uber.org/zap v1.26.0 // indirect
 	go.yaml.in/yaml/v2 v2.4.2 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
-	golang.org/x/mod v0.32.0 // indirect
+	golang.org/x/mod v0.37.0 // indirect
-	golang.org/x/sys v0.41.0 // indirect
+	golang.org/x/sys v0.46.0 // indirect
-	golang.org/x/text v0.34.0 // indirect
+	golang.org/x/text v0.39.0 // indirect
-	golang.org/x/tools v0.41.0 // indirect
+	golang.org/x/tools v0.47.0 // indirect
 	google.golang.org/protobuf v1.36.8 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )

go.sum

@@ -4,18 +4,22 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
-github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
-github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
 github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
 github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
 github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
+github.com/getsentry/sentry-go v0.49.0 h1:Ehejknu1l023Ub7QoRBVLAI7g3Jnhqku4oWx4B4Sh5s=
+github.com/getsentry/sentry-go v0.49.0/go.mod h1:nuMJAoCfe1u0Bts2ocyNI+TW8HT84vRMqwA5Qq/SKUI=
 github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
 github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
 github.com/go-chi/cors v1.2.2 h1:Jmey33TE+b+rB7fT8MUy1u0I4L+NARQlK6LhzKPSyQE=
 github.com/go-chi/cors v1.2.2/go.mod h1:sSbTewc+6wYHBBCW7ytsFSn836hqM7JxpglAy2Vzc58=
 github.com/go-chi/httprate v0.16.0 h1:8V5DH9j6pSK6UQoBsTpvMyFxycqaKEIToyPKzHJjUa8=
 github.com/go-chi/httprate v0.16.0/go.mod h1:A8lo+qRhk+s9LiuP5saS7XCGDXRXMcrueq0NfIuCa/I=
+github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
+github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
 github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
 github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
@@ -38,8 +42,12 @@ github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
 github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
-github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pingcap/errors v0.11.4 h1:lFuQV/oaUMGcD2tqt+01ROSmJs75VG1ToEOkZIZ4nE4=
-github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/pingcap/errors v0.11.4/go.mod h1:Oi8TUi2kEtXXLMJk9l1cGmz20kV3TaQ0usTwv5KuLY8=
+github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
+github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
+github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
 github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
 github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
@@ -48,8 +56,8 @@ github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9Z
 github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA=
 github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
 github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
-github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
+github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
-github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
+github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
 github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
 github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
 github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
@@ -84,18 +92,18 @@ go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
 go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
 go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
 go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
-golang.org/x/mod v0.32.0 h1:9F4d3PHLljb6x//jOyokMv3eX+YDeepZSEo3mFJy93c=
+golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
-golang.org/x/mod v0.32.0/go.mod h1:SgipZ/3h2Ci89DlEtEXWUk/HteuRin+HHhN+WbNhguU=
+golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
-golang.org/x/net v0.50.0 h1:ucWh9eiCGyDR3vtzso0WMQinm2Dnt8cFMuQa9K33J60=
+golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
-golang.org/x/net v0.50.0/go.mod h1:UgoSli3F/pBgdJBHCTc+tp3gmrU4XswgGRgtnwWTfyM=
+golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
-golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
+golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
-golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
+golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
-golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
+golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
-golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
+golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
-golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
+golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
-golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
+golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
-golang.org/x/tools v0.41.0 h1:a9b8iMweWG+S0OBnlU36rzLp20z1Rp10w+IY2czHTQc=
+golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
-golang.org/x/tools v0.41.0/go.mod h1:XSY6eDqxVNiYgezAVqqCeihT4j1U2CCsqvH3WhQpnlg=
+golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
 google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
 google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

internal/server/export_test.go

@@ -3,6 +3,8 @@ package server
 import (
 	"net/http"
 	"time"
+
+	"github.com/go-chi/chi/v5"
 )
 
 // RequestTimeout exports the handler execution budget applied by
@@ -22,3 +24,15 @@ func SetListenPort(s *Server, port int) {
 func HTTPServerOf(s *Server) *http.Server {
 	return s.httpServer
 }
+
+// EnableSentry runs the Sentry setup that the start hook runs, without
+// starting the HTTP server.
+func EnableSentry(s *Server) error {
+	return s.enableSentry()
+}
+
+// RouterOf returns the router SetupRoutes built, so a test can add a
+// route that panics.
+func RouterOf(s *Server) *chi.Mux {
+	return s.router
+}

internal/server/routes.go

@@ -4,6 +4,7 @@ import (
 	"net/http"
 	"time"
 
+	sentryhttp "github.com/getsentry/sentry-go/http"
 	"github.com/go-chi/chi/v5"
 	chimw "github.com/go-chi/chi/v5/middleware"
 	"github.com/prometheus/client_golang/prometheus/promhttp"
@@ -25,6 +26,16 @@ func (s *Server) SetupRoutes() {
 	s.router.Use(s.mw.Logging())
 	s.router.Use(chimw.Timeout(requestTimeout))
 
+	// Report panics in handlers to Sentry when DNSWATCHER_SENTRY_DSN is
+	// set. Repanic passes each panic on to chimw.Recoverer above, which
+	// still answers the request.
+	if s.sentryEnabled {
+		sentryHandler := sentryhttp.New(sentryhttp.Options{
+			Repanic: true,
+		})
+		s.router.Use(sentryHandler.Handle)
+	}
+
 	// Public, unauthenticated, read-only routes, the only ones
 	// REPO_POLICIES.md allows wildcard CORS on. CORS is middleware of
 	// this whole router, not of a Group, so that it also answers

internal/server/sentry_test.go

@@ -0,0 +1,190 @@
+package server_test
+
+import (
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/getsentry/sentry-go"
+	"github.com/spf13/viper"
+	"go.uber.org/fx"
+
+	"sneak.berlin/go/dnswatcher/internal/server"
+)
+
+// The tests below set env vars and touch the global state of viper and
+// of Sentry, so they cannot use t.Parallel.
+
+// standInDelay is how long the Sentry stand-in takes to answer. It
+// records a report only then, so a report that Shutdown did not wait
+// for has not been recorded yet when Shutdown returns.
+const standInDelay = 100 * time.Millisecond
+
+// sentryStandIn is a local HTTP server in place of Sentry's, so that
+// nothing a test reports leaves the host. It keeps the body of every
+// request it receives.
+type sentryStandIn struct {
+	server *httptest.Server
+	mu     sync.Mutex
+	bodies []string
+}
+
+func newSentryStandIn(t *testing.T) *sentryStandIn {
+	t.Helper()
+
+	standIn := &sentryStandIn{}
+	standIn.server = httptest.NewServer(http.HandlerFunc(
+		func(_ http.ResponseWriter, r *http.Request) {
+			body, err := io.ReadAll(r.Body)
+			if err != nil {
+				t.Errorf("reading request to the Sentry stand-in: %v", err)
+			}
+
+			time.Sleep(standInDelay)
+
+			standIn.mu.Lock()
+			defer standIn.mu.Unlock()
+
+			standIn.bodies = append(standIn.bodies, string(body))
+		},
+	))
+	t.Cleanup(standIn.server.Close)
+
+	return standIn
+}
+
+// dsn returns a DSN that points Sentry at the stand-in.
+func (s *sentryStandIn) dsn(t *testing.T) string {
+	t.Helper()
+
+	dsn, err := url.Parse(s.server.URL)
+	if err != nil {
+		t.Fatalf("parsing the stand-in URL: %v", err)
+	}
+
+	dsn.User = url.User("public-key")
+	dsn.Path = "/1"
+
+	return dsn.String()
+}
+
+// received reports whether a request to the stand-in contained text.
+func (s *sentryStandIn) received(text string) bool {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	for _, body := range s.bodies {
+		if strings.Contains(body, text) {
+			return true
+		}
+	}
+
+	return false
+}
+
+func TestSentryUnsetDoesNothing(t *testing.T) {
+	viper.Reset()
+	t.Setenv("DNSWATCHER_TARGETS", "example.com")
+	t.Setenv("DNSWATCHER_SENTRY_DSN", "")
+
+	srv := buildServer(t)
+
+	err := server.EnableSentry(srv)
+	if err != nil {
+		t.Fatalf("Sentry setup with no DSN: %v", err)
+	}
+
+	if sentry.CurrentHub().Client() != nil {
+		t.Error("Sentry was set up with no DSN configured")
+	}
+}
+
+// TestSentryReportsHandlerPanic checks that with a valid DSN a panic in
+// a handler is reported to Sentry, still reaches chimw.Recoverer, and
+// has been sent by the time Shutdown returns, and that nothing else is
+// sent to Sentry.
+func TestSentryReportsHandlerPanic(t *testing.T) {
+	standIn := newSentryStandIn(t)
+
+	viper.Reset()
+	t.Setenv("DNSWATCHER_TARGETS", "example.com")
+	t.Setenv("DNSWATCHER_SENTRY_DSN", standIn.dsn(t))
+
+	srv := buildServer(t)
+
+	err := server.EnableSentry(srv)
+	if err != nil {
+		t.Fatalf("Sentry setup with a valid DSN: %v", err)
+	}
+
+	// Sentry's client is global: close it so later tests find none.
+	t.Cleanup(func() {
+		sentry.CurrentHub().Client().Close()
+		sentry.CurrentHub().BindClient(nil)
+	})
+
+	const panicMessage = "handler panic in the Sentry test"
+
+	srv.SetupRoutes()
+	server.RouterOf(srv).Get(
+		"/panic",
+		func(http.ResponseWriter, *http.Request) {
+			panic(panicMessage)
+		},
+	)
+
+	// An ordinary request first: with client reports on, Sentry would
+	// add a count of its dropped transaction to the panic report.
+	serve(srv, httptest.NewRequestWithContext(
+		t.Context(), http.MethodGet, "/.well-known/healthcheck", nil,
+	))
+
+	rec := serve(srv, httptest.NewRequestWithContext(
+		t.Context(), http.MethodGet, "/panic", nil,
+	))
+	if rec.Code != http.StatusInternalServerError {
+		t.Errorf(
+			"status %d, want %d from the recoverer",
+			rec.Code, http.StatusInternalServerError,
+		)
+	}
+
+	err = srv.Shutdown(t.Context())
+	if err != nil {
+		t.Fatalf("Shutdown: %v", err)
+	}
+
+	if !standIn.received(panicMessage) {
+		t.Error("the panic had not been sent to Sentry when Shutdown returned")
+	}
+
+	if standIn.received("client_report") {
+		t.Error("Sentry was sent a client report, not only the panic")
+	}
+}
+
+func TestSentryInvalidDSNStopsStartup(t *testing.T) {
+	viper.Reset()
+	t.Setenv("DNSWATCHER_TARGETS", "example.com")
+	t.Setenv("DNSWATCHER_DATA_DIR", t.TempDir())
+	// Sentry cannot parse this: it has no public key before the host.
+	t.Setenv("DNSWATCHER_SENTRY_DSN", "https://sentry.test/1")
+
+	app := newServerApp(fx.Invoke(func(*server.Server) {}))
+
+	err := app.Start(t.Context())
+	if err == nil {
+		_ = app.Stop(t.Context())
+
+		t.Fatal("startup succeeded with an invalid DSN")
+	}
+
+	if !strings.Contains(err.Error(), "invalid DNSWATCHER_SENTRY_DSN") {
+		t.Errorf("startup error does not name the setting: %v", err)
+	}
+}

internal/server/server.go

@@ -9,6 +9,7 @@ import (
 	"net/http"
 	"time"
 
+	"github.com/getsentry/sentry-go"
 	"github.com/go-chi/chi/v5"
 	"go.uber.org/fx"
 
@@ -33,6 +34,10 @@ type Params struct {
 // shutdownTimeout is how long to wait for graceful shutdown.
 const shutdownTimeout = 30 * time.Second
 
+// sentryFlushTimeout is how long shutdown waits for Sentry to send the
+// error reports it still holds.
+const sentryFlushTimeout = 2 * time.Second
+
 // Socket-level timeouts for the HTTP server.
 //
 // These bound time spent on the connection itself and are a distinct
@@ -84,6 +89,7 @@ const (
 type Server struct {
 	startupTime   time.Time
 	port          int
+	sentryEnabled bool
 	log           *slog.Logger
 	router        *chi.Mux
 	httpServer    *http.Server
@@ -108,6 +114,12 @@ func New(
 	lifecycle.Append(fx.Hook{
 		OnStart: func(_ context.Context) error {
 			srv.startupTime = time.Now()
+
+			err := srv.enableSentry()
+			if err != nil {
+				return err
+			}
+
 			go srv.Run()
 
 			return nil
@@ -152,8 +164,11 @@ func (s *Server) Run() {
 	}
 }
 
-// Shutdown gracefully shuts down the server.
+// Shutdown gracefully shuts down the server, then sends the error
+// reports Sentry still holds.
 func (s *Server) Shutdown(ctx context.Context) error {
+	defer s.flushSentry()
+
 	if s.httpServer == nil {
 		return nil
 	}
@@ -184,3 +199,45 @@ func (s *Server) ServeHTTP(
 ) {
 	s.router.ServeHTTP(writer, request)
 }
+
+// enableSentry turns on Sentry error reporting when
+// DNSWATCHER_SENTRY_DSN is set, and does nothing when it is not. A DSN
+// that Sentry cannot parse is an error, so that startup stops instead
+// of running without the error reporting the operator asked for.
+func (s *Server) enableSentry() error {
+	if s.params.Config.SentryDSN == "" {
+		return nil
+	}
+
+	err := sentry.Init(sentry.ClientOptions{
+		Dsn:     s.params.Config.SentryDSN,
+		Release: s.params.Globals.Appname + "-" + s.params.Globals.Version,
+		// Use the transport that queues each report as it is made. With
+		// the default one, Flush can return before sending a report made
+		// just before it, such as one from the last request at shutdown.
+		DisableTelemetryBuffer: true,
+		// Send panic reports only, not Sentry's counts of what it dropped,
+		// such as the transaction it starts for every request.
+		DisableClientReports: true,
+	})
+	if err != nil {
+		return fmt.Errorf("invalid DNSWATCHER_SENTRY_DSN: %w", err)
+	}
+
+	s.log.Info("sentry error reporting activated")
+	s.sentryEnabled = true
+
+	return nil
+}
+
+// flushSentry sends the error reports Sentry still holds, waiting at
+// most sentryFlushTimeout.
+func (s *Server) flushSentry() {
+	if !s.sentryEnabled {
+		return
+	}
+
+	if !sentry.Flush(sentryFlushTimeout) {
+		s.log.Warn("sentry flush timed out; some error reports were not sent")
+	}
+}

internal/server/server_test.go

@@ -17,18 +17,13 @@ import (
 	"sneak.berlin/go/dnswatcher/internal/state"
 )
 
-// buildServer wires a *server.Server exactly as cmd/dnswatcher does,
+// newServerApp builds an fx app holding a *server.Server wired exactly
-// minus the watcher/resolver subtree that would touch live DNS. fx
+// as cmd/dnswatcher wires it, minus the watcher/resolver subtree that
-// builds the object graph but the lifecycle is never started, so no
+// would touch live DNS, plus the given option. config.New reads viper,
-// OnStart hook runs and nothing listens or resolves. The caller must
+// so the caller must first configure it, which is also why the caller
-// first configure viper (config.New reads it), which is also why the
+// cannot run in parallel.
-// caller cannot run in parallel.
+func newServerApp(option fx.Option) *fx.App {
-func buildServer(t *testing.T) *server.Server {
+	return fx.New(
-	t.Helper()
-
-	var srv *server.Server
-
-	app := fx.New(
 		fx.NopLogger,
 		fx.Provide(
 			globals.New,
@@ -41,8 +36,18 @@ func buildServer(t *testing.T) *server.Server {
 			handlers.New,
 			server.New,
 		),
-		fx.Populate(&srv),
+		option,
 	)
+}
+
+// buildServer builds the server without starting the app's lifecycle,
+// so no OnStart hook runs and nothing listens or resolves.
+func buildServer(t *testing.T) *server.Server {
+	t.Helper()
+
+	var srv *server.Server
+
+	app := newServerApp(fx.Populate(&srv))
 
 	err := app.Err()
 	if err != nil {

internal/state/state.go

@@ -35,8 +35,12 @@ type Params struct {
 }
 
 // DomainState holds the monitoring state for an apex domain.
+// NameserverAddresses holds the sorted addresses each nameserver's name
+// resolves to, by nameserver name. A state file written before it
+// existed loads with it nil.
 type DomainState struct {
 	Nameservers         []string            `json:"nameservers"`
+	NameserverAddresses map[string][]string `json:"nameserverAddresses"`
 	LastChecked         time.Time           `json:"lastChecked"`
 }
 

internal/state/state_test.go

@@ -4,6 +4,7 @@ import (
 	"encoding/json"
 	"os"
 	"path/filepath"
+	"reflect"
 	"strings"
 	"sync"
 	"testing"
@@ -37,6 +38,10 @@ func populateState(t *testing.T, s *state.State) {
 
 	s.SetDomainState("example.com", &state.DomainState{
 		Nameservers: []string{testNS1, testNS2},
+		NameserverAddresses: map[string][]string{
+			testNS1: {testIP, testIPv4},
+			testNS2: {testIPv4},
+		},
 		LastChecked: now,
 	})
 
@@ -123,6 +128,64 @@ func TestSaveLoadRoundTrip_Domains(t *testing.T) {
 	if len(dom.Nameservers) != 2 {
 		t.Errorf("expected 2 nameservers, got %d", len(dom.Nameservers))
 	}
+
+	want := map[string][]string{
+		testNS1: {testIP, testIPv4},
+		testNS2: {testIPv4},
+	}
+	if !reflect.DeepEqual(dom.NameserverAddresses, want) {
+		t.Errorf(
+			"nameserver addresses: got %v, want %v",
+			dom.NameserverAddresses, want,
+		)
+	}
+}
+
+// TestLoadStateFromBeforeNameserverAddresses loads a state file written
+// before nameserver addresses were saved.
+func TestLoadStateFromBeforeNameserverAddresses(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+
+	data := []byte(`{
+		"version": 1,
+		"lastUpdated": "2026-02-19T12:00:00Z",
+		"domains": {
+			"example.com": {
+				"nameservers": ["ns1.example.com.", "ns2.example.com."],
+				"lastChecked": "2026-02-19T12:00:00Z"
+			}
+		}
+	}`)
+
+	err := os.WriteFile(filepath.Join(dir, "state.json"), data, 0o600)
+	if err != nil {
+		t.Fatalf("writing state file: %v", err)
+	}
+
+	s := state.NewForTestWithDataDir(dir)
+
+	err = s.Load()
+	if err != nil {
+		t.Fatalf("Load() error: %v", err)
+	}
+
+	dom, ok := s.GetDomainState("example.com")
+	if !ok {
+		t.Fatal("missing domain example.com")
+	}
+
+	if !reflect.DeepEqual(dom.Nameservers, []string{testNS1, testNS2}) {
+		t.Errorf("nameservers: got %v", dom.Nameservers)
+	}
+
+	if dom.NameserverAddresses != nil {
+		t.Errorf(
+			"nameserver addresses: got %v, want none",
+			dom.NameserverAddresses,
+		)
+	}
 }
 
 // TestSaveLoadRoundTrip_Hostnames verifies hostname data survives a save/load cycle.

internal/watcher/export_test.go

@@ -48,6 +48,25 @@ func (w *Watcher) DetectHostnameChanges(
 	w.detectHostnameChanges(ctx, hostname, prev, current)
 }
 
+// ResolveNameserverAddresses exports resolveNameserverAddresses for
+// testing.
+func (w *Watcher) ResolveNameserverAddresses(
+	ctx context.Context,
+	nameservers []string,
+	prev map[string][]string,
+) map[string][]string {
+	return w.resolveNameserverAddresses(ctx, nameservers, prev)
+}
+
+// DetectNSAddressChanges exports detectNSAddressChanges for testing.
+func (w *Watcher) DetectNSAddressChanges(
+	ctx context.Context,
+	domain string,
+	prev, current map[string][]string,
+) {
+	w.detectNSAddressChanges(ctx, domain, prev, current)
+}
+
 // BuildHostnameState exports buildHostnameState for testing.
 func BuildHostnameState(
 	results map[string]*resolver.NameserverResponse,

internal/watcher/nsaddress_test.go

@@ -0,0 +1,151 @@
+package watcher_test
+
+import (
+	"context"
+	"log/slog"
+	"reflect"
+	"testing"
+
+	"sneak.berlin/go/dnswatcher/internal/livednstest"
+	"sneak.berlin/go/dnswatcher/internal/resolver"
+	"sneak.berlin/go/dnswatcher/internal/watcher"
+)
+
+const domain = "example.net"
+
+func TestNSAddressChangeAlerts(t *testing.T) {
+	t.Parallel()
+
+	// Each case is the nameserver addresses saved by the previous check
+	// and by the current one.
+	tests := []struct {
+		name          string
+		prev, current map[string][]string
+		want          int
+	}{
+		{
+			"same addresses",
+			map[string][]string{nsA: {ip1, ip2}},
+			map[string][]string{nsA: {ip1, ip2}},
+			0,
+		},
+		{
+			"same addresses in another order",
+			map[string][]string{nsA: {ip2, ip1}},
+			map[string][]string{nsA: {ip1, ip2}},
+			0,
+		},
+		{
+			"address replaced",
+			map[string][]string{nsA: {ip1}},
+			map[string][]string{nsA: {ip2}},
+			1,
+		},
+		{
+			"address added",
+			map[string][]string{nsA: {ip1}},
+			map[string][]string{nsA: {ip1, ip2}},
+			1,
+		},
+		{
+			"two nameservers changed",
+			map[string][]string{nsA: {ip1}, nsB: {ip2}},
+			map[string][]string{nsA: {ip3}, nsB: {ip3}},
+			2,
+		},
+		{
+			"nameserver added",
+			map[string][]string{nsA: {ip1}},
+			map[string][]string{nsA: {ip1}, nsB: {ip2}},
+			0,
+		},
+		{
+			"nameserver removed",
+			map[string][]string{nsA: {ip1}, nsB: {ip2}},
+			map[string][]string{nsA: {ip1}},
+			0,
+		},
+		{
+			"state file from before addresses were saved",
+			nil,
+			map[string][]string{nsA: {ip1}, nsB: {ip2}},
+			0,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			notifier := &mockNotifier{}
+			w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
+
+			w.DetectNSAddressChanges(t.Context(), domain, tt.prev, tt.current)
+
+			got := len(notifier.getNotifications())
+			if got != tt.want {
+				t.Errorf("sent %d address changes, want %d", got, tt.want)
+			}
+		})
+	}
+}
+
+func TestNSAddressChangeAlertNamesDomainNameserverAndAddresses(
+	t *testing.T,
+) {
+	t.Parallel()
+
+	notifier := &mockNotifier{}
+	w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
+
+	w.DetectNSAddressChanges(
+		t.Context(), domain,
+		map[string][]string{nsA: {ip1}},
+		map[string][]string{nsA: {ip2, ip3}},
+	)
+
+	want := notification{
+		Title: "NS Address Change: " + domain,
+		Message: "Domain: " + domain + "\nNameserver: " + nsA +
+			"\nOld: " + ip1 + "\nNew: " + ip2 + ", " + ip3,
+		Priority: "warning",
+	}
+
+	got := notifier.getNotifications()
+	if len(got) != 1 || got[0] != want {
+		t.Errorf("sent %v, want %v", got, want)
+	}
+}
+
+// TestNameserverWithNoAddressKeepsPrevious looks up nameserver names
+// with no address: two under .invalid, whose lookup fails with an
+// error, and one that does not exist under a real zone, which live DNS
+// answers with no address and no error. Each one with addresses saved
+// by the previous check keeps them; the one without gets none.
+func TestNameserverWithNoAddressKeepsPrevious(t *testing.T) {
+	t.Parallel()
+
+	w := watcher.NewForTest(
+		nil, nil, resolver.NewFromLogger(slog.Default()), nil, nil, nil,
+	)
+
+	nonexistentNS := "this-surely-does-not-exist-xyz." + testSmallDomain + "."
+
+	prev := map[string][]string{oldNS1: {oldIP}, nonexistentNS: {oldIP}}
+
+	var got map[string][]string
+
+	// The result is the same whether or not live DNS answers, so the
+	// lookup is not retried.
+	_ = livednstest.Run(func(ctx context.Context) error {
+		got = w.ResolveNameserverAddresses(
+			ctx, []string{oldNS1, oldNS2, nonexistentNS}, prev,
+		)
+
+		return nil
+	})
+
+	if !reflect.DeepEqual(got, prev) {
+		t.Errorf("saved %v, want %v", got, prev)
+	}
+}

internal/watcher/watcher.go

@@ -56,6 +56,7 @@ type Watcher struct {
 	tlsCheck         TLSChecker
 	notify           Notifier
 	cancel           context.CancelFunc
+	done             chan struct{} // closed when Run returns
 	firstRun         bool
 	expiryNotifiedMu sync.Mutex
 	expiryNotified   map[string]time.Time
@@ -79,31 +80,47 @@ func New(
 	}
 
 	lifecycle.Append(fx.Hook{
-		OnStart: func(_ context.Context) error {
+		OnStart: func(startCtx context.Context) error {
-			// Use context.Background() — the fx startup context
+			// The fx startup context expires after startup
-			// expires after startup completes, so deriving from it
+			// completes, so the watcher's context drops its
-			// would cancel the watcher immediately. The watcher's
+			// cancellation. The watcher's lifetime is controlled
-			// lifetime is controlled by w.cancel in OnStop.
+			// by w.cancel in OnStop.
-			ctx, cancel := context.WithCancel(context.Background())
+			ctx, cancel := context.WithCancel(
+				context.WithoutCancel(startCtx),
+			)
 			w.cancel = cancel
+			w.done = make(chan struct{})
 
-			go w.Run(ctx) //nolint:contextcheck // intentionally not derived from startCtx
+			go func() {
+				defer close(w.done)
+
+				w.Run(ctx)
+			}()
 
 			return nil
 		},
-		OnStop: func(_ context.Context) error {
+		OnStop: func(ctx context.Context) error {
-			if w.cancel != nil {
 			w.cancel()
-			}
 
+			// Run saves state as it returns. Waiting for it here
+			// means the save is done before shutdown goes on.
+			select {
+			case <-w.done:
 				return nil
+			case <-ctx.Done():
+				return fmt.Errorf(
+					"waiting for the watcher to stop: %w",
+					ctx.Err(),
+				)
+			}
 		},
 	})
 
 	return w, nil
 }
 
-// Run starts the monitoring loop with periodic scheduling.
+// Run starts the monitoring loop with periodic scheduling. When ctx
+// is cancelled, it saves state and returns.
 func (w *Watcher) Run(ctx context.Context) {
 	w.log.Info(
 		"watcher starting",
@@ -125,6 +142,7 @@ func (w *Watcher) Run(ctx context.Context) {
 	for {
 		select {
 		case <-ctx.Done():
+			w.saveState()
 			w.log.Info("watcher stopped")
 
 			return
@@ -216,12 +234,24 @@ func (w *Watcher) checkDomain(
 	now := time.Now().UTC()
 
 	prev, hasPrev := w.state.GetDomainState(domain)
+
+	var prevAddresses map[string][]string
+	if hasPrev {
+		prevAddresses = prev.NameserverAddresses
+	}
+
+	addresses := w.resolveNameserverAddresses(
+		ctx, nameservers, prevAddresses,
+	)
+
 	if hasPrev && !w.firstRun {
 		w.detectNSChanges(ctx, domain, prev.Nameservers, nameservers)
+		w.detectNSAddressChanges(ctx, domain, prevAddresses, addresses)
 	}
 
 	w.state.SetDomainState(domain, &state.DomainState{
 		Nameservers:         nameservers,
+		NameserverAddresses: addresses,
 		LastChecked:         now,
 	})
 
@@ -290,6 +320,73 @@ func (w *Watcher) detectNSChanges(
 	)
 }
 
+// resolveNameserverAddresses returns the sorted addresses each
+// nameserver's name resolves to. A nameserver whose lookup fails or
+// finds no address keeps its addresses from prev: the resolver finds no
+// address, without an error, when every server it asks times out, and
+// that is not an address change.
+func (w *Watcher) resolveNameserverAddresses(
+	ctx context.Context,
+	nameservers []string,
+	prev map[string][]string,
+) map[string][]string {
+	addresses := make(map[string][]string, len(nameservers))
+
+	for _, ns := range nameservers {
+		ips, err := w.resolver.ResolveIPAddresses(ctx, ns)
+		if err == nil && len(ips) > 0 {
+			sort.Strings(ips)
+			addresses[ns] = ips
+
+			continue
+		}
+
+		w.log.Error(
+			"no addresses found for nameserver",
+			"nameserver", ns,
+			"error", err,
+		)
+
+		if prevIPs, ok := prev[ns]; ok {
+			addresses[ns] = prevIPs
+		}
+	}
+
+	return addresses
+}
+
+// detectNSAddressChanges notifies when a nameserver in both checks
+// resolves to different addresses. A nameserver added or removed is
+// reported by detectNSChanges alone, and one with no addresses saved by
+// the previous check, as in a state file from before they were saved,
+// is not compared.
+func (w *Watcher) detectNSAddressChanges(
+	ctx context.Context,
+	domain string,
+	prev, current map[string][]string,
+) {
+	for ns, cur := range current {
+		old, ok := prev[ns]
+		if !ok || sliceEqual(old, cur) {
+			continue
+		}
+
+		msg := fmt.Sprintf(
+			"Domain: %s\nNameserver: %s\nOld: %s\nNew: %s",
+			domain, ns,
+			strings.Join(old, ", "),
+			strings.Join(cur, ", "),
+		)
+
+		w.notify.SendNotification(
+			ctx,
+			"NS Address Change: "+domain,
+			msg,
+			"warning",
+		)
+	}
+}
+
 func (w *Watcher) checkHostname(
 	ctx context.Context,
 	hostname string,

internal/watcher/watcher_test.go

@@ -4,7 +4,9 @@ import (
 	"context"
 	"fmt"
 	"log/slog"
+	"os"
 	"slices"
+	"strings"
 	"sync"
 	"testing"
 	"time"
@@ -26,9 +28,15 @@ import (
 // so tests assert on what the watcher does with the answers, never on
 // the records these zones publish. testHost's nameservers and addresses
 // stay the same from one check to the next, which the tests that check
-// it twice rely on.
+// it twice rely on, and testSmallDomain's nameservers stay the same
+// between a test looking them up and its check. A domain check looks up
+// each nameserver's addresses, about a second per nameserver, so the
+// tests that check a domain use testSmallDomain, which has two
+// nameservers, and check it once. The tests that query testDomain's
+// nameservers directly do no domain check.
 const (
 	testDomain      = "google.com"
+	testSmallDomain = "example.com"
 	testHost        = "cloudflare.com"
 	testIssuer      = "DigiCert"
 )
@@ -135,6 +143,7 @@ type testDeps struct {
 	notifier    *mockNotifier
 	state       *state.State
 	config      *config.Config
+	log         *logger.Logger
 }
 
 func newTestWatcher(
@@ -143,6 +152,23 @@ func newTestWatcher(
 ) (*watcher.Watcher, *testDeps) {
 	t.Helper()
 
+	deps := newTestDeps(t, cfg)
+
+	w := watcher.NewForTest(
+		deps.config,
+		deps.state,
+		resolver.NewFromLogger(slog.Default()),
+		deps.portChecker,
+		deps.tlsChecker,
+		deps.notifier,
+	)
+
+	return w, deps
+}
+
+func newTestDeps(t *testing.T, cfg *config.Config) *testDeps {
+	t.Helper()
+
 	deps := &testDeps{
 		portChecker: &mockPortChecker{},
 		tlsChecker: &mockTLSChecker{
@@ -157,30 +183,21 @@ func newTestWatcher(
 		t.Fatalf("globals.New: %v", err)
 	}
 
-	log, err := logger.New(nil, logger.Params{Globals: g})
+	deps.log, err = logger.New(nil, logger.Params{Globals: g})
 	if err != nil {
 		t.Fatalf("logger.New: %v", err)
 	}
 
 	// The watcher saves state after every check, into cfg.DataDir.
 	deps.state, err = state.New(fxtest.NewLifecycle(t), state.Params{
-		Logger: log,
+		Logger: deps.log,
 		Config: cfg,
 	})
 	if err != nil {
 		t.Fatalf("state.New: %v", err)
 	}
 
-	w := watcher.NewForTest(
+	return deps
-		deps.config,
-		deps.state,
-		resolver.NewFromLogger(slog.Default()),
-		deps.portChecker,
-		deps.tlsChecker,
-		deps.notifier,
-	)
-
-	return w, deps
 }
 
 func defaultTestConfig(t *testing.T) *config.Config {
@@ -196,8 +213,10 @@ func defaultTestConfig(t *testing.T) *config.Config {
 
 // checkOnce runs the watcher's checks once and returns an error when a
 // configured name has no hostname state saved by this check, or that
-// state holds no address. Either live DNS gave no answer for the name,
+// state holds no address, or a configured domain's nameserver has no
-// or the watcher saved no fresh result for it.
+// address saved or still has oldIP, which the tests save and live DNS
+// never returns. Either live DNS gave no answer for the name, or the
+// watcher saved no fresh result for it.
 func checkOnce(
 	ctx context.Context,
 	w *watcher.Watcher,
@@ -221,6 +240,20 @@ func checkOnce(
 		}
 	}
 
+	for _, name := range deps.config.Domains {
+		ds, _ := deps.state.GetDomainState(name)
+		for _, ns := range ds.Nameservers {
+			ips := ds.NameserverAddresses[ns]
+			if len(ips) == 0 || slices.Contains(ips, oldIP) {
+				return fmt.Errorf(
+					"%s: nameserver %s: %w, or the watcher saved "+
+						"no fresh addresses for it",
+					name, ns, livednstest.ErrNoAnswer,
+				)
+			}
+		}
+	}
+
 	return nil
 }
 
@@ -262,6 +295,26 @@ func runChecks(
 	return deps
 }
 
+// lookupNameservers returns the nameservers live DNS lists for domain,
+// for a test to save in the state its check starts from.
+func lookupNameservers(t *testing.T, domain string) []string {
+	t.Helper()
+
+	res := resolver.NewFromLogger(slog.Default())
+
+	var nameservers []string
+
+	livednstest.Retry(t, "LookupNS("+domain+")", func(ctx context.Context) error {
+		var err error
+
+		nameservers, err = res.LookupNS(ctx, domain)
+
+		return err
+	})
+
+	return nameservers
+}
+
 // addresses returns the A and AAAA values saved for a hostname.
 func addresses(hs *state.HostnameState) []string {
 	var ips []string
@@ -314,7 +367,7 @@ func TestFirstRunBaseline(t *testing.T) {
 	t.Parallel()
 
 	cfg := defaultTestConfig(t)
-	cfg.Domains = []string{testDomain}
+	cfg.Domains = []string{testSmallDomain}
 	cfg.Hostnames = []string{testHost}
 
 	deps := runChecks(t, cfg, nil, nil)
@@ -367,7 +420,7 @@ func TestDomainPortAndTLSChecks(t *testing.T) {
 	t.Parallel()
 
 	cfg := defaultTestConfig(t)
-	cfg.Domains = []string{testDomain}
+	cfg.Domains = []string{testSmallDomain}
 
 	deps := runChecks(t, cfg, nil, nil)
 
@@ -406,23 +459,106 @@ func TestNSChangeDetection(t *testing.T) {
 	t.Parallel()
 
 	cfg := defaultTestConfig(t)
-	cfg.Domains = []string{testDomain}
+	cfg.Domains = []string{testSmallDomain}
 
 	// The saved state lists nameservers that live DNS does not.
 	deps := runChecks(t, cfg, func(deps *testDeps) {
-		deps.state.SetDomainState(testDomain, &state.DomainState{
+		deps.state.SetDomainState(testSmallDomain, &state.DomainState{
 			Nameservers: []string{oldNS1, oldNS2},
 		})
 	}, nil)
 
-	assertNotified(t, deps, "NS Change: "+testDomain, "warning")
+	assertNotified(t, deps, "NS Change: "+testSmallDomain, "warning")
 
-	ds, _ := deps.state.GetDomainState(testDomain)
+	ds, _ := deps.state.GetDomainState(testSmallDomain)
 	if slices.Contains(ds.Nameservers, oldNS1) {
 		t.Errorf("saved nameservers not updated: %v", ds.Nameservers)
 	}
 }
 
+func TestNSAddressChangeDetection(t *testing.T) {
+	t.Parallel()
+
+	cfg := defaultTestConfig(t)
+	cfg.Domains = []string{testSmallDomain}
+
+	nameservers := lookupNameservers(t, testSmallDomain)
+
+	// The saved state lists the nameservers live DNS lists, each at an
+	// address live DNS never returns.
+	deps := runChecks(t, cfg, func(deps *testDeps) {
+		nsAddresses := make(map[string][]string, len(nameservers))
+		for _, ns := range nameservers {
+			nsAddresses[ns] = []string{oldIP}
+		}
+
+		deps.state.SetDomainState(testSmallDomain, &state.DomainState{
+			Nameservers:         nameservers,
+			NameserverAddresses: nsAddresses,
+		})
+	}, nil)
+
+	title := "NS Address Change: " + testSmallDomain
+	ds, _ := deps.state.GetDomainState(testSmallDomain)
+
+	// One alert per nameserver, naming it and the address it had.
+	for _, ns := range ds.Nameservers {
+		prefix := "Domain: " + testSmallDomain + "\nNameserver: " + ns +
+			"\nOld: " + oldIP + "\nNew: "
+
+		sent := 0
+
+		for _, n := range deps.notifier.getNotifications() {
+			if n.Title == title && strings.HasPrefix(n.Message, prefix) {
+				sent++
+			}
+		}
+
+		if sent != 1 {
+			t.Errorf("sent %d address changes for %s, want 1", sent, ns)
+		}
+	}
+
+	if n := countNotifications(deps, title); n != len(ds.Nameservers) {
+		t.Errorf(
+			"sent %d address changes for %d nameservers",
+			n, len(ds.Nameservers),
+		)
+	}
+
+	if n := countNotifications(deps, "NS Change: "+testSmallDomain); n != 0 {
+		t.Errorf("sent %d NS changes, want 0", n)
+	}
+}
+
+func TestNSAddedAndRemovedIsNoAddressChange(t *testing.T) {
+	t.Parallel()
+
+	cfg := defaultTestConfig(t)
+	cfg.Domains = []string{testSmallDomain}
+
+	nameservers := lookupNameservers(t, testSmallDomain)
+
+	// The saved state lists oldNS1, which live DNS does not, in place of
+	// the first nameserver live DNS lists, so that the check finds that
+	// one added and oldNS1 removed. Only oldNS1 has addresses saved.
+	deps := runChecks(t, cfg, func(deps *testDeps) {
+		deps.state.SetDomainState(testSmallDomain, &state.DomainState{
+			Nameservers:         append([]string{oldNS1}, nameservers[1:]...),
+			NameserverAddresses: map[string][]string{oldNS1: {oldIP}},
+		})
+	}, nil)
+
+	if n := countNotifications(deps, "NS Change: "+testSmallDomain); n != 1 {
+		t.Errorf("sent %d NS changes, want 1", n)
+	}
+
+	title := "NS Address Change: " + testSmallDomain
+	if n := countNotifications(deps, title); n != 0 {
+		t.Errorf("sent %d address changes, want 0", n)
+	}
+}
+
 func TestRecordChangeDetection(t *testing.T) {
 	t.Parallel()
 
@@ -539,6 +675,85 @@ func TestGracefulShutdown(t *testing.T) {
 	}
 }
 
+// TestStopSavesState stops a watcher built by New the way fx stops it,
+// and checks that a change made to the state after the last check is in
+// the state file afterwards. The state's own stop hook never runs here,
+// so only the watcher can have saved it. Nothing is configured to
+// check, so no DNS is involved.
+func TestStopSavesState(t *testing.T) {
+	t.Parallel()
+
+	cfg := defaultTestConfig(t)
+	deps := newTestDeps(t, cfg)
+	lc := fxtest.NewLifecycle(t)
+
+	_, err := watcher.New(lc, watcher.Params{
+		Logger:    deps.log,
+		Config:    cfg,
+		State:     deps.state,
+		Resolver:  resolver.NewFromLogger(slog.Default()),
+		PortCheck: deps.portChecker,
+		TLSCheck:  deps.tlsChecker,
+		Notify:    deps.notifier,
+	})
+	if err != nil {
+		t.Fatalf("watcher.New: %v", err)
+	}
+
+	lc.RequireStart()
+
+	// The first check saves state once. Wait for that save before
+	// changing the state, so the change can reach the file only
+	// through the save made at stop.
+	deadline := time.Now().Add(5 * time.Second)
+
+	for {
+		_, err = os.Stat(cfg.StatePath())
+		if err == nil {
+			break
+		}
+
+		if time.Now().After(deadline) {
+			t.Fatalf("the first check saved no state: %v", err)
+		}
+
+		time.Sleep(10 * time.Millisecond)
+	}
+
+	deps.state.SetDomainState(testDomain, &state.DomainState{
+		Nameservers: []string{oldNS1},
+	})
+
+	ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
+	defer cancel()
+
+	err = lc.Stop(ctx)
+	if err != nil {
+		t.Fatalf("stopping the watcher: %v", err)
+	}
+
+	saved, err := state.New(fxtest.NewLifecycle(t), state.Params{
+		Logger: deps.log,
+		Config: cfg,
+	})
+	if err != nil {
+		t.Fatalf("state.New: %v", err)
+	}
+
+	err = saved.Load()
+	if err != nil {
+		t.Fatalf("loading the state file: %v", err)
+	}
+
+	ds, ok := saved.GetDomainState(testDomain)
+	if !ok || !slices.Equal(ds.Nameservers, []string{oldNS1}) {
+		t.Errorf(
+			"state file after stop has %+v for %s, want nameservers %v",
+			ds, testDomain, []string{oldNS1},
+		)
+	}
+}
+
 func TestDNSRunsBeforePortAndTLSChecks(t *testing.T) {
 	t.Parallel()