watcher: a hostname whose zone's nameservers all stop answering gets NS Failure (closes #284)

When none of the nameservers of a hostname's zone answered, the walk to
them failed and the check was only logged. The walk now returns a
NoAnswerError naming the nameservers of the zone it reached through a
referral when none of them answers. When those are the nameservers the
previous check saved for the name, each is saved as failed with the
reason, so it gets an NS Failure and later an NS Recovery. A domain with
no delegation of its own, whose NS lookup is that same walk, gets the
same. A walk that fails before that zone, as when the host loses its
network, is still only logged.

Model: opus-5-5

README.md

@@ -78,8 +78,9 @@ notification endpoint set, changes show only on the dashboard; see
   answer NXDOMAIN, that it does not exist, has no nameservers and is shown as
   not existing (see Web Dashboard and HTTP API). A domain that exists but has no
   delegation of its own, such as `octocat.github.io`, has no nameservers either.
-  When the parent zone's servers do not answer, the check fails and the set from
-  the previous check is kept.
+  When the parent zone's servers do not answer, or the lookup of the domain's
+  own records (below) fails, the check fails and the set from the previous check
+  is kept.
 - Any change triggers a notification:
     - NS added to or removed from that set. A domain that had nameservers on the
       previous check and no longer exists gets one with all of them removed.
@@ -99,11 +100,18 @@ notification endpoint set, changes show only on the dashboard; see
   records, stored per nameserver. Their changes are notified as a hostname's
   are, as a record change, NS query failure, NS recovery, inconsistency or CNAME
   address change, in a message that starts `Domain:` where a hostname's starts
-  `Hostname:`. A domain listed in `DNSWATCHER_SKIP_RECORD_NOTIFICATIONS` gets no
-  record change or inconsistency notification. A domain with no delegation of
-  its own has these records asked at the servers of the zone it is in, as a
-  hostname has. A domain that does not exist has none: they are not asked for,
-  and those saved by an earlier check are removed without a notification.
+  `Hostname:`. A nameserver removed from the domain's delegation gets only the
+  NS change notification above. A domain listed in
+  `DNSWATCHER_SKIP_RECORD_NOTIFICATIONS` gets no record change or inconsistency
+  notification. A domain with no delegation of its own has these records asked
+  at the servers of the zone it is in, as a hostname has. When none of those
+  servers answers, the lookup of its NS record set fails, and they are saved as
+  failed and notified as a hostname's are (see below). A nameserver the domain's
+  records were asked at on the previous check but not on this one gets a
+  hostname's NS change notification (see below), unless it was in the domain's
+  delegation on the previous check, as when that delegation is withdrawn. A
+  domain that does not exist has none: they are not asked for, and those saved
+  by an earlier check are removed without a notification.
 
 ### DNS Hostname Monitoring (Subdomains)
 
@@ -115,7 +123,13 @@ notification endpoint set, changes show only on the dashboard; see
   trace moves from a name to its parent only when the servers asked answer that
   the name has no delegation of its own, or does not exist. When they do not
   answer, the check fails and the hostname's records from the previous check are
-  kept.
+  kept, unless they are the nameservers the previous check saved for the
+  hostname. Each of those is then saved as failed, with the reason, as a
+  domain's nameservers are when none of them answers, so it gets an NS query
+  failure notification, and an NS recovery notification when it answers again
+  (see below). A trace that gets no answer before it reaches them, as when the
+  host running dnswatcher loses its network, fails the check without a
+  notification.
 - Queries **each** authoritative nameserver independently for **all** record
   types: A, AAAA, CNAME, MX, TXT, SRV, CAA, NS.
 - Each record type is a query of its own. When a nameserver answers some types
@@ -156,6 +170,10 @@ notification endpoint set, changes show only on the dashboard; see
     - **NS recovery**: A previously-unreachable nameserver starts responding
       again. Its records are not compared with those from before it failed, so a
       change made while it was failing is not reported as a record change.
+    - **NS change**: A nameserver of the zone the hostname is in on the previous
+      check is not one now, whether or not it answered. One notification lists
+      every such nameserver on a `Removed:` line. Nothing is sent for a
+      nameserver that joins the zone.
     - **Inconsistency detected**: Two nameservers return different record sets
       for the same hostname and did not already differ on the previous check.
       Every pair of nameservers is compared. The alert is sent once for each
@@ -213,8 +231,14 @@ notification endpoint set, changes show only on the dashboard; see
     - Certificate CN, issuer, or SANs changed (replacement detected, reports old
       and new CN and issuer).
     - TLS connection failure to a previously-reachable IP:443 (handshake error,
-      timeout, connection refused after previously succeeding).
+      timeout, or connection refused when port 443 closed after the port check
+      that found it open).
     - TLS recovery: a previously-failing IP:443 now completes a handshake again.
+- A port check that finds port 443 closed removes the certificate entries for
+  that IP address, without a notification of its own: the port change
+  notification reports the closed port. When the port opens again, the next TLS
+  check saves the certificate as new, without a certificate change or recovery
+  notification.
 
 ### Notifications
 
@@ -237,7 +261,9 @@ includes:
 
 - **DNS record changes**: Which hostname or domain, which nameserver, what
   record type, old values, new values.
-- **DNS NS changes**: Which domain, which nameservers were added/removed.
+- **DNS NS changes**: Which domain, which nameservers were added to or removed
+  from its delegation. For a hostname, or a domain with no delegation of its
+  own, which nameservers left the zone it is in.
 - **NS address changes**: Which domain, which nameserver, its old and new
   addresses.
 - **CNAME address changes**: Which hostname or domain, the old and new addresses
@@ -282,8 +308,10 @@ clears them.
   port entry left with no name goes, so the dashboard and `/api/v1/status` no
   longer list or count it. The first check's port checks remove the port entries
   of addresses no configured name has.
-- Each port check also removes the certificate entries for an address their name
-  no longer resolves to, except while none of the name's nameservers answer.
+- Each port check also removes, without a notification, the certificate entries
+  for an address whose port 443 it found closed, and those for an address their
+  name no longer resolves to, except while none of the name's nameservers
+  answer.
 
 ### Web Dashboard
 
@@ -423,7 +451,7 @@ following precedence (highest to lowest):
 
 | Variable                               | Description                                                                                                                           | Default               |
 | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
-| `PORT`                                 | HTTP listen port                                                                                                                      | `8080`                |
+| `PORT`                                 | HTTP listen port; a port that cannot be bound, such as one already in use, stops startup                                              | `8080`                |
 | `DNSWATCHER_DEBUG`                     | Enable debug logging                                                                                                                  | `false`               |
 | `DNSWATCHER_DATA_DIR`                  | Directory for state file                                                                                                              | `/var/lib/dnswatcher` |
 | `DNSWATCHER_TARGETS`                   | Comma-separated DNS names (auto-classified via PSL)                                                                                   | `""`                  |
@@ -437,7 +465,7 @@ following precedence (highest to lowest):
 | `DNSWATCHER_SENTRY_DSN`                | Sentry DSN for error reporting                                                                                                        | `""`                  |
 | `DNSWATCHER_MAINTENANCE_MODE`          | Only sets `maintenanceMode` in the health check response; changes nothing else                                                        | `false`               |
 | `DNSWATCHER_METRICS_USERNAME`          | Basic auth username for /metrics, which is served only when this is set                                                               | `""`                  |
-| `DNSWATCHER_METRICS_PASSWORD`          | Basic auth password for /metrics                                                                                                      | `""`                  |
+| `DNSWATCHER_METRICS_PASSWORD`          | Basic auth password for /metrics; required when `DNSWATCHER_METRICS_USERNAME` is set, and startup stops without it                    | `""`                  |
 | `DNSWATCHER_SEND_TEST_NOTIFICATION`    | Send a test notification after first scan completes                                                                                   | `false`               |
 
 **`DNSWATCHER_TARGETS` is required.** dnswatcher will refuse to start if no
@@ -646,10 +674,20 @@ for A, AAAA and CNAME, the previous check's list is kept, or `null` when no
 earlier check saved one. A state file without it loads, and the first check
 after that saves it without a notification.
 
+A port entry's key is its address and port, and a certificate entry's key its
+address, port 443 and hostname, joined by colons. An IPv6 address is written in
+brackets in these keys, as in `[2001:db8::1]:443` and
+`[2001:db8::1]:443:www.example.com`. `/api/v1/status` and the dashboard show the
+same keys.
+
 A port entry's `hostnames` lists every name that resolves to its address,
 domains included. A port entry in the older format, with one `hostname` instead
 of the `hostnames` list, loads as a list of that one name.
 
+A state file that leaves out any of `domains`, `hostnames`, `ports` and
+`certificates`, or holds `null` for one, loads with that section empty. A file
+holding `{}` loads as empty state.
+
 ---
 
 ## Entrypoints

TODO.md

@@ -10,15 +10,35 @@
 
 # Status
 
-pre-1.0. No git tags. Work lands on `next` by PR. Open work for 1.0 is tracked
-on the 1.0 milestone: https://git.eeqj.de/sneak/dnswatcher/milestone/7
+pre-1.0. Work lands on `next` by PR. Open work for 1.0 is tracked on the 1.0
+milestone: https://git.eeqj.de/sneak/dnswatcher/milestone/7
 
 # Next Step
 
-trial run of the finished image: https://git.eeqj.de/sneak/dnswatcher/issues/149
+1.0 tracking issue: https://git.eeqj.de/sneak/dnswatcher/issues/66
 
 # Completed Steps
 
+- 2026-10-13: a hostname or a domain whose zone's nameservers all stop answering
+  gets an NS Failure for each, then an NS Recovery (closes #284).
+- 2026-10-13: an IPv6 address is written in brackets before its port, as in
+  `[2001:db8::1]:443`, in keys and Port Change notifications (closes #270).
+- 2026-10-13: `TODO.md` names only open issues as next and future steps, and its
+  Status says nothing about git tags (closes #281).
+- 2026-10-13: a port check that finds port 443 closed removes that address's
+  certificate entries, so none is left shown as `ok` (closes #265).
+- 2026-10-13: a state file holding `{}`, or leaving out a section, loads with
+  that section empty; the first check no longer crashes on it (closes #267).
+- 2026-10-13: a stop before the first scan completes sends no startup
+  notification; a scan that ran to the end still sends it (closes #271).
+- 2026-10-13: a nameserver leaving a domain's delegation sends only NS Change;
+  one leaving a hostname's zone sends NS Change, not NS Failure (closes #262).
+- 2026-10-13: `DNSWATCHER_METRICS_USERNAME` set without
+  `DNSWATCHER_METRICS_PASSWORD` stops startup (closes #268).
+- 2026-10-13: a save syncs the state file to disk before renaming it into place,
+  so a crash cannot leave it empty and stop the next start (closes #264).
+- 2026-10-13: a `PORT` that cannot be bound stops startup with an error naming
+  the address, instead of running without the HTTP server (closes #266).
 - 2026-10-13: a bad `PORT`, `DNSWATCHER_TLS_EXPIRY_WARNING` or true/false
   setting stops startup instead of being read as 0 or false (closes #269).
 - 2026-10-13: a check saving the state while the dashboard or `/api/v1/status`
@@ -187,6 +207,4 @@ trial run of the finished image: https://git.eeqj.de/sneak/dnswatcher/issues/149
 
 # Future Steps
 
-- 1.0 readiness: run it with a real config and read the logs:
-  https://git.eeqj.de/sneak/dnswatcher/issues/66
-- review toward 1.0: https://git.eeqj.de/sneak/dnswatcher/issues/144
+- DNSSEC validation, after 1.0: https://git.eeqj.de/sneak/dnswatcher/issues/59

internal/config/config.go

@@ -61,6 +61,13 @@ var ErrInvalidTLSExpiryWarning = errors.New(
 // unset and means the default.
 var ErrInvalidBool = errors.New("value must be true or false")
 
+// ErrNoMetricsPassword is returned when DNSWATCHER_METRICS_USERNAME is set and
+// DNSWATCHER_METRICS_PASSWORD is empty. Basic Auth on /metrics would otherwise
+// let in anyone who sends the username with an empty password.
+var ErrNoMetricsPassword = errors.New(
+	"DNSWATCHER_METRICS_PASSWORD must be set when DNSWATCHER_METRICS_USERNAME is set",
+)
+
 // Params contains dependencies for Config.
 type Params struct {
 	fx.In
@@ -108,6 +115,10 @@ func New(_ fx.Lifecycle, params Params) (*Config, error) {
 		return nil, err
 	}
 
+	if cfg.MetricsUsername != "" && cfg.MetricsPassword == "" {
+		return nil, ErrNoMetricsPassword
+	}
+
 	configureDebugLogging(cfg, params)
 
 	return cfg, nil

internal/config/config_test.go

@@ -354,6 +354,24 @@ func TestNew_SkipRecordNotificationsNotInTargetsStopsStartup(t *testing.T) {
 	require.ErrorContains(t, err, `"www.example.net"`)
 }
 
+// TestNew_MetricsUsernameWithoutPasswordStopsStartup checks an unset and an
+// empty password; TestNew_EnvironmentOverrides checks a username with one.
+func TestNew_MetricsUsernameWithoutPasswordStopsStartup(t *testing.T) {
+	viper.Reset()
+	t.Setenv("DNSWATCHER_TARGETS", "example.com")
+	t.Setenv("DNSWATCHER_METRICS_USERNAME", "prom")
+
+	_, err := config.New(nil, newTestParams(t))
+	require.ErrorIs(t, err, config.ErrNoMetricsPassword, "password unset")
+	require.ErrorContains(t, err, "DNSWATCHER_METRICS_PASSWORD")
+
+	viper.Reset()
+	t.Setenv("DNSWATCHER_METRICS_PASSWORD", "")
+
+	_, err = config.New(nil, newTestParams(t))
+	require.ErrorIs(t, err, config.ErrNoMetricsPassword, "password empty")
+}
+
 func TestNew_CustomDNSIntervalDuration(t *testing.T) {
 	viper.Reset()
 	t.Setenv("DNSWATCHER_TARGETS", "example.com")

internal/resolver/errors.go

@@ -2,6 +2,23 @@ package resolver
 
 import "errors"
 
+// NoAnswerError is returned when the walk to a name's nameservers
+// reached a zone through a referral and none of that zone's
+// nameservers, Nameservers as the referral names them, answered. Err
+// says why they did not.
+type NoAnswerError struct {
+	Nameservers []string
+	Err         error
+}
+
+func (e *NoAnswerError) Error() string {
+	return e.Err.Error()
+}
+
+func (e *NoAnswerError) Unwrap() error {
+	return e.Err
+}
+
 // Sentinel errors returned by the resolver.
 var (
 	// ErrNoNameservers is returned when no authoritative NS

internal/resolver/export_test.go

@@ -31,9 +31,11 @@ func NewWithQueryTimeout(log *slog.Logger, timeout time.Duration) *Resolver {
 func (r *Resolver) FollowDelegation(
 	ctx context.Context,
 	domain string,
+	zone string,
+	nameservers []string,
 	servers []string,
 ) ([]string, error) {
-	return r.followDelegation(ctx, domain, servers)
+	return r.followDelegation(ctx, domain, zone, nameservers, servers)
 }
 
 // FindAuthoritativeNameserversFrom exports findAuthoritativeNameservers

internal/resolver/iterative.go

@@ -204,20 +204,24 @@ func glueIPs(nsNames []string, glue map[string][]net.IP) []string {
 	return ips
 }
 
-// followDelegation follows referrals from servers, the root servers, to
-// domain and returns the NS set of domain's delegation. When the servers
-// of the zone domain is in answer that domain does not exist, the error
-// is ErrNXDomain. When they answer that it has no delegation of its own,
+// followDelegation follows referrals to domain from servers, the
+// addresses of the servers of zone, and returns the NS set of domain's
+// delegation. nameservers are the names a referral gave the servers of
+// zone; the walk to a name's nameservers starts at the root servers, the
+// servers of zone ".", which no referral names. When the servers of the
+// zone domain is in answer that domain does not exist, the error is
+// ErrNXDomain. When they answer that it has no delegation of its own,
 // because it is not the zone's apex, the set is empty and there is no
-// error. Any other error means that no such answer came.
+// error. Any other error means that no such answer came; it is a
+// *NoAnswerError when the servers that did not answer are those of a
+// zone a referral named.
 func (r *Resolver) followDelegation(
 	ctx context.Context,
 	domain string,
+	zone string,
+	nameservers []string,
 	servers []string,
 ) ([]string, error) {
-	// servers are the root servers, the servers of zone ".".
-	zone := "."
-
 	var withoutAddresses []string
 
 	for range maxDelegation {
@@ -228,6 +232,12 @@ func (r *Resolver) followDelegation(
 		resp, err := r.queryZone(
 			ctx, servers, withoutAddresses, zone, domain, dns.TypeNS, 0,
 		)
+
+		// A query that ctx cut short says nothing about the servers.
+		if err != nil && len(nameservers) > 0 && checkCtx(ctx) == nil {
+			return nil, &NoAnswerError{Nameservers: nameservers, Err: err}
+		}
+
 		if err != nil {
 			return nil, err
 		}
@@ -256,6 +266,7 @@ func (r *Resolver) followDelegation(
 			return r.resolveNSIterative(ctx, domain)
 		}
 
+		nameservers = authNS
 		servers, withoutAddresses = referralNameservers(resp)
 		zone = referralZone(resp)
 	}
@@ -626,7 +637,8 @@ func (r *Resolver) resolveARecord(
 // them. When the servers asked answer that the name has no delegation
 // of its own, or does not exist, it tries each parent name in turn, so
 // it returns the nameservers of the zone the name is in. When they do
-// not answer, it returns the error and tries no parent name.
+// not answer, it returns the error and tries no parent name; when a
+// referral named them, the error is a *NoAnswerError naming them.
 func (r *Resolver) FindAuthoritativeNameservers(
 	ctx context.Context,
 	domain string,
@@ -655,7 +667,7 @@ func (r *Resolver) findAuthoritativeNameservers(
 
 		candidate := strings.Join(labels[i:], ".") + "."
 
-		nsNames, err := r.followDelegation(ctx, candidate, servers)
+		nsNames, err := r.followDelegation(ctx, candidate, ".", nil, servers)
 		if err != nil && !errors.Is(err, ErrNXDomain) {
 			return nil, err
 		}
@@ -1067,7 +1079,7 @@ func (r *Resolver) LookupNS(
 	}
 
 	return r.followDelegation(
-		ctx, dns.Fqdn(strings.ToLower(domain)), rootServerList(),
+		ctx, dns.Fqdn(strings.ToLower(domain)), ".", nil, rootServerList(),
 	)
 }
 

internal/resolver/resolver_test.go

@@ -121,7 +121,9 @@ func TestFindAuthoritativeNameservers_DelegatedSubdomain(
 // delegation of its own, so the lookup returns that walk's error, about
 // www.google.com, and tries no parent name: trying google.com and com
 // would end in ErrNoNameservers, or in the error of a walk for one of
-// them.
+// them. The servers that did not answer are those the walk started at,
+// which no referral named, so the error is no NoAnswerError, as when
+// the host running dnswatcher has lost its network.
 func TestFindAuthoritativeNameservers_NoAnswer(t *testing.T) {
 	t.Parallel()
 
@@ -134,6 +136,9 @@ func TestFindAuthoritativeNameservers_NoAnswer(t *testing.T) {
 	require.NotErrorIs(t, err, resolver.ErrNoNameservers)
 	assert.Contains(t, err.Error(), "query www.google.com. @"+noAnswerAddress)
 	assert.Empty(t, nameservers)
+
+	var noAnswer *resolver.NoAnswerError
+	assert.NotErrorAs(t, err, &noAnswer)
 }
 
 func TestFindAuthoritativeNameservers_ReturnsSorted(
@@ -954,12 +959,34 @@ func TestFollowDelegation_NoAnswer(t *testing.T) {
 	r := resolver.NewWithQueryTimeout(slog.Default(), 100*time.Millisecond)
 
 	nameservers, err := r.FollowDelegation(
-		t.Context(), "google.com.", []string{noAnswerAddress},
+		t.Context(), "google.com.", ".", nil, []string{noAnswerAddress},
 	)
 	require.Error(t, err)
 	assert.Empty(t, nameservers)
 }
 
+// TestFollowDelegation_NoAnswerAfterReferral starts the walk for
+// www.example.net at the servers of example.net as a referral names
+// them, ns1.example.net and ns2.example.net, at 192.0.2.1, where nothing
+// answers. The error is a NoAnswerError naming them, which the watcher
+// compares with the nameservers it saved for the name.
+func TestFollowDelegation_NoAnswerAfterReferral(t *testing.T) {
+	t.Parallel()
+
+	r := resolver.NewWithQueryTimeout(slog.Default(), 100*time.Millisecond)
+	nameservers := []string{"ns1.example.net.", "ns2.example.net."}
+
+	_, err := r.FollowDelegation(
+		t.Context(), "www.example.net.", "example.net.", nameservers,
+		[]string{noAnswerAddress},
+	)
+
+	var noAnswer *resolver.NoAnswerError
+	require.ErrorAs(t, err, &noAnswer)
+	assert.Equal(t, nameservers, noAnswer.Nameservers)
+	assert.Contains(t, err.Error(), "query www.example.net. @"+noAnswerAddress)
+}
+
 // TestResolveNSIterative_NoDelegationOfItsOwn walks to the nameservers
 // of www.google.com as the fallback walk does. As in
 // TestLookupNS_NoDelegationOfItsOwn, the set is empty, with no error.

internal/server/export_test.go

@@ -13,8 +13,8 @@ import (
 const RequestTimeout time.Duration = requestTimeout
 
 // SetListenPort overrides the port Run binds. A test uses it to hand
-// Run an unbindable port so ListenAndServe fails immediately and Run
-// returns after storing its http.Server.
+// Run an unbindable port, so that Run stores its http.Server and then
+// returns an error without listening.
 func SetListenPort(s *Server, port int) {
 	s.port = port
 }

internal/server/server.go

@@ -6,6 +6,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"net"
 	"net/http"
 	"time"
 
@@ -112,7 +113,7 @@ func New(
 	}
 
 	lifecycle.Append(fx.Hook{
-		OnStart: func(_ context.Context) error {
+		OnStart: func(ctx context.Context) error {
 			srv.startupTime = time.Now()
 
 			err := srv.enableSentry()
@@ -120,9 +121,7 @@ func New(
 				return err
 			}
 
-			go srv.Run()
-
-			return nil
+			return srv.Run(ctx)
 		},
 		OnStop: func(ctx context.Context) error {
 			return srv.Shutdown(ctx)
@@ -149,8 +148,10 @@ func newHTTPServer(
 	}
 }
 
-// Run starts the HTTP server.
-func (s *Server) Run() {
+// Run binds the listen port, then serves HTTP on it in a new goroutine
+// and returns. A port that cannot be bound is returned as an error, so
+// that startup stops instead of running without the HTTP server.
+func (s *Server) Run(ctx context.Context) error {
 	s.SetupRoutes()
 
 	listenAddr := fmt.Sprintf(":%d", s.port)
@@ -158,10 +159,21 @@ func (s *Server) Run() {
 
 	s.log.Info("http server starting", "addr", listenAddr)
 
-	err := s.httpServer.ListenAndServe()
-	if err != nil && !errors.Is(err, http.ErrServerClosed) {
-		s.log.Error("http server error", "error", err)
+	lc := &net.ListenConfig{}
+
+	listener, err := lc.Listen(ctx, "tcp", listenAddr)
+	if err != nil {
+		return fmt.Errorf("starting http server: %w", err)
 	}
+
+	go func() {
+		err := s.httpServer.Serve(listener)
+		if err != nil && !errors.Is(err, http.ErrServerClosed) {
+			s.log.Error("http server error", "error", err)
+		}
+	}()
+
+	return nil
 }
 
 // Shutdown gracefully shuts down the server, then sends the error

internal/server/server_test.go

@@ -1,6 +1,8 @@
 package server_test
 
 import (
+	"net"
+	"strings"
 	"testing"
 
 	"github.com/spf13/viper"
@@ -58,15 +60,13 @@ func buildServer(t *testing.T) *server.Server {
 }
 
 // TestRunWiresSocketTimeouts pins that the http.Server the running
-// server actually serves — the one Run builds and hands to
-// ListenAndServe — carries every socket-level timeout, plus the two
+// server actually serves — the one Run builds and serves on the port
+// it binds — carries every socket-level timeout, plus the two
 // relationships the values must satisfy.
 //
-// Run is driven to completion with an unbindable port: it builds and
-// stores s.httpServer, then ListenAndServe fails at once and Run
-// returns without ever listening. The assertions run in the same
-// goroutine after Run returns, so reading s.httpServer is free of any
-// data race. Nothing here measures elapsed time.
+// Run is given an unbindable port: it builds and stores s.httpServer,
+// then the bind fails and Run returns the error without ever
+// listening. Nothing here measures elapsed time.
 //
 // ReadTimeout must be at least ReadHeaderTimeout. net/http reads the
 // headers under ReadHeaderTimeout, then sets the read deadline for the
@@ -84,7 +84,10 @@ func TestRunWiresSocketTimeouts(t *testing.T) {
 	srv := buildServer(t)
 	server.SetListenPort(srv, -1)
 
-	srv.Run()
+	err := srv.Run(t.Context())
+	if err == nil {
+		t.Fatal("Run bound port -1")
+	}
 
 	hs := server.HTTPServerOf(srv)
 	if hs == nil {
@@ -133,3 +136,38 @@ func TestRunWiresSocketTimeouts(t *testing.T) {
 		)
 	}
 }
+
+func TestPortInUseStopsStartup(t *testing.T) {
+	// Bound on every address, as Run binds, so that Run's bind fails.
+	lc := &net.ListenConfig{}
+
+	taken, err := lc.Listen(t.Context(), "tcp", ":0")
+	if err != nil {
+		t.Fatalf("binding a port for the test: %v", err)
+	}
+
+	defer func() { _ = taken.Close() }()
+
+	_, port, err := net.SplitHostPort(taken.Addr().String())
+	if err != nil {
+		t.Fatalf("reading the bound port: %v", err)
+	}
+
+	viper.Reset()
+	t.Setenv("DNSWATCHER_TARGETS", "example.com")
+	t.Setenv("DNSWATCHER_DATA_DIR", t.TempDir())
+	t.Setenv("PORT", port)
+
+	app := newServerApp(fx.Invoke(func(*server.Server) {}))
+
+	err = app.Start(t.Context())
+	if err == nil {
+		_ = app.Stop(t.Context())
+
+		t.Fatal("startup succeeded on a port that is already bound")
+	}
+
+	if !strings.Contains(err.Error(), ":"+port) {
+		t.Errorf("startup error does not name port %s: %v", port, err)
+	}
+}

internal/state/state.go

@@ -4,6 +4,7 @@ package state
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"maps"
@@ -215,6 +216,24 @@ func (s *State) Load() error {
 		return fmt.Errorf("parsing state file: %w", err)
 	}
 
+	// A section the file leaves out, or holds as null, loads empty: the
+	// setters write into these maps, and writing into a nil map panics.
+	if snapshot.Domains == nil {
+		snapshot.Domains = make(map[string]*DomainState)
+	}
+
+	if snapshot.Hostnames == nil {
+		snapshot.Hostnames = make(map[string]*HostnameState)
+	}
+
+	if snapshot.Ports == nil {
+		snapshot.Ports = make(map[string]*PortState)
+	}
+
+	if snapshot.Certificates == nil {
+		snapshot.Certificates = make(map[string]*CertificateState)
+	}
+
 	// A state file saved before each record value was stored once can
 	// hold a hostname's CNAME once for every record type asked for.
 	// Each value is kept once, so the first check does not see a
@@ -253,10 +272,20 @@ func (s *State) Save() error {
 		return fmt.Errorf("creating data directory: %w", err)
 	}
 
-	// Atomic write: write to temp file, then rename
+	// Atomic write: write to temp file, then rename. The temp file is
+	// synced before the rename and the directory after it, or a crash
+	// can leave the renamed state file empty.
 	tmpPath := path + ".tmp"
 
-	err = os.WriteFile(tmpPath, data, filePermissions)
+	//nolint:gosec // path is from trusted config
+	tmpFile, err := os.OpenFile(
+		tmpPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, filePermissions,
+	)
+	if err == nil {
+		_, err = tmpFile.Write(data)
+		err = errors.Join(err, tmpFile.Sync(), tmpFile.Close())
+	}
+
 	if err != nil {
 		return fmt.Errorf("writing temp state file: %w", err)
 	}
@@ -266,6 +295,15 @@ func (s *State) Save() error {
 		return fmt.Errorf("renaming state file: %w", err)
 	}
 
+	dir, err := os.Open(filepath.Dir(path))
+	if err == nil {
+		err = errors.Join(dir.Sync(), dir.Close())
+	}
+
+	if err != nil {
+		return fmt.Errorf("syncing data directory: %w", err)
+	}
+
 	s.log.Debug("state saved to disk", "path", path)
 
 	return nil

internal/state/state_test.go

@@ -618,6 +618,53 @@ func TestSaveAtomicWrite(t *testing.T) {
 	}
 }
 
+// TestSaveOverLeftoverTempFile verifies that a temp file left by a
+// crash, longer than the state Save writes, leaves nothing of itself in
+// the state file, and that no temp file remains after Save.
+func TestSaveOverLeftoverTempFile(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	statePath := filepath.Join(dir, "state.json")
+	tmpPath := statePath + ".tmp"
+
+	err := os.WriteFile(tmpPath, []byte(strings.Repeat("x", 64*1024)), 0o600)
+	if err != nil {
+		t.Fatalf("writing leftover temp file: %v", err)
+	}
+
+	s := state.NewForTestWithDataDir(dir)
+
+	populateState(t, s)
+
+	err = s.Save()
+	if err != nil {
+		t.Fatalf("Save() error: %v", err)
+	}
+
+	//nolint:gosec // test file with known path
+	data, err := os.ReadFile(statePath)
+	if err != nil {
+		t.Fatalf("reading state file: %v", err)
+	}
+
+	var snap state.Snapshot
+
+	err = json.Unmarshal(data, &snap)
+	if err != nil {
+		t.Fatalf("state file is not complete JSON: %v", err)
+	}
+
+	if len(snap.Domains) != 2 {
+		t.Errorf("state file has %d domains, want 2", len(snap.Domains))
+	}
+
+	_, err = os.Stat(tmpPath)
+	if !os.IsNotExist(err) {
+		t.Error("temp file should not exist after save")
+	}
+}
+
 // TestLoadMissingFile verifies that loading a nonexistent file is not an error.
 func TestLoadMissingFile(t *testing.T) {
 	t.Parallel()
@@ -648,6 +695,58 @@ func TestLoadMissingFile(t *testing.T) {
 	}
 }
 
+// TestLoadFileWithoutSections loads a state file holding {}, and files
+// that each leave out one section. Every section loads as an empty map.
+// TestFirstCheckAfterLoadingFileWithoutSections in the watcher tests
+// runs the first check after such a load.
+func TestLoadFileWithoutSections(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name string
+		file string
+	}{
+		{"no sections", `{}`},
+		{"no domains", `{"hostnames": {}, "ports": {}, "certificates": {}}`},
+		{"no hostnames", `{"domains": {}, "ports": {}, "certificates": {}}`},
+		{"no ports", `{"domains": {}, "hostnames": {}, "certificates": {}}`},
+		{"no certificates", `{"domains": {}, "hostnames": {}, "ports": {}}`},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			dir := t.TempDir()
+
+			err := os.WriteFile(
+				filepath.Join(dir, "state.json"), []byte(tt.file), 0o600,
+			)
+			if err != nil {
+				t.Fatalf("writing state file: %v", err)
+			}
+
+			s := state.NewForTestWithDataDir(dir)
+
+			err = s.Load()
+			if err != nil {
+				t.Fatalf("Load() error: %v", err)
+			}
+
+			want := state.Snapshot{
+				Domains:      map[string]*state.DomainState{},
+				Hostnames:    map[string]*state.HostnameState{},
+				Ports:        map[string]*state.PortState{},
+				Certificates: map[string]*state.CertificateState{},
+			}
+
+			if got := s.GetSnapshot(); !reflect.DeepEqual(got, want) {
+				t.Errorf("loaded %#v, want %#v", got, want)
+			}
+		})
+	}
+}
+
 // TestLoadCorruptFile verifies that a corrupt state file returns an error.
 func TestLoadCorruptFile(t *testing.T) {
 	t.Parallel()

internal/watcher/cancelled_test.go

@@ -83,6 +83,30 @@ func TestCancelledCheckSavesNothing(t *testing.T) {
 	}
 }
 
+// TestStopBeforeFirstScanSendsNoStartupNotification runs a watcher with
+// the startup notification on and its context already cancelled, so the
+// first scan is cut short before its lookup is sent. The scan did not
+// complete, so no startup notification may be sent.
+func TestStopBeforeFirstScanSendsNoStartupNotification(t *testing.T) {
+	t.Parallel()
+
+	cfg := defaultTestConfig(t)
+	cfg.Hostnames = []string{testHost}
+	cfg.SendTestNotification = true
+
+	w, deps := newTestWatcher(t, cfg)
+
+	ctx, cancel := context.WithCancel(t.Context())
+	cancel()
+
+	w.Run(ctx)
+
+	notifications := deps.notifier.getNotifications()
+	if len(notifications) != 0 {
+		t.Errorf("sent %v, want no notifications", notifications)
+	}
+}
+
 // newLoggingWatcher returns a watcher for a domain and a hostname, with
 // the real resolver, that writes what it logs at warning level or above
 // into the returned buffer.

internal/watcher/export_test.go

@@ -122,6 +122,34 @@ func (w *Watcher) RunTLSChecks(ctx context.Context) {
 	w.runTLSChecks(ctx)
 }
 
+// CheckHostnameResult exports checkHostnameResult for testing.
+func (w *Watcher) CheckHostnameResult(
+	ctx context.Context,
+	hostname string,
+	results map[string]*resolver.NameserverResponse,
+	err error,
+) bool {
+	return w.checkHostnameResult(ctx, hostname, results, err)
+}
+
+// CheckDomainResult exports checkDomainResult for testing.
+func (w *Watcher) CheckDomainResult(
+	ctx context.Context,
+	domain string,
+	nameservers []string,
+	err error,
+) {
+	w.checkDomainResult(ctx, domain, nameservers, err)
+}
+
+// FailedNameservers exports failedNameservers for testing.
+func FailedNameservers(
+	err error,
+	prev *state.HostnameState,
+) map[string]*resolver.NameserverResponse {
+	return failedNameservers(err, prev)
+}
+
 // BuildHostnameState exports buildHostnameState for testing.
 func BuildHostnameState(
 	results map[string]*resolver.NameserverResponse,

internal/watcher/interfaces.go

@@ -20,7 +20,10 @@ type DNSResolver interface {
 	) ([]string, error)
 
 	// LookupAllRecords queries all record types for a hostname,
-	// returning each nameserver's response keyed by nameserver.
+	// returning each nameserver's response keyed by nameserver. When
+	// the walk to the nameservers reached a zone through a referral and
+	// none of that zone's nameservers answered, the error is a
+	// *resolver.NoAnswerError naming them.
 	LookupAllRecords(
 		ctx context.Context,
 		hostname string,

internal/watcher/ipv6_test.go

@@ -0,0 +1,83 @@
+package watcher_test
+
+import (
+	"maps"
+	"slices"
+	"testing"
+
+	"sneak.berlin/go/dnswatcher/internal/state"
+	"sneak.berlin/go/dnswatcher/internal/watcher"
+)
+
+// TestIPv6AddressInBrackets runs the port checks, not the watcher's
+// first, on hostname state built here for a hostname at an IPv6 address
+// whose port 443 the previous check found closed. It is open now. Then
+// it runs the TLS checks and the port checks again. The port and
+// certificate keys and the Port Change notification write the address
+// in brackets, so its last group cannot be read as the port, and the
+// second port checks keep the certificate entry, whose address they read
+// back from its key. When a last port check finds port 443 closed, the
+// certificate entry is removed. Nothing is looked up.
+func TestIPv6AddressInBrackets(t *testing.T) {
+	t.Parallel()
+
+	const aaaaType = "AAAA"
+
+	cfg := defaultTestConfig(t)
+	cfg.Hostnames = []string{host}
+
+	deps := newTestDeps(t, cfg)
+	w := watcher.NewForTest(
+		cfg, deps.state, nil,
+		deps.portChecker, deps.tlsChecker, deps.notifier,
+	)
+	w.SetFirstRun(false)
+
+	deps.state.SetHostnameState(host, saved(
+		map[string]*state.NameserverRecordState{
+			nsA: answered(map[string][]string{aaaaType: {"2001:db8::1"}}),
+		},
+	))
+	deps.state.SetPortState("[2001:db8::1]:443", &state.PortState{
+		Hostnames: []string{host},
+	})
+
+	w.CheckAllPorts(t.Context())
+	w.RunTLSChecks(t.Context())
+	w.CheckAllPorts(t.Context())
+
+	snap := deps.state.GetSnapshot()
+
+	gotPorts := slices.Sorted(maps.Keys(snap.Ports))
+	wantPorts := []string{"[2001:db8::1]:443", "[2001:db8::1]:80"}
+
+	if !slices.Equal(gotPorts, wantPorts) {
+		t.Errorf("port entries %v, want %v", gotPorts, wantPorts)
+	}
+
+	gotCerts := slices.Sorted(maps.Keys(snap.Certificates))
+	wantCerts := []string{"[2001:db8::1]:443:www.example.net"}
+
+	if !slices.Equal(gotCerts, wantCerts) {
+		t.Errorf("certificate entries %v, want %v", gotCerts, wantCerts)
+	}
+
+	title := "Port Change: [2001:db8::1]:443"
+	want := `Hostnames: www.example.net
+Address: [2001:db8::1]:443
+Port now open`
+
+	got := deps.notifier.getNotifications()
+	if len(got) != 1 || got[0].Title != title || got[0].Message != want {
+		t.Errorf("sent %v, want one %q with message:\n%s", got, title, want)
+	}
+
+	deps.portChecker.closed = true
+
+	w.CheckAllPorts(t.Context())
+
+	gotCerts = slices.Sorted(maps.Keys(deps.state.GetSnapshot().Certificates))
+	if len(gotCerts) != 0 {
+		t.Errorf("certificate entries %v with port 443 closed, want none", gotCerts)
+	}
+}

internal/watcher/message_test.go

@@ -5,7 +5,6 @@ import (
 	"strings"
 	"testing"
 
-	"sneak.berlin/go/dnswatcher/internal/config"
 	"sneak.berlin/go/dnswatcher/internal/state"
 	"sneak.berlin/go/dnswatcher/internal/watcher"
 )
@@ -74,21 +73,19 @@ b.ns.example.net.: 192.0.2.2`,
 func TestDomainRecordNotificationsNameTheDomain(t *testing.T) {
 	t.Parallel()
 
-	notifier := &mockNotifier{}
-	w := watcher.NewForTest(
-		&config.Config{Domains: []string{domain}},
-		nil, nil, nil, nil, notifier,
-	)
+	cfg := defaultTestConfig(t)
+	cfg.Domains = []string{domain}
+
+	deps := newTestDeps(t, cfg)
+	w := watcher.NewForTest(cfg, deps.state, nil, nil, nil, deps.notifier)
 
 	// nsA's address changes, which also makes it differ from nsC; nsB
-	// fails; nsC answers again; nsD is gone.
-	nsD := "d.ns.example.net."
+	// fails; nsC answers again.
 	w.DetectHostnameChanges(t.Context(), domain,
 		saved(map[string]*state.NameserverRecordState{
 			nsA: answered(map[string][]string{"A": {ip1}}),
 			nsB: answered(map[string][]string{"A": {ip1}}),
 			nsC: failed(),
-			nsD: answered(map[string][]string{"A": {ip1}}),
 		}),
 		saved(map[string]*state.NameserverRecordState{
 			nsA: answered(map[string][]string{"A": {ip2}}),
@@ -102,18 +99,17 @@ func TestDomainRecordNotificationsNameTheDomain(t *testing.T) {
 		t.Context(), domain, cnameState(ip1), cnameState(ip2),
 	)
 
-	// NS Failure is sent for nsB failing and for nsD being gone.
 	want := map[string]int{
 		"Record Change":        1,
 		"Inconsistency":        1,
-		"NS Failure":           2,
+		"NS Failure":           1,
 		"NS Recovery":          1,
 		"CNAME Address Change": 1,
 	}
 
 	sent := make(map[string]int)
 
-	for _, n := range notifier.getNotifications() {
+	for _, n := range deps.notifier.getNotifications() {
 		kind, _, _ := strings.Cut(n.Title, ":")
 		sent[kind]++
 
@@ -128,6 +124,89 @@ func TestDomainRecordNotificationsNameTheDomain(t *testing.T) {
 	}
 }
 
+// nsB, which answered, and nsC, which did not, are no longer among the
+// nameservers. One NS Change lists both, for the hostname and for the
+// configured apex domain with no delegation of its own. Nothing is sent
+// for the domain whose delegation on the previous check listed them:
+// the check of its delegation sends its NS Change.
+func TestRemovedNameserversAreAnNSChange(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name   string
+		target string
+		// The domain's nameservers saved by the previous check.
+		delegation []string
+		// The NS Change sent, or "" for none.
+		want string
+	}{
+		{
+			name:   "hostname",
+			target: host,
+			want: `Hostname: www.example.net
+Removed: b.ns.example.net., c.ns.example.net.`,
+		},
+		{
+			name:       "domain with no delegation of its own",
+			target:     domain,
+			delegation: []string{},
+			want: `Domain: example.net
+Removed: b.ns.example.net., c.ns.example.net.`,
+		},
+		{
+			name:       "domain whose delegation listed them",
+			target:     domain,
+			delegation: []string{nsA, nsB, nsC},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			cfg := defaultTestConfig(t)
+			cfg.Domains = []string{domain}
+
+			deps := newTestDeps(t, cfg)
+			w := watcher.NewForTest(
+				cfg, deps.state, nil, nil, nil, deps.notifier,
+			)
+
+			deps.state.SetDomainState(domain, &state.DomainState{
+				Nameservers: tt.delegation,
+			})
+
+			prev := saved(map[string]*state.NameserverRecordState{
+				nsA: answered(map[string][]string{"A": {ip1}}),
+				nsB: answered(map[string][]string{"A": {ip1}}),
+				nsC: failed(),
+			})
+			current := saved(map[string]*state.NameserverRecordState{
+				nsA: answered(map[string][]string{"A": {ip1}}),
+			})
+
+			w.DetectHostnameChanges(t.Context(), tt.target, prev, current)
+
+			got := deps.notifier.getNotifications()
+
+			if tt.want == "" {
+				if len(got) != 0 {
+					t.Errorf("sent %v, want nothing", got)
+				}
+
+				return
+			}
+
+			title := "NS Change: " + tt.target
+			if len(got) != 1 || got[0].Title != title ||
+				got[0].Message != tt.want {
+				t.Errorf("sent %v, want one %q with message:\n%s",
+					got, title, tt.want)
+			}
+		})
+	}
+}
+
 // The startup notification counts the configured domains and hostnames,
 // although the state's hostnames also hold the apex domain's own
 // records. Nothing is looked up: the watcher has no resolver.

internal/watcher/nsfailure_test.go

@@ -2,6 +2,7 @@ package watcher_test
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"log/slog"
 	"strings"
@@ -14,6 +15,10 @@ import (
 	"sneak.berlin/go/dnswatcher/internal/watcher"
 )
 
+// errNoReply is why a walk that got no reply from a zone's servers
+// failed.
+var errNoReply = errors.New("all servers failed: i/o timeout")
+
 // answered is what a check saves for a nameserver that answered with
 // these records.
 func answered(records map[string][]string) *state.NameserverRecordState {
@@ -183,6 +188,134 @@ func TestNSFailureAlertNamesHostnameNameserverAndReason(t *testing.T) {
 	}
 }
 
+// TestZoneNameserversAllStopAnswering runs three checks of host from
+// the point where its lookup has returned. On the first, nsA and nsB,
+// the nameservers of its zone, answer. On the second, the lookup fails
+// with the error the walk to them returns when neither answers (see
+// the resolver's TestFollowDelegation_NoAnswerAfterReferral), given
+// here because no live zone has nameservers that never answer. For a
+// hostname it is the lookup of its records that fails, and for a domain
+// with no delegation of its own the lookup of its delegation. Both are
+// saved as failed, with the reason, and each gets an NS Failure. On the
+// third, both answer, and each gets an NS Recovery. A domain's records
+// are checked as a hostname's, so its first and third checks are too.
+func TestZoneNameserversAllStopAnswering(t *testing.T) {
+	t.Parallel()
+
+	records := map[string][]string{"A": {ip1}}
+	bothAnswer := map[string]*resolver.NameserverResponse{
+		nsA: response(records), nsB: response(records),
+	}
+	noAnswer := &resolver.NoAnswerError{
+		Nameservers: []string{nsA, nsB}, Err: errNoReply,
+	}
+
+	tests := []struct {
+		name     string
+		isDomain bool
+	}{
+		{"hostname", false},
+		{"domain with no delegation of its own", true},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			cfg := defaultTestConfig(t)
+			if tt.isDomain {
+				cfg.Domains = []string{host}
+			} else {
+				cfg.Hostnames = []string{host}
+			}
+
+			// The watcher is given each lookup's result, so it has no
+			// resolver.
+			deps := newTestDeps(t, cfg)
+			w := watcher.NewForTest(
+				cfg, deps.state, nil,
+				deps.portChecker, deps.tlsChecker, deps.notifier,
+			)
+			w.SetFirstRun(false)
+
+			w.CheckHostnameResult(t.Context(), host, bothAnswer, nil)
+
+			if tt.isDomain {
+				w.CheckDomainResult(t.Context(), host, nil, noAnswer)
+			} else {
+				w.CheckHostnameResult(t.Context(), host, nil, noAnswer)
+			}
+
+			current, _ := deps.state.GetHostnameState(host)
+
+			for _, ns := range []string{nsA, nsB} {
+				got := current.RecordsByNameserver[ns]
+				if got.Status != failed().Status || got.Error != errNoReply.Error() {
+					t.Errorf(
+						"%s saved with status %q, error %q; want %q, %q",
+						ns, got.Status, got.Error, failed().Status, errNoReply,
+					)
+				}
+			}
+
+			if n := countNotifications(deps, "NS Failure: "+host); n != 2 {
+				t.Errorf("sent %d NS Failure notifications, want 2", n)
+			}
+
+			w.CheckHostnameResult(t.Context(), host, bothAnswer, nil)
+
+			if n := countNotifications(deps, "NS Recovery: "+host); n != 2 {
+				t.Errorf("sent %d NS Recovery notifications, want 2", n)
+			}
+		})
+	}
+}
+
+// TestFailedNameserversOnlyForThoseOfThePreviousCheck gives
+// FailedNameservers lookup errors after a check in which nsA and nsB
+// answered. Only an error that names both makes them failed.
+func TestFailedNameserversOnlyForThoseOfThePreviousCheck(t *testing.T) {
+	t.Parallel()
+
+	records := map[string][]string{"A": {ip1}}
+	bothAnswered := saved(map[string]*state.NameserverRecordState{
+		nsA: answered(records), nsB: answered(records),
+	})
+	noAnswer := func(nameservers ...string) error {
+		return &resolver.NoAnswerError{Nameservers: nameservers, Err: errNoReply}
+	}
+
+	tests := []struct {
+		name       string
+		err        error
+		prev       *state.HostnameState
+		wantFailed bool
+	}{
+		{"both did not answer", noAnswer(nsA, nsB), bothAnswered, true},
+		{
+			"the servers of a parent zone did not answer",
+			noAnswer(nsC), bothAnswered, false,
+		},
+		{
+			"the zone's delegation lists only nsA now",
+			noAnswer(nsA), bothAnswered, false,
+		},
+		{"no root server answered", errNoReply, bothAnswered, false},
+		{"no previous check", noAnswer(nsA, nsB), nil, false},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			got := watcher.FailedNameservers(tt.err, tt.prev)
+			if (got != nil) != tt.wantFailed {
+				t.Errorf("returned %v, want failed: %v", got, tt.wantFailed)
+			}
+		})
+	}
+}
+
 // TestNameserverThatNeverAnswers asks a nameserver address where
 // nothing answers, 192.0.2.1, and checks what the watcher saves for it.
 // The deadline outlasts the resolver's first two-second try, as in the

internal/watcher/removed_test.go

@@ -2,7 +2,9 @@ package watcher_test
 
 import (
 	"maps"
+	"net"
 	"slices"
+	"strings"
 	"testing"
 
 	"sneak.berlin/go/dnswatcher/internal/state"
@@ -172,7 +174,7 @@ func TestCertificateStateForAnAddressGone(t *testing.T) {
 					"A": {ip1}, "AAAA": {ip6},
 				}),
 			}),
-			[]string{ip1 + ":443:" + host, ip6 + ":443:" + host},
+			[]string{ip1 + ":443:" + host, "[" + ip6 + "]:443:" + host},
 		},
 		{
 			"no nameserver answered",
@@ -182,7 +184,7 @@ func TestCertificateStateForAnAddressGone(t *testing.T) {
 			[]string{
 				ip1 + ":443:" + host,
 				ip2 + ":443:" + host,
-				ip6 + ":443:" + host,
+				"[" + ip6 + "]:443:" + host,
 			},
 		},
 	}
@@ -205,7 +207,8 @@ func TestCertificateStateForAnAddressGone(t *testing.T) {
 
 			for _, ip := range []string{ip1, ip2, ip6} {
 				deps.state.SetCertificateState(
-					ip+":443:"+host, &state.CertificateState{Status: "ok"},
+					net.JoinHostPort(ip, "443")+":"+host,
+					&state.CertificateState{Status: "ok"},
 				)
 			}
 
@@ -222,3 +225,67 @@ func TestCertificateStateForAnAddressGone(t *testing.T) {
 		})
 	}
 }
+
+// TestCertificateStateForAPortClosed runs the port checks on hostname
+// and port state built here for a configured hostname at ip1, whose
+// port 443 the previous check found open and whose certificate entry
+// is ok. The port is closed now: the certificate entry is removed, and
+// only the Port Change is notified. When the port opens again, the next
+// TLS check saves the certificate as new and notifies nothing about it.
+// Nothing is looked up.
+func TestCertificateStateForAPortClosed(t *testing.T) {
+	t.Parallel()
+
+	cfg := defaultTestConfig(t)
+	cfg.Hostnames = []string{host}
+
+	deps := newTestDeps(t, cfg)
+	w := watcher.NewForTest(
+		cfg, deps.state, nil,
+		deps.portChecker, deps.tlsChecker, deps.notifier,
+	)
+	w.SetFirstRun(false)
+
+	certKey := ip1 + ":443:" + host
+
+	deps.state.SetHostnameState(host, saved(
+		map[string]*state.NameserverRecordState{
+			nsA: answered(map[string][]string{"A": {ip1}}),
+		},
+	))
+	deps.state.SetPortState(ip1+":443", &state.PortState{
+		Open: true, Hostnames: []string{host},
+	})
+	deps.state.SetCertificateState(certKey, &state.CertificateState{
+		CommonName: host, Status: "ok",
+	})
+	deps.portChecker.closed = true
+
+	w.CheckAllPorts(t.Context())
+
+	if _, ok := deps.state.GetCertificateState(certKey); ok {
+		t.Errorf("certificate entry %s kept with port 443 closed", certKey)
+	}
+
+	sent := deps.notifier.getNotifications()
+	if title := "Port Change: " + ip1 + ":443"; len(sent) != 1 ||
+		sent[0].Title != title {
+		t.Errorf("sent %v, want one %q", sent, title)
+	}
+
+	deps.portChecker.closed = false
+
+	w.CheckAllPorts(t.Context())
+	w.RunTLSChecks(t.Context())
+
+	cert, ok := deps.state.GetCertificateState(certKey)
+	if !ok || cert.Status != "ok" {
+		t.Errorf("certificate entry %s after port 443 opened: %+v", certKey, cert)
+	}
+
+	for _, n := range deps.notifier.getNotifications() {
+		if strings.HasPrefix(n.Title, "TLS") {
+			t.Errorf("sent %v, want no TLS notification", n)
+		}
+	}
+}

internal/watcher/watcher.go

@@ -6,8 +6,10 @@ import (
 	"fmt"
 	"log/slog"
 	"maps"
+	"net"
 	"slices"
 	"sort"
+	"strconv"
 	"strings"
 	"time"
 
@@ -133,7 +135,12 @@ func (w *Watcher) Run(ctx context.Context) {
 
 	w.cleanupRemovedTargets()
 	w.RunOnce(ctx)
-	w.maybeSendTestNotification(ctx)
+
+	// RunOnce returns as usual when shutdown cuts the scan short, and
+	// the startup notification says the scan completed.
+	if ctx.Err() == nil {
+		w.maybeSendTestNotification(ctx)
+	}
 
 	dnsTicker := time.NewTicker(w.config.DNSInterval)
 	tlsTicker := time.NewTicker(w.config.TLSInterval)
@@ -284,12 +291,25 @@ func (w *Watcher) logFailedLookup(
 	w.log.Error(msg, args...)
 }
 
+// checkDomain looks up domain's delegation and checks domain with it
+// (see checkDomainResult).
 func (w *Watcher) checkDomain(
 	ctx context.Context,
 	domain string,
 ) {
 	nameservers, err := w.resolver.LookupNS(ctx, domain)
 
+	w.checkDomainResult(ctx, domain, nameservers, err)
+}
+
+// checkDomainResult finishes a check of domain from nameservers and err,
+// what the lookup of its delegation returned.
+func (w *Watcher) checkDomainResult(
+	ctx context.Context,
+	domain string,
+	nameservers []string,
+	err error,
+) {
 	// A domain that does not exist has no nameservers.
 	nxdomain := errors.Is(err, resolver.ErrNXDomain)
 	if nxdomain {
@@ -304,6 +324,12 @@ func (w *Watcher) checkDomain(
 			"error", err,
 		)
 
+		// For a domain with no delegation of its own, the servers that
+		// did not answer can be those of the zone it is in, at which
+		// its records are asked. They are then saved as failed, as a
+		// hostname's are.
+		w.checkHostnameResult(ctx, domain, nil, err)
+
 		return
 	}
 
@@ -322,6 +348,20 @@ func (w *Watcher) checkDomain(
 		ctx, nameservers, prevAddresses,
 	)
 
+	// A domain that does not exist has no records of its own: none are
+	// asked for, and those saved by an earlier check are removed.
+	// Otherwise the apex domain's records are also checked and saved as
+	// a hostname's, so that the port and TLS checks find its addresses.
+	// Notifications about them name it as a domain (see nameLine). When
+	// their lookup fails, the check fails as when the NS lookup does, so
+	// the saved delegation and the saved records come from the same
+	// check, as detectNSDisappearances needs.
+	if nxdomain {
+		w.state.DeleteHostnameState(domain)
+	} else if !w.checkHostname(ctx, domain) {
+		return
+	}
+
 	if hasPrev && !w.firstRun {
 		w.detectNSChanges(ctx, domain, prev.Nameservers, nameservers)
 		w.detectNSAddressChanges(ctx, domain, prevAddresses, addresses)
@@ -333,19 +373,6 @@ func (w *Watcher) checkDomain(
 		NXDomain:            nxdomain,
 		LastChecked:         now,
 	})
-
-	// A domain that does not exist has no records of its own: none are
-	// asked for, and those saved by an earlier check are removed.
-	if nxdomain {
-		w.state.DeleteHostnameState(domain)
-
-		return
-	}
-
-	// The apex domain's records are also checked and saved as a
-	// hostname's, so that the port and TLS checks find its addresses.
-	// Notifications about them name it as a domain (see nameLine).
-	w.checkHostname(ctx, domain)
 }
 
 func (w *Watcher) detectNSChanges(
@@ -456,10 +483,12 @@ func (w *Watcher) detectNSAddressChanges(
 	}
 }
 
+// checkHostname looks up hostname's records and checks them (see
+// checkHostnameResult). It reports whether it saved them.
 func (w *Watcher) checkHostname(
 	ctx context.Context,
 	hostname string,
-) {
+) bool {
 	results, err := w.resolver.LookupAllRecords(ctx, hostname)
 	if err != nil {
 		w.logFailedLookup(
@@ -468,15 +497,70 @@ func (w *Watcher) checkHostname(
 			"hostname", hostname,
 			"error", err,
 		)
-
-		return
 	}
 
+	return w.checkHostnameResult(ctx, hostname, results, err)
+}
+
+// checkHostnameResult finishes a check of hostname from results and err,
+// what the lookup of its records returned, and reports whether it saved
+// them. When the lookup failed, it saves nothing, unless the nameservers
+// that did not answer are those of the previous check (see
+// failedNameservers).
+func (w *Watcher) checkHostnameResult(
+	ctx context.Context,
+	hostname string,
+	results map[string]*resolver.NameserverResponse,
+	err error,
+) bool {
 	prev, _ := w.state.GetHostnameState(hostname)
 
+	if err != nil {
+		results = failedNameservers(err, prev)
+		if results == nil {
+			return false
+		}
+	}
+
 	w.updateHostnameState(
 		ctx, hostname, buildHostnameState(results, prev, time.Now().UTC()),
 	)
+
+	return true
+}
+
+// failedNameservers returns a failed response for each nameserver in
+// prev, a name's state from the previous check, with err as the reason,
+// when err says that the walk to the name's nameservers reached their
+// zone and none of them answered. For any other error, as when the walk
+// got no answer before it reached their zone, or when prev is nil, it
+// returns nil.
+func failedNameservers(
+	err error,
+	prev *state.HostnameState,
+) map[string]*resolver.NameserverResponse {
+	var noAnswer *resolver.NoAnswerError
+	if !errors.As(err, &noAnswer) || prev == nil {
+		return nil
+	}
+
+	saved := slices.Sorted(maps.Keys(prev.RecordsByNameserver))
+	if !slices.Equal(noAnswer.Nameservers, saved) {
+		return nil
+	}
+
+	results := make(map[string]*resolver.NameserverResponse)
+
+	for _, ns := range saved {
+		results[ns] = &resolver.NameserverResponse{
+			Nameserver: ns,
+			Records:    make(map[string][]string),
+			Status:     resolver.StatusError,
+			Error:      err.Error(),
+		}
+	}
+
+	return results
 }
 
 // updateHostnameState finishes a check of hostname from newState, built
@@ -772,28 +856,53 @@ func (w *Watcher) detectRecordChanges(
 	}
 }
 
+// detectNSDisappearances sends one NS Change listing the nameservers of
+// the previous check that are no longer those of the zone hostname is
+// in. For a configured apex domain it leaves out those of the domain's
+// saved delegation, which detectNSChanges notifies when they are removed
+// from it. checkDomain saves a new delegation only after this runs, and
+// only together with the records, so the saved delegation and the
+// nameservers in prev come from the same check.
 func (w *Watcher) detectNSDisappearances(
 	ctx context.Context,
 	hostname string,
 	prev, current *state.HostnameState,
 ) {
-	for ns, prevNS := range prev.RecordsByNameserver {
-		if _, ok := current.RecordsByNameserver[ns]; ok || prevNS.Status != statusOK {
-			continue
+	var delegation []string
+
+	if w.isDomain(hostname) {
+		if domainState, ok := w.state.GetDomainState(hostname); ok {
+			delegation = domainState.Nameservers
 		}
-
-		msg := fmt.Sprintf(
-			"%s\nNameserver: %s disappeared",
-			w.nameLine(hostname), ns,
-		)
-
-		w.notify.SendNotification(
-			ctx,
-			"NS Failure: "+hostname,
-			msg,
-			"error",
-		)
 	}
+
+	var removed []string
+
+	for ns := range prev.RecordsByNameserver {
+		_, ok := current.RecordsByNameserver[ns]
+		if !ok && !slices.Contains(delegation, ns) {
+			removed = append(removed, ns)
+		}
+	}
+
+	if len(removed) == 0 {
+		return
+	}
+
+	slices.Sort(removed)
+
+	msg := fmt.Sprintf(
+		"%s\nRemoved: %s",
+		w.nameLine(hostname),
+		strings.Join(removed, ", "),
+	)
+
+	w.notify.SendNotification(
+		ctx,
+		"NS Change: "+hostname,
+		msg,
+		"warning",
+	)
 }
 
 // detectNSFailures notifies when a nameserver that answered on the
@@ -943,7 +1052,8 @@ func (w *Watcher) checkAllPorts(ctx context.Context) {
 
 	// Phase 3: Remove port state entries that no longer have
 	// any hostname referencing them, and certificate entries for
-	// an address their name no longer has.
+	// an address whose port 443 is closed or that their name no
+	// longer has.
 	w.cleanupStalePorts(associations)
 	w.cleanupStaleCertificates()
 }
@@ -964,7 +1074,7 @@ func (w *Watcher) buildPortAssociations() map[string][]string {
 		ips := w.collectIPs(name)
 		for _, ip := range ips {
 			for _, port := range monitoredPorts {
-				key := fmt.Sprintf("%s:%d", ip, port)
+				key := net.JoinHostPort(ip, strconv.Itoa(port))
 				if assoc[key] == nil {
 					assoc[key] = make(map[string]bool)
 				}
@@ -989,18 +1099,15 @@ func (w *Watcher) buildPortAssociations() map[string][]string {
 	return result
 }
 
-// parsePortKey splits an "ip:port" key into its components.
+// parsePortKey splits an "ip:port" key, written by net.JoinHostPort,
+// into its components.
 func parsePortKey(key string) (string, int) {
-	lastColon := strings.LastIndex(key, ":")
-	if lastColon < 0 {
+	ip, port, err := net.SplitHostPort(key)
+	if err != nil {
 		return key, 0
 	}
 
-	ip := key[:lastColon]
-
-	var p int
-
-	_, err := fmt.Sscanf(key[lastColon+1:], "%d", &p)
+	p, err := strconv.Atoi(port)
 	if err != nil {
 		return ip, 0
 	}
@@ -1030,13 +1137,21 @@ func (w *Watcher) cleanupStalePorts(
 }
 
 // cleanupStaleCertificates removes the certificate entries for an
-// address their name no longer resolves to. An entry saved for a name
-// none of whose nameservers answered is kept: that name's addresses are
-// not known, not gone.
+// address whose port 443 was found closed, and for an address their
+// name no longer resolves to. An entry saved for a name none of whose
+// nameservers answered is kept unless its port is closed: that name's
+// addresses are not known, not gone.
 func (w *Watcher) cleanupStaleCertificates() {
 	for _, key := range w.state.GetAllCertificateKeys() {
 		ip, hostname := parseCertKey(key)
 
+		ps, ok := w.state.GetPortState(net.JoinHostPort(ip, strconv.Itoa(tlsPort)))
+		if ok && !ps.Open {
+			w.state.DeleteCertificateState(key)
+
+			continue
+		}
+
 		if slices.Contains(w.collectIPs(hostname), ip) ||
 			w.noNameserverAnswered(hostname) {
 			continue
@@ -1145,7 +1260,7 @@ func (w *Watcher) checkSinglePort(
 		return
 	}
 
-	key := fmt.Sprintf("%s:%d", ip, port)
+	key := net.JoinHostPort(ip, strconv.Itoa(port))
 	now := time.Now().UTC()
 	prev, hasPrev := w.state.GetPortState(key)
 
@@ -1207,7 +1322,7 @@ func (w *Watcher) checkTLSForHostname(
 	ips := w.collectIPs(hostname)
 
 	for _, ip := range ips {
-		portKey := fmt.Sprintf("%s:%d", ip, tlsPort)
+		portKey := net.JoinHostPort(ip, strconv.Itoa(tlsPort))
 
 		ps, ok := w.state.GetPortState(portKey)
 		if !ok || !ps.Open {
@@ -1231,7 +1346,7 @@ func (w *Watcher) checkTLSCert(
 		return
 	}
 
-	certKey := fmt.Sprintf("%s:%d:%s", ip, tlsPort, hostname)
+	certKey := net.JoinHostPort(ip, strconv.Itoa(tlsPort)) + ":" + hostname
 	now := time.Now().UTC()
 	prev, hasPrev := w.state.GetCertificateState(certKey)
 

internal/watcher/watcher_test.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"fmt"
 	"log/slog"
+	"net"
 	"os"
 	"slices"
 	"strings"
@@ -139,6 +140,22 @@ func (m *mockNotifier) getNotifications() []notification {
 	return result
 }
 
+// cancelOnNotify is a mockNotifier that calls cancel at each
+// notification, as if shutdown cut the check short there.
+type cancelOnNotify struct {
+	*mockNotifier
+
+	cancel context.CancelFunc
+}
+
+func (n cancelOnNotify) SendNotification(
+	ctx context.Context,
+	title, message, priority string,
+) {
+	n.mockNotifier.SendNotification(ctx, title, message, priority)
+	n.cancel()
+}
+
 // --- Helpers to build a Watcher and run its checks against live DNS ---
 
 type testDeps struct {
@@ -461,6 +478,54 @@ func TestDomainPortAndTLSChecks(t *testing.T) {
 	}
 }
 
+// TestFirstCheckAfterLoadingFileWithoutSections loads a state file
+// holding {}, and one that leaves out the certificates section, and runs
+// the first check. It saves domain, hostname, port and certificate
+// entries; a section loaded as nil made it panic.
+func TestFirstCheckAfterLoadingFileWithoutSections(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name string
+		file string
+	}{
+		{"no sections", `{}`},
+		{"no certificates", `{"domains": {}, "hostnames": {}, "ports": {}}`},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			cfg := defaultTestConfig(t)
+			cfg.Domains = []string{testSmallDomain}
+
+			_, deps := runChecks(t, cfg, func(deps *testDeps) {
+				err := os.WriteFile(cfg.StatePath(), []byte(tt.file), 0o600)
+				if err != nil {
+					t.Fatalf("writing the state file: %v", err)
+				}
+
+				err = deps.state.Load()
+				if err != nil {
+					t.Fatalf("loading the state file: %v", err)
+				}
+			})
+
+			snap := deps.state.GetSnapshot()
+			if len(snap.Domains) == 0 || len(snap.Hostnames) == 0 ||
+				len(snap.Ports) == 0 || len(snap.Certificates) == 0 {
+				t.Errorf(
+					"saved %d domain, %d hostname, %d port and %d "+
+						"certificate entries, want at least one of each",
+					len(snap.Domains), len(snap.Hostnames),
+					len(snap.Ports), len(snap.Certificates),
+				)
+			}
+		})
+	}
+}
+
 func TestNSChangeDetection(t *testing.T) {
 	t.Parallel()
 
@@ -558,6 +623,11 @@ func TestDomainThatDoesNotExist(t *testing.T) {
 // nameservers and without nxdomain, and its records, asked at the
 // codeberg.page servers, are saved. Those are testSmallDomain's two
 // nameservers; github.io, the zone of the README's example, has eight.
+// The previous check saved a delegation to oldNS1, now withdrawn, and
+// records asked at oldNS1. A first check is cut short at its first
+// notification, as shutdown would, and a second runs in full: across
+// them the NS Change of its delegation removes oldNS1, and no other NS
+// Change lists it again.
 func TestDomainWithNoDelegationOfItsOwn(t *testing.T) {
 	t.Parallel()
 
@@ -569,9 +639,30 @@ func TestDomainWithNoDelegationOfItsOwn(t *testing.T) {
 	var deps *testDeps
 
 	livednstest.Retry(t, "watcher checks", func(ctx context.Context) error {
-		var w *watcher.Watcher
+		deps = newTestDeps(t, cfg)
 
-		w, deps = newTestWatcher(t, cfg)
+		firstCtx, cancel := context.WithCancel(ctx)
+		defer cancel()
+
+		w := watcher.NewForTest(
+			cfg,
+			deps.state,
+			resolver.NewFromLogger(slog.Default()),
+			deps.portChecker,
+			deps.tlsChecker,
+			cancelOnNotify{mockNotifier: deps.notifier, cancel: cancel},
+		)
+
+		deps.state.SetDomainState(domain, &state.DomainState{
+			Nameservers: []string{oldNS1},
+		})
+		deps.state.SetHostnameState(domain, saved(
+			map[string]*state.NameserverRecordState{
+				oldNS1: answered(map[string][]string{"A": {oldIP}}),
+			},
+		))
+
+		w.RunOnce(firstCtx)
 
 		err := checkOnce(ctx, w, deps)
 
@@ -593,6 +684,22 @@ func TestDomainWithNoDelegationOfItsOwn(t *testing.T) {
 	if _, ok := deps.state.GetHostnameState(domain); !ok {
 		t.Errorf("no records saved for %s", domain)
 	}
+
+	title := "NS Change: " + domain
+	want := "Domain: " + domain + "\nAdded: \nRemoved: " + oldNS1
+
+	var changes []notification
+
+	for _, n := range deps.notifier.getNotifications() {
+		if n.Title == title {
+			changes = append(changes, n)
+		}
+	}
+
+	if len(changes) != 1 || changes[0].Message != want {
+		t.Errorf("sent %v, want one %q with message:\n%s",
+			changes, title, want)
+	}
 }
 
 func TestNSAddressChangeDetection(t *testing.T) {
@@ -718,7 +825,8 @@ func TestPortStateChange(t *testing.T) {
 
 	hs, _ := deps.state.GetHostnameState(testHost)
 	assertNotified(
-		t, deps, "Port Change: "+addresses(hs)[0]+":443", "warning",
+		t, deps, "Port Change: "+net.JoinHostPort(addresses(hs)[0], "443"),
+		"warning",
 	)
 }
 
@@ -925,11 +1033,12 @@ func TestDNSRunsBeforePortAndTLSChecks(t *testing.T) {
 
 	// Port and TLS checks must use the addresses this check found.
 	for _, ip := range addresses(snap.Hostnames[testHost]) {
-		if _, ok := snap.Ports[ip+":80"]; !ok {
-			t.Error("port check used stale DNS: missing " + ip + ":80")
+		portKey := net.JoinHostPort(ip, "80")
+		if _, ok := snap.Ports[portKey]; !ok {
+			t.Error("port check used stale DNS: missing " + portKey)
 		}
 
-		certKey := ip + ":443:" + testHost
+		certKey := net.JoinHostPort(ip, "443") + ":" + testHost
 		if _, ok := snap.Certificates[certKey]; !ok {
 			t.Error("TLS check used stale DNS: missing " + certKey)
 		}
@@ -1049,7 +1158,7 @@ func TestSendTestNotification_Disabled(t *testing.T) {
 	}
 }
 
-func TestNSFailureAndRecovery(t *testing.T) {
+func TestNSChangeAndRecovery(t *testing.T) {
 	t.Parallel()
 
 	cfg := defaultTestConfig(t)
@@ -1059,7 +1168,7 @@ func TestNSFailureAndRecovery(t *testing.T) {
 
 	// The saved state has every nameserver live DNS lists as one that
 	// did not answer, and, as answering, one that live DNS does not
-	// list, which then disappears.
+	// list, which the check then finds removed.
 	_, deps := runChecks(t, cfg, func(deps *testDeps) {
 		byNameserver := map[string]*state.NameserverRecordState{
 			oldNS1: answered(map[string][]string{"A": {oldIP}}),
@@ -1071,7 +1180,7 @@ func TestNSFailureAndRecovery(t *testing.T) {
 		deps.state.SetHostnameState(testHost, saved(byNameserver))
 	})
 
-	assertNotified(t, deps, "NS Failure: "+testHost, "error")
+	assertNotified(t, deps, "NS Change: "+testHost, "warning")
 	assertNotified(t, deps, "NS Recovery: "+testHost, "success")
 
 	// A nameserver that did not answer has no records to compare, so