TODO.md: drop the Completed Steps log; keep only open work (closes #307)

Git holds the history the list recorded.

Model: opus-5-5

README.md

@@ -355,6 +355,17 @@ dnswatcher exposes a lightweight HTTP API for operational visibility:
 | `GET /api/v1/status`           | Current monitoring state      |
 | `GET /metrics`                 | Prometheus metrics, see below |
 
+The health check, at `/.well-known/healthcheck` and `/health`, follows the most
+recent round of DNS checks. When every lookup in that round failed because no
+root server gave a usable reply, as when none of them answers or the network
+intercepts DNS queries (see DNS Resolution Strategy), dnswatcher could not reach
+the DNS at all. The health check then gives `status` `error` and answers
+`503 Service Unavailable`, so the image's health check fails and Docker and
+upaas see the container as unhealthy. Otherwise it gives `ok` and answers
+`200 OK`, as it also does before the first round finishes. A round gives `ok`
+even when a name's own nameservers do not answer or a domain does not exist,
+since the root servers gave a usable reply to those lookups.
+
 In `/api/v1/status`, each nameserver entry and certificate entry whose `status`
 is `error` also has `error`, the reason, as in the state file (see State File
 Format). A domain's own records are in its entry in `domains`, under
@@ -561,6 +572,18 @@ This approach ensures:
 - Ability to detect split-horizon or inconsistent responses across authoritative
   servers.
 
+A root server never offers recursion, so a reply marked "recursion available"
+from a root server's address came from a recursive resolver on the network
+answering in its place, as on a network where a router rule, a VPN or a DNS
+filter sends all DNS to one resolver. The first such reply from a root server's
+address, a refusal included, stops the lookup at once with the error "this
+network intercepts DNS queries", naming that address, and no other root server
+is asked. A lookup that every root server refuses without that mark fails with
+the same error. Every lookup starts at the root servers, so on such a network
+every lookup fails with that error, whether or not that resolver has the name
+cached, and dnswatcher's tests fail there too. That is by design: nothing asks
+again another way or otherwise works around the interception.
+
 A watched name's records are stored as its nameservers return them, CNAME
 included. When they return a CNAME and no address, the chain of every CNAME
 target they gave is followed (with a depth limit to prevent loops) to the A and

TODO.md

@@ -17,213 +17,6 @@ milestone: https://git.eeqj.de/sneak/dnswatcher/milestone/7
 
 1.0 tracking issue: https://git.eeqj.de/sneak/dnswatcher/issues/66
 
-# Completed Steps
-
-- 2026-10-15: a query its caller's context ends before it is sent records no
-  reason about the nameserver, and a try the context's deadline cuts short is a
-  timeout, so a nameserver whose only failures were timeouts is `timeout`, not a
-  network error or `ok` (closes #298).
-- 2026-10-15: the resolver no longer says its query timeout grows on each try;
-  every try waits the same 2 seconds, and the unused `timeoutMultiplier` is gone
-  (closes #296).
-- 2026-10-15: `TESTING.md` Constraints give the resolver's 2-second query
-  timeout and second try, say a zone's servers are asked one at a time in a
-  random order, give the 60-second cap and 20-second target for `make test`, and
-  say a flaky live test is fixed with robustness, not with mocks (closes #289).
-- 2026-10-15: a request under `/s/` that names a directory, such as `/s/` or
-  `/s/css`, answers 404 instead of a listing of its files (closes #286).
-- 2026-10-15: a notification endpoint that answers with a redirect, or any
-  status other than 2xx, is a failed delivery, retried and logged (closes #287).
-- 2026-10-15: a nameserver is asked at each of its IPv4 addresses in turn, so it
-  has failed only when none of them gives a usable reply (closes #285).
-- 2026-10-15: the log names a notification endpoint (`slack`, `mattermost`,
-  `ntfy`) and never shows its URL's path or query (closes #288).
-- 2026-10-13: a hostname or a domain whose zone's nameservers all stop answering
-  gets an NS Failure for each, then an NS Recovery (closes #284).
-- 2026-10-13: an IPv6 address is written in brackets before its port, as in
-  `[2001:db8::1]:443`, in keys and Port Change notifications (closes #270).
-- 2026-10-13: `TODO.md` names only open issues as next and future steps, and its
-  Status says nothing about git tags (closes #281).
-- 2026-10-13: a port check that finds port 443 closed removes that address's
-  certificate entries, so none is left shown as `ok` (closes #265).
-- 2026-10-13: a state file holding `{}`, or leaving out a section, loads with
-  that section empty; the first check no longer crashes on it (closes #267).
-- 2026-10-13: a stop before the first scan completes sends no startup
-  notification; a scan that ran to the end still sends it (closes #271).
-- 2026-10-13: a nameserver leaving a domain's delegation sends only NS Change;
-  one leaving a hostname's zone sends NS Change, not NS Failure (closes #262).
-- 2026-10-13: `DNSWATCHER_METRICS_USERNAME` set without
-  `DNSWATCHER_METRICS_PASSWORD` stops startup (closes #268).
-- 2026-10-13: a save syncs the state file to disk before renaming it into place,
-  so a crash cannot leave it empty and stop the next start (closes #264).
-- 2026-10-13: a `PORT` that cannot be bound stops startup with an error naming
-  the address, instead of running without the HTTP server (closes #266).
-- 2026-10-13: a bad `PORT`, `DNSWATCHER_TLS_EXPIRY_WARNING` or true/false
-  setting stops startup instead of being read as 0 or false (closes #269).
-- 2026-10-13: a check saving the state while the dashboard or `/api/v1/status`
-  is read cannot crash the process (closes #261).
-- 2026-10-06: canonical files re-fetched from `sneak/prompts` at `dd4027b`; lint
-  and test are phases of the `Dockerfile`, every scripted build uncached (closes
-  #257).
-- 2026-10-05: a name listed in `DNSWATCHER_SKIP_RECORD_NOTIFICATIONS` gets no
-  Record Change or Inconsistency notification; a name listed there that is not
-  in `DNSWATCHER_TARGETS` stops startup (closes #255).
-- 2026-10-02: a nameserver whose query for one record type failed while the
-  others answered with no records is `ok`, not `nodata` (closes #253).
-- 2026-10-02: a domain that does not exist is shown so, with no nameservers; no
-  name gets a parent's nameservers when its own did not answer (closes #222).
-- 2026-10-02: the refused-query test sends one query to four operators' public
-  resolvers in turn until one replies, not eight to one operator (closes #251).
-- 2026-10-02: a name removed from `DNSWATCHER_TARGETS` leaves the state, and so
-  the dashboard and API, at startup, before the first check (closes #223).
-- 2026-10-02: a record type whose query to a nameserver fails keeps its previous
-  records and alerts nothing; the other types are still saved (closes #231).
-- 2026-10-02: a Port Change notification lists the port's domains on a
-  `Domains:` line and its hostnames on a `Hostnames:` line (closes #248).
-- 2026-10-02: the dashboard's Ports table and `/api/v1/status` port entries list
-  a port's domains apart from its hostnames (closes #245).
-- 2026-10-02: nameservers a referral names without addresses are looked up,
-  three deep at most; `pool.ntp.org`'s nameservers resolve (closes #221).
-- 2026-10-02: an apex domain is not counted or listed as a hostname; its records
-  show under Domains, and notifications about them say `Domain:` (closes #224).
-- 2026-10-02: the dashboard lists each nameserver's record types in one fixed
-  order, the README's, then any other type, not a random one (closes #226).
-- 2026-10-02: the dashboard and `/api/v1/status` show why a nameserver query or
-  a certificate check failed, which only the state file showed (closes #225).
-- 2026-10-02: a name's CNAME is stored once per nameserver, not once per record
-  type asked for; a state file with repeats loads each value once (closes #220).
-- 2026-10-02: a DNS lookup that shutdown cuts short logs no error; one that
-  fails otherwise, or runs out of time, still does (closes #229).
-- 2026-10-02: Record Change and Inconsistency notifications list only the record
-  types that differ, each with its values as plain text (closes #219).
-- 2026-10-02: the startup notification no longer says every notification
-  endpoint works; it says it is a test sent to each of them (closes #230).
-- 2026-10-02: a Mattermost webhook that answers an HTTP error is logged as
-  `mattermost notification failed`, not as a Slack failure (closes #227).
-- 2026-10-02: durations in the log are written as text such as `2m0s`, not as a
-  bare count of nanoseconds (closes #228).
-- 2026-10-02: a watched name whose nameservers answer with a CNAME and no
-  address gets port and TLS checks at the end of its CNAME chain (closes #203).
-- 2026-10-02: a resolver test that reads one record type from a nameserver's
-  answer asks again when that type is missing from it (closes #218).
-- 2026-10-02: a plain `docker build .` of a clone stamps its tag or short
-  commit, not `dev`: the build context now carries `.git` (closes #210).
-- 2026-10-02: a query a server refuses is not resent asking for recursion, and
-  every root server refusing is reported as DNS interception (closes #206).
-- 2026-10-02: a push to a branch cancels that branch's older CI run, and the
-  checkout leaves no token in `.git/config` (closes #216).
-- 2026-10-02: watcher tests send far fewer queries and a live attempt may take
-  18s; nameserver addresses are asked only for A, AAAA, CNAME (closes #214).
-- 2026-10-02: the resolver tries root servers, and every other server list it
-  walks, in a random order each time, not always from the top (closes #138).
-- 2026-10-02: a name listed more than once in `DNSWATCHER_TARGETS`, in any
-  letter case or with a trailing dot, is watched once (closes #207).
-- 2026-10-01: README checked against the code and corrected: metrics, CORS,
-  notification retries, CNAMEs, state file fields, Design tree (closes #108).
-- 2026-10-01: a certificate within the expiry warning period is warned about on
-  every TLS check, where some checks used to skip it at random (closes #204).
-- 2026-10-01: a domain's NS set is its delegation from the parent zone's
-  servers, not whichever of its own servers answered first (closes #200).
-- 2026-10-01: README has Getting Started, Rationale and TODO sections, and its
-  Architecture section is now Design, in the order policy sets (closes #173).
-- 2026-10-01: a zone's server that answers SERVFAIL or a referral leading no
-  closer is passed over for the next, as one that times out is (closes #197).
-- 2026-10-01: when none of a configured name's nameservers answered, the port
-  state saved for its addresses is kept, not removed (closes #193).
-- 2026-10-01: `ResolveIPAddresses` returns an error, not no addresses, when no
-  nameserver of the name's zone answered (closes #190).
-- 2026-10-01: `make fmt` and `make fmt-check` cover Markdown with prettier, run
-  in Docker at the version pinned by `yarn.lock` (closes #119).
-- 2026-10-01: `make fmt-check` fails on a file `goimports` would change; both
-  format scripts run `goimports` at its pinned commit, not from `PATH` (#119).
-- 2026-10-01: a hostname is queried at the servers of the zone it is in, found
-  by following delegations for the name, not its last two labels (closes #189).
-- 2026-10-01: each nameserver's addresses are saved with its domain, and a
-  change while it stays in the delegation is notified (closes #105).
-- 2026-10-01: the watcher saves state when it stops, and shutdown waits for that
-  save, so it no longer relies on the state's own stop hook (closes #114).
-- 2026-10-01: `DNSWATCHER_SENTRY_DSN` reports panics in HTTP handlers to Sentry,
-  and a DSN Sentry cannot parse stops startup (closes #107).
-- 2026-10-01: a port or TLS check that shutdown cuts short saves nothing and
-  sends no notification, as a cut-short DNS lookup already did (closes #185).
-- 2026-10-01: the client address from `X-Forwarded-For` is the last entry that
-  is not a trusted proxy, not the first, which the client sets (closes #181).
-- 2026-10-01: a nameserver that does not answer is saved as `error` with the
-  reason, and NS failure and NS recovery are notified (closes #104).
-- 2026-10-01: a `DNSWATCHER_DNS_INTERVAL` or `DNSWATCHER_TLS_INTERVAL` that is
-  not a positive duration stops startup; empty means the default (closes #177).
-- 2026-10-01: `/metrics` allows each client address 30 requests a minute,
-  counted before Basic Auth, and answers 429 beyond that (closes #101).
-- 2026-10-01: the image built by `make docker` reports the `git describe`
-  version, not `dev`, and the startup log now shows it (closes #109).
-- 2026-10-01: two notify shutdown tests always release the delivery they hold,
-  so a drain that returns early fails them instead of hanging (closes #176).
-- 2026-10-01: `script/install-precommit` asks git for the repository's git
-  directory, so `make hooks` also works where `.git` is a file (closes #129).
-- 2026-10-01: `TODO.md` brought up to date: open issues listed by URL, every
-  Completed Steps entry cut to at most two lines (closes #146).
-- 2026-10-01: wildcard CORS now applies only to the public routes, not to
-  `/metrics`, and allows only the methods they serve (closes #100).
-- 2026-10-01: `internal/state` and `internal/watcher` no longer export test-only
-  constructors: two moved to `export_test.go`, one is deleted (closes #111).
-- 2026-10-01: notify shutdown tests use one timing constant per meaning, name
-  the bound they check, and require the drain's debug line (closes #116).
-- 2026-09-29: the entrypoint chowns the data directory to `dnswatcher` and runs
-  dnswatcher as that user, so a host bind mount needs no chown (closes #166).
-- 2026-09-29: the live-DNS test package is renamed `internal/livednstest`;
-  `make lint` fails when program code imports it (closes #164).
-- 2026-09-29: `.golangci.yml` re-fetched from `sneak/prompts`, with
-  `gomodguard_v2` and the org `depguard` `test-support` rule (closes #123).
-- 2026-09-29: watcher and resolver tests that look something up in DNS use the
-  real resolver against live DNS servers (closes #159).
-- 2026-09-28: the inconsistency alert is sent once, when two nameservers start
-  to disagree; every pair of nameservers is compared (closes #158).
-- 2026-09-28: DNS names in record values (CNAME, MX, SRV and NS targets) are
-  lower-cased, so letter case alone is not a change (closes #157).
-- 2026-09-28: lint and tests run on every build: `script/cibuild` and
-  `script/docker` pass `--no-cache-filter=lint,builder` (closes #115).
-- 2026-09-28: the server timeout test drives `Run` and checks the timeouts on
-  the `http.Server` it serves (closes #120).
-- 2026-09-28: upaas deploy readiness: the image runs as user `dnswatcher` with a
-  `HEALTHCHECK`; README "Running under upaas" (closes #147).
-- 2026-09-21: added behavioural tests for `internal/globals`,
-  `internal/healthcheck`, and `internal/logger` (closes #110).
-- 2026-09-21: `go mod tidy` dropped the redundant `golang.org/x/sync`
-  `// indirect` line so `script/bootstrap` leaves a clean tree (#132)
-- 2026-08-10: comment-only corrections to `script/bootstrap`, `script/cibuild`
-  and `Dockerfile.lint`; no behaviour changed.
-- 2026-08-10: MIT `LICENSE` added at the repository root; the README's first
-  line and License section name the licence.
-- 2026-08-10: policy scaffold present: `REPO_POLICIES.md`, `.editorconfig`,
-  `.dockerignore`, CI workflow, `make fmt-check`, `make docker`, `make hooks`.
-- 2026-08-10: Go's test cache disabled in `script/test` (`-count=1`), so every
-  run queries live DNS; a failed run is rerun with `-v`.
-- 2026-08-10: live-DNS tests made robust rather than gated (#93): a limit on
-  concurrent lookups, retries, and a quorum across nameservers.
-- 2026-08-10: all linting moved into Docker: `script/lint` builds
-  `Dockerfile.lint`, and the root `Dockerfile` has its own lint stage.
-- 2026-08-09: in-flight notification deliveries are drained at shutdown, bounded
-  by the shutdown deadline (#106).
-- 2026-08-09: `http.Server` sets all four socket timeouts; `WriteTimeout` stays
-  above the 60s handler timeout (#99).
-- 2026-08-09: `SecurityHeaders()` middleware sets HSTS, CSP and the other
-  security headers `REPO_POLICIES.md` requires on every response.
-- 2026-08-07: golangci-lint bumped to v2.12.2 and `.golangci.yml` set to the org
-  config; fixed the resulting `goconst`, `dupl` and `lll` findings.
-- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
-  shims, README Entrypoints section
-- 2026-02-20: iterative DNS resolver implemented
-- 2026-02-20: CI actions and go install refs pinned to commit SHAs; Gitea
-  Actions workflow added
-- 2026-02-20: watcher monitoring orchestrator merged to main (#8)
-- 2026-02-20: DOMAINS/HOSTNAMES unified into single TARGETS config (#11)
-- 2026-02-19: TCP port connectivity checker, made concurrent with port
-  validation; gosec G704 SSRF findings fixed without suppression
-- 2026-02-19: TLS certificate inspector with no-peer-certificates error path and
-  IP SANs
-- 2026-02-19: gosec SSRF and formatting fixes on main
-- 2026-02-19: initial scaffold with per-nameserver DNS monitoring model
-
 # Future Steps
 
 - DNSSEC validation, after 1.0: https://git.eeqj.de/sneak/dnswatcher/issues/59

internal/handlers/healthcheck.go

@@ -2,16 +2,25 @@ package handlers
 
 import (
 	"net/http"
+
+	"sneak.berlin/go/dnswatcher/internal/healthcheck"
 )
 
-// HandleHealthCheck returns the health check handler.
+// HandleHealthCheck returns the health check handler. It answers 503
+// Service Unavailable when the health check's status is error, so that
+// a container health check fails, and 200 OK otherwise.
 func (h *Handlers) HandleHealthCheck() http.HandlerFunc {
 	return func(
 		writer http.ResponseWriter,
 		request *http.Request,
 	) {
-		h.respondJSON(
-			writer, request, h.hc.Check(), http.StatusOK,
-		)
+		resp := h.hc.Check()
+
+		code := http.StatusOK
+		if resp.Status == healthcheck.StatusError {
+			code = http.StatusServiceUnavailable
+		}
+
+		h.respondJSON(writer, request, resp, code)
 	}
 }

internal/handlers/healthcheck_test.go

@@ -0,0 +1,158 @@
+package handlers_test
+
+import (
+	"encoding/json"
+	"log/slog"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"go.uber.org/fx/fxtest"
+
+	"sneak.berlin/go/dnswatcher/internal/config"
+	"sneak.berlin/go/dnswatcher/internal/globals"
+	"sneak.berlin/go/dnswatcher/internal/handlers"
+	"sneak.berlin/go/dnswatcher/internal/healthcheck"
+	"sneak.berlin/go/dnswatcher/internal/logger"
+	"sneak.berlin/go/dnswatcher/internal/notify"
+	"sneak.berlin/go/dnswatcher/internal/portcheck"
+	"sneak.berlin/go/dnswatcher/internal/resolver"
+	"sneak.berlin/go/dnswatcher/internal/state"
+	"sneak.berlin/go/dnswatcher/internal/tlscheck"
+	"sneak.berlin/go/dnswatcher/internal/watcher"
+)
+
+// TestHealthCheckAnswers503WhenDNSUnreachable serves the health check
+// before the watcher's first round of DNS checks, when it answers 200
+// with status ok, and after a round that could not reach the DNS, when
+// it answers 503 with status error. The watcher's resolver starts every
+// walk at public recursive resolvers of four operators, whose replies
+// are marked recursion available, as on a network that intercepts DNS
+// queries. If none of them replies, no root server answered, and the
+// round could not reach the DNS either.
+func TestHealthCheckAnswers503WhenDNSUnreachable(t *testing.T) {
+	t.Parallel()
+
+	hnd, w := newHandlersWithWatcher(t, resolver.NewWithRootServers(
+		slog.Default(),
+		[]string{"1.1.1.1", "8.8.8.8", "9.9.9.9", "208.67.222.222"},
+	))
+
+	assertHealthCheck(t, hnd, http.StatusOK, healthcheck.StatusOK)
+
+	w.RunOnce(t.Context())
+
+	assertHealthCheck(
+		t, hnd, http.StatusServiceUnavailable, healthcheck.StatusError,
+	)
+}
+
+// newHandlersWithWatcher builds real Handlers whose health check
+// follows the watcher it returns, which watches example.org with res.
+// The lifecycle is never started, so the watcher runs a round of checks
+// only when the test calls RunOnce.
+func newHandlersWithWatcher(
+	t *testing.T,
+	res *resolver.Resolver,
+) (*handlers.Handlers, *watcher.Watcher) {
+	t.Helper()
+
+	glob, err := globals.New(nil)
+	if err != nil {
+		t.Fatalf("globals.New: %v", err)
+	}
+
+	log, err := logger.New(nil, logger.Params{Globals: glob})
+	if err != nil {
+		t.Fatalf("logger.New: %v", err)
+	}
+
+	cfg := &config.Config{
+		Hostnames: []string{"example.org"},
+		DataDir:   t.TempDir(),
+	}
+	lifecycle := fxtest.NewLifecycle(t)
+
+	st, err := state.New(lifecycle, state.Params{Logger: log, Config: cfg})
+	if err != nil {
+		t.Fatalf("state.New: %v", err)
+	}
+
+	notifier, err := notify.New(
+		lifecycle, notify.Params{Logger: log, Config: cfg},
+	)
+	if err != nil {
+		t.Fatalf("notify.New: %v", err)
+	}
+
+	w, err := watcher.New(lifecycle, watcher.Params{
+		Logger:    log,
+		Config:    cfg,
+		State:     st,
+		Resolver:  res,
+		PortCheck: portcheck.NewStandalone(),
+		TLSCheck:  tlscheck.NewStandalone(),
+		Notify:    notifier,
+	})
+	if err != nil {
+		t.Fatalf("watcher.New: %v", err)
+	}
+
+	hc, err := healthcheck.New(lifecycle, healthcheck.Params{
+		Globals: glob,
+		Config:  cfg,
+		Logger:  log,
+		Watcher: w,
+	})
+	if err != nil {
+		t.Fatalf("healthcheck.New: %v", err)
+	}
+
+	hnd, err := handlers.New(nil, handlers.Params{
+		Logger:      log,
+		Globals:     glob,
+		Healthcheck: hc,
+		State:       st,
+		Notify:      notifier,
+	})
+	if err != nil {
+		t.Fatalf("handlers.New: %v", err)
+	}
+
+	return hnd, w
+}
+
+// assertHealthCheck serves one request to the health check and checks
+// that it answers with code and a JSON body whose status is status.
+func assertHealthCheck(
+	t *testing.T,
+	hnd *handlers.Handlers,
+	code int,
+	status string,
+) {
+	t.Helper()
+
+	rec := httptest.NewRecorder()
+	req := httptest.NewRequestWithContext(
+		t.Context(), http.MethodGet, "/.well-known/healthcheck", nil,
+	)
+
+	hnd.HandleHealthCheck()(rec, req)
+
+	if rec.Code != code {
+		t.Errorf("code = %d, want %d", rec.Code, code)
+	}
+
+	var body struct {
+		Status string `json:"status"`
+	}
+
+	err := json.Unmarshal(rec.Body.Bytes(), &body)
+	if err != nil {
+		t.Fatalf("decoding response: %v", err)
+	}
+
+	if body.Status != status {
+		t.Errorf("status = %q, want %q", body.Status, status)
+	}
+}

internal/healthcheck/healthcheck.go

@@ -11,6 +11,13 @@ import (
 	"sneak.berlin/go/dnswatcher/internal/config"
 	"sneak.berlin/go/dnswatcher/internal/globals"
 	"sneak.berlin/go/dnswatcher/internal/logger"
+	"sneak.berlin/go/dnswatcher/internal/watcher"
+)
+
+// Values of Response.Status.
+const (
+	StatusOK    = "ok"
+	StatusError = "error"
 )
 
 // Params contains dependencies for Healthcheck.
@@ -20,6 +27,7 @@ type Params struct {
 	Globals *globals.Globals
 	Config  *config.Config
 	Logger  *logger.Logger
+	Watcher *watcher.Watcher
 }
 
 // Healthcheck provides health status information.
@@ -61,10 +69,17 @@ func New(
 	return healthcheck, nil
 }
 
-// Check returns the current health status.
+// Check returns the current health status. Status is StatusError when
+// the watcher's most recent round of DNS checks could not reach the DNS
+// (see watcher.Watcher.DNSUnreachable), and StatusOK otherwise.
 func (h *Healthcheck) Check() *Response {
+	status := StatusOK
+	if h.params.Watcher.DNSUnreachable() {
+		status = StatusError
+	}
+
 	return &Response{
-		Status:        "ok",
+		Status:        status,
 		Now:           time.Now().UTC().Format(time.RFC3339Nano),
 		UptimeSeconds: int64(h.uptime().Seconds()),
 		UptimeHuman:   h.uptime().String(),

internal/healthcheck/healthcheck_test.go

@@ -3,10 +3,12 @@ package healthcheck_test
 import (
 	"context"
 	"encoding/json"
+	"log/slog"
 	"testing"
 	"time"
 
 	"go.uber.org/fx"
+	"go.uber.org/fx/fxtest"
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -15,6 +17,12 @@ import (
 	"sneak.berlin/go/dnswatcher/internal/globals"
 	"sneak.berlin/go/dnswatcher/internal/healthcheck"
 	"sneak.berlin/go/dnswatcher/internal/logger"
+	"sneak.berlin/go/dnswatcher/internal/notify"
+	"sneak.berlin/go/dnswatcher/internal/portcheck"
+	"sneak.berlin/go/dnswatcher/internal/resolver"
+	"sneak.berlin/go/dnswatcher/internal/state"
+	"sneak.berlin/go/dnswatcher/internal/tlscheck"
+	"sneak.berlin/go/dnswatcher/internal/watcher"
 )
 
 // recordingLifecycle is a minimal fx.Lifecycle that records the hooks
@@ -30,12 +38,14 @@ func (l *recordingLifecycle) Append(hook fx.Hook) {
 
 // newHealthcheck builds a Healthcheck through the real constructor and
 // runs the registered OnStart hook so StartupTime is set the same way
-// the fx lifecycle would set it.
+// the fx lifecycle would set it. It follows the watcher it returns,
+// which watches example.org with res and has run no round of checks.
 func newHealthcheck(
 	t *testing.T,
+	res *resolver.Resolver,
 	maintenance bool,
 	version string,
-) *healthcheck.Healthcheck {
+) (*healthcheck.Healthcheck, *watcher.Watcher) {
 	t.Helper()
 
 	g := &globals.Globals{Appname: "dnswatcher", Version: version}
@@ -43,12 +53,21 @@ func newHealthcheck(
 	log, err := logger.New(nil, logger.Params{Globals: g})
 	require.NoError(t, err)
 
+	cfg := &config.Config{
+		MaintenanceMode: maintenance,
+		Hostnames:       []string{"example.org"},
+		DataDir:         t.TempDir(),
+	}
+
+	w := newWatcher(t, log, cfg, res)
+
 	lifecycle := &recordingLifecycle{}
 
 	hc, err := healthcheck.New(lifecycle, healthcheck.Params{
 		Globals: g,
-		Config:  &config.Config{MaintenanceMode: maintenance},
+		Config:  cfg,
 		Logger:  log,
+		Watcher: w,
 	})
 	require.NoError(t, err)
 
@@ -57,16 +76,53 @@ func newHealthcheck(
 	require.NotNil(t, lifecycle.hooks[0].OnStart)
 	require.NoError(t, lifecycle.hooks[0].OnStart(context.Background()))
 
-	return hc
+	return hc, w
+}
+
+// newWatcher builds a watcher through the real constructor, with res as
+// its resolver. Its lifecycle is never started, so it runs a round of
+// checks only when the test calls RunOnce.
+func newWatcher(
+	t *testing.T,
+	log *logger.Logger,
+	cfg *config.Config,
+	res *resolver.Resolver,
+) *watcher.Watcher {
+	t.Helper()
+
+	lifecycle := fxtest.NewLifecycle(t)
+
+	st, err := state.New(lifecycle, state.Params{Logger: log, Config: cfg})
+	require.NoError(t, err)
+
+	notifier, err := notify.New(
+		lifecycle, notify.Params{Logger: log, Config: cfg},
+	)
+	require.NoError(t, err)
+
+	w, err := watcher.New(lifecycle, watcher.Params{
+		Logger:    log,
+		Config:    cfg,
+		State:     st,
+		Resolver:  res,
+		PortCheck: portcheck.NewStandalone(),
+		TLSCheck:  tlscheck.NewStandalone(),
+		Notify:    notifier,
+	})
+	require.NoError(t, err)
+
+	return w
 }
 
 func TestCheckStatusAndPayloadShape(t *testing.T) {
 	t.Parallel()
 
-	hc := newHealthcheck(t, false, "v9.9.9")
+	hc, _ := newHealthcheck(
+		t, resolver.NewFromLogger(slog.Default()), false, "v9.9.9",
+	)
 	resp := hc.Check()
 
-	assert.Equal(t, "ok", resp.Status)
+	assert.Equal(t, healthcheck.StatusOK, resp.Status)
 
 	// The JSON shape and field names are part of the contract for the
 	// /health and /.well-known/healthcheck routes, so assert on the
@@ -114,7 +170,10 @@ func TestCheckMaintenanceModeReflectsConfig(t *testing.T) {
 		t.Run(tt.name, func(t *testing.T) {
 			t.Parallel()
 
-			hc := newHealthcheck(t, tt.maintenance, "test")
+			hc, _ := newHealthcheck(
+				t, resolver.NewFromLogger(slog.Default()), tt.maintenance,
+				"test",
+			)
 			resp := hc.Check()
 			assert.Equal(t, tt.maintenance, resp.Maintenance,
 				"maintenanceMode must mirror Config.MaintenanceMode")
@@ -125,10 +184,38 @@ func TestCheckMaintenanceModeReflectsConfig(t *testing.T) {
 func TestCheckSurfacesVersionAndAppname(t *testing.T) {
 	t.Parallel()
 
-	hc := newHealthcheck(t, false, "surfaced-version-123")
+	hc, _ := newHealthcheck(
+		t, resolver.NewFromLogger(slog.Default()), false,
+		"surfaced-version-123",
+	)
 	resp := hc.Check()
 
 	assert.Equal(t, "surfaced-version-123", resp.Version,
 		"version from globals must appear in the payload")
 	assert.Equal(t, "dnswatcher", resp.Appname)
 }
+
+// TestCheckStatusFollowsTheDNSChecks checks that the status is ok
+// before the watcher's first round of DNS checks, and error after a
+// round that could not reach the DNS. The watcher's resolver starts
+// every walk at public recursive resolvers of four operators, whose
+// replies are marked recursion available, as on a network that
+// intercepts DNS queries. If none of them replies, no root server
+// answered, and the round could not reach the DNS either.
+func TestCheckStatusFollowsTheDNSChecks(t *testing.T) {
+	t.Parallel()
+
+	res := resolver.NewWithRootServers(
+		slog.Default(),
+		[]string{"1.1.1.1", "8.8.8.8", "9.9.9.9", "208.67.222.222"},
+	)
+	hc, w := newHealthcheck(t, res, false, "test")
+
+	assert.Equal(t, healthcheck.StatusOK, hc.Check().Status,
+		"status before the first round")
+
+	w.RunOnce(t.Context())
+
+	assert.Equal(t, healthcheck.StatusError, hc.Check().Status,
+		"status after a round that could not reach the DNS")
+}

internal/resolver/errors.go

@@ -36,6 +36,12 @@ var (
 	// so whether the name has addresses is unknown.
 	ErrNoNameserverAnswered = errors.New("no nameserver answered")
 
+	// ErrNoRootServerAnswered is returned when the root servers were
+	// asked and none of them gave a usable reply, as when the host
+	// running dnswatcher has lost its network. When they all refused,
+	// the error is ErrIntercepted instead.
+	ErrNoRootServerAnswered = errors.New("no root server answered")
+
 	// ErrNoNameserverAddress is returned when a nameserver is to be
 	// asked and no address for it was given.
 	ErrNoNameserverAddress = errors.New("nameserver has no address")
@@ -53,9 +59,13 @@ var (
 		"reply truncated and its retry over TCP failed",
 	)
 
-	// ErrIntercepted is returned when every root server refused a
-	// query. Root servers refuse no query, so the refusals came from
-	// something on the network answering in their place.
+	// ErrIntercepted is returned when a root server's address gave a
+	// reply marked recursion available, a refusal included, which ends
+	// the lookup at once, or when every root server refused a query.
+	// Root servers do neither, so those replies came from something on
+	// the network answering in their place. On such a network
+	// dnswatcher fails with it, by design: nothing works around the
+	// interception.
 	ErrIntercepted = errors.New("this network intercepts DNS queries")
 
 	// ErrCNAMEDepthExceeded is returned when a CNAME chain

internal/resolver/iterative.go

@@ -115,7 +115,8 @@ func (r *Resolver) retryTCP(
 }
 
 // queryDNS sends a DNS query to a specific server IP, never asking it
-// for recursion. A reply of REFUSED is returned as ErrRefused.
+// for recursion. A reply of REFUSED is returned together with
+// ErrRefused, so that the caller can still read the reply's flags.
 func (r *Resolver) queryDNS(
 	ctx context.Context,
 	serverIP string,
@@ -139,7 +140,7 @@ func (r *Resolver) queryDNS(
 	}
 
 	if resp.Rcode == dns.RcodeRefused {
-		return nil, fmt.Errorf(
+		return resp, fmt.Errorf(
 			"query %s @%s: %w", name, serverIP, ErrRefused,
 		)
 	}
@@ -284,10 +285,16 @@ func shuffled(
 }
 
 // queryServers asks servers, the servers of zone, about name in a random
-// order until one gives a usable reply. A server that times out, refuses
-// or gives a reply that is not usable is passed over for the next. When
-// every server refused, the error says so, and when they are the root
-// servers it is ErrIntercepted.
+// order until one gives a usable reply. When they are the root servers,
+// the first reply marked recursion available, a refusal included, ends
+// the lookup with ErrIntercepted, naming the address it came from, and
+// no other root server is asked: root servers offer no recursion, so
+// that reply came from something on the network answering in their
+// place. Otherwise a server that times out, refuses or gives a reply
+// that is not usable is passed over for the next. When every server
+// refused, the error says so, and when they are the root servers it is
+// ErrIntercepted. When the root servers gave no usable reply otherwise,
+// it is ErrNoRootServerAnswered.
 func (r *Resolver) queryServers(
 	ctx context.Context,
 	servers []string,
@@ -304,7 +311,17 @@ func (r *Resolver) queryServers(
 			return nil, ErrContextCanceled
 		}
 
+		// queryDNS returns a refusal together with its reply, so a
+		// refusal's mark is checked here too.
 		resp, err := r.queryDNS(ctx, ip, name, qtype)
+		if resp != nil && zone == "." && resp.RecursionAvailable {
+			return nil, fmt.Errorf(
+				"query %s @%s: reply marked recursion available, "+
+					"so not from a root server: %w",
+				name, ip, ErrIntercepted,
+			)
+		}
+
 		if err == nil && !usableReply(resp, zone, name) {
 			err = fmt.Errorf(
 				"query %s @%s: %w", name, ip, ErrUnusableReply,
@@ -336,6 +353,10 @@ func (r *Resolver) queryServers(
 		)
 	}
 
+	if zone == "." {
+		return nil, fmt.Errorf("%w: %w", ErrNoRootServerAnswered, lastErr)
+	}
+
 	return nil, fmt.Errorf("all servers failed: %w", lastErr)
 }
 
@@ -512,7 +533,7 @@ func (r *Resolver) resolveNSIterative(
 	}
 
 	domain = dns.Fqdn(domain)
-	servers := rootServerList()
+	servers := r.rootServers
 	zone := "."
 
 	for range maxDelegation {
@@ -576,7 +597,7 @@ func (r *Resolver) resolveARecord(
 	}
 
 	hostname = dns.Fqdn(hostname)
-	servers := rootServerList()
+	servers := r.rootServers
 	zone := "."
 
 	var withoutAddresses []string
@@ -636,7 +657,7 @@ func (r *Resolver) FindAuthoritativeNameservers(
 	ctx context.Context,
 	domain string,
 ) ([]string, error) {
-	return r.findAuthoritativeNameservers(ctx, domain, rootServerList())
+	return r.findAuthoritativeNameservers(ctx, domain, r.rootServers)
 }
 
 // findAuthoritativeNameservers is FindAuthoritativeNameservers with each
@@ -1111,7 +1132,7 @@ func (r *Resolver) LookupNS(
 	}
 
 	return r.followDelegation(
-		ctx, dns.Fqdn(strings.ToLower(domain)), ".", nil, rootServerList(),
+		ctx, dns.Fqdn(strings.ToLower(domain)), ".", nil, r.rootServers,
 	)
 }
 

internal/resolver/resolver.go

@@ -43,10 +43,12 @@ type NameserverResponse struct {
 }
 
 // Resolver performs iterative DNS resolution from root servers.
+// rootServers are the addresses every walk from the root starts at.
 type Resolver struct {
-	log    *slog.Logger
-	client DNSClient
-	tcp    DNSClient
+	log         *slog.Logger
+	client      DNSClient
+	tcp         DNSClient
+	rootServers []string
 }
 
 // New creates a new Resolver instance for use with uber/fx.
@@ -55,9 +57,10 @@ func New(
 	params Params,
 ) (*Resolver, error) {
 	return &Resolver{
-		log:    params.Logger.Get(),
-		client: &udpClient{timeout: queryTimeoutDuration},
-		tcp:    &tcpClient{timeout: queryTimeoutDuration},
+		log:         params.Logger.Get(),
+		client:      &udpClient{timeout: queryTimeoutDuration},
+		tcp:         &tcpClient{timeout: queryTimeoutDuration},
+		rootServers: rootServerList(),
 	}, nil
 }
 
@@ -65,10 +68,22 @@ func New(
 // useful for testing without the fx lifecycle.
 func NewFromLogger(log *slog.Logger) *Resolver {
 	return &Resolver{
-		log:    log,
-		client: &udpClient{timeout: queryTimeoutDuration},
-		tcp:    &tcpClient{timeout: queryTimeoutDuration},
+		log:         log,
+		client:      &udpClient{timeout: queryTimeoutDuration},
+		tcp:         &tcpClient{timeout: queryTimeoutDuration},
+		rootServers: rootServerList(),
 	}
 }
 
+// NewWithRootServers creates a Resolver as NewFromLogger does, except
+// that every walk from the root starts at servers instead of at the root
+// servers. Tests use it to show what dnswatcher does when no root server
+// answers, or when something on the network answers in their place.
+func NewWithRootServers(log *slog.Logger, servers []string) *Resolver {
+	r := NewFromLogger(log)
+	r.rootServers = servers
+
+	return r
+}
+
 // Method implementations are in iterative.go.

internal/resolver/resolver_test.go

@@ -815,6 +815,7 @@ func TestQueryServers_EveryRootServerRefused(t *testing.T) {
 // but nothing answers at 192.0.2.1, a documentation address, so not
 // every server refused, wherever 192.0.2.1 falls in the random order:
 // the error is not ErrIntercepted and does not say every server refused.
+// No root server gave a usable reply, so it is ErrNoRootServerAnswered.
 func TestQueryServers_NotEveryRootServerRefused(t *testing.T) {
 	t.Parallel()
 
@@ -848,6 +849,102 @@ func TestQueryServers_NotEveryRootServerRefused(t *testing.T) {
 	require.NotErrorIs(t, err, resolver.ErrIntercepted)
 	// Both errors for every server refusing say "refused a query for".
 	require.NotContains(t, err.Error(), "refused a query for")
+	require.ErrorIs(t, err, resolver.ErrNoRootServerAnswered)
+}
+
+// TestLookupsWhenNoRootServerAnswers starts every walk at 192.0.2.1, a
+// documentation address where nothing answers, as if no root server
+// answered. The lookups the watcher makes, LookupNS for a domain and
+// LookupAllRecords for a hostname, both fail with
+// ErrNoRootServerAnswered, which the watcher reads.
+func TestLookupsWhenNoRootServerAnswers(t *testing.T) {
+	t.Parallel()
+
+	r := resolver.NewWithRootServers(slog.Default(), []string{noAnswerAddress})
+
+	t.Run("LookupNS", func(t *testing.T) {
+		t.Parallel()
+
+		_, err := r.LookupNS(t.Context(), "google.com")
+		require.ErrorIs(t, err, resolver.ErrNoRootServerAnswered)
+		require.NotErrorIs(t, err, resolver.ErrIntercepted)
+	})
+
+	t.Run("LookupAllRecords", func(t *testing.T) {
+		t.Parallel()
+
+		_, err := r.LookupAllRecords(t.Context(), "www.google.com")
+		require.ErrorIs(t, err, resolver.ErrNoRootServerAnswered)
+		require.NotErrorIs(t, err, resolver.ErrIntercepted)
+	})
+}
+
+// TestQueryServers_RecursiveResolversAsRootServers passes public
+// recursive resolvers to QueryServers as the servers of the root zone,
+// as a network that sends every DNS query to a resolver would. Their
+// replies are marked recursion available, which no root server's is:
+// SERVFAIL, REFUSED, or an answer from a resolver's cache, as 1.1.1.1
+// gives for com., which would be used if a root server gave it. The
+// first such reply ends the lookup with ErrIntercepted, naming the
+// resolver that gave it. In the first group the replies differ; every
+// resolver in the second refuses a query that does not ask for
+// recursion, as 9.9.9.9 does, so there a refusal ends the lookup
+// instead of being passed over. Each group has resolvers of four
+// operators, and a resolver that gives no reply is passed over for the
+// next, so one of them answering is enough.
+func TestQueryServers_RecursiveResolversAsRootServers(t *testing.T) {
+	t.Parallel()
+
+	r := newTestResolver(t)
+	groups := [][]string{
+		{"1.1.1.1", "8.8.8.8", "9.9.9.9", "208.67.222.222"},
+		{"64.6.64.6", "185.222.222.222", "4.2.2.1", "94.140.14.14"},
+	}
+
+	for _, resolvers := range groups {
+		for _, name := range []string{"com.", "desec.io."} {
+			var err error
+
+			livednstest.Retry(
+				t,
+				"QueryServers("+strings.Join(resolvers, ", ")+
+					" as root servers, "+name+")",
+				func(ctx context.Context) error {
+					_, err = r.QueryServers(
+						ctx, resolvers, ".", name, dns.TypeNS,
+					)
+
+					// When no resolver gave a reply marked recursion
+					// available, none may have replied at all, so the
+					// attempt is tried again.
+					if err != nil && !strings.Contains(
+						err.Error(), "marked recursion available",
+					) {
+						return fmt.Errorf(
+							"%w: %w", livednstest.ErrNoAnswer, err,
+						)
+					}
+
+					return nil
+				},
+			)
+
+			require.ErrorIs(t, err, resolver.ErrIntercepted)
+
+			// The error names the resolver whose reply ended the
+			// lookup.
+			messages := make([]string, 0, len(resolvers))
+			for _, ip := range resolvers {
+				messages = append(messages,
+					"query "+name+" @"+ip+": reply marked recursion "+
+						"available, so not from a root server: "+
+						"this network intercepts DNS queries",
+				)
+			}
+
+			require.Contains(t, messages, err.Error())
+		}
+	}
 }
 
 func TestQueryNameserver_RecordsSorted(t *testing.T) {

internal/server/server_test.go

@@ -15,14 +15,21 @@ import (
 	"sneak.berlin/go/dnswatcher/internal/logger"
 	"sneak.berlin/go/dnswatcher/internal/middleware"
 	"sneak.berlin/go/dnswatcher/internal/notify"
+	"sneak.berlin/go/dnswatcher/internal/portcheck"
+	"sneak.berlin/go/dnswatcher/internal/resolver"
 	"sneak.berlin/go/dnswatcher/internal/server"
 	"sneak.berlin/go/dnswatcher/internal/state"
+	"sneak.berlin/go/dnswatcher/internal/tlscheck"
+	"sneak.berlin/go/dnswatcher/internal/watcher"
 )
 
 // newServerApp builds an fx app holding a *server.Server wired exactly
-// as cmd/dnswatcher wires it, minus the watcher/resolver subtree that
-// would touch live DNS, plus the given option. config.New reads viper,
-// so the caller must first configure it, which is also why the caller
+// as cmd/dnswatcher wires it, plus the given option. The health check
+// follows the watcher, so the watcher is in the app too. Starting the
+// app starts the watcher's first round of checks, against live DNS,
+// before the server; the tests that start it expect the server to fail
+// to start, which stops the watcher again. config.New reads viper, so
+// the caller must first configure it, which is also why the caller
 // cannot run in parallel.
 func newServerApp(option fx.Option) *fx.App {
 	return fx.New(
@@ -33,11 +40,29 @@ func newServerApp(option fx.Option) *fx.App {
 			config.New,
 			state.New,
 			healthcheck.New,
+			resolver.New,
+			portcheck.New,
+			tlscheck.New,
 			notify.New,
+			watcher.New,
 			middleware.New,
 			handlers.New,
 			server.New,
 		),
+		fx.Provide(
+			func(r *resolver.Resolver) watcher.DNSResolver {
+				return r
+			},
+			func(p *portcheck.Checker) watcher.PortChecker {
+				return p
+			},
+			func(t *tlscheck.Checker) watcher.TLSChecker {
+				return t
+			},
+			func(n *notify.Service) watcher.Notifier {
+				return n
+			},
+		),
 		option,
 	)
 }

internal/watcher/dnsunreachable_test.go

@@ -0,0 +1,66 @@
+package watcher_test
+
+import (
+	"log/slog"
+	"testing"
+
+	"sneak.berlin/go/dnswatcher/internal/resolver"
+	"sneak.berlin/go/dnswatcher/internal/watcher"
+)
+
+// TestDNSUnreachableWhenNoRootServerGivesUsableReply runs one round of
+// checks of a domain and a hostname with every walk starting at real
+// servers that are not root servers. At 192.0.2.1, a documentation
+// address, nothing answers, as when no root server answers. Public
+// recursive resolvers of four operators mark their replies recursion
+// available, as on a network that intercepts DNS queries; one that gives
+// no reply is passed over for the next, and if none replies, no root
+// server answered. Either way every lookup failed because no root server
+// gave a usable reply, so the round could not reach the DNS.
+// TestFirstRunBaseline and TestDomainThatDoesNotExist check rounds in
+// which the root servers gave a usable reply.
+func TestDNSUnreachableWhenNoRootServerGivesUsableReply(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name    string
+		servers []string
+	}{
+		{"no root server answers", []string{"192.0.2.1"}},
+		{
+			"the network intercepts DNS queries",
+			[]string{"1.1.1.1", "8.8.8.8", "9.9.9.9", "208.67.222.222"},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			cfg := defaultTestConfig(t)
+			cfg.Domains = []string{testSmallDomain}
+			cfg.Hostnames = []string{testHost}
+
+			deps := newTestDeps(t, cfg)
+			w := watcher.NewForTest(
+				cfg,
+				deps.state,
+				resolver.NewWithRootServers(slog.Default(), tt.servers),
+				deps.portChecker,
+				deps.tlsChecker,
+				deps.notifier,
+			)
+
+			if w.DNSUnreachable() {
+				t.Error("DNSUnreachable is true before the first round")
+			}
+
+			w.RunOnce(t.Context())
+
+			if !w.DNSUnreachable() {
+				t.Error("DNSUnreachable is false after a round in " +
+					"which no root server gave a usable reply")
+			}
+		})
+	}
+}

internal/watcher/watcher.go

@@ -11,6 +11,7 @@ import (
 	"sort"
 	"strconv"
 	"strings"
+	"sync/atomic"
 	"time"
 
 	"go.uber.org/fx"
@@ -62,6 +63,11 @@ type Watcher struct {
 	cancel    context.CancelFunc
 	done      chan struct{} // closed when Run returns
 	firstRun  bool
+
+	// dnsUnreachable is what DNSUnreachable reports. Each round of DNS
+	// checks sets it, and the HTTP server reads it from its own
+	// goroutines.
+	dnsUnreachable atomic.Bool
 }
 
 // New creates a new Watcher instance wired into the fx lifecycle.
@@ -197,6 +203,18 @@ func (w *Watcher) RunOnce(ctx context.Context) {
 	w.firstRun = false
 }
 
+// DNSUnreachable reports whether the most recent round of DNS checks
+// could not reach the DNS at all: every lookup in it failed because no
+// root server gave a usable reply, as when none of them answered or the
+// network intercepts DNS queries. A round reached the DNS even when a
+// name's own nameservers did not answer or a domain does not exist,
+// since the root servers gave a usable reply to those lookups. It is
+// false until the first round finishes, and a round that shutdown cut
+// short changes nothing. It may be called while a round runs.
+func (w *Watcher) DNSUnreachable() bool {
+	return w.dnsUnreachable.Load()
+}
+
 func (w *Watcher) detectFirstRun() {
 	snap := w.state.GetSnapshot()
 	hasState := len(snap.Domains) > 0 ||
@@ -265,15 +283,40 @@ func (w *Watcher) cleanupRemovedTargets() {
 // runDNSChecks performs DNS resolution for all configured domains
 // and hostnames, updating state with freshly resolved records.
 // This must complete before port or TLS checks run so those
-// checks operate on current IP addresses.
+// checks operate on current IP addresses. Unless ctx was cut short, it
+// then records whether this round could reach the DNS (see
+// DNSUnreachable).
 func (w *Watcher) runDNSChecks(ctx context.Context) {
+	errs := make([]error, 0, len(w.config.Domains)+len(w.config.Hostnames))
+
 	for _, domain := range w.config.Domains {
-		w.checkDomain(ctx, domain)
+		errs = append(errs, w.checkDomain(ctx, domain))
 	}
 
 	for _, hostname := range w.config.Hostnames {
-		w.checkHostname(ctx, hostname)
+		_, err := w.checkHostname(ctx, hostname)
+		errs = append(errs, err)
 	}
+
+	if ctx.Err() == nil {
+		w.dnsUnreachable.Store(noRootServerGaveUsableReply(errs))
+	}
+}
+
+// noRootServerGaveUsableReply reports whether errs, the errors of the
+// lookups in a round of DNS checks, nil for a lookup that succeeded,
+// show that every lookup failed because no root server gave a usable
+// reply: each is ErrNoRootServerAnswered or ErrIntercepted. A round with
+// no lookups shows nothing.
+func noRootServerGaveUsableReply(errs []error) bool {
+	for _, err := range errs {
+		if !errors.Is(err, resolver.ErrNoRootServerAnswered) &&
+			!errors.Is(err, resolver.ErrIntercepted) {
+			return false
+		}
+	}
+
+	return len(errs) > 0
 }
 
 // logFailedLookup logs a failed DNS lookup at error level, unless ctx
@@ -292,14 +335,16 @@ func (w *Watcher) logFailedLookup(
 }
 
 // checkDomain looks up domain's delegation and checks domain with it
-// (see checkDomainResult).
+// (see checkDomainResult). It returns the lookup's error.
 func (w *Watcher) checkDomain(
 	ctx context.Context,
 	domain string,
-) {
+) error {
 	nameservers, err := w.resolver.LookupNS(ctx, domain)
 
 	w.checkDomainResult(ctx, domain, nameservers, err)
+
+	return err
 }
 
 // checkDomainResult finishes a check of domain from nameservers and err,
@@ -358,7 +403,7 @@ func (w *Watcher) checkDomainResult(
 	// check, as detectNSDisappearances needs.
 	if nxdomain {
 		w.state.DeleteHostnameState(domain)
-	} else if !w.checkHostname(ctx, domain) {
+	} else if saved, _ := w.checkHostname(ctx, domain); !saved {
 		return
 	}
 
@@ -484,11 +529,12 @@ func (w *Watcher) detectNSAddressChanges(
 }
 
 // checkHostname looks up hostname's records and checks them (see
-// checkHostnameResult). It reports whether it saved them.
+// checkHostnameResult). It reports whether it saved them, and returns
+// the lookup's error.
 func (w *Watcher) checkHostname(
 	ctx context.Context,
 	hostname string,
-) bool {
+) (bool, error) {
 	results, err := w.resolver.LookupAllRecords(ctx, hostname)
 	if err != nil {
 		w.logFailedLookup(
@@ -499,7 +545,7 @@ func (w *Watcher) checkHostname(
 		)
 	}
 
-	return w.checkHostnameResult(ctx, hostname, results, err)
+	return w.checkHostnameResult(ctx, hostname, results, err), err
 }
 
 // checkHostnameResult finishes a check of hostname from results and err,

internal/watcher/watcher_test.go

@@ -385,11 +385,16 @@ func TestFirstRunBaseline(t *testing.T) {
 	cfg.Domains = []string{testSmallDomain}
 	cfg.Hostnames = []string{testHost}
 
-	_, deps := runChecks(t, cfg, nil)
+	w, deps := runChecks(t, cfg, nil)
 
 	assertNoNotifications(t, deps)
 	assertStatePopulated(t, deps)
 
+	if w.DNSUnreachable() {
+		t.Error("DNSUnreachable is true after a round whose lookups " +
+			"succeeded")
+	}
+
 	// testHost answers with an address, so the check saves an empty list
 	// of CNAME addresses for it; nil would mean the check did not look
 	// at whether to follow a CNAME.
@@ -551,7 +556,9 @@ func TestNSChangeDetection(t *testing.T) {
 // with nameservers and records saved by an earlier check. The .com
 // servers answer that it does not exist, so it is saved with nxdomain
 // set and no nameservers, an NS Change removes them all, and its saved
-// records are removed rather than asked for at the .com servers.
+// records are removed rather than asked for at the .com servers. The
+// round's only lookup failed, but the root servers and the .com servers
+// answered, so the round reached the DNS.
 func TestDomainThatDoesNotExist(t *testing.T) {
 	t.Parallel()
 
@@ -560,11 +567,12 @@ func TestDomainThatDoesNotExist(t *testing.T) {
 	cfg := defaultTestConfig(t)
 	cfg.Domains = []string{domain}
 
-	var deps *testDeps
+	var (
+		w    *watcher.Watcher
+		deps *testDeps
+	)
 
 	livednstest.Retry(t, "watcher checks", func(ctx context.Context) error {
-		var w *watcher.Watcher
-
 		w, deps = newTestWatcher(t, cfg)
 
 		deps.state.SetDomainState(domain, &state.DomainState{
@@ -602,6 +610,11 @@ func TestDomainThatDoesNotExist(t *testing.T) {
 		t.Errorf("records saved for %s: %v", domain, hs.RecordsByNameserver)
 	}
 
+	if w.DNSUnreachable() {
+		t.Error("DNSUnreachable is true after a round that reached " +
+			"the .com servers")
+	}
+
 	assertNotified(t, deps, "NS Change: "+domain, "warning")
 
 	// That is the only notification, and it removes both nameservers,