The health check answers healthy while every DNS lookup fails #302

New Issue

2026-10-16T11:12:15+02:00

clawbot commented

2026-10-16 11:12:15 +02:00

/.well-known/healthcheck answers healthy while every DNS lookup fails, for example on a network that intercepts DNS queries (#300). It reports only that the process is running, so upaas and any monitor see a healthy app that checks nothing. Noted by the reviewer of #301; it was already so before that PR.

Done when: the health check reports unhealthy when the most recent round of checks could not reach the DNS at all, with a test against the real DNS (never mocked).

Model: opus-5-5

`/.well-known/healthcheck` answers healthy while every DNS lookup fails, for example on a network that intercepts DNS queries (https://git.eeqj.de/sneak/dnswatcher/issues/300). It reports only that the process is running, so upaas and any monitor see a healthy app that checks nothing. Noted by the reviewer of https://git.eeqj.de/sneak/dnswatcher/pulls/301; it was already so before that PR. **Done when:** the health check reports unhealthy when the most recent round of checks could not reach the DNS at all, with a test against the real DNS (never mocked). Model: opus-5-5

clawbot commented

2026-10-18 15:12:15 +02:00

Plan

"Could not reach the DNS at all" means: in the most recent round of DNS checks, no lookup got a reply from a root server. Each one failed because no root server answered, or with ErrIntercepted ("this network intercepts DNS queries"). A round where the root servers answered but a target's own nameservers did not, or where a domain does not exist, did reach the DNS. dnswatcher reports those through its notifications, and the health check stays ok.

Changes:

internal/resolver (iterative resolution): when the root servers are asked and none gives a usable reply, and they did not all refuse (that case is already ErrIntercepted), the error wraps a new sentinel ErrNoRootServerAnswered ("no root server answered"), next to ErrNoNameserverAnswered. Make sure it reaches the watcher through LookupNS and LookupAllRecords with %w.
internal/watcher (runs the checks): at the end of each round of DNS checks (runDNSChecks) that its context did not cut short, record whether every lookup in the round failed with ErrIntercepted or ErrNoRootServerAnswered. Add one method the health check reads it with, safe to call from the HTTP server while a round runs. Until the first round finishes nothing is known, and the health check stays ok.
internal/healthcheck (health status): takes the watcher. Check gives status error when the most recent round could not reach the DNS, and ok otherwise.
internal/handlers: the health check answers 503 when status is error and 200 otherwise, on both /.well-known/healthcheck and /health. The image's wget health check then fails, and Docker and upaas see the container as unhealthy.
README.md: the health check entry says it follows the most recent round of DNS checks, what error means, and that it then answers 503. TODO.md: a Completed Steps entry in the same commit.

Tests (real DNS, never mocked; no stand-in resolver or watcher):

A watcher whose resolver starts its walks at real servers that are not root servers gets error and 503 after one round. Use a public recursive resolver, whose reply is marked recursion available (ErrIntercepted), and 192.0.2.1, a documentation address where nothing answers (ErrNoRootServerAnswered).
A watcher using the real root servers gets ok, both with a normal target and with a target that does not exist (every lookup fails, but the DNS was reached).
For other packages' tests to start a resolver's walks at given servers, add one exported constructor next to NewFromLogger, which is already there for tests, such as NewWithRootServers(log, servers). Nothing else in production code exists only for tests.
Use internal/livednstest for retries, and keep make test within its 20-second target.

Out of scope: the dashboard, /api/v1/status, metrics.

Model: opus-5-5

**Plan** "Could not reach the DNS at all" means: in the most recent round of DNS checks, no lookup got a reply from a root server. Each one failed because no root server answered, or with `ErrIntercepted` ("this network intercepts DNS queries"). A round where the root servers answered but a target's own nameservers did not, or where a domain does not exist, did reach the DNS. dnswatcher reports those through its notifications, and the health check stays `ok`. Changes: - `internal/resolver` (iterative resolution): when the root servers are asked and none gives a usable reply, and they did not all refuse (that case is already `ErrIntercepted`), the error wraps a new sentinel `ErrNoRootServerAnswered` ("no root server answered"), next to `ErrNoNameserverAnswered`. Make sure it reaches the watcher through `LookupNS` and `LookupAllRecords` with `%w`. - `internal/watcher` (runs the checks): at the end of each round of DNS checks (`runDNSChecks`) that its context did not cut short, record whether every lookup in the round failed with `ErrIntercepted` or `ErrNoRootServerAnswered`. Add one method the health check reads it with, safe to call from the HTTP server while a round runs. Until the first round finishes nothing is known, and the health check stays `ok`. - `internal/healthcheck` (health status): takes the watcher. `Check` gives `status` `error` when the most recent round could not reach the DNS, and `ok` otherwise. - `internal/handlers`: the health check answers 503 when `status` is `error` and 200 otherwise, on both `/.well-known/healthcheck` and `/health`. The image's `wget` health check then fails, and Docker and upaas see the container as unhealthy. - `README.md`: the health check entry says it follows the most recent round of DNS checks, what `error` means, and that it then answers 503. `TODO.md`: a Completed Steps entry in the same commit. Tests (real DNS, never mocked; no stand-in resolver or watcher): - A watcher whose resolver starts its walks at real servers that are not root servers gets `error` and 503 after one round. Use a public recursive resolver, whose reply is marked recursion available (`ErrIntercepted`), and `192.0.2.1`, a documentation address where nothing answers (`ErrNoRootServerAnswered`). - A watcher using the real root servers gets `ok`, both with a normal target and with a target that does not exist (every lookup fails, but the DNS was reached). - For other packages' tests to start a resolver's walks at given servers, add one exported constructor next to `NewFromLogger`, which is already there for tests, such as `NewWithRootServers(log, servers)`. Nothing else in production code exists only for tests. - Use `internal/livednstest` for retries, and keep `make test` within its 20-second target. Out of scope: the dashboard, `/api/v1/status`, metrics. Model: opus-5-5

clawbot referenced this issue from a commit

2026-10-18 15:30:33 +02:00

healthcheck: answer 503 when the last round of DNS checks reached no root server (closes #302)

clawbot referenced this issue from a commit

2026-10-18 15:30:47 +02:00

healthcheck: answer 503 when the last round of DNS checks reached no root server (closes #302)

clawbot referenced a pull request that will close this issue

2026-10-18 15:33:39 +02:00

healthcheck: answer 503 when no root server gave a usable reply in the last round of DNS checks (closes #302) #304

clawbot referenced this issue from a commit

2026-10-18 16:07:59 +02:00

docs: health check wording follows the code (rework for #302)