healthcheck: answer 503 when no root server gave a usable reply in the last round of DNS checks (closes #302)

The health check answered ok while every DNS lookup failed, as on a network that intercepts DNS queries. A lookup whose root servers give no usable reply now fails with ErrNoRootServerAnswered. At the end of each round of DNS checks that shutdown did not cut short, the watcher records whether every lookup failed with it or with ErrIntercepted; the health check then gives status error, and /.well-known/healthcheck and /health answer 503. Until the first round finishes it stays ok. The health check now depends on the watcher, so fx starts the watcher before the HTTP server and stops the server first. NewWithRootServers lets tests start a resolver's walks at real servers that are not root servers. Model: opus-5-5
2026-10-18 17:24:49 +02:00
parent b23cf43abf
commit 774f7caff6
14 changed files with 534 additions and 45 deletions
@@ -355,6 +355,17 @@ dnswatcher exposes a lightweight HTTP API for operational visibility:
 | `GET /api/v1/status`           | Current monitoring state      |
 | `GET /metrics`                 | Prometheus metrics, see below |

+The health check, at `/.well-known/healthcheck` and `/health`, follows the most
+recent round of DNS checks. When every lookup in that round failed because no
+root server gave a usable reply, as when none of them answers or the network
+intercepts DNS queries (see DNS Resolution Strategy), dnswatcher could not reach
+the DNS at all. The health check then gives `status` `error` and answers
+`503 Service Unavailable`, so the image's health check fails and Docker and
+upaas see the container as unhealthy. Otherwise it gives `ok` and answers
+`200 OK`, as it also does before the first round finishes. A round gives `ok`
+even when a name's own nameservers do not answer or a domain does not exist,
+since the root servers gave a usable reply to those lookups.
+
 In `/api/v1/status`, each nameserver entry and certificate entry whose `status`
 is `error` also has `error`, the reason, as in the state file (see State File
 Format). A domain's own records are in its entry in `domains`, under