diff --git a/TODO.md b/TODO.md index 7462510..8af2f80 100644 --- a/TODO.md +++ b/TODO.md @@ -1,24 +1,27 @@ # Workflow -* branch (from `main`) +* branch (from `next`) * do the work in Next Step * move Next Step to the top of Completed Steps * move the top item of Future Steps into Next Step * commit (`TODO.md` changes in the same commit as the work) -* merge to `main` if the branch is not protected, otherwise open a PR * push +* open a PR against `next` # Status -pre-1.0. No git tags. +pre-1.0. No git tags. Work lands on `next` by PR. Open work for 1.0 is tracked +on the 1.0 milestone: https://git.eeqj.de/sneak/dnswatcher/milestone/7 # Next Step -Add the README sections required by policy (Description, Getting Started, -Rationale, Design, TODO, License, Author) if any are still missing. +NS failure and NS recovery notifications: +https://git.eeqj.de/sneak/dnswatcher/issues/104 # Completed Steps +- 2026-10-01: `TODO.md` brought up to date: open issues listed by URL, every + Completed Steps entry cut to at most two lines (closes #146). - 2026-10-01: wildcard CORS now applies only to the public routes, not to `/metrics`, and allows only the methods they serve (closes #100). - 2026-10-01: `internal/state` and `internal/watcher` no longer export test-only @@ -27,259 +30,83 @@ Rationale, Design, TODO, License, Author) if any are still missing. the bound they check, and require the drain's debug line (closes #116). - 2026-09-29: the entrypoint chowns the data directory to `dnswatcher` and runs dnswatcher as that user, so a host bind mount needs no chown (closes #166). -- 2026-09-29: the live-DNS test package is renamed `internal/livednstest` and - added to the `test-support` `deny` list in `.golangci.yml`, so `make lint` - fails when program code imports it (closes #164). -- 2026-09-29: `.golangci.yml` re-fetched unchanged from `sneak/prompts`. It - replaces the deprecated `gomodguard` with `gomodguard_v2`, so `make lint` no - longer warns about it, and turns on `depguard` with the org `test-support` - rule, which rejects `net/http/httptest` except in test files and in files - under a directory whose name ends in `test`. This repo had no `deny` entries - of its own to carry forward (closes #123). -- 2026-09-29: nothing stands in for DNS any more. Watcher tests that look - something up in DNS use the real resolver against live DNS servers and test - record and nameserver changes by preparing the saved state a check starts - from; the resolver timeout test queries an address that never answers, and - `NewFromLoggerWithClient`, used only by its stand-in client, is gone. The - live-DNS retry and concurrency limit moved to `internal/livednstest`, which - both test packages use. `TESTING.md` states the README's rule (closes #159). -- 2026-09-28: the inconsistency alert is sent once, on the check where two - nameservers start to disagree or where a nameserver that disagrees first - appears, instead of on every check while they disagree, and not again after - a restart. Every pair of nameservers is compared, not only neighbours in - sorted order of name (closes #158). +- 2026-09-29: the live-DNS test package is renamed `internal/livednstest`; + `make lint` fails when program code imports it (closes #164). +- 2026-09-29: `.golangci.yml` re-fetched from `sneak/prompts`, with + `gomodguard_v2` and the org `depguard` `test-support` rule (closes #123). +- 2026-09-29: watcher and resolver tests that look something up in DNS use the + real resolver against live DNS servers (closes #159). +- 2026-09-28: the inconsistency alert is sent once, when two nameservers start + to disagree; every pair of nameservers is compared (closes #158). - 2026-09-28: DNS names in record values (CNAME, MX, SRV and NS targets) are - lower-cased, so nameservers that answer in different letter case no longer - count as inconsistent or as a record change (closes #157). -- 2026-09-28: `script/cibuild` and `script/docker` now pass - `--no-cache-filter=lint,builder` so lint and tests run every build (closes - #115). -- 2026-09-28: the server timeout test now drives `Run` and checks the - `http.Server` it serves carries the timeouts; corrected the `ReadTimeout` - note in that test (closes #120). -- 2026-09-28: upaas deploy readiness — runtime image runs as unprivileged - `dnswatcher`, Docker `HEALTHCHECK`, startup fails when the data directory is - not writable, README "Running under upaas" (closes #147). + lower-cased, so letter case alone is not a change (closes #157). +- 2026-09-28: lint and tests run on every build: `script/cibuild` and + `script/docker` pass `--no-cache-filter=lint,builder` (closes #115). +- 2026-09-28: the server timeout test drives `Run` and checks the timeouts on + the `http.Server` it serves (closes #120). +- 2026-09-28: upaas deploy readiness: the image runs as user `dnswatcher` with a + `HEALTHCHECK`; README "Running under upaas" (closes #147). - 2026-09-21: added behavioural tests for `internal/globals`, `internal/healthcheck`, and `internal/logger` (closes #110). - 2026-09-21: `go mod tidy` dropped the redundant `golang.org/x/sync` `// indirect` line so `script/bootstrap` leaves a clean tree (#132) -- 2026-08-10: comment-only corrections to `script/bootstrap`, - `script/cibuild`, and `Dockerfile.lint`. The `goimports` pin in - `script/bootstrap` was justified by a claim that `script/fmt-check` - runs it on the host; it does not (it runs `gofmt -l .` only), so the - header now credits `script/fmt` alone. `script/cibuild` still claimed - the `Dockerfile` runs `make check`, which stopped being true when - linting moved to its own stage; it now describes the lint stage - (`make fmt-check` plus `golangci-lint`) and the builder stage - (`make test`, `make build`). The `docker`-missing warning in - `script/bootstrap` reads as one sentence instead of three fragments - each re-prefixed with `bootstrap:`. `Dockerfile.lint` now records the - residual risk of omitting `golangci-lint config verify`: unknown - top-level keys in `.golangci.yml` are silently ignored, so a mistyped - key lints clean while applying nothing. No behaviour changed -- 2026-08-10: MIT `LICENSE` added at the repository root, closing the - last gap in `REPO_POLICIES.md`'s required-minimum file list and - removing the all-rights-reserved default that would otherwise have - shipped with a 1.0 tag. The licence choice is the standing org policy - (any public repo lacking a licence gets MIT; a private repo with no - licence is already all-rights-reserved), and this repo is public. The - file holds the canonical MIT text byte-for-byte with only the - copyright line filled in (`Copyright (c) 2026 sneak`); no clauses were - added, removed, or reflowed. `README.md`'s first line now names the - licence, as the Description requirement demands, and the License - section states MIT and points at the file instead of saying the choice - is pending. `make fmt` covers only Go sources (`gofmt -s`, - `goimports`), so it cannot reflow `LICENSE` -- 2026-08-10: the policy scaffold (`REPO_POLICIES.md`, `.editorconfig`, - `.dockerignore`, `.gitea/workflows/check.yml`, and the `fmt-check`, - `docker`, and hooks Makefile targets) is present; it landed piecemeal - across the scripts-to-rule-them-all and policy commits rather than as - the single commit this file once planned -- 2026-08-10: Go's test cache disabled for `script/test` via `-count=1`, - so every invocation actually executes. A cached pass replays an - earlier run's output without querying DNS at all, which in this repo - means the suite's entire premise goes unexercised while the run - reports green in under a second. The conditional verbose rerun that - `REPO_POLICIES.md` mandates was added at the same time (the primary - run had been unconditionally `-v`): quiet first, `-v` only on - failure, `-count=1` on both, and exit 1 forced regardless of the - rerun's result so a flake passing the second time cannot turn the - build green. `-timeout 90s` left alone as the deliberate backstop - above the 60s hard cap. Uncached suite runs ~4s, well inside the 20s - target -- 2026-08-10: live-DNS test flakiness addressed by robustness rather - than gating, per the owner's ruling on #93: new - `internal/resolver/livedns_test.go` adds a package-wide concurrency - gate (so parallel tests stop bursting at the first root server), - retry with exponential backoff on transport failures only, and - quorum instead of unanimity for multi-nameserver assertions. Quorum - tolerates silence only: every per-nameserver status must be in a - closed allowlist (`ok`/`timeout`/`error`, or - `nxdomain`/`timeout`/`error`), so a wrong answer from a minority — - `nodata` today, any status added later — fails the test instead of - sliding through under the majority. The - `make test` cap moved to the new org-wide 60s hard cap / 20s target - with a 90s `-timeout` backstop; `REPO_POLICIES.md` re-vendored - byte-identical from `sneak/prompts`. No mocks, no `-short`, no build - tags, no skips, and no change to production resolver behaviour -- 2026-08-10: all linting moved into Docker: new root `Dockerfile.lint` - on the digest-pinned `golangci/golangci-lint:v2.12.2` image, - `script/lint` reduced to a thin wrapper that builds it with - `--no-cache-filter=lint` so the linter actually executes every run, - golangci-lint install dropped from `script/bootstrap` (goimports - stays, `script/fmt` needs it on the host), and the root `Dockerfile` - given its own lint stage so its build no longer recurses through - `make check` into `script/lint`. `golangci-lint config verify` is - deliberately omitted: it fetches its schema over an unpinned live - HTTPS call -- 2026-08-09: in-flight notification deliveries are now drained at - shutdown (#106): `notify.New` registers an fx `OnStop` hook that waits - on a `sync.WaitGroup` of tracked delivery goroutines, bounded by the - `OnStop` context; on expiry the outstanding count is logged at warn - level and parked retry backoffs are released instead of being dropped - silently, and deliveries submitted after the drain begins are refused - so shutdown cannot be extended indefinitely; an `OnStop` context that - is already expired on entry with nothing outstanding drains quietly - rather than warning about deliveries that were never abandoned -- 2026-08-09: `http.Server` now sets all four socket-level timeouts - (`ReadTimeout` 15s, `ReadHeaderTimeout` 10s, `WriteTimeout` 75s, - `IdleTimeout` 120s) as named constants in `internal/server/server.go`, - closing the slowloris / unreaped-keep-alive exposure required by - `REPO_POLICIES.md` before 1.0; `WriteTimeout` is deliberately greater - than the 60s `chimw.Timeout` handler budget so that budget stays - reachable, and tests in `internal/server` pin both the non-zero - values and that relationship (#99) -- 2026-08-09: security response headers middleware - (`SecurityHeaders()` in `internal/middleware/middleware.go`) - registered globally in `internal/server/routes.go`, so HSTS, CSP, - `X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy`, and - `Permissions-Policy` are set on every response including `/s/...` and - `/metrics`; the CSP needs no `unsafe-inline`/`unsafe-eval` because the - dashboard ships no JavaScript and no inline styles; HSTS is emitted - unconditionally per policy (TLS-terminating proxy in front). Remaining - 1.0 hardening items — `http.Server` timeouts, request body limits, - rate limiting, CORS scoping — are tracked separately -- 2026-08-07: golangci-lint bumped to v2.12.2 (commit-pinned installs - in `Dockerfile` and `script/bootstrap`); `.golangci.yml` set to the - org-standard v2-schema config used across the org's repos - (owner-authorized; same file is being landed as canonical via prompts - PR #24), with settings under `linters.settings` so the - lll/funlen/cyclop/dupl thresholds apply; fixed the resulting - `goconst`, `dupl`, and `lll` findings; the informational `gomodguard` - deprecation warning under this config is accepted -- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, - Makefile shims, README Entrypoints section -- 2026-02-20: iterative DNS resolver implemented; tests made hermetic - with mocked DNS (origin/feature/resolver, unmerged) -- 2026-02-20: CI actions and go install refs pinned to commit SHAs; - Gitea Actions workflow for make check (origin/ci/make-check, unmerged) +- 2026-08-10: comment-only corrections to `script/bootstrap`, `script/cibuild` + and `Dockerfile.lint`; no behaviour changed. +- 2026-08-10: MIT `LICENSE` added at the repository root; the README's first + line and License section name the licence. +- 2026-08-10: policy scaffold present: `REPO_POLICIES.md`, `.editorconfig`, + `.dockerignore`, CI workflow, `make fmt-check`, `make docker`, `make hooks`. +- 2026-08-10: Go's test cache disabled in `script/test` (`-count=1`), so every + run queries live DNS; a failed run is rerun with `-v`. +- 2026-08-10: live-DNS tests made robust rather than gated (#93): a limit on + concurrent lookups, retries, and a quorum across nameservers. +- 2026-08-10: all linting moved into Docker: `script/lint` builds + `Dockerfile.lint`, and the root `Dockerfile` has its own lint stage. +- 2026-08-09: in-flight notification deliveries are drained at shutdown, bounded + by the shutdown deadline (#106). +- 2026-08-09: `http.Server` sets all four socket timeouts; `WriteTimeout` stays + above the 60s handler timeout (#99). +- 2026-08-09: `SecurityHeaders()` middleware sets HSTS, CSP and the other + security headers `REPO_POLICIES.md` requires on every response. +- 2026-08-07: golangci-lint bumped to v2.12.2 and `.golangci.yml` set to the org + config; fixed the resulting `goconst`, `dupl` and `lll` findings. +- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile + shims, README Entrypoints section +- 2026-02-20: iterative DNS resolver implemented +- 2026-02-20: CI actions and go install refs pinned to commit SHAs; Gitea + Actions workflow added - 2026-02-20: watcher monitoring orchestrator merged to main (#8) - 2026-02-20: DOMAINS/HOSTNAMES unified into single TARGETS config (#11) - 2026-02-19: TCP port connectivity checker, made concurrent with port validation; gosec G704 SSRF findings fixed without suppression - (feature branches, unmerged) -- 2026-02-19: TLS certificate inspector with no-peer-certificates error - path and IP SANs (feature branch, unmerged) +- 2026-02-19: TLS certificate inspector with no-peer-certificates error path and + IP SANs - 2026-02-19: gosec SSRF and formatting fixes on main - 2026-02-19: initial scaffold with per-nameserver DNS monitoring model # Future Steps -Compliance: - -- Pin Dockerfile base images by sha256 and ensure the Docker build runs - make check - -Branch reconciliation: - -- Sync local checkout with origin: local main is 8 commits behind - origin/main; local feature/resolver has diverged from - origin/feature/resolver, which already implements the resolver -- Merge in-flight branches to main once green: feature/resolver, - ci/make-check, feature/portcheck-implementation, - feature/tlscheck-implementation - -Resolver (plan from untracked TODO.md; largely implemented on -origin/feature/resolver, verify each item before closing): - -- Add github.com/miekg/dns dependency -- roots.go: hardcoded IANA root server list (a through m, IPv4/IPv6), - rootServers() returning ip:53 strings -- query.go: low-level query(ctx, server, name, qtype): UDP with TCP - fallback on truncation, RD=0, context respected, 5s per-query timeout, - returns raw *dns.Msg -- trace.go: iterative delegation chasing from roots: referral detection - (NOERROR, empty answer, NS in authority), glue extraction with - bailiwick check, out-of-bailiwick NS resolved with recursion guard, - delegation depth limit (20), retry across nameservers on failure, do - not chase CNAMEs inside trace -- FindAuthoritativeNameservers: NS set via trace, sorted, FQDN - normalized, trailing dot handled; must pass its 9 tests -- QueryNameserver: resolve NS host to IPs, query A/AAAA/CNAME/MX/TXT/ - SRV/CAA/NS, build NameserverResponse with status mapping (OK, - NXDomain, NoData, Error), documented record formatting, sorted values, - lame delegation detection; must pass its 16 tests -- QueryAllNameservers: find NS set for parent domain (public suffix - list), query all NS in parallel with bounded concurrency, return map - even when all fail, context cancellation; must pass its 4 tests -- LookupNS: thin wrapper over FindAuthoritativeNameservers, sorted, - identical results; must pass its 3 tests -- ResolveIPAddresses: collect A/AAAA from all NS, follow CNAME chains - with MaxCNAMEDepth, dedupe, sort, NXDOMAIN returns empty slice with - nil error; must pass its 9 tests -- All 39 resolver tests pass, make check green, merge to main - -Watcher (internal/watcher/watcher.go): - -- Scheduling loop in Run(ctx): initial check on startup, separate - tickers for DNS/port and TLS intervals, persist state via state.Save() - after each cycle, clean shutdown on context cancel -- Domain check: LookupNS, compare to stored state, store silently on - first run, notify with old/new NS lists on change -- Hostname check: QueryAllNameservers, compare per-NS records; notify on - record changes, NS failure, NS recovery, inconsistency detected, - inconsistency resolved, empty response; store silently on first run -- Port check: ResolveIPAddresses, check ports 80 and 443 per IP, notify - on open/closed transitions, handle new and disappeared IPs -- TLS check: for each open IP:443, CheckCertificate; notify on expiry - warning, certificate change (CN/issuer/SANs), TLS failure/recovery - -Port checker (internal/portcheck/portcheck.go): - -- Tests against known-open ports and RFC documentation IPs -- CheckPort: net.DialTimeout (5s), context respected; (true, nil) open, - (false, nil) closed/timeout/refused, error only for unexpected - failures - -TLS checker (internal/tlscheck/tlscheck.go): - -- Tests against known public HTTPS servers, verify fields populated -- CheckCertificate: tls.Dial to specific IP:443 with hostname as SNI; - extract subject CN, issuer CN and org, NotAfter, SANs; error on - handshake failure - -Notification service (internal/notify/notify.go, Slack/Mattermost/ntfy -backends exist): - -- Structured notification types: DNS change, port change, TLS expiry, - TLS change, NS failure, NS recovery, NS inconsistency -- Per-backend formatting: Slack/Mattermost attachment colors (red - failures/expiry, yellow warnings, green recoveries, blue info); ntfy - priorities (urgent failures, high warnings, default changes, low - recoveries); include hostname, nameserver, old/new values, timestamps - -HTTP API handlers: - -- Wire *state.State and *watcher.Watcher into handler params -- GET /api/v1/status: full state snapshot as JSON -- GET /api/v1/domains: domain states with NS records and last-checked -- GET /api/v1/hostnames: hostname states with per-NS record data - -Infrastructure notes (from untracked TODO.md): - -- Module path sneak.berlin/go/dnswatcher differs from the git.eeqj.de - remote intentionally; do not "fix" it -- Dependencies: github.com/miekg/dns, golang.org/x/net/publicsuffix -- DNS is never mocked; tests that look something up in DNS query live DNS - servers (README, "No DNS mocking. Ever.") +- nameserver IP address changes: https://git.eeqj.de/sneak/dnswatcher/issues/105 +- `DNSWATCHER_SENTRY_DSN` does nothing: + https://git.eeqj.de/sneak/dnswatcher/issues/107 +- invalid DNS or TLS interval silently replaced by the default: + https://git.eeqj.de/sneak/dnswatcher/issues/177 +- rate limit on `/metrics` Basic Auth: + https://git.eeqj.de/sneak/dnswatcher/issues/101 +- images report version `dev`: https://git.eeqj.de/sneak/dnswatcher/issues/109 +- trial run of the finished image: + https://git.eeqj.de/sneak/dnswatcher/issues/149 +- 1.0 readiness: run it with a real config and read the logs: + https://git.eeqj.de/sneak/dnswatcher/issues/66 +- `goimports` in `make fmt-check`, Markdown formatting: + https://git.eeqj.de/sneak/dnswatcher/issues/119 +- final state save at shutdown: https://git.eeqj.de/sneak/dnswatcher/issues/114 +- `internal/notify` shutdown tests hang when a drain returns early: + https://git.eeqj.de/sneak/dnswatcher/issues/176 +- README accuracy sweep: https://git.eeqj.de/sneak/dnswatcher/issues/108 +- README sections required by policy: + https://git.eeqj.de/sneak/dnswatcher/issues/173 +- `script/install-precommit` in a linked worktree: + https://git.eeqj.de/sneak/dnswatcher/issues/129 +- fixed root server order: https://git.eeqj.de/sneak/dnswatcher/issues/138 +- review toward 1.0: https://git.eeqj.de/sneak/dnswatcher/issues/144