netwatch/TODO.md

Workflow

• branch (from main)
• do the work in Next Step
• move Next Step to the top of Completed Steps
• move the top item of Future Steps into Next Step
• commit (TODO.md changes in the same commit as the work)
• merge to main if the branch is not protected, otherwise open a PR
• push

Status

pre-1.0. No git tags. feat/reportbuf-storage is merged; the backend, the CI workflow, and the backend repo standard files are all on main. Frontend and backend are both functional. Working toward the 1.0.0 milestone by closing the remaining repo-compliance issues on the tracker.

Next Step

Confirm the .gitea/workflows/check.yml run is green (main always green policy). The workflow file is already on main; what is unverified is that its latest run passes.

Completed Steps

• 2026-10-04: script/ and Makefile follow the org models (issue #28): make dev shims to the new script/dev, the Vite dev server, and the new make build to script/build, the frontend production build. .prettierignore no longer leaves out backend/, so make fmt and make fmt-check cover backend/README.md; it leaves out backend/.golangci.yml by name, the org standard file whose sha256 backend/script/lint checks. script/install-precommit and the date on script/bootstrap's pins are the org model again; script/bootstrap, script/fmt and script/fmt-check each say in a comment why they differ from it
• 2026-10-04: a frontend build on Node 26 or newer, such as make test on a host with Node 26, no longer prints Node's warning that module.register() is deprecated (issue #32); the build in Dockerfile runs on Node 22, which never printed it. The call was in @tailwindcss/node, which @tailwindcss/vite brings in at its own exact version; tailwind 4.3.1 calls module.registerHooks() instead where Node has it. yarn.lock now has @tailwindcss/vite and tailwindcss at 4.3.3, inside the ranges package.json already allowed, and tailwind's own dependencies moved with them. The built CSS changes only in how it is written out, in tailwind's Firefox focus-ring rule, which no longer applies to iframes (the page has none, so nothing on it looks different), and in tailwind's default sans-serif font list, which the page does not use: body sets a monospace font
• 2026-10-03: the frontend's unit tests cover what the page computes (issue #21): humanDuration, the latency colours of a figure and of a sparkline either side of each boundary, a target's min, max, average and median latency over an empty history, an all-unreachable one and a mixed one, and each of the four health states either side of its thresholds. package.json has a test script, so yarn run test and npm run test run them, and script/frontend-test runs it: quietly, and if a test fails, again with every test listed, and then fails. src/main.js now exports humanDuration, HostState, latencyClass and latencyHex for the tests; nothing it does changed
• 2026-10-03: the backend's logs are one stream (issue #27): fx logs its own steps of starting and stopping through the backend's logger, so off a terminal every line the backend's own logger and fx write is JSON, where fx used to write plain text to stderr. A config file that is found but cannot be read now stops the start with its error, logged as JSON like a bad setting, where it used to end in a Go panic. A test runs the server as a child process and checks both. The backend logs its name, version and architecture once at start. The health check's uptime keys are now uptime_seconds and uptime_human; its path, content type, "status":"ok" and 200 are unchanged. SENTRY_DSN, METRICS_USERNAME and METRICS_PASSWORD are still read and still unused, and left out of backend/README.md, until issues #94 and #95 wire them up
• 2026-10-03: the frontend has a real linter (issue #47, and item 2 of issue #28): eslint with its recommended rules, set in eslint.config.js, runs in a new frontend-lint stage of Dockerfile, which the frontend stage waits on, as the builder stage waits on the Go lint stage. script/lint builds both stages without the cache and runs no linter on the host. script/frontend-lint runs eslint where it used to repeat the prettier check that script/fmt-check runs on the host, and script/frontend-check, run by the frontend stage, is now the tests and the format check. script/bootstrap wants node 22.13.0 or newer, as eslint 10 does
• 2026-10-03: the tap-target check in make frontend-viewport-test expects one visible pin button per WAN host row (issue #46), where it expected at least 10 of the 26, so pin buttons missing from only some rows now fail it. The host row count the harness gathers, which the app-rendered check also reads, now counts only the WAN host rows: the local host rows have no pin button
• 2026-10-03: each target's row shows its result as soon as its check ends (issue #91), where every row waited for the round's slowest check, up to 24 seconds at a 30-second interval. Every row is still redrawn, and sorting, the summary, the health box and offline detection still run, once, when the round's last check ends, so no row reads "paused" after a pause and resume during the round. A check that ends after the user pauses or after its round is given up shows nothing, and the first round is still discarded as a whole
• 2026-10-03: root no longer acts outside /data when it prepares DATA_DIR (issue #80): bin/entrypoint.sh runs netwatch-server prepare-data-dir, which refuses a DATA_DIR that is not /data or a path below it written in full, then creates DATA_DIR, gives /data and everything in it to netwatch and sets the modes, all through a Go os.Root opened on /data. That refuses any path leading out of /data, so neither a symbolic link already there nor one a host process swaps in during the start can make root create or change anything elsewhere, and DATA_DIR=/etc no longer gives /etc to netwatch. The README.md section "Running under upaas" says which values are accepted
• 2026-10-03: DATA_DIR_MAX_BYTES is now how much of the report files is kept (issue #54): when a report would take them past it, the oldest report files are deleted to make room, each deletion logged, and at start files already past it are deleted the same way. A file still being written is never deleted. A report is refused with 507 only when the reports waiting to be written fill the cap on their own, and then no file is deleted. The reports of a failed write stop counting, and the part of its file written is removed. A file that cannot be deleted still counts until the next start; one already deleted by hand counts as freed
• 2026-10-03: backend/script/lint says what went wrong with its .golangci.yml check (issue #34). On a hash mismatch it says to compare the file with the org standard: if they differ, restore the org standard; if they are the same, the org standard changed, so update GOLANGCI_CONFIG_SHA256 in that script. It used to say only to restore the file, which loops once the org standard itself has moved. A missing .golangci.yml, and a sha256sum that is missing or prints no hash, each get their own message instead of being reported as a mismatch; every one still fails the lint
• 2026-10-03: the Go tests run with the race detector and coverage (issue #88): backend/script/test runs go test -timeout 30s -race -cover ./... and, if that fails, runs it again with -v and fails. Go's -timeout bounds the tests, not their compile; the root script/test no longer puts one 30-second timeout around both halves, which a cold Go build cache could use up on compiling alone. The race detector needs a C compiler: the builder stage of Dockerfile has gcc and musl-dev, and script/bootstrap installs gcc, with the C library headers on apt and apk, when gcc is missing; the binary is still built with CGO_ENABLED=0. New tests cover the health check's answer, a valid report's answer, a report file's exact contents, and the flush when the buffer reaches 10 MiB; the handlers' TestImport stub is gone
• 2026-10-03: each target check times out after 80% of the refresh interval (issue #78), 24 seconds at 30 seconds, where it was capped at 3 seconds. A round started early, after an interval change or when the recovery probe finds a target answering, gives up the last round's checks if they are still waiting, so rounds never overlap; the recovery probe gives up its own checks after half a second. The frontend has its first unit tests, run by script/frontend-test with Node's built-in test runner; for them, index.html now links src/styles.css, which src/main.js used to import
• 2026-09-29: the container sets up its own data directory (issue #75): bin/entrypoint.sh, still as root, creates DATA_DIR if missing and gives it and /data to the netwatch user with mode 750 before starting the backend as that user, so an empty host directory owned by root, or one holding files from another uid, works with no step on the host. It stops the start instead when a symbolic link is on the path to DATA_DIR, since root would change whatever the link points to. The README.md first-run step that created and chowned the host directory is gone, and the image no longer sets that ownership at build time
• 2026-09-29: CI can no longer pass on checks that did not run (issue #37): script/cibuild is now the org model, byte for byte. It runs script/bootstrap and script/check, then builds the image with --no-cache and the version from git describe as the VERSION build argument, where it used to be a plain docker build . whose check steps could come from the build cache. The workflow puts ~/.local/bin, where bootstrap links what it installs, on the step's PATH, and bootstrap now installs its pinned node when the installed one is older than the frontend's dependencies need
• 2026-09-29: backend/.golangci.yml re-vendored from sneak/prompts (issue #41): gomodguard, deprecated in golangci-lint v2.12.0, is disabled and its successor gomodguard_v2 enabled with the org block list, so lint runs print no deprecation warning. The new file also turns depguard on with its test-support rule, which keeps net/http/httptest out of non-test code; netwatch adds no entries of its own to that rule. backend/script/lint checks the new sha256
• 2026-09-29: nginx sends the security headers REPO_POLICIES.md requires on every response (issue #18), including errors, /assets/ and what it passes on from the backend, whose own copies it drops so each header goes out once. They live in security-headers.conf, which nginx.conf includes. The content security policy allows no inline script or style, so the status dot's grey in src/main.js is now a class; connect-src is * because probed hosts redirect to others, and the browser checks each redirect against it
• 2026-09-29: the request log is bounded (issue #60): the method, URL, protocol, User-Agent, Referer, request ID (which chi takes from the client's X-Request-Id header) and client address it writes are each cut to 128 bytes, the bound the report handler already used, so one request can no longer put about 1 MiB per field into a log line. That bound and its helper now live in the logger package, shared by both
• 2026-09-29: nginx takes the client address from X-Forwarded-For only on requests from the reverse proxies named in the container's TRUSTED_PROXIES (issue #64), and by default from none, where it trusted every RFC1918 address before, so a client could write a new address on each request and escape the rate limit. bin/entrypoint.sh writes one set_real_ip_from line per entry into /etc/nginx/trusted-proxies.conf, which nginx.conf includes, refusing an entry that is not an IP address or CIDR, as netwatch-server check-cidr finds; it starts the backend with TRUSTED_PROXIES=127.0.0.1/32, since nginx is its only client
• 2026-09-29: report file names can no longer collide (issue #61): each is reports-<timestamp>-<number>.jsonl.zst, where the number goes up by one for each file the server starts to write, so two flushes in the same millisecond, such as a flush for size and the final flush at shutdown, each get a file of their own instead of the second one failing. A failed write uses up its number, leaving a gap if the file could not be created and otherwise a file under that number that may be incomplete.
• 2026-09-29: ready to run under upaas (issue #59): the image has a HEALTHCHECK that requests /.well-known/healthcheck through nginx on the port from PORT. The backend no longer reads a bad PORT as 0 or a bad DEBUG as false: those, and a BIND_ADDRESS that is not an IP address, stop it from starting with an error naming the variable, as the limits, CORS_ALLOWED_ORIGINS and, now by name, TRUSTED_PROXIES already did. bin/entrypoint.sh also refuses a PORT outside 1 to 65535, and 8081, where the backend listens inside the container, naming PORT. README.md has a "Running under upaas" section, whose first-run steps create the host directory for /data owned by uid 1000; the image does not change its owner
• 2026-09-29: nginx listens on PORT (issue #26), 8080 when unset or empty: the nginx image renders nginx.conf as a template at container start, filling in PORT and no other variable. bin/entrypoint.sh refuses to start when PORT is not digits only. server_tokens off keeps the nginx version out of responses. script/frontend-viewport-test renders the template the same way. Gzip and a 50x.html error page are not added
• 2026-09-29: bounded the report endpoint (issue #20): POST /api/v1/reports still needs no credentials, but each client address, as resolved through TRUSTED_PROXIES, may send REPORTS_PER_MINUTE (default 60) reports a minute, counted by go-chi/httprate, and past that gets 429 with Retry-After; the report files in DATA_DIR, counted from start with those already there, may total at most DATA_DIR_MAX_BYTES (default 1 GiB), past which reports get 507; and the wildcard CORS is gone: no CORS headers unless CORS_ALLOWED_ORIGINS lists origins, and an entry that is not a plain scheme://host[:port] origin, * included, stops the server from starting. Deleting report files frees room only at the next start; pruning is issue #54
• 2026-09-28: one container image (issue #52): the root Dockerfile builds the only image, and Dockerfile.backend is gone. nginx serves the frontend on port 8080 and proxies /api/ and /.well-known/healthcheck to the backend, which listens on 127.0.0.1:8081 in the same container; the new BIND_ADDRESS setting sets its listen address. bin/entrypoint.sh starts both, passes TERM and INT on to both, and exits non-zero when either exits on its own. The backend runs as user netwatch and stores reports on the /data volume. script/docker is the org model again
• 2026-09-28: unified the gate (issue #16): the root make check covers the Go backend as well as the frontend, and the pre-commit hook with it; the backend moved onto scripts-to-rule-them-all (backend/script/*, backend/Makefile as shims, its duplicate hook installer removed); script/cibuild builds both images and is the workflow's only build step. The root make lint runs golangci-lint only in Docker, by building the lint stage of Dockerfile.backend without the cache. script/bootstrap installs the pinned Go unless the installed one is at least what backend/go.mod asks for, links what it installs into ~/.local/bin without replacing anything it did not create, and installs no linter. Root make test runs both halves within one 30-second timeout. When VERSION is unset or empty, the backend binary's version falls back to git describe inside a git checkout, then to dev
• 2026-09-28: frontend reporting client (issue #53): a Reporter class posts collected samples to /api/v1/reports every reportInterval (default 60s) as a per-host delta, with the report-building step a pure exported function of host state; a per-host mark advances only on a delivered POST; at most one report POST is pending at a time and it is abandoned after half the interval, so a slow POST never overlaps the next report and a mark never moves backwards; the samples of an abandoned POST are sent again at the next interval, so a backend that stored them but answered late receives them twice; the per-browser client id works in insecure (plain-HTTP) contexts; vite.config.js proxies /api to the local backend for yarn dev
• 2026-09-28: report ingest correctness (issue #23): a storage failure now returns 500 instead of a false ok; oversize bodies return 413 (distinguished from malformed JSON, which stays 400); a MaxBodyBytes middleware caps every route, not just the report route; the raw attacker-controlled geo blob is no longer logged (only its length), and client_id, timestamp and decode error text are length-bounded before logging; a decodeJSON handler helper was added; panic recovery now routes the stack through slog instead of chi's plain-text stderr; and writing a report file now returns its error, so a failed final flush on shutdown makes the process exit non-zero instead of losing the buffered reports silently
• 2026-09-21: shutdown lifecycle correctness. The process now shuts down through fx instead of os.Exit, so every component's OnStop runs and buffered reports are flushed to disk on SIGTERM — previously a full flush window of telemetry was silently lost on every restart. The http.Server is now built before its serving goroutine starts, so shutdown can no longer race or nil-deref it; a listen failure exits non-zero via fx.Shutdowner; reportbuf OnStop is idempotent; and writeTimeout now exceeds the chi per-request budget so that budget is actually reachable. Dead startupTime, exitCode, and cancelFunc fields were removed
• 2026-09-21: backend HTTP hardening (issue #19): added ReadHeaderTimeout and IdleTimeout to the server, a SecurityHeaders middleware (HSTS, tight CSP, frame/sniff/referrer/permissions headers) registered before CORS, and trusted-proxy client IP resolution honouring X-Forwarded-For / X-Real-IP only from a TRUSTED_PROXIES allowlist (loopback plus RFC1918 by default)
• 2026-08-10: adopted the org-standard backend/.golangci.yml verbatim and moved the pinned golangci-lint from v2.7.2 to v2.12.2 (the lint stage of Dockerfile.backend now pins the golangci/golangci-lint:v2.12.2 image by digest); the previous config declared version: "2" but used v1 schema keys, so every threshold in it was inert and its green result was meaningless. backend/Makefile's lint target now asserts the config's sha256 against the canonical file first, so drift from the org standard fails the build instead of silently degrading to defaults
• 2026-08-10: every interactive control now meets the 44x44 CSS px minimum tap target (.pin-btn, #interval-select, the debug-log label and, on narrow viewports, #pause-btn). The pin button's hit area grows via matching negative margins, so its layout footprint and row density are unchanged
• 2026-08-10: per-host status line wraps below the 768px breakpoint instead of forcing horizontal page scroll at 320px
• 2026-08-09: Dockerfile.backend reworked to the mandated Go multistage lint-stage pattern: separate lint stage on the hash-pinned golangci/golangci-lint image, COPY --from=lint stage dependency, CGO_ENABLED=0 static build driven by ARG VERSION, and no more COPY .git
• 2026-08-09: dotfile compliance — lifted backend/.editorconfig to the repo root so root = true covers the frontend too, and replaced .gitignore with the org model (OS, editor, node, and environment/secrets sections) plus this repo's dist/ and *.log. .env, .env.*, *.pem, and *.key are now ignored repo-wide, not just under backend/. Excluding .git from .dockerignore stays deferred: both images read git metadata at build time (COPY .git in Dockerfile.backend, git rev-parse in vite.config.js)
• 2026-08-09: automated responsive-layout harness (make frontend-viewport-test): digest-pinned headless Chrome driven over CDP against the built dist/, viewport widths derived from the breakpoints in src/styles.css (#13). The tap-target and host-row checks each fail when they measured nothing; the overflow, viewport-edge and clipped-text checks have no such guard of their own and rely on the app-rendered check, which fails the run when the app did not render. Found two real layout defects, filed as #42 and #43
• 2026-07-07 Adopted scripts-to-rule-them-all: script/ entrypoints, Makefile shims, README Entrypoints section
• 2026-02-27: backend with buffered zstd-compressed report storage; CI workflow and backend repo standard files; backend Dockerfile fixed (Go 1.25, golangci-lint) and moved to repo root (feat/reportbuf-storage)
• 2026-02-26: host row layout redesigned with CSS grid; overflow and spacing fixes; nginx config extracted; port hardcoded to 8080
• 2026-02-26: debug log panel, median stats, recovery probe, Docker build fix, S3 Singapore endpoint added
• 2026-02-23: summary box redesign, host pinning, local and UTC clocks, checks counter
• 2026-02-23: hosts sorted by latency; GET instead of HEAD for latency; timeout derived from interval; Hetzner regional endpoints; 3s interval
• 2026-01-29: initial NetWatch network latency monitor

Future Steps

• Wire script/frontend-viewport-test into CI as its own step (deliberately not part of make check today; the decision has real CI-runtime cost and is tracked separately)
• Compliance top-up as one small commit: add .editorconfig and add the hooks target to the Makefile
• After merge, confirm .gitea/workflows/check.yml is on main and CI is green (main always green policy)
• Decide what to do with untracked resume.sh: commit it, gitignore it, or delete it