netwatch/TODO.md

Workflow

• branch (from main)
• do the work in Next Step
• move Next Step to the top of Completed Steps
• move the top item of Future Steps into Next Step
• commit (TODO.md changes in the same commit as the work)
• merge to main if the branch is not protected, otherwise open a PR
• push

Status

pre-1.0. No git tags. feat/reportbuf-storage is merged; the backend, the CI workflow, and the backend repo standard files are all on main. Frontend and backend are both functional. Working toward the 1.0.0 milestone by closing the remaining repo-compliance issues on the tracker.

Next Step

Confirm the .gitea/workflows/check.yml run is green (main always green policy). The workflow file is already on main; what is unverified is that its latest run passes.

Completed Steps

• 2026-09-29: nginx sends the security headers REPO_POLICIES.md requires on every response (issue #18), including errors, /assets/ and what it passes on from the backend, whose own copies it drops so each header goes out once. They live in security-headers.conf, which nginx.conf includes. The content security policy allows no inline script or style, so the status dot's grey in src/main.js is now a class; connect-src is * because probed hosts redirect to others, and the browser checks each redirect against it
• 2026-09-29: nginx takes the client address from X-Forwarded-For only on requests from the reverse proxies named in the container's TRUSTED_PROXIES (issue #64), and by default from none, where it trusted every RFC1918 address before, so a client could write a new address on each request and escape the rate limit. bin/entrypoint.sh writes one set_real_ip_from line per entry into /etc/nginx/trusted-proxies.conf, which nginx.conf includes, refusing an entry that is not an IP address or CIDR, as netwatch-server check-cidr finds; it starts the backend with TRUSTED_PROXIES=127.0.0.1/32, since nginx is its only client
• 2026-09-29: report file names can no longer collide (issue #61): each is reports-<timestamp>-<number>.jsonl.zst, where the number goes up by one for each file the server starts to write, so two flushes in the same millisecond, such as a flush for size and the final flush at shutdown, each get a file of their own instead of the second one failing. A failed write uses up its number, leaving a gap if the file could not be created and otherwise a file under that number that may be incomplete.
• 2026-09-29: ready to run under upaas (issue #59): the image has a HEALTHCHECK that requests /.well-known/healthcheck through nginx on the port from PORT. The backend no longer reads a bad PORT as 0 or a bad DEBUG as false: those, and a BIND_ADDRESS that is not an IP address, stop it from starting with an error naming the variable, as the limits, CORS_ALLOWED_ORIGINS and, now by name, TRUSTED_PROXIES already did. bin/entrypoint.sh also refuses a PORT outside 1 to 65535, and 8081, where the backend listens inside the container, naming PORT. README.md has a "Running under upaas" section, whose first-run steps create the host directory for /data owned by uid 1000; the image does not change its owner
• 2026-09-29: nginx listens on PORT (issue #26), 8080 when unset or empty: the nginx image renders nginx.conf as a template at container start, filling in PORT and no other variable. bin/entrypoint.sh refuses to start when PORT is not digits only. server_tokens off keeps the nginx version out of responses. script/frontend-viewport-test renders the template the same way. Gzip and a 50x.html error page are not added
• 2026-09-29: bounded the report endpoint (issue #20): POST /api/v1/reports still needs no credentials, but each client address, as resolved through TRUSTED_PROXIES, may send REPORTS_PER_MINUTE (default 60) reports a minute, counted by go-chi/httprate, and past that gets 429 with Retry-After; the report files in DATA_DIR, counted from start with those already there, may total at most DATA_DIR_MAX_BYTES (default 1 GiB), past which reports get 507; and the wildcard CORS is gone: no CORS headers unless CORS_ALLOWED_ORIGINS lists origins, and an entry that is not a plain scheme://host[:port] origin, * included, stops the server from starting. Deleting report files frees room only at the next start; pruning is issue #54
• 2026-09-28: one container image (issue #52): the root Dockerfile builds the only image, and Dockerfile.backend is gone. nginx serves the frontend on port 8080 and proxies /api/ and /.well-known/healthcheck to the backend, which listens on 127.0.0.1:8081 in the same container; the new BIND_ADDRESS setting sets its listen address. bin/entrypoint.sh starts both, passes TERM and INT on to both, and exits non-zero when either exits on its own. The backend runs as user netwatch and stores reports on the /data volume. script/docker is the org model again
• 2026-09-28: unified the gate (issue #16): the root make check covers the Go backend as well as the frontend, and the pre-commit hook with it; the backend moved onto scripts-to-rule-them-all (backend/script/*, backend/Makefile as shims, its duplicate hook installer removed); script/cibuild builds both images and is the workflow's only build step. The root make lint runs golangci-lint only in Docker, by building the lint stage of Dockerfile.backend without the cache. script/bootstrap installs the pinned Go unless the installed one is at least what backend/go.mod asks for, links what it installs into ~/.local/bin without replacing anything it did not create, and installs no linter. Root make test runs both halves within one 30-second timeout. When VERSION is unset or empty, the backend binary's version falls back to git describe inside a git checkout, then to dev
• 2026-09-28: frontend reporting client (issue #53): a Reporter class posts collected samples to /api/v1/reports every reportInterval (default 60s) as a per-host delta, with the report-building step a pure exported function of host state; a per-host mark advances only on a delivered POST; at most one report POST is pending at a time and it is abandoned after half the interval, so a slow POST never overlaps the next report and a mark never moves backwards; the samples of an abandoned POST are sent again at the next interval, so a backend that stored them but answered late receives them twice; the per-browser client id works in insecure (plain-HTTP) contexts; vite.config.js proxies /api to the local backend for yarn dev
• 2026-09-28: report ingest correctness (issue #23): a storage failure now returns 500 instead of a false ok; oversize bodies return 413 (distinguished from malformed JSON, which stays 400); a MaxBodyBytes middleware caps every route, not just the report route; the raw attacker-controlled geo blob is no longer logged (only its length), and client_id, timestamp and decode error text are length-bounded before logging; a decodeJSON handler helper was added; panic recovery now routes the stack through slog instead of chi's plain-text stderr; and writing a report file now returns its error, so a failed final flush on shutdown makes the process exit non-zero instead of losing the buffered reports silently
• 2026-09-21: shutdown lifecycle correctness. The process now shuts down through fx instead of os.Exit, so every component's OnStop runs and buffered reports are flushed to disk on SIGTERM — previously a full flush window of telemetry was silently lost on every restart. The http.Server is now built before its serving goroutine starts, so shutdown can no longer race or nil-deref it; a listen failure exits non-zero via fx.Shutdowner; reportbuf OnStop is idempotent; and writeTimeout now exceeds the chi per-request budget so that budget is actually reachable. Dead startupTime, exitCode, and cancelFunc fields were removed
• 2026-09-21: backend HTTP hardening (issue #19): added ReadHeaderTimeout and IdleTimeout to the server, a SecurityHeaders middleware (HSTS, tight CSP, frame/sniff/referrer/permissions headers) registered before CORS, and trusted-proxy client IP resolution honouring X-Forwarded-For / X-Real-IP only from a TRUSTED_PROXIES allowlist (loopback plus RFC1918 by default)
• 2026-08-10: adopted the org-standard backend/.golangci.yml verbatim and moved the pinned golangci-lint from v2.7.2 to v2.12.2 (the lint stage of Dockerfile.backend now pins the golangci/golangci-lint:v2.12.2 image by digest); the previous config declared version: "2" but used v1 schema keys, so every threshold in it was inert and its green result was meaningless. backend/Makefile's lint target now asserts the config's sha256 against the canonical file first, so drift from the org standard fails the build instead of silently degrading to defaults
• 2026-08-10: every interactive control now meets the 44x44 CSS px minimum tap target (.pin-btn, #interval-select, the debug-log label and, on narrow viewports, #pause-btn). The pin button's hit area grows via matching negative margins, so its layout footprint and row density are unchanged
• 2026-08-10: per-host status line wraps below the 768px breakpoint instead of forcing horizontal page scroll at 320px
• 2026-08-09: Dockerfile.backend reworked to the mandated Go multistage lint-stage pattern: separate lint stage on the hash-pinned golangci/golangci-lint image, COPY --from=lint stage dependency, CGO_ENABLED=0 static build driven by ARG VERSION, and no more COPY .git
• 2026-08-09: dotfile compliance — lifted backend/.editorconfig to the repo root so root = true covers the frontend too, and replaced .gitignore with the org model (OS, editor, node, and environment/secrets sections) plus this repo's dist/ and *.log. .env, .env.*, *.pem, and *.key are now ignored repo-wide, not just under backend/. Excluding .git from .dockerignore stays deferred: both images read git metadata at build time (COPY .git in Dockerfile.backend, git rev-parse in vite.config.js)
• 2026-08-09: automated responsive-layout harness (make frontend-viewport-test): digest-pinned headless Chrome driven over CDP against the built dist/, viewport widths derived from the breakpoints in src/styles.css (#13). Every check carries a presence guard so none of them can pass against a page it is not actually measuring. Found two real layout defects, filed as #42 and #43
• 2026-07-07 Adopted scripts-to-rule-them-all: script/ entrypoints, Makefile shims, README Entrypoints section
• 2026-02-27: backend with buffered zstd-compressed report storage; CI workflow and backend repo standard files; backend Dockerfile fixed (Go 1.25, golangci-lint) and moved to repo root (feat/reportbuf-storage)
• 2026-02-26: host row layout redesigned with CSS grid; overflow and spacing fixes; nginx config extracted; port hardcoded to 8080
• 2026-02-26: debug log panel, median stats, recovery probe, Docker build fix, S3 Singapore endpoint added
• 2026-02-23: summary box redesign, host pinning, local and UTC clocks, checks counter
• 2026-02-23: hosts sorted by latency; GET instead of HEAD for latency; timeout derived from interval; Hetzner regional endpoints; 3s interval
• 2026-01-29: initial NetWatch network latency monitor

Future Steps

• Wire script/frontend-viewport-test into CI as its own step (deliberately not part of make check today; the decision has real CI-runtime cost and is tracked separately)
• Compliance top-up as one small commit: add .editorconfig and add the hooks target to the Makefile
• After merge, confirm .gitea/workflows/check.yml is on main and CI is green (main always green policy)
• Decide what to do with untracked resume.sh: commit it, gitignore it, or delete it
• Upstream fix needed in sneak/prompts: the org-standard .golangci.yml enables gomodguard, which golangci-lint v2.12.2 reports as deprecated since v2.12.0 and replaced by gomodguard_v2, so every backend lint run prints a deprecation warning. The file is standardized and must never be edited in this repo, so nothing can be done here beyond tracking it — tracked at https://git.eeqj.de/sneak/netwatch/issues/41