diff --git a/README.md b/README.md index 3f51cc2..04ba350 100644 --- a/README.md +++ b/README.md @@ -445,8 +445,8 @@ says what it checks and what it cannot. ## TODO -The to-do list is [TODO.md](TODO.md): where the work stands, the next step, the -open work, and what has been done. +The to-do list is [TODO.md](TODO.md): where the work stands, the next step and +the open work. ## License diff --git a/TODO.md b/TODO.md index d9f3255..7bd574e 100644 --- a/TODO.md +++ b/TODO.md @@ -2,704 +2,24 @@ - branch from `next` - do the work in Next Step -- move Next Step to the top of Completed Steps +- delete Next Step - move the top item of Future Steps into Next Step - commit (`TODO.md` changes in the same commit as the work) - push the branch and open a PR against `next` # Status -pre-1.0. Release candidates are tagged on `next`, the first being `v1.0.0-rc1`; -`git tag --merged origin/next` lists them. `main` is the stable branch and -`next` the development branch, which every PR targets. The frontend and the Go -backend ship as one Docker image. The Gitea workflow -`.gitea/workflows/check.yml` runs `script/cibuild` on every push, and cancels -the older run on the same branch. Working toward 1.0.0. +pre-1.0. Release candidates are tagged on `next`; `git tag --merged origin/next` +lists them. `main` is the stable branch and `next` the development branch, which +every PR targets. The frontend and the Go backend ship as one Docker image. The +Gitea workflow `.gitea/workflows/check.yml` runs `script/cibuild` on every push, +and cancels the older run on the same branch. Working toward 1.0.0. # Next Step Run `make frontend-viewport-test` in CI as its own step; it is not part of `make check`, as it needs Docker and takes minutes. -# Completed Steps - -- 2026-11-10: the CI time limit of 20 minutes is on the step that runs - `script/cibuild` ([#199](https://git.eeqj.de/sneak/netwatch/issues/199)). The - runner ignored it on the job. The `Dockerfile`, `script/frontend-test` and - `script/frontend-viewport-test` no longer wrap commands in `timeout`. -- 2026-10-26: `README.md`, `TODO.md` and `test/viewport/README.md` are rewritten - sentence by sentence ([#189](https://git.eeqj.de/sneak/netwatch/issues/189)). - Each sentence reads on the first pass: one idea, under about 30 words, the - verb early. Every statement matches the code on `next`. -- 2026-10-26: a target whose home page answers with a redirect is checked at - `/robots.txt` on the same host, which answers without one - ([#194](https://git.eeqj.de/sneak/netwatch/issues/194)). These are the AWS - Console, Akamai, B2 and the 8 S3 regions. The browser followed each redirect, - so their figures also timed the page it led to. The Google Cloud Console is - checked at `/robots.txt` too, as its home page redirects `curl`'s requests. - The S3 regions are checked with a HEAD request, as Chrome closes their - connection after most GET requests. `README.md` says what a figure includes. -- 2026-10-26: the six Hetzner targets answer wherever the page is served from - ([#191](https://git.eeqj.de/sneak/netwatch/issues/191)). They close the - connection without an answer when the request has a `Referer` header. The - browser sent one unless the page came with the image's - `Referrer-Policy: no-referrer` header, which a static host and `make dev` do - not send. Each check now sets `referrerPolicy: 'no-referrer'`. -- 2026-10-19: the server's `OnStart` hook in `internal/server/server.go` starts - `listenAndServe` in an anonymous function, as the Go style guide asks - ([#162](https://git.eeqj.de/sneak/netwatch/issues/162)). It was - `go s.listenAndServe()`. Nothing the server does changed. -- 2026-10-19: `.gitignore`, `.editorconfig` and `.golangci.yml` are the - sneak/prompts copies at `0653579` - ([#174](https://git.eeqj.de/sneak/netwatch/issues/174)). `.gitignore` is the - standard file followed by this repository's own entries. `.editorconfig` is - the standard file alone, which carries the `[*.go]` section. `.golangci.yml` - disables `canonicalheader`, as the canonical copy does: golangci-lint v2.14.0 - misses its findings at random. -- 2026-10-19: `package.json` has no `version` field and is marked - `"private": true` ([#158](https://git.eeqj.de/sneak/netwatch/issues/158)). It - said `1.0.0`, a release that has not been made, and nothing read it. The git - tag is the one version; the image and `make build` take it from - `git describe --tags`. The package is never published. -- 2026-10-19: Status names `v1.0.0-rc1`, the first release candidate, and says - release candidates are tagged on `next` - ([#157](https://git.eeqj.de/sneak/netwatch/issues/157)). It said there were no - git tags. -- 2026-10-19: `netwatch-server` starts the server only when given no arguments - ([#179](https://git.eeqj.de/sneak/netwatch/issues/179)). `check-cidr CIDR` and - `prepare-data-dir DATA_DIR` behave as before. Any other arguments print a - one-line usage message naming both forms to stderr and exit 2; they used to - start the server. Unit tests cover each case. -- 2026-10-19: `main()` in `cmd/netwatch-server/main.go` is one call into the new - package `internal/cli`, as the Go style guide asks - ([#160](https://git.eeqj.de/sneak/netwatch/issues/160)). `internal/cli` runs - the `check-cidr` and `prepare-data-dir` subcommands, with the same arguments, - messages and exit codes, and otherwise the server with the `fx` wiring. - `Appname` and `Version` stay in `main`, where the build sets `Version`. `main` - gives `cli.Main` stderr and the function that runs the server; the tests - replace both. Unit tests give each subcommand input it takes and input it - refuses. -- 2026-10-19: when `VERSION` is unset or empty, `script/build` stamps the - version from `git describe --tags --always --dirty`, as `script/docker` does - ([#156](https://git.eeqj.de/sneak/netwatch/issues/156)). Without `--tags` it - skipped lightweight tags, so `make run` could report a commit where the image - built from the same tree reported the tag. -- 2026-10-19: `.dockerignore` keeps Go test binaries and coverage output at the - root out of the build context, with `/*.test` and `/*.out` in the repository's - own section ([#155](https://git.eeqj.de/sneak/netwatch/issues/155)). Such - files were copied into the lint, test and build stages. The entries match only - at the root, so the same files in a subdirectory still reach the image. -- 2026-10-19: `.gitea/workflows/check.yml` is the sneak/prompts copy at - `0653579` ([#153](https://git.eeqj.de/sneak/netwatch/issues/153)). Its - checkout step sets `fetch-depth: 0`, so CI clones all history and tags. The - image built from a tagged commit then gets the tag as its version; it got the - short commit. The step also sets `persist-credentials: false`, so the job's - token is not left in `.git/config`. A new push cancels the older run on the - same branch, and a run stops after 20 minutes. -- 2026-10-19: `script/bootstrap` keeps the `yarn` on `PATH` only when - `yarn --version` is exactly `YARN_VERSION`; otherwise it installs that version - ([#154](https://git.eeqj.de/sneak/netwatch/issues/154)). Any yarn used to be - kept. It uses the `node` on `PATH` only when that node has the major version - of `NODE_VERSION` and is at least `NODE_MIN_VERSION`; a node 24 was kept - before. After installing node or yarn it looks the tool up through `PATH` - again and stops if that one fails the same check, as it does for Go. It prints - the node and yarn versions it uses. Every script that runs yarn on the host, - except `script/bootstrap`, runs it through the new `script/yarn`. That script - puts `~/.local/bin` first on `PATH`. It uses the first `node` there when it - passes the same test, and otherwise runs yarn under the pinned node in nvm. - `make build` ran yarn under whatever node was first on `PATH`. -- 2026-10-19: `bin/entrypoint.sh` has each step in a function, and a `main` at - the end calls them in order, as the code style guide's shell section asks - ([#163](https://git.eeqj.de/sneak/netwatch/issues/163)). It is still POSIX sh - with no `set -e`, and does what it did, with the same messages and exit codes. -- 2026-10-19: while report files cannot be written, the health check answers 503 - with `"status":"error"` and the same fields, so the image's `HEALTHCHECK` - reports the container `unhealthy` - ([#142](https://git.eeqj.de/sneak/netwatch/issues/142)). It answered 200 - whatever happened, so the platform saw a healthy container while every report - was refused with 500. It answers 200 at start, before any write has failed, - and again from the first write that succeeds. -- 2026-10-19: report file writes no longer overlap - ([#150](https://git.eeqj.de/sneak/netwatch/issues/150)), so the reports of - failed writes are written in the order they arrived. Two writes that failed at - the same time put their reports back in the wrong order when the older failed - first. One write runs at a time. Each takes the waiting reports only when it - starts, after a failed write has put its own back. The write that `Append` - starts at 10 MiB waits for any write under way, and `Append` starts no other - while it waits. Files written since the start are deleted to make room in the - order they were written, after those found at start. -- 2026-10-19: `DEBUG` unset or empty leaves debug logging off, and any other - value, `false` and `0` included, turns it on, as the code style guide says - ([#159](https://git.eeqj.de/sneak/netwatch/issues/159)). No value of it stops - the start any more. Before, only `1`, `t`, `T`, `TRUE`, `true`, `True`, `0`, - `f`, `F`, `FALSE`, `false` and `False` were accepted. `README.md` no longer - gives `false` as its default. -- 2026-10-19: check builds keep Go's modules and compiled packages between runs - ([#118](https://git.eeqj.de/sneak/netwatch/issues/118)), on sneak's rulings in - [prompts#113](https://git.eeqj.de/sneak/prompts/issues/113) and - [prompts#130](https://git.eeqj.de/sneak/prompts/issues/130). - `REPO_POLICIES.md`, `script/test`, `script/lint`, `script/docker` and - `script/cibuild` are the sneak/prompts copies at `0653579`. The scripts pass a - new `CACHEBUST` value where they passed `--no-cache`, and the lint and test - builds write no image. In each stage of `Dockerfile`, `ARG CACHEBUST` comes - after the system packages and the Go module download. Every step that lints, - tests or compiles comes after it, as does the node stages' `yarn install`. - Go's build cache and golangci-lint's are cache mounts, and both `go test` runs - carry `-count=1`. The test phase's Go caches used to be in memory, empty on - every build. -- 2026-10-19: the three unit tests of a discarded check also check that the - host's history is still empty, its status still `pending`, and the console - untouched ([#151](https://git.eeqj.de/sneak/netwatch/issues/151)). The - discarded cases are a round given up, a pause, and the first round. The tests - checked only the row. So a change that recorded the result and skipped only - the row would have kept them passing. -- 2026-10-18: run on its own, the backend stops at start when `DATA_DIR` cannot - be created or read, or when no file can be created in it - ([#148](https://git.eeqj.de/sneak/netwatch/issues/148)). The error names - `DATA_DIR` and its value. The first two cases named only the path, and the - third was found only at the first write of a report file. At start the server - creates a file in `DATA_DIR` and removes it. The report file number is an - `atomic.Int64`, as the Go style guide asks for int types - ([#161](https://git.eeqj.de/sneak/netwatch/issues/161)); file names are - unchanged. -- 2026-10-18: the image's nginx sends the page with `Cache-Control: no-cache` - ([#141](https://git.eeqj.de/sneak/netwatch/issues/141)), so a browser that - keeps a copy checks with the server that it is current before using it again. - The page went out with no `Cache-Control`. So a browser could keep it after a - deploy, ask for files under `/assets/` that the new build no longer has, and - show an empty page. A path that `location /` answers with the page through its - `try_files` fallback gets the same header. `location /` includes the security - headers again, as `/assets/` does, since its own `add_header` drops the - server's. `/assets/` keeps its immutable caching. - `make frontend-viewport-test` fails unless `/` and a path that falls back to - the page carry `Cache-Control: no-cache` and `Content-Security-Policy`. -- 2026-10-18: `README.md` says what the page loses while reports are refused or - the page is paused ([#145](https://git.eeqj.de/sneak/netwatch/issues/145)). It - says how fast one address can replace the stored reports - ([#146](https://git.eeqj.de/sneak/netwatch/issues/146)). It says that in the - image nginx answers a body over 1 MiB itself - ([#147](https://git.eeqj.de/sneak/netwatch/issues/147)). nginx's - `client_max_body_size` is 1 MiB by default, so in the image it answers a - larger body with 413 and its own HTML page, and the backend never sees it. The - reporter, the body limit and nginx are unchanged. -- 2026-10-18: a browser that cannot format the header's clocks still probes - ([#149](https://git.eeqj.de/sneak/netwatch/issues/149)). The clocks name the - local time zone through `Intl.DateTimeFormat`'s `shortGeneric` option, which - Chrome before 95, Firefox before 91 and Safari before 15.4 reject with an - error. It came before the first round started, so such a browser never checked - a target. The clocks now start last, after the rounds, and an error formatting - them leaves both blank. The footer's color legend reads `>=500ms` for red, - where it read `>500ms` - ([#152](https://git.eeqj.de/sneak/netwatch/issues/152)). -- 2026-10-18: `netwatch-server prepare-data-dir` no longer changes the owner of - a file outside `/data` through a hard link in the volume - ([#143](https://git.eeqj.de/sneak/netwatch/issues/143)). A file with a second - name (hard link) keeps its owner, and the start logs its path. Owners are set - on the opened directory, file or link, never by name, so a hard link swapped - in after the check is caught too. A symbolic link is opened without following - it, and the link itself is given to the netwatch user. So the server can still - follow one in a sticky directory that anyone can write to. Devices, named - pipes and sockets keep their owner. Setting the owner on an open file that is - not a directory is built only for Linux. `prepare-data-dir` runs only in the - Linux image, and macOS has neither `O_PATH` nor `AT_EMPTY_PATH`. On any other - system the package still builds, and `prepare-data-dir` fails at the first - file that is not a directory. -- 2026-10-18: from a trusted proxy, the backend takes the client address from - the right-most `X-Forwarded-For` entry that is not in `TRUSTED_PROXIES` - ([#144](https://git.eeqj.de/sneak/netwatch/issues/144)). It took the left-most - entry, which the client writes itself behind a proxy that appends to the - header. So a client could choose the address the rate limit counts and the - request log records. An entry written as an address with a port is read as - that address. The direct peer is used when the header holds only trusted - addresses, or when the search meets an entry that is not an address at all, - such as `unknown`. `X-Real-IP` is read only without `X-Forwarded-For`, and - every `X-Forwarded-For` header line is read, not only the first. The image's - nginx, and the `TRUSTED_PROXIES=127.0.0.1/32` it starts the backend with, are - unchanged. -- 2026-10-18: after an interval change, the summary's Reachable slot reads - `--/--` in white, as at page load, until the second round since the change - ends ([#140](https://git.eeqj.de/sneak/netwatch/issues/140)). It read 0/26 in - red from the change on, as no WAN host had a check recorded, for up to two - intervals with the network fine. -- 2026-10-14: stopping the server waits for a report file write already under - way, then runs the final flush - ([#122](https://git.eeqj.de/sneak/netwatch/issues/122)). Such a write is the - periodic flush's, or the one `Append` starts when 10 MiB are waiting. It used - to return at once, and the process exiting cut the write off, losing its - reports. A write under way that fails puts its reports back in memory, so the - final flush writes them. The stop fails, and the process exits non-zero, only - if that flush fails too. -- 2026-10-14: `make test` no longer runs the frontend's production build, which - the tests do not need ([#118](https://git.eeqj.de/sneak/netwatch/issues/118)). - `yarn build` moved out of the stage that runs the frontend's unit tests, now - named `frontend-test`, into a new `frontend-build` stage. That stage starts - from `frontend-test` and, like the `builder` stage, waits on the lint and test - phases, so the image still cannot be built unless they pass. The runtime stage - copies `dist/` from it, and the `git` that `vite.config.js` reads the commit - with is installed there. No build cache was added. -- 2026-10-14: opening the debug log no longer raises the measured latencies - ([#131](https://git.eeqj.de/sneak/netwatch/issues/131)). With the panel open, - each line written drew the panel again with every line before it, up to 1000. - A check that ends while the page draws is timed only when the drawing is done. - Each line is now added at the end of the open panel, and its first line is - dropped once it holds more than 1000. It scrolls to its newest line once a - frame rather than once a line. The whole panel is drawn only when it is - opened. -- 2026-10-14: unset, the backend's own `TRUSTED_PROXIES` trusts `10.0.0.0/8`, - `172.16.0.0/12` and `192.168.0.0/16`, and no longer loopback - ([#129](https://git.eeqj.de/sneak/netwatch/issues/129)). A list replaces that. - Set and empty, it trusts no proxy; it used to get the default, as viper reads - an empty environment variable as unset. A value it cannot parse still stops - the start, and the image still starts the backend with - `TRUSTED_PROXIES=127.0.0.1/32`. -- 2026-10-14: while report files cannot be written, `POST /api/v1/reports` - answers 500, and storing recovers once writing works again, with no restart - ([#123](https://git.eeqj.de/sneak/netwatch/issues/123)). A failed write puts - its reports back in the report buffer, still counted against - `DATA_DIR_MAX_BYTES`, where they used to be lost. The next flush, within a - minute, tries them again. Until a write succeeds, every report is refused and - nothing of it is kept, so the buffer does not grow while writes fail. -- 2026-10-14: a request whose handler panics is logged with status 500 and - counted with `code` 500 in the metrics, not 200 - ([#125](https://git.eeqj.de/sneak/netwatch/issues/125)). `Recoverer` is still - the first middleware. `Write500OnPanic` sits after the request log and after - the metrics. It writes the 500 as the panic passes and lets the panic go on to - Sentry and `Recoverer`. `Recoverer` now writes a 500 only when no status was - written. -- 2026-10-14: with `TRUSTED_PROXIES` unset, the container's nginx trusts the - three private ranges of RFC 1918, `10.0.0.0/8`, `172.16.0.0/12` and - `192.168.0.0/16` ([#119](https://git.eeqj.de/sneak/netwatch/issues/119)), as - sneak ruled on 2026-09-22 for every app. It trusted no proxy before. A list - replaces them, and set and empty it trusts none. The backend inside the - container still trusts only `127.0.0.1/32`. -- 2026-10-14: `POST /api/v1/reports` takes only a report sent with - `Content-Type: application/json`, parameters such as `charset` allowed - ([#126](https://git.eeqj.de/sneak/netwatch/issues/126)). It refuses any other - with 415 and the usual error body, storing nothing. A page on another origin - can send a `text/plain` POST without a CORS preflight, so any site could make - its visitors' browsers store reports. A JSON POST waits for the preflight, - which only an origin in `CORS_ALLOWED_ORIGINS` passes. The page already sent - `application/json`. -- 2026-10-14: changing the interval no longer drops the samples not yet reported - to the backend ([#124](https://git.eeqj.de/sneak/netwatch/issues/124)). The - interval menu resets each host with the new `HostState.reset`, which moves the - history's entries to `clearedHistory`, keeping the last 100 of all it has - cleared. `buildReport` reads both, so the next report has the samples recorded - before the change that no report has delivered. Until its next check, such a - host is sent with the status of its last check, not `pending`. The page's - history still starts over. `web/src/main.js` exports `Reporter` for the tests. -- 2026-10-14: the repo has the layout `REPO_POLICIES.md` gives - ([#30](https://git.eeqj.de/sneak/netwatch/issues/30)). The Go module is at the - root (`go.mod`, `go.sum`, `.golangci.yml`, `cmd/netwatch-server/`, - `internal/`), so its path `sneak.berlin/go/netwatch` names the repository. - `go.mod` ignores `node_modules/`, where npm packages can ship `.go` files that - `make tidy` would otherwise read as this module's packages. The page's source - is in `web/` (`index.html`, `src/`), which `vite.config.js` makes Vite's root; - the build still goes to `dist/`. `nginx.conf` and `security-headers.conf` are - in `configs/`. `backend/` is gone. Its `LICENSE` and `.dockerignore` are - deleted, its `.gitignore` entries are in the root one, and its `README.md` is - folded into the root `README.md`. Its scripts are in the root `script/`: - `script/build` builds the server binary after the frontend, and `script/run` - and `script/clean` came from `backend/script/`. `make test`, `make lint`, - `make fmt` and `make fmt-check` do what its `test`, `lint`, `fmt` and - `fmt-check` did. `script/bootstrap` no longer installs gcc, which only the - host Go tests needed. The image and the page are unchanged. -- 2026-10-07: the files shared from `sneak/prompts` were its copies at commit - `dd4027b` ([#113](https://git.eeqj.de/sneak/netwatch/issues/113)), with this - repository's own entries after the shared content in `.gitignore`, - `.dockerignore` and `.editorconfig`. `make lint` and `make test` each built - one phase of `Dockerfile` without the cache. `lint` ran golangci-lint v2.14.0 - over `backend/` and, through the `frontend-lint` stage, eslint. `test` ran the - Go tests on the Debian Go image and, through the `frontend` stage, the - frontend's unit tests and build. The builder stage waits on both. Without a - `VERSION` build argument it takes the version from `git describe` on the - `.git` in the build context. Neither linter nor the tests run on the host for - `make check`. `backend/.golangci.yml` was no longer checked against a sha256. - prettier formats only the JavaScript, CSS, HTML and Markdown, so the shared - `.golangci.yml` stays as fetched. `script/fmt` and `script/fmt-check` put - `~/.local/bin` on `PATH`. `script/frontend-lint` and `script/frontend-check` - are gone. -- 2026-10-07: the six Hetzner targets answer again, and failed checks are - written to the browser console - ([#114](https://git.eeqj.de/sneak/netwatch/issues/114)). The Hetzner - speed-test servers close the connection without an answer when the URL has a - query string, and every check added `?_cb=` and the time. Checks now fetch - each target's URL as written, with `cache: 'no-store'` as before. Each - recorded check that fails writes one `console.error` line, and the same line - to the debug log. The line gives the target's name and URL, the time, what - failed and how long the request took. A target that answers after failed - checks writes one `console.info` line. Checks the page does not record write - none, so the debug log no longer lists failures in the first round or in the - recovery probe. -- 2026-10-04: the page's footer no longer says "IPv4 only" - ([#111](https://git.eeqj.de/sneak/netwatch/issues/111)). Each check is a - `fetch`, the browser picks IPv4 or IPv6 for each WAN host, and the local - targets are IPv4 addresses. -- 2026-10-04: `README.md`, `TODO.md` and `test/viewport/README.md` say what the - tree does ([#24](https://git.eeqj.de/sneak/netwatch/issues/24)). The README's - Getting Started leads with `make` targets. A new Backend section says what - `netwatch-server` stores, its routes and how the image builds and runs it; it - pointed to `backend/README.md` for its settings. The checks are GET requests. - The 26 WAN hosts, the four health states, the summary's figures and the - features the list lacked are described as the page has them. The README's TODO - section points here. This file's Workflow branches from `next` and opens the - PR against `next`. Status says where the repo stands, and Next Step and Future - Steps hold only open work, linked to its issue where one exists. The viewport - test's README names Node's test runner, not `vitest`. -- 2026-10-04: in `src/main.js`, a target's min, max, median and average latency - come from one list of its answers - ([#102](https://git.eeqj.de/sneak/netwatch/issues/102)). The summary's figures - use the same function, so the median is written once. The latency color limits - are one table in `CONFIG`, read by both the figure's and the sparkline's - color. The health thresholds, the debug log's length and the gateway check's - timeout are `CONFIG` entries too. So are the recovery probe's host count and - interval, how often the rows are sorted, and the delay before the first - sparkline resize. Nothing the page does or shows changed; the footer's color - legend still writes the limits out as text. -- 2026-10-04: password guesses at `/metrics` are rate limited - ([#104](https://git.eeqj.de/sneak/netwatch/issues/104)). Each client address, - resolved through `TRUSTED_PROXIES` as for reports, may make 60 requests to - `/metrics` a minute, counted by `go-chi/httprate` apart from its reports. Past - that it gets 429 and its basic auth credentials are not checked. The limit is - a constant in `internal/server/routes.go`. -- 2026-10-04: the backend reports errors to Sentry - ([#95](https://git.eeqj.de/sneak/netwatch/issues/95)). With `SENTRY_DSN` set, - it sets up `sentry-go` with the release `netwatch-server-` and its version. It - reports each panic in a handler through `sentryhttp`, the last middleware - every request goes through. `sentryhttp` panics again, so the request still - gets the 500 from the panic recovery. On shutdown it waits up to 2 seconds for - Sentry to finish sending. A DSN Sentry refuses stops the start with an error - naming `SENTRY_DSN`. With it empty, Sentry is not set up and nothing is sent - to it. -- 2026-10-04: a target's name and URL and a debug log message show as the - characters they are and are never read as HTML - ([#29](https://git.eeqj.de/sneak/netwatch/issues/29)). A host row escapes the - name and URL it writes into its markup, and the debug log sets each line as - text. `README.md` no longer calls `CONFIG` frozen: the interval menu sets its - `updateInterval`, and the values computed from it follow. `AppState` declares - the recovery probe's two properties. The sparkline axis functions lose the - parameters they did not use, and the comment on a target's history names both - kinds of entry it holds. -- 2026-10-04: a dependency can be added without running yarn or go by hand - ([#45](https://git.eeqj.de/sneak/netwatch/issues/45)). - `make add-dependency PACKAGE=@` shims to the new - `script/add-dependency`, which runs `yarn add --dev`, so `package.json` and - `yarn.lock` change together, then `yarn install --frozen-lockfile`. The same - command moves a package to another version. `make tidy` shims to the new - `script/tidy`, which runs `go mod tidy`. A Go module is added by importing it, - or moved by editing its `require` line, then `make tidy`. `script/bootstrap` - still installs with `--frozen-lockfile`. -- 2026-10-04: the backend serves Prometheus metrics - ([#94](https://git.eeqj.de/sneak/netwatch/issues/94)). With `METRICS_USERNAME` - and `METRICS_PASSWORD` both set, it records request duration and response size - through `go-http-metrics`. It serves them, with Go's runtime and process - metrics, at `GET /metrics` behind basic auth with those credentials. nginx - passes `/metrics` to it as it does `/api/`. With neither set there are no - metrics and `/metrics` is 404. One without the other stops the start with an - error naming both, and so does a `METRICS_USERNAME` containing `:`. Only - requests that reach the health check or `POST /api/v1/reports` are recorded, - not `/metrics` itself and not every request as `GO_HTTP_SERVER_CONVENTIONS.md` - shows. The labels are the request's path and method, which clients can make up - without end. For that, `POST /api/v1/reports` is registered by its full path - instead of inside a `/api/v1` route group. -- 2026-10-04: `script/` and `Makefile` follow the org models - ([#28](https://git.eeqj.de/sneak/netwatch/issues/28)). `make dev` shims to the - new `script/dev`, the Vite dev server, and the new `make build` to - `script/build`, the frontend production build. `.prettierignore` no longer - left out `backend/`, so `make fmt` and `make fmt-check` covered - `backend/README.md`; it left out `backend/.golangci.yml` by name. - `script/install-precommit` and the date on `script/bootstrap`'s pins are the - org model again. `script/bootstrap`, `script/fmt` and `script/fmt-check` each - say in a comment why they differ from it. -- 2026-10-04: a frontend build on Node 26 or newer no longer prints Node's - warning that `module.register()` is deprecated - ([#32](https://git.eeqj.de/sneak/netwatch/issues/32)). The build in - `Dockerfile` runs on Node 22, which never printed it. The call was in - `@tailwindcss/node`, which `@tailwindcss/vite` brings in at its own exact - version. tailwind 4.3.1 calls `module.registerHooks()` instead where Node has - it. `yarn.lock` has `@tailwindcss/vite` and `tailwindcss` at 4.3.3, inside the - ranges `package.json` already allowed, and tailwind's own dependencies moved - with them. The built CSS changed only in how it is written out, in tailwind's - Firefox focus-ring rule, and in tailwind's default sans-serif font list, which - the page does not use. -- 2026-10-03: the frontend's unit tests cover what the page computes - ([#21](https://git.eeqj.de/sneak/netwatch/issues/21)). They cover - `humanDuration`, and the latency colours of a figure and of a sparkline either - side of each boundary. They cover a target's min, max, average and median - latency, and each of the four health states either side of its thresholds. - `package.json` has a `test` script, so `yarn run test` and `npm run test` run - them, and `script/frontend-test` runs it. `src/main.js` exports - `humanDuration`, `HostState`, `latencyClass` and `latencyHex` for the tests; - nothing it does changed. -- 2026-10-03: the backend's logs are one stream - ([#27](https://git.eeqj.de/sneak/netwatch/issues/27)). fx logs its own steps - of starting and stopping through the backend's logger. So off a terminal, - every line the backend and fx write is JSON; fx used to write plain text to - stderr. A config file that is found but cannot be read stops the start with - its error, logged as JSON like a bad setting. It used to end in a Go panic. - The backend logs its name, version and architecture once at start. The health - check's uptime keys are `uptime_seconds` and `uptime_human`; its path, content - type, `"status":"ok"` and 200 are unchanged. `SENTRY_DSN`, `METRICS_USERNAME` - and `METRICS_PASSWORD` were still read and unused until - [#94](https://git.eeqj.de/sneak/netwatch/issues/94) and - [#95](https://git.eeqj.de/sneak/netwatch/issues/95) wired them up. -- 2026-10-03: the frontend has a real linter - ([#47](https://git.eeqj.de/sneak/netwatch/issues/47), and item 2 of - [#28](https://git.eeqj.de/sneak/netwatch/issues/28)): `eslint` with its - recommended rules, set in `eslint.config.js`, runs in a new `frontend-lint` - stage of `Dockerfile`. `script/lint` builds the lint stages and runs no linter - on the host. `script/bootstrap` wants node 22.13.0 or newer, as eslint 10 - does. -- 2026-10-03: the tap-target check in `make frontend-viewport-test` expects one - visible pin button per WAN host row - ([#46](https://git.eeqj.de/sneak/netwatch/issues/46)), where it expected at - least 10 of the 26. Pin buttons missing from only some rows now fail it. The - host row count the test gathers, which the `app-rendered` check also reads, - counts only the WAN host rows: the local host rows have no pin button. -- 2026-10-03: each target's row shows its result as soon as its check ends - ([#91](https://git.eeqj.de/sneak/netwatch/issues/91)). Every row waited for - the round's slowest check, up to 24 seconds at a 30-second interval. Every row - is still redrawn, and sorting, the summary, the health box and offline - detection still run, once, when the round's last check ends. So no row reads - "paused" after a pause and resume during the round. A check that ends after - the user pauses or after its round is given up shows nothing. The first round - is still discarded as a whole. -- 2026-10-03: root no longer acts outside `/data` when it prepares `DATA_DIR` - ([#80](https://git.eeqj.de/sneak/netwatch/issues/80)). `bin/entrypoint.sh` - runs `netwatch-server prepare-data-dir`, which refuses a `DATA_DIR` that is - not `/data` or a path below it written in full. Then it creates `DATA_DIR`, - gives `/data` and everything in it to `netwatch` and sets the modes, all - through a Go `os.Root` opened on `/data`. That refuses any path leading out of - `/data`. So no symbolic link, already there or swapped in by a host process - during the start, can make root create or change anything elsewhere. - `DATA_DIR=/etc` no longer gives `/etc` to `netwatch`. The `README.md` section - "Running under upaas" says which values are accepted. -- 2026-10-03: `DATA_DIR_MAX_BYTES` is how much of the report files is kept - ([#54](https://git.eeqj.de/sneak/netwatch/issues/54)). When a report would - take them past it, the oldest report files are deleted to make room, each - deletion logged. At start, files already past it are deleted the same way. A - file still being written is never deleted. A report is refused with 507 only - when the reports waiting to be written fill the cap on their own, and then no - file is deleted. A file that cannot be deleted still counts until the next - start; one already deleted by hand counts as freed. -- 2026-10-03: `backend/script/lint` said what went wrong with its - `.golangci.yml` check ([#34](https://git.eeqj.de/sneak/netwatch/issues/34)). - On a hash mismatch it said to compare the file with the org standard. If they - differed, restore the org standard. If they were the same, the org standard - had changed, so update `GOLANGCI_CONFIG_SHA256` in that script. It used to say - only to restore the file, which loops once the org standard itself has moved. - A missing `.golangci.yml`, and a `sha256sum` that is missing or prints no - hash, each got their own message instead of being reported as a mismatch. -- 2026-10-03: the Go tests run with the race detector and coverage - ([#88](https://git.eeqj.de/sneak/netwatch/issues/88)): - `go test -timeout 30s -race -cover ./...` and, if that fails, the same again - with `-v`, then a failing exit. Go's `-timeout` bounds the tests, not their - compile. The race detector needs a C compiler; the binary is still built with - `CGO_ENABLED=0`. New tests cover the health check's answer, a valid report's - answer, a report file's exact contents, and the flush when the buffer reaches - 10 MiB. -- 2026-10-03: each target check times out after 80% of the refresh interval - ([#78](https://git.eeqj.de/sneak/netwatch/issues/78)), 24 seconds at 30 - seconds, where it was capped at 3 seconds. A round started early gives up the - last round's checks if they are still waiting, so rounds never overlap. A - round starts early after an interval change, or when the recovery probe finds - a target answering. The recovery probe gives up its own checks after half a - second. The frontend has its first unit tests, run by `script/frontend-test` - with Node's built-in test runner. For them, `index.html` links - `src/styles.css`, which `src/main.js` used to import. -- 2026-09-29: the container sets up its own data directory - ([#75](https://git.eeqj.de/sneak/netwatch/issues/75)). `bin/entrypoint.sh`, - still as root, created `DATA_DIR` if missing and gave it and `/data` to the - `netwatch` user with mode 750 before starting the backend as that user. So an - empty host directory owned by root, or one holding files from another uid, - works with no step on the host. It stopped the start when a symbolic link was - on the path to `DATA_DIR`, since root would change whatever the link points - to. The `README.md` first-run step that created and chowned the host directory - is gone, and the image no longer sets that ownership at build time. -- 2026-09-29: CI can no longer pass on checks that did not run - ([#37](https://git.eeqj.de/sneak/netwatch/issues/37)). `script/cibuild` became - the org model, byte for byte. It runs `script/bootstrap` and `script/check`, - then builds the image with the version from `git describe` as the `VERSION` - build argument. It used to be a plain `docker build .` whose check steps could - come from the build cache. The workflow put `~/.local/bin`, where bootstrap - links what it installs, on the step's `PATH`. bootstrap installs its pinned - node when the installed one is older than the frontend's dependencies need. -- 2026-09-29: `backend/.golangci.yml` was re-vendored from `sneak/prompts` - ([#41](https://git.eeqj.de/sneak/netwatch/issues/41)). `gomodguard`, - deprecated in golangci-lint v2.12.0, is disabled and its successor - `gomodguard_v2` enabled with the org block list, so lint runs print no - deprecation warning. The file also turns `depguard` on with its `test-support` - rule, which keeps `net/http/httptest` out of non-test code; netwatch adds no - entries of its own to that rule. -- 2026-09-29: nginx sends the security headers `REPO_POLICIES.md` requires on - every response ([#18](https://git.eeqj.de/sneak/netwatch/issues/18)), errors - and `/assets/` included. On what it passes on from the backend, it drops the - backend's own copies, so each header goes out once. They live in - `security-headers.conf`, which `nginx.conf` includes. The content security - policy allows no inline script or style, so the status dot's grey in - `src/main.js` is a class. `connect-src` is `*` because probed hosts redirect - to others, and the browser checks each redirect against it. -- 2026-09-29: the request log is bounded - ([#60](https://git.eeqj.de/sneak/netwatch/issues/60)). Each field it writes - from the request is cut to 128 bytes, the bound the report handler already - used. The fields are the method, URL, protocol, `User-Agent`, `Referer`, - request ID and client address. chi takes the request ID from the client's - `X-Request-Id` header. One request could put about 1 MiB per field into a log - line. That bound and its helper live in the `logger` package, shared by both. -- 2026-09-29: nginx takes the client address from `X-Forwarded-For` only on - requests from the reverse proxies named in the container's `TRUSTED_PROXIES` - ([#64](https://git.eeqj.de/sneak/netwatch/issues/64)), and at the time by - default from none. It trusted every RFC1918 address before, so a client could - write a new address on each request and escape the rate limit. - `bin/entrypoint.sh` writes one `set_real_ip_from` line per entry into - `/etc/nginx/trusted-proxies.conf`, which `nginx.conf` includes. It refuses an - entry that is not an IP address or CIDR, as `netwatch-server check-cidr` - finds. It starts the backend with `TRUSTED_PROXIES=127.0.0.1/32`, since nginx - is its only client. -- 2026-09-29: report file names can no longer collide - ([#61](https://git.eeqj.de/sneak/netwatch/issues/61)). Each is - `reports--.jsonl.zst`, where the number goes up by one for - each file the server starts to write. So two flushes in the same millisecond - each get a file of their own; the second one used to fail. A flush for size - and the final flush at shutdown can fall in the same millisecond. A failed - write uses up its number. -- 2026-09-29: ready to run under upaas - ([#59](https://git.eeqj.de/sneak/netwatch/issues/59)). The image has a - `HEALTHCHECK` that requests `/.well-known/healthcheck` through nginx on the - port from `PORT`. The backend no longer reads a bad `PORT` as 0 or a bad - `DEBUG` as false. Those, and a `BIND_ADDRESS` that is not an IP address, stop - it from starting with an error naming the variable. The limits, - `CORS_ALLOWED_ORIGINS` and, now by name, `TRUSTED_PROXIES` already did. - `bin/entrypoint.sh` also refuses a `PORT` outside 1 to 65535, and `8081`, - where the backend listens inside the container, naming `PORT`. `README.md` got - a "Running under upaas" section. -- 2026-09-29: nginx listens on `PORT` - ([#26](https://git.eeqj.de/sneak/netwatch/issues/26)), 8080 when unset or - empty. The nginx image renders `nginx.conf` as a template at container start, - filling in `PORT` and no other variable. `bin/entrypoint.sh` refuses to start - when `PORT` is not digits only. `server_tokens off` keeps the nginx version - out of responses. `script/frontend-viewport-test` renders the template the - same way. Gzip and a `50x.html` error page are not added. -- 2026-09-29: the report endpoint is bounded - ([#20](https://git.eeqj.de/sneak/netwatch/issues/20)). `POST /api/v1/reports` - still needs no credentials. Each client address, as resolved through - `TRUSTED_PROXIES`, may send `REPORTS_PER_MINUTE` (default 60) reports a - minute, counted by `go-chi/httprate`, and past that gets 429 with - `Retry-After`. The report files in `DATA_DIR`, counted from start with those - already there, may total at most `DATA_DIR_MAX_BYTES` (default 1 GiB); past - that, reports got 507. The wildcard CORS is gone: no CORS headers unless - `CORS_ALLOWED_ORIGINS` lists origins. An entry that is not a plain - `scheme://host[:port]` origin, `*` included, stops the server from starting. - Deleting report files freed room only at the next start, until - [#54](https://git.eeqj.de/sneak/netwatch/issues/54). -- 2026-09-28: one container image - ([#52](https://git.eeqj.de/sneak/netwatch/issues/52)). The root `Dockerfile` - builds the only image, and `Dockerfile.backend` is gone. nginx serves the - frontend on port 8080 and proxies `/api/` and `/.well-known/healthcheck` to - the backend, which listens on `127.0.0.1:8081` in the same container. The new - `BIND_ADDRESS` setting sets its listen address. `bin/entrypoint.sh` starts - both, passes TERM and INT on to both, and exits non-zero when either exits on - its own. The backend runs as user `netwatch` and stores reports on the `/data` - volume. `script/docker` is the org model again. -- 2026-09-28: one gate ([#16](https://git.eeqj.de/sneak/netwatch/issues/16)). - The root `make check` covers the Go backend as well as the frontend, and the - pre-commit hook with it. The backend moved onto scripts-to-rule-them-all: - `backend/script/*`, `backend/Makefile` as shims, its duplicate hook installer - removed. `script/cibuild` built both images and was the workflow's only build - step. The root `make lint` ran golangci-lint only in Docker, by building the - lint stage of `Dockerfile.backend` without the cache. `script/bootstrap` - installed the pinned Go unless the installed one was at least what - `backend/go.mod` asked for. It links what it installs into `~/.local/bin` - without replacing anything it did not create, and installs no linter. When - `VERSION` is unset or empty, the backend binary's version falls back to - `git describe` inside a git checkout, then to `dev`. -- 2026-09-28: frontend reporting client - ([#53](https://git.eeqj.de/sneak/netwatch/issues/53)). A `Reporter` class - posts collected samples to `/api/v1/reports` every `reportInterval` (default - 60s) as a per-host delta. The report-building step is a pure exported function - of host state. A per-host mark advances only on a delivered POST. At most one - report POST is pending at a time, and it is abandoned after half the interval. - So a slow POST never overlaps the next report, and a mark never moves - backwards. The samples of an abandoned POST are sent again at the next - interval, so a backend that stored them but answered late receives them twice. - The per-browser client id works in insecure (plain-HTTP) contexts. - `vite.config.js` proxies `/api` to the local backend for `yarn dev`. -- 2026-09-28: report ingest correctness - ([#23](https://git.eeqj.de/sneak/netwatch/issues/23)). A storage failure - returns 500 instead of a false `ok`. An oversize body returns 413, apart from - malformed JSON, which stays 400. A `MaxBodyBytes` middleware caps every route, - not just the report route. The raw attacker-controlled `geo` blob is no longer - logged, only its length, and `client_id`, `timestamp` and decode error text - are length-bounded before logging. A `decodeJSON` handler helper was added. - Panic recovery routes the stack through slog instead of chi's plain-text - stderr. Writing a report file returns its error, so a failed final flush on - shutdown makes the process exit non-zero instead of losing the buffered - reports silently. -- 2026-09-21: shutdown lifecycle correctness. The process shuts down through fx - instead of `os.Exit`, so every component's `OnStop` runs and buffered reports - are flushed to disk on `SIGTERM`. Before, a full flush window of telemetry was - lost on every restart. The `http.Server` is built before its serving goroutine - starts, so shutdown can no longer race or nil-deref it. A listen failure exits - non-zero via `fx.Shutdowner`. `reportbuf`'s `OnStop` is idempotent. - `writeTimeout` exceeds the chi per-request budget, so that budget is - reachable. Dead `startupTime`, `exitCode` and `cancelFunc` fields were - removed. -- 2026-09-21: backend HTTP hardening - ([#19](https://git.eeqj.de/sneak/netwatch/issues/19)): `ReadHeaderTimeout` and - `IdleTimeout` on the server, and a `SecurityHeaders` middleware (HSTS, tight - CSP, frame, sniff, referrer and permissions headers) registered before CORS. - `X-Forwarded-For` and `X-Real-IP` are honoured only from a `TRUSTED_PROXIES` - allowlist, at the time loopback plus RFC1918 by default. -- 2026-08-10: adopted the org-standard `backend/.golangci.yml` verbatim and - moved the pinned golangci-lint from v2.7.2 to v2.12.2; the `lint` stage of - `Dockerfile.backend` pinned the `golangci/golangci-lint:v2.12.2` image by - digest. The previous config declared `version: "2"` but used v1 schema keys, - so every threshold in it was inert and its green result meant nothing. - `backend/Makefile`'s `lint` target asserted the config's sha256 against the - canonical file first, so drift from the org standard failed the build instead - of silently degrading to defaults. -- 2026-08-10: every interactive control meets the 44x44 CSS px minimum tap - target: `.pin-btn`, `#interval-select`, the debug-log label and, on narrow - viewports, `#pause-btn`. The pin button's hit area grows via matching negative - margins, so its layout footprint and row density are unchanged. -- 2026-08-10: the per-host status line wraps below the 768px breakpoint instead - of forcing horizontal page scroll at 320px. -- 2026-08-09: `Dockerfile.backend` was reworked to the Go multistage lint-stage - pattern. It got a separate `lint` stage on the hash-pinned - `golangci/golangci-lint` image, and a `COPY --from=lint` stage dependency. The - build became a `CGO_ENABLED=0` static build driven by `ARG VERSION`, with no - more `COPY .git`. -- 2026-08-09: dotfile compliance. `backend/.editorconfig` moved to the repo root - so `root = true` covers the frontend too. `.gitignore` became the org model - (OS, editor, node, and environment/secrets sections) plus this repo's `dist/` - and `*.log`. `.env`, `.env.*`, `*.pem` and `*.key` are ignored repo-wide, not - just under `backend/`. Excluding `.git` from `.dockerignore` stayed deferred: - both images read git metadata at build time. -- 2026-08-09: automated responsive-layout test (`make frontend-viewport-test`): - a digest-pinned headless Chrome driven over CDP against the built `dist/`, at - viewport widths derived from the breakpoints in `src/styles.css` - ([#13](https://git.eeqj.de/sneak/netwatch/issues/13)). The tap-target and - host-row checks each fail when they measured nothing. The overflow, - viewport-edge and clipped-text checks have no such guard of their own and rely - on the `app-rendered` check, which fails the run when the app did not render. - It found two real layout defects, filed as - [#42](https://git.eeqj.de/sneak/netwatch/issues/42) and - [#43](https://git.eeqj.de/sneak/netwatch/issues/43). -- 2026-07-07: adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile - shims, README Entrypoints section. -- 2026-02-27: backend with buffered zstd-compressed report storage; CI workflow - and backend repo standard files; backend Dockerfile fixed (Go 1.25, - golangci-lint) and moved to repo root (`feat/reportbuf-storage`). -- 2026-02-26: host row layout redesigned with CSS grid; overflow and spacing - fixes; nginx config extracted; port hardcoded to 8080. -- 2026-02-26: debug log panel, median stats, recovery probe, Docker build fix, - S3 Singapore endpoint added. -- 2026-02-23: summary box redesign, host pinning, local and UTC clocks, checks - counter. -- 2026-02-23: hosts sorted by latency; GET instead of HEAD for latency; timeout - derived from interval; Hetzner regional endpoints; 3s interval. -- 2026-01-29: initial NetWatch network latency monitor. - # Future Steps - A backend test that posts a report to `POST /api/v1/reports` and checks the