From a3b18b6ce8af8ced836c2e9ca92df752d72e85b2 Mon Sep 17 00:00:00 2001 From: clawbot Date: Mon, 26 Oct 2026 04:41:53 +0000 Subject: [PATCH] README.md: rework against the second review The four findings: only a recorded failed check is logged, and the discarded round, given-up checks and the recovery probe log nothing; prepare-data-dir gives directories, regular files and symbolic links to the netwatch user, and a hard link keeps its owner and is logged; the cli entry names what it runs in three short sentences; the 80% timeout and the early-round rule have their reasons back, so rounds never overlap. The full pass also fixed claims broader than the code (a row always shows its stats; a report is posted every 60 seconds; a file is written every minute; each refusal answers JSON; every answer to a report carries rate-limit headers; every setting applies in the image, where BIND_ADDRESS has no effect; fmt formats all files), six sentences whose subject was split from its verb (the recovery probe, the forwarded entry with a port, the failed write's retry and its file, both rate limits, the nvm archive), and the viewport test's "same ... as" comparison. Model: fable-5-1 --- README.md | 136 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 75 insertions(+), 61 deletions(-) diff --git a/README.md b/README.md index e27fbb0..4ad498b 100644 --- a/README.md +++ b/README.md @@ -46,9 +46,9 @@ reports are stored in `data/reports`. The page shows one row per host. A row has the host's name, its URL, its latest latency in milliseconds and a graph of its last 100 checks. Below the figure, the row shows the minimum, median, average and maximum of the host's answers in -that history. A round of checks runs every 3 seconds by default. The header -shows the interval, how long the history spans, and whether the page is running -or paused. +that history. After a failed check, it shows `timeout` or `unreachable` there +instead. A round of checks runs every 3 seconds by default. The header shows the +interval, how long the history spans, and whether the page is running or paused. ### Targets @@ -103,9 +103,14 @@ of the measurement. It carries no `Referer` header and no query string: the Hetzner servers close the connection without an answer when either is present. A check times out after 80% of the interval: 2.4 seconds at the default of 3 -seconds, 24 seconds at 30 seconds. A check that times out, or answers after that -limit, is recorded as a timeout. A check that fails with a network error is -recorded as unreachable. +seconds, 24 seconds at 30 seconds. So a round's checks have all finished before +the next round is due. A check that times out, or answers after that limit, is +recorded as a timeout. A check that fails with a network error is recorded as +unreachable. + +A round can start early, after an interval change or when the recovery probe +finds a target answering. It then gives up the last round's checks that are +still waiting, so rounds never overlap. The first round after the page loads is discarded, as is the first round after an interval change. Its connections are all new, so its figures run high. @@ -152,9 +157,9 @@ Below the health box, a summary line covers the WAN hosts only: changed When no WAN host answers, the page starts a recovery probe. Every half second it -checks 4 WAN hosts, picked at random when the probe starts, and gives up the -checks it started half a second before. As soon as one answers, a new round -starts at once. +checks 4 WAN hosts and gives up the checks it started half a second before. The +4 are picked at random when the probe starts. As soon as one answers, a new +round starts at once. ### Row order @@ -178,14 +183,15 @@ Pinned rows are sorted by name, and a line separates them from the rest. - The header shows the local time with its time zone, and the UTC time. The footer shows the color legend and links to the commit the page was built from. -Each check that fails writes one line to the browser console with -`console.error`, and the same line to the debug log. The line gives the target's -name and URL, the time, what failed and how long the request took. The three -kinds of failure are a timeout, an answer over the time limit, and a network -error. For a network error the line quotes the error the browser gives the page, -such as `TypeError: Failed to fetch`, which does not say why. When a target -answers again after failed checks, one `console.info` line gives its latency and -how many checks in a row had failed. +Each failed check the page records writes one line to the browser console with +`console.error`, and the same line to the debug log. The discarded first round, +the checks a round gives up and the recovery probe write none. The line gives +the target's name and URL, the time, what failed and how long the request took. +The three kinds of failure are a timeout, an answer over the time limit, and a +network error. For a network error the line quotes the error the browser gives +the page, such as `TypeError: Failed to fetch`, which does not say why. When a +target answers again after failed checks, one `console.info` line gives its +latency and how many checks in a row had failed. ### Browsers @@ -194,8 +200,9 @@ custom properties. ## Reporting -Every 60 seconds the page posts a report to `/api/v1/reports` on the origin it -was served from. The report is JSON with four fields: +Every 60 seconds the page gathers its undelivered samples and posts them as a +report to `/api/v1/reports` on the origin it was served from. When there are +none, it posts nothing. The report is JSON with four fields: - `clientId`: a random id, made once per browser and kept in `localStorage`. When the browser has no usable storage, a new id is made for the session. @@ -258,9 +265,11 @@ The image: [Settings](#settings)) - stores the report files in `/data/reports`, on the `/data` volume. Before the backend starts, `netwatch-server prepare-data-dir` creates that directory and - gives `/data` and everything in it to user `netwatch`. A host directory - mounted at `/data` therefore ends up owned by uid 1000. The command changes - nothing outside `/data`, whatever links it meets there. + gives `/data` and the directories, regular files and symbolic links in it to + user `netwatch`. A file with a second name (hard link) keeps its owner, and + its path is logged. A host directory mounted at `/data` therefore ends up + owned by uid 1000. The command changes nothing outside `/data`, whatever links + it meets there. - takes the client address from `X-Forwarded-For` only on a request from a reverse proxy named in `TRUSTED_PROXIES`. nginx resolves the address and passes that one address to the backend. The backend is started with @@ -289,7 +298,9 @@ The [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs: - Port: container port `8080`. - Volume: container path `/data`. The reports are kept in `/data/reports`. - Environment variables: none is required. Every setting in - [Settings](#settings) applies, with three differences in the image: + [Settings](#settings) applies, with four differences in the image: + - `BIND_ADDRESS` has no effect. The backend listens on `127.0.0.1` inside + the container. - `PORT` is nginx's public port, from 1 to 65535 and not `8081`, where the backend listens inside the container. - `DATA_DIR` must be `/data` or a path below it, written in full, with no @@ -359,11 +370,11 @@ A request from a peer inside the set takes its client address from `X-Forwarded-For`. Each proxy appends the address it took the request from, so the entries on the right were written by trusted proxies and the ones further left by the client. The client address is the right-most entry that is not in -the set. An entry written with a port, such as `198.51.100.7:5000` or -`[2001:db8::1]:443`, is read as that address. The peer's address is used when -every entry is in the set. It is also used when the search from the right meets -an entry that is not an address at all, such as `unknown`. `X-Real-IP` is read -only when `X-Forwarded-For` is missing or empty. +the set. An entry may carry a port, such as `198.51.100.7:5000` or +`[2001:db8::1]:443`. It is read as the address alone. The peer's address is used +when every entry is in the set. It is also used when the search from the right +meets an entry that is not an address at all, such as `unknown`. `X-Real-IP` is +read only when `X-Forwarded-For` is missing or empty. ### Routes @@ -381,8 +392,9 @@ Every request body is capped at 1 MiB, and every request at 60 seconds. ### Report storage The server keeps the reports it takes in memory and writes them to `DATA_DIR` as -files of JSON lines, compressed with zstd. It writes a file every minute, -whenever 10 MiB of reports are waiting, and when it stops. +files of JSON lines, compressed with zstd. It writes the waiting reports to a +new file every minute, whenever 10 MiB of them are waiting, and when it stops. +With none waiting, it writes no file. The files are named `reports--.jsonl.zst`. The timestamp is UTC to the millisecond, so the names sort by time. The number starts at 1 when @@ -391,11 +403,11 @@ So two files started in the same millisecond still get different names. A file is created with `O_EXCL`, so it never overwrites another. A write can fail, as when the disk is full. The reports of a failed write stay -in memory, and the next write, within a minute, tries them again. One file is -written at a time, so a file holds its reports in the order they arrived, those -of a failed write included. A failed write uses up its number. Its file, if it -was created, is removed; if removing it fails too, the file stays and counts -toward `DATA_DIR_MAX_BYTES` from the next start. +in memory. The next write tries them again, within a minute. One file is written +at a time, so a file holds its reports in the order they arrived, those of a +failed write included. A failed write uses up its number. A file it created is +removed. If removing it fails too, the file stays and counts toward +`DATA_DIR_MAX_BYTES` from the next start. From a failed write until a write succeeds, `POST /api/v1/reports` refuses every report with 500 and keeps nothing of it, and the health check answers 503. The @@ -407,7 +419,8 @@ them too, and if it fails, the process exits non-zero. ### Report limits `POST /api/v1/reports` takes reports from anyone who can reach it, so it is -bounded in four ways. Each refusal answers with the body `{"status":"error"}`. +bounded in four ways. Each refusal by the backend answers with the body +`{"status":"error"}`. - JSON only. A report must be sent with `Content-Type: application/json`, with or without parameters such as `charset`, as the page sends it. Any other @@ -422,15 +435,15 @@ bounded in four ways. Each refusal answers with the body `{"status":"error"}`. end of the report is read. A body that stops being valid JSON sooner is answered 400. In the image, nginx refuses a body over 1 MiB itself, with its own HTML error page, and the backend never sees the request. -- Rate limit. Each client address, resolved through `TRUSTED_PROXIES`, may send - `REPORTS_PER_MINUTE` reports a minute. Past that it gets 429 with - `Retry-After: 60`. The minute slides: reports from the minute before still - count, fading out over the current one. An address that sends at most half the - limit in any 60 seconds is never refused. Above that, it may be. The page - sends one report a minute from each open tab. So the default of 60 refuses - nothing from up to 30 tabs behind one address, however their reports bunch up. - Every answer to a report carries the headers `X-RateLimit-Limit`, - `X-RateLimit-Remaining` and `X-RateLimit-Reset`. +- Rate limit. Each client address may send `REPORTS_PER_MINUTE` reports a + minute. [Trusted proxies](#trusted-proxies) says how that address is found. + Past that it gets 429 with `Retry-After: 60`. The minute slides: reports from + the minute before still count, fading out over the current one. An address + that sends at most half the limit in any 60 seconds is never refused. Above + that, it may be. The page sends one report a minute from each open tab. So the + default of 60 refuses nothing from up to 30 tabs behind one address, however + their reports bunch up. Every answer the route gives carries the headers + `X-RateLimit-Limit`, `X-RateLimit-Remaining` and `X-RateLimit-Reset`. - Size cap. The report files in `DATA_DIR` may total at most `DATA_DIR_MAX_BYTES`, counting the files already there at start. Reports waiting in memory count at their uncompressed size until they are written. @@ -484,8 +497,8 @@ Only those two routes are recorded because the labels are the request's path and method. Recording every request would let clients add labels without bound. `/metrics` has its own rate limit, so that its password cannot be guessed -quickly. Each client address, resolved through `TRUSTED_PROXIES`, may make 60 -requests to it a minute, whatever their credentials. Past that it gets 429 with +quickly. Each client address may make 60 requests to it a minute, whatever its +credentials. The address is found as for reports. Past that it gets 429 with `Retry-After: 60`, and its credentials are not checked. The minute slides as it does for reports, so a scraper polling every 2 seconds or less often is never refused. @@ -549,10 +562,10 @@ The page is built with Vite and Tailwind CSS v4. All its code is in `netwatch-server` is a Go HTTP server wired with `fx`. Its packages are in `internal/`: -- `cli` runs the command line: the `check-cidr` and `prepare-data-dir` - subcommands, which `bin/entrypoint.sh` runs before the server starts, and, - given no arguments, the server. Any other arguments get a usage message and - exit code 2. +- `cli` runs the command line. Given no arguments, it runs the server. It also + runs the `check-cidr` and `prepare-data-dir` subcommands, which + `bin/entrypoint.sh` runs before the server starts. Any other arguments get a + usage message and exit code 2. - `config` reads the settings and refuses values the server cannot use. - `server` builds the router and the HTTP server and registers the routes. - `middleware` holds the request log, the security headers, CORS, the body @@ -582,8 +595,8 @@ except `script/bootstrap`. - `script/bootstrap`: install the dependencies. It uses the `node` on `PATH` when that node has the pinned major version and is new enough for the page's - dependencies. Otherwise it installs the pinned node through nvm, fetched as a - hash-verified archive. It uses the `yarn` on `PATH` only when it is exactly + dependencies. Otherwise it installs nvm from a hash-verified archive, and the + pinned node through it. It uses the `yarn` on `PATH` only when it is exactly the pinned version. Otherwise it installs that version through corepack, or through npm where there is no corepack. Then it runs `yarn install --frozen-lockfile`. It uses the `go` on `PATH` only when it is @@ -612,8 +625,8 @@ except `script/bootstrap`. - `script/lint`: build the `lint` phase of `Dockerfile`, writing no image. The phase runs golangci-lint over the Go packages, and eslint in its `frontend-lint` stage. -- `script/fmt`: format all files (writes): prettier over the JavaScript, CSS, - HTML and Markdown, then gofmt over `cmd/` and `internal/`. It runs on the +- `script/fmt`: run the formatters, which write: prettier over the JavaScript, + CSS, HTML and Markdown, then gofmt over `cmd/` and `internal/`. It runs on the host, with `~/.local/bin` put on `PATH`. - `script/fmt-check`: check the formatting (read-only), with the same scope as `script/fmt`. @@ -662,12 +675,13 @@ The yarn packages are installed again on every build. call the functions it exports; no browser is involved. `make frontend-viewport-test` builds the page and checks its layout at several -window widths in a headless Chrome, served by the same nginx image and -configuration as the Docker image. It tests each breakpoint declared in -`web/src/styles.css`, today the one at `max-width: 768px`, and the widths one -pixel either side of it. It also tests four fixed widths: 320px, 1280px and two -phone landscape sizes. [test/viewport/README.md](test/viewport/README.md) says -what it checks and what it cannot. +window widths in a headless Chrome. The page is served with the nginx image and +the `configs/nginx.conf` that the Docker image ships. It tests each breakpoint +declared in `web/src/styles.css`, today the one at `max-width: 768px`, and the +widths one pixel either side of it. It also tests four fixed widths: 320px, +1280px and two phone landscape sizes. +[test/viewport/README.md](test/viewport/README.md) says what it checks and what +it cannot. ## TODO