sneak/netwatch
1
0
Fork 0
Code Issues 2 Pull Requests Actions 8 Packages Projects Releases Wiki Activity

README.md: rework against the second review

Browse Source 
The four findings: only a recorded failed check is logged, and the
discarded round, given-up checks and the recovery probe log nothing;
prepare-data-dir gives directories, regular files and symbolic links to
the netwatch user, and a hard link keeps its owner and is logged; the
cli entry names what it runs in three short sentences; the 80% timeout
and the early-round rule have their reasons back, so rounds never
overlap.

The full pass also fixed claims broader than the code (a row always
shows its stats; a report is posted every 60 seconds; a file is written
every minute; each refusal answers JSON; every answer to a report
carries rate-limit headers; every setting applies in the image, where
BIND_ADDRESS has no effect; fmt formats all files), six sentences whose
subject was split from its verb (the recovery probe, the forwarded
entry with a port, the failed write's retry and its file, both rate
limits, the nvm archive), and the viewport test's "same ... as"
comparison.

Model: fable-5-1
This commit is contained in:
clawbot
2026-10-26 05:41:11 +00:00
committed by sneak
parent 5788ea43e3
commit ddd3a83fdf
1 changed files with 75 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+75 -61
View File
@@ -46,9 +46,9 @@ reports are stored in `data/reports`.
The page shows one row per host. A row has the host's name, its URL, its latest
latency in milliseconds and a graph of its last 100 checks. Below the figure,
the row shows the minimum, median, average and maximum of the host's answers in
that history. A round of checks runs every 3 seconds by default. The header
shows the interval, how long the history spans, and whether the page is running
or paused.
that history. After a failed check, it shows `timeout` or `unreachable` there
instead. A round of checks runs every 3 seconds by default. The header shows the
interval, how long the history spans, and whether the page is running or paused.
### Targets
@@ -103,9 +103,14 @@ of the measurement. It carries no `Referer` header and no query string: the
Hetzner servers close the connection without an answer when either is present.
A check times out after 80% of the interval: 2.4 seconds at the default of 3
seconds, 24 seconds at 30 seconds. A check that times out, or answers after that
limit, is recorded as a timeout. A check that fails with a network error is
recorded as unreachable.
seconds, 24 seconds at 30 seconds. So a round's checks have all finished before
the next round is due. A check that times out, or answers after that limit, is
recorded as a timeout. A check that fails with a network error is recorded as
unreachable.
A round can start early, after an interval change or when the recovery probe
finds a target answering. It then gives up the last round's checks that are
still waiting, so rounds never overlap.
The first round after the page loads is discarded, as is the first round after
an interval change. Its connections are all new, so its figures run high.
@@ -152,9 +157,9 @@ Below the health box, a summary line covers the WAN hosts only:
changed
When no WAN host answers, the page starts a recovery probe. Every half second it
checks 4 WAN hosts, picked at random when the probe starts, and gives up the
checks it started half a second before. As soon as one answers, a new round
starts at once.
checks 4 WAN hosts and gives up the checks it started half a second before. The
4 are picked at random when the probe starts. As soon as one answers, a new
round starts at once.
### Row order
@@ -178,14 +183,15 @@ Pinned rows are sorted by name, and a line separates them from the rest.
- The header shows the local time with its time zone, and the UTC time. The
footer shows the color legend and links to the commit the page was built from.
Each check that fails writes one line to the browser console with
`console.error`, and the same line to the debug log. The line gives the target's
name and URL, the time, what failed and how long the request took. The three
kinds of failure are a timeout, an answer over the time limit, and a network
error. For a network error the line quotes the error the browser gives the page,
such as `TypeError: Failed to fetch`, which does not say why. When a target
answers again after failed checks, one `console.info` line gives its latency and
how many checks in a row had failed.
Each failed check the page records writes one line to the browser console with
`console.error`, and the same line to the debug log. The discarded first round,
the checks a round gives up and the recovery probe write none. The line gives
the target's name and URL, the time, what failed and how long the request took.
The three kinds of failure are a timeout, an answer over the time limit, and a
network error. For a network error the line quotes the error the browser gives
the page, such as `TypeError: Failed to fetch`, which does not say why. When a
target answers again after failed checks, one `console.info` line gives its
latency and how many checks in a row had failed.
### Browsers
@@ -194,8 +200,9 @@ custom properties.
## Reporting
Every 60 seconds the page posts a report to `/api/v1/reports` on the origin it
was served from. The report is JSON with four fields:
Every 60 seconds the page gathers its undelivered samples and posts them as a
report to `/api/v1/reports` on the origin it was served from. When there are
none, it posts nothing. The report is JSON with four fields:
- `clientId`: a random id, made once per browser and kept in `localStorage`.
When the browser has no usable storage, a new id is made for the session.
@@ -258,9 +265,11 @@ The image:
[Settings](#settings))
- stores the report files in `/data/reports`, on the `/data` volume. Before the
backend starts, `netwatch-server prepare-data-dir` creates that directory and
gives `/data` and everything in it to user `netwatch`. A host directory
mounted at `/data` therefore ends up owned by uid 1000. The command changes
nothing outside `/data`, whatever links it meets there.
gives `/data` and the directories, regular files and symbolic links in it to
user `netwatch`. A file with a second name (hard link) keeps its owner, and
its path is logged. A host directory mounted at `/data` therefore ends up
owned by uid 1000. The command changes nothing outside `/data`, whatever links
it meets there.
- takes the client address from `X-Forwarded-For` only on a request from a
reverse proxy named in `TRUSTED_PROXIES`. nginx resolves the address and
passes that one address to the backend. The backend is started with
@@ -289,7 +298,9 @@ The [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
- Port: container port `8080`.
- Volume: container path `/data`. The reports are kept in `/data/reports`.
- Environment variables: none is required. Every setting in
[Settings](#settings) applies, with three differences in the image:
[Settings](#settings) applies, with four differences in the image:
- `BIND_ADDRESS` has no effect. The backend listens on `127.0.0.1` inside
the container.
- `PORT` is nginx's public port, from 1 to 65535 and not `8081`, where the
backend listens inside the container.
- `DATA_DIR` must be `/data` or a path below it, written in full, with no
@@ -359,11 +370,11 @@ A request from a peer inside the set takes its client address from
`X-Forwarded-For`. Each proxy appends the address it took the request from, so
the entries on the right were written by trusted proxies and the ones further
left by the client. The client address is the right-most entry that is not in
the set. An entry written with a port, such as `198.51.100.7:5000` or
`[2001:db8::1]:443`, is read as that address. The peer's address is used when
every entry is in the set. It is also used when the search from the right meets
an entry that is not an address at all, such as `unknown`. `X-Real-IP` is read
only when `X-Forwarded-For` is missing or empty.
the set. An entry may carry a port, such as `198.51.100.7:5000` or
`[2001:db8::1]:443`. It is read as the address alone. The peer's address is used
when every entry is in the set. It is also used when the search from the right
meets an entry that is not an address at all, such as `unknown`. `X-Real-IP` is
read only when `X-Forwarded-For` is missing or empty.
### Routes
@@ -381,8 +392,9 @@ Every request body is capped at 1 MiB, and every request at 60 seconds.
### Report storage
The server keeps the reports it takes in memory and writes them to `DATA_DIR` as
files of JSON lines, compressed with zstd. It writes a file every minute,
whenever 10 MiB of reports are waiting, and when it stops.
files of JSON lines, compressed with zstd. It writes the waiting reports to a
new file every minute, whenever 10 MiB of them are waiting, and when it stops.
With none waiting, it writes no file.
The files are named `reports-<timestamp>-<number>.jsonl.zst`. The timestamp is
UTC to the millisecond, so the names sort by time. The number starts at 1 when
@@ -391,11 +403,11 @@ So two files started in the same millisecond still get different names. A file
is created with `O_EXCL`, so it never overwrites another.
A write can fail, as when the disk is full. The reports of a failed write stay
in memory, and the next write, within a minute, tries them again. One file is
written at a time, so a file holds its reports in the order they arrived, those
of a failed write included. A failed write uses up its number. Its file, if it
was created, is removed; if removing it fails too, the file stays and counts
toward `DATA_DIR_MAX_BYTES` from the next start.
in memory. The next write tries them again, within a minute. One file is written
at a time, so a file holds its reports in the order they arrived, those of a
failed write included. A failed write uses up its number. A file it created is
removed. If removing it fails too, the file stays and counts toward
`DATA_DIR_MAX_BYTES` from the next start.
From a failed write until a write succeeds, `POST /api/v1/reports` refuses every
report with 500 and keeps nothing of it, and the health check answers 503. The
@@ -407,7 +419,8 @@ them too, and if it fails, the process exits non-zero.
### Report limits
`POST /api/v1/reports` takes reports from anyone who can reach it, so it is
bounded in four ways. Each refusal answers with the body `{"status":"error"}`.
bounded in four ways. Each refusal by the backend answers with the body
`{"status":"error"}`.
- JSON only. A report must be sent with `Content-Type: application/json`, with
or without parameters such as `charset`, as the page sends it. Any other
@@ -422,15 +435,15 @@ bounded in four ways. Each refusal answers with the body `{"status":"error"}`.
end of the report is read. A body that stops being valid JSON sooner is
answered 400. In the image, nginx refuses a body over 1 MiB itself, with its
own HTML error page, and the backend never sees the request.
- Rate limit. Each client address, resolved through `TRUSTED_PROXIES`, may send
`REPORTS_PER_MINUTE` reports a minute. Past that it gets 429 with
`Retry-After: 60`. The minute slides: reports from the minute before still
count, fading out over the current one. An address that sends at most half the
limit in any 60 seconds is never refused. Above that, it may be. The page
sends one report a minute from each open tab. So the default of 60 refuses
nothing from up to 30 tabs behind one address, however their reports bunch up.
Every answer to a report carries the headers `X-RateLimit-Limit`,
`X-RateLimit-Remaining` and `X-RateLimit-Reset`.
- Rate limit. Each client address may send `REPORTS_PER_MINUTE` reports a
minute. [Trusted proxies](#trusted-proxies) says how that address is found.
Past that it gets 429 with `Retry-After: 60`. The minute slides: reports from
the minute before still count, fading out over the current one. An address
that sends at most half the limit in any 60 seconds is never refused. Above
that, it may be. The page sends one report a minute from each open tab. So the
default of 60 refuses nothing from up to 30 tabs behind one address, however
their reports bunch up. Every answer the route gives carries the headers
`X-RateLimit-Limit`, `X-RateLimit-Remaining` and `X-RateLimit-Reset`.
- Size cap. The report files in `DATA_DIR` may total at most
`DATA_DIR_MAX_BYTES`, counting the files already there at start. Reports
waiting in memory count at their uncompressed size until they are written.
@@ -484,8 +497,8 @@ Only those two routes are recorded because the labels are the request's path and
method. Recording every request would let clients add labels without bound.
`/metrics` has its own rate limit, so that its password cannot be guessed
quickly. Each client address, resolved through `TRUSTED_PROXIES`, may make 60
requests to it a minute, whatever their credentials. Past that it gets 429 with
quickly. Each client address may make 60 requests to it a minute, whatever its
credentials. The address is found as for reports. Past that it gets 429 with
`Retry-After: 60`, and its credentials are not checked. The minute slides as it
does for reports, so a scraper polling every 2 seconds or less often is never
refused.
@@ -549,10 +562,10 @@ The page is built with Vite and Tailwind CSS v4. All its code is in
`netwatch-server` is a Go HTTP server wired with `fx`. Its packages are in
`internal/`:
- `cli` runs the command line: the `check-cidr` and `prepare-data-dir`
subcommands, which `bin/entrypoint.sh` runs before the server starts, and,
given no arguments, the server. Any other arguments get a usage message and
exit code 2.
- `cli` runs the command line. Given no arguments, it runs the server. It also
runs the `check-cidr` and `prepare-data-dir` subcommands, which
`bin/entrypoint.sh` runs before the server starts. Any other arguments get a
usage message and exit code 2.
- `config` reads the settings and refuses values the server cannot use.
- `server` builds the router and the HTTP server and registers the routes.
- `middleware` holds the request log, the security headers, CORS, the body
@@ -582,8 +595,8 @@ except `script/bootstrap`.
- `script/bootstrap`: install the dependencies. It uses the `node` on `PATH`
when that node has the pinned major version and is new enough for the page's
dependencies. Otherwise it installs the pinned node through nvm, fetched as a
hash-verified archive. It uses the `yarn` on `PATH` only when it is exactly
dependencies. Otherwise it installs nvm from a hash-verified archive, and the
pinned node through it. It uses the `yarn` on `PATH` only when it is exactly
the pinned version. Otherwise it installs that version through corepack, or
through npm where there is no corepack. Then it runs
`yarn install --frozen-lockfile`. It uses the `go` on `PATH` only when it is
@@ -612,8 +625,8 @@ except `script/bootstrap`.
- `script/lint`: build the `lint` phase of `Dockerfile`, writing no image. The
phase runs golangci-lint over the Go packages, and eslint in its
`frontend-lint` stage.
- `script/fmt`: format all files (writes): prettier over the JavaScript, CSS,
HTML and Markdown, then gofmt over `cmd/` and `internal/`. It runs on the
- `script/fmt`: run the formatters, which write: prettier over the JavaScript,
CSS, HTML and Markdown, then gofmt over `cmd/` and `internal/`. It runs on the
host, with `~/.local/bin` put on `PATH`.
- `script/fmt-check`: check the formatting (read-only), with the same scope as
`script/fmt`.
@@ -662,12 +675,13 @@ The yarn packages are installed again on every build.
call the functions it exports; no browser is involved.
`make frontend-viewport-test` builds the page and checks its layout at several
window widths in a headless Chrome, served by the same nginx image and
configuration as the Docker image. It tests each breakpoint declared in
`web/src/styles.css`, today the one at `max-width: 768px`, and the widths one
pixel either side of it. It also tests four fixed widths: 320px, 1280px and two
phone landscape sizes. [test/viewport/README.md](test/viewport/README.md) says
what it checks and what it cannot.
window widths in a headless Chrome. The page is served with the nginx image and
the `configs/nginx.conf` that the Docker image ships. It tests each breakpoint
declared in `web/src/styles.css`, today the one at `max-width: 768px`, and the
widths one pixel either side of it. It also tests four fixed widths: 320px,
1280px and two phone landscape sizes.
[test/viewport/README.md](test/viewport/README.md) says what it checks and what
it cannot.
## TODO