The four findings: only a recorded failed check is logged, and the discarded round, given-up checks and the recovery probe log nothing; prepare-data-dir gives directories, regular files and symbolic links to the netwatch user, and a hard link keeps its owner and is logged; the cli entry names what it runs in three short sentences; the 80% timeout and the early-round rule have their reasons back, so rounds never overlap. The full pass also fixed claims broader than the code (a row always shows its stats; a report is posted every 60 seconds; a file is written every minute; each refusal answers JSON; every answer to a report carries rate-limit headers; every setting applies in the image, where BIND_ADDRESS has no effect; fmt formats all files), six sentences whose subject was split from its verb (the recovery probe, the forwarded entry with a port, the failed write's retry and its file, both rate limits, the nvm archive), and the viewport test's "same ... as" comparison. Model: fable-5-1
This commit is contained in:
@@ -46,9 +46,9 @@ reports are stored in `data/reports`.
|
||||
The page shows one row per host. A row has the host's name, its URL, its latest
|
||||
latency in milliseconds and a graph of its last 100 checks. Below the figure,
|
||||
the row shows the minimum, median, average and maximum of the host's answers in
|
||||
that history. A round of checks runs every 3 seconds by default. The header
|
||||
shows the interval, how long the history spans, and whether the page is running
|
||||
or paused.
|
||||
that history. After a failed check, it shows `timeout` or `unreachable` there
|
||||
instead. A round of checks runs every 3 seconds by default. The header shows the
|
||||
interval, how long the history spans, and whether the page is running or paused.
|
||||
|
||||
### Targets
|
||||
|
||||
@@ -103,9 +103,14 @@ of the measurement. It carries no `Referer` header and no query string: the
|
||||
Hetzner servers close the connection without an answer when either is present.
|
||||
|
||||
A check times out after 80% of the interval: 2.4 seconds at the default of 3
|
||||
seconds, 24 seconds at 30 seconds. A check that times out, or answers after that
|
||||
limit, is recorded as a timeout. A check that fails with a network error is
|
||||
recorded as unreachable.
|
||||
seconds, 24 seconds at 30 seconds. So a round's checks have all finished before
|
||||
the next round is due. A check that times out, or answers after that limit, is
|
||||
recorded as a timeout. A check that fails with a network error is recorded as
|
||||
unreachable.
|
||||
|
||||
A round can start early, after an interval change or when the recovery probe
|
||||
finds a target answering. It then gives up the last round's checks that are
|
||||
still waiting, so rounds never overlap.
|
||||
|
||||
The first round after the page loads is discarded, as is the first round after
|
||||
an interval change. Its connections are all new, so its figures run high.
|
||||
@@ -152,9 +157,9 @@ Below the health box, a summary line covers the WAN hosts only:
|
||||
changed
|
||||
|
||||
When no WAN host answers, the page starts a recovery probe. Every half second it
|
||||
checks 4 WAN hosts, picked at random when the probe starts, and gives up the
|
||||
checks it started half a second before. As soon as one answers, a new round
|
||||
starts at once.
|
||||
checks 4 WAN hosts and gives up the checks it started half a second before. The
|
||||
4 are picked at random when the probe starts. As soon as one answers, a new
|
||||
round starts at once.
|
||||
|
||||
### Row order
|
||||
|
||||
@@ -178,14 +183,15 @@ Pinned rows are sorted by name, and a line separates them from the rest.
|
||||
- The header shows the local time with its time zone, and the UTC time. The
|
||||
footer shows the color legend and links to the commit the page was built from.
|
||||
|
||||
Each check that fails writes one line to the browser console with
|
||||
`console.error`, and the same line to the debug log. The line gives the target's
|
||||
name and URL, the time, what failed and how long the request took. The three
|
||||
kinds of failure are a timeout, an answer over the time limit, and a network
|
||||
error. For a network error the line quotes the error the browser gives the page,
|
||||
such as `TypeError: Failed to fetch`, which does not say why. When a target
|
||||
answers again after failed checks, one `console.info` line gives its latency and
|
||||
how many checks in a row had failed.
|
||||
Each failed check the page records writes one line to the browser console with
|
||||
`console.error`, and the same line to the debug log. The discarded first round,
|
||||
the checks a round gives up and the recovery probe write none. The line gives
|
||||
the target's name and URL, the time, what failed and how long the request took.
|
||||
The three kinds of failure are a timeout, an answer over the time limit, and a
|
||||
network error. For a network error the line quotes the error the browser gives
|
||||
the page, such as `TypeError: Failed to fetch`, which does not say why. When a
|
||||
target answers again after failed checks, one `console.info` line gives its
|
||||
latency and how many checks in a row had failed.
|
||||
|
||||
### Browsers
|
||||
|
||||
@@ -194,8 +200,9 @@ custom properties.
|
||||
|
||||
## Reporting
|
||||
|
||||
Every 60 seconds the page posts a report to `/api/v1/reports` on the origin it
|
||||
was served from. The report is JSON with four fields:
|
||||
Every 60 seconds the page gathers its undelivered samples and posts them as a
|
||||
report to `/api/v1/reports` on the origin it was served from. When there are
|
||||
none, it posts nothing. The report is JSON with four fields:
|
||||
|
||||
- `clientId`: a random id, made once per browser and kept in `localStorage`.
|
||||
When the browser has no usable storage, a new id is made for the session.
|
||||
@@ -258,9 +265,11 @@ The image:
|
||||
[Settings](#settings))
|
||||
- stores the report files in `/data/reports`, on the `/data` volume. Before the
|
||||
backend starts, `netwatch-server prepare-data-dir` creates that directory and
|
||||
gives `/data` and everything in it to user `netwatch`. A host directory
|
||||
mounted at `/data` therefore ends up owned by uid 1000. The command changes
|
||||
nothing outside `/data`, whatever links it meets there.
|
||||
gives `/data` and the directories, regular files and symbolic links in it to
|
||||
user `netwatch`. A file with a second name (hard link) keeps its owner, and
|
||||
its path is logged. A host directory mounted at `/data` therefore ends up
|
||||
owned by uid 1000. The command changes nothing outside `/data`, whatever links
|
||||
it meets there.
|
||||
- takes the client address from `X-Forwarded-For` only on a request from a
|
||||
reverse proxy named in `TRUSTED_PROXIES`. nginx resolves the address and
|
||||
passes that one address to the backend. The backend is started with
|
||||
@@ -289,7 +298,9 @@ The [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
|
||||
- Port: container port `8080`.
|
||||
- Volume: container path `/data`. The reports are kept in `/data/reports`.
|
||||
- Environment variables: none is required. Every setting in
|
||||
[Settings](#settings) applies, with three differences in the image:
|
||||
[Settings](#settings) applies, with four differences in the image:
|
||||
- `BIND_ADDRESS` has no effect. The backend listens on `127.0.0.1` inside
|
||||
the container.
|
||||
- `PORT` is nginx's public port, from 1 to 65535 and not `8081`, where the
|
||||
backend listens inside the container.
|
||||
- `DATA_DIR` must be `/data` or a path below it, written in full, with no
|
||||
@@ -359,11 +370,11 @@ A request from a peer inside the set takes its client address from
|
||||
`X-Forwarded-For`. Each proxy appends the address it took the request from, so
|
||||
the entries on the right were written by trusted proxies and the ones further
|
||||
left by the client. The client address is the right-most entry that is not in
|
||||
the set. An entry written with a port, such as `198.51.100.7:5000` or
|
||||
`[2001:db8::1]:443`, is read as that address. The peer's address is used when
|
||||
every entry is in the set. It is also used when the search from the right meets
|
||||
an entry that is not an address at all, such as `unknown`. `X-Real-IP` is read
|
||||
only when `X-Forwarded-For` is missing or empty.
|
||||
the set. An entry may carry a port, such as `198.51.100.7:5000` or
|
||||
`[2001:db8::1]:443`. It is read as the address alone. The peer's address is used
|
||||
when every entry is in the set. It is also used when the search from the right
|
||||
meets an entry that is not an address at all, such as `unknown`. `X-Real-IP` is
|
||||
read only when `X-Forwarded-For` is missing or empty.
|
||||
|
||||
### Routes
|
||||
|
||||
@@ -381,8 +392,9 @@ Every request body is capped at 1 MiB, and every request at 60 seconds.
|
||||
### Report storage
|
||||
|
||||
The server keeps the reports it takes in memory and writes them to `DATA_DIR` as
|
||||
files of JSON lines, compressed with zstd. It writes a file every minute,
|
||||
whenever 10 MiB of reports are waiting, and when it stops.
|
||||
files of JSON lines, compressed with zstd. It writes the waiting reports to a
|
||||
new file every minute, whenever 10 MiB of them are waiting, and when it stops.
|
||||
With none waiting, it writes no file.
|
||||
|
||||
The files are named `reports-<timestamp>-<number>.jsonl.zst`. The timestamp is
|
||||
UTC to the millisecond, so the names sort by time. The number starts at 1 when
|
||||
@@ -391,11 +403,11 @@ So two files started in the same millisecond still get different names. A file
|
||||
is created with `O_EXCL`, so it never overwrites another.
|
||||
|
||||
A write can fail, as when the disk is full. The reports of a failed write stay
|
||||
in memory, and the next write, within a minute, tries them again. One file is
|
||||
written at a time, so a file holds its reports in the order they arrived, those
|
||||
of a failed write included. A failed write uses up its number. Its file, if it
|
||||
was created, is removed; if removing it fails too, the file stays and counts
|
||||
toward `DATA_DIR_MAX_BYTES` from the next start.
|
||||
in memory. The next write tries them again, within a minute. One file is written
|
||||
at a time, so a file holds its reports in the order they arrived, those of a
|
||||
failed write included. A failed write uses up its number. A file it created is
|
||||
removed. If removing it fails too, the file stays and counts toward
|
||||
`DATA_DIR_MAX_BYTES` from the next start.
|
||||
|
||||
From a failed write until a write succeeds, `POST /api/v1/reports` refuses every
|
||||
report with 500 and keeps nothing of it, and the health check answers 503. The
|
||||
@@ -407,7 +419,8 @@ them too, and if it fails, the process exits non-zero.
|
||||
### Report limits
|
||||
|
||||
`POST /api/v1/reports` takes reports from anyone who can reach it, so it is
|
||||
bounded in four ways. Each refusal answers with the body `{"status":"error"}`.
|
||||
bounded in four ways. Each refusal by the backend answers with the body
|
||||
`{"status":"error"}`.
|
||||
|
||||
- JSON only. A report must be sent with `Content-Type: application/json`, with
|
||||
or without parameters such as `charset`, as the page sends it. Any other
|
||||
@@ -422,15 +435,15 @@ bounded in four ways. Each refusal answers with the body `{"status":"error"}`.
|
||||
end of the report is read. A body that stops being valid JSON sooner is
|
||||
answered 400. In the image, nginx refuses a body over 1 MiB itself, with its
|
||||
own HTML error page, and the backend never sees the request.
|
||||
- Rate limit. Each client address, resolved through `TRUSTED_PROXIES`, may send
|
||||
`REPORTS_PER_MINUTE` reports a minute. Past that it gets 429 with
|
||||
`Retry-After: 60`. The minute slides: reports from the minute before still
|
||||
count, fading out over the current one. An address that sends at most half the
|
||||
limit in any 60 seconds is never refused. Above that, it may be. The page
|
||||
sends one report a minute from each open tab. So the default of 60 refuses
|
||||
nothing from up to 30 tabs behind one address, however their reports bunch up.
|
||||
Every answer to a report carries the headers `X-RateLimit-Limit`,
|
||||
`X-RateLimit-Remaining` and `X-RateLimit-Reset`.
|
||||
- Rate limit. Each client address may send `REPORTS_PER_MINUTE` reports a
|
||||
minute. [Trusted proxies](#trusted-proxies) says how that address is found.
|
||||
Past that it gets 429 with `Retry-After: 60`. The minute slides: reports from
|
||||
the minute before still count, fading out over the current one. An address
|
||||
that sends at most half the limit in any 60 seconds is never refused. Above
|
||||
that, it may be. The page sends one report a minute from each open tab. So the
|
||||
default of 60 refuses nothing from up to 30 tabs behind one address, however
|
||||
their reports bunch up. Every answer the route gives carries the headers
|
||||
`X-RateLimit-Limit`, `X-RateLimit-Remaining` and `X-RateLimit-Reset`.
|
||||
- Size cap. The report files in `DATA_DIR` may total at most
|
||||
`DATA_DIR_MAX_BYTES`, counting the files already there at start. Reports
|
||||
waiting in memory count at their uncompressed size until they are written.
|
||||
@@ -484,8 +497,8 @@ Only those two routes are recorded because the labels are the request's path and
|
||||
method. Recording every request would let clients add labels without bound.
|
||||
|
||||
`/metrics` has its own rate limit, so that its password cannot be guessed
|
||||
quickly. Each client address, resolved through `TRUSTED_PROXIES`, may make 60
|
||||
requests to it a minute, whatever their credentials. Past that it gets 429 with
|
||||
quickly. Each client address may make 60 requests to it a minute, whatever its
|
||||
credentials. The address is found as for reports. Past that it gets 429 with
|
||||
`Retry-After: 60`, and its credentials are not checked. The minute slides as it
|
||||
does for reports, so a scraper polling every 2 seconds or less often is never
|
||||
refused.
|
||||
@@ -549,10 +562,10 @@ The page is built with Vite and Tailwind CSS v4. All its code is in
|
||||
`netwatch-server` is a Go HTTP server wired with `fx`. Its packages are in
|
||||
`internal/`:
|
||||
|
||||
- `cli` runs the command line: the `check-cidr` and `prepare-data-dir`
|
||||
subcommands, which `bin/entrypoint.sh` runs before the server starts, and,
|
||||
given no arguments, the server. Any other arguments get a usage message and
|
||||
exit code 2.
|
||||
- `cli` runs the command line. Given no arguments, it runs the server. It also
|
||||
runs the `check-cidr` and `prepare-data-dir` subcommands, which
|
||||
`bin/entrypoint.sh` runs before the server starts. Any other arguments get a
|
||||
usage message and exit code 2.
|
||||
- `config` reads the settings and refuses values the server cannot use.
|
||||
- `server` builds the router and the HTTP server and registers the routes.
|
||||
- `middleware` holds the request log, the security headers, CORS, the body
|
||||
@@ -582,8 +595,8 @@ except `script/bootstrap`.
|
||||
|
||||
- `script/bootstrap`: install the dependencies. It uses the `node` on `PATH`
|
||||
when that node has the pinned major version and is new enough for the page's
|
||||
dependencies. Otherwise it installs the pinned node through nvm, fetched as a
|
||||
hash-verified archive. It uses the `yarn` on `PATH` only when it is exactly
|
||||
dependencies. Otherwise it installs nvm from a hash-verified archive, and the
|
||||
pinned node through it. It uses the `yarn` on `PATH` only when it is exactly
|
||||
the pinned version. Otherwise it installs that version through corepack, or
|
||||
through npm where there is no corepack. Then it runs
|
||||
`yarn install --frozen-lockfile`. It uses the `go` on `PATH` only when it is
|
||||
@@ -612,8 +625,8 @@ except `script/bootstrap`.
|
||||
- `script/lint`: build the `lint` phase of `Dockerfile`, writing no image. The
|
||||
phase runs golangci-lint over the Go packages, and eslint in its
|
||||
`frontend-lint` stage.
|
||||
- `script/fmt`: format all files (writes): prettier over the JavaScript, CSS,
|
||||
HTML and Markdown, then gofmt over `cmd/` and `internal/`. It runs on the
|
||||
- `script/fmt`: run the formatters, which write: prettier over the JavaScript,
|
||||
CSS, HTML and Markdown, then gofmt over `cmd/` and `internal/`. It runs on the
|
||||
host, with `~/.local/bin` put on `PATH`.
|
||||
- `script/fmt-check`: check the formatting (read-only), with the same scope as
|
||||
`script/fmt`.
|
||||
@@ -662,12 +675,13 @@ The yarn packages are installed again on every build.
|
||||
call the functions it exports; no browser is involved.
|
||||
|
||||
`make frontend-viewport-test` builds the page and checks its layout at several
|
||||
window widths in a headless Chrome, served by the same nginx image and
|
||||
configuration as the Docker image. It tests each breakpoint declared in
|
||||
`web/src/styles.css`, today the one at `max-width: 768px`, and the widths one
|
||||
pixel either side of it. It also tests four fixed widths: 320px, 1280px and two
|
||||
phone landscape sizes. [test/viewport/README.md](test/viewport/README.md) says
|
||||
what it checks and what it cannot.
|
||||
window widths in a headless Chrome. The page is served with the nginx image and
|
||||
the `configs/nginx.conf` that the Docker image ships. It tests each breakpoint
|
||||
declared in `web/src/styles.css`, today the one at `max-width: 768px`, and the
|
||||
widths one pixel either side of it. It also tests four fixed widths: 320px,
|
||||
1280px and two phone landscape sizes.
|
||||
[test/viewport/README.md](test/viewport/README.md) says what it checks and what
|
||||
it cannot.
|
||||
|
||||
## TODO
|
||||
|
||||
|
||||
Reference in New Issue
Block a user