From 2513303b3c54fb1df9ee5f48f39cf08f92d48811 Mon Sep 17 00:00:00 2001 From: clawbot Date: Mon, 26 Oct 2026 03:27:21 +0000 Subject: [PATCH] README.md rewritten sentence by sentence for a newcomer Every sentence is under about 30 words, with one idea each. The page, its targets and what a figure measures come first, then reporting, deployment, the settings reference, the design, and development last. No tables; every identifier in backticks. The code on next decides what is true. Model: fable-5-1 --- README.md | 1122 ++++++++++++++++++++++++++++------------------------- 1 file changed, 587 insertions(+), 535 deletions(-) diff --git a/README.md b/README.md index 6592f5b..7bce60f 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,17 @@ -NetWatch is an MIT-licensed JavaScript single-page application by -[@sneak](https://sneak.berlin) that provides real-time network latency -monitoring to common internet hosts, displayed with color-coded figures and -sparkline graphs, served from a static bucket or from its Docker image, where a -small Go backend stores the measurements the page reports. +NetWatch is an MIT-licensed single-page web application by +[@sneak](https://sneak.berlin) that shows the latency from your browser to 26 +well-known internet hosts. The page is JavaScript and does all its measuring in +the browser. A small Go backend, `netwatch-server`, stores the measurements the +page reports. The page and the backend ship together in one Docker image. + +## Rationale + +When a network misbehaves, it helps to see at a glance which hosts answer and +how fast. NetWatch is one page with a latency figure and a small graph for each +host, refreshed every few seconds. The page measures from the browser, so it +shows the path your own traffic takes. It can be served from any static file +host. The backend only stores what the page reports. Without it, the page works +the same and nothing is stored. ## Getting Started @@ -10,10 +19,10 @@ small Go backend stores the measurements the page reports. # Install the dependencies and the git pre-commit hook make setup -# Run the page on the Vite dev server +# Serve the page from the Vite dev server make dev -# Build netwatch-server and run it on port 8080 +# Build the page and netwatch-server, then run the server on port 8080 make run # Run the tests, both linters and the format check @@ -22,593 +31,636 @@ make check # Build the page into dist/ and netwatch-server into the repo root make build -# Build the image and run it +# Build the Docker image and run it make docker docker run -p 8080:8080 netwatch ``` -`make check` and `make docker` need Docker. `make dev` passes `/api` to -`http://127.0.0.1:8080`, where `make run` starts `netwatch-server` with its -defaults, so the reports the page posts are stored in `data/reports`. +`make check` and `make docker` need Docker. `make dev` serves the page at the +address Vite prints and passes `/api` requests to `http://127.0.0.1:8080`. +`make run` starts `netwatch-server` there with its defaults, so the page's +reports are stored in `data/reports`. -## Entrypoints +## The page -This repository adheres to the -[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) -standard: normalized scripts in `script/` are the entrypoints for the -development workflow, and the Makefile targets are thin shims that call them. -The scripts cover both the frontend and the Go backend, so `make check` fails if -either one is broken. We provide: +The page shows one row per host. A row has the host's name, its URL, its latest +latency in milliseconds and a graph of its last 100 checks. Below the figure, +the row shows the minimum, median, average and maximum of the host's answers in +that history. A round of checks runs every 3 seconds by default. The header +shows the interval, how long the history spans, and whether the page is running +or paused. -- `script/bootstrap` — install all dependencies (the pinned node via nvm unless - the `node` on `PATH` has the pinned major version and is new enough for the - frontend's dependencies, yarn `YARN_VERSION` via corepack, or npm where there - is no corepack, unless the `yarn` on `PATH` is exactly that version, - `yarn install --frozen-lockfile`, Go `GO_VERSION` unless the `go` on `PATH` is - exactly that version, and the Go modules), linking what it installs itself - into `~/.local/bin`, which has to be on `PATH`. It installs no Go linter and - not Docker: `make test` and `make lint` run in Docker -- `script/setup` — make a fresh clone ready for development: bootstrap plus the - git pre-commit hook -- `script/dev` — run the Vite dev server, which proxies `/api` to a locally - running `netwatch-server` -- `script/build` — build the frontend for production into `dist/`, then the - static `netwatch-server` binary into the repo root with its version stamped - in: `VERSION` from the environment, or when that is unset or empty, - `git describe --tags --always --dirty` inside a git checkout, as - `script/docker` does, then `dev` -- `script/run` — build with `script/build`, then run `netwatch-server` with its - defaults (see [Settings](#settings)) -- `script/clean` — remove what `script/build` writes -- `script/projectname` — print the project name (used for the Docker image tag) -- `script/yarn` — run yarn with the given arguments under the node - `script/bootstrap` would use: with `~/.local/bin` first on `PATH`, the first - `node` on that `PATH` when it has the pinned major version and is new enough - for the frontend's dependencies, and otherwise the pinned node in nvm. Every - script that runs yarn on the host runs it through this one, except - `script/bootstrap`, which picks its own node and yarn -- `script/test` — build the `test` phase of `Dockerfile`, writing no image: the - frontend's unit tests in its `frontend-test` stage, and the backend's Go tests - with the race detector and coverage. The frontend's production build is left - to the image build, in the `frontend-build` stage -- `script/lint` — build the `lint` phase of `Dockerfile`, writing no image: - eslint in its `frontend-lint` stage, and golangci-lint over the Go packages -- `script/fmt` — format all files (writes): prettier over the JavaScript, CSS, - HTML and Markdown, then gofmt over `cmd/` and `internal/`. It runs on the - host, as `script/fmt-check` does, with `~/.local/bin` put on `PATH` -- `script/fmt-check` — check formatting (read-only): prettier, then gofmt -- `script/check` — run test, lint, and fmt-check -- `script/add-dependency` — add a frontend package, or move one to another - version: `make add-dependency PACKAGE=@` runs `yarn add --dev`, - which changes `package.json` and `yarn.lock` together, then - `yarn install --frozen-lockfile` -- `script/tidy` — run `go mod tidy`: to add a Go module, import it and run - `make tidy`; to move one to another version, edit its `require` line in - `go.mod`, then run `make tidy` -- `script/frontend-test` — run the unit tests in `test/unit/` on the host with - Node's built-in test runner, through the `test` script in `package.json`, and - if any fails, run them again listing every test, and fail; then the production - build. Each test run has a 90-second timeout. `make test` runs the same tests - in Docker, without the production build -- `script/frontend-fmt` — format the JavaScript, CSS, HTML and Markdown with - prettier (writes) -- `script/frontend-fmt-check` — check prettier formatting (read-only) -- `script/frontend-viewport-test` — responsive-layout verification of the built - frontend in a containerised headless Chrome (see - [test/viewport/README.md](test/viewport/README.md)). Before the browser - starts, it checks that the page, and a path that falls back to it, are served - with `Cache-Control: no-cache` and `Content-Security-Policy`. Not part of - `script/check`: it takes minutes. -- `script/docker` — build the image from `Dockerfile`, tagged `netwatch` via - `script/projectname` -- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`, - then builds the image as `script/docker` does -- `script/precommit` — run by the git pre-commit hook; runs `script/check` -- `script/install-precommit` — install the git pre-commit hook +### Targets -`script/test`, `script/lint`, `script/docker` and `script/cibuild` each pass a -new `CACHEBUST` value, so every step in `Dockerfile` that lints, tests or -compiles runs again even when nothing has changed. Docker keeps the system -packages and the Go modules from earlier builds until the base image, the step -that installs them, or `go.mod` and `go.sum` change. The `frontend-build` stage -is the exception: it installs git again on every image build, because it starts -from the `frontend-test` stage, whose steps run again on every build. Go's build -cache and golangci-lint's are kept as well, and each reuses a result only when -everything it was made from is unchanged. Both `go test` runs carry `-count=1`, -so no test result is reused. The yarn packages are installed again on every -build. +The page checks 26 hosts on the internet, called the WAN hosts: -## Responsive layout +- `datavi.be`, pinned to the top at start +- the Anthropic API and the OpenAI API +- the AWS Console, the Google Cloud Console and Microsoft Azure +- Cloudflare, Fastly, Akamai, Google, GitHub and Backblaze B2 +- 8 S3 regional endpoints: Cape Town, London, Bahrain, Tokyo, Singapore, Sydney, + Oregon and São Paulo +- 6 Hetzner speed-test servers: Nuremberg, Falkenstein, Helsinki, Ashburn, + Hillsboro and Singapore -The narrow-viewport layout lives in the `max-width: 768px` media block in -`web/src/styles.css`. It is verified automatically by -`make frontend-viewport-test`, which drives a digest-pinned headless Chrome -against the built `dist/` and asserts on computed layout at widths derived from -that CSS — on every breakpoint it declares and one pixel either side of it, plus -a 320px floor, a desktop baseline and two landscape sizes. See -[test/viewport/README.md](test/viewport/README.md) for what it covers and what -it genuinely cannot. +Two local targets follow under the heading "Local Network". They are left out of +the summary and the health box. -## Rationale +- `Local CPE` is the cable modem at `http://192.168.100.1`. Its row is always + shown. +- `Local Gateway` is found when the page loads. The page sends a request to + `http://192.168.1.1`, `http://192.168.0.1`, `http://192.168.8.1` and + `http://10.0.0.1`. The first to answer within 1.5 seconds becomes this target. + If none answers, there is no row for it. -When debugging network issues, it's useful to have a persistent at-a-glance view -of latency and reachability to multiple well-known internet endpoints. NetWatch -provides this as a single page that does all its measuring in the browser, so it -can be served from anywhere static files are served. The backend in its Docker -image only stores the measurements the page reports; without it, the page works -the same and nothing is stored. +Browsers block a page served from a public address from reaching private +addresses (Private Network Access). So the local targets answer only when the +page is served from `localhost` or from a private address, and only on a network +that has them. -## Design +The host list is written in `web/src/main.js`. There is no setting for it. -The page's source is in `web/`, the Go backend's in `cmd/` and `internal/`, and -nginx's configuration in `configs/`. +### What a figure measures -The page is built with Vite and Tailwind CSS v4. Its code is all in -`web/src/main.js`, with a class-based architecture: +Each check is a GET request to the target's URL. The figure is the time from +sending the request until the response headers arrive, in milliseconds. Unlike +ping, it includes the time the server takes to answer. When the browser opens a +new connection, it also includes the DNS lookup, the connection setup and the +TLS handshake. Chrome opens a new connection for every check of the Hetzner +servers, and for more targets at longer intervals. -- **`CONFIG`**: Configuration object (update interval, timeouts, axis ticks, - etc.). The interval menu sets `updateInterval`, the one value the page writes - into `CONFIG`; the timeouts, the time the history spans and the x-axis ticks - are computed from it -- **`HostState`**: Per-host state management — history buffer, latency tracking, - status transitions -- **`AppState`**: Top-level state container — WAN hosts, local hosts, pause - state, aggregate stats -- **`SparklineRenderer`**: Canvas 2D sparkline drawing with fixed axes, - color-coded line segments, error regions, and DPR-aware scaling -- **UI functions**: `buildUI()` constructs the DOM, `updateHostRow()` / - `updateSummary()` / `updateHealthBox()` handle incremental updates -- **`tick()`**: Main loop — measures all hosts in parallel, pushing each host's - sample and redrawing its row as soon as its check ends, then redraws every - row, the summary and the health box once the last check ends. The first round, - after loading or an interval change, is discarded. The rows are sorted when - the last check ends in round 2, the first one kept, and in rounds 11, 21, 31 - and so on. When paused, pushes blank markers (no probes, no false outage) -- **`Reporter`**: Posts collected samples to the backend +Any answer counts as reachable, an error page included. The request is made in +the browser's `no-cors` mode, so the page cannot read the answer, only time it. -### Reporting +The browser follows redirects, and the figure then includes the request to the +new URL too. So a target whose home page redirects is checked at `/robots.txt` +on the same host, which answers without a redirect. The S3 regions are checked +with a HEAD request. Chrome keeps the connection to an S3 region open after a +HEAD request, but closes it after most GET requests. -Every `reportInterval` (default 60s) the page POSTs a JSON report to the -same-origin path `/api/v1/reports`: a random per-browser `clientId` kept in -`localStorage`, `geo` sent as null, and each host's unreported, non-paused -samples (timestamp, latency, error). A per-host high-water mark makes every -report a delta, so only new samples are sent; the mark advances only on a -delivered report, and while paused nothing is sent. Changing the interval clears -each host's history on the page; the samples in it not yet reported still go in -the next report. Delivery failure is quiet — one debug-log line per outage, -never blocking probing. The report-building step is a pure function of host -state. +The request is sent with `cache: 'no-store'`, so the browser's cache stays out +of the measurement. It carries no `Referer` header and no query string: the +Hetzner servers close the connection without an answer when either is present. -A sample in the history that no report has delivered is sent only as long as the -history holds it, so after a refused or failed report the next report carries -the samples the history still has. Samples an interval change cleared from the -history are kept apart, the last 100 per host, until they are delivered. The -history keeps each host's last 100 rounds: 100 seconds at the 1-second interval, -200 seconds at 2 seconds, 5 minutes at the default of 3 seconds and 100 minutes -at 60 seconds. At 1 second, one refused or failed report loses the oldest 20 -seconds of each host's samples, as the next delivered report comes 2 minutes -after the last. At 2 seconds and longer one loses nothing; samples are lost from -the third refused report in a row at 2 seconds, from the fifth at 3 seconds, and -later still at the longer intervals. +A check times out after 80% of the interval: 2.4 seconds at the default of 3 +seconds, 24 seconds at 30 seconds. A check that times out, or answers after that +limit, is recorded as a timeout. A check that fails with a network error is +recorded as unreachable. -No report is sent while the page is paused, but each paused round still adds an -entry to the history, so each report due during a pause counts as a refused one. -A pause longer than the history spans loses the samples in the history recorded -between the last delivered report and the pause: up to a minute's worth for each -host if the report before the pause was delivered. +The first round after the page loads is discarded, as is the first round after +an interval change. Its connections are all new, so its figures run high. -### Backend +The browser chooses between IPv4 and IPv6 for each target, as for any request. +The local targets are IPv4 addresses. -`netwatch-server` is a small Go HTTP server that stores the reports the page -posts. It keeps them in memory and writes them to `DATA_DIR` as zstd-compressed -files of JSON lines: every minute, whenever 10 MiB are waiting, and when it -stops. Its routes: +### Colors -- `POST /api/v1/reports` — takes a report, without credentials; each client - address may send a limited number a minute, and the report files are capped in - size, the oldest deleted first -- `GET /.well-known/healthcheck` — answers 200 with `"status":"ok"`, the - server's version and its uptime; while reports cannot be stored, as the last - write of a report file failed, it answers 503 with `"status":"error"` and the - same fields (see [Report storage](#report-storage)) -- `GET /metrics` — Prometheus metrics behind basic auth, only when - `METRICS_USERNAME` and `METRICS_PASSWORD` are set; each client address may - make a limited number of requests to it a minute +A latency figure and its line in the graph take one color: -It is an `fx`-wired Go application: `cmd/netwatch-server/` starts it, and its -packages are in `internal/`: +- green: under 50 ms +- lime: under 100 ms +- yellow: under 200 ms +- orange: under 500 ms +- red: 500 ms and over +- gray: no answer -- **`cli`**: Runs `netwatch-server` from its command line: the `check-cidr` and - `prepare-data-dir` subcommands, which `bin/entrypoint.sh` runs before it - starts the server, and, given no arguments, the server, with the `fx` wiring. - It refuses any other arguments with a usage message and exit code 2. -- **`config`**: Loads configuration from environment variables and config files - via Viper. -- **`handlers`**: HTTP request handlers for the API (health check, report - ingestion). -- **`reportbuf`**: The in-memory buffer of reports and the files it writes them - to. -- **`server`**: Chi-based HTTP server with middleware wiring and route - registration. -- **`healthcheck`**, **`middleware`**, **`logger`**, **`globals`**: Supporting - infrastructure. +### The graph -In the image, the `test` phase of `Dockerfile` tests it, the `builder` stage -builds it, and `bin/entrypoint.sh` runs it as user `netwatch` on -`127.0.0.1:8081`, behind nginx (see [Deployment](#deployment)). Outside the -image, `make run` builds it and runs it on port 8080. +The graph in each row plots the host's last 100 checks. At the default interval +that is 5 minutes; the labels on its time axis say how long. The vertical axis +runs from 0 to 1000 ms. A latency above 1000 ms is drawn at the top, and the +figure still shows its real value. The rounds in which a check failed are shaded +red. -#### Settings +### Health and summary -| Variable | Default | Description | -| ---------------------- | -------------------- | -------------------------------------------------------------------------------------------------------- | -| `BIND_ADDRESS` | empty | IP address to listen on; empty listens on every interface | -| `PORT` | `8080` | HTTP listen port | -| `DATA_DIR` | `./data/reports` | Directory for compressed reports | -| `DATA_DIR_MAX_BYTES` | `1073741824` (1 GiB) | Most bytes of report files kept in `DATA_DIR`, oldest deleted first; see [Report limits](#report-limits) | -| `DEBUG` | empty | Debug logging; any non-empty value, `false` and `0` included, turns it on | -| `TRUSTED_PROXIES` | RFC 1918 ranges | Comma-separated CIDRs whose `X-Forwarded-For` / `X-Real-IP` headers are trusted for client IP resolution | -| `REPORTS_PER_MINUTE` | `60` | Reports each client address may send a minute; see [Report limits](#report-limits) | -| `CORS_ALLOWED_ORIGINS` | empty | Comma-separated origins whose pages may call the API; see [CORS](#cors) | -| `METRICS_USERNAME` | empty | Basic auth user name for `/metrics`; see [Metrics](#metrics) | -| `METRICS_PASSWORD` | empty | Basic auth password for `/metrics`; see [Metrics](#metrics) | -| `SENTRY_DSN` | empty | DSN of the Sentry project to send errors to; see [Sentry](#sentry) | +Above the rows, a health box sums up the WAN hosts' latest results. A host +counts as failed when its last check timed out or found it unreachable. -`TRUSTED_PROXIES` defaults to `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`. A list -replaces the default; set and empty, the server trusts no proxy. Loopback is not -in the default, so a reverse proxy on the same host has to be named, as in -`TRUSTED_PROXIES=127.0.0.1/32`. A request whose direct peer is outside this set -has its forwarded headers ignored, and the direct peer is logged and -rate-limited instead. From a direct peer inside the set, the client address is -the right-most `X-Forwarded-For` entry that is not in the set: each proxy -appends the address it took the request from, so the entries further left were -written by the client and are not used. An entry written as an address with a -port, such as `198.51.100.7:5000` or `[2001:db8::1]:443`, is read as that -address. If the search from the right finds only addresses in the set, or meets -an entry that is not an address at all, such as `unknown`, the direct peer is -used. `X-Real-IP` is read only when `X-Forwarded-For` is missing or empty. The -container image does not use this default; see [Deployment](#deployment). +- `OFFLINE` (red): more than 10 hosts failed and at most 4 answered +- `DEGRADED` (orange): otherwise, more than 4 hosts failed +- `SLOW` (yellow): otherwise, more than 3 hosts took over 1000 ms +- `HEALTHY` (green): otherwise -A variable set to a value the server cannot use, such as `PORT=abc` or a -`BIND_ADDRESS` that is not an IP address, stops it from starting, with an error -naming the variable. An empty variable counts as unset, except -`TRUSTED_PROXIES`. No value of `DEBUG` stops the start. +Below the health box, a summary line covers the WAN hosts only: -#### Report storage +- `Reachable`: how many answered in the last round, out of 26 +- `Now`: the minimum, median, average and maximum of their latest figures +- the history's span, such as `5m`, with the minimum and maximum over the whole + history +- `Checks`: the number of rounds run since the page loaded or the interval + changed -Reports are written as `reports--.jsonl.zst` files in -`DATA_DIR`. The timestamp is in UTC to the millisecond, so the names sort by -time. The number starts at 1 when the server starts and goes up by one for each -file the server starts to write, so two files written in the same millisecond -still get different names. A failed write uses up its number and leaves a gap in -the numbers: its file, if it was created, is removed. The file stays, counted -toward `DATA_DIR_MAX_BYTES` from the next start, only if removing it fails too. -Files are created with `O_EXCL` to prevent overwrites. +When no WAN host answers, the page starts a recovery probe. Every half second it +checks 4 WAN hosts, picked at random when the probe starts, and gives up the +checks it started half a second before. As soon as one answers, a new round +starts at once. -The reports of a failed write stay in memory, and the next flush, within a -minute, tries again to write them. The server writes one file at a time, so a -file holds its reports in the order they arrived, those of a failed write -included. From the failed write until a write succeeds, `POST /api/v1/reports` -refuses every report with 500 and keeps nothing of it, and the health check -answers 503 with `"status":"error"`. The page sends the refused samples again in -its next report, as long as it still holds them: at the 1-second interval one -refused report already loses some of those in the history (see -[Reporting](#reporting)). Reports already answered 200 are lost only if the -server stops before a write succeeds; the final flush at stop tries them too, -and if it fails, the process exits non-zero. +### Row order -#### Report limits +The WAN rows are sorted by their latest latency, fastest first and unreachable +last. The rows are sorted when the second round ends, then every 10 rounds. The +pin button at the left of a row keeps the row at the top. Pinned rows are sorted +by name, and a line separates them from the rest. `datavi.be` is pinned at +start. -`POST /api/v1/reports` takes reports from anyone who can reach it, without -credentials, so it is bounded instead. Each refusal below answers with the same -`{"status":"error"}` body as any other error, except nginx's refusal of a body -over the limit in the image. +### Controls -- **JSON only.** A report must be sent with `Content-Type: application/json`, - parameters such as `charset` allowed, as the page sends it; any other is - refused with 415, and nothing of it is stored. A page on another origin can - send a `text/plain` POST, or a form, without a CORS preflight; refusing them - keeps any site from making its visitors' browsers store reports (see - [CORS](#cors)). -- **Body limit.** A report body may be at most 1 MiB, the backend's limit for - any request. Run on its own, the backend answers 413 before the route to a - request that declares a longer body. Without a declared length, the route - answers 413 only when the report it reads runs past 1 MiB; nothing after the - end of the report is read, so a longer body whose report ends sooner is - stored, and one that stops being valid JSON sooner is answered 400. In the - image, nginx refuses a body over 1 MiB first, with or without a declared - length, as its `client_max_body_size`, which `configs/nginx.conf` does not - set, is 1 MiB by default. nginx answers 413 with its own HTML error page, and - the backend never sees the request. -- **Rate limit.** Each client address, resolved through `TRUSTED_PROXIES`, may - send `REPORTS_PER_MINUTE` reports a minute; past that it gets 429 with - `Retry-After: 60`. The minute slides: reports from the minute before still - count, fading out over the current one, so an address is sure never to be - refused only while it sends at most half of `REPORTS_PER_MINUTE` in any 60 - seconds. The page sends one report a minute from each open tab, so the default - of 60 refuses nothing from up to 30 tabs behind one address, such as a - household or an office sharing it, however their reports bunch up. Report - responses also carry `X-RateLimit-Limit`, `X-RateLimit-Remaining` and - `X-RateLimit-Reset` headers. -- **Size cap.** The report files in `DATA_DIR` may total at most - `DATA_DIR_MAX_BYTES`, counting the files already there at start. Reports - waiting in memory count at their uncompressed size until they are written, - those of a failed write included; the part of its file written is removed. - When a report would take the total past the cap, the oldest report files are - deleted to make room, and each deletion is logged with the file's name and - size; a file still being written is never deleted. A report is refused with - 507, and nothing of it is stored, only when the reports waiting to be written - fill the cap on their own, and then no file is deleted. At start, report files - past the cap, as after lowering it, are deleted the same way. So the cap is - how much of the newest reports is kept: the default of 1 GiB is small enough - for any host; set it to the space you can give `DATA_DIR`. +- `Pause` stops the checks. The graphs keep scrolling and leave a gap for each + paused round; no outage is drawn. The figures turn gray until `Resume`. +- The `Interval` menu sets the time between rounds: 1, 2, 3, 5, 10, 15, 30 or 60 + seconds. Changing it clears the history on the page and starts a new round at + once. +- The pin button in each WAN row pins and unpins the row. +- Each host's URL is a link that opens in a new tab. +- The `Debug log` checkbox in the footer opens a panel with the page's last 1000 + log lines. Each line has a level: error, warning, notice, info or debug. +- The header shows the local time with its time zone, and the UTC time. The + footer shows the color legend and links to the commit the page was built from. -At the defaults, one address may send 60 reports of up to 1 MiB a minute, and a -report made up to compress badly takes about as much room in the report files as -its request body, so in about 17 minutes one address can fill the 1 GiB cap with -its own reports, every report stored before them deleted to make room. Lowering -`REPORTS_PER_MINUTE` or raising `DATA_DIR_MAX_BYTES` makes that take longer. +Each check that fails writes one line to the browser console with +`console.error`, and the same line to the debug log. The line gives the target's +name and URL, the time, what failed and how long the request took. The three +kinds of failure are a timeout, an answer over the time limit, and a network +error. For a network error the line quotes the error the browser gives the page, +such as `TypeError: Failed to fetch`, which does not say why. When a target +answers again after failed checks, one `console.info` line gives its latency and +how many checks in a row had failed. -#### CORS +### Browsers -The page calls the API from the origin it is served from, so by default the -server sends no CORS headers, and browsers let no other origin's pages call it. -To serve the page from elsewhere, list that origin in `CORS_ALLOWED_ORIGINS` -(for example `https://netwatch.example.com`); pages from a listed origin may -`GET` and `POST` with a `Content-Type` header. Each entry must be a plain -origin, `scheme://host` with an optional `:port`, as browsers send it: no path, -not even a trailing `/`, and no `*`. Any other entry stops the server from -starting, with an error naming `CORS_ALLOWED_ORIGINS`. +The page needs a browser with ES modules, the Fetch API, the Canvas API and CSS +custom properties. -#### Metrics +## Reporting -With both `METRICS_USERNAME` and `METRICS_PASSWORD` set, the server serves -Prometheus metrics at `GET /metrics` to requests with those as their basic auth -credentials, and answers any other with 401. For each request that reaches the -health check or `POST /api/v1/reports`, those the rate limit refuses included, -the metrics record its duration and response size, labelled with its path, -method and status; they also count those requests in progress, and include Go's -runtime and process metrics. No other request is recorded: not those to -`/metrics` itself, and not those answered before they reach either route, such -as a CORS preflight, or a request refused with 404 for a path no route has or -405 for a method its route does not take. Run on its own, the backend also -answers 413 before the route to a request declaring a body length over the 1 MiB -limit, which is not recorded. A body over the limit sent without a declared -length reaches the route and is recorded with the status the route answers, -which is 413 only when the report it reads runs past the limit: nothing after -the end of the report is read. In the image neither reaches the backend: nginx -answers both with 413 itself (see [Report limits](#report-limits)), and nothing -of them is recorded. Clients can make up any number of paths and methods, and -each would add labels to the metrics for as long as the server runs. +Every 60 seconds the page posts a report to `/api/v1/reports` on the origin it +was served from. The report is JSON with four fields: -`/metrics` is rate limited, so that its password cannot be guessed quickly: each -client address, resolved through `TRUSTED_PROXIES`, may make 60 requests to it a -minute, whatever their credentials. Past that it gets 429 with -`Retry-After: 60`, and its credentials are not checked. The minute slides as it -does for reports (see [Report limits](#report-limits)), so a scraper polling -every 2 seconds or less often is never refused. This allowance is apart from the -one for reports. +- `clientId`: a random id, made once per browser and kept in `localStorage`. + When the browser has no usable storage, a new id is made for the session. +- `geo`: always `null` +- `hosts`: one entry per host with new samples. The entry has the host's `name`, + `url` and `status`, and `history`, a list of samples. A sample has the check's + time `t` in Unix milliseconds, its `latency` (`null` for a failed check) and + its `error` (`"timeout"`, `"unreachable"` or `null`). +- `timestamp`: the time of the report -#### Sentry +Each report carries only the samples no earlier report has delivered. For each +host, the page remembers the time of the newest sample it delivered, and moves +that mark only when the server answers 2xx. Paused rounds are never sent. While +the page is paused, no report is sent at all. One report is in flight at a time. +A report with no answer after 30 seconds is abandoned, and its samples go in the +next report. A server that stored them but answered late therefore gets them +twice. -With `SENTRY_DSN` set, each panic in a handler is reported to that Sentry -project under the release `netwatch-server-` followed by the server's version. -On shutdown the server waits up to 2 seconds for Sentry to finish sending. +A failed delivery is quiet. The page writes one debug log line when deliveries +start failing and one when they recover. The checks go on. -### Monitoring targets +A sample waits for delivery only as long as the history holds it. The history +keeps each host's last 100 rounds: 5 minutes at the default interval of 3 +seconds. At 1 second it spans 100 seconds, at 2 seconds 200 seconds, and at 60 +seconds 100 minutes. Changing the interval clears the history, but the cleared +samples not yet delivered are kept apart, the last 100 per host, until they are +delivered. -- **26 WAN hosts**: datavi.be (pinned at start), Anthropic API, OpenAI API, AWS - Console, Google Cloud Console, Microsoft Azure, Cloudflare, Fastly CDN, - Akamai, Google, GitHub, B2, 8 S3 regional endpoints (Cape Town, London, - Bahrain, Tokyo, Singapore, Sydney, Oregon, São Paulo) and 6 Hetzner speed test - servers (Nuremberg, Falkenstein, Helsinki, Ashburn, Hillsboro, Singapore) -- **Local CPE**: Cable modem at 192.168.100.1 (always monitored) -- **Local Gateway**: Auto-detected on startup by probing common default gateway - addresses (192.168.1.1, 192.168.0.1, 192.168.8.1, 10.0.0.1); first responder - wins. Note: modern browsers enforce Private Network Access restrictions that - block public-origin pages from reaching RFC1918 addresses, so local targets - only work when NetWatch is served from localhost or a private address. +So at the 1-second interval, one refused or failed report loses the oldest 20 +seconds of each host's samples: the next delivered report comes 2 minutes after +the last. At 2 seconds and longer, one failed report loses nothing. Samples are +lost from the third failed report in a row at 2 seconds, from the fifth at 3 +seconds, and later still at longer intervals. -Local hosts are tracked separately from WAN stats. - -### Latency measurement - -Each check is a GET request to the target's URL as written, with -`mode: 'no-cors'`, `cache: 'no-store'` and `referrerPolicy: 'no-referrer'`. Its -figure is the time from sending the request until the answer's headers arrive, -timed with `performance.now()`. Unlike ping, this includes the time the server -takes to answer. It also includes the DNS lookup, connection setup and TLS -whenever the browser opens a new connection. Chrome opens one for every check of -the Hetzner speed-test servers, and for more targets at longer intervals. - -The S3 regions are checked with a HEAD request instead. Chrome keeps the -connection to an S3 region open after a HEAD request, but closes it after most -GET requests. The Hetzner speed-test servers do not answer a HEAD request. - -The browser follows every redirect, and the figure then includes the request to -the new URL too. So a target whose home page answers with a redirect is checked -at `/robots.txt` on the same host, which answers without one. An S3 region -answers it with 403, and any answer counts as reachable. - -The Hetzner speed-test servers close the connection without an answer when the -URL has a query string or the request has a `Referer` header. So no query string -is added, and no `Referer` header is sent, wherever the page is served from. -`no-store` keeps the browser's cache out of the measurement. Each check times -out after 80% of the refresh interval (24 seconds at 30 seconds) and is then -recorded as a timeout, so a round's checks have all finished before the next -round is due. When no WAN host answers, a recovery probe checks 4 WAN hosts, -picked at random when it starts, every half second, giving up the checks it -started half a second before. As soon as one answers, a new round starts at -once, as it does after an interval change. A round started early gives up the -last round's checks if they are still waiting, and that round records nothing -more, so rounds never overlap. The browser chooses between IPv4 and IPv6 for -each target, as for any request; the local targets are IPv4 addresses. - -Each recorded check that fails writes one line to the browser console with -`console.error`, and the same line to the debug log: the target's name and URL, -the time, whether it timed out, answered over the time limit or hit a network -error (with the error the browser gives the page, such as -`TypeError: Failed to fetch`, which does not say why), and how long the request -took. A target that answers after failed checks writes one `console.info` line -with its latency and how many checks in a row had failed. - -### Color coding - -| Latency | Color | -| ----------- | ------ | -| < 50ms | Green | -| < 100ms | Lime | -| < 200ms | Yellow | -| < 500ms | Orange | -| >= 500ms | Red | -| Unreachable | Gray | - -### Output structure - -``` -dist/ -├── index.html -└── assets/ - ├── index-*.css - └── index-*.js -``` - -## Features - -- A round of checks every 3 seconds by default; the interval menu sets 1, 2, 3, - 5, 10, 15, 30 or 60 seconds and clears the history -- Sparklines of each target's last 100 rounds: 300 seconds at 3 seconds -- The first round after loading or an interval change is discarded, as DNS and - TLS setup inflate its latencies -- Health indicator from the WAN hosts' latest results: OFFLINE (red) when more - than 10 fail and at most 4 answer, otherwise DEGRADED (orange) when more than - 4 fail, otherwise SLOW (yellow) when more than 3 take over 1000ms, otherwise - HEALTHY (green) -- Summary stats across WAN hosts only: how many answered, the min, median, - average and max of their latest latencies, the min and max over the whole - history, and the number of rounds run (`Checks`) -- Fixed chart axes: Y-axis 0–1000ms, higher latencies drawn at the top; X-axis - the time the history spans -- Color-coded latency figures and sparkline line segments -- WAN host rows sorted by latest latency, unreachable last; pinned rows stay on - top, in name order -- Play/pause: pause stops probes but history keeps scrolling (blank gaps, no - false outage) -- Debug log panel, behind a checkbox in the footer, with five levels (error, - warning, notice, info, debug) and the last 1000 lines -- Local and UTC clocks -- Clickable service URLs -- A footer link to the commit the page was built from -- Canvas-based sparkline rendering with devicePixelRatio scaling -- Zero runtime dependencies: all resources bundled into build artifacts +Each paused round still adds an entry to the history, so each report due during +a pause counts as a failed one. A pause longer than the history spans loses the +samples recorded between the last delivered report and the pause: up to a +minute's worth for each host. ## Deployment -`make build` writes the page to `dist/`, which any static file host (S3, GCS, -Cloudflare Pages, Vercel, Netlify, GitHub Pages) can serve; with no backend -there, its reports fail quietly and nothing is stored. Or run the Docker image -behind a reverse proxy. +### Static files -The Docker image, built from `Dockerfile` by `make docker`, is the whole service -in one container: nginx serves the built frontend and passes `/api/`, -`/.well-known/healthcheck` and `/metrics` to the Go backend, `netwatch-server`. -`bin/entrypoint.sh` starts it listening only inside the container, on -`127.0.0.1:8081`, with `TRUSTED_PROXIES=127.0.0.1/32`, so it takes the client -address nginx passes on and no other. The image: +`make build` writes the page to `dist/`: an `index.html` and an `assets/` +directory with one CSS file and one JavaScript file. Any static file host can +serve it. Without a backend at `/api/v1/reports` on that host, the reports fail +quietly and nothing is stored. -- Listens on port 8080 by default (override with `PORT` env var) -- Takes the client address from `X-Forwarded-For` only on requests from the - reverse proxies named in `TRUSTED_PROXIES`, by default from any address in - `10.0.0.0/8`, `172.16.0.0/12` or `192.168.0.0/16`. `bin/entrypoint.sh` checks - each entry with `netwatch-server check-cidr`, which parses it as the backend - parses its own `TRUSTED_PROXIES` -- Sends access logs to stdout -- Sends the page with `Cache-Control: no-cache`, so a browser that keeps a copy - checks with the server that it is current before using it again; only the - files under `/assets/`, whose names change whenever their contents do, are - cached as immutable -- Sends the security headers `REPO_POLICIES.md` requires on every response, as - `configs/security-headers.conf` sets them, in place of the backend's own -- Stores reports in `DATA_DIR`, `/data/reports` by default, on the `/data` - volume. Before the backend starts, the image creates `DATA_DIR` and gives it - and `/data` to user `netwatch` (uid 1000), which the backend runs as, with - `netwatch-server prepare-data-dir`, which acts on nothing outside `/data`, so - a host directory bind-mounted at `/data` ends up owned by uid 1000 -- Writes buffered reports to disk on `docker stop`, and exits non-zero if nginx - or the backend exits on its own, so the platform restarts it +### The Docker image -## Running under upaas +`make docker` builds the image `netwatch` from `Dockerfile`. It is the whole +service in one container. nginx serves the page and passes `/api/`, +`/.well-known/healthcheck` and `/metrics` to `netwatch-server`, which runs in +the same container. The backend listens on `127.0.0.1:8081` only, as user +`netwatch` (uid 1000). `bin/entrypoint.sh` starts both and watches them. -What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs: +The image: -- **Port:** container port `8080`. -- **Volume:** container path `/data`; the reports are kept in `/data/reports`. -- **Environment variables:** none is required. An empty one counts as unset, - except `TRUSTED_PROXIES`, and one set to a value netwatch cannot use stops the - container at start, with the reason in its log. - - `PORT`, default `8080`: the container port, from 1 to 65535. `8081` cannot - be used: the backend listens on it inside the container - - `REPORTS_PER_MINUTE`, default `60`: reports each client address may send a - minute - - `DATA_DIR_MAX_BYTES`, default `1073741824` (1 GiB): the most room the - report files may take; the oldest are deleted to stay under it - - `CORS_ALLOWED_ORIGINS`, default empty: other origins whose pages may call - the API - - `DEBUG`, default empty: debug logging; any non-empty value, `false` and - `0` included, turns it on - - `DATA_DIR`, default `/data/reports`: the directory the reports are kept - in: `/data` or a path below it, with no `.` or `..` part and no extra `/`. - The container also stops if the path goes through a symbolic link that - leads out of `/data` or is written as a full path - - `TRUSTED_PROXIES`, default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`: the - addresses the reverse proxy in front of the container connects from, as IP - addresses or CIDRs separated by commas. nginx takes the client address - from `X-Forwarded-For` only on a request from one of them, and the rate - limit counts that address. A list replaces the default. Set and empty, - `X-Forwarded-For` is ignored and every client behind the proxy shares the - proxy's one allowance of `REPORTS_PER_MINUTE`. Any client that connects - from one of these addresses can write its own `X-Forwarded-For`, and - through a port Docker publishes, every client may connect from the Docker - network's gateway, such as `172.17.0.1`, which the default includes; where - that matters, name only the addresses nothing but the proxy connects from. - - `METRICS_USERNAME` and `METRICS_PASSWORD`, default empty: with both set, - the backend records Prometheus metrics of its requests and serves them at - `/metrics` on the container port, to requests with this user name and - password as their basic auth credentials. With neither set, there are no - metrics and `/metrics` is not found. One set without the other, or a user - name containing `:`, stops the container - - `SENTRY_DSN`, default empty: set to a Sentry project's DSN, the backend - sends its errors to that Sentry project: each request whose handling - crashes, which still gets a 500 response. A value Sentry does not accept - stops the container. Empty, the backend sends nothing to Sentry -- **Health check:** the image's `HEALTHCHECK` requests - `/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless - both nginx and the backend answer. It also fails while reports cannot be - stored, as when the volume is full or `DATA_DIR` cannot be written: the health - check answers 503 from a failed write of a report file until a write succeeds, - and the backend tries the write again every minute. Three failed checks in a - row make the container `unhealthy`, and one that passes makes it `healthy` - again. upaas reads the container's health once, 60 seconds after a deploy, and - fails the deploy unless it is `healthy`. A container that turns `unhealthy` - after that is neither restarted nor reported by upaas, whose `unless-stopped` - restart policy ignores health; only `docker ps` and `docker inspect` show it. - The container also stops when either process exits. +- listens on port `8080`; the `PORT` variable changes it (see + [Settings](#settings)) +- stores the report files in `/data/reports`, on the `/data` volume. Before the + backend starts, `netwatch-server prepare-data-dir` creates that directory and + gives `/data` and everything in it to user `netwatch`. A host directory + mounted at `/data` therefore ends up owned by uid 1000. The command changes + nothing outside `/data`, whatever links it meets there. +- takes the client address from `X-Forwarded-For` only on a request from a + reverse proxy named in `TRUSTED_PROXIES`. nginx resolves the address and + passes that one address to the backend. The backend is started with + `TRUSTED_PROXIES=127.0.0.1/32`, so it trusts nginx alone. +- writes nginx's access log to stdout, and the backend's log as JSON to stdout +- sends the page with `Cache-Control: no-cache`, so a browser that keeps a copy + asks the server whether it is current before using it again. The files under + `/assets/` are cached as immutable; their names change whenever their contents + do. +- sends the security headers in `configs/security-headers.conf` on every + response, in place of the backend's own +- refuses a request body over 1 MiB with nginx's own 413 error page; the backend + never sees such a request +- requests `/.well-known/healthcheck` through nginx every 30 seconds, as its + `HEALTHCHECK`. The check fails unless both nginx and the backend answer, and + while report files cannot be written. Three failures in a row make the + container `unhealthy`; one pass makes it `healthy` again. +- on `docker stop`, writes the waiting reports to disk and exits 0. If nginx or + the backend exits on its own, the other is stopped and the container exits + non-zero, so the platform restarts it. -## Browser Compatibility +### Running under upaas -Requires a modern browser with ES modules, Fetch API, Canvas API, and CSS custom -properties. +The [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs: -## Limitations +- Port: container port `8080`. +- Volume: container path `/data`. The reports are kept in `/data/reports`. +- Environment variables: none is required. Every setting in + [Settings](#settings) applies, with three differences in the image: + - `PORT` is nginx's public port, from 1 to 65535 and not `8081`, where the + backend listens inside the container. + - `DATA_DIR` must be `/data` or a path below it, written in full, with no + `.` or `..` part and no extra `/`. A path through a symbolic link that + leads out of `/data` is refused too. + - `TRUSTED_PROXIES` names the proxies nginx trusts, as IP addresses or CIDRs + separated by commas. The backend's own setting is fixed to `127.0.0.1/32`. +- Health: the image's `HEALTHCHECK`, described above. -- **CORS**: The checks are cross-origin requests in `no-cors` mode, so the page - cannot read the answer, only time it: any answer counts as reachable, an error - page included. -- **Local targets**: The cable modem at 192.168.100.1 and the detected gateway - answer only on a network that has them, and only when NetWatch is served from - localhost or a private address (see Monitoring targets). -- **Network conditions**: Measurements reflect browser-to-endpoint latency, - which includes your local network, ISP, and internet routing. +A variable set to a value the container cannot use stops it at start, with the +reason in its log. + +## Settings + +`netwatch-server` reads its settings from environment variables. It also reads a +`.env` file in the directory it starts in, and a YAML file named +`netwatch-server.yaml` in `/etc/netwatch-server/` or +`~/.config/netwatch-server/`, when one exists. A variable in the environment +wins over a file. + +A variable set to a value the server cannot use stops it from starting, with an +error naming the variable. An empty variable counts as unset, except +`TRUSTED_PROXIES`. + +- `BIND_ADDRESS`, default empty: the IP address to listen on. Empty listens on + every interface. The image sets it to `127.0.0.1`. +- `PORT`, default `8080`: the port to listen on, from 1 to 65535. In the image + this is nginx's port; the backend listens on `8081`. +- `DATA_DIR`, default `./data/reports` (`/data/reports` in the image): the + directory for the report files. The server creates it if it is missing, and + stops at start if it cannot create a file in it. +- `DATA_DIR_MAX_BYTES`, default `1073741824` (1 GiB): the most bytes of report + files kept in `DATA_DIR`, oldest deleted first. A positive whole number. See + [Report limits](#report-limits). +- `REPORTS_PER_MINUTE`, default `60`: how many reports each client address may + send a minute. A positive whole number. See [Report limits](#report-limits). +- `TRUSTED_PROXIES`, default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`: the + reverse proxies whose `X-Forwarded-For` and `X-Real-IP` headers are trusted, + as CIDRs separated by commas. See [Trusted proxies](#trusted-proxies). +- `CORS_ALLOWED_ORIGINS`, default empty: other origins whose pages may call the + API, separated by commas. See [CORS](#cors). +- `METRICS_USERNAME` and `METRICS_PASSWORD`, default empty: the basic auth + credentials for `/metrics`. See [Metrics](#metrics). +- `SENTRY_DSN`, default empty: the DSN of a Sentry project to report crashes to. + See [Sentry](#sentry). +- `DEBUG`, default empty: any non-empty value turns debug logging on, `false` + and `0` included. No value of it stops the start. + +### Trusted proxies + +Each `TRUSTED_PROXIES` entry must be a CIDR, such as `127.0.0.1/32`; a lone +address is refused. A list replaces the default. Set and empty, the server +trusts no proxy. Loopback is not in the default, so a reverse proxy on the same +host has to be named. + +A request from a peer outside this set has its forwarded headers ignored. The +peer's own address is logged and rate limited. + +A request from a peer inside the set takes its client address from +`X-Forwarded-For`. Each proxy appends the address it took the request from, so +the entries on the right were written by trusted proxies and the ones further +left by the client. The client address is the right-most entry that is not in +the set. An entry written with a port, such as `198.51.100.7:5000` or +`[2001:db8::1]:443`, is read as that address. The peer's address is used when +every entry is in the set. It is also used when the search from the right meets +an entry that is not an address at all, such as `unknown`. `X-Real-IP` is read +only when `X-Forwarded-For` is missing or empty. + +### Routes + +- `POST /api/v1/reports` takes a report from the page. It needs no credentials; + [Report limits](#report-limits) bound it instead. +- `GET /.well-known/healthcheck` answers 200 with a JSON object: `appname`, + `version`, `now`, `uptime_seconds`, `uptime_human` and `status`, which is + `"ok"`. While report files cannot be written, it answers 503 with the same + fields and `"status":"error"`. +- `GET /metrics` serves Prometheus metrics, only when `METRICS_USERNAME` and + `METRICS_PASSWORD` are set. See [Metrics](#metrics). + +Every request body is capped at 1 MiB, and every request at 60 seconds. + +### Report storage + +The server keeps the reports it takes in memory and writes them to `DATA_DIR` as +files of JSON lines, compressed with zstd. It writes a file every minute, +whenever 10 MiB of reports are waiting, and when it stops. + +The files are named `reports--.jsonl.zst`. The timestamp is +UTC to the millisecond, so the names sort by time. The number starts at 1 when +the server starts, and goes up by one for each file the server starts to write. +So two files started in the same millisecond still get different names. A file +is created with `O_EXCL`, so it never overwrites another. + +A write can fail, as when the disk is full. The reports of a failed write stay +in memory, and the next write, within a minute, tries them again. One file is +written at a time, so a file holds its reports in the order they arrived, those +of a failed write included. A failed write uses up its number. Its file, if it +was created, is removed; if removing it fails too, the file stays and counts +toward `DATA_DIR_MAX_BYTES` from the next start. + +From a failed write until a write succeeds, `POST /api/v1/reports` refuses every +report with 500 and keeps nothing of it, and the health check answers 503. The +page sends the refused samples again in its next report, as long as it still +holds them (see [Reporting](#reporting)). Reports already answered 200 are lost +only if the server stops before a write succeeds. The final write at stop tries +them too, and if it fails, the process exits non-zero. + +### Report limits + +`POST /api/v1/reports` takes reports from anyone who can reach it, so it is +bounded in four ways. Each refusal answers with the body `{"status":"error"}`. + +- JSON only. A report must be sent with `Content-Type: application/json`, with + or without parameters such as `charset`, as the page sends it. Any other + content type is refused with 415, and nothing of it is stored. A page on + another origin can send a `text/plain` POST, or a form, without a CORS + preflight. Refusing those keeps any site from making its visitors' browsers + store reports (see [CORS](#cors)). +- Body limit. A report body may be at most 1 MiB, the backend's limit for any + request. A request that declares a longer body is answered 413 before it + reaches the route. A body sent without a declared length is read up to 1 MiB. + The route answers 413 only when the report runs past that; nothing after the + end of the report is read. A body that stops being valid JSON sooner is + answered 400. In the image, nginx refuses a body over 1 MiB itself, with its + own HTML error page, and the backend never sees the request. +- Rate limit. Each client address, resolved through `TRUSTED_PROXIES`, may send + `REPORTS_PER_MINUTE` reports a minute. Past that it gets 429 with + `Retry-After: 60`. The minute slides: reports from the minute before still + count, fading out over the current one. An address is sure never to be refused + only while it sends at most half the limit in any 60 seconds. The page sends + one report a minute from each open tab. So the default of 60 refuses nothing + from up to 30 tabs behind one address, however their reports bunch up. Every + answer to a report carries the headers `X-RateLimit-Limit`, + `X-RateLimit-Remaining` and `X-RateLimit-Reset`. +- Size cap. The report files in `DATA_DIR` may total at most + `DATA_DIR_MAX_BYTES`, counting the files already there at start. Reports + waiting in memory count at their uncompressed size until they are written. + When a report would take the total past the cap, the oldest report files are + deleted to make room, and each deletion is logged with the file's name and + size. A file still being written is never deleted. A report is refused with + 507 only when the reports waiting to be written fill the cap on their own; + then no file is deleted. At start, report files past the cap are deleted the + same way, as after lowering it. + +So the cap decides how much of the newest reports is kept. The default of 1 GiB +fits on any host; set it to the space you can give `DATA_DIR`. At the defaults, +one address may send 60 reports of up to 1 MiB a minute. A report made up to +compress badly takes about as much room in the files as in its request body. So +in about 17 minutes one address can fill the 1 GiB cap with its own reports, and +every report stored before them is deleted to make room. A lower +`REPORTS_PER_MINUTE` or a higher `DATA_DIR_MAX_BYTES` makes that take longer. + +### CORS + +The page calls the API from the origin it is served from. By default the server +sends no CORS headers, so browsers let no other origin's pages call it. To serve +the page from elsewhere, list that origin in `CORS_ALLOWED_ORIGINS`, for example +`https://netwatch.example.com`. Pages from a listed origin may `GET` and `POST` +with a `Content-Type` header. + +Each entry must be a plain origin as browsers send it: `scheme://host`, with an +optional `:port`, no path, no trailing `/` and no `*`. Any other entry stops the +server from starting. + +### Metrics + +With both `METRICS_USERNAME` and `METRICS_PASSWORD` set, the server serves +Prometheus metrics at `GET /metrics`. A request with those as its basic auth +credentials gets them; any other gets 401. With neither set, there are no +metrics and `/metrics` answers 404. One set without the other stops the start, +and so does a user name containing `:`. + +The metrics record each request that reaches the health check or +`POST /api/v1/reports`. For each, they record its duration and response size, +labelled with its path, method and status, and they count such requests in +progress. Requests the rate limit refuses are included. Go's runtime and process +metrics are included too. No other request is recorded: + +- not a request to `/metrics` itself +- not a CORS preflight +- not a 404 for an unknown path, or a 405 for the wrong method +- not a 413 answered before the route + +Only those two routes are recorded because the labels are the request's path and +method. Recording every request would let clients add labels without bound. + +`/metrics` has its own rate limit, so that its password cannot be guessed +quickly. Each client address, resolved through `TRUSTED_PROXIES`, may make 60 +requests to it a minute, whatever their credentials. Past that it gets 429 with +`Retry-After: 60`, and its credentials are not checked. The minute slides as it +does for reports, so a scraper polling every 2 seconds or less often is never +refused. + +### Sentry + +With `SENTRY_DSN` set, each request whose handling crashes is reported to that +Sentry project, under the release `netwatch-server-` followed by the server's +version. The request still gets a 500 answer. A DSN Sentry does not accept stops +the start. On shutdown the server waits up to 2 seconds for Sentry to finish +sending. Empty, the server sends nothing to Sentry. + +### Logging + +The backend logs JSON to stdout, or plain text when stdout is a terminal. Each +line names the source location it was written from. Each request is logged with +its method, URL, status, client address and duration, among other fields. Every +value taken from a request is cut to 128 bytes before it is logged. + +## Design + +The repository holds the page, the backend and the image: + +- `web/`: the page's source. `index.html` loads `src/main.js` and + `src/styles.css`. +- `cmd/netwatch-server/` and `internal/`: the Go backend. `cmd/` holds only + `main`, which calls into `internal/cli`. +- `configs/`: nginx's configuration for the image, `nginx.conf` and + `security-headers.conf`. +- `bin/entrypoint.sh`: the image's entrypoint. +- `Dockerfile`: the lint and test phases, the two build stages and the image. +- `script/`: the development commands; `Makefile` targets call them. +- `test/unit/`: the page's unit tests. `test/viewport/`: the layout test. + +### The page + +The page is built with Vite and Tailwind CSS v4. All its code is in +`web/src/main.js`: + +- `CONFIG` holds the timing and display constants. The interval menu sets its + `updateInterval`; the check timeout, the history span and the time axis are + computed from it. +- `WAN_HOSTS` is the list of WAN targets, with the URL and method each is + checked with. +- `HostState` is one host's history, latest latency and status. +- `AppState` holds the WAN hosts, the local hosts, the pause state and the round + count, and computes the summary and the health. +- `measureLatency` runs one check. +- `tick` runs one round. It checks every host in parallel and draws each row as + soon as its check ends. When the last check ends, it redraws every row, the + summary and the health box, and sorts the rows when a sort is due. +- `SparklineRenderer` draws a graph on a canvas, scaled to the device's pixel + ratio. +- `buildUI` builds the page; `updateHostRow`, `updateSummary` and + `updateHealthBox` redraw its parts. +- `Reporter` builds and posts the reports. `buildReport` is a pure function of + the hosts' state. + +### The backend + +`netwatch-server` is a Go HTTP server wired with `fx`. Its packages are in +`internal/`: + +- `cli` runs the command line: the `check-cidr` and `prepare-data-dir` + subcommands, which `bin/entrypoint.sh` runs before the server starts, and, + given no arguments, the server. Any other arguments get a usage message and + exit code 2. +- `config` reads the settings and refuses values the server cannot use. +- `server` builds the router and the HTTP server and registers the routes. +- `middleware` holds the request log, the security headers, CORS, the body + limit, the rate limit, the client address resolution and the metrics. +- `handlers` answers the health check and takes reports. +- `reportbuf` is the in-memory buffer of reports and the files it writes them + to. It also holds `PrepareDataDir`, which `prepare-data-dir` runs. +- `healthcheck`, `logger` and `globals` hold the health status, the logger and + the name and version the build stamps in. + +In the image, the `test` phase of `Dockerfile` tests the backend, the `builder` +stage builds it, and `bin/entrypoint.sh` runs it as user `netwatch` on +`127.0.0.1:8081`, behind nginx. Outside the image, `make run` builds it and runs +it on port 8080. + +## Development + +### Entrypoints + +This repository adheres to the +[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) +standard. The scripts in `script/` are the entrypoints for the development +workflow, and the Makefile targets are thin shims that call them. The scripts +cover both the page and the Go backend, so `make check` fails if either one is +broken. Every script that runs yarn on the host does so through `script/yarn`, +except `script/bootstrap`. + +- `script/bootstrap`: install the dependencies. It uses the `node` on `PATH` + when that node has the pinned major version and is new enough for the page's + dependencies. Otherwise it installs the pinned node through nvm, fetched as a + hash-verified archive. It uses the `yarn` on `PATH` only when it is exactly + the pinned version. Otherwise it installs that version through corepack, or + through npm where there is no corepack. Then it runs + `yarn install --frozen-lockfile`. It uses the `go` on `PATH` only when it is + exactly the pinned version. Otherwise it installs that release from a + hash-verified archive. Then it runs `go mod download`. What it installs itself + is linked into `~/.local/bin`, which has to be on `PATH`. It installs no Go + linter and no Docker; `make test` and `make lint` run in Docker. +- `script/setup`: make a fresh clone ready for development: `script/bootstrap`, + then the git pre-commit hook. +- `script/dev`: run the Vite dev server, which passes `/api` to a + `netwatch-server` running locally. +- `script/build`: build the page into `dist/`, then the static `netwatch-server` + binary into the repo root. The binary's version is `VERSION` from the + environment. When that is unset or empty, it is + `git describe --tags --always --dirty`, as in `script/docker`, or `dev` + outside a git checkout. +- `script/run`: `script/build`, then run `netwatch-server` with its defaults. +- `script/clean`: remove what `script/build` writes. +- `script/projectname`: print the project name, `netwatch`, which + `script/docker` uses as the image tag. +- `script/yarn`: run yarn with the given arguments under the node + `script/bootstrap` would use. +- `script/test`: build the `test` phase of `Dockerfile`, writing no image. The + phase runs the backend's Go tests with the race detector and coverage, and the + page's unit tests in its `frontend-test` stage. +- `script/lint`: build the `lint` phase of `Dockerfile`, writing no image. The + phase runs golangci-lint over the Go packages, and eslint in its + `frontend-lint` stage. +- `script/fmt`: format all files (writes): prettier over the JavaScript, CSS, + HTML and Markdown, then gofmt over `cmd/` and `internal/`. It runs on the + host, with `~/.local/bin` put on `PATH`. +- `script/fmt-check`: check the formatting (read-only), with the same scope as + `script/fmt`. +- `script/check`: `script/test`, `script/lint` and `script/fmt-check`. +- `script/add-dependency`: add a page dependency, or move one to another + version. `make add-dependency PACKAGE=@` runs `yarn add --dev`, + which changes `package.json` and `yarn.lock` together, then + `yarn install --frozen-lockfile`. +- `script/tidy`: run `go mod tidy`. To add a Go module, import it and run + `make tidy`. To move one to another version, edit its `require` line in + `go.mod`, then run `make tidy`. +- `script/frontend-test`: run the unit tests in `test/unit/` on the host, with + Node's built-in test runner, then the production build. If a test fails, the + tests run again listing every test, and the script fails. Each test run has a + 90-second timeout. +- `script/frontend-fmt` and `script/frontend-fmt-check`: the prettier half of + `script/fmt` and `script/fmt-check`. +- `script/frontend-viewport-test`: check the built page's layout in a + containerised headless Chrome; see + [test/viewport/README.md](test/viewport/README.md). It is not part of + `script/check`, as it takes minutes. +- `script/docker`: build the image from `Dockerfile`, tagged `netwatch`. +- `script/cibuild`: the CI entrypoint. It runs `script/bootstrap` and + `script/check`, then builds the image as `script/docker` does. The Gitea + workflow `.gitea/workflows/check.yml` runs it on every push. +- `script/precommit`: run by the git pre-commit hook; it runs `script/check`. +- `script/install-precommit`: install the git pre-commit hook. `make hooks` + calls it. + +### Build caching + +`script/test`, `script/lint`, `script/docker` and `script/cibuild` each pass a +new `CACHEBUST` build argument. In `Dockerfile`, every step that lints, tests or +compiles comes after `ARG CACHEBUST`, so it runs again on every build, even when +nothing has changed. Docker keeps the layers above that line, the system +packages and the Go modules, until the base image, their own lines, or `go.mod` +and `go.sum` change. Go's build cache and golangci-lint's cache are kept between +builds too; each reuses a result only when everything it was made from is +unchanged. Both `go test` runs carry `-count=1`, so no test result is reused. +The yarn packages are installed again on every build. + +### Tests + +`make test` runs the backend's Go tests and the page's unit tests in +`test/unit/`, in Docker. The unit tests import `web/src/main.js` in Node and +call the functions it exports; no browser is involved. + +`make frontend-viewport-test` builds the page and checks its layout at several +window widths in a headless Chrome, served by the same nginx image and +configuration as the Docker image. It tests each breakpoint declared in +`web/src/styles.css`, today the one at `max-width: 768px`, and the widths one +pixel either side of it. It also tests four fixed widths: 320px, 1280px and two +phone landscape sizes. [test/viewport/README.md](test/viewport/README.md) says +what it checks and what it cannot. ## TODO