From df400831da572e798b42bc8adcc654704430d7d4 Mon Sep 17 00:00:00 2001 From: clawbot Date: Mon, 26 Oct 2026 06:25:11 +0000 Subject: [PATCH] README.md: rework against the fifth review, mostly by cutting Fixes the seven findings and drops mechanism detail the code and its comments hold. 5,864 words to 3,500. Model: fable-5-1 --- README.md | 631 +++++++++++++++++------------------------------------- 1 file changed, 194 insertions(+), 437 deletions(-) diff --git a/README.md b/README.md index 87c6372..dda55f4 100644 --- a/README.md +++ b/README.md @@ -36,23 +36,22 @@ make docker docker run -p 8080:8080 netwatch ``` -`make check` and `make docker` need Docker. `make dev` serves the page at the -address Vite prints and passes `/api` requests to `http://127.0.0.1:8080`. -`make run` starts `netwatch-server` there with its defaults, so the page's -reports are stored in `data/reports`. +`make check` and `make docker` need Docker. `make run` starts `netwatch-server` +on port 8080 with its defaults, so the page's reports are stored in +`data/reports`. `make dev` passes the page's `/api` requests there. ## The page -The page shows one row per host. A row has the host's name, its URL, its latest -latency in milliseconds and a graph of its last 100 checks. Below the figure, -the row shows the minimum, median, average and maximum of the host's answers in -that history. After a failed check, it shows `timeout` or `unreachable` there -instead. A round of checks runs every 3 seconds by default. The header shows the -interval, how long the history spans, and whether the page is running or paused. +The page shows one row per host: the host's name, its URL, its latest latency in +milliseconds and a graph of its last 100 checks. Below the figure are the +minimum, median, average and maximum of the host's answers in that history, or +`timeout` or `unreachable` after a failed check. A round of checks runs every 3 +seconds by default. The header shows the interval, the history's span, and +whether the page is running or paused. ### Targets -The page checks 26 hosts on the internet, called the WAN hosts: +The page checks 26 hosts on the internet, the WAN hosts: - `datavi.be`, pinned to the top at start - the Anthropic API and the OpenAI API @@ -63,79 +62,32 @@ The page checks 26 hosts on the internet, called the WAN hosts: - 6 Hetzner speed-test servers: Nuremberg, Falkenstein, Helsinki, Ashburn, Hillsboro and Singapore -Two local targets follow under the heading "Local Network". They are left out of -the summary and the health box. +Two local targets follow under the heading "Local Network", left out of the +summary and the health box. `Local CPE` is the cable modem at +`http://192.168.100.1`. `Local Gateway` is the first of `http://192.168.1.1`, +`http://192.168.0.1`, `http://192.168.8.1` and `http://10.0.0.1` to answer +within 1.5 seconds of the page loading; with no answer, it has no row. Browsers +block a page served from a public address from reaching private addresses. So +the local targets answer only when the page is served from `localhost` or from a +private address. -- `Local CPE` is the cable modem at `http://192.168.100.1`. Its row is always - shown. -- `Local Gateway` is found when the page loads. The page sends a request to - `http://192.168.1.1`, `http://192.168.0.1`, `http://192.168.8.1` and - `http://10.0.0.1`. The first to answer within 1.5 seconds becomes this target. - If none answers, there is no row for it. - -Browsers block a page served from a public address from reaching private -addresses (Private Network Access). So the local targets answer only when the -page is served from `localhost` or from a private address, and only on a network -that has them. - -The host list is written in `web/src/main.js`. There is no setting for it. +The host list is written in `web/src/main.js`; there is no setting for it. ### What a figure measures -A check is a request to the target's URL: GET, or HEAD for the S3 regions. The -figure is the time from sending the request until the response headers arrive, -in milliseconds. Unlike ping, it includes the time the server takes to answer. -When the browser opens a new connection, it also includes the DNS lookup, the -connection setup and the TLS handshake. Chrome opens a new connection for every -check of the Hetzner servers, and for more targets at longer intervals. +A check is a request to the target's URL: `GET`, or `HEAD` for the S3 regions. +The figure is the time from sending the request until the response headers +arrive, in milliseconds. Unlike ping, it includes the time the server takes to +answer. On a new connection it also includes the DNS lookup, the connection +setup and the TLS handshake. Any answer counts as reachable, an error page +included. The browser follows redirects, so a target whose home page redirects +is checked at `/robots.txt` on the same host. -Any answer counts as reachable, an error page included. The request is made in -the browser's `no-cors` mode, so the page cannot read the answer, only time it. - -The browser follows redirects, and the figure then includes the request to the -new URL too. So a target whose home page redirects is checked at `/robots.txt` -on the same host, which answers without a redirect. The S3 regions are checked -with a HEAD request. Chrome keeps the connection to an S3 region open after a -HEAD request, but closes it after most GET requests. - -The request is sent with `cache: 'no-store'`, so the browser's cache stays out -of the measurement. It carries no `Referer` header and no query string: the -Hetzner servers close the connection without an answer when either is present. - -A check times out after 80% of the interval: 2.4 seconds at the default of 3 -seconds, 24 seconds at 30 seconds. So a round's checks have all finished before -the next round is due. A check that times out is recorded as a timeout, and so -is one that answers after that limit. A check that fails with a network error is -recorded as unreachable. - -A round can start early, after an interval change or when the recovery probe -finds a target answering. It then gives up the last round's checks that are -still waiting, so rounds never overlap. - -The first round after the page loads is discarded, as is the first round after -an interval change. Its connections are all new, so its figures run high. - -The browser chooses between IPv4 and IPv6 for each target, as for any request. -The local targets are IPv4 addresses. - -### Colors - -A latency figure and its line in the graph take one color: - -- green: under 50 ms -- lime: under 100 ms -- yellow: under 200 ms -- orange: under 500 ms -- red: 500 ms and over -- gray: no answer - -### The graph - -The graph in each row plots the host's last 100 checks. At the default interval -that is 5 minutes; the labels on its time axis say how long. The vertical axis -runs from 0 to 1000 ms. A latency above 1000 ms is drawn at the top, and the -figure still shows its real value. The rounds in which a check failed are shaded -red. +A check times out after 80% of the interval: 2.4 seconds at the default. A check +that answers after that limit counts as a timeout too. A check that fails with a +network error is recorded as unreachable. The first round after the page loads +is discarded, as its figures include the connection setup. So is the first round +after an interval change. ### Health and summary @@ -147,156 +99,93 @@ counts as failed when its last check timed out or found it unreachable. - `SLOW` (yellow): otherwise, more than 3 hosts took over 1000 ms - `HEALTHY` (green): otherwise -Below the health box, a summary line covers the WAN hosts only: +Below it, a summary line covers the WAN hosts only: -- `Reachable`: how many answered in the last round, out of 26 +- `Reachable`: how many of the 26 answered their last check - `Now`: the minimum, median, average and maximum of their latest figures -- the history's span, such as `5m`, with the minimum and maximum over the whole +- the history's span, such as `5m`: the minimum and maximum over the whole history -- `Checks`: the number of rounds run since the page loaded or the interval - changed +- `Checks`: the rounds run since the page loaded or the interval changed -When no WAN host answers, the page starts a recovery probe. Every half second it -checks 4 WAN hosts and gives up the checks it started half a second before. The -4 are picked at random when the probe starts. As soon as one answers, a new -round starts at once. +When no WAN host answers, the page checks 4 of them, picked at random, every +half second. As soon as one answers, a new round starts. -### Row order +### Rows, colors and controls -The WAN rows are sorted by their latest latency, fastest first and unreachable -last. The rows are sorted when the second round ends, and again in rounds 11, -21, 31 and so on. The pin button at the left of a row keeps the row at the top. -Pinned rows are sorted by name, and a line separates them from the rest. -`datavi.be` is pinned at start. +A figure and its line in the graph take the color of the legend in the footer, +from green under 50 ms to red from 500 ms up; gray is no answer. The graph's +axis ends at 1000 ms; a higher latency is drawn at the top, and the figure still +shows its real value. -### Controls +From time to time, the page sorts the WAN rows by their latest latency: fastest +first, failed last. The pin button at the left of a row keeps the row at the +top; pinned rows are sorted by name. `datavi.be` is pinned at start. -- `Pause` stops the checks. The graphs keep scrolling and leave a gap for each - paused round; no outage is drawn. The figures turn gray until `Resume`. -- The `Interval` menu sets the time between rounds: 1, 2, 3, 5, 10, 15, 30 or 60 - seconds. Changing it clears the history on the page and starts a new round at - once. -- The pin button in each WAN row pins and unpins the row. -- Each host's URL is a link that opens in a new tab. -- The `Debug log` checkbox in the footer opens a panel with the page's last 1000 - log lines. Each line has a level: error, warning, notice, info or debug. -- The header shows the local time with its time zone, and the UTC time. The - footer shows the color legend and links to the commit the page was built from. - -When the page records a failed check, it writes one line to the browser console -with `console.error`, and the same line to the debug log. Nothing is written for -the discarded first round, for a check a round gives up, or by the recovery -probe. The line gives the target's name and URL, the time, what failed and how -long the request took. The three kinds of failure are a timeout, an answer over -the time limit, and a network error. For a network error the line quotes the -error the browser gives the page, such as `TypeError: Failed to fetch`, which -does not say why. When a target answers again after failed checks, one -`console.info` line gives its latency and how many checks in a row had failed. - -### Browsers - -The page needs a browser with ES modules, the Fetch API, the Canvas API and CSS -custom properties. +`Pause` stops the checks: the graphs keep scrolling with a gap for each paused +round, and the figures turn gray. The `Interval` menu sets the time between +rounds: 1, 2, 3, 5, 10, 15, 30 or 60 seconds. Changing it clears the history and +starts a new round at once. The `Debug log` checkbox in the footer opens a panel +with the page's last 1000 log lines. ## Reporting -Every 60 seconds the page gathers its undelivered samples and posts them as a -report to `/api/v1/reports` on the origin it was served from. When there are -none, it posts nothing. The report is JSON with four fields: +Every 60 seconds the page posts its undelivered samples as a report to +`/api/v1/reports` on the origin it was served from. The report is JSON with four +fields: -- `clientId`: a random id, made once per browser and kept in `localStorage`. - When the browser has no usable storage, a new id is made for the session. +- `clientId`: a random id, made once per browser and kept in `localStorage`, or + once per session when the browser has no usable storage - `geo`: always `null` -- `hosts`: one entry per host with new samples. The entry has the host's `name`, - `url` and `status`, and `history`, a list of samples. A sample has the check's - time `t` in Unix milliseconds, its `latency` (`null` for a failed check) and - its `error` (`"timeout"`, `"unreachable"` or `null`). +- `hosts`: one entry per host with new samples: its `name`, `url` and `status`, + and `history`, its samples. A sample has the check's time `t` in Unix + milliseconds, its `latency` (`null` for a failed check) and its `error` + (`"timeout"`, `"unreachable"` or `null`). - `timestamp`: the time of the report -Each report carries only the samples no earlier report has delivered. For each -host, the page remembers the time of the newest sample it delivered, and moves -that mark only when the server answers 2xx. Paused rounds are never sent. While -the page is paused, no report is sent at all. One report is in flight at a time. -A report with no answer after 30 seconds is abandoned, and its samples go in the -next report. So a server that stored them but answered late gets them twice. +Each report carries only the samples no earlier report has delivered. A sample +waits only as long as the history holds it: the last 100 rounds of each host. +Nothing is sent while the page is paused. A report with no answer after 30 +seconds is abandoned, and its samples go in the next one. So a server that +stored them but answered late gets them twice. -A failed delivery is quiet. The page writes one debug log line when deliveries -start failing and one when they recover. The checks go on. - -A sample waits for delivery only as long as the history holds it. The history -keeps each host's last 100 rounds: 5 minutes at the default interval of 3 -seconds. At 1 second it spans 100 seconds, at 2 seconds 200 seconds, and at 60 -seconds 100 minutes. Changing the interval clears the history. The cleared -samples are kept apart, the last 100 per host, and those not yet delivered go in -the next report. - -So at the 1-second interval, one refused or failed report loses the oldest 20 -seconds of each host's samples: the next delivered report comes 2 minutes after -the last. At 2 seconds and longer, one failed report loses nothing. Samples are -lost from the third failed report in a row at 2 seconds, from the fifth at 3 -seconds, and later still at longer intervals. - -Each paused round still adds an entry to the history, so each report due during -a pause counts as a failed one. A pause longer than the history spans loses the -samples recorded between the last delivered report and the pause: up to a -minute's worth for each host. +A failed delivery is quiet, as on a static file host with no backend: one debug +log line when deliveries start failing, one when they recover. The checks go on. ## Deployment -### Static files +`make docker` builds the image `netwatch` from `Dockerfile`: the whole service +in one container. nginx serves the page and passes `/api/`, +`/.well-known/healthcheck` and `/metrics` to `netwatch-server` in the same +container. The backend listens on `127.0.0.1:8081` only, as user `netwatch` (uid +1000). `bin/entrypoint.sh` starts both and watches them. -`make build` writes the page to `dist/`: an `index.html` and an `assets/` -directory with one CSS file and one JavaScript file. Any static file host can -serve it. Without a backend at `/api/v1/reports` on that host, the reports fail -quietly and nothing is stored. +Before the backend starts, `bin/entrypoint.sh` runs two subcommands of +`netwatch-server`. `check-cidr` checks each `TRUSTED_PROXIES` entry (see +[Running under upaas](#running-under-upaas)). `prepare-data-dir` creates the +report directory and gives `/data`, with the directories, regular files and +symbolic links in it, to user `netwatch`. It changes nothing outside `/data`. -### The Docker image +The image also: -`make docker` builds the image `netwatch` from `Dockerfile`. It is the whole -service in one container. nginx serves the page and passes `/api/`, -`/.well-known/healthcheck` and `/metrics` to `netwatch-server`, which runs in -the same container. The backend listens on `127.0.0.1:8081` only, as user -`netwatch` (uid 1000). `bin/entrypoint.sh` starts both and watches them. - -The image: - -- listens on port `8080`; the `PORT` variable changes it (see - [Settings](#settings)) -- stores the report files in `/data/reports`, on the `/data` volume. Before the - backend starts, `netwatch-server prepare-data-dir` creates that directory and - gives `/data` and the directories, regular files and symbolic links in it to - user `netwatch`. A file with a second name (hard link) keeps its owner, and - its path is logged. A host directory mounted at `/data` therefore ends up - owned by uid 1000. The command changes nothing outside `/data`, whatever links - it meets there. -- takes the client address from `X-Forwarded-For` only on a request from a - reverse proxy named in `TRUSTED_PROXIES`. nginx resolves the address and - passes that one address to the backend. The backend is started with - `TRUSTED_PROXIES=127.0.0.1/32`, so it trusts nginx alone. -- writes nginx's access log to stdout, and the backend's log as JSON to stdout -- sends the page with `Cache-Control: no-cache`, so a browser that keeps a copy - asks the server whether it is current before using it again. The files under - `/assets/` are cached as immutable; their names change whenever their contents - do. -- sends the security headers in `configs/security-headers.conf` on every - response, in place of the backend's own -- refuses a request body over 1 MiB with nginx's own 413 error page; the backend - never sees such a request +- takes the client address from `X-Forwarded-For` only on a request from a proxy + named in `TRUSTED_PROXIES`, and passes that one address to the backend. The + backend is started with `TRUSTED_PROXIES=127.0.0.1/32`, so it trusts nginx + alone. +- writes nginx's access log and the backend's JSON log to stdout +- refuses a request body over 1 MiB with nginx's own 413 error page - requests `/.well-known/healthcheck` through nginx every 30 seconds, as its - `HEALTHCHECK`. The check fails unless both nginx and the backend answer. It - also fails while report files cannot be written. Three failures in a row make - the container `unhealthy`; one pass makes it `healthy` again. -- on `docker stop`, writes the waiting reports to disk and exits 0. If that - final write fails, the container exits non-zero. If nginx or the backend exits - on its own, the other is stopped and the container exits non-zero, so the - platform restarts it. + `HEALTHCHECK`. The check fails unless both nginx and the backend answer, and + while report files cannot be written. +- on `docker stop`, writes the waiting reports to disk and exits 0, or non-zero + if that write fails. If nginx or the backend exits on its own, the other is + stopped and the container exits non-zero. ### Running under upaas The [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs: - Port: container port `8080`. -- Volume: container path `/data`. The reports are kept in `/data/reports`. +- Volume: container path `/data`. - Environment variables: none is required. Every setting in [Settings](#settings) applies, with four differences in the image: - `BIND_ADDRESS` has no effect. The backend listens on `127.0.0.1` inside @@ -307,14 +196,16 @@ The [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs: `.` or `..` part and no extra `/`. A path through a symbolic link is refused unless the link is relative and stays inside `/data`. - `TRUSTED_PROXIES` names the proxies nginx trusts, as IP addresses or CIDRs - separated by commas. The backend's own setting is fixed to `127.0.0.1/32`. - With `TRUSTED_PROXIES` set and empty, nginx trusts no proxy. Every client - behind the proxy then shares the proxy's one allowance of - `REPORTS_PER_MINUTE`. Any client that connects from an address in the set - can write its own `X-Forwarded-For`. A connection through a port Docker - publishes can arrive from the Docker network's gateway, such as - `172.17.0.1`. The default includes that address. Where that matters, name - only the addresses that nothing but the proxy connects from. + separated by commas; the backend's own setting is fixed to `127.0.0.1/32`. + `bin/entrypoint.sh` gives a lone address a `/32` or `/128` and checks each + entry with `netwatch-server check-cidr`, which refuses what the backend + would refuse. Set and empty, nginx trusts no proxy, and every client + behind a proxy then shares the proxy's one allowance of + `REPORTS_PER_MINUTE`. A client that connects from a trusted address can + write its own `X-Forwarded-For`. A port Docker publishes can bring + connections from the Docker network's gateway, such as `172.17.0.1`, which + the default includes. Where that matters, name only the proxy's own + addresses. - Health: the image's `HEALTHCHECK`, described above. The container stops at start when a variable holds a value it cannot use. The @@ -350,30 +241,24 @@ names the variable. An empty variable counts as unset, except `TRUSTED_PROXIES`. API, separated by commas. See [CORS](#cors). - `METRICS_USERNAME` and `METRICS_PASSWORD`, default empty: the basic auth credentials for `/metrics`. See [Metrics](#metrics). -- `SENTRY_DSN`, default empty: the DSN of a Sentry project to report crashes to. - See [Sentry](#sentry). +- `SENTRY_DSN`, default empty: the DSN of a Sentry project. Each request whose + handling crashes is reported there; the request still gets a 500. - `DEBUG`, default empty: any non-empty value turns debug logging on, `false` and `0` included. Every value is accepted. ### Trusted proxies Each `TRUSTED_PROXIES` entry must be a CIDR, such as `127.0.0.1/32`; a lone -address is refused. A list replaces the default. Set and empty, the server +address is refused. A list replaces the default; set and empty, the server trusts no proxy. Loopback is not in the default, so a reverse proxy on the same host has to be named. -A request from a peer outside this set has its forwarded headers ignored. The -peer's own address is logged and rate limited. - -A request from a peer inside the set takes its client address from -`X-Forwarded-For`. Each proxy appends the address it took the request from, so -the entries on the right were written by trusted proxies and the ones further -left by the client. The client address is the right-most entry that is not in -the set. An entry may carry a port, such as `198.51.100.7:5000` or -`[2001:db8::1]:443`. It is read as the address alone. The peer's address is used -when every entry is in the set. It is also used when the search from the right -meets an entry that is not an address at all, such as `unknown`. `X-Real-IP` is -read only when `X-Forwarded-For` is missing or empty. +A request from a peer outside this set has its forwarded headers ignored; the +peer's own address is logged and rate limited. From a peer inside the set, the +client address is the right-most `X-Forwarded-For` entry that is not in the set. +The peer's address is used when every entry is in the set, or when the search +from the right meets an entry that is not an address, such as `unknown`. +`X-Real-IP` is read only when `X-Forwarded-For` is missing or empty. ### Routes @@ -391,29 +276,14 @@ Every request body is capped at 1 MiB, and every request at 60 seconds. ### Report storage The server keeps the reports it takes in memory and writes them to `DATA_DIR` as -files of JSON lines, compressed with zstd. It writes the waiting reports to a -new file every minute, whenever 10 MiB of them are waiting, and when it stops. -With none waiting, it writes no file. +files of JSON lines, compressed with zstd. It writes a new file every minute, +whenever 10 MiB of reports are waiting, and when it stops. With none waiting, it +writes no file. The files are named `reports--.jsonl.zst`; +the timestamp is UTC to the millisecond, so the names sort by time. -The files are named `reports--.jsonl.zst`. The timestamp is -UTC to the millisecond, so the names sort by time. The number starts at 1 when -the server starts, and goes up by one for each file the server starts to write. -So two files started in the same millisecond still get different names. A file -is created with `O_EXCL`, so it never overwrites another. - -A write can fail, as when the disk is full. The reports of a failed write stay -in memory. The next write tries them again, within a minute. One file is written -at a time, so a file holds its reports in the order they arrived, those of a -failed write included. A failed write uses up its number. A file it created is -removed. If removing it fails too, the file stays and counts toward -`DATA_DIR_MAX_BYTES` from the next start. - -From a failed write until a write succeeds, `POST /api/v1/reports` refuses every -report with 500 and keeps nothing of it, and the health check answers 503. The -page sends the refused samples again in its next report, as long as it still -holds them (see [Reporting](#reporting)). Reports already answered 200 are lost -only if the server stops before a write succeeds. The final write at stop tries -them too, and if it fails, the process exits non-zero. +The reports of a failed write, as on a full disk, stay in memory for the next +write to try again. Until a write succeeds, `POST /api/v1/reports` refuses every +report with 500 and the health check answers 503. ### Report limits @@ -421,102 +291,48 @@ them too, and if it fails, the process exits non-zero. bounded in four ways. Each refusal by the backend answers with the body `{"status":"error"}`. -- JSON only. A report must be sent with `Content-Type: application/json`, with - or without parameters such as `charset`, as the page sends it. Any other - content type is refused with 415, and nothing of it is stored. A page on - another origin can send a `text/plain` POST, or a form, without a CORS - preflight. Refusing those keeps any site from making its visitors' browsers - store reports (see [CORS](#cors)). -- Body limit. A report body may be at most 1 MiB, the backend's limit for any - request. A request that declares a longer body is answered 413 before it - reaches the route. A body sent without a declared length is read up to 1 MiB. - The route answers 413 only when the report runs past that; nothing after the - end of the report is read. A body that stops being valid JSON sooner is - answered 400. In the image, nginx refuses a body over 1 MiB itself, with its - own HTML error page, and the backend never sees the request. +- JSON only. A report must be sent with `Content-Type: application/json`; any + other content type is refused with 415, and malformed JSON with 400. +- Body limit. A report body may be at most 1 MiB. A request that declares a + longer body, or a report that runs past that, is refused with 413. - Rate limit. Each client address may send `REPORTS_PER_MINUTE` reports a - minute. [Trusted proxies](#trusted-proxies) says how that address is found. - Past that it gets 429 with `Retry-After: 60`. The minute slides: reports from - the minute before still count, fading out over the current one. An address - that sends at most half the limit in any 60 seconds is never refused. Above - that, it may be. The page sends one report a minute from each open tab. So the - default of 60 refuses nothing from up to 30 tabs behind one address, however - their reports bunch up. Every answer the route gives carries the headers - `X-RateLimit-Limit`, `X-RateLimit-Remaining` and `X-RateLimit-Reset`. + minute; past that it gets 429 with `Retry-After: 60`. The page sends one + report a minute from each open tab, so the default of 60 refuses nothing from + up to 30 tabs behind one address. - Size cap. The report files in `DATA_DIR` may total at most - `DATA_DIR_MAX_BYTES`, counting the files already there at start. Reports - waiting in memory count at their uncompressed size until they are written. - When a report would take the total past the cap, the oldest report files are - deleted to make room, and each deletion is logged with the file's name and - size. A file still being written is never deleted. A report is refused with - 507 only when the reports waiting to be written fill the cap on their own; - then no file is deleted. At start, report files past the cap are deleted the - same way, as after lowering it. + `DATA_DIR_MAX_BYTES`, counting the files already there at start and the + reports waiting in memory. When a report would take the total past the cap, + the oldest files are deleted to make room. A report is refused with 507 when + deleting the oldest files cannot make room. -So the cap decides how much of the newest reports is kept. The default of 1 GiB -fits on any host; set it to the space you can give `DATA_DIR`. At the defaults, -one address may send 60 reports of up to 1 MiB a minute. A report made up to -compress badly takes about as much room in the files as in its request body. So -in about 17 minutes one address can fill the 1 GiB cap with its own reports, and -every report stored before them is deleted to make room. A lower -`REPORTS_PER_MINUTE` or a higher `DATA_DIR_MAX_BYTES` makes that take longer. +At the defaults, one address can fill the 1 GiB cap with its own reports in +about 17 minutes, and every report stored before them is deleted to make room. ### CORS -The page calls the API from the origin it is served from. By default the server -sends no CORS headers, so browsers let no other origin's pages call it. -`CORS_ALLOWED_ORIGINS` lets pages served from the listed origins call the API, -for example `https://netwatch.example.com`. The page as built posts to its own -origin only, so it needs no entry. Pages from a listed origin may `GET` and -`POST` with a `Content-Type` header. - -Each entry must be a plain origin as browsers send it: `scheme://host`, with an -optional `:port`, no path, no trailing `/` and no `*`. Any other entry stops the -server from starting. +The page needs no entry here: it calls the API from the origin it is served +from. By default the server sends no CORS headers, and browsers let no other +origin's pages call it. `CORS_ALLOWED_ORIGINS` lists other origins, such as +`https://netwatch.example.com`, whose pages may then call the API with `GET` and +`POST`. Each entry must be a plain origin as browsers send it: `scheme://host`, +with an optional `:port`, no path, no trailing `/` and no `*`. ### Metrics With both `METRICS_USERNAME` and `METRICS_PASSWORD` set, the server serves Prometheus metrics at `GET /metrics`. A request with those as its basic auth -credentials gets them; any other gets 401. With neither set, there are no -metrics and `/metrics` answers 404. One set without the other stops the start, -and so does a user name containing `:`. - -The metrics record each request that reaches the health check or -`POST /api/v1/reports`. For each, they record its duration and response size, -labelled with its path, method and status, and they count such requests in -progress. Requests the rate limit refuses are included. Go's runtime and process -metrics are included too. No other request is recorded: - -- not a request to `/metrics` itself -- not a CORS preflight -- not a 404 for an unknown path, or a 405 for the wrong method -- not a 413 answered before the route - -Only those two routes are recorded because the labels are the request's path and -method. Recording every request would let clients add labels without bound. - -`/metrics` has its own rate limit, so that its password cannot be guessed -quickly. Each client address may make 60 requests to it a minute, whatever its -credentials. The address is found as for reports. Past that it gets 429 with -`Retry-After: 60`, and its credentials are not checked. The minute slides as it -does for reports, so a scraper polling every 2 seconds or less often is never -refused. - -### Sentry - -With `SENTRY_DSN` set, each request whose handling crashes is reported to that -Sentry project, under the release `netwatch-server-` followed by the server's -version. The request still gets a 500 answer. The server does not start if -Sentry rejects the DSN. On shutdown the server waits up to 2 seconds for Sentry -to finish sending. Empty, the server sends nothing to Sentry. +credentials gets them; any other gets 401. With neither set, `/metrics` +answers 404. The metrics record each request that reaches the health check or +`POST /api/v1/reports`: its duration and response size, labelled with its path, +method and status. Each client address may make 60 requests to `/metrics` a +minute. Past that it gets 429 with `Retry-After: 60`, and its credentials are +not checked. ### Logging The backend logs JSON to stdout, or plain text when stdout is a terminal. Each -line names the source location it was written from. Each request is logged with -its method, URL, status, client address and duration, among other fields. Every -value taken from a request is cut to 128 bytes before it is logged. +request is logged with its method, URL, status, client address and duration. +Every value taken from a request is cut to 128 bytes before it is logged. ## Design @@ -538,24 +354,17 @@ The repository holds the page, the backend and the image: The page is built with Vite and Tailwind CSS v4. All its code is in `web/src/main.js`: -- `CONFIG` holds the timing and display constants. The interval menu sets its - `updateInterval`; the check timeout, the history span and the time axis are - computed from it. -- `WAN_HOSTS` is the list of WAN targets, each with its URL. The S3 entries also - name HEAD as their method. -- `HostState` is one host's history, latest latency and status. -- `AppState` holds the WAN hosts, the local hosts, the pause state and the round - count, and computes the summary and the health. -- `measureLatency` runs one check. -- `tick` runs one round. It checks every host in parallel and draws each row as - soon as its check ends. When the last check ends, it redraws every row, the - summary and the health box, and sorts the rows when a sort is due. A discarded - round draws nothing. -- `SparklineRenderer` draws a graph on a canvas, scaled to the device's pixel - ratio. -- `buildUI` builds the page; `updateHostRow`, `updateSummary` and - `updateHealthBox` redraw its parts. -- `Reporter` builds and posts the reports. `buildReport` is a pure function of +- `CONFIG`: the timing and display constants. The interval menu sets its + `updateInterval`; the check timeout and the history span follow from it. +- `WAN_HOSTS`: the WAN targets, each with its URL and, for the S3 entries, the + method `HEAD`. +- `HostState` and `AppState`: one host's history and status, and the whole + page's state. +- `measureLatency`: one check. `tick`: one round, which checks every host in + parallel and draws each row as soon as its check ends. +- `SparklineRenderer`: the graphs. `buildUI`, `updateHostRow`, `updateSummary` + and `updateHealthBox`: the page and its parts. +- `Reporter`: builds and posts the reports. `buildReport` is a pure function of the hosts' state. ### The backend @@ -563,24 +372,17 @@ The page is built with Vite and Tailwind CSS v4. All its code is in `netwatch-server` is a Go HTTP server. The `fx` library starts and stops its parts in dependency order. The backend's packages are in `internal/`: -- `cli` runs the command line. Given no arguments, it runs the server. It also - runs the `check-cidr` and `prepare-data-dir` subcommands, which - `bin/entrypoint.sh` runs before the server starts. Any other arguments get a - usage message and exit code 2. -- `config` reads the settings and refuses values the server cannot use. -- `server` builds the router and the HTTP server and registers the routes. -- `middleware` holds the request log, the security headers, CORS, the body - limit, the rate limit, the client address resolution and the metrics. -- `handlers` answers the health check and takes reports. -- `reportbuf` is the in-memory buffer of reports and the files it writes them - to. It also holds `PrepareDataDir`, which `prepare-data-dir` runs. -- `healthcheck`, `logger` and `globals` hold the health status, the logger and - the name and version the build stamps in. - -In the image, the `test` phase of `Dockerfile` tests the backend, the `builder` -stage builds it, and `bin/entrypoint.sh` runs it as user `netwatch` on -`127.0.0.1:8081`, behind nginx. Outside the image, `make run` builds it and runs -it on port 8080. +- `cli`: the command line: the server when given no arguments, or the + `check-cidr` and `prepare-data-dir` subcommands that `bin/entrypoint.sh` runs. +- `config`: reads the settings and refuses values the server cannot use. +- `server`: the router, the routes and the HTTP server. +- `middleware`: the request log, the security headers, CORS, the body limit, the + rate limit, the client address resolution and the metrics. +- `handlers`: the health check and the report route. +- `reportbuf`: the in-memory buffer of reports and the files it writes them to. + It also holds `PrepareDataDir`, which `prepare-data-dir` runs. +- `healthcheck`, `logger` and `globals`: the health status, the logger, and the + name and version the build stamps in. ## Development @@ -589,100 +391,55 @@ it on port 8080. This repository adheres to the [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) standard. The scripts in `script/` are the entrypoints for the development -workflow, and the `Makefile` targets are thin shims that call them. The scripts -cover both the page and the Go backend, so `make check` fails if either one is -broken. Every script that runs yarn on the host does so through `script/yarn`, -except `script/bootstrap`. +workflow, and the `Makefile` targets are thin shims that call them. -- `script/bootstrap`: install the dependencies. It uses the `node` on `PATH` - when that node has the pinned major version and is new enough for the page's - dependencies. Otherwise it installs nvm from a hash-verified archive, and the - pinned node through it. It uses the `yarn` on `PATH` only when it is exactly - the pinned version. Otherwise it installs that version through corepack, or - through npm where there is no corepack. Then it runs - `yarn install --frozen-lockfile`. It uses the `go` on `PATH` only when it is - exactly the pinned version. Otherwise it installs that release from a - hash-verified archive. Then it runs `go mod download`. What it installs itself - is linked into `~/.local/bin`, which has to be on `PATH`. It installs no Go - linter and no Docker; `make test` and `make lint` run in Docker. -- `script/setup`: make a fresh clone ready for development: `script/bootstrap`, - then the git pre-commit hook. -- `script/dev`: run the Vite dev server, which passes `/api` to a - `netwatch-server` running locally. +- `script/bootstrap`: install the dependencies. It installs the pinned node, + yarn and Go when the ones on `PATH` do not pass its checks, and links them + into `~/.local/bin`, which has to be on `PATH`. It installs no Go linter and + no Docker. +- `script/setup`: `script/bootstrap`, then the git pre-commit hook. +- `script/dev`: run the Vite dev server. - `script/build`: build the page into `dist/`, then the static `netwatch-server` - binary into the repo root. The binary's version is `VERSION` from the - environment. When that is unset or empty, it is - `git describe --tags --always --dirty`, as in `script/docker`, or `dev` - outside a git checkout. + binary into the repo root. Its version is `VERSION` from the environment, else + `git describe --tags --always --dirty`, else `dev`. - `script/run`: `script/build`, then run `netwatch-server` with its defaults. - `script/clean`: remove what `script/build` writes. -- `script/projectname`: print the project name, `netwatch`, which - `script/docker` uses as the image tag. -- `script/yarn`: run yarn with the given arguments under the node - `script/bootstrap` would use. -- `script/test`: build the `test` phase of `Dockerfile`, writing no image. The - phase runs the backend's Go tests with the race detector and coverage, and the - page's unit tests in its `frontend-test` stage. -- `script/lint`: build the `lint` phase of `Dockerfile`, writing no image. The - phase runs golangci-lint over the Go packages, and eslint in its - `frontend-lint` stage. -- `script/fmt`: run the formatters, which write: prettier over the JavaScript, - CSS, HTML and Markdown, then gofmt over `cmd/` and `internal/`. It runs on the - host, with `~/.local/bin` put on `PATH`. +- `script/projectname`: print `netwatch`, the image tag `script/docker` uses. +- `script/yarn`: run yarn under the node `script/bootstrap` would use. +- `script/test`: build the `test` phase of `Dockerfile`, writing no image: the + Go tests with the race detector and coverage, and the page's unit tests. +- `script/lint`: build the `lint` phase of `Dockerfile`, writing no image: + golangci-lint over the Go packages and eslint over the JavaScript. +- `script/fmt`: run the formatters: prettier over the JavaScript, CSS, HTML and + Markdown, then gofmt over `cmd/` and `internal/`. They write the files in + place. - `script/fmt-check`: check the formatting (read-only), with the same scope as `script/fmt`. - `script/check`: `script/test`, `script/lint` and `script/fmt-check`. - `script/add-dependency`: add a page dependency, or move one to another - version. `make add-dependency PACKAGE=@` runs `yarn add --dev`, - which changes `package.json` and `yarn.lock` together, then - `yarn install --frozen-lockfile`. -- `script/tidy`: run `go mod tidy`. To add a Go module, import it and run - `make tidy`. To move one to another version, edit its `require` line in - `go.mod`, then run `make tidy`. + version: `make add-dependency PACKAGE=@`. +- `script/tidy`: run `go mod tidy`. - `script/frontend-test`: run the unit tests in `test/unit/` on the host, with - Node's built-in test runner, then the production build. If a test fails, the - tests run again listing every test, and the script fails. Each test run has a - 90-second timeout. + Node's built-in test runner, then the production build. - `script/frontend-fmt` and `script/frontend-fmt-check`: the prettier half of `script/fmt` and `script/fmt-check`. - `script/frontend-viewport-test`: check the built page's layout in a - containerised headless Chrome; see - [test/viewport/README.md](test/viewport/README.md). It is not part of - `script/check`, as it takes minutes. + containerised headless Chrome, not as part of `script/check`; see + [test/viewport/README.md](test/viewport/README.md). - `script/docker`: build the image from `Dockerfile`, tagged `netwatch`. -- `script/cibuild`: the CI entrypoint. It runs `script/bootstrap` and - `script/check`, then builds the image as `script/docker` does. The Gitea - workflow `.gitea/workflows/check.yml` runs it on every push. +- `script/cibuild`: `script/bootstrap`, `script/check`, then the image build. + The Gitea workflow `.gitea/workflows/check.yml` runs it on every push. - `script/precommit`: run by the git pre-commit hook; it runs `script/check`. - `script/install-precommit`: install the git pre-commit hook. `make hooks` calls it. -### Build caching - -`script/test`, `script/lint`, `script/docker` and `script/cibuild` each pass a -new `CACHEBUST` build argument. In `Dockerfile`, every step that lints, tests or -compiles comes after `ARG CACHEBUST`, so it runs again on every build, even when -nothing has changed. Docker keeps the layers above that line, the system -packages and the Go modules, until the base image, their own lines, or `go.mod` -and `go.sum` change. Go's build cache and golangci-lint's cache are kept between -builds too; each reuses a result only when everything it was made from is -unchanged. Both `go test` runs carry `-count=1`, so no test result is reused. -The yarn packages are installed again on every build. - ### Tests -`make test` runs the backend's Go tests and the page's unit tests in -`test/unit/`, in Docker. The unit tests import `web/src/main.js` in Node and -call the functions it exports; no browser is involved. - -`make frontend-viewport-test` builds the page and checks its layout at several -window widths in a headless Chrome. The page is served with the nginx image and -the `configs/nginx.conf` that the Docker image ships. It tests each breakpoint -declared in `web/src/styles.css`, today the one at `max-width: 768px`, and the -widths one pixel either side of it. It also tests four fixed widths: 320px, -1280px and two phone landscape sizes. -[test/viewport/README.md](test/viewport/README.md) says what it checks and what -it cannot. +`make test` runs the backend's Go tests and the page's unit tests in Docker. The +unit tests import `web/src/main.js` in Node and call the functions it exports; +no browser is involved. `make frontend-viewport-test` checks the built page's +layout in a headless Chrome; [test/viewport/README.md](test/viewport/README.md) +says what it checks and what it cannot. ## TODO