Targets whose home page redirects are checked at /robots.txt (closes #194)

The browser follows a redirect, so the figures for the AWS Console, Akamai,
B2 and the S3 regions also timed the page it led to. Each of these targets
is now checked at /robots.txt on its own host, which answers without a
redirect. So is the Google Cloud Console, whose home page redirects curl's
requests. The S3 regions are checked with a HEAD request, as Chrome closes
their connection after most GET requests. README.md says what a figure
includes.

Model: opus-5-5

.editorconfig

@@ -11,7 +11,8 @@ insert_final_newline = true
 [Makefile]
 indent_style = tab
 
-# This repository's own entries, after the shared content above.
-
 [*.go]
 indent_style = tab
+
+# This repository's own sections, such as one for another language it
+# uses, go below this comment, and a re-vendor keeps them.

.gitignore

@@ -27,7 +27,7 @@ node_modules/
 # Environment files. `*.env` covers bare `.env` and the `prod.env`
 # convention. Only the templates `example.env` and `sample.env` are
 # re-included below. A repository that commits any other template adds
-# its own negation after these lines, for example `!.env.example`.
+# its own negation at the end of this file, for example `!.env.example`.
 *.[eE][nN][vV]
 .[eE][nN][vV].*
 .[eE][nN][vV][rR][cC]
@@ -46,7 +46,9 @@ node_modules/
 [iI][dD]_[eE][dD]25519
 [iI][dD]_[eE][dD]25519_[sS][kK]
 
-# This repository's own entries, after the shared content above.
+# This repository's own entries, such as its build outputs, go below
+# this comment, and a re-vendor keeps them. Anchor a binary built at the
+# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
 
 # Build output
 dist/

.golangci.yml

@@ -25,6 +25,8 @@ linters:
     # silenced by disabling that name, not by enabling the successor.
     - wsl              # Deprecated, replaced by wsl_v5
     - gomodguard       # Deprecated, replaced by gomodguard_v2
+    # Misses findings at random in v2.14.0; back once a pinned release fixes it
+    - canonicalheader
   settings:
     lll:
       line-length: 88

README.md

@@ -413,20 +413,36 @@ Local hosts are tracked separately from WAN stats.
 
 ### Latency measurement
 
-GET requests to each target's URL as written, with `mode: 'no-cors'` and
-`cache: 'no-store'`, timed with `performance.now()`. No query string is added:
-the Hetzner speed-test servers close the connection without an answer when the
-URL has one, and `no-store` keeps the browser's cache out of the measurement.
-Each check times out after 80% of the refresh interval (24 seconds at 30
-seconds) and is then recorded as a timeout, so a round's checks have all
-finished before the next round is due. When no WAN host answers, a recovery
-probe checks 4 WAN hosts, picked at random when it starts, every half second,
-giving up the checks it started half a second before. As soon as one answers, a
-new round starts at once, as it does after an interval change. A round started
-early gives up the last round's checks if they are still waiting, and that round
-records nothing more, so rounds never overlap. The browser chooses between IPv4
-and IPv6 for each target, as for any request; the local targets are IPv4
-addresses.
+Each check is a GET request to the target's URL as written, with
+`mode: 'no-cors'`, `cache: 'no-store'` and `referrerPolicy: 'no-referrer'`. Its
+figure is the time from sending the request until the answer's headers arrive,
+timed with `performance.now()`. Unlike ping, this includes the time the server
+takes to answer. It also includes the DNS lookup, connection setup and TLS
+whenever the browser opens a new connection. Chrome opens one for every check of
+the Hetzner speed-test servers, and for more targets at longer intervals.
+
+The S3 regions are checked with a HEAD request instead. Chrome keeps the
+connection to an S3 region open after a HEAD request, but closes it after most
+GET requests. The Hetzner speed-test servers do not answer a HEAD request.
+
+The browser follows every redirect, and the figure then includes the request to
+the new URL too. So a target whose home page answers with a redirect is checked
+at `/robots.txt` on the same host, which answers without one. An S3 region
+answers it with 403, and any answer counts as reachable.
+
+The Hetzner speed-test servers close the connection without an answer when the
+URL has a query string or the request has a `Referer` header. So no query string
+is added, and no `Referer` header is sent, wherever the page is served from.
+`no-store` keeps the browser's cache out of the measurement. Each check times
+out after 80% of the refresh interval (24 seconds at 30 seconds) and is then
+recorded as a timeout, so a round's checks have all finished before the next
+round is due. When no WAN host answers, a recovery probe checks 4 WAN hosts,
+picked at random when it starts, every half second, giving up the checks it
+started half a second before. As soon as one answers, a new round starts at
+once, as it does after an interval change. A round started early gives up the
+last round's checks if they are still waiting, and that round records nothing
+more, so rounds never overlap. The browser chooses between IPv4 and IPv6 for
+each target, as for any request; the local targets are IPv4 addresses.
 
 Each recorded check that fails writes one line to the browser console with
 `console.error`, and the same line to the debug log: the target's name and URL,

TODO.md

@@ -23,6 +23,37 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of
 
 # Completed Steps
 
+- 2026-10-26: a target whose home page answers with a redirect is checked at
+  `/robots.txt` on the same host, which answers without one
+  ([#194](https://git.eeqj.de/sneak/netwatch/issues/194)). These are the AWS
+  Console, Akamai, B2 and the 8 S3 regions. The browser followed each redirect,
+  so their figures also timed the page it led to. The Google Cloud Console is
+  checked at `/robots.txt` too, as its home page redirects `curl`'s requests.
+  The S3 regions are checked with a HEAD request, as Chrome closes their
+  connection after most GET requests. `README.md` says what a figure includes
+- 2026-10-26: the six Hetzner targets answer wherever the page is served from
+  ([#191](https://git.eeqj.de/sneak/netwatch/issues/191)). They close the
+  connection without an answer when the request has a `Referer` header. The
+  browser sent one unless the page came with the image's
+  `Referrer-Policy: no-referrer` header, which a static host and `make dev` do
+  not send. Each check now sets `referrerPolicy: 'no-referrer'`
+- 2026-10-19: the server's `OnStart` hook in `internal/server/server.go` starts
+  `listenAndServe` with `go func() { s.listenAndServe() }()`, as the Go style
+  guide asks ([#162](https://git.eeqj.de/sneak/netwatch/issues/162)); it was
+  `go s.listenAndServe()`. No other `go` statement in the Go code called a
+  function directly. Nothing the server does changed
+- 2026-10-19: `.gitignore`, `.editorconfig` and `.golangci.yml` are the
+  sneak/prompts copies at `0653579`
+  ([#174](https://git.eeqj.de/sneak/netwatch/issues/174)). `.gitignore` is the
+  standard file followed by this repository's own entries; `.editorconfig` is
+  the standard file alone, which now carries the `[*.go]` section.
+  `.golangci.yml` disables `canonicalheader`, as the canonical copy does,
+  because golangci-lint v2.14.0 misses its findings at random
+- 2026-10-19: `package.json` has no `version` field and is marked
+  `"private": true` ([#158](https://git.eeqj.de/sneak/netwatch/issues/158)). It
+  said `1.0.0`, a release that has not been made, and nothing read it; the git
+  tag is the one version, which the image and `make build` take from
+  `git describe --tags`. The package is never published
 - 2026-10-19: Status names `v1.0.0-rc1`, the first release candidate, and says
   release candidates are tagged on `next`, where it said there were no git tags
   ([#157](https://git.eeqj.de/sneak/netwatch/issues/157)). The rest of Status

configs/security-headers.conf

@@ -8,10 +8,9 @@ add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" alway
 # Scripts and styles load only from the page's own origin. Inline ones
 # are blocked, style attributes in markup included, so style elements
 # through classes or element.style. data: images are for the favicon
-# in web/index.html. connect-src is * because the browser checks each
-# probe in web/src/main.js against it, and also every redirect the probe
-# follows, and several of those hosts redirect to others; a list of
-# hosts here would block those probes. It also covers the reports the
+# in web/index.html. connect-src is * because the page checks every
+# target in web/src/main.js. A list of hosts here would have to change
+# with every change to those targets. It also covers the reports the
 # page sends to its own origin.
 add_header Content-Security-Policy "default-src 'self'; connect-src *; img-src 'self' data:; object-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'" always;
 

internal/server/server.go

@@ -79,7 +79,7 @@ func New(
 			s.SetupRoutes()
 			s.httpServer = s.newHTTPServer()
 
-			go s.listenAndServe()
+			go func() { s.listenAndServe() }()
 
 			return nil
 		},

package.json

@@ -1,6 +1,6 @@
 {
     "name": "netwatch",
-    "version": "1.0.0",
+    "private": true,
     "description": "Real-time network latency monitor SPA",
     "type": "module",
     "scripts": {

test/unit/main.test.js

@@ -180,6 +180,91 @@ test("a check fetches the target's URL as written, with no query string added",
     );
 });
 
+test("a check sends no Referer header, wherever the page is served from", async (t) => {
+    mockTargets(t, () => 10);
+    const check = measureLatency("https://fsn1-speed.hetzner.com");
+    t.mock.timers.tick(10);
+    assert.notEqual(await settled(check), "still waiting");
+    assert.equal(
+        fetch.mock.calls[0].arguments[1].referrerPolicy,
+        "no-referrer",
+    );
+});
+
+// These targets' home pages answer with a redirect. A check follows it and
+// times the request to the new URL too, often on another host. The Google
+// Cloud Console redirects only requests that do not say they come from
+// another site, such as curl's. Each target is checked at /robots.txt on
+// its own host, which answers without a redirect. An S3 region answers it
+// with 403, and any answer counts as reachable.
+test("a target whose home page redirects is checked at /robots.txt on its own host", () => {
+    const checked = new Map(
+        new AppState([]).wan.map((host) => [host.name, host.url]),
+    );
+    for (const [name, url] of [
+        ["AWS Console", "https://console.aws.amazon.com/robots.txt"],
+        ["Google Cloud Console", "https://console.cloud.google.com/robots.txt"],
+        ["Akamai", "https://www.akamai.com/robots.txt"],
+        ["B2", "https://api.backblazeb2.com/robots.txt"],
+        [
+            "S3 af-south-1 (Cape Town)",
+            "https://s3.af-south-1.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 eu-west-2 (London)",
+            "https://s3.eu-west-2.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 me-south-1 (Bahrain)",
+            "https://s3.me-south-1.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 ap-northeast-1 (Tokyo)",
+            "https://s3.ap-northeast-1.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 ap-southeast-1 (Singapore)",
+            "https://s3.ap-southeast-1.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 ap-southeast-2 (Sydney)",
+            "https://s3.ap-southeast-2.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 us-west-2 (Oregon)",
+            "https://s3.us-west-2.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 sa-east-1 (São Paulo)",
+            "https://s3.sa-east-1.amazonaws.com/robots.txt",
+        ],
+    ]) {
+        assert.equal(checked.get(name), url);
+    }
+});
+
+// Chrome keeps the connection to an S3 region open after a HEAD check, but
+// closes it after most GET checks. The Hetzner servers do not answer HEAD.
+test("a round checks each S3 region with a HEAD request and every other target with a GET request", async (t) => {
+    const state = new AppState([
+        { name: "Local CPE", url: "http://192.168.100.1" },
+    ]);
+    mockTargets(t, () => 10);
+    const round = tick(state, new AbortController().signal);
+    t.mock.timers.tick(10);
+    assert.notEqual(await settled(round), "still waiting");
+    const methods = new Map(
+        fetch.mock.calls.map((call) => [
+            call.arguments[0],
+            call.arguments[1].method,
+        ]),
+    );
+    for (const host of state.allHosts) {
+        const method = host.name.startsWith("S3 ") ? "HEAD" : "GET";
+        assert.equal(methods.get(host.url), method, host.name);
+    }
+});
+
 test("at a 30000ms interval, a target answering after 1000ms shows in its row while another target's check is still waiting", async (t) => {
     CONFIG.updateInterval = 30000;
     const state = new AppState([

web/src/main.js

@@ -67,51 +67,66 @@ export const CONFIG = {
 
 // WAN endpoints to monitor. These are used for the aggregate health/stats
 // display (min/max/avg). Ordered: personal, LLM APIs, big-3 cloud, then
-// CDN/hosting/other.
+// CDN/hosting/other. A check follows a redirect and times the request to
+// the new URL too. So a target whose home page redirects is checked at
+// /robots.txt on its own host. That URL answers without a redirect.
 const WAN_HOSTS = [
     { name: "datavi.be", url: "https://datavi.be" },
     { name: "Anthropic API", url: "https://api.anthropic.com" },
     { name: "OpenAI API", url: "https://api.openai.com" },
-    { name: "AWS Console", url: "https://console.aws.amazon.com" },
-    { name: "Google Cloud Console", url: "https://console.cloud.google.com" },
+    { name: "AWS Console", url: "https://console.aws.amazon.com/robots.txt" },
+    {
+        name: "Google Cloud Console",
+        url: "https://console.cloud.google.com/robots.txt",
+    },
     { name: "Microsoft Azure", url: "https://portal.azure.com" },
     { name: "Cloudflare", url: "https://www.cloudflare.com" },
     { name: "Fastly CDN", url: "https://www.fastly.com" },
-    { name: "Akamai", url: "https://www.akamai.com" },
+    { name: "Akamai", url: "https://www.akamai.com/robots.txt" },
     { name: "Google", url: "https://www.google.com" },
     { name: "GitHub", url: "https://github.com" },
-    { name: "B2", url: "https://api.backblazeb2.com" },
+    { name: "B2", url: "https://api.backblazeb2.com/robots.txt" },
+    // Chrome keeps the connection to an S3 region open after a HEAD check,
+    // but closes it after most GET checks.
     {
         name: "S3 af-south-1 (Cape Town)",
-        url: "https://s3.af-south-1.amazonaws.com",
+        url: "https://s3.af-south-1.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 eu-west-2 (London)",
-        url: "https://s3.eu-west-2.amazonaws.com",
+        url: "https://s3.eu-west-2.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 me-south-1 (Bahrain)",
-        url: "https://s3.me-south-1.amazonaws.com",
+        url: "https://s3.me-south-1.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 ap-northeast-1 (Tokyo)",
-        url: "https://s3.ap-northeast-1.amazonaws.com",
+        url: "https://s3.ap-northeast-1.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 ap-southeast-1 (Singapore)",
-        url: "https://s3.ap-southeast-1.amazonaws.com",
+        url: "https://s3.ap-southeast-1.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 ap-southeast-2 (Sydney)",
-        url: "https://s3.ap-southeast-2.amazonaws.com",
+        url: "https://s3.ap-southeast-2.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 us-west-2 (Oregon)",
-        url: "https://s3.us-west-2.amazonaws.com",
+        url: "https://s3.us-west-2.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 sa-east-1 (São Paulo)",
-        url: "https://s3.sa-east-1.amazonaws.com",
+        url: "https://s3.sa-east-1.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     // Hetzner regional speed test servers — genuine per-DC endpoints
     {
@@ -271,6 +286,7 @@ export class HostState {
     constructor(host, pinned = false) {
         this.name = host.name;
         this.url = host.url;
+        this.method = host.method;
         // Each entry is either a check's result, { timestamp, latency,
         // error }, or a round skipped while paused, { timestamp,
         // latency: null, paused: true }.
@@ -567,16 +583,18 @@ export class Reporter {
 
 // --- Latency Measurement -----------------------------------------------------
 
-// Checks one target. The check times out after CONFIG.requestTimeout; the
-// caller can give it up sooner through the optional signal, which also ends
-// it as a timeout. A failed check's reason says what went wrong and after
-// how long; a check that answered has none.
+// Checks one target with the given HTTP method, GET when none is given. The
+// check times out after CONFIG.requestTimeout; the caller can give it up
+// sooner through the optional signal, which also ends it as a timeout. A
+// failed check's reason says what went wrong and after how long; a check
+// that answered has none.
 //
-// The URL is fetched as written, with nothing added to it: the Hetzner
-// speed-test servers close the connection without an answer when the URL
-// has a query string, and cache: "no-store" keeps the browser's cache out
-// of the measurement.
-export async function measureLatency(url, signal) {
+// The Hetzner speed-test servers close the connection without an answer
+// when the URL has a query string or the request has a Referer header. So
+// the URL is fetched as written, and referrerPolicy: "no-referrer" sends
+// no Referer header, wherever the page is served from. cache: "no-store"
+// keeps the browser's cache out of the measurement.
+export async function measureLatency(url, signal, method = "GET") {
     const controller = new AbortController();
     const timeoutId = setTimeout(
         () => controller.abort(),
@@ -588,9 +606,10 @@ export async function measureLatency(url, signal) {
 
     try {
         await fetch(url, {
-            method: "GET",
+            method,
             mode: "no-cors",
             cache: "no-store",
+            referrerPolicy: "no-referrer",
             signal: controller.signal,
         });
         const latency = Math.round(performance.now() - start);
@@ -1290,7 +1309,7 @@ export async function tick(state, signal, onOffline) {
     // the round started.
     await Promise.all(
         state.allHosts.map(async (host) => {
-            const r = await measureLatency(host.url, signal);
+            const r = await measureLatency(host.url, signal, host.method);
             if (state.paused || signal.aborted || state.tickCount === 0) {
                 return;
             }
@@ -1363,7 +1382,7 @@ function startRecoveryProbe(state, startRounds) {
         const checks = new AbortController();
         state._recoveryProbeChecks = checks;
         for (const host of canaries) {
-            measureLatency(host.url, checks.signal).then((r) => {
+            measureLatency(host.url, checks.signal, host.method).then((r) => {
                 if (r.error !== null || checks.signal.aborted) return;
                 log.notice("Recovery probe: connectivity detected");
                 stopRecoveryProbe(state);