Targets whose home page redirects are checked at /robots.txt (closes #190)

The browser follows a redirect, so the figures for the AWS Console, Akamai,
B2 and the S3 regions also timed the page it led to, for B2 and S3 on
another host. The Google Cloud Console redirects only requests that do not
say they come from another site; browsers say so and get a 403 directly.
Each of these targets is now checked at /robots.txt on its own host, which
answers without a redirect. README.md says what a figure includes. Chrome
opens a new connection for most S3 checks, so far regions now read higher.

Model: opus-5-5

README.md

@@ -413,20 +413,32 @@ Local hosts are tracked separately from WAN stats.
 
 ### Latency measurement
 
-GET requests to each target's URL as written, with `mode: 'no-cors'` and
-`cache: 'no-store'`, timed with `performance.now()`. No query string is added:
-the Hetzner speed-test servers close the connection without an answer when the
-URL has one, and `no-store` keeps the browser's cache out of the measurement.
-Each check times out after 80% of the refresh interval (24 seconds at 30
-seconds) and is then recorded as a timeout, so a round's checks have all
-finished before the next round is due. When no WAN host answers, a recovery
-probe checks 4 WAN hosts, picked at random when it starts, every half second,
-giving up the checks it started half a second before. As soon as one answers, a
-new round starts at once, as it does after an interval change. A round started
-early gives up the last round's checks if they are still waiting, and that round
-records nothing more, so rounds never overlap. The browser chooses between IPv4
-and IPv6 for each target, as for any request; the local targets are IPv4
-addresses.
+Each check is a GET request to the target's URL as written, with
+`mode: 'no-cors'`, `cache: 'no-store'` and `referrerPolicy: 'no-referrer'`. Its
+figure is the time from sending the request until the answer's headers arrive,
+timed with `performance.now()`. Unlike ping, this includes the time the server
+takes to answer. It also includes the DNS lookup, connection setup and TLS
+whenever the browser opens a new connection. Chrome opens one for every check of
+the Hetzner speed-test servers and for most checks of the S3 regions.
+
+The browser follows every redirect, and the figure then includes the request to
+the new URL too. So a target whose home page answers with a redirect is checked
+at `/robots.txt` on the same host, which answers without one. An S3 region
+answers it with 403, and any answer counts as reachable.
+
+The Hetzner speed-test servers close the connection without an answer when the
+URL has a query string or the request has a `Referer` header. So no query string
+is added, and no `Referer` header is sent, wherever the page is served from.
+`no-store` keeps the browser's cache out of the measurement. Each check times
+out after 80% of the refresh interval (24 seconds at 30 seconds) and is then
+recorded as a timeout, so a round's checks have all finished before the next
+round is due. When no WAN host answers, a recovery probe checks 4 WAN hosts,
+picked at random when it starts, every half second, giving up the checks it
+started half a second before. As soon as one answers, a new round starts at
+once, as it does after an interval change. A round started early gives up the
+last round's checks if they are still waiting, and that round records nothing
+more, so rounds never overlap. The browser chooses between IPv4 and IPv6 for
+each target, as for any request; the local targets are IPv4 addresses.
 
 Each recorded check that fails writes one line to the browser console with
 `console.error`, and the same line to the debug log: the target's name and URL,

TODO.md

@@ -23,6 +23,24 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of
 
 # Completed Steps
 
+- 2026-10-26: a target whose home page answers with a redirect is checked at
+  `/robots.txt` on the same host, which answers without one
+  ([#190](https://git.eeqj.de/sneak/netwatch/issues/190)). These are the AWS
+  Console, Akamai, B2 and the 8 S3 regions, and the Google Cloud Console, whose
+  home page redirects a request that does not say it comes from another site.
+  The browser follows a redirect, so their figures also timed the page it led
+  to, for B2 and the S3 regions on another host. Chrome opens a new connection
+  for most checks of an S3 region's `/robots.txt`, so those figures now include
+  connection setup and TLS, where they included a request to `aws.amazon.com`.
+  `README.md` says what a figure is: the time until the answer's headers arrive,
+  which includes the server's time to answer, and the DNS lookup, connection
+  setup and TLS whenever the browser opens a new connection
+- 2026-10-26: the six Hetzner targets answer wherever the page is served from
+  ([#191](https://git.eeqj.de/sneak/netwatch/issues/191)). They close the
+  connection without an answer when the request has a `Referer` header. The
+  browser sent one unless the page came with the image's
+  `Referrer-Policy: no-referrer` header, which a static host and `make dev` do
+  not send. Each check now sets `referrerPolicy: 'no-referrer'`
 - 2026-10-19: the server's `OnStart` hook in `internal/server/server.go` starts
   `listenAndServe` with `go func() { s.listenAndServe() }()`, as the Go style
   guide asks ([#162](https://git.eeqj.de/sneak/netwatch/issues/162)); it was

test/unit/main.test.js

@@ -180,6 +180,69 @@ test("a check fetches the target's URL as written, with no query string added",
     );
 });
 
+test("a check sends no Referer header, wherever the page is served from", async (t) => {
+    mockTargets(t, () => 10);
+    const check = measureLatency("https://fsn1-speed.hetzner.com");
+    t.mock.timers.tick(10);
+    assert.notEqual(await settled(check), "still waiting");
+    assert.equal(
+        fetch.mock.calls[0].arguments[1].referrerPolicy,
+        "no-referrer",
+    );
+});
+
+// These targets' home pages answer with a redirect, the Google Cloud
+// Console's when the request does not say it comes from another site. A
+// check follows it and times the request to the new URL too, often on
+// another host. Each is checked at /robots.txt on the host its name names,
+// which answers without a redirect: an S3 region with 403, which counts as
+// reachable.
+test("a target whose home page redirects is checked at /robots.txt on its own host", () => {
+    const checked = new Map(
+        new AppState([]).wan.map((host) => [host.name, host.url]),
+    );
+    for (const [name, url] of [
+        ["AWS Console", "https://console.aws.amazon.com/robots.txt"],
+        ["Google Cloud Console", "https://console.cloud.google.com/robots.txt"],
+        ["Akamai", "https://www.akamai.com/robots.txt"],
+        ["B2", "https://api.backblazeb2.com/robots.txt"],
+        [
+            "S3 af-south-1 (Cape Town)",
+            "https://s3.af-south-1.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 eu-west-2 (London)",
+            "https://s3.eu-west-2.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 me-south-1 (Bahrain)",
+            "https://s3.me-south-1.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 ap-northeast-1 (Tokyo)",
+            "https://s3.ap-northeast-1.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 ap-southeast-1 (Singapore)",
+            "https://s3.ap-southeast-1.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 ap-southeast-2 (Sydney)",
+            "https://s3.ap-southeast-2.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 us-west-2 (Oregon)",
+            "https://s3.us-west-2.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 sa-east-1 (São Paulo)",
+            "https://s3.sa-east-1.amazonaws.com/robots.txt",
+        ],
+    ]) {
+        assert.equal(checked.get(name), url);
+    }
+});
+
 test("at a 30000ms interval, a target answering after 1000ms shows in its row while another target's check is still waiting", async (t) => {
     CONFIG.updateInterval = 30000;
     const state = new AppState([

web/src/main.js

@@ -67,51 +67,56 @@ export const CONFIG = {
 
 // WAN endpoints to monitor. These are used for the aggregate health/stats
 // display (min/max/avg). Ordered: personal, LLM APIs, big-3 cloud, then
-// CDN/hosting/other.
+// CDN/hosting/other. A check follows a redirect and times the request to
+// the new URL too, so a target whose home page redirects is checked at
+// /robots.txt, which its own host answers without one.
 const WAN_HOSTS = [
     { name: "datavi.be", url: "https://datavi.be" },
     { name: "Anthropic API", url: "https://api.anthropic.com" },
     { name: "OpenAI API", url: "https://api.openai.com" },
-    { name: "AWS Console", url: "https://console.aws.amazon.com" },
-    { name: "Google Cloud Console", url: "https://console.cloud.google.com" },
+    { name: "AWS Console", url: "https://console.aws.amazon.com/robots.txt" },
+    {
+        name: "Google Cloud Console",
+        url: "https://console.cloud.google.com/robots.txt",
+    },
     { name: "Microsoft Azure", url: "https://portal.azure.com" },
     { name: "Cloudflare", url: "https://www.cloudflare.com" },
     { name: "Fastly CDN", url: "https://www.fastly.com" },
-    { name: "Akamai", url: "https://www.akamai.com" },
+    { name: "Akamai", url: "https://www.akamai.com/robots.txt" },
     { name: "Google", url: "https://www.google.com" },
     { name: "GitHub", url: "https://github.com" },
-    { name: "B2", url: "https://api.backblazeb2.com" },
+    { name: "B2", url: "https://api.backblazeb2.com/robots.txt" },
     {
         name: "S3 af-south-1 (Cape Town)",
-        url: "https://s3.af-south-1.amazonaws.com",
+        url: "https://s3.af-south-1.amazonaws.com/robots.txt",
     },
     {
         name: "S3 eu-west-2 (London)",
-        url: "https://s3.eu-west-2.amazonaws.com",
+        url: "https://s3.eu-west-2.amazonaws.com/robots.txt",
     },
     {
         name: "S3 me-south-1 (Bahrain)",
-        url: "https://s3.me-south-1.amazonaws.com",
+        url: "https://s3.me-south-1.amazonaws.com/robots.txt",
     },
     {
         name: "S3 ap-northeast-1 (Tokyo)",
-        url: "https://s3.ap-northeast-1.amazonaws.com",
+        url: "https://s3.ap-northeast-1.amazonaws.com/robots.txt",
     },
     {
         name: "S3 ap-southeast-1 (Singapore)",
-        url: "https://s3.ap-southeast-1.amazonaws.com",
+        url: "https://s3.ap-southeast-1.amazonaws.com/robots.txt",
     },
     {
         name: "S3 ap-southeast-2 (Sydney)",
-        url: "https://s3.ap-southeast-2.amazonaws.com",
+        url: "https://s3.ap-southeast-2.amazonaws.com/robots.txt",
     },
     {
         name: "S3 us-west-2 (Oregon)",
-        url: "https://s3.us-west-2.amazonaws.com",
+        url: "https://s3.us-west-2.amazonaws.com/robots.txt",
     },
     {
         name: "S3 sa-east-1 (São Paulo)",
-        url: "https://s3.sa-east-1.amazonaws.com",
+        url: "https://s3.sa-east-1.amazonaws.com/robots.txt",
     },
     // Hetzner regional speed test servers — genuine per-DC endpoints
     {
@@ -572,10 +577,11 @@ export class Reporter {
 // it as a timeout. A failed check's reason says what went wrong and after
 // how long; a check that answered has none.
 //
-// The URL is fetched as written, with nothing added to it: the Hetzner
-// speed-test servers close the connection without an answer when the URL
-// has a query string, and cache: "no-store" keeps the browser's cache out
-// of the measurement.
+// The Hetzner speed-test servers close the connection without an answer
+// when the URL has a query string or the request has a Referer header. So
+// the URL is fetched as written, and referrerPolicy: "no-referrer" sends
+// no Referer header, wherever the page is served from. cache: "no-store"
+// keeps the browser's cache out of the measurement.
 export async function measureLatency(url, signal) {
     const controller = new AbortController();
     const timeoutId = setTimeout(
@@ -591,6 +597,7 @@ export async function measureLatency(url, signal) {
             method: "GET",
             mode: "no-cors",
             cache: "no-store",
+            referrerPolicy: "no-referrer",
             signal: controller.signal,
         });
         const latency = Math.round(performance.now() - start);