Targets whose home page redirects are checked at /robots.txt (closes #194)

The browser follows a redirect, so the figures for the AWS Console, Akamai,
B2 and the S3 regions also timed the page it led to. Each of these targets
is now checked at /robots.txt on its own host, which answers without a
redirect. So is the Google Cloud Console, whose home page redirects curl's
requests. The S3 regions are checked with a HEAD request, as Chrome closes
their connection after most GET requests. README.md says what a figure
includes.

Model: opus-5-5

README.md

@@ -419,7 +419,11 @@ figure is the time from sending the request until the answer's headers arrive,
 timed with `performance.now()`. Unlike ping, this includes the time the server
 takes to answer. It also includes the DNS lookup, connection setup and TLS
 whenever the browser opens a new connection. Chrome opens one for every check of
-the Hetzner speed-test servers and for most checks of the S3 regions.
+the Hetzner speed-test servers, and for more targets at longer intervals.
+
+The S3 regions are checked with a HEAD request instead. Chrome keeps the
+connection to an S3 region open after a HEAD request, but closes it after most
+GET requests. The Hetzner speed-test servers do not answer a HEAD request.
 
 The browser follows every redirect, and the figure then includes the request to
 the new URL too. So a target whose home page answers with a redirect is checked

TODO.md

@@ -26,15 +26,11 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of
 - 2026-10-26: a target whose home page answers with a redirect is checked at
   `/robots.txt` on the same host, which answers without one
   ([#194](https://git.eeqj.de/sneak/netwatch/issues/194)). These are the AWS
-  Console, Akamai, B2 and the 8 S3 regions, and the Google Cloud Console, whose
+  Console, Akamai, B2 and the 8 S3 regions. The browser followed each redirect,
-  home page redirects a request that does not say it comes from another site.
+  so their figures also timed the page it led to. The Google Cloud Console is
-  The browser follows a redirect, so their figures also timed the page it led
+  checked at `/robots.txt` too, as its home page redirects `curl`'s requests.
-  to, for B2 and the S3 regions on another host. Chrome opens a new connection
+  The S3 regions are checked with a HEAD request, as Chrome closes their
-  for most checks of an S3 region's `/robots.txt`, so those figures now include
+  connection after most GET requests. `README.md` says what a figure includes
-  connection setup and TLS, where they included a request to `aws.amazon.com`.
-  `README.md` says what a figure is: the time until the answer's headers arrive,
-  which includes the server's time to answer, and the DNS lookup, connection
-  setup and TLS whenever the browser opens a new connection
 - 2026-10-26: the six Hetzner targets answer wherever the page is served from
   ([#191](https://git.eeqj.de/sneak/netwatch/issues/191)). They close the
   connection without an answer when the request has a `Referer` header. The

configs/security-headers.conf

@@ -8,10 +8,9 @@ add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" alway
 # Scripts and styles load only from the page's own origin. Inline ones
 # are blocked, style attributes in markup included, so style elements
 # through classes or element.style. data: images are for the favicon
-# in web/index.html. connect-src is * because the browser checks each
+# in web/index.html. connect-src is * because the page checks every
-# probe in web/src/main.js against it, and also every redirect the probe
+# target in web/src/main.js. A list of hosts here would have to change
-# follows, and several of those hosts redirect to others; a list of
+# with every change to those targets. It also covers the reports the
-# hosts here would block those probes. It also covers the reports the
 # page sends to its own origin.
 add_header Content-Security-Policy "default-src 'self'; connect-src *; img-src 'self' data:; object-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'" always;
 

test/unit/main.test.js

@@ -191,12 +191,12 @@ test("a check sends no Referer header, wherever the page is served from", async
     );
 });
 
-// These targets' home pages answer with a redirect, the Google Cloud
+// These targets' home pages answer with a redirect. A check follows it and
-// Console's when the request does not say it comes from another site. A
+// times the request to the new URL too, often on another host. The Google
-// check follows it and times the request to the new URL too, often on
+// Cloud Console redirects only requests that do not say they come from
-// another host. Each is checked at /robots.txt on the host its name names,
+// another site, such as curl's. Each target is checked at /robots.txt on
-// which answers without a redirect: an S3 region with 403, which counts as
+// its own host, which answers without a redirect. An S3 region answers it
-// reachable.
+// with 403, and any answer counts as reachable.
 test("a target whose home page redirects is checked at /robots.txt on its own host", () => {
     const checked = new Map(
         new AppState([]).wan.map((host) => [host.name, host.url]),
@@ -243,6 +243,28 @@ test("a target whose home page redirects is checked at /robots.txt on its own ho
     }
 });
 
+// Chrome keeps the connection to an S3 region open after a HEAD check, but
+// closes it after most GET checks. The Hetzner servers do not answer HEAD.
+test("a round checks each S3 region with a HEAD request and every other target with a GET request", async (t) => {
+    const state = new AppState([
+        { name: "Local CPE", url: "http://192.168.100.1" },
+    ]);
+    mockTargets(t, () => 10);
+    const round = tick(state, new AbortController().signal);
+    t.mock.timers.tick(10);
+    assert.notEqual(await settled(round), "still waiting");
+    const methods = new Map(
+        fetch.mock.calls.map((call) => [
+            call.arguments[0],
+            call.arguments[1].method,
+        ]),
+    );
+    for (const host of state.allHosts) {
+        const method = host.name.startsWith("S3 ") ? "HEAD" : "GET";
+        assert.equal(methods.get(host.url), method, host.name);
+    }
+});
+
 test("at a 30000ms interval, a target answering after 1000ms shows in its row while another target's check is still waiting", async (t) => {
     CONFIG.updateInterval = 30000;
     const state = new AppState([

web/src/main.js

@@ -86,37 +86,47 @@ const WAN_HOSTS = [
     { name: "Google", url: "https://www.google.com" },
     { name: "GitHub", url: "https://github.com" },
     { name: "B2", url: "https://api.backblazeb2.com/robots.txt" },
+    // Chrome keeps the connection to an S3 region open after a HEAD check,
+    // but closes it after most GET checks.
     {
         name: "S3 af-south-1 (Cape Town)",
         url: "https://s3.af-south-1.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 eu-west-2 (London)",
         url: "https://s3.eu-west-2.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 me-south-1 (Bahrain)",
         url: "https://s3.me-south-1.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 ap-northeast-1 (Tokyo)",
         url: "https://s3.ap-northeast-1.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 ap-southeast-1 (Singapore)",
         url: "https://s3.ap-southeast-1.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 ap-southeast-2 (Sydney)",
         url: "https://s3.ap-southeast-2.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 us-west-2 (Oregon)",
         url: "https://s3.us-west-2.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     {
         name: "S3 sa-east-1 (São Paulo)",
         url: "https://s3.sa-east-1.amazonaws.com/robots.txt",
+        method: "HEAD",
     },
     // Hetzner regional speed test servers — genuine per-DC endpoints
     {
@@ -276,6 +286,7 @@ export class HostState {
     constructor(host, pinned = false) {
         this.name = host.name;
         this.url = host.url;
+        this.method = host.method;
         // Each entry is either a check's result, { timestamp, latency,
         // error }, or a round skipped while paused, { timestamp,
         // latency: null, paused: true }.
@@ -572,17 +583,18 @@ export class Reporter {
 
 // --- Latency Measurement -----------------------------------------------------
 
-// Checks one target. The check times out after CONFIG.requestTimeout; the
+// Checks one target with the given HTTP method, GET when none is given. The
-// caller can give it up sooner through the optional signal, which also ends
+// check times out after CONFIG.requestTimeout; the caller can give it up
-// it as a timeout. A failed check's reason says what went wrong and after
+// sooner through the optional signal, which also ends it as a timeout. A
-// how long; a check that answered has none.
+// failed check's reason says what went wrong and after how long; a check
+// that answered has none.
 //
 // The Hetzner speed-test servers close the connection without an answer
 // when the URL has a query string or the request has a Referer header. So
 // the URL is fetched as written, and referrerPolicy: "no-referrer" sends
 // no Referer header, wherever the page is served from. cache: "no-store"
 // keeps the browser's cache out of the measurement.
-export async function measureLatency(url, signal) {
+export async function measureLatency(url, signal, method = "GET") {
     const controller = new AbortController();
     const timeoutId = setTimeout(
         () => controller.abort(),
@@ -594,7 +606,7 @@ export async function measureLatency(url, signal) {
 
     try {
         await fetch(url, {
-            method: "GET",
+            method,
             mode: "no-cors",
             cache: "no-store",
             referrerPolicy: "no-referrer",
@@ -1297,7 +1309,7 @@ export async function tick(state, signal, onOffline) {
     // the round started.
     await Promise.all(
         state.allHosts.map(async (host) => {
-            const r = await measureLatency(host.url, signal);
+            const r = await measureLatency(host.url, signal, host.method);
             if (state.paused || signal.aborted || state.tickCount === 0) {
                 return;
             }
@@ -1370,7 +1382,7 @@ function startRecoveryProbe(state, startRounds) {
         const checks = new AbortController();
         state._recoveryProbeChecks = checks;
         for (const host of canaries) {
-            measureLatency(host.url, checks.signal).then((r) => {
+            measureLatency(host.url, checks.signal, host.method).then((r) => {
                 if (r.error !== null || checks.signal.aborted) return;
                 log.notice("Recovery probe: connectivity detected");
                 stopRecoveryProbe(state);