Targets whose home page redirects are checked at /robots.txt (closes #194)

The browser follows a redirect, so the figures for the AWS Console, Akamai,
B2 and the S3 regions also timed the page it led to, for B2 and S3 on
another host. The Google Cloud Console redirects only requests that do not
say they come from another site; browsers say so and get a 403 directly.
Each of these targets is now checked at /robots.txt on its own host, which
answers without a redirect. README.md says what a figure includes. Chrome
opens a new connection for most S3 checks, so far regions now read higher.

Model: opus-5-5

README.md

@@ -419,11 +419,7 @@ figure is the time from sending the request until the answer's headers arrive,
 timed with `performance.now()`. Unlike ping, this includes the time the server
 takes to answer. It also includes the DNS lookup, connection setup and TLS
 whenever the browser opens a new connection. Chrome opens one for every check of
-the Hetzner speed-test servers, and for more targets at longer intervals.
-
-The S3 regions are checked with a HEAD request instead. Chrome keeps the
-connection to an S3 region open after a HEAD request, but closes it after most
-GET requests. The Hetzner speed-test servers do not answer a HEAD request.
+the Hetzner speed-test servers and for most checks of the S3 regions.
 
 The browser follows every redirect, and the figure then includes the request to
 the new URL too. So a target whose home page answers with a redirect is checked

TODO.md

@@ -26,11 +26,15 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of
 - 2026-10-26: a target whose home page answers with a redirect is checked at
   `/robots.txt` on the same host, which answers without one
   ([#194](https://git.eeqj.de/sneak/netwatch/issues/194)). These are the AWS
-  Console, Akamai, B2 and the 8 S3 regions. The browser followed each redirect,
-  so their figures also timed the page it led to. The Google Cloud Console is
-  checked at `/robots.txt` too, as its home page redirects `curl`'s requests.
-  The S3 regions are checked with a HEAD request, as Chrome closes their
-  connection after most GET requests. `README.md` says what a figure includes
+  Console, Akamai, B2 and the 8 S3 regions, and the Google Cloud Console, whose
+  home page redirects a request that does not say it comes from another site.
+  The browser follows a redirect, so their figures also timed the page it led
+  to, for B2 and the S3 regions on another host. Chrome opens a new connection
+  for most checks of an S3 region's `/robots.txt`, so those figures now include
+  connection setup and TLS, where they included a request to `aws.amazon.com`.
+  `README.md` says what a figure is: the time until the answer's headers arrive,
+  which includes the server's time to answer, and the DNS lookup, connection
+  setup and TLS whenever the browser opens a new connection
 - 2026-10-26: the six Hetzner targets answer wherever the page is served from
   ([#191](https://git.eeqj.de/sneak/netwatch/issues/191)). They close the
   connection without an answer when the request has a `Referer` header. The

configs/security-headers.conf

@@ -8,9 +8,10 @@ add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" alway
 # Scripts and styles load only from the page's own origin. Inline ones
 # are blocked, style attributes in markup included, so style elements
 # through classes or element.style. data: images are for the favicon
-# in web/index.html. connect-src is * because the page checks every
-# target in web/src/main.js. A list of hosts here would have to change
-# with every change to those targets. It also covers the reports the
+# in web/index.html. connect-src is * because the browser checks each
+# probe in web/src/main.js against it, and also every redirect the probe
+# follows, and several of those hosts redirect to others; a list of
+# hosts here would block those probes. It also covers the reports the
 # page sends to its own origin.
 add_header Content-Security-Policy "default-src 'self'; connect-src *; img-src 'self' data:; object-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'" always;
 

test/unit/main.test.js

@@ -191,12 +191,12 @@ test("a check sends no Referer header, wherever the page is served from", async
     );
 });
 
-// These targets' home pages answer with a redirect. A check follows it and
-// times the request to the new URL too, often on another host. The Google
-// Cloud Console redirects only requests that do not say they come from
-// another site, such as curl's. Each target is checked at /robots.txt on
-// its own host, which answers without a redirect. An S3 region answers it
-// with 403, and any answer counts as reachable.
+// These targets' home pages answer with a redirect, the Google Cloud
+// Console's when the request does not say it comes from another site. A
+// check follows it and times the request to the new URL too, often on
+// another host. Each is checked at /robots.txt on the host its name names,
+// which answers without a redirect: an S3 region with 403, which counts as
+// reachable.
 test("a target whose home page redirects is checked at /robots.txt on its own host", () => {
     const checked = new Map(
         new AppState([]).wan.map((host) => [host.name, host.url]),
@@ -243,28 +243,6 @@ test("a target whose home page redirects is checked at /robots.txt on its own ho
     }
 });
 
-// Chrome keeps the connection to an S3 region open after a HEAD check, but
-// closes it after most GET checks. The Hetzner servers do not answer HEAD.
-test("a round checks each S3 region with a HEAD request and every other target with a GET request", async (t) => {
-    const state = new AppState([
-        { name: "Local CPE", url: "http://192.168.100.1" },
-    ]);
-    mockTargets(t, () => 10);
-    const round = tick(state, new AbortController().signal);
-    t.mock.timers.tick(10);
-    assert.notEqual(await settled(round), "still waiting");
-    const methods = new Map(
-        fetch.mock.calls.map((call) => [
-            call.arguments[0],
-            call.arguments[1].method,
-        ]),
-    );
-    for (const host of state.allHosts) {
-        const method = host.name.startsWith("S3 ") ? "HEAD" : "GET";
-        assert.equal(methods.get(host.url), method, host.name);
-    }
-});
-
 test("at a 30000ms interval, a target answering after 1000ms shows in its row while another target's check is still waiting", async (t) => {
     CONFIG.updateInterval = 30000;
     const state = new AppState([

web/src/main.js

@@ -86,47 +86,37 @@ const WAN_HOSTS = [
     { name: "Google", url: "https://www.google.com" },
     { name: "GitHub", url: "https://github.com" },
     { name: "B2", url: "https://api.backblazeb2.com/robots.txt" },
-    // Chrome keeps the connection to an S3 region open after a HEAD check,
-    // but closes it after most GET checks.
     {
         name: "S3 af-south-1 (Cape Town)",
         url: "https://s3.af-south-1.amazonaws.com/robots.txt",
-        method: "HEAD",
     },
     {
         name: "S3 eu-west-2 (London)",
         url: "https://s3.eu-west-2.amazonaws.com/robots.txt",
-        method: "HEAD",
     },
     {
         name: "S3 me-south-1 (Bahrain)",
         url: "https://s3.me-south-1.amazonaws.com/robots.txt",
-        method: "HEAD",
     },
     {
         name: "S3 ap-northeast-1 (Tokyo)",
         url: "https://s3.ap-northeast-1.amazonaws.com/robots.txt",
-        method: "HEAD",
     },
     {
         name: "S3 ap-southeast-1 (Singapore)",
         url: "https://s3.ap-southeast-1.amazonaws.com/robots.txt",
-        method: "HEAD",
     },
     {
         name: "S3 ap-southeast-2 (Sydney)",
         url: "https://s3.ap-southeast-2.amazonaws.com/robots.txt",
-        method: "HEAD",
     },
     {
         name: "S3 us-west-2 (Oregon)",
         url: "https://s3.us-west-2.amazonaws.com/robots.txt",
-        method: "HEAD",
     },
     {
         name: "S3 sa-east-1 (São Paulo)",
         url: "https://s3.sa-east-1.amazonaws.com/robots.txt",
-        method: "HEAD",
     },
     // Hetzner regional speed test servers — genuine per-DC endpoints
     {
@@ -286,7 +276,6 @@ export class HostState {
     constructor(host, pinned = false) {
         this.name = host.name;
         this.url = host.url;
-        this.method = host.method;
         // Each entry is either a check's result, { timestamp, latency,
         // error }, or a round skipped while paused, { timestamp,
         // latency: null, paused: true }.
@@ -583,18 +572,17 @@ export class Reporter {
 
 // --- Latency Measurement -----------------------------------------------------
 
-// Checks one target with the given HTTP method, GET when none is given. The
-// check times out after CONFIG.requestTimeout; the caller can give it up
-// sooner through the optional signal, which also ends it as a timeout. A
-// failed check's reason says what went wrong and after how long; a check
-// that answered has none.
+// Checks one target. The check times out after CONFIG.requestTimeout; the
+// caller can give it up sooner through the optional signal, which also ends
+// it as a timeout. A failed check's reason says what went wrong and after
+// how long; a check that answered has none.
 //
 // The Hetzner speed-test servers close the connection without an answer
 // when the URL has a query string or the request has a Referer header. So
 // the URL is fetched as written, and referrerPolicy: "no-referrer" sends
 // no Referer header, wherever the page is served from. cache: "no-store"
 // keeps the browser's cache out of the measurement.
-export async function measureLatency(url, signal, method = "GET") {
+export async function measureLatency(url, signal) {
     const controller = new AbortController();
     const timeoutId = setTimeout(
         () => controller.abort(),
@@ -606,7 +594,7 @@ export async function measureLatency(url, signal, method = "GET") {
 
     try {
         await fetch(url, {
-            method,
+            method: "GET",
             mode: "no-cors",
             cache: "no-store",
             referrerPolicy: "no-referrer",
@@ -1309,7 +1297,7 @@ export async function tick(state, signal, onOffline) {
     // the round started.
     await Promise.all(
         state.allHosts.map(async (host) => {
-            const r = await measureLatency(host.url, signal, host.method);
+            const r = await measureLatency(host.url, signal);
             if (state.paused || signal.aborted || state.tickCount === 0) {
                 return;
             }
@@ -1382,7 +1370,7 @@ function startRecoveryProbe(state, startRounds) {
         const checks = new AbortController();
         state._recoveryProbeChecks = checks;
         for (const host of canaries) {
-            measureLatency(host.url, checks.signal, host.method).then((r) => {
+            measureLatency(host.url, checks.signal).then((r) => {
                 if (r.error !== null || checks.signal.aborted) return;
                 log.notice("Recovery probe: connectivity detected");
                 stopRecoveryProbe(state);