From 8be53283d37b6b1aa343eb2ac1ebc3d2fe8a9eb9 Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Mon, 26 Oct 2026 01:36:08 +0000 Subject: [PATCH] Targets whose home page redirects are checked at /robots.txt (closes #194) The browser follows a redirect, so the figures for the AWS Console, Akamai, B2 and the S3 regions also timed the page it led to. Each of these targets is now checked at /robots.txt on its own host, which answers without a redirect. So is the Google Cloud Console, whose home page redirects curl's requests. The S3 regions are checked with a HEAD request, as Chrome closes their connection after most GET requests. README.md says what a figure includes. Model: opus-5-5 --- README.md | 46 ++++++++++++++-------- TODO.md | 8 ++++ configs/security-headers.conf | 7 ++-- test/unit/main.test.js | 74 +++++++++++++++++++++++++++++++++++ web/src/main.js | 59 ++++++++++++++++++---------- 5 files changed, 153 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index a64d869..6592f5b 100644 --- a/README.md +++ b/README.md @@ -413,22 +413,36 @@ Local hosts are tracked separately from WAN stats. ### Latency measurement -GET requests to each target's URL as written, with `mode: 'no-cors'`, -`cache: 'no-store'` and `referrerPolicy: 'no-referrer'`, timed with -`performance.now()`. The Hetzner speed-test servers close the connection without -an answer when the URL has a query string or the request has a `Referer` header. -So no query string is added, and no `Referer` header is sent, wherever the page -is served from. `no-store` keeps the browser's cache out of the measurement. -Each check times out after 80% of the refresh interval (24 seconds at 30 -seconds) and is then recorded as a timeout, so a round's checks have all -finished before the next round is due. When no WAN host answers, a recovery -probe checks 4 WAN hosts, picked at random when it starts, every half second, -giving up the checks it started half a second before. As soon as one answers, a -new round starts at once, as it does after an interval change. A round started -early gives up the last round's checks if they are still waiting, and that round -records nothing more, so rounds never overlap. The browser chooses between IPv4 -and IPv6 for each target, as for any request; the local targets are IPv4 -addresses. +Each check is a GET request to the target's URL as written, with +`mode: 'no-cors'`, `cache: 'no-store'` and `referrerPolicy: 'no-referrer'`. Its +figure is the time from sending the request until the answer's headers arrive, +timed with `performance.now()`. Unlike ping, this includes the time the server +takes to answer. It also includes the DNS lookup, connection setup and TLS +whenever the browser opens a new connection. Chrome opens one for every check of +the Hetzner speed-test servers, and for more targets at longer intervals. + +The S3 regions are checked with a HEAD request instead. Chrome keeps the +connection to an S3 region open after a HEAD request, but closes it after most +GET requests. The Hetzner speed-test servers do not answer a HEAD request. + +The browser follows every redirect, and the figure then includes the request to +the new URL too. So a target whose home page answers with a redirect is checked +at `/robots.txt` on the same host, which answers without one. An S3 region +answers it with 403, and any answer counts as reachable. + +The Hetzner speed-test servers close the connection without an answer when the +URL has a query string or the request has a `Referer` header. So no query string +is added, and no `Referer` header is sent, wherever the page is served from. +`no-store` keeps the browser's cache out of the measurement. Each check times +out after 80% of the refresh interval (24 seconds at 30 seconds) and is then +recorded as a timeout, so a round's checks have all finished before the next +round is due. When no WAN host answers, a recovery probe checks 4 WAN hosts, +picked at random when it starts, every half second, giving up the checks it +started half a second before. As soon as one answers, a new round starts at +once, as it does after an interval change. A round started early gives up the +last round's checks if they are still waiting, and that round records nothing +more, so rounds never overlap. The browser chooses between IPv4 and IPv6 for +each target, as for any request; the local targets are IPv4 addresses. Each recorded check that fails writes one line to the browser console with `console.error`, and the same line to the debug log: the target's name and URL, diff --git a/TODO.md b/TODO.md index 2bb64ce..5731196 100644 --- a/TODO.md +++ b/TODO.md @@ -23,6 +23,14 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of # Completed Steps +- 2026-10-26: a target whose home page answers with a redirect is checked at + `/robots.txt` on the same host, which answers without one + ([#194](https://git.eeqj.de/sneak/netwatch/issues/194)). These are the AWS + Console, Akamai, B2 and the 8 S3 regions. The browser followed each redirect, + so their figures also timed the page it led to. The Google Cloud Console is + checked at `/robots.txt` too, as its home page redirects `curl`'s requests. + The S3 regions are checked with a HEAD request, as Chrome closes their + connection after most GET requests. `README.md` says what a figure includes - 2026-10-26: the six Hetzner targets answer wherever the page is served from ([#191](https://git.eeqj.de/sneak/netwatch/issues/191)). They close the connection without an answer when the request has a `Referer` header. The diff --git a/configs/security-headers.conf b/configs/security-headers.conf index 4f6e413..bfdf8bd 100644 --- a/configs/security-headers.conf +++ b/configs/security-headers.conf @@ -8,10 +8,9 @@ add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" alway # Scripts and styles load only from the page's own origin. Inline ones # are blocked, style attributes in markup included, so style elements # through classes or element.style. data: images are for the favicon -# in web/index.html. connect-src is * because the browser checks each -# probe in web/src/main.js against it, and also every redirect the probe -# follows, and several of those hosts redirect to others; a list of -# hosts here would block those probes. It also covers the reports the +# in web/index.html. connect-src is * because the page checks every +# target in web/src/main.js. A list of hosts here would have to change +# with every change to those targets. It also covers the reports the # page sends to its own origin. add_header Content-Security-Policy "default-src 'self'; connect-src *; img-src 'self' data:; object-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'" always; diff --git a/test/unit/main.test.js b/test/unit/main.test.js index c10a492..d265b44 100644 --- a/test/unit/main.test.js +++ b/test/unit/main.test.js @@ -191,6 +191,80 @@ test("a check sends no Referer header, wherever the page is served from", async ); }); +// These targets' home pages answer with a redirect. A check follows it and +// times the request to the new URL too, often on another host. The Google +// Cloud Console redirects only requests that do not say they come from +// another site, such as curl's. Each target is checked at /robots.txt on +// its own host, which answers without a redirect. An S3 region answers it +// with 403, and any answer counts as reachable. +test("a target whose home page redirects is checked at /robots.txt on its own host", () => { + const checked = new Map( + new AppState([]).wan.map((host) => [host.name, host.url]), + ); + for (const [name, url] of [ + ["AWS Console", "https://console.aws.amazon.com/robots.txt"], + ["Google Cloud Console", "https://console.cloud.google.com/robots.txt"], + ["Akamai", "https://www.akamai.com/robots.txt"], + ["B2", "https://api.backblazeb2.com/robots.txt"], + [ + "S3 af-south-1 (Cape Town)", + "https://s3.af-south-1.amazonaws.com/robots.txt", + ], + [ + "S3 eu-west-2 (London)", + "https://s3.eu-west-2.amazonaws.com/robots.txt", + ], + [ + "S3 me-south-1 (Bahrain)", + "https://s3.me-south-1.amazonaws.com/robots.txt", + ], + [ + "S3 ap-northeast-1 (Tokyo)", + "https://s3.ap-northeast-1.amazonaws.com/robots.txt", + ], + [ + "S3 ap-southeast-1 (Singapore)", + "https://s3.ap-southeast-1.amazonaws.com/robots.txt", + ], + [ + "S3 ap-southeast-2 (Sydney)", + "https://s3.ap-southeast-2.amazonaws.com/robots.txt", + ], + [ + "S3 us-west-2 (Oregon)", + "https://s3.us-west-2.amazonaws.com/robots.txt", + ], + [ + "S3 sa-east-1 (São Paulo)", + "https://s3.sa-east-1.amazonaws.com/robots.txt", + ], + ]) { + assert.equal(checked.get(name), url); + } +}); + +// Chrome keeps the connection to an S3 region open after a HEAD check, but +// closes it after most GET checks. The Hetzner servers do not answer HEAD. +test("a round checks each S3 region with a HEAD request and every other target with a GET request", async (t) => { + const state = new AppState([ + { name: "Local CPE", url: "http://192.168.100.1" }, + ]); + mockTargets(t, () => 10); + const round = tick(state, new AbortController().signal); + t.mock.timers.tick(10); + assert.notEqual(await settled(round), "still waiting"); + const methods = new Map( + fetch.mock.calls.map((call) => [ + call.arguments[0], + call.arguments[1].method, + ]), + ); + for (const host of state.allHosts) { + const method = host.name.startsWith("S3 ") ? "HEAD" : "GET"; + assert.equal(methods.get(host.url), method, host.name); + } +}); + test("at a 30000ms interval, a target answering after 1000ms shows in its row while another target's check is still waiting", async (t) => { CONFIG.updateInterval = 30000; const state = new AppState([ diff --git a/web/src/main.js b/web/src/main.js index 99eacd9..85d16f1 100644 --- a/web/src/main.js +++ b/web/src/main.js @@ -67,51 +67,66 @@ export const CONFIG = { // WAN endpoints to monitor. These are used for the aggregate health/stats // display (min/max/avg). Ordered: personal, LLM APIs, big-3 cloud, then -// CDN/hosting/other. +// CDN/hosting/other. A check follows a redirect and times the request to +// the new URL too. So a target whose home page redirects is checked at +// /robots.txt on its own host. That URL answers without a redirect. const WAN_HOSTS = [ { name: "datavi.be", url: "https://datavi.be" }, { name: "Anthropic API", url: "https://api.anthropic.com" }, { name: "OpenAI API", url: "https://api.openai.com" }, - { name: "AWS Console", url: "https://console.aws.amazon.com" }, - { name: "Google Cloud Console", url: "https://console.cloud.google.com" }, + { name: "AWS Console", url: "https://console.aws.amazon.com/robots.txt" }, + { + name: "Google Cloud Console", + url: "https://console.cloud.google.com/robots.txt", + }, { name: "Microsoft Azure", url: "https://portal.azure.com" }, { name: "Cloudflare", url: "https://www.cloudflare.com" }, { name: "Fastly CDN", url: "https://www.fastly.com" }, - { name: "Akamai", url: "https://www.akamai.com" }, + { name: "Akamai", url: "https://www.akamai.com/robots.txt" }, { name: "Google", url: "https://www.google.com" }, { name: "GitHub", url: "https://github.com" }, - { name: "B2", url: "https://api.backblazeb2.com" }, + { name: "B2", url: "https://api.backblazeb2.com/robots.txt" }, + // Chrome keeps the connection to an S3 region open after a HEAD check, + // but closes it after most GET checks. { name: "S3 af-south-1 (Cape Town)", - url: "https://s3.af-south-1.amazonaws.com", + url: "https://s3.af-south-1.amazonaws.com/robots.txt", + method: "HEAD", }, { name: "S3 eu-west-2 (London)", - url: "https://s3.eu-west-2.amazonaws.com", + url: "https://s3.eu-west-2.amazonaws.com/robots.txt", + method: "HEAD", }, { name: "S3 me-south-1 (Bahrain)", - url: "https://s3.me-south-1.amazonaws.com", + url: "https://s3.me-south-1.amazonaws.com/robots.txt", + method: "HEAD", }, { name: "S3 ap-northeast-1 (Tokyo)", - url: "https://s3.ap-northeast-1.amazonaws.com", + url: "https://s3.ap-northeast-1.amazonaws.com/robots.txt", + method: "HEAD", }, { name: "S3 ap-southeast-1 (Singapore)", - url: "https://s3.ap-southeast-1.amazonaws.com", + url: "https://s3.ap-southeast-1.amazonaws.com/robots.txt", + method: "HEAD", }, { name: "S3 ap-southeast-2 (Sydney)", - url: "https://s3.ap-southeast-2.amazonaws.com", + url: "https://s3.ap-southeast-2.amazonaws.com/robots.txt", + method: "HEAD", }, { name: "S3 us-west-2 (Oregon)", - url: "https://s3.us-west-2.amazonaws.com", + url: "https://s3.us-west-2.amazonaws.com/robots.txt", + method: "HEAD", }, { name: "S3 sa-east-1 (São Paulo)", - url: "https://s3.sa-east-1.amazonaws.com", + url: "https://s3.sa-east-1.amazonaws.com/robots.txt", + method: "HEAD", }, // Hetzner regional speed test servers — genuine per-DC endpoints { @@ -271,6 +286,7 @@ export class HostState { constructor(host, pinned = false) { this.name = host.name; this.url = host.url; + this.method = host.method; // Each entry is either a check's result, { timestamp, latency, // error }, or a round skipped while paused, { timestamp, // latency: null, paused: true }. @@ -567,17 +583,18 @@ export class Reporter { // --- Latency Measurement ----------------------------------------------------- -// Checks one target. The check times out after CONFIG.requestTimeout; the -// caller can give it up sooner through the optional signal, which also ends -// it as a timeout. A failed check's reason says what went wrong and after -// how long; a check that answered has none. +// Checks one target with the given HTTP method, GET when none is given. The +// check times out after CONFIG.requestTimeout; the caller can give it up +// sooner through the optional signal, which also ends it as a timeout. A +// failed check's reason says what went wrong and after how long; a check +// that answered has none. // // The Hetzner speed-test servers close the connection without an answer // when the URL has a query string or the request has a Referer header. So // the URL is fetched as written, and referrerPolicy: "no-referrer" sends // no Referer header, wherever the page is served from. cache: "no-store" // keeps the browser's cache out of the measurement. -export async function measureLatency(url, signal) { +export async function measureLatency(url, signal, method = "GET") { const controller = new AbortController(); const timeoutId = setTimeout( () => controller.abort(), @@ -589,7 +606,7 @@ export async function measureLatency(url, signal) { try { await fetch(url, { - method: "GET", + method, mode: "no-cors", cache: "no-store", referrerPolicy: "no-referrer", @@ -1292,7 +1309,7 @@ export async function tick(state, signal, onOffline) { // the round started. await Promise.all( state.allHosts.map(async (host) => { - const r = await measureLatency(host.url, signal); + const r = await measureLatency(host.url, signal, host.method); if (state.paused || signal.aborted || state.tickCount === 0) { return; } @@ -1365,7 +1382,7 @@ function startRecoveryProbe(state, startRounds) { const checks = new AbortController(); state._recoveryProbeChecks = checks; for (const host of canaries) { - measureLatency(host.url, checks.signal).then((r) => { + measureLatency(host.url, checks.signal, host.method).then((r) => { if (r.error !== null || checks.signal.aborted) return; log.notice("Recovery probe: connectivity detected"); stopRecoveryProbe(state);