From fbd7e5fe85042f40b80c64bd93a6724dd0e03e46 Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Mon, 26 Oct 2026 01:36:08 +0000 Subject: [PATCH] Targets whose home page redirects are checked at /robots.txt (closes #190) The browser follows a redirect, so the figures for the AWS Console, Akamai, B2 and the S3 regions also timed the page it led to, for B2 and S3 on another host. The Google Cloud Console redirects only requests that do not say they come from another site; browsers say so and get a 403 directly. Each of these targets is now checked at /robots.txt on its own host, which answers without a redirect. README.md says what a figure includes. Chrome opens a new connection for most S3 checks, so far regions now read higher. Model: opus-5-5 --- README.md | 42 +++++++++++++++++++++------------- TODO.md | 12 ++++++++++ test/unit/main.test.js | 52 ++++++++++++++++++++++++++++++++++++++++++ web/src/main.js | 31 ++++++++++++++----------- 4 files changed, 108 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index a64d869..56e73d6 100644 --- a/README.md +++ b/README.md @@ -413,22 +413,32 @@ Local hosts are tracked separately from WAN stats. ### Latency measurement -GET requests to each target's URL as written, with `mode: 'no-cors'`, -`cache: 'no-store'` and `referrerPolicy: 'no-referrer'`, timed with -`performance.now()`. The Hetzner speed-test servers close the connection without -an answer when the URL has a query string or the request has a `Referer` header. -So no query string is added, and no `Referer` header is sent, wherever the page -is served from. `no-store` keeps the browser's cache out of the measurement. -Each check times out after 80% of the refresh interval (24 seconds at 30 -seconds) and is then recorded as a timeout, so a round's checks have all -finished before the next round is due. When no WAN host answers, a recovery -probe checks 4 WAN hosts, picked at random when it starts, every half second, -giving up the checks it started half a second before. As soon as one answers, a -new round starts at once, as it does after an interval change. A round started -early gives up the last round's checks if they are still waiting, and that round -records nothing more, so rounds never overlap. The browser chooses between IPv4 -and IPv6 for each target, as for any request; the local targets are IPv4 -addresses. +Each check is a GET request to the target's URL as written, with +`mode: 'no-cors'`, `cache: 'no-store'` and `referrerPolicy: 'no-referrer'`. Its +figure is the time from sending the request until the answer's headers arrive, +timed with `performance.now()`. Unlike ping, this includes the time the server +takes to answer. It also includes the DNS lookup, connection setup and TLS +whenever the browser opens a new connection. Chrome opens one for every check of +the Hetzner speed-test servers and for most checks of the S3 regions. + +The browser follows every redirect, and the figure then includes the request to +the new URL too. So a target whose home page answers with a redirect is checked +at `/robots.txt` on the same host, which answers without one. An S3 region +answers it with 403, and any answer counts as reachable. + +The Hetzner speed-test servers close the connection without an answer when the +URL has a query string or the request has a `Referer` header. So no query string +is added, and no `Referer` header is sent, wherever the page is served from. +`no-store` keeps the browser's cache out of the measurement. Each check times +out after 80% of the refresh interval (24 seconds at 30 seconds) and is then +recorded as a timeout, so a round's checks have all finished before the next +round is due. When no WAN host answers, a recovery probe checks 4 WAN hosts, +picked at random when it starts, every half second, giving up the checks it +started half a second before. As soon as one answers, a new round starts at +once, as it does after an interval change. A round started early gives up the +last round's checks if they are still waiting, and that round records nothing +more, so rounds never overlap. The browser chooses between IPv4 and IPv6 for +each target, as for any request; the local targets are IPv4 addresses. Each recorded check that fails writes one line to the browser console with `console.error`, and the same line to the debug log: the target's name and URL, diff --git a/TODO.md b/TODO.md index 2bb64ce..43dfe0b 100644 --- a/TODO.md +++ b/TODO.md @@ -23,6 +23,18 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of # Completed Steps +- 2026-10-26: a target whose home page answers with a redirect is checked at + `/robots.txt` on the same host, which answers without one + ([#190](https://git.eeqj.de/sneak/netwatch/issues/190)). These are the AWS + Console, Akamai, B2 and the 8 S3 regions, and the Google Cloud Console, whose + home page redirects a request that does not say it comes from another site. + The browser follows a redirect, so their figures also timed the page it led + to, for B2 and the S3 regions on another host. Chrome opens a new connection + for most checks of an S3 region's `/robots.txt`, so those figures now include + connection setup and TLS, where they included a request to `aws.amazon.com`. + `README.md` says what a figure is: the time until the answer's headers arrive, + which includes the server's time to answer, and the DNS lookup, connection + setup and TLS whenever the browser opens a new connection - 2026-10-26: the six Hetzner targets answer wherever the page is served from ([#191](https://git.eeqj.de/sneak/netwatch/issues/191)). They close the connection without an answer when the request has a `Referer` header. The diff --git a/test/unit/main.test.js b/test/unit/main.test.js index c10a492..5632484 100644 --- a/test/unit/main.test.js +++ b/test/unit/main.test.js @@ -191,6 +191,58 @@ test("a check sends no Referer header, wherever the page is served from", async ); }); +// These targets' home pages answer with a redirect, the Google Cloud +// Console's when the request does not say it comes from another site. A +// check follows it and times the request to the new URL too, often on +// another host. Each is checked at /robots.txt on the host its name names, +// which answers without a redirect: an S3 region with 403, which counts as +// reachable. +test("a target whose home page redirects is checked at /robots.txt on its own host", () => { + const checked = new Map( + new AppState([]).wan.map((host) => [host.name, host.url]), + ); + for (const [name, url] of [ + ["AWS Console", "https://console.aws.amazon.com/robots.txt"], + ["Google Cloud Console", "https://console.cloud.google.com/robots.txt"], + ["Akamai", "https://www.akamai.com/robots.txt"], + ["B2", "https://api.backblazeb2.com/robots.txt"], + [ + "S3 af-south-1 (Cape Town)", + "https://s3.af-south-1.amazonaws.com/robots.txt", + ], + [ + "S3 eu-west-2 (London)", + "https://s3.eu-west-2.amazonaws.com/robots.txt", + ], + [ + "S3 me-south-1 (Bahrain)", + "https://s3.me-south-1.amazonaws.com/robots.txt", + ], + [ + "S3 ap-northeast-1 (Tokyo)", + "https://s3.ap-northeast-1.amazonaws.com/robots.txt", + ], + [ + "S3 ap-southeast-1 (Singapore)", + "https://s3.ap-southeast-1.amazonaws.com/robots.txt", + ], + [ + "S3 ap-southeast-2 (Sydney)", + "https://s3.ap-southeast-2.amazonaws.com/robots.txt", + ], + [ + "S3 us-west-2 (Oregon)", + "https://s3.us-west-2.amazonaws.com/robots.txt", + ], + [ + "S3 sa-east-1 (São Paulo)", + "https://s3.sa-east-1.amazonaws.com/robots.txt", + ], + ]) { + assert.equal(checked.get(name), url); + } +}); + test("at a 30000ms interval, a target answering after 1000ms shows in its row while another target's check is still waiting", async (t) => { CONFIG.updateInterval = 30000; const state = new AppState([ diff --git a/web/src/main.js b/web/src/main.js index 99eacd9..5430a0f 100644 --- a/web/src/main.js +++ b/web/src/main.js @@ -67,51 +67,56 @@ export const CONFIG = { // WAN endpoints to monitor. These are used for the aggregate health/stats // display (min/max/avg). Ordered: personal, LLM APIs, big-3 cloud, then -// CDN/hosting/other. +// CDN/hosting/other. A check follows a redirect and times the request to +// the new URL too, so a target whose home page redirects is checked at +// /robots.txt, which its own host answers without one. const WAN_HOSTS = [ { name: "datavi.be", url: "https://datavi.be" }, { name: "Anthropic API", url: "https://api.anthropic.com" }, { name: "OpenAI API", url: "https://api.openai.com" }, - { name: "AWS Console", url: "https://console.aws.amazon.com" }, - { name: "Google Cloud Console", url: "https://console.cloud.google.com" }, + { name: "AWS Console", url: "https://console.aws.amazon.com/robots.txt" }, + { + name: "Google Cloud Console", + url: "https://console.cloud.google.com/robots.txt", + }, { name: "Microsoft Azure", url: "https://portal.azure.com" }, { name: "Cloudflare", url: "https://www.cloudflare.com" }, { name: "Fastly CDN", url: "https://www.fastly.com" }, - { name: "Akamai", url: "https://www.akamai.com" }, + { name: "Akamai", url: "https://www.akamai.com/robots.txt" }, { name: "Google", url: "https://www.google.com" }, { name: "GitHub", url: "https://github.com" }, - { name: "B2", url: "https://api.backblazeb2.com" }, + { name: "B2", url: "https://api.backblazeb2.com/robots.txt" }, { name: "S3 af-south-1 (Cape Town)", - url: "https://s3.af-south-1.amazonaws.com", + url: "https://s3.af-south-1.amazonaws.com/robots.txt", }, { name: "S3 eu-west-2 (London)", - url: "https://s3.eu-west-2.amazonaws.com", + url: "https://s3.eu-west-2.amazonaws.com/robots.txt", }, { name: "S3 me-south-1 (Bahrain)", - url: "https://s3.me-south-1.amazonaws.com", + url: "https://s3.me-south-1.amazonaws.com/robots.txt", }, { name: "S3 ap-northeast-1 (Tokyo)", - url: "https://s3.ap-northeast-1.amazonaws.com", + url: "https://s3.ap-northeast-1.amazonaws.com/robots.txt", }, { name: "S3 ap-southeast-1 (Singapore)", - url: "https://s3.ap-southeast-1.amazonaws.com", + url: "https://s3.ap-southeast-1.amazonaws.com/robots.txt", }, { name: "S3 ap-southeast-2 (Sydney)", - url: "https://s3.ap-southeast-2.amazonaws.com", + url: "https://s3.ap-southeast-2.amazonaws.com/robots.txt", }, { name: "S3 us-west-2 (Oregon)", - url: "https://s3.us-west-2.amazonaws.com", + url: "https://s3.us-west-2.amazonaws.com/robots.txt", }, { name: "S3 sa-east-1 (São Paulo)", - url: "https://s3.sa-east-1.amazonaws.com", + url: "https://s3.sa-east-1.amazonaws.com/robots.txt", }, // Hetzner regional speed test servers — genuine per-DC endpoints {