diff --git a/README.md b/README.md index a64d869..56e73d6 100644 --- a/README.md +++ b/README.md @@ -413,22 +413,32 @@ Local hosts are tracked separately from WAN stats. ### Latency measurement -GET requests to each target's URL as written, with `mode: 'no-cors'`, -`cache: 'no-store'` and `referrerPolicy: 'no-referrer'`, timed with -`performance.now()`. The Hetzner speed-test servers close the connection without -an answer when the URL has a query string or the request has a `Referer` header. -So no query string is added, and no `Referer` header is sent, wherever the page -is served from. `no-store` keeps the browser's cache out of the measurement. -Each check times out after 80% of the refresh interval (24 seconds at 30 -seconds) and is then recorded as a timeout, so a round's checks have all -finished before the next round is due. When no WAN host answers, a recovery -probe checks 4 WAN hosts, picked at random when it starts, every half second, -giving up the checks it started half a second before. As soon as one answers, a -new round starts at once, as it does after an interval change. A round started -early gives up the last round's checks if they are still waiting, and that round -records nothing more, so rounds never overlap. The browser chooses between IPv4 -and IPv6 for each target, as for any request; the local targets are IPv4 -addresses. +Each check is a GET request to the target's URL as written, with +`mode: 'no-cors'`, `cache: 'no-store'` and `referrerPolicy: 'no-referrer'`. Its +figure is the time from sending the request until the answer's headers arrive, +timed with `performance.now()`. Unlike ping, this includes the time the server +takes to answer. It also includes the DNS lookup, connection setup and TLS +whenever the browser opens a new connection. Chrome opens one for every check of +the Hetzner speed-test servers and for most checks of the S3 regions. + +The browser follows every redirect, and the figure then includes the request to +the new URL too. So a target whose home page answers with a redirect is checked +at `/robots.txt` on the same host, which answers without one. An S3 region +answers it with 403, and any answer counts as reachable. + +The Hetzner speed-test servers close the connection without an answer when the +URL has a query string or the request has a `Referer` header. So no query string +is added, and no `Referer` header is sent, wherever the page is served from. +`no-store` keeps the browser's cache out of the measurement. Each check times +out after 80% of the refresh interval (24 seconds at 30 seconds) and is then +recorded as a timeout, so a round's checks have all finished before the next +round is due. When no WAN host answers, a recovery probe checks 4 WAN hosts, +picked at random when it starts, every half second, giving up the checks it +started half a second before. As soon as one answers, a new round starts at +once, as it does after an interval change. A round started early gives up the +last round's checks if they are still waiting, and that round records nothing +more, so rounds never overlap. The browser chooses between IPv4 and IPv6 for +each target, as for any request; the local targets are IPv4 addresses. Each recorded check that fails writes one line to the browser console with `console.error`, and the same line to the debug log: the target's name and URL, diff --git a/TODO.md b/TODO.md index 2bb64ce..43dfe0b 100644 --- a/TODO.md +++ b/TODO.md @@ -23,6 +23,18 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of # Completed Steps +- 2026-10-26: a target whose home page answers with a redirect is checked at + `/robots.txt` on the same host, which answers without one + ([#190](https://git.eeqj.de/sneak/netwatch/issues/190)). These are the AWS + Console, Akamai, B2 and the 8 S3 regions, and the Google Cloud Console, whose + home page redirects a request that does not say it comes from another site. + The browser follows a redirect, so their figures also timed the page it led + to, for B2 and the S3 regions on another host. Chrome opens a new connection + for most checks of an S3 region's `/robots.txt`, so those figures now include + connection setup and TLS, where they included a request to `aws.amazon.com`. + `README.md` says what a figure is: the time until the answer's headers arrive, + which includes the server's time to answer, and the DNS lookup, connection + setup and TLS whenever the browser opens a new connection - 2026-10-26: the six Hetzner targets answer wherever the page is served from ([#191](https://git.eeqj.de/sneak/netwatch/issues/191)). They close the connection without an answer when the request has a `Referer` header. The diff --git a/test/unit/main.test.js b/test/unit/main.test.js index c10a492..5632484 100644 --- a/test/unit/main.test.js +++ b/test/unit/main.test.js @@ -191,6 +191,58 @@ test("a check sends no Referer header, wherever the page is served from", async ); }); +// These targets' home pages answer with a redirect, the Google Cloud +// Console's when the request does not say it comes from another site. A +// check follows it and times the request to the new URL too, often on +// another host. Each is checked at /robots.txt on the host its name names, +// which answers without a redirect: an S3 region with 403, which counts as +// reachable. +test("a target whose home page redirects is checked at /robots.txt on its own host", () => { + const checked = new Map( + new AppState([]).wan.map((host) => [host.name, host.url]), + ); + for (const [name, url] of [ + ["AWS Console", "https://console.aws.amazon.com/robots.txt"], + ["Google Cloud Console", "https://console.cloud.google.com/robots.txt"], + ["Akamai", "https://www.akamai.com/robots.txt"], + ["B2", "https://api.backblazeb2.com/robots.txt"], + [ + "S3 af-south-1 (Cape Town)", + "https://s3.af-south-1.amazonaws.com/robots.txt", + ], + [ + "S3 eu-west-2 (London)", + "https://s3.eu-west-2.amazonaws.com/robots.txt", + ], + [ + "S3 me-south-1 (Bahrain)", + "https://s3.me-south-1.amazonaws.com/robots.txt", + ], + [ + "S3 ap-northeast-1 (Tokyo)", + "https://s3.ap-northeast-1.amazonaws.com/robots.txt", + ], + [ + "S3 ap-southeast-1 (Singapore)", + "https://s3.ap-southeast-1.amazonaws.com/robots.txt", + ], + [ + "S3 ap-southeast-2 (Sydney)", + "https://s3.ap-southeast-2.amazonaws.com/robots.txt", + ], + [ + "S3 us-west-2 (Oregon)", + "https://s3.us-west-2.amazonaws.com/robots.txt", + ], + [ + "S3 sa-east-1 (São Paulo)", + "https://s3.sa-east-1.amazonaws.com/robots.txt", + ], + ]) { + assert.equal(checked.get(name), url); + } +}); + test("at a 30000ms interval, a target answering after 1000ms shows in its row while another target's check is still waiting", async (t) => { CONFIG.updateInterval = 30000; const state = new AppState([ diff --git a/web/src/main.js b/web/src/main.js index 99eacd9..5430a0f 100644 --- a/web/src/main.js +++ b/web/src/main.js @@ -67,51 +67,56 @@ export const CONFIG = { // WAN endpoints to monitor. These are used for the aggregate health/stats // display (min/max/avg). Ordered: personal, LLM APIs, big-3 cloud, then -// CDN/hosting/other. +// CDN/hosting/other. A check follows a redirect and times the request to +// the new URL too, so a target whose home page redirects is checked at +// /robots.txt, which its own host answers without one. const WAN_HOSTS = [ { name: "datavi.be", url: "https://datavi.be" }, { name: "Anthropic API", url: "https://api.anthropic.com" }, { name: "OpenAI API", url: "https://api.openai.com" }, - { name: "AWS Console", url: "https://console.aws.amazon.com" }, - { name: "Google Cloud Console", url: "https://console.cloud.google.com" }, + { name: "AWS Console", url: "https://console.aws.amazon.com/robots.txt" }, + { + name: "Google Cloud Console", + url: "https://console.cloud.google.com/robots.txt", + }, { name: "Microsoft Azure", url: "https://portal.azure.com" }, { name: "Cloudflare", url: "https://www.cloudflare.com" }, { name: "Fastly CDN", url: "https://www.fastly.com" }, - { name: "Akamai", url: "https://www.akamai.com" }, + { name: "Akamai", url: "https://www.akamai.com/robots.txt" }, { name: "Google", url: "https://www.google.com" }, { name: "GitHub", url: "https://github.com" }, - { name: "B2", url: "https://api.backblazeb2.com" }, + { name: "B2", url: "https://api.backblazeb2.com/robots.txt" }, { name: "S3 af-south-1 (Cape Town)", - url: "https://s3.af-south-1.amazonaws.com", + url: "https://s3.af-south-1.amazonaws.com/robots.txt", }, { name: "S3 eu-west-2 (London)", - url: "https://s3.eu-west-2.amazonaws.com", + url: "https://s3.eu-west-2.amazonaws.com/robots.txt", }, { name: "S3 me-south-1 (Bahrain)", - url: "https://s3.me-south-1.amazonaws.com", + url: "https://s3.me-south-1.amazonaws.com/robots.txt", }, { name: "S3 ap-northeast-1 (Tokyo)", - url: "https://s3.ap-northeast-1.amazonaws.com", + url: "https://s3.ap-northeast-1.amazonaws.com/robots.txt", }, { name: "S3 ap-southeast-1 (Singapore)", - url: "https://s3.ap-southeast-1.amazonaws.com", + url: "https://s3.ap-southeast-1.amazonaws.com/robots.txt", }, { name: "S3 ap-southeast-2 (Sydney)", - url: "https://s3.ap-southeast-2.amazonaws.com", + url: "https://s3.ap-southeast-2.amazonaws.com/robots.txt", }, { name: "S3 us-west-2 (Oregon)", - url: "https://s3.us-west-2.amazonaws.com", + url: "https://s3.us-west-2.amazonaws.com/robots.txt", }, { name: "S3 sa-east-1 (São Paulo)", - url: "https://s3.sa-east-1.amazonaws.com", + url: "https://s3.sa-east-1.amazonaws.com/robots.txt", }, // Hetzner regional speed test servers — genuine per-DC endpoints {