sneak/netwatch
1
0
Fork 0
Code Issues 3 Pull Requests 3 Actions 7 Packages Projects Releases Wiki Activity

Targets whose home page redirects are checked at /robots.txt (closes #194)
check / check (push) Canceled after 0s
Details

Browse Source 
The browser follows a redirect, so the figures for the AWS Console, Akamai,
B2 and the S3 regions also timed the page it led to, for B2 and S3 on
another host. The Google Cloud Console redirects only requests that do not
say they come from another site; browsers say so and get a 403 directly.
Each of these targets is now checked at /robots.txt on its own host, which
answers without a redirect. README.md says what a figure includes. Chrome
opens a new connection for most S3 checks, so far regions now read higher.

Model: opus-5-5
This commit is contained in:
clawbot
2026-10-26 01:42:36 +00:00
parent a72797a403
commit f7b5acf210
4 changed files with 108 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+26 -16
View File
@@ -413,22 +413,32 @@ Local hosts are tracked separately from WAN stats.
### Latency measurement
GET requests to each target's URL as written, with `mode: 'no-cors'`,
`cache: 'no-store'` and `referrerPolicy: 'no-referrer'`, timed with
`performance.now()`. The Hetzner speed-test servers close the connection without
an answer when the URL has a query string or the request has a `Referer` header.
So no query string is added, and no `Referer` header is sent, wherever the page
is served from. `no-store` keeps the browser's cache out of the measurement.
Each check times out after 80% of the refresh interval (24 seconds at 30
seconds) and is then recorded as a timeout, so a round's checks have all
finished before the next round is due. When no WAN host answers, a recovery
probe checks 4 WAN hosts, picked at random when it starts, every half second,
giving up the checks it started half a second before. As soon as one answers, a
new round starts at once, as it does after an interval change. A round started
early gives up the last round's checks if they are still waiting, and that round
records nothing more, so rounds never overlap. The browser chooses between IPv4
and IPv6 for each target, as for any request; the local targets are IPv4
addresses.
Each check is a GET request to the target's URL as written, with
`mode: 'no-cors'`, `cache: 'no-store'` and `referrerPolicy: 'no-referrer'`. Its
figure is the time from sending the request until the answer's headers arrive,
timed with `performance.now()`. Unlike ping, this includes the time the server
takes to answer. It also includes the DNS lookup, connection setup and TLS
whenever the browser opens a new connection. Chrome opens one for every check of
the Hetzner speed-test servers and for most checks of the S3 regions.
The browser follows every redirect, and the figure then includes the request to
the new URL too. So a target whose home page answers with a redirect is checked
at `/robots.txt` on the same host, which answers without one. An S3 region
answers it with 403, and any answer counts as reachable.
The Hetzner speed-test servers close the connection without an answer when the
URL has a query string or the request has a `Referer` header. So no query string
is added, and no `Referer` header is sent, wherever the page is served from.
`no-store` keeps the browser's cache out of the measurement. Each check times
out after 80% of the refresh interval (24 seconds at 30 seconds) and is then
recorded as a timeout, so a round's checks have all finished before the next
round is due. When no WAN host answers, a recovery probe checks 4 WAN hosts,
picked at random when it starts, every half second, giving up the checks it
started half a second before. As soon as one answers, a new round starts at
once, as it does after an interval change. A round started early gives up the
last round's checks if they are still waiting, and that round records nothing
more, so rounds never overlap. The browser chooses between IPv4 and IPv6 for
each target, as for any request; the local targets are IPv4 addresses.
Each recorded check that fails writes one line to the browser console with
`console.error`, and the same line to the debug log: the target's name and URL,
TODO.md
+12
View File
@@ -23,6 +23,18 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of
# Completed Steps
- 2026-10-26: a target whose home page answers with a redirect is checked at
`/robots.txt` on the same host, which answers without one
([#194](https://git.eeqj.de/sneak/netwatch/issues/194)). These are the AWS
Console, Akamai, B2 and the 8 S3 regions, and the Google Cloud Console, whose
home page redirects a request that does not say it comes from another site.
The browser follows a redirect, so their figures also timed the page it led
to, for B2 and the S3 regions on another host. Chrome opens a new connection
for most checks of an S3 region's `/robots.txt`, so those figures now include
connection setup and TLS, where they included a request to `aws.amazon.com`.
`README.md` says what a figure is: the time until the answer's headers arrive,
which includes the server's time to answer, and the DNS lookup, connection
setup and TLS whenever the browser opens a new connection
- 2026-10-26: the six Hetzner targets answer wherever the page is served from
([#191](https://git.eeqj.de/sneak/netwatch/issues/191)). They close the
connection without an answer when the request has a `Referer` header. The
test/unit/main.test.js
+52
View File
@@ -191,6 +191,58 @@ test("a check sends no Referer header, wherever the page is served from", async
);
});
// These targets' home pages answer with a redirect, the Google Cloud
// Console's when the request does not say it comes from another site. A
// check follows it and times the request to the new URL too, often on
// another host. Each is checked at /robots.txt on the host its name names,
// which answers without a redirect: an S3 region with 403, which counts as
// reachable.
test("a target whose home page redirects is checked at /robots.txt on its own host", () => {
const checked = new Map(
new AppState([]).wan.map((host) => [host.name, host.url]),
);
for (const [name, url] of [
["AWS Console", "https://console.aws.amazon.com/robots.txt"],
["Google Cloud Console", "https://console.cloud.google.com/robots.txt"],
["Akamai", "https://www.akamai.com/robots.txt"],
["B2", "https://api.backblazeb2.com/robots.txt"],
[
"S3 af-south-1 (Cape Town)",
"https://s3.af-south-1.amazonaws.com/robots.txt",
],
[
"S3 eu-west-2 (London)",
"https://s3.eu-west-2.amazonaws.com/robots.txt",
],
[
"S3 me-south-1 (Bahrain)",
"https://s3.me-south-1.amazonaws.com/robots.txt",
],
[
"S3 ap-northeast-1 (Tokyo)",
"https://s3.ap-northeast-1.amazonaws.com/robots.txt",
],
[
"S3 ap-southeast-1 (Singapore)",
"https://s3.ap-southeast-1.amazonaws.com/robots.txt",
],
[
"S3 ap-southeast-2 (Sydney)",
"https://s3.ap-southeast-2.amazonaws.com/robots.txt",
],
[
"S3 us-west-2 (Oregon)",
"https://s3.us-west-2.amazonaws.com/robots.txt",
],
[
"S3 sa-east-1 (São Paulo)",
"https://s3.sa-east-1.amazonaws.com/robots.txt",
],
]) {
assert.equal(checked.get(name), url);
}
});
test("at a 30000ms interval, a target answering after 1000ms shows in its row while another target's check is still waiting", async (t) => {
CONFIG.updateInterval = 30000;
const state = new AppState([
web/src/main.js
+18 -13
View File
@@ -67,51 +67,56 @@ export const CONFIG = {
// WAN endpoints to monitor. These are used for the aggregate health/stats
// display (min/max/avg). Ordered: personal, LLM APIs, big-3 cloud, then
// CDN/hosting/other.
// CDN/hosting/other. A check follows a redirect and times the request to
// the new URL too, so a target whose home page redirects is checked at
// /robots.txt, which its own host answers without one.
const WAN_HOSTS = [
{ name: "datavi.be", url: "https://datavi.be" },
{ name: "Anthropic API", url: "https://api.anthropic.com" },
{ name: "OpenAI API", url: "https://api.openai.com" },
{ name: "AWS Console", url: "https://console.aws.amazon.com" },
{ name: "Google Cloud Console", url: "https://console.cloud.google.com" },
{ name: "AWS Console", url: "https://console.aws.amazon.com/robots.txt" },
{
name: "Google Cloud Console",
url: "https://console.cloud.google.com/robots.txt",
},
{ name: "Microsoft Azure", url: "https://portal.azure.com" },
{ name: "Cloudflare", url: "https://www.cloudflare.com" },
{ name: "Fastly CDN", url: "https://www.fastly.com" },
{ name: "Akamai", url: "https://www.akamai.com" },
{ name: "Akamai", url: "https://www.akamai.com/robots.txt" },
{ name: "Google", url: "https://www.google.com" },
{ name: "GitHub", url: "https://github.com" },
{ name: "B2", url: "https://api.backblazeb2.com" },
{ name: "B2", url: "https://api.backblazeb2.com/robots.txt" },
{
name: "S3 af-south-1 (Cape Town)",
url: "https://s3.af-south-1.amazonaws.com",
url: "https://s3.af-south-1.amazonaws.com/robots.txt",
},
{
name: "S3 eu-west-2 (London)",
url: "https://s3.eu-west-2.amazonaws.com",
url: "https://s3.eu-west-2.amazonaws.com/robots.txt",
},
{
name: "S3 me-south-1 (Bahrain)",
url: "https://s3.me-south-1.amazonaws.com",
url: "https://s3.me-south-1.amazonaws.com/robots.txt",
},
{
name: "S3 ap-northeast-1 (Tokyo)",
url: "https://s3.ap-northeast-1.amazonaws.com",
url: "https://s3.ap-northeast-1.amazonaws.com/robots.txt",
},
{
name: "S3 ap-southeast-1 (Singapore)",
url: "https://s3.ap-southeast-1.amazonaws.com",
url: "https://s3.ap-southeast-1.amazonaws.com/robots.txt",
},
{
name: "S3 ap-southeast-2 (Sydney)",
url: "https://s3.ap-southeast-2.amazonaws.com",
url: "https://s3.ap-southeast-2.amazonaws.com/robots.txt",
},
{
name: "S3 us-west-2 (Oregon)",
url: "https://s3.us-west-2.amazonaws.com",
url: "https://s3.us-west-2.amazonaws.com/robots.txt",
},
{
name: "S3 sa-east-1 (São Paulo)",
url: "https://s3.sa-east-1.amazonaws.com",
url: "https://s3.sa-east-1.amazonaws.com/robots.txt",
},
// Hetzner regional speed test servers — genuine per-DC endpoints
{