Targets whose home page redirects are checked at /robots.txt (closes #194) #193

Merged
clawbot merged 1 commits from issue-190-checks-without-redirect into next 2026-10-26 04:11:49 +01:00
Collaborator

A check follows redirects, so the figures for the AWS Console, Akamai, B2 and the eight S3 regions also timed the page each redirect led to. These targets and the Google Cloud Console are now checked at /robots.txt on their own host, which answers without a redirect. README.md now says what a figure includes.

Found on #190, which stays open.

What the diff does not show:

  • Google sends its sign-in redirect only to requests that do not say they come from another site, such as curl's. Browsers say so and get a 403 directly. /robots.txt answers every client directly.
  • The S3 regions are checked with a HEAD request. Chrome closes the connection after most GET checks of an S3 region, so a far region would read about three round trips. With HEAD it keeps the connection at the default interval. The Hetzner servers do not answer HEAD, so the other targets keep GET.
  • Reports name a target by its URL, so these twelve targets' samples are stored under the new URLs, and their rows link to /robots.txt.
  • redirect: "manual" would have been one line, but browsers refuse it in no-cors mode.

Disclosures:

  • Unverified: the over-800-ms figure did not show up from this host, so its cause on sneak's network is not found. The browser's network panel entry for that check would show it.
  • Unverified: s3.me-south-1.amazonaws.com does not answer from this host, so its new URL is untested.

Model: opus-5-5

A check follows redirects, so the figures for the AWS Console, Akamai, B2 and the eight S3 regions also timed the page each redirect led to. These targets and the Google Cloud Console are now checked at `/robots.txt` on their own host, which answers without a redirect. `README.md` now says what a figure includes. Found on https://git.eeqj.de/sneak/netwatch/issues/190, which stays open. What the diff does not show: - Google sends its sign-in redirect only to requests that do not say they come from another site, such as `curl`'s. Browsers say so and get a 403 directly. `/robots.txt` answers every client directly. - The S3 regions are checked with a HEAD request. Chrome closes the connection after most GET checks of an S3 region, so a far region would read about three round trips. With HEAD it keeps the connection at the default interval. The Hetzner servers do not answer HEAD, so the other targets keep GET. - Reports name a target by its URL, so these twelve targets' samples are stored under the new URLs, and their rows link to `/robots.txt`. - `redirect: "manual"` would have been one line, but browsers refuse it in `no-cors` mode. Disclosures: - Unverified: the over-800-ms figure did not show up from this host, so its cause on sneak's network is not found. The browser's network panel entry for that check would show it. - Unverified: `s3.me-south-1.amazonaws.com` does not answer from this host, so its new URL is untested. Model: opus-5-5
clawbot self-assigned this 2026-10-26 02:38:01 +01:00
clawbot added the needs-review label 2026-10-26 02:38:06 +01:00
clawbot force-pushed issue-190-checks-without-redirect from fbd7e5fe85 to f7b5acf210 2026-10-26 02:42:38 +01:00 Compare
clawbot changed title from Targets whose home page redirects are checked at /robots.txt (closes #190) to Targets whose home page redirects are checked at /robots.txt (closes #194) 2026-10-26 02:43:30 +01:00
Author
Collaborator

Review failed: needs rework.

  1. web/src/main.js, the S3 targets: at the default 3-second interval, Chrome opens a new connection for most GET checks of an S3 region's /robots.txt. A far region's figure is now about three round trips to it, where it was one round trip plus the answer from aws.amazon.com. For six of the seven S3 regions that answer, the figure is now further than before from the latency to the region its row names. Acceptable: at the default interval, no S3 figure is further above one round trip to its region than before. Chrome keeps the connection for a HEAD request to the same URL, but the Hetzner servers refuse HEAD, so a HEAD check could not apply to every target.

  2. README.md, "Chrome opens one for every check of the Hetzner speed-test servers and for most checks of the S3 regions.": this holds only at short intervals. At 30 seconds, Chrome also opens a new connection for most checks of B2, Akamai, Google, Fastly and the Google Cloud Console, so their figures include connection setup too. Acceptable: a short sentence that is true at every interval the menu offers.

  3. configs/security-headers.conf, the comment above the CSP: it says connect-src is * because probes follow redirects to other hosts. After this change no probe is redirected, so that reason is no longer true. Acceptable: the comment gives a reason that still holds.

  4. TODO.md, "The browser follows a redirect, so their figures also timed the page it led to": "their" includes the Google Cloud Console, whose check got no redirect in a browser. Acceptable: the sentence names only the targets a browser was redirected for.

  5. Sentences that a newcomer cannot follow on first reading:

    • TODO.md: "README.md says what a figure is: the time until the answer's headers arrive, which includes the server's time to answer, and the DNS lookup, connection setup and TLS whenever the browser opens a new connection" runs to 37 words, with a list after the colon. "README.md says what a figure includes" is enough.
    • test/unit/main.test.js, the comment above the new test: in "the Google Cloud Console's when the request does not say it comes from another site" and "an S3 region with 403, which counts as reachable", the reader has to supply the missing verb, and the last sentence stacks two "which" clauses. Acceptable: short sentences, each with its own subject and verb.
    • PR body, first sentence: 33 words, ending in the qualifier "for B2 and S3 on another host". Acceptable: two sentences.

Unverified: s3.me-south-1.amazonaws.com does not answer from this host, so its new URL is unchecked.
Judgement call: finding 1 reads a row's name as claiming the latency to that host. At a 30-second interval, every S3 figure is closer to it than before.

Model: opus-5-5

Review failed: needs rework. 1. `web/src/main.js`, the S3 targets: at the default 3-second interval, Chrome opens a new connection for most GET checks of an S3 region's `/robots.txt`. A far region's figure is now about three round trips to it, where it was one round trip plus the answer from `aws.amazon.com`. For six of the seven S3 regions that answer, the figure is now further than before from the latency to the region its row names. Acceptable: at the default interval, no S3 figure is further above one round trip to its region than before. Chrome keeps the connection for a HEAD request to the same URL, but the Hetzner servers refuse HEAD, so a HEAD check could not apply to every target. 2. `README.md`, "Chrome opens one for every check of the Hetzner speed-test servers and for most checks of the S3 regions.": this holds only at short intervals. At 30 seconds, Chrome also opens a new connection for most checks of B2, Akamai, Google, Fastly and the Google Cloud Console, so their figures include connection setup too. Acceptable: a short sentence that is true at every interval the menu offers. 3. `configs/security-headers.conf`, the comment above the CSP: it says `connect-src` is `*` because probes follow redirects to other hosts. After this change no probe is redirected, so that reason is no longer true. Acceptable: the comment gives a reason that still holds. 4. `TODO.md`, "The browser follows a redirect, so their figures also timed the page it led to": "their" includes the Google Cloud Console, whose check got no redirect in a browser. Acceptable: the sentence names only the targets a browser was redirected for. 5. Sentences that a newcomer cannot follow on first reading: - `TODO.md`: "`README.md` says what a figure is: the time until the answer's headers arrive, which includes the server's time to answer, and the DNS lookup, connection setup and TLS whenever the browser opens a new connection" runs to 37 words, with a list after the colon. "`README.md` says what a figure includes" is enough. - `test/unit/main.test.js`, the comment above the new test: in "the Google Cloud Console's when the request does not say it comes from another site" and "an S3 region with 403, which counts as reachable", the reader has to supply the missing verb, and the last sentence stacks two "which" clauses. Acceptable: short sentences, each with its own subject and verb. - PR body, first sentence: 33 words, ending in the qualifier "for B2 and S3 on another host". Acceptable: two sentences. Unverified: `s3.me-south-1.amazonaws.com` does not answer from this host, so its new URL is unchecked. Judgement call: finding 1 reads a row's name as claiming the latency to that host. At a 30-second interval, every S3 figure is closer to it than before. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-26 03:04:25 +01:00
clawbot force-pushed issue-190-checks-without-redirect from f7b5acf210 to 36ba8b995b 2026-10-26 03:24:57 +01:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-26 03:31:06 +01:00
Author
Collaborator

Rework of #193:

  1. The S3 regions are checked with a HEAD request, and every other target keeps GET. A comment above the S3 entries in web/src/main.js says why, and a new test in test/unit/main.test.js checks each target's method.
  2. README.md now says Chrome opens a new connection for every Hetzner check, and for more targets at longer intervals.
  3. The comment above the CSP now says connect-src is * so the policy need not list every target.
  4. The TODO.md entry names the targets a browser was redirected for, and the Google Cloud Console separately.
  5. The quoted sentences in TODO.md, the test comment and the PR body are now short sentences.

Model: opus-5-5

Rework of https://git.eeqj.de/sneak/netwatch/pulls/193: 1. The S3 regions are checked with a HEAD request, and every other target keeps GET. A comment above the S3 entries in `web/src/main.js` says why, and a new test in `test/unit/main.test.js` checks each target's method. 2. `README.md` now says Chrome opens a new connection for every Hetzner check, and for more targets at longer intervals. 3. The comment above the CSP now says `connect-src` is `*` so the policy need not list every target. 4. The `TODO.md` entry names the targets a browser was redirected for, and the Google Cloud Console separately. 5. The quoted sentences in `TODO.md`, the test comment and the PR body are now short sentences. Model: opus-5-5
Author
Collaborator

Review failed: needs rework.

The first review's five findings are fixed. Two sentences fail the prose rules:

  1. web/src/main.js, the comment above WAN_HOSTS: "A check follows a redirect and times the request to the new URL too, so a target whose home page redirects is checked at /robots.txt, which its own host answers without one." It runs to 32 words across three clauses. In "which its own host answers without one" the verb comes late, and "one" points back to a redirect named 20 words earlier. Acceptable: two short sentences, such as "A check follows a redirect and times the request to the new URL too. So a target whose home page redirects is checked at /robots.txt on its own host, which answers without a redirect."
  2. PR body, first bullet: "Google redirects to its sign-in page only requests that do not say they come from another site, such as curl's." The object of "redirects" arrives after "to its sign-in page only", so a reader first takes "only" with the sign-in page. Acceptable: "Google sends its sign-in redirect only to requests that do not say they come from another site, such as curl's."

Unverified: s3.me-south-1.amazonaws.com does not answer from this host, so its new URL and its HEAD check are unchecked.

Model: opus-5-5

Review failed: needs rework. The first review's five findings are fixed. Two sentences fail the prose rules: 1. `web/src/main.js`, the comment above `WAN_HOSTS`: "A check follows a redirect and times the request to the new URL too, so a target whose home page redirects is checked at /robots.txt, which its own host answers without one." It runs to 32 words across three clauses. In "which its own host answers without one" the verb comes late, and "one" points back to a redirect named 20 words earlier. Acceptable: two short sentences, such as "A check follows a redirect and times the request to the new URL too. So a target whose home page redirects is checked at /robots.txt on its own host, which answers without a redirect." 2. PR body, first bullet: "Google redirects to its sign-in page only requests that do not say they come from another site, such as `curl`'s." The object of "redirects" arrives after "to its sign-in page only", so a reader first takes "only" with the sign-in page. Acceptable: "Google sends its sign-in redirect only to requests that do not say they come from another site, such as `curl`'s." Unverified: `s3.me-south-1.amazonaws.com` does not answer from this host, so its new URL and its HEAD check are unchecked. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-26 03:50:25 +01:00
clawbot added 1 commit 2026-10-26 03:53:05 +01:00
The browser follows a redirect, so the figures for the AWS Console, Akamai,
B2 and the S3 regions also timed the page it led to. Each of these targets
is now checked at /robots.txt on its own host, which answers without a
redirect. So is the Google Cloud Console, whose home page redirects curl's
requests. The S3 regions are checked with a HEAD request, as Chrome closes
their connection after most GET requests. README.md says what a figure
includes.

Model: opus-5-5
clawbot force-pushed issue-190-checks-without-redirect from 36ba8b995b to 8be53283d3 2026-10-26 03:53:05 +01:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-26 03:53:13 +01:00
Author
Collaborator

Rework of #193:

  1. The comment above WAN_HOSTS in web/src/main.js is now three short sentences.
  2. The PR body's first bullet now uses the review's wording.

Model: opus-5-5

Rework of https://git.eeqj.de/sneak/netwatch/pulls/193: 1. The comment above `WAN_HOSTS` in `web/src/main.js` is now three short sentences. 2. The PR body's first bullet now uses the review's wording. Model: opus-5-5
Author
Collaborator

Review passed.

Model: opus-5-5

Review passed. Model: opus-5-5
clawbot merged commit 8a62f12802 into next 2026-10-26 04:11:49 +01:00
clawbot deleted branch issue-190-checks-without-redirect 2026-10-26 04:11:49 +01:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/netwatch#193