Targets whose home page redirects are checked at /robots.txt (closes #194)

The browser follows a redirect, so the figures for the AWS Console, Akamai,
B2 and the S3 regions also timed the page it led to. Each of these targets
is now checked at /robots.txt on its own host, which answers without a
redirect. So is the Google Cloud Console, whose home page redirects curl's
requests. The S3 regions are checked with a HEAD request, as Chrome closes
their connection after most GET requests. README.md says what a figure
includes.

Model: opus-5-5

README.md

@@ -413,22 +413,36 @@ Local hosts are tracked separately from WAN stats.
 
 ### Latency measurement
 
-GET requests to each target's URL as written, with `mode: 'no-cors'`,
-`cache: 'no-store'` and `referrerPolicy: 'no-referrer'`, timed with
-`performance.now()`. The Hetzner speed-test servers close the connection without
-an answer when the URL has a query string or the request has a `Referer` header.
-So no query string is added, and no `Referer` header is sent, wherever the page
-is served from. `no-store` keeps the browser's cache out of the measurement.
-Each check times out after 80% of the refresh interval (24 seconds at 30
-seconds) and is then recorded as a timeout, so a round's checks have all
-finished before the next round is due. When no WAN host answers, a recovery
-probe checks 4 WAN hosts, picked at random when it starts, every half second,
-giving up the checks it started half a second before. As soon as one answers, a
-new round starts at once, as it does after an interval change. A round started
-early gives up the last round's checks if they are still waiting, and that round
-records nothing more, so rounds never overlap. The browser chooses between IPv4
-and IPv6 for each target, as for any request; the local targets are IPv4
-addresses.
+Each check is a GET request to the target's URL as written, with
+`mode: 'no-cors'`, `cache: 'no-store'` and `referrerPolicy: 'no-referrer'`. Its
+figure is the time from sending the request until the answer's headers arrive,
+timed with `performance.now()`. Unlike ping, this includes the time the server
+takes to answer. It also includes the DNS lookup, connection setup and TLS
+whenever the browser opens a new connection. Chrome opens one for every check of
+the Hetzner speed-test servers, and for more targets at longer intervals.
+
+The S3 regions are checked with a HEAD request instead. Chrome keeps the
+connection to an S3 region open after a HEAD request, but closes it after most
+GET requests. The Hetzner speed-test servers do not answer a HEAD request.
+
+The browser follows every redirect, and the figure then includes the request to
+the new URL too. So a target whose home page answers with a redirect is checked
+at `/robots.txt` on the same host, which answers without one. An S3 region
+answers it with 403, and any answer counts as reachable.
+
+The Hetzner speed-test servers close the connection without an answer when the
+URL has a query string or the request has a `Referer` header. So no query string
+is added, and no `Referer` header is sent, wherever the page is served from.
+`no-store` keeps the browser's cache out of the measurement. Each check times
+out after 80% of the refresh interval (24 seconds at 30 seconds) and is then
+recorded as a timeout, so a round's checks have all finished before the next
+round is due. When no WAN host answers, a recovery probe checks 4 WAN hosts,
+picked at random when it starts, every half second, giving up the checks it
+started half a second before. As soon as one answers, a new round starts at
+once, as it does after an interval change. A round started early gives up the
+last round's checks if they are still waiting, and that round records nothing
+more, so rounds never overlap. The browser chooses between IPv4 and IPv6 for
+each target, as for any request; the local targets are IPv4 addresses.
 
 Each recorded check that fails writes one line to the browser console with
 `console.error`, and the same line to the debug log: the target's name and URL,