Targets whose home page redirects are checked at /robots.txt (closes #194)

The browser follows a redirect, so the figures for the AWS Console, Akamai,
B2 and the S3 regions also timed the page it led to, for B2 and S3 on
another host. The Google Cloud Console redirects only requests that do not
say they come from another site; browsers say so and get a 403 directly.
Each of these targets is now checked at /robots.txt on its own host, which
answers without a redirect. README.md says what a figure includes. Chrome
opens a new connection for most S3 checks, so far regions now read higher.

Model: opus-5-5

test/unit/main.test.js

@@ -191,6 +191,58 @@ test("a check sends no Referer header, wherever the page is served from", async
     );
 });
 
+// These targets' home pages answer with a redirect, the Google Cloud
+// Console's when the request does not say it comes from another site. A
+// check follows it and times the request to the new URL too, often on
+// another host. Each is checked at /robots.txt on the host its name names,
+// which answers without a redirect: an S3 region with 403, which counts as
+// reachable.
+test("a target whose home page redirects is checked at /robots.txt on its own host", () => {
+    const checked = new Map(
+        new AppState([]).wan.map((host) => [host.name, host.url]),
+    );
+    for (const [name, url] of [
+        ["AWS Console", "https://console.aws.amazon.com/robots.txt"],
+        ["Google Cloud Console", "https://console.cloud.google.com/robots.txt"],
+        ["Akamai", "https://www.akamai.com/robots.txt"],
+        ["B2", "https://api.backblazeb2.com/robots.txt"],
+        [
+            "S3 af-south-1 (Cape Town)",
+            "https://s3.af-south-1.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 eu-west-2 (London)",
+            "https://s3.eu-west-2.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 me-south-1 (Bahrain)",
+            "https://s3.me-south-1.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 ap-northeast-1 (Tokyo)",
+            "https://s3.ap-northeast-1.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 ap-southeast-1 (Singapore)",
+            "https://s3.ap-southeast-1.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 ap-southeast-2 (Sydney)",
+            "https://s3.ap-southeast-2.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 us-west-2 (Oregon)",
+            "https://s3.us-west-2.amazonaws.com/robots.txt",
+        ],
+        [
+            "S3 sa-east-1 (São Paulo)",
+            "https://s3.sa-east-1.amazonaws.com/robots.txt",
+        ],
+    ]) {
+        assert.equal(checked.get(name), url);
+    }
+});
+
 test("at a 30000ms interval, a target answering after 1000ms shows in its row while another target's check is still waiting", async (t) => {
     CONFIG.updateInterval = 30000;
     const state = new AppState([