clawbot 36ba8b995b Targets whose home page redirects are checked at /robots.txt (closes #194) ...

The browser follows a redirect, so the figures for the AWS Console, Akamai,
B2 and the S3 regions also timed the page it led to. Each of these targets
is now checked at /robots.txt on its own host, which answers without a
redirect. So is the Google Cloud Console, whose home page redirects curl's
requests. The S3 regions are checked with a HEAD request, as Chrome closes
their connection after most GET requests. README.md says what a figure
includes.

Model: opus-5-5

2026-10-26 02:24:24 +00:00

..

nginx.conf

Page is sent with Cache-Control: no-cache (closes #141)

2026-10-19 03:10:39 +02:00

security-headers.conf

Targets whose home page redirects are checked at /robots.txt (closes #194)

2026-10-26 02:24:24 +00:00