Author SHA1 Message Date
clawbot a6d68262cf README.md: rework against the third review
check / check (push) Waiting to run
The symbolic link rule, the console line, fx and the Makefile name
are fixed as the review asked. The full pass fixed nine more
sentences: a split subject, a late verb or a claim broader than the
code.

Model: fable-5-1
2026-10-26 05:16:00 +00:00
clawbot a3b18b6ce8 README.md: rework against the second review
check / check (push) Canceled after 0s
The four findings: only a recorded failed check is logged, and the
discarded round, given-up checks and the recovery probe log nothing;
prepare-data-dir gives directories, regular files and symbolic links to
the netwatch user, and a hard link keeps its owner and is logged; the
cli entry names what it runs in three short sentences; the 80% timeout
and the early-round rule have their reasons back, so rounds never
overlap.

The full pass also fixed claims broader than the code (a row always
shows its stats; a report is posted every 60 seconds; a file is written
every minute; each refusal answers JSON; every answer to a report
carries rate-limit headers; every setting applies in the image, where
BIND_ADDRESS has no effect; fmt formats all files), six sentences whose
subject was split from its verb (the recovery probe, the forwarded
entry with a port, the failed write's retry and its file, both rate
limits, the nvm archive), and the viewport test's "same ... as"
comparison.

Model: fable-5-1
2026-10-26 04:41:53 +00:00
clawbot e6ca7f525d README.md: rework against the review findings
check / check (push) Canceled after 0s
The check sentence names HEAD for the S3 regions up front. The rate-limit
sentence states the safe rate plainly: an address that sends at most half
the limit in any 60 seconds is never refused. The upaas `TRUSTED_PROXIES`
entry again warns that a client connecting from a trusted address can
write its own `X-Forwarded-For`, that a connection through a published
Docker port can arrive from the Docker network's gateway, which the
default includes, and that set and empty, every client behind the proxy
shares one allowance. `TODO.md` gets the Completed Steps entry for
#189, now that
#196 has merged. For
#195.

Model: fable-5-1
2026-10-26 04:08:24 +00:00
clawbot d317002cdc README.md: name the rounds in which the rows are sorted
Model: fable-5-1
2026-10-26 04:00:16 +00:00
clawbot 7cb2d11616 README.md rewritten sentence by sentence for a newcomer
Every sentence is under about 30 words, with one idea each. The page,
its targets and what a figure measures come first, then reporting,
deployment, the settings reference, the design, and development last.
No tables; every identifier in backticks. The code on next decides
what is true.

Model: fable-5-1
2026-10-26 04:00:16 +00:00
clawbot 83a095b572 TODO.md rewritten sentence by sentence
check / check (push) Canceled after 0s
TODO.md is rewritten from scratch so that every sentence reads on the first pass: one idea each, under about 30 words, the verb early. Workflow, Next Step and Future Steps keep their substance. Each Completed Steps entry keeps its date, its issue link and what changed; entries describing a state since changed are in the past tense. Part of #189.

Model: fable-5-1 (rewrite, review); opus-5-5 (merge)
Co-authored-by: clawbot <clawbot@noreply.example.org>
2026-10-26 04:59:21 +01:00
clawbot 8a62f12802 Targets whose home page redirects are checked at /robots.txt (closes #194)
check / check (push) Canceled after 0s
A check follows a redirect and times the request to the new URL too. So the AWS Console, the Google Cloud Console, Akamai, B2 and the eight S3 regions are now checked at /robots.txt on their own host, which answers without a redirect. Before, the S3 and B2 figures also timed a page on another host. The S3 regions are checked with HEAD, because Chrome closes the connection after a GET of their /robots.txt and the figure would then include connection setup every time. README.md says what a figure includes.

Reports name a target by its URL, so these targets' samples are stored under the new URLs.

Model: opus-5-5
2026-10-26 04:11:48 +01:00
clawbot a72797a403 Checks send no Referer header, so Hetzner answers (closes #191)
check / check (push) Canceled after 0s
Each check now sets referrerPolicy: "no-referrer". The Hetzner speed-test servers close the connection without an answer when the request has a Referer header, and the browser sent one unless the page came with the image's Referrer-Policy: no-referrer header, as from a static host or make dev. The checks now send none wherever the page is served from. s3.me-south-1.amazonaws.com still times out from this machine; curl gets no answer from it either.

Model: opus-5-5
2026-10-26 02:10:43 +01:00
clawbot db5eb507c1 server.go starts listenAndServe in an anonymous function (closes #162)
check / check (push) Canceled after 0s
The server's OnStart hook started the HTTP listener with
`go s.listenAndServe()`; the Go style guide asks for a function called
with `go` to be wrapped in an anonymous function, so it is now
`go func() { s.listenAndServe() }()`. No other go statement in the Go
code called a function directly. Nothing the server does changed.

Model: opus-5-5
2026-10-19 08:24:49 +02:00
clawbot 6e4c44b3d6 Match .gitignore, .editorconfig and .golangci.yml to sneak/prompts (closes #174)
check / check (push) Canceled after 0s
REPO_POLICIES.md asks for the standard .gitignore and .editorconfig
followed by the repository's own entries, and for the canonical
.golangci.yml. The shared parts and .golangci.yml now match the
sneak/prompts copies at 065357920a0a09bd736bf33d4cb5ce935fdc2328 byte
for byte. .editorconfig has no entries of its own left, since the
standard file now carries the [*.go] section. .golangci.yml disables
canonicalheader, as the canonical copy does.

Model: opus-5-5
2026-10-19 08:22:05 +02:00
clawbot 91eb3d6771 package.json drops its version field and is private (closes #158)
check / check (push) Canceled after 0s
package.json said version 1.0.0, a release that has not been made.
Nothing reads the field, and the package is never published; the git
tag is the one version, which the image and make build take from
git describe --tags. The field is gone, so there is nothing to keep in
step with tags, and the package is marked "private": true.

Model: opus-5-5
2026-10-19 08:16:29 +02:00
clawbot edcf7f8c3c TODO.md Status says where release candidates are tagged (closes #157)
check / check (push) Canceled after 0s
Status said there were no git tags, but one is tagged on `next`. It now
says release candidates are tagged on `next`, names the first one, and
gives the command that lists them, so it stays true as more are tagged.
The other Status sentences match the tree and are unchanged.

Model: opus-5-5
2026-10-19 08:13:00 +02:00
clawbot 5b01b50bc7 netwatch-server refuses arguments it does not understand (closes #179)
check / check (push) Canceled after 0s
cli.Main starts the server only when given no arguments. check-cidr
CIDR and prepare-data-dir DATA_DIR run as before. Any other command
line, such as a subcommand with no value or two values, or a misspelt
subcommand, gets a one-line usage message naming both forms on stderr
and exit code 2. It used to start the server, which then held the
port.

Model: opus-5-5
2026-10-19 08:09:13 +02:00
clawbot 7a3c706fe6 main() is one call into internal/cli (closes #160)
check / check (push) Canceled after 0s
cmd/netwatch-server/main.go parsed the command line, ran the check-cidr
and prepare-data-dir subcommands and built the fx application itself;
the Go style guide puts all of that in internal/ or pkg/. The new
package internal/cli does it, with the same arguments, messages and
exit codes. main() is now os.Exit(cli.Main(os.Stderr, Appname, Version,
os.Args[1:], cli.RunServer)): Main takes the writer for errors and the
function that runs the server, so its tests give it a buffer and a
function that fails them. Appname and Version stay in main, where the
build sets Version. The function behind prepare-data-dir also takes the
volume and the user's name, so its own tests give it a temporary
directory and the user they run as.

Model: opus-5-5
2026-10-19 07:53:09 +02:00
clawbot 7a2b6b0494 script/build stamps lightweight tags too (closes #156)
check / check (push) Canceled after 0s
When VERSION is unset or empty, script/build now takes the version from
`git describe --tags --always --dirty`, the form script/docker and
script/cibuild use. Without --tags it ignored lightweight tags, so a
binary from `make build` could report a commit where the image built
from the same tree reported the tag. README.md Entrypoints names the
same form.

Model: opus-5-5
2026-10-19 07:45:17 +02:00
clawbot a42a220bb7 .dockerignore keeps root test binaries and coverage out (closes #155)
check / check (push) Canceled after 0s
A Go test binary or coverage file left at the repository root, which
.gitignore already ignores, went into the build context of every docker
build in script/ and was copied into the lint, test and build stages.
The repository's own section of .dockerignore now lists /*.test and
/*.out, root-anchored like its other entries.

Checked with a probe image that copies the whole context: x.test and
x.out planted at the root are no longer in it, and nothing else changed.

Model: opus-5-5
2026-10-19 07:42:43 +02:00
20 changed files with 1711 additions and 1182 deletions
+4 -1
View File
@@ -73,9 +73,12 @@
# This repository's own entries, after the shared content above: the
# frontend build, the viewport test's output, the binary `make build`
# writes and the reports `make run` stores, and log files at the root.
# writes and the reports `make run` stores, and Go test binaries,
# coverage output and log files at the root.
/dist
/tmp
/netwatch-server
/data
/*.test
/*.out
/*.log
+3 -2
View File
@@ -11,7 +11,8 @@ insert_final_newline = true
[Makefile]
indent_style = tab
# This repository's own entries, after the shared content above.
[*.go]
indent_style = tab
# This repository's own sections, such as one for another language it
# uses, go below this comment, and a re-vendor keeps them.
+4 -2
View File
@@ -27,7 +27,7 @@ node_modules/
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation after these lines, for example `!.env.example`.
# its own negation at the end of this file, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
@@ -46,7 +46,9 @@ node_modules/
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# This repository's own entries, after the shared content above.
# This repository's own entries, such as its build outputs, go below
# this comment, and a re-vendor keeps them. Anchor a binary built at the
# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
# Build output
dist/
+2
View File
@@ -25,6 +25,8 @@ linters:
# silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
# Misses findings at random in v2.14.0; back once a pinned release fixes it
- canonicalheader
settings:
lll:
line-length: 88
+607 -513
View File
File diff suppressed because it is too large Load Diff
+556 -541
View File
File diff suppressed because it is too large Load Diff
+4 -74
View File
@@ -1,22 +1,11 @@
// Package main is the entry point for netwatch-server.
// Package main is the entry point for netwatch-server; package cli
// does the work.
package main
import (
"fmt"
"os"
"os/user"
"sneak.berlin/go/netwatch/internal/config"
"sneak.berlin/go/netwatch/internal/globals"
"sneak.berlin/go/netwatch/internal/handlers"
"sneak.berlin/go/netwatch/internal/healthcheck"
"sneak.berlin/go/netwatch/internal/logger"
"sneak.berlin/go/netwatch/internal/middleware"
"sneak.berlin/go/netwatch/internal/reportbuf"
"sneak.berlin/go/netwatch/internal/server"
"go.uber.org/fx"
"go.uber.org/fx/fxevent"
"sneak.berlin/go/netwatch/internal/cli"
)
//nolint:gochecknoglobals // set via ldflags at build time
@@ -26,64 +15,5 @@ var (
)
func main() {
// "netwatch-server check-cidr CIDR" exits 1, with the error, if
// this server would refuse CIDR in its TRUSTED_PROXIES.
// bin/entrypoint.sh runs it on each entry it gives nginx.
if len(os.Args) == 3 && os.Args[1] == "check-cidr" {
_, err := middleware.ParseTrustedProxies(os.Args[2:])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
return
}
// "netwatch-server prepare-data-dir DATA_DIR" gets DATA_DIR ready
// for the netwatch user, or exits 1 with the error; see
// reportbuf.PrepareDataDir. bin/entrypoint.sh runs it as root
// before it starts this server as that user.
if len(os.Args) == 3 && os.Args[1] == "prepare-data-dir" {
netwatch, err := user.Lookup("netwatch")
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
err = reportbuf.PrepareDataDir("/data", os.Args[2], netwatch)
if err != nil {
fmt.Fprintf(os.Stderr, "DATA_DIR '%s': %v\n", os.Args[2], err)
os.Exit(1)
}
return
}
globals.Appname = Appname
globals.Version = Version
fx.New(
// fx logs each step of starting and stopping through the
// server's own logger, so off a terminal those lines are
// JSON like every other line.
fx.WithLogger(func(log *logger.Logger) fxevent.Logger {
return &fxevent.SlogLogger{Logger: log.Get()}
}),
fx.Provide(
config.New,
globals.New,
handlers.New,
healthcheck.New,
logger.New,
middleware.New,
reportbuf.New,
server.New,
),
fx.Invoke(
// First, so the name and version are logged even when
// a setting stops the start.
func(log *logger.Logger) { log.Identify() },
func(*server.Server) {},
),
).Run()
os.Exit(cli.Main(os.Stderr, Appname, Version, os.Args[1:], cli.RunServer))
}
+3 -4
View File
@@ -8,10 +8,9 @@ add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" alway
# Scripts and styles load only from the page's own origin. Inline ones
# are blocked, style attributes in markup included, so style elements
# through classes or element.style. data: images are for the favicon
# in web/index.html. connect-src is * because the browser checks each
# probe in web/src/main.js against it, and also every redirect the probe
# follows, and several of those hosts redirect to others; a list of
# hosts here would block those probes. It also covers the reports the
# in web/index.html. connect-src is * because the page checks every
# target in web/src/main.js. A list of hosts here would have to change
# with every change to those targets. It also covers the reports the
# page sends to its own origin.
add_header Content-Security-Policy "default-src 'self'; connect-src *; img-src 'self' data:; object-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'" always;
+143
View File
@@ -0,0 +1,143 @@
// Package cli runs netwatch-server from its command line: the
// check-cidr and prepare-data-dir subcommands, which bin/entrypoint.sh
// runs before it starts the server, and, given no arguments, the
// server, with the fx wiring.
package cli
import (
"fmt"
"io"
"os/user"
"sneak.berlin/go/netwatch/internal/config"
"sneak.berlin/go/netwatch/internal/globals"
"sneak.berlin/go/netwatch/internal/handlers"
"sneak.berlin/go/netwatch/internal/healthcheck"
"sneak.berlin/go/netwatch/internal/logger"
"sneak.berlin/go/netwatch/internal/middleware"
"sneak.berlin/go/netwatch/internal/reportbuf"
"sneak.berlin/go/netwatch/internal/server"
"go.uber.org/fx"
"go.uber.org/fx/fxevent"
)
// usageExitCode is what Main returns for a command line it does not
// understand, the usual exit code for a usage error.
const usageExitCode = 2
// Main runs netwatch-server with args, its command line after the
// program's name, and returns the exit code. With no args, Main runs
// the server by calling serve with appname and version, the name and
// version the server logs and reports. With "check-cidr CIDR" or
// "prepare-data-dir DATA_DIR" it runs that subcommand, which writes its
// errors to stderr. Any other args get a one-line usage message on
// stderr and exit code 2. main passes os.Stderr and RunServer; the
// tests pass a buffer and a function of their own, so that no test
// starts the server. When serve is RunServer and the server fails to
// start, to listen or to stop, fx's Run exits the process with status 1
// and Main never returns.
func Main(
stderr io.Writer,
appname, version string,
args []string,
serve func(appname, version string),
) int {
// "netwatch-server check-cidr CIDR" exits 1, with the error, if
// this server would refuse CIDR in its TRUSTED_PROXIES.
// bin/entrypoint.sh runs it on each entry it gives nginx.
if len(args) == 2 && args[0] == "check-cidr" {
return checkCIDR(stderr, args[1])
}
// "netwatch-server prepare-data-dir DATA_DIR" gets DATA_DIR ready
// for the netwatch user, or exits 1 with the error; see
// reportbuf.PrepareDataDir. bin/entrypoint.sh runs it as root
// before it starts this server as that user.
if len(args) == 2 && args[0] == "prepare-data-dir" {
return prepareDataDir(stderr, "/data", args[1], "netwatch")
}
// Any other command line is refused, so that a mistyped one does
// not start the server.
if len(args) > 0 {
_, _ = fmt.Fprintln(stderr,
"usage: netwatch-server [check-cidr CIDR | prepare-data-dir DATA_DIR]")
return usageExitCode
}
serve(appname, version)
return 0
}
// checkCIDR returns 0 if the server would take cidr in its
// TRUSTED_PROXIES, and otherwise writes the error to stderr and
// returns 1.
func checkCIDR(stderr io.Writer, cidr string) int {
_, err := middleware.ParseTrustedProxies([]string{cidr})
if err != nil {
_, _ = fmt.Fprintln(stderr, err)
return 1
}
return 0
}
// prepareDataDir gets dataDir ready for the user named username with
// reportbuf.PrepareDataDir, volume being where the data volume is
// mounted, and returns 0. Otherwise it writes the error to stderr and
// returns 1.
func prepareDataDir(stderr io.Writer, volume, dataDir, username string) int {
owner, err := user.Lookup(username)
if err != nil {
_, _ = fmt.Fprintln(stderr, err)
return 1
}
err = reportbuf.PrepareDataDir(volume, dataDir, owner)
if err != nil {
_, _ = fmt.Fprintf(stderr, "DATA_DIR '%s': %v\n", dataDir, err)
return 1
}
return 0
}
// RunServer runs the server until it is told to stop, with appname and
// version as the name and version it logs and reports. If the server
// fails to start, to listen or to stop, fx's Run exits the process
// with status 1 itself.
func RunServer(appname, version string) {
globals.Appname = appname
globals.Version = version
fx.New(
// fx logs each step of starting and stopping through the
// server's own logger, so off a terminal those lines are
// JSON like every other line.
fx.WithLogger(func(log *logger.Logger) fxevent.Logger {
return &fxevent.SlogLogger{Logger: log.Get()}
}),
fx.Provide(
config.New,
globals.New,
handlers.New,
healthcheck.New,
logger.New,
middleware.New,
reportbuf.New,
server.New,
),
fx.Invoke(
// First, so the name and version are logged even when
// a setting stops the start.
func(log *logger.Logger) { log.Identify() },
func(*server.Server) {},
),
).Run()
}
+219
View File
@@ -0,0 +1,219 @@
package cli_test
import (
"bytes"
"os"
"os/user"
"path/filepath"
"strconv"
"strings"
"testing"
"sneak.berlin/go/netwatch/internal/cli"
"sneak.berlin/go/netwatch/internal/reportbuf"
)
// TestMainRunsCheckCIDR: Main given check-cidr returns 0, writing
// nothing, for a CIDR the server takes in its TRUSTED_PROXIES, and 1,
// with an error naming TRUSTED_PROXIES and the CIDR, for one it
// refuses, and does not start the server.
func TestMainRunsCheckCIDR(t *testing.T) {
t.Parallel()
var stderr bytes.Buffer
code := cli.Main(&stderr, "netwatch", "test",
[]string{"check-cidr", "10.0.0.0/8"}, failIfServerStarts(t))
if code != 0 || stderr.Len() != 0 {
t.Errorf("check-cidr 10.0.0.0/8: exit %d, stderr %q; want exit 0 and no output",
code, stderr.String())
}
stderr.Reset()
code = cli.Main(&stderr, "netwatch", "test",
[]string{"check-cidr", "1.2.3/32"}, failIfServerStarts(t))
want := `TRUSTED_PROXIES "1.2.3/32": `
if code != 1 || !strings.HasPrefix(stderr.String(), want) {
t.Errorf("check-cidr 1.2.3/32: exit %d, stderr %q; want exit 1 and %q...",
code, stderr.String(), want)
}
}
// TestMainRunsPrepareDataDir: Main given prepare-data-dir runs the
// prepare-data-dir subcommand for the netwatch user, and does not start
// the server. The test image has no netwatch user, so that subcommand
// exits 1 with the error from looking the user up;
// TestPrepareDataDirRefusesPathOutsideVolume checks the path.
func TestMainRunsPrepareDataDir(t *testing.T) {
t.Parallel()
var stderr bytes.Buffer
code := cli.Main(&stderr, "netwatch", "test",
[]string{"prepare-data-dir", "/etc"}, failIfServerStarts(t))
want := user.UnknownUserError("netwatch").Error() + "\n"
if code != 1 || stderr.String() != want {
t.Errorf("prepare-data-dir /etc: exit %d, stderr %q; want exit 1 and %q",
code, stderr.String(), want)
}
}
// TestMainRunsServerWithNoArguments: Main given no arguments starts the
// server with the name and version it was given, writes nothing and
// returns 0.
func TestMainRunsServerWithNoArguments(t *testing.T) {
t.Parallel()
var (
stderr bytes.Buffer
gotAppname, gotVersion string
)
code := cli.Main(&stderr, "netwatch", "test", []string{},
func(appname, version string) {
gotAppname, gotVersion = appname, version
})
if code != 0 || stderr.Len() != 0 {
t.Errorf("exit %d, stderr %q; want exit 0 and no output",
code, stderr.String())
}
if gotAppname != "netwatch" || gotVersion != "test" {
t.Errorf("server started with %q %q; want %q %q",
gotAppname, gotVersion, "netwatch", "test")
}
}
// TestMainRefusesOtherCommandLines: Main given a subcommand with too
// few or too many values, or a misspelt subcommand, writes the usage
// message to stderr, returns 2 and does not start the server.
func TestMainRefusesOtherCommandLines(t *testing.T) {
t.Parallel()
want := "usage: netwatch-server " +
"[check-cidr CIDR | prepare-data-dir DATA_DIR]\n"
for _, commandLine := range []string{
"netwatch-server check-cidr",
"netwatch-server check-cidr 192.0.2.0/24 198.51.100.0/24",
"netwatch-server prepare-data-dir",
"netwatch-server prepare-data-dir /data/reports /data/other",
"netwatch-server check-cdir 192.0.2.0/24",
} {
var stderr bytes.Buffer
// Main takes the command line after the program's name.
args := strings.Fields(commandLine)[1:]
code := cli.Main(&stderr, "netwatch", "test", args, failIfServerStarts(t))
if code != 2 || stderr.String() != want {
t.Errorf("%q: exit %d, stderr %q; want exit 2 and %q",
commandLine, code, stderr.String(), want)
}
}
}
// TestCheckCIDRTakesCIDR: check-cidr exits 0, writing nothing, for a
// CIDR the server takes in its TRUSTED_PROXIES.
func TestCheckCIDRTakesCIDR(t *testing.T) {
t.Parallel()
for _, cidr := range []string{"10.0.0.0/8", "192.0.2.1/32", "2001:db8::/32"} {
var stderr bytes.Buffer
code := cli.CheckCIDR(&stderr, cidr)
if code != 0 || stderr.Len() != 0 {
t.Errorf("%q: exit %d, stderr %q; want exit 0 and no output",
cidr, code, stderr.String())
}
}
}
// TestCheckCIDRRefusesOtherInput: check-cidr exits 1, with an error
// naming TRUSTED_PROXIES and the input, for anything the server would
// refuse in its TRUSTED_PROXIES, such as 1.2.3/32, which nginx would
// look up as a hostname.
func TestCheckCIDRRefusesOtherInput(t *testing.T) {
t.Parallel()
for _, cidr := range []string{
"1.2.3/32", "example.com/32", "10.0.0.0/33", "10.0.0.1", "",
} {
var stderr bytes.Buffer
code := cli.CheckCIDR(&stderr, cidr)
want := "TRUSTED_PROXIES " + strconv.Quote(cidr) + ": "
if code != 1 || !strings.HasPrefix(stderr.String(), want) {
t.Errorf("%q: exit %d, stderr %q; want exit 1 and %q...",
cidr, code, stderr.String(), want)
}
}
}
// TestPrepareDataDirTakesPathInVolume: prepare-data-dir exits 0,
// writing nothing, for a DATA_DIR below the volume, and creates it.
func TestPrepareDataDirTakesPathInVolume(t *testing.T) {
t.Parallel()
volume := t.TempDir()
dataDir := filepath.Join(volume, "reports")
var stderr bytes.Buffer
code := cli.PrepareDataDir(&stderr, volume, dataDir, currentUsername(t))
if code != 0 || stderr.Len() != 0 {
t.Errorf("exit %d, stderr %q; want exit 0 and no output",
code, stderr.String())
}
info, err := os.Stat(dataDir)
if err != nil || !info.IsDir() {
t.Errorf("DATA_DIR not created: %v", err)
}
}
// TestPrepareDataDirRefusesPathOutsideVolume: prepare-data-dir exits
// 1, with an error naming DATA_DIR and its value, for a DATA_DIR
// outside the volume.
func TestPrepareDataDirRefusesPathOutsideVolume(t *testing.T) {
t.Parallel()
volume := t.TempDir()
var stderr bytes.Buffer
code := cli.PrepareDataDir(&stderr, volume, "/etc", currentUsername(t))
want := "DATA_DIR '/etc': " + reportbuf.ErrDataDirOutsideVolume.Error() + "\n"
if code != 1 || stderr.String() != want {
t.Errorf("exit %d, stderr %q; want exit 1 and %q",
code, stderr.String(), want)
}
}
// currentUsername is the name of the user the test runs as, the only
// owner a test not run as root can give files to.
func currentUsername(t *testing.T) string {
t.Helper()
current, err := user.Current()
if err != nil {
t.Fatal(err)
}
return current.Username
}
// failIfServerStarts is what the tests give Main in place of
// cli.RunServer: it fails the test instead of starting the server.
func failIfServerStarts(t *testing.T) func(appname, version string) {
t.Helper()
return func(string, string) {
t.Error("Main started the server")
}
}
+16
View File
@@ -0,0 +1,16 @@
package cli
import "io"
// CheckCIDR exposes the check-cidr subcommand to the external tests,
// which read what it writes to stderr.
func CheckCIDR(stderr io.Writer, cidr string) int {
return checkCIDR(stderr, cidr)
}
// PrepareDataDir exposes the prepare-data-dir subcommand to the
// external tests, which give it a volume and a user of their own in
// place of /data and netwatch.
func PrepareDataDir(stderr io.Writer, volume, dataDir, username string) int {
return prepareDataDir(stderr, volume, dataDir, username)
}
+4 -4
View File
@@ -13,8 +13,8 @@ import (
"go.uber.org/fx"
)
// requireConfigError builds the config as main does and fails the
// test unless that fails with an error naming setting. It uses
// requireConfigError builds the config as cli.RunServer does and fails
// the test unless that fails with an error naming setting. It uses
// fx.New, because fxtest.New fails the test itself on an error.
func requireConfigError(t *testing.T, setting string) {
t.Helper()
@@ -31,8 +31,8 @@ func requireConfigError(t *testing.T, setting string) {
}
}
// requireTrustedProxies builds the config as main does and fails the
// test unless its TrustedProxies are want.
// requireTrustedProxies builds the config as cli.RunServer does and
// fails the test unless its TrustedProxies are want.
func requireTrustedProxies(t *testing.T, want []string) {
t.Helper()
+1 -1
View File
@@ -4,7 +4,7 @@ package globals
import "go.uber.org/fx"
//nolint:gochecknoglobals // set from main before fx starts
//nolint:gochecknoglobals // set by cli.RunServer before fx starts
var (
// Appname is the application name.
Appname string
+7 -7
View File
@@ -334,9 +334,9 @@ func stopWhileHeld(
return <-stopped
}
// startApp starts a Buffer through fx, as main does, with the DATA_DIR
// and DATA_DIR_MAX_BYTES the calling test has set, and returns it with
// the app, for tests that stop the app themselves.
// startApp starts a Buffer through fx, as cli.RunServer does, with the
// DATA_DIR and DATA_DIR_MAX_BYTES the calling test has set, and returns
// it with the app, for tests that stop the app themselves.
func startApp(t *testing.T) (*reportbuf.Buffer, *fxtest.App) {
t.Helper()
@@ -358,8 +358,8 @@ func startApp(t *testing.T) (*reportbuf.Buffer, *fxtest.App) {
return buf, app
}
// startBuffer starts a Buffer through fx, as main does, with the
// DATA_DIR and DATA_DIR_MAX_BYTES the calling test has set.
// startBuffer starts a Buffer through fx, as cli.RunServer does, with
// the DATA_DIR and DATA_DIR_MAX_BYTES the calling test has set.
func startBuffer(t *testing.T) *reportbuf.Buffer {
t.Helper()
@@ -1120,8 +1120,8 @@ func TestHealthCheckFailsWhileWritesFail(t *testing.T) {
}
// startBufferWithHandlers starts a Buffer and the Handlers that store
// reports in it through fx, as main does, with the DATA_DIR the calling
// test has set.
// reports in it through fx, as cli.RunServer does, with the DATA_DIR
// the calling test has set.
func startBufferWithHandlers(
t *testing.T,
) (*reportbuf.Buffer, *handlers.Handlers) {
+3 -3
View File
@@ -22,9 +22,9 @@ import (
"go.uber.org/fx/fxtest"
)
// newServer builds a Server from the same constructors as main,
// configured from the environment. It is never started, so nothing
// listens.
// newServer builds a Server from the same constructors as
// cli.RunServer, configured from the environment. It is never started,
// so nothing listens.
func newServer(t *testing.T) *server.Server {
t.Helper()
+1 -1
View File
@@ -79,7 +79,7 @@ func New(
s.SetupRoutes()
s.httpServer = s.newHTTPServer()
go s.listenAndServe()
go func() { s.listenAndServe() }()
return nil
},
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "netwatch",
"version": "1.0.0",
"private": true,
"description": "Real-time network latency monitor SPA",
"type": "module",
"scripts": {
+4 -3
View File
@@ -10,9 +10,10 @@ main() {
cd "$ROOT"
"$ROOT/script/yarn" build
# VERSION comes from the environment. Unset or empty, it is git
# describe, or "dev" where there is no git or no repository history.
version="${VERSION:-$(git describe --always --dirty 2>/dev/null || echo dev)}"
# VERSION comes from the environment. Unset or empty, it is
# `git describe --tags --always --dirty`, as in script/docker, or
# "dev" where there is no git or no repository history.
version="${VERSION:-$(git describe --tags --always --dirty 2>/dev/null || echo dev)}"
CGO_ENABLED=0 go build -trimpath \
-ldflags "-s -w -X main.Version=$version" \
+85
View File
@@ -180,6 +180,91 @@ test("a check fetches the target's URL as written, with no query string added",
);
});
test("a check sends no Referer header, wherever the page is served from", async (t) => {
mockTargets(t, () => 10);
const check = measureLatency("https://fsn1-speed.hetzner.com");
t.mock.timers.tick(10);
assert.notEqual(await settled(check), "still waiting");
assert.equal(
fetch.mock.calls[0].arguments[1].referrerPolicy,
"no-referrer",
);
});
// These targets' home pages answer with a redirect. A check follows it and
// times the request to the new URL too, often on another host. The Google
// Cloud Console redirects only requests that do not say they come from
// another site, such as curl's. Each target is checked at /robots.txt on
// its own host, which answers without a redirect. An S3 region answers it
// with 403, and any answer counts as reachable.
test("a target whose home page redirects is checked at /robots.txt on its own host", () => {
const checked = new Map(
new AppState([]).wan.map((host) => [host.name, host.url]),
);
for (const [name, url] of [
["AWS Console", "https://console.aws.amazon.com/robots.txt"],
["Google Cloud Console", "https://console.cloud.google.com/robots.txt"],
["Akamai", "https://www.akamai.com/robots.txt"],
["B2", "https://api.backblazeb2.com/robots.txt"],
[
"S3 af-south-1 (Cape Town)",
"https://s3.af-south-1.amazonaws.com/robots.txt",
],
[
"S3 eu-west-2 (London)",
"https://s3.eu-west-2.amazonaws.com/robots.txt",
],
[
"S3 me-south-1 (Bahrain)",
"https://s3.me-south-1.amazonaws.com/robots.txt",
],
[
"S3 ap-northeast-1 (Tokyo)",
"https://s3.ap-northeast-1.amazonaws.com/robots.txt",
],
[
"S3 ap-southeast-1 (Singapore)",
"https://s3.ap-southeast-1.amazonaws.com/robots.txt",
],
[
"S3 ap-southeast-2 (Sydney)",
"https://s3.ap-southeast-2.amazonaws.com/robots.txt",
],
[
"S3 us-west-2 (Oregon)",
"https://s3.us-west-2.amazonaws.com/robots.txt",
],
[
"S3 sa-east-1 (São Paulo)",
"https://s3.sa-east-1.amazonaws.com/robots.txt",
],
]) {
assert.equal(checked.get(name), url);
}
});
// Chrome keeps the connection to an S3 region open after a HEAD check, but
// closes it after most GET checks. The Hetzner servers do not answer HEAD.
test("a round checks each S3 region with a HEAD request and every other target with a GET request", async (t) => {
const state = new AppState([
{ name: "Local CPE", url: "http://192.168.100.1" },
]);
mockTargets(t, () => 10);
const round = tick(state, new AbortController().signal);
t.mock.timers.tick(10);
assert.notEqual(await settled(round), "still waiting");
const methods = new Map(
fetch.mock.calls.map((call) => [
call.arguments[0],
call.arguments[1].method,
]),
);
for (const host of state.allHosts) {
const method = host.name.startsWith("S3 ") ? "HEAD" : "GET";
assert.equal(methods.get(host.url), method, host.name);
}
});
test("at a 30000ms interval, a target answering after 1000ms shows in its row while another target's check is still waiting", async (t) => {
CONFIG.updateInterval = 30000;
const state = new AppState([
+44 -25
View File
@@ -67,51 +67,66 @@ export const CONFIG = {
// WAN endpoints to monitor. These are used for the aggregate health/stats
// display (min/max/avg). Ordered: personal, LLM APIs, big-3 cloud, then
// CDN/hosting/other.
// CDN/hosting/other. A check follows a redirect and times the request to
// the new URL too. So a target whose home page redirects is checked at
// /robots.txt on its own host. That URL answers without a redirect.
const WAN_HOSTS = [
{ name: "datavi.be", url: "https://datavi.be" },
{ name: "Anthropic API", url: "https://api.anthropic.com" },
{ name: "OpenAI API", url: "https://api.openai.com" },
{ name: "AWS Console", url: "https://console.aws.amazon.com" },
{ name: "Google Cloud Console", url: "https://console.cloud.google.com" },
{ name: "AWS Console", url: "https://console.aws.amazon.com/robots.txt" },
{
name: "Google Cloud Console",
url: "https://console.cloud.google.com/robots.txt",
},
{ name: "Microsoft Azure", url: "https://portal.azure.com" },
{ name: "Cloudflare", url: "https://www.cloudflare.com" },
{ name: "Fastly CDN", url: "https://www.fastly.com" },
{ name: "Akamai", url: "https://www.akamai.com" },
{ name: "Akamai", url: "https://www.akamai.com/robots.txt" },
{ name: "Google", url: "https://www.google.com" },
{ name: "GitHub", url: "https://github.com" },
{ name: "B2", url: "https://api.backblazeb2.com" },
{ name: "B2", url: "https://api.backblazeb2.com/robots.txt" },
// Chrome keeps the connection to an S3 region open after a HEAD check,
// but closes it after most GET checks.
{
name: "S3 af-south-1 (Cape Town)",
url: "https://s3.af-south-1.amazonaws.com",
url: "https://s3.af-south-1.amazonaws.com/robots.txt",
method: "HEAD",
},
{
name: "S3 eu-west-2 (London)",
url: "https://s3.eu-west-2.amazonaws.com",
url: "https://s3.eu-west-2.amazonaws.com/robots.txt",
method: "HEAD",
},
{
name: "S3 me-south-1 (Bahrain)",
url: "https://s3.me-south-1.amazonaws.com",
url: "https://s3.me-south-1.amazonaws.com/robots.txt",
method: "HEAD",
},
{
name: "S3 ap-northeast-1 (Tokyo)",
url: "https://s3.ap-northeast-1.amazonaws.com",
url: "https://s3.ap-northeast-1.amazonaws.com/robots.txt",
method: "HEAD",
},
{
name: "S3 ap-southeast-1 (Singapore)",
url: "https://s3.ap-southeast-1.amazonaws.com",
url: "https://s3.ap-southeast-1.amazonaws.com/robots.txt",
method: "HEAD",
},
{
name: "S3 ap-southeast-2 (Sydney)",
url: "https://s3.ap-southeast-2.amazonaws.com",
url: "https://s3.ap-southeast-2.amazonaws.com/robots.txt",
method: "HEAD",
},
{
name: "S3 us-west-2 (Oregon)",
url: "https://s3.us-west-2.amazonaws.com",
url: "https://s3.us-west-2.amazonaws.com/robots.txt",
method: "HEAD",
},
{
name: "S3 sa-east-1 (São Paulo)",
url: "https://s3.sa-east-1.amazonaws.com",
url: "https://s3.sa-east-1.amazonaws.com/robots.txt",
method: "HEAD",
},
// Hetzner regional speed test servers — genuine per-DC endpoints
{
@@ -271,6 +286,7 @@ export class HostState {
constructor(host, pinned = false) {
this.name = host.name;
this.url = host.url;
this.method = host.method;
// Each entry is either a check's result, { timestamp, latency,
// error }, or a round skipped while paused, { timestamp,
// latency: null, paused: true }.
@@ -567,16 +583,18 @@ export class Reporter {
// --- Latency Measurement -----------------------------------------------------
// Checks one target. The check times out after CONFIG.requestTimeout; the
// caller can give it up sooner through the optional signal, which also ends
// it as a timeout. A failed check's reason says what went wrong and after
// how long; a check that answered has none.
// Checks one target with the given HTTP method, GET when none is given. The
// check times out after CONFIG.requestTimeout; the caller can give it up
// sooner through the optional signal, which also ends it as a timeout. A
// failed check's reason says what went wrong and after how long; a check
// that answered has none.
//
// The URL is fetched as written, with nothing added to it: the Hetzner
// speed-test servers close the connection without an answer when the URL
// has a query string, and cache: "no-store" keeps the browser's cache out
// of the measurement.
export async function measureLatency(url, signal) {
// The Hetzner speed-test servers close the connection without an answer
// when the URL has a query string or the request has a Referer header. So
// the URL is fetched as written, and referrerPolicy: "no-referrer" sends
// no Referer header, wherever the page is served from. cache: "no-store"
// keeps the browser's cache out of the measurement.
export async function measureLatency(url, signal, method = "GET") {
const controller = new AbortController();
const timeoutId = setTimeout(
() => controller.abort(),
@@ -588,9 +606,10 @@ export async function measureLatency(url, signal) {
try {
await fetch(url, {
method: "GET",
method,
mode: "no-cors",
cache: "no-store",
referrerPolicy: "no-referrer",
signal: controller.signal,
});
const latency = Math.round(performance.now() - start);
@@ -1290,7 +1309,7 @@ export async function tick(state, signal, onOffline) {
// the round started.
await Promise.all(
state.allHosts.map(async (host) => {
const r = await measureLatency(host.url, signal);
const r = await measureLatency(host.url, signal, host.method);
if (state.paused || signal.aborted || state.tickCount === 0) {
return;
}
@@ -1363,7 +1382,7 @@ function startRecoveryProbe(state, startRounds) {
const checks = new AbortController();
state._recoveryProbeChecks = checks;
for (const host of canaries) {
measureLatency(host.url, checks.signal).then((r) => {
measureLatency(host.url, checks.signal, host.method).then((r) => {
if (r.error !== null || checks.signal.aborted) return;
log.notice("Recovery probe: connectivity detected");
stopRecoveryProbe(state);