sneak/netwatch
1
0
Fork 0
Code Issues 2 Pull Requests 2 Actions 10 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/netwatch:next
Branches Tags
sneak/netwatch:main sneak/netwatch:next sneak/netwatch:prod sneak/netwatch:feature/mobile-detection-message sneak/netwatch:feat/responsive-mobile-redesign sneak/netwatch:feature/mobile-detection
..
compare: sneak/netwatch:7494fd6197ea729c9bcf38cc965d0e6314daf300
Branches Tags
sneak/netwatch:next sneak/netwatch:main sneak/netwatch:prod sneak/netwatch:feature/mobile-detection-message sneak/netwatch:feat/responsive-mobile-redesign sneak/netwatch:feature/mobile-detection
1 Commits
next .. 7494fd6197
Author SHA1 Message Date
sneak 7494fd6197 Add make add-dependency and make tidy (closes #45)
check / check (push) Successful in 4m0s
Details 
No entrypoint could change yarn.lock or go.mod: script/bootstrap
installs with --frozen-lockfile, so adding a package meant running
yarn by hand. make add-dependency PACKAGE=<name>@<version> shims to
the new script/add-dependency: yarn add --dev, then yarn install
--frozen-lockfile. make tidy shims to the new script/tidy, go mod
tidy in backend/; a Go module is added by importing it, or moved by
editing its require line, then make tidy. script/bootstrap is
unchanged.

Model: opus-5-5
 2026-10-04 02:25:34 +00:00
42 changed files with 619 additions and 1890 deletions
Expand all files Collapse all files
.dockerignore
+7 -79
View File
@@ -1,81 +1,9 @@
# .dockerignore does NOT use .gitignore semantics. Docker matches with
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
# `/` and an unprefixed pattern is anchored at the context root. Every
# depth-independent pattern therefore needs `**/`, or `config/.env` and
# `certs/server.key` still ship while this file reads as solved. Only
# genuinely root-anchored entries go unprefixed. Never transplant these
# into .gitignore, where `**/` is wrong.
#
# Matching is case-sensitive, so secrets use character ranges rather
# than an ALL-CAPS twin, which would still miss `Server.Key`.
#
# Extend with this repo's own host-built artifacts, written anchored:
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context.
# .git is sent without its config. Without a VERSION build argument the
# stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
# Each submodule keeps a config with the same exposure in its git directory
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root.
# KNOWN GAP: a repo running agents in subdirectories still ships
# `services/api/.claude/` and must add its own anchored entry.
node_modules
dist
tmp
.DS_Store
*.log
.claude
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Re-include a committed template with a negation if the
# build needs one: `!docs/example.env`.
**/*.[eE][nN][vV]
**/.[eE][nN][vV].*
**/.[eE][nN][vV][rR][cC]
# Private keys and the bundles carrying them. Public certificates
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
**/*.[pP][eE][mM]
**/*.[kK][eE][yY]
**/*.[pP]12
**/*.[pP][fF][xX]
**/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# Dependencies: restored inside the image, never copied in.
**/node_modules
# OS metadata.
**/.DS_Store
**/Thumbs.db
# Editor state: never a build input, and it churns COPY.
**/*.swp
**/*.swo
**/*~
**/*.bak
**/.idea
**/.vscode
**/*.sublime-*
# This repository's own entries, after the shared content above: the
# frontend build, the viewport test's output, what `make build` and
# `make run` in backend/ leave behind, and log files at the root.
/dist
/tmp
/backend/netwatch-server
/backend/data
/*.log
# .git is sent so the build can stamp the version, without its config.
.git/config
.editorconfig
-5
View File
@@ -10,8 +10,3 @@ insert_final_newline = true
[Makefile]
indent_style = tab
# This repository's own entries, after the shared content above.
[*.go]
indent_style = tab
.gitea/workflows/check.yml
+4 -1
View File
@@ -6,4 +6,7 @@ jobs:
steps:
# actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- run: script/cibuild
# script/cibuild bootstraps, runs every check and builds the
# image. script/bootstrap links what it installs into
# ~/.local/bin, so that has to be on PATH for the rest.
- run: PATH="$HOME/.local/bin:$PATH" script/cibuild
.gitignore
+5 -38
View File
@@ -11,51 +11,18 @@ Thumbs.db
.vscode/
*.sublime-*
# Agent scratch (worktrees of this repo, created and destroyed by
# in-flight tooling). Unanchored: .gitignore patterns already match at
# every depth, so no prefix is wanted here. This is not a .dockerignore
# entry and must not be given a `**/` prefix on the way into one.
.claude/
# Node
node_modules/
# Secrets. Unanchored like every entry above, so each matches at every
# depth. Matching is case-sensitive on Linux, so names use character
# ranges rather than a lowercase form that misses `Server.Key`.
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation after these lines, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# This repository's own entries, after the shared content above.
# Environment / secrets
.env
.env.*
*.pem
*.key
# Build output
dist/
tmp/
/backend/netwatch-server
# Go test binaries and coverage output
*.test
*.out
# Logs
*.log
.prettierignore
+5
View File
@@ -1,2 +1,7 @@
dist/
node_modules/
tmp/
yarn.lock
.claude/
# The org standard file, copied verbatim; backend/script/lint checks its sha256.
backend/.golangci.yml
Dockerfile
+77 -102
View File
@@ -1,121 +1,96 @@
# The one image netwatch ships: nginx serves the built frontend and
# passes /api/, /.well-known/healthcheck and /metrics to netwatch-server,
# the Go backend, which runs in the same container on loopback only.
# passes /api/ and /.well-known/healthcheck to netwatch-server, the Go
# backend, which runs in the same container on loopback only.
# bin/entrypoint.sh starts and watches both.
#
# The lint and test phases are the gates: `make lint` (script/lint)
# builds the lint stage alone and `make test` (script/test) the test
# stage alone, and the builder stage depends on both, so the image
# cannot be built unless they pass. Each covers the frontend as well,
# through a copy from a node stage. Inside them each tool is invoked
# directly, never through make or script/, whose lint and test are
# themselves docker builds.
# Frontend lint stage: eslint with the rules in eslint.config.js. The
# lint phase below runs it.
# Lint stage — fast feedback on formatting and lint issues. The
# golangci/golangci-lint image ships Go, gofmt, make and the linter, so
# nothing is installed here. The root make lint builds this stage alone.
# golangci/golangci-lint:v2.12.2 (2026-08-10)
FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
WORKDIR /src
COPY backend/go.mod backend/go.sum ./
RUN go mod download
COPY backend/ .
RUN make fmt-check
RUN make lint
# Backend build stage
# golang:1.25-alpine (2026-02-27)
FROM golang:1.25-alpine@sha256:f6751d823c26342f9506c03797d2527668d095b0a15f1862cddb4d927a7a4ced AS builder
# gcc and musl-dev are for make test: its race detector needs cgo, which
# Go turns on by itself once a C compiler is present. make build still
# sets CGO_ENABLED=0, so the binary stays static.
RUN apk add --no-cache gcc git make musl-dev
WORKDIR /src
# Force BuildKit to run the lint stage before proceeding. BuildKit runs
# stages in parallel by default; without this no-op copy a lint failure
# would not gate compilation.
COPY --from=lint /src/go.sum /dev/null
COPY backend/go.mod backend/go.sum ./
RUN go mod download
COPY backend/ .
RUN make test
# make build is a shim around backend/script/build, the one definition
# of the build command:
# CGO_ENABLED=0 go build -trimpath -ldflags "-s -w -X main.Version=..."
# That script reads VERSION from the environment, so it is handed over
# there rather than as a make variable.
#
# The version is the VERSION build argument when one is given, otherwise
# `git describe --tags --always` of the repo's .git: the tag on a tagged
# commit, tag-N-gHASH on a commit after one, the short commit when no
# tag is reachable. A version that still comes out empty, dev or unknown
# fails the build. .git goes to /git, not /src/.git, where go build would
# find it and record VCS details of a work tree holding only backend/.
COPY .git /git
ARG VERSION
RUN version="${VERSION:-$(git --git-dir=/git describe --tags --always)}"; \
case "$version" in ""|dev|unknown) \
echo "version is '$version' although .git is present" >&2; \
exit 1 ;; \
esac; \
VERSION="$version" make build
# Frontend lint stage — eslint over the JavaScript, as the lint stage
# above lints the Go. The root make lint builds this stage alone too.
# node:22-alpine as of 2026-02-22
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS frontend-lint
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile
COPY . .
RUN yarn eslint .
RUN script/frontend-lint
# Lint phase: golangci-lint over the backend with backend/.golangci.yml,
# and eslint through the copy from frontend-lint at the end. The
# golangci/golangci-lint image ships Go and the linter.
# golangci/golangci-lint:v2.14.0, 2026-09-24
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
WORKDIR /src
COPY backend/go.mod backend/go.sum ./
RUN go mod download
COPY backend/ .
RUN golangci-lint run --config .golangci.yml ./...
# Nothing is wanted from frontend-lint; the copy is what makes this
# phase run it.
COPY --from=frontend-lint /app/yarn.lock /dev/null
# Frontend stage: the unit tests in test/unit/, then the production
# build into dist/, which the runtime stage serves. The test phase below
# runs it. The tests print a dot each; if any fails, they run again with
# every test listed, and the step fails even if that run passes.
# NODE_OPTIONS chooses the reporter because yarn adds its arguments
# after the test files, where node would take a reporter option for one
# more file.
# Frontend stage
# node:22-alpine as of 2026-02-22
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS frontend
WORKDIR /app
# Force BuildKit to run the frontend-lint stage before proceeding, as
# the builder stage does with the lint stage: without this no-op copy an
# eslint failure would not gate the image.
COPY --from=frontend-lint /app/yarn.lock /dev/null
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile
# vite.config.js reads the commit for the page's footer with git.
RUN apk add --no-cache git
RUN apk add --no-cache git make
COPY . .
RUN NODE_OPTIONS=--test-reporter=dot timeout 90 yarn --silent run test || \
{ echo "--- Rerunning with every test listed for details ---"; \
NODE_OPTIONS=--test-reporter=spec timeout 90 yarn --silent run test; \
exit 1; }
RUN yarn build
# make frontend-check runs the frontend tests and format check; its test
# step runs the unit tests, then the production yarn build, so this both
# produces dist/ and gates the image on test and formatting regressions.
# This node stage has neither Go nor Docker; the frontend-lint, lint and
# builder stages above gate the rest.
RUN make frontend-check
# Test phase: the backend's tests with the race detector and coverage,
# and the frontend's through the copy from the frontend stage at the
# end. -race needs cgo and so a C compiler, which the Debian Go image
# ships and the alpine one does not. -timeout 90s is a backstop above
# the 60-second cap on the suite. The rerun with -v only shows details:
# the step fails however it ends, because the first run already failed.
# Go's module and build caches are in memory (tmpfs) for the go test
# step alone, so the image make test tags does not carry them and the
# build spends no time writing them into it. They start empty on every
# build, so no stored test result can stand in for a run.
# golang:1.25.7-trixie, 2026-10-07
FROM golang@sha256:2b174ffcf56c7ad0c47d30d2630693265639ddf2a5141149c2da34db921791b4 AS test
WORKDIR /src
COPY backend/ .
RUN --mount=type=tmpfs,target=/go/pkg/mod \
--mount=type=tmpfs,target=/root/.cache/go-build \
go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
# Nothing is wanted from the frontend stage; the copy is what makes
# this phase run its tests.
COPY --from=frontend /app/yarn.lock /dev/null
# Backend build stage. Nothing is wanted from the two phases; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# golang:1.25-alpine (2026-02-27)
FROM golang@sha256:f6751d823c26342f9506c03797d2527668d095b0a15f1862cddb4d927a7a4ced AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY backend/go.mod backend/go.sum backend/
RUN cd backend && go mod download
COPY . .
# backend/script/build is the one definition of the build command:
# CGO_ENABLED=0 go build -trimpath -ldflags "-s -w -X main.Version=..."
# It reads VERSION from the environment.
#
# The version is the VERSION build argument when one is given, otherwise
# `git describe --tags --always` on the .git in the build context: the
# tag on a tagged commit, tag-N-gHASH on a commit after one, the short
# commit when no tag is reachable. With .git present, a version that is
# still empty, dev or unknown fails the build: git is missing or could
# not read the checkout.
ARG VERSION
RUN version="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$version" in ""|dev|unknown) \
echo "version is '$version' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
VERSION="$version" backend/script/build
# Runtime stage, and the last one: a plain `docker build .` builds it
# and the stages it copies from, the two phases included.
# Runtime stage
# nginx:stable-alpine as of 2026-02-22
FROM nginx@sha256:15e96e59aa3b0aada3a121296e3bce117721f42d88f5f64217ef4b18f458c6ab
@@ -131,7 +106,7 @@ RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/templates/netwatch.conf.template
COPY security-headers.conf /etc/nginx/security-headers.conf
COPY --from=frontend /app/dist /usr/share/nginx/html
COPY --from=builder /src/backend/netwatch-server /usr/local/bin/netwatch-server
COPY --from=builder /src/netwatch-server /usr/local/bin/netwatch-server
COPY bin/entrypoint.sh /usr/local/bin/entrypoint.sh
# bin/entrypoint.sh creates DATA_DIR at start and gives it and /data to
Makefile
+10 -5
View File
@@ -1,5 +1,5 @@
.PHONY: bootstrap setup dev build test lint fmt fmt-check check \
add-dependency tidy frontend-viewport-test docker hooks
add-dependency tidy frontend-check frontend-viewport-test docker hooks
# Standard targets are thin shims; the implementations live in script/
# per the scripts-to-rule-them-all pattern (see the Entrypoints section
@@ -34,16 +34,21 @@ fmt-check:
check:
@script/check
# make add-dependency PACKAGE=<name>@<version>. PACKAGE reaches the
# script through the environment, so the shell never reads it as code.
# make add-dependency PACKAGE=<name>@<version>
add-dependency:
@script/add-dependency "$$PACKAGE"
@script/add-dependency "$(PACKAGE)"
tidy:
@script/tidy
# The frontend tests and format check, for Dockerfile's frontend stage,
# which has neither Go nor Docker. Use check everywhere else.
frontend-check:
@script/frontend-check
# Responsive-layout verification in a containerised browser. Kept out of
# check: it takes minutes.
# check: it needs Docker and takes minutes, where make test has to stay
# under 20 seconds.
frontend-viewport-test:
@script/frontend-viewport-test
README.md
+82 -151
View File
@@ -1,33 +1,30 @@
NetWatch is an MIT-licensed JavaScript single-page application by
[@sneak](https://sneak.berlin) that provides real-time network latency
monitoring to common internet hosts, displayed with color-coded figures and
sparkline graphs, served from a static bucket or from its Docker image, where a
small Go backend stores the measurements the page reports.
sparkline graphs, served from a static bucket or Docker container.
## Getting Started
```bash
# Install the dependencies and the git pre-commit hook
make setup
# Install dependencies
yarn install
# Run the page on the Vite dev server
make dev
# Development server
yarn dev
# Run the tests, both linters and the format check
make check
# Production build
yarn build
# Build the page into dist/
make build
# Preview production build
yarn preview
# Build the image and run it
make docker
# Docker
docker build -t netwatch .
docker run -p 8080:8080 netwatch
```
`make check` and `make docker` need Docker. `make dev` passes `/api` to
`http://127.0.0.1:8080`, where `make run` in `backend/` starts `netwatch-server`
with its defaults, so the reports the page posts are stored in
`backend/data/reports`.
`yarn dev` proxies `/api` to `http://127.0.0.1:8080`, so a locally running
`netwatch-server` (see `backend/`) receives the reports the page posts.
## Entrypoints
@@ -41,12 +38,12 @@ halves, so the root `make check` fails if either one is broken. We provide:
- `script/bootstrap` — install all dependencies (the pinned node via nvm unless
one new enough for the frontend's dependencies is installed, yarn via
corepack, `yarn install --frozen-lockfile`, Go `GO_VERSION` unless the `go` on
`PATH` is exactly that version, the Go modules, and gcc with the C library
headers unless gcc is installed, for the race detector in `make test` in
`backend/`), linking what it installs itself into `~/.local/bin`, which has to
be on `PATH`. It installs no Go linter and not Docker: `make test` and
`make lint` run in Docker
corepack, `yarn install --frozen-lockfile`, the pinned Go unless one at least
as new as `backend/go.mod` asks for is installed, the Go modules, and gcc with
the C library headers unless gcc is installed, for the race detector in
`make test`), linking what it installs itself into `~/.local/bin`, which has
to be on `PATH`. It installs no Go linter and not Docker: `make lint` runs
both linters in Docker
- `script/setup` — make a fresh clone ready for development: bootstrap plus the
git pre-commit hook
- `script/dev` — run the Vite dev server, which proxies `/api` to a locally
@@ -54,14 +51,11 @@ halves, so the root `make check` fails if either one is broken. We provide:
- `script/build` — build the frontend for production into `dist/`;
`backend/script/build` builds the Go server
- `script/projectname` — print the project name (used for the Docker image tag)
- `script/test` — build the `test` phase of `Dockerfile` without the cache: the
frontend's unit tests and production build in its `frontend` stage, and the
- `script/test` — run `script/frontend-test`, then `backend/script/test`, the
backend's Go tests with the race detector and coverage
- `script/lint` — build the `lint` phase of `Dockerfile` without the cache:
eslint in its `frontend-lint` stage, and golangci-lint over `backend/`
- `script/fmt` — format all files (writes): prettier over the JavaScript, CSS,
HTML and Markdown, then gofmt over `backend/`. It runs on the host, as
`script/fmt-check` does, with `~/.local/bin` put on `PATH`
- `script/lint` — run eslint, then golangci-lint, both in Docker, by building
the `frontend-lint` and `lint` stages of `Dockerfile` without the cache
- `script/fmt` — format all files (writes): prettier, then gofmt over `backend/`
- `script/fmt-check` — check formatting (read-only): prettier, then gofmt
- `script/check` — run test, lint, and fmt-check
- `script/add-dependency` — add a frontend package, or move one to another
@@ -71,18 +65,23 @@ halves, so the root `make check` fails if either one is broken. We provide:
- `script/tidy` — run `go mod tidy` in `backend/`: to add a Go module, import it
and run `make tidy`; to move one to another version, edit its `require` line
in `backend/go.mod`, then run `make tidy`
- `script/frontend-test` — run the unit tests in `test/unit/` on the host with
Node's built-in test runner, through the `test` script in `package.json`, and
if any fails, run them again listing every test, and fail; then the production
build. Each test run has a 90-second timeout. `make test` runs the same in
Docker
- `script/frontend-fmt` — format the JavaScript, CSS, HTML and Markdown with
prettier (writes), the markdown in `backend/` included
- `script/frontend-test` — run the unit tests in `test/unit/` with Node's
built-in test runner, through the `test` script in `package.json`, and if any
fails, run them again listing every test, and fail; then the production build.
Each run has a 30-second timeout
- `script/frontend-lint` — run eslint with the rules in `eslint.config.js`; it
runs inside the `frontend-lint` stage of `Dockerfile`, which `make lint`
builds
- `script/frontend-fmt` — format everything prettier understands (writes), the
markdown in `backend/` included
- `script/frontend-fmt-check` — check prettier formatting (read-only)
- `script/frontend-check` — run `script/frontend-test` and
`script/frontend-fmt-check`, for the frontend stage of `Dockerfile`, which has
neither Go nor Docker
- `script/frontend-viewport-test` — responsive-layout verification of the built
frontend in a containerised headless Chrome (see
[test/viewport/README.md](test/viewport/README.md)). Not part of
`script/check`: it takes minutes.
`script/check`: it needs Docker and takes minutes.
- `script/docker` — build the image from `Dockerfile` without the build cache,
tagged `netwatch` via `script/projectname`
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
@@ -95,30 +94,25 @@ halves, so the root `make check` fails if either one is broken. We provide:
The narrow-viewport layout lives in the `max-width: 768px` media block in
`src/styles.css`. It is verified automatically by `make frontend-viewport-test`,
which drives a digest-pinned headless Chrome against the built `dist/` and
asserts on computed layout at widths derived from that CSS — on every breakpoint
it declares and one pixel either side of it, plus a 320px floor, a desktop
baseline and two landscape sizes. See
[test/viewport/README.md](test/viewport/README.md) for what it covers and what
it genuinely cannot.
asserts on computed layout at widths derived from that CSS — one pixel either
side of every breakpoint it declares, plus a 320px floor, a desktop baseline and
two landscape sizes. See [test/viewport/README.md](test/viewport/README.md) for
what it covers and what it genuinely cannot.
## Rationale
When debugging network issues, it's useful to have a persistent at-a-glance view
of latency and reachability to multiple well-known internet endpoints. NetWatch
provides this as a single page that does all its measuring in the browser, so it
can be served from anywhere static files are served. The backend in its Docker
image only stores the measurements the page reports; without it, the page works
the same and nothing is stored.
provides this as a zero-dependency SPA that can be deployed anywhere static
files are served, with no backend required.
## Design
The page is built with Vite and Tailwind CSS v4. Its code is all in
`src/main.js`, with a class-based architecture:
The application is a single-page app built with Vite and Tailwind CSS v4. All
code lives in `src/main.js` with a class-based architecture:
- **`CONFIG`**: Configuration object (update interval, timeouts, axis ticks,
etc.). The interval menu sets `updateInterval`, the one value the page writes
into `CONFIG`; the timeouts, the time the history spans and the x-axis ticks
are computed from it
- **`CONFIG`**: Frozen configuration object (update interval, timeouts, axis
ticks, etc.)
- **`HostState`**: Per-host state management — history buffer, latency tracking,
status transitions
- **`AppState`**: Top-level state container — WAN hosts, local hosts, pause
@@ -129,10 +123,10 @@ The page is built with Vite and Tailwind CSS v4. Its code is all in
`updateSummary()` / `updateHealthBox()` handle incremental updates
- **`tick()`**: Main loop — measures all hosts in parallel, pushing each host's
sample and redrawing its row as soon as its check ends, then redraws every
row, the summary and the health box once the last check ends. The first round,
after loading or an interval change, is discarded. The rows are sorted when
the last check ends in round 2, the first one kept, and in rounds 11, 21, 31
and so on. When paused, pushes blank markers (no probes, no false outage)
row, the summary and the health box once the last check ends. The rows are
sorted then too, after the first round that is not discarded and every tenth
round after that. When paused, pushes blank markers (no probes, no false
outage)
- **`Reporter`**: Posts collected samples to the backend
### Reporting
@@ -146,35 +140,12 @@ delivered report, and while paused nothing is sent. Delivery failure is quiet
one debug-log line per outage, retried at the next interval, never blocking
probing. The report-building step is a pure function of host state.
### Backend
`netwatch-server`, in `backend/`, is a small Go HTTP server that stores the
reports the page posts. It keeps them in memory and writes them to `DATA_DIR` as
zstd-compressed files of JSON lines: every minute, whenever 10 MiB are waiting,
and when it stops. Its routes:
- `POST /api/v1/reports` — takes a report, without credentials; each client
address may send a limited number a minute, and the report files are capped in
size, the oldest deleted first
- `GET /.well-known/healthcheck` — answers 200 with `"status":"ok"`, the
server's version and its uptime
- `GET /metrics` — Prometheus metrics behind basic auth, only when
`METRICS_USERNAME` and `METRICS_PASSWORD` are set; each client address may
make a limited number of requests to it a minute
In the image, the `test` phase of `Dockerfile` tests it, the `builder` stage
builds it with `backend/script/build`, and `bin/entrypoint.sh` runs it as user
`netwatch` on `127.0.0.1:8081`, behind nginx. Outside the image, `make run` in
`backend/` builds it and runs it on port 8080. Its settings, report storage and
limits are in [backend/README.md](backend/README.md).
### Monitoring targets
- **26 WAN hosts**: datavi.be (pinned at start), Anthropic API, OpenAI API, AWS
Console, Google Cloud Console, Microsoft Azure, Cloudflare, Fastly CDN,
Akamai, Google, GitHub, B2, 8 S3 regional endpoints (Cape Town, London,
Bahrain, Tokyo, Singapore, Sydney, Oregon, São Paulo) and 6 Hetzner speed test
servers (Nuremberg, Falkenstein, Helsinki, Ashburn, Hillsboro, Singapore)
- **22 WAN hosts**: datavi.be, Anthropic API, OpenAI API, AWS Console, GCP
Console, Azure, Cloudflare, Fastly, Akamai, GitHub, B2, 7 S3 regional
endpoints (Cape Town, London, Bahrain, Tokyo, Sydney, Oregon, São Paulo), 4
GCS locational endpoints (Iowa, Belgium, Singapore, Sydney)
- **Local CPE**: Cable modem at 192.168.100.1 (always monitored)
- **Local Gateway**: Auto-detected on startup by probing common default gateway
addresses (192.168.1.1, 192.168.0.1, 192.168.8.1, 10.0.0.1); first responder
@@ -186,28 +157,15 @@ Local hosts are tracked separately from WAN stats.
### Latency measurement
GET requests to each target's URL as written, with `mode: 'no-cors'` and
`cache: 'no-store'`, timed with `performance.now()`. No query string is added:
the Hetzner speed-test servers close the connection without an answer when the
URL has one, and `no-store` keeps the browser's cache out of the measurement.
Each check times out after 80% of the refresh interval (24 seconds at 30
seconds) and is then recorded as a timeout, so a round's checks have all
finished before the next round is due. When no WAN host answers, a recovery
probe checks 4 WAN hosts, picked at random when it starts, every half second,
giving up the checks it started half a second before. As soon as one answers, a
new round starts at once, as it does after an interval change. A round started
early gives up the last round's checks if they are still waiting, and that round
records nothing more, so rounds never overlap. The browser chooses between IPv4
and IPv6 for each target, as for any request; the local targets are IPv4
addresses.
Each recorded check that fails writes one line to the browser console with
`console.error`, and the same line to the debug log: the target's name and URL,
the time, whether it timed out, answered over the time limit or hit a network
error (with the error the browser gives the page, such as
`TypeError: Failed to fetch`, which does not say why), and how long the request
took. A target that answers after failed checks writes one `console.info` line
with its latency and how many checks in a row had failed.
HEAD requests with `mode: 'no-cors'` and `cache: 'no-store'`, timed with
`performance.now()`. Each check times out after 80% of the refresh interval (24
seconds at 30 seconds) and is then recorded as a timeout, so a round's checks
have all finished before the next round is due. When no WAN host answers, a
recovery probe checks 4 random WAN hosts every half second, giving up the checks
it started half a second before. As soon as one answers, a new round starts at
once, as it does after an interval change. A round started early gives up the
last round's checks if they are still waiting, and that round records nothing
more, so rounds never overlap. IPv4 only.
### Color coding
@@ -232,44 +190,27 @@ dist/
## Features
- A round of checks every 3 seconds by default; the interval menu sets 1, 2, 3,
5, 10, 15, 30 or 60 seconds and clears the history
- Sparklines of each target's last 100 rounds: 300 seconds at 3 seconds
- The first round after loading or an interval change is discarded, as DNS and
TLS setup inflate its latencies
- Health indicator from the WAN hosts' latest results: OFFLINE (red) when more
than 10 fail and at most 4 answer, otherwise DEGRADED (orange) when more than
4 fail, otherwise SLOW (yellow) when more than 3 take over 1000ms, otherwise
HEALTHY (green)
- Summary stats across WAN hosts only: how many answered, the min, median,
average and max of their latest latencies, the min and max over the whole
history, and the number of rounds run (`Checks`)
- Fixed chart axes: Y-axis 0–1000ms, higher latencies drawn at the top; X-axis
the time the history spans
- Real-time monitoring with 2s update interval and 300s history sparklines
- Health indicator: green (HEALTHY) or red (DEGRADED) based on WAN reachability
- Summary stats: reachable count, min/max/avg latency across WAN hosts only
- Fixed chart axes: Y-axis 0–1000ms, X-axis 0–300s
- Color-coded latency figures and sparkline line segments
- WAN host rows sorted by latest latency, unreachable last; pinned rows stay on
top, in name order
- Play/pause: pause stops probes but history keeps scrolling (blank gaps, no
false outage)
- Debug log panel, behind a checkbox in the footer, with five levels (error,
warning, notice, info, debug) and the last 1000 lines
- Local and UTC clocks
- Clickable service URLs
- A footer link to the commit the page was built from
- Canvas-based sparkline rendering with devicePixelRatio scaling
- Zero runtime dependencies: all resources bundled into build artifacts
## Deployment
`make build` writes the page to `dist/`, which any static file host (S3, GCS,
Cloudflare Pages, Vercel, Netlify, GitHub Pages) can serve; with no backend
there, its reports fail quietly and nothing is stored. Or run the Docker image
behind a reverse proxy.
After running `yarn build`, deploy the contents of the `dist/` directory to any
static file host (S3, GCS, Cloudflare Pages, Vercel, Netlify, GitHub Pages) or
use the Docker image behind a reverse proxy.
The Docker image, built from `Dockerfile` by `make docker`, is the whole service
in one container: nginx serves the built frontend and passes `/api/`,
`/.well-known/healthcheck` and `/metrics` to the Go backend, `netwatch-server`,
which listens only inside the container, on `127.0.0.1:8081`. The image:
The Docker image, built from `Dockerfile`, is the whole service in one
container: nginx serves the built frontend and passes `/api/` and
`/.well-known/healthcheck` to the Go backend, `netwatch-server`, which listens
only inside the container, on `127.0.0.1:8081`. The image:
- Listens on port 8080 by default (override with `PORT` env var)
- Takes the client address from `X-Forwarded-For` only on requests from the
@@ -317,16 +258,6 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
connects from one can write its own `X-Forwarded-For`, and through a port
Docker publishes, every client may connect from the Docker network's
gateway, such as `172.17.0.1`.
- `METRICS_USERNAME` and `METRICS_PASSWORD`, default empty: with both set,
the backend records Prometheus metrics of its requests and serves them at
`/metrics` on the container port, to requests with this user name and
password as their basic auth credentials. With neither set, there are no
metrics and `/metrics` is not found. One set without the other, or a user
name containing `:`, stops the container
- `SENTRY_DSN`, default empty: set to a Sentry project's DSN, the backend
sends its errors to that Sentry project: each request whose handling
crashes, which still gets a 500 response. A value Sentry does not accept
stops the container. Empty, the backend sends nothing to Sentry
- **Health check:** the image's `HEALTHCHECK` requests
`/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless
both nginx and the backend answer. upaas reads the container's health 60
@@ -340,19 +271,19 @@ properties.
## Limitations
- **CORS**: The checks are cross-origin requests in `no-cors` mode, so the page
cannot read the answer, only time it: any answer counts as reachable, an error
page included.
- **Local targets**: The cable modem at 192.168.100.1 and the detected gateway
answer only on a network that has them, and only when NetWatch is served from
localhost or a private address (see Monitoring targets).
- **CORS**: Some hosts may block cross-origin HEAD requests. The app uses
`no-cors` mode which allows the request but provides opaque responses. Latency
is still measurable based on request timing.
- **Local gateway**: The 192.168.100.1 endpoint requires the host to be
accessible from your network.
- **Network conditions**: Measurements reflect browser-to-endpoint latency,
which includes your local network, ISP, and internet routing.
## TODO
The to-do list is [TODO.md](TODO.md): where the work stands, the next step, the
open work, and what has been done.
- Add configurable host list (environment variable or config file)
- Add latency history export (CSV/JSON)
- Add notification/alert when status changes to DEGRADED
## License
REPO_POLICIES.md
+82 -353
View File
@@ -1,6 +1,6 @@
---
title: Repository Policies
last_modified: 2026-10-04
last_modified: 2026-07-06
---
This document covers repository structure, tooling, and workflow standards. Code
@@ -60,28 +60,17 @@ style conventions are in separate documents:
prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
with the version; the Gitea workflow calls it. **`script/cibuild` runs
`script/bootstrap` first**, because the workflow checks out the repo and runs
nothing else, while `script/fmt-check` runs the formatter on the host: on a
pristine checkout with nothing installed the run dies there, after the
containerised gates have passed. **The bootstrap alone is not enough**:
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
source nvm for the pinned node version before invoking it, exactly as
`script/bootstrap`'s own install step does. A runner carrying nothing but
docker and git then gets through `script/check`. Four further scripts are our
own extensions to the standard: `script/check` runs `script/test`,
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
installs the git pre-commit hook (the `make hooks` target shims to it); and
`script/projectname` (literally that filename) simply outputs the project's
name. Scripts that need the name call `script/projectname` — e.g.
`script/docker` assembles its image tag from it — so those scripts stay
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
the hook itself. Model scripts are at
repo root and runs `docker build .`; the Gitea workflow calls it. Four further
scripts are our own extensions to the standard: `script/check` runs
`script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
what the git pre-commit hook runs, and it calls `script/check`;
`script/install-precommit` installs the git pre-commit hook (the `make hooks`
target shims to it); and `script/projectname` (literally that filename) simply
outputs the project's name. Scripts that need the name call
`script/projectname` — e.g. `script/docker` assembles its image tag from it —
so those scripts stay byte-identical across all repos. Repo-type-specific
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
`script/precommit`, not in the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the
README requirements below).
@@ -100,198 +89,87 @@ style conventions are in separate documents:
contributor should be able to understand the entire development workflow by
reading the Makefile.
- Every repo should have a `Dockerfile`, and it carries the repo's gates: a
`lint` phase and a `test` phase, with the final stage depending on both so the
image cannot be built unless they pass. For non-server repos the final stage
brings up a development environment; for server repos it is the runtime image.
The gate phases and the build stage start from their pinned base images and
install what those images lack either inline, as the canonical Go `Dockerfile`
below does for `git`, or by running `script/bootstrap`, as the `prompts`
repo's own `Dockerfile` does for its yarn packages. The development
environment stage installs development prerequisites by running
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- Every repo should have a `Dockerfile`. All Dockerfiles must run `make check`
as a build step so the build fails if the branch is not green. For non-server
repos, the Dockerfile should bring up a development environment and run
`make check`. For server repos, `make check` should run as an early build
stage before the final image is assembled. Dockerfiles install development
prerequisites by running `script/bootstrap` rather than duplicating installs
inline; COPY `script/` and the dependency manifests (`package.json` +
`yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
layer stays cached until dependencies change.
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
no separate lint file. `script/lint` and `script/test` each build one phase
and nothing else:
- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
repos use a multistage build where linting runs in an independent stage based
on the `golangci/golangci-lint` image (pinned by hash). This stage runs
`make fmt-check` and `make lint` before the full build begins. The build stage
then declares an explicit dependency on the lint stage via
`COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
linting before proceeding to compilation and tests. This ensures lint failures
surface in seconds rather than minutes, without blocking on dependency
download or compilation in the build stage.
```sh
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
docker build --no-cache --target test -t "$(script/projectname)-test" .
```
**A stage that is not the last one in the file is built only when the final
stage's chain depends on it, or when `--target` names it.** That is why the
two gates are always invoked by name here, and why the final stage carries a
`COPY --from=` of a harmless file from each of them: without that edge a
plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing.
**Every `docker build` in `script/` is tagged**, here and in
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
image behind on every invocation, on every developer host and every CI
runner; a tagged one replaces the previous image.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
themselves a `docker build` and would recurse into a daemon that does not
exist in a build step. Formatting is the exception and stays on the host:
`script/fmt` writes the working tree, and `script/fmt-check` is its
read-only twin.
**No lint verdict may come from a host invocation of the linter.** On a
shared host golangci-lint reads a result cache keyed on file content rather
than location, so a second checkout of the same content is served the first
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own cache, its own `TMPDIR` and
a digest-pinned binary, so neither is reachable.
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
a `COPY` layer only when the copied content changes, so on an unchanged tree
the check `RUN` is served from cache, nothing executes, and the build still
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
four, and there is no fifth — `script/check` runs the two gate phases and
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
not evidence that anything ran: a sub-second build reporting success is a
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
and friends destroy a build cache shared with every other build on the host.
When a check is added or changed, prove it works by planting a defect it must
catch and watching the run fail on it, then revert the defect. A green run
alone shows neither that the check ran nor that it covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Debian Go image. The canonical Go repo
`Dockerfile`:
The standard pattern for a Go repo Dockerfile is:
```dockerfile
# Lint phase
# Lint stage — fast feedback on formatting and lint issues
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN golangci-lint run --config .golangci.yml ./...
RUN make fmt-check
RUN make lint
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# Build stage
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
# Force BuildKit to run the lint stage before proceeding
COPY --from=lint /src/go.sum /dev/null
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make test
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
ARG VERSION=dev
RUN CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage, and the last one
# Runtime stage
FROM alpine@sha256:...
COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"]
```
Key points:
- The lint phase uses the `golangci/golangci-lint` image directly (it has
both Go and the linter), so nothing needs installing.
- `COPY --from=<phase> /src/go.sum /dev/null` is a no-op copy whose only
purpose is the ordering edge. BuildKit runs stages in parallel by default,
and a stage nothing depends on is not built at all, so without these two
lines a red gate would not fail the build.
- Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain
and nothing else.
- The lint stage uses the `golangci/golangci-lint` image directly (it
includes both Go and the linter), so there is no need to install the
linter separately.
- `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates
a stage dependency. BuildKit runs stages in parallel by default; without
this line, the build stage would not wait for lint to finish and a lint
failure might not fail the overall build.
- If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint phase must
(e.g. a web frontend compiled in a separate stage), the lint stage must
create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
- If the project requires CGO or system libraries for linting, install them
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
has no `apk`, so install with `apt-get` under the Debian package name
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them:
```dockerfile
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
The lint stage should not depend on the actual build output — it exists to
fail fast.
- If the project requires CGO or system libraries for linting (e.g.
`vips-dev`), install them in the lint stage with `apk add`.
- The build stage runs `make test` after compilation setup. Tests run in the
build stage, not the lint stage, because they may require compiled
artifacts or heavier dependencies.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` on push, and checks out the repo as its only other step.
That script bootstraps, runs the gate phases, and then builds the image, so a
successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. A separate
workflow limited to `main` by a `branches` list under `on: push` cannot be
checked by review: to try a change to it, add the feature branch to that list
and push, then remove the branch from the list again before merging. Keep any
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
from the feature branch publishes nothing.
runs `script/cibuild` (which runs `docker build .`) on push. Since the
Dockerfile already runs `make check`, a successful build implies all checks
pass.
- Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -311,21 +189,14 @@ style conventions are in separate documents:
module under test to verify it compiles/parses. There is no excuse for
`make test` to be a no-op.
- `make test` must complete in under 60 seconds. That is the hard cap, and a
suite that exceeds it fails. Under 20 seconds is the target. A suite between
20 and 60 seconds is still green, but the overage must be filed as an
improvement bug against that repo. Add a 90-second timeout to the test
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
hard cap so that it catches a genuinely hung test rather than a merely slow
one.
- `make test` must complete in under 20 seconds. Add a 30-second timeout in the
Makefile.
- **The test command should use the conditional verbose rerun pattern.** Run
tests without `-v` (verbose) first. If tests fail, automatically rerun with
`-v` to show full output. This keeps CI logs and `docker build` output clean
on success (just package/suite summaries) while providing full diagnostic
detail on failure (every test case, every assertion). The command lives in the
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the
Makefile form below is the same pattern for any repo-local invocation:
- **`make test` should use the conditional verbose rerun pattern.** Run tests
without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to
show full output. This keeps CI logs and `docker build` output clean on
success (just package/suite summaries) while providing full diagnostic detail
on failure (every test case, every assertion). The general shell pattern:
```makefile
test:
@@ -338,26 +209,11 @@ style conventions are in separate documents:
```makefile
test:
@go test -count=1 -timeout 90s -race -cover ./... || \
@go test -timeout 30s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
go test -timeout 30s -race -v ./...; exit 1; }
```
`-count=1` is required on both invocations: it defeats Go's test _result_
cache, so neither run can report a stored pass in place of running the
tests. It leaves the build cache alone, so it costs the runtime of the suite
and no recompilation.
That cache is Go's own, separate from Docker's layer cache. Go stores a
passing result in its cache directory (`GOCACHE`), and when the same tests
run again on unchanged code it prints that result, marked `(cached)`,
without running them. That matters on a developer's machine, where this
target runs and the directory lasts from one run to the next. The `test`
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
result for this repo's tests and nothing before its `go test` step runs a
test, so there is nothing to replay. `--no-cache` (above) is what makes that
step run on an unchanged tree.
Python example:
```makefile
@@ -383,84 +239,10 @@ style conventions are in separate documents:
must be in `.gitignore`. No exceptions.
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
setting up a new repo. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified.
- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns
across unmodified leaves secrets in the build context.** Docker matches with
`moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension, so
`*` does not cross `/` and a pattern without a leading `**/` is anchored at
the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key`
therefore excludes only the copies at the repository root, while `config/.env`
and `certs/server.key` still reach the context and can land in an image layer
— which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
secret names use character ranges — `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`,
and likewise for `.envrc` and the extensionless SSH keys. Where such a pattern
also catches something the build needs, re-include it with a negation
(`!docs/example.env`); deleting the pattern reopens the exposure for every
other file it covers. Fetch the standard `.dockerignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend
it with the repo's own artifacts.
- **In-repo agent scratch belongs in both files, written to each file's own
semantics.** `.claude/` holds one worktree per in-flight agent — an entire
additional checkout of the repo — so under `COPY . .` the build context
inflates by a multiple of the repo and another session's unreviewed work can
be copied into an image layer. In `.gitignore` the entry is `.claude/`,
unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/`
prefix, because the prefixed form would also delete any nested directory of
that name from the build. Anchoring carries a known gap that the canonical
`.dockerignore` states in its own comment, since consuming repos receive the
file and not the tracker: the directory is created in the agent's working
directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there.
- **A plain `docker build .` of a clone stamps the version that
`git describe --tags --always` gives**, derived from the `.git` in the build
context as the canonical `Dockerfile` above shows. Without its failure check,
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
and the build would still exit 0. `script/docker` and `script/cibuild` pass
the version they compute on the host; it takes precedence. They do this
byte-identically across repos:
```sh
# Own line: a failing command substitution inside an argument does not
# trip `set -e`, so the inline form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$(script/projectname)" .
```
`--always` makes an untagged repo yield an abbreviated commit hash rather
than failing, and the `[ -n "$version" ]` line is the single place the
fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so a repo that embeds a
tag-derived version must set `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build
a probe image that does `COPY . .`, and list what actually landed
(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context`
size is not a substitute: a nested secret is a few bytes, and BuildKit
transfers only the delta from the previous build.
editor files (`.swp`, `*~`), language build artifacts, and `node_modules/`.
Fetch the standard `.gitignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
a new repo.
- **No build artifacts in version control.** Code-derived data (compiled
bundles, minified output, generated assets) must never be committed to the
@@ -476,56 +258,9 @@ style conventions are in separate documents:
- Make all changes on a feature branch. You can do whatever you want on a
feature branch.
- `.golangci.yml` is standardized. The vendored copy in a consuming repo must
_NEVER_ be modified by an agent: fetch it from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it
byte-identical, so that no repo can quietly loosen its own linting. Linter
configuration changes are made to the canonical copy in the `prompts` repo and
reach consuming repos by re-vendoring; an agent may open a PR against
canonical, which only the user merges. One list is exempt from byte-identity,
because it cannot be written once for every repo: the `deny` list of the
`test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
image
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
directive must not name a newer Go minor version than the one golangci-lint
was built with, or golangci-lint refuses to lint it: this release lints
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
`PATH` only, so on an already-provisioned machine the pin is inert and a
version bump is a silent no-op — while the Dockerfile, installing into a clean
image, gets the pinned version, so a local `make check` and `make docker` can
disagree about what the tool even is. The canonical form:
- compares the installed version against the pin over the **whole** version
token; a parser that stops at the first `-` reports `2.12.2` for a host
running `2.12.2-rc1` and skips the install;
- treats absent, non-zero, empty or unrecognised `--version` output as a
mismatch, so the failure direction is a redundant install and never a
skipped one;
- after installing, re-resolves the binary the way callers do — `hash -r`,
then through `PATH`, not through the directory the installer wrote to —
and fails naming the resolved path, since an install that a shadowing
binary hides succeeds while changing nothing any caller sees;
- is actually called, and prints the version on both success paths: a
function defined and never invoked has the same exit status and the same
empty output as one that worked.
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- `.golangci.yml` is standardized and must _NEVER_ be modified by an agent, only
manually by the user. Fetch from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`.
- When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD).
@@ -639,14 +374,12 @@ style conventions are in separate documents:
settings.
- Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
and language-specific config). Everything else goes in a subdirectory.
Canonical subdirectory names:
only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
language-specific config). Everything else goes in a subdirectory. Canonical
subdirectory names:
- `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
body is a single call into `internal/` or `pkg/`, no project logic in
`cmd/`
- `cmd/` — Go command entrypoints
- `configs/` — configuration templates and examples
- `deploy/` — deployment manifests (k8s, compose, terraform)
- `docs/` — documentation and markdown (README.md stays in root)
@@ -673,7 +406,3 @@ style conventions are in separate documents:
- Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
TODO.md
+19 -110
View File
@@ -1,104 +1,28 @@
# Workflow
- branch from `next`
- branch (from `main`)
- do the work in Next Step
- move Next Step to the top of Completed Steps
- move the top item of Future Steps into Next Step
- commit (`TODO.md` changes in the same commit as the work)
- push the branch and open a PR against `next`
- merge to `main` if the branch is not protected, otherwise open a PR
- push
# Status
pre-1.0. No git tags. `main` is the stable branch and `next` the development
branch, which every PR targets. The frontend and the Go backend ship as one
Docker image, and the Gitea workflow `.gitea/workflows/check.yml` runs
`script/cibuild` on every push. Working toward 1.0.0.
pre-1.0. No git tags. `feat/reportbuf-storage` is merged; the backend, the CI
workflow, and the backend repo standard files are all on `main`. Frontend and
backend are both functional. Working toward the 1.0.0 milestone by closing the
remaining repo-compliance issues on the tracker.
# Next Step
Decide whether the repo moves to the layout `REPO_POLICIES.md` gives, with
`backend/` no longer repeating files from the root
([#30](https://git.eeqj.de/sneak/netwatch/issues/30)).
Confirm the `.gitea/workflows/check.yml` run is green (main always green
policy). The workflow file is already on `main`; what is unverified is that its
latest run passes.
# Completed Steps
- 2026-10-07: the files shared from `sneak/prompts` are its copies at commit
`dd4027b` ([#113](https://git.eeqj.de/sneak/netwatch/issues/113)), with this
repository's own entries after the shared content in `.gitignore`,
`.dockerignore` and `.editorconfig`. `make lint` and `make test` each build
one phase of `Dockerfile` without the cache: `lint` runs golangci-lint v2.14.0
over `backend/` and, through the `frontend-lint` stage, eslint; `test` runs
the Go tests on the Debian Go image and, through the `frontend` stage, the
frontend's unit tests and build. The builder stage waits on both, and without
a `VERSION` build argument takes the version from `git describe` on the `.git`
in the build context. Neither linter nor the tests run on the host for
`make check`; `backend/script/lint` runs the root one, and
`backend/.golangci.yml` is no longer checked against a sha256. prettier
formats only the JavaScript, CSS, HTML and Markdown, so the shared
`.golangci.yml` stays as fetched. `script/fmt` and `script/fmt-check` put
`~/.local/bin` on `PATH`, which the shared workflow no longer does.
`script/frontend-lint` and `script/frontend-check` are gone
- 2026-10-07: the six Hetzner targets answer again, and failed checks are
written to the browser console
([#114](https://git.eeqj.de/sneak/netwatch/issues/114)). The Hetzner
speed-test servers close the connection without an answer when the URL has a
query string, and every check added `?_cb=` and the time; checks now fetch
each target's URL as written, with `cache: 'no-store'` as before. Each
recorded check that fails writes one `console.error` line, and the same line
to the debug log, with the target's name and URL, the time, what failed and
how long the request took; a target that answers after failed checks writes
one `console.info` line. Checks the page does not record write none, so the
debug log no longer lists failures in the first round or in the recovery probe
- 2026-10-04: the page's footer no longer says "IPv4 only"
([#111](https://git.eeqj.de/sneak/netwatch/issues/111)): each check is a
`fetch`, the browser picks IPv4 or IPv6 for each WAN host, and the local
targets are IPv4 addresses. The rest of the footer is unchanged
- 2026-10-04: `README.md`, `TODO.md` and `test/viewport/README.md` say what the
tree does (issue #24). The README's Getting Started leads with `make` targets;
a new Backend section says what `netwatch-server` stores, its routes and how
the image builds and runs it, and points to `backend/README.md` for its
settings; the checks are GET requests; the 26 WAN hosts, the four health
states, the summary's figures and the features the list lacked are described
as the page has them; and its TODO section points here, as does the one in
`backend/README.md`, whose open items moved to Future Steps. This file's
Workflow branches from `next` and opens the PR against `next`, Status says
where the repo stands, and Next Step and Future Steps hold only open work,
linked to its issue where one exists. The viewport harness README names Node's
test runner, not `vitest`
- 2026-10-04: in `src/main.js` (issue #102), a target's min, max, median and
average latency come from one list of its answers, through the same function
the summary's figures use, so the median is written once. The latency color
limits are one table in `CONFIG`, read by both the figure's and the
sparkline's color. The health thresholds, the debug log's length, the gateway
check's timeout, the recovery probe's number of hosts and interval, how often
the rows are sorted and the delay before the first sparkline resize are
`CONFIG` entries too. A unit test checks the summary's figures. Nothing the
page does or shows changed; the footer's color legend still writes the limits
out as text
- 2026-10-04: password guesses at `/metrics` are rate limited (issue #104): each
client address, resolved through `TRUSTED_PROXIES` as for reports, may make 60
requests to `/metrics` a minute, counted by `go-chi/httprate` apart from its
reports; past that it gets 429 and its basic auth credentials are not checked.
The limit is a constant in `backend/internal/server/routes.go`. A test uses up
one client's allowance on wrong passwords, gets 429 with the right one, and
checks that another client behind the same nginx still gets in
- 2026-10-04: the backend reports errors to Sentry (issue #95). With
`SENTRY_DSN` set, it sets up `sentry-go` with the release `netwatch-server-`
and its version, reports each panic in a handler through `sentryhttp`, the
last of the middleware every request goes through, which panics again so the
request still gets the 500 from the panic recovery, and waits up to 2 seconds
on shutdown for Sentry to finish sending. A DSN Sentry refuses stops the start
with an error naming `SENTRY_DSN`. With it empty, Sentry is not set up and
nothing is sent to it
- 2026-10-04: a target's name and URL and a debug log message show as the
characters they are and are never read as HTML (issue #29): a host row escapes
the name and URL it writes into its markup, and the debug log sets each line
as text. A unit test checks a target whose name and URL hold `<`, `>`, `"`,
`&` and `'`. `README.md` no longer calls `CONFIG` frozen: the interval menu
sets its `updateInterval`, and the values computed from it follow. `AppState`
declares the recovery probe's two properties, the sparkline axis functions
lose the parameters they did not use, and the comment on a target's history
names both kinds of entry it holds. Nothing the page does changed
- 2026-10-04: a dependency can be added without running yarn or go by hand
(issue #45): `make add-dependency PACKAGE=<name>@<version>` shims to the new
`script/add-dependency`, which runs `yarn add --dev`, so `package.json` and
@@ -107,19 +31,6 @@ Decide whether the repo moves to the layout `REPO_POLICIES.md` gives, with
`script/tidy`, which runs `go mod tidy` in `backend/`: a Go module is added by
importing it, or moved by editing its `require` line, then `make tidy`.
`script/bootstrap` still installs with `--frozen-lockfile`
- 2026-10-04: the backend serves Prometheus metrics (issue #94). With
`METRICS_USERNAME` and `METRICS_PASSWORD` both set, it records request
duration and response size through `go-http-metrics` and serves them, with
Go's runtime and process metrics, at `GET /metrics` behind basic auth with
those credentials; nginx passes `/metrics` to it as it does `/api/`. With
neither set there are no metrics and `/metrics` is 404; one without the other
stops the start with an error naming both, and so does a `METRICS_USERNAME`
containing `:`, with an error naming it. Only requests that reach the health
check or `POST /api/v1/reports` are recorded, not `/metrics` itself and not
every request as `GO_HTTP_SERVER_CONVENTIONS.md` shows, because the labels are
the request's path and method, which clients can make up without end. For
that, `POST /api/v1/reports` is now registered by its full path instead of
inside a `/api/v1` route group; it answers as before
- 2026-10-04: `script/` and `Makefile` follow the org models (issue #28):
`make dev` shims to the new `script/dev`, the Vite dev server, and the new
`make build` to `script/build`, the frontend production build.
@@ -416,14 +327,12 @@ Decide whether the repo moves to the layout `REPO_POLICIES.md` gives, with
# Future Steps
- Run `make frontend-viewport-test` in CI as its own step; it is not part of
`make check`, as it needs Docker and takes minutes
- A backend test that posts a report to `POST /api/v1/reports` and checks the
compressed file it is written to
- A backend route that decompresses the stored reports and answers queries on
them
- Prometheus metrics for the backend's in-memory buffer: its size, the number of
flushes and the number of reports
- A configurable host list (an environment variable or a config file)
- Export of the latency history (CSV or JSON)
- A notification when the health status changes to DEGRADED
- Wire `script/frontend-viewport-test` into CI as its own step (deliberately not
part of `make check` today; the decision has real CI-runtime cost and is
tracked separately)
- Compliance top-up as one small commit: add .editorconfig and add the hooks
target to the Makefile
- After merge, confirm .gitea/workflows/check.yml is on main and CI is green
(main always green policy)
- Decide what to do with untracked resume.sh: commit it, gitignore it, or delete
it
backend/.golangci.yml
-1
View File
@@ -17,7 +17,6 @@ linters:
disable:
# Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
- godot # Requires comments to end with periods
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
backend/README.md
+23 -64
View File
@@ -28,21 +28,20 @@ docker run -p 8080:8080 netwatch
This directory follows the same
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
pattern as the repo root: the targets in `backend/Makefile` are thin shims over
`backend/script/`. The root `Dockerfile` runs `build`, and the root scripts call
`fmt` and `fmt-check`:
`backend/script/`. The root `Dockerfile` runs them, and the root scripts call
`test`, `fmt` and `fmt-check`:
- `script/build` — compile the static `netwatch-server` binary with its version
stamped in. The version is `VERSION` from the environment; when that is unset
or empty, it falls back to `git describe` inside a git checkout, then to `dev`
- `script/test` — run the Go tests on the host with the race detector and
coverage; the root `make test` runs them in the `test` phase of the root
`Dockerfile`. Go's `-timeout 90s` bounds the tests, not their compile, and
`-count=1` keeps Go from reporting a stored pass. If they fail, they run again
with `-v` for the details, and the script fails. The race detector needs a C
compiler
- `script/lint` — run the root `script/lint`, which builds the `lint` phase of
the root `Dockerfile`: golangci-lint over this directory, and eslint over the
frontend. golangci-lint never runs on the host
- `script/test` — run the Go tests with the race detector and coverage. Go's
`-timeout 30s` bounds the tests, not their compile. If they fail, they run
again with `-v` for the details, and the script fails. The race detector needs
a C compiler
- `script/lint` — check `.golangci.yml` against its pinned sha256, then run
golangci-lint. It runs inside the golangci-lint image of the lint stage of the
root `Dockerfile`; from a checkout, run `make lint` at the repo root, which
builds that stage
- `script/fmt` — format the Go sources (writes)
- `script/fmt-check` — check Go formatting (read-only)
- `script/run` — build and run the server locally
@@ -89,9 +88,6 @@ project layout:
| `TRUSTED_PROXIES` | loopback + RFC1918 | Comma-separated CIDRs whose `X-Forwarded-For` / `X-Real-IP` headers are trusted for client IP resolution |
| `REPORTS_PER_MINUTE` | `60` | Reports each client address may send a minute; see [Report limits](#report-limits) |
| `CORS_ALLOWED_ORIGINS` | empty | Comma-separated origins whose pages may call the API; see [CORS](#cors) |
| `METRICS_USERNAME` | empty | Basic auth user name for `/metrics`; see [Metrics](#metrics) |
| `METRICS_PASSWORD` | empty | Basic auth password for `/metrics`; see [Metrics](#metrics) |
| `SENTRY_DSN` | empty | DSN of the Sentry project to send errors to; see [Sentry](#sentry) |
`TRUSTED_PROXIES` defaults to
`127.0.0.1/32,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`. The loopback
@@ -107,16 +103,16 @@ starting, with an error naming the variable. An empty variable counts as unset.
### Container image
The root `Dockerfile` builds one image in which nginx listens on the public port
8080, serves the frontend, and proxies `/api/`, `/.well-known/healthcheck` and
`/metrics` to this server. The image's entrypoint, `bin/entrypoint.sh`, starts
the server as user `netwatch` (uid 1000) with `BIND_ADDRESS=127.0.0.1` and
`PORT=8081`, so only nginx reaches it, and with `TRUSTED_PROXIES=127.0.0.1/32`,
so it takes the client address nginx passes on and no other. `DATA_DIR` is
`/data/reports`, on the `/data` volume; before starting the server, the
entrypoint creates it and gives it and `/data` to `netwatch` with
`netwatch-server prepare-data-dir`, which acts on nothing outside `/data`. nginx
replaces the security headers this server sets with those in the root
`security-headers.conf`, so those are what clients of the image see.
8080, serves the frontend, and proxies `/api/` and `/.well-known/healthcheck` to
this server. The image's entrypoint, `bin/entrypoint.sh`, starts the server as
user `netwatch` (uid 1000) with `BIND_ADDRESS=127.0.0.1` and `PORT=8081`, so
only nginx reaches it, and with `TRUSTED_PROXIES=127.0.0.1/32`, so it takes the
client address nginx passes on and no other. `DATA_DIR` is `/data/reports`, on
the `/data` volume; before starting the server, the entrypoint creates it and
gives it and `/data` to `netwatch` with `netwatch-server prepare-data-dir`,
which acts on nothing outside `/data`. nginx replaces the security headers this
server sets with those in the root `security-headers.conf`, so those are what
clients of the image see.
The container's own `TRUSTED_PROXIES` goes to nginx instead: IP addresses or
CIDRs, separated by commas, of the reverse proxies in front of the container.
@@ -180,48 +176,11 @@ origin, `scheme://host` with an optional `:port`, as browsers send it: no path,
not even a trailing `/`, and no `*`. Any other entry stops the server from
starting, with an error naming `CORS_ALLOWED_ORIGINS`.
### Metrics
With both `METRICS_USERNAME` and `METRICS_PASSWORD` set, the server serves
Prometheus metrics at `GET /metrics` to requests with those as their basic auth
credentials, and answers any other with 401. For each request that reaches the
health check or `POST /api/v1/reports`, those the rate limit refuses included,
the metrics record its duration and response size, labelled with its path,
method and status; they also count those requests in progress, and include Go's
runtime and process metrics. No other request is recorded: not those to
`/metrics` itself, and not those answered before they reach either route, such
as a CORS preflight, or a request refused with 404 for a path no route has, 405
for a method its route does not take, or 413 for declaring a body length over
the 1 MiB limit. A report whose body goes over the limit without declaring its
length reaches the route, is answered 413 there, and is recorded with that
status. Clients can make up any number of paths and methods, and each would add
labels to the metrics for as long as the server runs. With neither set, nothing
is recorded and `/metrics` answers 404. One without the other stops the server
from starting, with an error naming both; so does a `METRICS_USERNAME`
containing `:`, which basic auth cannot carry, with an error naming it.
`/metrics` is rate limited, so that its password cannot be guessed quickly: each
client address, resolved through `TRUSTED_PROXIES`, may make 60 requests to it a
minute, whatever their credentials. Past that it gets 429 with
`Retry-After: 60`, and its credentials are not checked. The minute slides as it
does for reports (see [Report limits](#report-limits)), so a scraper polling
every 2 seconds or less often is never refused. This allowance is apart from the
one for reports.
### Sentry
With `SENTRY_DSN` set, the server sends its errors to that Sentry project: each
panic in a handler is reported there, under the release `netwatch-server-`
followed by the server's version, and the request still gets 500 from the
server's panic recovery. On shutdown the server waits up to 2 seconds for Sentry
to finish sending. A DSN Sentry refuses stops the server from starting, with an
error naming `SENTRY_DSN`. With it empty, Sentry is not set up, and nothing is
sent to it.
## TODO
The to-do list, this backend's open work included, is [TODO.md](../TODO.md) at
the repo root.
- Add integration test that POSTs a report and verifies the compressed output
- Add report decompression/query endpoint
- Add metrics (Prometheus) for buffer size, flush count, report count
## License
backend/cmd/netwatch-server/main_test.go
-22
View File
@@ -115,28 +115,6 @@ func TestMalformedConfigFileStopsTheStart(t *testing.T) {
}
}
// TestRefusedSentryDSNStopsTheStart: a SENTRY_DSN that Sentry refuses
// stops the start, and the error, naming SENTRY_DSN, is logged as JSON.
func TestRefusedSentryDSNStopsTheStart(t *testing.T) {
t.Setenv("SENTRY_DSN", "not-a-dsn")
ctx, cancel := context.WithTimeout(t.Context(), childTimeout)
defer cancel()
child, stdout, stderr := startServer(ctx, t, t.TempDir(), freePort(ctx, t))
err := child.Wait()
if child.ProcessState.ExitCode() != 1 {
t.Fatalf("server exit = %v, want exit status 1", err)
}
requireJSONLines(t, stdout, stderr)
if !strings.Contains(stdout.String(), "SENTRY_DSN") {
t.Fatalf("no error naming SENTRY_DSN in stdout:\n%s", stdout)
}
}
// startServer runs main() in a child process listening on
// 127.0.0.1:port, with home as its HOME and working directory and its
// data directory in home, so it touches nothing outside home. Its
backend/go.mod
+3 -14
View File
@@ -3,30 +3,20 @@ module sneak.berlin/go/netwatch
go 1.25.5
require (
github.com/99designs/basicauth-go v0.0.0-20230316000542-bf6f9cbbf0f8
github.com/getsentry/sentry-go v0.49.0
github.com/go-chi/chi/v5 v5.2.5
github.com/go-chi/cors v1.2.2
github.com/go-chi/httprate v0.16.0
github.com/joho/godotenv v1.5.1
github.com/klauspost/compress v1.19.1
github.com/prometheus/client_golang v1.24.1
github.com/slok/go-http-metrics v0.13.0
github.com/klauspost/compress v1.18.4
github.com/spf13/viper v1.21.0
go.uber.org/fx v1.24.0
)
require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.10 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
github.com/sagikazarmark/locafero v0.11.0 // indirect
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
github.com/spf13/afero v1.15.0 // indirect
@@ -38,7 +28,6 @@ require (
go.uber.org/multierr v1.10.0 // indirect
go.uber.org/zap v1.26.0 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/text v0.40.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect
golang.org/x/sys v0.30.0 // indirect
golang.org/x/text v0.28.0 // indirect
)
backend/go.sum
+18 -52
View File
@@ -1,65 +1,37 @@
github.com/99designs/basicauth-go v0.0.0-20230316000542-bf6f9cbbf0f8 h1:nMpu1t4amK3vJWBibQ5X/Nv0aXL+b69TQf2uK5PH7Go=
github.com/99designs/basicauth-go v0.0.0-20230316000542-bf6f9cbbf0f8/go.mod h1:3cARGAK9CfW3HoxCy1a0G4TKrdiKke8ftOMEOHyySYs=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/getsentry/sentry-go v0.49.0 h1:Ehejknu1l023Ub7QoRBVLAI7g3Jnhqku4oWx4B4Sh5s=
github.com/getsentry/sentry-go v0.49.0/go.mod h1:nuMJAoCfe1u0Bts2ocyNI+TW8HT84vRMqwA5Qq/SKUI=
github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
github.com/go-chi/cors v1.2.2 h1:Jmey33TE+b+rB7fT8MUy1u0I4L+NARQlK6LhzKPSyQE=
github.com/go-chi/cors v1.2.2/go.mod h1:sSbTewc+6wYHBBCW7ytsFSn836hqM7JxpglAy2Vzc58=
github.com/go-chi/httprate v0.16.0 h1:8V5DH9j6pSK6UQoBsTpvMyFxycqaKEIToyPKzHJjUa8=
github.com/go-chi/httprate v0.16.0/go.mod h1:A8lo+qRhk+s9LiuP5saS7XCGDXRXMcrueq0NfIuCa/I=
github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/compress v1.18.4 h1:RPhnKRAQ4Fh8zU2FY/6ZFDwTVTxgJ/EMydqSTzE9a2c=
github.com/klauspost/compress v1.18.4/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE=
github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/pingcap/errors v0.11.4 h1:lFuQV/oaUMGcD2tqt+01ROSmJs75VG1ToEOkZIZ4nE4=
github.com/pingcap/errors v0.11.4/go.mod h1:Oi8TUi2kEtXXLMJk9l1cGmz20kV3TaQ0usTwv5KuLY8=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU=
github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.70.1 h1:1HvjP4D5oL3t8RsPlwxA9onvvStjtIHYE5XuuwOi/PY=
github.com/prometheus/common v0.70.1/go.mod h1:VdFUQDMZK3VLkurFUVhia6uys/0suUp86TJz5qbJRhc=
github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI=
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
github.com/slok/go-http-metrics v0.13.0 h1:lQDyJJx9wKhmbliyUsZ2l6peGnXRHjsjoqPt5VYzcP8=
github.com/slok/go-http-metrics v0.13.0/go.mod h1:HIr7t/HbN2sJaunvnt9wKP9xoBBVZFo1/KiHU3b0w+4=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8/go.mod h1:3n1Cwaq1E1/1lhQhtRK2ts/ZwZEhjcQeJQ1RuC6Q/8U=
github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
@@ -70,8 +42,6 @@ github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
github.com/spf13/viper v1.21.0/go.mod h1:P0lhsswPGWD/1lZJ9ny3fYnVqxiegrlNrEmgLjbTCAY=
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
@@ -84,24 +54,20 @@ go.uber.org/dig v1.19.0 h1:BACLhebsYdpQ7IROQ1AGPjrXcP5dF80U3gKoFzbaq/4=
go.uber.org/dig v1.19.0/go.mod h1:Us0rSJiThwCv2GteUN0Q7OKvU7n5J4dxZ9JKUXozFdE=
go.uber.org/fx v1.24.0 h1:wE8mruvpg2kiiL1Vqd0CC+tr0/24XIB10Iwp2lLWzkg=
go.uber.org/fx v1.24.0/go.mod h1:AmDeGyS+ZARGKM4tlH4FY2Jr63VjbEDJHtqXTGP5hbo=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.uber.org/goleak v1.2.0 h1:xqgm/S+aQvhWFTtR0XK3Jvg7z8kGV8P4X14IzwN3Eqk=
go.uber.org/goleak v1.2.0/go.mod h1:XJYK+MuIchqpmGmUSAzotztawfKvYLUIgg7guXrwVUo=
go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng=
golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
backend/internal/config/config.go
-21
View File
@@ -44,15 +44,6 @@ var (
errNotPort = errors.New("must be a port number, 1 to 65535")
errNotBool = errors.New("must be true or false")
errNotIP = errors.New("must be an IP address, or empty")
errMetricsCredentials = errors.New(
"METRICS_USERNAME and METRICS_PASSWORD must be set together, " +
"or neither",
)
errMetricsUsernameColon = errors.New(
"METRICS_USERNAME must not contain \":\", " +
"which basic auth cannot carry in a user name",
)
)
// Params defines the dependencies for Config.
@@ -187,18 +178,6 @@ func (s *Config) check() error {
}
}
// The server records and serves metrics only with both set, so
// one alone is a mistake that would otherwise go unnoticed.
if (s.MetricsUsername == "") != (s.MetricsPassword == "") {
return errMetricsCredentials
}
// Basic auth splits the credentials at the first ":", so with one
// in the user name every request to /metrics would get 401.
if strings.Contains(s.MetricsUsername, ":") {
return errMetricsUsernameColon
}
return checkOrigins(s.CORSAllowedOrigins)
}
backend/internal/config/config_test.go
-26
View File
@@ -103,32 +103,6 @@ func TestDataDirMaxBytesMustBeANumber(t *testing.T) {
requireConfigError(t, "DATA_DIR_MAX_BYTES")
}
// TestMetricsCredentialsGoTogether: with only one of the two set, the
// server would quietly serve no metrics, so the start fails, naming
// both.
func TestMetricsCredentialsGoTogether(t *testing.T) {
for _, set := range []string{"METRICS_USERNAME", "METRICS_PASSWORD"} {
t.Run(set, func(t *testing.T) {
t.Setenv("METRICS_USERNAME", "")
t.Setenv("METRICS_PASSWORD", "")
t.Setenv(set, "prometheus")
requireConfigError(t, "METRICS_USERNAME")
requireConfigError(t, "METRICS_PASSWORD")
})
}
}
// TestMetricsUsernameMustNotContainColon: basic auth splits the
// credentials at the first ":", so such a user name would get 401 on
// every request to /metrics.
func TestMetricsUsernameMustNotContainColon(t *testing.T) {
t.Setenv("METRICS_USERNAME", "prom:etheus")
t.Setenv("METRICS_PASSWORD", "secret")
requireConfigError(t, "METRICS_USERNAME")
}
// TestCORSAllowedOriginsMustBeOrigins: "*" would let every origin in,
// and an entry that is not a plain origin would match no page.
func TestCORSAllowedOriginsMustBeOrigins(t *testing.T) {
backend/internal/middleware/middleware.go
-27
View File
@@ -18,14 +18,9 @@ import (
"sneak.berlin/go/netwatch/internal/globals"
"sneak.berlin/go/netwatch/internal/logger"
basicauth "github.com/99designs/basicauth-go"
"github.com/go-chi/chi/v5/middleware"
"github.com/go-chi/cors"
"github.com/go-chi/httprate"
"github.com/prometheus/client_golang/prometheus"
metrics "github.com/slok/go-http-metrics/metrics/prometheus"
ghmm "github.com/slok/go-http-metrics/middleware"
"github.com/slok/go-http-metrics/middleware/std"
"go.uber.org/fx"
)
@@ -369,25 +364,3 @@ func (s *Middleware) RateLimit(
),
)
}
// Metrics returns middleware that records each request's duration and
// response size, and the requests in progress, in registry. They are
// labelled by the request path.
func (s *Middleware) Metrics(
registry prometheus.Registerer,
) func(http.Handler) http.Handler {
mdlw := ghmm.New(ghmm.Config{
Recorder: metrics.NewRecorder(metrics.Config{Registry: registry}),
})
return std.HandlerProvider("", mdlw)
}
// MetricsAuth returns middleware that lets a request through only with
// METRICS_USERNAME and METRICS_PASSWORD as its basic auth credentials,
// and answers any other with 401.
func (s *Middleware) MetricsAuth() func(http.Handler) http.Handler {
return basicauth.New("metrics", map[string][]string{
s.params.Config.MetricsUsername: {s.params.Config.MetricsPassword},
})
}
backend/internal/middleware/middleware_test.go
+1 -1
View File
@@ -343,7 +343,7 @@ func TestLoggingCutsRequestStringsToBound(t *testing.T) {
http.MethodGet, "/"+long, http.NoBody)
req.Header.Set("User-Agent", long)
req.Header.Set("Referer", long)
req.Header.Set("X-Request-ID", long)
req.Header.Set("X-Request-Id", long)
handler.ServeHTTP(httptest.NewRecorder(), req)
backend/internal/server/export_test.go
-12
View File
@@ -1,21 +1,9 @@
package server
import "github.com/go-chi/chi/v5"
// Router exposes the router to the external tests, which add routes
// of their own to it after SetupRoutes.
func (s *Server) Router() *chi.Mux {
return s.router
}
// MaxRequestBodyBytes exposes the router-wide body limit to the
// external tests.
const MaxRequestBodyBytes = maxRequestBodyBytes
// MetricsRequestsPerMinute exposes the /metrics rate limit to the
// external tests.
const MetricsRequestsPerMinute = metricsRequestsPerMinute
// ListenAddr exposes the address the server listens on to the
// external tests.
func (s *Server) ListenAddr() string {
backend/internal/server/routes.go
+5 -49
View File
@@ -3,12 +3,8 @@ package server
import (
"time"
sentryhttp "github.com/getsentry/sentry-go/http"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/collectors"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
const (
@@ -18,12 +14,6 @@ const (
// can mount s.mw.MaxBodyBytes with a smaller value to lower
// its bound, but cannot raise it: this cap runs first.
maxRequestBodyBytes int64 = 1 << 20 // 1 MiB
// metricsRequestsPerMinute is how many requests to /metrics each
// client address may make a minute, whatever their credentials. A
// scraper polling every 2 seconds sends half of it, which httprate
// never refuses.
metricsRequestsPerMinute = 60
)
// SetupRoutes configures the chi router with middleware and
@@ -39,47 +29,13 @@ func (s *Server) SetupRoutes() {
s.router.Use(s.mw.MaxBodyBytes(maxRequestBodyBytes))
s.router.Use(middleware.Timeout(requestTimeout))
// Sentry reports a panic, then panics again, so that s.mw.Recoverer
// still answers 500.
if s.params.Config.SentryDSN != "" {
s.router.Use(sentryhttp.New(sentryhttp.Options{Repanic: true}).Handle)
}
// The metrics go in a registry of this server's own, not in
// Prometheus' default one, which takes them only once per process.
registry := prometheus.NewRegistry()
registry.MustRegister(
collectors.NewGoCollector(),
collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}),
s.router.Get(
"/.well-known/healthcheck",
s.h.HandleHealthCheck(),
)
// Requests are measured only once chi has matched them to one of
// these routes, by path and method. The metrics are labelled with
// both, which any client can make up, so measuring every request
// would let clients add labels without bound. A Route here would
// be matched by its path prefix alone, so each path is given in
// full.
s.router.Group(func(r chi.Router) {
// config.New refuses one of the two credentials without the
// other.
if s.params.Config.MetricsUsername != "" {
r.Use(s.mw.Metrics(registry))
}
r.Get("/.well-known/healthcheck", s.h.HandleHealthCheck())
s.router.Route("/api/v1", func(r chi.Router) {
r.With(s.mw.RateLimit(s.params.Config.ReportsPerMinute)).
Post("/api/v1/reports", s.h.HandleReport())
Post("/reports", s.h.HandleReport())
})
// The rate limit comes before the basic auth, so a client past it
// gets 429 and its password is not checked.
if s.params.Config.MetricsUsername != "" {
s.router.With(
s.mw.RateLimit(metricsRequestsPerMinute),
s.mw.MetricsAuth(),
).Get("/metrics", promhttp.HandlerFor(
registry, promhttp.HandlerOpts{},
).ServeHTTP)
}
}
backend/internal/server/routes_test.go
-200
View File
@@ -1,12 +1,10 @@
package server_test
import (
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"sneak.berlin/go/netwatch/internal/config"
"sneak.berlin/go/netwatch/internal/globals"
@@ -17,7 +15,6 @@ import (
"sneak.berlin/go/netwatch/internal/reportbuf"
"sneak.berlin/go/netwatch/internal/server"
"github.com/getsentry/sentry-go"
"go.uber.org/fx"
"go.uber.org/fx/fxtest"
)
@@ -110,203 +107,6 @@ func TestCORSAllowedOriginsReachTheRouter(t *testing.T) {
}
}
// TestNoMetricsWithoutCredentials: with neither metrics setting set,
// there is no /metrics.
func TestNoMetricsWithoutCredentials(t *testing.T) {
t.Setenv("METRICS_USERNAME", "")
t.Setenv("METRICS_PASSWORD", "")
srv := newServer(t)
srv.SetupRoutes()
rec := httptest.NewRecorder()
req := httptest.NewRequestWithContext(t.Context(),
http.MethodGet, "/metrics", http.NoBody)
srv.ServeHTTP(rec, req)
if rec.Code != http.StatusNotFound {
t.Fatalf("status = %d, want %d", rec.Code, http.StatusNotFound)
}
}
// TestMetricsBehindBasicAuth: with both metrics settings set, /metrics
// answers only with them as basic auth credentials, and shows a
// request to a route but not one to a path no route has.
func TestMetricsBehindBasicAuth(t *testing.T) {
t.Setenv("METRICS_USERNAME", "prometheus")
t.Setenv("METRICS_PASSWORD", "right")
srv := newServer(t)
srv.SetupRoutes()
get := func(path, username, password string) *httptest.ResponseRecorder {
rec := httptest.NewRecorder()
req := httptest.NewRequestWithContext(t.Context(),
http.MethodGet, path, http.NoBody)
if username != "" {
req.SetBasicAuth(username, password)
}
srv.ServeHTTP(rec, req)
return rec
}
get("/.well-known/healthcheck", "", "")
get("/api/v1/no-such-route", "", "")
for _, creds := range [][2]string{
{"", ""},
{"prometheus", "wrong"},
{"someone", "right"},
} {
rec := get("/metrics", creds[0], creds[1])
if rec.Code != http.StatusUnauthorized {
t.Errorf("credentials %q: status = %d, want %d",
creds, rec.Code, http.StatusUnauthorized)
}
}
rec := get("/metrics", "prometheus", "right")
if rec.Code != http.StatusOK {
t.Fatalf("right credentials: status = %d, want %d",
rec.Code, http.StatusOK)
}
body := rec.Body.String()
if !strings.Contains(body, `handler="/.well-known/healthcheck"`) {
t.Errorf("metrics show no health check request:\n%s", body)
}
if strings.Contains(body, "no-such-route") {
t.Errorf("metrics show a request to a path no route has:\n%s", body)
}
if !strings.Contains(body, "go_goroutines") {
t.Errorf("metrics show no Go runtime metrics:\n%s", body)
}
}
// TestMetricsAreRateLimited: a client that has used up its /metrics
// allowance on wrong passwords gets 429 even with the right one, which
// is then not checked, while another client behind the same nginx
// still gets in.
func TestMetricsAreRateLimited(t *testing.T) {
t.Setenv("METRICS_USERNAME", "prometheus")
t.Setenv("METRICS_PASSWORD", "right")
// As in the container: nginx connects from loopback and names the
// client in X-Forwarded-For.
t.Setenv("TRUSTED_PROXIES", "127.0.0.1/32")
srv := newServer(t)
srv.SetupRoutes()
get := func(client, password string) int {
rec := httptest.NewRecorder()
req := httptest.NewRequestWithContext(t.Context(),
http.MethodGet, "/metrics", http.NoBody)
req.RemoteAddr = "127.0.0.1:40000"
req.Header.Set("X-Forwarded-For", client)
req.SetBasicAuth("prometheus", password)
srv.ServeHTTP(rec, req)
return rec.Code
}
for i := range server.MetricsRequestsPerMinute {
if code := get("203.0.113.7", "wrong"); code != http.StatusUnauthorized {
t.Fatalf("guess %d: status = %d, want %d",
i+1, code, http.StatusUnauthorized)
}
}
if code := get("203.0.113.7", "right"); code != http.StatusTooManyRequests {
t.Fatalf("right password past the limit: status = %d, want %d",
code, http.StatusTooManyRequests)
}
if code := get("203.0.113.8", "right"); code != http.StatusOK {
t.Fatalf("another client: status = %d, want %d",
code, http.StatusOK)
}
}
// TestMetricsInTwoServers: two servers in one process can both have
// metrics on.
func TestMetricsInTwoServers(t *testing.T) {
t.Setenv("METRICS_USERNAME", "prometheus")
t.Setenv("METRICS_PASSWORD", "right")
for range 2 {
newServer(t).SetupRoutes()
}
}
// TestSentry: with SENTRY_DSN empty there is no Sentry client. With it
// pointing at a local server standing in for Sentry, a panic in a
// handler reaches that server, and the request still gets the 500 from
// the panic recovery.
func TestSentry(t *testing.T) {
const panicMessage = "handler panic for TestSentry"
// sentry.Init sets the client for the whole process; take it away
// again so that no other test reports to Sentry.
t.Cleanup(func() { sentry.CurrentHub().BindClient(nil) })
t.Setenv("SENTRY_DSN", "")
newServer(t)
if sentry.CurrentHub().Client() != nil {
t.Fatal("a Sentry client exists with SENTRY_DSN empty")
}
// The body of the first request the stand-in for Sentry receives.
received := make(chan string, 1)
sentryServer := httptest.NewServer(http.HandlerFunc(
func(_ http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
select {
case received <- string(body):
default:
}
},
))
defer sentryServer.Close()
t.Setenv("SENTRY_DSN",
"http://key@"+sentryServer.Listener.Addr().String()+"/1")
srv := newServer(t)
srv.SetupRoutes()
srv.Router().Get("/panic", func(http.ResponseWriter, *http.Request) {
panic(panicMessage)
})
rec := httptest.NewRecorder()
req := httptest.NewRequestWithContext(t.Context(),
http.MethodGet, "/panic", http.NoBody)
srv.ServeHTTP(rec, req)
if rec.Code != http.StatusInternalServerError {
t.Fatalf("status = %d, want %d",
rec.Code, http.StatusInternalServerError)
}
// Sentry sends from a goroutine of its own.
select {
case body := <-received:
if !strings.Contains(body, panicMessage) {
t.Fatalf("the Sentry server received no report of the panic:\n%s",
body)
}
case <-time.After(5 * time.Second):
t.Fatal("nothing reached the Sentry server")
}
}
// TestHealthCheckRejectsOversizeBody sends the health check, which
// never reads its body, a body one byte over the limit. Only the
// router-wide body limit can reject it.
backend/internal/server/server.go
+1 -40
View File
@@ -7,10 +7,8 @@ package server
import (
"context"
"fmt"
"log/slog"
"net/http"
"time"
"sneak.berlin/go/netwatch/internal/config"
"sneak.berlin/go/netwatch/internal/globals"
@@ -18,15 +16,10 @@ import (
"sneak.berlin/go/netwatch/internal/logger"
"sneak.berlin/go/netwatch/internal/middleware"
"github.com/getsentry/sentry-go"
"github.com/go-chi/chi/v5"
"go.uber.org/fx"
)
// sentryFlushTimeout is how long shutdown waits for Sentry to send
// what it still holds.
const sentryFlushTimeout = 2 * time.Second
// Params defines the dependencies for Server.
type Params struct {
fx.In
@@ -63,11 +56,6 @@ func New(
s.log = params.Logger.Get()
s.shutdowner = params.Shutdowner
err := s.enableSentry()
if err != nil {
return nil, err
}
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
// Build the router and http.Server synchronously
@@ -99,37 +87,10 @@ func (s *Server) ServeHTTP(
s.router.ServeHTTP(w, r)
}
// enableSentry sets Sentry up when SENTRY_DSN is set, so that
// SetupRoutes can report panics to it. With SENTRY_DSN empty it does
// nothing. A DSN Sentry refuses stops the start.
func (s *Server) enableSentry() error {
if s.params.Config.SentryDSN == "" {
return nil
}
err := sentry.Init(sentry.ClientOptions{
Dsn: s.params.Config.SentryDSN,
Release: s.params.Globals.Appname + "-" + s.params.Globals.Version,
})
if err != nil {
return fmt.Errorf("SENTRY_DSN: %w", err)
}
s.log.Info("sentry error reporting activated")
return nil
}
// shutdown gracefully stops the HTTP server within the
// deadline of the context fx provides for OnStop, then gives
// Sentry, if set up, time to send what it still holds.
// deadline of the context fx provides for OnStop.
func (s *Server) shutdown(ctx context.Context) error {
err := s.httpServer.Shutdown(ctx)
if s.params.Config.SentryDSN != "" {
sentry.Flush(sentryFlushTimeout)
}
if err != nil {
s.log.Error("server clean shutdown failed", "error", err)
backend/script/lint
+42 -5
View File
@@ -1,13 +1,50 @@
#!/bin/sh
# script/lint: lint the whole repo as the root make lint does, by
# building the lint phase of the root Dockerfile. golangci-lint never
# runs on the host (REPO_POLICIES.md).
# script/lint: run golangci-lint over the backend. This runs inside the
# lint stage of the root Dockerfile, whose digest-pinned golangci-lint
# image provides the linter; nothing installs golangci-lint on the host.
# From a checkout, run `make lint` at the repo root, which builds that
# stage.
#
# .golangci.yml is standardized org-wide and must never be edited here
# (REPO_POLICIES.md). Its last silent drift replaced the v2 schema with
# v1 keys, which left every threshold in the file inert while the build
# stayed green. So the file is first checked against the canonical
# copy's sha256: a local comparison, no network, nothing unpinned.
set -eu
ROOT="$(cd "$(dirname "$0")/../.." && pwd -P)"
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# The sha256 of the org standard .golangci.yml. When that file changes in
# sneak/prompts and is copied here again, this changes with it.
GOLANGCI_CONFIG_SHA256="a79b63a254602a5318db5d0e9a06bc71b84bf0c1d896305229d8bfed1d1b1776"
main() {
exec "$ROOT/script/lint"
cd "$ROOT"
if [ ! -f .golangci.yml ]; then
echo "backend/.golangci.yml is missing. Copy the org standard verbatim" >&2
echo "from https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml" >&2
exit 1
fi
actual="$(sha256sum .golangci.yml | cut -d' ' -f1)"
if [ -z "$actual" ]; then
echo "sha256sum is missing or printed no hash, so" >&2
echo "backend/.golangci.yml could not be checked." >&2
exit 1
fi
if [ "$actual" != "$GOLANGCI_CONFIG_SHA256" ]; then
echo "backend/.golangci.yml does not match GOLANGCI_CONFIG_SHA256" >&2
echo "in backend/script/lint." >&2
echo " expected $GOLANGCI_CONFIG_SHA256" >&2
echo " actual $actual" >&2
echo "Compare it with the org standard," >&2
echo "https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml" >&2
echo "- If they differ, it was edited here: restore the org standard" >&2
echo " verbatim. Do not edit it." >&2
echo "- If they are the same, the org standard changed: set" >&2
echo " GOLANGCI_CONFIG_SHA256 in backend/script/lint to the actual hash." >&2
exit 1
fi
golangci-lint run ./...
}
main "$@"
backend/script/test
+7 -10
View File
@@ -1,21 +1,18 @@
#!/bin/sh
# script/test: run the backend test suite on the host with the race
# detector and coverage. The root make test runs the same in the test
# phase of the root Dockerfile. Go's own -timeout bounds the tests and
# not their compile, so a cold build cache cannot fail it. -count=1
# keeps Go's test result cache out of both runs, so neither can report
# a stored pass. The race detector needs cgo, and so a C compiler. If
# the tests fail, they run again with -v for the details, and the
# script fails even if that run passes.
# script/test: run the backend test suite with the race detector and
# coverage. Go's own -timeout bounds the tests and not their compile,
# so a cold build cache cannot fail it. The race detector needs cgo,
# and so a C compiler. If the tests fail, they run again with -v for
# the details, and the script fails even if that run passes.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
go test -count=1 -timeout 90s -race -cover ./... || {
go test -timeout 30s -race -cover ./... || {
echo "--- Rerunning with -v for details ---"
go test -count=1 -timeout 90s -race -v ./...
go test -timeout 30s -race -v ./...
exit 1
}
}
eslint.config.js
+1 -1
View File
@@ -1,5 +1,5 @@
// eslint's recommended rules over every JavaScript file in the repo.
// make lint runs it, in the frontend-lint stage of Dockerfile.
// script/frontend-lint runs it, in the frontend-lint stage of Dockerfile.
import js from "@eslint/js";
import globals from "globals";
import { defineConfig } from "eslint/config";
nginx.conf
-6
View File
@@ -68,10 +68,4 @@ server {
location = /.well-known/healthcheck {
proxy_pass http://127.0.0.1:8081;
}
# The backend's Prometheus metrics, behind its own basic auth. Unless
# METRICS_USERNAME and METRICS_PASSWORD are set, it answers 404.
location = /metrics {
proxy_pass http://127.0.0.1:8081;
}
}
script/add-dependency
+4 -13
View File
@@ -3,25 +3,16 @@
# version, with yarn add, which changes package.json and yarn.lock
# together; then install from yarn.lock with --frozen-lockfile, as
# script/bootstrap does, to show it installs as written. --dev because
# no frontend package is needed when the page runs: it ships as the
# built dist/.
# every frontend package is used only at build time.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
usage() {
main() {
if [ -z "${1:-}" ]; then
echo "usage: make add-dependency PACKAGE=<name>@<version>" >&2
exit 2
}
main() {
# Exactly one package. A value beginning with - would reach yarn as
# an option; yarn would quietly drop all but the first of several
# packages given in one value.
[ "$#" -eq 1 ] || usage
case "$1" in
"" | -* | *[[:space:]]*) usage ;;
esac
fi
cd "$ROOT"
yarn add --dev "$1"
yarn install --frozen-lockfile
script/bootstrap
+15 -31
View File
@@ -6,16 +6,17 @@
# used directly if it is at least NODE_MIN_VERSION; otherwise it is
# installed at a pinned version via nvm (installing nvm itself first,
# from a hash-verified release archive, never curl | sh). Go, with its
# gofmt, is used directly only if it is exactly GO_VERSION; otherwise
# that release is installed from its hash-verified archive.
# gofmt, is used directly if it is at least the version backend/go.mod
# asks for; otherwise the pinned Go release is installed from its
# hash-verified archive.
#
# What this script installs outside the system package manager lives
# under $HOME and is linked into ~/.local/bin, where make and the git
# hook find it once that directory is on PATH. Nothing in ~/.local/bin
# that this script did not create is ever replaced.
#
# golangci-lint is not installed: make lint runs it in Docker, as make
# test runs the tests, and this script does not install Docker either.
# golangci-lint is not installed: make lint runs it in Docker, which
# this script does not install either.
#
# Unlike the org model: Go and gcc for backend/, a newer node for eslint.
set -eu
@@ -185,30 +186,20 @@ ensure_yarn() {
}
# go_ok: the go on PATH has its gofmt beside it (a Go release ships the
# two together) and go version reports exactly GO_VERSION, compared over
# the whole version, not a prefix of it. A go that fails or prints
# anything else does not pass. GOTOOLCHAIN=local makes go report itself
# rather than a toolchain it would fetch.
# two together) and is at least the version backend/go.mod asks for.
# GOTOOLCHAIN=local makes an older go fail here instead of fetching a
# newer toolchain for itself.
go_ok() {
if missing go; then return 1; fi
[ -x "$(dirname "$(command -v go)")/gofmt" ] || return 1
version="$(GOTOOLCHAIN=local go version 2>/dev/null)" || return 1
case "$version" in
"go version go$GO_VERSION "*) return 0 ;;
*) return 1 ;;
esac
(cd "$ROOT/backend" && GOTOOLCHAIN=local go list -m >/dev/null 2>&1)
}
# ensure_go: unless go_ok, install GO_VERSION and link its go and gofmt.
# They are linked on every run that needs them, so a deleted link is put
# back, and the archive is unpacked again if either binary is missing.
# Afterwards go is looked up through PATH again and must pass go_ok, so
# another go that hides the link stops bootstrap.
ensure_go() {
if go_ok; then
echo "bootstrap: using $(GOTOOLCHAIN=local go version)"
return 0
fi
if go_ok; then return 0; fi
go_dir="$TOOLCHAIN/go-$GO_VERSION"
if [ ! -x "$go_dir/bin/go" ] || [ ! -x "$go_dir/bin/gofmt" ]; then
# sha256 of each archive, from https://go.dev/dl/?mode=json
@@ -249,13 +240,6 @@ ensure_go() {
fi
link_bin "$go_dir/bin/go" go
link_bin "$go_dir/bin/gofmt" gofmt
hash -r
if ! go_ok; then
echo "bootstrap: after installing go $GO_VERSION in $go_dir," >&2
echo " the go on PATH, $(command -v go), is not it or has no gofmt" >&2
exit 1
fi
echo "bootstrap: installed $(GOTOOLCHAIN=local go version)"
}
main() {
@@ -272,9 +256,9 @@ main() {
if missing make; then pkg_install gnumake make make make; fi
if missing git; then pkg_install git git git git; fi
# The race detector in backend/'s make test needs cgo, which Go turns
# on only when it finds its C compiler, gcc on Linux. apt and apk
# ship the C library headers apart from gcc.
# The race detector in make test needs cgo, which Go turns on only
# when it finds its C compiler, gcc on Linux. apt and apk ship the C
# library headers apart from gcc.
if missing gcc; then
pkg_install gcc "gcc libc6-dev" gcc "gcc musl-dev"
fi
@@ -287,8 +271,8 @@ main() {
(cd "$ROOT/backend" && go mod download)
if missing docker; then
echo "bootstrap: docker not found; make test and make lint, and so" >&2
echo " make check and the pre-commit hook, need it" >&2
echo "bootstrap: docker not found; make lint, and so make check" >&2
echo " and the pre-commit hook, need it to run the linters" >&2
fi
if [ -n "$path_hint" ] && [ -d "$BIN_DIR" ]; then
echo "bootstrap: add $BIN_DIR to the front of your PATH, e.g." >&2
script/check
+1 -3
View File
@@ -1,8 +1,6 @@
#!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. test and lint are Docker
# phases; fmt-check is native, because a formatter writes the working
# tree. Must not modify any files.
# extension to scripts-to-rule-them-all. Must not modify any files.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
script/fmt
+2 -5
View File
@@ -1,6 +1,6 @@
#!/bin/sh
# script/fmt: format the whole repo (writes): prettier over the
# JavaScript, CSS, HTML and Markdown, then gofmt over the Go backend.
# script/fmt: format the whole repo (writes): prettier over everything
# it understands, then gofmt over the Go backend.
# The org model formats only markdown; this repo also has JS and Go.
set -eu
@@ -8,9 +8,6 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
# script/bootstrap links the node, yarn and gofmt it installs into
# ~/.local/bin, which the shell that called it may not have on PATH.
PATH="$HOME/.local/bin:$PATH"
"$ROOT/script/frontend-fmt"
"$ROOT/backend/script/fmt"
}
script/fmt-check
-3
View File
@@ -8,9 +8,6 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
# script/bootstrap links the node, yarn and gofmt it installs into
# ~/.local/bin, which the shell that called it may not have on PATH.
PATH="$HOME/.local/bin:$PATH"
"$ROOT/script/frontend-fmt-check"
"$ROOT/backend/script/fmt-check"
}
script/frontend-check
Executable
+18
View File
@@ -0,0 +1,18 @@
#!/bin/sh
# script/frontend-check: run the frontend tests and format check only.
# This exists for the frontend stage of Dockerfile, a node image with
# neither Go nor Docker; the Dockerfile's frontend-lint stage runs the
# frontend linter, and its lint and builder stages gate the backend.
# Everywhere else, use script/check, which covers the whole repo. Must
# not modify any files.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
"$ROOT/script/frontend-test"
"$ROOT/script/frontend-fmt-check"
}
main "$@"
script/frontend-fmt
+3 -5
View File
@@ -1,8 +1,6 @@
#!/bin/sh
# script/frontend-fmt: format the JavaScript, CSS, HTML and Markdown,
# repo-wide (writes), the markdown in backend/ included. Those are the
# languages REPO_POLICIES.md gives prettier; the YAML is left alone, as
# backend/.golangci.yml must stay the org standard byte for byte.
# script/frontend-fmt: format the frontend and every other file prettier
# understands, repo-wide (writes), the markdown in backend/ included.
# Prettier does not read Go; backend/script/fmt formats the Go sources.
set -eu
@@ -10,7 +8,7 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
yarn prettier --write '**/*.{js,css,html,md}'
yarn prettier --write .
}
main "$@"
script/frontend-fmt-check
+1 -1
View File
@@ -7,7 +7,7 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
yarn prettier --check '**/*.{js,css,html,md}'
yarn prettier --check .
}
main "$@"
script/frontend-lint
Executable
+15
View File
@@ -0,0 +1,15 @@
#!/bin/sh
# script/frontend-lint: run eslint over the frontend. This runs inside
# the frontend-lint stage of Dockerfile, the digest-pinned node image
# with the packages from yarn.lock. From a checkout, run `make lint`,
# which builds that stage.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
yarn eslint .
}
main "$@"
script/frontend-test
+5 -6
View File
@@ -1,8 +1,7 @@
#!/bin/sh
# script/frontend-test: run the frontend test suite on the host: the
# unit tests in test/unit/, through the test script in package.json,
# then the production build, which fails on broken code. make test runs
# the same in the frontend stage of Dockerfile. The tests print a dot
# script/frontend-test: run the frontend test suite: the unit tests in
# test/unit/, through the test script in package.json, then the
# production build, which fails on broken code. The tests print a dot
# each; if any fails, they run again with every test listed, and the
# script fails even if that run passes. NODE_OPTIONS chooses the
# reporter because yarn adds its arguments after the test files, where
@@ -13,9 +12,9 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
NODE_OPTIONS=--test-reporter=dot timeout 90 yarn --silent run test || {
NODE_OPTIONS=--test-reporter=dot timeout 30 yarn --silent run test || {
echo "--- Rerunning with every test listed for details ---"
NODE_OPTIONS=--test-reporter=spec timeout 90 yarn --silent run test
NODE_OPTIONS=--test-reporter=spec timeout 30 yarn --silent run test
exit 1
}
timeout 30 yarn build
script/lint
+13 -13
View File
@@ -1,23 +1,23 @@
#!/bin/sh
# script/lint: run the linter. Linting is a phase of the Dockerfile and
# this builds that phase alone; the linter is never installed or run on
# a developer host, where a shared result cache and a host-global lock
# make its answer untrustworthy.
# script/lint: lint the whole repo: eslint over the frontend, then the Go
# linter over backend/.
#
# The phase is not the last stage in the file, so it is built only when
# --target names it. --no-cache because a cached lint layer is a lint
# that did not run. The tag makes each build replace the previous image
# instead of leaving a dangling one behind.
# No linter runs on the host: this builds the frontend-lint and lint
# stages of Dockerfile, the digest-pinned node and golangci-lint images.
# The first runs eslint; the second runs the backend's fmt-check and
# lint targets. --no-cache makes each linter really run every time
# rather than reuse an earlier result, and each stage is built for its
# checks alone, so no image is kept.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --no-cache \
--target lint \
-t "$("$SCRIPT_DIR/projectname")-lint" .
timeout 300 docker build --no-cache --target frontend-lint \
--output type=cacheonly .
timeout 300 docker build --no-cache --target lint \
--output type=cacheonly .
}
main "$@"
script/test
+8 -10
View File
@@ -1,19 +1,17 @@
#!/bin/sh
# script/test: run the test suite. Testing is a phase of the Dockerfile
# and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when
# named, --no-cache because a cached test layer is a test that did not
# run, and a tag so each build replaces the previous image.
# script/test: run the test suite for the whole repo: the frontend at
# the repo root, then the Go backend in backend/. Each half has its own
# 30-second limit, and there is none around both: from a cold Go build
# cache, compiling the backend's tests with the race detector can take
# 30 seconds on its own, and Go's -timeout leaves the compile out.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --no-cache \
--target test \
-t "$("$SCRIPT_DIR/projectname")-test" .
script/frontend-test
backend/script/test
}
main "$@"
src/main.js
+109 -181
View File
@@ -8,8 +8,6 @@
// display their real value in the latency figure. The history buffer holds
// maxHistoryPoints samples (historyDuration / updateInterval).
// reportInterval is how often collected samples are POSTed to the backend.
// The interval menu changes updateInterval while the page runs; the
// getters compute their values from it each time they are read.
export const CONFIG = {
updateInterval: 3000,
maxHistoryPoints: 100,
@@ -30,39 +28,6 @@ export const CONFIG = {
return [0, 1, 2, 3, 4, 5].map((i) => Math.round((d * i) / 5));
},
canvasHeight: 96,
// A latency figure and its sparkline take the color of the first entry
// whose limit, in ms, the latency is below.
latencyColors: [
{ below: 50, hex: "#22c55e", className: "text-green-500" },
{ below: 100, hex: "#84cc16", className: "text-lime-500" },
{ below: 200, hex: "#eab308", className: "text-yellow-500" },
{ below: 500, hex: "#f97316", className: "text-orange-500" },
{ below: Infinity, hex: "#ef4444", className: "text-red-500" },
],
// The health is offline when more than offlineTimeouts WAN hosts timed
// out or were unreachable and at most offlineReachable answered;
// otherwise degraded when more than degradedTimeouts timed out or were
// unreachable; otherwise slow when more than slowHosts answered after
// more than slowLatency ms.
offlineTimeouts: 10,
offlineReachable: 4,
degradedTimeouts: 4,
slowHosts: 3,
slowLatency: 1000,
// The debug log keeps its last maxLogEntries lines.
maxLogEntries: 1000,
// A gateway candidate that has not answered after gatewayTimeout ms is
// passed over.
gatewayTimeout: 1500,
// When no WAN host answers, the recovery probe checks recoveryProbeHosts
// random ones every recoveryProbeInterval ms.
recoveryProbeHosts: 4,
recoveryProbeInterval: 500,
// The rows are sorted after the first round that is not discarded, then
// every roundsPerSort rounds.
roundsPerSort: 10,
// The sparklines are sized and drawn again resizeDelay ms after start.
resizeDelay: 100,
};
// WAN endpoints to monitor. These are used for the aggregate health/stats
@@ -147,8 +112,7 @@ const debugLog = [];
const log = (() => {
function append(level, message) {
debugLog.push({ timestamp: new Date(), level, message });
if (debugLog.length > CONFIG.maxLogEntries)
debugLog.splice(0, debugLog.length - CONFIG.maxLogEntries);
if (debugLog.length > 1000) debugLog.splice(0, debugLog.length - 1000);
const panel = document.getElementById("debug-panel");
if (panel && !panel.classList.contains("hidden")) renderDebugLog();
}
@@ -208,10 +172,7 @@ async function detectGateway() {
const result = await Promise.any(
GATEWAY_CANDIDATES.map(async (url) => {
const controller = new AbortController();
const timeoutId = setTimeout(
() => controller.abort(),
CONFIG.gatewayTimeout,
);
const timeoutId = setTimeout(() => controller.abort(), 1500);
try {
await fetch(url, {
method: "GET",
@@ -236,40 +197,14 @@ async function detectGateway() {
// --- App State ---------------------------------------------------------------
// The min, max, median and average of latencies, a list of numbers, or all
// null when it is empty. The median of an even count is the mean of the
// middle two; it and the average are rounded.
function latencyStats(latencies) {
if (latencies.length === 0)
return { min: null, max: null, med: null, avg: null };
const sorted = [...latencies].sort((a, b) => a - b);
const mid = Math.floor(sorted.length / 2);
return {
min: sorted[0],
max: sorted[sorted.length - 1],
med:
sorted.length % 2
? sorted[mid]
: Math.round((sorted[mid - 1] + sorted[mid]) / 2),
avg: Math.round(
latencies.reduce((a, b) => a + b, 0) / latencies.length,
),
};
}
export class HostState {
constructor(host, pinned = false) {
this.name = host.name;
this.url = host.url;
// Each entry is either a check's result, { timestamp, latency,
// error }, or a round skipped while paused, { timestamp,
// latency: null, paused: true }.
this.history = [];
this.history = []; // { timestamp, latency, paused }
this.lastLatency = null;
this.status = "pending"; // 'online' | 'offline' | 'error' | 'pending'
this.pinned = pinned;
// How many recorded checks in a row have failed, up to the last one.
this.consecutiveFailures = 0;
}
pushSample(timestamp, result) {
@@ -283,9 +218,6 @@ export class HostState {
if (result.error === "timeout") this.status = "error";
else if (result.error) this.status = "offline";
else this.status = "online";
this.consecutiveFailures = result.error
? this.consecutiveFailures + 1
: 0;
}
pushPaused(timestamp) {
@@ -293,16 +225,38 @@ export class HostState {
this._trim();
}
// The min, max, median and average latency of the checks in the history
// that got an answer.
historyStats() {
return latencyStats(
this.history
.filter((p) => p.latency !== null)
.map((p) => p.latency),
averageLatency() {
const valid = this.history.filter((p) => p.latency !== null);
if (valid.length === 0) return null;
return Math.round(
valid.reduce((s, p) => s + p.latency, 0) / valid.length,
);
}
minLatency() {
const valid = this.history.filter((p) => p.latency !== null);
if (valid.length === 0) return null;
return Math.min(...valid.map((p) => p.latency));
}
maxLatency() {
const valid = this.history.filter((p) => p.latency !== null);
if (valid.length === 0) return null;
return Math.max(...valid.map((p) => p.latency));
}
medianLatency() {
const sorted = this.history
.filter((p) => p.latency !== null)
.map((p) => p.latency)
.sort((a, b) => a - b);
if (sorted.length === 0) return null;
const mid = Math.floor(sorted.length / 2);
return sorted.length % 2
? sorted[mid]
: Math.round((sorted[mid - 1] + sorted[mid]) / 2);
}
_trim() {
while (this.history.length > CONFIG.maxHistoryPoints)
this.history.shift();
@@ -317,10 +271,6 @@ export class AppState {
this.local = localHosts.map((h) => new HostState(h));
this.paused = false;
this.tickCount = 0;
// The recovery probe's timer, null while it is not running, and the
// checks it started last.
this._recoveryProbeId = null;
this._recoveryProbeChecks = null;
}
get allHosts() {
@@ -329,13 +279,33 @@ export class AppState {
/** WAN-only stats from latest sample (excludes local) */
wanStats() {
const latencies = this.wan
.filter((h) => h.lastLatency !== null)
.map((h) => h.lastLatency);
const reachable = this.wan.filter((h) => h.lastLatency !== null);
const latencies = reachable.map((h) => h.lastLatency);
const total = this.wan.length;
if (latencies.length === 0)
return {
reachable: 0,
total,
min: null,
max: null,
med: null,
avg: null,
};
const sorted = [...latencies].sort((a, b) => a - b);
const mid = Math.floor(sorted.length / 2);
const med =
sorted.length % 2
? sorted[mid]
: Math.round((sorted[mid - 1] + sorted[mid]) / 2);
return {
reachable: latencies.length,
total: this.wan.length,
...latencyStats(latencies),
total,
min: Math.min(...latencies),
max: Math.max(...latencies),
med,
avg: Math.round(
latencies.reduce((a, b) => a + b, 0) / latencies.length,
),
};
}
@@ -361,16 +331,12 @@ export class AppState {
const timeouts = this.wan.filter(
(h) => h.status === "error" || h.status === "offline",
).length;
if (
timeouts > CONFIG.offlineTimeouts &&
reachable <= CONFIG.offlineReachable
)
return "offline";
if (timeouts > CONFIG.degradedTimeouts) return "degraded";
if (timeouts > 10 && reachable <= 4) return "offline";
if (timeouts > 4) return "degraded";
const slow = this.wan.filter(
(h) => h.lastLatency !== null && h.lastLatency > CONFIG.slowLatency,
(h) => h.lastLatency !== null && h.lastLatency > 1000,
).length;
if (slow > CONFIG.slowHosts) return "slow";
if (slow > 3) return "slow";
return "healthy";
}
@@ -539,13 +505,7 @@ class Reporter {
// Checks one target. The check times out after CONFIG.requestTimeout; the
// caller can give it up sooner through the optional signal, which also ends
// it as a timeout. A failed check's reason says what went wrong and after
// how long; a check that answered has none.
//
// The URL is fetched as written, with nothing added to it: the Hetzner
// speed-test servers close the connection without an answer when the URL
// has a query string, and cache: "no-store" keeps the browser's cache out
// of the measurement.
// it as a timeout.
export async function measureLatency(url, signal) {
const controller = new AbortController();
const timeoutId = setTimeout(
@@ -554,10 +514,13 @@ export async function measureLatency(url, signal) {
);
signal?.addEventListener("abort", () => controller.abort());
const targetUrl = new URL(url);
targetUrl.searchParams.set("_cb", Date.now().toString());
const start = performance.now();
try {
await fetch(url, {
await fetch(targetUrl.toString(), {
method: "GET",
mode: "no-cors",
cache: "no-store",
@@ -566,28 +529,18 @@ export async function measureLatency(url, signal) {
const latency = Math.round(performance.now() - start);
clearTimeout(timeoutId);
if (latency > CONFIG.maxLatency) {
return {
latency: null,
error: "timeout",
reason: `answered after ${latency} ms, over the ${CONFIG.maxLatency} ms limit`,
};
log.error(`${url} timeout (${latency}ms > ${CONFIG.maxLatency}ms)`);
return { latency: null, error: "timeout" };
}
return { latency, error: null, reason: null };
return { latency, error: null };
} catch (err) {
const took = Math.round(performance.now() - start);
clearTimeout(timeoutId);
if (err.name === "AbortError") {
return {
latency: null,
error: "timeout",
reason: `timed out after ${took} ms (limit ${CONFIG.requestTimeout} ms)`,
};
log.error(`${url} timeout (aborted)`);
return { latency: null, error: "timeout" };
}
return {
latency: null,
error: "unreachable",
reason: `network error (${err.name}: ${err.message}) after ${took} ms`,
};
log.error(`${url} unreachable`);
return { latency: null, error: "unreachable" };
}
}
@@ -595,13 +548,21 @@ export async function measureLatency(url, signal) {
export function latencyHex(latency) {
if (latency === null) return "#6b7280";
return CONFIG.latencyColors.find((c) => latency < c.below).hex;
if (latency < 50) return "#22c55e";
if (latency < 100) return "#84cc16";
if (latency < 200) return "#eab308";
if (latency < 500) return "#f97316";
return "#ef4444";
}
export function latencyClass(latency, status) {
if (status === "offline" || status === "error" || latency === null)
return "text-gray-500";
return CONFIG.latencyColors.find((c) => latency < c.below).className;
if (latency < 50) return "text-green-500";
if (latency < 100) return "text-lime-500";
if (latency < 200) return "text-yellow-500";
if (latency < 500) return "text-orange-500";
return "text-red-500";
}
// --- Sparkline Renderer ------------------------------------------------------
@@ -619,8 +580,8 @@ class SparklineRenderer {
const ch = h - m.top - m.bottom;
ctx.clearRect(0, 0, w, h);
SparklineRenderer._drawYAxis(ctx, w, m, ch);
SparklineRenderer._drawXAxis(ctx, h, m, cw);
SparklineRenderer._drawYAxis(ctx, w, h, m, ch);
SparklineRenderer._drawXAxis(ctx, w, h, m, cw);
const len = history.length;
const pw = cw / (CONFIG.maxHistoryPoints - 1);
@@ -636,7 +597,7 @@ class SparklineRenderer {
SparklineRenderer._drawTip(ctx, history, getX, getY);
}
static _drawYAxis(ctx, w, m, ch) {
static _drawYAxis(ctx, w, h, m, ch) {
ctx.font = "300 12px monospace";
ctx.textAlign = "right";
ctx.textBaseline = "middle";
@@ -653,7 +614,7 @@ class SparklineRenderer {
}
}
static _drawXAxis(ctx, h, m, cw) {
static _drawXAxis(ctx, w, h, m, cw) {
ctx.textAlign = "center";
ctx.textBaseline = "top";
for (const tick of CONFIG.xAxisTicks) {
@@ -741,18 +702,7 @@ class SparklineRenderer {
// horizontally.
const STATUS_TEXT_CLASS = "status-text text-xs text-right col-span-2 mt-5";
// Escapes text for HTML, so it shows as written inside an element or a
// quoted attribute and is never read as markup.
function escapeHTML(text) {
return text
.replaceAll("&", "&amp;")
.replaceAll("<", "&lt;")
.replaceAll(">", "&gt;")
.replaceAll('"', "&quot;")
.replaceAll("'", "&#39;");
}
export function hostRowHTML(host, index, showPin = true) {
function hostRowHTML(host, index, showPin = true) {
const pinColor = host.pinned
? "text-blue-500"
: "text-gray-600 hover:text-gray-400";
@@ -771,12 +721,12 @@ export function hostRowHTML(host, index, showPin = true) {
<div class="w-[420px] flex-shrink-0 grid grid-cols-[minmax(0,1fr)_auto] items-center">
<div class="flex items-center gap-2 min-w-[200px]">
<div class="w-3 h-3 rounded-full flex-shrink-0 bg-[#6b7280]"></div>
<span class="font-medium text-white truncate">${escapeHTML(host.name)}</span>
<span class="font-medium text-white truncate">${host.name}</span>
</div>
<div class="latency-value text-4xl font-bold tabular-nums text-right mt-3" data-host="${index}">
<span class="text-gray-500">---</span>
</div>
<a href="${escapeHTML(host.url)}" target="_blank" rel="noopener" class="text-xs text-gray-500 truncate block col-span-2 -mt-2">${escapeHTML(host.url)}</a>
<a href="${host.url}" target="_blank" rel="noopener" class="text-xs text-gray-500 truncate block col-span-2 -mt-2">${host.url}</a>
<div class="${STATUS_TEXT_CLASS} text-gray-500" data-host="${index}">waiting...</div>
</div>
<div class="flex-grow sparkline-container rounded overflow-hidden border border-gray-700/30">
@@ -875,7 +825,7 @@ function buildUI(state) {
</div>
<footer class="mt-8 text-center text-gray-600 text-xs">
<p>Latency measured via GET requests | CORS restrictions may affect some measurements</p>
<p>Latency measured via GET requests | IPv4 only | CORS restrictions may affect some measurements</p>
<p class="mt-2">
<span class="inline-block w-3 h-3 rounded-full bg-green-500 mr-1 align-middle"></span>&lt;50ms
<span class="inline-block w-3 h-3 rounded-full bg-lime-500 mr-1 ml-3 align-middle"></span>&lt;100ms
@@ -942,7 +892,10 @@ function updateHostRow(host, index) {
latencyEl.innerHTML = `<span class="text-gray-500">---</span>`;
}
const { min, med, avg, max } = host.historyStats();
const avg = host.averageLatency();
const med = host.medianLatency();
const min = host.minLatency();
const max = host.maxLatency();
if (host.status === "online" && avg !== null) {
statusEl.innerHTML = statusStatsHTML([
["min", min],
@@ -1089,16 +1042,14 @@ function renderDebugLog() {
info: "text-gray-300",
debug: "text-gray-500",
};
el.replaceChildren(
...debugLog.map((entry) => {
el.innerHTML = debugLog
.map((entry) => {
const ts = formatUTCTimestamp(entry.timestamp);
const cls = levelColors[entry.level] || "text-gray-400";
const lvl = entry.level.toUpperCase().padEnd(7);
const line = document.createElement("div");
line.className = levelColors[entry.level] || "text-gray-400";
line.textContent = `${ts} ${lvl} ${entry.message}`;
return line;
}),
);
return `<div class="${cls}">${ts} ${lvl} ${entry.message}</div>`;
})
.join("");
el.scrollTop = el.scrollHeight;
}
@@ -1154,26 +1105,6 @@ function sortAndRebuildWAN(state) {
// --- Main Loop ---------------------------------------------------------------
// Writes one line to the browser console, and the same line to the debug
// log, for a check the page is about to record: for every check that
// failed, and for a check that answered after checks that failed. Called
// before host.pushSample, while host.consecutiveFailures still counts the
// checks before this one.
function logCheck(host, result) {
const target = `${host.name} ${host.url} at ${new Date().toISOString()}`;
const failed = host.consecutiveFailures;
if (result.error) {
const line = `netwatch: check failed: ${target}: ${result.reason}`;
console.error(line);
log.error(line);
} else if (failed > 0) {
const checks = failed === 1 ? "check" : "checks";
const line = `netwatch: target recovered: ${target}: answered after ${result.latency} ms, following ${failed} failed ${checks} in a row`;
console.info(line);
log.info(line);
}
}
export async function tick(state, signal, onOffline) {
const ts = Date.now();
@@ -1207,7 +1138,6 @@ export async function tick(state, signal, onOffline) {
if (state.paused || signal.aborted || state.tickCount === 0) {
return;
}
logCheck(host, r);
host.pushSample(ts, r);
updateHostRow(host, state.allHosts.indexOf(host));
log.debug(`${host.name}: ${r.error ? r.error : r.latency + "ms"}`);
@@ -1230,9 +1160,8 @@ export async function tick(state, signal, onOffline) {
// rows whose check ended before the resume still read "paused"
state.allHosts.forEach((host, i) => updateHostRow(host, i));
// Sort after the first real check, then every CONFIG.roundsPerSort
// ticks thereafter
if (state.tickCount === 2 || state.tickCount % CONFIG.roundsPerSort === 1) {
// Sort after the first real check, then every 10 ticks thereafter
if (state.tickCount === 2 || state.tickCount % 10 === 1) {
sortAndRebuildWAN(state);
}
@@ -1255,10 +1184,9 @@ export async function tick(state, signal, onOffline) {
// --- Recovery Probe ----------------------------------------------------------
// When offline, check CONFIG.recoveryProbeHosts random WAN hosts every
// CONFIG.recoveryProbeInterval ms, giving up the checks started one interval
// before, so at most that many are ever waiting. As soon as one answers,
// stop probing and start a new round at once.
// When offline, check 4 random WAN hosts every 500ms, giving up the checks
// started 500ms before, so at most 4 are ever waiting. As soon as one
// answers, stop probing and start a new round at once.
function startRecoveryProbe(state, startRounds) {
if (state._recoveryProbeId) return; // already running
const candidates = [...state.wan];
@@ -1266,7 +1194,7 @@ function startRecoveryProbe(state, startRounds) {
const j = Math.floor(Math.random() * (i + 1));
[candidates[i], candidates[j]] = [candidates[j], candidates[i]];
}
const canaries = candidates.slice(0, CONFIG.recoveryProbeHosts);
const canaries = candidates.slice(0, 4);
log.notice(
`Recovery probe started (${canaries.map((h) => h.name).join(", ")})`,
);
@@ -1283,7 +1211,7 @@ function startRecoveryProbe(state, startRounds) {
startRounds();
});
}
}, CONFIG.recoveryProbeInterval);
}, 500);
}
function stopRecoveryProbe(state) {
@@ -1547,7 +1475,7 @@ async function init() {
});
window.addEventListener("resize", () => handleResize(state));
setTimeout(() => handleResize(state), CONFIG.resizeDelay);
setTimeout(() => handleResize(state), 100);
}
// Bootstrap only when loaded as the page: a real DOM containing the #app
test/unit/main.test.js
+21 -211
View File
@@ -8,7 +8,6 @@ import {
AppState,
CONFIG,
greyOutUI,
hostRowHTML,
HostState,
humanDuration,
latencyClass,
@@ -23,16 +22,12 @@ import {
// test looks it up and kept in elements under its selector until the next
// test starts. As on a page, writing its text replaces its markup; the
// status dot greyOutUI looks for in it is not there. Drawing a sparkline
// does nothing; it looks for the pixel ratio on window and finds none. In
// each test, console.error and console.info print nothing and keep what
// they are given.
const doNothing = () => {};
// does nothing; it looks for the pixel ratio on window and finds none.
let elements;
beforeEach((t) => {
beforeEach(() => {
elements = {};
t.mock.method(console, "error", doNothing);
t.mock.method(console, "info", doNothing);
});
const doNothing = () => {};
const canvasContext = {
clearRect: doNothing,
beginPath: doNothing,
@@ -73,10 +68,8 @@ function statusText(state, host) {
// Mocks the clock for test t, so that a check lasting seconds takes no real
// time, and replaces fetch with targets that each answer after
// answerAfter(url) milliseconds of that clock, or never when that is
// Infinity. The target at unreachableUrl, if one is given, does not
// answer: after that time its fetch fails with the error a browser gives
// the page for a network error. Both are restored when the test ends.
function mockTargets(t, answerAfter, unreachableUrl) {
// Infinity. Both are restored when the test ends.
function mockTargets(t, answerAfter) {
t.mock.timers.enable({ apis: ["setTimeout", "Date"] });
t.mock.method(performance, "now", () => Date.now());
t.mock.method(
@@ -84,12 +77,8 @@ function mockTargets(t, answerAfter, unreachableUrl) {
"fetch",
(url, { signal }) =>
new Promise((resolve, reject) => {
const answer =
url === unreachableUrl
? () => reject(new TypeError("Failed to fetch"))
: resolve;
if (answerAfter(url) !== Infinity) {
setTimeout(answer, answerAfter(url));
setTimeout(resolve, answerAfter(url));
}
signal.addEventListener("abort", () => reject(signal.reason));
}),
@@ -117,7 +106,6 @@ for (const interval of [10000, 30000]) {
assert.deepEqual(await settled(check), {
latency: slowAnswer,
error: null,
reason: null,
});
});
@@ -131,37 +119,10 @@ for (const interval of [10000, 30000]) {
assert.deepEqual(await settled(check), {
latency: null,
error: "timeout",
reason: `timed out after ${timeout} ms (limit ${timeout} ms)`,
});
});
}
// A browser can run a timer late, on a busy page or in a background tab.
// Moving the mocked clock on 30000ms at once runs the 24000ms timeout with
// the clock already at 30000ms.
test("at a 30000ms interval, a check whose timeout runs late gives how long the request took and the time limit", async (t) => {
CONFIG.updateInterval = 30000;
mockTargets(t, () => Infinity);
const check = measureLatency("https://target.test");
t.mock.timers.tick(30000);
assert.deepEqual(await settled(check), {
latency: null,
error: "timeout",
reason: "timed out after 30000 ms (limit 24000 ms)",
});
});
test("a check fetches the target's URL as written, with no query string added", async (t) => {
mockTargets(t, () => 10);
const check = measureLatency("https://fsn1-speed.hetzner.com");
t.mock.timers.tick(10);
assert.notEqual(await settled(check), "still waiting");
assert.equal(
fetch.mock.calls[0].arguments[0],
"https://fsn1-speed.hetzner.com",
);
});
test("at a 30000ms interval, a target answering after 1000ms shows in its row while another target's check is still waiting", async (t) => {
CONFIG.updateInterval = 30000;
const state = new AppState([
@@ -170,7 +131,7 @@ test("at a 30000ms interval, a target answering after 1000ms shows in its row wh
const answering = state.local[0];
const waiting = state.wan[0];
// No target but the answering one ever answers.
mockTargets(t, (url) => (url === answering.url ? 1000 : Infinity));
mockTargets(t, (url) => (url.startsWith(answering.url) ? 1000 : Infinity));
// The third tick: the first is discarded as a whole, and the second ends
// by sorting the rows, which rebuilds a page that is not here.
state.tickCount = 2;
@@ -197,7 +158,7 @@ test("at a 30000ms interval, a check still waiting when its round is given up do
{ name: "Answering", url: "https://answering.test" },
]);
const answering = state.local[0];
mockTargets(t, (url) => (url === answering.url ? 1000 : Infinity));
mockTargets(t, (url) => (url.startsWith(answering.url) ? 1000 : Infinity));
state.tickCount = 2;
const roundChecks = new AbortController();
@@ -215,7 +176,7 @@ test("at a 30000ms interval, a check still waiting when the user pauses does not
{ name: "Answering", url: "https://answering.test" },
]);
const answering = state.local[0];
mockTargets(t, (url) => (url === answering.url ? 1000 : Infinity));
mockTargets(t, (url) => (url.startsWith(answering.url) ? 1000 : Infinity));
state.tickCount = 2;
const round = tick(state, new AbortController().signal);
@@ -232,7 +193,7 @@ test("at a 30000ms interval, a check in the first round does not show in its row
{ name: "Answering", url: "https://answering.test" },
]);
const answering = state.local[0];
mockTargets(t, (url) => (url === answering.url ? 1000 : Infinity));
mockTargets(t, (url) => (url.startsWith(answering.url) ? 1000 : Infinity));
const round = tick(state, new AbortController().signal);
t.mock.timers.tick(1000);
@@ -246,7 +207,7 @@ test("at a 30000ms interval, after the user pauses and resumes during a round, n
{ name: "Answering", url: "https://answering.test" },
]);
const answering = state.local[0];
mockTargets(t, (url) => (url === answering.url ? 1000 : Infinity));
mockTargets(t, (url) => (url.startsWith(answering.url) ? 1000 : Infinity));
state.tickCount = 2;
const round = tick(state, new AbortController().signal);
@@ -269,141 +230,6 @@ test("at a 30000ms interval, after the user pauses and resumes during a round, n
}
});
// In the next tests a round checks one target, Target, at a 30000ms
// interval, so a check times out after 24000ms. The mocked clock starts at
// 1970-01-01T00:00:00.000Z.
// An app state with Target and no WAN targets, so its rounds check only
// Target, and whose next round is recorded: it is the third, as the first
// is discarded and the second ends by sorting the rows, which rebuilds a
// page that is not here. Returns it and Target.
function stateWithOneTarget() {
CONFIG.updateInterval = 30000;
const state = new AppState([
{ name: "Target", url: "https://target.test" },
]);
state.wan = [];
state.tickCount = 2;
return { state, target: state.local[0] };
}
// What this test wrote to the browser console with console[method].
function consoleLines(method) {
return console[method].mock.calls.map((call) => call.arguments[0]);
}
test("a check that fails with a network error writes one console line with the target, the time, the error and how long the request took", async (t) => {
const { state, target } = stateWithOneTarget();
mockTargets(t, () => 23, target.url);
const round = tick(state, new AbortController().signal);
t.mock.timers.tick(23);
assert.notEqual(await settled(round), "still waiting");
assert.deepEqual(consoleLines("error"), [
"netwatch: check failed: Target https://target.test at 1970-01-01T00:00:00.023Z: network error (TypeError: Failed to fetch) after 23 ms",
]);
assert.deepEqual(consoleLines("info"), []);
});
test("a check that times out writes one console line with the target, the time, how long the request took and the time limit", async (t) => {
const { state } = stateWithOneTarget();
mockTargets(t, () => Infinity);
const round = tick(state, new AbortController().signal);
t.mock.timers.tick(24000);
assert.notEqual(await settled(round), "still waiting");
assert.deepEqual(consoleLines("error"), [
"netwatch: check failed: Target https://target.test at 1970-01-01T00:00:24.000Z: timed out after 24000 ms (limit 24000 ms)",
]);
assert.deepEqual(consoleLines("info"), []);
});
// An answer that took longer than CONFIG.maxLatency is recorded as a
// timeout. The limit is the check's own timeout, so such an answer is one
// that came in just as the check timed out; here it is lowered to 500ms.
test("a check answered over the time limit writes one console line with the target, the time, how long the answer took and the limit", async (t) => {
const { state } = stateWithOneTarget();
t.mock.getter(CONFIG, "maxLatency", () => 500);
mockTargets(t, () => 1000);
const round = tick(state, new AbortController().signal);
t.mock.timers.tick(1000);
assert.notEqual(await settled(round), "still waiting");
assert.deepEqual(consoleLines("error"), [
"netwatch: check failed: Target https://target.test at 1970-01-01T00:00:01.000Z: answered after 1000 ms, over the 500 ms limit",
]);
assert.deepEqual(consoleLines("info"), []);
});
test("a check that answers writes nothing to the console", async (t) => {
const { state } = stateWithOneTarget();
mockTargets(t, () => 30);
const round = tick(state, new AbortController().signal);
t.mock.timers.tick(30);
assert.notEqual(await settled(round), "still waiting");
assert.deepEqual(consoleLines("error"), []);
assert.deepEqual(consoleLines("info"), []);
});
for (const [failed, checks] of [
[1, "1 failed check"],
[3, "3 failed checks"],
]) {
test(`a check that answers after ${checks} writes one console line saying the target recovered, and the next writes nothing`, async (t) => {
const { state, target } = stateWithOneTarget();
for (let i = 0; i < failed; i++) {
target.pushSample(Date.now(), {
latency: null,
error: "unreachable",
});
}
mockTargets(t, () => 40);
for (let i = 0; i < 2; i++) {
const round = tick(state, new AbortController().signal);
t.mock.timers.tick(40);
assert.notEqual(await settled(round), "still waiting");
}
assert.deepEqual(consoleLines("info"), [
`netwatch: target recovered: Target https://target.test at 1970-01-01T00:00:00.040Z: answered after 40 ms, following ${checks} in a row`,
]);
assert.deepEqual(consoleLines("error"), []);
});
}
test("a check that fails in the first round, which is discarded, writes nothing to the console", async (t) => {
const { state, target } = stateWithOneTarget();
state.tickCount = 0;
mockTargets(t, () => 23, target.url);
const round = tick(state, new AbortController().signal);
t.mock.timers.tick(23);
assert.notEqual(await settled(round), "still waiting");
assert.deepEqual(consoleLines("error"), []);
assert.deepEqual(consoleLines("info"), []);
});
// The page shows &lt; &gt; &quot; &amp; and &#39; in a row's markup as
// < > " & and '.
test(`a target whose name and URL hold < > " & and ' shows those characters in its row`, () => {
const host = new HostState({
name: `<b>"x" & 'y'</b>`,
url: `https://x.test/<b>?a="x"&b='y'`,
});
const row = hostRowHTML(host, 0);
assert.doesNotMatch(row, /<b>/);
assert.ok(
row.includes(
">&lt;b&gt;&quot;x&quot; &amp; &#39;y&#39;&lt;/b&gt;</span>",
),
);
assert.ok(
row.includes(
'href="https://x.test/&lt;b&gt;?a=&quot;x&quot;&amp;b=&#39;y&#39;"',
),
);
assert.ok(
row.includes(
">https://x.test/&lt;b&gt;?a=&quot;x&quot;&amp;b=&#39;y&#39;</a>",
),
);
});
for (const [seconds, text] of [
[0, "0s"],
[1, "1s"],
@@ -494,35 +320,19 @@ for (const { history, latencies, statistics } of [
},
]) {
test(`a target's min, max, average and median latency over ${history}`, () => {
const { min, max, avg, med } = hostAfter(latencies).historyStats();
assert.deepEqual({ min, max, average: avg, median: med }, statistics);
const host = hostAfter(latencies);
assert.deepEqual(
{
min: host.minLatency(),
max: host.maxLatency(),
average: host.averageLatency(),
median: host.medianLatency(),
},
statistics,
);
});
}
// The summary's figures come from each WAN target's last check, by the same
// rules as a target's own: here four answered, one was found unreachable
// and the rest have not been checked yet. The median, 22.5, and the
// average, 21.25, are rounded.
test("the summary's min, max, median and average latency over the WAN targets' last checks", () => {
const state = new AppState([]);
[30, 10, null, 25, 20].forEach((latency, i) =>
state.wan[i].pushSample(
Date.now(),
latency === null
? { latency: null, error: "unreachable" }
: { latency, error: null },
),
);
assert.deepEqual(state.wanStats(), {
reachable: 4,
total: state.wan.length,
min: 10,
max: 30,
med: 23,
avg: 21,
});
});
// An app state in which, of the WAN targets, the first timedOut timed out,
// the next unreachable were found unreachable, the next answered answered
// after latency ms, and the rest have not been checked yet.
test/viewport/README.md
+9 -10
View File
@@ -13,8 +13,8 @@ width derived from the app's own CSS. Screenshots land in `tmp/viewport/`
alongside a `results.json`; they are artifacts for a human to look at when
something fails, not the evidence. The assertions are the evidence.
The target is deliberately outside `make check`: it takes minutes, and
`make test` has to stay under 60 seconds.
The target is deliberately outside `make check`: it needs Docker and takes
minutes, and `make test` has to stay under 20 seconds.
## How the widths are chosen
@@ -77,7 +77,7 @@ the internet, so the app's latency probes cannot reach anything real. The
harness answers them itself from a fixed delay table, with a deterministic
fraction failed outright, so the rows render a realistic spread of one-, two-
and three-digit latencies plus some unreachable rows. That spread is what the
layout has to survive; a `---` placeholder in every row would not exercise it.
layout has to survive; 24 identical `---` placeholders would not exercise it.
## What this cannot verify
@@ -104,11 +104,10 @@ Everything else this issue was actually about — does the layout reflow, does
anything overflow, is content clipped, are the controls big enough — is a
function of viewport width and CSS, and is covered above.
## Relation to the unit tests
## Relation to the unit test framework (#21)
Complementary layers, not two stacks. The unit tests in `test/unit/`, which
`make test` runs with Node's built-in test runner, exercise the functions
`src/main.js` exports in-process with no browser. This harness exercises
rendered layout in a real engine and is the only thing here that can see a media
query. Neither replaces the other; assertions about computed styles and element
geometry belong here, assertions about functions belong in `test/unit/`.
Complementary layers, not two stacks. `vitest` (#21) will exercise module-level
logic in-process with no browser. This harness exercises rendered layout in a
real engine and is the only thing here that can see a media query. Neither
replaces the other; assertions about computed styles and element geometry belong
here, assertions about functions belong in `vitest`.