Author SHA1 Message Date
clawbot 6f8f42f2bf Target check timeout is 80% of the refresh interval (closes #78)
check / check (push) Successful in 1m54s
Each target check now times out after 80% of the refresh interval, 24
seconds at 30 seconds, where it was capped at 3 seconds, so slow, far
targets are recorded with their real time. A round asked for while the
last one's checks are still waiting, after an interval change or by the
recovery probe, is skipped, so rounds never overlap; the recovery probe
starts no new checks while its last ones wait.

The frontend has its first unit tests, run by script/frontend-test with
Node's built-in test runner. index.html now links src/styles.css, which
src/main.js imported, since Node cannot import CSS.

Model: opus-5-5
2026-09-29 10:32:03 +00:00
sneak 7dfb3b8c32 Merge branch 'main' into next
check / check (push) Successful in 2m6s
2026-09-29 12:04:10 +02:00
clawbot a5ca73c585 Container sets up its own data directory (closes #75) (#76)
check / check (push) Successful in 2m6s
Closes #75.

`bin/entrypoint.sh`, which already runs as root, now makes the data directory usable before the backend starts: it creates `DATA_DIR` if missing, gives it and `/data` to the `netwatch` user (`chown -R`), and sets mode 750 on both, the mode the backend gives a directory it creates. The backend still runs as `netwatch`. The README "Running under upaas" section loses its first-run step that created and chowned the host directory and names only the path to mount. The Dockerfile's build-time `mkdir` and `chown` of `/data` are gone, since the entrypoint now does this on every start.

What the diff does not show:

- The host directory mounted at `/data` ends up owned by uid 1000 with mode 750, and everything under `DATA_DIR` is chowned to uid 1000 on every start.
- If the directory cannot be created or chowned, the container stops with that tool's error before either process starts.

Recorded runs with `--mount type=bind`: an empty directory owned by root (mode 755, and again mode 700), and one holding a `reports` directory and report file owned by uid 1001 with mode 700. Each time the container turned healthy, `netwatch-server` ran as `netwatch`, and a posted report was written to `DATA_DIR`; a second start on the root-owned and the uid 1001 directories did the same.

Judgement call: `/data` itself is given to `netwatch` as well as `DATA_DIR`, so the backend can reach `DATA_DIR` inside a host directory with mode 700.

Model: opus-5-5
Reviewed-on: #76
Co-authored-by: clawbot <35+clawbot@noreply.example.org>
2026-09-29 12:03:59 +02:00
clawbot f423768975 cibuild: the org model, which runs every check uncached (closes #37)
check / check (push) Successful in 2m13s
script/cibuild was a plain docker build ., so on a tree Docker had
seen before every check step came from the build cache and the build
still passed. It is now the org model from sneak/prompts, byte for
byte: script/bootstrap, script/check, then docker build --no-cache
with the git describe version as the VERSION build argument.

The workflow puts ~/.local/bin, where bootstrap links what it
installs, on the step's PATH. Bootstrap now installs its pinned node
when the installed one is older than 22.12.0, the oldest the
frontend's dependencies accept (puppeteer-core's engines field), as
it already does for Go against backend/go.mod.

Model: opus-5-5
2026-09-29 11:55:52 +02:00
clawbot c226ceee01 chore(backend): re-vendor .golangci.yml with gomodguard_v2 (closes #41)
check / check (push) Successful in 24s
golangci-lint v2.12 deprecates gomodguard, which the org .golangci.yml
reached through "default: all", so every lint run printed a
deprecation warning. backend/.golangci.yml is now the current copy
from sneak/prompts, fetched unedited: gomodguard is disabled and
gomodguard_v2 enabled with the org block list. The new file also
turns depguard on with its test-support rule, which forbids
net/http/httptest outside test code. netwatch has no test-support
packages of its own to add to that rule, so the file is identical to
the canonical one. backend/script/lint checks the new sha256. The
backend raises no findings under the new rules.

Model: opus-5-5
2026-09-29 10:56:00 +02:00
sneak bd08e901ee next into main: netwatch as one container, ready for upaas (#49)
check / check (push) Successful in 12s
Reviewed-on: #49
2026-09-29 10:43:12 +02:00
clawbot d81da05748 nginx: security headers on every response (closes #18)
check / check (push) Successful in 21s
nginx sent none of the security headers REPO_POLICIES.md requires.
security-headers.conf now sets all six with always, included at server
level and again in /assets/, whose own add_header would otherwise drop
them. nginx hides the copies netwatch-server sets, so /api/ and the
health check carry each header once. The content security policy
allows no inline script or style; the host row's status dot took its
grey from a style attribute, now a class. connect-src is * because
several probed hosts redirect to other hosts and the browser checks
every redirect against it. Referrer-Policy is no-referrer, as the
backend already sends.

Model: opus-5-5
2026-09-29 10:22:12 +02:00
clawbot d74d1e311e fix(backend): cut request log fields to the log bound (closes #60)
check / check (push) Successful in 14s
The request log wrote the URL, User-Agent, Referer and other
request-supplied strings with no length limit, and the server accepts
headers up to 1 MiB, so one request could put about 1 MiB per field
into a log line. Every string the request log takes from the request,
including the request ID chi copies from X-Request-Id, is now cut to
the 128-byte bound the report handler already used. That bound and
its helper moved from the handlers package to the logger package so
both use the one copy.

Model: opus-5-5
2026-09-29 09:39:11 +02:00
clawbot 8833603eff nginx: trust X-Forwarded-For only from TRUSTED_PROXIES (closes #64)
check / check (push) Successful in 15s
nginx trusted X-Forwarded-For from every RFC1918 address, so a client
reaching it from one could write a new address on each request and
get a fresh rate-limit allowance. The container's TRUSTED_PROXIES now
names the reverse proxies nginx trusts, none by default.
bin/entrypoint.sh makes each entry a CIDR, checks it with the new
"netwatch-server check-cidr", which runs the server's own
TRUSTED_PROXIES parsing, and writes one set_real_ip_from line per
entry into /etc/nginx/trusted-proxies.conf, which nginx.conf includes.
The backend is started with TRUSTED_PROXIES=127.0.0.1/32, since nginx
is its only client. The viewport test mounts an empty file there.

Model: opus-5-5
2026-09-29 08:55:47 +02:00
clawbot 6022cc8b02 fix(backend): give each report file a name of its own (closes #61)
check / check (push) Successful in 13s
Report files were named by a millisecond timestamp and created with
O_EXCL, so two flushes in the same millisecond, such as a flush for
size and the final flush at shutdown, got the same name and the second
failed, losing its reports. Each name now carries a number after the
timestamp that goes up by one for each file the server starts to
write, so names still sort by time and never repeat within a run. A
failed write uses up its number, leaving a gap if the file could not
be created and otherwise a file under that number that may be
incomplete.

Model: opus-5-5
2026-09-29 08:05:26 +02:00
clawbot d2f219ca19 upaas: health check, settings checked at start, README section (closes #59)
check / check (push) Successful in 15s
The image's HEALTHCHECK requests /.well-known/healthcheck through
nginx on the port from PORT, so it fails unless both processes answer.
The backend reads PORT and DEBUG with strconv instead of viper, which
turned a bad PORT into 0 and a bad DEBUG into false. Those, and a
BIND_ADDRESS that is not an IP address, now stop the start with an
error naming the variable; the TRUSTED_PROXIES error names it too.
bin/entrypoint.sh also refuses a container PORT outside 1 to 65535,
or 8081, where the backend listens, naming PORT. README.md gains
"Running under upaas". Its first-run steps create the host directory
owned by uid 1000, so the image changes no ownership.

Model: opus-5-5
2026-09-29 06:39:10 +02:00
27 changed files with 733 additions and 147 deletions
+4 -2
View File
@@ -6,5 +6,7 @@ jobs:
steps: steps:
# actions/checkout v4.2.2, 2026-02-22 # actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
# script/cibuild builds the image, whose stages run every check. # script/cibuild bootstraps, runs every check and builds the
- run: script/cibuild # image. script/bootstrap links what it installs into
# ~/.local/bin, so that has to be on PATH for the rest.
- run: PATH="$HOME/.local/bin:$PATH" script/cibuild
+9 -1
View File
@@ -72,17 +72,25 @@ RUN addgroup -g 1000 -S netwatch && \
# conf.d; bin/entrypoint.sh says how. # conf.d; bin/entrypoint.sh says how.
RUN rm /etc/nginx/conf.d/default.conf RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/templates/netwatch.conf.template COPY nginx.conf /etc/nginx/templates/netwatch.conf.template
COPY security-headers.conf /etc/nginx/security-headers.conf
COPY --from=frontend /app/dist /usr/share/nginx/html COPY --from=frontend /app/dist /usr/share/nginx/html
COPY --from=builder /src/netwatch-server /usr/local/bin/netwatch-server COPY --from=builder /src/netwatch-server /usr/local/bin/netwatch-server
COPY bin/entrypoint.sh /usr/local/bin/entrypoint.sh COPY bin/entrypoint.sh /usr/local/bin/entrypoint.sh
# bin/entrypoint.sh creates DATA_DIR at start and gives it and /data to
# the netwatch user, whatever is mounted there.
ENV DATA_DIR=/data/reports ENV DATA_DIR=/data/reports
RUN mkdir -p /data/reports && chown -R netwatch:netwatch /data
VOLUME /data VOLUME /data
# The default public port; PORT changes it. # The default public port; PORT changes it.
EXPOSE 8080 EXPOSE 8080
# Requests the backend's health check through nginx, on the port from
# PORT, so it fails unless both answer. upaas reads the result 60
# seconds after a deploy and fails the deploy unless it is healthy.
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD wget -q -O /dev/null "http://127.0.0.1:${PORT:-8080}/.well-known/healthcheck"
# The nginx image stops its container with SIGQUIT; the entrypoint # The nginx image stops its container with SIGQUIT; the entrypoint
# acts on TERM and INT. # acts on TERM and INT.
STOPSIGNAL SIGTERM STOPSIGNAL SIGTERM
+58 -15
View File
@@ -36,12 +36,12 @@ The Go backend in `backend/` has its own `script/` directory and shim Makefile
(see [backend/README.md](backend/README.md)). The root scripts cover both (see [backend/README.md](backend/README.md)). The root scripts cover both
halves, so the root `make check` fails if either one is broken. We provide: halves, so the root `make check` fails if either one is broken. We provide:
- `script/bootstrap` — install all dependencies (pinned node via nvm if needed, - `script/bootstrap` — install all dependencies (the pinned node via nvm unless
yarn via corepack, `yarn install --frozen-lockfile`, the pinned Go unless one one new enough for the frontend's dependencies is installed, yarn via
at least as new as `backend/go.mod` asks for is installed, and the Go corepack, `yarn install --frozen-lockfile`, the pinned Go unless one at least
modules), linking what it installs itself into `~/.local/bin`, which has to be as new as `backend/go.mod` asks for is installed, and the Go modules), linking
on `PATH`. It installs no Go linter and not Docker: `make lint` runs the what it installs itself into `~/.local/bin`, which has to be on `PATH`. It
linter in Docker installs no Go linter and not Docker: `make lint` runs the linter in Docker
- `script/setup` — make a fresh clone ready for development: bootstrap plus the - `script/setup` — make a fresh clone ready for development: bootstrap plus the
git pre-commit hook git pre-commit hook
- `script/projectname` — print the project name (used for the Docker image tag) - `script/projectname` — print the project name (used for the Docker image tag)
@@ -52,8 +52,8 @@ halves, so the root `make check` fails if either one is broken. We provide:
- `script/fmt` — format all files (writes): prettier, then gofmt over `backend/` - `script/fmt` — format all files (writes): prettier, then gofmt over `backend/`
- `script/fmt-check` — check formatting (read-only): prettier, then gofmt - `script/fmt-check` — check formatting (read-only): prettier, then gofmt
- `script/check` — run test, lint, and fmt-check - `script/check` — run test, lint, and fmt-check
- `script/frontend-test` — run the production build as the frontend's test (no - `script/frontend-test` — run the unit tests in `test/unit/` with Node's
unit tests yet) built-in test runner, then the production build
- `script/frontend-lint` — run prettier in check mode - `script/frontend-lint` — run prettier in check mode
- `script/frontend-fmt` — format everything prettier understands (writes) - `script/frontend-fmt` — format everything prettier understands (writes)
- `script/frontend-fmt-check` — check prettier formatting (read-only) - `script/frontend-fmt-check` — check prettier formatting (read-only)
@@ -65,7 +65,8 @@ halves, so the root `make check` fails if either one is broken. We provide:
`script/check`: it needs Docker and takes minutes. `script/check`: it needs Docker and takes minutes.
- `script/docker` — build the image from `Dockerfile` without the build cache, - `script/docker` — build the image from `Dockerfile` without the build cache,
tagged `netwatch` via `script/projectname` tagged `netwatch` via `script/projectname`
- `script/cibuild` — CI entrypoint: builds the image - `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
then builds the image as `script/docker` does, without the build cache
- `script/precommit` — run by the git pre-commit hook; runs `script/check` - `script/precommit` — run by the git pre-commit hook; runs `script/check`
- `script/install-precommit` — install the git pre-commit hook - `script/install-precommit` — install the git pre-commit hook
@@ -135,8 +136,11 @@ Local hosts are tracked separately from WAN stats.
### Latency measurement ### Latency measurement
HEAD requests with `mode: 'no-cors'` and `cache: 'no-store'`, timed with HEAD requests with `mode: 'no-cors'` and `cache: 'no-store'`, timed with
`performance.now()`. 1-second timeout; anything over 1000ms is clamped to `performance.now()`. Each check times out after 80% of the refresh interval (24
unreachable. IPv4 only. seconds at 30 seconds) and is then recorded as a timeout, so a round's checks
have all finished before the next round is due. A round due while the last one
is still waiting, which happens only after an interval change or when the
recovery probe starts one, is skipped. IPv4 only.
### Color coding ### Color coding
@@ -184,16 +188,55 @@ container: nginx serves the built frontend and passes `/api/` and
only inside the container, on `127.0.0.1:8081`. The image: only inside the container, on `127.0.0.1:8081`. The image:
- Listens on port 8080 by default (override with `PORT` env var) - Listens on port 8080 by default (override with `PORT` env var)
- Trusts `X-Forwarded-For` from RFC1918 reverse proxies (10/8, 172.16/12, - Takes the client address from `X-Forwarded-For` only on requests from the
192.168/16) reverse proxies named in `TRUSTED_PROXIES`, and by default from none
- Sends access logs to stdout - Sends access logs to stdout
- Caches static assets with immutable headers - Caches static assets with immutable headers
- Sends the security headers `REPO_POLICIES.md` requires on every response, as
`security-headers.conf` sets them, in place of the backend's own
- Stores reports in `DATA_DIR`, `/data/reports` by default, on the `/data` - Stores reports in `DATA_DIR`, `/data/reports` by default, on the `/data`
volume. The backend runs as user `netwatch` (uid 1000), so a directory volume. Before the backend starts, the image creates `DATA_DIR` and gives it
bind-mounted at `/data` must be writable by uid 1000 and `/data` to user `netwatch` (uid 1000), which the backend runs as, so a
host directory bind-mounted at `/data` ends up owned by uid 1000
- Writes buffered reports to disk on `docker stop`, and exits non-zero if nginx - Writes buffered reports to disk on `docker stop`, and exits non-zero if nginx
or the backend exits on its own, so the platform restarts it or the backend exits on its own, so the platform restarts it
## Running under upaas
What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
- **Port:** container port `8080`.
- **Volume:** container path `/data`; the reports are kept in `/data/reports`.
- **Environment variables:** none is required. An empty one counts as unset, and
one set to a value netwatch cannot use stops the container at start, with the
reason in its log.
- `PORT`, default `8080`: the container port, from 1 to 65535. `8081` cannot
be used: the backend listens on it inside the container
- `REPORTS_PER_MINUTE`, default `60`: reports each client address may send a
minute
- `DATA_DIR_MAX_BYTES`, default `1073741824` (1 GiB): the most room the
report files may take
- `CORS_ALLOWED_ORIGINS`, default empty: other origins whose pages may call
the API
- `DEBUG`, default `false`: debug logging
- `DATA_DIR`, default `/data/reports`: leave unset; reports kept outside
`/data` do not survive a redeploy
- `TRUSTED_PROXIES`, default empty: set it to the address the reverse proxy
in front of the container connects from, as an IP address or CIDR; several
are separated by commas. nginx takes the client address from
`X-Forwarded-For` only on a request from one of them, and the rate limit
counts that address. Unset, `X-Forwarded-For` is ignored and every client
behind the proxy shares the proxy's one allowance of `REPORTS_PER_MINUTE`.
Name only addresses nothing but the proxy connects from: any client that
connects from one can write its own `X-Forwarded-For`, and through a port
Docker publishes, every client may connect from the Docker network's
gateway, such as `172.17.0.1`.
- **Health check:** the image's `HEALTHCHECK` requests
`/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless
both nginx and the backend answer. upaas reads the container's health 60
seconds after a deploy and fails the deploy unless it is `healthy`. The
container also stops when either process exits.
## Browser Compatibility ## Browser Compatibility
Requires a modern browser with ES modules, Fetch API, Canvas API, and CSS custom Requires a modern browser with ES modules, Fetch API, Canvas API, and CSS custom
+69 -10
View File
@@ -23,11 +23,76 @@ latest run passes.
# Completed Steps # Completed Steps
- 2026-09-29: each target check times out after 80% of the refresh interval
(issue #78), 24 seconds at 30 seconds, where it was capped at 3 seconds. A
round due while the last one's checks are still waiting is skipped, so rounds
never overlap, and the recovery probe starts no new checks while its last ones
are waiting. The frontend has its first unit tests, run by
`script/frontend-test` with Node's built-in test runner; for them,
`index.html` now links `src/styles.css`, which `src/main.js` used to import
- 2026-09-29: the container sets up its own data directory (issue #75):
`bin/entrypoint.sh`, still as root, creates `DATA_DIR` if missing and gives it
and `/data` to the `netwatch` user with mode 750 before starting the backend
as that user, so an empty host directory owned by root, or one holding files
from another uid, works with no step on the host. It stops the start instead
when a symbolic link is on the path to `DATA_DIR`, since root would change
whatever the link points to. The `README.md` first-run step that created and
chowned the host directory is gone, and the image no longer sets that
ownership at build time
- 2026-09-29: CI can no longer pass on checks that did not run (issue #37):
`script/cibuild` is now the org model, byte for byte. It runs
`script/bootstrap` and `script/check`, then builds the image with `--no-cache`
and the version from `git describe` as the `VERSION` build argument, where it
used to be a plain `docker build .` whose check steps could come from the
build cache. The workflow puts `~/.local/bin`, where bootstrap links what it
installs, on the step's `PATH`, and bootstrap now installs its pinned node
when the installed one is older than the frontend's dependencies need
- 2026-09-29: `backend/.golangci.yml` re-vendored from `sneak/prompts` (issue
#41): `gomodguard`, deprecated in golangci-lint v2.12.0, is disabled and its
successor `gomodguard_v2` enabled with the org block list, so lint runs print
no deprecation warning. The new file also turns `depguard` on with its
`test-support` rule, which keeps `net/http/httptest` out of non-test code;
netwatch adds no entries of its own to that rule. `backend/script/lint` checks
the new sha256
- 2026-09-29: nginx sends the security headers `REPO_POLICIES.md` requires on
every response (issue #18), including errors, `/assets/` and what it passes on
from the backend, whose own copies it drops so each header goes out once. They
live in `security-headers.conf`, which `nginx.conf` includes. The content
security policy allows no inline script or style, so the status dot's grey in
`src/main.js` is now a class; `connect-src` is `*` because probed hosts
redirect to others, and the browser checks each redirect against it
- 2026-09-29: the request log is bounded (issue #60): the method, URL, protocol,
`User-Agent`, `Referer`, request ID (which chi takes from the client's
`X-Request-Id` header) and client address it writes are each cut to 128 bytes,
the bound the report handler already used, so one request can no longer put
about 1 MiB per field into a log line. That bound and its helper now live in
the `logger` package, shared by both
- 2026-09-29: nginx takes the client address from `X-Forwarded-For` only on
requests from the reverse proxies named in the container's `TRUSTED_PROXIES`
(issue #64), and by default from none, where it trusted every RFC1918 address
before, so a client could write a new address on each request and escape the
rate limit. `bin/entrypoint.sh` writes one `set_real_ip_from` line per entry
into `/etc/nginx/trusted-proxies.conf`, which `nginx.conf` includes, refusing
an entry that is not an IP address or CIDR, as `netwatch-server check-cidr`
finds; it starts the backend with `TRUSTED_PROXIES=127.0.0.1/32`, since nginx
is its only client
- 2026-09-29: report file names can no longer collide (issue #61): each is - 2026-09-29: report file names can no longer collide (issue #61): each is
`reports-<timestamp>-<number>.jsonl.zst`, where the number counts the files `reports-<timestamp>-<number>.jsonl.zst`, where the number goes up by one for
written since the server started, so two flushes in the same millisecond, such each file the server starts to write, so two flushes in the same millisecond,
as a flush for size and the final flush at shutdown, each get a file of their such as a flush for size and the final flush at shutdown, each get a file of
own instead of the second one failing their own instead of the second one failing. A failed write uses up its
number, leaving a gap if the file could not be created and otherwise a file
under that number that may be incomplete.
- 2026-09-29: ready to run under upaas (issue #59): the image has a
`HEALTHCHECK` that requests `/.well-known/healthcheck` through nginx on the
port from `PORT`. The backend no longer reads a bad `PORT` as 0 or a bad
`DEBUG` as false: those, and a `BIND_ADDRESS` that is not an IP address, stop
it from starting with an error naming the variable, as the limits,
`CORS_ALLOWED_ORIGINS` and, now by name, `TRUSTED_PROXIES` already did.
`bin/entrypoint.sh` also refuses a `PORT` outside 1 to 65535, and `8081`,
where the backend listens inside the container, naming `PORT`. `README.md` has
a "Running under upaas" section, whose first-run steps create the host
directory for `/data` owned by uid 1000; the image does not change its owner
- 2026-09-29: nginx listens on `PORT` (issue #26), 8080 when unset or empty: the - 2026-09-29: nginx listens on `PORT` (issue #26), 8080 when unset or empty: the
nginx image renders `nginx.conf` as a template at container start, filling in nginx image renders `nginx.conf` as a template at container start, filling in
`PORT` and no other variable. `bin/entrypoint.sh` refuses to start when `PORT` `PORT` and no other variable. `bin/entrypoint.sh` refuses to start when `PORT`
@@ -157,9 +222,3 @@ latest run passes.
(main always green policy) (main always green policy)
- Decide what to do with untracked resume.sh: commit it, gitignore it, or delete - Decide what to do with untracked resume.sh: commit it, gitignore it, or delete
it it
- Upstream fix needed in `sneak/prompts`: the org-standard `.golangci.yml`
enables `gomodguard`, which golangci-lint v2.12.2 reports as deprecated since
v2.12.0 and replaced by `gomodguard_v2`, so every backend lint run prints a
deprecation warning. The file is standardized and must never be edited in this
repo, so nothing can be done here beyond tracking it — tracked at
<https://git.eeqj.de/sneak/netwatch/issues/41>
+66 -2
View File
@@ -10,14 +10,20 @@ run:
linters: linters:
default: all default: all
enable:
# Successor to the deprecated gomodguard. Named explicitly, rather than
# left to `default: all`, because it carries the module policy below.
- gomodguard_v2
disable: disable:
# Genuinely incompatible with project patterns # Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields - exhaustruct # Requires all struct fields
- depguard # Dependency allow/block lists
- godot # Requires comments to end with periods - godot # Requires comments to end with periods
- wsl # Deprecated, replaced by wsl_v5
- wrapcheck # Too verbose for internal packages - wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go - varnamelen # Short names like db, id are idiomatic Go
# Deprecated: the warning is attached to the old name, so it is
# silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
settings: settings:
lll: lll:
line-length: 88 line-length: 88
@@ -28,6 +34,64 @@ linters:
max-complexity: 15 max-complexity: 15
dupl: dupl:
threshold: 100 threshold: 100
depguard:
# Test-support code must not be compiled into the shipped binary. A
# test-support package exists to hand a test privileges the program
# itself must never have, so a file that is not a test must not import
# one. Test files, and the files inside a package whose directory name
# ends in `test`, are where that code belongs, and are exempt.
#
# The deny list below is the one part of this file a repository is
# expected to extend, and the only part it may. depguard matches an
# import path against a list of prefixes, so it cannot be told "any path
# whose last segment ends in test"; a repository's own test-support
# packages have to be named here one at a time, by full import path,
# under a module path that differs from repository to repository. Add
# them; change nothing else.
rules:
test-support:
list-mode: lax
files:
- "$all"
- "!$test"
- "!**/*test/**"
deny:
- pkg: net/http/httptest
desc: >-
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
# Only decisions already recorded in the Go package defaults are
# listed here. Every entry matches the module path exactly.
gomodguard_v2:
blocked:
- module: github.com/rs/zerolog
recommendations:
- log/slog
reason: "Structured logging is stdlib log/slog."
# One entry per pre-fork module path, because the later releases
# are separate paths. A prefix match would be shorter but would
# also reach github.com/go-redis/redismock, the test double for
# the successor these entries recommend.
- module: github.com/go-redis/redis
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v7
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v8
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/sergi/go-diff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "No unified diff output; use go-udiff."
- module: github.com/hexops/gotextdiff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "Unmaintained fork; use go-udiff."
issues: issues:
max-issues-per-linter: 0 max-issues-per-linter: 0
+30 -9
View File
@@ -87,9 +87,14 @@ Internal packages in `internal/` follow standard Go project layout:
| `CORS_ALLOWED_ORIGINS` | empty | Comma-separated origins whose pages may call the API; see [CORS](#cors) | | `CORS_ALLOWED_ORIGINS` | empty | Comma-separated origins whose pages may call the API; see [CORS](#cors) |
`TRUSTED_PROXIES` defaults to `127.0.0.1/32,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`. `TRUSTED_PROXIES` defaults to `127.0.0.1/32,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`.
The loopback entries cover the reverse proxy that shares the container; the The loopback entries cover a reverse proxy on the same host. A request whose
RFC1918 ranges match `nginx.conf`. A request whose direct peer is outside this direct peer is outside this set has its forwarded headers ignored, and the
set has its forwarded headers ignored, and the direct peer is logged instead. direct peer is logged and rate-limited instead. The container image does not use
this default; see [Container image](#container-image).
A variable set to a value the server cannot use, such as `PORT=abc`,
`DEBUG=maybe` or a `BIND_ADDRESS` that is not an IP address, stops it from
starting, with an error naming the variable. An empty variable counts as unset.
### Container image ### Container image
@@ -97,17 +102,33 @@ The root `Dockerfile` builds one image in which nginx listens on the public port
8080, serves the frontend, and proxies `/api/` and `/.well-known/healthcheck` to 8080, serves the frontend, and proxies `/api/` and `/.well-known/healthcheck` to
this server. The image's entrypoint, `bin/entrypoint.sh`, starts the server as this server. The image's entrypoint, `bin/entrypoint.sh`, starts the server as
user `netwatch` (uid 1000) with `BIND_ADDRESS=127.0.0.1` and `PORT=8081`, so user `netwatch` (uid 1000) with `BIND_ADDRESS=127.0.0.1` and `PORT=8081`, so
only nginx reaches it. `DATA_DIR` is `/data/reports`, on the `/data` volume, only nginx reaches it, and with `TRUSTED_PROXIES=127.0.0.1/32`, so it takes the
which `netwatch` owns. client address nginx passes on and no other. `DATA_DIR` is `/data/reports`, on
the `/data` volume; the entrypoint creates it and gives it and `/data` to
`netwatch` before starting the server. nginx replaces the security headers
this server sets with those in the root `security-headers.conf`, so those are
what clients of the image see.
The container's own `TRUSTED_PROXIES` goes to nginx instead: IP addresses or
CIDRs, separated by commas, of the reverse proxies in front of the container.
nginx takes the client address from `X-Forwarded-For` only on a request from one
of them. Unset or empty, nginx trusts no proxy, and the client address is the
one each request comes from, so every client behind a proxy shares one rate
limit. An entry that is not an IP address or CIDR, such as a hostname or
`1.2.3`, stops the container at start with an error naming `TRUSTED_PROXIES`:
the entrypoint checks each entry with `netwatch-server check-cidr`, which parses
it as this server parses its own `TRUSTED_PROXIES`.
### Report storage ### Report storage
Reports are written as `reports-<timestamp>-<number>.jsonl.zst` files in Reports are written as `reports-<timestamp>-<number>.jsonl.zst` files in
`DATA_DIR`. The timestamp is in UTC to the millisecond, so the names sort by `DATA_DIR`. The timestamp is in UTC to the millisecond, so the names sort by
time; the number counts the files the server has written since it started, so time. The number starts at 1 when the server starts and goes up by one for each
two files written in the same millisecond still get different names. Each file file the server starts to write, so two files written in the same millisecond
contains one JSON object per line, compressed with zstd. Files are created with still get different names. A failed write uses up its number, leaving a gap in
`O_EXCL` to prevent overwrites. the numbers if the file could not be created and otherwise a file under that
number that may be incomplete. Each file contains one JSON object per line,
compressed with zstd. Files are created with `O_EXCL` to prevent overwrites.
### Report limits ### Report limits
+16
View File
@@ -2,6 +2,9 @@
package main package main
import ( import (
"fmt"
"os"
"sneak.berlin/go/netwatch/internal/config" "sneak.berlin/go/netwatch/internal/config"
"sneak.berlin/go/netwatch/internal/globals" "sneak.berlin/go/netwatch/internal/globals"
"sneak.berlin/go/netwatch/internal/handlers" "sneak.berlin/go/netwatch/internal/handlers"
@@ -22,6 +25,19 @@ var (
) )
func main() { func main() {
// "netwatch-server check-cidr CIDR" exits 1, with the error, if
// this server would refuse CIDR in its TRUSTED_PROXIES.
// bin/entrypoint.sh runs it on each entry it gives nginx.
if len(os.Args) == 3 && os.Args[1] == "check-cidr" {
_, err := middleware.ParseTrustedProxies(os.Args[2:])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
return
}
globals.Appname = Appname globals.Appname = Appname
globals.Version = Version globals.Version = Version
globals.Buildarch = Buildarch globals.Buildarch = Buildarch
+56 -20
View File
@@ -6,7 +6,10 @@ import (
"errors" "errors"
"fmt" "fmt"
"log/slog" "log/slog"
"math"
"net/netip"
"net/url" "net/url"
"strconv"
"strings" "strings"
"sneak.berlin/go/netwatch/internal/globals" "sneak.berlin/go/netwatch/internal/globals"
@@ -18,10 +21,11 @@ import (
) )
// defaultTrustedProxies lists the networks whose forwarded // defaultTrustedProxies lists the networks whose forwarded
// headers are honoured by default. It covers the RFC1918 // headers are honoured by default: IPv4 and IPv6 loopback,
// ranges (to match nginx.conf) plus IPv4 and IPv6 loopback, // for a reverse proxy on the same host, and the RFC1918
// because the reverse proxy shares the container and reaches // ranges. The container image does not use it:
// the backend over loopback. // bin/entrypoint.sh gives the server 127.0.0.1/32, since
// nginx is its only client there.
const defaultTrustedProxies = "127.0.0.1/32,::1/128," + const defaultTrustedProxies = "127.0.0.1/32,::1/128," +
"10.0.0.0/8,172.16.0.0/12,192.168.0.0/16" "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
@@ -37,6 +41,9 @@ var (
errNotOrigin = errors.New( errNotOrigin = errors.New(
"must be an origin, scheme://host with an optional port", "must be an origin, scheme://host with an optional port",
) )
errNotPort = errors.New("must be a port number, 1 to 65535")
errNotBool = errors.New("must be true or false")
errNotIP = errors.New("must be an IP address, or empty")
) )
// Params defines the dependencies for Config. // Params defines the dependencies for Config.
@@ -65,7 +72,8 @@ type Config struct {
} }
// New loads configuration from env, .env files, and config // New loads configuration from env, .env files, and config
// files, returning a fully resolved Config. // files, returning a fully resolved Config. It fails, with an error
// naming the setting, on a value the server cannot use.
func New( func New(
_ fx.Lifecycle, _ fx.Lifecycle,
params Params, params Params,
@@ -103,15 +111,29 @@ func New(
} }
} }
// Read with strconv: viper's GetInt and GetBool would read a value
// they cannot parse as 0 or false instead of failing.
port, err := strconv.Atoi(viper.GetString("PORT"))
if err != nil || port < 1 || port > math.MaxUint16 {
return nil, fmt.Errorf("PORT %q: %w",
viper.GetString("PORT"), errNotPort)
}
debug, err := strconv.ParseBool(viper.GetString("DEBUG"))
if err != nil {
return nil, fmt.Errorf("DEBUG %q: %w",
viper.GetString("DEBUG"), errNotBool)
}
s := &Config{ s := &Config{
BindAddress: viper.GetString("BIND_ADDRESS"), BindAddress: viper.GetString("BIND_ADDRESS"),
CORSAllowedOrigins: splitList(viper.GetString("CORS_ALLOWED_ORIGINS")), CORSAllowedOrigins: splitList(viper.GetString("CORS_ALLOWED_ORIGINS")),
DataDir: viper.GetString("DATA_DIR"), DataDir: viper.GetString("DATA_DIR"),
DataDirMaxBytes: viper.GetInt64("DATA_DIR_MAX_BYTES"), DataDirMaxBytes: viper.GetInt64("DATA_DIR_MAX_BYTES"),
Debug: viper.GetBool("DEBUG"), Debug: debug,
MetricsPassword: viper.GetString("METRICS_PASSWORD"), MetricsPassword: viper.GetString("METRICS_PASSWORD"),
MetricsUsername: viper.GetString("METRICS_USERNAME"), MetricsUsername: viper.GetString("METRICS_USERNAME"),
Port: viper.GetInt("PORT"), Port: port,
ReportsPerMinute: viper.GetInt("REPORTS_PER_MINUTE"), ReportsPerMinute: viper.GetInt("REPORTS_PER_MINUTE"),
SentryDSN: viper.GetString("SENTRY_DSN"), SentryDSN: viper.GetString("SENTRY_DSN"),
TrustedProxies: splitList(viper.GetString("TRUSTED_PROXIES")), TrustedProxies: splitList(viper.GetString("TRUSTED_PROXIES")),
@@ -119,19 +141,7 @@ func New(
params: &params, params: &params,
} }
// viper reads a value that is not a number as 0, so this also err = s.check()
// catches a mistyped setting.
if s.ReportsPerMinute <= 0 {
return nil, fmt.Errorf("REPORTS_PER_MINUTE %q: %w",
viper.GetString("REPORTS_PER_MINUTE"), errNotPositive)
}
if s.DataDirMaxBytes <= 0 {
return nil, fmt.Errorf("DATA_DIR_MAX_BYTES %q: %w",
viper.GetString("DATA_DIR_MAX_BYTES"), errNotPositive)
}
err = checkOrigins(s.CORSAllowedOrigins)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -144,6 +154,32 @@ func New(
return s, nil return s, nil
} }
// check fails with an error naming the first setting here whose value
// the server cannot use. New checks PORT and DEBUG as it reads them,
// and the middleware checks TRUSTED_PROXIES as it parses it.
func (s *Config) check() error {
// viper reads a value that is not a number as 0, so this also
// catches a mistyped setting.
if s.ReportsPerMinute <= 0 {
return fmt.Errorf("REPORTS_PER_MINUTE %q: %w",
viper.GetString("REPORTS_PER_MINUTE"), errNotPositive)
}
if s.DataDirMaxBytes <= 0 {
return fmt.Errorf("DATA_DIR_MAX_BYTES %q: %w",
viper.GetString("DATA_DIR_MAX_BYTES"), errNotPositive)
}
if s.BindAddress != "" {
_, err := netip.ParseAddr(s.BindAddress)
if err != nil {
return fmt.Errorf("BIND_ADDRESS %q: %w", s.BindAddress, errNotIP)
}
}
return checkOrigins(s.CORSAllowedOrigins)
}
// checkOrigins fails on the first CORS_ALLOWED_ORIGINS entry that is // checkOrigins fails on the first CORS_ALLOWED_ORIGINS entry that is
// not a plain origin, scheme://host with an optional port, as browsers // not a plain origin, scheme://host with an optional port, as browsers
// send it; anything more, such as a trailing "/", would match no page. // send it; anything more, such as a trailing "/", would match no page.
+57
View File
@@ -29,6 +29,63 @@ func requireConfigError(t *testing.T, setting string) {
} }
} }
// TestSettingsLoadAsGiven: valid values pass the checks and are used
// as given. bin/entrypoint.sh starts the server with these
// BIND_ADDRESS and PORT values.
func TestSettingsLoadAsGiven(t *testing.T) {
t.Setenv("BIND_ADDRESS", "127.0.0.1")
t.Setenv("PORT", "8081")
t.Setenv("DEBUG", "true")
var cfg *config.Config
app := fx.New(
fx.NopLogger,
fx.Provide(globals.New, logger.New, config.New),
fx.Populate(&cfg),
)
err := app.Err()
if err != nil {
t.Fatalf("config error = %v", err)
}
if cfg.BindAddress != "127.0.0.1" || cfg.Port != 8081 || !cfg.Debug {
t.Fatalf("BindAddress, Port, Debug = %q, %d, %t; "+
"want \"127.0.0.1\", 8081, true",
cfg.BindAddress, cfg.Port, cfg.Debug)
}
}
// TestPortMustBeAPortNumber: viper reads a value that is not a number
// as 0, on which the server would listen on a random port.
func TestPortMustBeAPortNumber(t *testing.T) {
for _, value := range []string{"abc", "0", "65536", "8080.5"} {
t.Run(value, func(t *testing.T) {
t.Setenv("PORT", value)
requireConfigError(t, "PORT")
})
}
}
// TestDebugMustBeTrueOrFalse: viper reads any other value, such as
// "yes", as false.
func TestDebugMustBeTrueOrFalse(t *testing.T) {
t.Setenv("DEBUG", "yes")
requireConfigError(t, "DEBUG")
}
// TestBindAddressMustBeAnIPAddress: a host name would be looked up
// only once the server starts listening, and a mistyped one would stop
// it then with an error that does not name the setting.
func TestBindAddressMustBeAnIPAddress(t *testing.T) {
t.Setenv("BIND_ADDRESS", "localhost")
requireConfigError(t, "BIND_ADDRESS")
}
// TestReportsPerMinuteMustBePositive: unchecked, zero would panic // TestReportsPerMinuteMustBePositive: unchecked, zero would panic
// when the routes are built, and a negative rate would lift the // when the routes are built, and a negative rate would lift the
// limit. // limit.
-3
View File
@@ -2,9 +2,6 @@ package handlers
import "log/slog" import "log/slog"
// MaxLoggedFieldBytes exposes the log bound to the external tests.
const MaxLoggedFieldBytes = maxLoggedFieldBytes
// NewForTest builds a Handlers around a report sink and logger, // NewForTest builds a Handlers around a report sink and logger,
// bypassing the fx graph so handler behaviour (including the // bypassing the fx graph so handler behaviour (including the
// storage failure path) is exercisable in unit tests. // storage failure path) is exercisable in unit tests.
+4 -18
View File
@@ -5,14 +5,10 @@ import (
"errors" "errors"
"net/http" "net/http"
"sneak.berlin/go/netwatch/internal/logger"
"sneak.berlin/go/netwatch/internal/reportbuf" "sneak.berlin/go/netwatch/internal/reportbuf"
) )
// maxLoggedFieldBytes bounds untrusted text (string fields,
// decode error text) before it is logged, so a caller cannot
// inflate log volume with an oversized value.
const maxLoggedFieldBytes = 128
type reportSample struct { type reportSample struct {
T int64 `json:"t"` T int64 `json:"t"`
Latency *int `json:"latency"` Latency *int `json:"latency"`
@@ -83,7 +79,7 @@ func (s *Handlers) decodeErrorStatus(err error) int {
// The decoder's error text can quote request bytes (a whole // The decoder's error text can quote request bytes (a whole
// oversized number, for example), so it is bounded too. // oversized number, for example), so it is bounded too.
s.log.Error("failed to decode report", s.log.Error("failed to decode report",
"error", boundedForLog(err.Error()), "error", logger.BoundedForLog(err.Error()),
) )
return http.StatusBadRequest return http.StatusBadRequest
@@ -115,20 +111,10 @@ func (s *Handlers) logReportReceived(rpt report) {
} }
s.log.Info("report received", s.log.Info("report received",
"client_id", boundedForLog(rpt.ClientID), "client_id", logger.BoundedForLog(rpt.ClientID),
"timestamp", boundedForLog(rpt.Timestamp), "timestamp", logger.BoundedForLog(rpt.Timestamp),
"host_count", len(rpt.Hosts), "host_count", len(rpt.Hosts),
"total_samples", totalSamples, "total_samples", totalSamples,
"geo_bytes", len(rpt.Geo), "geo_bytes", len(rpt.Geo),
) )
} }
// boundedForLog truncates an untrusted string to a fixed byte
// bound so an attacker-controlled field cannot dominate the log.
func boundedForLog(s string) string {
if len(s) > maxLoggedFieldBytes {
return s[:maxLoggedFieldBytes]
}
return s
}
+6 -5
View File
@@ -12,6 +12,7 @@ import (
"testing" "testing"
"sneak.berlin/go/netwatch/internal/handlers" "sneak.berlin/go/netwatch/internal/handlers"
"sneak.berlin/go/netwatch/internal/logger"
"sneak.berlin/go/netwatch/internal/middleware" "sneak.berlin/go/netwatch/internal/middleware"
"sneak.berlin/go/netwatch/internal/reportbuf" "sneak.berlin/go/netwatch/internal/reportbuf"
) )
@@ -174,7 +175,7 @@ func TestHandleReportDoesNotLogRawGeo(t *testing.T) {
func TestHandleReportLogsClientIDCutToBound(t *testing.T) { func TestHandleReportLogsClientIDCutToBound(t *testing.T) {
t.Parallel() t.Parallel()
long := strings.Repeat("c", 2*handlers.MaxLoggedFieldBytes) long := strings.Repeat("c", 2*logger.MaxLoggedFieldBytes)
var logbuf bytes.Buffer var logbuf bytes.Buffer
@@ -197,16 +198,16 @@ func TestHandleReportLogsClientIDCutToBound(t *testing.T) {
t.Fatalf("log line not JSON: %v (%q)", err, logbuf.String()) t.Fatalf("log line not JSON: %v (%q)", err, logbuf.String())
} }
want := long[:handlers.MaxLoggedFieldBytes] want := long[:logger.MaxLoggedFieldBytes]
if logged["client_id"] != want { if logged["client_id"] != want {
t.Fatalf("logged client_id not cut to %d bytes: %q", t.Fatalf("logged client_id not cut to %d bytes: %q",
handlers.MaxLoggedFieldBytes, logged["client_id"]) logger.MaxLoggedFieldBytes, logged["client_id"])
} }
if logged["timestamp"] != want { if logged["timestamp"] != want {
t.Fatalf("logged timestamp not cut to %d bytes: %q", t.Fatalf("logged timestamp not cut to %d bytes: %q",
handlers.MaxLoggedFieldBytes, logged["timestamp"]) logger.MaxLoggedFieldBytes, logged["timestamp"])
} }
} }
@@ -215,7 +216,7 @@ func TestHandleReportDecodeErrorLogIsBounded(t *testing.T) {
// A number too large for its int64 field makes the decoder's // A number too large for its int64 field makes the decoder's
// error text quote the whole number. // error text quote the whole number.
huge := strings.Repeat("9", 2*handlers.MaxLoggedFieldBytes) huge := strings.Repeat("9", 2*logger.MaxLoggedFieldBytes)
var logbuf bytes.Buffer var logbuf bytes.Buffer
+15
View File
@@ -11,6 +11,21 @@ import (
"go.uber.org/fx" "go.uber.org/fx"
) )
// MaxLoggedFieldBytes bounds untrusted text (request fields,
// header values, decode error text) before it is logged, so a
// caller cannot inflate log volume with an oversized value.
const MaxLoggedFieldBytes = 128
// BoundedForLog truncates an untrusted string to a fixed byte
// bound so an attacker-controlled field cannot dominate the log.
func BoundedForLog(s string) string {
if len(s) > MaxLoggedFieldBytes {
return s[:MaxLoggedFieldBytes]
}
return s
}
// Params defines the dependencies for Logger. // Params defines the dependencies for Logger.
type Params struct { type Params struct {
fx.In fx.In
@@ -29,7 +29,3 @@ func ClientIP(
) string { ) string {
return clientIP(remoteAddr, header, trusted) return clientIP(remoteAddr, header, trusted)
} }
func ParseTrustedProxies(cidrs []string) ([]netip.Prefix, error) {
return parseTrustedProxies(cidrs)
}
+19 -16
View File
@@ -64,7 +64,7 @@ func New(
_ fx.Lifecycle, _ fx.Lifecycle,
params Params, params Params,
) (*Middleware, error) { ) (*Middleware, error) {
trusted, err := parseTrustedProxies(params.Config.TrustedProxies) trusted, err := ParseTrustedProxies(params.Config.TrustedProxies)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -77,16 +77,18 @@ func New(
return s, nil return s, nil
} }
// parseTrustedProxies converts CIDR strings into prefixes, // ParseTrustedProxies converts the TRUSTED_PROXIES entries into
// failing fast on any malformed entry. // prefixes, failing fast on any malformed entry. Each entry must be
func parseTrustedProxies(cidrs []string) ([]netip.Prefix, error) { // a CIDR; a lone address is refused. "netwatch-server check-cidr"
// runs it too.
func ParseTrustedProxies(cidrs []string) ([]netip.Prefix, error) {
prefixes := make([]netip.Prefix, 0, len(cidrs)) prefixes := make([]netip.Prefix, 0, len(cidrs))
for _, cidr := range cidrs { for _, cidr := range cidrs {
prefix, err := netip.ParsePrefix(cidr) prefix, err := netip.ParsePrefix(cidr)
if err != nil { if err != nil {
return nil, fmt.Errorf( return nil, fmt.Errorf(
"trusted proxy %q: %w", cidr, err, "TRUSTED_PROXIES %q: %w", cidr, err,
) )
} }
@@ -187,7 +189,10 @@ func addrInAny(s string, trusted []netip.Prefix) bool {
} }
// Logging returns middleware that logs each request with // Logging returns middleware that logs each request with
// timing, status code, and client information. // timing, status code, and client information. Every string
// taken from the request is cut to logger.MaxLoggedFieldBytes,
// including the request ID, which chi takes from the client's
// X-Request-Id header when one is sent.
func (s *Middleware) Logging() func(http.Handler) http.Handler { func (s *Middleware) Logging() func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler { return func(next http.Handler) http.Handler {
return http.HandlerFunc( return http.HandlerFunc(
@@ -200,21 +205,19 @@ func (s *Middleware) Logging() func(http.Handler) http.Handler {
latency := time.Since(start) latency := time.Since(start)
s.log.InfoContext(ctx, "request", s.log.InfoContext(ctx, "request",
"request_start", start, "request_start", start,
"method", r.Method, "method", logger.BoundedForLog(r.Method),
"url", r.URL.String(), "url", logger.BoundedForLog(r.URL.String()),
"useragent", r.UserAgent(), "useragent", logger.BoundedForLog(r.UserAgent()),
"request_id", "request_id",
ctx.Value( logger.BoundedForLog(middleware.GetReqID(ctx)),
middleware.RequestIDKey, "referer", logger.BoundedForLog(r.Referer()),
), "proto", logger.BoundedForLog(r.Proto),
"referer", r.Referer(),
"proto", r.Proto,
"remote_ip", "remote_ip",
clientIP( logger.BoundedForLog(clientIP(
r.RemoteAddr, r.RemoteAddr,
r.Header, r.Header,
s.trustedProxies, s.trustedProxies,
), )),
"status", lrw.statusCode, "status", lrw.statusCode,
"latency_ms", "latency_ms",
latency.Milliseconds(), latency.Milliseconds(),
+69 -3
View File
@@ -13,7 +13,10 @@ import (
"testing/synctest" "testing/synctest"
"time" "time"
"sneak.berlin/go/netwatch/internal/logger"
"sneak.berlin/go/netwatch/internal/middleware" "sneak.berlin/go/netwatch/internal/middleware"
chimiddleware "github.com/go-chi/chi/v5/middleware"
) )
const ( const (
@@ -36,15 +39,32 @@ func mustPrefixes(t *testing.T, cidrs ...string) []netip.Prefix {
return prefixes return prefixes
} }
// TestParseTrustedProxiesRejectsMalformed includes entries nginx would
// read as another address or look up as a hostname, in the CIDR form
// bin/entrypoint.sh gives "netwatch-server check-cidr".
func TestParseTrustedProxiesRejectsMalformed(t *testing.T) { func TestParseTrustedProxiesRejectsMalformed(t *testing.T) {
t.Parallel() t.Parallel()
_, err := middleware.ParseTrustedProxies([]string{"not-a-cidr"}) for _, cidr := range []string{
if err == nil { "not-a-cidr", "10.0.0.1", "1.2.3/32", "172.30/32", "10/32",
t.Fatal("expected error for malformed CIDR, got nil") "cafe/32", "999.1.1.1/32", "10.0.0.0/33", "::1/129",
"fe80::1%eth0/128",
} {
_, err := middleware.ParseTrustedProxies([]string{cidr})
if err == nil || !strings.Contains(err.Error(), "TRUSTED_PROXIES") {
t.Errorf("%q: error = %v, want one naming TRUSTED_PROXIES",
cidr, err)
}
} }
} }
func TestParseTrustedProxiesAcceptsCIDRs(t *testing.T) {
t.Parallel()
mustPrefixes(t, "172.17.0.1/32", "10.0.0.0/8", "2001:db8::1/128",
"2001:db8::/32", "::ffff:192.0.2.1/128")
}
type clientIPCase struct { type clientIPCase struct {
name string name string
remoteAddr string remoteAddr string
@@ -303,6 +323,52 @@ func TestRecovererRepanicsOnAbortHandler(t *testing.T) {
} }
} }
// TestLoggingCutsRequestStringsToBound sends an over-long URL and
// over-long header values, and checks the request log writes each
// one cut to logger.MaxLoggedFieldBytes.
func TestLoggingCutsRequestStringsToBound(t *testing.T) {
t.Parallel()
long := strings.Repeat("a", 2*logger.MaxLoggedFieldBytes)
var logbuf bytes.Buffer
mw := middleware.NewWithLogger(
slog.New(slog.NewJSONHandler(&logbuf, nil)),
)
handler := chimiddleware.RequestID(mw.Logging()(okHandler()))
req := httptest.NewRequestWithContext(t.Context(),
http.MethodGet, "/"+long, http.NoBody)
req.Header.Set("User-Agent", long)
req.Header.Set("Referer", long)
req.Header.Set("X-Request-Id", long)
handler.ServeHTTP(httptest.NewRecorder(), req)
var logged map[string]any
err := json.Unmarshal(logbuf.Bytes(), &logged)
if err != nil {
t.Fatalf("log line not JSON: %v (%q)", err, logbuf.String())
}
want := map[string]string{
"url": ("/" + long)[:logger.MaxLoggedFieldBytes],
"useragent": long[:logger.MaxLoggedFieldBytes],
"referer": long[:logger.MaxLoggedFieldBytes],
"request_id": long[:logger.MaxLoggedFieldBytes],
}
for field, value := range want {
if logged[field] != value {
t.Errorf("logged %s = %q, want it cut to %d bytes",
field, logged[field], logger.MaxLoggedFieldBytes)
}
}
}
// okHandler stands in for the route a middleware guards. // okHandler stands in for the route a middleware guards.
func okHandler() http.Handler { func okHandler() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+1 -1
View File
@@ -14,7 +14,7 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
GOLANGCI_CONFIG_SHA256="021cc83f4e6fc7c31b95b34b846723dfcf20b66b7baeea1dc40406e643346bcb" GOLANGCI_CONFIG_SHA256="a79b63a254602a5318db5d0e9a06bc71b84bf0c1d896305229d8bfed1d1b1776"
main() { main() {
cd "$ROOT" cd "$ROOT"
+72 -6
View File
@@ -10,8 +10,9 @@ set -u
# PORT is the public port nginx listens on, 8080 when unset or empty. # PORT is the public port nginx listens on, 8080 when unset or empty.
# nginx would take a value such as localhost or unix:/tmp/x.sock as an # nginx would take a value such as localhost or unix:/tmp/x.sock as an
# address and start anyway, so anything but digits stops the container # address and start anyway, and reports a bad port without naming
# here, before either process starts. # PORT, so a value that is not a usable port stops the container here,
# before either process starts.
export PORT="${PORT:-8080}" export PORT="${PORT:-8080}"
case "$PORT" in case "$PORT" in
*[!0-9]*) *[!0-9]*)
@@ -19,6 +20,69 @@ case "$PORT" in
exit 1 exit 1
;; ;;
esac esac
# The length is checked first because, for a number too big for it,
# the shell's test prints an error and is false, so the range checks
# alone would let it through.
if [ "${#PORT}" -gt 5 ] || [ "$PORT" -lt 1 ] || [ "$PORT" -gt 65535 ]; then
echo "entrypoint: PORT must be from 1 to 65535, not '$PORT'" >&2
exit 1
fi
if [ "$PORT" -eq 8081 ]; then
echo "entrypoint: PORT cannot be 8081, netwatch-server listens there" >&2
exit 1
fi
# TRUSTED_PROXIES names the reverse proxies in front of the container,
# as IP addresses or CIDRs separated by commas. nginx takes the client
# address from X-Forwarded-For only on a request from one of them, so
# unset or empty, it trusts no one. nginx.conf includes the file written
# here, one set_real_ip_from line per entry.
#
# nginx looks up an entry it cannot read as an address as a hostname,
# and trusts what it finds (1.2.3 is found as 1.2.0.3). So each entry
# is made a CIDR, a lone address getting /128 if it is IPv6 and /32 if
# not, and netwatch-server checks it with the parsing it gives its own
# TRUSTED_PROXIES. Its error, naming the CIDR, is dropped for the one
# below, naming the entry as written. set -f keeps a * in an entry from
# becoming a list of file names.
TRUSTED_PROXIES="${TRUSTED_PROXIES:-}"
set -f
for proxy in $(printf '%s' "$TRUSTED_PROXIES" | tr ',' ' '); do
case "$proxy" in
*/*) cidr="$proxy" ;;
*:*) cidr="$proxy/128" ;;
*) cidr="$proxy/32" ;;
esac
if ! netwatch-server check-cidr "$cidr" 2> /dev/null; then
echo "entrypoint: TRUSTED_PROXIES must be IP addresses or CIDRs" \
"separated by commas; '$proxy' is neither" >&2
exit 1
fi
echo "set_real_ip_from $cidr;"
done > /etc/nginx/trusted-proxies.conf
# netwatch-server keeps its report files in DATA_DIR, on the /data
# volume, which may be a host directory owned by root or by another
# uid. Both are given to the netwatch user here, with the mode the
# server gives a directory it creates, so the host directory needs no
# preparing.
#
# chown and chmod, run as root, change whatever a symbolic link on the
# path points to, anywhere in the container, and the netwatch user can
# put one in /data. So the start stops unless readlink -f, which
# follows every link on a path, gives /data and DATA_DIR back as they
# are. It also writes a path in full, so a DATA_DIR with '.', '..' or
# an extra '/' in it is refused too.
export DATA_DIR="${DATA_DIR:-/data/reports}"
mkdir -p "$DATA_DIR" || exit 1
if [ "$(readlink -f /data)" != /data ] ||
[ "$(readlink -f "$DATA_DIR")" != "$DATA_DIR" ]; then
echo "entrypoint: DATA_DIR must be a full path with no '.', '..'," \
"extra '/' or symbolic link on it or on /data, not '$DATA_DIR'" >&2
exit 1
fi
chown -R netwatch:netwatch /data "$DATA_DIR" || exit 1
chmod 750 /data "$DATA_DIR" || exit 1
# A stop signal is only noted here; the loop below acts on it. # A stop signal is only noted here; the loop below acts on it.
stop_requested="" stop_requested=""
@@ -26,10 +90,12 @@ trap 'stop_requested=yes' TERM INT
# netwatch-server runs as the netwatch user and listens on loopback # netwatch-server runs as the netwatch user and listens on loopback
# only, on a port other than the public one; nginx.conf proxies to this # only, on a port other than the public one; nginx.conf proxies to this
# address. The netwatch user has no login shell, hence -s /bin/sh. # address. Its only client is nginx, so it takes the client address
# busybox su replaces itself with the command instead of staying on as # nginx passes on from 127.0.0.1 alone, whatever TRUSTED_PROXIES the
# its parent, so $! is the server's own PID. # container has. The netwatch user has no login shell, hence -s
BIND_ADDRESS=127.0.0.1 PORT=8081 \ # /bin/sh. busybox su replaces itself with the command instead of
# staying on as its parent, so $! is the server's own PID.
BIND_ADDRESS=127.0.0.1 PORT=8081 TRUSTED_PROXIES=127.0.0.1/32 \
su -s /bin/sh netwatch -c 'exec netwatch-server' & su -s /bin/sh netwatch -c 'exec netwatch-server' &
backend=$! backend=$!
+3
View File
@@ -9,6 +9,9 @@
type="image/svg+xml" type="image/svg+xml"
href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><text y='.9em' font-size='90'>📡</text></svg>" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><text y='.9em' font-size='90'>📡</text></svg>"
/> />
<!-- Linked here, not imported by src/main.js, so the unit tests can
import that module in Node, which cannot import CSS. -->
<link rel="stylesheet" href="/src/styles.css" />
</head> </head>
<body class="bg-gray-900 text-white min-h-screen"> <body class="bg-gray-900 text-white min-h-screen">
<div id="app"></div> <div id="app"></div>
+22 -4
View File
@@ -8,13 +8,20 @@ server {
# Keep the nginx version out of the Server header and error pages. # Keep the nginx version out of the Server header and error pages.
server_tokens off; server_tokens off;
# The security headers, on every response. An add_header in a
# location drops every add_header from here, so a location with one
# of its own includes this file again.
include /etc/nginx/security-headers.conf;
root /usr/share/nginx/html; root /usr/share/nginx/html;
index index.html; index index.html;
# Trust RFC1918 reverse proxies for X-Forwarded-For # The client address comes from X-Forwarded-For only on a request
set_real_ip_from 10.0.0.0/8; # from the reverse proxies in TRUSTED_PROXIES: bin/entrypoint.sh
set_real_ip_from 172.16.0.0/12; # writes one set_real_ip_from line for each into this file, and
set_real_ip_from 192.168.0.0/16; # leaves it empty when TRUSTED_PROXIES is unset, so that by default
# the client address is the one each request comes from.
include /etc/nginx/trusted-proxies.conf;
real_ip_header X-Forwarded-For; real_ip_header X-Forwarded-For;
real_ip_recursive on; real_ip_recursive on;
@@ -30,6 +37,7 @@ server {
location /assets/ { location /assets/ {
expires 1y; expires 1y;
add_header Cache-Control "public, immutable"; add_header Cache-Control "public, immutable";
include /etc/nginx/security-headers.conf;
} }
# netwatch-server, the Go backend, runs in the same container and # netwatch-server, the Go backend, runs in the same container and
@@ -43,6 +51,16 @@ server {
proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Proto $scheme;
# netwatch-server sets the same security headers on its own
# responses. Its copies are dropped so that each header goes out
# once, as security-headers.conf sets it.
proxy_hide_header Strict-Transport-Security;
proxy_hide_header Content-Security-Policy;
proxy_hide_header X-Frame-Options;
proxy_hide_header X-Content-Type-Options;
proxy_hide_header Referrer-Policy;
proxy_hide_header Permissions-Policy;
location /api/ { location /api/ {
proxy_pass http://127.0.0.1:8081; proxy_pass http://127.0.0.1:8081;
} }
+25 -7
View File
@@ -3,12 +3,12 @@
# this repo. Idempotent: every install is guarded by a check so already # this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew, # installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes nothing is present. Node is # or apk (detected in that order); assumes nothing is present. Node is
# used directly if installed; otherwise it is installed at a pinned # used directly if it is at least NODE_MIN_VERSION; otherwise it is
# version via nvm (installing nvm itself first, from a hash-verified # installed at a pinned version via nvm (installing nvm itself first,
# release archive, never curl | sh). Go, with its gofmt, is used # from a hash-verified release archive, never curl | sh). Go, with its
# directly if it is at least the version backend/go.mod asks for; # gofmt, is used directly if it is at least the version backend/go.mod
# otherwise the pinned Go release is installed from its hash-verified # asks for; otherwise the pinned Go release is installed from its
# archive. # hash-verified archive.
# #
# What this script installs outside the system package manager lives # What this script installs outside the system package manager lives
# under $HOME and is linked into ~/.local/bin, where make and the git # under $HOME and is linked into ~/.local/bin, where make and the git
@@ -23,6 +23,10 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-07-07 # Pinned versions, 2026-07-07
NODE_VERSION="22.17.0" NODE_VERSION="22.17.0"
# The oldest node the frontend's dependencies accept: the "engines"
# field of puppeteer-core 25.5.0, the most demanding of them, asks for
# 22.12.0 or newer, 2026-09-29. An older installed node is not used.
NODE_MIN_VERSION="22.12.0"
NVM_VERSION="0.40.3" NVM_VERSION="0.40.3"
# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz # sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0" NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
@@ -136,8 +140,22 @@ ensure_nvm() {
rm -rf "$tmp" rm -rf "$tmp"
} }
# node_ok: the node on PATH is at least NODE_MIN_VERSION. node itself
# compares the two: major, then minor, then patch.
node_ok() {
if missing node; then return 1; fi
node -e '
const have = process.versions.node.split(".").map(Number);
const want = process.argv[1].split(".").map(Number);
for (let i = 0; i < 3; i++) {
if (have[i] !== want[i]) process.exit(have[i] > want[i] ? 0 : 1);
}
' "$NODE_MIN_VERSION"
}
# ensure_node: unless node_ok, install NODE_VERSION and link its node.
ensure_node() { ensure_node() {
if ! missing node; then return 0; fi if node_ok; then return 0; fi
ensure_nvm ensure_nvm
nvm_sh "nvm install $NODE_VERSION" nvm_sh "nvm install $NODE_VERSION"
link_bin "$HOME/.nvm/versions/node/v$NODE_VERSION/bin/node" node link_bin "$HOME/.nvm/versions/node/v$NODE_VERSION/bin/node" node
+20 -6
View File
@@ -1,15 +1,29 @@
#!/bin/sh #!/bin/sh
# script/cibuild: run the CI build: build the one image from Dockerfile, # script/cibuild: run the CI build. It bootstraps first: a CI runner
# whose stages run the checks as build steps (the backend's fmt-check, # checks out and runs this and nothing else, and script/fmt-check runs
# lint and tests, and the frontend's test, lint and fmt-check). This is # the formatter on the host, which a pristine checkout cannot do.
# the only build step the Gitea workflow runs. # --no-cache for the same reason as script/docker: the gate phases the
# final stage depends on are RUN steps, and a cached one is a check that
# did not run.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
timeout 300 docker build . "$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/check"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. VERSION is computed here because .dockerignore
# excludes .git, so `git describe` in a build stage yields an empty
# version without failing.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
} }
main "$@" main "$@"
+4 -3
View File
@@ -1,13 +1,14 @@
#!/bin/sh #!/bin/sh
# script/frontend-test: run the frontend test suite. The frontend has no # script/frontend-test: run the frontend test suite: the unit tests in
# unit tests; the production build serves as the test (fails on broken # test/unit/ with Node's built-in test runner, then the production
# code). # build, which fails on broken code.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
timeout 30 node --test test/unit/*.test.js
timeout 30 yarn build timeout 30 yarn build
} }
+5
View File
@@ -63,11 +63,16 @@ main() {
# nginx.conf is a template: the image renders it over its own # nginx.conf is a template: the image renders it over its own
# default.conf, with the same port and limit bin/entrypoint.sh uses. # default.conf, with the same port and limit bin/entrypoint.sh uses.
# The empty file it includes trusts no proxy, as bin/entrypoint.sh
# writes it when TRUSTED_PROXIES is unset. nginx.conf also includes
# the security headers, so the page runs under the shipped policy.
docker run -d --rm --name "$SERVER" \ docker run -d --rm --name "$SERVER" \
--network "$NETWORK" --network-alias netwatch \ --network "$NETWORK" --network-alias netwatch \
-e PORT=8080 -e NGINX_ENVSUBST_FILTER='^PORT$' \ -e PORT=8080 -e NGINX_ENVSUBST_FILTER='^PORT$' \
-v "$ROOT/dist:/usr/share/nginx/html:ro" \ -v "$ROOT/dist:/usr/share/nginx/html:ro" \
-v "$ROOT/nginx.conf:/etc/nginx/templates/default.conf.template:ro" \ -v "$ROOT/nginx.conf:/etc/nginx/templates/default.conf.template:ro" \
-v /dev/null:/etc/nginx/trusted-proxies.conf:ro \
-v "$ROOT/security-headers.conf:/etc/nginx/security-headers.conf:ro" \
"$SERVER_IMAGE" > /dev/null "$SERVER_IMAGE" > /dev/null
# The image's own entrypoint already exposes CDP on 9222 and passes # The image's own entrypoint already exposes CDP on 9222 and passes
+24
View File
@@ -0,0 +1,24 @@
# The security headers REPO_POLICIES.md requires on every response.
# nginx.conf includes this file, which Dockerfile copies to
# /etc/nginx/security-headers.conf. always sends each header on error
# responses too.
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
# Scripts and styles load only from the page's own origin. Inline ones
# are blocked, style attributes in markup included, so style elements
# through classes or element.style. data: images are for the favicon
# in index.html. connect-src is * because the browser checks each probe in
# src/main.js against it, and also every redirect the probe follows,
# and several of those hosts redirect to others; a list of hosts here
# would block those probes. It also covers the reports the page sends
# to its own origin.
add_header Content-Security-Policy "default-src 'self'; connect-src *; img-src 'self' data:; object-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'" always;
add_header X-Frame-Options DENY always;
add_header X-Content-Type-Options nosniff always;
# The probed hosts are not told where the page is served from.
add_header Referrer-Policy no-referrer always;
add_header Permissions-Policy "accelerometer=(), camera=(), display-capture=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), midi=(), payment=(), usb=()" always;
+27 -12
View File
@@ -1,14 +1,14 @@
import "./styles.css";
// --- Configuration ----------------------------------------------------------- // --- Configuration -----------------------------------------------------------
// Timing, axis labels, and display constants. Latency above maxLatency is // Timing, axis labels, and display constants. A target check times out
// clamped to "unreachable". The sparkline Y-axis is capped at // after requestTimeout, 80% of updateInterval, so a round's checks have
// all finished before the next round is due; latency above maxLatency is
// recorded as a timeout. The sparkline Y-axis is capped at
// graphMaxLatency — values above it pin to the top of the chart but still // graphMaxLatency — values above it pin to the top of the chart but still
// display their real value in the latency figure. The history buffer holds // display their real value in the latency figure. The history buffer holds
// maxHistoryPoints samples (historyDuration / updateInterval). // maxHistoryPoints samples (historyDuration / updateInterval).
// reportInterval is how often collected samples are POSTed to the backend. // reportInterval is how often collected samples are POSTed to the backend.
const CONFIG = { export const CONFIG = {
updateInterval: 3000, updateInterval: 3000,
maxHistoryPoints: 100, maxHistoryPoints: 100,
reportInterval: 60000, reportInterval: 60000,
@@ -16,7 +16,7 @@ const CONFIG = {
return (this.maxHistoryPoints * this.updateInterval) / 1000; return (this.maxHistoryPoints * this.updateInterval) / 1000;
}, },
get requestTimeout() { get requestTimeout() {
return Math.min(this.updateInterval - 100, 3000); return this.updateInterval * 0.8;
}, },
get maxLatency() { get maxLatency() {
return this.requestTimeout; return this.requestTimeout;
@@ -503,7 +503,7 @@ class Reporter {
// --- Latency Measurement ----------------------------------------------------- // --- Latency Measurement -----------------------------------------------------
async function measureLatency(url) { export async function measureLatency(url) {
const controller = new AbortController(); const controller = new AbortController();
const timeoutId = setTimeout( const timeoutId = setTimeout(
() => controller.abort(), () => controller.abort(),
@@ -716,7 +716,7 @@ function hostRowHTML(host, index, showPin = true) {
${pinBtn} ${pinBtn}
<div class="w-[420px] flex-shrink-0 grid grid-cols-[minmax(0,1fr)_auto] items-center"> <div class="w-[420px] flex-shrink-0 grid grid-cols-[minmax(0,1fr)_auto] items-center">
<div class="flex items-center gap-2 min-w-[200px]"> <div class="flex items-center gap-2 min-w-[200px]">
<div class="w-3 h-3 rounded-full flex-shrink-0" style="background-color: ${latencyHex(null)}"></div> <div class="w-3 h-3 rounded-full flex-shrink-0 bg-[#6b7280]"></div>
<span class="font-medium text-white truncate">${host.name}</span> <span class="font-medium text-white truncate">${host.name}</span>
</div> </div>
<div class="latency-value text-4xl font-bold tabular-nums text-right mt-3" data-host="${index}"> <div class="latency-value text-4xl font-bold tabular-nums text-right mt-3" data-host="${index}">
@@ -1181,11 +1181,16 @@ function startRecoveryProbe(state, triggerTick) {
log.notice( log.notice(
`Recovery probe started (${canaries.map((h) => h.name).join(", ")})`, `Recovery probe started (${canaries.map((h) => h.name).join(", ")})`,
); );
// A check can wait up to CONFIG.requestTimeout, far longer than
// 500ms, so no new checks start while the last ones are waiting.
let checking = false;
state._recoveryProbeId = setInterval(async () => { state._recoveryProbeId = setInterval(async () => {
if (state.paused) return; if (state.paused || checking) return;
checking = true;
const results = await Promise.all( const results = await Promise.all(
canaries.map((h) => measureLatency(h.url)), canaries.map((h) => measureLatency(h.url)),
); );
checking = false;
if (results.some((r) => r.error === null)) { if (results.some((r) => r.error === null)) {
log.notice("Recovery probe: connectivity detected"); log.notice("Recovery probe: connectivity detected");
stopRecoveryProbe(state); stopRecoveryProbe(state);
@@ -1374,8 +1379,18 @@ async function init() {
updateClocks(); updateClocks();
setInterval(updateClocks, 1000); setInterval(updateClocks, 1000);
function doTick() { // A round waits up to CONFIG.requestTimeout for its checks. A round
tick(state, () => startRecoveryProbe(state, doTick)); // asked for while one is still waiting, after an interval change or
// by the recovery probe, is skipped, so rounds never overlap.
let roundRunning = false;
async function doTick() {
if (roundRunning) return;
roundRunning = true;
try {
await tick(state, () => startRecoveryProbe(state, doTick));
} finally {
roundRunning = false;
}
} }
doTick(); doTick();
@@ -1444,7 +1459,7 @@ async function init() {
// Bootstrap only when loaded as the page: a real DOM containing the #app // Bootstrap only when loaded as the page: a real DOM containing the #app
// mount point this module renders into. Importing the module in a unit test // mount point this module renders into. Importing the module in a unit test
// (which has no #app) runs nothing, so buildReport can be tested in isolation. // (which has no #app) runs nothing, so its exports can be tested in isolation.
if (typeof document !== "undefined" && document.getElementById("app")) { if (typeof document !== "undefined" && document.getElementById("app")) {
if (document.readyState === "loading") { if (document.readyState === "loading") {
document.addEventListener("DOMContentLoaded", init); document.addEventListener("DOMContentLoaded", init);
+52
View File
@@ -0,0 +1,52 @@
// Unit tests for src/main.js, run by script/frontend-test with Node's
// built-in test runner. Importing the module does not start the page.
import { after, before, test } from "node:test";
import assert from "node:assert/strict";
import { createServer } from "node:http";
import { CONFIG, measureLatency } from "../../src/main.js";
// measureLatency writes timeouts to the debug log, which looks for its
// panel in the page. There is no page here.
globalThis.document = { getElementById: () => null };
// A target that answers after the number of milliseconds in the path,
// e.g. /600.
let server;
let target;
before(async () => {
server = createServer((req, res) => {
const delay = Number(new URL(req.url, "http://x").pathname.slice(1));
setTimeout(() => res.end(), delay);
});
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
target = `http://127.0.0.1:${server.address().port}`;
});
after(() => {
server.closeAllConnections();
server.close();
});
test("the timeout is 80% of the refresh interval", () => {
CONFIG.updateInterval = 30000;
assert.equal(CONFIG.requestTimeout, 24000);
CONFIG.updateInterval = 3000;
assert.equal(CONFIG.requestTimeout, 2400);
});
test("an answer within the timeout is recorded with its real time, a later one as a timeout", async () => {
// 600ms is past the 400ms timeout of a 500ms interval...
CONFIG.updateInterval = 500;
assert.deepEqual(await measureLatency(`${target}/600`), {
latency: null,
error: "timeout",
});
// ...and within the 1200ms timeout of a 1500ms interval.
CONFIG.updateInterval = 1500;
const { latency, error } = await measureLatency(`${target}/600`);
assert.equal(error, null);
assert.ok(latency >= 600 && latency < 1200, `latency ${latency}ms`);
});