1 Commits
Author SHA1 Message Date
sneak 71f9032041 DEBUG turns debug logging on for any non-empty value (closes #159)
check / check (push) Canceled after 0s
DEBUG was read with strconv.ParseBool, so a value such as "yes" stopped
the start. It now follows the code style guide: unset or empty is off,
and any other value, "false" and "0" included, turns debug logging on.
No value of DEBUG stops the start. The test that "yes" fails is replaced
by one covering unset, empty, "false", "0" and "yes". README.md no
longer gives false as the default, and says which values turn it on.

Model: opus-5-5
2026-10-19 03:45:44 +00:00
30 changed files with 369 additions and 1226 deletions
+1 -4
View File
@@ -73,12 +73,9 @@
# This repository's own entries, after the shared content above: the # This repository's own entries, after the shared content above: the
# frontend build, the viewport test's output, the binary `make build` # frontend build, the viewport test's output, the binary `make build`
# writes and the reports `make run` stores, and Go test binaries, # writes and the reports `make run` stores, and log files at the root.
# coverage output and log files at the root.
/dist /dist
/tmp /tmp
/netwatch-server /netwatch-server
/data /data
/*.test
/*.out
/*.log /*.log
+2 -3
View File
@@ -11,8 +11,7 @@ insert_final_newline = true
[Makefile] [Makefile]
indent_style = tab indent_style = tab
# This repository's own entries, after the shared content above.
[*.go] [*.go]
indent_style = tab indent_style = tab
# This repository's own sections, such as one for another language it
# uses, go below this comment, and a re-vendor keeps them.
-11
View File
@@ -1,20 +1,9 @@
name: check name: check
on: [push] on: [push]
# Free the shared runner: a new push cancels only the same branch's older run.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs: jobs:
check: check:
runs-on: ubuntu-latest runs-on: ubuntu-latest
# Free the shared runner from a hung build.
timeout-minutes: 20
steps: steps:
# actions/checkout v4.2.2, 2026-02-22 # actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
# script/cibuild needs no token, so none is left in .git/config.
with:
persist-credentials: false
# All history and tags, so git describe finds the version tag.
fetch-depth: 0
- run: script/cibuild - run: script/cibuild
+2 -4
View File
@@ -27,7 +27,7 @@ node_modules/
# Environment files. `*.env` covers bare `.env` and the `prod.env` # Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are # convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds # re-included below. A repository that commits any other template adds
# its own negation at the end of this file, for example `!.env.example`. # its own negation after these lines, for example `!.env.example`.
*.[eE][nN][vV] *.[eE][nN][vV]
.[eE][nN][vV].* .[eE][nN][vV].*
.[eE][nN][vV][rR][cC] .[eE][nN][vV][rR][cC]
@@ -46,9 +46,7 @@ node_modules/
[iI][dD]_[eE][dD]25519 [iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK] [iI][dD]_[eE][dD]25519_[sS][kK]
# This repository's own entries, such as its build outputs, go below # This repository's own entries, after the shared content above.
# this comment, and a re-vendor keeps them. Anchor a binary built at the
# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
# Build output # Build output
dist/ dist/
-2
View File
@@ -25,8 +25,6 @@ linters:
# silenced by disabling that name, not by enabling the successor. # silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5 - wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2 - gomodguard # Deprecated, replaced by gomodguard_v2
# Misses findings at random in v2.14.0; back once a pinned release fixes it
- canonicalheader
settings: settings:
lll: lll:
line-length: 88 line-length: 88
+17 -42
View File
@@ -41,13 +41,11 @@ The scripts cover both the frontend and the Go backend, so `make check` fails if
either one is broken. We provide: either one is broken. We provide:
- `script/bootstrap` — install all dependencies (the pinned node via nvm unless - `script/bootstrap` — install all dependencies (the pinned node via nvm unless
the `node` on `PATH` has the pinned major version and is new enough for the one new enough for the frontend's dependencies is installed, yarn via
frontend's dependencies, yarn `YARN_VERSION` via corepack, or npm where there corepack, `yarn install --frozen-lockfile`, Go `GO_VERSION` unless the `go` on
is no corepack, unless the `yarn` on `PATH` is exactly that version, `PATH` is exactly that version, and the Go modules), linking what it installs
`yarn install --frozen-lockfile`, Go `GO_VERSION` unless the `go` on `PATH` is itself into `~/.local/bin`, which has to be on `PATH`. It installs no Go
exactly that version, and the Go modules), linking what it installs itself linter and not Docker: `make test` and `make lint` run in Docker
into `~/.local/bin`, which has to be on `PATH`. It installs no Go linter and
not Docker: `make test` and `make lint` run in Docker
- `script/setup` — make a fresh clone ready for development: bootstrap plus the - `script/setup` — make a fresh clone ready for development: bootstrap plus the
git pre-commit hook git pre-commit hook
- `script/dev` — run the Vite dev server, which proxies `/api` to a locally - `script/dev` — run the Vite dev server, which proxies `/api` to a locally
@@ -55,18 +53,11 @@ either one is broken. We provide:
- `script/build` — build the frontend for production into `dist/`, then the - `script/build` — build the frontend for production into `dist/`, then the
static `netwatch-server` binary into the repo root with its version stamped static `netwatch-server` binary into the repo root with its version stamped
in: `VERSION` from the environment, or when that is unset or empty, in: `VERSION` from the environment, or when that is unset or empty,
`git describe --tags --always --dirty` inside a git checkout, as `git describe` inside a git checkout, then `dev`
`script/docker` does, then `dev`
- `script/run` — build with `script/build`, then run `netwatch-server` with its - `script/run` — build with `script/build`, then run `netwatch-server` with its
defaults (see [Settings](#settings)) defaults (see [Settings](#settings))
- `script/clean` — remove what `script/build` writes - `script/clean` — remove what `script/build` writes
- `script/projectname` — print the project name (used for the Docker image tag) - `script/projectname` — print the project name (used for the Docker image tag)
- `script/yarn` — run yarn with the given arguments under the node
`script/bootstrap` would use: with `~/.local/bin` first on `PATH`, the first
`node` on that `PATH` when it has the pinned major version and is new enough
for the frontend's dependencies, and otherwise the pinned node in nvm. Every
script that runs yarn on the host runs it through this one, except
`script/bootstrap`, which picks its own node and yarn
- `script/test` — build the `test` phase of `Dockerfile`, writing no image: the - `script/test` — build the `test` phase of `Dockerfile`, writing no image: the
frontend's unit tests in its `frontend-test` stage, and the backend's Go tests frontend's unit tests in its `frontend-test` stage, and the backend's Go tests
with the race detector and coverage. The frontend's production build is left with the race detector and coverage. The frontend's production build is left
@@ -208,9 +199,7 @@ stops. Its routes:
address may send a limited number a minute, and the report files are capped in address may send a limited number a minute, and the report files are capped in
size, the oldest deleted first size, the oldest deleted first
- `GET /.well-known/healthcheck` — answers 200 with `"status":"ok"`, the - `GET /.well-known/healthcheck` — answers 200 with `"status":"ok"`, the
server's version and its uptime; while reports cannot be stored, as the last server's version and its uptime
write of a report file failed, it answers 503 with `"status":"error"` and the
same fields (see [Report storage](#report-storage))
- `GET /metrics` — Prometheus metrics behind basic auth, only when - `GET /metrics` — Prometheus metrics behind basic auth, only when
`METRICS_USERNAME` and `METRICS_PASSWORD` are set; each client address may `METRICS_USERNAME` and `METRICS_PASSWORD` are set; each client address may
make a limited number of requests to it a minute make a limited number of requests to it a minute
@@ -218,10 +207,6 @@ stops. Its routes:
It is an `fx`-wired Go application: `cmd/netwatch-server/` starts it, and its It is an `fx`-wired Go application: `cmd/netwatch-server/` starts it, and its
packages are in `internal/`: packages are in `internal/`:
- **`cli`**: Runs `netwatch-server` from its command line: the `check-cidr` and
`prepare-data-dir` subcommands, which `bin/entrypoint.sh` runs before it
starts the server, and, given no arguments, the server, with the `fx` wiring.
It refuses any other arguments with a usage message and exit code 2.
- **`config`**: Loads configuration from environment variables and config files - **`config`**: Loads configuration from environment variables and config files
via Viper. via Viper.
- **`handlers`**: HTTP request handlers for the API (health check, report - **`handlers`**: HTTP request handlers for the API (health check, report
@@ -286,16 +271,13 @@ toward `DATA_DIR_MAX_BYTES` from the next start, only if removing it fails too.
Files are created with `O_EXCL` to prevent overwrites. Files are created with `O_EXCL` to prevent overwrites.
The reports of a failed write stay in memory, and the next flush, within a The reports of a failed write stay in memory, and the next flush, within a
minute, tries again to write them. The server writes one file at a time, so a minute, tries again to write them. From the failed write until a write succeeds,
file holds its reports in the order they arrived, those of a failed write `POST /api/v1/reports` refuses every report with 500 and keeps nothing of it.
included. From the failed write until a write succeeds, `POST /api/v1/reports` The page sends the refused samples again in its next report, as long as it still
refuses every report with 500 and keeps nothing of it, and the health check holds them: at the 1-second interval one refused report already loses some of
answers 503 with `"status":"error"`. The page sends the refused samples again in those in the history (see [Reporting](#reporting)). Reports already answered 200
its next report, as long as it still holds them: at the 1-second interval one are lost only if the server stops before a write succeeds; the final flush at
refused report already loses some of those in the history (see stop tries them too, and if it fails, the process exits non-zero.
[Reporting](#reporting)). Reports already answered 200 are lost only if the
server stops before a write succeeds; the final flush at stop tries them too,
and if it fails, the process exits non-zero.
#### Report limits #### Report limits
@@ -567,16 +549,9 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
stops the container. Empty, the backend sends nothing to Sentry stops the container. Empty, the backend sends nothing to Sentry
- **Health check:** the image's `HEALTHCHECK` requests - **Health check:** the image's `HEALTHCHECK` requests
`/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless `/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless
both nginx and the backend answer. It also fails while reports cannot be both nginx and the backend answer. upaas reads the container's health 60
stored, as when the volume is full or `DATA_DIR` cannot be written: the health seconds after a deploy and fails the deploy unless it is `healthy`. The
check answers 503 from a failed write of a report file until a write succeeds, container also stops when either process exits.
and the backend tries the write again every minute. Three failed checks in a
row make the container `unhealthy`, and one that passes makes it `healthy`
again. upaas reads the container's health once, 60 seconds after a deploy, and
fails the deploy unless it is `healthy`. A container that turns `unhealthy`
after that is neither restarted nor reported by upaas, whose `unless-stopped`
restart policy ignores health; only `docker ps` and `docker inspect` show it.
The container also stops when either process exits.
## Browser Compatibility ## Browser Compatibility
+4 -115
View File
@@ -9,12 +9,10 @@
# Status # Status
pre-1.0. Release candidates are tagged on `next`, the first being `v1.0.0-rc1`; pre-1.0. No git tags. `main` is the stable branch and `next` the development
`git tag --merged origin/next` lists them. `main` is the stable branch and branch, which every PR targets. The frontend and the Go backend ship as one
`next` the development branch, which every PR targets. The frontend and the Go Docker image, and the Gitea workflow `.gitea/workflows/check.yml` runs
backend ship as one Docker image, and the Gitea workflow `script/cibuild` on every push. Working toward 1.0.0.
`.gitea/workflows/check.yml` runs `script/cibuild` on every push, and cancels
the older run on the same branch. Working toward 1.0.0.
# Next Step # Next Step
@@ -23,115 +21,6 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of
# Completed Steps # Completed Steps
- 2026-10-19: the server's `OnStart` hook in `internal/server/server.go` starts
`listenAndServe` with `go func() { s.listenAndServe() }()`, as the Go style
guide asks ([#162](https://git.eeqj.de/sneak/netwatch/issues/162)); it was
`go s.listenAndServe()`. No other `go` statement in the Go code called a
function directly. Nothing the server does changed
- 2026-10-19: `.gitignore`, `.editorconfig` and `.golangci.yml` are the
sneak/prompts copies at `0653579`
([#174](https://git.eeqj.de/sneak/netwatch/issues/174)). `.gitignore` is the
standard file followed by this repository's own entries; `.editorconfig` is
the standard file alone, which now carries the `[*.go]` section.
`.golangci.yml` disables `canonicalheader`, as the canonical copy does,
because golangci-lint v2.14.0 misses its findings at random
- 2026-10-19: `package.json` has no `version` field and is marked
`"private": true` ([#158](https://git.eeqj.de/sneak/netwatch/issues/158)). It
said `1.0.0`, a release that has not been made, and nothing read it; the git
tag is the one version, which the image and `make build` take from
`git describe --tags`. The package is never published
- 2026-10-19: Status names `v1.0.0-rc1`, the first release candidate, and says
release candidates are tagged on `next`, where it said there were no git tags
([#157](https://git.eeqj.de/sneak/netwatch/issues/157)). The rest of Status
matches the tree
- 2026-10-19: `netwatch-server` starts the server only when given no arguments
([#179](https://git.eeqj.de/sneak/netwatch/issues/179)). `check-cidr CIDR` and
`prepare-data-dir DATA_DIR` behave as before; any other arguments, such as a
subcommand with no value or two, or a misspelt one, print a one-line usage
message naming both forms to stderr and exit 2. They used to start the server.
Unit tests call `cli.Main` with no arguments and check that it starts the
server, and with each subcommand given no value and two values and with a
misspelt subcommand, and check the exit code, the message and that the server
does not start
- 2026-10-19: `main()` in `cmd/netwatch-server/main.go` is one call into a new
package, `internal/cli`, as the Go style guide asks
([#160](https://git.eeqj.de/sneak/netwatch/issues/160)). `internal/cli` runs
the `check-cidr` and `prepare-data-dir` subcommands, with the same arguments,
messages and exit codes, and otherwise the server, with the `fx` wiring.
`Appname` and `Version` stay in `main`, where the build sets `Version`, and
`main` gives `cli.Main` stderr and the function that runs the server, which
the tests replace with a buffer and a function that fails them. Unit tests
call `cli.Main` with `check-cidr` and a CIDR the server takes and one it
refuses, and check the exit code and what it writes to stderr, and with
`prepare-data-dir` and a path outside `/data`, and check that it exits 1 with
the error from looking up the `netwatch` user, which the test image does not
have. Others give the functions behind the subcommands input they take and
input they refuse, and check the exit code and what they write to stderr
- 2026-10-19: when `VERSION` is unset or empty, `script/build` stamps the
version from `git describe --tags --always --dirty`, as `script/docker` does
([#156](https://git.eeqj.de/sneak/netwatch/issues/156)). Without `--tags` it
skipped lightweight tags, so `make run` could report a commit where the image
built from the same tree reported the tag. Checked once with a lightweight tag
on a scratch clone: `make build` stamped it
- 2026-10-19: `.dockerignore` keeps Go test binaries and coverage output at the
root out of the build context, with `/*.test` and `/*.out` in the repository's
own section ([#155](https://git.eeqj.de/sneak/netwatch/issues/155)). Such
files went into the build context of every `docker build` in `script/` and
were copied into the lint, test and build stages. Checked with a probe image
that copies the whole context: `x.test` and `x.out` planted at the root no
longer land in it, and nothing else changed. The entries match only at the
root, so the same files in a subdirectory still reach the image
- 2026-10-19: `.gitea/workflows/check.yml` is the sneak/prompts copy at
`0653579` ([#153](https://git.eeqj.de/sneak/netwatch/issues/153)). Its
checkout step sets `fetch-depth: 0`, so CI clones all history and tags, and
`git describe` in `script/cibuild` gives the image built from a tagged commit
the tag as its version; it gave the short commit. The checkout step also sets
`persist-credentials: false`, so the job's token is not left in `.git/config`.
A new push cancels the older run on the same branch, and a run stops after 20
minutes
- 2026-10-19: `script/bootstrap` keeps the `yarn` on `PATH` only when
`yarn --version` is exactly `YARN_VERSION`, and otherwise installs that
version as it did when no yarn was installed
([#154](https://git.eeqj.de/sneak/netwatch/issues/154)); any yarn used to be
kept. It uses the `node` on `PATH` only when that node has the major version
of `NODE_VERSION`, as `REPO_POLICIES.md` asks, and is at least
`NODE_MIN_VERSION`; a node 24 was kept before. After installing node or yarn
it looks the tool up through `PATH` again and stops if that one fails the same
check, as it does for Go, and it prints the node and yarn versions it uses.
Every script that runs yarn on the host, except `script/bootstrap`, which
picks its own node and yarn, runs it through the new `script/yarn`. That
script puts `~/.local/bin` first on `PATH`, uses the first `node` on that
`PATH` when it passes the same test, and otherwise runs yarn under the pinned
node in nvm; `make build` ran yarn under whatever node was first on `PATH`
- 2026-10-19: `bin/entrypoint.sh` has each of its steps in a function, and a
`main` at the end calls them in order, as the shell section of the code style
guide asks ([#163](https://git.eeqj.de/sneak/netwatch/issues/163)). It is
still POSIX sh with no `set -e`, and does what it did, with the same messages
and exit codes. Checked by running the image with each script: a normal start,
bad `PORT`, `TRUSTED_PROXIES` and `DATA_DIR` values, `docker stop`, INT, and
nginx exiting on its own
- 2026-10-19: while report files cannot be written, the health check answers 503
with `"status":"error"` and the same fields, so the image's `HEALTHCHECK`
reports the container `unhealthy`
([#142](https://git.eeqj.de/sneak/netwatch/issues/142)). It answered 200
`"status":"ok"` whatever happened, so the platform saw a healthy container
while every report was refused with 500. It answers 200 at start, before any
write has failed, and again from the first write that succeeds. A unit test
removes `DATA_DIR` to make a write fail and checks the 503, then puts it back
and checks the 200 after the next write
- 2026-10-19: report file writes no longer overlap
([#150](https://git.eeqj.de/sneak/netwatch/issues/150)), so the reports of
failed writes are written in the order they arrived. Two writes that failed at
the same time put their reports back in the wrong order when the older failed
first. One write runs at a time, and each takes the waiting reports only when
it starts, after a failed write has put its own back. The write `Append`
starts when 10 MiB are waiting waits for any write under way, and `Append`
starts no other while it waits. Files written since the start are deleted to
make room in the order they were written, after those found at start. A unit
test holds a write open, starts a second at 10 MiB, fails both and checks the
order in the next file. Others check that `Append` starts no second write at
10 MiB while one waits, that it starts one again the next time 10 MiB are
waiting, and that the first file written since the start is the first deleted
- 2026-10-19: `DEBUG` unset or empty leaves debug logging off, and any other - 2026-10-19: `DEBUG` unset or empty leaves debug logging off, and any other
value, `false` and `0` included, turns it on, as the code style guide says value, `false` and `0` included, turns it on, as the code style guide says
([#159](https://git.eeqj.de/sneak/netwatch/issues/159)). No value of it stops ([#159](https://git.eeqj.de/sneak/netwatch/issues/159)). No value of it stops
+63 -91
View File
@@ -13,27 +13,24 @@ set -u
# address and start anyway, and reports a bad port without naming # address and start anyway, and reports a bad port without naming
# PORT, so a value that is not a usable port stops the container here, # PORT, so a value that is not a usable port stops the container here,
# before either process starts. # before either process starts.
check_port() { export PORT="${PORT:-8080}"
export PORT="${PORT:-8080}" case "$PORT" in
case "$PORT" in *[!0-9]*)
*[!0-9]*) echo "entrypoint: PORT must be a port number, not '$PORT'" >&2
echo "entrypoint: PORT must be a port number, not '$PORT'" >&2
exit 1
;;
esac
# The length is checked first because, for a number too big for it,
# the shell's test prints an error and is false, so the range checks
# alone would let it through.
if [ "${#PORT}" -gt 5 ] || [ "$PORT" -lt 1 ] || [ "$PORT" -gt 65535 ]; then
echo "entrypoint: PORT must be from 1 to 65535, not '$PORT'" >&2
exit 1 exit 1
fi ;;
if [ "$PORT" -eq 8081 ]; then esac
echo "entrypoint: PORT cannot be 8081, netwatch-server" \ # The length is checked first because, for a number too big for it,
"listens there" >&2 # the shell's test prints an error and is false, so the range checks
exit 1 # alone would let it through.
fi if [ "${#PORT}" -gt 5 ] || [ "$PORT" -lt 1 ] || [ "$PORT" -gt 65535 ]; then
} echo "entrypoint: PORT must be from 1 to 65535, not '$PORT'" >&2
exit 1
fi
if [ "$PORT" -eq 8081 ]; then
echo "entrypoint: PORT cannot be 8081, netwatch-server listens there" >&2
exit 1
fi
# TRUSTED_PROXIES names the reverse proxies in front of the container, # TRUSTED_PROXIES names the reverse proxies in front of the container,
# as IP addresses or CIDRs separated by commas. nginx takes the client # as IP addresses or CIDRs separated by commas. nginx takes the client
@@ -50,23 +47,21 @@ check_port() {
# TRUSTED_PROXIES. Its error, naming the CIDR, is dropped for the one # TRUSTED_PROXIES. Its error, naming the CIDR, is dropped for the one
# below, naming the entry as written. set -f keeps a * in an entry from # below, naming the entry as written. set -f keeps a * in an entry from
# becoming a list of file names. # becoming a list of file names.
write_trusted_proxies() { TRUSTED_PROXIES="${TRUSTED_PROXIES-10.0.0.0/8,172.16.0.0/12,192.168.0.0/16}"
TRUSTED_PROXIES="${TRUSTED_PROXIES-10.0.0.0/8,172.16.0.0/12,192.168.0.0/16}" set -f
set -f for proxy in $(printf '%s' "$TRUSTED_PROXIES" | tr ',' ' '); do
for proxy in $(printf '%s' "$TRUSTED_PROXIES" | tr ',' ' '); do case "$proxy" in
case "$proxy" in */*) cidr="$proxy" ;;
*/*) cidr="$proxy" ;; *:*) cidr="$proxy/128" ;;
*:*) cidr="$proxy/128" ;; *) cidr="$proxy/32" ;;
*) cidr="$proxy/32" ;; esac
esac if ! netwatch-server check-cidr "$cidr" 2> /dev/null; then
if ! netwatch-server check-cidr "$cidr" 2> /dev/null; then echo "entrypoint: TRUSTED_PROXIES must be IP addresses or CIDRs" \
echo "entrypoint: TRUSTED_PROXIES must be IP addresses or CIDRs" \ "separated by commas; '$proxy' is neither" >&2
"separated by commas; '$proxy' is neither" >&2 exit 1
exit 1 fi
fi echo "set_real_ip_from $cidr;"
echo "set_real_ip_from $cidr;" done > /etc/nginx/trusted-proxies.conf
done > /etc/nginx/trusted-proxies.conf
}
# netwatch-server keeps its report files in DATA_DIR, on the /data # netwatch-server keeps its report files in DATA_DIR, on the /data
# volume, which may be a host directory owned by root or by another # volume, which may be a host directory owned by root or by another
@@ -78,10 +73,12 @@ write_trusted_proxies() {
# naming DATA_DIR, unless DATA_DIR is /data or a path below it, and it # naming DATA_DIR, unless DATA_DIR is /data or a path below it, and it
# acts on nothing outside /data, whatever symbolic or hard links it # acts on nothing outside /data, whatever symbolic or hard links it
# meets there. # meets there.
prepare_data_dir() { export DATA_DIR="${DATA_DIR:-/data/reports}"
export DATA_DIR="${DATA_DIR:-/data/reports}" netwatch-server prepare-data-dir "$DATA_DIR" || exit 1
netwatch-server prepare-data-dir "$DATA_DIR" || exit 1
} # A stop signal is only noted here; the loop below acts on it.
stop_requested=""
trap 'stop_requested=yes' TERM INT
# netwatch-server runs as the netwatch user and listens on loopback # netwatch-server runs as the netwatch user and listens on loopback
# only, on a port other than the public one; configs/nginx.conf proxies # only, on a port other than the public one; configs/nginx.conf proxies
@@ -90,11 +87,9 @@ prepare_data_dir() {
# the container has. The netwatch user has no login shell, hence -s # the container has. The netwatch user has no login shell, hence -s
# /bin/sh. busybox su replaces itself with the command instead of # /bin/sh. busybox su replaces itself with the command instead of
# staying on as its parent, so $! is the server's own PID. # staying on as its parent, so $! is the server's own PID.
start_backend() { BIND_ADDRESS=127.0.0.1 PORT=8081 TRUSTED_PROXIES=127.0.0.1/32 \
BIND_ADDRESS=127.0.0.1 PORT=8081 TRUSTED_PROXIES=127.0.0.1/32 \ su -s /bin/sh netwatch -c 'exec netwatch-server' &
su -s /bin/sh netwatch -c 'exec netwatch-server' & backend=$!
backend=$!
}
# nginx starts through the nginx image's own entrypoint, which applies # nginx starts through the nginx image's own entrypoint, which applies
# the image's start-up configuration and then replaces itself with # the image's start-up configuration and then replaces itself with
@@ -103,11 +98,9 @@ start_backend() {
# that rendering to PORT: a variable nginx itself uses, such as $uri, # that rendering to PORT: a variable nginx itself uses, such as $uri,
# would otherwise be replaced by an environment variable of the same # would otherwise be replaced by an environment variable of the same
# name. # name.
start_nginx() { NGINX_ENVSUBST_FILTER='^PORT$' \
NGINX_ENVSUBST_FILTER='^PORT$' \ /docker-entrypoint.sh nginx -g 'daemon off;' &
/docker-entrypoint.sh nginx -g 'daemon off;' & nginx=$!
nginx=$!
}
running() { running() {
kill -0 "$1" 2>/dev/null kill -0 "$1" 2>/dev/null
@@ -116,47 +109,26 @@ running() {
# POSIX sh cannot wait for whichever of two children exits first, so # POSIX sh cannot wait for whichever of two children exits first, so
# look once a second. The shell collects a child that has exited while # look once a second. The shell collects a child that has exited while
# it runs sleep, and running() is false for that child from then on. # it runs sleep, and running() is false for that child from then on.
wait_for_stop_or_exit() { while [ -z "$stop_requested" ] && running "$backend" && running "$nginx"; do
while [ -z "$stop_requested" ] && running "$backend" && running "$nginx"; do sleep 1
sleep 1 done
done
}
# Stop both, then wait until neither is left. # Stop both, then wait until neither is left.
stop_both() { kill -TERM "$backend" "$nginx" 2>/dev/null
kill -TERM "$backend" "$nginx" 2>/dev/null while running "$backend" || running "$nginx"; do
while running "$backend" || running "$nginx"; do sleep 1
sleep 1 done
done
} wait "$backend"
backend_status=$?
wait "$nginx"
nginx_status=$?
echo "entrypoint: netwatch-server exited $backend_status," \
"nginx exited $nginx_status"
# Success is a requested stop that both processes exited cleanly from. # Success is a requested stop that both processes exited cleanly from.
exit_with_status() { if [ -n "$stop_requested" ] && [ "$backend_status" -eq 0 ] &&
wait "$backend" [ "$nginx_status" -eq 0 ]; then
backend_status=$? exit 0
wait "$nginx" fi
nginx_status=$? exit 1
echo "entrypoint: netwatch-server exited $backend_status," \
"nginx exited $nginx_status"
if [ -n "$stop_requested" ] && [ "$backend_status" -eq 0 ] &&
[ "$nginx_status" -eq 0 ]; then
exit 0
fi
exit 1
}
main() {
check_port
write_trusted_proxies
prepare_data_dir
# A stop signal is only noted here; wait_for_stop_or_exit acts on it.
stop_requested=""
trap 'stop_requested=yes' TERM INT
start_backend
start_nginx
wait_for_stop_or_exit
stop_both
exit_with_status
}
main "$@"
+74 -4
View File
@@ -1,11 +1,22 @@
// Package main is the entry point for netwatch-server; package cli // Package main is the entry point for netwatch-server.
// does the work.
package main package main
import ( import (
"fmt"
"os" "os"
"os/user"
"sneak.berlin/go/netwatch/internal/cli" "sneak.berlin/go/netwatch/internal/config"
"sneak.berlin/go/netwatch/internal/globals"
"sneak.berlin/go/netwatch/internal/handlers"
"sneak.berlin/go/netwatch/internal/healthcheck"
"sneak.berlin/go/netwatch/internal/logger"
"sneak.berlin/go/netwatch/internal/middleware"
"sneak.berlin/go/netwatch/internal/reportbuf"
"sneak.berlin/go/netwatch/internal/server"
"go.uber.org/fx"
"go.uber.org/fx/fxevent"
) )
//nolint:gochecknoglobals // set via ldflags at build time //nolint:gochecknoglobals // set via ldflags at build time
@@ -15,5 +26,64 @@ var (
) )
func main() { func main() {
os.Exit(cli.Main(os.Stderr, Appname, Version, os.Args[1:], cli.RunServer)) // "netwatch-server check-cidr CIDR" exits 1, with the error, if
// this server would refuse CIDR in its TRUSTED_PROXIES.
// bin/entrypoint.sh runs it on each entry it gives nginx.
if len(os.Args) == 3 && os.Args[1] == "check-cidr" {
_, err := middleware.ParseTrustedProxies(os.Args[2:])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
return
}
// "netwatch-server prepare-data-dir DATA_DIR" gets DATA_DIR ready
// for the netwatch user, or exits 1 with the error; see
// reportbuf.PrepareDataDir. bin/entrypoint.sh runs it as root
// before it starts this server as that user.
if len(os.Args) == 3 && os.Args[1] == "prepare-data-dir" {
netwatch, err := user.Lookup("netwatch")
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
err = reportbuf.PrepareDataDir("/data", os.Args[2], netwatch)
if err != nil {
fmt.Fprintf(os.Stderr, "DATA_DIR '%s': %v\n", os.Args[2], err)
os.Exit(1)
}
return
}
globals.Appname = Appname
globals.Version = Version
fx.New(
// fx logs each step of starting and stopping through the
// server's own logger, so off a terminal those lines are
// JSON like every other line.
fx.WithLogger(func(log *logger.Logger) fxevent.Logger {
return &fxevent.SlogLogger{Logger: log.Get()}
}),
fx.Provide(
config.New,
globals.New,
handlers.New,
healthcheck.New,
logger.New,
middleware.New,
reportbuf.New,
server.New,
),
fx.Invoke(
// First, so the name and version are logged even when
// a setting stops the start.
func(log *logger.Logger) { log.Identify() },
func(*server.Server) {},
),
).Run()
} }
-143
View File
@@ -1,143 +0,0 @@
// Package cli runs netwatch-server from its command line: the
// check-cidr and prepare-data-dir subcommands, which bin/entrypoint.sh
// runs before it starts the server, and, given no arguments, the
// server, with the fx wiring.
package cli
import (
"fmt"
"io"
"os/user"
"sneak.berlin/go/netwatch/internal/config"
"sneak.berlin/go/netwatch/internal/globals"
"sneak.berlin/go/netwatch/internal/handlers"
"sneak.berlin/go/netwatch/internal/healthcheck"
"sneak.berlin/go/netwatch/internal/logger"
"sneak.berlin/go/netwatch/internal/middleware"
"sneak.berlin/go/netwatch/internal/reportbuf"
"sneak.berlin/go/netwatch/internal/server"
"go.uber.org/fx"
"go.uber.org/fx/fxevent"
)
// usageExitCode is what Main returns for a command line it does not
// understand, the usual exit code for a usage error.
const usageExitCode = 2
// Main runs netwatch-server with args, its command line after the
// program's name, and returns the exit code. With no args, Main runs
// the server by calling serve with appname and version, the name and
// version the server logs and reports. With "check-cidr CIDR" or
// "prepare-data-dir DATA_DIR" it runs that subcommand, which writes its
// errors to stderr. Any other args get a one-line usage message on
// stderr and exit code 2. main passes os.Stderr and RunServer; the
// tests pass a buffer and a function of their own, so that no test
// starts the server. When serve is RunServer and the server fails to
// start, to listen or to stop, fx's Run exits the process with status 1
// and Main never returns.
func Main(
stderr io.Writer,
appname, version string,
args []string,
serve func(appname, version string),
) int {
// "netwatch-server check-cidr CIDR" exits 1, with the error, if
// this server would refuse CIDR in its TRUSTED_PROXIES.
// bin/entrypoint.sh runs it on each entry it gives nginx.
if len(args) == 2 && args[0] == "check-cidr" {
return checkCIDR(stderr, args[1])
}
// "netwatch-server prepare-data-dir DATA_DIR" gets DATA_DIR ready
// for the netwatch user, or exits 1 with the error; see
// reportbuf.PrepareDataDir. bin/entrypoint.sh runs it as root
// before it starts this server as that user.
if len(args) == 2 && args[0] == "prepare-data-dir" {
return prepareDataDir(stderr, "/data", args[1], "netwatch")
}
// Any other command line is refused, so that a mistyped one does
// not start the server.
if len(args) > 0 {
_, _ = fmt.Fprintln(stderr,
"usage: netwatch-server [check-cidr CIDR | prepare-data-dir DATA_DIR]")
return usageExitCode
}
serve(appname, version)
return 0
}
// checkCIDR returns 0 if the server would take cidr in its
// TRUSTED_PROXIES, and otherwise writes the error to stderr and
// returns 1.
func checkCIDR(stderr io.Writer, cidr string) int {
_, err := middleware.ParseTrustedProxies([]string{cidr})
if err != nil {
_, _ = fmt.Fprintln(stderr, err)
return 1
}
return 0
}
// prepareDataDir gets dataDir ready for the user named username with
// reportbuf.PrepareDataDir, volume being where the data volume is
// mounted, and returns 0. Otherwise it writes the error to stderr and
// returns 1.
func prepareDataDir(stderr io.Writer, volume, dataDir, username string) int {
owner, err := user.Lookup(username)
if err != nil {
_, _ = fmt.Fprintln(stderr, err)
return 1
}
err = reportbuf.PrepareDataDir(volume, dataDir, owner)
if err != nil {
_, _ = fmt.Fprintf(stderr, "DATA_DIR '%s': %v\n", dataDir, err)
return 1
}
return 0
}
// RunServer runs the server until it is told to stop, with appname and
// version as the name and version it logs and reports. If the server
// fails to start, to listen or to stop, fx's Run exits the process
// with status 1 itself.
func RunServer(appname, version string) {
globals.Appname = appname
globals.Version = version
fx.New(
// fx logs each step of starting and stopping through the
// server's own logger, so off a terminal those lines are
// JSON like every other line.
fx.WithLogger(func(log *logger.Logger) fxevent.Logger {
return &fxevent.SlogLogger{Logger: log.Get()}
}),
fx.Provide(
config.New,
globals.New,
handlers.New,
healthcheck.New,
logger.New,
middleware.New,
reportbuf.New,
server.New,
),
fx.Invoke(
// First, so the name and version are logged even when
// a setting stops the start.
func(log *logger.Logger) { log.Identify() },
func(*server.Server) {},
),
).Run()
}
-219
View File
@@ -1,219 +0,0 @@
package cli_test
import (
"bytes"
"os"
"os/user"
"path/filepath"
"strconv"
"strings"
"testing"
"sneak.berlin/go/netwatch/internal/cli"
"sneak.berlin/go/netwatch/internal/reportbuf"
)
// TestMainRunsCheckCIDR: Main given check-cidr returns 0, writing
// nothing, for a CIDR the server takes in its TRUSTED_PROXIES, and 1,
// with an error naming TRUSTED_PROXIES and the CIDR, for one it
// refuses, and does not start the server.
func TestMainRunsCheckCIDR(t *testing.T) {
t.Parallel()
var stderr bytes.Buffer
code := cli.Main(&stderr, "netwatch", "test",
[]string{"check-cidr", "10.0.0.0/8"}, failIfServerStarts(t))
if code != 0 || stderr.Len() != 0 {
t.Errorf("check-cidr 10.0.0.0/8: exit %d, stderr %q; want exit 0 and no output",
code, stderr.String())
}
stderr.Reset()
code = cli.Main(&stderr, "netwatch", "test",
[]string{"check-cidr", "1.2.3/32"}, failIfServerStarts(t))
want := `TRUSTED_PROXIES "1.2.3/32": `
if code != 1 || !strings.HasPrefix(stderr.String(), want) {
t.Errorf("check-cidr 1.2.3/32: exit %d, stderr %q; want exit 1 and %q...",
code, stderr.String(), want)
}
}
// TestMainRunsPrepareDataDir: Main given prepare-data-dir runs the
// prepare-data-dir subcommand for the netwatch user, and does not start
// the server. The test image has no netwatch user, so that subcommand
// exits 1 with the error from looking the user up;
// TestPrepareDataDirRefusesPathOutsideVolume checks the path.
func TestMainRunsPrepareDataDir(t *testing.T) {
t.Parallel()
var stderr bytes.Buffer
code := cli.Main(&stderr, "netwatch", "test",
[]string{"prepare-data-dir", "/etc"}, failIfServerStarts(t))
want := user.UnknownUserError("netwatch").Error() + "\n"
if code != 1 || stderr.String() != want {
t.Errorf("prepare-data-dir /etc: exit %d, stderr %q; want exit 1 and %q",
code, stderr.String(), want)
}
}
// TestMainRunsServerWithNoArguments: Main given no arguments starts the
// server with the name and version it was given, writes nothing and
// returns 0.
func TestMainRunsServerWithNoArguments(t *testing.T) {
t.Parallel()
var (
stderr bytes.Buffer
gotAppname, gotVersion string
)
code := cli.Main(&stderr, "netwatch", "test", []string{},
func(appname, version string) {
gotAppname, gotVersion = appname, version
})
if code != 0 || stderr.Len() != 0 {
t.Errorf("exit %d, stderr %q; want exit 0 and no output",
code, stderr.String())
}
if gotAppname != "netwatch" || gotVersion != "test" {
t.Errorf("server started with %q %q; want %q %q",
gotAppname, gotVersion, "netwatch", "test")
}
}
// TestMainRefusesOtherCommandLines: Main given a subcommand with too
// few or too many values, or a misspelt subcommand, writes the usage
// message to stderr, returns 2 and does not start the server.
func TestMainRefusesOtherCommandLines(t *testing.T) {
t.Parallel()
want := "usage: netwatch-server " +
"[check-cidr CIDR | prepare-data-dir DATA_DIR]\n"
for _, commandLine := range []string{
"netwatch-server check-cidr",
"netwatch-server check-cidr 192.0.2.0/24 198.51.100.0/24",
"netwatch-server prepare-data-dir",
"netwatch-server prepare-data-dir /data/reports /data/other",
"netwatch-server check-cdir 192.0.2.0/24",
} {
var stderr bytes.Buffer
// Main takes the command line after the program's name.
args := strings.Fields(commandLine)[1:]
code := cli.Main(&stderr, "netwatch", "test", args, failIfServerStarts(t))
if code != 2 || stderr.String() != want {
t.Errorf("%q: exit %d, stderr %q; want exit 2 and %q",
commandLine, code, stderr.String(), want)
}
}
}
// TestCheckCIDRTakesCIDR: check-cidr exits 0, writing nothing, for a
// CIDR the server takes in its TRUSTED_PROXIES.
func TestCheckCIDRTakesCIDR(t *testing.T) {
t.Parallel()
for _, cidr := range []string{"10.0.0.0/8", "192.0.2.1/32", "2001:db8::/32"} {
var stderr bytes.Buffer
code := cli.CheckCIDR(&stderr, cidr)
if code != 0 || stderr.Len() != 0 {
t.Errorf("%q: exit %d, stderr %q; want exit 0 and no output",
cidr, code, stderr.String())
}
}
}
// TestCheckCIDRRefusesOtherInput: check-cidr exits 1, with an error
// naming TRUSTED_PROXIES and the input, for anything the server would
// refuse in its TRUSTED_PROXIES, such as 1.2.3/32, which nginx would
// look up as a hostname.
func TestCheckCIDRRefusesOtherInput(t *testing.T) {
t.Parallel()
for _, cidr := range []string{
"1.2.3/32", "example.com/32", "10.0.0.0/33", "10.0.0.1", "",
} {
var stderr bytes.Buffer
code := cli.CheckCIDR(&stderr, cidr)
want := "TRUSTED_PROXIES " + strconv.Quote(cidr) + ": "
if code != 1 || !strings.HasPrefix(stderr.String(), want) {
t.Errorf("%q: exit %d, stderr %q; want exit 1 and %q...",
cidr, code, stderr.String(), want)
}
}
}
// TestPrepareDataDirTakesPathInVolume: prepare-data-dir exits 0,
// writing nothing, for a DATA_DIR below the volume, and creates it.
func TestPrepareDataDirTakesPathInVolume(t *testing.T) {
t.Parallel()
volume := t.TempDir()
dataDir := filepath.Join(volume, "reports")
var stderr bytes.Buffer
code := cli.PrepareDataDir(&stderr, volume, dataDir, currentUsername(t))
if code != 0 || stderr.Len() != 0 {
t.Errorf("exit %d, stderr %q; want exit 0 and no output",
code, stderr.String())
}
info, err := os.Stat(dataDir)
if err != nil || !info.IsDir() {
t.Errorf("DATA_DIR not created: %v", err)
}
}
// TestPrepareDataDirRefusesPathOutsideVolume: prepare-data-dir exits
// 1, with an error naming DATA_DIR and its value, for a DATA_DIR
// outside the volume.
func TestPrepareDataDirRefusesPathOutsideVolume(t *testing.T) {
t.Parallel()
volume := t.TempDir()
var stderr bytes.Buffer
code := cli.PrepareDataDir(&stderr, volume, "/etc", currentUsername(t))
want := "DATA_DIR '/etc': " + reportbuf.ErrDataDirOutsideVolume.Error() + "\n"
if code != 1 || stderr.String() != want {
t.Errorf("exit %d, stderr %q; want exit 1 and %q",
code, stderr.String(), want)
}
}
// currentUsername is the name of the user the test runs as, the only
// owner a test not run as root can give files to.
func currentUsername(t *testing.T) string {
t.Helper()
current, err := user.Current()
if err != nil {
t.Fatal(err)
}
return current.Username
}
// failIfServerStarts is what the tests give Main in place of
// cli.RunServer: it fails the test instead of starting the server.
func failIfServerStarts(t *testing.T) func(appname, version string) {
t.Helper()
return func(string, string) {
t.Error("Main started the server")
}
}
-16
View File
@@ -1,16 +0,0 @@
package cli
import "io"
// CheckCIDR exposes the check-cidr subcommand to the external tests,
// which read what it writes to stderr.
func CheckCIDR(stderr io.Writer, cidr string) int {
return checkCIDR(stderr, cidr)
}
// PrepareDataDir exposes the prepare-data-dir subcommand to the
// external tests, which give it a volume and a user of their own in
// place of /data and netwatch.
func PrepareDataDir(stderr io.Writer, volume, dataDir, username string) int {
return prepareDataDir(stderr, volume, dataDir, username)
}
+4 -4
View File
@@ -13,8 +13,8 @@ import (
"go.uber.org/fx" "go.uber.org/fx"
) )
// requireConfigError builds the config as cli.RunServer does and fails // requireConfigError builds the config as main does and fails the
// the test unless that fails with an error naming setting. It uses // test unless that fails with an error naming setting. It uses
// fx.New, because fxtest.New fails the test itself on an error. // fx.New, because fxtest.New fails the test itself on an error.
func requireConfigError(t *testing.T, setting string) { func requireConfigError(t *testing.T, setting string) {
t.Helper() t.Helper()
@@ -31,8 +31,8 @@ func requireConfigError(t *testing.T, setting string) {
} }
} }
// requireTrustedProxies builds the config as cli.RunServer does and // requireTrustedProxies builds the config as main does and fails the
// fails the test unless its TrustedProxies are want. // test unless its TrustedProxies are want.
func requireTrustedProxies(t *testing.T, want []string) { func requireTrustedProxies(t *testing.T, want []string) {
t.Helper() t.Helper()
+1 -1
View File
@@ -4,7 +4,7 @@ package globals
import "go.uber.org/fx" import "go.uber.org/fx"
//nolint:gochecknoglobals // set by cli.RunServer before fx starts //nolint:gochecknoglobals // set from main before fx starts
var ( var (
// Appname is the application name. // Appname is the application name.
Appname string Appname string
+2 -10
View File
@@ -3,17 +3,9 @@ package handlers
import "net/http" import "net/http"
// HandleHealthCheck returns a handler for the health check // HandleHealthCheck returns a handler for the health check
// endpoint. It answers 200 while the status is "ok" and 503 otherwise, // endpoint.
// so that the image's HEALTHCHECK fails.
func (s *Handlers) HandleHealthCheck() http.HandlerFunc { func (s *Handlers) HandleHealthCheck() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
resp := s.hc.Healthcheck() s.respondJSON(w, r, s.hc.Healthcheck(), http.StatusOK)
code := http.StatusOK
if resp.Status != "ok" {
code = http.StatusServiceUnavailable
}
s.respondJSON(w, r, resp, code)
} }
} }
+9 -22
View File
@@ -9,19 +9,17 @@ import (
"testing" "testing"
"time" "time"
"sneak.berlin/go/netwatch/internal/config"
"sneak.berlin/go/netwatch/internal/globals" "sneak.berlin/go/netwatch/internal/globals"
"sneak.berlin/go/netwatch/internal/handlers" "sneak.berlin/go/netwatch/internal/handlers"
"sneak.berlin/go/netwatch/internal/healthcheck" "sneak.berlin/go/netwatch/internal/healthcheck"
"sneak.berlin/go/netwatch/internal/logger" "sneak.berlin/go/netwatch/internal/logger"
"sneak.berlin/go/netwatch/internal/reportbuf"
"go.uber.org/fx/fxtest" "go.uber.org/fx/fxtest"
) )
// newStartedHandlers builds Handlers with a real health check for the // newStartedHandlers builds Handlers with a real health check for the
// server named in g, and a report buffer in a temporary directory, and // server named in g, and starts them, which records the time the
// starts them, which records the time the uptime counts from. // uptime counts from.
func newStartedHandlers(t *testing.T, g *globals.Globals) *handlers.Handlers { func newStartedHandlers(t *testing.T, g *globals.Globals) *handlers.Handlers {
t.Helper() t.Helper()
@@ -32,23 +30,14 @@ func newStartedHandlers(t *testing.T, g *globals.Globals) *handlers.Handlers {
t.Fatalf("logger: %v", err) t.Fatalf("logger: %v", err)
} }
buf, err := reportbuf.New(lc, reportbuf.Params{
Config: &config.Config{DataDir: t.TempDir(), DataDirMaxBytes: 1 << 20},
Logger: log,
})
if err != nil {
t.Fatalf("report buffer: %v", err)
}
hc, err := healthcheck.New(lc, hc, err := healthcheck.New(lc,
healthcheck.Params{Buffer: buf, Globals: g, Logger: log}) healthcheck.Params{Globals: g, Logger: log})
if err != nil { if err != nil {
t.Fatalf("health check: %v", err) t.Fatalf("health check: %v", err)
} }
h, err := handlers.New(lc, handlers.Params{ h, err := handlers.New(lc,
Buffer: buf, Globals: g, Healthcheck: hc, Logger: log, handlers.Params{Globals: g, Healthcheck: hc, Logger: log})
})
if err != nil { if err != nil {
t.Fatalf("handlers: %v", err) t.Fatalf("handlers: %v", err)
} }
@@ -59,12 +48,10 @@ func newStartedHandlers(t *testing.T, g *globals.Globals) *handlers.Handlers {
return h return h
} }
// TestHandleHealthCheck checks the health check's answer at start, when // TestHandleHealthCheck checks the health check's answer: 200, a JSON
// no write of a report file has failed: 200, a JSON content type, and a // content type, and a JSON object with exactly the fields of
// JSON object with exactly the fields of healthcheck.HealthcheckResponse, // healthcheck.HealthcheckResponse, carrying this server's name and
// carrying "status":"ok", this server's name and version and an uptime // version and an uptime counted from its start.
// counted from its start. The answer while writes fail is checked in
// internal/reportbuf, whose tests can make a write fail.
func TestHandleHealthCheck(t *testing.T) { func TestHandleHealthCheck(t *testing.T) {
t.Parallel() t.Parallel()
+2 -10
View File
@@ -10,7 +10,6 @@ import (
"sneak.berlin/go/netwatch/internal/config" "sneak.berlin/go/netwatch/internal/config"
"sneak.berlin/go/netwatch/internal/globals" "sneak.berlin/go/netwatch/internal/globals"
"sneak.berlin/go/netwatch/internal/logger" "sneak.berlin/go/netwatch/internal/logger"
"sneak.berlin/go/netwatch/internal/reportbuf"
"go.uber.org/fx" "go.uber.org/fx"
) )
@@ -19,7 +18,6 @@ import (
type Params struct { type Params struct {
fx.In fx.In
Buffer *reportbuf.Buffer
Config *config.Config Config *config.Config
Globals *globals.Globals Globals *globals.Globals
Logger *logger.Logger Logger *logger.Logger
@@ -70,18 +68,12 @@ func New(
return s, nil return s, nil
} }
// Healthcheck returns the current health status of the application: // Healthcheck returns the current health status of the application.
// "ok", or "error" while report files cannot be written.
func (s *Healthcheck) Healthcheck() *HealthcheckResponse { func (s *Healthcheck) Healthcheck() *HealthcheckResponse {
status := "ok"
if s.params.Buffer.LastWriteFailed() {
status = "error"
}
return &HealthcheckResponse{ return &HealthcheckResponse{
Appname: s.params.Globals.Appname, Appname: s.params.Globals.Appname,
Now: time.Now().UTC().Format(time.RFC3339Nano), Now: time.Now().UTC().Format(time.RFC3339Nano),
Status: status, Status: "ok",
UptimeHuman: s.uptime().String(), UptimeHuman: s.uptime().String(),
UptimeSeconds: int64(s.uptime().Seconds()), UptimeSeconds: int64(s.uptime().Seconds()),
Version: s.params.Globals.Version, Version: s.params.Globals.Version,
+43 -63
View File
@@ -16,6 +16,7 @@ import (
"log/slog" "log/slog"
"os" "os"
"path/filepath" "path/filepath"
"slices"
"strings" "strings"
"sync" "sync"
"sync/atomic" "sync/atomic"
@@ -72,9 +73,11 @@ type Buffer struct {
// except in tests that hold the write open or make it fail. // except in tests that hold the write open or make it fail.
fileCreated func(f *os.File) fileCreated func(f *os.File)
// files are the report files that may be deleted to make room, // files are the report files that may be deleted to make room,
// oldest first: those in dataDir at start, in name order, then // in name order, which is oldest first: those in dataDir at
// each one this buffer writes, once it is complete. A file still // start, and each one this buffer writes, put in at its place by
// being written is not among them. filesBytes is their total size. // name once it is complete, even when an older file's write
// completes after a newer one's. A file still being written is
// not among them. filesBytes is their total size.
files []reportFile files []reportFile
filesBytes int64 filesBytes int64
log *slog.Logger log *slog.Logger
@@ -87,10 +90,6 @@ type Buffer struct {
// millisecond still get different names. // millisecond still get different names.
seq atomic.Int64 seq atomic.Int64
stopOnce sync.Once stopOnce sync.Once
// thresholdWriteWaiting is set while a write Append started at
// flushSizeThreshold waits for writeMu; Append starts no other
// while it is set.
thresholdWriteWaiting atomic.Bool
// usedBytes is what Append checks against maxBytes: the size // usedBytes is what Append checks against maxBytes: the size
// of the report files in dataDir, plus the reports waiting to // of the report files in dataDir, plus the reports waiting to
// be written to one at their uncompressed size. // be written to one at their uncompressed size.
@@ -100,11 +99,6 @@ type Buffer struct {
// refuses reports, and the reports of that write wait in buf for // refuses reports, and the reports of that write wait in buf for
// the next flush. // the next flush.
writeErr error writeErr error
// writeMu is held across each write of a report file, so writes
// never overlap: a failed write puts its reports back in buf before
// the next write takes them, which keeps them in the order they
// were appended.
writeMu sync.Mutex
// writers are the goroutines that write report files: flushLoop, // writers are the goroutines that write report files: flushLoop,
// and each one Append starts at flushSizeThreshold. OnStop waits // and each one Append starts at flushSizeThreshold. OnStop waits
// for them before the final flush. // for them before the final flush.
@@ -213,9 +207,8 @@ func New(
// write of a report file failed, until a write succeeds again. If the // write of a report file failed, until a write succeeds again. If the
// line would take usedBytes past maxBytes, the oldest report files are // line would take usedBytes past maxBytes, the oldest report files are
// deleted to make room; it stores nothing and returns ErrFull if that // deleted to make room; it stores nothing and returns ErrFull if that
// cannot make room. If the buffer reaches the size threshold, it starts // cannot make room. If the buffer reaches the size threshold, it is
// a write of the reports in the background (see flushAtThreshold), // drained and written to disk asynchronously.
// unless one it started is still waiting.
func (b *Buffer) Append(v any) error { func (b *Buffer) Append(v any) error {
line, err := json.Marshal(v) line, err := json.Marshal(v)
if err != nil { if err != nil {
@@ -245,9 +238,18 @@ func (b *Buffer) Append(v any) error {
b.buf.Write(line) b.buf.Write(line)
b.buf.WriteByte('\n') b.buf.WriteByte('\n')
if b.buf.Len() >= flushSizeThreshold && if b.buf.Len() >= flushSizeThreshold {
b.thresholdWriteWaiting.CompareAndSwap(false, true) { data := b.drainBuf()
b.writers.Go(b.flushAtThreshold) b.mu.Unlock()
b.writers.Go(func() {
writeErr := b.writeFile(data)
if writeErr != nil {
b.log.Error("flush reports failed", "error", writeErr)
}
})
return nil
} }
b.mu.Unlock() b.mu.Unlock()
@@ -255,16 +257,6 @@ func (b *Buffer) Append(v any) error {
return nil return nil
} }
// LastWriteFailed reports whether the last write of a report file to
// finish failed: false at start until a write fails, and false again
// once a write succeeds. While it is true, Append refuses reports.
func (b *Buffer) LastWriteFailed() bool {
b.mu.Lock()
defer b.mu.Unlock()
return b.writeErr != nil
}
// deleteOldestFiles deletes report files, oldest first, until n more // deleteOldestFiles deletes report files, oldest first, until n more
// bytes fit under maxBytes. It deletes none when the reports waiting // bytes fit under maxBytes. It deletes none when the reports waiting
// to be written leave no room for n even with every file gone, since // to be written leave no room for n even with every file gone, since
@@ -315,28 +307,21 @@ func (b *Buffer) flushLoop() {
} }
} }
// flushLocked waits for any write of a report file under way, then // flushLocked acquires the lock, drains the buffer, and
// writes the reports waiting in the buffer to a new one. // writes the data to a compressed file.
func (b *Buffer) flushLocked() error { func (b *Buffer) flushLocked() error {
b.writeMu.Lock() b.mu.Lock()
defer b.writeMu.Unlock()
return b.writeFile() if b.buf.Len() == 0 {
} b.mu.Unlock()
// flushAtThreshold is the write Append starts when the buffer reaches return nil
// flushSizeThreshold. Once any write under way is done, it lets Append
// start another and writes the reports waiting in the buffer.
func (b *Buffer) flushAtThreshold() {
b.writeMu.Lock()
defer b.writeMu.Unlock()
b.thresholdWriteWaiting.Store(false)
err := b.writeFile()
if err != nil {
b.log.Error("flush reports failed", "error", err)
} }
data := b.drainBuf()
b.mu.Unlock()
return b.writeFile(data)
} }
// drainBuf copies the buffer contents and resets it. // drainBuf copies the buffer contents and resets it.
@@ -349,22 +334,10 @@ func (b *Buffer) drainBuf() []byte {
return data return data
} }
// writeFile takes the reports waiting in the buffer, if any, and writes // writeFile writes data, reports drained from the buffer, to a new
// them to a new timestamped zstd-compressed JSONL file in the data // timestamped zstd-compressed JSONL file in the data directory. If the
// directory. If the write fails, it puts the reports back in the // write fails, it puts the reports back in the buffer.
// buffer. The caller must hold b.writeMu. func (b *Buffer) writeFile(data []byte) error {
func (b *Buffer) writeFile() error {
b.mu.Lock()
if b.buf.Len() == 0 {
b.mu.Unlock()
return nil
}
data := b.drainBuf()
b.mu.Unlock()
// The timestamp comes first, so the names sort by time; the number // The timestamp comes first, so the names sort by time; the number
// after it tells apart files named in the same millisecond. // after it tells apart files named in the same millisecond.
ts := b.now().UTC().Format("2006-01-02T15-04-05.000Z") ts := b.now().UTC().Format("2006-01-02T15-04-05.000Z")
@@ -395,7 +368,14 @@ func (b *Buffer) writeFile() error {
b.usedBytes -= int64(len(data)) b.usedBytes -= int64(len(data))
b.usedBytes += size b.usedBytes += size
b.filesBytes += size b.filesBytes += size
b.files = append(b.files, reportFile{name: name, size: size})
// At its place by name, not at the end: another write, of a newer
// file, may have completed while this one was being written.
i, _ := slices.BinarySearchFunc(b.files, name,
func(f reportFile, target string) int {
return strings.Compare(f.name, target)
})
b.files = slices.Insert(b.files, i, reportFile{name: name, size: size})
return nil return nil
} }
+114 -332
View File
@@ -334,9 +334,9 @@ func stopWhileHeld(
return <-stopped return <-stopped
} }
// startApp starts a Buffer through fx, as cli.RunServer does, with the // startApp starts a Buffer through fx, as main does, with the DATA_DIR
// DATA_DIR and DATA_DIR_MAX_BYTES the calling test has set, and returns // and DATA_DIR_MAX_BYTES the calling test has set, and returns it with
// it with the app, for tests that stop the app themselves. // the app, for tests that stop the app themselves.
func startApp(t *testing.T) (*reportbuf.Buffer, *fxtest.App) { func startApp(t *testing.T) (*reportbuf.Buffer, *fxtest.App) {
t.Helper() t.Helper()
@@ -358,8 +358,8 @@ func startApp(t *testing.T) (*reportbuf.Buffer, *fxtest.App) {
return buf, app return buf, app
} }
// startBuffer starts a Buffer through fx, as cli.RunServer does, with // startBuffer starts a Buffer through fx, as main does, with the
// the DATA_DIR and DATA_DIR_MAX_BYTES the calling test has set. // DATA_DIR and DATA_DIR_MAX_BYTES the calling test has set.
func startBuffer(t *testing.T) *reportbuf.Buffer { func startBuffer(t *testing.T) *reportbuf.Buffer {
t.Helper() t.Helper()
@@ -581,62 +581,6 @@ func TestWrittenFilesDeletedToMakeRoom(t *testing.T) {
} }
} }
// TestWrittenFilesDeletedOldestFirst writes three report files, then
// appends a report that fits only once one of them is deleted. The
// first one written must be the one deleted.
func TestWrittenFilesDeletedOldestFirst(t *testing.T) {
const maxBytes = 1000
report := map[string]string{"id": "written"}
t.Setenv("DATA_DIR", t.TempDir())
t.Setenv("DATA_DIR_MAX_BYTES", strconv.Itoa(maxBytes))
buf := startBuffer(t)
var written []string
buf.OnFileCreated(func(f *os.File) { written = append(written, f.Name()) })
for range 3 {
err := buf.Append(report)
if err != nil {
t.Fatalf("append report: %v", err)
}
err = buf.Flush()
if err != nil {
t.Fatalf("flush: %v", err)
}
}
// The three files hold the same report, so they are the same size.
info, err := os.Stat(written[0])
if err != nil {
t.Fatalf("stat %s: %v", written[0], err)
}
// A report that fits beside two of the files, so deleting one makes
// exactly the room it needs.
pad := maxBytes - 2*int(info.Size()) -
lineBytes(t, map[string]string{"id": ""})
err = buf.Append(map[string]string{"id": strings.Repeat("a", pad)})
if err != nil {
t.Fatalf("report that needs room: %v", err)
}
if exists(t, written[0]) {
t.Fatal("first report file written kept when room was needed")
}
for _, path := range written[1:] {
if !exists(t, path) {
t.Fatalf("%s deleted before the first report file written", path)
}
}
}
// TestFailedDeletionStillCounts makes deleting the oldest report file // TestFailedDeletionStillCounts makes deleting the oldest report file
// fail. It is still there, so it still takes room, and the next oldest // fail. It is still there, so it still takes room, and the next oldest
// is deleted in its place. // is deleted in its place.
@@ -722,16 +666,17 @@ func TestFileDeletedByHandFreesRoom(t *testing.T) {
} }
} }
// TestFileBeingWrittenIsNeverDeleted holds the write of a report file // TestFileBeingWrittenIsNeverDeleted holds the write of one report file
// open under a cap with room for two reports, then appends a second // open while a second write completes, then sends a report that needs
// report and a third that needs room. The file being written is the // room. Deleting either file would make it, and the one being written is
// only report file, and it is not deleted to make room, so the third // the older, but only the complete one may be deleted. Once the first
// report is refused. Once the write is complete, its file is deleted // write is complete, its file is deleted when room is needed.
// when room is needed.
func TestFileBeingWrittenIsNeverDeleted(t *testing.T) { func TestFileBeingWrittenIsNeverDeleted(t *testing.T) {
report := map[string]string{"id": "writing"} report := map[string]string{"id": "writing"}
t.Setenv("DATA_DIR", t.TempDir()) t.Setenv("DATA_DIR", t.TempDir())
// Room for two reports waiting to be written, but not for two
// beside a report file.
t.Setenv("DATA_DIR_MAX_BYTES", strconv.Itoa(2*lineBytes(t, report))) t.Setenv("DATA_DIR_MAX_BYTES", strconv.Itoa(2*lineBytes(t, report)))
buf := startBuffer(t) buf := startBuffer(t)
@@ -756,25 +701,36 @@ func TestFileBeingWrittenIsNeverDeleted(t *testing.T) {
writing := <-created writing := <-created
// Errorf, not Fatalf, until the write is released, so that a // Only the first write is held; the second goes through, and
// so do the writes after it, the final one at stop included.
var complete string
buf.OnFileCreated(func(f *os.File) { complete = f.Name() })
// Errorf, not Fatalf, until the first write is released, so that a
// failure here does not leave it held. // failure here does not leave it held.
err = buf.Append(report) err = buf.Append(report)
if err != nil { if err != nil {
t.Errorf("second report: %v", err) t.Errorf("second report: %v", err)
} }
err = buf.Flush()
if err != nil {
t.Errorf("flush of the second report: %v", err)
}
err = buf.Append(report) err = buf.Append(report)
if !errors.Is(err, reportbuf.ErrFull) { if err != nil {
t.Errorf("report that needs room: error = %v, want ErrFull", err) t.Errorf("report that needs room: %v", err)
} }
if !exists(t, writing) { if !exists(t, writing) {
t.Error("report file deleted while it was being written") t.Error("report file deleted while it was being written")
} }
// Only this write is held; the writes after it, the final one at if exists(t, complete) {
// stop included, go through. t.Error("complete report file kept, though room was needed")
buf.OnFileCreated(func(*os.File) {}) }
close(release) close(release)
@@ -783,11 +739,9 @@ func TestFileBeingWrittenIsNeverDeleted(t *testing.T) {
t.Fatalf("flush of the first report: %v", err) t.Fatalf("flush of the first report: %v", err)
} }
// The second report still waits to be written, so this one fits
// only once the file just written is deleted.
err = buf.Append(report) err = buf.Append(report)
if err != nil { if err != nil {
t.Fatalf("report after the write was complete: %v", err) t.Fatalf("report after the first write was complete: %v", err)
} }
if exists(t, writing) { if exists(t, writing) {
@@ -929,82 +883,6 @@ func TestFailedWriteKeepsReportsAppendedDuringIt(t *testing.T) {
} }
} }
// TestFailedWritesKeepReportsInOrder holds a write open, then appends a
// report that takes the buffer to FlushSizeThreshold, so that Append
// starts a second write. The held write fails first, then the second,
// which waited for it and took both reports. The next flush writes them
// in the order they were appended, the held write's first.
func TestFailedWritesKeepReportsInOrder(t *testing.T) {
dir := t.TempDir()
t.Setenv("DATA_DIR", dir)
buf := startBuffer(t)
created := make(chan struct{})
release := make(chan struct{})
// Each write is held until release is closed, then fails, as its
// file is closed under it.
buf.OnFileCreated(func(f *os.File) {
created <- struct{}{}
<-release
_ = f.Close()
})
err := buf.Append(map[string]string{"id": "held"})
if err != nil {
t.Fatalf("report of the held write: %v", err)
}
flushed := make(chan error)
go func() { flushed <- buf.Flush() }()
<-created
big := strings.Repeat("a", reportbuf.FlushSizeThreshold)
// Errorf, not Fatalf, until the held write is released, so that a
// failure here does not leave it held.
err = buf.Append(map[string]string{"id": big})
if err != nil {
t.Errorf("report that starts the second write: %v", err)
}
close(release)
err = <-flushed
if err == nil {
t.Error("held write succeeded, though its file was closed under it")
}
// The second write has created its file, so it has taken the
// reports.
<-created
// Writes from here on, the final one at stop included, succeed.
buf.OnFileCreated(func(*os.File) {})
// Flush waits for the second write to fail before it writes.
err = buf.Flush()
if err != nil {
t.Fatalf("flush after the failed writes: %v", err)
}
files, err := readReportFiles(dir)
if err != nil {
t.Fatalf("read report files: %v", err)
}
want := `{"id":"held"}` + "\n" + `{"id":"` + big + `"}` + "\n"
if len(files) != 1 || files[0] != want {
t.Fatalf("%d report files, want one holding the held write's "+
"report, then the second one", len(files))
}
}
// TestReportsRefusedWhileWritesFail posts reports through the report // TestReportsRefusedWhileWritesFail posts reports through the report
// handler, as the page does. Once a write has failed, a report is // handler, as the page does. Once a write has failed, a report is
// refused with 500 until a write succeeds again; the reports taken // refused with 500 until a write succeeds again; the reports taken
@@ -1074,54 +952,9 @@ func TestReportsRefusedWhileWritesFail(t *testing.T) {
} }
} }
// TestHealthCheckFailsWhileWritesFail asks the health check handler
// for its answer around a failed write. It answers 200 with
// "status":"ok" at start, 503 with "status":"error" from the failed
// write until a write succeeds, and 200 with "status":"ok" again after.
func TestHealthCheckFailsWhileWritesFail(t *testing.T) {
dir := t.TempDir()
t.Setenv("DATA_DIR", dir)
buf, h := startBufferWithHandlers(t)
checkHealthCheck(t, h, "at start", http.StatusOK, "ok")
// A flush writes a file only when a report is waiting.
err := buf.Append(map[string]string{"id": "kept"})
if err != nil {
t.Fatalf("append: %v", err)
}
// With the data directory gone, the write has nowhere to go.
err = os.Remove(dir)
if err != nil {
t.Fatalf("remove data dir: %v", err)
}
err = buf.Flush()
if err == nil {
t.Fatal("flush succeeded with the data directory gone")
}
checkHealthCheck(t, h, "after the failed write",
http.StatusServiceUnavailable, "error")
err = os.Mkdir(dir, 0o750)
if err != nil {
t.Fatalf("make data dir: %v", err)
}
err = buf.Flush()
if err != nil {
t.Fatalf("flush with the data directory back: %v", err)
}
checkHealthCheck(t, h, "after a write succeeded", http.StatusOK, "ok")
}
// startBufferWithHandlers starts a Buffer and the Handlers that store // startBufferWithHandlers starts a Buffer and the Handlers that store
// reports in it through fx, as cli.RunServer does, with the DATA_DIR // reports in it through fx, as main does, with the DATA_DIR the calling
// the calling test has set. // test has set.
func startBufferWithHandlers( func startBufferWithHandlers(
t *testing.T, t *testing.T,
) (*reportbuf.Buffer, *handlers.Handlers) { ) (*reportbuf.Buffer, *handlers.Handlers) {
@@ -1166,39 +999,6 @@ func postReport(t *testing.T, h *handlers.Handlers, clientID string) int {
return rec.Code return rec.Code
} }
// checkHealthCheck asks the health check handler for its answer, and
// fails the test, saying when it asked, unless the answer has status
// code wantCode and a "status" field of wantStatus.
func checkHealthCheck(
t *testing.T,
h *handlers.Handlers,
when string,
wantCode int,
wantStatus string,
) {
t.Helper()
rec := httptest.NewRecorder()
req := httptest.NewRequestWithContext(t.Context(),
http.MethodGet, "/.well-known/healthcheck", http.NoBody)
h.HandleHealthCheck().ServeHTTP(rec, req)
var body struct {
Status string `json:"status"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Fatalf("health check %s: body not JSON: %v (%q)",
when, err, rec.Body.String())
}
if rec.Code != wantCode || body.Status != wantStatus {
t.Fatalf("health check %s: %d with status %q, want %d with %q",
when, rec.Code, body.Status, wantCode, wantStatus)
}
}
// TestConcurrentAppendsStopAtCap appends from many goroutines at once // TestConcurrentAppendsStopAtCap appends from many goroutines at once
// with room for exactly roomFor reports: exactly that many must be // with room for exactly roomFor reports: exactly that many must be
// taken, which holds only if Append checks and counts each report // taken, which holds only if Append checks and counts each report
@@ -1350,106 +1150,6 @@ func TestFlushAtSizeThreshold(t *testing.T) {
} }
} }
// TestEachSizeThresholdStartsAWrite reaches FlushSizeThreshold twice,
// with no call to Flush and the periodic flush a minute away. Append
// must start a write each time, so that the reports end up in two
// report files. The test runs in a synctest bubble, so that
// synctest.Wait returns once each write is done.
func TestEachSizeThresholdStartsAWrite(t *testing.T) {
dir := t.TempDir()
t.Setenv("DATA_DIR", dir)
synctest.Test(t, func(t *testing.T) {
buf := startBuffer(t)
var appended string
for range 2 {
appended = appendToThreshold(t, buf)
synctest.Wait()
}
files, err := readReportFiles(dir)
if err != nil {
t.Fatalf("read report files: %v", err)
}
if !slices.Equal(files, []string{appended, appended}) {
t.Fatalf("%d report files, want two, each holding the %d "+
"bytes appended to reach the threshold",
len(files), len(appended))
}
})
}
// TestAppendStartsOneWaitingWrite holds a write open, then appends two
// reports of FlushSizeThreshold bytes each. Append starts a write at
// the first, which waits for the held one, and none at the second,
// since that write is still waiting. Every write fails, so a write
// started at the second report would find the reports back in the
// buffer and try them again: once the held write is released, two
// writes must have been started, the held one and one more.
func TestAppendStartsOneWaitingWrite(t *testing.T) {
t.Setenv("DATA_DIR", t.TempDir())
synctest.Test(t, func(t *testing.T) {
buf := startBuffer(t)
var writes atomic.Int64
release := make(chan struct{})
// Each write is held until release is closed, then fails, as its
// file is closed under it.
buf.OnFileCreated(func(f *os.File) {
writes.Add(1)
<-release
_ = f.Close()
})
err := buf.Append(map[string]string{"id": "one-waiting"})
if err != nil {
t.Fatalf("report of the held write: %v", err)
}
go func() { _ = buf.Flush() }()
// Wait returns once the flush's write is held.
synctest.Wait()
big := map[string]string{
"id": strings.Repeat("a", reportbuf.FlushSizeThreshold),
}
// Errorf, not Fatalf, until the held write is released, so that a
// failure here does not leave it held.
for range 2 {
err = buf.Append(big)
if err != nil {
t.Errorf("report of FlushSizeThreshold bytes: %v", err)
}
}
close(release)
// Wait returns once every write started is done.
synctest.Wait()
started := writes.Load()
// Writes from here on, the final one at stop included, succeed.
buf.OnFileCreated(func(*os.File) {})
if started != 2 {
t.Fatalf("%d writes started, want 2: the held one and one "+
"at the threshold", started)
}
})
}
// appendToThreshold appends reports to buf until it holds // appendToThreshold appends reports to buf until it holds
// FlushSizeThreshold bytes, so that the last append starts writing them // FlushSizeThreshold bytes, so that the last append starts writing them
// to a report file, and returns the lines appended. // to a report file, and returns the lines appended.
@@ -1595,3 +1295,85 @@ func hasReportFile(t *testing.T, dir string) bool {
return false return false
} }
// TestFilesDeletedOldestFirstWhenWritesOverlap holds the write of an
// older report file open until a newer one's write completes, then
// releases it. When room is needed, the older file is deleted first,
// though its write was the last to complete.
func TestFilesDeletedOldestFirstWhenWritesOverlap(t *testing.T) {
const maxBytes = 1000
report := map[string]string{"id": "overlap"}
t.Setenv("DATA_DIR", t.TempDir())
t.Setenv("DATA_DIR_MAX_BYTES", strconv.Itoa(maxBytes))
buf := startBuffer(t)
created := make(chan string)
release := make(chan struct{})
buf.OnFileCreated(func(f *os.File) {
created <- f.Name()
<-release
})
err := buf.Append(report)
if err != nil {
t.Fatalf("older report: %v", err)
}
flushed := make(chan error)
go func() { flushed <- buf.Flush() }()
older := <-created
// Only the older write is held; the newer one goes through, and
// so do the writes after it, the final one at stop included.
var newer string
buf.OnFileCreated(func(f *os.File) { newer = f.Name() })
// Errorf, not Fatalf, until the older write is released, so that a
// failure here does not leave it held.
err = buf.Append(report)
if err != nil {
t.Errorf("newer report: %v", err)
}
err = buf.Flush()
if err != nil {
t.Errorf("flush of the newer report: %v", err)
}
close(release)
err = <-flushed
if err != nil {
t.Fatalf("flush of the older report: %v", err)
}
info, err := os.Stat(newer)
if err != nil {
t.Fatalf("stat %s: %v", newer, err)
}
// A report that fits beside the newer file alone, so deleting the
// older one makes exactly the room it needs.
pad := maxBytes - int(info.Size()) - lineBytes(t, map[string]string{"id": ""})
err = buf.Append(map[string]string{"id": strings.Repeat("a", pad)})
if err != nil {
t.Fatalf("report that needs room: %v", err)
}
if exists(t, older) {
t.Fatal("older report file kept when room was needed")
}
if !exists(t, newer) {
t.Fatal("newer report file deleted before the older one")
}
}
+3 -3
View File
@@ -22,9 +22,9 @@ import (
"go.uber.org/fx/fxtest" "go.uber.org/fx/fxtest"
) )
// newServer builds a Server from the same constructors as // newServer builds a Server from the same constructors as main,
// cli.RunServer, configured from the environment. It is never started, // configured from the environment. It is never started, so nothing
// so nothing listens. // listens.
func newServer(t *testing.T) *server.Server { func newServer(t *testing.T) *server.Server {
t.Helper() t.Helper()
+1 -1
View File
@@ -79,7 +79,7 @@ func New(
s.SetupRoutes() s.SetupRoutes()
s.httpServer = s.newHTTPServer() s.httpServer = s.newHTTPServer()
go func() { s.listenAndServe() }() go s.listenAndServe()
return nil return nil
}, },
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "netwatch", "name": "netwatch",
"private": true, "version": "1.0.0",
"description": "Real-time network latency monitor SPA", "description": "Real-time network latency monitor SPA",
"type": "module", "type": "module",
"scripts": { "scripts": {
+2 -2
View File
@@ -23,8 +23,8 @@ main() {
"" | -* | *[[:space:]]*) usage ;; "" | -* | *[[:space:]]*) usage ;;
esac esac
cd "$ROOT" cd "$ROOT"
"$ROOT/script/yarn" add --dev "$1" yarn add --dev "$1"
"$ROOT/script/yarn" install --frozen-lockfile yarn install --frozen-lockfile
} }
main "$@" main "$@"
+14 -55
View File
@@ -3,14 +3,11 @@
# this repo. Idempotent: every install is guarded by a check so already # this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew, # installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes nothing is present. Node is # or apk (detected in that order); assumes nothing is present. Node is
# used directly only if it has the major version of NODE_VERSION and is # used directly if it is at least NODE_MIN_VERSION; otherwise it is
# at least NODE_MIN_VERSION; otherwise NODE_VERSION is installed via nvm # installed at a pinned version via nvm (installing nvm itself first,
# (installing nvm itself first, from a hash-verified release archive, # from a hash-verified release archive, never curl | sh). Go, with its
# never curl | sh). Yarn is used directly only if it is exactly # gofmt, is used directly only if it is exactly GO_VERSION; otherwise
# YARN_VERSION; otherwise that version is prepared through corepack, # that release is installed from its hash-verified archive.
# or installed with npm where there is no corepack.
# Go, with its gofmt, is used directly only if it is exactly GO_VERSION;
# otherwise that release is installed from its hash-verified archive.
# #
# What this script installs outside the system package manager lives # What this script installs outside the system package manager lives
# under $HOME and is linked into ~/.local/bin, where make and the git # under $HOME and is linked into ~/.local/bin, where make and the git
@@ -145,63 +142,32 @@ ensure_nvm() {
rm -rf "$tmp" rm -rf "$tmp"
} }
# node_ok: the node on PATH has the major version of NODE_VERSION and is # node_ok: the node on PATH is at least NODE_MIN_VERSION. node itself
# at least NODE_MIN_VERSION. Only the major is compared with the pin, # compares the two: major, then minor, then patch.
# as REPO_POLICIES.md asks. node itself compares the versions: major,
# then minor, then patch. A node that fails does not pass.
# script/bootstrap and script/yarn carry this function unchanged.
node_ok() { node_ok() {
if ! command -v node >/dev/null 2>&1; then return 1; fi if missing node; then return 1; fi
node -e ' node -e '
const have = process.versions.node.split(".").map(Number); const have = process.versions.node.split(".").map(Number);
const want = process.argv[1].split(".").map(Number); const want = process.argv[1].split(".").map(Number);
const pinned = process.argv[2].split(".").map(Number);
if (have[0] !== pinned[0]) process.exit(1);
for (let i = 0; i < 3; i++) { for (let i = 0; i < 3; i++) {
if (have[i] !== want[i]) process.exit(have[i] > want[i] ? 0 : 1); if (have[i] !== want[i]) process.exit(have[i] > want[i] ? 0 : 1);
} }
' "$NODE_MIN_VERSION" "$NODE_VERSION" ' "$NODE_MIN_VERSION"
} }
# ensure_node: unless node_ok, install NODE_VERSION and link its node. # ensure_node: unless node_ok, install NODE_VERSION and link its node.
# Afterwards node is looked up through PATH again and must pass node_ok.
ensure_node() { ensure_node() {
if node_ok; then if node_ok; then return 0; fi
echo "bootstrap: using node $(node --version)"
return 0
fi
ensure_nvm ensure_nvm
nvm_sh "nvm install $NODE_VERSION" nvm_sh "nvm install $NODE_VERSION"
link_bin "$HOME/.nvm/versions/node/v$NODE_VERSION/bin/node" node link_bin "$HOME/.nvm/versions/node/v$NODE_VERSION/bin/node" node
hash -r
if ! node_ok; then
echo "bootstrap: after installing node $NODE_VERSION through nvm," >&2
echo " the node on PATH, $(command -v node), is not it" >&2
exit 1
fi
echo "bootstrap: installed node $(node --version)"
} }
# yarn_ok: yarn --version on PATH prints exactly YARN_VERSION, compared # ensure_yarn: corepack writes its shims (pnpm and yarnpkg as well as
# over the whole version. A yarn that fails or prints anything else does # yarn) into $TOOLCHAIN rather than next to itself, and the npm fallback
# not pass. # installs there too; only yarn is linked.
yarn_ok() {
if missing yarn; then return 1; fi
version="$(yarn --version 2>/dev/null)" || return 1
[ "$version" = "$YARN_VERSION" ]
}
# ensure_yarn: unless yarn_ok, prepare YARN_VERSION and link its yarn.
# corepack writes its shims (pnpm and yarnpkg as well as yarn) into
# $TOOLCHAIN rather than next to itself, and the npm fallback installs
# there too; only yarn is linked. Afterwards yarn is looked up through
# PATH again and must pass yarn_ok, so another yarn that hides the link
# stops bootstrap.
ensure_yarn() { ensure_yarn() {
if yarn_ok; then if ! missing yarn; then return 0; fi
echo "bootstrap: using yarn $(yarn --version)"
return 0
fi
shims="$TOOLCHAIN/corepack-shims" shims="$TOOLCHAIN/corepack-shims"
mkdir -p "$shims" mkdir -p "$shims"
if ! missing corepack; then if ! missing corepack; then
@@ -216,13 +182,6 @@ ensure_yarn() {
shims="$TOOLCHAIN/npm-global/bin" shims="$TOOLCHAIN/npm-global/bin"
fi fi
link_bin "$shims/yarn" yarn link_bin "$shims/yarn" yarn
hash -r
if ! yarn_ok; then
echo "bootstrap: after installing yarn $YARN_VERSION in $shims," >&2
echo " the yarn on PATH, $(command -v yarn), does not report it" >&2
exit 1
fi
echo "bootstrap: installed yarn $(yarn --version)"
} }
# go_ok: the go on PATH has its gofmt beside it (a Go release ships the # go_ok: the go on PATH has its gofmt beside it (a Go release ships the
+4 -5
View File
@@ -8,12 +8,11 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
"$ROOT/script/yarn" build yarn build
# VERSION comes from the environment. Unset or empty, it is # VERSION comes from the environment. Unset or empty, it is git
# `git describe --tags --always --dirty`, as in script/docker, or # describe, or "dev" where there is no git or no repository history.
# "dev" where there is no git or no repository history. version="${VERSION:-$(git describe --always --dirty 2>/dev/null || echo dev)}"
version="${VERSION:-$(git describe --tags --always --dirty 2>/dev/null || echo dev)}"
CGO_ENABLED=0 go build -trimpath \ CGO_ENABLED=0 go build -trimpath \
-ldflags "-s -w -X main.Version=$version" \ -ldflags "-s -w -X main.Version=$version" \
+1 -1
View File
@@ -7,7 +7,7 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
"$ROOT/script/yarn" dev yarn dev
} }
main "$@" main "$@"
+1 -1
View File
@@ -10,7 +10,7 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
"$ROOT/script/yarn" prettier --write '**/*.{js,css,html,md}' yarn prettier --write '**/*.{js,css,html,md}'
} }
main "$@" main "$@"
+1 -1
View File
@@ -7,7 +7,7 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
"$ROOT/script/yarn" prettier --check '**/*.{js,css,html,md}' yarn prettier --check '**/*.{js,css,html,md}'
} }
main "$@" main "$@"
+3 -5
View File
@@ -14,14 +14,12 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
NODE_OPTIONS=--test-reporter=dot \ NODE_OPTIONS=--test-reporter=dot timeout 90 yarn --silent run test || {
timeout 90 "$ROOT/script/yarn" --silent run test || {
echo "--- Rerunning with every test listed for details ---" echo "--- Rerunning with every test listed for details ---"
NODE_OPTIONS=--test-reporter=spec \ NODE_OPTIONS=--test-reporter=spec timeout 90 yarn --silent run test
timeout 90 "$ROOT/script/yarn" --silent run test
exit 1 exit 1
} }
timeout 30 "$ROOT/script/yarn" build timeout 30 yarn build
} }
main "$@" main "$@"
-55
View File
@@ -1,55 +0,0 @@
#!/bin/sh
# script/yarn: run yarn with the given arguments under the node
# script/bootstrap would use. Every script that runs yarn on the host
# runs it through this one, except script/bootstrap, which picks its
# own node and yarn.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the pins in script/bootstrap.
NODE_VERSION="22.17.0"
NODE_MIN_VERSION="22.13.0"
# node_ok: the node on PATH has the major version of NODE_VERSION and is
# at least NODE_MIN_VERSION. Only the major is compared with the pin,
# as REPO_POLICIES.md asks. node itself compares the versions: major,
# then minor, then patch. A node that fails does not pass.
# script/bootstrap and script/yarn carry this function unchanged.
node_ok() {
if ! command -v node >/dev/null 2>&1; then return 1; fi
node -e '
const have = process.versions.node.split(".").map(Number);
const want = process.argv[1].split(".").map(Number);
const pinned = process.argv[2].split(".").map(Number);
if (have[0] !== pinned[0]) process.exit(1);
for (let i = 0; i < 3; i++) {
if (have[i] !== want[i]) process.exit(have[i] > want[i] ? 0 : 1);
}
' "$NODE_MIN_VERSION" "$NODE_VERSION"
}
# Run the yarn script/bootstrap installed: the yarn on PATH when the
# node on PATH passes node_ok, and otherwise yarn under the pinned node
# in nvm. nvm is a bash script, hence the subshell.
run_yarn() {
if node_ok && command -v yarn >/dev/null 2>&1; then
exec yarn "$@"
fi
if node_ok || [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "script/yarn: no yarn; run script/bootstrap first" >&2
exit 1
fi
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
}
main() {
cd "$ROOT"
# script/bootstrap links the node and yarn it installs into
# ~/.local/bin, which the shell that called it may not have on PATH.
PATH="$HOME/.local/bin:$PATH"
run_yarn "$@"
}
main "$@"