Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
415f2f41ac |
+1
-4
@@ -73,12 +73,9 @@
|
|||||||
|
|
||||||
# This repository's own entries, after the shared content above: the
|
# This repository's own entries, after the shared content above: the
|
||||||
# frontend build, the viewport test's output, the binary `make build`
|
# frontend build, the viewport test's output, the binary `make build`
|
||||||
# writes and the reports `make run` stores, and Go test binaries,
|
# writes and the reports `make run` stores, and log files at the root.
|
||||||
# coverage output and log files at the root.
|
|
||||||
/dist
|
/dist
|
||||||
/tmp
|
/tmp
|
||||||
/netwatch-server
|
/netwatch-server
|
||||||
/data
|
/data
|
||||||
/*.test
|
|
||||||
/*.out
|
|
||||||
/*.log
|
/*.log
|
||||||
|
|||||||
+2
-3
@@ -11,8 +11,7 @@ insert_final_newline = true
|
|||||||
[Makefile]
|
[Makefile]
|
||||||
indent_style = tab
|
indent_style = tab
|
||||||
|
|
||||||
|
# This repository's own entries, after the shared content above.
|
||||||
|
|
||||||
[*.go]
|
[*.go]
|
||||||
indent_style = tab
|
indent_style = tab
|
||||||
|
|
||||||
# This repository's own sections, such as one for another language it
|
|
||||||
# uses, go below this comment, and a re-vendor keeps them.
|
|
||||||
|
|||||||
@@ -1,20 +1,9 @@
|
|||||||
name: check
|
name: check
|
||||||
on: [push]
|
on: [push]
|
||||||
# Free the shared runner: a new push cancels only the same branch's older run.
|
|
||||||
concurrency:
|
|
||||||
group: ${{ github.workflow }}-${{ github.ref }}
|
|
||||||
cancel-in-progress: true
|
|
||||||
jobs:
|
jobs:
|
||||||
check:
|
check:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
# Free the shared runner from a hung build.
|
|
||||||
timeout-minutes: 20
|
|
||||||
steps:
|
steps:
|
||||||
# actions/checkout v4.2.2, 2026-02-22
|
# actions/checkout v4.2.2, 2026-02-22
|
||||||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
|
||||||
# script/cibuild needs no token, so none is left in .git/config.
|
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
# All history and tags, so git describe finds the version tag.
|
|
||||||
fetch-depth: 0
|
|
||||||
- run: script/cibuild
|
- run: script/cibuild
|
||||||
|
|||||||
+2
-4
@@ -27,7 +27,7 @@ node_modules/
|
|||||||
# Environment files. `*.env` covers bare `.env` and the `prod.env`
|
# Environment files. `*.env` covers bare `.env` and the `prod.env`
|
||||||
# convention. Only the templates `example.env` and `sample.env` are
|
# convention. Only the templates `example.env` and `sample.env` are
|
||||||
# re-included below. A repository that commits any other template adds
|
# re-included below. A repository that commits any other template adds
|
||||||
# its own negation at the end of this file, for example `!.env.example`.
|
# its own negation after these lines, for example `!.env.example`.
|
||||||
*.[eE][nN][vV]
|
*.[eE][nN][vV]
|
||||||
.[eE][nN][vV].*
|
.[eE][nN][vV].*
|
||||||
.[eE][nN][vV][rR][cC]
|
.[eE][nN][vV][rR][cC]
|
||||||
@@ -46,9 +46,7 @@ node_modules/
|
|||||||
[iI][dD]_[eE][dD]25519
|
[iI][dD]_[eE][dD]25519
|
||||||
[iI][dD]_[eE][dD]25519_[sS][kK]
|
[iI][dD]_[eE][dD]25519_[sS][kK]
|
||||||
|
|
||||||
# This repository's own entries, such as its build outputs, go below
|
# This repository's own entries, after the shared content above.
|
||||||
# this comment, and a re-vendor keeps them. Anchor a binary built at the
|
|
||||||
# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
|
|
||||||
|
|
||||||
# Build output
|
# Build output
|
||||||
dist/
|
dist/
|
||||||
|
|||||||
@@ -25,8 +25,6 @@ linters:
|
|||||||
# silenced by disabling that name, not by enabling the successor.
|
# silenced by disabling that name, not by enabling the successor.
|
||||||
- wsl # Deprecated, replaced by wsl_v5
|
- wsl # Deprecated, replaced by wsl_v5
|
||||||
- gomodguard # Deprecated, replaced by gomodguard_v2
|
- gomodguard # Deprecated, replaced by gomodguard_v2
|
||||||
# Misses findings at random in v2.14.0; back once a pinned release fixes it
|
|
||||||
- canonicalheader
|
|
||||||
settings:
|
settings:
|
||||||
lll:
|
lll:
|
||||||
line-length: 88
|
line-length: 88
|
||||||
|
|||||||
+17
-39
@@ -10,22 +10,11 @@
|
|||||||
# phase covers the frontend as well, through a copy from a node stage.
|
# phase covers the frontend as well, through a copy from a node stage.
|
||||||
# Inside them each tool is invoked directly, never through make or
|
# Inside them each tool is invoked directly, never through make or
|
||||||
# script/, whose lint and test are themselves docker builds.
|
# script/, whose lint and test are themselves docker builds.
|
||||||
#
|
|
||||||
# In every stage ARG CACHEBUST comes right after the steps that install
|
|
||||||
# system packages and download Go's modules, or right after FROM in a
|
|
||||||
# stage with neither, and every step that lints, tests or compiles
|
|
||||||
# comes after it, as does yarn install in the node stages. script/
|
|
||||||
# passes a new value on every build, which reruns every RUN step after
|
|
||||||
# it. Docker keeps the layers above it until the base image, their own
|
|
||||||
# lines or go.mod and go.sum change. The steps that compile mount Go's
|
|
||||||
# build cache, and golangci-lint run also mounts golangci-lint's; both
|
|
||||||
# survive from build to build.
|
|
||||||
|
|
||||||
# Frontend lint stage: eslint with the rules in eslint.config.js. The
|
# Frontend lint stage: eslint with the rules in eslint.config.js. The
|
||||||
# lint phase below runs it.
|
# lint phase below runs it.
|
||||||
# node:22-alpine as of 2026-02-22
|
# node:22-alpine as of 2026-02-22
|
||||||
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS frontend-lint
|
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS frontend-lint
|
||||||
ARG CACHEBUST
|
|
||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
COPY package.json yarn.lock ./
|
COPY package.json yarn.lock ./
|
||||||
RUN yarn install --frozen-lockfile
|
RUN yarn install --frozen-lockfile
|
||||||
@@ -40,11 +29,8 @@ FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a924
|
|||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
ARG CACHEBUST
|
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN golangci-lint run --config .golangci.yml ./...
|
||||||
--mount=type=cache,target=/root/.cache/golangci-lint \
|
|
||||||
golangci-lint run --config .golangci.yml ./...
|
|
||||||
# Nothing is wanted from frontend-lint; the copy is what makes this
|
# Nothing is wanted from frontend-lint; the copy is what makes this
|
||||||
# phase run it.
|
# phase run it.
|
||||||
COPY --from=frontend-lint /app/yarn.lock /dev/null
|
COPY --from=frontend-lint /app/yarn.lock /dev/null
|
||||||
@@ -57,7 +43,6 @@ COPY --from=frontend-lint /app/yarn.lock /dev/null
|
|||||||
# node would take a reporter option for one more file.
|
# node would take a reporter option for one more file.
|
||||||
# node:22-alpine as of 2026-02-22
|
# node:22-alpine as of 2026-02-22
|
||||||
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS frontend-test
|
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS frontend-test
|
||||||
ARG CACHEBUST
|
|
||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
COPY package.json yarn.lock ./
|
COPY package.json yarn.lock ./
|
||||||
RUN yarn install --frozen-lockfile
|
RUN yarn install --frozen-lockfile
|
||||||
@@ -73,19 +58,19 @@ RUN NODE_OPTIONS=--test-reporter=dot timeout 90 yarn --silent run test || \
|
|||||||
# ships and the alpine one does not. -timeout 90s is a backstop above
|
# ships and the alpine one does not. -timeout 90s is a backstop above
|
||||||
# the 60-second cap on the suite. The rerun with -v only shows details:
|
# the 60-second cap on the suite. The rerun with -v only shows details:
|
||||||
# the step fails however it ends, because the first run already failed.
|
# the step fails however it ends, because the first run already failed.
|
||||||
# -count=1 because the mounted cache also holds test results, which go
|
# Go's module and build caches are in memory (tmpfs) for the go test
|
||||||
# test would otherwise replay.
|
# step alone, so the image make test tags does not carry them and the
|
||||||
|
# build spends no time writing them into it. They start empty on every
|
||||||
|
# build, so no stored test result can stand in for a run.
|
||||||
# golang:1.25.7-trixie, 2026-10-07
|
# golang:1.25.7-trixie, 2026-10-07
|
||||||
FROM golang@sha256:2b174ffcf56c7ad0c47d30d2630693265639ddf2a5141149c2da34db921791b4 AS test
|
FROM golang@sha256:2b174ffcf56c7ad0c47d30d2630693265639ddf2a5141149c2da34db921791b4 AS test
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
|
||||||
RUN go mod download
|
|
||||||
ARG CACHEBUST
|
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN --mount=type=tmpfs,target=/go/pkg/mod \
|
||||||
go test -count=1 -timeout 90s -race -cover ./... || \
|
--mount=type=tmpfs,target=/root/.cache/go-build \
|
||||||
|
go test -timeout 90s -race -cover ./... || \
|
||||||
{ echo "--- Rerunning with -v for details ---"; \
|
{ echo "--- Rerunning with -v for details ---"; \
|
||||||
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
|
go test -timeout 90s -race -v ./...; exit 1; }
|
||||||
# Nothing is wanted from the frontend-test stage; the copy is what makes
|
# Nothing is wanted from the frontend-test stage; the copy is what makes
|
||||||
# this phase run its tests.
|
# this phase run its tests.
|
||||||
COPY --from=frontend-test /app/yarn.lock /dev/null
|
COPY --from=frontend-test /app/yarn.lock /dev/null
|
||||||
@@ -98,28 +83,23 @@ COPY --from=frontend-test /app/yarn.lock /dev/null
|
|||||||
FROM frontend-test AS frontend-build
|
FROM frontend-test AS frontend-build
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
COPY --from=test /src/go.sum /dev/null
|
COPY --from=test /src/go.sum /dev/null
|
||||||
# vite.config.js reads the commit for the page's footer with git. git is
|
# vite.config.js reads the commit for the page's footer with git.
|
||||||
# installed again on every image build, although it comes above
|
|
||||||
# ARG CACHEBUST, because this stage starts from frontend-test, whose
|
|
||||||
# steps run again on every build.
|
|
||||||
RUN apk add --no-cache git
|
RUN apk add --no-cache git
|
||||||
ARG CACHEBUST
|
|
||||||
RUN yarn build
|
RUN yarn build
|
||||||
|
|
||||||
# Backend build stage. Nothing is wanted from the two phases; the copies
|
# Backend build stage. Nothing is wanted from the two phases; the copies
|
||||||
# are what make BuildKit build them first, so nothing after them can
|
# are what make BuildKit build them first, so this stage cannot run
|
||||||
# run unless lint and test passed.
|
# unless lint and test passed.
|
||||||
# golang:1.25-alpine (2026-02-27)
|
# golang:1.25-alpine (2026-02-27)
|
||||||
FROM golang@sha256:f6751d823c26342f9506c03797d2527668d095b0a15f1862cddb4d927a7a4ced AS builder
|
FROM golang@sha256:f6751d823c26342f9506c03797d2527668d095b0a15f1862cddb4d927a7a4ced AS builder
|
||||||
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
|
COPY --from=test /src/go.sum /dev/null
|
||||||
RUN apk add --no-cache git
|
RUN apk add --no-cache git
|
||||||
|
# A tar-stream context keeps the sender's file owners, which git refuses.
|
||||||
|
RUN git config --system --add safe.directory /src
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
ARG CACHEBUST
|
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
|
||||||
COPY --from=test /src/go.sum /dev/null
|
|
||||||
# A tar-stream context keeps the sender's file owners, which git refuses.
|
|
||||||
RUN git config --system --add safe.directory /src
|
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# The same go build as in script/build, which builds the binary on the
|
# The same go build as in script/build, which builds the binary on the
|
||||||
@@ -132,8 +112,7 @@ COPY . .
|
|||||||
# still empty, dev or unknown fails the build: git is missing or could
|
# still empty, dev or unknown fails the build: git is missing or could
|
||||||
# not read the checkout.
|
# not read the checkout.
|
||||||
ARG VERSION
|
ARG VERSION
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN version="${VERSION:-$(git describe --tags --always)}"; \
|
||||||
version="${VERSION:-$(git describe --tags --always)}"; \
|
|
||||||
if [ -e .git ]; then \
|
if [ -e .git ]; then \
|
||||||
case "$version" in ""|dev|unknown) \
|
case "$version" in ""|dev|unknown) \
|
||||||
echo "version is '$version' although .git is present" >&2; \
|
echo "version is '$version' although .git is present" >&2; \
|
||||||
@@ -148,7 +127,6 @@ RUN --mount=type=cache,target=/root/.cache/go-build \
|
|||||||
# and the stages it copies from, the two phases included.
|
# and the stages it copies from, the two phases included.
|
||||||
# nginx:stable-alpine as of 2026-02-22
|
# nginx:stable-alpine as of 2026-02-22
|
||||||
FROM nginx@sha256:15e96e59aa3b0aada3a121296e3bce117721f42d88f5f64217ef4b18f458c6ab
|
FROM nginx@sha256:15e96e59aa3b0aada3a121296e3bce117721f42d88f5f64217ef4b18f458c6ab
|
||||||
ARG CACHEBUST
|
|
||||||
|
|
||||||
# netwatch-server runs as this user, which owns the report directory.
|
# netwatch-server runs as this user, which owns the report directory.
|
||||||
# nginx keeps the image's own arrangement: its main process runs as
|
# nginx keeps the image's own arrangement: its main process runs as
|
||||||
|
|||||||
@@ -41,13 +41,11 @@ The scripts cover both the frontend and the Go backend, so `make check` fails if
|
|||||||
either one is broken. We provide:
|
either one is broken. We provide:
|
||||||
|
|
||||||
- `script/bootstrap` — install all dependencies (the pinned node via nvm unless
|
- `script/bootstrap` — install all dependencies (the pinned node via nvm unless
|
||||||
the `node` on `PATH` has the pinned major version and is new enough for the
|
one new enough for the frontend's dependencies is installed, yarn via
|
||||||
frontend's dependencies, yarn `YARN_VERSION` via corepack, or npm where there
|
corepack, `yarn install --frozen-lockfile`, Go `GO_VERSION` unless the `go` on
|
||||||
is no corepack, unless the `yarn` on `PATH` is exactly that version,
|
`PATH` is exactly that version, and the Go modules), linking what it installs
|
||||||
`yarn install --frozen-lockfile`, Go `GO_VERSION` unless the `go` on `PATH` is
|
itself into `~/.local/bin`, which has to be on `PATH`. It installs no Go
|
||||||
exactly that version, and the Go modules), linking what it installs itself
|
linter and not Docker: `make test` and `make lint` run in Docker
|
||||||
into `~/.local/bin`, which has to be on `PATH`. It installs no Go linter and
|
|
||||||
not Docker: `make test` and `make lint` run in Docker
|
|
||||||
- `script/setup` — make a fresh clone ready for development: bootstrap plus the
|
- `script/setup` — make a fresh clone ready for development: bootstrap plus the
|
||||||
git pre-commit hook
|
git pre-commit hook
|
||||||
- `script/dev` — run the Vite dev server, which proxies `/api` to a locally
|
- `script/dev` — run the Vite dev server, which proxies `/api` to a locally
|
||||||
@@ -55,23 +53,16 @@ either one is broken. We provide:
|
|||||||
- `script/build` — build the frontend for production into `dist/`, then the
|
- `script/build` — build the frontend for production into `dist/`, then the
|
||||||
static `netwatch-server` binary into the repo root with its version stamped
|
static `netwatch-server` binary into the repo root with its version stamped
|
||||||
in: `VERSION` from the environment, or when that is unset or empty,
|
in: `VERSION` from the environment, or when that is unset or empty,
|
||||||
`git describe --tags --always --dirty` inside a git checkout, as
|
`git describe` inside a git checkout, then `dev`
|
||||||
`script/docker` does, then `dev`
|
|
||||||
- `script/run` — build with `script/build`, then run `netwatch-server` with its
|
- `script/run` — build with `script/build`, then run `netwatch-server` with its
|
||||||
defaults (see [Settings](#settings))
|
defaults (see [Settings](#settings))
|
||||||
- `script/clean` — remove what `script/build` writes
|
- `script/clean` — remove what `script/build` writes
|
||||||
- `script/projectname` — print the project name (used for the Docker image tag)
|
- `script/projectname` — print the project name (used for the Docker image tag)
|
||||||
- `script/yarn` — run yarn with the given arguments under the node
|
- `script/test` — build the `test` phase of `Dockerfile` without the cache: the
|
||||||
`script/bootstrap` would use: with `~/.local/bin` first on `PATH`, the first
|
|
||||||
`node` on that `PATH` when it has the pinned major version and is new enough
|
|
||||||
for the frontend's dependencies, and otherwise the pinned node in nvm. Every
|
|
||||||
script that runs yarn on the host runs it through this one, except
|
|
||||||
`script/bootstrap`, which picks its own node and yarn
|
|
||||||
- `script/test` — build the `test` phase of `Dockerfile`, writing no image: the
|
|
||||||
frontend's unit tests in its `frontend-test` stage, and the backend's Go tests
|
frontend's unit tests in its `frontend-test` stage, and the backend's Go tests
|
||||||
with the race detector and coverage. The frontend's production build is left
|
with the race detector and coverage. The frontend's production build is left
|
||||||
to the image build, in the `frontend-build` stage
|
to the image build, in the `frontend-build` stage
|
||||||
- `script/lint` — build the `lint` phase of `Dockerfile`, writing no image:
|
- `script/lint` — build the `lint` phase of `Dockerfile` without the cache:
|
||||||
eslint in its `frontend-lint` stage, and golangci-lint over the Go packages
|
eslint in its `frontend-lint` stage, and golangci-lint over the Go packages
|
||||||
- `script/fmt` — format all files (writes): prettier over the JavaScript, CSS,
|
- `script/fmt` — format all files (writes): prettier over the JavaScript, CSS,
|
||||||
HTML and Markdown, then gofmt over `cmd/` and `internal/`. It runs on the
|
HTML and Markdown, then gofmt over `cmd/` and `internal/`. It runs on the
|
||||||
@@ -99,25 +90,13 @@ either one is broken. We provide:
|
|||||||
starts, it checks that the page, and a path that falls back to it, are served
|
starts, it checks that the page, and a path that falls back to it, are served
|
||||||
with `Cache-Control: no-cache` and `Content-Security-Policy`. Not part of
|
with `Cache-Control: no-cache` and `Content-Security-Policy`. Not part of
|
||||||
`script/check`: it takes minutes.
|
`script/check`: it takes minutes.
|
||||||
- `script/docker` — build the image from `Dockerfile`, tagged `netwatch` via
|
- `script/docker` — build the image from `Dockerfile` without the build cache,
|
||||||
`script/projectname`
|
tagged `netwatch` via `script/projectname`
|
||||||
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
|
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
|
||||||
then builds the image as `script/docker` does
|
then builds the image as `script/docker` does, without the build cache
|
||||||
- `script/precommit` — run by the git pre-commit hook; runs `script/check`
|
- `script/precommit` — run by the git pre-commit hook; runs `script/check`
|
||||||
- `script/install-precommit` — install the git pre-commit hook
|
- `script/install-precommit` — install the git pre-commit hook
|
||||||
|
|
||||||
`script/test`, `script/lint`, `script/docker` and `script/cibuild` each pass a
|
|
||||||
new `CACHEBUST` value, so every step in `Dockerfile` that lints, tests or
|
|
||||||
compiles runs again even when nothing has changed. Docker keeps the system
|
|
||||||
packages and the Go modules from earlier builds until the base image, the step
|
|
||||||
that installs them, or `go.mod` and `go.sum` change. The `frontend-build` stage
|
|
||||||
is the exception: it installs git again on every image build, because it starts
|
|
||||||
from the `frontend-test` stage, whose steps run again on every build. Go's build
|
|
||||||
cache and golangci-lint's are kept as well, and each reuses a result only when
|
|
||||||
everything it was made from is unchanged. Both `go test` runs carry `-count=1`,
|
|
||||||
so no test result is reused. The yarn packages are installed again on every
|
|
||||||
build.
|
|
||||||
|
|
||||||
## Responsive layout
|
## Responsive layout
|
||||||
|
|
||||||
The narrow-viewport layout lives in the `max-width: 768px` media block in
|
The narrow-viewport layout lives in the `max-width: 768px` media block in
|
||||||
@@ -208,9 +187,7 @@ stops. Its routes:
|
|||||||
address may send a limited number a minute, and the report files are capped in
|
address may send a limited number a minute, and the report files are capped in
|
||||||
size, the oldest deleted first
|
size, the oldest deleted first
|
||||||
- `GET /.well-known/healthcheck` — answers 200 with `"status":"ok"`, the
|
- `GET /.well-known/healthcheck` — answers 200 with `"status":"ok"`, the
|
||||||
server's version and its uptime; while reports cannot be stored, as the last
|
server's version and its uptime
|
||||||
write of a report file failed, it answers 503 with `"status":"error"` and the
|
|
||||||
same fields (see [Report storage](#report-storage))
|
|
||||||
- `GET /metrics` — Prometheus metrics behind basic auth, only when
|
- `GET /metrics` — Prometheus metrics behind basic auth, only when
|
||||||
`METRICS_USERNAME` and `METRICS_PASSWORD` are set; each client address may
|
`METRICS_USERNAME` and `METRICS_PASSWORD` are set; each client address may
|
||||||
make a limited number of requests to it a minute
|
make a limited number of requests to it a minute
|
||||||
@@ -218,10 +195,6 @@ stops. Its routes:
|
|||||||
It is an `fx`-wired Go application: `cmd/netwatch-server/` starts it, and its
|
It is an `fx`-wired Go application: `cmd/netwatch-server/` starts it, and its
|
||||||
packages are in `internal/`:
|
packages are in `internal/`:
|
||||||
|
|
||||||
- **`cli`**: Runs `netwatch-server` from its command line: the `check-cidr` and
|
|
||||||
`prepare-data-dir` subcommands, which `bin/entrypoint.sh` runs before it
|
|
||||||
starts the server, and, given no arguments, the server, with the `fx` wiring.
|
|
||||||
It refuses any other arguments with a usage message and exit code 2.
|
|
||||||
- **`config`**: Loads configuration from environment variables and config files
|
- **`config`**: Loads configuration from environment variables and config files
|
||||||
via Viper.
|
via Viper.
|
||||||
- **`handlers`**: HTTP request handlers for the API (health check, report
|
- **`handlers`**: HTTP request handlers for the API (health check, report
|
||||||
@@ -286,16 +259,13 @@ toward `DATA_DIR_MAX_BYTES` from the next start, only if removing it fails too.
|
|||||||
Files are created with `O_EXCL` to prevent overwrites.
|
Files are created with `O_EXCL` to prevent overwrites.
|
||||||
|
|
||||||
The reports of a failed write stay in memory, and the next flush, within a
|
The reports of a failed write stay in memory, and the next flush, within a
|
||||||
minute, tries again to write them. The server writes one file at a time, so a
|
minute, tries again to write them. From the failed write until a write succeeds,
|
||||||
file holds its reports in the order they arrived, those of a failed write
|
`POST /api/v1/reports` refuses every report with 500 and keeps nothing of it.
|
||||||
included. From the failed write until a write succeeds, `POST /api/v1/reports`
|
The page sends the refused samples again in its next report, as long as it still
|
||||||
refuses every report with 500 and keeps nothing of it, and the health check
|
holds them: at the 1-second interval one refused report already loses some of
|
||||||
answers 503 with `"status":"error"`. The page sends the refused samples again in
|
those in the history (see [Reporting](#reporting)). Reports already answered 200
|
||||||
its next report, as long as it still holds them: at the 1-second interval one
|
are lost only if the server stops before a write succeeds; the final flush at
|
||||||
refused report already loses some of those in the history (see
|
stop tries them too, and if it fails, the process exits non-zero.
|
||||||
[Reporting](#reporting)). Reports already answered 200 are lost only if the
|
|
||||||
server stops before a write succeeds; the final flush at stop tries them too,
|
|
||||||
and if it fails, the process exits non-zero.
|
|
||||||
|
|
||||||
#### Report limits
|
#### Report limits
|
||||||
|
|
||||||
@@ -567,16 +537,9 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
|
|||||||
stops the container. Empty, the backend sends nothing to Sentry
|
stops the container. Empty, the backend sends nothing to Sentry
|
||||||
- **Health check:** the image's `HEALTHCHECK` requests
|
- **Health check:** the image's `HEALTHCHECK` requests
|
||||||
`/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless
|
`/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless
|
||||||
both nginx and the backend answer. It also fails while reports cannot be
|
both nginx and the backend answer. upaas reads the container's health 60
|
||||||
stored, as when the volume is full or `DATA_DIR` cannot be written: the health
|
seconds after a deploy and fails the deploy unless it is `healthy`. The
|
||||||
check answers 503 from a failed write of a report file until a write succeeds,
|
container also stops when either process exits.
|
||||||
and the backend tries the write again every minute. Three failed checks in a
|
|
||||||
row make the container `unhealthy`, and one that passes makes it `healthy`
|
|
||||||
again. upaas reads the container's health once, 60 seconds after a deploy, and
|
|
||||||
fails the deploy unless it is `healthy`. A container that turns `unhealthy`
|
|
||||||
after that is neither restarted nor reported by upaas, whose `unless-stopped`
|
|
||||||
restart policy ignores health; only `docker ps` and `docker inspect` show it.
|
|
||||||
The container also stops when either process exits.
|
|
||||||
|
|
||||||
## Browser Compatibility
|
## Browser Compatibility
|
||||||
|
|
||||||
|
|||||||
+86
-196
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Repository Policies
|
title: Repository Policies
|
||||||
last_modified: 2026-10-18
|
last_modified: 2026-10-04
|
||||||
---
|
---
|
||||||
|
|
||||||
This document covers repository structure, tooling, and workflow standards. Code
|
This document covers repository structure, tooling, and workflow standards. Code
|
||||||
@@ -54,39 +54,34 @@ style conventions are in separate documents:
|
|||||||
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
|
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
|
||||||
assumes nothing is present: base tools come from nix, apt, brew, or apk
|
assumes nothing is present: base tools come from nix, apt, brew, or apk
|
||||||
(detected in that order; apt runs noninteractive). For node it uses the
|
(detected in that order; apt runs noninteractive). For node it uses the
|
||||||
installed node only when its major version is the pinned one; otherwise (node
|
installed node if present; otherwise it installs a PINNED node version via
|
||||||
missing, or another major, such as the node 24 the CI runner image ships) it
|
nvm, first installing nvm itself if missing — from a hash-verified GitHub
|
||||||
installs the PINNED node version via nvm and installs yarn and the packages
|
release archive (never `curl | sh`), with bash installed as an explicit
|
||||||
under it. nvm itself is installed first if missing, from a hash-verified
|
prerequisite since nvm requires bash. yarn is then pinned via
|
||||||
GitHub release archive (never `curl | sh`), with bash installed as an explicit
|
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
|
||||||
prerequisite since nvm requires bash. Only the major version is compared
|
always exact versions. `script/cibuild` runs the CI build: it changes to the
|
||||||
because the `Dockerfile` stages start from a node image whose exact version is
|
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
|
||||||
not the pin, and nvm cannot install a prebuilt node on alpine. yarn is pinned
|
with the version; the Gitea workflow calls it. **`script/cibuild` runs
|
||||||
via `corepack prepare yarn@<version> --activate`. Never install "latest" or
|
|
||||||
"lts"; always exact versions. `script/cibuild` runs the CI build: it changes
|
|
||||||
to the repo root, runs `script/bootstrap`, runs `script/check`, and builds the
|
|
||||||
image with the version; the Gitea workflow calls it. **`script/cibuild` runs
|
|
||||||
`script/bootstrap` first**, because the workflow checks out the repo and runs
|
`script/bootstrap` first**, because the workflow checks out the repo and runs
|
||||||
nothing else, while `script/fmt-check` runs the formatter on the host: on a
|
nothing else, while `script/fmt-check` runs the formatter on the host: on a
|
||||||
pristine checkout with nothing installed the run dies there, after the
|
pristine checkout with nothing installed the run dies there, after the
|
||||||
containerised gates have passed. **The bootstrap alone is not enough**: when
|
containerised gates have passed. **The bootstrap alone is not enough**:
|
||||||
`script/bootstrap` installs node and yarn under nvm it leaves neither on the
|
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
|
||||||
`PATH` of the shell that called it, so a bare `yarn` either exits 127 or runs
|
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
|
||||||
under another node. The host entrypoints that need yarn — `script/fmt` and
|
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
|
||||||
`script/fmt-check` — therefore use the same test as `script/bootstrap`: when
|
source nvm for the pinned node version before invoking it, exactly as
|
||||||
the node on `PATH` has the pinned major version they run the `yarn` on `PATH`,
|
`script/bootstrap`'s own install step does. A runner carrying nothing but
|
||||||
and otherwise they load nvm through `$HOME/.nvm/nvm.sh` and run yarn under the
|
docker and git then gets through `script/check`. Four further scripts are our
|
||||||
pinned node. A runner carrying nothing but docker and git then gets through
|
own extensions to the standard: `script/check` runs `script/test`,
|
||||||
`script/check`. Four further scripts are our own extensions to the standard:
|
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
|
||||||
`script/check` runs `script/test`, `script/lint` and `script/fmt-check`;
|
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
|
||||||
`script/precommit` is what the git pre-commit hook runs, and it calls
|
installs the git pre-commit hook (the `make hooks` target shims to it); and
|
||||||
`script/check`; `script/install-precommit` installs the git pre-commit hook
|
`script/projectname` (literally that filename) simply outputs the project's
|
||||||
(the `make hooks` target shims to it); and `script/projectname` (literally
|
name. Scripts that need the name call `script/projectname` — e.g.
|
||||||
that filename) simply outputs the project's name. Scripts that need the name
|
`script/docker` assembles its image tag from it — so those scripts stay
|
||||||
call `script/projectname` — e.g. `script/docker` assembles its image tag from
|
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
|
||||||
it — so those scripts stay byte-identical across all repos. Repo-type-specific
|
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
|
||||||
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
|
the hook itself. Model scripts are at
|
||||||
`script/precommit`, not in the hook itself. Model scripts are at
|
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
|
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
|
||||||
must document the provided scripts in an **Entrypoints** section (see the
|
must document the provided scripts in an **Entrypoints** section (see the
|
||||||
README requirements below).
|
README requirements below).
|
||||||
@@ -123,10 +118,8 @@ style conventions are in separate documents:
|
|||||||
and nothing else:
|
and nothing else:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
|
||||||
--target lint --output type=cacheonly .
|
docker build --no-cache --target test -t "$(script/projectname)-test" .
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
|
||||||
--target test --output type=cacheonly .
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**A stage that is not the last one in the file is built only when the final
|
**A stage that is not the last one in the file is built only when the final
|
||||||
@@ -136,15 +129,10 @@ style conventions are in separate documents:
|
|||||||
plain `docker build .` builds the last stage alone and exits 0 having linted
|
plain `docker build .` builds the last stage alone and exits 0 having linted
|
||||||
and tested nothing.
|
and tested nothing.
|
||||||
|
|
||||||
**The gate builds write no image.** With `--output type=cacheonly` the phase
|
**Every `docker build` in `script/` is tagged**, here and in
|
||||||
runs and a failing step fails the build, but the result is not exported.
|
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
|
||||||
Nothing uses those images, and writing one out is slow: a Go test phase's
|
image behind on every invocation, on every developer host and every CI
|
||||||
image holds the toolchain. A build given neither `--output` nor `-t` writes
|
runner; a tagged one replaces the previous image.
|
||||||
an untagged image and leaves it dangling, on every developer host and every
|
|
||||||
CI runner. `script/cibuild` and `script/docker` build the image that ships
|
|
||||||
and tag it, so each build replaces the previous image; each assigns the tag
|
|
||||||
on its own line before the build, so `set -e` stops it where
|
|
||||||
`script/projectname` fails.
|
|
||||||
|
|
||||||
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
|
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
|
||||||
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
|
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
|
||||||
@@ -159,42 +147,22 @@ style conventions are in separate documents:
|
|||||||
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
|
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
|
||||||
exit non-zero with `parallel golangci-lint is running` — a status a caller
|
exit non-zero with `parallel golangci-lint is running` — a status a caller
|
||||||
cannot tell from real findings. Both have produced wrong verdicts in this
|
cannot tell from real findings. Both have produced wrong verdicts in this
|
||||||
org, in both directions. A container has its own `TMPDIR`, so the lock is
|
org, in both directions. A container has its own cache, its own `TMPDIR` and
|
||||||
out of reach. The lint phase does keep golangci-lint's cache between check
|
a digest-pinned binary, so neither is reachable.
|
||||||
builds (below), but it lints every checkout at `/src`, and a finding is
|
|
||||||
taken from that cache only for the same files, settings and golangci-lint
|
|
||||||
version.
|
|
||||||
|
|
||||||
- **Any build that runs checks is given a new `CACHEBUST` value.** Docker
|
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
|
||||||
invalidates a `COPY` layer only when the copied content changes, so on an
|
a `COPY` layer only when the copied content changes, so on an unchanged tree
|
||||||
unchanged tree the check `RUN` is served from cache, nothing executes, and the
|
the check `RUN` is served from cache, nothing executes, and the build still
|
||||||
build still exits 0. Every `docker build` in `script/` therefore passes
|
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
|
||||||
`--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the script's process
|
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
|
||||||
id, so no two builds share a value, not even the three in one
|
four, and there is no fifth — `script/check` runs the two gate phases and
|
||||||
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
|
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
|
||||||
`script/docker` are the four, and there is no fifth — `script/check` runs the
|
not evidence that anything ran: a sub-second build reporting success is a
|
||||||
two gate phases and `script/fmt-check`, and builds no image of its own. In
|
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
|
||||||
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps
|
and friends destroy a build cache shared with every other build on the host.
|
||||||
that install system packages and download Go's modules, or right after `FROM`
|
When a check is added or changed, prove it works by planting a defect it must
|
||||||
in a stage with neither, and every step that lints, tests or compiles comes
|
catch and watching the run fail on it, then revert the defect. A green run
|
||||||
after it. A step that installs anything else as well, such as a
|
alone shows neither that the check ran nor that it covers what it should.
|
||||||
`script/bootstrap` that also installs node packages, comes after it too. A new
|
|
||||||
value reruns every `RUN` step after that line; `COPY` steps are compared by
|
|
||||||
content and need nothing. Docker keeps the layers above the line until the
|
|
||||||
base image, their own lines or the files they copy change. A stage without the
|
|
||||||
`ARG` line is served from cache on an unchanged tree, so a repository
|
|
||||||
re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do
|
|
||||||
not pass `--no-cache`: it reruns every step too, but it also builds the layers
|
|
||||||
above the line again and empties every cache mount, so Go's build cache and
|
|
||||||
golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they
|
|
||||||
survive between check builds, would start empty every time. A bare
|
|
||||||
`docker build .` is not evidence that anything ran: a sub-second build
|
|
||||||
reporting success is a cache hit, not a result. Never invalidate by pruning —
|
|
||||||
`docker builder prune` and friends destroy a build cache shared with every
|
|
||||||
other build on the host. When a check is added or changed, prove it works by
|
|
||||||
planting a defect it must catch and watching the run fail on it, then revert
|
|
||||||
the defect. A green run alone shows neither that the check ran nor that it
|
|
||||||
covers what it should.
|
|
||||||
|
|
||||||
- **The gate phases are separate stages, and the build stage depends on both.**
|
- **The gate phases are separate stages, and the build stage depends on both.**
|
||||||
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
||||||
@@ -203,56 +171,40 @@ style conventions are in separate documents:
|
|||||||
`Dockerfile`:
|
`Dockerfile`:
|
||||||
|
|
||||||
```dockerfile
|
```dockerfile
|
||||||
# In every stage ARG CACHEBUST comes right after the steps that install
|
|
||||||
# system packages and download Go's modules, or right after FROM in a
|
|
||||||
# stage with neither, and every step that lints, tests or compiles
|
|
||||||
# comes after it. script/ passes a new value on every build, which
|
|
||||||
# reruns every RUN step after it. Docker keeps the layers above it
|
|
||||||
# until the base image, their own lines or go.mod and go.sum change.
|
|
||||||
# The steps that compile mount Go's build cache, and golangci-lint run
|
|
||||||
# also mounts golangci-lint's; both survive from build to build.
|
|
||||||
|
|
||||||
# Lint phase
|
# Lint phase
|
||||||
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
|
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
|
||||||
FROM golangci/golangci-lint@sha256:... AS lint
|
FROM golangci/golangci-lint@sha256:... AS lint
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
ARG CACHEBUST
|
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN golangci-lint run --config .golangci.yml ./...
|
||||||
--mount=type=cache,target=/root/.cache/golangci-lint \
|
|
||||||
golangci-lint run --config .golangci.yml ./...
|
|
||||||
|
|
||||||
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
|
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
|
||||||
# image ships and the alpine one does not. -count=1 because the mounted
|
# image ships and the alpine one does not.
|
||||||
# cache also holds test results, which go test would otherwise replay.
|
|
||||||
# golang:1.x, YYYY-MM-DD
|
# golang:1.x, YYYY-MM-DD
|
||||||
FROM golang@sha256:... AS test
|
FROM golang@sha256:... AS test
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
ARG CACHEBUST
|
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN go test -timeout 90s -race -cover ./... || \
|
||||||
go test -count=1 -timeout 90s -race -cover ./... || \
|
|
||||||
{ echo "--- Rerunning with -v for details ---"; \
|
{ echo "--- Rerunning with -v for details ---"; \
|
||||||
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
|
go test -timeout 90s -race -v ./...; exit 1; }
|
||||||
|
|
||||||
# Build stage. Nothing is wanted from either phase above; the copies
|
# Build stage. Nothing is wanted from either phase above; the copies
|
||||||
# are what make BuildKit build them first, so nothing after them can
|
# are what make BuildKit build them first, so this stage cannot run
|
||||||
# run unless lint and test passed.
|
# unless lint and test passed.
|
||||||
# golang:1.x-alpine, YYYY-MM-DD
|
# golang:1.x-alpine, YYYY-MM-DD
|
||||||
FROM golang@sha256:... AS builder
|
FROM golang@sha256:... AS builder
|
||||||
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
|
COPY --from=test /src/go.sum /dev/null
|
||||||
RUN apk add --no-cache git
|
RUN apk add --no-cache git
|
||||||
|
# A tar-stream context keeps the sender's file owners, which git refuses.
|
||||||
|
RUN git config --system --add safe.directory /src
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
ARG CACHEBUST
|
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
|
||||||
COPY --from=test /src/go.sum /dev/null
|
|
||||||
# A tar-stream context keeps the sender's file owners, which git refuses.
|
|
||||||
RUN git config --system --add safe.directory /src
|
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# The VERSION build arg when one is given, otherwise
|
# The VERSION build arg when one is given, otherwise
|
||||||
@@ -260,8 +212,7 @@ style conventions are in separate documents:
|
|||||||
# .git present, a version that is still empty, dev or unknown fails the
|
# .git present, a version that is still empty, dev or unknown fails the
|
||||||
# build: git is missing or could not read the checkout.
|
# build: git is missing or could not read the checkout.
|
||||||
ARG VERSION
|
ARG VERSION
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
|
||||||
VERSION="${VERSION:-$(git describe --tags --always)}"; \
|
|
||||||
if [ -e .git ]; then \
|
if [ -e .git ]; then \
|
||||||
case "$VERSION" in ""|dev|unknown) \
|
case "$VERSION" in ""|dev|unknown) \
|
||||||
echo "version is '$VERSION' although .git is present" >&2; \
|
echo "version is '$VERSION' although .git is present" >&2; \
|
||||||
@@ -274,7 +225,6 @@ style conventions are in separate documents:
|
|||||||
|
|
||||||
# Runtime stage, and the last one
|
# Runtime stage, and the last one
|
||||||
FROM alpine@sha256:...
|
FROM alpine@sha256:...
|
||||||
ARG CACHEBUST
|
|
||||||
COPY --from=builder /app /usr/local/bin/app
|
COPY --from=builder /app /usr/local/bin/app
|
||||||
ENTRYPOINT ["app"]
|
ENTRYPOINT ["app"]
|
||||||
```
|
```
|
||||||
@@ -289,26 +239,6 @@ style conventions are in separate documents:
|
|||||||
- Keep the runtime stage last, and if you add a stage after it, give it the
|
- Keep the runtime stage last, and if you add a stage after it, give it the
|
||||||
same two copies. A plain `docker build .` builds the last stage's chain
|
same two copies. A plain `docker build .` builds the last stage's chain
|
||||||
and nothing else.
|
and nothing else.
|
||||||
- Check builds keep these, and none of them can hand back a stale result:
|
|
||||||
- The layers above `ARG CACHEBUST`: the system packages and Go's
|
|
||||||
modules. Docker installs the packages again when the base image or the
|
|
||||||
install line changes, and downloads the modules again when `go.mod` or
|
|
||||||
`go.sum` changes. Go checks every module against `go.sum` as it
|
|
||||||
downloads it, and no later build can change a kept layer. The modules
|
|
||||||
are kept in a layer rather than a cache mount because a cache mount is
|
|
||||||
shared by every build on the host, and Go builds from whatever files
|
|
||||||
it holds without checking them again (only `go mod verify` does), so a
|
|
||||||
file changed there would go unnoticed.
|
|
||||||
- Go's build cache (`/root/.cache/go-build`, where Go keeps it for
|
|
||||||
root), mounted on the steps that compile: `golangci-lint run`, both
|
|
||||||
`go test` runs and `go build`. Go reuses a compiled package only when
|
|
||||||
all its inputs are unchanged. The cache also holds test results, which
|
|
||||||
is why both `go test` runs carry `-count=1`.
|
|
||||||
- golangci-lint's cache (`/root/.cache/golangci-lint`, where the pinned
|
|
||||||
image keeps it), mounted on `golangci-lint run`. golangci-lint reuses
|
|
||||||
a package's findings only when the package's files and those of every
|
|
||||||
package it imports, `go.mod`, the linters and their settings in
|
|
||||||
`.golangci.yml`, and the golangci-lint version are all unchanged.
|
|
||||||
- If the project uses `//go:embed` directives that reference build artifacts
|
- If the project uses `//go:embed` directives that reference build artifacts
|
||||||
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
||||||
create placeholder files so the embed directives resolve. Example:
|
create placeholder files so the embed directives resolve. Example:
|
||||||
@@ -317,20 +247,12 @@ style conventions are in separate documents:
|
|||||||
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
|
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
|
||||||
has no `apk`, so install with `apt-get` under the Debian package name
|
has no `apk`, so install with `apt-get` under the Debian package name
|
||||||
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
|
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
|
||||||
lists in the same `RUN`, so the layer does not keep them. The test phase,
|
lists in the same `RUN`, so the layer does not keep them:
|
||||||
on the Debian Go image, installs the libraries its tests need the same
|
|
||||||
way. Either install goes right after `FROM`, above `ARG CACHEBUST`, so
|
|
||||||
Docker keeps it until the base image or the install line changes:
|
|
||||||
|
|
||||||
```dockerfile
|
```dockerfile
|
||||||
FROM golangci/golangci-lint@sha256:... AS lint
|
|
||||||
RUN apt-get update \
|
RUN apt-get update \
|
||||||
&& apt-get install -y --no-install-recommends libvips-dev \
|
&& apt-get install -y --no-install-recommends libvips-dev \
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
WORKDIR /src
|
|
||||||
COPY go.mod go.sum ./
|
|
||||||
RUN go mod download
|
|
||||||
ARG CACHEBUST
|
|
||||||
```
|
```
|
||||||
|
|
||||||
- `.dockerignore` lets `.git` into the build context. It keeps out every git
|
- `.dockerignore` lets `.git` into the build context. It keeps out every git
|
||||||
@@ -356,39 +278,20 @@ style conventions are in separate documents:
|
|||||||
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
|
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
|
||||||
`docker build .` with no build arguments must succeed; a Dockerfile that
|
`docker build .` with no build arguments must succeed; a Dockerfile that
|
||||||
refuses an empty build argument drops that refusal and keeps the argument.
|
refuses an empty build argument drops that refusal and keeps the argument.
|
||||||
A checkout whose `.git` is a file (a linked worktree, or a repository
|
|
||||||
checked out as a submodule) is the exception: that file points to a git
|
|
||||||
directory outside the build context, so the build cannot read the version
|
|
||||||
and a plain `docker build .` fails; pass the version with
|
|
||||||
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
|
|
||||||
do.
|
|
||||||
|
|
||||||
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
||||||
runs `script/cibuild` on push, and checks out the repo as its only other step,
|
runs `script/cibuild` on push, and checks out the repo as its only other step.
|
||||||
with `persist-credentials: false`: `script/cibuild` needs no token, and
|
That script bootstraps, runs the gate phases, and then builds the image, so a
|
||||||
without it the checkout leaves the job's token in `.git/config` for every
|
successful run means every check passed; a bare `docker build .` does not
|
||||||
later step. The checkout step also sets `fetch-depth: 0`, which fetches the
|
|
||||||
tags `git describe` needs: by default it clones shallow with no tags, and a
|
|
||||||
tagged repository's CI build would stamp a bare short commit id. The
|
|
||||||
workflow's `concurrency` block groups runs by workflow and branch
|
|
||||||
(`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`,
|
|
||||||
so a new push cancels the older run on the same branch, queued or running, and
|
|
||||||
no other: runs for replaced commits do not hold up the shared runner.
|
|
||||||
`script/cibuild` bootstraps, runs the gate phases, and then builds the image,
|
|
||||||
so a successful run means every check passed; a bare `docker build .` does not
|
|
||||||
carry the same guarantee, because its gate phases may come from the cache. The
|
carry the same guarantee, because its gate phases may come from the cache. The
|
||||||
image build gets a `CACHEBUST` value of its own and so runs the gate phases a
|
image build is uncached and so runs the gate phases a second time. That is the
|
||||||
second time. That is the price of the rule above, and it is worth paying: the
|
price of the rule above, and it is worth paying: the image that ships is built
|
||||||
image that ships is built from a run of its own gates rather than from a cache
|
from a run of its own gates rather than from a cache entry. A separate
|
||||||
entry. The `check` job sets `timeout-minutes: 20`, so a hung build frees the
|
workflow limited to `main` by a `branches` list under `on: push` cannot be
|
||||||
shared runner after 20 minutes. That allows for the three Docker builds
|
checked by review: to try a change to it, add the feature branch to that list
|
||||||
described above (the test phase, the lint phase, then the image), each held to
|
and push, then remove the branch from the list again before merging. Keep any
|
||||||
the 5-minute Docker build limit below, plus the bootstrap. A separate workflow
|
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
|
||||||
limited to `main` by a `branches` list under `on: push` cannot be checked by
|
from the feature branch publishes nothing.
|
||||||
review: to try a change to it, add the feature branch to that list and push,
|
|
||||||
then remove the branch from the list again before merging. Keep any job in it
|
|
||||||
that publishes behind `if: github.ref_name == 'main'`, so the run from the
|
|
||||||
feature branch publishes nothing.
|
|
||||||
|
|
||||||
- Use platform-standard formatters: `black` for Python, `prettier` for
|
- Use platform-standard formatters: `black` for Python, `prettier` for
|
||||||
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
||||||
@@ -448,11 +351,12 @@ style conventions are in separate documents:
|
|||||||
That cache is Go's own, separate from Docker's layer cache. Go stores a
|
That cache is Go's own, separate from Docker's layer cache. Go stores a
|
||||||
passing result in its cache directory (`GOCACHE`), and when the same tests
|
passing result in its cache directory (`GOCACHE`), and when the same tests
|
||||||
run again on unchanged code it prints that result, marked `(cached)`,
|
run again on unchanged code it prints that result, marked `(cached)`,
|
||||||
without running them. The directory lasts from one run to the next on a
|
without running them. That matters on a developer's machine, where this
|
||||||
developer's machine, where this target runs, and in the `test` phase of the
|
target runs and the directory lasts from one run to the next. The `test`
|
||||||
`Dockerfile`, which mounts it to keep compiled packages between check
|
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
|
||||||
builds, so the phase's `go test` runs carry `-count=1` too. A new
|
result for this repo's tests and nothing before its `go test` step runs a
|
||||||
`CACHEBUST` value (above) is what makes that step run on an unchanged tree.
|
test, so there is nothing to replay. `--no-cache` (above) is what makes that
|
||||||
|
step run on an unchanged tree.
|
||||||
|
|
||||||
Python example:
|
Python example:
|
||||||
|
|
||||||
@@ -480,12 +384,9 @@ style conventions are in separate documents:
|
|||||||
|
|
||||||
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
|
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
|
||||||
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
|
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
|
||||||
`node_modules/`, and the repo's own build outputs. Fetch the standard
|
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
|
||||||
`.gitignore` from
|
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
|
setting up a new repo. These patterns are written to `.gitignore`'s own
|
||||||
a new repo. A repo's `.gitignore` is the standard file followed by the repo's
|
|
||||||
own entries, such as its binaries; a re-vendor replaces the standard part and
|
|
||||||
keeps those entries. These patterns are written to `.gitignore`'s own
|
|
||||||
semantics, in which an unanchored pattern already matches at every depth; they
|
semantics, in which an unanchored pattern already matches at every depth; they
|
||||||
are not a `.dockerignore` and must not be transplanted into one unmodified.
|
are not a `.dockerignore` and must not be transplanted into one unmodified.
|
||||||
|
|
||||||
@@ -533,15 +434,13 @@ style conventions are in separate documents:
|
|||||||
byte-identically across repos:
|
byte-identically across repos:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# The version and the tag each get their own line: a failing command
|
# Own line: a failing command substitution inside an argument does not
|
||||||
# substitution inside an argument does not trip `set -e`, so the inline
|
# trip `set -e`, so the inline form degrades to an empty constant.
|
||||||
# form degrades to an empty constant.
|
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
tag="$(script/projectname)"
|
docker build --no-cache \
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$tag" .
|
-t "$(script/projectname)" .
|
||||||
```
|
```
|
||||||
|
|
||||||
`--always` makes an untagged repo yield an abbreviated commit hash rather
|
`--always` makes an untagged repo yield an abbreviated commit hash rather
|
||||||
@@ -553,8 +452,8 @@ style conventions are in separate documents:
|
|||||||
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
|
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
|
||||||
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
|
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
|
||||||
the scripts stay byte-identical. One consequence for CI: the standard
|
the scripts stay byte-identical. One consequence for CI: the standard
|
||||||
checkout action clones shallow and fetches no tags, so the canonical
|
checkout action clones shallow and fetches no tags, so a repo that embeds a
|
||||||
`.gitea/workflows/check.yml` sets `fetch-depth: 0` on its checkout step.
|
tag-derived version must set `fetch-depth: 0` on its checkout step.
|
||||||
|
|
||||||
- **Verify `.dockerignore` by enumerating the image, not by reading the
|
- **Verify `.dockerignore` by enumerating the image, not by reading the
|
||||||
patterns.** Plant files at the root _and_ at least two directories deep, build
|
patterns.** Plant files at the root _and_ at least two directories deep, build
|
||||||
@@ -621,11 +520,6 @@ style conventions are in separate documents:
|
|||||||
function defined and never invoked has the same exit status and the same
|
function defined and never invoked has the same exit status and the same
|
||||||
empty output as one that worked.
|
empty output as one that worked.
|
||||||
|
|
||||||
Node is the exception to the whole-token comparison: the canonical
|
|
||||||
`script/bootstrap` compares only its major version, because the node 22
|
|
||||||
alpine image the `Dockerfile` stages start from is not the exact pin, and
|
|
||||||
nvm cannot install a prebuilt node on alpine.
|
|
||||||
|
|
||||||
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
|
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
|
||||||
|
|
||||||
A Go tool a repo needs on the host is installed with `go install` pinned to
|
A Go tool a repo needs on the host is installed with `go install` pinned to
|
||||||
@@ -742,11 +636,7 @@ style conventions are in separate documents:
|
|||||||
Never edit existing migrations after release.
|
Never edit existing migrations after release.
|
||||||
|
|
||||||
- All repos should have an `.editorconfig` enforcing the project's indentation
|
- All repos should have an `.editorconfig` enforcing the project's indentation
|
||||||
settings: the standard file from
|
settings.
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, which sets
|
|
||||||
tabs for `Makefile` and Go files, followed by the repo's own sections, such as
|
|
||||||
one for another language it uses. A re-vendor replaces the standard part and
|
|
||||||
keeps those sections.
|
|
||||||
|
|
||||||
- Avoid putting files in the repo root unless necessary. Root should contain
|
- Avoid putting files in the repo root unless necessary. Root should contain
|
||||||
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
|
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
|
||||||
|
|||||||
@@ -9,12 +9,10 @@
|
|||||||
|
|
||||||
# Status
|
# Status
|
||||||
|
|
||||||
pre-1.0. Release candidates are tagged on `next`, the first being `v1.0.0-rc1`;
|
pre-1.0. No git tags. `main` is the stable branch and `next` the development
|
||||||
`git tag --merged origin/next` lists them. `main` is the stable branch and
|
branch, which every PR targets. The frontend and the Go backend ship as one
|
||||||
`next` the development branch, which every PR targets. The frontend and the Go
|
Docker image, and the Gitea workflow `.gitea/workflows/check.yml` runs
|
||||||
backend ship as one Docker image, and the Gitea workflow
|
`script/cibuild` on every push. Working toward 1.0.0.
|
||||||
`.gitea/workflows/check.yml` runs `script/cibuild` on every push, and cancels
|
|
||||||
the older run on the same branch. Working toward 1.0.0.
|
|
||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
@@ -23,134 +21,12 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-10-19: the server's `OnStart` hook in `internal/server/server.go` starts
|
|
||||||
`listenAndServe` with `go func() { s.listenAndServe() }()`, as the Go style
|
|
||||||
guide asks ([#162](https://git.eeqj.de/sneak/netwatch/issues/162)); it was
|
|
||||||
`go s.listenAndServe()`. No other `go` statement in the Go code called a
|
|
||||||
function directly. Nothing the server does changed
|
|
||||||
- 2026-10-19: `.gitignore`, `.editorconfig` and `.golangci.yml` are the
|
|
||||||
sneak/prompts copies at `0653579`
|
|
||||||
([#174](https://git.eeqj.de/sneak/netwatch/issues/174)). `.gitignore` is the
|
|
||||||
standard file followed by this repository's own entries; `.editorconfig` is
|
|
||||||
the standard file alone, which now carries the `[*.go]` section.
|
|
||||||
`.golangci.yml` disables `canonicalheader`, as the canonical copy does,
|
|
||||||
because golangci-lint v2.14.0 misses its findings at random
|
|
||||||
- 2026-10-19: `package.json` has no `version` field and is marked
|
|
||||||
`"private": true` ([#158](https://git.eeqj.de/sneak/netwatch/issues/158)). It
|
|
||||||
said `1.0.0`, a release that has not been made, and nothing read it; the git
|
|
||||||
tag is the one version, which the image and `make build` take from
|
|
||||||
`git describe --tags`. The package is never published
|
|
||||||
- 2026-10-19: Status names `v1.0.0-rc1`, the first release candidate, and says
|
|
||||||
release candidates are tagged on `next`, where it said there were no git tags
|
|
||||||
([#157](https://git.eeqj.de/sneak/netwatch/issues/157)). The rest of Status
|
|
||||||
matches the tree
|
|
||||||
- 2026-10-19: `netwatch-server` starts the server only when given no arguments
|
|
||||||
([#179](https://git.eeqj.de/sneak/netwatch/issues/179)). `check-cidr CIDR` and
|
|
||||||
`prepare-data-dir DATA_DIR` behave as before; any other arguments, such as a
|
|
||||||
subcommand with no value or two, or a misspelt one, print a one-line usage
|
|
||||||
message naming both forms to stderr and exit 2. They used to start the server.
|
|
||||||
Unit tests call `cli.Main` with no arguments and check that it starts the
|
|
||||||
server, and with each subcommand given no value and two values and with a
|
|
||||||
misspelt subcommand, and check the exit code, the message and that the server
|
|
||||||
does not start
|
|
||||||
- 2026-10-19: `main()` in `cmd/netwatch-server/main.go` is one call into a new
|
|
||||||
package, `internal/cli`, as the Go style guide asks
|
|
||||||
([#160](https://git.eeqj.de/sneak/netwatch/issues/160)). `internal/cli` runs
|
|
||||||
the `check-cidr` and `prepare-data-dir` subcommands, with the same arguments,
|
|
||||||
messages and exit codes, and otherwise the server, with the `fx` wiring.
|
|
||||||
`Appname` and `Version` stay in `main`, where the build sets `Version`, and
|
|
||||||
`main` gives `cli.Main` stderr and the function that runs the server, which
|
|
||||||
the tests replace with a buffer and a function that fails them. Unit tests
|
|
||||||
call `cli.Main` with `check-cidr` and a CIDR the server takes and one it
|
|
||||||
refuses, and check the exit code and what it writes to stderr, and with
|
|
||||||
`prepare-data-dir` and a path outside `/data`, and check that it exits 1 with
|
|
||||||
the error from looking up the `netwatch` user, which the test image does not
|
|
||||||
have. Others give the functions behind the subcommands input they take and
|
|
||||||
input they refuse, and check the exit code and what they write to stderr
|
|
||||||
- 2026-10-19: when `VERSION` is unset or empty, `script/build` stamps the
|
|
||||||
version from `git describe --tags --always --dirty`, as `script/docker` does
|
|
||||||
([#156](https://git.eeqj.de/sneak/netwatch/issues/156)). Without `--tags` it
|
|
||||||
skipped lightweight tags, so `make run` could report a commit where the image
|
|
||||||
built from the same tree reported the tag. Checked once with a lightweight tag
|
|
||||||
on a scratch clone: `make build` stamped it
|
|
||||||
- 2026-10-19: `.dockerignore` keeps Go test binaries and coverage output at the
|
|
||||||
root out of the build context, with `/*.test` and `/*.out` in the repository's
|
|
||||||
own section ([#155](https://git.eeqj.de/sneak/netwatch/issues/155)). Such
|
|
||||||
files went into the build context of every `docker build` in `script/` and
|
|
||||||
were copied into the lint, test and build stages. Checked with a probe image
|
|
||||||
that copies the whole context: `x.test` and `x.out` planted at the root no
|
|
||||||
longer land in it, and nothing else changed. The entries match only at the
|
|
||||||
root, so the same files in a subdirectory still reach the image
|
|
||||||
- 2026-10-19: `.gitea/workflows/check.yml` is the sneak/prompts copy at
|
|
||||||
`0653579` ([#153](https://git.eeqj.de/sneak/netwatch/issues/153)). Its
|
|
||||||
checkout step sets `fetch-depth: 0`, so CI clones all history and tags, and
|
|
||||||
`git describe` in `script/cibuild` gives the image built from a tagged commit
|
|
||||||
the tag as its version; it gave the short commit. The checkout step also sets
|
|
||||||
`persist-credentials: false`, so the job's token is not left in `.git/config`.
|
|
||||||
A new push cancels the older run on the same branch, and a run stops after 20
|
|
||||||
minutes
|
|
||||||
- 2026-10-19: `script/bootstrap` keeps the `yarn` on `PATH` only when
|
|
||||||
`yarn --version` is exactly `YARN_VERSION`, and otherwise installs that
|
|
||||||
version as it did when no yarn was installed
|
|
||||||
([#154](https://git.eeqj.de/sneak/netwatch/issues/154)); any yarn used to be
|
|
||||||
kept. It uses the `node` on `PATH` only when that node has the major version
|
|
||||||
of `NODE_VERSION`, as `REPO_POLICIES.md` asks, and is at least
|
|
||||||
`NODE_MIN_VERSION`; a node 24 was kept before. After installing node or yarn
|
|
||||||
it looks the tool up through `PATH` again and stops if that one fails the same
|
|
||||||
check, as it does for Go, and it prints the node and yarn versions it uses.
|
|
||||||
Every script that runs yarn on the host, except `script/bootstrap`, which
|
|
||||||
picks its own node and yarn, runs it through the new `script/yarn`. That
|
|
||||||
script puts `~/.local/bin` first on `PATH`, uses the first `node` on that
|
|
||||||
`PATH` when it passes the same test, and otherwise runs yarn under the pinned
|
|
||||||
node in nvm; `make build` ran yarn under whatever node was first on `PATH`
|
|
||||||
- 2026-10-19: `bin/entrypoint.sh` has each of its steps in a function, and a
|
|
||||||
`main` at the end calls them in order, as the shell section of the code style
|
|
||||||
guide asks ([#163](https://git.eeqj.de/sneak/netwatch/issues/163)). It is
|
|
||||||
still POSIX sh with no `set -e`, and does what it did, with the same messages
|
|
||||||
and exit codes. Checked by running the image with each script: a normal start,
|
|
||||||
bad `PORT`, `TRUSTED_PROXIES` and `DATA_DIR` values, `docker stop`, INT, and
|
|
||||||
nginx exiting on its own
|
|
||||||
- 2026-10-19: while report files cannot be written, the health check answers 503
|
|
||||||
with `"status":"error"` and the same fields, so the image's `HEALTHCHECK`
|
|
||||||
reports the container `unhealthy`
|
|
||||||
([#142](https://git.eeqj.de/sneak/netwatch/issues/142)). It answered 200
|
|
||||||
`"status":"ok"` whatever happened, so the platform saw a healthy container
|
|
||||||
while every report was refused with 500. It answers 200 at start, before any
|
|
||||||
write has failed, and again from the first write that succeeds. A unit test
|
|
||||||
removes `DATA_DIR` to make a write fail and checks the 503, then puts it back
|
|
||||||
and checks the 200 after the next write
|
|
||||||
- 2026-10-19: report file writes no longer overlap
|
|
||||||
([#150](https://git.eeqj.de/sneak/netwatch/issues/150)), so the reports of
|
|
||||||
failed writes are written in the order they arrived. Two writes that failed at
|
|
||||||
the same time put their reports back in the wrong order when the older failed
|
|
||||||
first. One write runs at a time, and each takes the waiting reports only when
|
|
||||||
it starts, after a failed write has put its own back. The write `Append`
|
|
||||||
starts when 10 MiB are waiting waits for any write under way, and `Append`
|
|
||||||
starts no other while it waits. Files written since the start are deleted to
|
|
||||||
make room in the order they were written, after those found at start. A unit
|
|
||||||
test holds a write open, starts a second at 10 MiB, fails both and checks the
|
|
||||||
order in the next file. Others check that `Append` starts no second write at
|
|
||||||
10 MiB while one waits, that it starts one again the next time 10 MiB are
|
|
||||||
waiting, and that the first file written since the start is the first deleted
|
|
||||||
- 2026-10-19: `DEBUG` unset or empty leaves debug logging off, and any other
|
- 2026-10-19: `DEBUG` unset or empty leaves debug logging off, and any other
|
||||||
value, `false` and `0` included, turns it on, as the code style guide says
|
value, `false` and `0` included, turns it on, as the code style guide says
|
||||||
([#159](https://git.eeqj.de/sneak/netwatch/issues/159)). No value of it stops
|
([#159](https://git.eeqj.de/sneak/netwatch/issues/159)). No value of it stops
|
||||||
the start any more; before, only `1`, `t`, `T`, `TRUE`, `true`, `True`, `0`,
|
the start any more; before, only `1`, `t`, `T`, `TRUE`, `true`, `True`, `0`,
|
||||||
`f`, `F`, `FALSE`, `false` and `False` were accepted. `README.md` no longer
|
`f`, `F`, `FALSE`, `false` and `False` were accepted. `README.md` no longer
|
||||||
gives `false` as its default
|
gives `false` as its default
|
||||||
- 2026-10-19: check builds keep Go's modules and compiled packages between runs
|
|
||||||
([#118](https://git.eeqj.de/sneak/netwatch/issues/118)), on sneak's rulings in
|
|
||||||
[prompts#113](https://git.eeqj.de/sneak/prompts/issues/113) and
|
|
||||||
[prompts#130](https://git.eeqj.de/sneak/prompts/issues/130).
|
|
||||||
`REPO_POLICIES.md`, `script/test`, `script/lint`, `script/docker` and
|
|
||||||
`script/cibuild` are the sneak/prompts copies at `0653579`: the scripts pass a
|
|
||||||
new `CACHEBUST` value where they passed `--no-cache`, and the lint and test
|
|
||||||
builds write no image. In each stage of `Dockerfile`, `ARG CACHEBUST` comes
|
|
||||||
after the system packages and the Go module download, and before every step
|
|
||||||
that lints, tests or compiles and the node stages' `yarn install`. Go's build
|
|
||||||
cache and golangci-lint's are cache mounts, and both `go test` runs carry
|
|
||||||
`-count=1`. The test phase's Go caches used to be in memory, empty on every
|
|
||||||
build
|
|
||||||
- 2026-10-19: the three unit tests of a check whose result is discarded (its
|
- 2026-10-19: the three unit tests of a check whose result is discarded (its
|
||||||
round given up, the user paused, or the first round) also check that the
|
round given up, the user paused, or the first round) also check that the
|
||||||
host's history is still empty, its status still `pending`, and nothing was
|
host's history is still empty, its status still `pending`, and nothing was
|
||||||
@@ -751,3 +627,9 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of
|
|||||||
- A configurable host list (an environment variable or a config file)
|
- A configurable host list (an environment variable or a config file)
|
||||||
- Export of the latency history (CSV or JSON)
|
- Export of the latency history (CSV or JSON)
|
||||||
- A notification when the health status changes to DEGRADED
|
- A notification when the health status changes to DEGRADED
|
||||||
|
- `make test` within the 20-second target
|
||||||
|
([#118](https://git.eeqj.de/sneak/netwatch/issues/118)): every run compiles
|
||||||
|
the Go standard library and every dependency with the race detector from
|
||||||
|
nothing. Whether test builds may keep Go's compiled packages between runs
|
||||||
|
waits on sneak's ruling in
|
||||||
|
[prompts#113](https://git.eeqj.de/sneak/prompts/issues/113)
|
||||||
|
|||||||
+63
-91
@@ -13,27 +13,24 @@ set -u
|
|||||||
# address and start anyway, and reports a bad port without naming
|
# address and start anyway, and reports a bad port without naming
|
||||||
# PORT, so a value that is not a usable port stops the container here,
|
# PORT, so a value that is not a usable port stops the container here,
|
||||||
# before either process starts.
|
# before either process starts.
|
||||||
check_port() {
|
export PORT="${PORT:-8080}"
|
||||||
export PORT="${PORT:-8080}"
|
case "$PORT" in
|
||||||
case "$PORT" in
|
*[!0-9]*)
|
||||||
*[!0-9]*)
|
echo "entrypoint: PORT must be a port number, not '$PORT'" >&2
|
||||||
echo "entrypoint: PORT must be a port number, not '$PORT'" >&2
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
# The length is checked first because, for a number too big for it,
|
|
||||||
# the shell's test prints an error and is false, so the range checks
|
|
||||||
# alone would let it through.
|
|
||||||
if [ "${#PORT}" -gt 5 ] || [ "$PORT" -lt 1 ] || [ "$PORT" -gt 65535 ]; then
|
|
||||||
echo "entrypoint: PORT must be from 1 to 65535, not '$PORT'" >&2
|
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
;;
|
||||||
if [ "$PORT" -eq 8081 ]; then
|
esac
|
||||||
echo "entrypoint: PORT cannot be 8081, netwatch-server" \
|
# The length is checked first because, for a number too big for it,
|
||||||
"listens there" >&2
|
# the shell's test prints an error and is false, so the range checks
|
||||||
exit 1
|
# alone would let it through.
|
||||||
fi
|
if [ "${#PORT}" -gt 5 ] || [ "$PORT" -lt 1 ] || [ "$PORT" -gt 65535 ]; then
|
||||||
}
|
echo "entrypoint: PORT must be from 1 to 65535, not '$PORT'" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
if [ "$PORT" -eq 8081 ]; then
|
||||||
|
echo "entrypoint: PORT cannot be 8081, netwatch-server listens there" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
# TRUSTED_PROXIES names the reverse proxies in front of the container,
|
# TRUSTED_PROXIES names the reverse proxies in front of the container,
|
||||||
# as IP addresses or CIDRs separated by commas. nginx takes the client
|
# as IP addresses or CIDRs separated by commas. nginx takes the client
|
||||||
@@ -50,23 +47,21 @@ check_port() {
|
|||||||
# TRUSTED_PROXIES. Its error, naming the CIDR, is dropped for the one
|
# TRUSTED_PROXIES. Its error, naming the CIDR, is dropped for the one
|
||||||
# below, naming the entry as written. set -f keeps a * in an entry from
|
# below, naming the entry as written. set -f keeps a * in an entry from
|
||||||
# becoming a list of file names.
|
# becoming a list of file names.
|
||||||
write_trusted_proxies() {
|
TRUSTED_PROXIES="${TRUSTED_PROXIES-10.0.0.0/8,172.16.0.0/12,192.168.0.0/16}"
|
||||||
TRUSTED_PROXIES="${TRUSTED_PROXIES-10.0.0.0/8,172.16.0.0/12,192.168.0.0/16}"
|
set -f
|
||||||
set -f
|
for proxy in $(printf '%s' "$TRUSTED_PROXIES" | tr ',' ' '); do
|
||||||
for proxy in $(printf '%s' "$TRUSTED_PROXIES" | tr ',' ' '); do
|
case "$proxy" in
|
||||||
case "$proxy" in
|
*/*) cidr="$proxy" ;;
|
||||||
*/*) cidr="$proxy" ;;
|
*:*) cidr="$proxy/128" ;;
|
||||||
*:*) cidr="$proxy/128" ;;
|
*) cidr="$proxy/32" ;;
|
||||||
*) cidr="$proxy/32" ;;
|
esac
|
||||||
esac
|
if ! netwatch-server check-cidr "$cidr" 2> /dev/null; then
|
||||||
if ! netwatch-server check-cidr "$cidr" 2> /dev/null; then
|
echo "entrypoint: TRUSTED_PROXIES must be IP addresses or CIDRs" \
|
||||||
echo "entrypoint: TRUSTED_PROXIES must be IP addresses or CIDRs" \
|
"separated by commas; '$proxy' is neither" >&2
|
||||||
"separated by commas; '$proxy' is neither" >&2
|
exit 1
|
||||||
exit 1
|
fi
|
||||||
fi
|
echo "set_real_ip_from $cidr;"
|
||||||
echo "set_real_ip_from $cidr;"
|
done > /etc/nginx/trusted-proxies.conf
|
||||||
done > /etc/nginx/trusted-proxies.conf
|
|
||||||
}
|
|
||||||
|
|
||||||
# netwatch-server keeps its report files in DATA_DIR, on the /data
|
# netwatch-server keeps its report files in DATA_DIR, on the /data
|
||||||
# volume, which may be a host directory owned by root or by another
|
# volume, which may be a host directory owned by root or by another
|
||||||
@@ -78,10 +73,12 @@ write_trusted_proxies() {
|
|||||||
# naming DATA_DIR, unless DATA_DIR is /data or a path below it, and it
|
# naming DATA_DIR, unless DATA_DIR is /data or a path below it, and it
|
||||||
# acts on nothing outside /data, whatever symbolic or hard links it
|
# acts on nothing outside /data, whatever symbolic or hard links it
|
||||||
# meets there.
|
# meets there.
|
||||||
prepare_data_dir() {
|
export DATA_DIR="${DATA_DIR:-/data/reports}"
|
||||||
export DATA_DIR="${DATA_DIR:-/data/reports}"
|
netwatch-server prepare-data-dir "$DATA_DIR" || exit 1
|
||||||
netwatch-server prepare-data-dir "$DATA_DIR" || exit 1
|
|
||||||
}
|
# A stop signal is only noted here; the loop below acts on it.
|
||||||
|
stop_requested=""
|
||||||
|
trap 'stop_requested=yes' TERM INT
|
||||||
|
|
||||||
# netwatch-server runs as the netwatch user and listens on loopback
|
# netwatch-server runs as the netwatch user and listens on loopback
|
||||||
# only, on a port other than the public one; configs/nginx.conf proxies
|
# only, on a port other than the public one; configs/nginx.conf proxies
|
||||||
@@ -90,11 +87,9 @@ prepare_data_dir() {
|
|||||||
# the container has. The netwatch user has no login shell, hence -s
|
# the container has. The netwatch user has no login shell, hence -s
|
||||||
# /bin/sh. busybox su replaces itself with the command instead of
|
# /bin/sh. busybox su replaces itself with the command instead of
|
||||||
# staying on as its parent, so $! is the server's own PID.
|
# staying on as its parent, so $! is the server's own PID.
|
||||||
start_backend() {
|
BIND_ADDRESS=127.0.0.1 PORT=8081 TRUSTED_PROXIES=127.0.0.1/32 \
|
||||||
BIND_ADDRESS=127.0.0.1 PORT=8081 TRUSTED_PROXIES=127.0.0.1/32 \
|
su -s /bin/sh netwatch -c 'exec netwatch-server' &
|
||||||
su -s /bin/sh netwatch -c 'exec netwatch-server' &
|
backend=$!
|
||||||
backend=$!
|
|
||||||
}
|
|
||||||
|
|
||||||
# nginx starts through the nginx image's own entrypoint, which applies
|
# nginx starts through the nginx image's own entrypoint, which applies
|
||||||
# the image's start-up configuration and then replaces itself with
|
# the image's start-up configuration and then replaces itself with
|
||||||
@@ -103,11 +98,9 @@ start_backend() {
|
|||||||
# that rendering to PORT: a variable nginx itself uses, such as $uri,
|
# that rendering to PORT: a variable nginx itself uses, such as $uri,
|
||||||
# would otherwise be replaced by an environment variable of the same
|
# would otherwise be replaced by an environment variable of the same
|
||||||
# name.
|
# name.
|
||||||
start_nginx() {
|
NGINX_ENVSUBST_FILTER='^PORT$' \
|
||||||
NGINX_ENVSUBST_FILTER='^PORT$' \
|
/docker-entrypoint.sh nginx -g 'daemon off;' &
|
||||||
/docker-entrypoint.sh nginx -g 'daemon off;' &
|
nginx=$!
|
||||||
nginx=$!
|
|
||||||
}
|
|
||||||
|
|
||||||
running() {
|
running() {
|
||||||
kill -0 "$1" 2>/dev/null
|
kill -0 "$1" 2>/dev/null
|
||||||
@@ -116,47 +109,26 @@ running() {
|
|||||||
# POSIX sh cannot wait for whichever of two children exits first, so
|
# POSIX sh cannot wait for whichever of two children exits first, so
|
||||||
# look once a second. The shell collects a child that has exited while
|
# look once a second. The shell collects a child that has exited while
|
||||||
# it runs sleep, and running() is false for that child from then on.
|
# it runs sleep, and running() is false for that child from then on.
|
||||||
wait_for_stop_or_exit() {
|
while [ -z "$stop_requested" ] && running "$backend" && running "$nginx"; do
|
||||||
while [ -z "$stop_requested" ] && running "$backend" && running "$nginx"; do
|
sleep 1
|
||||||
sleep 1
|
done
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
# Stop both, then wait until neither is left.
|
# Stop both, then wait until neither is left.
|
||||||
stop_both() {
|
kill -TERM "$backend" "$nginx" 2>/dev/null
|
||||||
kill -TERM "$backend" "$nginx" 2>/dev/null
|
while running "$backend" || running "$nginx"; do
|
||||||
while running "$backend" || running "$nginx"; do
|
sleep 1
|
||||||
sleep 1
|
done
|
||||||
done
|
|
||||||
}
|
wait "$backend"
|
||||||
|
backend_status=$?
|
||||||
|
wait "$nginx"
|
||||||
|
nginx_status=$?
|
||||||
|
echo "entrypoint: netwatch-server exited $backend_status," \
|
||||||
|
"nginx exited $nginx_status"
|
||||||
|
|
||||||
# Success is a requested stop that both processes exited cleanly from.
|
# Success is a requested stop that both processes exited cleanly from.
|
||||||
exit_with_status() {
|
if [ -n "$stop_requested" ] && [ "$backend_status" -eq 0 ] &&
|
||||||
wait "$backend"
|
[ "$nginx_status" -eq 0 ]; then
|
||||||
backend_status=$?
|
exit 0
|
||||||
wait "$nginx"
|
fi
|
||||||
nginx_status=$?
|
exit 1
|
||||||
echo "entrypoint: netwatch-server exited $backend_status," \
|
|
||||||
"nginx exited $nginx_status"
|
|
||||||
if [ -n "$stop_requested" ] && [ "$backend_status" -eq 0 ] &&
|
|
||||||
[ "$nginx_status" -eq 0 ]; then
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
exit 1
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
|
||||||
check_port
|
|
||||||
write_trusted_proxies
|
|
||||||
prepare_data_dir
|
|
||||||
# A stop signal is only noted here; wait_for_stop_or_exit acts on it.
|
|
||||||
stop_requested=""
|
|
||||||
trap 'stop_requested=yes' TERM INT
|
|
||||||
start_backend
|
|
||||||
start_nginx
|
|
||||||
wait_for_stop_or_exit
|
|
||||||
stop_both
|
|
||||||
exit_with_status
|
|
||||||
}
|
|
||||||
|
|
||||||
main "$@"
|
|
||||||
|
|||||||
@@ -1,11 +1,22 @@
|
|||||||
// Package main is the entry point for netwatch-server; package cli
|
// Package main is the entry point for netwatch-server.
|
||||||
// does the work.
|
|
||||||
package main
|
package main
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
|
"os/user"
|
||||||
|
|
||||||
"sneak.berlin/go/netwatch/internal/cli"
|
"sneak.berlin/go/netwatch/internal/config"
|
||||||
|
"sneak.berlin/go/netwatch/internal/globals"
|
||||||
|
"sneak.berlin/go/netwatch/internal/handlers"
|
||||||
|
"sneak.berlin/go/netwatch/internal/healthcheck"
|
||||||
|
"sneak.berlin/go/netwatch/internal/logger"
|
||||||
|
"sneak.berlin/go/netwatch/internal/middleware"
|
||||||
|
"sneak.berlin/go/netwatch/internal/reportbuf"
|
||||||
|
"sneak.berlin/go/netwatch/internal/server"
|
||||||
|
|
||||||
|
"go.uber.org/fx"
|
||||||
|
"go.uber.org/fx/fxevent"
|
||||||
)
|
)
|
||||||
|
|
||||||
//nolint:gochecknoglobals // set via ldflags at build time
|
//nolint:gochecknoglobals // set via ldflags at build time
|
||||||
@@ -15,5 +26,64 @@ var (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
os.Exit(cli.Main(os.Stderr, Appname, Version, os.Args[1:], cli.RunServer))
|
// "netwatch-server check-cidr CIDR" exits 1, with the error, if
|
||||||
|
// this server would refuse CIDR in its TRUSTED_PROXIES.
|
||||||
|
// bin/entrypoint.sh runs it on each entry it gives nginx.
|
||||||
|
if len(os.Args) == 3 && os.Args[1] == "check-cidr" {
|
||||||
|
_, err := middleware.ParseTrustedProxies(os.Args[2:])
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// "netwatch-server prepare-data-dir DATA_DIR" gets DATA_DIR ready
|
||||||
|
// for the netwatch user, or exits 1 with the error; see
|
||||||
|
// reportbuf.PrepareDataDir. bin/entrypoint.sh runs it as root
|
||||||
|
// before it starts this server as that user.
|
||||||
|
if len(os.Args) == 3 && os.Args[1] == "prepare-data-dir" {
|
||||||
|
netwatch, err := user.Lookup("netwatch")
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = reportbuf.PrepareDataDir("/data", os.Args[2], netwatch)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "DATA_DIR '%s': %v\n", os.Args[2], err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
globals.Appname = Appname
|
||||||
|
globals.Version = Version
|
||||||
|
|
||||||
|
fx.New(
|
||||||
|
// fx logs each step of starting and stopping through the
|
||||||
|
// server's own logger, so off a terminal those lines are
|
||||||
|
// JSON like every other line.
|
||||||
|
fx.WithLogger(func(log *logger.Logger) fxevent.Logger {
|
||||||
|
return &fxevent.SlogLogger{Logger: log.Get()}
|
||||||
|
}),
|
||||||
|
fx.Provide(
|
||||||
|
config.New,
|
||||||
|
globals.New,
|
||||||
|
handlers.New,
|
||||||
|
healthcheck.New,
|
||||||
|
logger.New,
|
||||||
|
middleware.New,
|
||||||
|
reportbuf.New,
|
||||||
|
server.New,
|
||||||
|
),
|
||||||
|
fx.Invoke(
|
||||||
|
// First, so the name and version are logged even when
|
||||||
|
// a setting stops the start.
|
||||||
|
func(log *logger.Logger) { log.Identify() },
|
||||||
|
func(*server.Server) {},
|
||||||
|
),
|
||||||
|
).Run()
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,143 +0,0 @@
|
|||||||
// Package cli runs netwatch-server from its command line: the
|
|
||||||
// check-cidr and prepare-data-dir subcommands, which bin/entrypoint.sh
|
|
||||||
// runs before it starts the server, and, given no arguments, the
|
|
||||||
// server, with the fx wiring.
|
|
||||||
package cli
|
|
||||||
|
|
||||||
import (
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"os/user"
|
|
||||||
|
|
||||||
"sneak.berlin/go/netwatch/internal/config"
|
|
||||||
"sneak.berlin/go/netwatch/internal/globals"
|
|
||||||
"sneak.berlin/go/netwatch/internal/handlers"
|
|
||||||
"sneak.berlin/go/netwatch/internal/healthcheck"
|
|
||||||
"sneak.berlin/go/netwatch/internal/logger"
|
|
||||||
"sneak.berlin/go/netwatch/internal/middleware"
|
|
||||||
"sneak.berlin/go/netwatch/internal/reportbuf"
|
|
||||||
"sneak.berlin/go/netwatch/internal/server"
|
|
||||||
|
|
||||||
"go.uber.org/fx"
|
|
||||||
"go.uber.org/fx/fxevent"
|
|
||||||
)
|
|
||||||
|
|
||||||
// usageExitCode is what Main returns for a command line it does not
|
|
||||||
// understand, the usual exit code for a usage error.
|
|
||||||
const usageExitCode = 2
|
|
||||||
|
|
||||||
// Main runs netwatch-server with args, its command line after the
|
|
||||||
// program's name, and returns the exit code. With no args, Main runs
|
|
||||||
// the server by calling serve with appname and version, the name and
|
|
||||||
// version the server logs and reports. With "check-cidr CIDR" or
|
|
||||||
// "prepare-data-dir DATA_DIR" it runs that subcommand, which writes its
|
|
||||||
// errors to stderr. Any other args get a one-line usage message on
|
|
||||||
// stderr and exit code 2. main passes os.Stderr and RunServer; the
|
|
||||||
// tests pass a buffer and a function of their own, so that no test
|
|
||||||
// starts the server. When serve is RunServer and the server fails to
|
|
||||||
// start, to listen or to stop, fx's Run exits the process with status 1
|
|
||||||
// and Main never returns.
|
|
||||||
func Main(
|
|
||||||
stderr io.Writer,
|
|
||||||
appname, version string,
|
|
||||||
args []string,
|
|
||||||
serve func(appname, version string),
|
|
||||||
) int {
|
|
||||||
// "netwatch-server check-cidr CIDR" exits 1, with the error, if
|
|
||||||
// this server would refuse CIDR in its TRUSTED_PROXIES.
|
|
||||||
// bin/entrypoint.sh runs it on each entry it gives nginx.
|
|
||||||
if len(args) == 2 && args[0] == "check-cidr" {
|
|
||||||
return checkCIDR(stderr, args[1])
|
|
||||||
}
|
|
||||||
|
|
||||||
// "netwatch-server prepare-data-dir DATA_DIR" gets DATA_DIR ready
|
|
||||||
// for the netwatch user, or exits 1 with the error; see
|
|
||||||
// reportbuf.PrepareDataDir. bin/entrypoint.sh runs it as root
|
|
||||||
// before it starts this server as that user.
|
|
||||||
if len(args) == 2 && args[0] == "prepare-data-dir" {
|
|
||||||
return prepareDataDir(stderr, "/data", args[1], "netwatch")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Any other command line is refused, so that a mistyped one does
|
|
||||||
// not start the server.
|
|
||||||
if len(args) > 0 {
|
|
||||||
_, _ = fmt.Fprintln(stderr,
|
|
||||||
"usage: netwatch-server [check-cidr CIDR | prepare-data-dir DATA_DIR]")
|
|
||||||
|
|
||||||
return usageExitCode
|
|
||||||
}
|
|
||||||
|
|
||||||
serve(appname, version)
|
|
||||||
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkCIDR returns 0 if the server would take cidr in its
|
|
||||||
// TRUSTED_PROXIES, and otherwise writes the error to stderr and
|
|
||||||
// returns 1.
|
|
||||||
func checkCIDR(stderr io.Writer, cidr string) int {
|
|
||||||
_, err := middleware.ParseTrustedProxies([]string{cidr})
|
|
||||||
if err != nil {
|
|
||||||
_, _ = fmt.Fprintln(stderr, err)
|
|
||||||
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
// prepareDataDir gets dataDir ready for the user named username with
|
|
||||||
// reportbuf.PrepareDataDir, volume being where the data volume is
|
|
||||||
// mounted, and returns 0. Otherwise it writes the error to stderr and
|
|
||||||
// returns 1.
|
|
||||||
func prepareDataDir(stderr io.Writer, volume, dataDir, username string) int {
|
|
||||||
owner, err := user.Lookup(username)
|
|
||||||
if err != nil {
|
|
||||||
_, _ = fmt.Fprintln(stderr, err)
|
|
||||||
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
err = reportbuf.PrepareDataDir(volume, dataDir, owner)
|
|
||||||
if err != nil {
|
|
||||||
_, _ = fmt.Fprintf(stderr, "DATA_DIR '%s': %v\n", dataDir, err)
|
|
||||||
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
// RunServer runs the server until it is told to stop, with appname and
|
|
||||||
// version as the name and version it logs and reports. If the server
|
|
||||||
// fails to start, to listen or to stop, fx's Run exits the process
|
|
||||||
// with status 1 itself.
|
|
||||||
func RunServer(appname, version string) {
|
|
||||||
globals.Appname = appname
|
|
||||||
globals.Version = version
|
|
||||||
|
|
||||||
fx.New(
|
|
||||||
// fx logs each step of starting and stopping through the
|
|
||||||
// server's own logger, so off a terminal those lines are
|
|
||||||
// JSON like every other line.
|
|
||||||
fx.WithLogger(func(log *logger.Logger) fxevent.Logger {
|
|
||||||
return &fxevent.SlogLogger{Logger: log.Get()}
|
|
||||||
}),
|
|
||||||
fx.Provide(
|
|
||||||
config.New,
|
|
||||||
globals.New,
|
|
||||||
handlers.New,
|
|
||||||
healthcheck.New,
|
|
||||||
logger.New,
|
|
||||||
middleware.New,
|
|
||||||
reportbuf.New,
|
|
||||||
server.New,
|
|
||||||
),
|
|
||||||
fx.Invoke(
|
|
||||||
// First, so the name and version are logged even when
|
|
||||||
// a setting stops the start.
|
|
||||||
func(log *logger.Logger) { log.Identify() },
|
|
||||||
func(*server.Server) {},
|
|
||||||
),
|
|
||||||
).Run()
|
|
||||||
}
|
|
||||||
@@ -1,219 +0,0 @@
|
|||||||
package cli_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"os"
|
|
||||||
"os/user"
|
|
||||||
"path/filepath"
|
|
||||||
"strconv"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"sneak.berlin/go/netwatch/internal/cli"
|
|
||||||
"sneak.berlin/go/netwatch/internal/reportbuf"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestMainRunsCheckCIDR: Main given check-cidr returns 0, writing
|
|
||||||
// nothing, for a CIDR the server takes in its TRUSTED_PROXIES, and 1,
|
|
||||||
// with an error naming TRUSTED_PROXIES and the CIDR, for one it
|
|
||||||
// refuses, and does not start the server.
|
|
||||||
func TestMainRunsCheckCIDR(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
var stderr bytes.Buffer
|
|
||||||
|
|
||||||
code := cli.Main(&stderr, "netwatch", "test",
|
|
||||||
[]string{"check-cidr", "10.0.0.0/8"}, failIfServerStarts(t))
|
|
||||||
if code != 0 || stderr.Len() != 0 {
|
|
||||||
t.Errorf("check-cidr 10.0.0.0/8: exit %d, stderr %q; want exit 0 and no output",
|
|
||||||
code, stderr.String())
|
|
||||||
}
|
|
||||||
|
|
||||||
stderr.Reset()
|
|
||||||
|
|
||||||
code = cli.Main(&stderr, "netwatch", "test",
|
|
||||||
[]string{"check-cidr", "1.2.3/32"}, failIfServerStarts(t))
|
|
||||||
|
|
||||||
want := `TRUSTED_PROXIES "1.2.3/32": `
|
|
||||||
if code != 1 || !strings.HasPrefix(stderr.String(), want) {
|
|
||||||
t.Errorf("check-cidr 1.2.3/32: exit %d, stderr %q; want exit 1 and %q...",
|
|
||||||
code, stderr.String(), want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestMainRunsPrepareDataDir: Main given prepare-data-dir runs the
|
|
||||||
// prepare-data-dir subcommand for the netwatch user, and does not start
|
|
||||||
// the server. The test image has no netwatch user, so that subcommand
|
|
||||||
// exits 1 with the error from looking the user up;
|
|
||||||
// TestPrepareDataDirRefusesPathOutsideVolume checks the path.
|
|
||||||
func TestMainRunsPrepareDataDir(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
var stderr bytes.Buffer
|
|
||||||
|
|
||||||
code := cli.Main(&stderr, "netwatch", "test",
|
|
||||||
[]string{"prepare-data-dir", "/etc"}, failIfServerStarts(t))
|
|
||||||
|
|
||||||
want := user.UnknownUserError("netwatch").Error() + "\n"
|
|
||||||
if code != 1 || stderr.String() != want {
|
|
||||||
t.Errorf("prepare-data-dir /etc: exit %d, stderr %q; want exit 1 and %q",
|
|
||||||
code, stderr.String(), want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestMainRunsServerWithNoArguments: Main given no arguments starts the
|
|
||||||
// server with the name and version it was given, writes nothing and
|
|
||||||
// returns 0.
|
|
||||||
func TestMainRunsServerWithNoArguments(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
var (
|
|
||||||
stderr bytes.Buffer
|
|
||||||
gotAppname, gotVersion string
|
|
||||||
)
|
|
||||||
|
|
||||||
code := cli.Main(&stderr, "netwatch", "test", []string{},
|
|
||||||
func(appname, version string) {
|
|
||||||
gotAppname, gotVersion = appname, version
|
|
||||||
})
|
|
||||||
if code != 0 || stderr.Len() != 0 {
|
|
||||||
t.Errorf("exit %d, stderr %q; want exit 0 and no output",
|
|
||||||
code, stderr.String())
|
|
||||||
}
|
|
||||||
|
|
||||||
if gotAppname != "netwatch" || gotVersion != "test" {
|
|
||||||
t.Errorf("server started with %q %q; want %q %q",
|
|
||||||
gotAppname, gotVersion, "netwatch", "test")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestMainRefusesOtherCommandLines: Main given a subcommand with too
|
|
||||||
// few or too many values, or a misspelt subcommand, writes the usage
|
|
||||||
// message to stderr, returns 2 and does not start the server.
|
|
||||||
func TestMainRefusesOtherCommandLines(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
want := "usage: netwatch-server " +
|
|
||||||
"[check-cidr CIDR | prepare-data-dir DATA_DIR]\n"
|
|
||||||
|
|
||||||
for _, commandLine := range []string{
|
|
||||||
"netwatch-server check-cidr",
|
|
||||||
"netwatch-server check-cidr 192.0.2.0/24 198.51.100.0/24",
|
|
||||||
"netwatch-server prepare-data-dir",
|
|
||||||
"netwatch-server prepare-data-dir /data/reports /data/other",
|
|
||||||
"netwatch-server check-cdir 192.0.2.0/24",
|
|
||||||
} {
|
|
||||||
var stderr bytes.Buffer
|
|
||||||
|
|
||||||
// Main takes the command line after the program's name.
|
|
||||||
args := strings.Fields(commandLine)[1:]
|
|
||||||
|
|
||||||
code := cli.Main(&stderr, "netwatch", "test", args, failIfServerStarts(t))
|
|
||||||
if code != 2 || stderr.String() != want {
|
|
||||||
t.Errorf("%q: exit %d, stderr %q; want exit 2 and %q",
|
|
||||||
commandLine, code, stderr.String(), want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestCheckCIDRTakesCIDR: check-cidr exits 0, writing nothing, for a
|
|
||||||
// CIDR the server takes in its TRUSTED_PROXIES.
|
|
||||||
func TestCheckCIDRTakesCIDR(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, cidr := range []string{"10.0.0.0/8", "192.0.2.1/32", "2001:db8::/32"} {
|
|
||||||
var stderr bytes.Buffer
|
|
||||||
|
|
||||||
code := cli.CheckCIDR(&stderr, cidr)
|
|
||||||
if code != 0 || stderr.Len() != 0 {
|
|
||||||
t.Errorf("%q: exit %d, stderr %q; want exit 0 and no output",
|
|
||||||
cidr, code, stderr.String())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestCheckCIDRRefusesOtherInput: check-cidr exits 1, with an error
|
|
||||||
// naming TRUSTED_PROXIES and the input, for anything the server would
|
|
||||||
// refuse in its TRUSTED_PROXIES, such as 1.2.3/32, which nginx would
|
|
||||||
// look up as a hostname.
|
|
||||||
func TestCheckCIDRRefusesOtherInput(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, cidr := range []string{
|
|
||||||
"1.2.3/32", "example.com/32", "10.0.0.0/33", "10.0.0.1", "",
|
|
||||||
} {
|
|
||||||
var stderr bytes.Buffer
|
|
||||||
|
|
||||||
code := cli.CheckCIDR(&stderr, cidr)
|
|
||||||
|
|
||||||
want := "TRUSTED_PROXIES " + strconv.Quote(cidr) + ": "
|
|
||||||
if code != 1 || !strings.HasPrefix(stderr.String(), want) {
|
|
||||||
t.Errorf("%q: exit %d, stderr %q; want exit 1 and %q...",
|
|
||||||
cidr, code, stderr.String(), want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrepareDataDirTakesPathInVolume: prepare-data-dir exits 0,
|
|
||||||
// writing nothing, for a DATA_DIR below the volume, and creates it.
|
|
||||||
func TestPrepareDataDirTakesPathInVolume(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
volume := t.TempDir()
|
|
||||||
dataDir := filepath.Join(volume, "reports")
|
|
||||||
|
|
||||||
var stderr bytes.Buffer
|
|
||||||
|
|
||||||
code := cli.PrepareDataDir(&stderr, volume, dataDir, currentUsername(t))
|
|
||||||
if code != 0 || stderr.Len() != 0 {
|
|
||||||
t.Errorf("exit %d, stderr %q; want exit 0 and no output",
|
|
||||||
code, stderr.String())
|
|
||||||
}
|
|
||||||
|
|
||||||
info, err := os.Stat(dataDir)
|
|
||||||
if err != nil || !info.IsDir() {
|
|
||||||
t.Errorf("DATA_DIR not created: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrepareDataDirRefusesPathOutsideVolume: prepare-data-dir exits
|
|
||||||
// 1, with an error naming DATA_DIR and its value, for a DATA_DIR
|
|
||||||
// outside the volume.
|
|
||||||
func TestPrepareDataDirRefusesPathOutsideVolume(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
volume := t.TempDir()
|
|
||||||
|
|
||||||
var stderr bytes.Buffer
|
|
||||||
|
|
||||||
code := cli.PrepareDataDir(&stderr, volume, "/etc", currentUsername(t))
|
|
||||||
|
|
||||||
want := "DATA_DIR '/etc': " + reportbuf.ErrDataDirOutsideVolume.Error() + "\n"
|
|
||||||
if code != 1 || stderr.String() != want {
|
|
||||||
t.Errorf("exit %d, stderr %q; want exit 1 and %q",
|
|
||||||
code, stderr.String(), want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// currentUsername is the name of the user the test runs as, the only
|
|
||||||
// owner a test not run as root can give files to.
|
|
||||||
func currentUsername(t *testing.T) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
current, err := user.Current()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return current.Username
|
|
||||||
}
|
|
||||||
|
|
||||||
// failIfServerStarts is what the tests give Main in place of
|
|
||||||
// cli.RunServer: it fails the test instead of starting the server.
|
|
||||||
func failIfServerStarts(t *testing.T) func(appname, version string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
return func(string, string) {
|
|
||||||
t.Error("Main started the server")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,16 +0,0 @@
|
|||||||
package cli
|
|
||||||
|
|
||||||
import "io"
|
|
||||||
|
|
||||||
// CheckCIDR exposes the check-cidr subcommand to the external tests,
|
|
||||||
// which read what it writes to stderr.
|
|
||||||
func CheckCIDR(stderr io.Writer, cidr string) int {
|
|
||||||
return checkCIDR(stderr, cidr)
|
|
||||||
}
|
|
||||||
|
|
||||||
// PrepareDataDir exposes the prepare-data-dir subcommand to the
|
|
||||||
// external tests, which give it a volume and a user of their own in
|
|
||||||
// place of /data and netwatch.
|
|
||||||
func PrepareDataDir(stderr io.Writer, volume, dataDir, username string) int {
|
|
||||||
return prepareDataDir(stderr, volume, dataDir, username)
|
|
||||||
}
|
|
||||||
@@ -13,8 +13,8 @@ import (
|
|||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
)
|
)
|
||||||
|
|
||||||
// requireConfigError builds the config as cli.RunServer does and fails
|
// requireConfigError builds the config as main does and fails the
|
||||||
// the test unless that fails with an error naming setting. It uses
|
// test unless that fails with an error naming setting. It uses
|
||||||
// fx.New, because fxtest.New fails the test itself on an error.
|
// fx.New, because fxtest.New fails the test itself on an error.
|
||||||
func requireConfigError(t *testing.T, setting string) {
|
func requireConfigError(t *testing.T, setting string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
@@ -31,8 +31,8 @@ func requireConfigError(t *testing.T, setting string) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// requireTrustedProxies builds the config as cli.RunServer does and
|
// requireTrustedProxies builds the config as main does and fails the
|
||||||
// fails the test unless its TrustedProxies are want.
|
// test unless its TrustedProxies are want.
|
||||||
func requireTrustedProxies(t *testing.T, want []string) {
|
func requireTrustedProxies(t *testing.T, want []string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ package globals
|
|||||||
|
|
||||||
import "go.uber.org/fx"
|
import "go.uber.org/fx"
|
||||||
|
|
||||||
//nolint:gochecknoglobals // set by cli.RunServer before fx starts
|
//nolint:gochecknoglobals // set from main before fx starts
|
||||||
var (
|
var (
|
||||||
// Appname is the application name.
|
// Appname is the application name.
|
||||||
Appname string
|
Appname string
|
||||||
|
|||||||
@@ -3,17 +3,9 @@ package handlers
|
|||||||
import "net/http"
|
import "net/http"
|
||||||
|
|
||||||
// HandleHealthCheck returns a handler for the health check
|
// HandleHealthCheck returns a handler for the health check
|
||||||
// endpoint. It answers 200 while the status is "ok" and 503 otherwise,
|
// endpoint.
|
||||||
// so that the image's HEALTHCHECK fails.
|
|
||||||
func (s *Handlers) HandleHealthCheck() http.HandlerFunc {
|
func (s *Handlers) HandleHealthCheck() http.HandlerFunc {
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
resp := s.hc.Healthcheck()
|
s.respondJSON(w, r, s.hc.Healthcheck(), http.StatusOK)
|
||||||
|
|
||||||
code := http.StatusOK
|
|
||||||
if resp.Status != "ok" {
|
|
||||||
code = http.StatusServiceUnavailable
|
|
||||||
}
|
|
||||||
|
|
||||||
s.respondJSON(w, r, resp, code)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,19 +9,17 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/netwatch/internal/config"
|
|
||||||
"sneak.berlin/go/netwatch/internal/globals"
|
"sneak.berlin/go/netwatch/internal/globals"
|
||||||
"sneak.berlin/go/netwatch/internal/handlers"
|
"sneak.berlin/go/netwatch/internal/handlers"
|
||||||
"sneak.berlin/go/netwatch/internal/healthcheck"
|
"sneak.berlin/go/netwatch/internal/healthcheck"
|
||||||
"sneak.berlin/go/netwatch/internal/logger"
|
"sneak.berlin/go/netwatch/internal/logger"
|
||||||
"sneak.berlin/go/netwatch/internal/reportbuf"
|
|
||||||
|
|
||||||
"go.uber.org/fx/fxtest"
|
"go.uber.org/fx/fxtest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// newStartedHandlers builds Handlers with a real health check for the
|
// newStartedHandlers builds Handlers with a real health check for the
|
||||||
// server named in g, and a report buffer in a temporary directory, and
|
// server named in g, and starts them, which records the time the
|
||||||
// starts them, which records the time the uptime counts from.
|
// uptime counts from.
|
||||||
func newStartedHandlers(t *testing.T, g *globals.Globals) *handlers.Handlers {
|
func newStartedHandlers(t *testing.T, g *globals.Globals) *handlers.Handlers {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -32,23 +30,14 @@ func newStartedHandlers(t *testing.T, g *globals.Globals) *handlers.Handlers {
|
|||||||
t.Fatalf("logger: %v", err)
|
t.Fatalf("logger: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
buf, err := reportbuf.New(lc, reportbuf.Params{
|
|
||||||
Config: &config.Config{DataDir: t.TempDir(), DataDirMaxBytes: 1 << 20},
|
|
||||||
Logger: log,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("report buffer: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
hc, err := healthcheck.New(lc,
|
hc, err := healthcheck.New(lc,
|
||||||
healthcheck.Params{Buffer: buf, Globals: g, Logger: log})
|
healthcheck.Params{Globals: g, Logger: log})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("health check: %v", err)
|
t.Fatalf("health check: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
h, err := handlers.New(lc, handlers.Params{
|
h, err := handlers.New(lc,
|
||||||
Buffer: buf, Globals: g, Healthcheck: hc, Logger: log,
|
handlers.Params{Globals: g, Healthcheck: hc, Logger: log})
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("handlers: %v", err)
|
t.Fatalf("handlers: %v", err)
|
||||||
}
|
}
|
||||||
@@ -59,12 +48,10 @@ func newStartedHandlers(t *testing.T, g *globals.Globals) *handlers.Handlers {
|
|||||||
return h
|
return h
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestHandleHealthCheck checks the health check's answer at start, when
|
// TestHandleHealthCheck checks the health check's answer: 200, a JSON
|
||||||
// no write of a report file has failed: 200, a JSON content type, and a
|
// content type, and a JSON object with exactly the fields of
|
||||||
// JSON object with exactly the fields of healthcheck.HealthcheckResponse,
|
// healthcheck.HealthcheckResponse, carrying this server's name and
|
||||||
// carrying "status":"ok", this server's name and version and an uptime
|
// version and an uptime counted from its start.
|
||||||
// counted from its start. The answer while writes fail is checked in
|
|
||||||
// internal/reportbuf, whose tests can make a write fail.
|
|
||||||
func TestHandleHealthCheck(t *testing.T) {
|
func TestHandleHealthCheck(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -10,7 +10,6 @@ import (
|
|||||||
"sneak.berlin/go/netwatch/internal/config"
|
"sneak.berlin/go/netwatch/internal/config"
|
||||||
"sneak.berlin/go/netwatch/internal/globals"
|
"sneak.berlin/go/netwatch/internal/globals"
|
||||||
"sneak.berlin/go/netwatch/internal/logger"
|
"sneak.berlin/go/netwatch/internal/logger"
|
||||||
"sneak.berlin/go/netwatch/internal/reportbuf"
|
|
||||||
|
|
||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
)
|
)
|
||||||
@@ -19,7 +18,6 @@ import (
|
|||||||
type Params struct {
|
type Params struct {
|
||||||
fx.In
|
fx.In
|
||||||
|
|
||||||
Buffer *reportbuf.Buffer
|
|
||||||
Config *config.Config
|
Config *config.Config
|
||||||
Globals *globals.Globals
|
Globals *globals.Globals
|
||||||
Logger *logger.Logger
|
Logger *logger.Logger
|
||||||
@@ -70,18 +68,12 @@ func New(
|
|||||||
return s, nil
|
return s, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Healthcheck returns the current health status of the application:
|
// Healthcheck returns the current health status of the application.
|
||||||
// "ok", or "error" while report files cannot be written.
|
|
||||||
func (s *Healthcheck) Healthcheck() *HealthcheckResponse {
|
func (s *Healthcheck) Healthcheck() *HealthcheckResponse {
|
||||||
status := "ok"
|
|
||||||
if s.params.Buffer.LastWriteFailed() {
|
|
||||||
status = "error"
|
|
||||||
}
|
|
||||||
|
|
||||||
return &HealthcheckResponse{
|
return &HealthcheckResponse{
|
||||||
Appname: s.params.Globals.Appname,
|
Appname: s.params.Globals.Appname,
|
||||||
Now: time.Now().UTC().Format(time.RFC3339Nano),
|
Now: time.Now().UTC().Format(time.RFC3339Nano),
|
||||||
Status: status,
|
Status: "ok",
|
||||||
UptimeHuman: s.uptime().String(),
|
UptimeHuman: s.uptime().String(),
|
||||||
UptimeSeconds: int64(s.uptime().Seconds()),
|
UptimeSeconds: int64(s.uptime().Seconds()),
|
||||||
Version: s.params.Globals.Version,
|
Version: s.params.Globals.Version,
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ import (
|
|||||||
"log/slog"
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
@@ -72,9 +73,11 @@ type Buffer struct {
|
|||||||
// except in tests that hold the write open or make it fail.
|
// except in tests that hold the write open or make it fail.
|
||||||
fileCreated func(f *os.File)
|
fileCreated func(f *os.File)
|
||||||
// files are the report files that may be deleted to make room,
|
// files are the report files that may be deleted to make room,
|
||||||
// oldest first: those in dataDir at start, in name order, then
|
// in name order, which is oldest first: those in dataDir at
|
||||||
// each one this buffer writes, once it is complete. A file still
|
// start, and each one this buffer writes, put in at its place by
|
||||||
// being written is not among them. filesBytes is their total size.
|
// name once it is complete, even when an older file's write
|
||||||
|
// completes after a newer one's. A file still being written is
|
||||||
|
// not among them. filesBytes is their total size.
|
||||||
files []reportFile
|
files []reportFile
|
||||||
filesBytes int64
|
filesBytes int64
|
||||||
log *slog.Logger
|
log *slog.Logger
|
||||||
@@ -87,10 +90,6 @@ type Buffer struct {
|
|||||||
// millisecond still get different names.
|
// millisecond still get different names.
|
||||||
seq atomic.Int64
|
seq atomic.Int64
|
||||||
stopOnce sync.Once
|
stopOnce sync.Once
|
||||||
// thresholdWriteWaiting is set while a write Append started at
|
|
||||||
// flushSizeThreshold waits for writeMu; Append starts no other
|
|
||||||
// while it is set.
|
|
||||||
thresholdWriteWaiting atomic.Bool
|
|
||||||
// usedBytes is what Append checks against maxBytes: the size
|
// usedBytes is what Append checks against maxBytes: the size
|
||||||
// of the report files in dataDir, plus the reports waiting to
|
// of the report files in dataDir, plus the reports waiting to
|
||||||
// be written to one at their uncompressed size.
|
// be written to one at their uncompressed size.
|
||||||
@@ -100,11 +99,6 @@ type Buffer struct {
|
|||||||
// refuses reports, and the reports of that write wait in buf for
|
// refuses reports, and the reports of that write wait in buf for
|
||||||
// the next flush.
|
// the next flush.
|
||||||
writeErr error
|
writeErr error
|
||||||
// writeMu is held across each write of a report file, so writes
|
|
||||||
// never overlap: a failed write puts its reports back in buf before
|
|
||||||
// the next write takes them, which keeps them in the order they
|
|
||||||
// were appended.
|
|
||||||
writeMu sync.Mutex
|
|
||||||
// writers are the goroutines that write report files: flushLoop,
|
// writers are the goroutines that write report files: flushLoop,
|
||||||
// and each one Append starts at flushSizeThreshold. OnStop waits
|
// and each one Append starts at flushSizeThreshold. OnStop waits
|
||||||
// for them before the final flush.
|
// for them before the final flush.
|
||||||
@@ -213,9 +207,8 @@ func New(
|
|||||||
// write of a report file failed, until a write succeeds again. If the
|
// write of a report file failed, until a write succeeds again. If the
|
||||||
// line would take usedBytes past maxBytes, the oldest report files are
|
// line would take usedBytes past maxBytes, the oldest report files are
|
||||||
// deleted to make room; it stores nothing and returns ErrFull if that
|
// deleted to make room; it stores nothing and returns ErrFull if that
|
||||||
// cannot make room. If the buffer reaches the size threshold, it starts
|
// cannot make room. If the buffer reaches the size threshold, it is
|
||||||
// a write of the reports in the background (see flushAtThreshold),
|
// drained and written to disk asynchronously.
|
||||||
// unless one it started is still waiting.
|
|
||||||
func (b *Buffer) Append(v any) error {
|
func (b *Buffer) Append(v any) error {
|
||||||
line, err := json.Marshal(v)
|
line, err := json.Marshal(v)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -245,9 +238,18 @@ func (b *Buffer) Append(v any) error {
|
|||||||
b.buf.Write(line)
|
b.buf.Write(line)
|
||||||
b.buf.WriteByte('\n')
|
b.buf.WriteByte('\n')
|
||||||
|
|
||||||
if b.buf.Len() >= flushSizeThreshold &&
|
if b.buf.Len() >= flushSizeThreshold {
|
||||||
b.thresholdWriteWaiting.CompareAndSwap(false, true) {
|
data := b.drainBuf()
|
||||||
b.writers.Go(b.flushAtThreshold)
|
b.mu.Unlock()
|
||||||
|
|
||||||
|
b.writers.Go(func() {
|
||||||
|
writeErr := b.writeFile(data)
|
||||||
|
if writeErr != nil {
|
||||||
|
b.log.Error("flush reports failed", "error", writeErr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
b.mu.Unlock()
|
b.mu.Unlock()
|
||||||
@@ -255,16 +257,6 @@ func (b *Buffer) Append(v any) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// LastWriteFailed reports whether the last write of a report file to
|
|
||||||
// finish failed: false at start until a write fails, and false again
|
|
||||||
// once a write succeeds. While it is true, Append refuses reports.
|
|
||||||
func (b *Buffer) LastWriteFailed() bool {
|
|
||||||
b.mu.Lock()
|
|
||||||
defer b.mu.Unlock()
|
|
||||||
|
|
||||||
return b.writeErr != nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// deleteOldestFiles deletes report files, oldest first, until n more
|
// deleteOldestFiles deletes report files, oldest first, until n more
|
||||||
// bytes fit under maxBytes. It deletes none when the reports waiting
|
// bytes fit under maxBytes. It deletes none when the reports waiting
|
||||||
// to be written leave no room for n even with every file gone, since
|
// to be written leave no room for n even with every file gone, since
|
||||||
@@ -315,28 +307,21 @@ func (b *Buffer) flushLoop() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// flushLocked waits for any write of a report file under way, then
|
// flushLocked acquires the lock, drains the buffer, and
|
||||||
// writes the reports waiting in the buffer to a new one.
|
// writes the data to a compressed file.
|
||||||
func (b *Buffer) flushLocked() error {
|
func (b *Buffer) flushLocked() error {
|
||||||
b.writeMu.Lock()
|
b.mu.Lock()
|
||||||
defer b.writeMu.Unlock()
|
|
||||||
|
|
||||||
return b.writeFile()
|
if b.buf.Len() == 0 {
|
||||||
}
|
b.mu.Unlock()
|
||||||
|
|
||||||
// flushAtThreshold is the write Append starts when the buffer reaches
|
return nil
|
||||||
// flushSizeThreshold. Once any write under way is done, it lets Append
|
|
||||||
// start another and writes the reports waiting in the buffer.
|
|
||||||
func (b *Buffer) flushAtThreshold() {
|
|
||||||
b.writeMu.Lock()
|
|
||||||
defer b.writeMu.Unlock()
|
|
||||||
|
|
||||||
b.thresholdWriteWaiting.Store(false)
|
|
||||||
|
|
||||||
err := b.writeFile()
|
|
||||||
if err != nil {
|
|
||||||
b.log.Error("flush reports failed", "error", err)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
data := b.drainBuf()
|
||||||
|
b.mu.Unlock()
|
||||||
|
|
||||||
|
return b.writeFile(data)
|
||||||
}
|
}
|
||||||
|
|
||||||
// drainBuf copies the buffer contents and resets it.
|
// drainBuf copies the buffer contents and resets it.
|
||||||
@@ -349,22 +334,10 @@ func (b *Buffer) drainBuf() []byte {
|
|||||||
return data
|
return data
|
||||||
}
|
}
|
||||||
|
|
||||||
// writeFile takes the reports waiting in the buffer, if any, and writes
|
// writeFile writes data, reports drained from the buffer, to a new
|
||||||
// them to a new timestamped zstd-compressed JSONL file in the data
|
// timestamped zstd-compressed JSONL file in the data directory. If the
|
||||||
// directory. If the write fails, it puts the reports back in the
|
// write fails, it puts the reports back in the buffer.
|
||||||
// buffer. The caller must hold b.writeMu.
|
func (b *Buffer) writeFile(data []byte) error {
|
||||||
func (b *Buffer) writeFile() error {
|
|
||||||
b.mu.Lock()
|
|
||||||
|
|
||||||
if b.buf.Len() == 0 {
|
|
||||||
b.mu.Unlock()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
data := b.drainBuf()
|
|
||||||
b.mu.Unlock()
|
|
||||||
|
|
||||||
// The timestamp comes first, so the names sort by time; the number
|
// The timestamp comes first, so the names sort by time; the number
|
||||||
// after it tells apart files named in the same millisecond.
|
// after it tells apart files named in the same millisecond.
|
||||||
ts := b.now().UTC().Format("2006-01-02T15-04-05.000Z")
|
ts := b.now().UTC().Format("2006-01-02T15-04-05.000Z")
|
||||||
@@ -395,7 +368,14 @@ func (b *Buffer) writeFile() error {
|
|||||||
b.usedBytes -= int64(len(data))
|
b.usedBytes -= int64(len(data))
|
||||||
b.usedBytes += size
|
b.usedBytes += size
|
||||||
b.filesBytes += size
|
b.filesBytes += size
|
||||||
b.files = append(b.files, reportFile{name: name, size: size})
|
|
||||||
|
// At its place by name, not at the end: another write, of a newer
|
||||||
|
// file, may have completed while this one was being written.
|
||||||
|
i, _ := slices.BinarySearchFunc(b.files, name,
|
||||||
|
func(f reportFile, target string) int {
|
||||||
|
return strings.Compare(f.name, target)
|
||||||
|
})
|
||||||
|
b.files = slices.Insert(b.files, i, reportFile{name: name, size: size})
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -334,9 +334,9 @@ func stopWhileHeld(
|
|||||||
return <-stopped
|
return <-stopped
|
||||||
}
|
}
|
||||||
|
|
||||||
// startApp starts a Buffer through fx, as cli.RunServer does, with the
|
// startApp starts a Buffer through fx, as main does, with the DATA_DIR
|
||||||
// DATA_DIR and DATA_DIR_MAX_BYTES the calling test has set, and returns
|
// and DATA_DIR_MAX_BYTES the calling test has set, and returns it with
|
||||||
// it with the app, for tests that stop the app themselves.
|
// the app, for tests that stop the app themselves.
|
||||||
func startApp(t *testing.T) (*reportbuf.Buffer, *fxtest.App) {
|
func startApp(t *testing.T) (*reportbuf.Buffer, *fxtest.App) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -358,8 +358,8 @@ func startApp(t *testing.T) (*reportbuf.Buffer, *fxtest.App) {
|
|||||||
return buf, app
|
return buf, app
|
||||||
}
|
}
|
||||||
|
|
||||||
// startBuffer starts a Buffer through fx, as cli.RunServer does, with
|
// startBuffer starts a Buffer through fx, as main does, with the
|
||||||
// the DATA_DIR and DATA_DIR_MAX_BYTES the calling test has set.
|
// DATA_DIR and DATA_DIR_MAX_BYTES the calling test has set.
|
||||||
func startBuffer(t *testing.T) *reportbuf.Buffer {
|
func startBuffer(t *testing.T) *reportbuf.Buffer {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -581,62 +581,6 @@ func TestWrittenFilesDeletedToMakeRoom(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestWrittenFilesDeletedOldestFirst writes three report files, then
|
|
||||||
// appends a report that fits only once one of them is deleted. The
|
|
||||||
// first one written must be the one deleted.
|
|
||||||
func TestWrittenFilesDeletedOldestFirst(t *testing.T) {
|
|
||||||
const maxBytes = 1000
|
|
||||||
|
|
||||||
report := map[string]string{"id": "written"}
|
|
||||||
|
|
||||||
t.Setenv("DATA_DIR", t.TempDir())
|
|
||||||
t.Setenv("DATA_DIR_MAX_BYTES", strconv.Itoa(maxBytes))
|
|
||||||
|
|
||||||
buf := startBuffer(t)
|
|
||||||
|
|
||||||
var written []string
|
|
||||||
|
|
||||||
buf.OnFileCreated(func(f *os.File) { written = append(written, f.Name()) })
|
|
||||||
|
|
||||||
for range 3 {
|
|
||||||
err := buf.Append(report)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("append report: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = buf.Flush()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("flush: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// The three files hold the same report, so they are the same size.
|
|
||||||
info, err := os.Stat(written[0])
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("stat %s: %v", written[0], err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A report that fits beside two of the files, so deleting one makes
|
|
||||||
// exactly the room it needs.
|
|
||||||
pad := maxBytes - 2*int(info.Size()) -
|
|
||||||
lineBytes(t, map[string]string{"id": ""})
|
|
||||||
|
|
||||||
err = buf.Append(map[string]string{"id": strings.Repeat("a", pad)})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("report that needs room: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if exists(t, written[0]) {
|
|
||||||
t.Fatal("first report file written kept when room was needed")
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, path := range written[1:] {
|
|
||||||
if !exists(t, path) {
|
|
||||||
t.Fatalf("%s deleted before the first report file written", path)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFailedDeletionStillCounts makes deleting the oldest report file
|
// TestFailedDeletionStillCounts makes deleting the oldest report file
|
||||||
// fail. It is still there, so it still takes room, and the next oldest
|
// fail. It is still there, so it still takes room, and the next oldest
|
||||||
// is deleted in its place.
|
// is deleted in its place.
|
||||||
@@ -722,16 +666,17 @@ func TestFileDeletedByHandFreesRoom(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestFileBeingWrittenIsNeverDeleted holds the write of a report file
|
// TestFileBeingWrittenIsNeverDeleted holds the write of one report file
|
||||||
// open under a cap with room for two reports, then appends a second
|
// open while a second write completes, then sends a report that needs
|
||||||
// report and a third that needs room. The file being written is the
|
// room. Deleting either file would make it, and the one being written is
|
||||||
// only report file, and it is not deleted to make room, so the third
|
// the older, but only the complete one may be deleted. Once the first
|
||||||
// report is refused. Once the write is complete, its file is deleted
|
// write is complete, its file is deleted when room is needed.
|
||||||
// when room is needed.
|
|
||||||
func TestFileBeingWrittenIsNeverDeleted(t *testing.T) {
|
func TestFileBeingWrittenIsNeverDeleted(t *testing.T) {
|
||||||
report := map[string]string{"id": "writing"}
|
report := map[string]string{"id": "writing"}
|
||||||
|
|
||||||
t.Setenv("DATA_DIR", t.TempDir())
|
t.Setenv("DATA_DIR", t.TempDir())
|
||||||
|
// Room for two reports waiting to be written, but not for two
|
||||||
|
// beside a report file.
|
||||||
t.Setenv("DATA_DIR_MAX_BYTES", strconv.Itoa(2*lineBytes(t, report)))
|
t.Setenv("DATA_DIR_MAX_BYTES", strconv.Itoa(2*lineBytes(t, report)))
|
||||||
|
|
||||||
buf := startBuffer(t)
|
buf := startBuffer(t)
|
||||||
@@ -756,25 +701,36 @@ func TestFileBeingWrittenIsNeverDeleted(t *testing.T) {
|
|||||||
|
|
||||||
writing := <-created
|
writing := <-created
|
||||||
|
|
||||||
// Errorf, not Fatalf, until the write is released, so that a
|
// Only the first write is held; the second goes through, and
|
||||||
|
// so do the writes after it, the final one at stop included.
|
||||||
|
var complete string
|
||||||
|
|
||||||
|
buf.OnFileCreated(func(f *os.File) { complete = f.Name() })
|
||||||
|
|
||||||
|
// Errorf, not Fatalf, until the first write is released, so that a
|
||||||
// failure here does not leave it held.
|
// failure here does not leave it held.
|
||||||
err = buf.Append(report)
|
err = buf.Append(report)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Errorf("second report: %v", err)
|
t.Errorf("second report: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
err = buf.Flush()
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("flush of the second report: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
err = buf.Append(report)
|
err = buf.Append(report)
|
||||||
if !errors.Is(err, reportbuf.ErrFull) {
|
if err != nil {
|
||||||
t.Errorf("report that needs room: error = %v, want ErrFull", err)
|
t.Errorf("report that needs room: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if !exists(t, writing) {
|
if !exists(t, writing) {
|
||||||
t.Error("report file deleted while it was being written")
|
t.Error("report file deleted while it was being written")
|
||||||
}
|
}
|
||||||
|
|
||||||
// Only this write is held; the writes after it, the final one at
|
if exists(t, complete) {
|
||||||
// stop included, go through.
|
t.Error("complete report file kept, though room was needed")
|
||||||
buf.OnFileCreated(func(*os.File) {})
|
}
|
||||||
|
|
||||||
close(release)
|
close(release)
|
||||||
|
|
||||||
@@ -783,11 +739,9 @@ func TestFileBeingWrittenIsNeverDeleted(t *testing.T) {
|
|||||||
t.Fatalf("flush of the first report: %v", err)
|
t.Fatalf("flush of the first report: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The second report still waits to be written, so this one fits
|
|
||||||
// only once the file just written is deleted.
|
|
||||||
err = buf.Append(report)
|
err = buf.Append(report)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("report after the write was complete: %v", err)
|
t.Fatalf("report after the first write was complete: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if exists(t, writing) {
|
if exists(t, writing) {
|
||||||
@@ -929,82 +883,6 @@ func TestFailedWriteKeepsReportsAppendedDuringIt(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestFailedWritesKeepReportsInOrder holds a write open, then appends a
|
|
||||||
// report that takes the buffer to FlushSizeThreshold, so that Append
|
|
||||||
// starts a second write. The held write fails first, then the second,
|
|
||||||
// which waited for it and took both reports. The next flush writes them
|
|
||||||
// in the order they were appended, the held write's first.
|
|
||||||
func TestFailedWritesKeepReportsInOrder(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
t.Setenv("DATA_DIR", dir)
|
|
||||||
|
|
||||||
buf := startBuffer(t)
|
|
||||||
|
|
||||||
created := make(chan struct{})
|
|
||||||
release := make(chan struct{})
|
|
||||||
|
|
||||||
// Each write is held until release is closed, then fails, as its
|
|
||||||
// file is closed under it.
|
|
||||||
buf.OnFileCreated(func(f *os.File) {
|
|
||||||
created <- struct{}{}
|
|
||||||
|
|
||||||
<-release
|
|
||||||
|
|
||||||
_ = f.Close()
|
|
||||||
})
|
|
||||||
|
|
||||||
err := buf.Append(map[string]string{"id": "held"})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("report of the held write: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
flushed := make(chan error)
|
|
||||||
|
|
||||||
go func() { flushed <- buf.Flush() }()
|
|
||||||
|
|
||||||
<-created
|
|
||||||
|
|
||||||
big := strings.Repeat("a", reportbuf.FlushSizeThreshold)
|
|
||||||
|
|
||||||
// Errorf, not Fatalf, until the held write is released, so that a
|
|
||||||
// failure here does not leave it held.
|
|
||||||
err = buf.Append(map[string]string{"id": big})
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("report that starts the second write: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
close(release)
|
|
||||||
|
|
||||||
err = <-flushed
|
|
||||||
if err == nil {
|
|
||||||
t.Error("held write succeeded, though its file was closed under it")
|
|
||||||
}
|
|
||||||
|
|
||||||
// The second write has created its file, so it has taken the
|
|
||||||
// reports.
|
|
||||||
<-created
|
|
||||||
|
|
||||||
// Writes from here on, the final one at stop included, succeed.
|
|
||||||
buf.OnFileCreated(func(*os.File) {})
|
|
||||||
|
|
||||||
// Flush waits for the second write to fail before it writes.
|
|
||||||
err = buf.Flush()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("flush after the failed writes: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
files, err := readReportFiles(dir)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("read report files: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := `{"id":"held"}` + "\n" + `{"id":"` + big + `"}` + "\n"
|
|
||||||
if len(files) != 1 || files[0] != want {
|
|
||||||
t.Fatalf("%d report files, want one holding the held write's "+
|
|
||||||
"report, then the second one", len(files))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestReportsRefusedWhileWritesFail posts reports through the report
|
// TestReportsRefusedWhileWritesFail posts reports through the report
|
||||||
// handler, as the page does. Once a write has failed, a report is
|
// handler, as the page does. Once a write has failed, a report is
|
||||||
// refused with 500 until a write succeeds again; the reports taken
|
// refused with 500 until a write succeeds again; the reports taken
|
||||||
@@ -1074,54 +952,9 @@ func TestReportsRefusedWhileWritesFail(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestHealthCheckFailsWhileWritesFail asks the health check handler
|
|
||||||
// for its answer around a failed write. It answers 200 with
|
|
||||||
// "status":"ok" at start, 503 with "status":"error" from the failed
|
|
||||||
// write until a write succeeds, and 200 with "status":"ok" again after.
|
|
||||||
func TestHealthCheckFailsWhileWritesFail(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
t.Setenv("DATA_DIR", dir)
|
|
||||||
|
|
||||||
buf, h := startBufferWithHandlers(t)
|
|
||||||
|
|
||||||
checkHealthCheck(t, h, "at start", http.StatusOK, "ok")
|
|
||||||
|
|
||||||
// A flush writes a file only when a report is waiting.
|
|
||||||
err := buf.Append(map[string]string{"id": "kept"})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("append: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// With the data directory gone, the write has nowhere to go.
|
|
||||||
err = os.Remove(dir)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("remove data dir: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = buf.Flush()
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("flush succeeded with the data directory gone")
|
|
||||||
}
|
|
||||||
|
|
||||||
checkHealthCheck(t, h, "after the failed write",
|
|
||||||
http.StatusServiceUnavailable, "error")
|
|
||||||
|
|
||||||
err = os.Mkdir(dir, 0o750)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("make data dir: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = buf.Flush()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("flush with the data directory back: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkHealthCheck(t, h, "after a write succeeded", http.StatusOK, "ok")
|
|
||||||
}
|
|
||||||
|
|
||||||
// startBufferWithHandlers starts a Buffer and the Handlers that store
|
// startBufferWithHandlers starts a Buffer and the Handlers that store
|
||||||
// reports in it through fx, as cli.RunServer does, with the DATA_DIR
|
// reports in it through fx, as main does, with the DATA_DIR the calling
|
||||||
// the calling test has set.
|
// test has set.
|
||||||
func startBufferWithHandlers(
|
func startBufferWithHandlers(
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
) (*reportbuf.Buffer, *handlers.Handlers) {
|
) (*reportbuf.Buffer, *handlers.Handlers) {
|
||||||
@@ -1166,39 +999,6 @@ func postReport(t *testing.T, h *handlers.Handlers, clientID string) int {
|
|||||||
return rec.Code
|
return rec.Code
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkHealthCheck asks the health check handler for its answer, and
|
|
||||||
// fails the test, saying when it asked, unless the answer has status
|
|
||||||
// code wantCode and a "status" field of wantStatus.
|
|
||||||
func checkHealthCheck(
|
|
||||||
t *testing.T,
|
|
||||||
h *handlers.Handlers,
|
|
||||||
when string,
|
|
||||||
wantCode int,
|
|
||||||
wantStatus string,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
req := httptest.NewRequestWithContext(t.Context(),
|
|
||||||
http.MethodGet, "/.well-known/healthcheck", http.NoBody)
|
|
||||||
h.HandleHealthCheck().ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
var body struct {
|
|
||||||
Status string `json:"status"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := json.Unmarshal(rec.Body.Bytes(), &body)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("health check %s: body not JSON: %v (%q)",
|
|
||||||
when, err, rec.Body.String())
|
|
||||||
}
|
|
||||||
|
|
||||||
if rec.Code != wantCode || body.Status != wantStatus {
|
|
||||||
t.Fatalf("health check %s: %d with status %q, want %d with %q",
|
|
||||||
when, rec.Code, body.Status, wantCode, wantStatus)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestConcurrentAppendsStopAtCap appends from many goroutines at once
|
// TestConcurrentAppendsStopAtCap appends from many goroutines at once
|
||||||
// with room for exactly roomFor reports: exactly that many must be
|
// with room for exactly roomFor reports: exactly that many must be
|
||||||
// taken, which holds only if Append checks and counts each report
|
// taken, which holds only if Append checks and counts each report
|
||||||
@@ -1350,106 +1150,6 @@ func TestFlushAtSizeThreshold(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestEachSizeThresholdStartsAWrite reaches FlushSizeThreshold twice,
|
|
||||||
// with no call to Flush and the periodic flush a minute away. Append
|
|
||||||
// must start a write each time, so that the reports end up in two
|
|
||||||
// report files. The test runs in a synctest bubble, so that
|
|
||||||
// synctest.Wait returns once each write is done.
|
|
||||||
func TestEachSizeThresholdStartsAWrite(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
t.Setenv("DATA_DIR", dir)
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
buf := startBuffer(t)
|
|
||||||
|
|
||||||
var appended string
|
|
||||||
|
|
||||||
for range 2 {
|
|
||||||
appended = appendToThreshold(t, buf)
|
|
||||||
|
|
||||||
synctest.Wait()
|
|
||||||
}
|
|
||||||
|
|
||||||
files, err := readReportFiles(dir)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("read report files: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(files, []string{appended, appended}) {
|
|
||||||
t.Fatalf("%d report files, want two, each holding the %d "+
|
|
||||||
"bytes appended to reach the threshold",
|
|
||||||
len(files), len(appended))
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestAppendStartsOneWaitingWrite holds a write open, then appends two
|
|
||||||
// reports of FlushSizeThreshold bytes each. Append starts a write at
|
|
||||||
// the first, which waits for the held one, and none at the second,
|
|
||||||
// since that write is still waiting. Every write fails, so a write
|
|
||||||
// started at the second report would find the reports back in the
|
|
||||||
// buffer and try them again: once the held write is released, two
|
|
||||||
// writes must have been started, the held one and one more.
|
|
||||||
func TestAppendStartsOneWaitingWrite(t *testing.T) {
|
|
||||||
t.Setenv("DATA_DIR", t.TempDir())
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
buf := startBuffer(t)
|
|
||||||
|
|
||||||
var writes atomic.Int64
|
|
||||||
|
|
||||||
release := make(chan struct{})
|
|
||||||
|
|
||||||
// Each write is held until release is closed, then fails, as its
|
|
||||||
// file is closed under it.
|
|
||||||
buf.OnFileCreated(func(f *os.File) {
|
|
||||||
writes.Add(1)
|
|
||||||
|
|
||||||
<-release
|
|
||||||
|
|
||||||
_ = f.Close()
|
|
||||||
})
|
|
||||||
|
|
||||||
err := buf.Append(map[string]string{"id": "one-waiting"})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("report of the held write: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
go func() { _ = buf.Flush() }()
|
|
||||||
|
|
||||||
// Wait returns once the flush's write is held.
|
|
||||||
synctest.Wait()
|
|
||||||
|
|
||||||
big := map[string]string{
|
|
||||||
"id": strings.Repeat("a", reportbuf.FlushSizeThreshold),
|
|
||||||
}
|
|
||||||
|
|
||||||
// Errorf, not Fatalf, until the held write is released, so that a
|
|
||||||
// failure here does not leave it held.
|
|
||||||
for range 2 {
|
|
||||||
err = buf.Append(big)
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("report of FlushSizeThreshold bytes: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
close(release)
|
|
||||||
|
|
||||||
// Wait returns once every write started is done.
|
|
||||||
synctest.Wait()
|
|
||||||
|
|
||||||
started := writes.Load()
|
|
||||||
|
|
||||||
// Writes from here on, the final one at stop included, succeed.
|
|
||||||
buf.OnFileCreated(func(*os.File) {})
|
|
||||||
|
|
||||||
if started != 2 {
|
|
||||||
t.Fatalf("%d writes started, want 2: the held one and one "+
|
|
||||||
"at the threshold", started)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// appendToThreshold appends reports to buf until it holds
|
// appendToThreshold appends reports to buf until it holds
|
||||||
// FlushSizeThreshold bytes, so that the last append starts writing them
|
// FlushSizeThreshold bytes, so that the last append starts writing them
|
||||||
// to a report file, and returns the lines appended.
|
// to a report file, and returns the lines appended.
|
||||||
@@ -1595,3 +1295,85 @@ func hasReportFile(t *testing.T, dir string) bool {
|
|||||||
|
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestFilesDeletedOldestFirstWhenWritesOverlap holds the write of an
|
||||||
|
// older report file open until a newer one's write completes, then
|
||||||
|
// releases it. When room is needed, the older file is deleted first,
|
||||||
|
// though its write was the last to complete.
|
||||||
|
func TestFilesDeletedOldestFirstWhenWritesOverlap(t *testing.T) {
|
||||||
|
const maxBytes = 1000
|
||||||
|
|
||||||
|
report := map[string]string{"id": "overlap"}
|
||||||
|
|
||||||
|
t.Setenv("DATA_DIR", t.TempDir())
|
||||||
|
t.Setenv("DATA_DIR_MAX_BYTES", strconv.Itoa(maxBytes))
|
||||||
|
|
||||||
|
buf := startBuffer(t)
|
||||||
|
|
||||||
|
created := make(chan string)
|
||||||
|
release := make(chan struct{})
|
||||||
|
|
||||||
|
buf.OnFileCreated(func(f *os.File) {
|
||||||
|
created <- f.Name()
|
||||||
|
|
||||||
|
<-release
|
||||||
|
})
|
||||||
|
|
||||||
|
err := buf.Append(report)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("older report: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
flushed := make(chan error)
|
||||||
|
|
||||||
|
go func() { flushed <- buf.Flush() }()
|
||||||
|
|
||||||
|
older := <-created
|
||||||
|
|
||||||
|
// Only the older write is held; the newer one goes through, and
|
||||||
|
// so do the writes after it, the final one at stop included.
|
||||||
|
var newer string
|
||||||
|
|
||||||
|
buf.OnFileCreated(func(f *os.File) { newer = f.Name() })
|
||||||
|
|
||||||
|
// Errorf, not Fatalf, until the older write is released, so that a
|
||||||
|
// failure here does not leave it held.
|
||||||
|
err = buf.Append(report)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("newer report: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = buf.Flush()
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("flush of the newer report: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
close(release)
|
||||||
|
|
||||||
|
err = <-flushed
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("flush of the older report: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
info, err := os.Stat(newer)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("stat %s: %v", newer, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A report that fits beside the newer file alone, so deleting the
|
||||||
|
// older one makes exactly the room it needs.
|
||||||
|
pad := maxBytes - int(info.Size()) - lineBytes(t, map[string]string{"id": ""})
|
||||||
|
|
||||||
|
err = buf.Append(map[string]string{"id": strings.Repeat("a", pad)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("report that needs room: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if exists(t, older) {
|
||||||
|
t.Fatal("older report file kept when room was needed")
|
||||||
|
}
|
||||||
|
|
||||||
|
if !exists(t, newer) {
|
||||||
|
t.Fatal("newer report file deleted before the older one")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -22,9 +22,9 @@ import (
|
|||||||
"go.uber.org/fx/fxtest"
|
"go.uber.org/fx/fxtest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// newServer builds a Server from the same constructors as
|
// newServer builds a Server from the same constructors as main,
|
||||||
// cli.RunServer, configured from the environment. It is never started,
|
// configured from the environment. It is never started, so nothing
|
||||||
// so nothing listens.
|
// listens.
|
||||||
func newServer(t *testing.T) *server.Server {
|
func newServer(t *testing.T) *server.Server {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
|
|||||||
@@ -79,7 +79,7 @@ func New(
|
|||||||
s.SetupRoutes()
|
s.SetupRoutes()
|
||||||
s.httpServer = s.newHTTPServer()
|
s.httpServer = s.newHTTPServer()
|
||||||
|
|
||||||
go func() { s.listenAndServe() }()
|
go s.listenAndServe()
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
},
|
},
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "netwatch",
|
"name": "netwatch",
|
||||||
"private": true,
|
"version": "1.0.0",
|
||||||
"description": "Real-time network latency monitor SPA",
|
"description": "Real-time network latency monitor SPA",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
|
|||||||
@@ -23,8 +23,8 @@ main() {
|
|||||||
"" | -* | *[[:space:]]*) usage ;;
|
"" | -* | *[[:space:]]*) usage ;;
|
||||||
esac
|
esac
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
"$ROOT/script/yarn" add --dev "$1"
|
yarn add --dev "$1"
|
||||||
"$ROOT/script/yarn" install --frozen-lockfile
|
yarn install --frozen-lockfile
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+14
-55
@@ -3,14 +3,11 @@
|
|||||||
# this repo. Idempotent: every install is guarded by a check so already
|
# this repo. Idempotent: every install is guarded by a check so already
|
||||||
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
||||||
# or apk (detected in that order); assumes nothing is present. Node is
|
# or apk (detected in that order); assumes nothing is present. Node is
|
||||||
# used directly only if it has the major version of NODE_VERSION and is
|
# used directly if it is at least NODE_MIN_VERSION; otherwise it is
|
||||||
# at least NODE_MIN_VERSION; otherwise NODE_VERSION is installed via nvm
|
# installed at a pinned version via nvm (installing nvm itself first,
|
||||||
# (installing nvm itself first, from a hash-verified release archive,
|
# from a hash-verified release archive, never curl | sh). Go, with its
|
||||||
# never curl | sh). Yarn is used directly only if it is exactly
|
# gofmt, is used directly only if it is exactly GO_VERSION; otherwise
|
||||||
# YARN_VERSION; otherwise that version is prepared through corepack,
|
# that release is installed from its hash-verified archive.
|
||||||
# or installed with npm where there is no corepack.
|
|
||||||
# Go, with its gofmt, is used directly only if it is exactly GO_VERSION;
|
|
||||||
# otherwise that release is installed from its hash-verified archive.
|
|
||||||
#
|
#
|
||||||
# What this script installs outside the system package manager lives
|
# What this script installs outside the system package manager lives
|
||||||
# under $HOME and is linked into ~/.local/bin, where make and the git
|
# under $HOME and is linked into ~/.local/bin, where make and the git
|
||||||
@@ -145,63 +142,32 @@ ensure_nvm() {
|
|||||||
rm -rf "$tmp"
|
rm -rf "$tmp"
|
||||||
}
|
}
|
||||||
|
|
||||||
# node_ok: the node on PATH has the major version of NODE_VERSION and is
|
# node_ok: the node on PATH is at least NODE_MIN_VERSION. node itself
|
||||||
# at least NODE_MIN_VERSION. Only the major is compared with the pin,
|
# compares the two: major, then minor, then patch.
|
||||||
# as REPO_POLICIES.md asks. node itself compares the versions: major,
|
|
||||||
# then minor, then patch. A node that fails does not pass.
|
|
||||||
# script/bootstrap and script/yarn carry this function unchanged.
|
|
||||||
node_ok() {
|
node_ok() {
|
||||||
if ! command -v node >/dev/null 2>&1; then return 1; fi
|
if missing node; then return 1; fi
|
||||||
node -e '
|
node -e '
|
||||||
const have = process.versions.node.split(".").map(Number);
|
const have = process.versions.node.split(".").map(Number);
|
||||||
const want = process.argv[1].split(".").map(Number);
|
const want = process.argv[1].split(".").map(Number);
|
||||||
const pinned = process.argv[2].split(".").map(Number);
|
|
||||||
if (have[0] !== pinned[0]) process.exit(1);
|
|
||||||
for (let i = 0; i < 3; i++) {
|
for (let i = 0; i < 3; i++) {
|
||||||
if (have[i] !== want[i]) process.exit(have[i] > want[i] ? 0 : 1);
|
if (have[i] !== want[i]) process.exit(have[i] > want[i] ? 0 : 1);
|
||||||
}
|
}
|
||||||
' "$NODE_MIN_VERSION" "$NODE_VERSION"
|
' "$NODE_MIN_VERSION"
|
||||||
}
|
}
|
||||||
|
|
||||||
# ensure_node: unless node_ok, install NODE_VERSION and link its node.
|
# ensure_node: unless node_ok, install NODE_VERSION and link its node.
|
||||||
# Afterwards node is looked up through PATH again and must pass node_ok.
|
|
||||||
ensure_node() {
|
ensure_node() {
|
||||||
if node_ok; then
|
if node_ok; then return 0; fi
|
||||||
echo "bootstrap: using node $(node --version)"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
ensure_nvm
|
ensure_nvm
|
||||||
nvm_sh "nvm install $NODE_VERSION"
|
nvm_sh "nvm install $NODE_VERSION"
|
||||||
link_bin "$HOME/.nvm/versions/node/v$NODE_VERSION/bin/node" node
|
link_bin "$HOME/.nvm/versions/node/v$NODE_VERSION/bin/node" node
|
||||||
hash -r
|
|
||||||
if ! node_ok; then
|
|
||||||
echo "bootstrap: after installing node $NODE_VERSION through nvm," >&2
|
|
||||||
echo " the node on PATH, $(command -v node), is not it" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
echo "bootstrap: installed node $(node --version)"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# yarn_ok: yarn --version on PATH prints exactly YARN_VERSION, compared
|
# ensure_yarn: corepack writes its shims (pnpm and yarnpkg as well as
|
||||||
# over the whole version. A yarn that fails or prints anything else does
|
# yarn) into $TOOLCHAIN rather than next to itself, and the npm fallback
|
||||||
# not pass.
|
# installs there too; only yarn is linked.
|
||||||
yarn_ok() {
|
|
||||||
if missing yarn; then return 1; fi
|
|
||||||
version="$(yarn --version 2>/dev/null)" || return 1
|
|
||||||
[ "$version" = "$YARN_VERSION" ]
|
|
||||||
}
|
|
||||||
|
|
||||||
# ensure_yarn: unless yarn_ok, prepare YARN_VERSION and link its yarn.
|
|
||||||
# corepack writes its shims (pnpm and yarnpkg as well as yarn) into
|
|
||||||
# $TOOLCHAIN rather than next to itself, and the npm fallback installs
|
|
||||||
# there too; only yarn is linked. Afterwards yarn is looked up through
|
|
||||||
# PATH again and must pass yarn_ok, so another yarn that hides the link
|
|
||||||
# stops bootstrap.
|
|
||||||
ensure_yarn() {
|
ensure_yarn() {
|
||||||
if yarn_ok; then
|
if ! missing yarn; then return 0; fi
|
||||||
echo "bootstrap: using yarn $(yarn --version)"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
shims="$TOOLCHAIN/corepack-shims"
|
shims="$TOOLCHAIN/corepack-shims"
|
||||||
mkdir -p "$shims"
|
mkdir -p "$shims"
|
||||||
if ! missing corepack; then
|
if ! missing corepack; then
|
||||||
@@ -216,13 +182,6 @@ ensure_yarn() {
|
|||||||
shims="$TOOLCHAIN/npm-global/bin"
|
shims="$TOOLCHAIN/npm-global/bin"
|
||||||
fi
|
fi
|
||||||
link_bin "$shims/yarn" yarn
|
link_bin "$shims/yarn" yarn
|
||||||
hash -r
|
|
||||||
if ! yarn_ok; then
|
|
||||||
echo "bootstrap: after installing yarn $YARN_VERSION in $shims," >&2
|
|
||||||
echo " the yarn on PATH, $(command -v yarn), does not report it" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
echo "bootstrap: installed yarn $(yarn --version)"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# go_ok: the go on PATH has its gofmt beside it (a Go release ships the
|
# go_ok: the go on PATH has its gofmt beside it (a Go release ships the
|
||||||
|
|||||||
+4
-5
@@ -8,12 +8,11 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
"$ROOT/script/yarn" build
|
yarn build
|
||||||
|
|
||||||
# VERSION comes from the environment. Unset or empty, it is
|
# VERSION comes from the environment. Unset or empty, it is git
|
||||||
# `git describe --tags --always --dirty`, as in script/docker, or
|
# describe, or "dev" where there is no git or no repository history.
|
||||||
# "dev" where there is no git or no repository history.
|
version="${VERSION:-$(git describe --always --dirty 2>/dev/null || echo dev)}"
|
||||||
version="${VERSION:-$(git describe --tags --always --dirty 2>/dev/null || echo dev)}"
|
|
||||||
|
|
||||||
CGO_ENABLED=0 go build -trimpath \
|
CGO_ENABLED=0 go build -trimpath \
|
||||||
-ldflags "-s -w -X main.Version=$version" \
|
-ldflags "-s -w -X main.Version=$version" \
|
||||||
|
|||||||
+9
-14
@@ -2,12 +2,9 @@
|
|||||||
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
||||||
# checks out and runs this and nothing else, and script/fmt-check runs
|
# checks out and runs this and nothing else, and script/fmt-check runs
|
||||||
# the formatter on the host, which a pristine checkout cannot do.
|
# the formatter on the host, which a pristine checkout cannot do.
|
||||||
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST
|
# --no-cache for the same reason as script/docker: the gate phases the
|
||||||
# line, since a cached gate phase is a check that did not run; unlike
|
# final stage depends on are RUN steps, and a cached one is a check that
|
||||||
# --no-cache, it keeps the layers above that line and the cache mounts.
|
# did not run.
|
||||||
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
|
|
||||||
# before every step that lints, tests or compiles; a stage without that
|
|
||||||
# line is served from cache on an unchanged tree, with no warning.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -17,17 +14,15 @@ main() {
|
|||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
"$SCRIPT_DIR/bootstrap"
|
"$SCRIPT_DIR/bootstrap"
|
||||||
"$SCRIPT_DIR/check"
|
"$SCRIPT_DIR/check"
|
||||||
# The version and the tag each get their own line: a failing
|
# Own line: a failing command substitution inside an argument does
|
||||||
# command substitution inside an argument does not trip `set -e`,
|
# not trip `set -e`, so the inline form degrades silently to an
|
||||||
# so the inline form degrades silently to an empty constant. The
|
# empty constant. The VERSION build argument takes precedence over
|
||||||
# VERSION build argument takes precedence over the version a build
|
# the version a build stage derives from the .git in the context.
|
||||||
# stage derives from the .git in the context.
|
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
tag="$("$SCRIPT_DIR/projectname")"
|
docker build --no-cache \
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$tag" .
|
-t "$("$SCRIPT_DIR/projectname")" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+1
-1
@@ -7,7 +7,7 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
"$ROOT/script/yarn" dev
|
yarn dev
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+8
-14
@@ -1,12 +1,8 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/docker: build the Docker image tagged with the project name.
|
# script/docker: build the Docker image tagged with the project name.
|
||||||
# Identical in all repos; the tag comes from script/projectname.
|
# Identical in all repos; the tag comes from script/projectname.
|
||||||
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST
|
# --no-cache because the gate phases the final stage depends on are RUN
|
||||||
# line, since a cached gate phase is a check that did not run; unlike
|
# steps, and a cached one is a check that did not run.
|
||||||
# --no-cache, it keeps the layers above that line and the cache mounts.
|
|
||||||
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
|
|
||||||
# before every step that lints, tests or compiles; a stage without that
|
|
||||||
# line is served from cache on an unchanged tree, with no warning.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -14,17 +10,15 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
# The version and the tag each get their own line: a failing
|
# Own line: a failing command substitution inside an argument does
|
||||||
# command substitution inside an argument does not trip `set -e`,
|
# not trip `set -e`, so the inline form degrades silently to an
|
||||||
# so the inline form degrades silently to an empty constant. The
|
# empty constant. The VERSION build argument takes precedence over
|
||||||
# VERSION build argument takes precedence over the version a build
|
# the version a build stage derives from the .git in the context.
|
||||||
# stage derives from the .git in the context.
|
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
tag="$("$SCRIPT_DIR/projectname")"
|
docker build --no-cache \
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$tag" .
|
-t "$("$SCRIPT_DIR/projectname")" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+1
-1
@@ -10,7 +10,7 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
"$ROOT/script/yarn" prettier --write '**/*.{js,css,html,md}'
|
yarn prettier --write '**/*.{js,css,html,md}'
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
"$ROOT/script/yarn" prettier --check '**/*.{js,css,html,md}'
|
yarn prettier --check '**/*.{js,css,html,md}'
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
@@ -14,14 +14,12 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
NODE_OPTIONS=--test-reporter=dot \
|
NODE_OPTIONS=--test-reporter=dot timeout 90 yarn --silent run test || {
|
||||||
timeout 90 "$ROOT/script/yarn" --silent run test || {
|
|
||||||
echo "--- Rerunning with every test listed for details ---"
|
echo "--- Rerunning with every test listed for details ---"
|
||||||
NODE_OPTIONS=--test-reporter=spec \
|
NODE_OPTIONS=--test-reporter=spec timeout 90 yarn --silent run test
|
||||||
timeout 90 "$ROOT/script/yarn" --silent run test
|
|
||||||
exit 1
|
exit 1
|
||||||
}
|
}
|
||||||
timeout 30 "$ROOT/script/yarn" build
|
timeout 30 yarn build
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+5
-10
@@ -5,14 +5,9 @@
|
|||||||
# make its answer untrustworthy.
|
# make its answer untrustworthy.
|
||||||
#
|
#
|
||||||
# The phase is not the last stage in the file, so it is built only when
|
# The phase is not the last stage in the file, so it is built only when
|
||||||
# --target names it. A new CACHEBUST value reruns every RUN step after
|
# --target names it. --no-cache because a cached lint layer is a lint
|
||||||
# the ARG CACHEBUST line, since a cached lint layer is a lint that did
|
# that did not run. The tag makes each build replace the previous image
|
||||||
# not run; unlike --no-cache, it keeps the layers above that line and
|
# instead of leaving a dangling one behind.
|
||||||
# the cache mounts. This relies on every stage of the Dockerfile
|
|
||||||
# declaring ARG CACHEBUST before every step that lints, tests or
|
|
||||||
# compiles; a stage without that line is served from cache on an
|
|
||||||
# unchanged tree, with no warning.
|
|
||||||
# --output type=cacheonly writes no image, since nothing uses one.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -20,9 +15,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
docker build --no-cache \
|
||||||
--target lint \
|
--target lint \
|
||||||
--output type=cacheonly .
|
-t "$("$SCRIPT_DIR/projectname")-lint" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+4
-9
@@ -2,13 +2,8 @@
|
|||||||
# script/test: run the test suite. Testing is a phase of the Dockerfile
|
# script/test: run the test suite. Testing is a phase of the Dockerfile
|
||||||
# and this builds that phase alone, on the same terms as script/lint:
|
# and this builds that phase alone, on the same terms as script/lint:
|
||||||
# --target because a phase that is not the last stage is built only when
|
# --target because a phase that is not the last stage is built only when
|
||||||
# named, and a new CACHEBUST value because a cached test layer is a test
|
# named, --no-cache because a cached test layer is a test that did not
|
||||||
# that did not run; unlike --no-cache, it keeps the layers above the ARG
|
# run, and a tag so each build replaces the previous image.
|
||||||
# CACHEBUST line and the cache mounts. This relies on every stage of the
|
|
||||||
# Dockerfile declaring ARG CACHEBUST before every step that lints, tests
|
|
||||||
# or compiles; a stage without that line is served from cache on an
|
|
||||||
# unchanged tree, with no warning. --output type=cacheonly writes no
|
|
||||||
# image, since nothing uses one.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -16,9 +11,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
docker build --no-cache \
|
||||||
--target test \
|
--target test \
|
||||||
--output type=cacheonly .
|
-t "$("$SCRIPT_DIR/projectname")-test" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
-55
@@ -1,55 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
# script/yarn: run yarn with the given arguments under the node
|
|
||||||
# script/bootstrap would use. Every script that runs yarn on the host
|
|
||||||
# runs it through this one, except script/bootstrap, which picks its
|
|
||||||
# own node and yarn.
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|
||||||
|
|
||||||
# Must match the pins in script/bootstrap.
|
|
||||||
NODE_VERSION="22.17.0"
|
|
||||||
NODE_MIN_VERSION="22.13.0"
|
|
||||||
|
|
||||||
# node_ok: the node on PATH has the major version of NODE_VERSION and is
|
|
||||||
# at least NODE_MIN_VERSION. Only the major is compared with the pin,
|
|
||||||
# as REPO_POLICIES.md asks. node itself compares the versions: major,
|
|
||||||
# then minor, then patch. A node that fails does not pass.
|
|
||||||
# script/bootstrap and script/yarn carry this function unchanged.
|
|
||||||
node_ok() {
|
|
||||||
if ! command -v node >/dev/null 2>&1; then return 1; fi
|
|
||||||
node -e '
|
|
||||||
const have = process.versions.node.split(".").map(Number);
|
|
||||||
const want = process.argv[1].split(".").map(Number);
|
|
||||||
const pinned = process.argv[2].split(".").map(Number);
|
|
||||||
if (have[0] !== pinned[0]) process.exit(1);
|
|
||||||
for (let i = 0; i < 3; i++) {
|
|
||||||
if (have[i] !== want[i]) process.exit(have[i] > want[i] ? 0 : 1);
|
|
||||||
}
|
|
||||||
' "$NODE_MIN_VERSION" "$NODE_VERSION"
|
|
||||||
}
|
|
||||||
|
|
||||||
# Run the yarn script/bootstrap installed: the yarn on PATH when the
|
|
||||||
# node on PATH passes node_ok, and otherwise yarn under the pinned node
|
|
||||||
# in nvm. nvm is a bash script, hence the subshell.
|
|
||||||
run_yarn() {
|
|
||||||
if node_ok && command -v yarn >/dev/null 2>&1; then
|
|
||||||
exec yarn "$@"
|
|
||||||
fi
|
|
||||||
if node_ok || [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
|
||||||
echo "script/yarn: no yarn; run script/bootstrap first" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
|
|
||||||
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
|
||||||
cd "$ROOT"
|
|
||||||
# script/bootstrap links the node and yarn it installs into
|
|
||||||
# ~/.local/bin, which the shell that called it may not have on PATH.
|
|
||||||
PATH="$HOME/.local/bin:$PATH"
|
|
||||||
run_yarn "$@"
|
|
||||||
}
|
|
||||||
|
|
||||||
main "$@"
|
|
||||||
Reference in New Issue
Block a user