clawbot
4d2bf12dae
check / check (push) Failing after 20s
Report files were named by a millisecond timestamp and created with O_EXCL, so two flushes in the same millisecond, such as a flush for size and the final flush at shutdown, got the same name and the second failed, losing its reports. Each name now carries a number after the timestamp that counts the files written since the server started, so names still sort by time and never repeat. The new test flushes pairs until one falls within one millisecond; the 1 ms pauses earlier tests used to dodge the collision are gone. Model: opus-5-5
160 lines
9.9 KiB
Markdown
160 lines
9.9 KiB
Markdown
# Workflow
|
|
|
|
- branch (from `main`)
|
|
- do the work in Next Step
|
|
- move Next Step to the top of Completed Steps
|
|
- move the top item of Future Steps into Next Step
|
|
- commit (`TODO.md` changes in the same commit as the work)
|
|
- merge to `main` if the branch is not protected, otherwise open a PR
|
|
- push
|
|
|
|
# Status
|
|
|
|
pre-1.0. No git tags. `feat/reportbuf-storage` is merged; the backend, the CI
|
|
workflow, and the backend repo standard files are all on `main`. Frontend and
|
|
backend are both functional. Working toward the 1.0.0 milestone by closing the
|
|
remaining repo-compliance issues on the tracker.
|
|
|
|
# Next Step
|
|
|
|
Confirm the `.gitea/workflows/check.yml` run is green (main always green
|
|
policy). The workflow file is already on `main`; what is unverified is that its
|
|
latest run passes.
|
|
|
|
# Completed Steps
|
|
|
|
- 2026-09-29: report file names can no longer collide (issue #61): each is
|
|
`reports-<timestamp>-<number>.jsonl.zst`, where the number counts the files
|
|
written since the server started, so two flushes in the same millisecond, such
|
|
as a flush for size and the final flush at shutdown, each get a file of their
|
|
own instead of the second one failing
|
|
- 2026-09-29: bounded the report endpoint (issue #20): `POST /api/v1/reports`
|
|
still needs no credentials, but each client address, as resolved through
|
|
`TRUSTED_PROXIES`, may send `REPORTS_PER_MINUTE` (default 60) reports a
|
|
minute, counted by `go-chi/httprate`, and past that gets 429 with
|
|
`Retry-After`; the report files in `DATA_DIR`, counted from start with those
|
|
already there, may total at most `DATA_DIR_MAX_BYTES` (default 1 GiB), past
|
|
which reports get 507; and the wildcard CORS is gone: no CORS headers unless
|
|
`CORS_ALLOWED_ORIGINS` lists origins, and an entry that is not a plain
|
|
`scheme://host[:port]` origin, `*` included, stops the server from starting.
|
|
Deleting report files frees room only at the next start; pruning is issue #54
|
|
- 2026-09-28: one container image (issue #52): the root `Dockerfile` builds the
|
|
only image, and `Dockerfile.backend` is gone. nginx serves the frontend on
|
|
port 8080 and proxies `/api/` and `/.well-known/healthcheck` to the backend,
|
|
which listens on `127.0.0.1:8081` in the same container; the new
|
|
`BIND_ADDRESS` setting sets its listen address. `bin/entrypoint.sh` starts
|
|
both, passes TERM and INT on to both, and exits non-zero when either exits on
|
|
its own. The backend runs as user `netwatch` and stores reports on the `/data`
|
|
volume. `script/docker` is the org model again
|
|
- 2026-09-28: unified the gate (issue #16): the root `make check` covers the Go
|
|
backend as well as the frontend, and the pre-commit hook with it; the backend
|
|
moved onto scripts-to-rule-them-all (`backend/script/*`, `backend/Makefile` as
|
|
shims, its duplicate hook installer removed); `script/cibuild` builds both
|
|
images and is the workflow's only build step. The root `make lint` runs
|
|
golangci-lint only in Docker, by building the lint stage of
|
|
`Dockerfile.backend` without the cache. `script/bootstrap` installs the pinned
|
|
Go unless the installed one is at least what `backend/go.mod` asks for, links
|
|
what it installs into `~/.local/bin` without replacing anything it did not
|
|
create, and installs no linter. Root `make test` runs both halves within one
|
|
30-second timeout. When `VERSION` is unset or empty, the backend binary's
|
|
version falls back to `git describe` inside a git checkout, then to `dev`
|
|
- 2026-09-28: frontend reporting client (issue #53): a `Reporter` class posts
|
|
collected samples to `/api/v1/reports` every `reportInterval` (default 60s) as
|
|
a per-host delta, with the report-building step a pure exported function of
|
|
host state; a per-host mark advances only on a delivered POST; at most one
|
|
report POST is pending at a time and it is abandoned after half the interval,
|
|
so a slow POST never overlaps the next report and a mark never moves
|
|
backwards; the samples of an abandoned POST are sent again at the next
|
|
interval, so a backend that stored them but answered late receives them twice;
|
|
the per-browser client id works in insecure (plain-HTTP) contexts;
|
|
`vite.config.js` proxies `/api` to the local backend for `yarn dev`
|
|
- 2026-09-28: report ingest correctness (issue #23): a storage failure now
|
|
returns 500 instead of a false `ok`; oversize bodies return 413 (distinguished
|
|
from malformed JSON, which stays 400); a `MaxBodyBytes` middleware caps every
|
|
route, not just the report route; the raw attacker-controlled `geo` blob is no
|
|
longer logged (only its length), and `client_id`, `timestamp` and decode error
|
|
text are length-bounded before logging; a `decodeJSON` handler helper was
|
|
added; panic recovery now routes the stack through slog instead of chi's
|
|
plain-text stderr; and writing a report file now returns its error, so a
|
|
failed final flush on shutdown makes the process exit non-zero instead of
|
|
losing the buffered reports silently
|
|
- 2026-09-21: shutdown lifecycle correctness. The process now shuts down through
|
|
fx instead of `os.Exit`, so every component's `OnStop` runs and buffered
|
|
reports are flushed to disk on `SIGTERM` — previously a full flush window of
|
|
telemetry was silently lost on every restart. The `http.Server` is now built
|
|
before its serving goroutine starts, so shutdown can no longer race or
|
|
nil-deref it; a listen failure exits non-zero via `fx.Shutdowner`; `reportbuf`
|
|
`OnStop` is idempotent; and `writeTimeout` now exceeds the chi per-request
|
|
budget so that budget is actually reachable. Dead `startupTime`, `exitCode`,
|
|
and `cancelFunc` fields were removed
|
|
- 2026-09-21: backend HTTP hardening (issue #19): added `ReadHeaderTimeout` and
|
|
`IdleTimeout` to the server, a `SecurityHeaders` middleware (HSTS, tight CSP,
|
|
frame/sniff/referrer/permissions headers) registered before CORS, and
|
|
trusted-proxy client IP resolution honouring `X-Forwarded-For` / `X-Real-IP`
|
|
only from a `TRUSTED_PROXIES` allowlist (loopback plus RFC1918 by default)
|
|
- 2026-08-10: adopted the org-standard `backend/.golangci.yml` verbatim and
|
|
moved the pinned golangci-lint from v2.7.2 to v2.12.2 (the `lint` stage of
|
|
`Dockerfile.backend` now pins the `golangci/golangci-lint:v2.12.2` image by
|
|
digest); the previous config declared `version: "2"` but used v1 schema keys,
|
|
so every threshold in it was inert and its green result was meaningless.
|
|
`backend/Makefile`'s `lint` target now asserts the config's sha256 against the
|
|
canonical file first, so drift from the org standard fails the build instead
|
|
of silently degrading to defaults
|
|
- 2026-08-10: every interactive control now meets the 44x44 CSS px minimum tap
|
|
target (`.pin-btn`, `#interval-select`, the debug-log label and, on narrow
|
|
viewports, `#pause-btn`). The pin button's hit area grows via matching
|
|
negative margins, so its layout footprint and row density are unchanged
|
|
- 2026-08-10: per-host status line wraps below the 768px breakpoint instead of
|
|
forcing horizontal page scroll at 320px
|
|
- 2026-08-09: `Dockerfile.backend` reworked to the mandated Go multistage
|
|
lint-stage pattern: separate `lint` stage on the hash-pinned
|
|
`golangci/golangci-lint` image, `COPY --from=lint` stage dependency,
|
|
`CGO_ENABLED=0` static build driven by `ARG VERSION`, and no more `COPY .git`
|
|
- 2026-08-09: dotfile compliance — lifted `backend/.editorconfig` to the repo
|
|
root so `root = true` covers the frontend too, and replaced `.gitignore` with
|
|
the org model (OS, editor, node, and environment/secrets sections) plus this
|
|
repo's `dist/` and `*.log`. `.env`, `.env.*`, `*.pem`, and `*.key` are now
|
|
ignored repo-wide, not just under `backend/`. Excluding `.git` from
|
|
`.dockerignore` stays deferred: both images read git metadata at build time
|
|
(`COPY .git` in `Dockerfile.backend`, `git rev-parse` in `vite.config.js`)
|
|
- 2026-08-09: automated responsive-layout harness
|
|
(`make frontend-viewport-test`): digest-pinned headless Chrome driven over CDP
|
|
against the built `dist/`, viewport widths derived from the breakpoints in
|
|
`src/styles.css` ([#13](https://git.eeqj.de/sneak/netwatch/issues/13)). Every
|
|
check carries a presence guard so none of them can pass against a page it is
|
|
not actually measuring. Found two real layout defects, filed as
|
|
[#42](https://git.eeqj.de/sneak/netwatch/issues/42) and
|
|
[#43](https://git.eeqj.de/sneak/netwatch/issues/43)
|
|
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
|
|
shims, README Entrypoints section
|
|
- 2026-02-27: backend with buffered zstd-compressed report storage; CI workflow
|
|
and backend repo standard files; backend Dockerfile fixed (Go 1.25,
|
|
golangci-lint) and moved to repo root (feat/reportbuf-storage)
|
|
- 2026-02-26: host row layout redesigned with CSS grid; overflow and spacing
|
|
fixes; nginx config extracted; port hardcoded to 8080
|
|
- 2026-02-26: debug log panel, median stats, recovery probe, Docker build fix,
|
|
S3 Singapore endpoint added
|
|
- 2026-02-23: summary box redesign, host pinning, local and UTC clocks, checks
|
|
counter
|
|
- 2026-02-23: hosts sorted by latency; GET instead of HEAD for latency; timeout
|
|
derived from interval; Hetzner regional endpoints; 3s interval
|
|
- 2026-01-29: initial NetWatch network latency monitor
|
|
|
|
# Future Steps
|
|
|
|
- Wire `script/frontend-viewport-test` into CI as its own step (deliberately not
|
|
part of `make check` today; the decision has real CI-runtime cost and is
|
|
tracked separately)
|
|
- Compliance top-up as one small commit: add .editorconfig and add the hooks
|
|
target to the Makefile
|
|
- After merge, confirm .gitea/workflows/check.yml is on main and CI is green
|
|
(main always green policy)
|
|
- Decide what to do with untracked resume.sh: commit it, gitignore it, or delete
|
|
it
|
|
- Upstream fix needed in `sneak/prompts`: the org-standard `.golangci.yml`
|
|
enables `gomodguard`, which golangci-lint v2.12.2 reports as deprecated since
|
|
v2.12.0 and replaced by `gomodguard_v2`, so every backend lint run prints a
|
|
deprecation warning. The file is standardized and must never be edited in this
|
|
repo, so nothing can be done here beyond tracking it — tracked at
|
|
<https://git.eeqj.de/sneak/netwatch/issues/41>
|