netwatch/backend

netwatch-server is an MIT-licensed Go HTTP backend by @sneak that receives telemetry reports from the NetWatch SPA and persists them as zstd-compressed JSONL files on disk.

Getting Started

From this directory:

# Build and run locally
make run

From the repo root, which is also the build context of Dockerfile.backend:

# Run tests, lint, and format check over the frontend and this backend
make check

# Build both images, including netwatch-server
make docker
docker run -p 8080:8080 netwatch-server

Entrypoints

This directory follows the same Scripts to Rule Them All pattern as the repo root: the targets in backend/Makefile are thin shims over backend/script/. Dockerfile.backend runs them, and the root scripts call test, fmt and fmt-check:

• script/build — compile the static netwatch-server binary with its version and architecture stamped in. The version is VERSION from the environment; when that is unset or empty, it falls back to git describe inside a git checkout, then to dev
• script/test — run the Go tests under a 30-second timeout
• script/lint — check .golangci.yml against its pinned sha256, then run golangci-lint. It runs inside the golangci-lint image of the lint stage of Dockerfile.backend; from a checkout, run make lint at the repo root, which builds that stage
• script/fmt — format the Go sources (writes)
• script/fmt-check — check Go formatting (read-only)
• script/run — build and run the server locally
• script/clean — remove build artifacts

There is no check, hooks or docker target here: the root make check covers this directory, the root make hooks installs the repo's only pre-commit hook, and the root make docker builds this image.

Rationale

The NetWatch frontend collects latency measurements from the browser but has no way to persist or aggregate them. This backend provides a minimal POST /api/v1/reports endpoint that buffers incoming reports in memory and flushes them to compressed files on disk for later analysis.

Design

The server is structured as an fx-wired Go application under cmd/netwatch-server/. Internal packages in internal/ follow standard Go project layout:

• config: Loads configuration from environment variables and config files via Viper.
• handlers: HTTP request handlers for the API (health check, report ingestion).
• reportbuf: In-memory buffer that accumulates JSONL report lines and flushes to zstd-compressed files when the buffer reaches 10 MiB or every 60 seconds.
• server: Chi-based HTTP server with middleware wiring and route registration.
• healthcheck, middleware, logger, globals: Supporting infrastructure.

Configuration

Variable	Default	Description
PORT	8080	HTTP listen port
DATA_DIR	./data/reports	Directory for compressed reports
DATA_DIR_MAX_BYTES	1073741824 (1 GiB)	Most the report files in DATA_DIR may total; see Report limits
DEBUG	false	Enable debug logging
TRUSTED_PROXIES	loopback + RFC1918	Comma-separated CIDRs whose X-Forwarded-For / X-Real-IP headers are trusted for client IP resolution
REPORTS_PER_MINUTE	60	Reports each client address may send a minute; see Report limits
CORS_ALLOWED_ORIGINS	empty	Comma-separated origins whose pages may call the API; see CORS

TRUSTED_PROXIES defaults to 127.0.0.1/32,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16. The loopback entries cover the reverse proxy that shares the container; the RFC1918 ranges match nginx.conf. A request whose direct peer is outside this set has its forwarded headers ignored, and the direct peer is logged instead.

Report storage

Reports are written as reports-<timestamp>.jsonl.zst files in DATA_DIR. Each file contains one JSON object per line, compressed with zstd. Files are created with O_EXCL to prevent overwrites.

Report limits

POST /api/v1/reports takes reports from anyone who can reach it, without credentials, so it is bounded instead. Both refusals below answer with the same {"status":"error"} body as any other error.

• Rate limit. Each client address, resolved through TRUSTED_PROXIES, may send REPORTS_PER_MINUTE reports a minute, all at once if it likes. Past that it gets 429 with a Retry-After header until its allowance refills, at one report every 60 / REPORTS_PER_MINUTE seconds. The page sends one report a minute from each open tab, so the default of 60 refuses nothing from up to 60 tabs behind one address, such as a household or an office sharing it, even when all their reports arrive together.
• Size cap. The report files in DATA_DIR may total at most DATA_DIR_MAX_BYTES, counting the files already there at start. Reports waiting in memory count at their uncompressed size until they are written, so a report that would take the total past the cap is refused with 507, and nothing of it is stored. Deleting report files frees room only at the next start, when the files are counted again. The default of 1 GiB is small enough for any host; set it to the space you can give DATA_DIR.

CORS

The page calls the API from the origin it is served from, so by default the server sends no CORS headers, and browsers let no other origin's pages call it. To serve the page from elsewhere, list that origin in CORS_ALLOWED_ORIGINS (for example https://netwatch.example.com); pages from a listed origin may GET and POST with a Content-Type header.

TODO

• Add integration test that POSTs a report and verifies the compressed output
• Add report decompression/query endpoint
• Add metrics (Prometheus) for buffer size, flush count, report count
• Add retention policy to prune old report files

License

MIT. See LICENSE.

Author

@sneak