clawbot
e683ac3b61
A buffer failure on POST /api/v1/reports now returns 500 instead of a false `ok`, so clients can retry. An over-limit body returns 413 (errors.As on `*http.MaxBytesError`); malformed JSON stays 400. A MaxBodyBytes middleware (1 MiB) caps every route, rejecting an oversized Content-Length up front and capping the read otherwise; a route group can only lower that limit. The raw geo blob is no longer logged, only its length; client_id, timestamp and decode error text are length-bounded before logging. A decodeJSON handler helper is added. Panic recovery routes the stack through slog as structured JSON. Storage failure uses 500: a full buffer or write error is server-side and retryable. Model: opus-5-5
netwatch-server is an MIT-licensed Go HTTP backend by @sneak that receives telemetry reports from the NetWatch SPA and persists them as zstd-compressed JSONL files on disk.
Getting Started
# Build and run locally
make run
# Run tests, lint, and format check
make check
# Docker
docker build -t netwatch-server .
docker run -p 8080:8080 netwatch-server
Rationale
The NetWatch frontend collects latency measurements from the browser but has no
way to persist or aggregate them. This backend provides a minimal
POST /api/v1/reports endpoint that buffers incoming reports in memory and
flushes them to compressed files on disk for later analysis.
Design
The server is structured as an fx-wired Go application under cmd/netwatch-server/.
Internal packages in internal/ follow standard Go project layout:
config: Loads configuration from environment variables and config files via Viper.handlers: HTTP request handlers for the API (health check, report ingestion).reportbuf: In-memory buffer that accumulates JSONL report lines and flushes to zstd-compressed files when the buffer reaches 10 MiB or every 60 seconds.server: Chi-based HTTP server with middleware wiring and route registration.healthcheck,middleware,logger,globals: Supporting infrastructure.
Configuration
| Variable | Default | Description |
|---|---|---|
PORT |
8080 |
HTTP listen port |
DATA_DIR |
./data/reports |
Directory for compressed reports |
DEBUG |
false |
Enable debug logging |
TRUSTED_PROXIES |
loopback + RFC1918 | Comma-separated CIDRs whose X-Forwarded-For / X-Real-IP headers are trusted for client IP resolution |
TRUSTED_PROXIES defaults to 127.0.0.1/32,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16.
The loopback entries cover the reverse proxy that shares the container; the
RFC1918 ranges match nginx.conf. A request whose direct peer is outside this
set has its forwarded headers ignored, and the direct peer is logged instead.
Report storage
Reports are written as reports-<timestamp>.jsonl.zst files in DATA_DIR.
Each file contains one JSON object per line, compressed with zstd. Files are
created with O_EXCL to prevent overwrites.
TODO
- Add integration test that POSTs a report and verifies the compressed output
- Add report decompression/query endpoint
- Add metrics (Prometheus) for buffer size, flush count, report count
- Add retention policy to prune old report files
License
MIT. See LICENSE.