clawbot
2bf52ba7ff
check / check (push) Successful in 58s
POST /api/v1/reports stays unauthenticated but is bounded. Each client address, as the trusted-proxy logic resolves it, may send REPORTS_PER_MINUTE reports a minute (default 60, all at once if it likes), using golang.org/x/time/rate; past that it gets 429 with Retry-After. Buckets that have refilled are dropped once a minute, so idle addresses do not pile up. reportbuf refuses a report that would take the report files past DATA_DIR_MAX_BYTES (default 1 GiB) with ErrFull, answered with 507; the count starts from the files already in DATA_DIR, and reports not yet written count at their uncompressed size. CORS adds nothing unless CORS_ALLOWED_ORIGINS lists origins. A limit that is not a positive number stops the server from starting. Model: opus-5-5
141 lines
6.6 KiB
Markdown
141 lines
6.6 KiB
Markdown
netwatch-server is an MIT-licensed Go HTTP backend by
|
|
[@sneak](https://sneak.berlin) that receives telemetry reports from the NetWatch
|
|
SPA and persists them as zstd-compressed JSONL files on disk.
|
|
|
|
## Getting Started
|
|
|
|
From this directory:
|
|
|
|
```bash
|
|
# Build and run locally
|
|
make run
|
|
```
|
|
|
|
From the repo root, which is also the build context of `Dockerfile.backend`:
|
|
|
|
```bash
|
|
# Run tests, lint, and format check over the frontend and this backend
|
|
make check
|
|
|
|
# Build both images, including netwatch-server
|
|
make docker
|
|
docker run -p 8080:8080 netwatch-server
|
|
```
|
|
|
|
## Entrypoints
|
|
|
|
This directory follows the same
|
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
|
pattern as the repo root: the targets in `backend/Makefile` are thin shims over
|
|
`backend/script/`. `Dockerfile.backend` runs them, and the root scripts call
|
|
`test`, `fmt` and `fmt-check`:
|
|
|
|
- `script/build` — compile the static `netwatch-server` binary with its version
|
|
and architecture stamped in. The version is `VERSION` from the environment;
|
|
when that is unset or empty, it falls back to `git describe` inside a git
|
|
checkout, then to `dev`
|
|
- `script/test` — run the Go tests under a 30-second timeout
|
|
- `script/lint` — check `.golangci.yml` against its pinned sha256, then run
|
|
golangci-lint. It runs inside the golangci-lint image of the lint stage of
|
|
`Dockerfile.backend`; from a checkout, run `make lint` at the repo root, which
|
|
builds that stage
|
|
- `script/fmt` — format the Go sources (writes)
|
|
- `script/fmt-check` — check Go formatting (read-only)
|
|
- `script/run` — build and run the server locally
|
|
- `script/clean` — remove build artifacts
|
|
|
|
There is no `check`, `hooks` or `docker` target here: the root `make check`
|
|
covers this directory, the root `make hooks` installs the repo's only pre-commit
|
|
hook, and the root `make docker` builds this image.
|
|
|
|
## Rationale
|
|
|
|
The NetWatch frontend collects latency measurements from the browser but has no
|
|
way to persist or aggregate them. This backend provides a minimal
|
|
`POST /api/v1/reports` endpoint that buffers incoming reports in memory and
|
|
flushes them to compressed files on disk for later analysis.
|
|
|
|
## Design
|
|
|
|
The server is structured as an `fx`-wired Go application under `cmd/netwatch-server/`.
|
|
Internal packages in `internal/` follow standard Go project layout:
|
|
|
|
- **`config`**: Loads configuration from environment variables and config files
|
|
via Viper.
|
|
- **`handlers`**: HTTP request handlers for the API (health check, report
|
|
ingestion).
|
|
- **`reportbuf`**: In-memory buffer that accumulates JSONL report lines and
|
|
flushes to zstd-compressed files when the buffer reaches 10 MiB or every 60
|
|
seconds.
|
|
- **`server`**: Chi-based HTTP server with middleware wiring and route
|
|
registration.
|
|
- **`healthcheck`**, **`middleware`**, **`logger`**, **`globals`**: Supporting
|
|
infrastructure.
|
|
|
|
### Configuration
|
|
|
|
| Variable | Default | Description |
|
|
| ---------------------- | -------------------- | -------------------------------------------------------------------------------------------------------- |
|
|
| `PORT` | `8080` | HTTP listen port |
|
|
| `DATA_DIR` | `./data/reports` | Directory for compressed reports |
|
|
| `DATA_DIR_MAX_BYTES` | `1073741824` (1 GiB) | Most the report files in `DATA_DIR` may total; see [Report limits](#report-limits) |
|
|
| `DEBUG` | `false` | Enable debug logging |
|
|
| `TRUSTED_PROXIES` | loopback + RFC1918 | Comma-separated CIDRs whose `X-Forwarded-For` / `X-Real-IP` headers are trusted for client IP resolution |
|
|
| `REPORTS_PER_MINUTE` | `60` | Reports each client address may send a minute; see [Report limits](#report-limits) |
|
|
| `CORS_ALLOWED_ORIGINS` | empty | Comma-separated origins whose pages may call the API; see [CORS](#cors) |
|
|
|
|
`TRUSTED_PROXIES` defaults to `127.0.0.1/32,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`.
|
|
The loopback entries cover the reverse proxy that shares the container; the
|
|
RFC1918 ranges match `nginx.conf`. A request whose direct peer is outside this
|
|
set has its forwarded headers ignored, and the direct peer is logged instead.
|
|
|
|
### Report storage
|
|
|
|
Reports are written as `reports-<timestamp>.jsonl.zst` files in `DATA_DIR`.
|
|
Each file contains one JSON object per line, compressed with zstd. Files are
|
|
created with `O_EXCL` to prevent overwrites.
|
|
|
|
### Report limits
|
|
|
|
`POST /api/v1/reports` takes reports from anyone who can reach it, without
|
|
credentials, so it is bounded instead. Both refusals below answer with the same
|
|
`{"status":"error"}` body as any other error.
|
|
|
|
- **Rate limit.** Each client address, resolved through `TRUSTED_PROXIES`, may
|
|
send `REPORTS_PER_MINUTE` reports a minute, all at once if it likes. Past that
|
|
it gets 429 with a `Retry-After` header until its allowance refills, at one
|
|
report every 60 / `REPORTS_PER_MINUTE` seconds. The page sends one report a
|
|
minute from each open tab, so the default of 60 refuses nothing from up to 60
|
|
tabs behind one address, such as a household or an office sharing it, even
|
|
when all their reports arrive together.
|
|
- **Size cap.** The report files in `DATA_DIR` may total at most
|
|
`DATA_DIR_MAX_BYTES`, counting the files already there at start. Reports
|
|
waiting in memory count at their uncompressed size until they are written, so
|
|
a report that would take the total past the cap is refused with 507, and
|
|
nothing of it is stored. Deleting report files frees room only at the next
|
|
start, when the files are counted again. The default of 1 GiB is small enough
|
|
for any host; set it to the space you can give `DATA_DIR`.
|
|
|
|
### CORS
|
|
|
|
The page calls the API from the origin it is served from, so by default the
|
|
server sends no CORS headers, and browsers let no other origin's pages call it.
|
|
To serve the page from elsewhere, list that origin in `CORS_ALLOWED_ORIGINS`
|
|
(for example `https://netwatch.example.com`); pages from a listed origin may
|
|
`GET` and `POST` with a `Content-Type` header.
|
|
|
|
## TODO
|
|
|
|
- Add integration test that POSTs a report and verifies the compressed output
|
|
- Add report decompression/query endpoint
|
|
- Add metrics (Prometheus) for buffer size, flush count, report count
|
|
- Add retention policy to prune old report files
|
|
|
|
## License
|
|
|
|
MIT. See [LICENSE](LICENSE).
|
|
|
|
## Author
|
|
|
|
[@sneak](https://sneak.berlin)
|