netwatch/backend/README.md

netwatch-server is an MIT-licensed Go HTTP backend by
[@sneak](https://sneak.berlin) that receives telemetry reports from the NetWatch
SPA and persists them as zstd-compressed JSONL files on disk.

## Getting Started

From this directory:

```bash
# Build and run locally
make run
```

From the repo root, whose `Dockerfile` builds the one image that ships this
backend behind nginx (see [Container image](#container-image)):

```bash
# Run tests, lint, and format check over the frontend and this backend
make check

# Build the image: nginx, the frontend and this backend
make docker
docker run -p 8080:8080 netwatch
```

## Entrypoints

This directory follows the same
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
pattern as the repo root: the targets in `backend/Makefile` are thin shims over
`backend/script/`. The root `Dockerfile` runs them, and the root scripts call
`test`, `fmt` and `fmt-check`:

- `script/build` — compile the static `netwatch-server` binary with its version
  stamped in. The version is `VERSION` from the environment;
  when that is unset or empty, it falls back to `git describe` inside a git
  checkout, then to `dev`
- `script/test` — run the Go tests under a 30-second timeout
- `script/lint` — check `.golangci.yml` against its pinned sha256, then run
  golangci-lint. It runs inside the golangci-lint image of the lint stage of
  the root `Dockerfile`; from a checkout, run `make lint` at the repo root,
  which builds that stage
- `script/fmt` — format the Go sources (writes)
- `script/fmt-check` — check Go formatting (read-only)
- `script/run` — build and run the server locally
- `script/clean` — remove build artifacts

There is no `check`, `hooks` or `docker` target here: the root `make check`
covers this directory, the root `make hooks` installs the repo's only pre-commit
hook, and the root `make docker` builds the image that contains this backend.

## Rationale

The NetWatch frontend collects latency measurements from the browser but has no
way to persist or aggregate them. This backend provides a minimal
`POST /api/v1/reports` endpoint that buffers incoming reports in memory and
flushes them to compressed files on disk for later analysis.

## Design

The server is structured as an `fx`-wired Go application under `cmd/netwatch-server/`.
Internal packages in `internal/` follow standard Go project layout:

- **`config`**: Loads configuration from environment variables and config files
  via Viper.
- **`handlers`**: HTTP request handlers for the API (health check, report
  ingestion).
- **`reportbuf`**: In-memory buffer that accumulates JSONL report lines and
  flushes to zstd-compressed files when the buffer reaches 10 MiB or every 60
  seconds.
- **`server`**: Chi-based HTTP server with middleware wiring and route
  registration.
- **`healthcheck`**, **`middleware`**, **`logger`**, **`globals`**: Supporting
  infrastructure.

### Configuration

| Variable               | Default              | Description                                                                                              |
| ---------------------- | -------------------- | -------------------------------------------------------------------------------------------------------- |
| `BIND_ADDRESS`         | empty                | IP address to listen on; empty listens on every interface                                                |
| `PORT`                 | `8080`               | HTTP listen port                                                                                         |
| `DATA_DIR`             | `./data/reports`     | Directory for compressed reports                                                                         |
| `DATA_DIR_MAX_BYTES`   | `1073741824` (1 GiB) | Most bytes of report files kept in `DATA_DIR`, oldest deleted first; see [Report limits](#report-limits) |
| `DEBUG`                | `false`              | Enable debug logging                                                                                     |
| `TRUSTED_PROXIES`      | loopback + RFC1918   | Comma-separated CIDRs whose `X-Forwarded-For` / `X-Real-IP` headers are trusted for client IP resolution |
| `REPORTS_PER_MINUTE`   | `60`                 | Reports each client address may send a minute; see [Report limits](#report-limits)                       |
| `CORS_ALLOWED_ORIGINS` | empty                | Comma-separated origins whose pages may call the API; see [CORS](#cors)                                  |

`TRUSTED_PROXIES` defaults to `127.0.0.1/32,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`.
The loopback entries cover a reverse proxy on the same host. A request whose
direct peer is outside this set has its forwarded headers ignored, and the
direct peer is logged and rate-limited instead. The container image does not use
this default; see [Container image](#container-image).

A variable set to a value the server cannot use, such as `PORT=abc`,
`DEBUG=maybe` or a `BIND_ADDRESS` that is not an IP address, stops it from
starting, with an error naming the variable. An empty variable counts as unset.

### Container image

The root `Dockerfile` builds one image in which nginx listens on the public port
8080, serves the frontend, and proxies `/api/` and `/.well-known/healthcheck` to
this server. The image's entrypoint, `bin/entrypoint.sh`, starts the server as
user `netwatch` (uid 1000) with `BIND_ADDRESS=127.0.0.1` and `PORT=8081`, so
only nginx reaches it, and with `TRUSTED_PROXIES=127.0.0.1/32`, so it takes the
client address nginx passes on and no other. `DATA_DIR` is `/data/reports`, on
the `/data` volume; the entrypoint creates it and gives it and `/data` to
`netwatch` before starting the server. nginx replaces the security headers
this server sets with those in the root `security-headers.conf`, so those are
what clients of the image see.

The container's own `TRUSTED_PROXIES` goes to nginx instead: IP addresses or
CIDRs, separated by commas, of the reverse proxies in front of the container.
nginx takes the client address from `X-Forwarded-For` only on a request from one
of them. Unset or empty, nginx trusts no proxy, and the client address is the
one each request comes from, so every client behind a proxy shares one rate
limit. An entry that is not an IP address or CIDR, such as a hostname or
`1.2.3`, stops the container at start with an error naming `TRUSTED_PROXIES`:
the entrypoint checks each entry with `netwatch-server check-cidr`, which parses
it as this server parses its own `TRUSTED_PROXIES`.

### Report storage

Reports are written as `reports-<timestamp>-<number>.jsonl.zst` files in
`DATA_DIR`. The timestamp is in UTC to the millisecond, so the names sort by
time. The number starts at 1 when the server starts and goes up by one for each
file the server starts to write, so two files written in the same millisecond
still get different names. A failed write uses up its number, leaving a gap in
the numbers if the file could not be created and otherwise a file under that
number that may be incomplete. Each file contains one JSON object per line,
compressed with zstd. Files are created with `O_EXCL` to prevent overwrites.

### Report limits

`POST /api/v1/reports` takes reports from anyone who can reach it, without
credentials, so it is bounded instead. Both refusals below answer with the same
`{"status":"error"}` body as any other error.

- **Rate limit.** Each client address, resolved through `TRUSTED_PROXIES`, may
  send `REPORTS_PER_MINUTE` reports a minute; past that it gets 429 with
  `Retry-After: 60`. The minute slides: reports from the minute before still
  count, fading out over the current one, so an address is sure never to be
  refused only while it sends at most half of `REPORTS_PER_MINUTE` in any 60
  seconds. The page sends one report a minute from each open tab, so the default
  of 60 refuses nothing from up to 30 tabs behind one address, such as a
  household or an office sharing it, however their reports bunch up. Report
  responses also carry `X-RateLimit-Limit`, `X-RateLimit-Remaining` and
  `X-RateLimit-Reset` headers.
- **Size cap.** The report files in `DATA_DIR` may total at most
  `DATA_DIR_MAX_BYTES`, counting the files already there at start. Reports
  waiting in memory count at their uncompressed size until they are written.
  When a report would take the total past the cap, the oldest report files are
  deleted to make room, and each deletion is logged with the file's name and
  size; a file still being written is never deleted. A report is refused with
  507, and nothing of it is stored, only when the reports not yet written leave
  no room for it on their own, and then no file is deleted. At start, report
  files past the cap, as after lowering it, are deleted the same way. So the cap
  is how much of the newest reports is kept: the default of 1 GiB is small
  enough for any host; set it to the space you can give `DATA_DIR`.

### CORS

The page calls the API from the origin it is served from, so by default the
server sends no CORS headers, and browsers let no other origin's pages call it.
To serve the page from elsewhere, list that origin in `CORS_ALLOWED_ORIGINS`
(for example `https://netwatch.example.com`); pages from a listed origin may
`GET` and `POST` with a `Content-Type` header. Each entry must be a plain
origin, `scheme://host` with an optional `:port`, as browsers send it: no path,
not even a trailing `/`, and no `*`. Any other entry stops the server from
starting, with an error naming `CORS_ALLOWED_ORIGINS`.

## TODO

- Add integration test that POSTs a report and verifies the compressed output
- Add report decompression/query endpoint
- Add metrics (Prometheus) for buffer size, flush count, report count

## License

MIT. See [LICENSE](LICENSE).

## Author

[@sneak](https://sneak.berlin)