5 Commits
Author SHA1 Message Date
clawbot 186f932eb8 Footer no longer says IPv4 only (closes #111)
check / check (push) Successful in 2m48s
Each check is a fetch: the browser picks IPv4 or IPv6 for each WAN
host, and the local targets are IPv4 addresses, so "IPv4 only" was
wrong for the page as a whole. The footer's first line drops it and
its separator; the rest of the footer is unchanged. TODO.md moves this
to Completed Steps and the layout decision from Future Steps into Next
Step.

Model: opus-5-5
2026-10-04 07:38:06 +02:00
clawbot 64e142c17f README, TODO and the viewport README say what the tree does (closes #24)
check / check (push) Successful in 2m57s
README.md: Getting Started leads with make targets; a Backend section
gives netwatch-server's routes and how the image builds and runs it;
the checks are GET requests; the WAN host list, health states, summary
figures, sorting and missing features match src/main.js; the TODO
section points to TODO.md, which holds the one to-do list.

backend/README.md: its TODO section points to TODO.md too, whose
Future Steps take its three open items.

TODO.md: Workflow branches from next and opens the PR against next;
Status, Next Step and Future Steps describe the open work, linked to
its issue where one exists.

test/viewport/README.md: the unit tests run on Node's test runner,
not vitest.

Model: opus-5-5
2026-10-04 07:03:05 +02:00
clawbot 161f955ae2 Latency statistics written once, thresholds read from CONFIG (closes #102)
check / check (push) Successful in 2m50s
A target's min, max, median and average latency now come from one list
of its answers, through latencyStats(), which the summary's figures use
too, so the median is written once. latencyHex() and latencyClass() read
one table of color limits in CONFIG. The health thresholds, the debug
log's length, the gateway check's timeout, the recovery probe's number
of hosts and interval, how often the rows are sorted and the delay
before the first sparkline resize are CONFIG entries, read where the
numbers were. HostState's minLatency(), maxLatency(), averageLatency()
and medianLatency() are gone; the statistics test reads historyStats().
Nothing the page does or shows changes. A unit test now covers the
summary's figures.

Model: opus-5-5
2026-10-04 06:51:24 +02:00
clawbot d40e67d4ab Rate limit password attempts on /metrics (closes #104)
check / check (push) Successful in 3m32s
Each client address may make 60 requests to /metrics a minute,
through the same httprate middleware and TRUSTED_PROXIES
resolution the report route uses, with an allowance of its own.
The limit runs before the basic auth, so past it the answer is 429
and the password is not checked. backend/README.md says so; a test
uses up one client's allowance on wrong passwords, gets 429 with
the right one, and checks that another client behind the same
nginx still gets in.

Model: opus-5-5
2026-10-04 06:19:06 +02:00
clawbot dc2d240725 Report handler panics to Sentry when SENTRY_DSN is set (closes #95)
check / check (push) Successful in 3m6s
With SENTRY_DSN set, the server initialises sentry-go with the release
netwatch-server-<version>, adds the sentryhttp middleware with Repanic
as the last router-wide middleware, after the timeout, and flushes
Sentry for 2 seconds on shutdown. A DSN Sentry refuses stops the start
with an error naming SENTRY_DSN. With it empty, nothing is set up.

The metrics middleware stays on the matched routes only, so it runs
inside the Sentry middleware rather than before it.

Model: opus-5-5
2026-10-04 05:57:02 +02:00
13 changed files with 540 additions and 190 deletions
+107 -55
View File
@@ -1,30 +1,33 @@
NetWatch is an MIT-licensed JavaScript single-page application by NetWatch is an MIT-licensed JavaScript single-page application by
[@sneak](https://sneak.berlin) that provides real-time network latency [@sneak](https://sneak.berlin) that provides real-time network latency
monitoring to common internet hosts, displayed with color-coded figures and monitoring to common internet hosts, displayed with color-coded figures and
sparkline graphs, served from a static bucket or Docker container. sparkline graphs, served from a static bucket or from its Docker image, where a
small Go backend stores the measurements the page reports.
## Getting Started ## Getting Started
```bash ```bash
# Install dependencies # Install the dependencies and the git pre-commit hook
yarn install make setup
# Development server # Run the page on the Vite dev server
yarn dev make dev
# Production build # Run the tests, both linters and the format check
yarn build make check
# Preview production build # Build the page into dist/
yarn preview make build
# Docker # Build the image and run it
docker build -t netwatch . make docker
docker run -p 8080:8080 netwatch docker run -p 8080:8080 netwatch
``` ```
`yarn dev` proxies `/api` to `http://127.0.0.1:8080`, so a locally running `make check` and `make docker` need Docker. `make dev` passes `/api` to
`netwatch-server` (see `backend/`) receives the reports the page posts. `http://127.0.0.1:8080`, where `make run` in `backend/` starts `netwatch-server`
with its defaults, so the reports the page posts are stored in
`backend/data/reports`.
## Entrypoints ## Entrypoints
@@ -94,22 +97,25 @@ halves, so the root `make check` fails if either one is broken. We provide:
The narrow-viewport layout lives in the `max-width: 768px` media block in The narrow-viewport layout lives in the `max-width: 768px` media block in
`src/styles.css`. It is verified automatically by `make frontend-viewport-test`, `src/styles.css`. It is verified automatically by `make frontend-viewport-test`,
which drives a digest-pinned headless Chrome against the built `dist/` and which drives a digest-pinned headless Chrome against the built `dist/` and
asserts on computed layout at widths derived from that CSS — one pixel either asserts on computed layout at widths derived from that CSS — on every breakpoint
side of every breakpoint it declares, plus a 320px floor, a desktop baseline and it declares and one pixel either side of it, plus a 320px floor, a desktop
two landscape sizes. See [test/viewport/README.md](test/viewport/README.md) for baseline and two landscape sizes. See
what it covers and what it genuinely cannot. [test/viewport/README.md](test/viewport/README.md) for what it covers and what
it genuinely cannot.
## Rationale ## Rationale
When debugging network issues, it's useful to have a persistent at-a-glance view When debugging network issues, it's useful to have a persistent at-a-glance view
of latency and reachability to multiple well-known internet endpoints. NetWatch of latency and reachability to multiple well-known internet endpoints. NetWatch
provides this as a zero-dependency SPA that can be deployed anywhere static provides this as a single page that does all its measuring in the browser, so it
files are served, with no backend required. can be served from anywhere static files are served. The backend in its Docker
image only stores the measurements the page reports; without it, the page works
the same and nothing is stored.
## Design ## Design
The application is a single-page app built with Vite and Tailwind CSS v4. All The page is built with Vite and Tailwind CSS v4. Its code is all in
code lives in `src/main.js` with a class-based architecture: `src/main.js`, with a class-based architecture:
- **`CONFIG`**: Configuration object (update interval, timeouts, axis ticks, - **`CONFIG`**: Configuration object (update interval, timeouts, axis ticks,
etc.). The interval menu sets `updateInterval`, the one value the page writes etc.). The interval menu sets `updateInterval`, the one value the page writes
@@ -125,10 +131,10 @@ code lives in `src/main.js` with a class-based architecture:
`updateSummary()` / `updateHealthBox()` handle incremental updates `updateSummary()` / `updateHealthBox()` handle incremental updates
- **`tick()`**: Main loop — measures all hosts in parallel, pushing each host's - **`tick()`**: Main loop — measures all hosts in parallel, pushing each host's
sample and redrawing its row as soon as its check ends, then redraws every sample and redrawing its row as soon as its check ends, then redraws every
row, the summary and the health box once the last check ends. The rows are row, the summary and the health box once the last check ends. The first round,
sorted then too, after the first round that is not discarded and every tenth after loading or an interval change, is discarded. The rows are sorted when
round after that. When paused, pushes blank markers (no probes, no false the last check ends in round 2, the first one kept, and in rounds 11, 21, 31
outage) and so on. When paused, pushes blank markers (no probes, no false outage)
- **`Reporter`**: Posts collected samples to the backend - **`Reporter`**: Posts collected samples to the backend
### Reporting ### Reporting
@@ -142,12 +148,35 @@ delivered report, and while paused nothing is sent. Delivery failure is quiet
one debug-log line per outage, retried at the next interval, never blocking one debug-log line per outage, retried at the next interval, never blocking
probing. The report-building step is a pure function of host state. probing. The report-building step is a pure function of host state.
### Backend
`netwatch-server`, in `backend/`, is a small Go HTTP server that stores the
reports the page posts. It keeps them in memory and writes them to `DATA_DIR` as
zstd-compressed files of JSON lines: every minute, whenever 10 MiB are waiting,
and when it stops. Its routes:
- `POST /api/v1/reports` — takes a report, without credentials; each client
address may send a limited number a minute, and the report files are capped in
size, the oldest deleted first
- `GET /.well-known/healthcheck` — answers 200 with `"status":"ok"`, the
server's version and its uptime
- `GET /metrics` — Prometheus metrics behind basic auth, only when
`METRICS_USERNAME` and `METRICS_PASSWORD` are set; each client address may
make a limited number of requests to it a minute
In the image, the `builder` stage of `Dockerfile` tests it and builds it with
`backend/script/build`, and `bin/entrypoint.sh` runs it as user `netwatch` on
`127.0.0.1:8081`, behind nginx. Outside the image, `make run` in `backend/`
builds it and runs it on port 8080. Its settings, report storage and limits are
in [backend/README.md](backend/README.md).
### Monitoring targets ### Monitoring targets
- **22 WAN hosts**: datavi.be, Anthropic API, OpenAI API, AWS Console, GCP - **26 WAN hosts**: datavi.be (pinned at start), Anthropic API, OpenAI API, AWS
Console, Azure, Cloudflare, Fastly, Akamai, GitHub, B2, 7 S3 regional Console, Google Cloud Console, Microsoft Azure, Cloudflare, Fastly CDN,
endpoints (Cape Town, London, Bahrain, Tokyo, Sydney, Oregon, São Paulo), 4 Akamai, Google, GitHub, B2, 8 S3 regional endpoints (Cape Town, London,
GCS locational endpoints (Iowa, Belgium, Singapore, Sydney) Bahrain, Tokyo, Singapore, Sydney, Oregon, São Paulo) and 6 Hetzner speed test
servers (Nuremberg, Falkenstein, Helsinki, Ashburn, Hillsboro, Singapore)
- **Local CPE**: Cable modem at 192.168.100.1 (always monitored) - **Local CPE**: Cable modem at 192.168.100.1 (always monitored)
- **Local Gateway**: Auto-detected on startup by probing common default gateway - **Local Gateway**: Auto-detected on startup by probing common default gateway
addresses (192.168.1.1, 192.168.0.1, 192.168.8.1, 10.0.0.1); first responder addresses (192.168.1.1, 192.168.0.1, 192.168.8.1, 10.0.0.1); first responder
@@ -159,15 +188,17 @@ Local hosts are tracked separately from WAN stats.
### Latency measurement ### Latency measurement
HEAD requests with `mode: 'no-cors'` and `cache: 'no-store'`, timed with GET requests with `mode: 'no-cors'`, `cache: 'no-store'` and a cache-busting
`performance.now()`. Each check times out after 80% of the refresh interval (24 query parameter, timed with `performance.now()`. Each check times out after 80%
seconds at 30 seconds) and is then recorded as a timeout, so a round's checks of the refresh interval (24 seconds at 30 seconds) and is then recorded as a
have all finished before the next round is due. When no WAN host answers, a timeout, so a round's checks have all finished before the next round is due.
recovery probe checks 4 random WAN hosts every half second, giving up the checks When no WAN host answers, a recovery probe checks 4 WAN hosts, picked at random
it started half a second before. As soon as one answers, a new round starts at when it starts, every half second, giving up the checks it started half a second
once, as it does after an interval change. A round started early gives up the before. As soon as one answers, a new round starts at once, as it does after an
last round's checks if they are still waiting, and that round records nothing interval change. A round started early gives up the last round's checks if they
more, so rounds never overlap. IPv4 only. are still waiting, and that round records nothing more, so rounds never overlap.
The browser chooses between IPv4 and IPv6 for each target, as for any request;
the local targets are IPv4 addresses.
### Color coding ### Color coding
@@ -192,25 +223,42 @@ dist/
## Features ## Features
- Real-time monitoring with 2s update interval and 300s history sparklines - A round of checks every 3 seconds by default; the interval menu sets 1, 2, 3,
- Health indicator: green (HEALTHY) or red (DEGRADED) based on WAN reachability 5, 10, 15, 30 or 60 seconds and clears the history
- Summary stats: reachable count, min/max/avg latency across WAN hosts only - Sparklines of each target's last 100 rounds: 300 seconds at 3 seconds
- Fixed chart axes: Y-axis 0–1000ms, X-axis 0–300s - The first round after loading or an interval change is discarded, as DNS and
TLS setup inflate its latencies
- Health indicator from the WAN hosts' latest results: OFFLINE (red) when more
than 10 fail and at most 4 answer, otherwise DEGRADED (orange) when more than
4 fail, otherwise SLOW (yellow) when more than 3 take over 1000ms, otherwise
HEALTHY (green)
- Summary stats across WAN hosts only: how many answered, the min, median,
average and max of their latest latencies, the min and max over the whole
history, and the number of rounds run (`Checks`)
- Fixed chart axes: Y-axis 0–1000ms, higher latencies drawn at the top; X-axis
the time the history spans
- Color-coded latency figures and sparkline line segments - Color-coded latency figures and sparkline line segments
- WAN host rows sorted by latest latency, unreachable last; pinned rows stay on
top, in name order
- Play/pause: pause stops probes but history keeps scrolling (blank gaps, no - Play/pause: pause stops probes but history keeps scrolling (blank gaps, no
false outage) false outage)
- Debug log panel, behind a checkbox in the footer, with five levels (error,
warning, notice, info, debug) and the last 1000 lines
- Local and UTC clocks
- Clickable service URLs - Clickable service URLs
- A footer link to the commit the page was built from
- Canvas-based sparkline rendering with devicePixelRatio scaling - Canvas-based sparkline rendering with devicePixelRatio scaling
- Zero runtime dependencies: all resources bundled into build artifacts - Zero runtime dependencies: all resources bundled into build artifacts
## Deployment ## Deployment
After running `yarn build`, deploy the contents of the `dist/` directory to any `make build` writes the page to `dist/`, which any static file host (S3, GCS,
static file host (S3, GCS, Cloudflare Pages, Vercel, Netlify, GitHub Pages) or Cloudflare Pages, Vercel, Netlify, GitHub Pages) can serve; with no backend
use the Docker image behind a reverse proxy. there, its reports fail quietly and nothing is stored. Or run the Docker image
behind a reverse proxy.
The Docker image, built from `Dockerfile`, is the whole service in one The Docker image, built from `Dockerfile` by `make docker`, is the whole service
container: nginx serves the built frontend and passes `/api/`, in one container: nginx serves the built frontend and passes `/api/`,
`/.well-known/healthcheck` and `/metrics` to the Go backend, `netwatch-server`, `/.well-known/healthcheck` and `/metrics` to the Go backend, `netwatch-server`,
which listens only inside the container, on `127.0.0.1:8081`. The image: which listens only inside the container, on `127.0.0.1:8081`. The image:
@@ -266,6 +314,10 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
password as their basic auth credentials. With neither set, there are no password as their basic auth credentials. With neither set, there are no
metrics and `/metrics` is not found. One set without the other, or a user metrics and `/metrics` is not found. One set without the other, or a user
name containing `:`, stops the container name containing `:`, stops the container
- `SENTRY_DSN`, default empty: set to a Sentry project's DSN, the backend
sends its errors to that Sentry project: each request whose handling
crashes, which still gets a 500 response. A value Sentry does not accept
stops the container. Empty, the backend sends nothing to Sentry
- **Health check:** the image's `HEALTHCHECK` requests - **Health check:** the image's `HEALTHCHECK` requests
`/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless `/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless
both nginx and the backend answer. upaas reads the container's health 60 both nginx and the backend answer. upaas reads the container's health 60
@@ -279,19 +331,19 @@ properties.
## Limitations ## Limitations
- **CORS**: Some hosts may block cross-origin HEAD requests. The app uses - **CORS**: The checks are cross-origin requests in `no-cors` mode, so the page
`no-cors` mode which allows the request but provides opaque responses. Latency cannot read the answer, only time it: any answer counts as reachable, an error
is still measurable based on request timing. page included.
- **Local gateway**: The 192.168.100.1 endpoint requires the host to be - **Local targets**: The cable modem at 192.168.100.1 and the detected gateway
accessible from your network. answer only on a network that has them, and only when NetWatch is served from
localhost or a private address (see Monitoring targets).
- **Network conditions**: Measurements reflect browser-to-endpoint latency, - **Network conditions**: Measurements reflect browser-to-endpoint latency,
which includes your local network, ISP, and internet routing. which includes your local network, ISP, and internet routing.
## TODO ## TODO
- Add configurable host list (environment variable or config file) The to-do list is [TODO.md](TODO.md): where the work stands, the next step, the
- Add latency history export (CSV/JSON) open work, and what has been done.
- Add notification/alert when status changes to DEGRADED
## License ## License
+61 -19
View File
@@ -1,28 +1,68 @@
# Workflow # Workflow
- branch (from `main`) - branch from `next`
- do the work in Next Step - do the work in Next Step
- move Next Step to the top of Completed Steps - move Next Step to the top of Completed Steps
- move the top item of Future Steps into Next Step - move the top item of Future Steps into Next Step
- commit (`TODO.md` changes in the same commit as the work) - commit (`TODO.md` changes in the same commit as the work)
- merge to `main` if the branch is not protected, otherwise open a PR - push the branch and open a PR against `next`
- push
# Status # Status
pre-1.0. No git tags. `feat/reportbuf-storage` is merged; the backend, the CI pre-1.0. No git tags. `main` is the stable branch and `next` the development
workflow, and the backend repo standard files are all on `main`. Frontend and branch, which every PR targets. The frontend and the Go backend ship as one
backend are both functional. Working toward the 1.0.0 milestone by closing the Docker image, and the Gitea workflow `.gitea/workflows/check.yml` runs
remaining repo-compliance issues on the tracker. `script/cibuild` on every push. Working toward 1.0.0.
# Next Step # Next Step
Confirm the `.gitea/workflows/check.yml` run is green (main always green Decide whether the repo moves to the layout `REPO_POLICIES.md` gives, with
policy). The workflow file is already on `main`; what is unverified is that its `backend/` no longer repeating files from the root
latest run passes. ([#30](https://git.eeqj.de/sneak/netwatch/issues/30)).
# Completed Steps # Completed Steps
- 2026-10-04: the page's footer no longer says "IPv4 only"
([#111](https://git.eeqj.de/sneak/netwatch/issues/111)): each check is a
`fetch`, the browser picks IPv4 or IPv6 for each WAN host, and the local
targets are IPv4 addresses. The rest of the footer is unchanged
- 2026-10-04: `README.md`, `TODO.md` and `test/viewport/README.md` say what the
tree does (issue #24). The README's Getting Started leads with `make` targets;
a new Backend section says what `netwatch-server` stores, its routes and how
the image builds and runs it, and points to `backend/README.md` for its
settings; the checks are GET requests; the 26 WAN hosts, the four health
states, the summary's figures and the features the list lacked are described
as the page has them; and its TODO section points here, as does the one in
`backend/README.md`, whose open items moved to Future Steps. This file's
Workflow branches from `next` and opens the PR against `next`, Status says
where the repo stands, and Next Step and Future Steps hold only open work,
linked to its issue where one exists. The viewport harness README names Node's
test runner, not `vitest`
- 2026-10-04: in `src/main.js` (issue #102), a target's min, max, median and
average latency come from one list of its answers, through the same function
the summary's figures use, so the median is written once. The latency color
limits are one table in `CONFIG`, read by both the figure's and the
sparkline's color. The health thresholds, the debug log's length, the gateway
check's timeout, the recovery probe's number of hosts and interval, how often
the rows are sorted and the delay before the first sparkline resize are
`CONFIG` entries too. A unit test checks the summary's figures. Nothing the
page does or shows changed; the footer's color legend still writes the limits
out as text
- 2026-10-04: password guesses at `/metrics` are rate limited (issue #104): each
client address, resolved through `TRUSTED_PROXIES` as for reports, may make 60
requests to `/metrics` a minute, counted by `go-chi/httprate` apart from its
reports; past that it gets 429 and its basic auth credentials are not checked.
The limit is a constant in `backend/internal/server/routes.go`. A test uses up
one client's allowance on wrong passwords, gets 429 with the right one, and
checks that another client behind the same nginx still gets in
- 2026-10-04: the backend reports errors to Sentry (issue #95). With
`SENTRY_DSN` set, it sets up `sentry-go` with the release `netwatch-server-`
and its version, reports each panic in a handler through `sentryhttp`, the
last of the middleware every request goes through, which panics again so the
request still gets the 500 from the panic recovery, and waits up to 2 seconds
on shutdown for Sentry to finish sending. A DSN Sentry refuses stops the start
with an error naming `SENTRY_DSN`. With it empty, Sentry is not set up and
nothing is sent to it
- 2026-10-04: a target's name and URL and a debug log message show as the - 2026-10-04: a target's name and URL and a debug log message show as the
characters they are and are never read as HTML (issue #29): a host row escapes characters they are and are never read as HTML (issue #29): a host row escapes
the name and URL it writes into its markup, and the debug log sets each line the name and URL it writes into its markup, and the debug log sets each line
@@ -349,12 +389,14 @@ latest run passes.
# Future Steps # Future Steps
- Wire `script/frontend-viewport-test` into CI as its own step (deliberately not - Run `make frontend-viewport-test` in CI as its own step; it is not part of
part of `make check` today; the decision has real CI-runtime cost and is `make check`, as it needs Docker and takes minutes
tracked separately) - A backend test that posts a report to `POST /api/v1/reports` and checks the
- Compliance top-up as one small commit: add .editorconfig and add the hooks compressed file it is written to
target to the Makefile - A backend route that decompresses the stored reports and answers queries on
- After merge, confirm .gitea/workflows/check.yml is on main and CI is green them
(main always green policy) - Prometheus metrics for the backend's in-memory buffer: its size, the number of
- Decide what to do with untracked resume.sh: commit it, gitignore it, or delete flushes and the number of reports
it - A configurable host list (an environment variable or a config file)
- Export of the latency history (CSV or JSON)
- A notification when the health status changes to DEGRADED
+21 -3
View File
@@ -90,6 +90,7 @@ project layout:
| `CORS_ALLOWED_ORIGINS` | empty | Comma-separated origins whose pages may call the API; see [CORS](#cors) | | `CORS_ALLOWED_ORIGINS` | empty | Comma-separated origins whose pages may call the API; see [CORS](#cors) |
| `METRICS_USERNAME` | empty | Basic auth user name for `/metrics`; see [Metrics](#metrics) | | `METRICS_USERNAME` | empty | Basic auth user name for `/metrics`; see [Metrics](#metrics) |
| `METRICS_PASSWORD` | empty | Basic auth password for `/metrics`; see [Metrics](#metrics) | | `METRICS_PASSWORD` | empty | Basic auth password for `/metrics`; see [Metrics](#metrics) |
| `SENTRY_DSN` | empty | DSN of the Sentry project to send errors to; see [Sentry](#sentry) |
`TRUSTED_PROXIES` defaults to `TRUSTED_PROXIES` defaults to
`127.0.0.1/32,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`. The loopback `127.0.0.1/32,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`. The loopback
@@ -198,11 +199,28 @@ is recorded and `/metrics` answers 404. One without the other stops the server
from starting, with an error naming both; so does a `METRICS_USERNAME` from starting, with an error naming both; so does a `METRICS_USERNAME`
containing `:`, which basic auth cannot carry, with an error naming it. containing `:`, which basic auth cannot carry, with an error naming it.
`/metrics` is rate limited, so that its password cannot be guessed quickly: each
client address, resolved through `TRUSTED_PROXIES`, may make 60 requests to it a
minute, whatever their credentials. Past that it gets 429 with
`Retry-After: 60`, and its credentials are not checked. The minute slides as it
does for reports (see [Report limits](#report-limits)), so a scraper polling
every 2 seconds or less often is never refused. This allowance is apart from the
one for reports.
### Sentry
With `SENTRY_DSN` set, the server sends its errors to that Sentry project: each
panic in a handler is reported there, under the release `netwatch-server-`
followed by the server's version, and the request still gets 500 from the
server's panic recovery. On shutdown the server waits up to 2 seconds for Sentry
to finish sending. A DSN Sentry refuses stops the server from starting, with an
error naming `SENTRY_DSN`. With it empty, Sentry is not set up, and nothing is
sent to it.
## TODO ## TODO
- Add integration test that POSTs a report and verifies the compressed output The to-do list, this backend's open work included, is [TODO.md](../TODO.md) at
- Add report decompression/query endpoint the repo root.
- Add metrics (Prometheus) for buffer size, flush count, report count
## License ## License
+22
View File
@@ -115,6 +115,28 @@ func TestMalformedConfigFileStopsTheStart(t *testing.T) {
} }
} }
// TestRefusedSentryDSNStopsTheStart: a SENTRY_DSN that Sentry refuses
// stops the start, and the error, naming SENTRY_DSN, is logged as JSON.
func TestRefusedSentryDSNStopsTheStart(t *testing.T) {
t.Setenv("SENTRY_DSN", "not-a-dsn")
ctx, cancel := context.WithTimeout(t.Context(), childTimeout)
defer cancel()
child, stdout, stderr := startServer(ctx, t, t.TempDir(), freePort(ctx, t))
err := child.Wait()
if child.ProcessState.ExitCode() != 1 {
t.Fatalf("server exit = %v, want exit status 1", err)
}
requireJSONLines(t, stdout, stderr)
if !strings.Contains(stdout.String(), "SENTRY_DSN") {
t.Fatalf("no error naming SENTRY_DSN in stdout:\n%s", stdout)
}
}
// startServer runs main() in a child process listening on // startServer runs main() in a child process listening on
// 127.0.0.1:port, with home as its HOME and working directory and its // 127.0.0.1:port, with home as its HOME and working directory and its
// data directory in home, so it touches nothing outside home. Its // data directory in home, so it touches nothing outside home. Its
+1
View File
@@ -4,6 +4,7 @@ go 1.25.5
require ( require (
github.com/99designs/basicauth-go v0.0.0-20230316000542-bf6f9cbbf0f8 github.com/99designs/basicauth-go v0.0.0-20230316000542-bf6f9cbbf0f8
github.com/getsentry/sentry-go v0.49.0
github.com/go-chi/chi/v5 v5.2.5 github.com/go-chi/chi/v5 v5.2.5
github.com/go-chi/cors v1.2.2 github.com/go-chi/cors v1.2.2
github.com/go-chi/httprate v0.16.0 github.com/go-chi/httprate v0.16.0
+16 -8
View File
@@ -4,18 +4,22 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8= github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0= github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k= github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0= github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/getsentry/sentry-go v0.49.0 h1:Ehejknu1l023Ub7QoRBVLAI7g3Jnhqku4oWx4B4Sh5s=
github.com/getsentry/sentry-go v0.49.0/go.mod h1:nuMJAoCfe1u0Bts2ocyNI+TW8HT84vRMqwA5Qq/SKUI=
github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug= github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0= github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
github.com/go-chi/cors v1.2.2 h1:Jmey33TE+b+rB7fT8MUy1u0I4L+NARQlK6LhzKPSyQE= github.com/go-chi/cors v1.2.2 h1:Jmey33TE+b+rB7fT8MUy1u0I4L+NARQlK6LhzKPSyQE=
github.com/go-chi/cors v1.2.2/go.mod h1:sSbTewc+6wYHBBCW7ytsFSn836hqM7JxpglAy2Vzc58= github.com/go-chi/cors v1.2.2/go.mod h1:sSbTewc+6wYHBBCW7ytsFSn836hqM7JxpglAy2Vzc58=
github.com/go-chi/httprate v0.16.0 h1:8V5DH9j6pSK6UQoBsTpvMyFxycqaKEIToyPKzHJjUa8= github.com/go-chi/httprate v0.16.0 h1:8V5DH9j6pSK6UQoBsTpvMyFxycqaKEIToyPKzHJjUa8=
github.com/go-chi/httprate v0.16.0/go.mod h1:A8lo+qRhk+s9LiuP5saS7XCGDXRXMcrueq0NfIuCa/I= github.com/go-chi/httprate v0.16.0/go.mod h1:A8lo+qRhk+s9LiuP5saS7XCGDXRXMcrueq0NfIuCa/I=
github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs= github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM= github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
@@ -36,8 +40,12 @@ github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4= github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY= github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pingcap/errors v0.11.4 h1:lFuQV/oaUMGcD2tqt+01ROSmJs75VG1ToEOkZIZ4nE4=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pingcap/errors v0.11.4/go.mod h1:Oi8TUi2kEtXXLMJk9l1cGmz20kV3TaQ0usTwv5KuLY8=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU= github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU=
github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE= github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk= github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
@@ -46,8 +54,8 @@ github.com/prometheus/common v0.70.1 h1:1HvjP4D5oL3t8RsPlwxA9onvvStjtIHYE5XuuwOi
github.com/prometheus/common v0.70.1/go.mod h1:VdFUQDMZK3VLkurFUVhia6uys/0suUp86TJz5qbJRhc= github.com/prometheus/common v0.70.1/go.mod h1:VdFUQDMZK3VLkurFUVhia6uys/0suUp86TJz5qbJRhc=
github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI= github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI=
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY= github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8= github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc= github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik= github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
github.com/slok/go-http-metrics v0.13.0 h1:lQDyJJx9wKhmbliyUsZ2l6peGnXRHjsjoqPt5VYzcP8= github.com/slok/go-http-metrics v0.13.0 h1:lQDyJJx9wKhmbliyUsZ2l6peGnXRHjsjoqPt5VYzcP8=
@@ -93,7 +101,7 @@ golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+12
View File
@@ -1,9 +1,21 @@
package server package server
import "github.com/go-chi/chi/v5"
// Router exposes the router to the external tests, which add routes
// of their own to it after SetupRoutes.
func (s *Server) Router() *chi.Mux {
return s.router
}
// MaxRequestBodyBytes exposes the router-wide body limit to the // MaxRequestBodyBytes exposes the router-wide body limit to the
// external tests. // external tests.
const MaxRequestBodyBytes = maxRequestBodyBytes const MaxRequestBodyBytes = maxRequestBodyBytes
// MetricsRequestsPerMinute exposes the /metrics rate limit to the
// external tests.
const MetricsRequestsPerMinute = metricsRequestsPerMinute
// ListenAddr exposes the address the server listens on to the // ListenAddr exposes the address the server listens on to the
// external tests. // external tests.
func (s *Server) ListenAddr() string { func (s *Server) ListenAddr() string {
+21 -4
View File
@@ -3,6 +3,7 @@ package server
import ( import (
"time" "time"
sentryhttp "github.com/getsentry/sentry-go/http"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
"github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus"
@@ -17,6 +18,12 @@ const (
// can mount s.mw.MaxBodyBytes with a smaller value to lower // can mount s.mw.MaxBodyBytes with a smaller value to lower
// its bound, but cannot raise it: this cap runs first. // its bound, but cannot raise it: this cap runs first.
maxRequestBodyBytes int64 = 1 << 20 // 1 MiB maxRequestBodyBytes int64 = 1 << 20 // 1 MiB
// metricsRequestsPerMinute is how many requests to /metrics each
// client address may make a minute, whatever their credentials. A
// scraper polling every 2 seconds sends half of it, which httprate
// never refuses.
metricsRequestsPerMinute = 60
) )
// SetupRoutes configures the chi router with middleware and // SetupRoutes configures the chi router with middleware and
@@ -32,6 +39,12 @@ func (s *Server) SetupRoutes() {
s.router.Use(s.mw.MaxBodyBytes(maxRequestBodyBytes)) s.router.Use(s.mw.MaxBodyBytes(maxRequestBodyBytes))
s.router.Use(middleware.Timeout(requestTimeout)) s.router.Use(middleware.Timeout(requestTimeout))
// Sentry reports a panic, then panics again, so that s.mw.Recoverer
// still answers 500.
if s.params.Config.SentryDSN != "" {
s.router.Use(sentryhttp.New(sentryhttp.Options{Repanic: true}).Handle)
}
// The metrics go in a registry of this server's own, not in // The metrics go in a registry of this server's own, not in
// Prometheus' default one, which takes them only once per process. // Prometheus' default one, which takes them only once per process.
registry := prometheus.NewRegistry() registry := prometheus.NewRegistry()
@@ -59,10 +72,14 @@ func (s *Server) SetupRoutes() {
Post("/api/v1/reports", s.h.HandleReport()) Post("/api/v1/reports", s.h.HandleReport())
}) })
// The rate limit comes before the basic auth, so a client past it
// gets 429 and its password is not checked.
if s.params.Config.MetricsUsername != "" { if s.params.Config.MetricsUsername != "" {
s.router.With(s.mw.MetricsAuth()). s.router.With(
Get("/metrics", promhttp.HandlerFor( s.mw.RateLimit(metricsRequestsPerMinute),
registry, promhttp.HandlerOpts{}, s.mw.MetricsAuth(),
).ServeHTTP) ).Get("/metrics", promhttp.HandlerFor(
registry, promhttp.HandlerOpts{},
).ServeHTTP)
} }
} }
+111
View File
@@ -1,10 +1,12 @@
package server_test package server_test
import ( import (
"io"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"strings" "strings"
"testing" "testing"
"time"
"sneak.berlin/go/netwatch/internal/config" "sneak.berlin/go/netwatch/internal/config"
"sneak.berlin/go/netwatch/internal/globals" "sneak.berlin/go/netwatch/internal/globals"
@@ -15,6 +17,7 @@ import (
"sneak.berlin/go/netwatch/internal/reportbuf" "sneak.berlin/go/netwatch/internal/reportbuf"
"sneak.berlin/go/netwatch/internal/server" "sneak.berlin/go/netwatch/internal/server"
"github.com/getsentry/sentry-go"
"go.uber.org/fx" "go.uber.org/fx"
"go.uber.org/fx/fxtest" "go.uber.org/fx/fxtest"
) )
@@ -185,6 +188,50 @@ func TestMetricsBehindBasicAuth(t *testing.T) {
} }
} }
// TestMetricsAreRateLimited: a client that has used up its /metrics
// allowance on wrong passwords gets 429 even with the right one, which
// is then not checked, while another client behind the same nginx
// still gets in.
func TestMetricsAreRateLimited(t *testing.T) {
t.Setenv("METRICS_USERNAME", "prometheus")
t.Setenv("METRICS_PASSWORD", "right")
// As in the container: nginx connects from loopback and names the
// client in X-Forwarded-For.
t.Setenv("TRUSTED_PROXIES", "127.0.0.1/32")
srv := newServer(t)
srv.SetupRoutes()
get := func(client, password string) int {
rec := httptest.NewRecorder()
req := httptest.NewRequestWithContext(t.Context(),
http.MethodGet, "/metrics", http.NoBody)
req.RemoteAddr = "127.0.0.1:40000"
req.Header.Set("X-Forwarded-For", client)
req.SetBasicAuth("prometheus", password)
srv.ServeHTTP(rec, req)
return rec.Code
}
for i := range server.MetricsRequestsPerMinute {
if code := get("203.0.113.7", "wrong"); code != http.StatusUnauthorized {
t.Fatalf("guess %d: status = %d, want %d",
i+1, code, http.StatusUnauthorized)
}
}
if code := get("203.0.113.7", "right"); code != http.StatusTooManyRequests {
t.Fatalf("right password past the limit: status = %d, want %d",
code, http.StatusTooManyRequests)
}
if code := get("203.0.113.8", "right"); code != http.StatusOK {
t.Fatalf("another client: status = %d, want %d",
code, http.StatusOK)
}
}
// TestMetricsInTwoServers: two servers in one process can both have // TestMetricsInTwoServers: two servers in one process can both have
// metrics on. // metrics on.
func TestMetricsInTwoServers(t *testing.T) { func TestMetricsInTwoServers(t *testing.T) {
@@ -196,6 +243,70 @@ func TestMetricsInTwoServers(t *testing.T) {
} }
} }
// TestSentry: with SENTRY_DSN empty there is no Sentry client. With it
// pointing at a local server standing in for Sentry, a panic in a
// handler reaches that server, and the request still gets the 500 from
// the panic recovery.
func TestSentry(t *testing.T) {
const panicMessage = "handler panic for TestSentry"
// sentry.Init sets the client for the whole process; take it away
// again so that no other test reports to Sentry.
t.Cleanup(func() { sentry.CurrentHub().BindClient(nil) })
t.Setenv("SENTRY_DSN", "")
newServer(t)
if sentry.CurrentHub().Client() != nil {
t.Fatal("a Sentry client exists with SENTRY_DSN empty")
}
// The body of the first request the stand-in for Sentry receives.
received := make(chan string, 1)
sentryServer := httptest.NewServer(http.HandlerFunc(
func(_ http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
select {
case received <- string(body):
default:
}
},
))
defer sentryServer.Close()
t.Setenv("SENTRY_DSN",
"http://key@"+sentryServer.Listener.Addr().String()+"/1")
srv := newServer(t)
srv.SetupRoutes()
srv.Router().Get("/panic", func(http.ResponseWriter, *http.Request) {
panic(panicMessage)
})
rec := httptest.NewRecorder()
req := httptest.NewRequestWithContext(t.Context(),
http.MethodGet, "/panic", http.NoBody)
srv.ServeHTTP(rec, req)
if rec.Code != http.StatusInternalServerError {
t.Fatalf("status = %d, want %d",
rec.Code, http.StatusInternalServerError)
}
// Sentry sends from a goroutine of its own.
select {
case body := <-received:
if !strings.Contains(body, panicMessage) {
t.Fatalf("the Sentry server received no report of the panic:\n%s",
body)
}
case <-time.After(5 * time.Second):
t.Fatal("nothing reached the Sentry server")
}
}
// TestHealthCheckRejectsOversizeBody sends the health check, which // TestHealthCheckRejectsOversizeBody sends the health check, which
// never reads its body, a body one byte over the limit. Only the // never reads its body, a body one byte over the limit. Only the
// router-wide body limit can reject it. // router-wide body limit can reject it.
+40 -1
View File
@@ -7,8 +7,10 @@ package server
import ( import (
"context" "context"
"fmt"
"log/slog" "log/slog"
"net/http" "net/http"
"time"
"sneak.berlin/go/netwatch/internal/config" "sneak.berlin/go/netwatch/internal/config"
"sneak.berlin/go/netwatch/internal/globals" "sneak.berlin/go/netwatch/internal/globals"
@@ -16,10 +18,15 @@ import (
"sneak.berlin/go/netwatch/internal/logger" "sneak.berlin/go/netwatch/internal/logger"
"sneak.berlin/go/netwatch/internal/middleware" "sneak.berlin/go/netwatch/internal/middleware"
"github.com/getsentry/sentry-go"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"go.uber.org/fx" "go.uber.org/fx"
) )
// sentryFlushTimeout is how long shutdown waits for Sentry to send
// what it still holds.
const sentryFlushTimeout = 2 * time.Second
// Params defines the dependencies for Server. // Params defines the dependencies for Server.
type Params struct { type Params struct {
fx.In fx.In
@@ -56,6 +63,11 @@ func New(
s.log = params.Logger.Get() s.log = params.Logger.Get()
s.shutdowner = params.Shutdowner s.shutdowner = params.Shutdowner
err := s.enableSentry()
if err != nil {
return nil, err
}
lc.Append(fx.Hook{ lc.Append(fx.Hook{
OnStart: func(_ context.Context) error { OnStart: func(_ context.Context) error {
// Build the router and http.Server synchronously // Build the router and http.Server synchronously
@@ -87,10 +99,37 @@ func (s *Server) ServeHTTP(
s.router.ServeHTTP(w, r) s.router.ServeHTTP(w, r)
} }
// enableSentry sets Sentry up when SENTRY_DSN is set, so that
// SetupRoutes can report panics to it. With SENTRY_DSN empty it does
// nothing. A DSN Sentry refuses stops the start.
func (s *Server) enableSentry() error {
if s.params.Config.SentryDSN == "" {
return nil
}
err := sentry.Init(sentry.ClientOptions{
Dsn: s.params.Config.SentryDSN,
Release: s.params.Globals.Appname + "-" + s.params.Globals.Version,
})
if err != nil {
return fmt.Errorf("SENTRY_DSN: %w", err)
}
s.log.Info("sentry error reporting activated")
return nil
}
// shutdown gracefully stops the HTTP server within the // shutdown gracefully stops the HTTP server within the
// deadline of the context fx provides for OnStop. // deadline of the context fx provides for OnStop, then gives
// Sentry, if set up, time to send what it still holds.
func (s *Server) shutdown(ctx context.Context) error { func (s *Server) shutdown(ctx context.Context) error {
err := s.httpServer.Shutdown(ctx) err := s.httpServer.Shutdown(ctx)
if s.params.Config.SentryDSN != "" {
sentry.Flush(sentryFlushTimeout)
}
if err != nil { if err != nil {
s.log.Error("server clean shutdown failed", "error", err) s.log.Error("server clean shutdown failed", "error", err)
+94 -83
View File
@@ -30,6 +30,39 @@ export const CONFIG = {
return [0, 1, 2, 3, 4, 5].map((i) => Math.round((d * i) / 5)); return [0, 1, 2, 3, 4, 5].map((i) => Math.round((d * i) / 5));
}, },
canvasHeight: 96, canvasHeight: 96,
// A latency figure and its sparkline take the color of the first entry
// whose limit, in ms, the latency is below.
latencyColors: [
{ below: 50, hex: "#22c55e", className: "text-green-500" },
{ below: 100, hex: "#84cc16", className: "text-lime-500" },
{ below: 200, hex: "#eab308", className: "text-yellow-500" },
{ below: 500, hex: "#f97316", className: "text-orange-500" },
{ below: Infinity, hex: "#ef4444", className: "text-red-500" },
],
// The health is offline when more than offlineTimeouts WAN hosts timed
// out or were unreachable and at most offlineReachable answered;
// otherwise degraded when more than degradedTimeouts timed out or were
// unreachable; otherwise slow when more than slowHosts answered after
// more than slowLatency ms.
offlineTimeouts: 10,
offlineReachable: 4,
degradedTimeouts: 4,
slowHosts: 3,
slowLatency: 1000,
// The debug log keeps its last maxLogEntries lines.
maxLogEntries: 1000,
// A gateway candidate that has not answered after gatewayTimeout ms is
// passed over.
gatewayTimeout: 1500,
// When no WAN host answers, the recovery probe checks recoveryProbeHosts
// random ones every recoveryProbeInterval ms.
recoveryProbeHosts: 4,
recoveryProbeInterval: 500,
// The rows are sorted after the first round that is not discarded, then
// every roundsPerSort rounds.
roundsPerSort: 10,
// The sparklines are sized and drawn again resizeDelay ms after start.
resizeDelay: 100,
}; };
// WAN endpoints to monitor. These are used for the aggregate health/stats // WAN endpoints to monitor. These are used for the aggregate health/stats
@@ -114,7 +147,8 @@ const debugLog = [];
const log = (() => { const log = (() => {
function append(level, message) { function append(level, message) {
debugLog.push({ timestamp: new Date(), level, message }); debugLog.push({ timestamp: new Date(), level, message });
if (debugLog.length > 1000) debugLog.splice(0, debugLog.length - 1000); if (debugLog.length > CONFIG.maxLogEntries)
debugLog.splice(0, debugLog.length - CONFIG.maxLogEntries);
const panel = document.getElementById("debug-panel"); const panel = document.getElementById("debug-panel");
if (panel && !panel.classList.contains("hidden")) renderDebugLog(); if (panel && !panel.classList.contains("hidden")) renderDebugLog();
} }
@@ -174,7 +208,10 @@ async function detectGateway() {
const result = await Promise.any( const result = await Promise.any(
GATEWAY_CANDIDATES.map(async (url) => { GATEWAY_CANDIDATES.map(async (url) => {
const controller = new AbortController(); const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), 1500); const timeoutId = setTimeout(
() => controller.abort(),
CONFIG.gatewayTimeout,
);
try { try {
await fetch(url, { await fetch(url, {
method: "GET", method: "GET",
@@ -199,6 +236,27 @@ async function detectGateway() {
// --- App State --------------------------------------------------------------- // --- App State ---------------------------------------------------------------
// The min, max, median and average of latencies, a list of numbers, or all
// null when it is empty. The median of an even count is the mean of the
// middle two; it and the average are rounded.
function latencyStats(latencies) {
if (latencies.length === 0)
return { min: null, max: null, med: null, avg: null };
const sorted = [...latencies].sort((a, b) => a - b);
const mid = Math.floor(sorted.length / 2);
return {
min: sorted[0],
max: sorted[sorted.length - 1],
med:
sorted.length % 2
? sorted[mid]
: Math.round((sorted[mid - 1] + sorted[mid]) / 2),
avg: Math.round(
latencies.reduce((a, b) => a + b, 0) / latencies.length,
),
};
}
export class HostState { export class HostState {
constructor(host, pinned = false) { constructor(host, pinned = false) {
this.name = host.name; this.name = host.name;
@@ -230,38 +288,16 @@ export class HostState {
this._trim(); this._trim();
} }
averageLatency() { // The min, max, median and average latency of the checks in the history
const valid = this.history.filter((p) => p.latency !== null); // that got an answer.
if (valid.length === 0) return null; historyStats() {
return Math.round( return latencyStats(
valid.reduce((s, p) => s + p.latency, 0) / valid.length, this.history
.filter((p) => p.latency !== null)
.map((p) => p.latency),
); );
} }
minLatency() {
const valid = this.history.filter((p) => p.latency !== null);
if (valid.length === 0) return null;
return Math.min(...valid.map((p) => p.latency));
}
maxLatency() {
const valid = this.history.filter((p) => p.latency !== null);
if (valid.length === 0) return null;
return Math.max(...valid.map((p) => p.latency));
}
medianLatency() {
const sorted = this.history
.filter((p) => p.latency !== null)
.map((p) => p.latency)
.sort((a, b) => a - b);
if (sorted.length === 0) return null;
const mid = Math.floor(sorted.length / 2);
return sorted.length % 2
? sorted[mid]
: Math.round((sorted[mid - 1] + sorted[mid]) / 2);
}
_trim() { _trim() {
while (this.history.length > CONFIG.maxHistoryPoints) while (this.history.length > CONFIG.maxHistoryPoints)
this.history.shift(); this.history.shift();
@@ -288,33 +324,13 @@ export class AppState {
/** WAN-only stats from latest sample (excludes local) */ /** WAN-only stats from latest sample (excludes local) */
wanStats() { wanStats() {
const reachable = this.wan.filter((h) => h.lastLatency !== null); const latencies = this.wan
const latencies = reachable.map((h) => h.lastLatency); .filter((h) => h.lastLatency !== null)
const total = this.wan.length; .map((h) => h.lastLatency);
if (latencies.length === 0)
return {
reachable: 0,
total,
min: null,
max: null,
med: null,
avg: null,
};
const sorted = [...latencies].sort((a, b) => a - b);
const mid = Math.floor(sorted.length / 2);
const med =
sorted.length % 2
? sorted[mid]
: Math.round((sorted[mid - 1] + sorted[mid]) / 2);
return { return {
reachable: latencies.length, reachable: latencies.length,
total, total: this.wan.length,
min: Math.min(...latencies), ...latencyStats(latencies),
max: Math.max(...latencies),
med,
avg: Math.round(
latencies.reduce((a, b) => a + b, 0) / latencies.length,
),
}; };
} }
@@ -340,12 +356,16 @@ export class AppState {
const timeouts = this.wan.filter( const timeouts = this.wan.filter(
(h) => h.status === "error" || h.status === "offline", (h) => h.status === "error" || h.status === "offline",
).length; ).length;
if (timeouts > 10 && reachable <= 4) return "offline"; if (
if (timeouts > 4) return "degraded"; timeouts > CONFIG.offlineTimeouts &&
reachable <= CONFIG.offlineReachable
)
return "offline";
if (timeouts > CONFIG.degradedTimeouts) return "degraded";
const slow = this.wan.filter( const slow = this.wan.filter(
(h) => h.lastLatency !== null && h.lastLatency > 1000, (h) => h.lastLatency !== null && h.lastLatency > CONFIG.slowLatency,
).length; ).length;
if (slow > 3) return "slow"; if (slow > CONFIG.slowHosts) return "slow";
return "healthy"; return "healthy";
} }
@@ -557,21 +577,13 @@ export async function measureLatency(url, signal) {
export function latencyHex(latency) { export function latencyHex(latency) {
if (latency === null) return "#6b7280"; if (latency === null) return "#6b7280";
if (latency < 50) return "#22c55e"; return CONFIG.latencyColors.find((c) => latency < c.below).hex;
if (latency < 100) return "#84cc16";
if (latency < 200) return "#eab308";
if (latency < 500) return "#f97316";
return "#ef4444";
} }
export function latencyClass(latency, status) { export function latencyClass(latency, status) {
if (status === "offline" || status === "error" || latency === null) if (status === "offline" || status === "error" || latency === null)
return "text-gray-500"; return "text-gray-500";
if (latency < 50) return "text-green-500"; return CONFIG.latencyColors.find((c) => latency < c.below).className;
if (latency < 100) return "text-lime-500";
if (latency < 200) return "text-yellow-500";
if (latency < 500) return "text-orange-500";
return "text-red-500";
} }
// --- Sparkline Renderer ------------------------------------------------------ // --- Sparkline Renderer ------------------------------------------------------
@@ -845,7 +857,7 @@ function buildUI(state) {
</div> </div>
<footer class="mt-8 text-center text-gray-600 text-xs"> <footer class="mt-8 text-center text-gray-600 text-xs">
<p>Latency measured via GET requests | IPv4 only | CORS restrictions may affect some measurements</p> <p>Latency measured via GET requests | CORS restrictions may affect some measurements</p>
<p class="mt-2"> <p class="mt-2">
<span class="inline-block w-3 h-3 rounded-full bg-green-500 mr-1 align-middle"></span>&lt;50ms <span class="inline-block w-3 h-3 rounded-full bg-green-500 mr-1 align-middle"></span>&lt;50ms
<span class="inline-block w-3 h-3 rounded-full bg-lime-500 mr-1 ml-3 align-middle"></span>&lt;100ms <span class="inline-block w-3 h-3 rounded-full bg-lime-500 mr-1 ml-3 align-middle"></span>&lt;100ms
@@ -912,10 +924,7 @@ function updateHostRow(host, index) {
latencyEl.innerHTML = `<span class="text-gray-500">---</span>`; latencyEl.innerHTML = `<span class="text-gray-500">---</span>`;
} }
const avg = host.averageLatency(); const { min, med, avg, max } = host.historyStats();
const med = host.medianLatency();
const min = host.minLatency();
const max = host.maxLatency();
if (host.status === "online" && avg !== null) { if (host.status === "online" && avg !== null) {
statusEl.innerHTML = statusStatsHTML([ statusEl.innerHTML = statusStatsHTML([
["min", min], ["min", min],
@@ -1182,8 +1191,9 @@ export async function tick(state, signal, onOffline) {
// rows whose check ended before the resume still read "paused" // rows whose check ended before the resume still read "paused"
state.allHosts.forEach((host, i) => updateHostRow(host, i)); state.allHosts.forEach((host, i) => updateHostRow(host, i));
// Sort after the first real check, then every 10 ticks thereafter // Sort after the first real check, then every CONFIG.roundsPerSort
if (state.tickCount === 2 || state.tickCount % 10 === 1) { // ticks thereafter
if (state.tickCount === 2 || state.tickCount % CONFIG.roundsPerSort === 1) {
sortAndRebuildWAN(state); sortAndRebuildWAN(state);
} }
@@ -1206,9 +1216,10 @@ export async function tick(state, signal, onOffline) {
// --- Recovery Probe ---------------------------------------------------------- // --- Recovery Probe ----------------------------------------------------------
// When offline, check 4 random WAN hosts every 500ms, giving up the checks // When offline, check CONFIG.recoveryProbeHosts random WAN hosts every
// started 500ms before, so at most 4 are ever waiting. As soon as one // CONFIG.recoveryProbeInterval ms, giving up the checks started one interval
// answers, stop probing and start a new round at once. // before, so at most that many are ever waiting. As soon as one answers,
// stop probing and start a new round at once.
function startRecoveryProbe(state, startRounds) { function startRecoveryProbe(state, startRounds) {
if (state._recoveryProbeId) return; // already running if (state._recoveryProbeId) return; // already running
const candidates = [...state.wan]; const candidates = [...state.wan];
@@ -1216,7 +1227,7 @@ function startRecoveryProbe(state, startRounds) {
const j = Math.floor(Math.random() * (i + 1)); const j = Math.floor(Math.random() * (i + 1));
[candidates[i], candidates[j]] = [candidates[j], candidates[i]]; [candidates[i], candidates[j]] = [candidates[j], candidates[i]];
} }
const canaries = candidates.slice(0, 4); const canaries = candidates.slice(0, CONFIG.recoveryProbeHosts);
log.notice( log.notice(
`Recovery probe started (${canaries.map((h) => h.name).join(", ")})`, `Recovery probe started (${canaries.map((h) => h.name).join(", ")})`,
); );
@@ -1233,7 +1244,7 @@ function startRecoveryProbe(state, startRounds) {
startRounds(); startRounds();
}); });
} }
}, 500); }, CONFIG.recoveryProbeInterval);
} }
function stopRecoveryProbe(state) { function stopRecoveryProbe(state) {
@@ -1497,7 +1508,7 @@ async function init() {
}); });
window.addEventListener("resize", () => handleResize(state)); window.addEventListener("resize", () => handleResize(state));
setTimeout(() => handleResize(state), 100); setTimeout(() => handleResize(state), CONFIG.resizeDelay);
} }
// Bootstrap only when loaded as the page: a real DOM containing the #app // Bootstrap only when loaded as the page: a real DOM containing the #app
+26 -10
View File
@@ -347,19 +347,35 @@ for (const { history, latencies, statistics } of [
}, },
]) { ]) {
test(`a target's min, max, average and median latency over ${history}`, () => { test(`a target's min, max, average and median latency over ${history}`, () => {
const host = hostAfter(latencies); const { min, max, avg, med } = hostAfter(latencies).historyStats();
assert.deepEqual( assert.deepEqual({ min, max, average: avg, median: med }, statistics);
{
min: host.minLatency(),
max: host.maxLatency(),
average: host.averageLatency(),
median: host.medianLatency(),
},
statistics,
);
}); });
} }
// The summary's figures come from each WAN target's last check, by the same
// rules as a target's own: here four answered, one was found unreachable
// and the rest have not been checked yet. The median, 22.5, and the
// average, 21.25, are rounded.
test("the summary's min, max, median and average latency over the WAN targets' last checks", () => {
const state = new AppState([]);
[30, 10, null, 25, 20].forEach((latency, i) =>
state.wan[i].pushSample(
Date.now(),
latency === null
? { latency: null, error: "unreachable" }
: { latency, error: null },
),
);
assert.deepEqual(state.wanStats(), {
reachable: 4,
total: state.wan.length,
min: 10,
max: 30,
med: 23,
avg: 21,
});
});
// An app state in which, of the WAN targets, the first timedOut timed out, // An app state in which, of the WAN targets, the first timedOut timed out,
// the next unreachable were found unreachable, the next answered answered // the next unreachable were found unreachable, the next answered answered
// after latency ms, and the rest have not been checked yet. // after latency ms, and the rest have not been checked yet.
+8 -7
View File
@@ -77,7 +77,7 @@ the internet, so the app's latency probes cannot reach anything real. The
harness answers them itself from a fixed delay table, with a deterministic harness answers them itself from a fixed delay table, with a deterministic
fraction failed outright, so the rows render a realistic spread of one-, two- fraction failed outright, so the rows render a realistic spread of one-, two-
and three-digit latencies plus some unreachable rows. That spread is what the and three-digit latencies plus some unreachable rows. That spread is what the
layout has to survive; 24 identical `---` placeholders would not exercise it. layout has to survive; a `---` placeholder in every row would not exercise it.
## What this cannot verify ## What this cannot verify
@@ -104,10 +104,11 @@ Everything else this issue was actually about — does the layout reflow, does
anything overflow, is content clipped, are the controls big enough — is a anything overflow, is content clipped, are the controls big enough — is a
function of viewport width and CSS, and is covered above. function of viewport width and CSS, and is covered above.
## Relation to the unit test framework (#21) ## Relation to the unit tests
Complementary layers, not two stacks. `vitest` (#21) will exercise module-level Complementary layers, not two stacks. The unit tests in `test/unit/`, which
logic in-process with no browser. This harness exercises rendered layout in a `make test` runs with Node's built-in test runner, exercise the functions
real engine and is the only thing here that can see a media query. Neither `src/main.js` exports in-process with no browser. This harness exercises
replaces the other; assertions about computed styles and element geometry belong rendered layout in a real engine and is the only thing here that can see a media
here, assertions about functions belong in `vitest`. query. Neither replaces the other; assertions about computed styles and element
geometry belong here, assertions about functions belong in `test/unit/`.