4 Commits
Author SHA1 Message Date
clawbot 3848fbc6f4 Debug log adds each line instead of drawing the whole panel (closes #131)
check / check (push) Canceled after 0s
With the debug log open, every line written drew the panel again with
up to 1000 lines, and a check that ends while the page draws is timed
only when the drawing is done, so latencies measured with the panel open
read too high. Each line is now appended to the open panel and its first
line dropped once it holds more than maxLogEntries; the whole panel is
drawn only when it is opened. Scrolling to the newest line reads the
panel's height, which lays out all its lines, so it runs once a frame,
with at most one frame request waiting.

Model: opus-5-5
2026-10-14 11:03:40 +00:00
clawbot cbcde098dc Unset TRUSTED_PROXIES trusts the RFC 1918 ranges (closes #119)
check / check (push) Canceled after 0s
With TRUSTED_PROXIES unset, bin/entrypoint.sh gives nginx one
set_real_ip_from line for each of 10.0.0.0/8, 172.16.0.0/12 and
192.168.0.0/16, as sneak ruled on 2026-09-22 for every app. A list
replaces them, and set and empty trusts no proxy; each entry is checked
as before. The backend inside the container still trusts only
127.0.0.1/32. README.md, configs/nginx.conf and the comment in
script/frontend-viewport-test say the new default.

Model: opus-5-5
2026-10-14 13:01:34 +02:00
clawbot 73b39e485b Reports are taken only as application/json (closes #126)
check / check (push) Canceled after 0s
A page on another origin can POST text/plain or a form without a CORS
preflight, so its visitors' browsers could store reports although the
server sends no CORS headers. decodeJSON now refuses a request whose
Content-Type is not application/json (parameters such as charset
allowed), and the report handler answers it 415 with the usual error
body, storing nothing. A JSON POST waits for the preflight, which only
an origin in CORS_ALLOWED_ORIGINS passes; the page already sends
application/json.

Model: opus-5-5
2026-10-14 12:43:25 +02:00
clawbot 03bc282871 Interval change keeps unreported samples for the reporter (closes #124)
check / check (push) Canceled after 0s
The interval menu cleared every host's history, and the reporter builds
each report from that history, so the samples recorded since the last
delivered report were never sent. The menu now resets each host with
HostState.reset, which moves the history's entries to clearedHistory,
keeping the last 100 it has cleared, and buildReport reads both. A host
still pending after the reset is sent with the status of its last check.
The per-host marks are unchanged, so a sample a delivered report had is
not sent again, and the page's history still starts over. Reporter is
exported for the new unit tests.

Model: opus-5-5
2026-10-14 12:23:13 +02:00
11 changed files with 432 additions and 72 deletions
+30 -20
View File
@@ -149,9 +149,11 @@ same-origin path `/api/v1/reports`: a random per-browser `clientId` kept in
`localStorage`, `geo` sent as null, and each host's unreported, non-paused `localStorage`, `geo` sent as null, and each host's unreported, non-paused
samples (timestamp, latency, error). A per-host high-water mark makes every samples (timestamp, latency, error). A per-host high-water mark makes every
report a delta, so only new samples are sent; the mark advances only on a report a delta, so only new samples are sent; the mark advances only on a
delivered report, and while paused nothing is sent. Delivery failure is quiet — delivered report, and while paused nothing is sent. Changing the interval clears
one debug-log line per outage, retried at the next interval, never blocking each host's history on the page; the samples in it not yet reported still go in
probing. The report-building step is a pure function of host state. the next report. Delivery failure is quiet — one debug-log line per outage,
retried at the next interval, never blocking probing. The report-building step
is a pure function of host state.
### Backend ### Backend
@@ -229,9 +231,15 @@ Files are created with `O_EXCL` to prevent overwrites.
#### Report limits #### Report limits
`POST /api/v1/reports` takes reports from anyone who can reach it, without `POST /api/v1/reports` takes reports from anyone who can reach it, without
credentials, so it is bounded instead. Both refusals below answer with the same credentials, so it is bounded instead. Each refusal below answers with the same
`{"status":"error"}` body as any other error. `{"status":"error"}` body as any other error.
- **JSON only.** A report must be sent with `Content-Type: application/json`,
parameters such as `charset` allowed, as the page sends it; any other is
refused with 415, and nothing of it is stored. A page on another origin can
send a `text/plain` POST, or a form, without a CORS preflight; refusing them
keeps any site from making its visitors' browsers store reports (see
[CORS](#cors)).
- **Rate limit.** Each client address, resolved through `TRUSTED_PROXIES`, may - **Rate limit.** Each client address, resolved through `TRUSTED_PROXIES`, may
send `REPORTS_PER_MINUTE` reports a minute; past that it gets 429 with send `REPORTS_PER_MINUTE` reports a minute; past that it gets 429 with
`Retry-After: 60`. The minute slides: reports from the minute before still `Retry-After: 60`. The minute slides: reports from the minute before still
@@ -404,9 +412,10 @@ address nginx passes on and no other. The image:
- Listens on port 8080 by default (override with `PORT` env var) - Listens on port 8080 by default (override with `PORT` env var)
- Takes the client address from `X-Forwarded-For` only on requests from the - Takes the client address from `X-Forwarded-For` only on requests from the
reverse proxies named in `TRUSTED_PROXIES`, and by default from none. reverse proxies named in `TRUSTED_PROXIES`, by default from any address in
`bin/entrypoint.sh` checks each entry with `netwatch-server check-cidr`, which `10.0.0.0/8`, `172.16.0.0/12` or `192.168.0.0/16`. `bin/entrypoint.sh` checks
parses it as the backend parses its own `TRUSTED_PROXIES` each entry with `netwatch-server check-cidr`, which parses it as the backend
parses its own `TRUSTED_PROXIES`
- Sends access logs to stdout - Sends access logs to stdout
- Caches static assets with immutable headers - Caches static assets with immutable headers
- Sends the security headers `REPO_POLICIES.md` requires on every response, as - Sends the security headers `REPO_POLICIES.md` requires on every response, as
@@ -425,9 +434,9 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
- **Port:** container port `8080`. - **Port:** container port `8080`.
- **Volume:** container path `/data`; the reports are kept in `/data/reports`. - **Volume:** container path `/data`; the reports are kept in `/data/reports`.
- **Environment variables:** none is required. An empty one counts as unset, and - **Environment variables:** none is required. An empty one counts as unset,
one set to a value netwatch cannot use stops the container at start, with the except `TRUSTED_PROXIES`, and one set to a value netwatch cannot use stops the
reason in its log. container at start, with the reason in its log.
- `PORT`, default `8080`: the container port, from 1 to 65535. `8081` cannot - `PORT`, default `8080`: the container port, from 1 to 65535. `8081` cannot
be used: the backend listens on it inside the container be used: the backend listens on it inside the container
- `REPORTS_PER_MINUTE`, default `60`: reports each client address may send a - `REPORTS_PER_MINUTE`, default `60`: reports each client address may send a
@@ -441,16 +450,17 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
in: `/data` or a path below it, with no `.` or `..` part and no extra `/`. in: `/data` or a path below it, with no `.` or `..` part and no extra `/`.
The container also stops if the path goes through a symbolic link that The container also stops if the path goes through a symbolic link that
leads out of `/data` or is written as a full path leads out of `/data` or is written as a full path
- `TRUSTED_PROXIES`, default empty: set it to the address the reverse proxy - `TRUSTED_PROXIES`, default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`: the
in front of the container connects from, as an IP address or CIDR; several addresses the reverse proxy in front of the container connects from, as IP
are separated by commas. nginx takes the client address from addresses or CIDRs separated by commas. nginx takes the client address
`X-Forwarded-For` only on a request from one of them, and the rate limit from `X-Forwarded-For` only on a request from one of them, and the rate
counts that address. Unset, `X-Forwarded-For` is ignored and every client limit counts that address. A list replaces the default. Set and empty,
behind the proxy shares the proxy's one allowance of `REPORTS_PER_MINUTE`. `X-Forwarded-For` is ignored and every client behind the proxy shares the
Name only addresses nothing but the proxy connects from: any client that proxy's one allowance of `REPORTS_PER_MINUTE`. Any client that connects
connects from one can write its own `X-Forwarded-For`, and through a port from one of these addresses can write its own `X-Forwarded-For`, and
Docker publishes, every client may connect from the Docker network's through a port Docker publishes, every client may connect from the Docker
gateway, such as `172.17.0.1`. network's gateway, such as `172.17.0.1`, which the default includes; where
that matters, name only the addresses nothing but the proxy connects from.
- `METRICS_USERNAME` and `METRICS_PASSWORD`, default empty: with both set, - `METRICS_USERNAME` and `METRICS_PASSWORD`, default empty: with both set,
the backend records Prometheus metrics of its requests and serves them at the backend records Prometheus metrics of its requests and serves them at
`/metrics` on the container port, to requests with this user name and `/metrics` on the container port, to requests with this user name and
+38
View File
@@ -21,6 +21,44 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of
# Completed Steps # Completed Steps
- 2026-10-14: opening the debug log no longer raises the measured latencies
([#131](https://git.eeqj.de/sneak/netwatch/issues/131)). With the panel open,
each line written drew the panel again with every line before it, up to 1000,
and a check that ends while the page draws is timed only when the drawing is
done. Each line is now added at the end of the open panel, its first line is
dropped once it holds more than 1000, and it scrolls to its newest line once a
frame rather than once a line; the whole panel is drawn only when it is
opened. A unit test writes more lines than the panel keeps and checks that the
lines already shown are left as they are
- 2026-10-14: with `TRUSTED_PROXIES` unset, the container's nginx trusts the
three private ranges of RFC 1918, `10.0.0.0/8`, `172.16.0.0/12` and
`192.168.0.0/16` ([#119](https://git.eeqj.de/sneak/netwatch/issues/119)), as
sneak ruled on 2026-09-22 for every app; it trusted no proxy before. A list
replaces them, and set and empty it trusts none. The backend inside the
container still trusts only `127.0.0.1/32`
- 2026-10-14: `POST /api/v1/reports` takes only a report sent with
`Content-Type: application/json`, parameters such as `charset` allowed, and
refuses any other with 415 and the usual error body, storing nothing
([#126](https://git.eeqj.de/sneak/netwatch/issues/126)). A page on another
origin can send a `text/plain` POST without a CORS preflight, so any site
could make its visitors' browsers store reports; a JSON POST waits for the
preflight, which only an origin in `CORS_ALLOWED_ORIGINS` passes. The page
already sent `application/json`. A test posts a report as `text/plain`, as
`application/x-www-form-urlencoded`, as `multipart/form-data`, with no
`Content-Type`, and as `application/json` with and without a charset, and
checks which are stored
- 2026-10-14: changing the interval no longer drops the samples not yet reported
to the backend ([#124](https://git.eeqj.de/sneak/netwatch/issues/124)). The
interval menu resets each host with the new `HostState.reset`, which moves the
history's entries to `clearedHistory`, keeping the last 100 of all it has
cleared, and `buildReport` reads both, so the next report has the samples
recorded before the change that no report has delivered. Until its next check,
such a host is sent with the status of its last check, not `pending`. The
page's history still starts over, and the per-host marks still leave out of a
report the samples a delivered one had. Unit tests reset a host as the
interval menu does and find its unreported samples in the next report, and
only the last 100 after more were cleared; `web/src/main.js` now exports
`Reporter` for them
- 2026-10-14: the repo has the layout `REPO_POLICIES.md` gives - 2026-10-14: the repo has the layout `REPO_POLICIES.md` gives
([#30](https://git.eeqj.de/sneak/netwatch/issues/30)). The Go module is at the ([#30](https://git.eeqj.de/sneak/netwatch/issues/30)). The Go module is at the
root (`go.mod`, `go.sum`, `.golangci.yml`, `cmd/netwatch-server/`, root (`go.mod`, `go.sum`, `.golangci.yml`, `cmd/netwatch-server/`,
+6 -4
View File
@@ -34,9 +34,11 @@ fi
# TRUSTED_PROXIES names the reverse proxies in front of the container, # TRUSTED_PROXIES names the reverse proxies in front of the container,
# as IP addresses or CIDRs separated by commas. nginx takes the client # as IP addresses or CIDRs separated by commas. nginx takes the client
# address from X-Forwarded-For only on a request from one of them, so # address from X-Forwarded-For only on a request from one of them.
# unset or empty, it trusts no one. configs/nginx.conf includes the file # Unset, they are the three private ranges of RFC 1918; set and empty,
# written here, one set_real_ip_from line per entry. # nginx trusts no one, which is why the default below follows - and not
# :-. configs/nginx.conf includes the file written here, one
# set_real_ip_from line per entry.
# #
# nginx looks up an entry it cannot read as an address as a hostname, # nginx looks up an entry it cannot read as an address as a hostname,
# and trusts what it finds (1.2.3 is found as 1.2.0.3). So each entry # and trusts what it finds (1.2.3 is found as 1.2.0.3). So each entry
@@ -45,7 +47,7 @@ fi
# TRUSTED_PROXIES. Its error, naming the CIDR, is dropped for the one # TRUSTED_PROXIES. Its error, naming the CIDR, is dropped for the one
# below, naming the entry as written. set -f keeps a * in an entry from # below, naming the entry as written. set -f keeps a * in an entry from
# becoming a list of file names. # becoming a list of file names.
TRUSTED_PROXIES="${TRUSTED_PROXIES:-}" TRUSTED_PROXIES="${TRUSTED_PROXIES-10.0.0.0/8,172.16.0.0/12,192.168.0.0/16}"
set -f set -f
for proxy in $(printf '%s' "$TRUSTED_PROXIES" | tr ',' ' '); do for proxy in $(printf '%s' "$TRUSTED_PROXIES" | tr ',' ' '); do
case "$proxy" in case "$proxy" in
+4 -3
View File
@@ -18,9 +18,10 @@ server {
# The client address comes from X-Forwarded-For only on a request # The client address comes from X-Forwarded-For only on a request
# from the reverse proxies in TRUSTED_PROXIES: bin/entrypoint.sh # from the reverse proxies in TRUSTED_PROXIES: bin/entrypoint.sh
# writes one set_real_ip_from line for each into this file, and # writes one set_real_ip_from line for each into this file. Unset,
# leaves it empty when TRUSTED_PROXIES is unset, so that by default # TRUSTED_PROXIES is the three private ranges of RFC 1918; set and
# the client address is the one each request comes from. # empty, the file is empty and the client address is the one each
# request comes from.
include /etc/nginx/trusted-proxies.conf; include /etc/nginx/trusted-proxies.conf;
real_ip_header X-Forwarded-For; real_ip_header X-Forwarded-For;
real_ip_recursive on; real_ip_recursive on;
+21 -4
View File
@@ -5,7 +5,9 @@ package handlers
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"errors"
"log/slog" "log/slog"
"mime"
"net/http" "net/http"
"sneak.berlin/go/netwatch/internal/globals" "sneak.berlin/go/netwatch/internal/globals"
@@ -18,6 +20,10 @@ import (
const jsonContentType = "application/json; charset=utf-8" const jsonContentType = "application/json; charset=utf-8"
// errNotJSON is what decodeJSON returns for a request whose
// Content-Type is not application/json.
var errNotJSON = errors.New("content type is not application/json")
// reportAppender is the subset of the report buffer the handlers // reportAppender is the subset of the report buffer the handlers
// depend on. Defining it here keeps the storage failure path // depend on. Defining it here keeps the storage failure path
// exercisable with a stub in tests. // exercisable with a stub in tests.
@@ -80,14 +86,25 @@ func (s *Handlers) respondJSON(
} }
} }
// decodeJSON decodes the request body into v. The body is // decodeJSON decodes the request body into v. It returns errNotJSON,
// expected to already be bounded by the body-size middleware, so // reading nothing, unless the request's Content-Type is
// a caller can distinguish an over-limit body from malformed // application/json, with or without parameters such as charset. The
// JSON by testing the returned error for *http.MaxBytesError. // body is expected to already be bounded by the body-size middleware,
// so a caller can distinguish an over-limit body from malformed JSON
// by testing the returned error for *http.MaxBytesError.
func (s *Handlers) decodeJSON( func (s *Handlers) decodeJSON(
_ http.ResponseWriter, _ http.ResponseWriter,
r *http.Request, r *http.Request,
v any, v any,
) error { ) error {
// A page on another origin can POST text/plain or a form without
// a CORS preflight, so its visitors' browsers would send it
// whatever the CORS settings. A JSON POST waits for the
// preflight, so only JSON is taken.
mediaType, _, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
if err != nil || mediaType != "application/json" {
return errNotJSON
}
return json.NewDecoder(r.Body).Decode(v) return json.NewDecoder(r.Body).Decode(v)
} }
+9 -3
View File
@@ -30,7 +30,7 @@ type report struct {
} }
// HandleReport returns a handler that accepts telemetry // HandleReport returns a handler that accepts telemetry
// reports from NetWatch clients. // reports from NetWatch clients, sent as application/json.
func (s *Handlers) HandleReport() http.HandlerFunc { func (s *Handlers) HandleReport() http.HandlerFunc {
type response struct { type response struct {
Status string `json:"status"` Status string `json:"status"`
@@ -66,9 +66,15 @@ func (s *Handlers) HandleReport() http.HandlerFunc {
} }
// decodeErrorStatus logs a report decode failure and returns the // decodeErrorStatus logs a report decode failure and returns the
// status to send: 413 when the body exceeded the size limit, // status to send: 415 when the report was not sent as JSON, 413 when
// otherwise 400 for malformed JSON. // the body exceeded the size limit, otherwise 400 for malformed JSON.
func (s *Handlers) decodeErrorStatus(err error) int { func (s *Handlers) decodeErrorStatus(err error) int {
if errors.Is(err, errNotJSON) {
s.log.Warn("report refused: Content-Type is not application/json")
return http.StatusUnsupportedMediaType
}
var tooLarge *http.MaxBytesError var tooLarge *http.MaxBytesError
if errors.As(err, &tooLarge) { if errors.As(err, &tooLarge) {
s.log.Warn("report body too large", "limit_bytes", tooLarge.Limit) s.log.Warn("report body too large", "limit_bytes", tooLarge.Limit)
+103
View File
@@ -27,6 +27,17 @@ type stubAppender struct {
func (s stubAppender) Append(any) error { return s.err } func (s stubAppender) Append(any) error { return s.err }
// recordingAppender keeps every report the handler stores.
type recordingAppender struct {
reports []any
}
func (a *recordingAppender) Append(v any) error {
a.reports = append(a.reports, v)
return nil
}
func newTestHandlers(buf stubAppender, out io.Writer) *handlers.Handlers { func newTestHandlers(buf stubAppender, out io.Writer) *handlers.Handlers {
return handlers.NewForTest(buf, slog.New(slog.NewJSONHandler(out, nil))) return handlers.NewForTest(buf, slog.New(slog.NewJSONHandler(out, nil)))
} }
@@ -83,6 +94,7 @@ func TestHandleReportAcceptsValidReports(t *testing.T) {
http.MethodPost, "/api/v1/reports", http.MethodPost, "/api/v1/reports",
strings.NewReader(tt.body), strings.NewReader(tt.body),
) )
req.Header.Set("Content-Type", "application/json")
h.HandleReport().ServeHTTP(rec, req) h.HandleReport().ServeHTTP(rec, req)
@@ -103,6 +115,89 @@ func TestHandleReportAcceptsValidReports(t *testing.T) {
} }
} }
// TestHandleReportStoresOnlyJSON checks that a report is stored only
// when sent as application/json, as the page sends it. One sent as
// text/plain or as a form, which a page on another origin can send
// without a CORS preflight, is refused with 415.
func TestHandleReportStoresOnlyJSON(t *testing.T) {
t.Parallel()
tests := []struct {
name string
contentType string
wantStatus int
wantStored int
}{
{
name: "text/plain",
contentType: "text/plain;charset=UTF-8",
wantStatus: http.StatusUnsupportedMediaType,
wantStored: 0,
},
{
name: "a urlencoded form",
contentType: "application/x-www-form-urlencoded",
wantStatus: http.StatusUnsupportedMediaType,
wantStored: 0,
},
{
name: "a multipart form",
contentType: "multipart/form-data; boundary=x",
wantStatus: http.StatusUnsupportedMediaType,
wantStored: 0,
},
{
name: "no Content-Type",
contentType: "",
wantStatus: http.StatusUnsupportedMediaType,
wantStored: 0,
},
{
name: "application/json",
contentType: "application/json",
wantStatus: http.StatusOK,
wantStored: 1,
},
{
name: "application/json with a charset",
contentType: "application/json;charset=UTF-8",
wantStatus: http.StatusOK,
wantStored: 1,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
var stored recordingAppender
h := handlers.NewForTest(&stored, slog.New(slog.DiscardHandler))
rec := httptest.NewRecorder()
req := httptest.NewRequestWithContext(t.Context(),
http.MethodPost, "/api/v1/reports",
strings.NewReader(`{"clientId":"c1","hosts":[]}`),
)
if tt.contentType != "" {
req.Header.Set("Content-Type", tt.contentType)
}
h.HandleReport().ServeHTTP(rec, req)
if rec.Code != tt.wantStatus {
t.Errorf("status = %d, want %d", rec.Code, tt.wantStatus)
}
if len(stored.reports) != tt.wantStored {
t.Errorf("reports stored = %d, want %d",
len(stored.reports), tt.wantStored)
}
})
}
}
func TestHandleReportStorageFailureIsNon2xx(t *testing.T) { func TestHandleReportStorageFailureIsNon2xx(t *testing.T) {
t.Parallel() t.Parallel()
@@ -113,6 +208,7 @@ func TestHandleReportStorageFailureIsNon2xx(t *testing.T) {
http.MethodPost, "/api/v1/reports", http.MethodPost, "/api/v1/reports",
strings.NewReader(`{"clientId":"c1","hosts":[]}`), strings.NewReader(`{"clientId":"c1","hosts":[]}`),
) )
req.Header.Set("Content-Type", "application/json")
h.HandleReport().ServeHTTP(rec, req) h.HandleReport().ServeHTTP(rec, req)
@@ -138,6 +234,7 @@ func TestHandleReportFullIs507(t *testing.T) {
http.MethodPost, "/api/v1/reports", http.MethodPost, "/api/v1/reports",
strings.NewReader(`{"clientId":"c1","hosts":[]}`), strings.NewReader(`{"clientId":"c1","hosts":[]}`),
) )
req.Header.Set("Content-Type", "application/json")
h.HandleReport().ServeHTTP(rec, req) h.HandleReport().ServeHTTP(rec, req)
@@ -161,6 +258,7 @@ func TestHandleReportMalformedJSONIs400(t *testing.T) {
http.MethodPost, "/api/v1/reports", http.MethodPost, "/api/v1/reports",
strings.NewReader(`{not json`), strings.NewReader(`{not json`),
) )
req.Header.Set("Content-Type", "application/json")
h.HandleReport().ServeHTTP(rec, req) h.HandleReport().ServeHTTP(rec, req)
@@ -185,6 +283,8 @@ func TestHandleReportOversizeIs413(t *testing.T) {
http.MethodPost, "/api/v1/reports", http.MethodPost, "/api/v1/reports",
strings.NewReader(`{"clientId":"`+strings.Repeat("x", 200)+`"}`), strings.NewReader(`{"clientId":"`+strings.Repeat("x", 200)+`"}`),
) )
req.Header.Set("Content-Type", "application/json")
// No declared length, so only the middleware's read cap can // No declared length, so only the middleware's read cap can
// stop this body. // stop this body.
req.ContentLength = -1 req.ContentLength = -1
@@ -213,6 +313,7 @@ func TestHandleReportDoesNotLogRawGeo(t *testing.T) {
`{"clientId":"c1","geo":{"raw":"`+sentinel+`"},"hosts":[]}`, `{"clientId":"c1","geo":{"raw":"`+sentinel+`"},"hosts":[]}`,
), ),
) )
req.Header.Set("Content-Type", "application/json")
h.HandleReport().ServeHTTP(rec, req) h.HandleReport().ServeHTTP(rec, req)
@@ -245,6 +346,7 @@ func TestHandleReportLogsClientIDCutToBound(t *testing.T) {
`{"clientId":"`+long+`","timestamp":"`+long+`","hosts":[]}`, `{"clientId":"`+long+`","timestamp":"`+long+`","hosts":[]}`,
), ),
) )
req.Header.Set("Content-Type", "application/json")
h.HandleReport().ServeHTTP(rec, req) h.HandleReport().ServeHTTP(rec, req)
@@ -284,6 +386,7 @@ func TestHandleReportDecodeErrorLogIsBounded(t *testing.T) {
http.MethodPost, "/api/v1/reports", http.MethodPost, "/api/v1/reports",
strings.NewReader(`{"hosts":[{"history":[{"t":`+huge+`}]}]}`), strings.NewReader(`{"hosts":[{"history":[{"t":`+huge+`}]}]}`),
) )
req.Header.Set("Content-Type", "application/json")
h.HandleReport().ServeHTTP(rec, req) h.HandleReport().ServeHTTP(rec, req)
+1
View File
@@ -66,6 +66,7 @@ func TestReportsAreRateLimited(t *testing.T) {
http.MethodPost, "/api/v1/reports", http.MethodPost, "/api/v1/reports",
strings.NewReader(`{"clientId":"c1","hosts":[]}`), strings.NewReader(`{"clientId":"c1","hosts":[]}`),
) )
req.Header.Set("Content-Type", "application/json")
srv.ServeHTTP(rec, req) srv.ServeHTTP(rec, req)
return rec.Code return rec.Code
+2 -2
View File
@@ -64,8 +64,8 @@ main() {
# configs/nginx.conf is a template: the image renders it over its own # configs/nginx.conf is a template: the image renders it over its own
# default.conf, with the same port and limit bin/entrypoint.sh uses. # default.conf, with the same port and limit bin/entrypoint.sh uses.
# The empty file it includes trusts no proxy, as bin/entrypoint.sh # The empty file it includes trusts no proxy, as bin/entrypoint.sh
# writes it when TRUSTED_PROXIES is unset. It also includes the # writes it when TRUSTED_PROXIES is set and empty. It also includes
# security headers, so the page runs under the shipped policy. # the security headers, so the page runs under the shipped policy.
docker run -d --rm --name "$SERVER" \ docker run -d --rm --name "$SERVER" \
--network "$NETWORK" --network-alias netwatch \ --network "$NETWORK" --network-alias netwatch \
-e PORT=8080 -e NGINX_ENVSUBST_FILTER='^PORT$' \ -e PORT=8080 -e NGINX_ENVSUBST_FILTER='^PORT$' \
+139 -8
View File
@@ -13,19 +13,22 @@ import {
humanDuration, humanDuration,
latencyClass, latencyClass,
latencyHex, latencyHex,
log,
measureLatency, measureLatency,
Reporter,
tick, tick,
} from "../../web/src/main.js"; } from "../../web/src/main.js";
// There is no page here, so the tests stand in for it. The debug log looks // There is no page here, so the tests stand in for it. The debug log looks
// for its panel by id and finds none. Each element of a host's row that // for its panel by id and finds none, an element the page creates is a
// tick or greyOutUI draws into is a plain object, made the first time a // plain object, and no frame the page asks for ever comes. Each element of
// test looks it up and kept in elements under its selector until the next // a host's row that tick or greyOutUI draws into is a plain object, made
// test starts. As on a page, writing its text replaces its markup; the // the first time a test looks it up and kept in elements under its
// status dot greyOutUI looks for in it is not there. Drawing a sparkline // selector until the next test starts. As on a page, writing its text
// does nothing; it looks for the pixel ratio on window and finds none. In // replaces its markup; the status dot greyOutUI looks for in it is not
// each test, console.error and console.info print nothing and keep what // there. Drawing a sparkline does nothing; it looks for the pixel ratio on
// they are given. // window and finds none. In each test, console.error and console.info
// print nothing and keep what they are given.
const doNothing = () => {}; const doNothing = () => {};
let elements; let elements;
beforeEach((t) => { beforeEach((t) => {
@@ -45,8 +48,11 @@ const canvasContext = {
arc: doNothing, arc: doNothing,
}; };
globalThis.window = {}; globalThis.window = {};
globalThis.requestAnimationFrame = doNothing;
globalThis.cancelAnimationFrame = doNothing;
globalThis.document = { globalThis.document = {
getElementById: () => null, getElementById: () => null,
createElement: () => ({}),
querySelector: (selector) => querySelector: (selector) =>
(elements[selector] ??= { (elements[selector] ??= {
getContext: () => canvasContext, getContext: () => canvasContext,
@@ -378,6 +384,131 @@ test("a check that fails in the first round, which is discarded, writes nothing
assert.deepEqual(consoleLines("info"), []); assert.deepEqual(consoleLines("info"), []);
}); });
// The reports Reporter sent in this test, each as the hosts it sent.
function reportedHosts() {
return fetch.mock.calls.map(
(call) => JSON.parse(call.arguments[1].body).hosts,
);
}
// In the next two tests every report is delivered, and only Target has
// checks to report. target.reset() does to Target what the interval menu
// does to each target.
// The interval changes after the first report. The second report is sent
// before Target's first check since then, and the third after it. A fourth
// would hold no check the first three did not, so it is not sent.
test("checks recorded before an interval change and not yet reported go in the next report, with the status of the last check, and no check is reported twice", async (t) => {
const state = new AppState([
{ name: "Target", url: "https://target.test" },
]);
const target = state.local[0];
const reporter = new Reporter(state, "client", CONFIG.reportInterval);
t.mock.method(globalThis, "fetch", async () => ({ ok: true }));
target.pushSample(1000, { latency: 10, error: null });
await reporter.flush();
target.pushSample(2000, { latency: 20, error: null });
target.pushSample(3000, { latency: null, error: "timeout" });
target.reset();
await reporter.flush();
target.pushSample(4000, { latency: 40, error: null });
await reporter.flush();
await reporter.flush();
const name = "Target";
const url = "https://target.test";
assert.deepEqual(reportedHosts(), [
[
{
name,
url,
status: "online",
history: [{ t: 1000, latency: 10, error: null }],
},
],
[
{
name,
url,
status: "error",
history: [
{ t: 2000, latency: 20, error: null },
{ t: 3000, latency: null, error: "timeout" },
],
},
],
[
{
name,
url,
status: "online",
history: [{ t: 4000, latency: 40, error: null }],
},
],
]);
// The page's history starts over.
assert.deepEqual(target.history, [
{ timestamp: 4000, latency: 40, error: null },
]);
});
// The interval changes twice, after 60 checks each time, before the first
// report. Of the 120 checks the interval menu cleared, the report has the
// last 100 (CONFIG.maxHistoryPoints).
test("of the checks interval changes cleared before they were reported, the next report has the last 100", async (t) => {
const state = new AppState([
{ name: "Target", url: "https://target.test" },
]);
const target = state.local[0];
const reporter = new Reporter(state, "client", CONFIG.reportInterval);
t.mock.method(globalThis, "fetch", async () => ({ ok: true }));
for (let n = 1; n <= 120; n++) {
target.pushSample(n * 1000, { latency: 10, error: null });
if (n % 60 === 0) target.reset();
}
await reporter.flush();
const sent = reportedHosts()[0][0].history;
assert.equal(sent.length, 100);
assert.equal(sent[0].t, 21000);
assert.equal(sent.at(-1).t, 120000);
});
// panel stands in for the open debug log panel, and for the element in it
// that holds its lines, whichever the page looks up by id; the lines are
// kept in order in lines. Drawing the whole panel, as opening it does,
// replaces every line with a new one.
test("with the debug log open, writing more lines than it keeps adds each to the panel and leaves the lines already shown as they are", (t) => {
const lines = [];
const panel = {
classList: { contains: () => false },
children: lines,
get firstElementChild() {
return lines[0];
},
append: (line) => lines.push(line),
removeChild: (line) => lines.splice(lines.indexOf(line), 1),
replaceChildren: (...newLines) =>
lines.splice(0, lines.length, ...newLines),
};
t.mock.method(globalThis.document, "getElementById", () => panel);
const kept = CONFIG.maxLogEntries;
for (let i = 1; i <= kept; i++) log.info(`line ${i}`);
const shown = [...lines];
for (let i = kept + 1; i <= kept + 5; i++) log.info(`line ${i}`);
assert.deepEqual(
lines.map((line) => line.textContent.match(/line \d+$/)?.[0]),
Array.from({ length: kept }, (_, i) => `line ${i + 6}`),
);
// The first five lines shown were dropped, and the rest are the same
// lines, not drawn again.
shown.slice(5).forEach((line, i) => assert.equal(lines[i], line));
});
// The page shows &lt; &gt; &quot; &amp; and &#39; in a row's markup as // The page shows &lt; &gt; &quot; &amp; and &#39; in a row's markup as
// < > " & and '. // < > " & and '.
test(`a target whose name and URL hold < > " & and ' shows those characters in its row`, () => { test(`a target whose name and URL hold < > " & and ' shows those characters in its row`, () => {
+79 -28
View File
@@ -144,13 +144,15 @@ const WAN_HOSTS = [
const debugLog = []; const debugLog = [];
const log = (() => { export const log = (() => {
function append(level, message) { function append(level, message) {
debugLog.push({ timestamp: new Date(), level, message }); const entry = { timestamp: new Date(), level, message };
debugLog.push(entry);
if (debugLog.length > CONFIG.maxLogEntries) if (debugLog.length > CONFIG.maxLogEntries)
debugLog.splice(0, debugLog.length - CONFIG.maxLogEntries); debugLog.splice(0, debugLog.length - CONFIG.maxLogEntries);
const panel = document.getElementById("debug-panel"); const panel = document.getElementById("debug-panel");
if (panel && !panel.classList.contains("hidden")) renderDebugLog(); if (panel && !panel.classList.contains("hidden"))
appendDebugLogLine(entry);
} }
return { return {
error: (msg) => append("error", msg), error: (msg) => append("error", msg),
@@ -257,6 +259,14 @@ function latencyStats(latencies) {
}; };
} }
// The status a check's error gives its host: "error" for a timeout,
// "offline" for any other error, and "online" for none.
function statusFromError(error) {
if (error === "timeout") return "error";
if (error) return "offline";
return "online";
}
export class HostState { export class HostState {
constructor(host, pinned = false) { constructor(host, pinned = false) {
this.name = host.name; this.name = host.name;
@@ -265,6 +275,10 @@ export class HostState {
// error }, or a round skipped while paused, { timestamp, // error }, or a round skipped while paused, { timestamp,
// latency: null, paused: true }. // latency: null, paused: true }.
this.history = []; this.history = [];
// The last 100 (CONFIG.maxHistoryPoints) entries the interval menu
// has cleared from the history, which the reporter may not have
// sent yet.
this.clearedHistory = [];
this.lastLatency = null; this.lastLatency = null;
this.status = "pending"; // 'online' | 'offline' | 'error' | 'pending' this.status = "pending"; // 'online' | 'offline' | 'error' | 'pending'
this.pinned = pinned; this.pinned = pinned;
@@ -280,9 +294,7 @@ export class HostState {
}); });
this._trim(); this._trim();
this.lastLatency = result.latency; this.lastLatency = result.latency;
if (result.error === "timeout") this.status = "error"; this.status = statusFromError(result.error);
else if (result.error) this.status = "offline";
else this.status = "online";
this.consecutiveFailures = result.error this.consecutiveFailures = result.error
? this.consecutiveFailures + 1 ? this.consecutiveFailures + 1
: 0; : 0;
@@ -293,6 +305,18 @@ export class HostState {
this._trim(); this._trim();
} }
// What the interval menu does to each host: the history starts over and
// the host waits for its next check. The history's entries move to
// clearedHistory, so the reporter still sends those it has not.
reset() {
this.clearedHistory = [...this.clearedHistory, ...this.history].slice(
-CONFIG.maxHistoryPoints,
);
this.history = [];
this.lastLatency = null;
this.status = "pending";
}
// The min, max, median and average latency of the checks in the history // The min, max, median and average latency of the checks in the history
// that got an answer. // that got an answer.
historyStats() { historyStats() {
@@ -429,9 +453,12 @@ function getClientId() {
// Build the delta report body the backend decodes, plus the new per-host // Build the delta report body the backend decodes, plus the new per-host
// high-water marks. Pure function of the passed state: `hosts` is an array // high-water marks. Pure function of the passed state: `hosts` is an array
// of { name, url, status, history }, `since` maps a host url to the Unix-ms // of { name, url, status, clearedHistory, history }, `since` maps a host url
// timestamp of the last sample already reported for it, and `now` is a Date. // to the Unix-ms timestamp of the last sample already reported for it, and
// Only non-paused samples newer than the mark are included. Returns null // `now` is a Date. Only non-paused samples newer than the mark are included,
// from clearedHistory as well as history. A host whose status is pending,
// as the interval menu leaves it until its next check, is sent with the
// status of its last check, which is the last sample sent. Returns null
// when no host has an unreported sample. // when no host has an unreported sample.
export function buildReport(hosts, clientId, now, since) { export function buildReport(hosts, clientId, now, since) {
const reportHosts = []; const reportHosts = [];
@@ -440,7 +467,7 @@ export function buildReport(hosts, clientId, now, since) {
const mark = since.get(host.url) ?? 0; const mark = since.get(host.url) ?? 0;
const samples = []; const samples = [];
let high = mark; let high = mark;
for (const p of host.history) { for (const p of [...host.clearedHistory, ...host.history]) {
if (p.paused) continue; if (p.paused) continue;
if (p.timestamp <= mark) continue; if (p.timestamp <= mark) continue;
samples.push({ samples.push({
@@ -454,7 +481,10 @@ export function buildReport(hosts, clientId, now, since) {
reportHosts.push({ reportHosts.push({
name: host.name, name: host.name,
url: host.url, url: host.url,
status: host.status, status:
host.status === "pending"
? statusFromError(samples.at(-1).error)
: host.status,
history: samples, history: samples,
}); });
marks.set(host.url, high); marks.set(host.url, high);
@@ -481,7 +511,7 @@ export function buildReport(hosts, clientId, now, since) {
// Only one report is ever in flight, and it is abandoned after half the // Only one report is ever in flight, and it is abandoned after half the
// interval, so a slow POST can neither overlap the next report (which would // interval, so a slow POST can neither overlap the next report (which would
// carry the same samples) nor stall reporting for good. // carry the same samples) nor stall reporting for good.
class Reporter { export class Reporter {
constructor(state, clientId, intervalMs) { constructor(state, clientId, intervalMs) {
this.state = state; this.state = state;
this.clientId = clientId; this.clientId = clientId;
@@ -1079,9 +1109,8 @@ function updateHealthBox(state) {
// --- Debug Log Renderer ------------------------------------------------------ // --- Debug Log Renderer ------------------------------------------------------
function renderDebugLog() { // One line of the debug log panel, its message set as text.
const el = document.getElementById("debug-log"); function debugLogLine(entry) {
if (!el) return;
const levelColors = { const levelColors = {
error: "text-red-400", error: "text-red-400",
warning: "text-orange-400", warning: "text-orange-400",
@@ -1089,19 +1118,43 @@ function renderDebugLog() {
info: "text-gray-300", info: "text-gray-300",
debug: "text-gray-500", debug: "text-gray-500",
}; };
el.replaceChildren( const ts = formatUTCTimestamp(entry.timestamp);
...debugLog.map((entry) => { const lvl = entry.level.toUpperCase().padEnd(7);
const ts = formatUTCTimestamp(entry.timestamp); const line = document.createElement("div");
const lvl = entry.level.toUpperCase().padEnd(7); line.className = levelColors[entry.level] || "text-gray-400";
const line = document.createElement("div"); line.textContent = `${ts} ${lvl} ${entry.message}`;
line.className = levelColors[entry.level] || "text-gray-400"; return line;
line.textContent = `${ts} ${lvl} ${entry.message}`; }
return line;
}), // Draws every line of the debug log into the panel, when it is opened.
); function renderDebugLog() {
const el = document.getElementById("debug-log");
if (!el) return;
el.replaceChildren(...debugLog.map((entry) => debugLogLine(entry)));
el.scrollTop = el.scrollHeight; el.scrollTop = el.scrollHeight;
} }
// The request that scrolls the open panel to its newest line in the next
// frame. One waits at most, also in a hidden tab, where frames do not run.
let debugLogScrollFrame = 0;
// Adds entry to the open panel as its last line, and drops the first line
// once the panel holds more than maxLogEntries. A check that ends while the
// page is busy is timed only when the page is done, so the lines already
// shown are left as they are, and the panel is scrolled once a frame
// rather than once a line: scrolling it lays out all of its lines again.
function appendDebugLogLine(entry) {
const el = document.getElementById("debug-log");
if (!el) return;
el.append(debugLogLine(entry));
if (el.children.length > CONFIG.maxLogEntries)
el.removeChild(el.firstElementChild);
cancelAnimationFrame(debugLogScrollFrame);
debugLogScrollFrame = requestAnimationFrame(() => {
el.scrollTop = el.scrollHeight;
});
}
// --- Sorting ----------------------------------------------------------------- // --- Sorting -----------------------------------------------------------------
// Sort WAN hosts: pinned hosts first (alphabetically), then unpinned hosts // Sort WAN hosts: pinned hosts first (alphabetically), then unpinned hosts
@@ -1502,9 +1555,7 @@ async function init() {
// Clear all host history and reset state // Clear all host history and reset state
for (const host of state.allHosts) { for (const host of state.allHosts) {
host.history = []; host.reset();
host.lastLatency = null;
host.status = "pending";
} }
state.tickCount = 0; state.tickCount = 0;