Unusable DATA_DIR error names DATA_DIR; report file number is atomic.Int64 (closes #148, closes #161)

When the backend runs on its own, a DATA_DIR that could not be created
or read stopped the start with an error that named the path but not
DATA_DIR, and one where no file could be created let it start, to fail
at the first write of a report file. The start now creates a file in
DATA_DIR and removes it, and all three cases fail with
DATA_DIR "<value>": <reason>, the form config.New uses for the other
settings. Tests start the buffer with DATA_DIR below a regular file and
with DATA_DIR set to /proc.

The report file sequence counter is atomic.Int64 instead of
atomic.Uint64, as the Go style guide asks for int types. File names are
unchanged.

Model: opus-5-5

README.md

@@ -86,7 +86,9 @@ either one is broken. We provide:
 - `script/frontend-fmt-check` — check prettier formatting (read-only)
 - `script/frontend-viewport-test` — responsive-layout verification of the built
   frontend in a containerised headless Chrome (see
-  [test/viewport/README.md](test/viewport/README.md)). Not part of
+  [test/viewport/README.md](test/viewport/README.md)). Before the browser
+  starts, it checks that the page, and a path that falls back to it, are served
+  with `Cache-Control: no-cache` and `Content-Security-Policy`. Not part of
   `script/check`: it takes minutes.
 - `script/docker` — build the image from `Dockerfile` without the build cache,
   tagged `netwatch` via `script/projectname`
@@ -153,8 +155,26 @@ report a delta, so only new samples are sent; the mark advances only on a
 delivered report, and while paused nothing is sent. Changing the interval clears
 each host's history on the page; the samples in it not yet reported still go in
 the next report. Delivery failure is quiet — one debug-log line per outage,
-retried at the next interval, never blocking probing. The report-building step
-is a pure function of host state.
+never blocking probing. The report-building step is a pure function of host
+state.
+
+A sample in the history that no report has delivered is sent only as long as the
+history holds it, so after a refused or failed report the next report carries
+the samples the history still has. Samples an interval change cleared from the
+history are kept apart, the last 100 per host, until they are delivered. The
+history keeps each host's last 100 rounds: 100 seconds at the 1-second interval,
+200 seconds at 2 seconds, 5 minutes at the default of 3 seconds and 100 minutes
+at 60 seconds. At 1 second, one refused or failed report loses the oldest 20
+seconds of each host's samples, as the next delivered report comes 2 minutes
+after the last. At 2 seconds and longer one loses nothing; samples are lost from
+the third refused report in a row at 2 seconds, from the fifth at 3 seconds, and
+later still at the longer intervals.
+
+No report is sent while the page is paused, but each paused round still adds an
+entry to the history, so each report due during a pause counts as a refused one.
+A pause longer than the history spans loses the samples in the history recorded
+between the last delivered report and the pause: up to a minute's worth for each
+host if the report before the pause was delivered.
 
 ### Backend
 
@@ -240,17 +260,19 @@ Files are created with `O_EXCL` to prevent overwrites.
 
 The reports of a failed write stay in memory, and the next flush, within a
 minute, tries again to write them. From the failed write until a write succeeds,
-`POST /api/v1/reports` refuses every report with 500 and keeps nothing of it, so
-the page keeps its samples and sends them again at its next report. Reports
-already answered 200 are lost only if the server stops before a write succeeds;
-the final flush at stop tries them too, and if it fails, the process exits
-non-zero.
+`POST /api/v1/reports` refuses every report with 500 and keeps nothing of it.
+The page sends the refused samples again in its next report, as long as it still
+holds them: at the 1-second interval one refused report already loses some of
+those in the history (see [Reporting](#reporting)). Reports already answered 200
+are lost only if the server stops before a write succeeds; the final flush at
+stop tries them too, and if it fails, the process exits non-zero.
 
 #### Report limits
 
 `POST /api/v1/reports` takes reports from anyone who can reach it, without
 credentials, so it is bounded instead. Each refusal below answers with the same
-`{"status":"error"}` body as any other error.
+`{"status":"error"}` body as any other error, except nginx's refusal of a body
+over the limit in the image.
 
 - **JSON only.** A report must be sent with `Content-Type: application/json`,
   parameters such as `charset` allowed, as the page sends it; any other is
@@ -258,6 +280,16 @@ credentials, so it is bounded instead. Each refusal below answers with the same
   send a `text/plain` POST, or a form, without a CORS preflight; refusing them
   keeps any site from making its visitors' browsers store reports (see
   [CORS](#cors)).
+- **Body limit.** A report body may be at most 1 MiB, the backend's limit for
+  any request. Run on its own, the backend answers 413 before the route to a
+  request that declares a longer body. Without a declared length, the route
+  answers 413 only when the report it reads runs past 1 MiB; nothing after the
+  end of the report is read, so a longer body whose report ends sooner is
+  stored, and one that stops being valid JSON sooner is answered 400. In the
+  image, nginx refuses a body over 1 MiB first, with or without a declared
+  length, as its `client_max_body_size`, which `configs/nginx.conf` does not
+  set, is 1 MiB by default. nginx answers 413 with its own HTML error page, and
+  the backend never sees the request.
 - **Rate limit.** Each client address, resolved through `TRUSTED_PROXIES`, may
   send `REPORTS_PER_MINUTE` reports a minute; past that it gets 429 with
   `Retry-After: 60`. The minute slides: reports from the minute before still
@@ -281,6 +313,12 @@ credentials, so it is bounded instead. Each refusal below answers with the same
   how much of the newest reports is kept: the default of 1 GiB is small enough
   for any host; set it to the space you can give `DATA_DIR`.
 
+At the defaults, one address may send 60 reports of up to 1 MiB a minute, and a
+report made up to compress badly takes about as much room in the report files as
+its request body, so in about 17 minutes one address can fill the 1 GiB cap with
+its own reports, every report stored before them deleted to make room. Lowering
+`REPORTS_PER_MINUTE` or raising `DATA_DIR_MAX_BYTES` makes that take longer.
+
 #### CORS
 
 The page calls the API from the origin it is served from, so by default the
@@ -302,12 +340,16 @@ the metrics record its duration and response size, labelled with its path,
 method and status; they also count those requests in progress, and include Go's
 runtime and process metrics. No other request is recorded: not those to
 `/metrics` itself, and not those answered before they reach either route, such
-as a CORS preflight, or a request refused with 404 for a path no route has, 405
-for a method its route does not take, or 413 for declaring a body length over
-the 1 MiB limit. A report whose body goes over the limit without declaring its
-length reaches the route, is answered 413 there, and is recorded with that
-status. Clients can make up any number of paths and methods, and each would add
-labels to the metrics for as long as the server runs.
+as a CORS preflight, or a request refused with 404 for a path no route has or
+405 for a method its route does not take. Run on its own, the backend also
+answers 413 before the route to a request declaring a body length over the 1 MiB
+limit, which is not recorded. A body over the limit sent without a declared
+length reaches the route and is recorded with the status the route answers,
+which is 413 only when the report it reads runs past the limit: nothing after
+the end of the report is read. In the image neither reaches the backend: nginx
+answers both with 413 itself (see [Report limits](#report-limits)), and nothing
+of them is recorded. Clients can make up any number of paths and methods, and
+each would add labels to the metrics for as long as the server runs.
 
 `/metrics` is rate limited, so that its password cannot be guessed quickly: each
 client address, resolved through `TRUSTED_PROXIES`, may make 60 requests to it a
@@ -435,7 +477,10 @@ address nginx passes on and no other. The image:
   each entry with `netwatch-server check-cidr`, which parses it as the backend
   parses its own `TRUSTED_PROXIES`
 - Sends access logs to stdout
-- Caches static assets with immutable headers
+- Sends the page with `Cache-Control: no-cache`, so a browser that keeps a copy
+  checks with the server that it is current before using it again; only the
+  files under `/assets/`, whose names change whenever their contents do, are
+  cached as immutable
 - Sends the security headers `REPO_POLICIES.md` requires on every response, as
   `configs/security-headers.conf` sets them, in place of the backend's own
 - Stores reports in `DATA_DIR`, `/data/reports` by default, on the `/data`

TODO.md

@@ -21,6 +21,46 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of
 
 # Completed Steps
 
+- 2026-10-18: when the backend runs on its own, a `DATA_DIR` that cannot be
+  created or read, or where no file can be created, stops the start with an
+  error naming `DATA_DIR` and its value, as the other settings' errors do
+  ([#148](https://git.eeqj.de/sneak/netwatch/issues/148)); the first two named
+  only the path, and the third was found only at the first write of a report
+  file. At start the server creates a file in `DATA_DIR` and removes it. Unit
+  tests start the buffer with `DATA_DIR` below a regular file and with
+  `DATA_DIR` set to `/proc`, and check the error. The report file number is an
+  `atomic.Int64`, as the Go style guide asks for int types
+  ([#161](https://git.eeqj.de/sneak/netwatch/issues/161)); file names are
+  unchanged
+- 2026-10-18: the image's nginx sends the page with `Cache-Control: no-cache`
+  ([#141](https://git.eeqj.de/sneak/netwatch/issues/141)), so a browser that
+  keeps a copy checks with the server that it is current before using it again.
+  The page went out with no `Cache-Control`, so a browser could keep it after a
+  deploy and ask for files under `/assets/` that the new build no longer has,
+  and show an empty page. A path that `location /` answers with the page through
+  its `try_files` fallback gets the same header. `location /` now includes the
+  security headers again, as `/assets/` does, since its own `add_header` drops
+  the server's. `/assets/` keeps its immutable caching.
+  `make frontend-viewport-test` fails unless `/` and a path that falls back to
+  the page carry `Cache-Control: no-cache` and `Content-Security-Policy`, which
+  comes from the same file as the other security headers
+- 2026-10-18: `README.md` says what the page loses while reports are refused or
+  the page is paused ([#145](https://git.eeqj.de/sneak/netwatch/issues/145)),
+  how fast one address can replace the stored reports
+  ([#146](https://git.eeqj.de/sneak/netwatch/issues/146)), and that in the image
+  nginx answers a body over 1 MiB itself
+  ([#147](https://git.eeqj.de/sneak/netwatch/issues/147)). A sample in the
+  history that no report has delivered is sent only as long as the history holds
+  it: at the 1-second interval one refused or failed report loses the oldest 20
+  seconds of each host's samples, and a pause longer than the history loses
+  every sample between the last delivered report and the pause. Samples an
+  interval change cleared are kept apart, the last 100 per host, until they are
+  delivered. At the defaults, one address sending 1 MiB reports made up to
+  compress badly can fill the 1 GiB cap in about 17 minutes. nginx's
+  `client_max_body_size` is also 1 MiB by default, so in the image it answers a
+  larger body with 413 and its own HTML page, and the backend never sees or
+  records it; the backend's own 413 sentences now say they apply to it run on
+  its own. The reporter, the body limit and nginx are unchanged
 - 2026-10-18: a browser that cannot format the header's clocks still probes
   ([#149](https://git.eeqj.de/sneak/netwatch/issues/149)). The clocks name the
   local time zone through `Intl.DateTimeFormat`'s `shortGeneric` option, which

configs/nginx.conf

@@ -30,8 +30,13 @@ server {
     access_log /dev/stdout combined;
     error_log  /dev/stderr warn;
 
+    # With no-cache, a browser never uses a copy of the page that names
+    # assets a newer build no longer has without first checking with the
+    # server that it is current.
     location / {
         try_files $uri $uri/ /index.html;
+        add_header Cache-Control no-cache;
+        include /etc/nginx/security-headers.conf;
     }
 
     # Cache static assets aggressively

internal/reportbuf/reportbuf.go

@@ -88,7 +88,7 @@ type Buffer struct {
 	now func() time.Time
 	// seq numbers the report files, so that two named in the same
 	// millisecond still get different names.
-	seq      atomic.Uint64
+	seq      atomic.Int64
 	stopOnce sync.Once
 	// usedBytes is what Append checks against maxBytes: the size
 	// of the report files in dataDir, plus the reports waiting to
@@ -127,16 +127,33 @@ func New(
 
 	lc.Append(fx.Hook{
 		OnStart: func(_ context.Context) error {
+			// The errors name DATA_DIR, as config.New's do for the
+			// settings it checks.
 			err := os.MkdirAll(b.dataDir, dirPerms)
 			if err != nil {
-				return fmt.Errorf("create data dir: %w", err)
+				return fmt.Errorf("DATA_DIR %q: %w", b.dataDir, err)
+			}
+
+			// Creating and removing a file stops the start when no
+			// file can be created in DATA_DIR, instead of the first
+			// write of a report file failing after reports were
+			// answered 200. A full disk is not caught: the empty
+			// file is still created.
+			f, err := os.CreateTemp(b.dataDir, "")
+			if err != nil {
+				return fmt.Errorf("DATA_DIR %q: %w", b.dataDir, err)
+			}
+
+			err = errors.Join(f.Close(), os.Remove(f.Name()))
+			if err != nil {
+				return fmt.Errorf("DATA_DIR %q: %w", b.dataDir, err)
 			}
 
 			// Report files left by earlier runs count too, and are
 			// the first to be deleted to make room.
 			files, err := reportFiles(b.dataDir)
 			if err != nil {
-				return err
+				return fmt.Errorf("DATA_DIR %q: %w", b.dataDir, err)
 			}
 
 			b.mu.Lock()

internal/reportbuf/reportbuf_test.go

@@ -108,6 +108,68 @@ func TestFailedFinalFlushFailsStop(t *testing.T) {
 	}
 }
 
+// TestUncreatableDataDirStopsStart sets DATA_DIR below a regular file,
+// where no directory can be created, even by root, as the tests run in
+// the backend image. The start must fail with an error naming DATA_DIR
+// and its value, as the errors for the other settings do.
+func TestUncreatableDataDirStopsStart(t *testing.T) {
+	file := filepath.Join(t.TempDir(), "file")
+	writeBytes(t, file, 1)
+
+	dir := filepath.Join(file, "reports")
+	t.Setenv("DATA_DIR", dir)
+
+	app := fxtest.New(t,
+		fx.Provide(
+			globals.New,
+			logger.New,
+			config.New,
+			reportbuf.New,
+		),
+		fx.Invoke(func(*reportbuf.Buffer) {}),
+	)
+
+	err := app.Start(t.Context())
+	if err == nil {
+		t.Fatal("start succeeded with DATA_DIR below a regular file")
+	}
+
+	want := fmt.Sprintf("DATA_DIR %q: ", dir)
+	if !strings.Contains(err.Error(), want) {
+		t.Fatalf("start error = %q, want it to contain %q", err, want)
+	}
+}
+
+// TestUnwritableDataDirStopsStart sets DATA_DIR to /proc, which exists
+// and can be read, but where no file can be created, even by root, as
+// the tests run in the backend image: root ignores a directory the test
+// makes read-only. The start must fail with an error naming DATA_DIR and
+// its value, rather than the first write of a report file.
+func TestUnwritableDataDirStopsStart(t *testing.T) {
+	dir := "/proc"
+	t.Setenv("DATA_DIR", dir)
+
+	app := fxtest.New(t,
+		fx.Provide(
+			globals.New,
+			logger.New,
+			config.New,
+			reportbuf.New,
+		),
+		fx.Invoke(func(*reportbuf.Buffer) {}),
+	)
+
+	err := app.Start(t.Context())
+	if err == nil {
+		t.Fatal("start succeeded with DATA_DIR /proc")
+	}
+
+	want := fmt.Sprintf("DATA_DIR %q: ", dir)
+	if !strings.Contains(err.Error(), want) {
+		t.Fatalf("start error = %q, want it to contain %q", err, want)
+	}
+}
+
 // TestStopWaitsForPeriodicWrite holds open the write the periodic flush
 // starts, then stops the app. The stop must wait until the write is
 // released, and the report must then be in its file. The test runs in

script/frontend-viewport-test

@@ -7,6 +7,8 @@
 # headless Chrome at it over CDP, and asserts on computed layout at every
 # viewport width derived from the app's own CSS. See
 # test/viewport/README.md for what this covers and what it cannot.
+# Before the browser starts, it checks that the page is served with
+# Cache-Control: no-cache and Content-Security-Policy.
 #
 # Deliberately not part of script/check: it needs Docker and takes far
 # longer than the 20s budget make test has to stay inside.
@@ -75,6 +77,32 @@ main() {
         -v "$ROOT/configs/security-headers.conf:/etc/nginx/security-headers.conf:ro" \
         "$SERVER_IMAGE" > /dev/null
 
+    # The page, and /no-such-page, which location / answers with the
+    # page through its try_files fallback, must go out with
+    # Cache-Control: no-cache. That add_header in its location drops the
+    # server's own unless the location includes the security headers
+    # again; only Content-Security-Policy is checked, as it comes from
+    # the same file as the rest. curl tries up to ten more times, a
+    # second apart, while nginx starts. It runs without -S, which would
+    # print an error for every failed attempt, even when a later one
+    # succeeds.
+    for path in / /no-such-page; do
+        if ! headers="$(docker exec "$SERVER" curl -s -o /dev/null -D - \
+            --retry 10 --retry-connrefused --retry-delay 1 \
+            "http://127.0.0.1:8080$path")"; then
+            echo "frontend-viewport-test: nginx did not answer $path" >&2
+            exit 1
+        fi
+        headers="$(printf '%s\n' "$headers" | tr -d '\r')"
+        if ! printf '%s\n' "$headers" | grep -qix 'Cache-Control: no-cache' ||
+            ! printf '%s\n' "$headers" | grep -qi '^Content-Security-Policy:'; then
+            echo "frontend-viewport-test: $path needs" \
+                "Cache-Control: no-cache and Content-Security-Policy; got:" >&2
+            printf '%s\n' "$headers" >&2
+            exit 1
+        fi
+    done
+
     # The image's own entrypoint already exposes CDP on 9222 and passes
     # --no-sandbox, so only extra flags belong here; re-specifying the
     # debugging port collides with it and leaves the endpoint bound to