Entrypoint acts as root on nothing outside /data (closes #80)

`bin/entrypoint.sh` now runs `netwatch-server prepare-data-dir`, which
refuses a `DATA_DIR` that is not `/data` or a path below it written in
full, then creates `DATA_DIR`, gives `/data` and everything in it to
`netwatch`, and sets mode 750 on `/data` and `DATA_DIR`. Every step goes
through a Go `os.Root` opened on `/data`, and the modes are set on the
opened directories rather than by name, so neither a symbolic link
already there nor one a host process swaps in while the container
starts can make root create or change anything outside `/data`. The
README says which `DATA_DIR` values are accepted.

Model: opus-5-5

Dockerfile

@@ -20,7 +20,10 @@ RUN make lint
 # golang:1.25-alpine (2026-02-27)
 FROM golang:1.25-alpine@sha256:f6751d823c26342f9506c03797d2527668d095b0a15f1862cddb4d927a7a4ced AS builder
 
-RUN apk add --no-cache git make
+# gcc and musl-dev are for make test: its race detector needs cgo, which
+# Go turns on by itself once a C compiler is present. make build still
+# sets CGO_ENABLED=0, so the binary stays static.
+RUN apk add --no-cache gcc git make musl-dev
 
 WORKDIR /src
 

README.md

@@ -39,14 +39,16 @@ halves, so the root `make check` fails if either one is broken. We provide:
 - `script/bootstrap` — install all dependencies (the pinned node via nvm unless
   one new enough for the frontend's dependencies is installed, yarn via
   corepack, `yarn install --frozen-lockfile`, the pinned Go unless one at least
-  as new as `backend/go.mod` asks for is installed, and the Go modules), linking
-  what it installs itself into `~/.local/bin`, which has to be on `PATH`. It
-  installs no Go linter and not Docker: `make lint` runs the linter in Docker
+  as new as `backend/go.mod` asks for is installed, the Go modules, and gcc with
+  the C library headers unless gcc is installed, for the race detector in
+  `make test`), linking what it installs itself into `~/.local/bin`, which has
+  to be on `PATH`. It installs no Go linter and not Docker: `make lint` runs the
+  linter in Docker
 - `script/setup` — make a fresh clone ready for development: bootstrap plus the
   git pre-commit hook
 - `script/projectname` — print the project name (used for the Docker image tag)
-- `script/test` — run `script/frontend-test`, then the backend's Go tests, both
-  within one 30-second timeout
+- `script/test` — run `script/frontend-test`, then the backend's Go tests, each
+  under its own 30-second timeout
 - `script/lint` — run `script/frontend-lint`, then golangci-lint in Docker, by
   building the lint stage of `Dockerfile` without the cache
 - `script/fmt` — format all files (writes): prettier, then gofmt over `backend/`
@@ -222,8 +224,10 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
     - `CORS_ALLOWED_ORIGINS`, default empty: other origins whose pages may call
       the API
     - `DEBUG`, default `false`: debug logging
-    - `DATA_DIR`, default `/data/reports`: leave unset; reports kept outside
-      `/data` do not survive a redeploy
+    - `DATA_DIR`, default `/data/reports`: the directory the reports are kept
+      in: `/data` or a path below it, with no `.` or `..` part and no extra `/`.
+      The container also stops if the path goes through a symbolic link that
+      leads out of `/data` or is written as a full path
     - `TRUSTED_PROXIES`, default empty: set it to the address the reverse proxy
       in front of the container connects from, as an IP address or CIDR; several
       are separated by commas. nginx takes the client address from

TODO.md

@@ -23,6 +23,27 @@ latest run passes.
 
 # Completed Steps
 
+- 2026-10-03: root no longer acts outside `/data` when it prepares `DATA_DIR`
+  (issue #80): `bin/entrypoint.sh` runs `netwatch-server prepare-data-dir`,
+  which refuses a `DATA_DIR` that is not `/data` or a path below it written in
+  full, then creates `DATA_DIR`, gives `/data` and everything in it to
+  `netwatch` and sets the modes, all through a Go `os.Root` opened on `/data`.
+  That refuses any path leading out of `/data`, so neither a symbolic link
+  already there nor one a host process swaps in during the start can make root
+  create or change anything elsewhere, and `DATA_DIR=/etc` no longer gives
+  `/etc` to `netwatch`. The `README.md` section "Running under upaas" says which
+  values are accepted
+- 2026-10-03: the Go tests run with the race detector and coverage (issue #88):
+  `backend/script/test` runs `go test -timeout 30s -race -cover ./...` and, if
+  that fails, runs it again with `-v` and fails. Go's `-timeout` bounds the
+  tests, not their compile; the root `script/test` no longer puts one 30-second
+  timeout around both halves, which a cold Go build cache could use up on
+  compiling alone. The race detector needs a C compiler: the builder stage of
+  `Dockerfile` has gcc and musl-dev, and `script/bootstrap` installs gcc, with
+  the C library headers on apt and apk, when gcc is missing; the binary is still
+  built with `CGO_ENABLED=0`. New tests cover the health check's answer, a valid
+  report's answer, a report file's exact contents, and the flush when the buffer
+  reaches 10 MiB; the handlers' `TestImport` stub is gone
 - 2026-10-03: each target check times out after 80% of the refresh interval
   (issue #78), 24 seconds at 30 seconds, where it was capped at 3 seconds. A
   round started early, after an interval change or when the recovery probe finds

backend/README.md

@@ -35,7 +35,10 @@ pattern as the repo root: the targets in `backend/Makefile` are thin shims over
   stamped in. The version is `VERSION` from the environment;
   when that is unset or empty, it falls back to `git describe` inside a git
   checkout, then to `dev`
-- `script/test` — run the Go tests under a 30-second timeout
+- `script/test` — run the Go tests with the race detector and coverage. Go's
+  `-timeout 30s` bounds the tests, not their compile. If they fail, they run
+  again with `-v` for the details, and the script fails. The race detector needs
+  a C compiler
 - `script/lint` — check `.golangci.yml` against its pinned sha256, then run
   golangci-lint. It runs inside the golangci-lint image of the lint stage of
   the root `Dockerfile`; from a checkout, run `make lint` at the repo root,
@@ -104,8 +107,9 @@ this server. The image's entrypoint, `bin/entrypoint.sh`, starts the server as
 user `netwatch` (uid 1000) with `BIND_ADDRESS=127.0.0.1` and `PORT=8081`, so
 only nginx reaches it, and with `TRUSTED_PROXIES=127.0.0.1/32`, so it takes the
 client address nginx passes on and no other. `DATA_DIR` is `/data/reports`, on
-the `/data` volume; the entrypoint creates it and gives it and `/data` to
-`netwatch` before starting the server. nginx replaces the security headers
+the `/data` volume; before starting the server, the entrypoint creates it and
+gives it and `/data` to `netwatch` with `netwatch-server prepare-data-dir`,
+which acts on nothing outside `/data`. nginx replaces the security headers
 this server sets with those in the root `security-headers.conf`, so those are
 what clients of the image see.
 

backend/cmd/netwatch-server/main.go

@@ -4,6 +4,7 @@ package main
 import (
 	"fmt"
 	"os"
+	"os/user"
 
 	"sneak.berlin/go/netwatch/internal/config"
 	"sneak.berlin/go/netwatch/internal/globals"
@@ -37,6 +38,26 @@ func main() {
 		return
 	}
 
+	// "netwatch-server prepare-data-dir DATA_DIR" gets DATA_DIR ready
+	// for the netwatch user, or exits 1 with the error; see
+	// reportbuf.PrepareDataDir. bin/entrypoint.sh runs it as root
+	// before it starts this server as that user.
+	if len(os.Args) == 3 && os.Args[1] == "prepare-data-dir" {
+		netwatch, err := user.Lookup("netwatch")
+		if err != nil {
+			fmt.Fprintln(os.Stderr, err)
+			os.Exit(1)
+		}
+
+		err = reportbuf.PrepareDataDir("/data", os.Args[2], netwatch)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "DATA_DIR '%s': %v\n", os.Args[2], err)
+			os.Exit(1)
+		}
+
+		return
+	}
+
 	globals.Appname = Appname
 	globals.Version = Version
 

backend/internal/handlers/handlers_test.go

@@ -1,13 +0,0 @@
-package handlers_test
-
-import (
-	"testing"
-
-	_ "sneak.berlin/go/netwatch/internal/handlers"
-)
-
-func TestImport(t *testing.T) {
-	t.Parallel()
-	// Compilation check — verifies the package parses
-	// and all imports resolve.
-}

backend/internal/handlers/healthcheck_test.go

@@ -0,0 +1,120 @@
+package handlers_test
+
+import (
+	"encoding/json"
+	"maps"
+	"net/http"
+	"net/http/httptest"
+	"slices"
+	"testing"
+	"time"
+
+	"sneak.berlin/go/netwatch/internal/globals"
+	"sneak.berlin/go/netwatch/internal/handlers"
+	"sneak.berlin/go/netwatch/internal/healthcheck"
+	"sneak.berlin/go/netwatch/internal/logger"
+
+	"go.uber.org/fx/fxtest"
+)
+
+// newStartedHandlers builds Handlers with a real health check for the
+// server named in g, and starts them, which records the time the
+// uptime counts from.
+func newStartedHandlers(t *testing.T, g *globals.Globals) *handlers.Handlers {
+	t.Helper()
+
+	lc := fxtest.NewLifecycle(t)
+
+	log, err := logger.New(lc, logger.Params{Globals: g})
+	if err != nil {
+		t.Fatalf("logger: %v", err)
+	}
+
+	hc, err := healthcheck.New(lc,
+		healthcheck.Params{Globals: g, Logger: log})
+	if err != nil {
+		t.Fatalf("health check: %v", err)
+	}
+
+	h, err := handlers.New(lc,
+		handlers.Params{Globals: g, Healthcheck: hc, Logger: log})
+	if err != nil {
+		t.Fatalf("handlers: %v", err)
+	}
+
+	lc.RequireStart()
+	t.Cleanup(lc.RequireStop)
+
+	return h
+}
+
+// TestHandleHealthCheck checks the health check's answer: 200, a JSON
+// content type, and a JSON object with exactly the fields of
+// healthcheck.Response, carrying this server's name and version and
+// an uptime counted from its start.
+func TestHandleHealthCheck(t *testing.T) {
+	t.Parallel()
+
+	g := &globals.Globals{Appname: "netwatch-server", Version: "v1.2.3"}
+	h := newStartedHandlers(t, g)
+
+	rec := httptest.NewRecorder()
+	req := httptest.NewRequestWithContext(t.Context(),
+		http.MethodGet, "/.well-known/healthcheck", http.NoBody)
+
+	h.HandleHealthCheck().ServeHTTP(rec, req)
+
+	if rec.Code != http.StatusOK {
+		t.Fatalf("status = %d, want %d", rec.Code, http.StatusOK)
+	}
+
+	contentType := rec.Header().Get("Content-Type")
+	if contentType != "application/json; charset=utf-8" {
+		t.Errorf("Content-Type = %q, want %q",
+			contentType, "application/json; charset=utf-8")
+	}
+
+	var body map[string]any
+
+	err := json.Unmarshal(rec.Body.Bytes(), &body)
+	if err != nil {
+		t.Fatalf("body not a JSON object: %v (%q)", err, rec.Body.String())
+	}
+
+	fields := []string{
+		"appname", "now", "status", "uptimeHuman", "uptimeSeconds", "version",
+	}
+	if got := slices.Sorted(maps.Keys(body)); !slices.Equal(got, fields) {
+		t.Fatalf("fields = %v, want %v", got, fields)
+	}
+
+	for field, want := range map[string]string{
+		"appname": g.Appname, "status": "ok", "version": g.Version,
+	} {
+		if body[field] != want {
+			t.Errorf("%s = %v, want %q", field, body[field], want)
+		}
+	}
+
+	now, _ := body["now"].(string)
+
+	at, err := time.Parse(time.RFC3339Nano, now)
+	if err != nil || time.Since(at).Abs() > time.Minute {
+		t.Errorf("now = %q, want the current time in RFC 3339 (%v)", now, err)
+	}
+
+	// Started just now, so the uptime is well under a minute.
+	human, _ := body["uptimeHuman"].(string)
+
+	uptime, err := time.ParseDuration(human)
+	if err != nil || uptime > time.Minute {
+		t.Errorf("uptimeHuman = %q, want a duration under a minute (%v)",
+			human, err)
+	}
+
+	seconds, ok := body["uptimeSeconds"].(float64)
+	if !ok || seconds < 0 || seconds > time.Minute.Seconds() {
+		t.Errorf("uptimeSeconds = %v, want a number of seconds under a minute",
+			body["uptimeSeconds"])
+	}
+}

backend/internal/handlers/report_test.go

@@ -46,6 +46,63 @@ func decodeStatus(t *testing.T, body []byte) string {
 	return resp.Status
 }
 
+// TestHandleReportAcceptsValidReports checks the answer to a valid
+// report, one with no hosts and one shaped as the frontend sends them:
+// 200 and {"status":"ok"} as JSON.
+func TestHandleReportAcceptsValidReports(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name string
+		body string
+	}{
+		{
+			name: "no hosts",
+			body: `{"clientId":"c1","geo":null,"hosts":[],` +
+				`"timestamp":"2026-10-03T12:00:00.000Z"}`,
+		},
+		{
+			name: "a host with a latency and an error sample",
+			body: `{"clientId":"c1","geo":null,"hosts":[{` +
+				`"name":"Example","url":"https://example.com/",` +
+				`"status":"error","history":[` +
+				`{"t":1790000000000,"latency":42,"error":null},` +
+				`{"t":1790000003000,"latency":null,"error":"timeout"}]}],` +
+				`"timestamp":"2026-10-03T12:00:00.000Z"}`,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			h := newTestHandlers(stubAppender{}, io.Discard)
+
+			rec := httptest.NewRecorder()
+			req := httptest.NewRequestWithContext(t.Context(),
+				http.MethodPost, "/api/v1/reports",
+				strings.NewReader(tt.body),
+			)
+
+			h.HandleReport().ServeHTTP(rec, req)
+
+			if rec.Code != http.StatusOK {
+				t.Fatalf("status = %d, want %d", rec.Code, http.StatusOK)
+			}
+
+			contentType := rec.Header().Get("Content-Type")
+			if contentType != "application/json; charset=utf-8" {
+				t.Errorf("Content-Type = %q, want %q",
+					contentType, "application/json; charset=utf-8")
+			}
+
+			if got := rec.Body.String(); got != "{\"status\":\"ok\"}\n" {
+				t.Errorf("body = %q, want %q", got, "{\"status\":\"ok\"}\n")
+			}
+		})
+	}
+}
+
 func TestHandleReportStorageFailureIsNon2xx(t *testing.T) {
 	t.Parallel()
 

backend/internal/reportbuf/datadir.go

@@ -0,0 +1,113 @@
+package reportbuf
+
+import (
+	"errors"
+	"io/fs"
+	"os"
+	"os/user"
+	"path/filepath"
+	"strconv"
+	"syscall"
+)
+
+// ErrDataDirOutsideVolume is returned by PrepareDataDir for a DATA_DIR
+// that is not the volume or a path below it, written in full.
+var ErrDataDirOutsideVolume = errors.New(
+	"must be /data or a path below it, with no '.', '..' or extra '/'")
+
+// PrepareDataDir gets dir, the server's DATA_DIR, ready for owner, the
+// user the server runs as, so that a host directory mounted at volume,
+// /data in the image, needs no preparing: it creates dir, gives volume
+// and everything in it to owner, and gives volume and dir the mode the
+// server gives a directory it creates. dir must be volume or a path
+// below it, with no '.', '..', empty part or '/' at the end.
+//
+// bin/entrypoint.sh runs this as root, which would follow a symbolic
+// link anywhere, so every step goes through an os.Root opened on
+// volume: it follows a link only when it is written as a relative
+// path that stays inside volume, and refuses any other. A process on
+// the host can swap a link onto a path in volume at any moment while
+// this runs. Even then, the os.Root checks each link as it reaches it,
+// MkdirAll never creates a directory through a link, Lchown never
+// changes what a link points to, and the modes are set on directories
+// already opened (see chmodDir), so the most that process can do is
+// make a step fail or wait, or act on something else inside volume.
+func PrepareDataDir(volume, dir string, owner *user.User) error {
+	// rel is dir as a path from volume; IsLocal is false for one that
+	// leads out of it.
+	rel, err := filepath.Rel(volume, dir)
+	if err != nil || dir != filepath.Clean(dir) || !filepath.IsLocal(rel) {
+		return ErrDataDirOutsideVolume
+	}
+
+	uid, err := strconv.Atoi(owner.Uid)
+	if err != nil {
+		return err
+	}
+
+	gid, err := strconv.Atoi(owner.Gid)
+	if err != nil {
+		return err
+	}
+
+	root, err := os.OpenRoot(volume)
+	if err != nil {
+		return err
+	}
+
+	defer func() { _ = root.Close() }()
+
+	err = root.MkdirAll(rel, dirPerms)
+	if err != nil {
+		return err
+	}
+
+	// WalkDir does not follow a symbolic link, and Lchown gives the
+	// link itself to owner, not what it points to.
+	err = fs.WalkDir(root.FS(), ".",
+		func(name string, _ fs.DirEntry, err error) error {
+			if err != nil {
+				return err
+			}
+
+			return root.Lchown(name, uid, gid)
+		})
+	if err != nil {
+		return err
+	}
+
+	err = chmodDir(root, ".")
+	if err != nil {
+		return err
+	}
+
+	return chmodDir(root, rel)
+}
+
+// chmodDir gives name, a directory inside root, the mode the server
+// gives a directory it creates. Root.Chmod would not hold: on Linux it
+// checks that name is not a symbolic link, then sets the mode by name,
+// following a link swapped in between. So chmodDir opens name through
+// root and sets the mode on the open directory. It refuses anything
+// but a directory: a directory has no second name (hard link), so the
+// one opened is inside root, where any other file could be a hard link
+// to one outside.
+func chmodDir(root *os.Root, name string) error {
+	dir, err := root.Open(name)
+	if err != nil {
+		return err
+	}
+
+	defer func() { _ = dir.Close() }()
+
+	info, err := dir.Stat()
+	if err != nil {
+		return err
+	}
+
+	if !info.IsDir() {
+		return &fs.PathError{Op: "chmod", Path: name, Err: syscall.ENOTDIR}
+	}
+
+	return dir.Chmod(dirPerms)
+}

backend/internal/reportbuf/datadir_test.go

@@ -0,0 +1,271 @@
+package reportbuf_test
+
+import (
+	"errors"
+	"io/fs"
+	"os"
+	"os/user"
+	"path/filepath"
+	"strconv"
+	"syscall"
+	"testing"
+
+	"sneak.berlin/go/netwatch/internal/reportbuf"
+)
+
+// reports is the last part of DATA_DIR in these tests, as in the
+// image's /data/reports.
+const reports = "reports"
+
+// currentUser is the user the test runs as, the only owner a test not
+// run as root can give files to.
+func currentUser() *user.User {
+	return &user.User{
+		Uid: strconv.Itoa(os.Getuid()),
+		Gid: strconv.Itoa(os.Getgid()),
+	}
+}
+
+// tempDirMode700 is a new directory in a t.TempDir with mode 0700, so
+// a test can tell that PrepareDataDir left its mode alone.
+func tempDirMode700(t *testing.T) string {
+	t.Helper()
+
+	dir := filepath.Join(t.TempDir(), "d")
+
+	err := os.Mkdir(dir, 0o700)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	return dir
+}
+
+func requireMode(t *testing.T, path string, want fs.FileMode) {
+	t.Helper()
+
+	info, err := os.Stat(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if info.Mode() != want {
+		t.Errorf("%s: mode %v, want %v", path, info.Mode(), want)
+	}
+}
+
+func requireMissing(t *testing.T, path string) {
+	t.Helper()
+
+	_, err := os.Lstat(path)
+	if !errors.Is(err, fs.ErrNotExist) {
+		t.Errorf("%s: created, or Lstat failed: %v", path, err)
+	}
+}
+
+func requireOwner(t *testing.T, path string, uid, gid uint32) {
+	t.Helper()
+
+	info, err := os.Lstat(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	stat, _ := info.Sys().(*syscall.Stat_t)
+	if stat.Uid != uid || stat.Gid != gid {
+		t.Errorf("%s: owner %d:%d, want %d:%d", path, stat.Uid, stat.Gid,
+			uid, gid)
+	}
+}
+
+func TestPrepareDataDirCreatesDataDir(t *testing.T) {
+	t.Parallel()
+
+	volume := t.TempDir()
+	dir := filepath.Join(volume, "a", reports)
+
+	err := reportbuf.PrepareDataDir(volume, dir, currentUser())
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	requireMode(t, volume, fs.ModeDir|0o750)
+	requireMode(t, dir, fs.ModeDir|0o750)
+}
+
+// TestPrepareDataDirSetsModeOfExistingDataDir: a DATA_DIR already on
+// the host with another mode gets the mode too, not only a new one.
+func TestPrepareDataDirSetsModeOfExistingDataDir(t *testing.T) {
+	t.Parallel()
+
+	volume := t.TempDir()
+	dir := filepath.Join(volume, reports)
+
+	err := os.Mkdir(dir, 0o700)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = reportbuf.PrepareDataDir(volume, dir, currentUser())
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	requireMode(t, dir, fs.ModeDir|0o750)
+}
+
+func TestPrepareDataDirTakesTheVolumeItself(t *testing.T) {
+	t.Parallel()
+
+	volume := t.TempDir()
+
+	err := reportbuf.PrepareDataDir(volume, volume, currentUser())
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	requireMode(t, volume, fs.ModeDir|0o750)
+}
+
+// TestPrepareDataDirRefusesDataDirOutsideVolume covers a DATA_DIR that
+// is relative, outside the volume, or not written in full.
+func TestPrepareDataDirRefusesDataDirOutsideVolume(t *testing.T) {
+	t.Parallel()
+
+	volume := tempDirMode700(t)
+	for _, dir := range []string{
+		reports, volume + "/../new", volume + "/", volume + "//" + reports,
+		volume + "/./" + reports, volume + "/" + reports + "/..", volume + "x",
+		"/etc",
+	} {
+		err := reportbuf.PrepareDataDir(volume, dir, currentUser())
+		if !errors.Is(err, reportbuf.ErrDataDirOutsideVolume) {
+			t.Errorf("%q: error = %v, want ErrDataDirOutsideVolume", dir, err)
+		}
+	}
+
+	requireMissing(t, filepath.Join(filepath.Dir(volume), "new"))
+	requireMode(t, volume, fs.ModeDir|0o700)
+}
+
+// TestPrepareDataDirRefusesLinkOutOfVolume puts a symbolic link to a
+// directory outside the volume on the path to DATA_DIR, written as a
+// full path and as one that climbs out with '..', and as DATA_DIR
+// itself, where the mode would be set through it.
+func TestPrepareDataDirRefusesLinkOutOfVolume(t *testing.T) {
+	t.Parallel()
+
+	for _, tc := range []struct {
+		name      string
+		climbsOut bool
+		link, dir string
+	}{
+		{"full path", false, "x", "x/reports"},
+		{"climbs out", true, "x", "x/reports"},
+		{"DATA_DIR itself", false, reports, reports},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+
+			outside := tempDirMode700(t)
+			volume := t.TempDir()
+
+			climbOut, err := filepath.Rel(volume, outside)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			target := outside
+			if tc.climbsOut {
+				target = climbOut
+			}
+
+			err = os.Symlink(target, filepath.Join(volume, tc.link))
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			err = reportbuf.PrepareDataDir(volume,
+				filepath.Join(volume, tc.dir), currentUser())
+			if err == nil {
+				t.Error("no error")
+			}
+
+			requireMissing(t, filepath.Join(outside, reports))
+			requireMode(t, outside, fs.ModeDir|0o700)
+		})
+	}
+}
+
+// TestPrepareDataDirRefusesDanglingLink: DATA_DIR is a symbolic link
+// to a name in the volume that does not exist, which is not created.
+func TestPrepareDataDirRefusesDanglingLink(t *testing.T) {
+	t.Parallel()
+
+	volume := t.TempDir()
+
+	err := os.Symlink("missing", filepath.Join(volume, reports))
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = reportbuf.PrepareDataDir(volume, filepath.Join(volume, reports),
+		currentUser())
+	if err == nil {
+		t.Error("no error")
+	}
+
+	requireMissing(t, filepath.Join(volume, "missing"))
+}
+
+// TestPrepareDataDirGivesVolumeToOwner gives everything in the volume
+// to a uid and gid that own nothing, which only root can do. A
+// symbolic link in the volume to a directory outside it is given to
+// them itself; what it points to is left as it was.
+func TestPrepareDataDirGivesVolumeToOwner(t *testing.T) {
+	t.Parallel()
+
+	if os.Geteuid() != 0 {
+		t.Skip("only root can give files to another uid")
+	}
+
+	outside := t.TempDir()
+	volume := t.TempDir()
+	old := filepath.Join(volume, "old")
+
+	err := os.WriteFile(filepath.Join(outside, "f"), nil, 0o600)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = os.Mkdir(old, 0o700)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = os.WriteFile(filepath.Join(old, "f"), nil, 0o600)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = os.Symlink(outside, filepath.Join(old, "link"))
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = reportbuf.PrepareDataDir(volume, filepath.Join(volume, reports),
+		&user.User{Uid: "4242", Gid: "4343"})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	for _, path := range []string{
+		volume, filepath.Join(volume, reports), old,
+		filepath.Join(old, "f"), filepath.Join(old, "link"),
+	} {
+		requireOwner(t, path, 4242, 4343)
+	}
+
+	requireOwner(t, outside, 0, 0)
+	requireOwner(t, filepath.Join(outside, "f"), 0, 0)
+}

backend/internal/reportbuf/export_test.go

@@ -2,6 +2,10 @@ package reportbuf
 
 import "time"
 
+// FlushSizeThreshold exposes the buffer size at which Append starts
+// writing a report file to the external tests.
+const FlushSizeThreshold = flushSizeThreshold
+
 // Flush writes the buffered reports to a file now, as the periodic
 // flush does, so tests need not wait a minute for it.
 func (b *Buffer) Flush() error {

backend/internal/reportbuf/reportbuf_test.go

@@ -334,7 +334,11 @@ func TestTwoFlushesInOneMillisecond(t *testing.T) {
 		}
 	}
 
-	files := readReportFiles(t, dir)
+	files, err := readReportFiles(dir)
+	if err != nil {
+		t.Fatalf("read report files: %v", err)
+	}
+
 	if len(files) != flushes {
 		t.Fatalf("%d report files after %d flushes", len(files), flushes)
 	}
@@ -347,6 +351,88 @@ func TestTwoFlushesInOneMillisecond(t *testing.T) {
 	}
 }
 
+// TestReportFileHoldsTheLinesAppended flushes three reports and reads
+// their file back: it must decompress to exactly their JSON lines, in
+// the order they were appended.
+func TestReportFileHoldsTheLinesAppended(t *testing.T) {
+	dir := t.TempDir()
+	t.Setenv("DATA_DIR", dir)
+
+	buf := startBuffer(t)
+
+	for _, id := range []int{1, 2, 3} {
+		err := buf.Append(map[string]int{"id": id})
+		if err != nil {
+			t.Fatalf("append report %d: %v", id, err)
+		}
+	}
+
+	err := buf.Flush()
+	if err != nil {
+		t.Fatalf("flush: %v", err)
+	}
+
+	files, err := readReportFiles(dir)
+	if err != nil {
+		t.Fatalf("read report files: %v", err)
+	}
+
+	want := `{"id":1}` + "\n" + `{"id":2}` + "\n" + `{"id":3}` + "\n"
+	if len(files) != 1 || files[0] != want {
+		t.Fatalf("report files = %q, want one holding %q", files, want)
+	}
+}
+
+// TestFlushAtSizeThreshold appends reports until the buffer holds
+// FlushSizeThreshold bytes. The append that gets it there must write
+// them all to one report file, with no call to Flush and the periodic
+// flush a minute away, and no earlier append may write one.
+func TestFlushAtSizeThreshold(t *testing.T) {
+	dir := t.TempDir()
+	t.Setenv("DATA_DIR", dir)
+
+	buf := startBuffer(t)
+
+	// Large reports, so the threshold takes a few hundred appends.
+	pad := strings.Repeat("a", 64<<10)
+
+	var appended strings.Builder
+
+	for id := 0; appended.Len() < reportbuf.FlushSizeThreshold; id++ {
+		report := map[string]any{"id": id, "pad": pad}
+
+		err := buf.Append(report)
+		if err != nil {
+			t.Fatalf("append report %d: %v", id, err)
+		}
+
+		line, err := json.Marshal(report)
+		if err != nil {
+			t.Fatalf("marshal report %d: %v", id, err)
+		}
+
+		appended.Write(line)
+		appended.WriteByte('\n')
+	}
+
+	// Append writes the file in the background, so wait for it.
+	deadline := time.Now().Add(10 * time.Second)
+
+	for {
+		files, err := readReportFiles(dir)
+		if err == nil && len(files) == 1 && files[0] == appended.String() {
+			return
+		}
+
+		if time.Now().After(deadline) {
+			t.Fatalf("%d report files (error: %v), want one holding the "+
+				"%d bytes appended", len(files), err, appended.Len())
+		}
+
+		time.Sleep(10 * time.Millisecond)
+	}
+}
+
 // reportFilesBytes returns the total size of the report files in dir.
 func reportFilesBytes(t *testing.T, dir string) int64 {
 	t.Helper()
@@ -371,20 +457,19 @@ func reportFilesBytes(t *testing.T, dir string) int64 {
 }
 
 // readReportFiles returns the decompressed contents of each report
-// file in dir.
-func readReportFiles(t *testing.T, dir string) []string {
-	t.Helper()
-
+// file in dir. A file still being written does not decompress, so it
+// gives an error.
+func readReportFiles(dir string) ([]string, error) {
 	files := os.DirFS(dir)
 
 	names, err := fs.Glob(files, "reports-*.jsonl.zst")
 	if err != nil {
-		t.Fatalf("list report files: %v", err)
+		return nil, fmt.Errorf("list report files: %w", err)
 	}
 
 	dec, err := zstd.NewReader(nil)
 	if err != nil {
-		t.Fatalf("create zstd decoder: %v", err)
+		return nil, fmt.Errorf("create zstd decoder: %w", err)
 	}
 	defer dec.Close()
 
@@ -393,18 +478,18 @@ func readReportFiles(t *testing.T, dir string) []string {
 	for _, name := range names {
 		compressed, readErr := fs.ReadFile(files, name)
 		if readErr != nil {
-			t.Fatalf("read %s: %v", name, readErr)
+			return nil, fmt.Errorf("read %s: %w", name, readErr)
 		}
 
 		data, decErr := dec.DecodeAll(compressed, nil)
 		if decErr != nil {
-			t.Fatalf("decompress %s: %v", name, decErr)
+			return nil, fmt.Errorf("decompress %s: %w", name, decErr)
 		}
 
 		contents = append(contents, string(data))
 	}
 
-	return contents
+	return contents, nil
 }
 
 func writeBytes(t *testing.T, path string, n int) {

backend/script/test

@@ -1,12 +1,20 @@
 #!/bin/sh
-# script/test: run the backend test suite.
+# script/test: run the backend test suite with the race detector and
+# coverage. Go's own -timeout bounds the tests and not their compile,
+# so a cold build cache cannot fail it. The race detector needs cgo,
+# and so a C compiler. If the tests fail, they run again with -v for
+# the details, and the script fails even if that run passes.
 set -eu
 
 ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
 
 main() {
     cd "$ROOT"
-    timeout 30 go test ./...
+    go test -timeout 30s -race -cover ./... || {
+        echo "--- Rerunning with -v for details ---"
+        go test -timeout 30s -race -v ./...
+        exit 1
+    }
 }
 
 main "$@"

bin/entrypoint.sh

@@ -63,26 +63,13 @@ done > /etc/nginx/trusted-proxies.conf
 
 # netwatch-server keeps its report files in DATA_DIR, on the /data
 # volume, which may be a host directory owned by root or by another
-# uid. Both are given to the netwatch user here, with the mode the
-# server gives a directory it creates, so the host directory needs no
-# preparing.
-#
-# chown and chmod, run as root, change whatever a symbolic link on the
-# path points to, anywhere in the container, and the netwatch user can
-# put one in /data. So the start stops unless readlink -f, which
-# follows every link on a path, gives /data and DATA_DIR back as they
-# are. It also writes a path in full, so a DATA_DIR with '.', '..' or
-# an extra '/' in it is refused too.
+# uid. Here, as root, netwatch-server prepare-data-dir creates DATA_DIR
+# and gives /data and everything in it to the netwatch user, so the
+# host directory needs no preparing. It stops the start, naming
+# DATA_DIR, unless DATA_DIR is /data or a path below it, and it acts on
+# nothing outside /data, whatever symbolic links it meets there.
 export DATA_DIR="${DATA_DIR:-/data/reports}"
-mkdir -p "$DATA_DIR" || exit 1
-if [ "$(readlink -f /data)" != /data ] ||
-    [ "$(readlink -f "$DATA_DIR")" != "$DATA_DIR" ]; then
-    echo "entrypoint: DATA_DIR must be a full path with no '.', '..'," \
-        "extra '/' or symbolic link on it or on /data, not '$DATA_DIR'" >&2
-    exit 1
-fi
-chown -R netwatch:netwatch /data "$DATA_DIR" || exit 1
-chmod 750 /data "$DATA_DIR" || exit 1
+netwatch-server prepare-data-dir "$DATA_DIR" || exit 1
 
 # A stop signal is only noted here; the loop below acts on it.
 stop_requested=""

script/bootstrap

@@ -64,7 +64,8 @@ detect_pkgmgr() {
     fi
 }
 
-# pkg_install <nix-attr> <apt-pkg> <brew-formula> <apk-pkg>
+# pkg_install <nix-attr> <apt-pkgs> <brew-formula> <apk-pkgs>: the apt
+# and apk arguments may each list several packages, separated by spaces.
 pkg_install() {
     detect_pkgmgr
     case "$PKGMGR" in
@@ -74,10 +75,10 @@ pkg_install() {
                 $SUDO env DEBIAN_FRONTEND=noninteractive apt-get update
                 APT_UPDATED=1
             fi
-            $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2"
+            $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y $2
             ;;
         brew) brew install "$3" ;;
-        apk) apk add --no-cache "$4" ;;
+        apk) apk add --no-cache $4 ;;
     esac
 }
 
@@ -253,6 +254,12 @@ main() {
 
     if missing make; then pkg_install gnumake make make make; fi
     if missing git; then pkg_install git git git git; fi
+    # The race detector in make test needs cgo, which Go turns on only
+    # when it finds its C compiler, gcc on Linux. apt and apk ship the C
+    # library headers apart from gcc.
+    if missing gcc; then
+        pkg_install gcc "gcc libc6-dev" gcc "gcc musl-dev"
+    fi
 
     ensure_node
     ensure_yarn

script/test

@@ -1,14 +1,17 @@
 #!/bin/sh
 # script/test: run the test suite for the whole repo: the frontend at
-# the repo root, then the Go backend in backend/. Both halves together
-# get 30 seconds; each also keeps its own limit for the Dockerfiles.
+# the repo root, then the Go backend in backend/. Each half has its own
+# 30-second limit, and there is none around both: from a cold Go build
+# cache, compiling the backend's tests with the race detector can take
+# 30 seconds on its own, and Go's -timeout leaves the compile out.
 set -eu
 
 ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
 
 main() {
     cd "$ROOT"
-    timeout 30 sh -c 'script/frontend-test && backend/script/test'
+    script/frontend-test
+    backend/script/test
 }
 
 main "$@"