Bootstrap keeps yarn at its pinned version and node at its pinned major (closes #154)

script/bootstrap kept any yarn on PATH, so YARN_VERSION applied
only where none was installed. It now keeps the yarn on PATH only when
yarn --version is exactly YARN_VERSION, and otherwise installs that
version. The node on PATH is used only when it has the pinned major
version, as REPO_POLICIES.md asks, and is at least NODE_MIN_VERSION.
After installing either, bootstrap looks the tool up through PATH
again, stops, naming the path, if that one fails the same check, and
prints the version it uses, as for Go. Every other host script that
runs yarn now goes through the new script/yarn, which uses the first
node on PATH, ~/.local/bin first, when it passes the same test, and
otherwise the pinned node in nvm.

Model: opus-5-5

README.md

@@ -207,7 +207,9 @@ stops. Its routes:
   address may send a limited number a minute, and the report files are capped in
   size, the oldest deleted first
 - `GET /.well-known/healthcheck` — answers 200 with `"status":"ok"`, the
-  server's version and its uptime
+  server's version and its uptime; while reports cannot be stored, as the last
+  write of a report file failed, it answers 503 with `"status":"error"` and the
+  same fields (see [Report storage](#report-storage))
 - `GET /metrics` — Prometheus metrics behind basic auth, only when
   `METRICS_USERNAME` and `METRICS_PASSWORD` are set; each client address may
   make a limited number of requests to it a minute
@@ -282,10 +284,11 @@ The reports of a failed write stay in memory, and the next flush, within a
 minute, tries again to write them. The server writes one file at a time, so a
 file holds its reports in the order they arrived, those of a failed write
 included. From the failed write until a write succeeds, `POST /api/v1/reports`
-refuses every report with 500 and keeps nothing of it. The page sends the
-refused samples again in its next report, as long as it still holds them: at the
-1-second interval one refused report already loses some of those in the history
-(see [Reporting](#reporting)). Reports already answered 200 are lost only if the
+refuses every report with 500 and keeps nothing of it, and the health check
+answers 503 with `"status":"error"`. The page sends the refused samples again in
+its next report, as long as it still holds them: at the 1-second interval one
+refused report already loses some of those in the history (see
+[Reporting](#reporting)). Reports already answered 200 are lost only if the
 server stops before a write succeeds; the final flush at stop tries them too,
 and if it fails, the process exits non-zero.
 
@@ -559,9 +562,16 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
       stops the container. Empty, the backend sends nothing to Sentry
 - **Health check:** the image's `HEALTHCHECK` requests
   `/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless
-  both nginx and the backend answer. upaas reads the container's health 60
-  seconds after a deploy and fails the deploy unless it is `healthy`. The
-  container also stops when either process exits.
+  both nginx and the backend answer. It also fails while reports cannot be
+  stored, as when the volume is full or `DATA_DIR` cannot be written: the health
+  check answers 503 from a failed write of a report file until a write succeeds,
+  and the backend tries the write again every minute. Three failed checks in a
+  row make the container `unhealthy`, and one that passes makes it `healthy`
+  again. upaas reads the container's health once, 60 seconds after a deploy, and
+  fails the deploy unless it is `healthy`. A container that turns `unhealthy`
+  after that is neither restarted nor reported by upaas, whose `unless-stopped`
+  restart policy ignores health; only `docker ps` and `docker inspect` show it.
+  The container also stops when either process exits.
 
 ## Browser Compatibility
 

TODO.md

@@ -35,6 +35,22 @@ Run `make frontend-viewport-test` in CI as its own step; it is not part of
   script puts `~/.local/bin` first on `PATH`, uses the first `node` on that
   `PATH` when it passes the same test, and otherwise runs yarn under the pinned
   node in nvm; `make build` ran yarn under whatever node was first on `PATH`
+- 2026-10-19: `bin/entrypoint.sh` has each of its steps in a function, and a
+  `main` at the end calls them in order, as the shell section of the code style
+  guide asks ([#163](https://git.eeqj.de/sneak/netwatch/issues/163)). It is
+  still POSIX sh with no `set -e`, and does what it did, with the same messages
+  and exit codes. Checked by running the image with each script: a normal start,
+  bad `PORT`, `TRUSTED_PROXIES` and `DATA_DIR` values, `docker stop`, INT, and
+  nginx exiting on its own
+- 2026-10-19: while report files cannot be written, the health check answers 503
+  with `"status":"error"` and the same fields, so the image's `HEALTHCHECK`
+  reports the container `unhealthy`
+  ([#142](https://git.eeqj.de/sneak/netwatch/issues/142)). It answered 200
+  `"status":"ok"` whatever happened, so the platform saw a healthy container
+  while every report was refused with 500. It answers 200 at start, before any
+  write has failed, and again from the first write that succeeds. A unit test
+  removes `DATA_DIR` to make a write fail and checks the 503, then puts it back
+  and checks the 200 after the next write
 - 2026-10-19: report file writes no longer overlap
   ([#150](https://git.eeqj.de/sneak/netwatch/issues/150)), so the reports of
   failed writes are written in the order they arrived. Two writes that failed at

bin/entrypoint.sh

@@ -13,24 +13,27 @@ set -u
 # address and start anyway, and reports a bad port without naming
 # PORT, so a value that is not a usable port stops the container here,
 # before either process starts.
-export PORT="${PORT:-8080}"
-case "$PORT" in
-    *[!0-9]*)
-        echo "entrypoint: PORT must be a port number, not '$PORT'" >&2
+check_port() {
+    export PORT="${PORT:-8080}"
+    case "$PORT" in
+        *[!0-9]*)
+            echo "entrypoint: PORT must be a port number, not '$PORT'" >&2
+            exit 1
+            ;;
+    esac
+    # The length is checked first because, for a number too big for it,
+    # the shell's test prints an error and is false, so the range checks
+    # alone would let it through.
+    if [ "${#PORT}" -gt 5 ] || [ "$PORT" -lt 1 ] || [ "$PORT" -gt 65535 ]; then
+        echo "entrypoint: PORT must be from 1 to 65535, not '$PORT'" >&2
         exit 1
-        ;;
-esac
-# The length is checked first because, for a number too big for it,
-# the shell's test prints an error and is false, so the range checks
-# alone would let it through.
-if [ "${#PORT}" -gt 5 ] || [ "$PORT" -lt 1 ] || [ "$PORT" -gt 65535 ]; then
-    echo "entrypoint: PORT must be from 1 to 65535, not '$PORT'" >&2
-    exit 1
-fi
-if [ "$PORT" -eq 8081 ]; then
-    echo "entrypoint: PORT cannot be 8081, netwatch-server listens there" >&2
-    exit 1
-fi
+    fi
+    if [ "$PORT" -eq 8081 ]; then
+        echo "entrypoint: PORT cannot be 8081, netwatch-server" \
+            "listens there" >&2
+        exit 1
+    fi
+}
 
 # TRUSTED_PROXIES names the reverse proxies in front of the container,
 # as IP addresses or CIDRs separated by commas. nginx takes the client
@@ -47,21 +50,23 @@ fi
 # TRUSTED_PROXIES. Its error, naming the CIDR, is dropped for the one
 # below, naming the entry as written. set -f keeps a * in an entry from
 # becoming a list of file names.
-TRUSTED_PROXIES="${TRUSTED_PROXIES-10.0.0.0/8,172.16.0.0/12,192.168.0.0/16}"
-set -f
-for proxy in $(printf '%s' "$TRUSTED_PROXIES" | tr ',' ' '); do
-    case "$proxy" in
-        */*) cidr="$proxy" ;;
-        *:*) cidr="$proxy/128" ;;
-        *) cidr="$proxy/32" ;;
-    esac
-    if ! netwatch-server check-cidr "$cidr" 2> /dev/null; then
-        echo "entrypoint: TRUSTED_PROXIES must be IP addresses or CIDRs" \
-            "separated by commas; '$proxy' is neither" >&2
-        exit 1
-    fi
-    echo "set_real_ip_from $cidr;"
-done > /etc/nginx/trusted-proxies.conf
+write_trusted_proxies() {
+    TRUSTED_PROXIES="${TRUSTED_PROXIES-10.0.0.0/8,172.16.0.0/12,192.168.0.0/16}"
+    set -f
+    for proxy in $(printf '%s' "$TRUSTED_PROXIES" | tr ',' ' '); do
+        case "$proxy" in
+            */*) cidr="$proxy" ;;
+            *:*) cidr="$proxy/128" ;;
+            *) cidr="$proxy/32" ;;
+        esac
+        if ! netwatch-server check-cidr "$cidr" 2> /dev/null; then
+            echo "entrypoint: TRUSTED_PROXIES must be IP addresses or CIDRs" \
+                "separated by commas; '$proxy' is neither" >&2
+            exit 1
+        fi
+        echo "set_real_ip_from $cidr;"
+    done > /etc/nginx/trusted-proxies.conf
+}
 
 # netwatch-server keeps its report files in DATA_DIR, on the /data
 # volume, which may be a host directory owned by root or by another
@@ -73,12 +78,10 @@ done > /etc/nginx/trusted-proxies.conf
 # naming DATA_DIR, unless DATA_DIR is /data or a path below it, and it
 # acts on nothing outside /data, whatever symbolic or hard links it
 # meets there.
-export DATA_DIR="${DATA_DIR:-/data/reports}"
-netwatch-server prepare-data-dir "$DATA_DIR" || exit 1
-
-# A stop signal is only noted here; the loop below acts on it.
-stop_requested=""
-trap 'stop_requested=yes' TERM INT
+prepare_data_dir() {
+    export DATA_DIR="${DATA_DIR:-/data/reports}"
+    netwatch-server prepare-data-dir "$DATA_DIR" || exit 1
+}
 
 # netwatch-server runs as the netwatch user and listens on loopback
 # only, on a port other than the public one; configs/nginx.conf proxies
@@ -87,9 +90,11 @@ trap 'stop_requested=yes' TERM INT
 # the container has. The netwatch user has no login shell, hence -s
 # /bin/sh. busybox su replaces itself with the command instead of
 # staying on as its parent, so $! is the server's own PID.
-BIND_ADDRESS=127.0.0.1 PORT=8081 TRUSTED_PROXIES=127.0.0.1/32 \
-    su -s /bin/sh netwatch -c 'exec netwatch-server' &
-backend=$!
+start_backend() {
+    BIND_ADDRESS=127.0.0.1 PORT=8081 TRUSTED_PROXIES=127.0.0.1/32 \
+        su -s /bin/sh netwatch -c 'exec netwatch-server' &
+    backend=$!
+}
 
 # nginx starts through the nginx image's own entrypoint, which applies
 # the image's start-up configuration and then replaces itself with
@@ -98,9 +103,11 @@ backend=$!
 # that rendering to PORT: a variable nginx itself uses, such as $uri,
 # would otherwise be replaced by an environment variable of the same
 # name.
-NGINX_ENVSUBST_FILTER='^PORT$' \
-    /docker-entrypoint.sh nginx -g 'daemon off;' &
-nginx=$!
+start_nginx() {
+    NGINX_ENVSUBST_FILTER='^PORT$' \
+        /docker-entrypoint.sh nginx -g 'daemon off;' &
+    nginx=$!
+}
 
 running() {
     kill -0 "$1" 2>/dev/null
@@ -109,26 +116,47 @@ running() {
 # POSIX sh cannot wait for whichever of two children exits first, so
 # look once a second. The shell collects a child that has exited while
 # it runs sleep, and running() is false for that child from then on.
-while [ -z "$stop_requested" ] && running "$backend" && running "$nginx"; do
-    sleep 1
-done
+wait_for_stop_or_exit() {
+    while [ -z "$stop_requested" ] && running "$backend" && running "$nginx"; do
+        sleep 1
+    done
+}
 
 # Stop both, then wait until neither is left.
-kill -TERM "$backend" "$nginx" 2>/dev/null
-while running "$backend" || running "$nginx"; do
-    sleep 1
-done
-
-wait "$backend"
-backend_status=$?
-wait "$nginx"
-nginx_status=$?
-echo "entrypoint: netwatch-server exited $backend_status," \
-    "nginx exited $nginx_status"
+stop_both() {
+    kill -TERM "$backend" "$nginx" 2>/dev/null
+    while running "$backend" || running "$nginx"; do
+        sleep 1
+    done
+}
 
 # Success is a requested stop that both processes exited cleanly from.
-if [ -n "$stop_requested" ] && [ "$backend_status" -eq 0 ] &&
-    [ "$nginx_status" -eq 0 ]; then
-    exit 0
-fi
-exit 1
+exit_with_status() {
+    wait "$backend"
+    backend_status=$?
+    wait "$nginx"
+    nginx_status=$?
+    echo "entrypoint: netwatch-server exited $backend_status," \
+        "nginx exited $nginx_status"
+    if [ -n "$stop_requested" ] && [ "$backend_status" -eq 0 ] &&
+        [ "$nginx_status" -eq 0 ]; then
+        exit 0
+    fi
+    exit 1
+}
+
+main() {
+    check_port
+    write_trusted_proxies
+    prepare_data_dir
+    # A stop signal is only noted here; wait_for_stop_or_exit acts on it.
+    stop_requested=""
+    trap 'stop_requested=yes' TERM INT
+    start_backend
+    start_nginx
+    wait_for_stop_or_exit
+    stop_both
+    exit_with_status
+}
+
+main "$@"

internal/handlers/healthcheck.go

@@ -3,9 +3,17 @@ package handlers
 import "net/http"
 
 // HandleHealthCheck returns a handler for the health check
-// endpoint.
+// endpoint. It answers 200 while the status is "ok" and 503 otherwise,
+// so that the image's HEALTHCHECK fails.
 func (s *Handlers) HandleHealthCheck() http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
-		s.respondJSON(w, r, s.hc.Healthcheck(), http.StatusOK)
+		resp := s.hc.Healthcheck()
+
+		code := http.StatusOK
+		if resp.Status != "ok" {
+			code = http.StatusServiceUnavailable
+		}
+
+		s.respondJSON(w, r, resp, code)
 	}
 }

internal/handlers/healthcheck_test.go

@@ -9,17 +9,19 @@ import (
 	"testing"
 	"time"
 
+	"sneak.berlin/go/netwatch/internal/config"
 	"sneak.berlin/go/netwatch/internal/globals"
 	"sneak.berlin/go/netwatch/internal/handlers"
 	"sneak.berlin/go/netwatch/internal/healthcheck"
 	"sneak.berlin/go/netwatch/internal/logger"
+	"sneak.berlin/go/netwatch/internal/reportbuf"
 
 	"go.uber.org/fx/fxtest"
 )
 
 // newStartedHandlers builds Handlers with a real health check for the
-// server named in g, and starts them, which records the time the
-// uptime counts from.
+// server named in g, and a report buffer in a temporary directory, and
+// starts them, which records the time the uptime counts from.
 func newStartedHandlers(t *testing.T, g *globals.Globals) *handlers.Handlers {
 	t.Helper()
 
@@ -30,14 +32,23 @@ func newStartedHandlers(t *testing.T, g *globals.Globals) *handlers.Handlers {
 		t.Fatalf("logger: %v", err)
 	}
 
+	buf, err := reportbuf.New(lc, reportbuf.Params{
+		Config: &config.Config{DataDir: t.TempDir(), DataDirMaxBytes: 1 << 20},
+		Logger: log,
+	})
+	if err != nil {
+		t.Fatalf("report buffer: %v", err)
+	}
+
 	hc, err := healthcheck.New(lc,
-		healthcheck.Params{Globals: g, Logger: log})
+		healthcheck.Params{Buffer: buf, Globals: g, Logger: log})
 	if err != nil {
 		t.Fatalf("health check: %v", err)
 	}
 
-	h, err := handlers.New(lc,
-		handlers.Params{Globals: g, Healthcheck: hc, Logger: log})
+	h, err := handlers.New(lc, handlers.Params{
+		Buffer: buf, Globals: g, Healthcheck: hc, Logger: log,
+	})
 	if err != nil {
 		t.Fatalf("handlers: %v", err)
 	}
@@ -48,10 +59,12 @@ func newStartedHandlers(t *testing.T, g *globals.Globals) *handlers.Handlers {
 	return h
 }
 
-// TestHandleHealthCheck checks the health check's answer: 200, a JSON
-// content type, and a JSON object with exactly the fields of
-// healthcheck.HealthcheckResponse, carrying this server's name and
-// version and an uptime counted from its start.
+// TestHandleHealthCheck checks the health check's answer at start, when
+// no write of a report file has failed: 200, a JSON content type, and a
+// JSON object with exactly the fields of healthcheck.HealthcheckResponse,
+// carrying "status":"ok", this server's name and version and an uptime
+// counted from its start. The answer while writes fail is checked in
+// internal/reportbuf, whose tests can make a write fail.
 func TestHandleHealthCheck(t *testing.T) {
 	t.Parallel()
 

internal/healthcheck/healthcheck.go

@@ -10,6 +10,7 @@ import (
 	"sneak.berlin/go/netwatch/internal/config"
 	"sneak.berlin/go/netwatch/internal/globals"
 	"sneak.berlin/go/netwatch/internal/logger"
+	"sneak.berlin/go/netwatch/internal/reportbuf"
 
 	"go.uber.org/fx"
 )
@@ -18,6 +19,7 @@ import (
 type Params struct {
 	fx.In
 
+	Buffer  *reportbuf.Buffer
 	Config  *config.Config
 	Globals *globals.Globals
 	Logger  *logger.Logger
@@ -68,12 +70,18 @@ func New(
 	return s, nil
 }
 
-// Healthcheck returns the current health status of the application.
+// Healthcheck returns the current health status of the application:
+// "ok", or "error" while report files cannot be written.
 func (s *Healthcheck) Healthcheck() *HealthcheckResponse {
+	status := "ok"
+	if s.params.Buffer.LastWriteFailed() {
+		status = "error"
+	}
+
 	return &HealthcheckResponse{
 		Appname:       s.params.Globals.Appname,
 		Now:           time.Now().UTC().Format(time.RFC3339Nano),
-		Status:        "ok",
+		Status:        status,
 		UptimeHuman:   s.uptime().String(),
 		UptimeSeconds: int64(s.uptime().Seconds()),
 		Version:       s.params.Globals.Version,

internal/reportbuf/reportbuf.go

@@ -255,6 +255,16 @@ func (b *Buffer) Append(v any) error {
 	return nil
 }
 
+// LastWriteFailed reports whether the last write of a report file to
+// finish failed: false at start until a write fails, and false again
+// once a write succeeds. While it is true, Append refuses reports.
+func (b *Buffer) LastWriteFailed() bool {
+	b.mu.Lock()
+	defer b.mu.Unlock()
+
+	return b.writeErr != nil
+}
+
 // deleteOldestFiles deletes report files, oldest first, until n more
 // bytes fit under maxBytes. It deletes none when the reports waiting
 // to be written leave no room for n even with every file gone, since

internal/reportbuf/reportbuf_test.go

@@ -1074,6 +1074,51 @@ func TestReportsRefusedWhileWritesFail(t *testing.T) {
 	}
 }
 
+// TestHealthCheckFailsWhileWritesFail asks the health check handler
+// for its answer around a failed write. It answers 200 with
+// "status":"ok" at start, 503 with "status":"error" from the failed
+// write until a write succeeds, and 200 with "status":"ok" again after.
+func TestHealthCheckFailsWhileWritesFail(t *testing.T) {
+	dir := t.TempDir()
+	t.Setenv("DATA_DIR", dir)
+
+	buf, h := startBufferWithHandlers(t)
+
+	checkHealthCheck(t, h, "at start", http.StatusOK, "ok")
+
+	// A flush writes a file only when a report is waiting.
+	err := buf.Append(map[string]string{"id": "kept"})
+	if err != nil {
+		t.Fatalf("append: %v", err)
+	}
+
+	// With the data directory gone, the write has nowhere to go.
+	err = os.Remove(dir)
+	if err != nil {
+		t.Fatalf("remove data dir: %v", err)
+	}
+
+	err = buf.Flush()
+	if err == nil {
+		t.Fatal("flush succeeded with the data directory gone")
+	}
+
+	checkHealthCheck(t, h, "after the failed write",
+		http.StatusServiceUnavailable, "error")
+
+	err = os.Mkdir(dir, 0o750)
+	if err != nil {
+		t.Fatalf("make data dir: %v", err)
+	}
+
+	err = buf.Flush()
+	if err != nil {
+		t.Fatalf("flush with the data directory back: %v", err)
+	}
+
+	checkHealthCheck(t, h, "after a write succeeded", http.StatusOK, "ok")
+}
+
 // startBufferWithHandlers starts a Buffer and the Handlers that store
 // reports in it through fx, as main does, with the DATA_DIR the calling
 // test has set.
@@ -1121,6 +1166,39 @@ func postReport(t *testing.T, h *handlers.Handlers, clientID string) int {
 	return rec.Code
 }
 
+// checkHealthCheck asks the health check handler for its answer, and
+// fails the test, saying when it asked, unless the answer has status
+// code wantCode and a "status" field of wantStatus.
+func checkHealthCheck(
+	t *testing.T,
+	h *handlers.Handlers,
+	when string,
+	wantCode int,
+	wantStatus string,
+) {
+	t.Helper()
+
+	rec := httptest.NewRecorder()
+	req := httptest.NewRequestWithContext(t.Context(),
+		http.MethodGet, "/.well-known/healthcheck", http.NoBody)
+	h.HandleHealthCheck().ServeHTTP(rec, req)
+
+	var body struct {
+		Status string `json:"status"`
+	}
+
+	err := json.Unmarshal(rec.Body.Bytes(), &body)
+	if err != nil {
+		t.Fatalf("health check %s: body not JSON: %v (%q)",
+			when, err, rec.Body.String())
+	}
+
+	if rec.Code != wantCode || body.Status != wantStatus {
+		t.Fatalf("health check %s: %d with status %q, want %d with %q",
+			when, rec.Code, body.Status, wantCode, wantStatus)
+	}
+}
+
 // TestConcurrentAppendsStopAtCap appends from many goroutines at once
 // with room for exactly roomFor reports: exactly that many must be
 // taken, which holds only if Append checks and counts each report