Health check answers 503 while report files cannot be written (closes #142)

The health check answered 200 "status":"ok" whatever happened, so a
container whose report files could not be written stayed healthy while
every report was refused with 500. The report buffer now says whether
its last write of a report file failed. While it has, the health
check's status is "error" and the handler answers 503 with the same
fields, which fails the image's HEALTHCHECK. At start no write has
failed, and the first write that succeeds brings back 200. A test
removes DATA_DIR to make a write fail and checks the 503, then the 200
once a write succeeds again.

Model: opus-5-5

README.md

@@ -199,7 +199,9 @@ stops. Its routes:
   address may send a limited number a minute, and the report files are capped in
   size, the oldest deleted first
 - `GET /.well-known/healthcheck` — answers 200 with `"status":"ok"`, the
-  server's version and its uptime
+  server's version and its uptime; while reports cannot be stored, as the last
+  write of a report file failed, it answers 503 with `"status":"error"` and the
+  same fields (see [Report storage](#report-storage))
 - `GET /metrics` — Prometheus metrics behind basic auth, only when
   `METRICS_USERNAME` and `METRICS_PASSWORD` are set; each client address may
   make a limited number of requests to it a minute
@@ -274,10 +276,11 @@ The reports of a failed write stay in memory, and the next flush, within a
 minute, tries again to write them. The server writes one file at a time, so a
 file holds its reports in the order they arrived, those of a failed write
 included. From the failed write until a write succeeds, `POST /api/v1/reports`
-refuses every report with 500 and keeps nothing of it. The page sends the
-refused samples again in its next report, as long as it still holds them: at the
-1-second interval one refused report already loses some of those in the history
-(see [Reporting](#reporting)). Reports already answered 200 are lost only if the
+refuses every report with 500 and keeps nothing of it, and the health check
+answers 503 with `"status":"error"`. The page sends the refused samples again in
+its next report, as long as it still holds them: at the 1-second interval one
+refused report already loses some of those in the history (see
+[Reporting](#reporting)). Reports already answered 200 are lost only if the
 server stops before a write succeeds; the final flush at stop tries them too,
 and if it fails, the process exits non-zero.
 
@@ -551,9 +554,16 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for netwatch needs:
       stops the container. Empty, the backend sends nothing to Sentry
 - **Health check:** the image's `HEALTHCHECK` requests
   `/.well-known/healthcheck` through nginx every 30 seconds, so it fails unless
-  both nginx and the backend answer. upaas reads the container's health 60
-  seconds after a deploy and fails the deploy unless it is `healthy`. The
-  container also stops when either process exits.
+  both nginx and the backend answer. It also fails while reports cannot be
+  stored, as when the volume is full or `DATA_DIR` cannot be written: the health
+  check answers 503 from a failed write of a report file until a write succeeds,
+  and the backend tries the write again every minute. Three failed checks in a
+  row make the container `unhealthy`, and one that passes makes it `healthy`
+  again. upaas reads the container's health once, 60 seconds after a deploy, and
+  fails the deploy unless it is `healthy`. A container that turns `unhealthy`
+  after that is neither restarted nor reported by upaas, whose `unless-stopped`
+  restart policy ignores health; only `docker ps` and `docker inspect` show it.
+  The container also stops when either process exits.
 
 ## Browser Compatibility