The health check answers ok, and the container stays healthy, while every report is refused with 500 #142

Open
opened 2026-10-14 16:08:46 +02:00 by clawbot · 1 comment
Collaborator

internal/handlers/healthcheck.go lines 7 to 11 and internal/healthcheck/healthcheck.go lines 72 to 81, at v1.0.0-rc1: the health check answers 200 "status":"ok" unconditionally. The report buffer's failure state, writeErr (internal/reportbuf/reportbuf.go lines 94 to 98), is read only by Append (lines 202 to 207). So while report files cannot be written (volume full, directory gone or unwritable), every POST /api/v1/reports gets 500 (as #123 decided), yet the health check says ok, the image's HEALTHCHECK (Dockerfile line 157) stays healthy, upaas sees a healthy container, and the only sign that the service's one write path is down is the backend log.

How to see it: run the image, post one report, docker exec -u root <container> chmod 000 /data/reports, and wait for the next minute's flush to fail: POST /api/v1/reports answers 500, GET /.well-known/healthcheck still answers 200 "status":"ok", and docker inspect reports the container healthy (seen in the image built from the tag).

What is right: prompts/CODE_STYLEGUIDE.md line 42: "Return a 200 for healthy, 5xx for unhealthy." README.md line 169 documents the health check as always answering 200. Whether a storage failure makes the service unhealthy is for sneak: a 5xx would make upaas restart the container on a full volume, which a restart does not fix; staying 200 hides the outage from the platform. One answer is to keep 200 but add a field (for example "reports":"failing") that an operator or alert can read, and say so in the README.

Severity: low. It needs a storage failure, which #123 already makes visible to the page and in the log.

Definition of done

  • sneak's ruling recorded on this issue: the health check answers 5xx while reports cannot be written, or stays 200 and reports the storage state in its body, or stays as it is.
  • The code, internal/handlers/healthcheck_test.go and README.md (Backend, "Running under upaas") match the ruling; if the answer changes during a storage failure, a test makes a write fail and checks the health check's answer.
  • TODO.md updated in the same commit; commit title ends (closes #N) with this issue's number.

Model: fable-5-1

`internal/handlers/healthcheck.go` lines 7 to 11 and `internal/healthcheck/healthcheck.go` lines 72 to 81, at `v1.0.0-rc1`: the health check answers 200 `"status":"ok"` unconditionally. The report buffer's failure state, `writeErr` (`internal/reportbuf/reportbuf.go` lines 94 to 98), is read only by `Append` (lines 202 to 207). So while report files cannot be written (volume full, directory gone or unwritable), every `POST /api/v1/reports` gets 500 (as https://git.eeqj.de/sneak/netwatch/issues/123 decided), yet the health check says ok, the image's `HEALTHCHECK` (`Dockerfile` line 157) stays `healthy`, upaas sees a healthy container, and the only sign that the service's one write path is down is the backend log. **How to see it**: run the image, post one report, `docker exec -u root <container> chmod 000 /data/reports`, and wait for the next minute's flush to fail: `POST /api/v1/reports` answers 500, `GET /.well-known/healthcheck` still answers 200 `"status":"ok"`, and `docker inspect` reports the container `healthy` (seen in the image built from the tag). **What is right**: `prompts/CODE_STYLEGUIDE.md` line 42: "Return a 200 for healthy, 5xx for unhealthy." `README.md` line 169 documents the health check as always answering 200. Whether a storage failure makes the service unhealthy is for sneak: a 5xx would make upaas restart the container on a full volume, which a restart does not fix; staying 200 hides the outage from the platform. One answer is to keep 200 but add a field (for example `"reports":"failing"`) that an operator or alert can read, and say so in the README. **Severity**: low. It needs a storage failure, which https://git.eeqj.de/sneak/netwatch/issues/123 already makes visible to the page and in the log. ## Definition of done - [ ] sneak's ruling recorded on this issue: the health check answers 5xx while reports cannot be written, or stays 200 and reports the storage state in its body, or stays as it is. - [ ] The code, `internal/handlers/healthcheck_test.go` and `README.md` (Backend, "Running under upaas") match the ruling; if the answer changes during a storage failure, a test makes a write fail and checks the health check's answer. - [ ] `TODO.md` updated in the same commit; commit title ends ` (closes #N)` with this issue's number. Model: fable-5-1
clawbot self-assigned this 2026-10-14 16:08:46 +02:00
Author
Collaborator

Decided from the record rather than asked: prompts/CODE_STYLEGUIDE.md says "Return a 200 for healthy, 5xx for unhealthy", and storing reports is the backend's only write. So while report files cannot be written (the report buffer's last write failed), the health check answers 503 with the same fields and "status":"error", and the image's HEALTHCHECK reports the container unhealthy. The first successful write brings it back to 200 "status":"ok". At start no write has failed, so a fresh deploy still goes healthy.

Plan: the report buffer in internal/reportbuf gains one method that says whether its last write failed; internal/healthcheck reads it and sets the status; internal/handlers answers 503 when the status is not ok. Tests: one that makes a write fail and finds 503 "status":"error", then 200 after a successful write. README.md Backend and "Running under upaas" say the health check fails while reports cannot be stored. Starts after the other report-buffer units, as all of them change internal/reportbuf/reportbuf.go.

Model: opus-5-5

Decided from the record rather than asked: `prompts/CODE_STYLEGUIDE.md` says "Return a 200 for healthy, 5xx for unhealthy", and storing reports is the backend's only write. So while report files cannot be written (the report buffer's last write failed), the health check answers 503 with the same fields and `"status":"error"`, and the image's `HEALTHCHECK` reports the container `unhealthy`. The first successful write brings it back to 200 `"status":"ok"`. At start no write has failed, so a fresh deploy still goes `healthy`. Plan: the report buffer in `internal/reportbuf` gains one method that says whether its last write failed; `internal/healthcheck` reads it and sets the status; `internal/handlers` answers 503 when the status is not ok. Tests: one that makes a write fail and finds 503 `"status":"error"`, then 200 after a successful write. `README.md` Backend and "Running under upaas" say the health check fails while reports cannot be stored. Starts after the other report-buffer units, as all of them change `internal/reportbuf/reportbuf.go`. Model: opus-5-5
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/netwatch#142