CI can report success without running lint or tests, because script/cibuild replays a cached image #119

New Issue

2026-08-11T14:38:28+02:00

clawbot commented

2026-08-11 14:38:28 +02:00

The Gitea workflow runs script/cibuild, which is a plain docker build .. On a runner with a warm layer cache, the make fmt-check / make lint / make test / make build layers replay from cache and the build exits 0 without executing any of them.

Evidence, from two independent reviews today: a plain script/cibuild completed in 1.451s, exit 0, every stage cached. Historic green checks on this repo report "Successful in 3-4s", which is the same thing — those runs proved nothing. Real runs take 3-5 minutes.

Why it matters: next and main are required to stay green, and CI is the automated guard on that. A gate that can pass without running is not a gate. Every merge today was validated by cache-defeated runs done by hand instead; that should not be the only thing standing between a red branch and main.

Note .dockerignore excludes *.md, so docs-only commits legitimately hit cache in full — that part is fine and should stay.

Definition of done

A CI run executes the lint and test stages for any commit that changes code, and cannot report success by replaying them from cache.
A deliberately broken test on a scratch branch makes CI go red. This is the acceptance check — without it, the fix is unverified.
Docs-only commits need not pay for a full uncached build.

Implementation notes

Scoped invalidation (docker build --no-cache-filter=lint,builder) is what the reviews used to get real signal, and is cheaper than --no-cache.
Related but distinct: #106 (script/lint runs the host golangci-lint, so make check is not authoritative for lint) and #109 (run all linting in Docker). Fixing those does not fix this one.
Branch from next, PR based on next, single commit, title ending (closes #N).

The Gitea workflow runs `script/cibuild`, which is a plain `docker build .`. On a runner with a warm layer cache, the `make fmt-check` / `make lint` / `make test` / `make build` layers replay from cache and the build exits 0 without executing any of them. Evidence, from two independent reviews today: a plain `script/cibuild` completed in **1.451s, exit 0, every stage cached**. Historic green checks on this repo report "Successful in 3-4s", which is the same thing — those runs proved nothing. Real runs take 3-5 minutes. Why it matters: `next` and `main` are required to stay green, and CI is the automated guard on that. A gate that can pass without running is not a gate. Every merge today was validated by cache-defeated runs done by hand instead; that should not be the only thing standing between a red branch and `main`. Note `.dockerignore` excludes `*.md`, so docs-only commits legitimately hit cache in full — that part is fine and should stay. ## Definition of done - A CI run executes the lint and test stages for any commit that changes code, and cannot report success by replaying them from cache. - A deliberately broken test on a scratch branch makes CI go red. This is the acceptance check — without it, the fix is unverified. - Docs-only commits need not pay for a full uncached build. ## Implementation notes - Scoped invalidation (`docker build --no-cache-filter=lint,builder`) is what the reviews used to get real signal, and is cheaper than `--no-cache`. - Related but distinct: https://git.eeqj.de/sneak/webhooker/issues/106 (`script/lint` runs the host golangci-lint, so `make check` is not authoritative for lint) and https://git.eeqj.de/sneak/webhooker/issues/109 (run all linting in Docker). Fixing those does not fix this one. - Branch from `next`, PR based on `next`, single commit, title ending ` (closes #N)`.

clawbot commented

2026-08-11 15:11:42 +02:00

Second failure mode, same root: consecutive merges cancel each other's runs, and a cancelled run is recorded as failure.

Measured on next just now:

e50a79c, 15a6117, d51cd0f — failure, description Has been cancelled
84b758b (current head) — success, description Successful in 5s

So the three intermediate commits read as red despite being green, and the one that reads green ran nothing. Net: no commit on next today has been genuinely validated by CI. The gate is currently held by hand-run cache-defeated Docker builds.

This matters beyond tidiness — next is supposed to be mergeable to main without notice, and its history currently shows red markers that do not mean what they appear to mean.

Add to the definition of done: a superseded run must not leave a failure status on a commit that was never actually tested — either let runs complete per-commit, or record cancellation as neutral rather than failed.

Second failure mode, same root: consecutive merges cancel each other's runs, and a cancelled run is recorded as `failure`. Measured on `next` just now: - `e50a79c`, `15a6117`, `d51cd0f` — `failure`, description `Has been cancelled` - `84b758b` (current head) — `success`, description `Successful in 5s` So the three intermediate commits read as red despite being green, and the one that reads green ran nothing. Net: no commit on `next` today has been genuinely validated by CI. The gate is currently held by hand-run cache-defeated Docker builds. This matters beyond tidiness — `next` is supposed to be mergeable to `main` without notice, and its history currently shows red markers that do not mean what they appear to mean. Add to the definition of done: a superseded run must not leave a `failure` status on a commit that was never actually tested — either let runs complete per-commit, or record cancellation as neutral rather than failed.