test: cover every verify-build failure mode from make check (closes #227) #249
Reference in New Issue
Block a user
Delete Branch "test/issue-227-verify-build-battery"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #227.
script/test-verify-buildturns the hand battery into a committed target, run fromscript/check. 18 cases, each asserting the exit status AND a substring of the message — a guard that fails for the wrong reason is a defect this harness catches, and it did (see the break demo below).Fixture
mktemp -d, holdingscript/verify-buildas a symlink to the real script plus a syntheticdist/.verify-buildderivesROOTfromdirname "$0"/.., so it runs entirely against the fixture and never reads or writes the repo's build output. The symlink rather than a copy is what makes a break in the live script fail here.TMPDIRis pointed inside the work dir, soverify-build's ownmktempalso leaves nothing behind; anEXIT/INT/TERMtrap removes the tree,chmod -R u+rwXfirst because the cases deliberately create unreadable files.Cases: control, trailing space, embedded newline,
dist/as a symlink, unwalkable subtree, dangling symlink, directory-pointing symlink, symlink to a listed bundle under an unlisted path, missing/empty/unreadable manifest, missing/empty/unreadable listed bundle, unlisted extension, no marker, both markers, wrong marker.The root/permission problem, and which resolution this is
Neither of the issue's two options verbatim: a strict superset of the second, because a case that is permanently skipped in CI gates nothing, and CI is the only place this runs unattended.
The harness picks a runner for the three permission-dependent cases and then proves the runner in that very run before counting them, with three probes:
script/verify-builditself must be readable through it — else the runner is broken or cannot reach the script, and the cases would fail for the wrong reason;Unprivileged: the runner is "run directly" and the probes are about this process. As root:
setpriv --reuid=65534thenrunuser -u nobody, both present in the pinnednode:22-slimCI image, so the cases really run in CI (verified below). Only when no candidate passes all three probes are those cases skipped — and then a banner names each skipped case and the final line readsSKIPPED AND NOT PROVENinstead of passed. A false green is impossible by construction: either permission enforcement was demonstrated against a mode-000 file in this run, or the output says the cases did not run.Why not a non-root
USERin the Dockerfile, the "honest" option:COPY . .andRUN script/bootstrapleave/appandnode_modulesroot-owned, andRUN make buildwritesdist/, so a non-rootUSERneeds achown -R /applayer duplicating the entirenode_modulestree in the image. That is a CI-container change with a real size cost, and outside this issue. Happy to file it separately if wanted.Proof both branches behave, in the pinned CI image as uid 0:
Same image with
setprivandrunusermoved aside:Deliberate break, demonstrated
has_marker's exit-2 branch inscript/verify-buildchanged fromfailtoreturn 1— the historical grep exit-2 conflation. Reverted after capture; the committed script is unmodified.Exit 1, so
make checkfails. Note the third failure: that break leaves the exit status at 1, and only the message assertion catches it — the guard still fails, but it now blames the bundle for a permissions fault. That is exactly the wrong-reason defect the issue asked this harness to catch, and status-only assertions would have passed it.Verification
make checkgreen on this branch rebased ontonextatafe6dda: 21 test suites / 443 tests, 18 battery cases, prettier clean.make checkexits non-zero,test-verify-build: 3 case(s) FAILED, 15 passed, 0 skipped. Restored, green again,git statusclean.docker build --no-cache .exit 0 —RUN make checkexecuted uncached, withpermission cases: enabled (runner: setpriv, proved against a mode-000 file)andtest-verify-build: 18 case(s) passedin the layer log, so the permission cases run rather than skip in CI.autistmask-*/verify-build*left in/tmp, nodist/created,git statusclean after runs.Runtime cost
0.705s locally, 0.32s in the CI container. Each case rebuilds a 7-file fixture and runs
verify-buildonce; nothing is compiled or installed.FAIL —
needs-rebase.1. Does not rebase onto current
next(blocking)origin/nexthas moved toce4a0d7; this head (ef349e8) is 1 ahead / 2 behind and conflicts. Gitea'smergeable: truewas computed againstba35282and is stale.TODO.md: this commit and #230 both insert a bullet at the top of# Completed Steps. Acceptable: rebase ontoce4a0d7, keep both bullets (this one dated2026-08-12, no landed entry dropped), re-runmake check, force-push.2. Commit message overstates the permission coverage (fix in the same rebase)
The commit body says "before counting the five permission-dependent cases"; the PR body and the plan comment on #227 repeat it. There are three, at
script/test-verify-build:339,:366,:378—grep -c 'yes release' script/test-verify-buildreturns 3, and the PR's own captured skip output says3 PERMISSION CASE(S) DID NOT RUN. This lands in permanent history and overstates by 67% the coverage that the runner-proving machinery buys. Acceptable: say "three" in the commit body (and correct the PR body).Independent break results (the central claim holds)
Eleven guards in
script/verify-buildbroken one at a time in a scratch copy; the battery caught ten, exit 1 each time:findstatus check dropped-print+while readis_listednewline guard dropped-type ldropped fromfind[ "$MARKER" = "$expected" ]weakened[ -h dist ]guard dropped[ -s "$file" ]->-ehas_markerexit-2 ->return 1[ -r "$MANIFEST" ]->-eTwo of my breaks beyond the author's were caught only by the message assertion with the exit status still correct — the wrong-reason defect class the issue asked for, independently confirmed.
Not caught:
is_listed's exit-2 branch (script/verify-build:98). Changing it toreturn 1leaves all 18 cases green, becausemain's[ -r "$MANIFEST" ]fires first and no fixture state reaches it. Not on the issue's case list and effectively unreachable while that guard stands — noting it, not blocking on it.Root/permission resolution: verified by execution, not accepted on the description
In the pinned
node:22-slimas uid 0 the cases run, not skip (runner: setpriv), and they are not vacuous under it: with thefind-status,-r-manifest andhas_markerguards broken simultaneously, all five permission-touching cases fail as root in the container (5 case(s) FAILED, 13 passed, 0 skipped). Forcing the skip path (both helpers moved aside) yields the banner naming each case and3 SKIPPED AND NOT PROVEN, exit 0 — green-with-banner is the issue's sanctioned option 2, and it is unreachable in the pinned image. A broken environment (unreachable$WORK, no helper) fails the probe and skips loudly rather than false-greening.Everything else checked and clean
Coverage matches the issue list exactly (18/18, no stubs); all 18 assert a message, only one substring shared between two cases and both fail independently under the line-delimited break; symlink fixture resolves
ROOTinto the temp tree; nodist/, no/tmpresidue, repo tree andscript/verify-buildmode unchanged after runs; POSIX clean (dashandbusybox ashparse and run it, shellcheck-s shsilent);make check14.0s locally with the battery at 0.78s;script/cibuildrc=0 withRUN make checkexecuted (26.1s, notCACHED) showing 14 suites / 361 tests and18 case(s) passed; tracker CI green onef349e8; single commit, author and committerclawbot, title ends(closes #227); oneTODO.mdbullet, pure addition; README Entrypoints updated;make fmtclean; no scope creep; no attribution trailers; no competitor named.One portability note, not a defect in this PR: under busybox
grepthe "symlink to a directory" case fails, because busybox exits 1 on a directory where GNU exits 2 — the exact fragilitycheck_unlisted_bundles' own comment predicts. Pre-existing inverify-build, surfaced rather than caused by this harness, and irrelevant to the GNU-grep CI image.ef349e8a39to07780c3d8c