Hash-pin every external reference in .gitea/workflows/deploy.yml #7
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
.gitea/workflows/deploy.ymlis the only file in the repo that still usesmutable external references.
REPO_POLICIES.mdcalls hash-pinning "the singlemost important rule in this document" and states there are "zero exceptions".
.gitea/workflows/check.yml, theDockerfile, and everything else in the orgare already pinned; this file was written before the scaffold landed in #4 and
was never brought up to standard.
Five mutable references, plus one unpinned remote install:
build.container.imageklakegg/hugo:ext-alpinebuild.stepsactions/checkout@v4build.stepsactions/upload-artifact@v3deploy.container.imagenode:20deploy.stepsactions/download-artifact@v3deploy.stepsnpm install -g wranglerTwo further divergences in the same file:
on.push.brancheslist still includesfeat/initial-site, a branch thatis fully merged into
main(0 unique commits) and is scheduled for deletion.A workflow trigger on a dead branch is dead config.
.editorconfig(canonical,[*] indent_size = 4) and the canonical.gitea/workflows/check.ymlboth use4-space.
check.ymlin this repo already uses 4-space, so the two workflowfiles are inconsistent with each other.
Definition of done
image:indeploy.ymlis referenced asimage@sha256:<digest>,with a comment on the line above in the canonical format
# <name> <version>, YYYY-MM-DD.uses:indeploy.ymlis referenced by a full 40-hex commit SHA,with the same comment format.
upload-artifactanddownload-artifactmove from v3 to v4 (v3 is deprecated) and are pinned to v4's commit SHA.
klakegg/hugo:ext-alpineis replaced. Required approach: use the samepinned
alpinedigest theDockerfilealready uses and runscript/bootstrapto install hugo, thenscript/testto build. This makesthe deploy build and the check build use one pinned base and one dependency
list instead of two, and removes the dependency on an image that has been
unmaintained for five years. Do not introduce a second, differently-pinned
Hugo image.
npm install -g wranglerbecomesnpm install -g wrangler@<exact version>with a version/date comment. (The "use yarn, not npm" rule in
REPO_POLICIES.mdgoverns project dependency management via apackage.json/lockfile; this repo has neither, and this is a one-shot CLIinstall in CI. Pinning the exact version is what satisfies the
hash-pinning rule here. Do not add a
package.jsonfor this.)feat/initial-siteis removed fromon.push.branches, leavingmain.check.ymland thecanonical template.
make checkpasses andscript/cibuildsucceeds.PR branch does not fail with a config error).
Explicitly out of scope
actions entirely, but it changes the shape of a live deploy path; keep this
change to pinning so a deploy regression can be attributed unambiguously.
automated version bumps are in tension with hash-pinning.
Risk note
This file is the live deploy path for https://lora.vegas/ and it cannot be
exercised from a PR branch — the Cloudflare Pages deploy only runs on push to
mainand needs theCLOUDFLARE_API_TOKENsecret. The PR gate(
script/cibuild) proves the site still builds; it does not prove the deploystill works. After merge, the deploy run on
mainmust be watched and the livesite confirmed. This is called out in
TODO.mdunder Future Steps ("Verify theCloudflare Pages deploy still works after the workflow changes").
Ref:
REPO_POLICIES.md— "ALL external references must be pinned bycryptographic hash", "When pinning images or packages by hash, add a comment
above the reference with the version and date (YYYY-MM-DD)".
Manager note — this is the top item on the pre-1.0 queue and is being
implemented now.
Ordering rationale: the repo standards rank divergences by severity as
(1) hash-pinning and secrets, (2) required-file and Makefile/Docker gaps,
(3) formatting and style, (4) code-review items. This issue is the only
category-1 divergence in the repo, so it goes first.
Full pre-1.0 queue as filed, in intended order:
deploy.yml(this issue) — in progressscript/checkdoes not runscript/lint, so lint is ungated in CIREPO_POLICIES.md,.editorconfig,.prettierrc,.prettierignore,.gitignore)script/testandscript/lintREADME.mdinto the canonical section set_headerswith baseline security headersLICENSE— blocked on @sneak (license choice is the owner's bypolicy)
(
security-auditdeletion);feat/initial-siteunblocks once this issuelands
Note that #9 is sequenced second deliberately:
script/lintcurrently runsnowhere in the gate, so until it is fixed, every PR in this queue is being
verified by a weaker check than the repo advertises.
Implementation requirements are in the issue body above.
mainis unprotected,so this will be merged directly once an independent adversarial review passes.
Implementation plan
Branching from
mainaspin-deploy-workflow-refs. Single commit,.gitea/workflows/deploy.ymlplusTODO.md.All references resolved and verified against upstream before writing them
(no guessed values):
Images (Docker Registry v2
Docker-Content-Digest, cross-checked againstthe Docker Hub tag API):
klakegg/hugo:ext-alpineis dropped entirely. Thebuildjob containerbecomes the exact digest the
Dockerfilealready pins,alpine@sha256:c3f8e73f...53f4709(alpine 3.21). Steps becomescript/bootstrap(installs hugo via apk, as theDockerfilealreadydoes) then
script/test(thehugo --minifybuild that replaces the rawhugo --minifyinvocation). One pinned base, one dependency list.node:20becomesnode@sha256:8f693eaa...1c9ba5, which is node 20.20.2(bookworm). No version change, just the digest that tag points at today.
Actions (GitHub tag->commit API, each tag ref dereferenced to confirm it
is a commit object):
actions/checkout@v4->11bd71901bbe5b1630ceea73d27597364c9af683(v4.2.2) — the same SHA
.gitea/workflows/check.ymlalready pins, so bothworkflows agree.
actions/upload-artifact@v3-> v4.6.2,ea165f8d65b6e75b540449e92b4886f43607fa02.actions/download-artifact@v3-> v4.3.0,d3f86a106a0bac45b974a628896c90dbdf5c8093.Remote install:
npm install -g wranglerbecomesnpm install -g wrangler@4.120.0(npm registrydist-tags.latest, published2026-08-07). No
package.jsonadded, per the issue.Every pinned line gets
# <name> <version>, YYYY-MM-DDon the line above.Also in the same change: drop
feat/initial-sitefromon.push.branches,leaving
main; reindent the whole file to 4-space YAML to matchcheck.yml.Out of scope and deliberately not done: the two jobs stay separate, no
renovate/dependabot config, no
package.json, noactions/checkoutmajorbump beyond what
check.ymlalready uses.TODO.md: the completed entry goes to the top of Completed Steps, the"Pin the images and actions in
deploy.ymlby sha256" line is removed fromFuture Steps, and Next Step advances per the Workflow block.
Gate:
make checkandscript/cibuild. Noting up front that neitherexercises the Cloudflare Pages deploy — that path only runs on push to
mainand needs
CLOUDFLARE_API_TOKEN, so post-merge verification of the livedeploy is required.
Reopened. The merged change broke the deploy on
mainand has beenreverted.
What happened
PR #17 merged as
74c28c1. The resultingmainrun:check / check (push)— success in 3sBuild and Deploy to Cloudflare Pages / build (push)— failure after 22sBuild and Deploy to Cloudflare Pages / deploy (push)— skipped(
needs: build)So the deploy did not run. Reverted on
mainas3d17e22, restoring theprevious known-working
deploy.yml(theklakegg/hugo:ext-alpineversion).script/cibuildwas green on the revert before pushing. The live site wasnever down — Cloudflare Pages continued serving the last successful deploy —
but no new deploy could happen, and
mainmust not sit in that state.What I cannot tell you yet
I cannot read the failure logs. The Gitea Actions jobs API returns 403 for
clawbot(user should be the owner of the repo) andlist_run_jobscomesback empty, so I have the job's status and duration and nothing else. The
failing run is https://git.eeqj.de/sneak/lora.vegas/actions/runs/25 — @sneak,
you can read those logs and I cannot.
The 22-second duration says it did real work before failing rather than dying
instantly on a config parse error, but I am not going to guess further without
evidence. Candidate causes worth checking in the log, in rough order of my
suspicion: how
act_runnerprovidesnodefor JavaScript actions in acontainer job (it may inject a glibc-linked node that cannot execute against
musl, which no amount of
apk add nodejswould fix), whethershell: shishonoured as a job default, and whether the
apk addstep itself failed.The real lesson — this is a process failure, not just a bad patch
This change was reviewed twice, adversarially, by two independent reviewers.
The second reviewer went as far as executing the pinned
actions/checkoutandactions/upload-artifactbundles under the container's own node and confirmingthey load. It still shipped a break.
The reason is structural and was named in the issue from the start:
deploy.ymlonly triggers on push to
main, so no amount of pre-merge review could executeit. Every verification anyone performed simulated the environment. None of it
was the environment.
script/cibuildcannot help here either —docker buildruns
RUNunder/bin/shwith no Actions runtime at all.New hard requirement for the next attempt
Reviewing harder will not fix an unobservable path. The next attempt must make
the path observable before it reaches
main:deploy.yml'son.push.branchesfor theworking branch, so the
buildjob actually executes underact_runnerduring development. Pass/fail is visible through the commit-status API,
which
clawbotcan read even without log access.buildjob goes green on the branch.branches: [main].The
deployjob will still be unexercised (it needsCLOUDFLARE_API_TOKENandwould publish), but
buildis where this failed and where it can be proven.If a log excerpt from run 25 can be pasted into this issue, that shortcuts the
whole loop — otherwise the next attempt proceeds by iterating against the
branch signal.
Everything else about the reverted work stands: all six pins were independently
re-resolved and correct, and the reverted commits remain in history at
3f91a7candb157bfdfor reuse.Manager note — refinement to the "make it observable" requirement above, so
the next attempt does not create a second problem while fixing the first.
Adding the working branch to
on.push.branchesmakes both jobs run on thatbranch, not just
build. Thedeployjob would then executewrangler pages deploy public --project-name=lora-vegas --branch=<branch>against the real Cloudflare Pages project using the real
CLOUDFLARE_API_TOKEN. On Cloudflare Pages a non-production branch nameproduces a preview deployment rather than overwriting production, so this is
not catastrophic, but it still spends the token and litters the project with
preview deployments on every iteration. That is not acceptable as a routine
development loop.
So the requirement is: temporary branch trigger plus a guard on the deploy
job so it cannot run off
main:With that, a branch push exercises exactly the job that failed (
build) andthe
deployjob reports as skipped, touching nothing external.Worth keeping the
if:guard permanently even after the temporary trigger isremoved. It costs one line and it means any future branch trigger — added
deliberately or by accident — cannot reach Cloudflare. Defence in depth against
precisely the class of mistake this issue is now about. I would keep it; if
there is a reason not to, say so and it comes out in the final commit along
with the temporary trigger.
Sequence for the next attempt:
main(3d17e22, the revert).3f91a7candb157bfd— the six pins wereindependently verified correct and should be reused as-is, not re-resolved.
if:guard.buildjob result from the commit-status API. Iterateagainst that real signal until it is green.
branches: [main]andthe
if:guard.mainrun — which bythen will be the second time that exact job configuration has run, not the
first.
This is actionable without waiting on the run 25 logs. If those logs do get
pasted here they will still shortcut step 4 considerably.
Recovery confirmed. The revert (
3d17e22) produced a fully greenmainrun:check / check (push)— success in 4sBuild and Deploy to Cloudflare Pages / build (push)— success in 5sBuild and Deploy to Cloudflare Pages / deploy (push)— success in 19shttps://lora.vegas/returns HTTP 200 with the expected title and content. Thedeploy path is healthy again and a real deployment has completed end to end.
One diagnostic detail worth recording, because it narrows the search for the
next attempt: the restored
buildjob succeeds in 5 seconds, while thereverted alpine-based
buildjob failed after 22 seconds. The old job isfast because
klakegg/hugo:ext-alpineships hugo preinstalled and does nopackage installation at all. Twenty-two seconds is about what
apk add --no-cache nodejs git tarfollowed byscript/bootstrapwould cost beforesomething downstream failed — so the failure most likely occurred at or after
the checkout step, not at container startup or YAML parsing.
That is consistent with the
act_runner/node hypothesis and inconsistent with"the
apk addstep itself failed" (which would have failed faster and wouldnot have gotten as far). Still a hypothesis, not a diagnosis — the run 25 logs
remain the fast path.
This also means the comparison is not purely like-for-like on cost: the current
approach trades a 5-second build for a ~20-second one in exchange for dropping
an abandoned image and reusing the repo's own pinned base. That trade is still
clearly worth making, but it is worth stating plainly rather than discovering
later.
Implementation plan (second attempt)
Branching from
mainat3d17e22aspin-deploy-refs-observable.The six pinned values from the reverted work are reused verbatim; they were
independently re-resolved twice and are not in question. What is in question is
whether the
buildjob can execute at all on a bare musl alpine underact_runner, and that is what this attempt actually measures instead ofarguing about.
Observability strategy
I have the same access limitation as before: the Actions jobs/logs API 403s for
clawbot, so I cannot read run 25's log. The only readable signal is thecommit-status API, which reports one entry per job with
context,statusand
description.So the plan is to encode the diagnosis as job topology rather than as log
output. Alongside the temporary branch trigger on
deploy.yml, I will add atemporary
.gitea/workflows/probe.ymltriggered only on this branch,containing several small jobs that each isolate one hypothesis. Every job shows
up as its own
contextin the commit-status API, so a single push bisects thefailure in parallel rather than one hypothesis per round trip.
Probe jobs, first round:
actions/checkout— control; tells me whetheract_runnerinjects anodeof its own (if this passes, the wholenode-prerequisite theory is wrong).
apk add --no-cache nodejs git tar+ pinned checkout —exactly the reverted
b157bfdprefix. Isolates checkout from everythingdownstream.
script/bootstrap+script/test— isolates the site buildinside the Actions container from the JS-action machinery.
actions/upload-artifactv4 (pinned) — isolates theartifact upload, which is the one step whose protocol changed in this issue
(v3 -> v4) and which the ~22s timing is consistent with reaching.
node:20-alpinepinned by digest + pinned checkout — the fallback imagedirection, measured rather than assumed.
Then iterate on real results until the real
buildjob indeploy.ymlisgreen on the branch.
Deploy guard
Per the refinement above,
deploygetsif: github.ref_name == 'main'so abranch push cannot reach Cloudflare. That guard stays permanently. The
temporary branch trigger and the whole
probe.ymlfile are removed in thefinal commit, leaving
branches: [main].Scope
Unchanged from the issue: two jobs stay separate, no renovate/dependabot, no
package.json, no re-resolution of the pinned values. If the evidence showsthe mandated "reuse the
Dockerfile's alpine digest" requirement cannot workunder
act_runner, I will deviate and say so explicitly in the PR body withthe observed evidence, since that requirement is @sneak's to re-evaluate.
Gate:
make checkandscript/cibuild, plus the real branch run. I will stateplainly in the PR body that the
deployjob remains unexercised — it needsCLOUDFLARE_API_TOKENand would publish — and I will not claim otherwise.Second attempt is up as #22, branch
pin-deploy-refs-observable, labelledneeds-review.The
buildjob is green on the branch under a real runner. Observed at73f912cvia the commit-status API:The observability requirement worked, and it earned its keep immediately: it
found two breaks, not one, and neither was the thing everyone was
suspecting.
The image was never the problem. A probe job of bare alpine + checkout
failed while the same job with
apk add nodejs git tarfirst passed, and afurther job adding
script/bootstrap+script/testalso passed. Thepinned alpine digest, the prerequisite step and the site build are all fine.
What failed was
actions/upload-artifactv4 — two otherwise identicaljobs, one ending in the v4 upload (failed after 11s) and one without it
(passed in 5s). Artifacts v4 is a different wire protocol and this instance
does not serve it. So the v3 -> v4 bump in DoD item 2 is the thing that
broke run 25, not the
klakeggreplacement. Filed as #20.A second break was waiting behind it, in the
deployjob, which nobodyhad ever run either.
wrangler@4.120.0requires node >= 22 and refuses tostart on the pinned node 20 container;
npm installonly warns aboutengines, so the install step passes and the deploy step fails. And theunpinned
npm install -g wranglerit replaced was never installinglatestanyway — npm picks the newest version the running node satisfies, which on
node 20 is 4.86.0. So the "pin the current version" change silently moved
the deploy onto a wrangler that cannot run. Pinned at 4.86.0, the version
actually in production. Filed as #21.
Both deviations from the definition of done are stated explicitly in the PR
body with the evidence, since they are yours to re-evaluate. DoD item 3 (reuse
the
Dockerfile's alpine digest) is kept — the evidence says it works.The temporary branch trigger and the temporary probe workflow are removed in
the final commit
54ed637. Theif: github.ref_name == 'main'guard is keptpermanently, as suggested.
Still unproven and not claimed: the
deployjob has never executed. A probepair rehearsed it end to end — same pinned node digest, same pinned download
action, same pinned wrangler, real site tarball downloaded and extracted —
stopping at
wrangler pages deploy --helpinstead of publishing. Themainrun after merge is still the first time the real publish path runs, and it must
be watched with the live site confirmed.
Manager note — the root cause is found, and the requirement I wrote in this
issue is what broke production.
PR #22 (#22) developed against a real
act_runnersignal, using a temporary branch trigger plus theif:guard.Round 1 reproduced the
mainfailure exactly, then probe jobs bisected it.The image replacement was never the cause. Two otherwise-identical probe
jobs differing only by a trailing
upload-artifactv4 step: 5s pass withoutit, 11s fail with it. Bare alpine plus
apk add nodejs git tarplus checkoutpasses, and adding
script/bootstrapandscript/teststill passes. Theklakegg→ pinned-alpine swap works fine.What broke the deploy was DoD item 2 of this issue — my requirement that
upload-artifactanddownload-artifactmove from v3 to v4. Artifacts v4 is adifferent wire protocol, and this Gitea instance does not serve it. I wrote
that requirement because v3 is deprecated on GitHub, and I did not check
whether the Gitea instance we actually run on implements v4. That is the same
mistake I have been criticising in this thread: reasoning about the environment
instead of testing it.
Worse, it survived review by being actively confirmed. The second reviewer of
PR #17 explicitly cleared the v4 change on the grounds that Gitea 1.25.4 is
"well past artifact-v4 support". That was a plausible-sounding inference that
happened to be false, and it is exactly the kind of finding an adversarial
review is supposed to catch rather than manufacture.
DoD item 2 is hereby amended. The artifact actions stay on v3, pinned by
commit SHA at the exact commits
@v3was already resolving to. Pinning is therequirement; the major-version bump was my addition and it was wrong. The
policy rule is "pin by cryptographic hash" — it says nothing about chasing
upstream majors, and doing so on an unverified assumption is how this issue
took down the deploy.
A second break was hiding behind the first
wrangler@4.120.0declaresengines.node >=22.0.0and exits 1 on the pinnednode 20 container.
npm installonly warns aboutengines, so the installstep would have gone green and the deploy step would have failed — a second
outage, revealed only after the first was fixed.
The subtle part is worth recording. The unpinned
npm install -g wrangleritreplaced was not installing
latest: npm resolves a bare package name tothe newest version the running node satisfies, which on node 20 is 4.86.0. So
"pin the version that
latestcurrently points at" silently moved the deployonto a wrangler that cannot start on our node. Pinning a mutable reference
means pinning what it actually resolves to in this environment, not what it
resolves to in general. PR #22 pins 4.86.0 and leaves the node digest alone.
Both deviations are tracked: #20 (artifacts v3/v4) and #21 (wrangler/node
version relationship).
Status
PR #22 has a green
buildjob on a realact_runnerrun — the first time anyattempt at this issue has demonstrated that. An independent reviewer who was
not involved in any prior round is reviewing now, with instructions to treat
all prior reasoning including mine as suspect.
The
deployjob remains unexercised and unexercisable pre-merge. It will bewatched on
mainimmediately after merge, with an immediate revert if itfails.
Done. Merged and verified on
main.PR #22 merged as
9959cb5. Themainrun — including the job that has neversuccessfully executed with pinned references until now:
check / check (push)— success in 4sBuild and Deploy to Cloudflare Pages / build (push)— success in 9sBuild and Deploy to Cloudflare Pages / deploy (push)— success in 18shttps://lora.vegas/returns HTTP 200, 4226 bytes, with the expected title andthe mesh channel and Signal group content intact. DoD item 8 is now
genuinely satisfied — the deploy path has run end to end with every external
reference pinned, which is the thing this issue was actually about and which no
previous attempt ever demonstrated.
Final state of
deploy.yml: both container images pinned by digest, all threeuses:pinned by 40-hex commit SHA, the wrangler install pinned to an exactversion that actually runs on the pinned node,
branches: [main]only, theif: github.ref_name == 'main'guard on the deploy job, 4-space YAML. Theabandoned
klakegg/hugo:ext-alpineis gone; the deploy build and the checkbuild now share one pinned base and one dependency list.
What this cost and what it bought
Two merges, one production break, one revert, three implementation rounds and
four reviews. The break was caused by a requirement I wrote (the artifact v3 →
v4 bump) on an assumption I never tested, and it was confirmed rather than
caught by a review that reasoned its way to a false conclusion about this Gitea
instance.
What actually fixed it was not reviewing harder. It was making the path
executable before merging it — a temporary branch trigger plus a guard so
branch runs could not reach Cloudflare, then bisecting with probe jobs against
real runner output. That found the true cause in one pass and incidentally
exposed a second latent outage (
wrangler@4.120.0requires node >= 22 andexits 1 on the pinned node 20 image;
npm installonly warns aboutengines,so the install step would have gone green and the deploy step would have
failed).
The generalisable lessons, recorded here because they outlive this issue:
review. Both reviews of PR #17 were reasoning about an environment neither
could run. Where a workflow triggers only on
main, make it temporarilyobservable rather than trusting analysis.
pinning. Chasing upstream majors on an unverified assumption is what took
the deploy down.
environment.
npm install -g wrangleron node 20 resolves to 4.86.0, notto
latest. Pinning "what latest points at" silently changed behaviour.script/cibuildwas green for every broken revision, becausedocker buildruns
RUNunder/bin/shwith no Actions runtime at all.Follow-ups, none blocking
runs.using: 'node16'durabilitynote).
submodules: recursiveis a no-op here (no.gitmodules,themesis a plain tree), which also undercuts the statedjustification for installing
git; andapk add taris likely redundantagainst busybox tar.
Closing.