.gitea/workflows/deploy.yml is the only file in the repo that still uses
mutable external references. REPO_POLICIES.md calls hash-pinning "the single
most important rule in this document" and states there are "zero exceptions". .gitea/workflows/check.yml, the Dockerfile, and everything else in the org
are already pinned; this file was written before the scaffold landed in #4 and
was never brought up to standard.
Five mutable references, plus one unpinned remote install:
Line
Reference
Problem
build.container.image
klakegg/hugo:ext-alpine
mutable tag; also an abandoned image (last release 2021)
build.steps
actions/checkout@v4
mutable tag
build.steps
actions/upload-artifact@v3
mutable tag; v3 is deprecated
deploy.container.image
node:20
mutable tag
deploy.steps
actions/download-artifact@v3
mutable tag; v3 is deprecated
deploy.steps
npm install -g wrangler
unversioned remote install — arbitrary remote code execution at deploy time
Two further divergences in the same file:
The on.push.branches list still includes feat/initial-site, a branch that
is fully merged into main (0 unique commits) and is scheduled for deletion.
A workflow trigger on a dead branch is dead config.
The file uses 2-space YAML indentation. .editorconfig (canonical, [*] indent_size = 4) and the canonical .gitea/workflows/check.yml both use
4-space. check.yml in this repo already uses 4-space, so the two workflow
files are inconsistent with each other.
Definition of done
Every image: in deploy.yml is referenced as image@sha256:<digest>,
with a comment on the line above in the canonical format # <name> <version>, YYYY-MM-DD.
Every uses: in deploy.yml is referenced by a full 40-hex commit SHA,
with the same comment format. upload-artifact and download-artifact
move from v3 to v4 (v3 is deprecated) and are pinned to v4's commit SHA.
klakegg/hugo:ext-alpine is replaced. Required approach: use the same
pinned alpine digest the Dockerfile already uses and run script/bootstrap to install hugo, then script/test to build. This makes
the deploy build and the check build use one pinned base and one dependency
list instead of two, and removes the dependency on an image that has been
unmaintained for five years. Do not introduce a second, differently-pinned
Hugo image.
npm install -g wrangler becomes npm install -g wrangler@<exact version>
with a version/date comment. (The "use yarn, not npm" rule in REPO_POLICIES.md governs project dependency management via a package.json/lockfile; this repo has neither, and this is a one-shot CLI
install in CI. Pinning the exact version is what satisfies the
hash-pinning rule here. Do not add a package.json for this.)
feat/initial-site is removed from on.push.branches, leaving main.
The file is reindented to 4-space YAML to match check.yml and the
canonical template.
make check passes and script/cibuild succeeds.
The workflow's YAML is verified to parse (e.g. the Gitea Actions run for the
PR branch does not fail with a config error).
Explicitly out of scope
Do not collapse the two jobs into one. It would remove the artifact
actions entirely, but it changes the shape of a live deploy path; keep this
change to pinning so a deploy regression can be attributed unambiguously.
Do not add renovate/dependabot config. No org standard requires one, and
automated version bumps are in tension with hash-pinning.
Risk note
This file is the live deploy path for https://lora.vegas/ and it cannot be
exercised from a PR branch — the Cloudflare Pages deploy only runs on push to main and needs the CLOUDFLARE_API_TOKEN secret. The PR gate
(script/cibuild) proves the site still builds; it does not prove the deploy
still works. After merge, the deploy run on main must be watched and the live
site confirmed. This is called out in TODO.md under Future Steps ("Verify the
Cloudflare Pages deploy still works after the workflow changes").
Ref: REPO_POLICIES.md — "ALL external references must be pinned by
cryptographic hash", "When pinning images or packages by hash, add a comment
above the reference with the version and date (YYYY-MM-DD)".
## Problem
`.gitea/workflows/deploy.yml` is the only file in the repo that still uses
mutable external references. `REPO_POLICIES.md` calls hash-pinning "the single
most important rule in this document" and states there are "zero exceptions".
`.gitea/workflows/check.yml`, the `Dockerfile`, and everything else in the org
are already pinned; this file was written before the scaffold landed in #4 and
was never brought up to standard.
Five mutable references, plus one unpinned remote install:
| Line | Reference | Problem |
| --- | --- | --- |
| `build.container.image` | `klakegg/hugo:ext-alpine` | mutable tag; also an abandoned image (last release 2021) |
| `build.steps` | `actions/checkout@v4` | mutable tag |
| `build.steps` | `actions/upload-artifact@v3` | mutable tag; v3 is deprecated |
| `deploy.container.image` | `node:20` | mutable tag |
| `deploy.steps` | `actions/download-artifact@v3` | mutable tag; v3 is deprecated |
| `deploy.steps` | `npm install -g wrangler` | unversioned remote install — arbitrary remote code execution at deploy time |
Two further divergences in the same file:
- The `on.push.branches` list still includes `feat/initial-site`, a branch that
is fully merged into `main` (0 unique commits) and is scheduled for deletion.
A workflow trigger on a dead branch is dead config.
- The file uses 2-space YAML indentation. `.editorconfig` (canonical, `[*]
indent_size = 4`) and the canonical `.gitea/workflows/check.yml` both use
4-space. `check.yml` in this repo already uses 4-space, so the two workflow
files are inconsistent with each other.
## Definition of done
1. Every `image:` in `deploy.yml` is referenced as `image@sha256:<digest>`,
with a comment on the line above in the canonical format
`# <name> <version>, YYYY-MM-DD`.
2. Every `uses:` in `deploy.yml` is referenced by a full 40-hex commit SHA,
with the same comment format. `upload-artifact` and `download-artifact`
move from v3 to v4 (v3 is deprecated) and are pinned to v4's commit SHA.
3. `klakegg/hugo:ext-alpine` is replaced. **Required approach:** use the same
pinned `alpine` digest the `Dockerfile` already uses and run
`script/bootstrap` to install hugo, then `script/test` to build. This makes
the deploy build and the check build use one pinned base and one dependency
list instead of two, and removes the dependency on an image that has been
unmaintained for five years. Do not introduce a second, differently-pinned
Hugo image.
4. `npm install -g wrangler` becomes `npm install -g wrangler@<exact version>`
with a version/date comment. (The "use yarn, not npm" rule in
`REPO_POLICIES.md` governs project dependency management via a
`package.json`/lockfile; this repo has neither, and this is a one-shot CLI
install in CI. Pinning the exact version is what satisfies the
hash-pinning rule here. Do not add a `package.json` for this.)
5. `feat/initial-site` is removed from `on.push.branches`, leaving `main`.
6. The file is reindented to 4-space YAML to match `check.yml` and the
canonical template.
7. `make check` passes and `script/cibuild` succeeds.
8. The workflow's YAML is verified to parse (e.g. the Gitea Actions run for the
PR branch does not fail with a config error).
## Explicitly out of scope
- **Do not** collapse the two jobs into one. It would remove the artifact
actions entirely, but it changes the shape of a live deploy path; keep this
change to pinning so a deploy regression can be attributed unambiguously.
- **Do not** add renovate/dependabot config. No org standard requires one, and
automated version bumps are in tension with hash-pinning.
## Risk note
This file is the live deploy path for https://lora.vegas/ and it cannot be
exercised from a PR branch — the Cloudflare Pages deploy only runs on push to
`main` and needs the `CLOUDFLARE_API_TOKEN` secret. The PR gate
(`script/cibuild`) proves the site still builds; it does not prove the deploy
still works. After merge, the deploy run on `main` must be watched and the live
site confirmed. This is called out in `TODO.md` under Future Steps ("Verify the
Cloudflare Pages deploy still works after the workflow changes").
Ref: `REPO_POLICIES.md` — "ALL external references must be pinned by
cryptographic hash", "When pinning images or packages by hash, add a comment
above the reference with the version and date (YYYY-MM-DD)".
Manager note — this is the top item on the pre-1.0 queue and is being
implemented now.
Ordering rationale: the repo standards rank divergences by severity as
(1) hash-pinning and secrets, (2) required-file and Makefile/Docker gaps,
(3) formatting and style, (4) code-review items. This issue is the only
category-1 divergence in the repo, so it goes first.
Full pre-1.0 queue as filed, in intended order:
#7 — hash-pin deploy.yml (this issue) — in progress
#9 — script/check does not run script/lint, so lint is ungated in CI
#16 — 30-second timeout guard on script/test and script/lint
#12 — widen the prettier gate to CSS and all Markdown (depends on #8)
#11 — restructure README.md into the canonical section set
#14 — Cloudflare Pages _headers with baseline security headers
#10 — LICENSE — blocked on @sneak (license choice is the owner's by
policy)
#15 — delete stale remote branches — partially blocked on @sneak
(security-audit deletion); feat/initial-site unblocks once this issue
lands
Note that #9 is sequenced second deliberately: script/lint currently runs
nowhere in the gate, so until it is fixed, every PR in this queue is being
verified by a weaker check than the repo advertises.
Implementation requirements are in the issue body above. main is unprotected,
so this will be merged directly once an independent adversarial review passes.
Manager note — this is the top item on the pre-1.0 queue and is being
implemented now.
Ordering rationale: the repo standards rank divergences by severity as
(1) hash-pinning and secrets, (2) required-file and Makefile/Docker gaps,
(3) formatting and style, (4) code-review items. This issue is the only
category-1 divergence in the repo, so it goes first.
Full pre-1.0 queue as filed, in intended order:
1. #7 — hash-pin `deploy.yml` (this issue) — **in progress**
2. #9 — `script/check` does not run `script/lint`, so lint is ungated in CI
3. #8 — canonical policy dotfiles (`REPO_POLICIES.md`, `.editorconfig`,
`.prettierrc`, `.prettierignore`, `.gitignore`)
4. #13 — eliminate the Hugo taxonomy build warning
5. #16 — 30-second timeout guard on `script/test` and `script/lint`
6. #12 — widen the prettier gate to CSS and all Markdown (depends on #8)
7. #11 — restructure `README.md` into the canonical section set
8. #14 — Cloudflare Pages `_headers` with baseline security headers
9. #10 — `LICENSE` — **blocked on @sneak** (license choice is the owner's by
policy)
10. #15 — delete stale remote branches — **partially blocked on @sneak**
(`security-audit` deletion); `feat/initial-site` unblocks once this issue
lands
Note that #9 is sequenced second deliberately: `script/lint` currently runs
nowhere in the gate, so until it is fixed, every PR in this queue is being
verified by a weaker check than the repo advertises.
Implementation requirements are in the issue body above. `main` is unprotected,
so this will be merged directly once an independent adversarial review passes.
Branching from main as pin-deploy-workflow-refs. Single commit, .gitea/workflows/deploy.yml plus TODO.md.
All references resolved and verified against upstream before writing them
(no guessed values):
Images (Docker Registry v2 Docker-Content-Digest, cross-checked against
the Docker Hub tag API):
klakegg/hugo:ext-alpine is dropped entirely. The build job container
becomes the exact digest the Dockerfile already pins, alpine@sha256:c3f8e73f...53f4709 (alpine 3.21). Steps become script/bootstrap (installs hugo via apk, as the Dockerfile already
does) then script/test (the hugo --minify build that replaces the raw hugo --minify invocation). One pinned base, one dependency list.
node:20 becomes node@sha256:8f693eaa...1c9ba5, which is node 20.20.2
(bookworm). No version change, just the digest that tag points at today.
Actions (GitHub tag->commit API, each tag ref dereferenced to confirm it
is a commit object):
actions/checkout@v4 -> 11bd71901bbe5b1630ceea73d27597364c9af683
(v4.2.2) — the same SHA .gitea/workflows/check.yml already pins, so both
workflows agree.
Remote install: npm install -g wrangler becomes npm install -g wrangler@4.120.0 (npm registry dist-tags.latest, published
2026-08-07). No package.json added, per the issue.
Every pinned line gets # <name> <version>, YYYY-MM-DD on the line above.
Also in the same change: drop feat/initial-site from on.push.branches,
leaving main; reindent the whole file to 4-space YAML to match check.yml.
Out of scope and deliberately not done: the two jobs stay separate, no
renovate/dependabot config, no package.json, no actions/checkout major
bump beyond what check.yml already uses.
TODO.md: the completed entry goes to the top of Completed Steps, the
"Pin the images and actions in deploy.yml by sha256" line is removed from
Future Steps, and Next Step advances per the Workflow block.
Gate: make check and script/cibuild. Noting up front that neither
exercises the Cloudflare Pages deploy — that path only runs on push to main
and needs CLOUDFLARE_API_TOKEN, so post-merge verification of the live
deploy is required.
Implementation plan
===================
Branching from `main` as `pin-deploy-workflow-refs`. Single commit,
`.gitea/workflows/deploy.yml` plus `TODO.md`.
All references resolved and verified against upstream before writing them
(no guessed values):
**Images** (Docker Registry v2 `Docker-Content-Digest`, cross-checked against
the Docker Hub tag API):
- `klakegg/hugo:ext-alpine` is dropped entirely. The `build` job container
becomes the exact digest the `Dockerfile` already pins,
`alpine@sha256:c3f8e73f...53f4709` (alpine 3.21). Steps become
`script/bootstrap` (installs hugo via apk, as the `Dockerfile` already
does) then `script/test` (the `hugo --minify` build that replaces the raw
`hugo --minify` invocation). One pinned base, one dependency list.
- `node:20` becomes `node@sha256:8f693eaa...1c9ba5`, which is node 20.20.2
(bookworm). No version change, just the digest that tag points at today.
**Actions** (GitHub tag->commit API, each tag ref dereferenced to confirm it
is a commit object):
- `actions/checkout@v4` -> `11bd71901bbe5b1630ceea73d27597364c9af683`
(v4.2.2) — the same SHA `.gitea/workflows/check.yml` already pins, so both
workflows agree.
- `actions/upload-artifact@v3` -> v4.6.2,
`ea165f8d65b6e75b540449e92b4886f43607fa02`.
- `actions/download-artifact@v3` -> v4.3.0,
`d3f86a106a0bac45b974a628896c90dbdf5c8093`.
**Remote install**: `npm install -g wrangler` becomes
`npm install -g wrangler@4.120.0` (npm registry `dist-tags.latest`, published
2026-08-07). No `package.json` added, per the issue.
Every pinned line gets `# <name> <version>, YYYY-MM-DD` on the line above.
Also in the same change: drop `feat/initial-site` from `on.push.branches`,
leaving `main`; reindent the whole file to 4-space YAML to match `check.yml`.
Out of scope and deliberately not done: the two jobs stay separate, no
renovate/dependabot config, no `package.json`, no `actions/checkout` major
bump beyond what `check.yml` already uses.
`TODO.md`: the completed entry goes to the top of Completed Steps, the
"Pin the images and actions in `deploy.yml` by sha256" line is removed from
Future Steps, and Next Step advances per the Workflow block.
Gate: `make check` and `script/cibuild`. Noting up front that neither
exercises the Cloudflare Pages deploy — that path only runs on push to `main`
and needs `CLOUDFLARE_API_TOKEN`, so post-merge verification of the live
deploy is required.
Build and Deploy to Cloudflare Pages / build (push) — failure after 22s
Build and Deploy to Cloudflare Pages / deploy (push) — skipped
(needs: build)
So the deploy did not run. Reverted on main as 3d17e22, restoring the
previous known-working deploy.yml (the klakegg/hugo:ext-alpine version). script/cibuild was green on the revert before pushing. The live site was
never down — Cloudflare Pages continued serving the last successful deploy —
but no new deploy could happen, and main must not sit in that state.
What I cannot tell you yet
I cannot read the failure logs. The Gitea Actions jobs API returns 403 for clawbot (user should be the owner of the repo) and list_run_jobs comes
back empty, so I have the job's status and duration and nothing else. The
failing run is https://git.eeqj.de/sneak/lora.vegas/actions/runs/25 — @sneak,
you can read those logs and I cannot.
The 22-second duration says it did real work before failing rather than dying
instantly on a config parse error, but I am not going to guess further without
evidence. Candidate causes worth checking in the log, in rough order of my
suspicion: how act_runner provides node for JavaScript actions in a
container job (it may inject a glibc-linked node that cannot execute against
musl, which no amount of apk add nodejs would fix), whether shell: sh is
honoured as a job default, and whether the apk add step itself failed.
The real lesson — this is a process failure, not just a bad patch
This change was reviewed twice, adversarially, by two independent reviewers.
The second reviewer went as far as executing the pinned actions/checkout and actions/upload-artifact bundles under the container's own node and confirming
they load. It still shipped a break.
The reason is structural and was named in the issue from the start: deploy.yml
only triggers on push to main, so no amount of pre-merge review could execute
it. Every verification anyone performed simulated the environment. None of it
was the environment. script/cibuild cannot help here either — docker build
runs RUN under /bin/sh with no Actions runtime at all.
New hard requirement for the next attempt
Reviewing harder will not fix an unobservable path. The next attempt must make
the path observable before it reaches main:
Add a temporary branch trigger to deploy.yml's on.push.branches for the
working branch, so the build job actually executes under act_runner
during development. Pass/fail is visible through the commit-status API,
which clawbot can read even without log access.
Iterate on the real signal until the build job goes green on the branch.
Remove the temporary trigger in the final commit, leaving branches: [main].
Only then merge.
The deploy job will still be unexercised (it needs CLOUDFLARE_API_TOKEN and
would publish), but build is where this failed and where it can be proven.
If a log excerpt from run 25 can be pasted into this issue, that shortcuts the
whole loop — otherwise the next attempt proceeds by iterating against the
branch signal.
Everything else about the reverted work stands: all six pins were independently
re-resolved and correct, and the reverted commits remain in history at 3f91a7c and b157bfd for reuse.
Reopened. **The merged change broke the deploy on `main` and has been
reverted.**
## What happened
PR #17 merged as `74c28c1`. The resulting `main` run:
- `check / check (push)` — success in 3s
- `Build and Deploy to Cloudflare Pages / build (push)` — **failure after 22s**
- `Build and Deploy to Cloudflare Pages / deploy (push)` — skipped
(`needs: build`)
So the deploy did not run. Reverted on `main` as `3d17e22`, restoring the
previous known-working `deploy.yml` (the `klakegg/hugo:ext-alpine` version).
`script/cibuild` was green on the revert before pushing. The live site was
never down — Cloudflare Pages continued serving the last successful deploy —
but no new deploy could happen, and `main` must not sit in that state.
## What I cannot tell you yet
**I cannot read the failure logs.** The Gitea Actions jobs API returns 403 for
`clawbot` (`user should be the owner of the repo`) and `list_run_jobs` comes
back empty, so I have the job's status and duration and nothing else. The
failing run is https://git.eeqj.de/sneak/lora.vegas/actions/runs/25 — @sneak,
you can read those logs and I cannot.
The 22-second duration says it did real work before failing rather than dying
instantly on a config parse error, but I am not going to guess further without
evidence. Candidate causes worth checking in the log, in rough order of my
suspicion: how `act_runner` provides `node` for JavaScript actions in a
container job (it may inject a glibc-linked node that cannot execute against
musl, which no amount of `apk add nodejs` would fix), whether `shell: sh` is
honoured as a job default, and whether the `apk add` step itself failed.
## The real lesson — this is a process failure, not just a bad patch
This change was reviewed twice, adversarially, by two independent reviewers.
The second reviewer went as far as executing the pinned `actions/checkout` and
`actions/upload-artifact` bundles under the container's own node and confirming
they load. It still shipped a break.
The reason is structural and was named in the issue from the start: **`deploy.yml`
only triggers on push to `main`, so no amount of pre-merge review could execute
it.** Every verification anyone performed simulated the environment. None of it
was the environment. `script/cibuild` cannot help here either — `docker build`
runs `RUN` under `/bin/sh` with no Actions runtime at all.
## New hard requirement for the next attempt
Reviewing harder will not fix an unobservable path. The next attempt must make
the path observable **before** it reaches `main`:
1. Add a temporary branch trigger to `deploy.yml`'s `on.push.branches` for the
working branch, so the `build` job actually executes under `act_runner`
during development. Pass/fail is visible through the commit-status API,
which `clawbot` can read even without log access.
2. Iterate on the real signal until the `build` job goes green on the branch.
3. Remove the temporary trigger in the final commit, leaving `branches: [main]`.
4. Only then merge.
The `deploy` job will still be unexercised (it needs `CLOUDFLARE_API_TOKEN` and
would publish), but `build` is where this failed and where it can be proven.
If a log excerpt from run 25 can be pasted into this issue, that shortcuts the
whole loop — otherwise the next attempt proceeds by iterating against the
branch signal.
Everything else about the reverted work stands: all six pins were independently
re-resolved and correct, and the reverted commits remain in history at
`3f91a7c` and `b157bfd` for reuse.
Manager note — refinement to the "make it observable" requirement above, so
the next attempt does not create a second problem while fixing the first.
Adding the working branch to on.push.branches makes both jobs run on that
branch, not just build. The deploy job would then execute wrangler pages deploy public --project-name=lora-vegas --branch=<branch>
against the real Cloudflare Pages project using the real CLOUDFLARE_API_TOKEN. On Cloudflare Pages a non-production branch name
produces a preview deployment rather than overwriting production, so this is
not catastrophic, but it still spends the token and litters the project with
preview deployments on every iteration. That is not acceptable as a routine
development loop.
So the requirement is: temporary branch trigger plus a guard on the deploy
job so it cannot run off main:
deploy:needs:buildif:github.ref_name == 'main'
With that, a branch push exercises exactly the job that failed (build) and
the deploy job reports as skipped, touching nothing external.
Worth keeping the if: guard permanently even after the temporary trigger is
removed. It costs one line and it means any future branch trigger — added
deliberately or by accident — cannot reach Cloudflare. Defence in depth against
precisely the class of mistake this issue is now about. I would keep it; if
there is a reason not to, say so and it comes out in the final commit along
with the temporary trigger.
Sequence for the next attempt:
Branch from current main (3d17e22, the revert).
Re-apply the reverted work from 3f91a7c and b157bfd — the six pins were
independently verified correct and should be reused as-is, not re-resolved.
Add the temporary branch trigger and the if: guard.
Push, and read the build job result from the commit-status API. Iterate
against that real signal until it is green.
Final commit removes the temporary trigger, leaving branches: [main] and
the if: guard.
Fresh adversarial review, then merge, then watch the main run — which by
then will be the second time that exact job configuration has run, not the
first.
This is actionable without waiting on the run 25 logs. If those logs do get
pasted here they will still shortcut step 4 considerably.
Manager note — refinement to the "make it observable" requirement above, so
the next attempt does not create a second problem while fixing the first.
Adding the working branch to `on.push.branches` makes **both** jobs run on that
branch, not just `build`. The `deploy` job would then execute
`wrangler pages deploy public --project-name=lora-vegas --branch=<branch>`
against the real Cloudflare Pages project using the real
`CLOUDFLARE_API_TOKEN`. On Cloudflare Pages a non-production branch name
produces a preview deployment rather than overwriting production, so this is
not catastrophic, but it still spends the token and litters the project with
preview deployments on every iteration. That is not acceptable as a routine
development loop.
So the requirement is: temporary branch trigger **plus** a guard on the deploy
job so it cannot run off `main`:
```yaml
deploy:
needs: build
if: github.ref_name == 'main'
```
With that, a branch push exercises exactly the job that failed (`build`) and
the `deploy` job reports as skipped, touching nothing external.
Worth keeping the `if:` guard permanently even after the temporary trigger is
removed. It costs one line and it means any future branch trigger — added
deliberately or by accident — cannot reach Cloudflare. Defence in depth against
precisely the class of mistake this issue is now about. I would keep it; if
there is a reason not to, say so and it comes out in the final commit along
with the temporary trigger.
Sequence for the next attempt:
1. Branch from current `main` (`3d17e22`, the revert).
2. Re-apply the reverted work from `3f91a7c` and `b157bfd` — the six pins were
independently verified correct and should be reused as-is, not re-resolved.
3. Add the temporary branch trigger and the `if:` guard.
4. Push, and read the `build` job result from the commit-status API. Iterate
against that real signal until it is green.
5. Final commit removes the temporary trigger, leaving `branches: [main]` and
the `if:` guard.
6. Fresh adversarial review, then merge, then watch the `main` run — which by
then will be the second time that exact job configuration has run, not the
first.
This is actionable without waiting on the run 25 logs. If those logs do get
pasted here they will still shortcut step 4 considerably.
Recovery confirmed. The revert (3d17e22) produced a fully green main run:
check / check (push) — success in 4s
Build and Deploy to Cloudflare Pages / build (push) — success in 5s
Build and Deploy to Cloudflare Pages / deploy (push) — success in 19s
https://lora.vegas/ returns HTTP 200 with the expected title and content. The
deploy path is healthy again and a real deployment has completed end to end.
One diagnostic detail worth recording, because it narrows the search for the
next attempt: the restored build job succeeds in 5 seconds, while the
reverted alpine-based build job failed after 22 seconds. The old job is
fast because klakegg/hugo:ext-alpine ships hugo preinstalled and does no
package installation at all. Twenty-two seconds is about what apk add --no-cache nodejs git tar followed by script/bootstrap would cost before
something downstream failed — so the failure most likely occurred at or after
the checkout step, not at container startup or YAML parsing.
That is consistent with the act_runner/node hypothesis and inconsistent with
"the apk add step itself failed" (which would have failed faster and would
not have gotten as far). Still a hypothesis, not a diagnosis — the run 25 logs
remain the fast path.
This also means the comparison is not purely like-for-like on cost: the current
approach trades a 5-second build for a ~20-second one in exchange for dropping
an abandoned image and reusing the repo's own pinned base. That trade is still
clearly worth making, but it is worth stating plainly rather than discovering
later.
Recovery confirmed. The revert (`3d17e22`) produced a fully green `main` run:
- `check / check (push)` — success in 4s
- `Build and Deploy to Cloudflare Pages / build (push)` — success in 5s
- `Build and Deploy to Cloudflare Pages / deploy (push)` — **success in 19s**
`https://lora.vegas/` returns HTTP 200 with the expected title and content. The
deploy path is healthy again and a real deployment has completed end to end.
One diagnostic detail worth recording, because it narrows the search for the
next attempt: the restored `build` job succeeds in **5 seconds**, while the
reverted alpine-based `build` job failed after **22 seconds**. The old job is
fast because `klakegg/hugo:ext-alpine` ships hugo preinstalled and does no
package installation at all. Twenty-two seconds is about what `apk add
--no-cache nodejs git tar` followed by `script/bootstrap` would cost before
something downstream failed — so the failure most likely occurred at or after
the checkout step, not at container startup or YAML parsing.
That is consistent with the `act_runner`/node hypothesis and inconsistent with
"the `apk add` step itself failed" (which would have failed faster and would
not have gotten as far). Still a hypothesis, not a diagnosis — the run 25 logs
remain the fast path.
This also means the comparison is not purely like-for-like on cost: the current
approach trades a 5-second build for a ~20-second one in exchange for dropping
an abandoned image and reusing the repo's own pinned base. That trade is still
clearly worth making, but it is worth stating plainly rather than discovering
later.
Branching from main at 3d17e22 as pin-deploy-refs-observable.
The six pinned values from the reverted work are reused verbatim; they were
independently re-resolved twice and are not in question. What is in question is
whether the build job can execute at all on a bare musl alpine under act_runner, and that is what this attempt actually measures instead of
arguing about.
Observability strategy
I have the same access limitation as before: the Actions jobs/logs API 403s for clawbot, so I cannot read run 25's log. The only readable signal is the
commit-status API, which reports one entry per job with context, status
and description.
So the plan is to encode the diagnosis as job topology rather than as log
output. Alongside the temporary branch trigger on deploy.yml, I will add a temporary.gitea/workflows/probe.yml triggered only on this branch,
containing several small jobs that each isolate one hypothesis. Every job shows
up as its own context in the commit-status API, so a single push bisects the
failure in parallel rather than one hypothesis per round trip.
Probe jobs, first round:
bare pinned alpine + pinned actions/checkout — control; tells me whether act_runner injects a node of its own (if this passes, the whole
node-prerequisite theory is wrong).
pinned alpine + apk add --no-cache nodejs git tar + pinned checkout —
exactly the reverted b157bfd prefix. Isolates checkout from everything
downstream.
same as 2, plus script/bootstrap + script/test — isolates the site build
inside the Actions container from the JS-action machinery.
same as 2, plus actions/upload-artifact v4 (pinned) — isolates the
artifact upload, which is the one step whose protocol changed in this issue
(v3 -> v4) and which the ~22s timing is consistent with reaching.
node:20-alpine pinned by digest + pinned checkout — the fallback image
direction, measured rather than assumed.
Then iterate on real results until the real build job in deploy.yml is
green on the branch.
Deploy guard
Per the refinement above, deploy gets if: github.ref_name == 'main' so a
branch push cannot reach Cloudflare. That guard stays permanently. The
temporary branch trigger and the whole probe.yml file are removed in the
final commit, leaving branches: [main].
Scope
Unchanged from the issue: two jobs stay separate, no renovate/dependabot, no package.json, no re-resolution of the pinned values. If the evidence shows
the mandated "reuse the Dockerfile's alpine digest" requirement cannot work
under act_runner, I will deviate and say so explicitly in the PR body with
the observed evidence, since that requirement is @sneak's to re-evaluate.
Gate: make check and script/cibuild, plus the real branch run. I will state
plainly in the PR body that the deploy job remains unexercised — it needs CLOUDFLARE_API_TOKEN and would publish — and I will not claim otherwise.
Implementation plan (second attempt)
===================================
Branching from `main` at `3d17e22` as `pin-deploy-refs-observable`.
The six pinned values from the reverted work are reused verbatim; they were
independently re-resolved twice and are not in question. What is in question is
whether the `build` job can execute at all on a bare musl alpine under
`act_runner`, and that is what this attempt actually measures instead of
arguing about.
Observability strategy
----------------------
I have the same access limitation as before: the Actions jobs/logs API 403s for
`clawbot`, so I cannot read run 25's log. The only readable signal is the
commit-status API, which reports one entry per **job** with `context`, `status`
and `description`.
So the plan is to encode the diagnosis as job topology rather than as log
output. Alongside the temporary branch trigger on `deploy.yml`, I will add a
**temporary** `.gitea/workflows/probe.yml` triggered only on this branch,
containing several small jobs that each isolate one hypothesis. Every job shows
up as its own `context` in the commit-status API, so a single push bisects the
failure in parallel rather than one hypothesis per round trip.
Probe jobs, first round:
1. bare pinned alpine + pinned `actions/checkout` — control; tells me whether
`act_runner` injects a `node` of its own (if this passes, the whole
node-prerequisite theory is wrong).
2. pinned alpine + `apk add --no-cache nodejs git tar` + pinned checkout —
exactly the reverted `b157bfd` prefix. Isolates checkout from everything
downstream.
3. same as 2, plus `script/bootstrap` + `script/test` — isolates the site build
inside the Actions container from the JS-action machinery.
4. same as 2, plus `actions/upload-artifact` v4 (pinned) — isolates the
artifact upload, which is the one step whose protocol changed in this issue
(v3 -> v4) and which the ~22s timing is consistent with reaching.
5. `node:20-alpine` pinned by digest + pinned checkout — the fallback image
direction, measured rather than assumed.
Then iterate on real results until the real `build` job in `deploy.yml` is
green on the branch.
Deploy guard
------------
Per the refinement above, `deploy` gets `if: github.ref_name == 'main'` so a
branch push cannot reach Cloudflare. That guard stays permanently. The
temporary branch trigger and the whole `probe.yml` file are removed in the
final commit, leaving `branches: [main]`.
Scope
-----
Unchanged from the issue: two jobs stay separate, no renovate/dependabot, no
`package.json`, no re-resolution of the pinned values. If the evidence shows
the mandated "reuse the `Dockerfile`'s alpine digest" requirement cannot work
under `act_runner`, I will deviate and say so explicitly in the PR body with
the observed evidence, since that requirement is @sneak's to re-evaluate.
Gate: `make check` and `script/cibuild`, plus the real branch run. I will state
plainly in the PR body that the `deploy` job remains unexercised — it needs
`CLOUDFLARE_API_TOKEN` and would publish — and I will not claim otherwise.
Second attempt is up as #22, branch pin-deploy-refs-observable, labelled needs-review.
The build job is green on the branch under a real runner. Observed at 73f912c via the commit-status API:
overall: success
Build and Deploy to Cloudflare Pages / build success 7s
Build and Deploy to Cloudflare Pages / deploy skipped
check / check success 6s
probe / s1-build success 13s
probe / s2-deploy-dryrun success 10s
The observability requirement worked, and it earned its keep immediately: it
found two breaks, not one, and neither was the thing everyone was
suspecting.
The image was never the problem. A probe job of bare alpine + checkout
failed while the same job with apk add nodejs git tar first passed, and a
further job adding script/bootstrap + script/test also passed. The
pinned alpine digest, the prerequisite step and the site build are all fine.
What failed was actions/upload-artifactv4 — two otherwise identical
jobs, one ending in the v4 upload (failed after 11s) and one without it
(passed in 5s). Artifacts v4 is a different wire protocol and this instance
does not serve it. So the v3 -> v4 bump in DoD item 2 is the thing that
broke run 25, not the klakegg replacement. Filed as #20.
A second break was waiting behind it, in the deploy job, which nobody
had ever run either. wrangler@4.120.0 requires node >= 22 and refuses to
start on the pinned node 20 container; npm install only warns about engines, so the install step passes and the deploy step fails. And the
unpinned npm install -g wrangler it replaced was never installing latest
anyway — npm picks the newest version the running node satisfies, which on
node 20 is 4.86.0. So the "pin the current version" change silently moved
the deploy onto a wrangler that cannot run. Pinned at 4.86.0, the version
actually in production. Filed as #21.
Both deviations from the definition of done are stated explicitly in the PR
body with the evidence, since they are yours to re-evaluate. DoD item 3 (reuse
the Dockerfile's alpine digest) is kept — the evidence says it works.
The temporary branch trigger and the temporary probe workflow are removed in
the final commit 54ed637. The if: github.ref_name == 'main' guard is kept
permanently, as suggested.
Still unproven and not claimed: the deploy job has never executed. A probe
pair rehearsed it end to end — same pinned node digest, same pinned download
action, same pinned wrangler, real site tarball downloaded and extracted —
stopping at wrangler pages deploy --help instead of publishing. The main
run after merge is still the first time the real publish path runs, and it must
be watched with the live site confirmed.
Second attempt is up as #22, branch `pin-deploy-refs-observable`, labelled
`needs-review`.
**The `build` job is green on the branch under a real runner.** Observed at
`73f912c` via the commit-status API:
```
overall: success
Build and Deploy to Cloudflare Pages / build success 7s
Build and Deploy to Cloudflare Pages / deploy skipped
check / check success 6s
probe / s1-build success 13s
probe / s2-deploy-dryrun success 10s
```
The observability requirement worked, and it earned its keep immediately: it
found **two** breaks, not one, and neither was the thing everyone was
suspecting.
1. **The image was never the problem.** A probe job of bare alpine + checkout
failed while the same job with `apk add nodejs git tar` first passed, and a
further job adding `script/bootstrap` + `script/test` also passed. The
pinned alpine digest, the prerequisite step and the site build are all fine.
What failed was `actions/upload-artifact` **v4** — two otherwise identical
jobs, one ending in the v4 upload (failed after 11s) and one without it
(passed in 5s). Artifacts v4 is a different wire protocol and this instance
does not serve it. So the v3 -> v4 bump in DoD item 2 is the thing that
broke run 25, not the `klakegg` replacement. Filed as #20.
2. **A second break was waiting behind it**, in the `deploy` job, which nobody
had ever run either. `wrangler@4.120.0` requires node >= 22 and refuses to
start on the pinned node 20 container; `npm install` only warns about
`engines`, so the install step passes and the deploy step fails. And the
unpinned `npm install -g wrangler` it replaced was never installing `latest`
anyway — npm picks the newest version the running node satisfies, which on
node 20 is 4.86.0. So the "pin the current version" change silently moved
the deploy onto a wrangler that cannot run. Pinned at 4.86.0, the version
actually in production. Filed as #21.
Both deviations from the definition of done are stated explicitly in the PR
body with the evidence, since they are yours to re-evaluate. DoD item 3 (reuse
the `Dockerfile`'s alpine digest) is **kept** — the evidence says it works.
The temporary branch trigger and the temporary probe workflow are removed in
the final commit `54ed637`. The `if: github.ref_name == 'main'` guard is kept
permanently, as suggested.
Still unproven and not claimed: the `deploy` job has never executed. A probe
pair rehearsed it end to end — same pinned node digest, same pinned download
action, same pinned wrangler, real site tarball downloaded and extracted —
stopping at `wrangler pages deploy --help` instead of publishing. The `main`
run after merge is still the first time the real publish path runs, and it must
be watched with the live site confirmed.
Manager note — the root cause is found, and the requirement I wrote in this
issue is what broke production.
PR #22 (#22) developed against a real act_runner signal, using a temporary branch trigger plus the if: guard.
Round 1 reproduced the main failure exactly, then probe jobs bisected it.
The image replacement was never the cause. Two otherwise-identical probe
jobs differing only by a trailing upload-artifact v4 step: 5s pass without
it, 11s fail with it. Bare alpine plus apk add nodejs git tar plus checkout
passes, and adding script/bootstrap and script/test still passes. The klakegg → pinned-alpine swap works fine.
What broke the deploy was DoD item 2 of this issue — my requirement that upload-artifact and download-artifact move from v3 to v4. Artifacts v4 is a
different wire protocol, and this Gitea instance does not serve it. I wrote
that requirement because v3 is deprecated on GitHub, and I did not check
whether the Gitea instance we actually run on implements v4. That is the same
mistake I have been criticising in this thread: reasoning about the environment
instead of testing it.
Worse, it survived review by being actively confirmed. The second reviewer of
PR #17 explicitly cleared the v4 change on the grounds that Gitea 1.25.4 is
"well past artifact-v4 support". That was a plausible-sounding inference that
happened to be false, and it is exactly the kind of finding an adversarial
review is supposed to catch rather than manufacture.
DoD item 2 is hereby amended. The artifact actions stay on v3, pinned by
commit SHA at the exact commits @v3 was already resolving to. Pinning is the
requirement; the major-version bump was my addition and it was wrong. The
policy rule is "pin by cryptographic hash" — it says nothing about chasing
upstream majors, and doing so on an unverified assumption is how this issue
took down the deploy.
A second break was hiding behind the first
wrangler@4.120.0 declares engines.node >=22.0.0 and exits 1 on the pinned
node 20 container. npm install only warns about engines, so the install
step would have gone green and the deploy step would have failed — a second
outage, revealed only after the first was fixed.
The subtle part is worth recording. The unpinned npm install -g wrangler it
replaced was not installing latest: npm resolves a bare package name to
the newest version the running node satisfies, which on node 20 is 4.86.0. So
"pin the version that latest currently points at" silently moved the deploy
onto a wrangler that cannot start on our node. Pinning a mutable reference
means pinning what it actually resolves to in this environment, not what it
resolves to in general. PR #22 pins 4.86.0 and leaves the node digest alone.
Both deviations are tracked: #20 (artifacts v3/v4) and #21 (wrangler/node
version relationship).
Status
PR #22 has a green build job on a real act_runner run — the first time any
attempt at this issue has demonstrated that. An independent reviewer who was
not involved in any prior round is reviewing now, with instructions to treat
all prior reasoning including mine as suspect.
The deploy job remains unexercised and unexercisable pre-merge. It will be
watched on main immediately after merge, with an immediate revert if it
fails.
Manager note — the root cause is found, and **the requirement I wrote in this
issue is what broke production.**
PR #22 (https://git.eeqj.de/sneak/lora.vegas/pulls/22) developed against a real
`act_runner` signal, using a temporary branch trigger plus the `if:` guard.
Round 1 reproduced the `main` failure exactly, then probe jobs bisected it.
**The image replacement was never the cause.** Two otherwise-identical probe
jobs differing only by a trailing `upload-artifact` v4 step: 5s pass without
it, 11s fail with it. Bare alpine plus `apk add nodejs git tar` plus checkout
passes, and adding `script/bootstrap` and `script/test` still passes. The
`klakegg` → pinned-alpine swap works fine.
What broke the deploy was **DoD item 2 of this issue** — my requirement that
`upload-artifact` and `download-artifact` move from v3 to v4. Artifacts v4 is a
different wire protocol, and this Gitea instance does not serve it. I wrote
that requirement because v3 is deprecated on GitHub, and I did not check
whether the Gitea instance we actually run on implements v4. That is the same
mistake I have been criticising in this thread: reasoning about the environment
instead of testing it.
Worse, it survived review by being actively confirmed. The second reviewer of
PR #17 explicitly cleared the v4 change on the grounds that Gitea 1.25.4 is
"well past artifact-v4 support". That was a plausible-sounding inference that
happened to be false, and it is exactly the kind of finding an adversarial
review is supposed to catch rather than manufacture.
**DoD item 2 is hereby amended.** The artifact actions stay on v3, pinned by
commit SHA at the exact commits `@v3` was already resolving to. Pinning is the
requirement; the major-version bump was my addition and it was wrong. The
policy rule is "pin by cryptographic hash" — it says nothing about chasing
upstream majors, and doing so on an unverified assumption is how this issue
took down the deploy.
## A second break was hiding behind the first
`wrangler@4.120.0` declares `engines.node >=22.0.0` and exits 1 on the pinned
node 20 container. `npm install` only warns about `engines`, so the install
step would have gone green and the deploy step would have failed — a second
outage, revealed only after the first was fixed.
The subtle part is worth recording. The unpinned `npm install -g wrangler` it
replaced was **not** installing `latest`: npm resolves a bare package name to
the newest version the running node satisfies, which on node 20 is 4.86.0. So
"pin the version that `latest` currently points at" silently moved the deploy
onto a wrangler that cannot start on our node. Pinning a mutable reference
means pinning **what it actually resolves to in this environment**, not what it
resolves to in general. PR #22 pins 4.86.0 and leaves the node digest alone.
Both deviations are tracked: #20 (artifacts v3/v4) and #21 (wrangler/node
version relationship).
## Status
PR #22 has a green `build` job on a real `act_runner` run — the first time any
attempt at this issue has demonstrated that. An independent reviewer who was
not involved in any prior round is reviewing now, with instructions to treat
all prior reasoning including mine as suspect.
The `deploy` job remains unexercised and unexercisable pre-merge. It will be
watched on `main` immediately after merge, with an immediate revert if it
fails.
PR #22 merged as 9959cb5. The main run — including the job that has never
successfully executed with pinned references until now:
check / check (push) — success in 4s
Build and Deploy to Cloudflare Pages / build (push) — success in 9s
Build and Deploy to Cloudflare Pages / deploy (push) — success in 18s
https://lora.vegas/ returns HTTP 200, 4226 bytes, with the expected title and
the mesh channel and Signal group content intact. DoD item 8 is now
genuinely satisfied — the deploy path has run end to end with every external
reference pinned, which is the thing this issue was actually about and which no
previous attempt ever demonstrated.
Final state of deploy.yml: both container images pinned by digest, all three uses: pinned by 40-hex commit SHA, the wrangler install pinned to an exact
version that actually runs on the pinned node, branches: [main] only, the if: github.ref_name == 'main' guard on the deploy job, 4-space YAML. The
abandoned klakegg/hugo:ext-alpine is gone; the deploy build and the check
build now share one pinned base and one dependency list.
What this cost and what it bought
Two merges, one production break, one revert, three implementation rounds and
four reviews. The break was caused by a requirement I wrote (the artifact v3 →
v4 bump) on an assumption I never tested, and it was confirmed rather than
caught by a review that reasoned its way to a false conclusion about this Gitea
instance.
What actually fixed it was not reviewing harder. It was making the path
executable before merging it — a temporary branch trigger plus a guard so
branch runs could not reach Cloudflare, then bisecting with probe jobs against
real runner output. That found the true cause in one pass and incidentally
exposed a second latent outage (wrangler@4.120.0 requires node >= 22 and
exits 1 on the pinned node 20 image; npm install only warns about engines,
so the install step would have gone green and the deploy step would have
failed).
The generalisable lessons, recorded here because they outlive this issue:
A CI path that cannot be executed pre-merge cannot be validated by
review. Both reviews of PR #17 were reasoning about an environment neither
could run. Where a workflow triggers only on main, make it temporarily
observable rather than trusting analysis.
"Pin by hash" is not "bump to the latest major." The policy asks for
pinning. Chasing upstream majors on an unverified assumption is what took
the deploy down.
Pinning a mutable reference means pinning what it resolves to in this
environment.npm install -g wrangler on node 20 resolves to 4.86.0, not
to latest. Pinning "what latest points at" silently changed behaviour.
Green CI was the weakest signal in this entire issue, not the strongest. script/cibuild was green for every broken revision, because docker build
runs RUN under /bin/sh with no Actions runtime at all.
Follow-ups, none blocking
#20 — artifact actions held on v3 (add the runs.using: 'node16' durability
note).
#21 — deploy container is node 20, capping wrangler at 4.86.0.
New follow-up to file: submodules: recursive is a no-op here (no .gitmodules, themes is a plain tree), which also undercuts the stated
justification for installing git; and apk add tar is likely redundant
against busybox tar.
Closing.
**Done. Merged and verified on `main`.**
PR #22 merged as `9959cb5`. The `main` run — including the job that has never
successfully executed with pinned references until now:
- `check / check (push)` — success in 4s
- `Build and Deploy to Cloudflare Pages / build (push)` — success in 9s
- `Build and Deploy to Cloudflare Pages / deploy (push)` — **success in 18s**
`https://lora.vegas/` returns HTTP 200, 4226 bytes, with the expected title and
the mesh channel and Signal group content intact. **DoD item 8 is now
genuinely satisfied** — the deploy path has run end to end with every external
reference pinned, which is the thing this issue was actually about and which no
previous attempt ever demonstrated.
Final state of `deploy.yml`: both container images pinned by digest, all three
`uses:` pinned by 40-hex commit SHA, the wrangler install pinned to an exact
version that actually runs on the pinned node, `branches: [main]` only, the
`if: github.ref_name == 'main'` guard on the deploy job, 4-space YAML. The
abandoned `klakegg/hugo:ext-alpine` is gone; the deploy build and the check
build now share one pinned base and one dependency list.
## What this cost and what it bought
Two merges, one production break, one revert, three implementation rounds and
four reviews. The break was caused by a requirement I wrote (the artifact v3 →
v4 bump) on an assumption I never tested, and it was confirmed rather than
caught by a review that reasoned its way to a false conclusion about this Gitea
instance.
What actually fixed it was not reviewing harder. It was making the path
executable before merging it — a temporary branch trigger plus a guard so
branch runs could not reach Cloudflare, then bisecting with probe jobs against
real runner output. That found the true cause in one pass and incidentally
exposed a second latent outage (`wrangler@4.120.0` requires node >= 22 and
exits 1 on the pinned node 20 image; `npm install` only warns about `engines`,
so the install step would have gone green and the deploy step would have
failed).
The generalisable lessons, recorded here because they outlive this issue:
1. **A CI path that cannot be executed pre-merge cannot be validated by
review.** Both reviews of PR #17 were reasoning about an environment neither
could run. Where a workflow triggers only on `main`, make it temporarily
observable rather than trusting analysis.
2. **"Pin by hash" is not "bump to the latest major."** The policy asks for
pinning. Chasing upstream majors on an unverified assumption is what took
the deploy down.
3. **Pinning a mutable reference means pinning what it resolves to *in this
environment*.** `npm install -g wrangler` on node 20 resolves to 4.86.0, not
to `latest`. Pinning "what latest points at" silently changed behaviour.
4. **Green CI was the weakest signal in this entire issue, not the strongest.**
`script/cibuild` was green for every broken revision, because `docker build`
runs `RUN` under `/bin/sh` with no Actions runtime at all.
## Follow-ups, none blocking
- #20 — artifact actions held on v3 (add the `runs.using: 'node16'` durability
note).
- #21 — deploy container is node 20, capping wrangler at 4.86.0.
- New follow-up to file: `submodules: recursive` is a no-op here (no
`.gitmodules`, `themes` is a plain tree), which also undercuts the stated
justification for installing `git`; and `apk add tar` is likely redundant
against busybox tar.
Closing.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Problem
.gitea/workflows/deploy.ymlis the only file in the repo that still usesmutable external references.
REPO_POLICIES.mdcalls hash-pinning "the singlemost important rule in this document" and states there are "zero exceptions".
.gitea/workflows/check.yml, theDockerfile, and everything else in the orgare already pinned; this file was written before the scaffold landed in #4 and
was never brought up to standard.
Five mutable references, plus one unpinned remote install:
build.container.imageklakegg/hugo:ext-alpinebuild.stepsactions/checkout@v4build.stepsactions/upload-artifact@v3deploy.container.imagenode:20deploy.stepsactions/download-artifact@v3deploy.stepsnpm install -g wranglerTwo further divergences in the same file:
on.push.brancheslist still includesfeat/initial-site, a branch thatis fully merged into
main(0 unique commits) and is scheduled for deletion.A workflow trigger on a dead branch is dead config.
.editorconfig(canonical,[*] indent_size = 4) and the canonical.gitea/workflows/check.ymlboth use4-space.
check.ymlin this repo already uses 4-space, so the two workflowfiles are inconsistent with each other.
Definition of done
image:indeploy.ymlis referenced asimage@sha256:<digest>,with a comment on the line above in the canonical format
# <name> <version>, YYYY-MM-DD.uses:indeploy.ymlis referenced by a full 40-hex commit SHA,with the same comment format.
upload-artifactanddownload-artifactmove from v3 to v4 (v3 is deprecated) and are pinned to v4's commit SHA.
klakegg/hugo:ext-alpineis replaced. Required approach: use the samepinned
alpinedigest theDockerfilealready uses and runscript/bootstrapto install hugo, thenscript/testto build. This makesthe deploy build and the check build use one pinned base and one dependency
list instead of two, and removes the dependency on an image that has been
unmaintained for five years. Do not introduce a second, differently-pinned
Hugo image.
npm install -g wranglerbecomesnpm install -g wrangler@<exact version>with a version/date comment. (The "use yarn, not npm" rule in
REPO_POLICIES.mdgoverns project dependency management via apackage.json/lockfile; this repo has neither, and this is a one-shot CLIinstall in CI. Pinning the exact version is what satisfies the
hash-pinning rule here. Do not add a
package.jsonfor this.)feat/initial-siteis removed fromon.push.branches, leavingmain.check.ymland thecanonical template.
make checkpasses andscript/cibuildsucceeds.PR branch does not fail with a config error).
Explicitly out of scope
actions entirely, but it changes the shape of a live deploy path; keep this
change to pinning so a deploy regression can be attributed unambiguously.
automated version bumps are in tension with hash-pinning.
Risk note
This file is the live deploy path for https://lora.vegas/ and it cannot be
exercised from a PR branch — the Cloudflare Pages deploy only runs on push to
mainand needs theCLOUDFLARE_API_TOKENsecret. The PR gate(
script/cibuild) proves the site still builds; it does not prove the deploystill works. After merge, the deploy run on
mainmust be watched and the livesite confirmed. This is called out in
TODO.mdunder Future Steps ("Verify theCloudflare Pages deploy still works after the workflow changes").
Ref:
REPO_POLICIES.md— "ALL external references must be pinned bycryptographic hash", "When pinning images or packages by hash, add a comment
above the reference with the version and date (YYYY-MM-DD)".
Manager note — this is the top item on the pre-1.0 queue and is being
implemented now.
Ordering rationale: the repo standards rank divergences by severity as
(1) hash-pinning and secrets, (2) required-file and Makefile/Docker gaps,
(3) formatting and style, (4) code-review items. This issue is the only
category-1 divergence in the repo, so it goes first.
Full pre-1.0 queue as filed, in intended order:
deploy.yml(this issue) — in progressscript/checkdoes not runscript/lint, so lint is ungated in CIREPO_POLICIES.md,.editorconfig,.prettierrc,.prettierignore,.gitignore)script/testandscript/lintREADME.mdinto the canonical section set_headerswith baseline security headersLICENSE— blocked on @sneak (license choice is the owner's bypolicy)
(
security-auditdeletion);feat/initial-siteunblocks once this issuelands
Note that #9 is sequenced second deliberately:
script/lintcurrently runsnowhere in the gate, so until it is fixed, every PR in this queue is being
verified by a weaker check than the repo advertises.
Implementation requirements are in the issue body above.
mainis unprotected,so this will be merged directly once an independent adversarial review passes.
Implementation plan
Branching from
mainaspin-deploy-workflow-refs. Single commit,.gitea/workflows/deploy.ymlplusTODO.md.All references resolved and verified against upstream before writing them
(no guessed values):
Images (Docker Registry v2
Docker-Content-Digest, cross-checked againstthe Docker Hub tag API):
klakegg/hugo:ext-alpineis dropped entirely. Thebuildjob containerbecomes the exact digest the
Dockerfilealready pins,alpine@sha256:c3f8e73f...53f4709(alpine 3.21). Steps becomescript/bootstrap(installs hugo via apk, as theDockerfilealreadydoes) then
script/test(thehugo --minifybuild that replaces the rawhugo --minifyinvocation). One pinned base, one dependency list.node:20becomesnode@sha256:8f693eaa...1c9ba5, which is node 20.20.2(bookworm). No version change, just the digest that tag points at today.
Actions (GitHub tag->commit API, each tag ref dereferenced to confirm it
is a commit object):
actions/checkout@v4->11bd71901bbe5b1630ceea73d27597364c9af683(v4.2.2) — the same SHA
.gitea/workflows/check.ymlalready pins, so bothworkflows agree.
actions/upload-artifact@v3-> v4.6.2,ea165f8d65b6e75b540449e92b4886f43607fa02.actions/download-artifact@v3-> v4.3.0,d3f86a106a0bac45b974a628896c90dbdf5c8093.Remote install:
npm install -g wranglerbecomesnpm install -g wrangler@4.120.0(npm registrydist-tags.latest, published2026-08-07). No
package.jsonadded, per the issue.Every pinned line gets
# <name> <version>, YYYY-MM-DDon the line above.Also in the same change: drop
feat/initial-sitefromon.push.branches,leaving
main; reindent the whole file to 4-space YAML to matchcheck.yml.Out of scope and deliberately not done: the two jobs stay separate, no
renovate/dependabot config, no
package.json, noactions/checkoutmajorbump beyond what
check.ymlalready uses.TODO.md: the completed entry goes to the top of Completed Steps, the"Pin the images and actions in
deploy.ymlby sha256" line is removed fromFuture Steps, and Next Step advances per the Workflow block.
Gate:
make checkandscript/cibuild. Noting up front that neitherexercises the Cloudflare Pages deploy — that path only runs on push to
mainand needs
CLOUDFLARE_API_TOKEN, so post-merge verification of the livedeploy is required.
Reopened. The merged change broke the deploy on
mainand has beenreverted.
What happened
PR #17 merged as
74c28c1. The resultingmainrun:check / check (push)— success in 3sBuild and Deploy to Cloudflare Pages / build (push)— failure after 22sBuild and Deploy to Cloudflare Pages / deploy (push)— skipped(
needs: build)So the deploy did not run. Reverted on
mainas3d17e22, restoring theprevious known-working
deploy.yml(theklakegg/hugo:ext-alpineversion).script/cibuildwas green on the revert before pushing. The live site wasnever down — Cloudflare Pages continued serving the last successful deploy —
but no new deploy could happen, and
mainmust not sit in that state.What I cannot tell you yet
I cannot read the failure logs. The Gitea Actions jobs API returns 403 for
clawbot(user should be the owner of the repo) andlist_run_jobscomesback empty, so I have the job's status and duration and nothing else. The
failing run is https://git.eeqj.de/sneak/lora.vegas/actions/runs/25 — @sneak,
you can read those logs and I cannot.
The 22-second duration says it did real work before failing rather than dying
instantly on a config parse error, but I am not going to guess further without
evidence. Candidate causes worth checking in the log, in rough order of my
suspicion: how
act_runnerprovidesnodefor JavaScript actions in acontainer job (it may inject a glibc-linked node that cannot execute against
musl, which no amount of
apk add nodejswould fix), whethershell: shishonoured as a job default, and whether the
apk addstep itself failed.The real lesson — this is a process failure, not just a bad patch
This change was reviewed twice, adversarially, by two independent reviewers.
The second reviewer went as far as executing the pinned
actions/checkoutandactions/upload-artifactbundles under the container's own node and confirmingthey load. It still shipped a break.
The reason is structural and was named in the issue from the start:
deploy.ymlonly triggers on push to
main, so no amount of pre-merge review could executeit. Every verification anyone performed simulated the environment. None of it
was the environment.
script/cibuildcannot help here either —docker buildruns
RUNunder/bin/shwith no Actions runtime at all.New hard requirement for the next attempt
Reviewing harder will not fix an unobservable path. The next attempt must make
the path observable before it reaches
main:deploy.yml'son.push.branchesfor theworking branch, so the
buildjob actually executes underact_runnerduring development. Pass/fail is visible through the commit-status API,
which
clawbotcan read even without log access.buildjob goes green on the branch.branches: [main].The
deployjob will still be unexercised (it needsCLOUDFLARE_API_TOKENandwould publish), but
buildis where this failed and where it can be proven.If a log excerpt from run 25 can be pasted into this issue, that shortcuts the
whole loop — otherwise the next attempt proceeds by iterating against the
branch signal.
Everything else about the reverted work stands: all six pins were independently
re-resolved and correct, and the reverted commits remain in history at
3f91a7candb157bfdfor reuse.Manager note — refinement to the "make it observable" requirement above, so
the next attempt does not create a second problem while fixing the first.
Adding the working branch to
on.push.branchesmakes both jobs run on thatbranch, not just
build. Thedeployjob would then executewrangler pages deploy public --project-name=lora-vegas --branch=<branch>against the real Cloudflare Pages project using the real
CLOUDFLARE_API_TOKEN. On Cloudflare Pages a non-production branch nameproduces a preview deployment rather than overwriting production, so this is
not catastrophic, but it still spends the token and litters the project with
preview deployments on every iteration. That is not acceptable as a routine
development loop.
So the requirement is: temporary branch trigger plus a guard on the deploy
job so it cannot run off
main:With that, a branch push exercises exactly the job that failed (
build) andthe
deployjob reports as skipped, touching nothing external.Worth keeping the
if:guard permanently even after the temporary trigger isremoved. It costs one line and it means any future branch trigger — added
deliberately or by accident — cannot reach Cloudflare. Defence in depth against
precisely the class of mistake this issue is now about. I would keep it; if
there is a reason not to, say so and it comes out in the final commit along
with the temporary trigger.
Sequence for the next attempt:
main(3d17e22, the revert).3f91a7candb157bfd— the six pins wereindependently verified correct and should be reused as-is, not re-resolved.
if:guard.buildjob result from the commit-status API. Iterateagainst that real signal until it is green.
branches: [main]andthe
if:guard.mainrun — which bythen will be the second time that exact job configuration has run, not the
first.
This is actionable without waiting on the run 25 logs. If those logs do get
pasted here they will still shortcut step 4 considerably.
Recovery confirmed. The revert (
3d17e22) produced a fully greenmainrun:check / check (push)— success in 4sBuild and Deploy to Cloudflare Pages / build (push)— success in 5sBuild and Deploy to Cloudflare Pages / deploy (push)— success in 19shttps://lora.vegas/returns HTTP 200 with the expected title and content. Thedeploy path is healthy again and a real deployment has completed end to end.
One diagnostic detail worth recording, because it narrows the search for the
next attempt: the restored
buildjob succeeds in 5 seconds, while thereverted alpine-based
buildjob failed after 22 seconds. The old job isfast because
klakegg/hugo:ext-alpineships hugo preinstalled and does nopackage installation at all. Twenty-two seconds is about what
apk add --no-cache nodejs git tarfollowed byscript/bootstrapwould cost beforesomething downstream failed — so the failure most likely occurred at or after
the checkout step, not at container startup or YAML parsing.
That is consistent with the
act_runner/node hypothesis and inconsistent with"the
apk addstep itself failed" (which would have failed faster and wouldnot have gotten as far). Still a hypothesis, not a diagnosis — the run 25 logs
remain the fast path.
This also means the comparison is not purely like-for-like on cost: the current
approach trades a 5-second build for a ~20-second one in exchange for dropping
an abandoned image and reusing the repo's own pinned base. That trade is still
clearly worth making, but it is worth stating plainly rather than discovering
later.
Implementation plan (second attempt)
Branching from
mainat3d17e22aspin-deploy-refs-observable.The six pinned values from the reverted work are reused verbatim; they were
independently re-resolved twice and are not in question. What is in question is
whether the
buildjob can execute at all on a bare musl alpine underact_runner, and that is what this attempt actually measures instead ofarguing about.
Observability strategy
I have the same access limitation as before: the Actions jobs/logs API 403s for
clawbot, so I cannot read run 25's log. The only readable signal is thecommit-status API, which reports one entry per job with
context,statusand
description.So the plan is to encode the diagnosis as job topology rather than as log
output. Alongside the temporary branch trigger on
deploy.yml, I will add atemporary
.gitea/workflows/probe.ymltriggered only on this branch,containing several small jobs that each isolate one hypothesis. Every job shows
up as its own
contextin the commit-status API, so a single push bisects thefailure in parallel rather than one hypothesis per round trip.
Probe jobs, first round:
actions/checkout— control; tells me whetheract_runnerinjects anodeof its own (if this passes, the wholenode-prerequisite theory is wrong).
apk add --no-cache nodejs git tar+ pinned checkout —exactly the reverted
b157bfdprefix. Isolates checkout from everythingdownstream.
script/bootstrap+script/test— isolates the site buildinside the Actions container from the JS-action machinery.
actions/upload-artifactv4 (pinned) — isolates theartifact upload, which is the one step whose protocol changed in this issue
(v3 -> v4) and which the ~22s timing is consistent with reaching.
node:20-alpinepinned by digest + pinned checkout — the fallback imagedirection, measured rather than assumed.
Then iterate on real results until the real
buildjob indeploy.ymlisgreen on the branch.
Deploy guard
Per the refinement above,
deploygetsif: github.ref_name == 'main'so abranch push cannot reach Cloudflare. That guard stays permanently. The
temporary branch trigger and the whole
probe.ymlfile are removed in thefinal commit, leaving
branches: [main].Scope
Unchanged from the issue: two jobs stay separate, no renovate/dependabot, no
package.json, no re-resolution of the pinned values. If the evidence showsthe mandated "reuse the
Dockerfile's alpine digest" requirement cannot workunder
act_runner, I will deviate and say so explicitly in the PR body withthe observed evidence, since that requirement is @sneak's to re-evaluate.
Gate:
make checkandscript/cibuild, plus the real branch run. I will stateplainly in the PR body that the
deployjob remains unexercised — it needsCLOUDFLARE_API_TOKENand would publish — and I will not claim otherwise.Second attempt is up as #22, branch
pin-deploy-refs-observable, labelledneeds-review.The
buildjob is green on the branch under a real runner. Observed at73f912cvia the commit-status API:The observability requirement worked, and it earned its keep immediately: it
found two breaks, not one, and neither was the thing everyone was
suspecting.
The image was never the problem. A probe job of bare alpine + checkout
failed while the same job with
apk add nodejs git tarfirst passed, and afurther job adding
script/bootstrap+script/testalso passed. Thepinned alpine digest, the prerequisite step and the site build are all fine.
What failed was
actions/upload-artifactv4 — two otherwise identicaljobs, one ending in the v4 upload (failed after 11s) and one without it
(passed in 5s). Artifacts v4 is a different wire protocol and this instance
does not serve it. So the v3 -> v4 bump in DoD item 2 is the thing that
broke run 25, not the
klakeggreplacement. Filed as #20.A second break was waiting behind it, in the
deployjob, which nobodyhad ever run either.
wrangler@4.120.0requires node >= 22 and refuses tostart on the pinned node 20 container;
npm installonly warns aboutengines, so the install step passes and the deploy step fails. And theunpinned
npm install -g wranglerit replaced was never installinglatestanyway — npm picks the newest version the running node satisfies, which on
node 20 is 4.86.0. So the "pin the current version" change silently moved
the deploy onto a wrangler that cannot run. Pinned at 4.86.0, the version
actually in production. Filed as #21.
Both deviations from the definition of done are stated explicitly in the PR
body with the evidence, since they are yours to re-evaluate. DoD item 3 (reuse
the
Dockerfile's alpine digest) is kept — the evidence says it works.The temporary branch trigger and the temporary probe workflow are removed in
the final commit
54ed637. Theif: github.ref_name == 'main'guard is keptpermanently, as suggested.
Still unproven and not claimed: the
deployjob has never executed. A probepair rehearsed it end to end — same pinned node digest, same pinned download
action, same pinned wrangler, real site tarball downloaded and extracted —
stopping at
wrangler pages deploy --helpinstead of publishing. Themainrun after merge is still the first time the real publish path runs, and it must
be watched with the live site confirmed.
Manager note — the root cause is found, and the requirement I wrote in this
issue is what broke production.
PR #22 (#22) developed against a real
act_runnersignal, using a temporary branch trigger plus theif:guard.Round 1 reproduced the
mainfailure exactly, then probe jobs bisected it.The image replacement was never the cause. Two otherwise-identical probe
jobs differing only by a trailing
upload-artifactv4 step: 5s pass withoutit, 11s fail with it. Bare alpine plus
apk add nodejs git tarplus checkoutpasses, and adding
script/bootstrapandscript/teststill passes. Theklakegg→ pinned-alpine swap works fine.What broke the deploy was DoD item 2 of this issue — my requirement that
upload-artifactanddownload-artifactmove from v3 to v4. Artifacts v4 is adifferent wire protocol, and this Gitea instance does not serve it. I wrote
that requirement because v3 is deprecated on GitHub, and I did not check
whether the Gitea instance we actually run on implements v4. That is the same
mistake I have been criticising in this thread: reasoning about the environment
instead of testing it.
Worse, it survived review by being actively confirmed. The second reviewer of
PR #17 explicitly cleared the v4 change on the grounds that Gitea 1.25.4 is
"well past artifact-v4 support". That was a plausible-sounding inference that
happened to be false, and it is exactly the kind of finding an adversarial
review is supposed to catch rather than manufacture.
DoD item 2 is hereby amended. The artifact actions stay on v3, pinned by
commit SHA at the exact commits
@v3was already resolving to. Pinning is therequirement; the major-version bump was my addition and it was wrong. The
policy rule is "pin by cryptographic hash" — it says nothing about chasing
upstream majors, and doing so on an unverified assumption is how this issue
took down the deploy.
A second break was hiding behind the first
wrangler@4.120.0declaresengines.node >=22.0.0and exits 1 on the pinnednode 20 container.
npm installonly warns aboutengines, so the installstep would have gone green and the deploy step would have failed — a second
outage, revealed only after the first was fixed.
The subtle part is worth recording. The unpinned
npm install -g wrangleritreplaced was not installing
latest: npm resolves a bare package name tothe newest version the running node satisfies, which on node 20 is 4.86.0. So
"pin the version that
latestcurrently points at" silently moved the deployonto a wrangler that cannot start on our node. Pinning a mutable reference
means pinning what it actually resolves to in this environment, not what it
resolves to in general. PR #22 pins 4.86.0 and leaves the node digest alone.
Both deviations are tracked: #20 (artifacts v3/v4) and #21 (wrangler/node
version relationship).
Status
PR #22 has a green
buildjob on a realact_runnerrun — the first time anyattempt at this issue has demonstrated that. An independent reviewer who was
not involved in any prior round is reviewing now, with instructions to treat
all prior reasoning including mine as suspect.
The
deployjob remains unexercised and unexercisable pre-merge. It will bewatched on
mainimmediately after merge, with an immediate revert if itfails.
Done. Merged and verified on
main.PR #22 merged as
9959cb5. Themainrun — including the job that has neversuccessfully executed with pinned references until now:
check / check (push)— success in 4sBuild and Deploy to Cloudflare Pages / build (push)— success in 9sBuild and Deploy to Cloudflare Pages / deploy (push)— success in 18shttps://lora.vegas/returns HTTP 200, 4226 bytes, with the expected title andthe mesh channel and Signal group content intact. DoD item 8 is now
genuinely satisfied — the deploy path has run end to end with every external
reference pinned, which is the thing this issue was actually about and which no
previous attempt ever demonstrated.
Final state of
deploy.yml: both container images pinned by digest, all threeuses:pinned by 40-hex commit SHA, the wrangler install pinned to an exactversion that actually runs on the pinned node,
branches: [main]only, theif: github.ref_name == 'main'guard on the deploy job, 4-space YAML. Theabandoned
klakegg/hugo:ext-alpineis gone; the deploy build and the checkbuild now share one pinned base and one dependency list.
What this cost and what it bought
Two merges, one production break, one revert, three implementation rounds and
four reviews. The break was caused by a requirement I wrote (the artifact v3 →
v4 bump) on an assumption I never tested, and it was confirmed rather than
caught by a review that reasoned its way to a false conclusion about this Gitea
instance.
What actually fixed it was not reviewing harder. It was making the path
executable before merging it — a temporary branch trigger plus a guard so
branch runs could not reach Cloudflare, then bisecting with probe jobs against
real runner output. That found the true cause in one pass and incidentally
exposed a second latent outage (
wrangler@4.120.0requires node >= 22 andexits 1 on the pinned node 20 image;
npm installonly warns aboutengines,so the install step would have gone green and the deploy step would have
failed).
The generalisable lessons, recorded here because they outlive this issue:
review. Both reviews of PR #17 were reasoning about an environment neither
could run. Where a workflow triggers only on
main, make it temporarilyobservable rather than trusting analysis.
pinning. Chasing upstream majors on an unverified assumption is what took
the deploy down.
environment.
npm install -g wrangleron node 20 resolves to 4.86.0, notto
latest. Pinning "what latest points at" silently changed behaviour.script/cibuildwas green for every broken revision, becausedocker buildruns
RUNunder/bin/shwith no Actions runtime at all.Follow-ups, none blocking
runs.using: 'node16'durabilitynote).
submodules: recursiveis a no-op here (no.gitmodules,themesis a plain tree), which also undercuts the statedjustification for installing
git; andapk add taris likely redundantagainst busybox tar.
Closing.
clawbot referenced this issue2026-09-03 15:17:15 +02:00