Hash-pin every external reference in .gitea/workflows/deploy.yml #7

Closed
opened 2026-08-09 03:41:16 +02:00 by clawbot · 9 comments
Collaborator

Problem

.gitea/workflows/deploy.yml is the only file in the repo that still uses
mutable external references. REPO_POLICIES.md calls hash-pinning "the single
most important rule in this document" and states there are "zero exceptions".
.gitea/workflows/check.yml, the Dockerfile, and everything else in the org
are already pinned; this file was written before the scaffold landed in #4 and
was never brought up to standard.

Five mutable references, plus one unpinned remote install:

Line Reference Problem
build.container.image klakegg/hugo:ext-alpine mutable tag; also an abandoned image (last release 2021)
build.steps actions/checkout@v4 mutable tag
build.steps actions/upload-artifact@v3 mutable tag; v3 is deprecated
deploy.container.image node:20 mutable tag
deploy.steps actions/download-artifact@v3 mutable tag; v3 is deprecated
deploy.steps npm install -g wrangler unversioned remote install — arbitrary remote code execution at deploy time

Two further divergences in the same file:

  • The on.push.branches list still includes feat/initial-site, a branch that
    is fully merged into main (0 unique commits) and is scheduled for deletion.
    A workflow trigger on a dead branch is dead config.
  • The file uses 2-space YAML indentation. .editorconfig (canonical, [*] indent_size = 4) and the canonical .gitea/workflows/check.yml both use
    4-space. check.yml in this repo already uses 4-space, so the two workflow
    files are inconsistent with each other.

Definition of done

  1. Every image: in deploy.yml is referenced as image@sha256:<digest>,
    with a comment on the line above in the canonical format
    # <name> <version>, YYYY-MM-DD.
  2. Every uses: in deploy.yml is referenced by a full 40-hex commit SHA,
    with the same comment format. upload-artifact and download-artifact
    move from v3 to v4 (v3 is deprecated) and are pinned to v4's commit SHA.
  3. klakegg/hugo:ext-alpine is replaced. Required approach: use the same
    pinned alpine digest the Dockerfile already uses and run
    script/bootstrap to install hugo, then script/test to build. This makes
    the deploy build and the check build use one pinned base and one dependency
    list instead of two, and removes the dependency on an image that has been
    unmaintained for five years. Do not introduce a second, differently-pinned
    Hugo image.
  4. npm install -g wrangler becomes npm install -g wrangler@<exact version>
    with a version/date comment. (The "use yarn, not npm" rule in
    REPO_POLICIES.md governs project dependency management via a
    package.json/lockfile; this repo has neither, and this is a one-shot CLI
    install in CI. Pinning the exact version is what satisfies the
    hash-pinning rule here. Do not add a package.json for this.)
  5. feat/initial-site is removed from on.push.branches, leaving main.
  6. The file is reindented to 4-space YAML to match check.yml and the
    canonical template.
  7. make check passes and script/cibuild succeeds.
  8. The workflow's YAML is verified to parse (e.g. the Gitea Actions run for the
    PR branch does not fail with a config error).

Explicitly out of scope

  • Do not collapse the two jobs into one. It would remove the artifact
    actions entirely, but it changes the shape of a live deploy path; keep this
    change to pinning so a deploy regression can be attributed unambiguously.
  • Do not add renovate/dependabot config. No org standard requires one, and
    automated version bumps are in tension with hash-pinning.

Risk note

This file is the live deploy path for https://lora.vegas/ and it cannot be
exercised from a PR branch — the Cloudflare Pages deploy only runs on push to
main and needs the CLOUDFLARE_API_TOKEN secret. The PR gate
(script/cibuild) proves the site still builds; it does not prove the deploy
still works. After merge, the deploy run on main must be watched and the live
site confirmed. This is called out in TODO.md under Future Steps ("Verify the
Cloudflare Pages deploy still works after the workflow changes").

Ref: REPO_POLICIES.md — "ALL external references must be pinned by
cryptographic hash", "When pinning images or packages by hash, add a comment
above the reference with the version and date (YYYY-MM-DD)".

## Problem `.gitea/workflows/deploy.yml` is the only file in the repo that still uses mutable external references. `REPO_POLICIES.md` calls hash-pinning "the single most important rule in this document" and states there are "zero exceptions". `.gitea/workflows/check.yml`, the `Dockerfile`, and everything else in the org are already pinned; this file was written before the scaffold landed in #4 and was never brought up to standard. Five mutable references, plus one unpinned remote install: | Line | Reference | Problem | | --- | --- | --- | | `build.container.image` | `klakegg/hugo:ext-alpine` | mutable tag; also an abandoned image (last release 2021) | | `build.steps` | `actions/checkout@v4` | mutable tag | | `build.steps` | `actions/upload-artifact@v3` | mutable tag; v3 is deprecated | | `deploy.container.image` | `node:20` | mutable tag | | `deploy.steps` | `actions/download-artifact@v3` | mutable tag; v3 is deprecated | | `deploy.steps` | `npm install -g wrangler` | unversioned remote install — arbitrary remote code execution at deploy time | Two further divergences in the same file: - The `on.push.branches` list still includes `feat/initial-site`, a branch that is fully merged into `main` (0 unique commits) and is scheduled for deletion. A workflow trigger on a dead branch is dead config. - The file uses 2-space YAML indentation. `.editorconfig` (canonical, `[*] indent_size = 4`) and the canonical `.gitea/workflows/check.yml` both use 4-space. `check.yml` in this repo already uses 4-space, so the two workflow files are inconsistent with each other. ## Definition of done 1. Every `image:` in `deploy.yml` is referenced as `image@sha256:<digest>`, with a comment on the line above in the canonical format `# <name> <version>, YYYY-MM-DD`. 2. Every `uses:` in `deploy.yml` is referenced by a full 40-hex commit SHA, with the same comment format. `upload-artifact` and `download-artifact` move from v3 to v4 (v3 is deprecated) and are pinned to v4's commit SHA. 3. `klakegg/hugo:ext-alpine` is replaced. **Required approach:** use the same pinned `alpine` digest the `Dockerfile` already uses and run `script/bootstrap` to install hugo, then `script/test` to build. This makes the deploy build and the check build use one pinned base and one dependency list instead of two, and removes the dependency on an image that has been unmaintained for five years. Do not introduce a second, differently-pinned Hugo image. 4. `npm install -g wrangler` becomes `npm install -g wrangler@<exact version>` with a version/date comment. (The "use yarn, not npm" rule in `REPO_POLICIES.md` governs project dependency management via a `package.json`/lockfile; this repo has neither, and this is a one-shot CLI install in CI. Pinning the exact version is what satisfies the hash-pinning rule here. Do not add a `package.json` for this.) 5. `feat/initial-site` is removed from `on.push.branches`, leaving `main`. 6. The file is reindented to 4-space YAML to match `check.yml` and the canonical template. 7. `make check` passes and `script/cibuild` succeeds. 8. The workflow's YAML is verified to parse (e.g. the Gitea Actions run for the PR branch does not fail with a config error). ## Explicitly out of scope - **Do not** collapse the two jobs into one. It would remove the artifact actions entirely, but it changes the shape of a live deploy path; keep this change to pinning so a deploy regression can be attributed unambiguously. - **Do not** add renovate/dependabot config. No org standard requires one, and automated version bumps are in tension with hash-pinning. ## Risk note This file is the live deploy path for https://lora.vegas/ and it cannot be exercised from a PR branch — the Cloudflare Pages deploy only runs on push to `main` and needs the `CLOUDFLARE_API_TOKEN` secret. The PR gate (`script/cibuild`) proves the site still builds; it does not prove the deploy still works. After merge, the deploy run on `main` must be watched and the live site confirmed. This is called out in `TODO.md` under Future Steps ("Verify the Cloudflare Pages deploy still works after the workflow changes"). Ref: `REPO_POLICIES.md` — "ALL external references must be pinned by cryptographic hash", "When pinning images or packages by hash, add a comment above the reference with the version and date (YYYY-MM-DD)".
Author
Collaborator

Manager note — this is the top item on the pre-1.0 queue and is being
implemented now.

Ordering rationale: the repo standards rank divergences by severity as
(1) hash-pinning and secrets, (2) required-file and Makefile/Docker gaps,
(3) formatting and style, (4) code-review items. This issue is the only
category-1 divergence in the repo, so it goes first.

Full pre-1.0 queue as filed, in intended order:

  1. #7 — hash-pin deploy.yml (this issue) — in progress
  2. #9script/check does not run script/lint, so lint is ungated in CI
  3. #8 — canonical policy dotfiles (REPO_POLICIES.md, .editorconfig,
    .prettierrc, .prettierignore, .gitignore)
  4. #13 — eliminate the Hugo taxonomy build warning
  5. #16 — 30-second timeout guard on script/test and script/lint
  6. #12 — widen the prettier gate to CSS and all Markdown (depends on #8)
  7. #11 — restructure README.md into the canonical section set
  8. #14 — Cloudflare Pages _headers with baseline security headers
  9. #10LICENSEblocked on @sneak (license choice is the owner's by
    policy)
  10. #15 — delete stale remote branches — partially blocked on @sneak
    (security-audit deletion); feat/initial-site unblocks once this issue
    lands

Note that #9 is sequenced second deliberately: script/lint currently runs
nowhere in the gate, so until it is fixed, every PR in this queue is being
verified by a weaker check than the repo advertises.

Implementation requirements are in the issue body above. main is unprotected,
so this will be merged directly once an independent adversarial review passes.

Manager note — this is the top item on the pre-1.0 queue and is being implemented now. Ordering rationale: the repo standards rank divergences by severity as (1) hash-pinning and secrets, (2) required-file and Makefile/Docker gaps, (3) formatting and style, (4) code-review items. This issue is the only category-1 divergence in the repo, so it goes first. Full pre-1.0 queue as filed, in intended order: 1. #7 — hash-pin `deploy.yml` (this issue) — **in progress** 2. #9 — `script/check` does not run `script/lint`, so lint is ungated in CI 3. #8 — canonical policy dotfiles (`REPO_POLICIES.md`, `.editorconfig`, `.prettierrc`, `.prettierignore`, `.gitignore`) 4. #13 — eliminate the Hugo taxonomy build warning 5. #16 — 30-second timeout guard on `script/test` and `script/lint` 6. #12 — widen the prettier gate to CSS and all Markdown (depends on #8) 7. #11 — restructure `README.md` into the canonical section set 8. #14 — Cloudflare Pages `_headers` with baseline security headers 9. #10 — `LICENSE` — **blocked on @sneak** (license choice is the owner's by policy) 10. #15 — delete stale remote branches — **partially blocked on @sneak** (`security-audit` deletion); `feat/initial-site` unblocks once this issue lands Note that #9 is sequenced second deliberately: `script/lint` currently runs nowhere in the gate, so until it is fixed, every PR in this queue is being verified by a weaker check than the repo advertises. Implementation requirements are in the issue body above. `main` is unprotected, so this will be merged directly once an independent adversarial review passes.
Author
Collaborator

Implementation plan

Branching from main as pin-deploy-workflow-refs. Single commit,
.gitea/workflows/deploy.yml plus TODO.md.

All references resolved and verified against upstream before writing them
(no guessed values):

Images (Docker Registry v2 Docker-Content-Digest, cross-checked against
the Docker Hub tag API):

  • klakegg/hugo:ext-alpine is dropped entirely. The build job container
    becomes the exact digest the Dockerfile already pins,
    alpine@sha256:c3f8e73f...53f4709 (alpine 3.21). Steps become
    script/bootstrap (installs hugo via apk, as the Dockerfile already
    does) then script/test (the hugo --minify build that replaces the raw
    hugo --minify invocation). One pinned base, one dependency list.
  • node:20 becomes node@sha256:8f693eaa...1c9ba5, which is node 20.20.2
    (bookworm). No version change, just the digest that tag points at today.

Actions (GitHub tag->commit API, each tag ref dereferenced to confirm it
is a commit object):

  • actions/checkout@v4 -> 11bd71901bbe5b1630ceea73d27597364c9af683
    (v4.2.2) — the same SHA .gitea/workflows/check.yml already pins, so both
    workflows agree.
  • actions/upload-artifact@v3 -> v4.6.2,
    ea165f8d65b6e75b540449e92b4886f43607fa02.
  • actions/download-artifact@v3 -> v4.3.0,
    d3f86a106a0bac45b974a628896c90dbdf5c8093.

Remote install: npm install -g wrangler becomes
npm install -g wrangler@4.120.0 (npm registry dist-tags.latest, published
2026-08-07). No package.json added, per the issue.

Every pinned line gets # <name> <version>, YYYY-MM-DD on the line above.

Also in the same change: drop feat/initial-site from on.push.branches,
leaving main; reindent the whole file to 4-space YAML to match check.yml.

Out of scope and deliberately not done: the two jobs stay separate, no
renovate/dependabot config, no package.json, no actions/checkout major
bump beyond what check.yml already uses.

TODO.md: the completed entry goes to the top of Completed Steps, the
"Pin the images and actions in deploy.yml by sha256" line is removed from
Future Steps, and Next Step advances per the Workflow block.

Gate: make check and script/cibuild. Noting up front that neither
exercises the Cloudflare Pages deploy — that path only runs on push to main
and needs CLOUDFLARE_API_TOKEN, so post-merge verification of the live
deploy is required.

Implementation plan =================== Branching from `main` as `pin-deploy-workflow-refs`. Single commit, `.gitea/workflows/deploy.yml` plus `TODO.md`. All references resolved and verified against upstream before writing them (no guessed values): **Images** (Docker Registry v2 `Docker-Content-Digest`, cross-checked against the Docker Hub tag API): - `klakegg/hugo:ext-alpine` is dropped entirely. The `build` job container becomes the exact digest the `Dockerfile` already pins, `alpine@sha256:c3f8e73f...53f4709` (alpine 3.21). Steps become `script/bootstrap` (installs hugo via apk, as the `Dockerfile` already does) then `script/test` (the `hugo --minify` build that replaces the raw `hugo --minify` invocation). One pinned base, one dependency list. - `node:20` becomes `node@sha256:8f693eaa...1c9ba5`, which is node 20.20.2 (bookworm). No version change, just the digest that tag points at today. **Actions** (GitHub tag->commit API, each tag ref dereferenced to confirm it is a commit object): - `actions/checkout@v4` -> `11bd71901bbe5b1630ceea73d27597364c9af683` (v4.2.2) — the same SHA `.gitea/workflows/check.yml` already pins, so both workflows agree. - `actions/upload-artifact@v3` -> v4.6.2, `ea165f8d65b6e75b540449e92b4886f43607fa02`. - `actions/download-artifact@v3` -> v4.3.0, `d3f86a106a0bac45b974a628896c90dbdf5c8093`. **Remote install**: `npm install -g wrangler` becomes `npm install -g wrangler@4.120.0` (npm registry `dist-tags.latest`, published 2026-08-07). No `package.json` added, per the issue. Every pinned line gets `# <name> <version>, YYYY-MM-DD` on the line above. Also in the same change: drop `feat/initial-site` from `on.push.branches`, leaving `main`; reindent the whole file to 4-space YAML to match `check.yml`. Out of scope and deliberately not done: the two jobs stay separate, no renovate/dependabot config, no `package.json`, no `actions/checkout` major bump beyond what `check.yml` already uses. `TODO.md`: the completed entry goes to the top of Completed Steps, the "Pin the images and actions in `deploy.yml` by sha256" line is removed from Future Steps, and Next Step advances per the Workflow block. Gate: `make check` and `script/cibuild`. Noting up front that neither exercises the Cloudflare Pages deploy — that path only runs on push to `main` and needs `CLOUDFLARE_API_TOKEN`, so post-merge verification of the live deploy is required.
clawbot reopened this issue 2026-08-09 04:37:37 +02:00
Author
Collaborator

Reopened. The merged change broke the deploy on main and has been
reverted.

What happened

PR #17 merged as 74c28c1. The resulting main run:

  • check / check (push) — success in 3s
  • Build and Deploy to Cloudflare Pages / build (push)failure after 22s
  • Build and Deploy to Cloudflare Pages / deploy (push) — skipped
    (needs: build)

So the deploy did not run. Reverted on main as 3d17e22, restoring the
previous known-working deploy.yml (the klakegg/hugo:ext-alpine version).
script/cibuild was green on the revert before pushing. The live site was
never down — Cloudflare Pages continued serving the last successful deploy —
but no new deploy could happen, and main must not sit in that state.

What I cannot tell you yet

I cannot read the failure logs. The Gitea Actions jobs API returns 403 for
clawbot (user should be the owner of the repo) and list_run_jobs comes
back empty, so I have the job's status and duration and nothing else. The
failing run is https://git.eeqj.de/sneak/lora.vegas/actions/runs/25@sneak,
you can read those logs and I cannot.

The 22-second duration says it did real work before failing rather than dying
instantly on a config parse error, but I am not going to guess further without
evidence. Candidate causes worth checking in the log, in rough order of my
suspicion: how act_runner provides node for JavaScript actions in a
container job (it may inject a glibc-linked node that cannot execute against
musl, which no amount of apk add nodejs would fix), whether shell: sh is
honoured as a job default, and whether the apk add step itself failed.

The real lesson — this is a process failure, not just a bad patch

This change was reviewed twice, adversarially, by two independent reviewers.
The second reviewer went as far as executing the pinned actions/checkout and
actions/upload-artifact bundles under the container's own node and confirming
they load. It still shipped a break.

The reason is structural and was named in the issue from the start: deploy.yml
only triggers on push to main, so no amount of pre-merge review could execute
it.
Every verification anyone performed simulated the environment. None of it
was the environment. script/cibuild cannot help here either — docker build
runs RUN under /bin/sh with no Actions runtime at all.

New hard requirement for the next attempt

Reviewing harder will not fix an unobservable path. The next attempt must make
the path observable before it reaches main:

  1. Add a temporary branch trigger to deploy.yml's on.push.branches for the
    working branch, so the build job actually executes under act_runner
    during development. Pass/fail is visible through the commit-status API,
    which clawbot can read even without log access.
  2. Iterate on the real signal until the build job goes green on the branch.
  3. Remove the temporary trigger in the final commit, leaving branches: [main].
  4. Only then merge.

The deploy job will still be unexercised (it needs CLOUDFLARE_API_TOKEN and
would publish), but build is where this failed and where it can be proven.

If a log excerpt from run 25 can be pasted into this issue, that shortcuts the
whole loop — otherwise the next attempt proceeds by iterating against the
branch signal.

Everything else about the reverted work stands: all six pins were independently
re-resolved and correct, and the reverted commits remain in history at
3f91a7c and b157bfd for reuse.

Reopened. **The merged change broke the deploy on `main` and has been reverted.** ## What happened PR #17 merged as `74c28c1`. The resulting `main` run: - `check / check (push)` — success in 3s - `Build and Deploy to Cloudflare Pages / build (push)` — **failure after 22s** - `Build and Deploy to Cloudflare Pages / deploy (push)` — skipped (`needs: build`) So the deploy did not run. Reverted on `main` as `3d17e22`, restoring the previous known-working `deploy.yml` (the `klakegg/hugo:ext-alpine` version). `script/cibuild` was green on the revert before pushing. The live site was never down — Cloudflare Pages continued serving the last successful deploy — but no new deploy could happen, and `main` must not sit in that state. ## What I cannot tell you yet **I cannot read the failure logs.** The Gitea Actions jobs API returns 403 for `clawbot` (`user should be the owner of the repo`) and `list_run_jobs` comes back empty, so I have the job's status and duration and nothing else. The failing run is https://git.eeqj.de/sneak/lora.vegas/actions/runs/25 — @sneak, you can read those logs and I cannot. The 22-second duration says it did real work before failing rather than dying instantly on a config parse error, but I am not going to guess further without evidence. Candidate causes worth checking in the log, in rough order of my suspicion: how `act_runner` provides `node` for JavaScript actions in a container job (it may inject a glibc-linked node that cannot execute against musl, which no amount of `apk add nodejs` would fix), whether `shell: sh` is honoured as a job default, and whether the `apk add` step itself failed. ## The real lesson — this is a process failure, not just a bad patch This change was reviewed twice, adversarially, by two independent reviewers. The second reviewer went as far as executing the pinned `actions/checkout` and `actions/upload-artifact` bundles under the container's own node and confirming they load. It still shipped a break. The reason is structural and was named in the issue from the start: **`deploy.yml` only triggers on push to `main`, so no amount of pre-merge review could execute it.** Every verification anyone performed simulated the environment. None of it was the environment. `script/cibuild` cannot help here either — `docker build` runs `RUN` under `/bin/sh` with no Actions runtime at all. ## New hard requirement for the next attempt Reviewing harder will not fix an unobservable path. The next attempt must make the path observable **before** it reaches `main`: 1. Add a temporary branch trigger to `deploy.yml`'s `on.push.branches` for the working branch, so the `build` job actually executes under `act_runner` during development. Pass/fail is visible through the commit-status API, which `clawbot` can read even without log access. 2. Iterate on the real signal until the `build` job goes green on the branch. 3. Remove the temporary trigger in the final commit, leaving `branches: [main]`. 4. Only then merge. The `deploy` job will still be unexercised (it needs `CLOUDFLARE_API_TOKEN` and would publish), but `build` is where this failed and where it can be proven. If a log excerpt from run 25 can be pasted into this issue, that shortcuts the whole loop — otherwise the next attempt proceeds by iterating against the branch signal. Everything else about the reverted work stands: all six pins were independently re-resolved and correct, and the reverted commits remain in history at `3f91a7c` and `b157bfd` for reuse.
Author
Collaborator

Manager note — refinement to the "make it observable" requirement above, so
the next attempt does not create a second problem while fixing the first.

Adding the working branch to on.push.branches makes both jobs run on that
branch, not just build. The deploy job would then execute
wrangler pages deploy public --project-name=lora-vegas --branch=<branch>
against the real Cloudflare Pages project using the real
CLOUDFLARE_API_TOKEN. On Cloudflare Pages a non-production branch name
produces a preview deployment rather than overwriting production, so this is
not catastrophic, but it still spends the token and litters the project with
preview deployments on every iteration. That is not acceptable as a routine
development loop.

So the requirement is: temporary branch trigger plus a guard on the deploy
job so it cannot run off main:

    deploy:
        needs: build
        if: github.ref_name == 'main'

With that, a branch push exercises exactly the job that failed (build) and
the deploy job reports as skipped, touching nothing external.

Worth keeping the if: guard permanently even after the temporary trigger is
removed. It costs one line and it means any future branch trigger — added
deliberately or by accident — cannot reach Cloudflare. Defence in depth against
precisely the class of mistake this issue is now about. I would keep it; if
there is a reason not to, say so and it comes out in the final commit along
with the temporary trigger.

Sequence for the next attempt:

  1. Branch from current main (3d17e22, the revert).
  2. Re-apply the reverted work from 3f91a7c and b157bfd — the six pins were
    independently verified correct and should be reused as-is, not re-resolved.
  3. Add the temporary branch trigger and the if: guard.
  4. Push, and read the build job result from the commit-status API. Iterate
    against that real signal until it is green.
  5. Final commit removes the temporary trigger, leaving branches: [main] and
    the if: guard.
  6. Fresh adversarial review, then merge, then watch the main run — which by
    then will be the second time that exact job configuration has run, not the
    first.

This is actionable without waiting on the run 25 logs. If those logs do get
pasted here they will still shortcut step 4 considerably.

Manager note — refinement to the "make it observable" requirement above, so the next attempt does not create a second problem while fixing the first. Adding the working branch to `on.push.branches` makes **both** jobs run on that branch, not just `build`. The `deploy` job would then execute `wrangler pages deploy public --project-name=lora-vegas --branch=<branch>` against the real Cloudflare Pages project using the real `CLOUDFLARE_API_TOKEN`. On Cloudflare Pages a non-production branch name produces a preview deployment rather than overwriting production, so this is not catastrophic, but it still spends the token and litters the project with preview deployments on every iteration. That is not acceptable as a routine development loop. So the requirement is: temporary branch trigger **plus** a guard on the deploy job so it cannot run off `main`: ```yaml deploy: needs: build if: github.ref_name == 'main' ``` With that, a branch push exercises exactly the job that failed (`build`) and the `deploy` job reports as skipped, touching nothing external. Worth keeping the `if:` guard permanently even after the temporary trigger is removed. It costs one line and it means any future branch trigger — added deliberately or by accident — cannot reach Cloudflare. Defence in depth against precisely the class of mistake this issue is now about. I would keep it; if there is a reason not to, say so and it comes out in the final commit along with the temporary trigger. Sequence for the next attempt: 1. Branch from current `main` (`3d17e22`, the revert). 2. Re-apply the reverted work from `3f91a7c` and `b157bfd` — the six pins were independently verified correct and should be reused as-is, not re-resolved. 3. Add the temporary branch trigger and the `if:` guard. 4. Push, and read the `build` job result from the commit-status API. Iterate against that real signal until it is green. 5. Final commit removes the temporary trigger, leaving `branches: [main]` and the `if:` guard. 6. Fresh adversarial review, then merge, then watch the `main` run — which by then will be the second time that exact job configuration has run, not the first. This is actionable without waiting on the run 25 logs. If those logs do get pasted here they will still shortcut step 4 considerably.
Author
Collaborator

Recovery confirmed. The revert (3d17e22) produced a fully green main run:

  • check / check (push) — success in 4s
  • Build and Deploy to Cloudflare Pages / build (push) — success in 5s
  • Build and Deploy to Cloudflare Pages / deploy (push)success in 19s

https://lora.vegas/ returns HTTP 200 with the expected title and content. The
deploy path is healthy again and a real deployment has completed end to end.

One diagnostic detail worth recording, because it narrows the search for the
next attempt: the restored build job succeeds in 5 seconds, while the
reverted alpine-based build job failed after 22 seconds. The old job is
fast because klakegg/hugo:ext-alpine ships hugo preinstalled and does no
package installation at all. Twenty-two seconds is about what apk add --no-cache nodejs git tar followed by script/bootstrap would cost before
something downstream failed — so the failure most likely occurred at or after
the checkout step, not at container startup or YAML parsing.

That is consistent with the act_runner/node hypothesis and inconsistent with
"the apk add step itself failed" (which would have failed faster and would
not have gotten as far). Still a hypothesis, not a diagnosis — the run 25 logs
remain the fast path.

This also means the comparison is not purely like-for-like on cost: the current
approach trades a 5-second build for a ~20-second one in exchange for dropping
an abandoned image and reusing the repo's own pinned base. That trade is still
clearly worth making, but it is worth stating plainly rather than discovering
later.

Recovery confirmed. The revert (`3d17e22`) produced a fully green `main` run: - `check / check (push)` — success in 4s - `Build and Deploy to Cloudflare Pages / build (push)` — success in 5s - `Build and Deploy to Cloudflare Pages / deploy (push)` — **success in 19s** `https://lora.vegas/` returns HTTP 200 with the expected title and content. The deploy path is healthy again and a real deployment has completed end to end. One diagnostic detail worth recording, because it narrows the search for the next attempt: the restored `build` job succeeds in **5 seconds**, while the reverted alpine-based `build` job failed after **22 seconds**. The old job is fast because `klakegg/hugo:ext-alpine` ships hugo preinstalled and does no package installation at all. Twenty-two seconds is about what `apk add --no-cache nodejs git tar` followed by `script/bootstrap` would cost before something downstream failed — so the failure most likely occurred at or after the checkout step, not at container startup or YAML parsing. That is consistent with the `act_runner`/node hypothesis and inconsistent with "the `apk add` step itself failed" (which would have failed faster and would not have gotten as far). Still a hypothesis, not a diagnosis — the run 25 logs remain the fast path. This also means the comparison is not purely like-for-like on cost: the current approach trades a 5-second build for a ~20-second one in exchange for dropping an abandoned image and reusing the repo's own pinned base. That trade is still clearly worth making, but it is worth stating plainly rather than discovering later.
Author
Collaborator

Implementation plan (second attempt)

Branching from main at 3d17e22 as pin-deploy-refs-observable.

The six pinned values from the reverted work are reused verbatim; they were
independently re-resolved twice and are not in question. What is in question is
whether the build job can execute at all on a bare musl alpine under
act_runner, and that is what this attempt actually measures instead of
arguing about.

Observability strategy

I have the same access limitation as before: the Actions jobs/logs API 403s for
clawbot, so I cannot read run 25's log. The only readable signal is the
commit-status API, which reports one entry per job with context, status
and description.

So the plan is to encode the diagnosis as job topology rather than as log
output. Alongside the temporary branch trigger on deploy.yml, I will add a
temporary .gitea/workflows/probe.yml triggered only on this branch,
containing several small jobs that each isolate one hypothesis. Every job shows
up as its own context in the commit-status API, so a single push bisects the
failure in parallel rather than one hypothesis per round trip.

Probe jobs, first round:

  1. bare pinned alpine + pinned actions/checkout — control; tells me whether
    act_runner injects a node of its own (if this passes, the whole
    node-prerequisite theory is wrong).
  2. pinned alpine + apk add --no-cache nodejs git tar + pinned checkout —
    exactly the reverted b157bfd prefix. Isolates checkout from everything
    downstream.
  3. same as 2, plus script/bootstrap + script/test — isolates the site build
    inside the Actions container from the JS-action machinery.
  4. same as 2, plus actions/upload-artifact v4 (pinned) — isolates the
    artifact upload, which is the one step whose protocol changed in this issue
    (v3 -> v4) and which the ~22s timing is consistent with reaching.
  5. node:20-alpine pinned by digest + pinned checkout — the fallback image
    direction, measured rather than assumed.

Then iterate on real results until the real build job in deploy.yml is
green on the branch.

Deploy guard

Per the refinement above, deploy gets if: github.ref_name == 'main' so a
branch push cannot reach Cloudflare. That guard stays permanently. The
temporary branch trigger and the whole probe.yml file are removed in the
final commit, leaving branches: [main].

Scope

Unchanged from the issue: two jobs stay separate, no renovate/dependabot, no
package.json, no re-resolution of the pinned values. If the evidence shows
the mandated "reuse the Dockerfile's alpine digest" requirement cannot work
under act_runner, I will deviate and say so explicitly in the PR body with
the observed evidence, since that requirement is @sneak's to re-evaluate.

Gate: make check and script/cibuild, plus the real branch run. I will state
plainly in the PR body that the deploy job remains unexercised — it needs
CLOUDFLARE_API_TOKEN and would publish — and I will not claim otherwise.

Implementation plan (second attempt) =================================== Branching from `main` at `3d17e22` as `pin-deploy-refs-observable`. The six pinned values from the reverted work are reused verbatim; they were independently re-resolved twice and are not in question. What is in question is whether the `build` job can execute at all on a bare musl alpine under `act_runner`, and that is what this attempt actually measures instead of arguing about. Observability strategy ---------------------- I have the same access limitation as before: the Actions jobs/logs API 403s for `clawbot`, so I cannot read run 25's log. The only readable signal is the commit-status API, which reports one entry per **job** with `context`, `status` and `description`. So the plan is to encode the diagnosis as job topology rather than as log output. Alongside the temporary branch trigger on `deploy.yml`, I will add a **temporary** `.gitea/workflows/probe.yml` triggered only on this branch, containing several small jobs that each isolate one hypothesis. Every job shows up as its own `context` in the commit-status API, so a single push bisects the failure in parallel rather than one hypothesis per round trip. Probe jobs, first round: 1. bare pinned alpine + pinned `actions/checkout` — control; tells me whether `act_runner` injects a `node` of its own (if this passes, the whole node-prerequisite theory is wrong). 2. pinned alpine + `apk add --no-cache nodejs git tar` + pinned checkout — exactly the reverted `b157bfd` prefix. Isolates checkout from everything downstream. 3. same as 2, plus `script/bootstrap` + `script/test` — isolates the site build inside the Actions container from the JS-action machinery. 4. same as 2, plus `actions/upload-artifact` v4 (pinned) — isolates the artifact upload, which is the one step whose protocol changed in this issue (v3 -> v4) and which the ~22s timing is consistent with reaching. 5. `node:20-alpine` pinned by digest + pinned checkout — the fallback image direction, measured rather than assumed. Then iterate on real results until the real `build` job in `deploy.yml` is green on the branch. Deploy guard ------------ Per the refinement above, `deploy` gets `if: github.ref_name == 'main'` so a branch push cannot reach Cloudflare. That guard stays permanently. The temporary branch trigger and the whole `probe.yml` file are removed in the final commit, leaving `branches: [main]`. Scope ----- Unchanged from the issue: two jobs stay separate, no renovate/dependabot, no `package.json`, no re-resolution of the pinned values. If the evidence shows the mandated "reuse the `Dockerfile`'s alpine digest" requirement cannot work under `act_runner`, I will deviate and say so explicitly in the PR body with the observed evidence, since that requirement is @sneak's to re-evaluate. Gate: `make check` and `script/cibuild`, plus the real branch run. I will state plainly in the PR body that the `deploy` job remains unexercised — it needs `CLOUDFLARE_API_TOKEN` and would publish — and I will not claim otherwise.
Author
Collaborator

Second attempt is up as #22, branch pin-deploy-refs-observable, labelled
needs-review.

The build job is green on the branch under a real runner. Observed at
73f912c via the commit-status API:

overall: success
Build and Deploy to Cloudflare Pages / build   success   7s
Build and Deploy to Cloudflare Pages / deploy  skipped
check / check                                  success   6s
probe / s1-build                               success  13s
probe / s2-deploy-dryrun                       success  10s

The observability requirement worked, and it earned its keep immediately: it
found two breaks, not one, and neither was the thing everyone was
suspecting.

  1. The image was never the problem. A probe job of bare alpine + checkout
    failed while the same job with apk add nodejs git tar first passed, and a
    further job adding script/bootstrap + script/test also passed. The
    pinned alpine digest, the prerequisite step and the site build are all fine.
    What failed was actions/upload-artifact v4 — two otherwise identical
    jobs, one ending in the v4 upload (failed after 11s) and one without it
    (passed in 5s). Artifacts v4 is a different wire protocol and this instance
    does not serve it. So the v3 -> v4 bump in DoD item 2 is the thing that
    broke run 25, not the klakegg replacement. Filed as #20.

  2. A second break was waiting behind it, in the deploy job, which nobody
    had ever run either. wrangler@4.120.0 requires node >= 22 and refuses to
    start on the pinned node 20 container; npm install only warns about
    engines, so the install step passes and the deploy step fails. And the
    unpinned npm install -g wrangler it replaced was never installing latest
    anyway — npm picks the newest version the running node satisfies, which on
    node 20 is 4.86.0. So the "pin the current version" change silently moved
    the deploy onto a wrangler that cannot run. Pinned at 4.86.0, the version
    actually in production. Filed as #21.

Both deviations from the definition of done are stated explicitly in the PR
body with the evidence, since they are yours to re-evaluate. DoD item 3 (reuse
the Dockerfile's alpine digest) is kept — the evidence says it works.

The temporary branch trigger and the temporary probe workflow are removed in
the final commit 54ed637. The if: github.ref_name == 'main' guard is kept
permanently, as suggested.

Still unproven and not claimed: the deploy job has never executed. A probe
pair rehearsed it end to end — same pinned node digest, same pinned download
action, same pinned wrangler, real site tarball downloaded and extracted —
stopping at wrangler pages deploy --help instead of publishing. The main
run after merge is still the first time the real publish path runs, and it must
be watched with the live site confirmed.

Second attempt is up as #22, branch `pin-deploy-refs-observable`, labelled `needs-review`. **The `build` job is green on the branch under a real runner.** Observed at `73f912c` via the commit-status API: ``` overall: success Build and Deploy to Cloudflare Pages / build success 7s Build and Deploy to Cloudflare Pages / deploy skipped check / check success 6s probe / s1-build success 13s probe / s2-deploy-dryrun success 10s ``` The observability requirement worked, and it earned its keep immediately: it found **two** breaks, not one, and neither was the thing everyone was suspecting. 1. **The image was never the problem.** A probe job of bare alpine + checkout failed while the same job with `apk add nodejs git tar` first passed, and a further job adding `script/bootstrap` + `script/test` also passed. The pinned alpine digest, the prerequisite step and the site build are all fine. What failed was `actions/upload-artifact` **v4** — two otherwise identical jobs, one ending in the v4 upload (failed after 11s) and one without it (passed in 5s). Artifacts v4 is a different wire protocol and this instance does not serve it. So the v3 -> v4 bump in DoD item 2 is the thing that broke run 25, not the `klakegg` replacement. Filed as #20. 2. **A second break was waiting behind it**, in the `deploy` job, which nobody had ever run either. `wrangler@4.120.0` requires node >= 22 and refuses to start on the pinned node 20 container; `npm install` only warns about `engines`, so the install step passes and the deploy step fails. And the unpinned `npm install -g wrangler` it replaced was never installing `latest` anyway — npm picks the newest version the running node satisfies, which on node 20 is 4.86.0. So the "pin the current version" change silently moved the deploy onto a wrangler that cannot run. Pinned at 4.86.0, the version actually in production. Filed as #21. Both deviations from the definition of done are stated explicitly in the PR body with the evidence, since they are yours to re-evaluate. DoD item 3 (reuse the `Dockerfile`'s alpine digest) is **kept** — the evidence says it works. The temporary branch trigger and the temporary probe workflow are removed in the final commit `54ed637`. The `if: github.ref_name == 'main'` guard is kept permanently, as suggested. Still unproven and not claimed: the `deploy` job has never executed. A probe pair rehearsed it end to end — same pinned node digest, same pinned download action, same pinned wrangler, real site tarball downloaded and extracted — stopping at `wrangler pages deploy --help` instead of publishing. The `main` run after merge is still the first time the real publish path runs, and it must be watched with the live site confirmed.
Author
Collaborator

Manager note — the root cause is found, and the requirement I wrote in this
issue is what broke production.

PR #22 (#22) developed against a real
act_runner signal, using a temporary branch trigger plus the if: guard.
Round 1 reproduced the main failure exactly, then probe jobs bisected it.

The image replacement was never the cause. Two otherwise-identical probe
jobs differing only by a trailing upload-artifact v4 step: 5s pass without
it, 11s fail with it. Bare alpine plus apk add nodejs git tar plus checkout
passes, and adding script/bootstrap and script/test still passes. The
klakegg → pinned-alpine swap works fine.

What broke the deploy was DoD item 2 of this issue — my requirement that
upload-artifact and download-artifact move from v3 to v4. Artifacts v4 is a
different wire protocol, and this Gitea instance does not serve it. I wrote
that requirement because v3 is deprecated on GitHub, and I did not check
whether the Gitea instance we actually run on implements v4. That is the same
mistake I have been criticising in this thread: reasoning about the environment
instead of testing it.

Worse, it survived review by being actively confirmed. The second reviewer of
PR #17 explicitly cleared the v4 change on the grounds that Gitea 1.25.4 is
"well past artifact-v4 support". That was a plausible-sounding inference that
happened to be false, and it is exactly the kind of finding an adversarial
review is supposed to catch rather than manufacture.

DoD item 2 is hereby amended. The artifact actions stay on v3, pinned by
commit SHA at the exact commits @v3 was already resolving to. Pinning is the
requirement; the major-version bump was my addition and it was wrong. The
policy rule is "pin by cryptographic hash" — it says nothing about chasing
upstream majors, and doing so on an unverified assumption is how this issue
took down the deploy.

A second break was hiding behind the first

wrangler@4.120.0 declares engines.node >=22.0.0 and exits 1 on the pinned
node 20 container. npm install only warns about engines, so the install
step would have gone green and the deploy step would have failed — a second
outage, revealed only after the first was fixed.

The subtle part is worth recording. The unpinned npm install -g wrangler it
replaced was not installing latest: npm resolves a bare package name to
the newest version the running node satisfies, which on node 20 is 4.86.0. So
"pin the version that latest currently points at" silently moved the deploy
onto a wrangler that cannot start on our node. Pinning a mutable reference
means pinning what it actually resolves to in this environment, not what it
resolves to in general. PR #22 pins 4.86.0 and leaves the node digest alone.

Both deviations are tracked: #20 (artifacts v3/v4) and #21 (wrangler/node
version relationship).

Status

PR #22 has a green build job on a real act_runner run — the first time any
attempt at this issue has demonstrated that. An independent reviewer who was
not involved in any prior round is reviewing now, with instructions to treat
all prior reasoning including mine as suspect.

The deploy job remains unexercised and unexercisable pre-merge. It will be
watched on main immediately after merge, with an immediate revert if it
fails.

Manager note — the root cause is found, and **the requirement I wrote in this issue is what broke production.** PR #22 (https://git.eeqj.de/sneak/lora.vegas/pulls/22) developed against a real `act_runner` signal, using a temporary branch trigger plus the `if:` guard. Round 1 reproduced the `main` failure exactly, then probe jobs bisected it. **The image replacement was never the cause.** Two otherwise-identical probe jobs differing only by a trailing `upload-artifact` v4 step: 5s pass without it, 11s fail with it. Bare alpine plus `apk add nodejs git tar` plus checkout passes, and adding `script/bootstrap` and `script/test` still passes. The `klakegg` → pinned-alpine swap works fine. What broke the deploy was **DoD item 2 of this issue** — my requirement that `upload-artifact` and `download-artifact` move from v3 to v4. Artifacts v4 is a different wire protocol, and this Gitea instance does not serve it. I wrote that requirement because v3 is deprecated on GitHub, and I did not check whether the Gitea instance we actually run on implements v4. That is the same mistake I have been criticising in this thread: reasoning about the environment instead of testing it. Worse, it survived review by being actively confirmed. The second reviewer of PR #17 explicitly cleared the v4 change on the grounds that Gitea 1.25.4 is "well past artifact-v4 support". That was a plausible-sounding inference that happened to be false, and it is exactly the kind of finding an adversarial review is supposed to catch rather than manufacture. **DoD item 2 is hereby amended.** The artifact actions stay on v3, pinned by commit SHA at the exact commits `@v3` was already resolving to. Pinning is the requirement; the major-version bump was my addition and it was wrong. The policy rule is "pin by cryptographic hash" — it says nothing about chasing upstream majors, and doing so on an unverified assumption is how this issue took down the deploy. ## A second break was hiding behind the first `wrangler@4.120.0` declares `engines.node >=22.0.0` and exits 1 on the pinned node 20 container. `npm install` only warns about `engines`, so the install step would have gone green and the deploy step would have failed — a second outage, revealed only after the first was fixed. The subtle part is worth recording. The unpinned `npm install -g wrangler` it replaced was **not** installing `latest`: npm resolves a bare package name to the newest version the running node satisfies, which on node 20 is 4.86.0. So "pin the version that `latest` currently points at" silently moved the deploy onto a wrangler that cannot start on our node. Pinning a mutable reference means pinning **what it actually resolves to in this environment**, not what it resolves to in general. PR #22 pins 4.86.0 and leaves the node digest alone. Both deviations are tracked: #20 (artifacts v3/v4) and #21 (wrangler/node version relationship). ## Status PR #22 has a green `build` job on a real `act_runner` run — the first time any attempt at this issue has demonstrated that. An independent reviewer who was not involved in any prior round is reviewing now, with instructions to treat all prior reasoning including mine as suspect. The `deploy` job remains unexercised and unexercisable pre-merge. It will be watched on `main` immediately after merge, with an immediate revert if it fails.
Author
Collaborator

Done. Merged and verified on main.

PR #22 merged as 9959cb5. The main run — including the job that has never
successfully executed with pinned references until now:

  • check / check (push) — success in 4s
  • Build and Deploy to Cloudflare Pages / build (push) — success in 9s
  • Build and Deploy to Cloudflare Pages / deploy (push)success in 18s

https://lora.vegas/ returns HTTP 200, 4226 bytes, with the expected title and
the mesh channel and Signal group content intact. DoD item 8 is now
genuinely satisfied
— the deploy path has run end to end with every external
reference pinned, which is the thing this issue was actually about and which no
previous attempt ever demonstrated.

Final state of deploy.yml: both container images pinned by digest, all three
uses: pinned by 40-hex commit SHA, the wrangler install pinned to an exact
version that actually runs on the pinned node, branches: [main] only, the
if: github.ref_name == 'main' guard on the deploy job, 4-space YAML. The
abandoned klakegg/hugo:ext-alpine is gone; the deploy build and the check
build now share one pinned base and one dependency list.

What this cost and what it bought

Two merges, one production break, one revert, three implementation rounds and
four reviews. The break was caused by a requirement I wrote (the artifact v3 →
v4 bump) on an assumption I never tested, and it was confirmed rather than
caught by a review that reasoned its way to a false conclusion about this Gitea
instance.

What actually fixed it was not reviewing harder. It was making the path
executable before merging it — a temporary branch trigger plus a guard so
branch runs could not reach Cloudflare, then bisecting with probe jobs against
real runner output. That found the true cause in one pass and incidentally
exposed a second latent outage (wrangler@4.120.0 requires node >= 22 and
exits 1 on the pinned node 20 image; npm install only warns about engines,
so the install step would have gone green and the deploy step would have
failed).

The generalisable lessons, recorded here because they outlive this issue:

  1. A CI path that cannot be executed pre-merge cannot be validated by
    review.
    Both reviews of PR #17 were reasoning about an environment neither
    could run. Where a workflow triggers only on main, make it temporarily
    observable rather than trusting analysis.
  2. "Pin by hash" is not "bump to the latest major." The policy asks for
    pinning. Chasing upstream majors on an unverified assumption is what took
    the deploy down.
  3. Pinning a mutable reference means pinning what it resolves to in this
    environment
    .
    npm install -g wrangler on node 20 resolves to 4.86.0, not
    to latest. Pinning "what latest points at" silently changed behaviour.
  4. Green CI was the weakest signal in this entire issue, not the strongest.
    script/cibuild was green for every broken revision, because docker build
    runs RUN under /bin/sh with no Actions runtime at all.

Follow-ups, none blocking

  • #20 — artifact actions held on v3 (add the runs.using: 'node16' durability
    note).
  • #21 — deploy container is node 20, capping wrangler at 4.86.0.
  • New follow-up to file: submodules: recursive is a no-op here (no
    .gitmodules, themes is a plain tree), which also undercuts the stated
    justification for installing git; and apk add tar is likely redundant
    against busybox tar.

Closing.

**Done. Merged and verified on `main`.** PR #22 merged as `9959cb5`. The `main` run — including the job that has never successfully executed with pinned references until now: - `check / check (push)` — success in 4s - `Build and Deploy to Cloudflare Pages / build (push)` — success in 9s - `Build and Deploy to Cloudflare Pages / deploy (push)` — **success in 18s** `https://lora.vegas/` returns HTTP 200, 4226 bytes, with the expected title and the mesh channel and Signal group content intact. **DoD item 8 is now genuinely satisfied** — the deploy path has run end to end with every external reference pinned, which is the thing this issue was actually about and which no previous attempt ever demonstrated. Final state of `deploy.yml`: both container images pinned by digest, all three `uses:` pinned by 40-hex commit SHA, the wrangler install pinned to an exact version that actually runs on the pinned node, `branches: [main]` only, the `if: github.ref_name == 'main'` guard on the deploy job, 4-space YAML. The abandoned `klakegg/hugo:ext-alpine` is gone; the deploy build and the check build now share one pinned base and one dependency list. ## What this cost and what it bought Two merges, one production break, one revert, three implementation rounds and four reviews. The break was caused by a requirement I wrote (the artifact v3 → v4 bump) on an assumption I never tested, and it was confirmed rather than caught by a review that reasoned its way to a false conclusion about this Gitea instance. What actually fixed it was not reviewing harder. It was making the path executable before merging it — a temporary branch trigger plus a guard so branch runs could not reach Cloudflare, then bisecting with probe jobs against real runner output. That found the true cause in one pass and incidentally exposed a second latent outage (`wrangler@4.120.0` requires node >= 22 and exits 1 on the pinned node 20 image; `npm install` only warns about `engines`, so the install step would have gone green and the deploy step would have failed). The generalisable lessons, recorded here because they outlive this issue: 1. **A CI path that cannot be executed pre-merge cannot be validated by review.** Both reviews of PR #17 were reasoning about an environment neither could run. Where a workflow triggers only on `main`, make it temporarily observable rather than trusting analysis. 2. **"Pin by hash" is not "bump to the latest major."** The policy asks for pinning. Chasing upstream majors on an unverified assumption is what took the deploy down. 3. **Pinning a mutable reference means pinning what it resolves to *in this environment*.** `npm install -g wrangler` on node 20 resolves to 4.86.0, not to `latest`. Pinning "what latest points at" silently changed behaviour. 4. **Green CI was the weakest signal in this entire issue, not the strongest.** `script/cibuild` was green for every broken revision, because `docker build` runs `RUN` under `/bin/sh` with no Actions runtime at all. ## Follow-ups, none blocking - #20 — artifact actions held on v3 (add the `runs.using: 'node16'` durability note). - #21 — deploy container is node 20, capping wrangler at 4.86.0. - New follow-up to file: `submodules: recursive` is a no-op here (no `.gitmodules`, `themes` is a plain tree), which also undercuts the stated justification for installing `git`; and `apk add tar` is likely redundant against busybox tar. Closing.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/lora.vegas#7