sneak/dnswatcher
1
0
Fork 0
Code Issues 37 Pull Requests 8 Actions Packages Projects Releases Wiki Activity

DECISION/ACCESS NEEDED: the CI job log Gitea returns for a commit does not correspond to that commit #126

New Issue
Open
opened 2026-08-09 08:42:43 +02:00 by clawbot · 2 comments
clawbot commented 2026-08-09 08:42:43 +02:00
Collaborator
Copy Link

@sneak — this needs your access. The Actions API returns 403 user should be the owner of the repo for the clawbot account, so I cannot investigate it myself.

What was observed

While re-reviewing PR #122, a reviewer tried to corroborate the local Docker results against CI. The PR head ff66ecc shows a green check / check (push) run. The job log Gitea's API returns for that run is:

  • dated 2026-02-21, roughly six months before the commit existed;
  • showing a native go build, not script/cibuild / docker build . as .gitea/workflows/check.yml specifies;
  • containing pre-#93 DNS skips, which no longer exist anywhere in this codebase.

So the log describes a build that predates this branch entirely. The green check mark on that PR is not evidence about that code.

Why this matters beyond one PR

"CI green" has been cited as corroborating evidence on every PR in this milestone. If the association between commit and run/log is unreliable, none of those citations mean anything.

This is now the fifth check in this repo found to look authoritative while not being so, joining script/cibuild (#115, served a cached green in 0.262s having run nothing), script/bootstrap (#117, pinned linter install skipped so the running linter may not be the pinned one), script/lint (#121, shared cache returned another codebase's findings), and script/docker (#124, same cache hole as #115).

The one piece of good news: no merge decision in this milestone rested on CI. Every merge-ready label was granted on locally reproduced evidence — negative controls with planted sentinels, mutation tests, and repeated cache-bypassed -race runs. In hindsight that was the correct instinct rather than luck, and it is the reason this discovery costs nothing retroactively.

What I could not determine

Whether this is:

  1. a Gitea Actions log retention or log-association bug (run metadata correct, log lookup returning the wrong artifact);
  2. a stale or misconfigured runner replaying an old job definition rather than the current .gitea/workflows/check.yml;
  3. an artefact of the API endpoint the reviewer used, with the web UI showing something different; or
  4. genuinely correct, with a runner somewhere still executing a six-month-old workflow definition.

Option 2 and option 4 are materially worse than 1 and 3: they would mean CI has not actually been running script/cibuild on this repo, and the green marks reflect a build nobody has inspected.

What would settle it

  1. Open the run for ff66ecc in the Gitea web UI and compare against what the API returns. If they differ, it is a log-association problem (1 or 3) and the runner is probably fine.
  2. Check the registered runner(s) for this repo — version, labels, and whether any cached workflow definition is being reused.
  3. Trigger a fresh run and confirm the log shows script/cibuild and a docker build, with RUN make check genuinely executing.
  4. If the runner is executing a stale definition, note that until #115's fix (PR #122) merges, even a correct runner could return a cached RUN make check layer — so item 3 should be checked after #122 lands, or with the cache cleared.

Definition of done

Either:

  • the mismatch is explained and, if it is a real defect, fixed — with a fresh run on a current commit whose log demonstrably shows script/cibuild running the full check; or
  • a decision is recorded that Gitea CI is not to be treated as evidence for this repo, in which case I will stop citing it in review verdicts and rely solely on locally reproduced runs.

Either outcome is fine. What is not fine is leaving a green tick that reviewers might reasonably read as meaning something.

Note

I have not changed any process on the strength of this. Reviews here already require locally reproduced evidence and will continue to, regardless of how this resolves.

@sneak — this needs your access. The Actions API returns **403 `user should be the owner of the repo`** for the `clawbot` account, so I cannot investigate it myself. ## What was observed While re-reviewing [PR #122](https://git.eeqj.de/sneak/dnswatcher/pulls/122), a reviewer tried to corroborate the local Docker results against CI. The PR head `ff66ecc` shows a **green** `check / check (push)` run. The job log Gitea's API returns for that run is: - dated **2026-02-21**, roughly six months before the commit existed; - showing a **native `go build`**, not `script/cibuild` / `docker build .` as `.gitea/workflows/check.yml` specifies; - containing **pre-#93 DNS skips**, which no longer exist anywhere in this codebase. So the log describes a build that predates this branch entirely. The green check mark on that PR is not evidence about that code. ## Why this matters beyond one PR "CI green" has been cited as corroborating evidence on **every PR in this milestone**. If the association between commit and run/log is unreliable, none of those citations mean anything. This is now the **fifth** check in this repo found to look authoritative while not being so, joining `script/cibuild` (#115, served a cached green in 0.262s having run nothing), `script/bootstrap` (#117, pinned linter install skipped so the running linter may not be the pinned one), `script/lint` (#121, shared cache returned another codebase's findings), and `script/docker` (#124, same cache hole as #115). **The one piece of good news:** no merge decision in this milestone rested on CI. Every `merge-ready` label was granted on locally reproduced evidence — negative controls with planted sentinels, mutation tests, and repeated cache-bypassed `-race` runs. In hindsight that was the correct instinct rather than luck, and it is the reason this discovery costs nothing retroactively. ## What I could not determine Whether this is: 1. a Gitea Actions **log retention or log-association bug** (run metadata correct, log lookup returning the wrong artifact); 2. a **stale or misconfigured runner** replaying an old job definition rather than the current `.gitea/workflows/check.yml`; 3. an **artefact of the API endpoint** the reviewer used, with the web UI showing something different; or 4. genuinely correct, with a runner somewhere still executing a six-month-old workflow definition. Option 2 and option 4 are materially worse than 1 and 3: they would mean CI has not actually been running `script/cibuild` on this repo, and the green marks reflect a build nobody has inspected. ## What would settle it 1. Open the run for `ff66ecc` in the Gitea web UI and compare against what the API returns. If they differ, it is a log-association problem (1 or 3) and the runner is probably fine. 2. Check the registered runner(s) for this repo — version, labels, and whether any cached workflow definition is being reused. 3. Trigger a fresh run and confirm the log shows `script/cibuild` and a `docker build`, with `RUN make check` genuinely executing. 4. If the runner is executing a stale definition, note that until #115's fix (PR #122) merges, even a correct runner could return a cached `RUN make check` layer — so item 3 should be checked **after** #122 lands, or with the cache cleared. ## Definition of done Either: - the mismatch is explained and, if it is a real defect, fixed — with a fresh run on a current commit whose log demonstrably shows `script/cibuild` running the full check; **or** - a decision is recorded that Gitea CI is not to be treated as evidence for this repo, in which case I will stop citing it in review verdicts and rely solely on locally reproduced runs. Either outcome is fine. What is not fine is leaving a green tick that reviewers might reasonably read as meaning something. ## Note I have not changed any process on the strength of this. Reviews here already require locally reproduced evidence and will continue to, regardless of how this resolves.
clawbot added this to the 1.0 milestone 2026-08-09 08:42:43 +02:00
sneak was assigned by clawbot 2026-08-09 08:42:43 +02:00
clawbot commented 2026-08-09 08:47:33 +02:00
Author
Collaborator
Copy Link

[manager] RESOLVED to the benign explanation. The gate is sound; the defect is in log association only. Downgrading this from "CI may never have run" to "CI logs are unreadable/misassociated".

The experiment

A log read was impossible (Actions API returns 403 for this account), so I ran a natural experiment instead — pushing a commit that cannot pass and seeing whether CI noticed.

Branch ci-probe-126 off main (9347a28), one new file internal/config/zz_ci_probe_test.go containing a single t.Fatal("CI-PROBE-126: ..."). make check cannot pass on that tree.

Result — CI reported failure:

state:       failure
context:     check / check (push)
description: Failing after 23s
sha:         4251e743cd03a55f19bdffc6db976eccf05da853
created_at:  2026-08-09T08:46:41+02:00
target_url:  /sneak/dnswatcher/actions/runs/109/jobs/0

What this establishes

  1. The runner executes the real, current gate. It went red on a defect introduced seconds earlier. A replayed stale job definition cannot do that.
  2. It is content-sensitive, not returning a canned status — the same workflow reports green on healthy commits and red on this one.
  3. The timestamp is contemporaneous (08:46:41, seconds after the push), not a six-month-old artefact.
  4. Duration tracks pipeline structure: 23s for a fail, against the 58s green previously recorded on ff66ecc. A short fail and a longer pass is what a real pipeline produces.

The layer-cache defect (#115) cannot confound this. That bug only serves a cached RUN make check on a byte-identical tree. This commit adds a new file, so COPY . . invalidates and everything below it must rebuild — a cached green was not available. The check step had to run, and it ran, and it failed.

Revised conclusion

Of the four candidate explanations in the issue body, 1 or 3 hold — a log-retention/association bug, or an artefact of the log endpoint — and 2 and 4 are excluded. There is no stale runner. CI has been executing script/cibuild on this repository.

So the green ticks on #97, #112, #113, #118 and #122 do reflect real builds of those commits. That does not change any merge decision, because none of them rested on CI in the first place — every merge-ready label in this milestone was granted on locally reproduced evidence — but it is a relief rather than a new problem.

What remains, and what it is worth

The logs are still wrong: the API returned a 2026-02-21 log, showing a native go build with pre-#93 DNS skips, for a run on a commit that did not exist then. Combined with the 403 on the Actions API for this account, that means CI failures on this repo are currently undiagnosable by an agent — we can see that something failed, never why. Every failure has to be reproduced locally to be understood.

That is an annoyance rather than a correctness problem, and it is squarely in your court since it needs owner rights. Worth checking whether the web UI shows the correct log for run 109 (fail) versus what the API returns — if they differ, it is purely an API/retention artefact and can probably be ignored.

Housekeeping

PR #127 (the probe) is closed and branch ci-probe-126 is deleted from origin. The failing test existed only on that branch, never touched main, and is gone. Nothing to clean up.

Credit for the approach goes to a manager on a sibling repo, who pointed out that a deliberately-broken push proves more than a log read can — and whose own repo produced the same shape of result independently.

**[manager] RESOLVED to the benign explanation. The gate is sound; the defect is in log association only.** Downgrading this from "CI may never have run" to "CI logs are unreadable/misassociated". ## The experiment A log read was impossible (Actions API returns 403 for this account), so I ran a natural experiment instead — pushing a commit that **cannot** pass and seeing whether CI noticed. Branch `ci-probe-126` off `main` (`9347a28`), one new file `internal/config/zz_ci_probe_test.go` containing a single `t.Fatal("CI-PROBE-126: ...")`. `make check` cannot pass on that tree. **Result — CI reported `failure`:** ``` state: failure context: check / check (push) description: Failing after 23s sha: 4251e743cd03a55f19bdffc6db976eccf05da853 created_at: 2026-08-09T08:46:41+02:00 target_url: /sneak/dnswatcher/actions/runs/109/jobs/0 ``` ## What this establishes 1. **The runner executes the real, current gate.** It went red on a defect introduced seconds earlier. A replayed stale job definition cannot do that. 2. **It is content-sensitive**, not returning a canned status — the same workflow reports green on healthy commits and red on this one. 3. **The timestamp is contemporaneous** (08:46:41, seconds after the push), not a six-month-old artefact. 4. **Duration tracks pipeline structure**: 23s for a fail, against the 58s green previously recorded on `ff66ecc`. A short fail and a longer pass is what a real pipeline produces. **The layer-cache defect (#115) cannot confound this.** That bug only serves a cached `RUN make check` on a *byte-identical* tree. This commit adds a new file, so `COPY . .` invalidates and everything below it must rebuild — a cached green was not available. The check step had to run, and it ran, and it failed. ## Revised conclusion Of the four candidate explanations in the issue body, **1 or 3 hold** — a log-retention/association bug, or an artefact of the log endpoint — and **2 and 4 are excluded**. There is no stale runner. CI has been executing `script/cibuild` on this repository. So the green ticks on #97, #112, #113, #118 and #122 do reflect real builds of those commits. That does not change any merge decision, because none of them rested on CI in the first place — every `merge-ready` label in this milestone was granted on locally reproduced evidence — but it is a relief rather than a new problem. ## What remains, and what it is worth The **logs** are still wrong: the API returned a 2026-02-21 log, showing a native `go build` with pre-#93 DNS skips, for a run on a commit that did not exist then. Combined with the 403 on the Actions API for this account, that means **CI failures on this repo are currently undiagnosable by an agent** — we can see *that* something failed, never *why*. Every failure has to be reproduced locally to be understood. That is an annoyance rather than a correctness problem, and it is squarely in your court since it needs owner rights. Worth checking whether the web UI shows the correct log for run 109 (fail) versus what the API returns — if they differ, it is purely an API/retention artefact and can probably be ignored. ## Housekeeping PR #127 (the probe) is closed and branch `ci-probe-126` is deleted from `origin`. The failing test existed only on that branch, never touched `main`, and is gone. Nothing to clean up. Credit for the approach goes to a manager on a sibling repo, who pointed out that a deliberately-broken push proves more than a log read can — and whose own repo produced the same shape of result independently.
clawbot commented 2026-08-09 17:05:06 +02:00
Author
Collaborator
Copy Link

[manager] Second, different CI symptom — recording here since this issue already tracks CI reliability.

While reviewing PR #131, CI never ran on head db933f3: run 111 sat at "Waiting to run" for roughly 20 minutes. PR #128 had completed in 54s shortly before, so the runner was working moments earlier.

Queued, not failed — so this is not a needs-checks situation and it did not block the review, which substituted a real docker build --no-cache-filter=builder instead.

Distinct from the original symptom here (a log returned for a run that predates the commit), so it may or may not share a cause. Both point at the same place: the runner and its queue are worth a look when you triage this.

Practical effect, unchanged: CI on this repo is currently neither diagnosable (Actions API is 403 for this account, logs misassociated) nor reliably timely. Reviews here already record verdicts from locally reproduced evidence rather than the green tick, so nothing is blocked — but a missing tick should be read as "CI did not report", not as a problem with the branch.

**[manager] Second, different CI symptom — recording here since this issue already tracks CI reliability.** While reviewing [PR #131](https://git.eeqj.de/sneak/dnswatcher/pulls/131), CI **never ran** on head `db933f3`: run 111 sat at "Waiting to run" for roughly 20 minutes. [PR #128](https://git.eeqj.de/sneak/dnswatcher/pulls/128) had completed in 54s shortly before, so the runner was working moments earlier. Queued, not failed — so this is not a `needs-checks` situation and it did not block the review, which substituted a real `docker build --no-cache-filter=builder` instead. Distinct from the original symptom here (a log returned for a run that predates the commit), so it may or may not share a cause. Both point at the same place: the runner and its queue are worth a look when you triage this. **Practical effect, unchanged:** CI on this repo is currently neither diagnosable (Actions API is 403 for this account, logs misassociated) nor reliably timely. Reviews here already record verdicts from locally reproduced evidence rather than the green tick, so nothing is blocked — but a missing tick should be read as "CI did not report", not as a problem with the branch.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
merge-ready
All checks pass, reviewed, rebased, ready for merge
 needs-checks
make check does not pass cleanly
 needs-rebase
PR has merge conflicts or is behind main
 needs-review
Code review not yet done
 needs-rework
Code review found issues to fix
No Label
Milestone
No items
No Milestone 1.0
Projects
Clear projects
No project
Assignees
Clear assignees
clawbot sneak
No Assignees sneak
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/dnswatcher#126
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?