blocked: no Actions runner is picking up jobs — every CI run sits queued forever #220

New Issue

clawbot · 2026-08-11T14:45:42+02:00

clawbot commented

2026-08-11 14:45:42 +02:00

Needs you: this is infrastructure, and clawbot gets 403 user should be the owner of the repo on the Actions API, so I cannot diagnose it further.

Every push creates a check / check (push) run that is never executed. Run IDs keep incrementing — 452, 460, 472, 486 — and all sit at pending / "Waiting to run" indefinitely. Current next head cf5f582 has been queued since 14:41 with created_at == updated_at, i.e. never touched. Six independent reviewers hit this on six different heads today.

Symptom is consistent with no runner registered or the runner being offline: jobs are being created correctly, nothing is consuming them.

Why it matters before 1.0.0

Nothing is blocked right now — every unit is gated on make check, and reviewers additionally run the containerized script/cibuild path with the make check layer confirmed executing uncached. So the verification is real. But it is all being done by agents on this machine, and the repo's own CI gate is not actually enforcing anything. A branch that went red would be caught by a worker or a reviewer, not by CI.

That also means "CI green on head" cannot currently be part of the merge gate, which is where I would normally want it — particularly for the next -> main milestone PR.

What I need from you

Either bring a runner back for this repo, or tell me CI is intentionally off for now and I will stop treating a queued run as a defect and stop asking reviewers to confirm it.

No other work is waiting on this.

Needs you: this is infrastructure, and `clawbot` gets `403 user should be the owner of the repo` on the Actions API, so I cannot diagnose it further. Every push creates a `check / check (push)` run that is never executed. Run IDs keep incrementing — 452, 460, 472, 486 — and all sit at `pending` / "Waiting to run" indefinitely. Current `next` head `cf5f582` has been queued since 14:41 with `created_at` == `updated_at`, i.e. never touched. Six independent reviewers hit this on six different heads today. Symptom is consistent with no runner registered or the runner being offline: jobs are being created correctly, nothing is consuming them. ## Why it matters before 1.0.0 Nothing is blocked right now — every unit is gated on `make check`, and reviewers additionally run the containerized `script/cibuild` path with the `make check` layer confirmed executing uncached. So the verification is real. But it is all being done by agents on this machine, and **the repo's own CI gate is not actually enforcing anything**. A branch that went red would be caught by a worker or a reviewer, not by CI. That also means "CI green on head" cannot currently be part of the merge gate, which is where I would normally want it — particularly for the `next` -> `main` milestone PR. ## What I need from you Either bring a runner back for this repo, or tell me CI is intentionally off for now and I will stop treating a queued run as a defect and stop asking reviewers to confirm it. No other work is waiting on this.

sneak was assigned by clawbot

2026-08-11 14:45:42 +02:00

clawbot commented

2026-08-11 14:46:46 +02:00

Correction to the diagnosis above, which changes what to look at: the runs are not all merely queued. Two commits already on next report failure with the description "Has been cancelled" — b9bc226 and 19cb1ca — while other heads sit pending / "Waiting to run".

So the pattern is a mix of cancelled and never-started, not a uniformly empty queue. That points at a runner that registers, takes work and dies or is killed, rather than simply being absent — worth checking the runner process and its logs rather than only whether one is registered.

It also has a start time. The runner was healthy earlier today: the last run that actually executed was on d93eda3 at 14:15, reported "Successful in 22s". Jobs stopped being picked up at roughly 14:31.

Update — it is intermittent, not dead. Run 477 on 2589473 subsequently executed and reported success in 30s. But the very next push, run 493 on next head 9b957ff, is back to pending / "Waiting to run". So the runner comes back, takes a job, then stops again.

That changes the likely cause: not a deregistered runner but one that is crashing, being OOM-killed, or losing its connection and reconnecting. Whatever supervises it is restarting it often enough to occasionally drain one job. Worth looking at the runner's resource limits and restart history rather than its registration.

Same ask, unchanged, and still nothing blocked on it.

Correction to the diagnosis above, which changes what to look at: the runs are not all merely queued. Two commits already on `next` report `failure` with the description **"Has been cancelled"** — `b9bc226` and `19cb1ca` — while other heads sit `pending` / "Waiting to run". So the pattern is a mix of cancelled and never-started, not a uniformly empty queue. That points at a runner that registers, takes work and dies or is killed, rather than simply being absent — worth checking the runner process and its logs rather than only whether one is registered. **It also has a start time.** The runner was healthy earlier today: the last run that actually executed was on `d93eda3` at 14:15, reported "Successful in 22s". Jobs stopped being picked up at roughly **14:31**. **Update — it is intermittent, not dead.** Run 477 on `2589473` subsequently executed and reported `success` in 30s. But the very next push, run 493 on `next` head `9b957ff`, is back to `pending` / "Waiting to run". So the runner comes back, takes a job, then stops again. That changes the likely cause: not a deregistered runner but one that is crashing, being OOM-killed, or losing its connection and reconnecting. Whatever supervises it is restarting it often enough to occasionally drain one job. Worth looking at the runner's resource limits and restart history rather than its registration. Same ask, unchanged, and still nothing blocked on it.

clawbot commented

2026-08-11 14:56:39 +02:00

This is worse than a stalled runner, and it changes the severity. The runner is attaching results from unrelated jobs to the wrong commits.

check / check (push) reported success on b9ac3cf (head of #196), "Successful in 25s", run 478. A reviewer pulled the actual job behind that status. Job 502:

fetched and checked out refs/remotes/origin/fix/approval-address-title at e7a960c6 — a different branch entirely
reported Test Suites: 2 passed, 2 total / Tests: 14 passed, 14 total
carries timestamps reading 2026-02-27
its html_url points at run 204 while its run_id is 478

The commit it claimed to verify has 8 suites and 165 tests. So the green tick was for someone else's code, from months ago.

A stuck queue is an inconvenience. A green check that is not a check of the code under it is actively dangerous — it is the one failure mode that can let something red reach main while every signal says otherwise. If a run identifier can be reused or a status can be bound to the wrong ref, no CI verdict in this repo is currently trustworthy, including any that look green on the milestone PR.

Practically, for now: I am treating tracker CI status as carrying no information, positive or negative, and gating merges purely on make check plus the containerized script/cibuild path run by a reviewer who is independent of the author. That is real verification and it is what has caught every defect today. But it means the repo has no working CI gate at all, rather than a slow one, and I would not want to tag 1.0.0 on that basis without you having looked at this.

Still nothing blocked. Same ask, higher stakes than when I filed it.

This is worse than a stalled runner, and it changes the severity. **The runner is attaching results from unrelated jobs to the wrong commits.** `check / check (push)` reported `success` on `b9ac3cf` (head of https://git.eeqj.de/sneak/AutistMask/pulls/196), "Successful in 25s", run 478. A reviewer pulled the actual job behind that status. Job 502: - fetched and checked out `refs/remotes/origin/fix/approval-address-title` at `e7a960c6` — a different branch entirely - reported `Test Suites: 2 passed, 2 total` / `Tests: 14 passed, 14 total` - carries timestamps reading **2026-02-27** - its `html_url` points at run 204 while its `run_id` is 478 The commit it claimed to verify has 8 suites and 165 tests. So the green tick was for someone else's code, from months ago. A stuck queue is an inconvenience. A green check that is not a check of the code under it is actively dangerous — it is the one failure mode that can let something red reach `main` while every signal says otherwise. If a run identifier can be reused or a status can be bound to the wrong ref, no CI verdict in this repo is currently trustworthy, including any that look green on the milestone PR. Practically, for now: I am treating tracker CI status as carrying **no information**, positive or negative, and gating merges purely on `make check` plus the containerized `script/cibuild` path run by a reviewer who is independent of the author. That is real verification and it is what has caught every defect today. But it means the repo has no working CI gate at all, rather than a slow one, and I would not want to tag 1.0.0 on that basis without you having looked at this. Still nothing blocked. Same ask, higher stakes than when I filed it.