infra: the mac-studio CI runner has run out of docker subnets, so every e2e job fails before checkout #396

Open
opened 2026-09-21 09:56:29 +02:00 by clawbot · 1 comment
Collaborator

Since about 09:32 today every e2e workflow job on the mac-studio runner fails before checkout, on every branch (next at 99292b9, and every issue branch pushed since, including a docs-only one). The job log ends with:

Cleaning up network for job e2e-chrome, and network name is: GITEA-ACTIONS-TASK-6285_WORKFLOW-e2e_JOB-e2e-chrome-e2e-chrome-network
Error response from daemon: all predefined address pools have been fully subnetted

That is the runner host's docker daemon out of subnet space for per-job networks, which is what happens when job networks are created and never removed. Nothing in the repo changed to cause it and nothing in the repo can fix it. The check workflow still runs. Both e2e jobs were green on next at 1197d21 this morning.

What is needed, on the runner host: docker network ls to confirm the leaked GITEA-ACTIONS-* networks, docker network prune (or removing those networks by name) while no job is running, and, if it recurs, a larger default-address-pools in the daemon config or a runner upgrade if this is a known cleanup defect in that act_runner version (v0.3.0).

Until this is cleared, next shows red in the e2e jobs on every push. The units landing meanwhile are each gated by an independent reviewer's local make check on the rebased tree, and #290 already documents that the e2e jobs report rather than gate; that does not make a red next acceptable, so this is flagged rather than worked around.

Definition of done

  • The e2e jobs on next start and run to a result again.
  • If the cause was leaked job networks, the fix that stops the leak (config, runner version) is recorded here.

model: claude-fable-5

Since about 09:32 today every `e2e` workflow job on the `mac-studio` runner fails before checkout, on every branch (`next` at `99292b9`, and every issue branch pushed since, including a docs-only one). The job log ends with: ``` Cleaning up network for job e2e-chrome, and network name is: GITEA-ACTIONS-TASK-6285_WORKFLOW-e2e_JOB-e2e-chrome-e2e-chrome-network Error response from daemon: all predefined address pools have been fully subnetted ``` That is the runner host's docker daemon out of subnet space for per-job networks, which is what happens when job networks are created and never removed. Nothing in the repo changed to cause it and nothing in the repo can fix it. The `check` workflow still runs. Both e2e jobs were green on `next` at `1197d21` this morning. What is needed, on the runner host: `docker network ls` to confirm the leaked `GITEA-ACTIONS-*` networks, `docker network prune` (or removing those networks by name) while no job is running, and, if it recurs, a larger `default-address-pools` in the daemon config or a runner upgrade if this is a known cleanup defect in that act_runner version (`v0.3.0`). Until this is cleared, `next` shows red in the e2e jobs on every push. The units landing meanwhile are each gated by an independent reviewer's local `make check` on the rebased tree, and https://git.eeqj.de/sneak/AutistMask/issues/290 already documents that the e2e jobs report rather than gate; that does not make a red `next` acceptable, so this is flagged rather than worked around. ## Definition of done - [ ] The e2e jobs on `next` start and run to a result again. - [ ] If the cause was leaked job networks, the fix that stops the leak (config, runner version) is recorded here. model: claude-fable-5
sneak was assigned by clawbot 2026-09-21 09:56:29 +02:00
Author
Collaborator

Correction to the report above: the check workflow no longer runs either. Since about 09:56 every job of every workflow on this runner, check included, fails within one second of starting, on next (now at 9ac7df0) and on every issue branch. So next shows red in all three jobs, and no push to this repo can get a CI result until the runner host is cleared as described above.

Units keep landing on next gated by an independent reviewer's local make check on the rebased tree; the tree now on next is byte-identical to the one the reviewer of #397 ran it on.

Model: fable-5-1

Correction to the report above: the `check` workflow no longer runs either. Since about 09:56 every job of every workflow on this runner, `check` included, fails within one second of starting, on `next` (now at `9ac7df0`) and on every issue branch. So `next` shows red in all three jobs, and no push to this repo can get a CI result until the runner host is cleared as described above. Units keep landing on `next` gated by an independent reviewer's local `make check` on the rebased tree; the tree now on `next` is byte-identical to the one the reviewer of https://git.eeqj.de/sneak/AutistMask/pulls/397 ran it on. Model: fable-5-1
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/AutistMask#396