Flaky test: t.TempDir cleanup race in internal/service/webhook TestExtractBranch #198

Closed
opened 2026-09-22 11:01:25 +02:00 by clawbot · 1 comment
Collaborator

Observed during the Docker gate on #194 (the finding is unrelated to that PR, which touches no webhook code).

TestExtractBranch/extracts_develop_branch in internal/service/webhook flaked once with a t.TempDir cleanup failure: RemoveAll: directory not empty. Cause is an async deployment goroutine still writing into the subtest's temp directory after the subtest has returned, so Go's automatic t.TempDir cleanup races the late write. A re-run passed (host re-runs were 0/8), so it is intermittent.

Why it matters: make check runs go test -race, and a flaky cleanup failure can turn the gate red on an otherwise-good commit. Because the CI gate is what protects next and main, an intermittently-red gate is a real problem even though the production code is fine.

Definition of done

  • Identify the goroutine (deployment/async path) that writes under the test's t.TempDir after the relevant subtest returns.
  • Make the test deterministically wait for that async work to finish before it returns (e.g. synchronize on completion, inject a controllable clock/hook, or scope the temp dir so the late writer cannot touch a dir being cleaned up) — do not merely retry or sleep.
  • Demonstrate stability: the affected test passes repeatedly under -race (e.g. -count=20) with no RemoveAll: directory not empty.
  • make check green via the Docker gate. Lands via PR from a feature branch off next.

Model: opus-4-8

Observed during the Docker gate on https://git.eeqj.de/sneak/upaas/pulls/194 (the finding is unrelated to that PR, which touches no webhook code). `TestExtractBranch/extracts_develop_branch` in `internal/service/webhook` flaked once with a `t.TempDir` cleanup failure: `RemoveAll: directory not empty`. Cause is an async deployment goroutine still writing into the subtest`'`s temp directory after the subtest has returned, so Go`'`s automatic `t.TempDir` cleanup races the late write. A re-run passed (host re-runs were 0/8), so it is intermittent. Why it matters: `make check` runs `go test -race`, and a flaky cleanup failure can turn the gate red on an otherwise-good commit. Because the CI gate is what protects `next` and `main`, an intermittently-red gate is a real problem even though the production code is fine. ## Definition of done - Identify the goroutine (deployment/async path) that writes under the test`'`s `t.TempDir` after the relevant subtest returns. - Make the test deterministically wait for that async work to finish before it returns (e.g. synchronize on completion, inject a controllable clock/hook, or scope the temp dir so the late writer cannot touch a dir being cleaned up) — do not merely retry or sleep. - Demonstrate stability: the affected test passes repeatedly under `-race` (e.g. `-count=20`) with no `RemoveAll: directory not empty`. - `make check` green via the Docker gate. Lands via PR from a feature branch off `next`. Model: opus-4-8
Author
Collaborator

Fixed in #201.

The webhook Service now tracks its deployment goroutines in a sync.WaitGroup and exposes WaitForDeployments; the webhook tests wait on it instead of sleeping 100ms, so the async writer can no longer race t.TempDir cleanup. go test -race -run TestExtractBranch -count=20 ./internal/service/webhook is green with no RemoveAll errors, and make check is green via the Docker gate.

Model: opus-4-8

Fixed in https://git.eeqj.de/sneak/upaas/pulls/201. The webhook `Service` now tracks its deployment goroutines in a `sync.WaitGroup` and exposes `WaitForDeployments`; the webhook tests wait on it instead of sleeping 100ms, so the async writer can no longer race `t.TempDir` cleanup. `go test -race -run TestExtractBranch -count=20 ./internal/service/webhook` is green with no `RemoveAll` errors, and `make check` is green via the Docker gate. Model: opus-4-8
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/upaas#198