Root background loops at context.Background() (closes #97) #100
Reference in New Issue
Block a user
Delete Branch "issue-97-lifecycle-context"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Fixes the two pre-existing sites on
mainwhere a long-lived goroutine derived its lifetime from an fxOnStarthook context. fx builds that context withcontext.WithTimeout(ctx, StartTimeout)— 15 seconds by default — and cancels it when the start phase ends, so both loops died shortly after boot.The two fixes
internal/delivery/engine.go—Engine.startrooted the whole worker pool,recoverPending, andretrySweepin the hook context. Every worker returned on<-ctx.Done()about fifteen seconds into the process, after which the application kept receiving and persisting inbound events while nothing forwarded them:deliveryChfilled and began logging "delivery channel full" with no consumer left.internal/database/retention.go—RetentionReaper.starthad the same defect. Under the default one-hourRETENTION_SWEEP_INTERVALthe loop was cancelled forty-five minutes before its first tick, so the reaper never ran a single sweep and per-webhook event databases grew without bound.Both now use
context.WithCancel(context.Background()). Their lifetime is bounded byOnStop, which already cancels and waits on theWaitGroup, so shutdown behaviour is unchanged. In each component thelc.Append(fx.Hook{...})call moves into aregisterHooksmethod, theOnStartparameter is named_so the trap cannot be reintroduced by someone silencing an unused-parameter warning, and a doc comment at eachstartexplains why the hook context must not be used.StartTimeoutis deliberately not lengthened — that would treat the symptom.Tests
Four new tests, two per component, in
internal/delivery/engine_lifecycle_test.goandinternal/database/retention_lifecycle_test.go.Each drives the genuine registered hook: the test builds a recording
fx.Lifecycle, calls the component's realregisterHooksthrough anExport...shim, and invokes the recordedOnStart/OnStop— the exact functions the application runs.OnStartis handed an already-cancelled context, which is fx's start-phase cancellation taken to its limit. Passing a plaincontext.Background()would prove nothing, since that is precisely the bug.TestEngine_WorkersOutliveStartHookContext— after the pool has settled, seeds a log-target delivery and asserts it reachesdelivered.TestEngine_StopHookStopsWorkers— proves the pool is live, then assertsOnStopreturns within a bounded timeout (it blocks onwg.Wait(), so returning at all proves every goroutine observed the cancellation), and that a task notified afterwards stayspending.TestRetentionReaper_LoopOutlivesStartHookContext— runs at a 10ms interval and asserts a long-expired event is reaped.TestRetentionReaper_StopHookStopsLoop— same bounded-timeout shutdown assertion, then asserts a newly seeded expired chain survives.One correctness note on the engine test as first written. Driving
Notifyimmediately afterOnStartpassed against the unfixed code: a worker'sselecthad both a readyctx.Done()and a readydeliveryCh, Go chooses between ready cases at random, and with ten workers a doomed pool still delivered the task. The helper now waits a settle window afterOnStartbefore any work is enqueued, and the test seeds its delivery only afterwards so restart recovery cannot enqueue during startup. With an empty queue and a done context a broken pool has nothing butctx.Done()ready, so it is deterministically gone by the time the task arrives.Mutation evidence
Each fix was reverted in turn (hook context passed back into
start,starttaking acontext.Contextagain) and the suite re-run throughmake test; then restored.Engine.startreverted--- FAIL: TestEngine_WorkersOutliveStartHookContext (5.87s),--- FAIL: TestEngine_StopHookStopsWorkers (5.85s)RetentionReaper.startreverted--- FAIL: TestRetentionReaper_LoopOutlivesStartHookContext (5.38s),--- FAIL: TestRetentionReaper_StopHookStopsLoop (5.39s)The first engine mutation run is also what exposed the
select-race weakness described above:TestEngine_WorkersOutliveStartHookContextpassed against the bug before the helper was hardened, and fails against it after.Tree-wide sweep
Every
fx.Hookregistration in the tree was checked. All sevenOnStarthooks now take_ context.Context, so no goroutine anywhere can inherit a start-phase context:internal/delivery/engine.go(fixed here)internal/database/retention.go(fixed here)internal/database/database.gointernal/handlers/handlers.gointernal/healthcheck/healthcheck.gointernal/server/server.gointernal/session/session.goThe five untouched hooks already used
_and start no long-lived goroutine from a hook context. This confirms the issue's observation at4f5ecb1still holds at implementation time.Lint findings
funcorder,internal/delivery/engine.go— introduced here: extractingregisterHooksplaced an unexported method ahead of the exportedScheduleRetry. Fixed by movingregisterHooksbelowScheduleRetry.unparam,internal/delivery/engine_integration_test.go— introduced here.iWaitForStatus'sexpectedparameter only ever receiveddatabase.DeliveryStatusDelivered; onmainit had two call sites, below unparam's reporting threshold, and the two added by this change pushed it over. Confirmed by runningmake linton a clean4f5ecb1worktree, where the finding does not appear. Fixed at the root rather than suppressed: the helper is nowiWaitForDelivered(t, db, deliveryID).gosecG704,internal/delivery/client_ssrf_test.go:78— pre-existing and not from this change, which does not touch that file. It reproduces on a clean4f5ecb1worktree and is an artifact of the host linter (v2.10.1) being older than the CI pin. Deliberately not fixed here. It does not appear underscript/cibuild, which lints inside the pinnedgolangci/golangci-lint:v2.12.2image; that build is green.Verification
make fmt— clean, includingTODO.md.make check— tests andfmt-checkgreen; the only remaining output is the pre-existing host-onlygosecG704 above.script/cibuild— exit 0. This is the authoritative run: it executesmake fmt-check,make lint, andmake testinside the pinned v2.12.2 image, with nogosec/G704output anywhere in the log and an uncached test run..golangci.ymlis untouched and the v2.12.2 Dockerfile pin is unchanged.Relationship to PR #95
PR #95 introduces a third instance of this defect in its own new
internal/delivery/archive_sweeper.goand fixes it there, so the two changes do not collide. That file does not exist onmainand is not touched or included here. The fix in this PR deliberately mirrors #95's shape — theregisterHooksextraction, the_hook parameter, the//nolint:contextcheckon the hook, and the explanatory comment onstart— so all three sites read identically once both land.Summary of what this builds and how it was verified.
Built. Two one-line lifetime fixes plus the scaffolding that makes them permanent.
Engine.startandRetentionReaper.startno longer take acontext.Contextat all; each derives its loop context fromcontext.WithCancel(context.Background()). Each component'slc.Append(fx.Hook{...})moved into aregisterHooksmethod whoseOnStarttakes_ context.Context, carrying a//nolint:contextcheckand a doc comment onstartexplaining why the hook context is poison for a long-lived goroutine.OnStopis unchanged and still cancels then waits on theWaitGroup.Verified.
script/cibuild— exit 0, the authoritative run.make fmt-check,make lint, andmake testall execute inside the pinnedgolangci/golangci-lint:v2.12.2image, tests uncached. Grepping the full build log forgosecandG704returns zero hits.make checklocally — all packages pass; the only output is the pre-existinggosecG704 atinternal/delivery/client_ssrf_test.go:78, which reproduces identically on a clean4f5ecb1worktree and is a host-linter-version artifact in a file this change does not touch. Left alone deliberately.make fmt— clean,TODO.mdincluded.Engine.startto take the hook context failsTestEngine_WorkersOutliveStartHookContextandTestEngine_StopHookStopsWorkers; revertingRetentionReaper.startfailsTestRetentionReaper_LoopOutlivesStartHookContextandTestRetentionReaper_StopHookStopsLoop. Each mutation was restored and the suite re-run green.Worth a reviewer's attention. The first mutation run caught a real hole in the engine regression test. As originally written it enqueued work immediately after
OnStart, and it passed against the unfixed code — a worker'sselectsaw both a readyctx.Done()and a readydeliveryCh, Go picks among ready cases at random, and one of ten workers won often enough to deliver the task. A test that passes against the bug is worth nothing, so the helper now settles the pool afterOnStartbefore any work exists, and the test seeds its delivery only afterwards so restart recovery cannot enqueue during startup. Against the bug, the pool is deterministically gone before the task arrives.Sweep. All seven
fx.HookOnStartregistrations in the tree now take_ context.Context; the five this change does not touch already did and start no long-lived goroutine. No further instances of the pattern exist onmain.Not included.
internal/delivery/archive_sweeper.go— the third instance of this bug lives in PR #95's new file and is fixed there. It does not exist onmainand is untouched here. This PR intentionally mirrors #95's shape so all three sites read the same way once both land.Review of PR #100 —
20a050bagainstmain4f5ecb1Verdict: PASS.
Reviewed adversarially against issue #97's Definition of done. Every load-bearing claim in the PR description was re-verified independently rather than taken on trust; where I could execute a check instead of reading one, I did.
Definition of done — all five items met
startmethods root atcontext.Background()internal/delivery/engine.go:235,internal/database/retention.go:95wg.Wait()returns-racemake checkgreen via repo entrypoints1. Mutation verification (executed)
Performed in a throwaway worktree, never in the PR checkout. Each mutation restored
startto accepting the hook context and passing it tocontext.WithCancel, restoring the hook tofunc(ctx context.Context);ExportStartwas pointed atcontext.Background()so the mutation was isolated to the hook path exactly. Each mutation was run 5 times throughmake test, because the caller flagged the original defect as a random-select flake.Engine.startrevertedTestEngine_WorkersOutliveStartHookContextFAIL 5/5;TestEngine_StopHookStopsWorkersFAIL 5/5RetentionReaper.startrevertedTestRetentionReaper_LoopOutlivesStartHookContextFAIL 5/5;TestRetentionReaper_StopHookStopsLoopFAIL 5/5Both mutations were reverted afterwards and the tree confirmed clean via
git status.On the hardening of
TestEngine_WorkersOutliveStartHookContext. The concern that the fix is merely "less flaky" does not hold up, and the reason is structural rather than statistical. The original hole existed becauseNotifyraced the pool's firstselect, where bothctx.Done()anddeliveryChwere ready and Go picks among ready cases at random.startEngineViaHooknow returns only afterhookSettleDelay, and the test seeds its delivery after that — so at the moment the pool makes its firstselect, the queue is provably empty andctx.Done()is the only ready case. The random choice is eliminated, not merely biased: there is no second ready case to choose. The helper's doc comment also correctly forbids seeding pending or retrying deliveries before the call, which is the one thing that could reintroduce work during startup viarecoverPending.The residual timing assumption is that 250ms suffices for a pre-cancelled context to be observed. That held 5/5 under concurrent load on my host. Critically, its failure mode is asymmetric: an insufficient settle window can only reduce mutation sensitivity, never produce a false failure against correct code. See the non-blocking note below.
2. Shutdown is not traded away (executed)
Both shutdown tests are meaningful, not vacuous: each first proves the loop is live (a delivered task / a reaped event) so a fast
OnStopcannot pass by stopping something already dead, then assertsOnStopreturns within a bounded timeout, then asserts the component is genuinely inert afterwards. Sincestop()blocks onwg.Wait(), returning at all proves every goroutine observed cancellation.Eight full
-racesuite runs: no hangs, no race reports, no package-timeout pressure. Package durationsinternal/delivery3.686s to 4.223s andinternal/database1.489s to 2.069s against the 30s per-package timeout — ample headroom, andmake teststays well inside the 20s policy budget.3. The
unparamfix did not weaken any assertion (verified by reading)git grep iWaitForStatus 4f5ecb1shows exactly two call sites onmain, both passingdatabase.DeliveryStatusDelivered. No test anywhere waited onfailed,retrying, orpendingthrough this helper — those statuses are asserted directly elsewhere and are untouched. Collapsing toiWaitForDeliveredis a strict no-op for coverage. Fixing at the root rather than suppressing was the right call.4. Tree-wide sweep (verified independently, not taken on trust)
Grepped every
OnStartin the tree myself. Exactly seven registrations, all nowfunc(_ context.Context) error:internal/delivery/engine.go,internal/database/retention.go,internal/database/database.go,internal/handlers/handlers.go,internal/healthcheck/healthcheck.go,internal/server/server.go,internal/session/session.go. The PR's list is accurate and complete. The only othercontext.WithCancel(context.Background())for a long-lived goroutine isinternal/server/server.go:130, which was already correct.5.
internal/delivery/archive_sweeper.go(verified)Absent from the tree and absent from the diff. The eight changed files are confined to the two components, their
export_test.goshims, the two new test files, one integration-test helper rename, andTODO.md. No scope creep.6. Nothing legitimately provided by the hook context is lost (verified by reading fx source)
This is the direction most likely to hide a regression, so I checked it against the pinned dependency rather than reasoning from memory. In fx v1.20.1,
App.Startwraps the lifecycle inwithRollback, which on any start failure callsapp.lifecycle.Stop(ctx)— andLifecycle.StoprunsOnStoponly for hooks whoseOnStartalready completed. So if startup fails after these hooks run, both components'OnStopstill executes, cancelling the loop and joining theWaitGroup. The goroutines cannot outlive a failedapp.Start.Nothing else was lost: the repo carries no tracing or OpenTelemetry, and the only
ctx.Valueread in the tree is an unrelated request ID ininternal/middleware/middleware.go. No hook context ever carried values here.7. Suppressions (executed)
Exactly two additions matching
nolintin the entire diff, both//nolint:contextcheck, both on the hook registration. I tested necessity by deleting both and re-runningmake lint: this produces exactly two new findings,Function start should pass the context parameter (contextcheck)atinternal/database/retention.goandinternal/delivery/engine.go. The suppressions are necessary, minimal, correctly scoped, and the rationale comments are accurate.8. Repo policy (verified)
.golangci.ymlsha256021cc83f4e6fc7c31b95b34b846723dfcf20b66b7baeea1dc40406e643346bcb— unmodified. Not in the diff. The v2.12.2 pin is intact.Root background loops at context.Background() (closes #97)ends with the required trailer.TODO.mdupdated in the same commit.make fmtproduces no diff, includingTODO.md.make checkmodifies no files; tree clean after every run.9. CI and build verification (executed)
20a050b: success,check / check (push), 3m3s.git merge-treeagainst currentorigin/main4f5ecb1merges clean. No rebase needed.script/cibuild: exit 0 — but I must record a caveat the PR description does not. On my run every Docker layer wasCACHED, includingmake testandmake lint, so mycibuildinvocation executed no tests and is not by itself independent evidence. Because Docker layers are content-addressed over the copied source, a cachedmake test/make lintlayer does still attest that those commands succeeded on this exact tree, and the fresh Gitea CI run on20a050bis genuinely uncached. Independent evidence is supplied by my own host runs:make checkwith all four new tests passing, plus the 8 clean and 10 mutation suite runs above.make checkon20a050bexits 2 solely oninternal/delivery/client_ssrf_test.go:78:28: G704 (gosec). I confirmed this against a clean4f5ecb1worktree: byte-identical single finding,1 issues: gosec: 1. Pre-existing, host-linter-version artifact, in a file this PR does not touch, absent under the pinned CI image. The PR's characterisation is correct. This PR introduces zero new lint findings.Non-blocking observations
None of these gate the merge; the first is worth a follow-up issue, the rest are noted for the record.
hookSettleDelayis a wall-clock assumption, not a happens-before edge (internal/delivery/engine_lifecycle_test.go:29). The reasoning behind it is correct and the comment is unusually good, but the guarantee rests on 250ms being enough rather than on a synchronisation event. A fully deterministic form would have the buggy pool signal its own exit — for example, exporting the workerWaitGroupand joining it, or a counter of live workers polled withrequire.Eventually. Worth doing if this test is ever seen to weaken. Not a defect today: the assumption held 5/5 under load, and it cannot cause a false CI failure against correct code.OnStopperforms an unboundedwg.Wait()and ignores its context in both components. If a delivery is wedged, fx'sStopTimeoutfires and the app exits while the hook goroutine is still blocked. This is unchanged frommainand not introduced here, but it is the natural companion defect to the one being fixed and would make a reasonable follow-up.cancel:RetentionReaper.stopguardsif r.cancel != nil,Engine.stopdoes not. Pre-existing onmain, unreachable in practice since fx only runsOnStopafter a successfulOnStart. Cosmetic asymmetry only.recordingLifecycleis defined twice, once in each new test file. They are in different packages (delivery_testanddatabase_test) so this is legal and arguably preferable to a shared test module, but it is duplication a future reader may trip over.script/testalways passes-v, which diverges from the conditional-verbose-rerun pattern inREPO_POLICIES.md. Entirely pre-existing and out of scope for this PR.Summary
The two fixes are real, minimal, and correct. The regression tests drive the genuine registered hooks rather than a reimplementation, they fail deterministically against the bug in both directions, and the shutdown tests close the obvious way this fix could have gone wrong. The tree-wide sweep is accurate. The
unparamrefactor loses nothing. Failed-startup cleanup is preserved by fx's rollback path. Policy is clean and CI is green on the head commit.The one thing I would not repeat is resting the verification story on
script/cibuildexit 0 when the layers were cached — that claim needed the host-side evidence to stand up. It does stand up.Recommend
merge-ready.Manager note
Independent review verdict: PASS, no blocking findings. The reviewer did not author this change.
This is the release-blocker, so I asked for a higher evidentiary bar than usual and got it.
Why I am confident
selectbetween a readyctx.Done()and a readydeliveryCh. Because the helper now settles the pool before any work is enqueued, the buggy pool's firstselecthas no second ready case — the randomness is eliminated rather than merely biased. The residual 250ms wall-clock assumption held 5/5 under load, and its failure mode is asymmetric: it can only reduce mutation sensitivity, never cause a false CI failure.App.StartuseswithRollback, which callslifecycle.Stopon failure and runsOnStoponly for hooks already started — so goroutines cannot outlive a failedapp.Start. No tracing spans or hook-context values exist in the tree, so nothing was lost.OnStarthooks, all now_ context.Context.unparamrefactor loses no coverage — only twoiWaitForStatuscall sites existed onmain, both waiting onDelivered. No test waited onfailedthrough that helper.One correction to the PR's own narrative
The PR body says
script/cibuildran with tests uncached. On the reviewer's run every Docker layer wasCACHED, includingmake testandmake lint, so that command by itself proved nothing. The conclusion still holds — layers are content-addressed over the copied source, and the Gitea CI run on20a050bis genuinely fresh and green — but the reviewer substituted host-side evidence rather than accept the claim. Recording it because "cibuild exit 0" is load-bearing in a lot of our PR bodies and it is worth knowing when it is and is not evidence.Non-blocking, tracked
Filed as #102:
OnStopin both components ignores its context and callswg.Wait()unbounded, so a wedged goroutine hangs shutdown forever. That is the exact mirror of the bug this PR fixes — there a long-lived goroutine wrongly inherited the start context, here shutdown wrongly ignores the stop context. Pre-existing and unchanged by this PR, so not a gate, but it belongs on the 1.0 list. TheEngine.stopmissingcancel != nilguard and the duplicatedrecordingLifecyclehelper went into the same issue.Labeled
merge-readyand assigned to @sneak.Merge ordering
I would land this one first. It fixes the highest-severity defect on
mainand PR #95 also touchesinternal/delivery/engine.go, so #95 should rebase onto the newmainrather than the other way round.Verification re-check: the green is real
A fleet-wide warning came in that
script/cibuildcan report a green it did not earn. It is a plaindocker build .with no cache control, and the Dockerfile doesCOPY . .thenRUN make check, so on an unchanged tree Docker serves the check layer from cache — the suite never runs and the build still exits 0. Observed elsewhere as a SUCCESS in 0.262 seconds with every layerCACHED, against 64.3 seconds forced uncached.That matters here because this repo's host golangci-lint (v2.10.1) disagrees with the pinned v2.12.2, so the pinned-linter result is exactly what a cached layer would leave unproven — and it is load-bearing in this PR's verification narrative. The reviewer had already flagged that their own
script/cibuildrun was fully cache-hit.Re-checked against Gitea CI, which builds each commit on a runner independently of any local Docker cache:
20a050b—check / check (push): success in 3m3s (run 104)A cached build finishes in under a second. Three minutes is a genuine execution of
make fmt-check,make lint, andmake testinside the pinned v2.12.2 image. The pinned-linter claim stands on its own evidence. No re-label, no pull-back.For completeness, every currently merge-ready PR was re-checked the same way and all five have genuine multi-minute CI runs: #87
f32284a2m37s, #9108c9c1a3m6s, #92985464d6m3s, #9613de7cd2m43s, #10020a050b3m3s.Going forward a local
script/cibuildexit 0 is only cited as evidence when it demonstrably ran — wall time checked and noCACHEDon the check layers — otherwise the Gitea CI run with its duration is the evidence. The upstream template fix (anARG CHECK_EPOCHaboveRUN make check) is filed in thepromptsrepo as #26; nothing to change in this repo.CI integrity: proven by a red/green pair on this very branch
A concern was raised across the fleet that a Gitea
successtick might not reflect a real run — at another repo, the job log returned for a green head commit was dated roughly six months before that commit existed and showed a build step that no longer exists in the codebase. Since several webhooker PRs were cleared on CI evidence after their localscript/cibuildruns came back cached, that would have voided the clearance.Checked. webhooker's CI is genuinely executing the real gate.
The Actions API is closed to
clawbotin every direction (get_run→ 404,list_jobs→ 403 "user should be the owner of the repo",list_run_jobs→ empty,get_job_log_preview→ 500), so a job-log read was not possible. This branch supplied a better test by accident.While recovering an interrupted agent's work I pushed a deliberately-labelled WIP commit that I knew carried three lint findings (
funcorder,unparam, and one more). It was then fixed and force-pushed. Same branch, same files, about twenty minutes apart:ce1e46bfailure— "Failing after 1m2s", run 103,2026-08-09T06:57:42+02:0020a050bsuccess— "Successful in 3m3s", run 104,2026-08-09T07:18:20+02:00That establishes four things a single log read could not:
Scope of the claim
This proves the gate ran and discriminated correctly on these commits in this repo. It does not prove that every individual green in the merge queue was a fully uncached execution end to end, and it says nothing about the other repo, where the reported symptom is real and remains under investigation. If anything it narrows that: whatever is wrong there is not a site-wide Gitea Actions defect.
No labels or assignments were changed on the strength of the alarm — the check came first, and the evidence held.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.