Eviction loop is not context-cancellable; shutdown ignores its deadline #102
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Surfaced by the
contextchecklinter while bringing #55's code under the canonical.golangci.ymlin PR #54. Filing rather than silencing: it is a real design choice, not a style nit.What the linter reports
What is actually going on
internal/handlers/handlers.goregisters the eviction lifecycle on fx:Both hooks discard the
context.Contextfx hands them.Cache.evictionLoop(internal/imgcache/eviction.go) then makes its own:Every database call in the eviction and reconciliation passes descends from that
context.Background(), so nothing in the loop is cancellable. Shutdown relies entirely on theevictionStopchannel:StopEvictionblocks on<-c.evictionDone, and the loop only observesevictionStopbetween passes. So an eviction or reconciliation pass that is already in flight whenOnStopfires runs to completion, however long that takes, and the shutdown deadline fx passes toOnStophas no effect on it. A reconciliation pass walks the whole variant and source cache directory tree, so on a large cache this is not a negligible amount of work to be unable to interrupt.Discarding the
OnStartcontext is correct and must stay that way: the loop has to outliveOnStart, so it cannot inherit that context. The gap is that nothing replaces it.Suggested fix
Give
Cacheits own cancellable context for the eviction goroutine:StartEvictionderivesctx, cancel := context.WithCancel(context.Background())and storescancelon theCache.evictionLoopuses thatctxinstead of making its own, and selects onctx.Done()alongsideevictionStop.StopEvictioncallscancel()before closingevictionStop, so an in-flight pass unwinds via context cancellation instead of running to completion. The<-c.evictionDonewait then returns promptly.Worth deciding as part of this: whether
StopEvictionshould take acontext.ContextsoOnStopcan pass fx's shutdown deadline straight through, and whether an interrupted eviction pass needs anything beyond the transaction rollback it already gets (it should not:evictSourceBlobdeletes rows in one transaction before unlinking, so a cancelled pass leaves accounting consistent and the next reconciliation pass picks up any orphaned file).Why not in PR #54
That PR replaces the linter config and fixes the findings mechanically, explicitly without behavior changes. This one changes shutdown semantics of concurrency-sensitive code that had just passed adversarial review on #55, so it belongs in its own change with its own tests. PR #54 carries a
//nolint:contextcheckon theOnStarthook that points at this issue.Correction to this issue's body, which is now out of date in one respect.
The "Why not in PR #54" section says that PR replaces the linter config and fixes findings "explicitly without behavior changes". That framing was accurate when this issue was filed, but PR #54's round-4 review established that the PR is not a pure no-op, and its description has since been rewritten to disclose three behavior deltas:
Cache.StoreVariantnow takes acontext.Contextand usesExecContext— on a cancelled request the accounting row is skipped wheremaincommitted it.MetadataStorage.Storeno longer leaks.tmp-*.jsonfiles — its cleanup defer was dead onmain(unnamed result parameter, so the closure read an outererrleft nil by the successfulos.CreateTemp, while all three failure paths shadowed it). A genuine latent bug fix.signing_keyvalidation error text gainedvalue too short:.The reasoning for deferring this issue is unaffected, and if anything is stronger now. The argument was never that PR #54 contains literally zero behavior change — it is that changing the shutdown semantics of concurrency-sensitive code does not belong in a lint-conformance pass. The three deltas above are each local and individually reviewable; making the eviction loop cancellable is a design change to code that had just cleared adversarial review on #55, and it deserves its own change with its own tests.
Leaving the body as-is rather than rewriting it, so the record of what was believed when stays intact — but treat this comment as the correction.