Run the reconciliation pass at startup and once a day (closes #250)
check / check (push) Waiting to run
check / check (push) Waiting to run
The pass makes a database query for every cached file and ran on every eviction tick, so on a large cache one pass could outlast the tick and the next started at once. It now runs on a timer of its own that starts when a pass ends, a day long. Eviction and the removal of expired negative cache rows keep the eviction interval. Model: opus-5-5
This commit is contained in:
@@ -18,6 +18,11 @@ import (
|
||||
// write-pressure wakeups triggered by stores.
|
||||
const DefaultEvictionInterval = 5 * time.Minute
|
||||
|
||||
// reconciliationInterval is how long after one reconciliation pass ends
|
||||
// the next one starts. A pass reads a row for every cached file, so its
|
||||
// cost grows with the cache, and it runs this rarely for that reason.
|
||||
const reconciliationInterval = 24 * time.Hour
|
||||
|
||||
// evictionBatchSize is how many LRU candidates of each class (variants
|
||||
// and source blobs) one eviction pass fetches from the database.
|
||||
const evictionBatchSize = 100
|
||||
@@ -437,15 +442,17 @@ func (c *Cache) notifyWritePressure() {
|
||||
}
|
||||
}
|
||||
|
||||
// StartEviction launches the background eviction goroutine, which
|
||||
// reconciles the database accounting with the cache directories and
|
||||
// removes the expired negative cache rows at startup and again on every
|
||||
// periodic tick thereafter, and evicts to the configured limit on the
|
||||
// given periodic interval and on write-pressure notifications. On a
|
||||
// disabled cache it only removes the expired negative cache rows, as
|
||||
// the negative cache is used either way. It is a no-op when already
|
||||
// started. The goroutine outlives the caller, so it runs with its own
|
||||
// context, which StopEviction cancels.
|
||||
// StartEviction launches the background eviction goroutine. At startup
|
||||
// it reconciles the database accounting with the cache directories,
|
||||
// removes the expired negative cache rows and evicts to the configured
|
||||
// limit. After that it reconciles again a day after each reconciliation
|
||||
// pass ends, removes the expired negative cache rows on every tick of
|
||||
// the given interval, and evicts to the limit on every tick, after each
|
||||
// reconciliation pass and on write-pressure notifications. On a disabled
|
||||
// cache it only removes the expired negative cache rows, as the negative
|
||||
// cache is used either way. It is a no-op when already started. The
|
||||
// goroutine outlives the caller, so it runs with its own context, which
|
||||
// StopEviction cancels.
|
||||
func (c *Cache) StartEviction(interval time.Duration) {
|
||||
if c.evictionCancel != nil {
|
||||
return
|
||||
@@ -486,6 +493,16 @@ func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
|
||||
c.runCleanExpired(ctx)
|
||||
c.runEvictionPass(ctx)
|
||||
|
||||
// Reconciliation walks the cache directories and reads a row for
|
||||
// every file, so it runs on a timer of its own rather than on the
|
||||
// periodic tick or on write-pressure wakeups. The timer starts only
|
||||
// once a pass has ended, so passes never run back to back, however
|
||||
// long one takes. Running it daily, not just at startup, bounds how
|
||||
// long a store's best-effort accounting insert can stay silently
|
||||
// unaccounted for to about one day, plus the time the passes take.
|
||||
reconciliationTimer := time.NewTimer(reconciliationInterval)
|
||||
defer reconciliationTimer.Stop()
|
||||
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
|
||||
@@ -493,17 +510,10 @@ func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
// Reconciliation walks the cache directories, so it only
|
||||
// runs on the periodic ticker rather than on every
|
||||
// write-pressure wakeup, keeping it off the per-store hot
|
||||
// path. Reusing the eviction interval itself (rather than a
|
||||
// separate, longer one) is a deliberate choice: it is the
|
||||
// simplest option that still bounds how long a store's
|
||||
// best-effort accounting insert can stay silently
|
||||
// unaccounted for to one interval, on a process that is
|
||||
// already running this loop regardless.
|
||||
case <-reconciliationTimer.C:
|
||||
c.runReconciliationPass(ctx)
|
||||
reconciliationTimer.Reset(reconciliationInterval)
|
||||
case <-ticker.C:
|
||||
c.runCleanExpired(ctx)
|
||||
case <-c.evictionPressure:
|
||||
}
|
||||
@@ -556,19 +566,18 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
|
||||
|
||||
// reconcileAccounting synchronizes the database size accounting with
|
||||
// the actual contents of the cache directories. It runs at startup and
|
||||
// again on every periodic eviction tick thereafter, off the request
|
||||
// hot path: it adopts variant files that predate the accounting table
|
||||
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
|
||||
// insert under transient DB contention), drops accounting rows whose
|
||||
// files are missing, removes source blob files the database does not
|
||||
// know (and rows whose files are gone), removes source and variant files
|
||||
// that are empty or not the size their rows record, sweeps stale temp
|
||||
// files left behind by crashed writes, and last checks the total cache
|
||||
// usage against the tables. It reads the tables a page at a time.
|
||||
// Running it periodically, not just once, bounds how long such drift can
|
||||
// accumulate unaccounted for on a long-running process to one eviction
|
||||
// interval. Once ctx is cancelled, it stops at the next file or row and
|
||||
// returns ctx's error.
|
||||
// then once a day, off the request hot path: it adopts variant files
|
||||
// that predate the accounting table (or whose accounting insert failed,
|
||||
// e.g. StoreVariant's best-effort insert under transient DB contention),
|
||||
// drops accounting rows whose files are missing, removes source blob
|
||||
// files the database does not know (and rows whose files are gone),
|
||||
// removes source and variant files that are empty or not the size their
|
||||
// rows record, sweeps stale temp files left behind by crashed writes,
|
||||
// and last checks the total cache usage against the tables. It reads the
|
||||
// tables a page at a time. Running it daily, not just once, bounds how
|
||||
// long such drift can accumulate unaccounted for on a long-running
|
||||
// process to about one day. Once ctx is cancelled, it stops at the next
|
||||
// file or row and returns ctx's error.
|
||||
func (c *Cache) reconcileAccounting(ctx context.Context) error {
|
||||
if c.disabled {
|
||||
return nil
|
||||
|
||||
Reference in New Issue
Block a user