Replace LRU eviction in restore with reference-counted sweeper

Restore previously capped the blob disk cache at 4× the configured
blob_size_limit (so 40 GB by default). With large or heavily-deduped
snapshots a chunk-by-chunk file walk could blow past that cap and
trigger LRU eviction of blobs that were still needed by later files,
forcing repeated re-downloads — observed during a real restore as
single-stream throughput collapsing to under 1 MB/s.

Restore now allocates the cache with no practical size cap and drives
eviction explicitly:

  * An in-memory set of restored file IDs accumulates as files finish.
  * Every blob_size_limit/100 bytes of restored data (≈100 sweeps per
    blob's worth of writes) the sweeper iterates the cache. For each
    cached blob it queries the snapshot's local SQLite DB for every
    file that references any chunk in the blob and deletes the cache
    entry only when every such file is already in the restored set.
  * blobStillNeeded returns true on any error so an unreadable DB
    never causes premature eviction.

The cache itself gains Delete(key) and Keys() so the sweeper can drive
removal without touching internal LRU state.
This commit is contained in:
2026-06-17 07:15:22 +02:00
parent cf8a527d35
commit 683fb0b103
3 changed files with 171 additions and 3 deletions
+28
View File
@@ -181,6 +181,34 @@ func (c *blobDiskCache) Has(key string) bool {
return ok
}
// Delete removes a blob from the cache and its disk file. No-op if absent.
// Used by restore's sweep logic to free blobs whose chunks have all been
// restored (so they will never be needed again during this restore).
func (c *blobDiskCache) Delete(key string) {
c.mu.Lock()
defer c.mu.Unlock()
e, ok := c.items[key]
if !ok {
return
}
c.unlink(e)
delete(c.items, key)
c.curBytes -= e.size
_ = os.Remove(c.path(key))
}
// Keys returns a snapshot of all cached keys. Safe for iteration without
// holding the cache lock; the cache may change concurrently.
func (c *blobDiskCache) Keys() []string {
c.mu.Lock()
defer c.mu.Unlock()
keys := make([]string, 0, len(c.items))
for k := range c.items {
keys = append(keys, k)
}
return keys
}
// Size returns current total cached bytes.
func (c *blobDiskCache) Size() int64 {
c.mu.Lock()