clawbot
66ea7f5f45
check / check (push) Waiting to run
A restore of selected paths, or one where files failed under --skip-errors, kept the blobs those files share in the temporary cache until it ended: the sweeper freed a blob only once every file in the snapshot database that uses it had been restored, and files outside the selection or that failed never are. The sweeper now asks the restore plan, through pendingFileUsesBlob, whether any pending file still uses a cached blob. The plan holds only this run's files and drops each one once it is restored or has failed. The new test backs up 16 KiB files so it stays quick; the locality backup helper takes the file size for that. Model: opus-5-5
62 lines
1.7 KiB
Go
62 lines
1.7 KiB
Go
package vaultik
|
|
|
|
// restoreSweeper frees cached blobs once no file still to be restored in
|
|
// this run needs them. It works as follows:
|
|
//
|
|
// 1. After each chunk is restored, chunkRestored accumulates a running
|
|
// byte count.
|
|
// 2. When the accumulator crosses a threshold (one hundredth of the
|
|
// configured blob size — so a sweep runs about a hundred times per
|
|
// blob's worth of restored bytes), the sweeper iterates every key in
|
|
// the cache and deletes each blob that the restore plan no longer
|
|
// needs.
|
|
//
|
|
// The plan drops a file once it is restored or has failed, and holds
|
|
// only the files this run restores, so files outside the requested paths
|
|
// never keep a blob.
|
|
type restoreSweeper struct {
|
|
plan *restorePlan
|
|
cache *blobDiskCache
|
|
threshold int64
|
|
bytesAccum int64
|
|
}
|
|
|
|
// newRestoreSweeper returns a sweeper that triggers eviction every
|
|
// `threshold` bytes restored. Callers should pass blob_size_limit/100.
|
|
func newRestoreSweeper(
|
|
plan *restorePlan,
|
|
cache *blobDiskCache,
|
|
threshold int64,
|
|
) *restoreSweeper {
|
|
if threshold <= 0 {
|
|
threshold = 1
|
|
}
|
|
|
|
return &restoreSweeper{
|
|
plan: plan,
|
|
cache: cache,
|
|
threshold: threshold,
|
|
}
|
|
}
|
|
|
|
// chunkRestored accounts n bytes against the sweep threshold and runs a
|
|
// sweep if the threshold has been crossed since the last sweep.
|
|
func (s *restoreSweeper) chunkRestored(n int64) {
|
|
s.bytesAccum += n
|
|
if s.bytesAccum < s.threshold {
|
|
return
|
|
}
|
|
|
|
s.bytesAccum = 0
|
|
s.sweep()
|
|
}
|
|
|
|
// sweep deletes every cached blob that no pending file references.
|
|
func (s *restoreSweeper) sweep() {
|
|
for _, blobHash := range s.cache.Keys() {
|
|
if !s.plan.pendingFileUsesBlob(blobHash) {
|
|
s.cache.Delete(blobHash)
|
|
}
|
|
}
|
|
}
|