Keep unreferenced blobs newer than prune_grace_period (closes #387)
check / check (push) Failing after 5m50s
check / check (push) Failing after 5m50s
A backup uploads its blobs before the manifest that lists them, and the process lock covers one user on one machine, so a prune from another host or user sharing the destination deleted a running backup's blobs. Prune now deletes an unreferenced blob only once its listed time is older than the new config key prune_grace_period (default 24h; a negative value is refused). A newer blob, or one listed with no time, is kept and counted (blobs_too_new). A backup uploads a blob already on the destination again instead of skipping it, so it gets a fresh time. The rclone backend takes each object's time from the listing (use_server_modtime). Cost: after database delete, the next backup uploads all its data again. Side effect: a rebuilt blob replaces the stored copy; if compression_level or the recipients changed since, older snapshots listing it fail verify. Model: opus-5-5
This commit is contained in:
@@ -53,6 +53,14 @@ func NewRcloneStorer(ctx context.Context, remote, path string) (*RcloneStorer, e
|
||||
rclonePath += path
|
||||
}
|
||||
|
||||
// On S3, Swift and some other remotes rclone keeps a time of its own in
|
||||
// each object's metadata, which costs one request per object to read.
|
||||
// rclone's use_server_modtime setting makes ModTime the time the
|
||||
// listing gives instead: when the object was stored, which is what
|
||||
// prune compares with prune_grace_period.
|
||||
ctx, ci := fs.AddConfig(ctx)
|
||||
ci.UseServerModTime = true
|
||||
|
||||
// Create the rclone filesystem
|
||||
fsys, err := fs.NewFs(ctx, rclonePath)
|
||||
if err != nil {
|
||||
@@ -211,8 +219,9 @@ func (r *RcloneStorer) ListStream(
|
||||
|
||||
if prefix == "" || strings.HasPrefix(key, prefix) {
|
||||
ch <- ObjectInfo{
|
||||
Key: key,
|
||||
Size: obj.Size(),
|
||||
Key: key,
|
||||
Size: obj.Size(),
|
||||
ModTime: obj.ModTime(ctx),
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user