Keep unreferenced blobs newer than prune_grace_period (closes #387)

A backup uploads its blobs before the manifest that lists them, and the
process lock covers one user on one machine, so a prune from another host
or user sharing the destination deleted a running backup's blobs.

Prune now deletes an unreferenced blob only once its listed time is older
than the new config key prune_grace_period (default 24h; a negative value
is refused). A newer blob, or one listed with no time, is kept and counted
(blobs_too_new). A backup uploads a blob already on the destination again
instead of skipping it, so it gets a fresh time. The rclone backend takes
each object's time from the listing (use_server_modtime).

Cost: after database delete, the next backup uploads all its data again.
Side effect: a rebuilt blob replaces the stored copy; if compression_level
or the recipients changed since, older snapshots listing it fail verify.

Model: opus-5-5

internal/cli/config.go

@@ -239,6 +239,13 @@ storage_url: ""
 # Default: 3
 # compression_level: 3
 
+# How old a blob that no snapshot lists must be before prune deletes it.
+# A backup uploads its blobs before the manifest that lists them, so on a
+# destination shared between hosts or users this must be longer than the
+# longest backup run. Accepts: 36h, 90m, etc. 0s deletes them at once.
+# Default: 24h
+# prune_grace_period: 24h
+
 # Hostname used in snapshot IDs. Default: system hostname.
 # hostname: myserver
 

internal/cli/database.go

@@ -48,9 +48,9 @@ This will erase all local tracking of:
 - The storage-binding record
 
 The remote storage is NOT affected. After deletion, the next backup
-will perform a full scan and re-deduplicate against existing remote
-blobs, and the local index will re-bind to the currently configured
-storage destination on that run.
+will perform a full scan and upload all the data again, and the local
+index will re-bind to the currently configured storage destination on
+that run.
 
 Use --force to skip the confirmation prompt.`,
 		Args: cobra.NoArgs,

internal/cli/entry_prune_quiet_test.go

@@ -5,6 +5,7 @@ import (
 	"path/filepath"
 	"strings"
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -14,8 +15,8 @@ import (
 const flagCron = "--cron"
 
 // plantUnreferencedBlob writes a blob that no snapshot references to the
-// destination store at storeDir, so that a prune deletes it. Returns its
-// path.
+// destination store at storeDir, dated long before any prune grace period,
+// so that a prune deletes it. Returns its path.
 func plantUnreferencedBlob(t *testing.T, storeDir string) string {
 	t.Helper()
 
@@ -25,6 +26,9 @@ func plantUnreferencedBlob(t *testing.T, storeDir string) string {
 	require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o750))
 	require.NoError(t, os.WriteFile(path, []byte("unreferenced"), 0o600))
 
+	longAgo := time.Unix(0, 0)
+	require.NoError(t, os.Chtimes(path, longAgo, longAgo))
+
 	return path
 }
 

internal/cli/prune.go

@@ -20,7 +20,10 @@ Local cleanup drops incomplete snapshots and any files, chunks, or
 blobs no longer referenced by a completed snapshot. Remote cleanup
 scans every snapshot manifest in the destination store, builds the
 set of still-referenced blob hashes, and deletes any blob not in that
-set.
+set that is older than prune_grace_period (default 24h). A newer one
+is kept: it may belong to a backup still running, on this host or
+another sharing the destination store, that has not yet uploaded its
+manifest.
 
 Snapshot create --prune runs the same cleanup automatically; this
 command is the manual entry point for the same work (e.g. after a

internal/cli/snapshot.go

@@ -132,8 +132,8 @@ on Linux), then /etc/vaultik/config.yml.`,
 		"Run in cron mode (silent unless warning or error)")
 	cmd.Flags().BoolVar(&opts.Prune, "prune", false,
 		"After backup, drop older snapshots of the same name and remove "+
-			"orphaned blobs (not after a backup that skipped unreadable "+
-			"files)")
+			"orphaned blobs older than prune_grace_period (not after a "+
+			"backup that skipped unreadable files)")
 	cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
 		"With --prune: keep snapshots newer than this duration "+
 			"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+