Keep unreferenced blobs newer than prune_grace_period (closes #387)

A backup uploads its blobs before the manifest that lists them, and the
process lock is per user per machine, so a prune from another host or
user sharing the destination deleted a running backup's blobs; that
backup still reported success and could not be restored.

Prune now deletes an unreferenced blob only once it is older than the new
config key prune_grace_period (default 24h), and reports how many it kept
(blobs_too_new under --json). The storage listing now carries each
object's modification time. remote nuke is unchanged. The README says what
the lock covers and that a shared destination relies on the grace period
or needs one prefix per host.

Model: opus-5-5

README.md

@@ -456,8 +456,7 @@ exits non-zero.
 * `--json`: Output stats as JSON. Also skips the confirmation prompt, as
   `--force` does. A blob that cannot be deleted is counted in
   `blobs_failed`, and the document is printed before the non-zero exit.
-  A blob kept for being newer than `prune_grace_period`, or because the
-  listing of the destination gives no time for it, is counted in
+  A blob kept for being newer than `prune_grace_period` is counted in
   `blobs_too_new` and not in `blobs_found`.
 
 **`info`**: Display system configuration, storage settings, encryption
@@ -493,8 +492,8 @@ deletes what is left.
 
 **`database delete`**: Delete the local SQLite state database file
 entirely. Remote storage is unaffected; the next backup will do a full
-scan and upload all the data again, and the local index will re-bind to
-the currently configured storage destination.
+scan and re-deduplicate against existing remote blobs, and the local
+index will re-bind to the currently configured storage destination.
 Use this after changing `storage_url`, or `s3.endpoint`, `s3.bucket` or
 `s3.prefix`, to a different destination.
 * `--force`: Skip confirmation prompt

docs/DATAMODEL.md

@@ -90,7 +90,7 @@ Stores information about packed, compressed, and encrypted blob files.
 - `created_ts` (INTEGER NOT NULL) - Creation timestamp
 - `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
 - `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression
-- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption. Once uploaded, the size of the object in storage
+- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption. Once uploaded, the size of the object in storage: when the blob was already stored, under another compression level or other recipients, this can differ from the size of the local copy
 - `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded)
 
 **Purpose:** Tracks blob lifecycle from creation through upload. The UUID primary key allows immediate association of chunks with blobs.
@@ -185,7 +185,7 @@ Tracks blob upload metrics.
    - `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after packing
 
 4. **Upload**
-   - `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as uploaded, with the size of the object in storage (a blob already stored is uploaded again, so that a prune does not take it for an unreferenced blob older than `prune_grace_period`)
+   - `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as uploaded, with the size of the object in storage (the upload is skipped when the blob is already stored)
    - `INSERT INTO uploads` - Record upload metrics with snapshot_id
    - `INSERT INTO snapshot_blobs` - Associate blob with snapshot
 

docs/REPOSTRUCTURE.md

@@ -150,7 +150,7 @@ These are known, deliberate properties of the format and the tooling, recorded s
 3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of 2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a blob holds hundreds of chunks, so individual chunk lengths are not visible in the blob's size; lowering the limit toward the chunk size begins to expose them.
 4. **Decrypted data on local disk.** Several commands stage plaintext under `$TMPDIR`: `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600` and removed on success, but a `kill -9` or a power loss leaves them behind — delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR` should be trusted to the same degree as the restore target.
 5. **Store permissions differ per command.** The backed-up host needs only PUT to run `snapshot create`: it writes blobs and metadata and neither reads nor deletes them. Other commands need more — `snapshot verify`, `snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`, `snapshot remove`, and `remote nuke` also delete. The recommended cron line uses `--prune`, which runs `prune` on the backed-up host, so granting that host `--prune` gives it credentials that can delete its own backups. To keep the source host to PUT only, prune from a separate trusted host instead.
-6. **Changing recipients does not re-encrypt existing data.** Deduplicated chunks already on the destination stay encrypted to the recipients in force when they were written. A new snapshot that reuses them cannot be restored with a newly added recipient's key alone, because those reused objects were never encrypted to it. To make everything readable by a new key, run `vaultik database delete` and take a full backup to a fresh destination or prefix.
+6. **Changing recipients does not re-encrypt existing data.** Deduplicated chunks and same-named blobs already on the destination stay encrypted to the recipients in force when they were written. A new snapshot that reuses them cannot be restored with a newly added recipient's key alone, because those reused objects were never encrypted to it. To make everything readable by a new key, run `vaultik database delete` and take a full backup to a fresh destination or prefix.
 7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients. Long-lived ciphertext held by a third party (the destination operator) has no fallback if X25519 is ever broken: there is no second recipient type and no post-quantum option.
 
 ## Consistency Guarantees

internal/cli/database.go

@@ -48,9 +48,9 @@ This will erase all local tracking of:
 - The storage-binding record
 
 The remote storage is NOT affected. After deletion, the next backup
-will perform a full scan and upload all the data again, and the local
-index will re-bind to the currently configured storage destination on
-that run.
+will perform a full scan and re-deduplicate against existing remote
+blobs, and the local index will re-bind to the currently configured
+storage destination on that run.
 
 Use --force to skip the confirmation prompt.`,
 		Args: cobra.NoArgs,

internal/config/config.go

@@ -69,8 +69,6 @@ var (
 	errFileURLNotAbsolute = errors.New(
 		"a file:// storage_url needs an absolute path, " +
 			"as in file:///path/to/backup")
-	errNegativePruneGracePeriod = errors.New(
-		"prune_grace_period must not be negative")
 	errStorageNotConfigured = errors.New(
 		"storage not configured; set storage_url or provide s3.endpoint + " +
 			"s3.bucket + credentials")
@@ -368,7 +366,6 @@ func Parse(data []byte) (*Config, error) {
 //     exceeds the configured limit
 //   - Compression level must be between 1 and 19
 //   - S3 part size must be between 5MiB and 5GiB, the part sizes S3 accepts
-//   - Prune grace period must not be negative
 //
 // Returns an error describing the first validation failure encountered.
 func (c *Config) Validate() error {
@@ -418,10 +415,6 @@ func (c *Config) Validate() error {
 		return errBadS3PartSize
 	}
 
-	if c.PruneGracePeriod < 0 {
-		return errNegativePruneGracePeriod
-	}
-
 	return nil
 }
 

internal/config/config_test.go

@@ -412,44 +412,6 @@ func TestDocumentedSizeExamples(t *testing.T) {
 	}
 }
 
-// TestValidatePruneGracePeriod checks that a negative prune_grace_period
-// is refused rather than acting like 0s, which deletes unreferenced blobs
-// at once, and that 0s and the documented examples are accepted.
-func TestValidatePruneGracePeriod(t *testing.T) {
-	t.Parallel()
-
-	const withoutGracePeriod = "snapshots:\n" +
-		"  test:\n" +
-		"    paths: [/tmp/vaultik-test-source]\n" +
-		"storage_url: file:///tmp/vaultik-test-storage\n"
-
-	tests := []struct {
-		setting string
-		wantErr error
-	}{
-		{"prune_grace_period: 0s", nil},
-		{"prune_grace_period: 90m", nil},
-		{"prune_grace_period: 36h", nil},
-		{"prune_grace_period: -1h", errNegativePruneGracePeriod},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.setting, func(t *testing.T) {
-			t.Parallel()
-
-			cfg, err := Parse([]byte(withoutGracePeriod + tt.setting + "\n"))
-			if err != nil {
-				t.Fatalf("Parse() unexpected error: %v", err)
-			}
-
-			err = cfg.Validate()
-			if !errors.Is(err, tt.wantErr) {
-				t.Fatalf("Validate() error = %v, want %v", err, tt.wantErr)
-			}
-		})
-	}
-}
-
 // TestLoadRejectsRelativeFileURL checks that a file:// storage_url with a
 // relative path fails at load. Accepted, it would resolve against the
 // working directory, so a backup run from another directory would write

internal/database/blobs.go

@@ -155,7 +155,9 @@ func (r *BlobRepository) UpdateFinished(
 }
 
 // UpdateUploaded marks a blob as uploaded and records the size of the
-// stored object as its compressed size.
+// stored object as its compressed size. That size can differ from the
+// local copy's when an identical blob was already stored with another
+// compression level or other recipients.
 func (r *BlobRepository) UpdateUploaded(
 	ctx context.Context, tx *sql.Tx, id string, storedSize int64,
 ) error {

internal/snapshot/scanner.go

@@ -1641,7 +1641,8 @@ func (s *Scanner) handleBlobReady(
 	blobPath := fmt.Sprintf("blobs/%s/%s/%s",
 		finishedBlob.Hash[:2], finishedBlob.Hash[2:4], finishedBlob.Hash)
 
-	err := s.uploadBlob(ctx, blobPath, blobWithReader, startTime, result)
+	storedSize, blobExists, err := s.uploadBlobIfNeeded(
+		ctx, blobPath, blobWithReader, startTime, result)
 	if err != nil {
 		cleanupErr := s.cleanupBlobTempFile(blobWithReader)
 
@@ -1649,7 +1650,8 @@ func (s *Scanner) handleBlobReady(
 			fmt.Errorf("uploading blob %s: %w", finishedBlob.Hash, err), cleanupErr)
 	}
 
-	err = s.recordBlobMetadata(ctx, finishedBlob, startTime)
+	err = s.recordBlobMetadata(
+		ctx, finishedBlob, storedSize, blobExists, startTime)
 	if err != nil {
 		cleanupErr := s.cleanupBlobTempFile(blobWithReader)
 
@@ -1673,33 +1675,46 @@ func (s *Scanner) handleBlobReady(
 	return nil
 }
 
-// uploadBlob uploads the blob to storage. A blob already there, left by
-// an earlier run stopped before its manifest, is uploaded again rather
-// than skipped: keeping its old time would let a prune elsewhere delete
-// it as an unreferenced blob older than prune_grace_period before this
-// backup's manifest lists it.
-func (s *Scanner) uploadBlob(
+// uploadBlobIfNeeded uploads the blob to storage if it doesn't already
+// exist. It returns the size of the object in storage, which for an
+// existing object can differ from the local copy's, and whether it
+// already existed.
+func (s *Scanner) uploadBlobIfNeeded(
 	ctx context.Context,
 	blobPath string,
 	blobWithReader *blob.WithReader,
 	startTime time.Time,
 	result *ScanResult,
-) error {
+) (int64, bool, error) {
 	finishedBlob := blobWithReader.FinishedBlob
+
+	// Check if blob already exists (deduplication after restart)
 	destination := s.storage.Info().Location
 
+	stored, err := s.storage.Stat(ctx, blobPath)
+	if err == nil {
+		log.Info("Blob already exists in storage, skipping upload",
+			"hash", finishedBlob.Hash,
+			"size", humanize.Bytes(safeUint64(finishedBlob.Compressed)))
+		s.ui.Infof("Blob %s (%s) already exists at %s. Skipping upload.",
+			s.ui.Hex(finishedBlob.Hash), s.ui.Size(finishedBlob.Compressed),
+			s.ui.Path(destination))
+
+		return stored.Size, true, nil
+	}
+
 	s.ui.Beginf("Uploading blob %s (%s) to %s.",
 		s.ui.Hex(finishedBlob.Hash), s.ui.Size(finishedBlob.Compressed),
 		s.ui.Path(destination))
 
 	progressCallback := s.makeUploadProgressCallback(ctx, finishedBlob, startTime)
 
-	err := s.storage.PutWithProgress(ctx, blobPath, blobWithReader.Reader,
+	err = s.storage.PutWithProgress(ctx, blobPath, blobWithReader.Reader,
 		finishedBlob.Compressed, progressCallback)
 	if err != nil {
 		log.Error("Failed to upload blob", "hash", finishedBlob.Hash, "error", err)
 
-		return fmt.Errorf("uploading blob to storage: %w", err)
+		return 0, false, fmt.Errorf("uploading blob to storage: %w", err)
 	}
 
 	uploadDuration := time.Since(startTime)
@@ -1730,7 +1745,7 @@ func (s *Scanner) uploadBlob(
 		stats.BytesUploaded.Add(finishedBlob.Compressed)
 	}
 
-	return nil
+	return finishedBlob.Compressed, false, nil
 }
 
 // makeUploadProgressCallback creates a progress callback for blob uploads.
@@ -1800,6 +1815,8 @@ func (s *Scanner) makeUploadProgressCallback(
 func (s *Scanner) recordBlobMetadata(
 	ctx context.Context,
 	finishedBlob *blob.FinishedBlob,
+	storedSize int64,
+	blobExists bool,
 	startTime time.Time,
 ) error {
 	finishedBlobID, err := types.ParseBlobID(finishedBlob.ID)
@@ -1811,7 +1828,7 @@ func (s *Scanner) recordBlobMetadata(
 
 	return s.repos.WithTx(ctx, func(txCtx context.Context, tx *sql.Tx) error {
 		err := s.repos.Blobs.UpdateUploaded(
-			txCtx, tx, finishedBlob.ID, finishedBlob.Compressed)
+			txCtx, tx, finishedBlob.ID, storedSize)
 		if err != nil {
 			return fmt.Errorf("updating blob upload timestamp: %w", err)
 		}
@@ -1822,17 +1839,19 @@ func (s *Scanner) recordBlobMetadata(
 			return fmt.Errorf("adding blob to snapshot: %w", err)
 		}
 
-		upload := &database.Upload{
-			BlobHash:   finishedBlob.Hash,
-			SnapshotID: s.snapshotID,
-			UploadedAt: startTime,
-			Size:       finishedBlob.Compressed,
-			DurationMs: uploadDuration.Milliseconds(),
-		}
+		if !blobExists {
+			upload := &database.Upload{
+				BlobHash:   finishedBlob.Hash,
+				SnapshotID: s.snapshotID,
+				UploadedAt: startTime,
+				Size:       finishedBlob.Compressed,
+				DurationMs: uploadDuration.Milliseconds(),
+			}
 
-		err = s.repos.Uploads.Create(txCtx, tx, upload)
-		if err != nil {
-			return fmt.Errorf("recording upload metrics: %w", err)
+			err := s.repos.Uploads.Create(txCtx, tx, upload)
+			if err != nil {
+				return fmt.Errorf("recording upload metrics: %w", err)
+			}
 		}
 
 		return nil

internal/storage/rclone.go

@@ -53,14 +53,6 @@ func NewRcloneStorer(ctx context.Context, remote, path string) (*RcloneStorer, e
 		rclonePath += path
 	}
 
-	// On S3, Swift and some other remotes rclone keeps a time of its own in
-	// each object's metadata, which costs one request per object to read.
-	// rclone's use_server_modtime setting makes ModTime the time the
-	// listing gives instead: when the object was stored, which is what
-	// prune compares with prune_grace_period.
-	ctx, ci := fs.AddConfig(ctx)
-	ci.UseServerModTime = true
-
 	// Create the rclone filesystem
 	fsys, err := fs.NewFs(ctx, rclonePath)
 	if err != nil {

internal/storage/rclone_test.go

@@ -5,17 +5,11 @@ import (
 	"context"
 	"errors"
 	"io"
-	"net/http"
-	"net/http/httptest"
 	"os"
 	"path/filepath"
 	"strings"
-	"sync/atomic"
 	"testing"
-	"time"
 
-	"github.com/johannesboyne/gofakes3"
-	"github.com/johannesboyne/gofakes3/backend/s3mem"
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/config"
 	"github.com/rclone/rclone/fs/config/configmap"
@@ -435,72 +429,3 @@ func TestNewRcloneStorerReadsRcloneConfigEnv(t *testing.T) {
 		t.Errorf("object is not in the remote's directory: %v", err)
 	}
 }
-
-// TestRcloneListStreamOnS3MakesNoRequestPerObject checks that listing an
-// rclone S3 remote takes each object's time from the listing itself. By
-// default rclone reads the time it keeps in each object's metadata, one
-// request per object, and every command that lists the destination store
-// would slow down as it grows.
-//
-//nolint:paralleltest // NewRcloneStorer installs the process-global rclone config
-func TestRcloneListStreamOnS3MakesNoRequestPerObject(t *testing.T) {
-	backend := s3mem.New()
-
-	err := backend.CreateBucket(s3TestBucket)
-	if err != nil {
-		t.Fatalf("create bucket: %v", err)
-	}
-
-	const body = "blob-bytes"
-
-	_, err = backend.PutObject(s3TestBucket, testBlobKey, nil,
-		strings.NewReader(body), int64(len(body)))
-	if err != nil {
-		t.Fatalf("seed object: %v", err)
-	}
-
-	var headRequests atomic.Int64
-
-	fakeS3 := gofakes3.New(backend).Server()
-	srv := httptest.NewServer(http.HandlerFunc(
-		func(w http.ResponseWriter, r *http.Request) {
-			if r.Method == http.MethodHead {
-				headRequests.Add(1)
-			}
-
-			fakeS3.ServeHTTP(w, r)
-		}))
-	t.Cleanup(srv.Close)
-
-	remote := ":s3,provider=Other" +
-		",access_key_id=" + s3TestAccessKeyID +
-		",secret_access_key=" + s3TestSecretAccessKey +
-		",endpoint='" + srv.URL + "'"
-	ctx := context.Background()
-
-	s, err := storage.NewRcloneStorer(ctx, remote, s3TestBucket)
-	if err != nil {
-		t.Fatalf("NewRcloneStorer: %v", err)
-	}
-
-	headRequests.Store(0)
-
-	var listed []storage.ObjectInfo
-
-	for info := range s.ListStream(ctx, "blobs/") {
-		listed = append(listed, info)
-	}
-
-	if len(listed) != 1 || listed[0].Err != nil {
-		t.Fatalf("ListStream(\"blobs/\") = %v, want the one object", listed)
-	}
-
-	if time.Since(listed[0].ModTime).Abs() > time.Minute {
-		t.Errorf("ListStream ModTime = %v, want within a minute of now",
-			listed[0].ModTime)
-	}
-
-	if n := headRequests.Load(); n != 0 {
-		t.Errorf("ListStream made %d HEAD request(s), want none", n)
-	}
-}

internal/vaultik/prune.go

@@ -166,8 +166,7 @@ type PruneBlobsResult struct {
 	BlobsFailed  int   `json:"blobs_failed,omitempty"`
 	BytesFreed   int64 `json:"bytes_freed"`
 	// BlobsTooNew counts the unreferenced blobs kept because they are
-	// newer than the prune grace period, or of unknown age. BlobsFound
-	// does not include them.
+	// newer than the prune grace period. BlobsFound does not include them.
 	BlobsTooNew int `json:"blobs_too_new"`
 }
 
@@ -396,8 +395,7 @@ func (v *Vaultik) listAllRemoteBlobs() (map[string]storage.ObjectInfo, error) {
 // findUnreferencedBlobs returns the hashes of the blobs no manifest
 // references that are older than the prune grace period, and their total
 // size. It also returns how many unreferenced blobs it left out for
-// being newer than that, or of unknown age because the listing gave no
-// time for them.
+// being newer than that.
 func (v *Vaultik) findUnreferencedBlobs(
 	allBlobs map[string]storage.ObjectInfo, referenced map[string]bool,
 ) ([]string, int64, int) {
@@ -414,7 +412,7 @@ func (v *Vaultik) findUnreferencedBlobs(
 			continue
 		}
 
-		if blob.ModTime.IsZero() || blob.ModTime.After(cutoff) {
+		if blob.ModTime.After(cutoff) {
 			tooNew++
 
 			continue
@@ -428,16 +426,15 @@ func (v *Vaultik) findUnreferencedBlobs(
 }
 
 // reportBlobsTooNew says how many unreferenced blobs prune kept for being
-// newer than the grace period or of unknown age. Under --json the
-// document carries the count instead.
+// newer than the grace period. Under --json the document carries the
+// count instead.
 func (v *Vaultik) reportBlobsTooNew(tooNew int, opts *PruneOptions) {
 	if tooNew == 0 || opts.JSON {
 		return
 	}
 
 	v.UI.Infof("Kept %d unreferenced blob(s) newer than "+
-		"prune_grace_period (%s), or of unknown age: a backup still "+
-		"running may need them.",
+		"prune_grace_period (%s): a backup still running may need them.",
 		tooNew, v.Config.PruneGracePeriod)
 }
 

internal/vaultik/prune_grace_period_test.go

@@ -84,128 +84,6 @@ func TestPruneDuringBackupKeepsItsBlobs(t *testing.T) {
 	assertRestoredTree(t, fs, restoreDir, testFiles)
 }
 
-// TestPruneDuringBackupKeepsBlobsOfKilledRun covers a backup that builds
-// the blobs an earlier run, stopped before its manifest upload, left on
-// the destination store, older than the grace period by the time of the
-// backup. A prune from another host runs before the backup's manifest
-// upload, and the backup must then restore.
-func TestPruneDuringBackupKeepsBlobsOfKilledRun(t *testing.T) {
-	log.Initialize(log.Config{})
-	t.Parallel()
-
-	fs := afero.NewOsFs()
-	tempDir := t.TempDir()
-	dataDir := filepath.Join(tempDir, "src")
-	storeDir := t.TempDir()
-	restoreDir := filepath.Join(tempDir, "restored")
-	dbPath := filepath.Join(tempDir, "index.sqlite")
-
-	ctx := context.Background()
-	testFiles := writeFaultSourceTree(t, fs, dataDir)
-
-	cfg := faultTestConfig()
-	cfg.IndexPath = dbPath
-	cfg.ChunkSize = config.Size(faultChunkSize)
-	cfg.Snapshots = map[string]config.SnapshotConfig{
-		"data": {Paths: []string{dataDir}},
-	}
-
-	inner, err := storage.NewFileStorer(storeDir)
-	require.NoError(t, err)
-
-	otherHostCfg := faultTestConfig()
-	otherHostCfg.PruneGracePeriod = time.Hour
-	otherHost := newReaderVaultik(ctx, otherHostCfg, inner, nil, fs)
-
-	var (
-		killedRun = true
-		pruned    bool
-		pruneErr  error
-	)
-
-	store := faultstore.New(inner)
-	store.OnPut = func(key string) faultstore.PutAction {
-		if !strings.HasSuffix(key, "/manifest.json.zst") {
-			return faultstore.PutNormal
-		}
-
-		if killedRun {
-			return faultstore.PutFail
-		}
-
-		pruned = true
-		pruneErr = otherHost.PruneBlobs(&vaultik.PruneOptions{Force: true})
-
-		return faultstore.PutNormal
-	}
-
-	db, err := database.New(ctx, dbPath)
-	require.NoError(t, err)
-
-	repos := database.NewRepositories(db)
-	v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
-	opts := &vaultik.SnapshotCreateOptions{Cron: true}
-
-	require.Error(t, v.CreateSnapshot(opts),
-		"the first run must stop at its manifest upload")
-
-	beforeGracePeriod := time.Now().Add(-2 * otherHostCfg.PruneGracePeriod)
-
-	for object := range inner.ListStream(ctx, "blobs/") {
-		require.NoError(t, object.Err)
-		require.NoError(t, os.Chtimes(filepath.Join(storeDir, object.Key),
-			beforeGracePeriod, beforeGracePeriod))
-	}
-
-	killedRun = false
-
-	require.NoError(t, v.CreateSnapshot(opts))
-	require.True(t, pruned, "the prune must run before the manifest upload")
-	require.NoError(t, pruneErr)
-
-	local, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
-	require.NoError(t, err)
-	require.Len(t, local, 1)
-	require.NoError(t, db.Close())
-
-	reader := newReaderVaultik(ctx, cfg, inner, nil, fs)
-	require.NoError(t, reader.Restore(&vaultik.RestoreOptions{
-		SnapshotID: local[0].ID.String(),
-		TargetDir:  restoreDir,
-		Verify:     true,
-	}), "a prune during the backup must not delete the blobs it rebuilt")
-
-	assertRestoredTree(t, fs, restoreDir, testFiles)
-}
-
-// TestPruneKeepsUnreferencedBlobWithNoTime checks that a blob no
-// manifest lists, whose listing gives no modification time (as an S3
-// listing entry without LastModified does), is kept and counted with
-// the blobs too new to delete: its age is unknown.
-func TestPruneKeepsUnreferencedBlobWithNoTime(t *testing.T) {
-	log.Initialize(log.Config{})
-	t.Parallel()
-
-	env := newListEnv(t)
-
-	// Written into the mock store's map directly rather than through
-	// Put, so its listing gives no time.
-	blobKey := "blobs/" + testBlobHashA[:2] + "/" + testBlobHashA[2:4] +
-		"/" + testBlobHashA
-	env.store.data[blobKey] = []byte("blob")
-
-	require.NoError(t, env.v.PruneBlobs(&vaultik.PruneOptions{JSON: true}))
-
-	assert.True(t, env.store.hasKey(blobKey),
-		"a blob of unknown age must be kept")
-
-	var result vaultik.PruneBlobsResult
-
-	require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &result))
-	assert.Equal(t, 0, result.BlobsDeleted)
-	assert.Equal(t, 1, result.BlobsTooNew)
-}
-
 // TestPruneDeletesUnreferencedBlobOlderThanGracePeriod checks that the
 // grace period only delays deletion: a blob no manifest lists is deleted
 // once it is older than the grace period, and kept and counted until

internal/vaultik/remove_snapshot_test.go

@@ -26,14 +26,11 @@ import (
 type testStorer struct {
 	mu   sync.Mutex
 	data map[string][]byte
-	// putTimes holds when Put stored each object; ListStream reports it.
-	putTimes map[string]time.Time
 }
 
 func newTestStorer() *testStorer {
 	return &testStorer{
-		data:     make(map[string][]byte),
-		putTimes: make(map[string]time.Time),
+		data: make(map[string][]byte),
 	}
 }
 
@@ -47,7 +44,6 @@ func (s *testStorer) Put(_ context.Context, key string, reader io.Reader) error
 	}
 
 	s.data[key] = data
-	s.putTimes[key] = time.Now().UTC()
 
 	return nil
 }
@@ -91,7 +87,6 @@ func (s *testStorer) Delete(_ context.Context, key string) error {
 	defer s.mu.Unlock()
 
 	delete(s.data, key)
-	delete(s.putTimes, key)
 
 	return nil
 }
@@ -125,9 +120,8 @@ func (s *testStorer) ListStream(
 		for key, data := range s.data {
 			if prefix == "" || strings.HasPrefix(key, prefix) {
 				ch <- storage.ObjectInfo{
-					Key:     key,
-					Size:    int64(len(data)),
-					ModTime: s.putTimes[key],
+					Key:  key,
+					Size: int64(len(data)),
 				}
 			}
 		}

internal/vaultik/reused_blob_verify_test.go

@@ -14,13 +14,13 @@ import (
 	"sneak.berlin/go/vaultik/internal/vaultik"
 )
 
-// TestVerifyPassesWhenBackupRebuildsStoredBlob backs up a file, then backs
-// it up again from an empty local index, as after `database delete`, with
-// an extra age recipient. The second backup builds the same blob, already
-// stored, and uploads it again; its copy is larger, because age writes a
-// header entry for each recipient. The second snapshot must record the
-// size of the object now stored, so shallow and deep verify pass.
-func TestVerifyPassesWhenBackupRebuildsStoredBlob(t *testing.T) {
+// TestVerifyPassesWhenBackupReusesStoredBlob backs up a file, then backs it
+// up again from an empty local index, as after `database delete`, with an
+// extra age recipient. The second backup builds the same blob, finds it
+// already stored and skips the upload, but its own copy is larger, because
+// age writes a header entry for each recipient. The second snapshot must
+// record the size of the stored object, so shallow and deep verify pass.
+func TestVerifyPassesWhenBackupReusesStoredBlob(t *testing.T) {
 	log.Initialize(log.Config{})
 	t.Parallel()
 
@@ -58,8 +58,8 @@ func TestVerifyPassesWhenBackupRebuildsStoredBlob(t *testing.T) {
 	snapshotID, result := runDedupSnapshot(ctx, t, env.sm, env.makeScanner(),
 		env.cfg.Hostname, "second", dataDir, secondDBPath)
 	require.Positive(t, result.BlobsCreated)
-	require.Equal(t, result.BlobsCreated, result.BlobsUploaded,
-		"the second backup should upload every blob again")
+	require.Zero(t, result.BlobsUploaded,
+		"the second backup should find every blob already stored")
 
 	require.NoError(t, env.db.Close())