Report unknown blob figures for a snapshot whose manifest cannot be read (closes #272)

When remote info could not read a snapshot's manifest, the orphan
figures were unknown but the snapshot's row still gave 0 blobs and 0 B,
in the table and in --json. The row's blob count and blob size are now
unknown, and null in --json.

A directory with no manifest, as an interrupted backup leaves, still
shows 0: the orphan figures count its blobs as orphaned, so it
references none.

The constant holding the "unknown" text is renamed from countUnknown to
unknownText, since it now also stands for a size.

Model: opus-5-5

README.md

@@ -412,8 +412,7 @@ destination store.
 entirely. Remote storage is unaffected; the next backup will do a full
 scan and re-deduplicate against existing remote blobs, and the local
 index will re-bind to the currently configured storage destination.
-Use this after changing `storage_url`, or `s3.endpoint`, `s3.bucket` or
-`s3.prefix`, to a different destination.
+Use this after changing `storage_url` to a different destination.
 * `--force`: Skip confirmation prompt
 
 ---

TODO.md

@@ -22,68 +22,6 @@ the tag exists and is exercised; what is left is merging `next` to
 
 # Completed Steps
 
-- 2026-10-13: Recorded the size of the stored object for a blob a backup
-  found already stored
-  ([issue #300](https://git.eeqj.de/sneak/vaultik/issues/300)). When the
-  upload was skipped, the blob row and the manifest kept the size of the
-  local copy, which differs from the stored object's when
-  `compression_level` or the recipients changed since it was written.
-  `snapshot verify` then failed on a good snapshot, shallow and deep,
-  and the listed sizes were wrong. Marking the blob uploaded now records
-  the size of the object in storage.
-
-- 2026-10-12: Bound the local index to the destination an `s3.*` config
-  names ([issue #298](https://git.eeqj.de/sneak/vaultik/issues/298)).
-  Only `storage_url` was checked, so after `s3.bucket` or `s3.prefix`
-  changed, the next backup treated every chunk as already stored,
-  uploaded nothing, and completed a snapshot that could not be restored.
-  Without `storage_url`, the local index is now bound to `s3.endpoint`,
-  `s3.bucket` and `s3.prefix`, and a change is refused as a changed
-  `storage_url` is.
-
-- 2026-10-12: Rejected a retention duration longer than about 292 years
-  ([issue #289](https://git.eeqj.de/sneak/vaultik/issues/289)). The
-  number given to `--older-than` or `--keep-newer-than` was multiplied
-  by the length of its unit with no limit, so from `293y`, `15251w`,
-  `3559mo` or `106752d`, or a sum of parts such as `200y200y`, the
-  result wrapped to a negative duration. The cutoff then lay in the
-  future, every snapshot of the name was deleted, the one just made
-  included, and the command exited 0. Such a duration is now rejected
-  as invalid and nothing is deleted.
-
-- 2026-10-08: Made a backup cancelled under `--skip-errors` stop at once
-  ([issue #286](https://git.eeqj.de/sneak/vaultik/issues/286)). After
-  Ctrl-C or SIGTERM, phase 2 treated the cancellation error from each
-  remaining file like an unreadable file: it opened the file, printed an
-  error line for it, counted it as failed and went on to the next one.
-  The run now stops at the first file after the cancellation, and no
-  file is counted as failed because of it.
-
-- 2026-10-08: Made `remote nuke` delete the `.partial` files that
-  uploads cut off part-way leave on the destination store
-  ([issue #281](https://git.eeqj.de/sneak/vaultik/issues/281)). The
-  file and rclone listings skip such a file, so the command left it in
-  place and still reported the store empty. It now removes them under
-  `metadata/` and `blobs/` as its last step.
-
-- 2026-10-08: Made a local index error while recording a directory or
-  symlink stop a backup under `--skip-errors`
-  ([issue #284](https://git.eeqj.de/sneak/vaultik/issues/284)). Phase 2
-  only records such an entry in the local index, since it has no data
-  to open or read, but an error doing so was skipped like an unreadable
-  file. The snapshot then completed without the entry, and a restore
-  did not recreate it. The error now stops the backup, as it does
-  without the flag.
-
-- 2026-10-08: Counted a file that a backup could not store as failed
-  ([issue #280](https://git.eeqj.de/sneak/vaultik/issues/280)). A file
-  that phase 1 counted and phase 2 could not open, because it was
-  unreadable under `--skip-errors` or removed in between, was reported
-  in the summary as unchanged with its bytes as backed up, and the
-  `snapshots` row's `file_count` and `total_size` included it. The
-  summary now counts it as failed, and its data total and the row leave
-  it out.
-
 - 2026-10-08: Made `remote info` report a snapshot's blob count and
   blob size as unknown when its manifest cannot be read
   ([issue #272](https://git.eeqj.de/sneak/vaultik/issues/272)). The
@@ -93,17 +31,6 @@ the tag exists and is exercised; what is left is merging `next` to
   manifest still shows 0, since the orphan figures count its blobs as
   orphaned.
 
-- 2026-10-08: Kept the progress line of a snapshot with more than one
-  path within 100%
-  ([issue #271](https://git.eeqj.de/sneak/vaultik/issues/271)). The
-  bytes and files processed counted every path, but the totals they
-  were divided by held only the current path's, so a second path
-  smaller than the first showed more than 100% and an ETA of `unknown`.
-  The totals now add up over the paths scanned so far. The rate behind
-  the ETA restarts when each path's scan phase ends, so that phase does
-  not lower the rate while the path is processed; until then the rate
-  is the previous path's.
-
 - 2026-10-08: Made a second `snapshot create` of one name succeed when
   it starts in the same second as the first
   ([issue #270](https://git.eeqj.de/sneak/vaultik/issues/270)). The

docs/DATAMODEL.md

@@ -90,7 +90,7 @@ Stores information about packed, compressed, and encrypted blob files.
 - `created_ts` (INTEGER NOT NULL) - Creation timestamp
 - `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
 - `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression
-- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption. Once uploaded, the size of the object in storage: when the blob was already stored, under another compression level or other recipients, this differs from the size of the local copy
+- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption
 - `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded)
 
 **Purpose:** Tracks blob lifecycle from creation through upload. The UUID primary key allows immediate association of chunks with blobs.
@@ -183,7 +183,7 @@ Tracks blob upload metrics.
    - `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after packing
 
 4. **Upload**
-   - `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as uploaded, with the size of the object in storage (the upload is skipped when the blob is already stored)
+   - `UPDATE blobs SET uploaded_ts = ?` - Mark blob as uploaded
    - `INSERT INTO uploads` - Record upload metrics with snapshot_id
    - `INSERT INTO snapshot_blobs` - Associate blob with snapshot
 

internal/database/blobs.go

@@ -159,16 +159,13 @@ func (r *BlobRepository) UpdateFinished(
 	return nil
 }
 
-// UpdateUploaded marks a blob as uploaded and records the size of the
-// stored object as its compressed size. That size can differ from the
-// local copy's when an identical blob was already stored with another
-// compression level or other recipients.
+// UpdateUploaded marks a blob as uploaded
 func (r *BlobRepository) UpdateUploaded(
-	ctx context.Context, tx *sql.Tx, id string, storedSize int64,
+	ctx context.Context, tx *sql.Tx, id string,
 ) error {
 	query := `
-		UPDATE blobs
-		SET uploaded_ts = ?, compressed_size = ?
+		UPDATE blobs 
+		SET uploaded_ts = ?
 		WHERE id = ?
 	`
 
@@ -176,9 +173,9 @@ func (r *BlobRepository) UpdateUploaded(
 
 	var err error
 	if tx != nil {
-		_, err = tx.ExecContext(ctx, query, now, storedSize, id)
+		_, err = tx.ExecContext(ctx, query, now, id)
 	} else {
-		_, err = r.db.ExecWithLog(ctx, query, now, storedSize, id)
+		_, err = r.db.ExecWithLog(ctx, query, now, id)
 	}
 
 	if err != nil {

internal/database/blobs_test.go

@@ -122,8 +122,8 @@ func TestBlobRepositoryUpdates(t *testing.T) {
 		t.Errorf("expected compressed size 500, got %d", updated.CompressedSize)
 	}
 
-	// Test UpdateUploaded, with a stored object larger than the local copy
-	err = repo.UpdateUploaded(ctx, nil, blob.ID.String(), 600)
+	// Test UpdateUploaded
+	err = repo.UpdateUploaded(ctx, nil, blob.ID.String())
 	if err != nil {
 		t.Fatalf("failed to update blob as uploaded: %v", err)
 	}
@@ -141,10 +141,6 @@ func TestBlobRepositoryUpdates(t *testing.T) {
 	if uploaded.UploadedTS.Before(now.Add(-1 * time.Second)) {
 		t.Error("uploaded timestamp should be around test time")
 	}
-
-	if uploaded.CompressedSize != 600 {
-		t.Errorf("expected the stored size 600, got %d", uploaded.CompressedSize)
-	}
 }
 
 func TestBlobRepositoryDuplicate(t *testing.T) {

internal/snapshot/progress.go

@@ -56,27 +56,23 @@ const (
 
 // ProgressStats holds atomic counters for progress tracking
 type ProgressStats struct {
-	FilesScanned   atomic.Int64 // Total files seen during scan (includes skipped)
-	FilesProcessed atomic.Int64 // Files actually processed in phase 2
-	FilesSkipped   atomic.Int64 // Files skipped due to no changes
-	BytesScanned   atomic.Int64 // Bytes from new/changed files only
-	BytesSkipped   atomic.Int64 // Bytes from unchanged files
-	BytesProcessed atomic.Int64 // Actual bytes processed (for ETA calculation)
-	ChunksCreated  atomic.Int64
-	BlobsCreated   atomic.Int64
-	BlobsUploaded  atomic.Int64
-	BytesUploaded  atomic.Int64
-	CurrentFile    atomic.Value // stores string
-	TotalSize      atomic.Int64 // Size to process in the paths scanned so far
-	TotalFiles     atomic.Int64 // Files to process in the paths scanned so far
-	StartTime      time.Time
-	mu             sync.RWMutex
-	lastDetailTime time.Time
-
-	// Guarded by mu: when the last scan phase ended, and
-	// BytesProcessed at that moment.
-	processStartTime  time.Time
-	processStartBytes int64
+	FilesScanned     atomic.Int64 // Total files seen during scan (includes skipped)
+	FilesProcessed   atomic.Int64 // Files actually processed in phase 2
+	FilesSkipped     atomic.Int64 // Files skipped due to no changes
+	BytesScanned     atomic.Int64 // Bytes from new/changed files only
+	BytesSkipped     atomic.Int64 // Bytes from unchanged files
+	BytesProcessed   atomic.Int64 // Actual bytes processed (for ETA calculation)
+	ChunksCreated    atomic.Int64
+	BlobsCreated     atomic.Int64
+	BlobsUploaded    atomic.Int64
+	BytesUploaded    atomic.Int64
+	CurrentFile      atomic.Value // stores string
+	TotalSize        atomic.Int64 // Total size to process (set after scan phase)
+	TotalFiles       atomic.Int64 // Total files to process in phase 2
+	ProcessStartTime atomic.Value // stores time.Time when processing starts
+	StartTime        time.Time
+	mu               sync.RWMutex
+	lastDetailTime   time.Time
 
 	// Upload tracking
 	CurrentUpload    atomic.Value // stores *UploadInfo
@@ -152,36 +148,10 @@ func (pr *ProgressReporter) GetStats() *ProgressStats {
 	return pr.stats
 }
 
-// AddTotalSize adds the size one path of the snapshot has to process to
-// the total, once that path's scan phase is done, and starts measuring
-// the processing rate again. The processed counts run across every path,
-// so the total does too. Restarting the rate keeps a path's scan phase,
-// which processes nothing, out of the rate while that path is processed.
-// During a later path's scan phase the rate is still the previous
-// path's, and falls as that scan goes on.
-func (pr *ProgressReporter) AddTotalSize(size int64) {
-	pr.stats.TotalSize.Add(size)
-
-	pr.stats.mu.Lock()
-	defer pr.stats.mu.Unlock()
-
-	pr.stats.processStartTime = time.Now().UTC()
-	pr.stats.processStartBytes = pr.stats.BytesProcessed.Load()
-}
-
-// processRate returns the bytes processed per second since the last scan
-// phase ended, or 0 before the first path's scan phase is done.
-func (s *ProgressStats) processRate() float64 {
-	s.mu.RLock()
-	defer s.mu.RUnlock()
-
-	if s.processStartTime.IsZero() {
-		return 0
-	}
-
-	processed := s.BytesProcessed.Load() - s.processStartBytes
-
-	return float64(processed) / time.Since(s.processStartTime).Seconds()
+// SetTotalSize sets the total size to process (after scan phase)
+func (pr *ProgressReporter) SetTotalSize(size int64) {
+	pr.stats.TotalSize.Store(size)
+	pr.stats.ProcessStartTime.Store(time.Now().UTC())
 }
 
 // Helper functions
@@ -387,12 +357,19 @@ func (pr *ProgressReporter) printSummaryStatus() {
 	// Calculate ETA if we have total size and are processing
 	etaStr := ""
 
-	processRate := pr.stats.processRate()
-	if totalSize > 0 && processRate > 0 {
-		remainingBytes := totalSize - bytesProcessed
-		remainingSeconds := float64(remainingBytes) / processRate
-		eta := time.Duration(remainingSeconds * float64(time.Second))
-		etaStr = " | ETA: " + formatDuration(eta)
+	if totalSize > 0 && bytesProcessed > 0 {
+		processStart, ok := pr.stats.ProcessStartTime.Load().(time.Time)
+		if ok && !processStart.IsZero() {
+			processElapsed := time.Since(processStart)
+
+			rate := float64(bytesProcessed) / processElapsed.Seconds()
+			if rate > 0 {
+				remainingBytes := totalSize - bytesProcessed
+				remainingSeconds := float64(remainingBytes) / rate
+				eta := time.Duration(remainingSeconds * float64(time.Second))
+				etaStr = " | ETA: " + formatDuration(eta)
+			}
+		}
 	}
 
 	rate := float64(bytesScanned+bytesSkipped) / elapsed.Seconds()
@@ -444,18 +421,25 @@ func (pr *ProgressReporter) printDetailedStatus() {
 	log.Info("Elapsed time", "duration", formatDuration(elapsed))
 
 	// Calculate and show ETA if we have data
-	processRate := pr.stats.processRate()
-	if totalSize > 0 && processRate > 0 {
-		remainingBytes := totalSize - bytesProcessed
-		remainingSeconds := float64(remainingBytes) / processRate
-		eta := time.Duration(remainingSeconds * float64(time.Second))
-		percentComplete := float64(bytesProcessed) / float64(totalSize) * percentScale
-		log.Info("Overall progress",
-			"percent", fmt.Sprintf("%.1f%%", percentComplete),
-			"processed", humanize.Bytes(safeUint64(bytesProcessed)),
-			"total", humanize.Bytes(safeUint64(totalSize)),
-			"rate", humanize.Bytes(uint64(processRate))+"/s",
-			"eta", formatDuration(eta))
+	if totalSize > 0 && bytesProcessed > 0 {
+		processStart, ok := pr.stats.ProcessStartTime.Load().(time.Time)
+		if ok && !processStart.IsZero() {
+			processElapsed := time.Since(processStart)
+
+			processRate := float64(bytesProcessed) / processElapsed.Seconds()
+			if processRate > 0 {
+				remainingBytes := totalSize - bytesProcessed
+				remainingSeconds := float64(remainingBytes) / processRate
+				eta := time.Duration(remainingSeconds * float64(time.Second))
+				percentComplete := float64(bytesProcessed) / float64(totalSize) * percentScale
+				log.Info("Overall progress",
+					"percent", fmt.Sprintf("%.1f%%", percentComplete),
+					"processed", humanize.Bytes(safeUint64(bytesProcessed)),
+					"total", humanize.Bytes(safeUint64(totalSize)),
+					"rate", humanize.Bytes(uint64(processRate))+"/s",
+					"eta", formatDuration(eta))
+			}
+		}
 	}
 
 	log.Info("Files processed",

internal/snapshot/progress_rate_test.go

@@ -1,54 +0,0 @@
-//nolint:testpackage // exercises the unexported processRate helper
-package snapshot
-
-import (
-	"testing"
-	"time"
-)
-
-// TestProcessRateNotLoweredByLaterScanPhase feeds the reporter two paths
-// that each take 10 seconds to process, the second after a 30-second scan
-// phase. That scan phase processes nothing, so the rate while the second
-// path is processed must be that path's own.
-func TestProcessRateNotLoweredByLaterScanPhase(t *testing.T) {
-	t.Parallel()
-
-	const (
-		pathSize       = 1000
-		processingTime = 10 * time.Second
-		scanTime       = 30 * time.Second
-	)
-
-	// Never started, but Stop releases its tickers and signal handler.
-	progress := NewProgressReporter()
-	defer progress.Stop()
-
-	stats := progress.GetStats()
-
-	// Moves the processing start time back instead of sleeping.
-	elapse := func(d time.Duration) {
-		stats.mu.Lock()
-		defer stats.mu.Unlock()
-
-		stats.processStartTime = stats.processStartTime.Add(-d)
-	}
-
-	progress.AddTotalSize(pathSize)
-	stats.BytesProcessed.Add(pathSize)
-	elapse(processingTime)
-
-	elapse(scanTime)
-	progress.AddTotalSize(pathSize)
-	stats.BytesProcessed.Add(pathSize)
-	elapse(processingTime)
-
-	want := pathSize / processingTime.Seconds()
-	got := stats.processRate()
-
-	// The test's own run time adds to the elapsed time, so got is a hair
-	// under want.
-	if got > want || got < want*0.99 {
-		t.Errorf("rate while the second path is processed is %.1f bytes/s, "+
-			"want %.1f", got, want)
-	}
-}

internal/snapshot/progress_test.go

@@ -1,94 +0,0 @@
-package snapshot_test
-
-import (
-	"context"
-	"path/filepath"
-	"strings"
-	"testing"
-
-	"github.com/spf13/afero"
-	"sneak.berlin/go/vaultik/internal/database"
-	"sneak.berlin/go/vaultik/internal/snapshot"
-)
-
-// TestProgressPercentWithSecondPathSmaller backs up two paths with one
-// scanner, as a snapshot with two paths does, the second path smaller
-// than the first. The progress line divides the bytes processed by the
-// total size, so both must count both paths to stay within 100%.
-func TestProgressPercentWithSecondPathSmaller(t *testing.T) {
-	t.Parallel()
-
-	fs := afero.NewMemMapFs()
-	files := map[string]string{
-		"/large/one.txt":   strings.Repeat("1", 4000),
-		"/large/two.txt":   strings.Repeat("2", 4000),
-		"/small/three.txt": strings.Repeat("3", 1000),
-	}
-
-	for path, content := range files {
-		err := fs.MkdirAll(filepath.Dir(path), 0755)
-		if err != nil {
-			t.Fatalf("mkdir: %v", err)
-		}
-
-		err = afero.WriteFile(fs, path, []byte(content), 0644)
-		if err != nil {
-			t.Fatalf("write %s: %v", path, err)
-		}
-	}
-
-	db, err := database.NewTestDB()
-	if err != nil {
-		t.Fatalf("create test db: %v", err)
-	}
-
-	t.Cleanup(func() {
-		cerr := db.Close()
-		if cerr != nil {
-			t.Errorf("close db: %v", cerr)
-		}
-	})
-
-	repos := database.NewRepositories(db)
-
-	scanner := snapshot.NewScanner(snapshot.ScannerConfig{
-		FS:               fs,
-		ChunkSize:        int64(1024 * 16),
-		Repositories:     repos,
-		MaxBlobSize:      int64(1024 * 1024),
-		CompressionLevel: 3,
-		AgeRecipients:    []string{testAgePublicKey},
-		EnableProgress:   true,
-	})
-
-	// Never started, but Stop releases its tickers and signal handler.
-	progress := scanner.GetProgress()
-	defer progress.Stop()
-
-	ctx := context.Background()
-	snapshotID := "test-snapshot-progress"
-	createTestSnapshotRecord(ctx, t, repos, snapshotID)
-
-	for _, path := range []string{"/large", "/small"} {
-		_, err := scanner.Scan(ctx, path, snapshotID)
-		if err != nil {
-			t.Fatalf("scanning %s: %v", path, err)
-		}
-	}
-
-	stats := progress.GetStats()
-
-	// Directories count toward the total size but produce no chunks, so
-	// the percentage ends just under 100%.
-	percent := float64(stats.BytesProcessed.Load()) /
-		float64(stats.TotalSize.Load()) * 100
-	if percent > 100 {
-		t.Errorf("progress after both paths is %.1f%%, want at most 100%%",
-			percent)
-	}
-
-	if stats.FilesProcessed.Load() != stats.TotalFiles.Load() {
-		t.Errorf("progress after both paths is %d of %d files, want all",
-			stats.FilesProcessed.Load(), stats.TotalFiles.Load())
-	}
-}

internal/snapshot/scanner.go

@@ -133,13 +133,10 @@ type ScannerConfig struct {
 
 // ScanResult contains the results of a scan operation. Files and bytes
 // are counted per file: BytesScanned is the size of the new and changed
-// files, BytesSkipped that of the unchanged ones. FilesFailed counts the
-// new and changed files that phase 2 could not store; FilesScanned
-// includes them and BytesScanned does not.
+// files, BytesSkipped that of the unchanged ones.
 type ScanResult struct {
 	FilesScanned   int
 	FilesSkipped   int
-	FilesFailed    int
 	FilesDeleted   int
 	BytesScanned   int64
 	BytesSkipped   int64
@@ -410,8 +407,8 @@ func (s *Scanner) summarizeScanPhase(
 	}
 
 	if s.progress != nil {
-		s.progress.AddTotalSize(totalSizeToProcess)
-		s.progress.GetStats().TotalFiles.Add(int64(len(filesToProcess)))
+		s.progress.SetTotalSize(totalSizeToProcess)
+		s.progress.GetStats().TotalFiles.Store(int64(len(filesToProcess)))
 	}
 
 	log.Info("Phase 1 complete",
@@ -1312,13 +1309,6 @@ func (s *Scanner) processPhase(
 
 	// Process each file
 	for _, fileToProcess := range filesToProcess {
-		// Check context cancellation
-		select {
-		case <-ctx.Done():
-			return ctx.Err()
-		default:
-		}
-
 		// Update progress
 		if s.progress != nil {
 			s.progress.GetStats().CurrentFile.Store(fileToProcess.Path)
@@ -1355,32 +1345,13 @@ func (s *Scanner) processPhase(
 	return s.finalizeProcessPhase(ctx, result)
 }
 
-// processFileWithErrorHandling records a directory or symlink, or wraps
-// processFileStreaming for a regular file with error recovery for deleted
-// files and skip-errors mode. Returns (skipped, error).
+// processFileWithErrorHandling wraps processFileStreaming with error recovery for
+// deleted files and skip-errors mode. Returns (skipped, error).
 func (s *Scanner) processFileWithErrorHandling(
 	ctx context.Context, fileToProcess *FileToProcess, result *ScanResult,
 ) (bool, error) {
-	// A directory or symlink has no data to open or read; it is only
-	// recorded in the local index. An error recording it stops the run
-	// even under --skip-errors, or the snapshot would complete without it.
-	mode := os.FileMode(fileToProcess.File.Mode)
-	if mode&os.ModeSymlink != 0 || mode.IsDir() {
-		err := s.recordNonRegularFile(ctx, fileToProcess)
-		if err != nil {
-			return false, fmt.Errorf("processing file %s: %w", fileToProcess.Path, err)
-		}
-
-		return false, nil
-	}
-
 	err := s.processFileStreaming(ctx, fileToProcess, result)
 	if err != nil {
-		// A cancelled run stops here even under --skip-errors, rather than
-		// counting this file as failed and going on to the next one.
-		if ctx.Err() != nil {
-			return false, fmt.Errorf("processing file %s: %w", fileToProcess.Path, err)
-		}
 		// A packer/database/encryption/upload failure means the chunk's data
 		// may not have been stored. Skipping the file would let the snapshot
 		// record a file whose chunk is in no blob and cannot be restored, so
@@ -1394,7 +1365,7 @@ func (s *Scanner) processFileWithErrorHandling(
 			log.Warn("File was deleted during backup, skipping",
 				"path", fileToProcess.Path)
 
-			countFailedFile(fileToProcess, result)
+			result.FilesSkipped++
 
 			return true, nil
 		}
@@ -1405,7 +1376,7 @@ func (s *Scanner) processFileWithErrorHandling(
 			s.ui.Errorf("Failed to process %s: %v. Skipping (--skip-errors).",
 				s.ui.Path(fileToProcess.Path), err)
 
-			countFailedFile(fileToProcess, result)
+			result.FilesSkipped++
 
 			return true, nil
 		}
@@ -1416,13 +1387,6 @@ func (s *Scanner) processFileWithErrorHandling(
 	return false, nil
 }
 
-// countFailedFile counts a file that phase 2 could not store as failed
-// and takes its size back out of BytesScanned, where phase 1 put it.
-func countFailedFile(fileToProcess *FileToProcess, result *ScanResult) {
-	result.FilesFailed++
-	result.BytesScanned -= fileToProcess.FileInfo.Size()
-}
-
 // printProcessingProgress prints a periodic progress line during the process phase,
 // showing files processed, bytes transferred, throughput, and ETA
 func (s *Scanner) printProcessingProgress(
@@ -1509,8 +1473,7 @@ func (s *Scanner) finalizeProcessPhase(ctx context.Context, result *ScanResult)
 			// snapshot references has uploaded_ts set, so deduplication and
 			// interrupted-run repair treat these blobs as trustworthy.
 			err = s.repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
-				err := s.repos.Blobs.UpdateUploaded(
-					ctx, tx, b.ID, b.Compressed)
+				err := s.repos.Blobs.UpdateUploaded(ctx, tx, b.ID)
 				if err != nil {
 					return fmt.Errorf("marking blob uploaded: %w", err)
 				}
@@ -1546,7 +1509,7 @@ func (s *Scanner) handleBlobReady(
 	blobPath := fmt.Sprintf("blobs/%s/%s/%s",
 		finishedBlob.Hash[:2], finishedBlob.Hash[2:4], finishedBlob.Hash)
 
-	storedSize, blobExists, err := s.uploadBlobIfNeeded(
+	blobExists, err := s.uploadBlobIfNeeded(
 		ctx, blobPath, blobWithReader, startTime, result)
 	if err != nil {
 		s.cleanupBlobTempFile(blobWithReader)
@@ -1554,8 +1517,7 @@ func (s *Scanner) handleBlobReady(
 		return fmt.Errorf("uploading blob %s: %w", finishedBlob.Hash, err)
 	}
 
-	err = s.recordBlobMetadata(
-		ctx, finishedBlob, storedSize, blobExists, startTime)
+	err = s.recordBlobMetadata(ctx, finishedBlob, blobExists, startTime)
 	if err != nil {
 		s.cleanupBlobTempFile(blobWithReader)
 
@@ -1577,22 +1539,20 @@ func (s *Scanner) handleBlobReady(
 }
 
 // uploadBlobIfNeeded uploads the blob to storage if it doesn't already
-// exist. It returns the size of the object in storage, which for an
-// existing object can differ from the local copy's, and whether it
-// already existed.
+// exist, returns whether it existed
 func (s *Scanner) uploadBlobIfNeeded(
 	ctx context.Context,
 	blobPath string,
 	blobWithReader *blob.WithReader,
 	startTime time.Time,
 	result *ScanResult,
-) (int64, bool, error) {
+) (bool, error) {
 	finishedBlob := blobWithReader.FinishedBlob
 
 	// Check if blob already exists (deduplication after restart)
 	destination := s.storage.Info().Location
 
-	stored, err := s.storage.Stat(ctx, blobPath)
+	_, err := s.storage.Stat(ctx, blobPath)
 	if err == nil {
 		log.Info("Blob already exists in storage, skipping upload",
 			"hash", finishedBlob.Hash,
@@ -1601,7 +1561,7 @@ func (s *Scanner) uploadBlobIfNeeded(
 			s.ui.Hex(finishedBlob.Hash), s.ui.Size(finishedBlob.Compressed),
 			s.ui.Path(destination))
 
-		return stored.Size, true, nil
+		return true, nil
 	}
 
 	s.ui.Beginf("Uploading blob %s (%s) to %s.",
@@ -1615,7 +1575,7 @@ func (s *Scanner) uploadBlobIfNeeded(
 	if err != nil {
 		log.Error("Failed to upload blob", "hash", finishedBlob.Hash, "error", err)
 
-		return 0, false, fmt.Errorf("uploading blob to storage: %w", err)
+		return false, fmt.Errorf("uploading blob to storage: %w", err)
 	}
 
 	uploadDuration := time.Since(startTime)
@@ -1646,7 +1606,7 @@ func (s *Scanner) uploadBlobIfNeeded(
 		stats.BytesUploaded.Add(finishedBlob.Compressed)
 	}
 
-	return finishedBlob.Compressed, false, nil
+	return false, nil
 }
 
 // makeUploadProgressCallback creates a progress callback for blob uploads.
@@ -1716,7 +1676,6 @@ func (s *Scanner) makeUploadProgressCallback(
 func (s *Scanner) recordBlobMetadata(
 	ctx context.Context,
 	finishedBlob *blob.FinishedBlob,
-	storedSize int64,
 	blobExists bool,
 	startTime time.Time,
 ) error {
@@ -1728,8 +1687,7 @@ func (s *Scanner) recordBlobMetadata(
 	uploadDuration := time.Since(startTime)
 
 	return s.repos.WithTx(ctx, func(txCtx context.Context, tx *sql.Tx) error {
-		err := s.repos.Blobs.UpdateUploaded(
-			txCtx, tx, finishedBlob.ID, storedSize)
+		err := s.repos.Blobs.UpdateUploaded(txCtx, tx, finishedBlob.ID)
 		if err != nil {
 			return fmt.Errorf("updating blob upload timestamp: %w", err)
 		}
@@ -1797,11 +1755,16 @@ func (e *packerError) Error() string { return e.err.Error() }
 
 func (e *packerError) Unwrap() error { return e.err }
 
-// processFileStreaming processes a regular file by streaming chunks directly
-// to the packer
+// processFileStreaming processes a file by streaming chunks directly to the packer
 func (s *Scanner) processFileStreaming(
 	ctx context.Context, fileToProcess *FileToProcess, result *ScanResult,
 ) error {
+	// Symlinks and directories have no data to chunk — just record them in the DB.
+	mode := os.FileMode(fileToProcess.File.Mode)
+	if mode&os.ModeSymlink != 0 || mode.IsDir() {
+		return s.recordNonRegularFile(ctx, fileToProcess)
+	}
+
 	file, err := s.fs.Open(fileToProcess.Path)
 	if err != nil {
 		return fmt.Errorf("opening file: %w", wrapPermissionError(fileToProcess.Path, err))

internal/snapshot/skip_errors_test.go

@@ -82,32 +82,6 @@ func (f *readFailFs) Open(name string) (afero.File, error) {
 	return file, nil
 }
 
-// cancelOnOpenFs cancels the run when the scanner opens the target file to
-// back it up, as Ctrl-C partway through processing would, and records each
-// file opened after that.
-type cancelOnOpenFs struct {
-	afero.Fs
-
-	target            string
-	cancel            context.CancelFunc
-	cancelled         bool
-	openedAfterCancel []string
-}
-
-//nolint:ireturn // afero.Fs.Open is defined to return the interface.
-func (f *cancelOnOpenFs) Open(name string) (afero.File, error) {
-	if f.cancelled {
-		f.openedAfterCancel = append(f.openedAfterCancel, name)
-	}
-
-	if name == f.target {
-		f.cancel()
-		f.cancelled = true
-	}
-
-	return f.Fs.Open(name)
-}
-
 // linkRemovedAfterLstatFs is the real filesystem, except that the symlink at
 // target is removed right after the walk lstats it, as happens when a link is
 // deleted during a backup. The scanner's readlink of it then fails.
@@ -154,16 +128,15 @@ func writeSkipErrorTestFile(t *testing.T, fs afero.Fs, path, content string) {
 	}
 }
 
-// runSkipErrorScan scans source on fs under ctx with the given skip-errors
-// setting, printing user-facing messages to uiw (nil discards them), and
-// returns the repositories (for inspection) and the scan error.
+// runSkipErrorScan scans source on fs with the given skip-errors setting,
+// printing user-facing messages to uiw (nil discards them), and returns the
+// repositories (for inspection) and the scan error.
 func runSkipErrorScan(
-	ctx context.Context, t *testing.T, fs afero.Fs, source string,
-	skipErrors bool, uiw *ui.Writer,
+	t *testing.T, fs afero.Fs, source string, skipErrors bool, uiw *ui.Writer,
 ) (*database.Repositories, error) {
 	t.Helper()
 
-	db, err := database.New(ctx, ":memory:")
+	db, err := database.NewTestDB()
 	if err != nil {
 		t.Fatalf("create test db: %v", err)
 	}
@@ -188,6 +161,7 @@ func runSkipErrorScan(
 		SkipErrors:       skipErrors,
 	})
 
+	ctx := context.Background()
 	snapshotID := "test-snapshot-skip-errors"
 	createTestSnapshotRecord(ctx, t, repos, snapshotID)
 
@@ -211,7 +185,7 @@ func TestScannerPackingFailureAbortsUnderSkipErrors(t *testing.T) {
 	writeSkipErrorTestFile(t, fs, "/source/file1.txt", "first file content")
 	writeSkipErrorTestFile(t, fs, "/source/file2.txt", "second file content")
 
-	repos, err := runSkipErrorScan(context.Background(), t, fs, "/source", true, nil)
+	repos, err := runSkipErrorScan(t, fs, "/source", true, nil)
 	if err == nil {
 		t.Fatal("expected scan to abort on the packer error, got nil")
 	}
@@ -238,7 +212,7 @@ func TestScannerReadErrorAbortsWithoutSkipErrors(t *testing.T) {
 	fs := &readFailFs{Fs: afero.NewMemMapFs(), target: target}
 	writeSkipErrorTestFile(t, fs, target, "content that cannot be read")
 
-	_, err := runSkipErrorScan(context.Background(), t, fs, "/source", false, nil)
+	_, err := runSkipErrorScan(t, fs, "/source", false, nil)
 	if err == nil {
 		t.Fatal("expected scan to fail on the read error, got nil")
 	}
@@ -254,7 +228,7 @@ func TestScannerReadErrorSkippedWithSkipErrors(t *testing.T) {
 	fs := &readFailFs{Fs: afero.NewMemMapFs(), target: target}
 	writeSkipErrorTestFile(t, fs, target, "content that cannot be read")
 
-	repos, err := runSkipErrorScan(context.Background(), t, fs, "/source", true, nil)
+	repos, err := runSkipErrorScan(t, fs, "/source", true, nil)
 	if err != nil {
 		t.Fatalf("expected scan to complete with --skip-errors, got %v", err)
 	}
@@ -293,7 +267,7 @@ func TestScannerUnreadableSymlinkAbortsWithoutSkipErrors(t *testing.T) {
 	sourceDir, linkPath := writeSymlinkSource(t)
 	fs := &linkRemovedAfterLstatFs{t: t, target: linkPath}
 
-	_, err := runSkipErrorScan(context.Background(), t, fs, sourceDir, false, nil)
+	_, err := runSkipErrorScan(t, fs, sourceDir, false, nil)
 	if !errors.Is(err, os.ErrNotExist) {
 		t.Fatalf("expected scan to fail on the removed symlink, got %v", err)
 	}
@@ -309,7 +283,7 @@ func TestScannerUnreadableSymlinkSkippedWithSkipErrors(t *testing.T) {
 	fs := &linkRemovedAfterLstatFs{t: t, target: linkPath}
 	uiw := ui.NewWithColor(io.Discard, false)
 
-	repos, err := runSkipErrorScan(context.Background(), t, fs, sourceDir, true, uiw)
+	repos, err := runSkipErrorScan(t, fs, sourceDir, true, uiw)
 	if err != nil {
 		t.Fatalf("expected scan to complete with --skip-errors, got %v", err)
 	}
@@ -328,56 +302,3 @@ func TestScannerUnreadableSymlinkSkippedWithSkipErrors(t *testing.T) {
 		t.Fatalf("expected %s not to be recorded", linkPath)
 	}
 }
-
-// TestScannerCancelStopsSkipErrorsRun checks that a --skip-errors backup
-// cancelled partway through processing returns the cancellation error,
-// opens no further file, and reports no file as failed.
-func TestScannerCancelStopsSkipErrorsRun(t *testing.T) {
-	t.Parallel()
-
-	tests := []struct {
-		name          string
-		targetContent string
-	}{
-		// The cancellation lands while the target is being read.
-		{name: "while reading a file", targetContent: "first file content"},
-		// An empty target has no chunks, so the cancellation goes unnoticed
-		// until the run moves on to the next file.
-		{name: "between files", targetContent: ""},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			t.Parallel()
-
-			const target = "/source/a.txt"
-
-			ctx, cancel := context.WithCancel(context.Background())
-			defer cancel()
-
-			fs := &cancelOnOpenFs{
-				Fs: afero.NewMemMapFs(), target: target, cancel: cancel,
-			}
-			writeSkipErrorTestFile(t, fs, target, tt.targetContent)
-			writeSkipErrorTestFile(t, fs, "/source/b.txt", "second file content")
-			writeSkipErrorTestFile(t, fs, "/source/c.txt", "third file content")
-
-			uiw := ui.NewWithColor(io.Discard, false)
-
-			_, err := runSkipErrorScan(ctx, t, fs, "/source", true, uiw)
-			if !errors.Is(err, context.Canceled) {
-				t.Fatalf("expected the cancellation error, got %v", err)
-			}
-
-			if len(fs.openedAfterCancel) != 0 {
-				t.Fatalf("expected no file opened after the cancellation, got %v",
-					fs.openedAfterCancel)
-			}
-
-			if uiw.ErrorCount() != 0 {
-				t.Fatalf("expected no file reported as failed, got %d error lines",
-					uiw.ErrorCount())
-			}
-		})
-	}
-}

internal/snapshot/snapshot.go

@@ -892,7 +892,7 @@ func (sm *SnapshotManager) getFileSize(path string) int64 {
 // BackupStats contains statistics from a backup operation
 type BackupStats struct {
 	FilesScanned  int
-	TotalSize     int64 // Total size of the files in the snapshot
+	TotalSize     int64 // Total size of all files examined
 	ChunksCreated int
 	BlobsCreated  int
 	BytesUploaded int64

internal/storage/faultstore/faultstore.go

@@ -171,11 +171,6 @@ func (f *Storer) ListStream(
 	return f.inner.ListStream(ctx, prefix)
 }
 
-// DeletePartialUploads delegates unchanged.
-func (f *Storer) DeletePartialUploads(ctx context.Context, prefix string) error {
-	return f.inner.DeletePartialUploads(ctx, prefix)
-}
-
 // Info delegates unchanged.
 func (f *Storer) Info() storage.Info {
 	return f.inner.Info()

internal/storage/file.go

@@ -242,42 +242,6 @@ func (f *FileStorer) ListStream(ctx context.Context, prefix string) <-chan Objec
 	return ch
 }
 
-// DeletePartialUploads removes every file under prefix whose name ends in
-// tempSuffix. A missing prefix has none to remove.
-func (f *FileStorer) DeletePartialUploads(ctx context.Context, prefix string) error {
-	basePath := f.fullPath(prefix)
-
-	exists, err := afero.Exists(f.fs, basePath)
-	if err != nil {
-		return fmt.Errorf("checking path: %w", err)
-	}
-
-	if !exists {
-		return nil
-	}
-
-	err = afero.Walk(f.fs, basePath, func(path string, info os.FileInfo, err error) error {
-		if err != nil {
-			return err
-		}
-
-		if ctx.Err() != nil {
-			return ctx.Err()
-		}
-
-		if info.IsDir() || !strings.HasSuffix(info.Name(), tempSuffix) {
-			return nil
-		}
-
-		return f.fs.Remove(path)
-	})
-	if err != nil {
-		return fmt.Errorf("walking directory: %w", err)
-	}
-
-	return nil
-}
-
 // Info returns human-readable storage location information.
 func (f *FileStorer) Info() Info {
 	return Info{

internal/storage/file_atomic_test.go

@@ -120,45 +120,3 @@ func TestFileStorer_ListSkipsPartialFiles(t *testing.T) {
 		t.Fatalf("ListStream should return only the real key, got %v", streamed)
 	}
 }
-
-// TestFileStorer_DeletePartialUploads checks that a leftover temp file is
-// removed and the object at the real key is kept.
-func TestFileStorer_DeletePartialUploads(t *testing.T) {
-	t.Parallel()
-
-	base := t.TempDir()
-
-	f, err := storage.NewFileStorer(base)
-	if err != nil {
-		t.Fatalf("NewFileStorer: %v", err)
-	}
-
-	ctx := context.Background()
-
-	err = f.Put(ctx, testBlobKey, strings.NewReader("blob-bytes"))
-	if err != nil {
-		t.Fatalf("Put: %v", err)
-	}
-
-	leftover := filepath.Join(base, testBlobKey+"-123456.partial")
-
-	err = os.WriteFile(leftover, []byte("half"), 0o600)
-	if err != nil {
-		t.Fatalf("writing leftover temp file: %v", err)
-	}
-
-	err = f.DeletePartialUploads(ctx, "blobs/")
-	if err != nil {
-		t.Fatalf("DeletePartialUploads: %v", err)
-	}
-
-	_, err = os.Stat(leftover)
-	if !os.IsNotExist(err) {
-		t.Errorf("leftover temp file was not removed: %v", err)
-	}
-
-	_, err = f.Stat(ctx, testBlobKey)
-	if err != nil {
-		t.Errorf("Stat of the real key: %v", err)
-	}
-}

internal/storage/rclone.go

@@ -213,31 +213,6 @@ func (r *RcloneStorer) ListStream(
 	return ch
 }
 
-// DeletePartialUploads removes every object under prefix whose name ends
-// in tempSuffix.
-func (r *RcloneStorer) DeletePartialUploads(ctx context.Context, prefix string) error {
-	var partial []fs.Object
-
-	err := operations.ListFn(ctx, r.fsys, func(obj fs.Object) {
-		key := obj.Remote()
-		if strings.HasPrefix(key, prefix) && strings.HasSuffix(key, tempSuffix) {
-			partial = append(partial, obj)
-		}
-	})
-	if err != nil {
-		return fmt.Errorf("listing objects: %w", err)
-	}
-
-	for _, obj := range partial {
-		err = obj.Remove(ctx)
-		if err != nil {
-			return fmt.Errorf("removing object: %w", err)
-		}
-	}
-
-	return nil
-}
-
 // Info returns human-readable storage location information.
 func (r *RcloneStorer) Info() Info {
 	location := r.remote

internal/storage/rclone_test.go

@@ -198,47 +198,6 @@ func TestRcloneStorerListSkipsPartialFiles(t *testing.T) {
 	}
 }
 
-// TestRcloneStorerDeletePartialUploads checks that a temporary file left
-// by a killed upload is removed and the object at the real key is kept.
-//
-//nolint:paralleltest // NewRcloneStorer installs the process-global rclone config
-func TestRcloneStorerDeletePartialUploads(t *testing.T) {
-	dir := t.TempDir()
-	ctx := context.Background()
-
-	s, err := storage.NewRcloneStorer(ctx, ":local", dir)
-	if err != nil {
-		t.Fatalf("NewRcloneStorer: %v", err)
-	}
-
-	err = s.Put(ctx, testBlobKey, strings.NewReader("blob-bytes"))
-	if err != nil {
-		t.Fatalf("Put: %v", err)
-	}
-
-	leftover := filepath.Join(dir, testBlobKey+"-123456.partial")
-
-	err = os.WriteFile(leftover, []byte("half"), 0o600)
-	if err != nil {
-		t.Fatalf("writing leftover temp file: %v", err)
-	}
-
-	err = s.DeletePartialUploads(ctx, "blobs/")
-	if err != nil {
-		t.Fatalf("DeletePartialUploads: %v", err)
-	}
-
-	_, err = os.Stat(leftover)
-	if !os.IsNotExist(err) {
-		t.Errorf("leftover temp file was not removed: %v", err)
-	}
-
-	_, err = s.Stat(ctx, testBlobKey)
-	if err != nil {
-		t.Errorf("Stat of the real key: %v", err)
-	}
-}
-
 // newRcloneStorerOnWrappedLocal registers name as rclone's local backend
 // wrapped by wrap, and builds an rclone backend on it rooted at a fresh
 // temp directory. wrap changes the features the local backend reports, so

internal/storage/s3.go

@@ -99,12 +99,6 @@ func (s *S3Storer) ListStream(ctx context.Context, prefix string) <-chan ObjectI
 	return ch
 }
 
-// DeletePartialUploads has nothing to remove: S3 shows an object only once
-// its upload has completed, so an upload cut off part-way leaves none.
-func (s *S3Storer) DeletePartialUploads(_ context.Context, _ string) error {
-	return nil
-}
-
 // Info returns human-readable storage location information.
 func (s *S3Storer) Info() Info {
 	return Info{

internal/storage/storer.go

@@ -71,12 +71,6 @@ type Storer interface {
 	// If an error occurs during listing, the final item will have Err set.
 	ListStream(ctx context.Context, prefix string) <-chan ObjectInfo
 
-	// DeletePartialUploads removes every object under prefix that an
-	// upload cut off part-way left under a temporary name ending in
-	// `.partial`. The file and rclone backends' List and ListStream skip
-	// such an object.
-	DeletePartialUploads(ctx context.Context, prefix string) error
-
 	// Info returns human-readable storage location information.
 	Info() Info
 }

internal/vaultik/destination_validation_test.go

@@ -257,10 +257,6 @@ func (s *stubLister) List(_ context.Context, _ string) ([]string, error) {
 	return nil, errStubUnused
 }
 
-func (s *stubLister) DeletePartialUploads(_ context.Context, _ string) error {
-	return errStubUnused
-}
-
 func (s *stubLister) Info() storage.Info {
 	return storage.Info{}
 }

internal/vaultik/helpers.go

@@ -3,7 +3,6 @@ package vaultik
 import (
 	"errors"
 	"fmt"
-	"math"
 	"regexp"
 	"strconv"
 	"strings"
@@ -48,10 +47,6 @@ const (
 	year  = 365 * day
 )
 
-// maxDuration is the largest value a time.Duration can hold, about 292
-// years.
-const maxDuration = time.Duration(math.MaxInt64)
-
 // A snapshot ID split on "_" has at least a hostname and a trailing
 // timestamp.
 const minSnapshotIDParts = 2
@@ -150,10 +145,10 @@ func parseSnapshotName(snapshotID, hostname string) string {
 // parseDuration parses a duration string with support for human-friendly units:
 // d/day/days, w/week/weeks, mo/month/months, y/year/years, plus standard Go
 // duration units. Following Go, m is minutes and mo is months. A bare number,
-// an unknown unit, a negative value, and a total longer than maxDuration are
-// all rejected. Outside the Go units the input must be whole numbers each
-// followed directly by a unit (2w3d), so 1.0y and 30 days are errors;
-// decimals work only in Go units (1.5h).
+// an unknown unit, and a negative value are all rejected. Outside the Go
+// units the input must be whole numbers each followed directly by a unit
+// (2w3d), so 1.0y and 30 days are errors; decimals work only in Go units
+// (1.5h).
 func parseDuration(s string) (time.Duration, error) {
 	if strings.HasPrefix(strings.TrimSpace(s), "-") {
 		return 0, errNegativeDuration
@@ -186,30 +181,19 @@ func parseDuration(s string) (time.Duration, error) {
 			return 0, fmt.Errorf("invalid number %q: %w", match[1], err)
 		}
 
-		var unitLength time.Duration
-
 		unit := strings.ToLower(match[2])
 		switch unit {
 		case "d", "day", "days":
-			unitLength = day
+			total += time.Duration(n) * day
 		case "w", "week", "weeks":
-			unitLength = week
+			total += time.Duration(n) * week
 		case "mo", "month", "months":
-			unitLength = month
+			total += time.Duration(n) * month
 		case "y", "year", "years":
-			unitLength = year
+			total += time.Duration(n) * year
 		default:
 			return 0, fmt.Errorf("%w %q", errUnknownTimeUnit, unit)
 		}
-
-		// Past maxDuration the sum wraps to a negative value, which would
-		// put the purge cutoff in the future.
-		if time.Duration(n) > (maxDuration-total)/unitLength {
-			return 0, fmt.Errorf("%w: %q is longer than about 292 years",
-				errInvalidDuration, s)
-		}
-
-		total += time.Duration(n) * unitLength
 	}
 
 	return total, nil

internal/vaultik/helpers_test.go

@@ -111,17 +111,6 @@ func TestParseDuration(t *testing.T) {
 		{"30 days", 0, true},
 		{"30 days ago", 0, true},
 		{"x7d", 0, true},
-		// The largest time.Duration is about 292 years. A larger value
-		// must be rejected, not wrapped to a negative one, which would
-		// put the cutoff in the future and select every snapshot.
-		{"292y", 292 * 365 * 24 * time.Hour, false},
-		{"293y", 0, true},
-		{"3559mo", 0, true},
-		{"15251w", 0, true},
-		{"106752d", 0, true},
-		{"2562048h", 0, true},
-		{"200y200y", 0, true},
-		{"292y6mo", 0, true},
 	}
 
 	for _, tt := range tests {

internal/vaultik/integration_test.go

@@ -155,12 +155,6 @@ func (m *MockStorer) ListStream(
 	return ch
 }
 
-// DeletePartialUploads has nothing to remove: Put stores each object
-// under its key at once.
-func (m *MockStorer) DeletePartialUploads(_ context.Context, _ string) error {
-	return nil
-}
-
 func (m *MockStorer) Info() storage.Info {
 	return storage.Info{
 		Type:     "mock",

internal/vaultik/nuke_remote_test.go

@@ -1,64 +0,0 @@
-package vaultik_test
-
-import (
-	"context"
-	"io/fs"
-	"os"
-	"path/filepath"
-	"testing"
-
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-	"sneak.berlin/go/vaultik/internal/log"
-	"sneak.berlin/go/vaultik/internal/snapshot"
-)
-
-// TestNukeRemoteLeavesNoFiles checks that remote nuke leaves no file
-// under a file:// destination, including the `.partial` files uploads
-// killed part-way leave next to a snapshot's metadata and next to where a
-// blob would have been.
-func TestNukeRemoteLeavesNoFiles(t *testing.T) {
-	log.Initialize(log.Config{})
-	t.Parallel()
-
-	ctx := context.Background()
-	storeDir := filepath.Join(t.TempDir(), "store")
-	v, repos, _ := backUpToFileDestination(ctx, t, storeDir)
-
-	snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
-	require.NoError(t, err)
-	require.Len(t, snapshots, 1)
-
-	snapshotKey := snapshot.RemoteSnapshotKey(snapshots[0].ID.String())
-	hash := testBlobHashA
-	leftovers := []string{
-		filepath.Join(storeDir, "metadata", snapshotKey,
-			"db.zst.age-123456.partial"),
-		filepath.Join(storeDir, "blobs", hash[:2], hash[2:4],
-			hash+"-123456.partial"),
-	}
-
-	for _, leftover := range leftovers {
-		require.NoError(t, os.MkdirAll(filepath.Dir(leftover), 0o750))
-		require.NoError(t, os.WriteFile(leftover, []byte("half an upload"), 0o600))
-	}
-
-	require.NoError(t, v.NukeRemote(true))
-
-	var files []string
-
-	err = filepath.WalkDir(storeDir,
-		func(path string, entry fs.DirEntry, err error) error {
-			if err != nil {
-				return err
-			}
-
-			if !entry.IsDir() {
-				files = append(files, path)
-			}
-
-			return nil
-		})
-	require.NoError(t, err)
-	assert.Empty(t, files)
-}

internal/vaultik/prune.go

@@ -24,9 +24,8 @@ var errNukeRequiresForce = errors.New(
 const metadataDirName = "metadata"
 
 // NukeRemote deletes every snapshot's metadata and every blob from remote
-// storage, along with any object an upload cut off part-way left under a
-// temporary `.partial` name. After this returns successfully the bucket
-// prefix is empty and the next backup starts from scratch.
+// storage. After this returns successfully the bucket prefix is empty and
+// the next backup starts from scratch.
 //
 // Refuses to run unless force is true. The caller is responsible for
 // confirming with the user.
@@ -49,15 +48,6 @@ func (v *Vaultik) NukeRemote(force bool) error {
 		return fmt.Errorf("pruning blobs: %w", err)
 	}
 
-	// The file and rclone listings skip `.partial` objects, so the two
-	// steps above never delete them.
-	for _, prefix := range []string{"metadata/", "blobs/"} {
-		err = v.Storage.DeletePartialUploads(v.ctx, prefix)
-		if err != nil {
-			return fmt.Errorf("deleting partial uploads: %w", err)
-		}
-	}
-
 	v.UI.Completef("Backup destination store is now empty.")
 
 	return nil

internal/vaultik/purge_per_name_test.go

@@ -267,29 +267,6 @@ func TestPurgeOlderThan_WithNameFilter(t *testing.T) {
 	assert.Contains(t, remaining, snapHomeT0)
 }
 
-func TestPurgeOlderThan_TooLargeDeletesNothing(t *testing.T) {
-	log.Initialize(log.Config{})
-	t.Parallel()
-
-	snapshotIDs := []string{snapHomeT0, snapHomeT1}
-
-	v := setupPurgeTest(t, snapHostname, snapshotIDs)
-
-	err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
-		OlderThan: "1000y",
-		Force:     true,
-	})
-	require.Error(t, err)
-
-	err = v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
-		OlderThan: "292y",
-		Force:     true,
-	})
-	require.NoError(t, err)
-
-	assert.ElementsMatch(t, snapshotIDs, listRemainingSnapshots(t, v))
-}
-
 func TestPurgeKeepLatest_ThreeNames(t *testing.T) {
 	log.Initialize(log.Config{})
 	t.Parallel()

internal/vaultik/remove_snapshot_test.go

@@ -125,12 +125,6 @@ func (s *testStorer) ListStream(
 	return ch
 }
 
-// DeletePartialUploads has nothing to remove: Put stores each object
-// under its key at once.
-func (s *testStorer) DeletePartialUploads(_ context.Context, _ string) error {
-	return nil
-}
-
 func (s *testStorer) Info() storage.Info {
 	return storage.Info{
 		Type:     testLabel,

internal/vaultik/reused_blob_verify_test.go

@@ -1,83 +0,0 @@
-package vaultik_test
-
-import (
-	"context"
-	"io"
-	"path/filepath"
-	"testing"
-
-	"filippo.io/age"
-	"github.com/spf13/afero"
-	"github.com/stretchr/testify/require"
-	"sneak.berlin/go/vaultik/internal/log"
-	"sneak.berlin/go/vaultik/internal/ui"
-	"sneak.berlin/go/vaultik/internal/vaultik"
-)
-
-// TestVerifyPassesWhenBackupReusesStoredBlob backs up a file, then backs it
-// up again from an empty local index, as after `database delete`, with an
-// extra age recipient. The second backup builds the same blob, finds it
-// already stored and skips the upload, but its own copy is larger, because
-// age writes a header entry for each recipient. The second snapshot must
-// record the size of the stored object, so shallow and deep verify pass.
-func TestVerifyPassesWhenBackupReusesStoredBlob(t *testing.T) {
-	log.Initialize(log.Config{})
-	t.Parallel()
-
-	fs := afero.NewOsFs()
-	tempDir := t.TempDir()
-
-	dataDir := filepath.Join(tempDir, "source")
-	storeDir := filepath.Join(tempDir, "remote")
-	secondDBPath := filepath.Join(tempDir, "second.sqlite")
-
-	chunkSize := int64(64 * 1024)
-	maxBlobSize := int64(512 * 1024)
-
-	require.NoError(t, fs.MkdirAll(dataDir, 0o755))
-	require.NoError(t, afero.WriteFile(fs,
-		filepath.Join(dataDir, "data.bin"),
-		bytesPattern("reuse-", int(chunkSize*3)), 0o644))
-
-	ctx := context.Background()
-
-	runFileStorageBackup(ctx, t, fs, dataDir, storeDir,
-		filepath.Join(tempDir, "first.sqlite"), chunkSize, maxBlobSize)
-
-	env := setupDedupBackupEnv(
-		ctx, t, fs, storeDir, secondDBPath, chunkSize, maxBlobSize)
-
-	defer func() { _ = env.db.Close() }()
-
-	extra, err := age.GenerateX25519Identity()
-	require.NoError(t, err)
-
-	env.cfg.AgeRecipients = append(env.cfg.AgeRecipients,
-		extra.Recipient().String())
-
-	snapshotID, result := runDedupSnapshot(ctx, t, env.sm, env.makeScanner(),
-		env.cfg.Hostname, "second", dataDir, secondDBPath)
-	require.Positive(t, result.BlobsCreated)
-	require.Zero(t, result.BlobsUploaded,
-		"the second backup should find every blob already stored")
-
-	require.NoError(t, env.db.Close())
-
-	verifier := &vaultik.Vaultik{
-		Config:  env.cfg,
-		Storage: env.storer,
-		Fs:      fs,
-		Stdout:  io.Discard,
-		Stderr:  io.Discard,
-		UI:      ui.NewWithColor(io.Discard, false),
-	}
-	verifier.SetContext(ctx)
-
-	require.NoError(t,
-		verifier.VerifySnapshotWithOptions(snapshotID, &vaultik.VerifyOptions{}),
-		"shallow verify should pass")
-	require.NoError(t,
-		verifier.VerifySnapshotWithOptions(snapshotID,
-			&vaultik.VerifyOptions{Deep: true}),
-		"deep verify should pass")
-}

internal/vaultik/snapshot.go

@@ -185,7 +185,6 @@ type snapshotStats struct {
 	totalBlobs         int
 	totalBytesSkipped  int64
 	totalFilesSkipped  int
-	totalFilesFailed   int
 	totalFilesDeleted  int
 	totalBytesDeleted  int64
 	totalBytesUploaded int64
@@ -316,7 +315,6 @@ func (v *Vaultik) scanAllDirectories(
 		stats.totalChunks += result.ChunksCreated
 		stats.totalBlobs += result.BlobsCreated
 		stats.totalFilesSkipped += result.FilesSkipped
-		stats.totalFilesFailed += result.FilesFailed
 		stats.totalBytesSkipped += result.BytesSkipped
 		stats.totalFilesDeleted += result.FilesDeleted
 		stats.totalBytesDeleted += result.BytesDeleted
@@ -328,7 +326,6 @@ func (v *Vaultik) scanAllDirectories(
 			"path", dir,
 			"files", result.FilesScanned,
 			"files_skipped", result.FilesSkipped,
-			"files_failed", result.FilesFailed,
 			"bytes", result.BytesScanned,
 			"bytes_skipped", result.BytesSkipped,
 			"chunks", result.ChunksCreated,
@@ -362,11 +359,9 @@ func (v *Vaultik) finalizeSnapshotMetadata(
 		return fmt.Errorf("getting snapshot blob sizes: %w", err)
 	}
 
-	// file_count and total_size leave out the files that could not be
-	// stored; stats.totalBytes already does.
 	extStats := snapshot.ExtendedBackupStats{
 		BackupStats: snapshot.BackupStats{
-			FilesScanned:  stats.totalFiles - stats.totalFilesFailed,
+			FilesScanned:  stats.totalFiles,
 			TotalSize:     stats.totalBytes + stats.totalBytesSkipped,
 			ChunksCreated: stats.totalChunks,
 			BlobsCreated:  stats.totalBlobs,
@@ -414,8 +409,7 @@ func (v *Vaultik) printSnapshotSummary(
 	snapshotID string, startTime time.Time, stats *snapshotStats,
 ) {
 	snapshotDuration := time.Since(startTime)
-	totalFilesChanged := stats.totalFiles - stats.totalFilesSkipped -
-		stats.totalFilesFailed
+	totalFilesChanged := stats.totalFiles - stats.totalFilesSkipped
 	totalBytesAll := stats.totalBytes + stats.totalBytesSkipped
 
 	var compressionRatio float64
@@ -432,10 +426,6 @@ func (v *Vaultik) printSnapshotSummary(
 		v.UI.Count(stats.totalFiles),
 		v.UI.Count(totalFilesChanged),
 		v.UI.Count(stats.totalFilesSkipped))
-	if stats.totalFilesFailed > 0 {
-		filesMsg += fmt.Sprintf(", %s failed", v.UI.Count(stats.totalFilesFailed))
-	}
-
 	if stats.totalFilesDeleted > 0 {
 		filesMsg += fmt.Sprintf(", %s deleted", v.UI.Count(stats.totalFilesDeleted))
 	}

internal/vaultik/snapshot_failed_file_test.go

@@ -1,123 +0,0 @@
-package vaultik_test
-
-import (
-	"bytes"
-	"context"
-	"fmt"
-	"os"
-	"path/filepath"
-	"testing"
-
-	"github.com/spf13/afero"
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-	"sneak.berlin/go/vaultik/internal/config"
-	"sneak.berlin/go/vaultik/internal/database"
-	"sneak.berlin/go/vaultik/internal/log"
-	"sneak.berlin/go/vaultik/internal/storage"
-	"sneak.berlin/go/vaultik/internal/ui"
-	"sneak.berlin/go/vaultik/internal/vaultik"
-)
-
-// openFailFs is the real filesystem, except that opening path fails
-// with err. Phase 1 of a backup only lstats a file, so it still counts
-// path; phase 2 is the first to open it.
-type openFailFs struct {
-	afero.OsFs
-
-	path string
-	err  error
-}
-
-//nolint:ireturn // afero.Fs.Open is defined to return the interface.
-func (f *openFailFs) Open(name string) (afero.File, error) {
-	if name == f.path {
-		return nil, &os.PathError{Op: "open", Path: name, Err: f.err}
-	}
-
-	return f.OsFs.Open(name)
-}
-
-// A file that phase 2 cannot open is reported as failed, not as
-// unchanged, and neither the summary's data total nor the snapshots row
-// counts it. See https://git.eeqj.de/sneak/vaultik/issues/280.
-func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
-	log.Initialize(log.Config{})
-	t.Parallel()
-
-	tests := []struct {
-		name       string
-		openErr    error
-		skipErrors bool
-	}{
-		// What a normal user gets opening a file with mode 000.
-		{name: "unopenable under skip-errors",
-			openErr: os.ErrPermission, skipErrors: true},
-		// What opening a file removed after phase 1 gives.
-		{name: "removed between the phases",
-			openErr: os.ErrNotExist, skipErrors: false},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			t.Parallel()
-
-			ctx := context.Background()
-
-			// The scan walks the source path with symlinks resolved, so
-			// failedPath must be spelled the same way to match.
-			tempDir, err := filepath.EvalSymlinks(t.TempDir())
-			require.NoError(t, err)
-
-			srcDir := filepath.Join(tempDir, "src")
-			failedPath := filepath.Join(srcDir, "failed.txt")
-			storedContent := []byte("this file is backed up")
-			storedSize := int64(len(storedContent))
-
-			fs := &openFailFs{path: failedPath, err: tt.openErr}
-			require.NoError(t, fs.MkdirAll(srcDir, 0o755))
-			require.NoError(t, afero.WriteFile(fs,
-				filepath.Join(srcDir, "stored.txt"), storedContent, 0o644))
-			require.NoError(t, afero.WriteFile(fs,
-				failedPath, []byte("this file cannot be opened"), 0o644))
-
-			cfg := faultTestConfig()
-			cfg.IndexPath = filepath.Join(tempDir, "index.sqlite")
-			cfg.Snapshots = map[string]config.SnapshotConfig{
-				"src": {Paths: []string{srcDir}},
-			}
-
-			store, err := storage.NewFileStorer(filepath.Join(tempDir, "remote"))
-			require.NoError(t, err)
-
-			db, err := database.New(ctx, cfg.IndexPath)
-			require.NoError(t, err)
-			t.Cleanup(func() { _ = db.Close() })
-
-			repos := database.NewRepositories(db)
-			out := &bytes.Buffer{}
-			v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
-			v.UI = ui.NewWithColor(out, false)
-
-			require.NoError(t, v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
-				SkipErrors: tt.skipErrors,
-				Snapshots:  []string{"src"},
-			}))
-
-			summary := out.String()
-			assert.Contains(t, summary,
-				"Files: 2 examined, 1 backed up, 0 unchanged, 1 failed.")
-			assert.Contains(t, summary,
-				fmt.Sprintf("Data: %s total (%s backed up).",
-					v.UI.Size(storedSize), v.UI.Size(storedSize)))
-
-			snap, err := repos.Snapshots.GetByID(ctx,
-				localSnapshotID(ctx, t, repos, "src"))
-			require.NoError(t, err)
-			require.NotNil(t, snap)
-
-			assert.Equal(t, int64(1), snap.FileCount)
-			assert.Equal(t, storedSize, snap.TotalSize)
-		})
-	}
-}

internal/vaultik/snapshot_skip_errors_test.go

@@ -1,75 +0,0 @@
-package vaultik_test
-
-import (
-	"context"
-	"fmt"
-	"path/filepath"
-	"testing"
-
-	"github.com/spf13/afero"
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-	"sneak.berlin/go/vaultik/internal/config"
-	"sneak.berlin/go/vaultik/internal/database"
-	"sneak.berlin/go/vaultik/internal/log"
-	"sneak.berlin/go/vaultik/internal/storage"
-	"sneak.berlin/go/vaultik/internal/vaultik"
-)
-
-// --skip-errors skips only a file that cannot be opened or read. A local
-// index error while a directory is recorded stops the backup, and the
-// snapshot is not recorded as complete. See
-// https://git.eeqj.de/sneak/vaultik/issues/284.
-func TestSkipErrorsBackupStopsOnIndexErrorRecordingDirectory(t *testing.T) {
-	log.Initialize(log.Config{})
-	t.Parallel()
-
-	ctx := context.Background()
-
-	// The scan walks the source path with symlinks resolved, so dirPath
-	// must be spelled the same way to match the row the scan inserts.
-	tempDir, err := filepath.EvalSymlinks(t.TempDir())
-	require.NoError(t, err)
-
-	srcDir := filepath.Join(tempDir, "src")
-	dirPath := filepath.Join(srcDir, "dir")
-
-	fs := afero.NewOsFs()
-	require.NoError(t, fs.MkdirAll(dirPath, 0o755))
-	require.NoError(t, afero.WriteFile(fs,
-		filepath.Join(dirPath, "file.txt"), []byte("file content"), 0o644))
-
-	cfg := faultTestConfig()
-	cfg.IndexPath = filepath.Join(tempDir, "index.sqlite")
-	cfg.Snapshots = map[string]config.SnapshotConfig{
-		"tree": {Paths: []string{srcDir}},
-	}
-
-	store, err := storage.NewFileStorer(filepath.Join(tempDir, "remote"))
-	require.NoError(t, err)
-
-	db, err := database.New(ctx, cfg.IndexPath)
-	require.NoError(t, err)
-	t.Cleanup(func() { _ = db.Close() })
-
-	// The local index refuses the directory's files row.
-	_, err = db.Conn().ExecContext(ctx, fmt.Sprintf(`
-		CREATE TRIGGER refuse_directory BEFORE INSERT ON files
-		WHEN NEW.path = '%s'
-		BEGIN SELECT RAISE(ABORT, 'simulated index error'); END`, dirPath))
-	require.NoError(t, err)
-
-	repos := database.NewRepositories(db)
-	v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
-
-	err = v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
-		SkipErrors: true,
-		Snapshots:  []string{"tree"},
-	})
-	require.ErrorContains(t, err, "simulated index error")
-
-	snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
-	require.NoError(t, err)
-	require.Len(t, snapshots, 1)
-	assert.Nil(t, snapshots[0].CompletedAt)
-}

internal/vaultik/storage_bind.go

@@ -9,7 +9,7 @@ import (
 )
 
 // errStorageBindingMismatch is returned when the local index database is
-// bound to a different destination than the configured one.
+// bound to a different destination than the configured storage_url.
 var errStorageBindingMismatch = errors.New(
 	"local index is bound to a different backup destination")
 
@@ -24,9 +24,8 @@ var errStorageBindingMismatch = errors.New(
 //
 // Behaviour:
 //   - On first use (empty stored value), record the configured
-//     destination (storage_url, or the s3.* endpoint, bucket and
-//     prefix) and log the binding.
-//   - When the stored value matches the configured destination, do
+//     storage_url and log the binding.
+//   - When the stored value matches the configured storage_url, do
 //     nothing and return nil.
 //   - When the two differ, refuse with an error that tells the user
 //     how to recover (revert the config, or run `vaultik database
@@ -47,10 +46,12 @@ func (v *Vaultik) EnsureStorageBinding() error {
 
 	configured := v.Config.StorageURL
 	if configured == "" {
-		// The s3.* form: bind on its endpoint, bucket and prefix,
-		// written the way storage_url would name the same destination.
-		s3 := v.Config.S3
-		configured = "s3://" + s3.Bucket + "/" + s3.Prefix + "?endpoint=" + s3.Endpoint
+		// Some legacy configs still use the split s3.* keys instead of
+		// storage_url. Falling back to a synthetic URL for those would
+		// hide the fact that the binding is loose. Instead, treat
+		// unset as "nothing to bind against" — the check is
+		// necessarily best-effort for those configs.
+		return nil
 	}
 
 	stored, err := v.Repositories.LocalMeta.Get(v.ctx, database.LocalMetaKeyStorageURL)
@@ -95,8 +96,7 @@ func buildBindingMismatchMessage(stored, configured string) string {
 		"future snapshots referencing blobs that don't exist at the new destination.\n" +
 		"\n" +
 		"To proceed, either:\n" +
-		"  - revert storage_url (or the s3.* endpoint, bucket and prefix) in your\n" +
-		"    config to the bound destination, or\n" +
+		"  - revert storage_url in your config to the bound destination, or\n" +
 		"  - run 'vaultik database delete' to discard the local index and rebuild it\n" +
 		"    from a fresh full backup against the new destination"
 }

internal/vaultik/storage_bind_test.go

@@ -76,51 +76,3 @@ func TestEnsureStorageBinding_MismatchRefuses(t *testing.T) {
 	assert.Contains(t, err.Error(), "file:///mnt/backups/new")
 	assert.Contains(t, err.Error(), "vaultik database delete")
 }
-
-// A config with no storage_url names its destination with s3.endpoint,
-// s3.bucket and s3.prefix; a change to any of them must be refused just
-// as a changed storage_url is.
-func TestEnsureStorageBinding_S3FieldsChangeRefuses(t *testing.T) {
-	log.Initialize(log.Config{})
-	t.Parallel()
-
-	bound := config.S3Config{
-		Endpoint: "https://s3.example.com",
-		Bucket:   "bucket-a",
-		Prefix:   "backups",
-	}
-
-	tests := []struct {
-		name    string
-		changed config.S3Config
-	}{
-		{"bucket", config.S3Config{
-			Endpoint: bound.Endpoint, Bucket: "bucket-b", Prefix: bound.Prefix,
-		}},
-		{"prefix", config.S3Config{
-			Endpoint: bound.Endpoint, Bucket: bound.Bucket, Prefix: "other",
-		}},
-		{"endpoint", config.S3Config{
-			Endpoint: "https://s3.other.com", Bucket: bound.Bucket, Prefix: bound.Prefix,
-		}},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			t.Parallel()
-
-			tv, _ := buildBindTestVaultik(t, "")
-
-			tv.Config.S3 = bound
-			require.NoError(t, tv.EnsureStorageBinding(), "first use binds")
-			require.NoError(t, tv.EnsureStorageBinding(), "same destination passes")
-
-			tv.Config.S3 = tt.changed
-			err := tv.EnsureStorageBinding()
-			require.Error(t, err)
-			assert.Contains(t, err.Error(),
-				"local index is bound to a different backup destination")
-			assert.Contains(t, err.Error(), "vaultik database delete")
-		})
-	}
-}