Merge feature/remote-id-hashing-and-resilient-list

Two related changes, both addressing leakage and brittleness around
the public bytes the destination store sees.

First, every remote storage path that previously embedded a human
snapshot ID (e.g. metadata/heraklion_berlin.sneak.fs.photos.2026.
catalog_2026-06-24T07:00:15Z/...) now uses the hashed remote key:

  RemoteSnapshotKey(id) = hex(SHA256(SHA256("vaultik|" + id)))

Applied at:

  * uploadSnapshotArtifacts (snapshot create write path)
  * the manifest.json.zst snapshot_id field — manifest is
    unencrypted, so the human ID would otherwise be readable to
    anyone with bucket-list permission
  * cleanupIncompleteSnapshots metadata-existence probe
  * snapshot restore / verify (downloadSnapshotDB,
    loadVerificationData)
  * downloadManifestByKey, deleteRemoteSnapshotByKey
  * CleanupLocalSnapshots reconciliation
  * the locally-driven removal paths (RemoveSnapshot,
    RemoveAllSnapshots, confirmAndExecutePurge)

The local index database keeps human IDs everywhere — the hash is a
boundary translation, not a rename. A directory listing of the
backup destination now looks like
"metadata/<64-hex>/{db.zst.age,manifest.json.zst}" with no host,
snapshot-name, or timestamp information visible.

Second, snapshot list no longer fails just because remote storage is
unreachable, and only consults the remote when the local machine can
plausibly decrypt:

  * Listing is always driven by the local index database — that's
    what holds the human IDs, timestamps, and per-snapshot stats
    that the table actually shows.
  * If no age secret key is configured, we skip remote listing
    entirely (the box is treated as a write-only backup machine —
    there's no value showing it remote-only keys it could never
    restore).
  * If a key IS configured, we try the remote listing; failures
    (volume unmounted, permission denied, network error) downgrade
    to a warning instead of aborting the command.
  * When the remote listing succeeds, we cross-reference by hashing
    each local human ID and diffing against the returned key set.
    Local-only snapshots get the existing "stale local record"
    cleanup hint; remote-only keys are surfaced as a single
    "NOTE: N remote snapshot(s) found in backup destination store
    but not in local database" line.

FileStorer construction also no longer does an eager mkdir — the
basePath is recorded and the directory is created lazily on first
write. A missing or unmounted destination during `snapshot list`
should NOT block the command, and now it doesn't.

RemoveAllSnapshots is rewritten to drive deletion from the local
index instead of from a remote listing, hashing each local ID to
find the corresponding remote key. Orphan remote keys (no matching
local snapshot) are handled separately and only deleted when
--remote is set. Existing tests are updated to hash storage paths
through the new RemoteSnapshotKey helper.

The hash format is a hard pre-1.0 break: existing remote snapshots
written under the human-ID path scheme are no longer readable; they
need to be either re-uploaded under the new scheme or manually
renamed. There is no fallback path; matching the project policy of
"no migrations pre-1.0."

README.md

@@ -100,9 +100,8 @@ vaultik [--config <path>] snapshot list [--json]
 vaultik [--config <path>] snapshot verify <snapshot-id> [--deep] [--json]
 vaultik [--config <path>] snapshot purge [--keep-latest | --older-than <duration>] [--snapshot <name>...] [--force]
 vaultik [--config <path>] snapshot remove <snapshot-id|--all> [--dry-run] [--force] [--remote] [--json]
-vaultik [--config <path>] snapshot prune
 vaultik [--config <path>] snapshot cleanup
-vaultik [--config <path>] restore <snapshot-id> <target-dir> [paths...] [--verify]
+vaultik [--config <path>] snapshot restore <snapshot-id> <target-dir> [paths...] [--verify]
 vaultik [--config <path>] prune [--force] [--json]
 vaultik [--config <path>] info
 vaultik [--config <path>] remote info [--json]
@@ -123,7 +122,7 @@ vaultik version
 
 ### environment variables
 
-* `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for `restore` and `verify --deep`)
+* `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for `snapshot restore` and `snapshot verify --deep`)
 * `VAULTIK_CONFIG`: Path to config file (overridden by `--config`)
 * `VAULTIK_INDEX_PATH`: Override local SQLite index path
 
@@ -157,11 +156,13 @@ existing file. Created with mode `0600` since it will contain credentials.
 **`config edit`**: Open the config file in `$EDITOR` (falls back to `vi`).
 
 **`config get`**: Print a config value addressed by dotted YAML path
-(e.g. `vaultik config get s3.bucket`). Non-scalar values print as YAML.
+(e.g. `vaultik config get storage_url`). Non-scalar values print as YAML.
 
 **`config set`**: Set a scalar config value by dotted YAML path
-(e.g. `vaultik config set compression_level 9`). Comments and formatting
-in the file are preserved; intermediate maps are created as needed.
+(e.g. `vaultik config set compression_level 9`,
+`vaultik config set storage_url "file:///mnt/backups"`). Comments and
+formatting in the file are preserved; intermediate maps are created as
+needed.
 
 **`snapshot create`**: Perform incremental backup of configured snapshots.
 * Optional snapshot names argument to create specific snapshots (default: all)
@@ -176,7 +177,11 @@ in the file are preserved; intermediate maps are created as needed.
 * `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
   this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`)
 
-**`snapshot list`**: List all snapshots with their timestamps and sizes.
+**`snapshot list`**: Show every snapshot known to the destination
+store with timestamps and three sizes per snapshot (compressed
+remote size; total uncompressed chunk size; size of chunks newly
+referenced by that snapshot). The uncompressed and "new chunk"
+columns show `<remote only>` for snapshots not in the local index.
 * `--json`: Output in JSON format
 
 **`snapshot verify`**: Verify snapshot integrity.
@@ -194,28 +199,31 @@ latest globally).
 * `--force`: Skip confirmation prompt
 
 **`snapshot remove`**: Remove a specific snapshot from the local database.
+Automatically cleans up local rows (files, chunks, blobs) that the removed
+snapshot was the last referrer for — you don't need a separate prune step
+after removal.
 * `--remote`: Also remove snapshot metadata from remote storage
 * `--all`: Remove all snapshots (requires `--force`)
 * `--dry-run`: Show what would be deleted without deleting
 * `--force`: Skip confirmation prompt
 * `--json`: Output result as JSON
 
-**`snapshot prune`**: Clean orphaned data from the local database (files,
-chunks, blobs not referenced by any snapshot).
-
 **`snapshot cleanup`**: Remove stale local snapshot records that have no
 corresponding metadata in remote storage. These are typically left behind
 by incomplete or interrupted backups. Does not touch remote storage.
 
-**`restore`**: Restore files from a backup snapshot.
+**`snapshot restore`**: Restore files from a backup snapshot.
 * Requires `VAULTIK_AGE_SECRET_KEY` environment variable
 * Optional path arguments to restore specific files/directories (default: all)
 * Preserves file permissions, timestamps, ownership (ownership requires root),
   symlinks, and empty directories
 * `--verify`: After restoring, verify every file's chunk hashes match
 
-**`prune`**: Remove unreferenced blobs from remote storage.
-* Scans all snapshot manifests for referenced blobs, deletes any blob not referenced
+**`prune`**: Tidy up everything that isn't needed. Removes orphaned local
+database rows (files, chunks, blobs no longer referenced by any completed
+snapshot) AND deletes unreferenced blobs from remote storage. `snapshot
+create --prune`, `snapshot remove`, and `snapshot purge` run the same
+cleanup automatically; this is the manual entry point for the same work.
 * `--force`: Skip confirmation prompt
 * `--json`: Output stats as JSON
 
@@ -385,13 +393,71 @@ Key fields:
 
 ## roadmap
 
-Items for future releases:
+Items still to do before / shortly after 1.0. Loosely ordered by
+priority.
 
-* Error-condition tests (network failures, disk full, corrupted/missing blobs)
-* Parallel blob downloads during restore
-* Bandwidth limiting (`--bwlimit`)
-* Security audit of encryption implementation
-* Man pages and richer `--help` examples
+### correctness and operability
+
+* **Security audit of the encryption implementation.** Pre-1.0
+  blocker if we're advertising "secure" at the top of this README.
+  age + zstd + content-defined chunking is mostly off-the-shelf
+  pieces, but the seams (key handling, recipient parsing, manifest
+  trust boundary, restore-time identity validation) need an outside
+  read.
+* **Error-condition tests.** Today's coverage is the happy path
+  plus a few specific regressions. Need fault-injection coverage:
+  network failures mid-blob, disk-full during restore, corrupted /
+  truncated / missing blobs, partial uploads, kill -9 between
+  manifest and db.zst.age writes.
+* **Verify restored content end-to-end in CI.** The current
+  integration test does this for a small synthetic snapshot but
+  not at scale. A nightly job against a multi-GB representative
+  snapshot would catch silent regressions in the chunker, packer,
+  or restore planner.
+
+### performance
+
+* **Parallel blob downloads during restore.** Single-stream right
+  now. With a fast S3 endpoint and a multi-core machine restore is
+  bound by per-blob fetch + decrypt + decompress; running N of
+  those in parallel against the disk cache would close most of the
+  remaining gap. Needs to interact correctly with the locality
+  planner and sweeper.
+* **Bandwidth limiting (`--bwlimit`).** Both upload and download.
+  Useful for backing up over a shared link. Tricky to make work
+  correctly with the parallel-download story.
+* **Restart of interrupted restore.** Today restore is restartable
+  in the sense that re-running it overwrites partial output; it
+  doesn't resume from where it stopped or skip already-present
+  files. A `--resume` mode that checks targets before fetching
+  blobs would matter for very large restores.
+
+### usability
+
+* **Man pages and richer `--help` examples.** Cobra generates
+  basic help; man pages would be a separate target.
+* **`--bwlimit` style human-readable size flags** across the
+  command surface where they're currently raw integers.
+* **`vaultik snapshot diff <a> <b>`** — show which files changed
+  between two snapshots without restoring either.
+* **Status reporting hook for `--cron`.** When a backup fails
+  silently in cron, the user has no idea. A configurable
+  webhook / email / `notify-send` hook on completion (success and
+  failure) would close the loop.
+
+### infrastructure
+
+* **Cross-machine restore documentation.** The "restore from
+  another host" workflow works but isn't documented as a
+  first-class operation in this README. Worth a dedicated section
+  once it's settled.
+* **Schema migrations.** Currently nonexistent — pre-1.0 schema
+  changes are handled by `vaultik database purge` plus a full
+  re-scan. Post-1.0 we'll need a migration story to keep existing
+  index databases usable across upgrades.
+* **Storage backend coverage tests.** S3, file://, and rclone://
+  all share the Storer interface but the rclone path is the least
+  exercised in CI.
 
 ---
 

internal/cli/config.go

@@ -285,7 +285,7 @@ func newConfigEditCommand() *cobra.Command {
 func newConfigGetCommand() *cobra.Command {
 	return &cobra.Command{
 		Use:   "get <key>",
-		Short: "Print a config value by dotted path (e.g. s3.bucket)",
+		Short: "Print a config value by dotted path (e.g. storage_url, compression_level)",
 		Args:  cobra.ExactArgs(1),
 		RunE: func(cmd *cobra.Command, args []string) error {
 			path, err := ResolveConfigPath()
@@ -328,9 +328,10 @@ the file back, preserving comments and formatting. Intermediate maps
 are created as needed.
 
 Examples:
+  vaultik config set storage_url "file:///mnt/backups"
+  vaultik config set storage_url "s3://bucket/prefix?endpoint=host&region=us-east-1"
   vaultik config set compression_level 9
-  vaultik config set s3.bucket mybucket
-  vaultik config set storage_url "file:///mnt/backups"`,
+  vaultik config set s3.bucket mybucket  # legacy S3 fields still supported`,
 		Args: cobra.ExactArgs(2),
 		RunE: func(cmd *cobra.Command, args []string) error {
 			path, err := ResolveConfigPath()

internal/cli/prune.go

@@ -16,14 +16,19 @@ func NewPruneCommand() *cobra.Command {
 
 	cmd := &cobra.Command{
 		Use:   "prune",
-		Short: "Remove unreferenced blobs",
-		Long: `Removes blobs that are not referenced by any snapshot.
+		Short: "Tidy local database and remote storage",
+		Long: `Removes orphaned data from both the local index database and
+unreferenced blobs from the backup destination store.
 
-This command scans all snapshots and their manifests to build a list of
-referenced blobs, then removes any blobs in storage that are not in this list.
+Local cleanup drops incomplete snapshots and any files, chunks, or
+blobs no longer referenced by a completed snapshot. Remote cleanup
+scans every snapshot manifest in the destination store, builds the
+set of still-referenced blob hashes, and deletes any blob not in that
+set.
 
-Use this command after deleting snapshots with 'vaultik purge' to reclaim
-storage space.`,
+Snapshot create --prune and snapshot remove run the same cleanup
+automatically; this command is the manual entry point for the same
+work (e.g. after a crashed backup or to reclaim storage).`,
 		Args: cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, args []string) error {
 			// Use unified config resolution
@@ -49,7 +54,7 @@ storage space.`,
 								// Start the prune operation in a goroutine
 								go func() {
 									// Run the prune operation
-									if err := v.PruneBlobs(opts); err != nil {
+									if err := v.Prune(opts); err != nil {
 										if err != context.Canceled {
 											if !opts.JSON {
 												log.Error("Prune operation failed", "error", err)

internal/cli/snapshot.go

@@ -25,7 +25,6 @@ func NewSnapshotCommand() *cobra.Command {
 	cmd.AddCommand(newSnapshotPurgeCommand())
 	cmd.AddCommand(newSnapshotVerifyCommand())
 	cmd.AddCommand(newSnapshotRemoveCommand())
-	cmd.AddCommand(newSnapshotPruneCommand())
 	cmd.AddCommand(newSnapshotCleanupCommand())
 	cmd.AddCommand(newSnapshotRestoreCommand())
 
@@ -415,64 +414,6 @@ Use --all --force to remove all snapshots.`,
 	return cmd
 }
 
-// newSnapshotPruneCommand creates the 'snapshot prune' subcommand
-func newSnapshotPruneCommand() *cobra.Command {
-	cmd := &cobra.Command{
-		Use:   "prune",
-		Short: "Remove orphaned data from local database",
-		Long: `Removes orphaned files, chunks, and blobs from the local database.
-
-This cleans up data that is no longer referenced by any snapshot, which can
-accumulate from incomplete backups or deleted snapshots.`,
-		Args: cobra.NoArgs,
-		RunE: func(cmd *cobra.Command, args []string) error {
-			// Use unified config resolution
-			configPath, err := ResolveConfigPath()
-			if err != nil {
-				return err
-			}
-
-			rootFlags := GetRootFlags()
-			return RunWithApp(cmd.Context(), AppOptions{
-				ConfigPath: configPath,
-				LogOptions: log.LogOptions{
-					Verbose: rootFlags.Verbose,
-					Debug:   rootFlags.Debug,
-					Quiet:   rootFlags.Quiet,
-				},
-				Modules: []fx.Option{},
-				Invokes: []fx.Option{
-					fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
-						lc.Append(fx.Hook{
-							OnStart: func(ctx context.Context) error {
-								go func() {
-									if _, err := v.PruneDatabase(); err != nil {
-										if err != context.Canceled {
-											log.Error("Failed to prune database", "error", err)
-											ReportError("Failed to prune database: %v", err)
-											os.Exit(1)
-										}
-									}
-									if err := v.Shutdowner.Shutdown(); err != nil {
-										log.Error("Failed to shutdown", "error", err)
-									}
-								}()
-								return nil
-							},
-							OnStop: func(ctx context.Context) error {
-								v.Cancel()
-								return nil
-							},
-						})
-					}),
-				},
-			})
-		},
-	}
-
-	return cmd
-}
-
 // newSnapshotCleanupCommand creates the 'snapshot cleanup' subcommand
 func newSnapshotCleanupCommand() *cobra.Command {
 	cmd := &cobra.Command{

internal/log/log.go

@@ -46,8 +46,12 @@ func Initialize(cfg Config) {
 	var level slog.Level
 
 	if cfg.Cron || cfg.Quiet {
-		// In quiet/cron mode, only show errors
-		level = slog.LevelError
+		// In cron/quiet mode keep warnings and errors visible — the
+		// whole point of --cron is to stay silent only on total
+		// success, so that anything cron emails to root is genuinely
+		// "something went wrong, look at it." A backup with stuck
+		// permission errors or skipped files should NOT be silent.
+		level = slog.LevelWarn
 	} else if cfg.Debug || strings.Contains(os.Getenv("GODEBUG"), "vaultik") {
 		level = slog.LevelDebug
 	} else if cfg.Verbose {

internal/snapshot/remotekey.go

@@ -0,0 +1,40 @@
+package snapshot
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+)
+
+// remoteKeyPrefix is mixed into the snapshot ID hash so the resulting
+// hex digest is domain-separated from any other "double SHA256 of a
+// string" identifier the user might also use. Keeping this stable is a
+// hard compatibility requirement: changing it invalidates every
+// existing snapshot's remote storage path.
+const remoteKeyPrefix = "vaultik|"
+
+// RemoteSnapshotKey returns the storage-side identifier for a snapshot
+// given its human snapshot ID. It is hex(SHA256(SHA256(prefix + id))).
+// The two SHA256 rounds match Bitcoin's "hash256" convention so the
+// output looks like a 64-character hex blob with no exploitable
+// structure visible to a remote observer.
+//
+// We use this in three places:
+//
+//   - the "metadata/<remote-key>/..." subdirectory on the storage
+//     backend so a directory listing of the bucket / file:// dest
+//     doesn't reveal hostnames, configured snapshot names, or backup
+//     timestamps;
+//   - the `snapshot_id` field of the unencrypted manifest.json.zst
+//     for the same reason;
+//   - any code path that needs to translate a known local snapshot ID
+//     into the path it would occupy on remote storage.
+//
+// The human ID stays the user-visible handle everywhere else — local
+// database joins, CLI arguments, summary lines, log fields — because
+// it's never written to the public bytes once this function gates
+// every storage-path construction.
+func RemoteSnapshotKey(snapshotID string) string {
+	first := sha256.Sum256([]byte(remoteKeyPrefix + snapshotID))
+	second := sha256.Sum256(first[:])
+	return hex.EncodeToString(second[:])
+}

internal/snapshot/scanner.go

@@ -1177,16 +1177,17 @@ func (s *Scanner) uploadBlobIfNeeded(ctx context.Context, blobPath string, blobW
 	finishedBlob := blobWithReader.FinishedBlob
 
 	// Check if blob already exists (deduplication after restart)
+	destination := s.storage.Info().Location
 	if _, err := s.storage.Stat(ctx, blobPath); err == nil {
 		log.Info("Blob already exists in storage, skipping upload",
 			"hash", finishedBlob.Hash, "size", humanize.Bytes(uint64(finishedBlob.Compressed)))
-		s.ui.Info("Blob %s (%s) already exists in backup destination store. Skipping upload.",
-			s.ui.Hex(finishedBlob.Hash), s.ui.Size(finishedBlob.Compressed))
+		s.ui.Info("Blob %s (%s) already exists at %s. Skipping upload.",
+			s.ui.Hex(finishedBlob.Hash), s.ui.Size(finishedBlob.Compressed), s.ui.Path(destination))
 		return true, nil
 	}
 
-	s.ui.Begin("Uploading blob %s (%s) to backup destination store.",
-		s.ui.Hex(finishedBlob.Hash), s.ui.Size(finishedBlob.Compressed))
+	s.ui.Begin("Uploading blob %s (%s) to %s.",
+		s.ui.Hex(finishedBlob.Hash), s.ui.Size(finishedBlob.Compressed), s.ui.Path(destination))
 
 	progressCallback := s.makeUploadProgressCallback(ctx, finishedBlob, startTime)
 

internal/snapshot/snapshot.go

@@ -314,10 +314,17 @@ func (sm *SnapshotManager) prepareExportDB(ctx context.Context, dbPath, snapshot
 	return finalData, tempDBPath, nil
 }
 
-// uploadSnapshotArtifacts uploads the database backup and blob manifest to S3
+// uploadSnapshotArtifacts uploads the database backup and blob manifest
+// to remote storage at metadata/<remote-key>/, where remote-key is the
+// double-SHA256 derivation of the snapshot ID (see RemoteSnapshotKey).
+// We never write the human-readable snapshot ID into any unencrypted
+// part of remote storage so a listing of the destination bucket leaks
+// no host, configuration, or scheduling information.
 func (sm *SnapshotManager) uploadSnapshotArtifacts(ctx context.Context, snapshotID string, dbData, manifestData []byte) error {
+	remoteKey := RemoteSnapshotKey(snapshotID)
+
 	// Upload database backup (compressed and encrypted)
-	dbKey := fmt.Sprintf("metadata/%s/db.zst.age", snapshotID)
+	dbKey := fmt.Sprintf("metadata/%s/db.zst.age", remoteKey)
 
 	dbUploadStart := time.Now()
 	if err := sm.storage.Put(ctx, dbKey, bytes.NewReader(dbData)); err != nil {
@@ -332,7 +339,7 @@ func (sm *SnapshotManager) uploadSnapshotArtifacts(ctx context.Context, snapshot
 		"speed", humanize.SI(dbUploadSpeed, "bps"))
 
 	// Upload blob manifest (compressed only, not encrypted)
-	manifestKey := fmt.Sprintf("metadata/%s/manifest.json.zst", snapshotID)
+	manifestKey := fmt.Sprintf("metadata/%s/manifest.json.zst", remoteKey)
 	manifestUploadStart := time.Now()
 	if err := sm.storage.Put(ctx, manifestKey, bytes.NewReader(manifestData)); err != nil {
 		return fmt.Errorf("uploading blob manifest: %w", err)
@@ -607,9 +614,11 @@ func (sm *SnapshotManager) generateBlobManifest(ctx context.Context, dbPath stri
 		}
 	}
 
-	// Create manifest
+	// Create manifest. SnapshotID in the unencrypted manifest is the
+	// double-SHA256 remote key, not the human ID, so the public bytes
+	// don't reveal hostname/snapshot-name/timestamp metadata.
 	manifest := &Manifest{
-		SnapshotID:          snapshotID,
+		SnapshotID:          RemoteSnapshotKey(snapshotID),
 		Timestamp:           time.Now().UTC().Format(time.RFC3339),
 		BlobCount:           len(blobs),
 		TotalCompressedSize: totalCompressedSize,
@@ -680,8 +689,9 @@ func (sm *SnapshotManager) CleanupIncompleteSnapshots(ctx context.Context, hostn
 
 	// Check each incomplete snapshot for metadata in storage
 	for _, snapshot := range incompleteSnapshots {
-		// Check if metadata exists in storage
-		metadataKey := fmt.Sprintf("metadata/%s/db.zst", snapshot.ID)
+		// Check if metadata exists in storage (paths use the hashed
+		// remote key so we don't leak host info to the listing).
+		metadataKey := fmt.Sprintf("metadata/%s/db.zst", RemoteSnapshotKey(snapshot.ID.String()))
 		_, err := sm.storage.Stat(ctx, metadataKey)
 
 		if err != nil {

internal/storage/file.go

@@ -19,15 +19,20 @@ type FileStorer struct {
 }
 
 // NewFileStorer creates a new filesystem storage backend.
-// The basePath directory will be created if it doesn't exist.
-// Uses the real OS filesystem by default; call SetFilesystem to override for testing.
+//
+// Construction is intentionally cheap and does not touch the filesystem.
+// The basePath is recorded; the directory is created lazily on first
+// write. Reads (Get/Stat/List) tolerate a missing basePath — a missing
+// or unmounted destination during `snapshot list` should NOT block the
+// command, it should degrade to "no remote snapshots reachable" with a
+// warning. Write operations (Put/PutWithProgress) call MkdirAll for the
+// per-blob parent directory, which also covers basePath on first use.
+//
+// Uses the real OS filesystem by default; call SetFilesystem to
+// override for testing.
 func NewFileStorer(basePath string) (*FileStorer, error) {
-	fs := afero.NewOsFs()
-	if err := fs.MkdirAll(basePath, 0755); err != nil {
-		return nil, fmt.Errorf("file:// storage: cannot create or access %s: %w (check that the volume is mounted and writable)", basePath, err)
-	}
 	return &FileStorer{
-		fs:       fs,
+		fs:       afero.NewOsFs(),
 		basePath: basePath,
 	}, nil
 }

internal/vaultik/blobcache.go

@@ -16,14 +16,22 @@ type blobDiskCacheEntry struct {
 	next *blobDiskCacheEntry
 }
 
-// blobDiskCache is an LRU cache that stores blobs on disk instead of in memory.
-// Blobs are written to a temp directory keyed by their hash. When total size
-// exceeds maxBytes, the least-recently-used entries are evicted (deleted from disk).
+// blobDiskCache stores blobs on disk keyed by hash. It exposes ReadAt
+// for slice reads (the restore path uses this so chunk extraction
+// never reads a whole blob into memory) plus Get/Put for whole-blob
+// access.
 //
-// The Get/ReadAt/peak-Len counters are debugging instrumentation used by
-// tests to assert that the restore code path uses ReadAt (which reads
-// only the requested slice of a blob) rather than Get (which reads the
-// full blob into memory).
+// Eviction policy is caller-controlled. The cache keeps an LRU list
+// internally and will fall back to LRU eviction if curBytes exceeds
+// maxBytes. Restore passes math.MaxInt64 as maxBytes and drives
+// eviction itself via Delete() through restoreSweeper, which deletes
+// each blob the moment every file that references its chunks has been
+// written. LRU never fires under that configuration; it is kept as a
+// safety net for callers that don't manage eviction themselves.
+//
+// Get/ReadAt/peak-Len counters are debugging instrumentation used by
+// tests to assert that the restore code path uses ReadAt rather than
+// Get and to bound peak disk-cache occupancy.
 type blobDiskCache struct {
 	mu       sync.Mutex
 	dir      string

internal/vaultik/info.go

@@ -22,14 +22,29 @@ func (v *Vaultik) ShowInfo() error {
 	v.printfStdout("Go Version:      %s\n", runtime.Version())
 	v.printlnStdout()
 
-	// Storage Configuration
+	// Storage Configuration. The backend is selected by storage_url
+	// (s3://, file://, rclone://); the legacy s3.* fields are only
+	// printed when they're actually populated, since the URL scheme
+	// is the primary configuration.
 	v.printfStdout("=== Storage Configuration ===\n")
-	v.printfStdout("S3 Bucket:       %s\n", v.Config.S3.Bucket)
+	storageInfo := v.Storage.Info()
+	v.printfStdout("Type:            %s\n", storageInfo.Type)
+	v.printfStdout("Location:        %s\n", storageInfo.Location)
+	if v.Config.StorageURL != "" {
+		v.printfStdout("Storage URL:     %s\n", v.Config.StorageURL)
+	}
+	if v.Config.S3.Bucket != "" {
+		v.printfStdout("S3 Bucket:       %s\n", v.Config.S3.Bucket)
+	}
 	if v.Config.S3.Prefix != "" {
 		v.printfStdout("S3 Prefix:       %s\n", v.Config.S3.Prefix)
 	}
-	v.printfStdout("S3 Endpoint:     %s\n", v.Config.S3.Endpoint)
-	v.printfStdout("S3 Region:       %s\n", v.Config.S3.Region)
+	if v.Config.S3.Endpoint != "" {
+		v.printfStdout("S3 Endpoint:     %s\n", v.Config.S3.Endpoint)
+	}
+	if v.Config.S3.Region != "" {
+		v.printfStdout("S3 Region:       %s\n", v.Config.S3.Region)
+	}
 	v.printlnStdout()
 
 	// Backup Settings
@@ -337,7 +352,7 @@ func (v *Vaultik) printRemoteInfoTable(result *RemoteInfoResult) {
 		humanize.Comma(int64(result.OrphanedBlobCount)), humanize.Bytes(uint64(result.OrphanedBlobSize)))
 
 	if result.OrphanedBlobCount > 0 {
-		v.printfStdout("\nRun 'vaultik prune --remote' to remove orphaned blobs.\n")
+		v.printfStdout("\nRun 'vaultik prune' to remove orphaned blobs.\n")
 	}
 }
 

internal/vaultik/integration_test.go

@@ -651,10 +651,12 @@ func TestEndToEndFileStorage(t *testing.T) {
 	require.NoError(t, sm.ExportSnapshotMetadata(ctx, dbPath, snapshotID))
 
 	// Verify the backup actually landed on disk under blobs/ and metadata/.
+	// The metadata subdirectory uses the hashed remote key, not the human
+	// snapshot ID, so the on-disk structure doesn't leak hostname/name/time.
 	blobInfo, err := os.Stat(filepath.Join(storeDir, "blobs"))
 	require.NoError(t, err)
 	require.True(t, blobInfo.IsDir())
-	metaInfo, err := os.Stat(filepath.Join(storeDir, "metadata", snapshotID))
+	metaInfo, err := os.Stat(filepath.Join(storeDir, "metadata", snapshot.RemoteSnapshotKey(snapshotID)))
 	require.NoError(t, err)
 	require.True(t, metaInfo.IsDir())
 

internal/vaultik/prune.go

@@ -48,6 +48,19 @@ type PruneBlobsResult struct {
 	BytesFreed   int64 `json:"bytes_freed"`
 }
 
+// Prune removes orphaned data from the local index database AND
+// unreferenced blobs from the backup destination store. This is the
+// single user-facing prune entry point — the split between local and
+// remote cleanup is an implementation detail. Calling code should
+// prefer this method over PruneDatabase or PruneBlobs individually
+// unless it specifically wants one half.
+func (v *Vaultik) Prune(opts *PruneOptions) error {
+	if _, err := v.PruneDatabase(); err != nil {
+		return fmt.Errorf("pruning local database: %w", err)
+	}
+	return v.PruneBlobs(opts)
+}
+
 // PruneBlobs removes unreferenced blobs from storage
 func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
 	log.Info("Starting prune operation")
@@ -110,20 +123,22 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
 // collectReferencedBlobs downloads all manifests and returns the set of referenced blob hashes
 func (v *Vaultik) collectReferencedBlobs() (map[string]bool, error) {
 	log.Info("Listing remote snapshots")
-	snapshotIDs, err := v.listUniqueSnapshotIDs()
+	// IDs returned by listUniqueSnapshotIDs are remote keys (hashed
+	// subdirectories under metadata/), not human snapshot IDs.
+	remoteKeys, err := v.listUniqueSnapshotIDs()
 	if err != nil {
-		return nil, fmt.Errorf("listing snapshot IDs: %w", err)
+		return nil, fmt.Errorf("listing snapshot keys: %w", err)
 	}
-	log.Info("Found manifests in remote storage", "count", len(snapshotIDs))
+	log.Info("Found manifests in remote storage", "count", len(remoteKeys))
 
 	allBlobsReferenced := make(map[string]bool)
 	manifestCount := 0
 
-	for _, snapshotID := range snapshotIDs {
-		log.Debug("Processing manifest", "snapshot_id", snapshotID)
-		manifest, err := v.downloadManifest(snapshotID)
+	for _, remoteKey := range remoteKeys {
+		log.Debug("Processing manifest", "remote_key", remoteKey)
+		manifest, err := v.downloadManifestByKey(remoteKey)
 		if err != nil {
-			log.Error("Failed to download manifest", "snapshot_id", snapshotID, "error", err)
+			log.Error("Failed to download manifest", "remote_key", remoteKey, "error", err)
 			continue
 		}
 		for _, blob := range manifest.Blobs {

internal/vaultik/remove_snapshot_test.go

@@ -132,7 +132,9 @@ func (s *testStorer) Info() storage.StorageInfo {
 	}
 }
 
-// addManifest creates a compressed manifest in storage
+// addManifest creates a compressed manifest in storage at the same
+// hashed path the production code uses. snapshotID is the human ID;
+// the storage path uses RemoteSnapshotKey(id).
 func addManifest(t *testing.T, store *testStorer, snapshotID string, blobHashes []string) {
 	t.Helper()
 
@@ -144,8 +146,9 @@ func addManifest(t *testing.T, store *testStorer, snapshotID string, blobHashes
 		}
 	}
 
+	remoteKey := snapshot.RemoteSnapshotKey(snapshotID)
 	manifest := &snapshot.Manifest{
-		SnapshotID: snapshotID,
+		SnapshotID: remoteKey,
 		BlobCount:  len(blobs),
 		Blobs:      blobs,
 	}
@@ -153,11 +156,19 @@ func addManifest(t *testing.T, store *testStorer, snapshotID string, blobHashes
 	data, err := snapshot.EncodeManifest(manifest, 3)
 	require.NoError(t, err)
 
-	key := "metadata/" + snapshotID + "/manifest.json.zst"
+	key := "metadata/" + remoteKey + "/manifest.json.zst"
 	err = store.Put(context.Background(), key, bytes.NewReader(data))
 	require.NoError(t, err)
 }
 
+// remoteKeyPath returns the storage-relative path to a snapshot's
+// metadata directory or manifest under the hashed remote-key scheme.
+// Tests use this in hasKey/asserts to avoid scattering RemoteSnapshotKey
+// calls throughout.
+func remoteKeyPath(snapshotID, suffix string) string {
+	return "metadata/" + snapshot.RemoteSnapshotKey(snapshotID) + "/" + suffix
+}
+
 // addBlob adds a fake blob to storage
 func addBlob(t *testing.T, store *testStorer, hash string) {
 	t.Helper()
@@ -198,7 +209,7 @@ func TestRemoveSnapshot_LocalOnly(t *testing.T) {
 	// Blobs should NOT be deleted (that's what prune is for)
 	assert.True(t, store.hasKey("blobs/aa/aa/"+blobA))
 	// Remote metadata should NOT be deleted (no --remote flag)
-	assert.True(t, store.hasKey("metadata/snapshot-001/manifest.json.zst"))
+	assert.True(t, store.hasKey(remoteKeyPath("snapshot-001", "manifest.json.zst")))
 
 	// Verify output
 	assert.Contains(t, tv.Stdout.String(), "Removed snapshot 'snapshot-001' from local database")
@@ -225,7 +236,7 @@ func TestRemoveSnapshot_WithRemote(t *testing.T) {
 	// Blobs should NOT be deleted
 	assert.True(t, store.hasKey("blobs/aa/aa/"+blobA))
 	// Remote metadata SHOULD be deleted
-	assert.False(t, store.hasKey("metadata/snapshot-001/manifest.json.zst"))
+	assert.False(t, store.hasKey(remoteKeyPath("snapshot-001", "manifest.json.zst")))
 
 	// Verify output mentions prune
 	assert.Contains(t, tv.Stdout.String(), "Removed snapshot 'snapshot-001' from local database")
@@ -255,7 +266,7 @@ func TestRemoveSnapshot_DryRun(t *testing.T) {
 	// Nothing should be deleted
 	assert.Equal(t, initialCount, store.keyCount())
 	assert.True(t, store.hasKey("blobs/aa/aa/"+blobA))
-	assert.True(t, store.hasKey("metadata/snapshot-001/manifest.json.zst"))
+	assert.True(t, store.hasKey(remoteKeyPath("snapshot-001", "manifest.json.zst")))
 
 	// Verify dry run message
 	assert.Contains(t, tv.Stdout.String(), "[Dry run - no changes made]")
@@ -299,8 +310,8 @@ func TestRemoveAllSnapshots_WithForce(t *testing.T) {
 	// Blobs should NOT be deleted
 	assert.True(t, store.hasKey("blobs/aa/aa/"+blobA))
 	// Remote metadata SHOULD be deleted
-	assert.False(t, store.hasKey("metadata/snapshot-001/manifest.json.zst"))
-	assert.False(t, store.hasKey("metadata/snapshot-002/manifest.json.zst"))
+	assert.False(t, store.hasKey(remoteKeyPath("snapshot-001", "manifest.json.zst")))
+	assert.False(t, store.hasKey(remoteKeyPath("snapshot-002", "manifest.json.zst")))
 
 	// Verify output
 	assert.Contains(t, tv.Stdout.String(), "Removed 2 snapshot(s)")
@@ -318,7 +329,10 @@ func TestRemoveAllSnapshots_DryRun(t *testing.T) {
 
 	tv := vaultik.NewForTesting(store)
 
-	opts := &vaultik.RemoveOptions{All: true, Force: true, DryRun: true}
+	// --remote is required to enumerate orphan remote keys; without
+	// it, RemoveAll only acts on local snapshots, and NewForTesting
+	// has no local DB.
+	opts := &vaultik.RemoveOptions{All: true, Force: true, DryRun: true, Remote: true}
 	result, err := tv.RemoveAllSnapshots(opts)
 
 	require.NoError(t, err)

internal/vaultik/restore.go

@@ -18,6 +18,7 @@ import (
 	"sneak.berlin/go/vaultik/internal/blobgen"
 	"sneak.berlin/go/vaultik/internal/database"
 	"sneak.berlin/go/vaultik/internal/log"
+	"sneak.berlin/go/vaultik/internal/snapshot"
 	"sneak.berlin/go/vaultik/internal/types"
 )
 
@@ -394,10 +395,12 @@ func (v *Vaultik) handleRestoreVerification(
 	return nil
 }
 
-// downloadSnapshotDB downloads and decrypts the snapshot metadata database
+// downloadSnapshotDB downloads and decrypts the snapshot metadata
+// database. The snapshotID is the human ID; we hash it to the remote
+// key for the storage path.
 func (v *Vaultik) downloadSnapshotDB(snapshotID string, identity age.Identity) (*database.DB, error) {
 	// Download encrypted database from storage
-	dbKey := fmt.Sprintf("metadata/%s/db.zst.age", snapshotID)
+	dbKey := fmt.Sprintf("metadata/%s/db.zst.age", snapshot.RemoteSnapshotKey(snapshotID))
 
 	reader, err := v.Storage.Get(v.ctx, dbKey)
 	if err != nil {

internal/vaultik/snapshot.go

@@ -8,16 +8,13 @@ import (
 	"regexp"
 	"sort"
 	"strings"
-	"sync"
 	"text/tabwriter"
 	"time"
 
 	"github.com/dustin/go-humanize"
-	"golang.org/x/sync/errgroup"
 	"sneak.berlin/go/vaultik/internal/database"
 	"sneak.berlin/go/vaultik/internal/log"
 	"sneak.berlin/go/vaultik/internal/snapshot"
-	"sneak.berlin/go/vaultik/internal/types"
 )
 
 // SnapshotCreateOptions contains options for the snapshot create command
@@ -92,8 +89,13 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
 		}
 	}
 
+	// Terminus must obey the --cron invariant: silent on total
+	// success only. UI.Complete is dropped in cron/quiet mode (that's
+	// the success path), but if any warnings fired during the run we
+	// emit the summary via UI.Warning so cron actually delivers
+	// something for the user to look at.
 	if v.UI.WarningCount() > 0 {
-		v.UI.Complete("Finished (with %d warnings).", v.UI.WarningCount())
+		v.UI.Warning("Finished with %d warning(s) — review the output above.", v.UI.WarningCount())
 	} else {
 		v.UI.Complete("Finished successfully.")
 	}
@@ -378,25 +380,43 @@ func (v *Vaultik) getSnapshotBlobSizes(snapshotID string) (compressed int64, unc
 	return compressed, uncompressed
 }
 
-// ListSnapshots lists all snapshots
+// ListSnapshots prints the table of snapshots, plus any reconciliation
+// warnings/notes between the local index and the backup destination
+// store.
+//
+// The local index database is always the primary source for the
+// table — it has the human snapshot IDs, timestamps, and per-snapshot
+// stats.
+//
+// If an age secret key is configured AND remote listing succeeds, we
+// cross-reference: any local snapshot whose hashed key isn't visible
+// remotely gets a "local-only" cleanup hint, and any remote key that
+// doesn't correspond to a known local snapshot gets reported in a
+// NOTE.
+//
+// If no age key is set the local machine is assumed write-only
+// (backup-only), so we skip remote listing entirely — there's no
+// value showing keys the user couldn't restore anyway.
+//
+// If remote listing fails (unmounted volume, permission denied,
+// network), we degrade to local-only with a warning. List never
+// fails just because the destination is unreachable.
 func (v *Vaultik) ListSnapshots(jsonOutput bool) error {
 	log.Info("Listing snapshots")
-	remoteSnapshots, err := v.listRemoteSnapshotIDs()
+
+	localSnaps, err := v.Repositories.Snapshots.ListRecent(v.ctx, 10000)
 	if err != nil {
-		return err
+		return fmt.Errorf("listing local snapshots: %w", err)
 	}
 
-	localSnapshotMap, err := v.reconcileLocalWithRemote(remoteSnapshots)
-	if err != nil {
-		return err
+	snapshots := make([]SnapshotInfo, 0, len(localSnaps))
+	for _, ls := range localSnaps {
+		if ls.CompletedAt == nil {
+			continue
+		}
+		snapshots = append(snapshots, v.snapshotInfoFromLocal(ls))
 	}
 
-	snapshots, err := v.buildSnapshotInfoList(remoteSnapshots, localSnapshotMap)
-	if err != nil {
-		return err
-	}
-
-	// Sort by timestamp (newest first)
 	sort.Slice(snapshots, func(i, j int) bool {
 		return snapshots[i].Timestamp.After(snapshots[j].Timestamp)
 	})
@@ -411,173 +431,85 @@ func (v *Vaultik) ListSnapshots(jsonOutput bool) error {
 		return err
 	}
 
-	// Warn about local snapshots that don't exist in remote storage.
-	var stale []string
-	for id := range localSnapshotMap {
-		if !remoteSnapshots[id] {
-			stale = append(stale, id)
+	if v.Config.AgeSecretKey == "" {
+		return nil
+	}
+
+	remoteKeys, err := v.listAllRemoteSnapshotKeys()
+	if err != nil {
+		v.UI.Warning("Could not list backup destination store: %v.", err)
+		return nil
+	}
+
+	localKeys := make(map[string]string, len(localSnaps))
+	for _, ls := range localSnaps {
+		if ls.CompletedAt == nil {
+			continue
+		}
+		localKeys[snapshot.RemoteSnapshotKey(ls.ID.String())] = ls.ID.String()
+	}
+	remoteSet := make(map[string]bool, len(remoteKeys))
+	for _, k := range remoteKeys {
+		remoteSet[k] = true
+	}
+
+	var localOnly []string
+	for key, humanID := range localKeys {
+		if !remoteSet[key] {
+			localOnly = append(localOnly, humanID)
 		}
 	}
-	if len(stale) > 0 {
-		v.UI.Warning("%d local snapshot record(s) not found in backup destination store:", len(stale))
-		for _, id := range stale {
+	var remoteOnlyCount int
+	for key := range remoteSet {
+		if _, ok := localKeys[key]; !ok {
+			remoteOnlyCount++
+		}
+	}
+
+	if len(localOnly) > 0 {
+		v.UI.Warning("%d local snapshot record(s) not found in backup destination store:", len(localOnly))
+		for _, id := range localOnly {
 			v.UI.Info("%s", v.UI.Snapshot(id))
 		}
 		v.UI.Info("Run 'vaultik snapshot cleanup' to remove stale local records.")
 	}
+	if remoteOnlyCount > 0 {
+		v.UI.Notice("NOTE: %d remote snapshot(s) found in backup destination store but not in local database.", remoteOnlyCount)
+	}
 
 	return nil
 }
 
-// listRemoteSnapshotIDs returns a set of snapshot IDs found in remote storage
-func (v *Vaultik) listRemoteSnapshotIDs() (map[string]bool, error) {
-	remoteSnapshots := make(map[string]bool)
-	objectCh := v.Storage.ListStream(v.ctx, "metadata/")
+// snapshotInfoFromLocal builds a SnapshotInfo row from a local snapshot
+// record. Failures from any per-snapshot stat query degrade that
+// column to its snapshot-row fallback but never fail the listing.
+func (v *Vaultik) snapshotInfoFromLocal(ls *database.Snapshot) SnapshotInfo {
+	idStr := ls.ID.String()
 
-	for object := range objectCh {
-		if object.Err != nil {
-			return nil, fmt.Errorf("listing remote snapshots: %w", object.Err)
-		}
-
-		parts := strings.Split(object.Key, "/")
-		if len(parts) >= 2 && parts[0] == "metadata" && parts[1] != "" {
-			if strings.HasPrefix(parts[1], ".") {
-				continue
-			}
-			remoteSnapshots[parts[1]] = true
-		}
-	}
-
-	return remoteSnapshots, nil
-}
-
-// reconcileLocalWithRemote builds a map of local snapshots keyed by ID for cross-referencing with remote
-func (v *Vaultik) reconcileLocalWithRemote(remoteSnapshots map[string]bool) (map[string]*database.Snapshot, error) {
-	localSnapshots, err := v.Repositories.Snapshots.ListRecent(v.ctx, 10000)
+	totalSize, err := v.Repositories.Snapshots.GetSnapshotTotalCompressedSize(v.ctx, idStr)
 	if err != nil {
-		return nil, fmt.Errorf("listing local snapshots: %w", err)
+		log.Warn("Failed to get total compressed size", "id", idStr, "error", err)
+		totalSize = ls.BlobSize
 	}
 
-	localSnapshotMap := make(map[string]*database.Snapshot)
-	for _, s := range localSnapshots {
-		localSnapshotMap[s.ID.String()] = s
+	uncompressedSize, err := v.Repositories.Snapshots.GetSnapshotUncompressedChunkSize(v.ctx, idStr)
+	if err != nil {
+		log.Warn("Failed to get uncompressed chunk size", "id", idStr, "error", err)
 	}
 
-	return localSnapshotMap, nil
-}
-
-// buildSnapshotInfoList constructs SnapshotInfo entries from remote IDs and local data
-func (v *Vaultik) buildSnapshotInfoList(remoteSnapshots map[string]bool, localSnapshotMap map[string]*database.Snapshot) ([]SnapshotInfo, error) {
-	snapshots := make([]SnapshotInfo, 0, len(remoteSnapshots))
-
-	// remoteOnly collects snapshot IDs that need a manifest download.
-	var remoteOnly []string
-
-	for snapshotID := range remoteSnapshots {
-		if localSnap, exists := localSnapshotMap[snapshotID]; exists && localSnap.CompletedAt != nil {
-			totalSize, err := v.Repositories.Snapshots.GetSnapshotTotalCompressedSize(v.ctx, snapshotID)
-			if err != nil {
-				log.Warn("Failed to get total compressed size", "id", snapshotID, "error", err)
-				totalSize = localSnap.BlobSize
-			}
-
-			uncompressedSize, err := v.Repositories.Snapshots.GetSnapshotUncompressedChunkSize(v.ctx, snapshotID)
-			if err != nil {
-				log.Warn("Failed to get uncompressed chunk size", "id", snapshotID, "error", err)
-			}
-
-			newChunkSize, err := v.Repositories.Snapshots.GetSnapshotNewChunkSize(v.ctx, snapshotID)
-			if err != nil {
-				log.Warn("Failed to get new chunk size", "id", snapshotID, "error", err)
-			}
-
-			snapshots = append(snapshots, SnapshotInfo{
-				ID:               localSnap.ID,
-				Timestamp:        localSnap.StartedAt,
-				CompressedSize:   totalSize,
-				UncompressedSize: uncompressedSize,
-				NewChunkSize:     newChunkSize,
-				LocallyTracked:   true,
-			})
-		} else {
-			timestamp, err := parseSnapshotTimestamp(snapshotID)
-			if err != nil {
-				log.Warn("Failed to parse snapshot timestamp", "id", snapshotID, "error", err)
-				continue
-			}
-
-			// Pre-add with zero size; will be filled by concurrent downloads.
-			snapshots = append(snapshots, SnapshotInfo{
-				ID:             types.SnapshotID(snapshotID),
-				Timestamp:      timestamp,
-				CompressedSize: 0,
-				LocallyTracked: false,
-			})
-			remoteOnly = append(remoteOnly, snapshotID)
-		}
+	newChunkSize, err := v.Repositories.Snapshots.GetSnapshotNewChunkSize(v.ctx, idStr)
+	if err != nil {
+		log.Warn("Failed to get new chunk size", "id", idStr, "error", err)
 	}
 
-	// Download manifests concurrently for remote-only snapshots.
-	if len(remoteOnly) > 0 {
-		// maxConcurrentManifestDownloads bounds parallel manifest fetches to
-		// avoid overwhelming the S3 endpoint while still being much faster
-		// than serial downloads.
-		const maxConcurrentManifestDownloads = 10
-
-		type manifestResult struct {
-			snapshotID string
-			size       int64
-		}
-
-		var (
-			mu      sync.Mutex
-			results []manifestResult
-		)
-
-		g, gctx := errgroup.WithContext(v.ctx)
-		g.SetLimit(maxConcurrentManifestDownloads)
-
-		for _, sid := range remoteOnly {
-			g.Go(func() error {
-				manifestPath := fmt.Sprintf("metadata/%s/manifest.json.zst", sid)
-				reader, err := v.Storage.Get(gctx, manifestPath)
-				if err != nil {
-					return fmt.Errorf("downloading manifest for %s: %w", sid, err)
-				}
-				defer func() { _ = reader.Close() }()
-
-				manifest, err := snapshot.DecodeManifest(reader)
-				if err != nil {
-					return fmt.Errorf("decoding manifest for %s: %w", sid, err)
-				}
-
-				mu.Lock()
-				results = append(results, manifestResult{
-					snapshotID: sid,
-					size:       manifest.TotalCompressedSize,
-				})
-				mu.Unlock()
-				return nil
-			})
-		}
-
-		if err := g.Wait(); err != nil {
-			return nil, fmt.Errorf("fetching manifest sizes: %w", err)
-		}
-
-		// Build a lookup from results and patch the pre-added entries.
-		sizeMap := make(map[string]int64, len(results))
-		for _, r := range results {
-			sizeMap[r.snapshotID] = r.size
-		}
-		for i := range snapshots {
-			if sz, ok := sizeMap[string(snapshots[i].ID)]; ok {
-				snapshots[i].CompressedSize = sz
-			}
-		}
+	return SnapshotInfo{
+		ID:               ls.ID,
+		Timestamp:        ls.StartedAt,
+		CompressedSize:   totalSize,
+		UncompressedSize: uncompressedSize,
+		NewChunkSize:     newChunkSize,
+		LocallyTracked:   true,
 	}
-
-	return snapshots, nil
 }
 
 // printSnapshotTable renders the snapshot list as a formatted table
@@ -763,14 +695,23 @@ func (v *Vaultik) confirmAndExecutePurge(toDelete []SnapshotInfo, force, quiet b
 		if err := v.deleteSnapshotFromLocalDB(snapshotID); err != nil {
 			log.Error("Failed to delete from local database", "snapshot_id", snapshotID, "error", err)
 		}
-		if err := v.deleteSnapshotFromRemote(snapshotID); err != nil {
+		if err := v.deleteRemoteSnapshotByKey(snapshot.RemoteSnapshotKey(snapshotID)); err != nil {
 			return fmt.Errorf("deleting snapshot %s from remote: %w", snapshotID, err)
 		}
 	}
 
+	// Tidy up local DB orphans now so users don't have to run a
+	// separate command after a purge. Guarded against nil for tests
+	// that don't wire up a SnapshotManager.
+	if v.SnapshotManager != nil {
+		if err := v.SnapshotManager.CleanupOrphanedData(v.ctx); err != nil {
+			log.Warn("Failed to clean up orphaned local data after purge", "error", err)
+		}
+	}
+
 	if !quiet {
 		v.printfStdout("Deleted %d snapshot(s)\n", len(toDelete))
-		v.printlnStdout("\nNote: Run 'vaultik prune' to clean up unreferenced blobs.")
+		v.printlnStdout("\nNote: Run 'vaultik prune' to clean up unreferenced remote blobs.")
 	}
 
 	return nil
@@ -799,8 +740,9 @@ func (v *Vaultik) VerifySnapshotWithOptions(snapshotID string, opts *VerifyOptio
 
 	v.printVerifyHeader(snapshotID, opts)
 
-	// Download and parse manifest
-	manifest, err := v.downloadManifest(snapshotID)
+	// Download and parse manifest. The caller supplies a human
+	// snapshot ID; we hash it to address remote storage.
+	manifest, err := v.downloadManifestByKey(snapshot.RemoteSnapshotKey(snapshotID))
 	if err != nil {
 		if opts.JSON {
 			result.Status = "failed"
@@ -915,12 +857,18 @@ func (v *Vaultik) outputVerifyJSON(result *VerifyResult) error {
 
 // CleanupLocalSnapshots removes local snapshot records that have no
 // corresponding metadata in remote storage. These are typically left
-// behind by incomplete or interrupted backups.
+// behind by incomplete or interrupted backups. Each local snapshot's
+// human ID is hashed via RemoteSnapshotKey and compared against the
+// remote listing.
 func (v *Vaultik) CleanupLocalSnapshots() error {
-	remoteSnapshots, err := v.listRemoteSnapshotIDs()
+	remoteKeys, err := v.listAllRemoteSnapshotKeys()
 	if err != nil {
 		return err
 	}
+	remoteSet := make(map[string]bool, len(remoteKeys))
+	for _, k := range remoteKeys {
+		remoteSet[k] = true
+	}
 
 	localSnapshots, err := v.Repositories.Snapshots.ListRecent(v.ctx, 10000)
 	if err != nil {
@@ -930,7 +878,7 @@ func (v *Vaultik) CleanupLocalSnapshots() error {
 	var removed int
 	for _, snap := range localSnapshots {
 		id := snap.ID.String()
-		if !remoteSnapshots[id] {
+		if !remoteSet[snapshot.RemoteSnapshotKey(id)] {
 			v.printfStdout("Removing stale local record: %s\n", id)
 			if err := v.deleteSnapshotFromLocalDB(id); err != nil {
 				log.Error("Failed to delete local snapshot", "snapshot_id", id, "error", err)
@@ -950,8 +898,12 @@ func (v *Vaultik) CleanupLocalSnapshots() error {
 
 // Helper methods that were previously on SnapshotApp
 
-func (v *Vaultik) downloadManifest(snapshotID string) (*snapshot.Manifest, error) {
-	manifestPath := fmt.Sprintf("metadata/%s/manifest.json.zst", snapshotID)
+// downloadManifestByKey fetches the manifest at
+// metadata/<remoteKey>/manifest.json.zst. The remoteKey is the double-
+// SHA256 derivation produced by snapshot.RemoteSnapshotKey, not the
+// human snapshot ID. Callers that have a human ID must hash first.
+func (v *Vaultik) downloadManifestByKey(remoteKey string) (*snapshot.Manifest, error) {
+	manifestPath := fmt.Sprintf("metadata/%s/manifest.json.zst", remoteKey)
 
 	reader, err := v.Storage.Get(v.ctx, manifestPath)
 	if err != nil {
@@ -1086,12 +1038,22 @@ func (v *Vaultik) RemoveSnapshot(snapshotID string, opts *RemoveOptions) (*Remov
 	// If --remote, also remove from remote storage
 	if opts.Remote {
 		log.Info("Removing snapshot metadata from remote storage", "snapshot_id", snapshotID)
-		if err := v.deleteSnapshotFromRemote(snapshotID); err != nil {
+		if err := v.deleteRemoteSnapshotByKey(snapshot.RemoteSnapshotKey(snapshotID)); err != nil {
 			return result, fmt.Errorf("removing from remote storage: %w", err)
 		}
 		result.RemoteRemoved = true
 	}
 
+	// Clean up the local rows that just became orphaned (files, chunks,
+	// blob_chunks, blobs no longer referenced by any snapshot). This
+	// used to be a separate `vaultik snapshot prune` step; running it
+	// inline means `snapshot remove` leaves no ghost rows behind.
+	if v.SnapshotManager != nil {
+		if err := v.SnapshotManager.CleanupOrphanedData(v.ctx); err != nil {
+			log.Warn("Failed to clean up orphaned local data after removal", "error", err)
+		}
+	}
+
 	// Output result
 	if opts.JSON {
 		return result, v.outputRemoveJSON(result)
@@ -1101,20 +1063,48 @@ func (v *Vaultik) RemoveSnapshot(snapshotID string, opts *RemoveOptions) (*Remov
 	v.printfStdout("Removed snapshot '%s' from local database\n", snapshotID)
 	if opts.Remote {
 		v.printlnStdout("Removed snapshot metadata from remote storage")
-		v.printlnStdout("\nNote: Blobs were not removed. Run 'vaultik prune' to remove orphaned blobs.")
+		v.printlnStdout("\nNote: Remote blobs were not removed. Run 'vaultik prune' to remove orphaned blobs.")
 	}
 
 	return result, nil
 }
 
-// RemoveAllSnapshots removes all snapshots from local database and optionally from remote
+// RemoveAllSnapshots removes every snapshot known to the local
+// database from the local index, and (with --remote) every snapshot
+// metadata directory in remote storage. Both sides are processed so a
+// "remove --all" leaves nothing behind, even when the local DB and
+// remote storage have diverged.
 func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error) {
-	snapshotIDs, err := v.listAllRemoteSnapshotIDs()
+	localSnaps, err := v.localSnapshotIDs()
 	if err != nil {
-		return nil, err
+		return nil, fmt.Errorf("listing local snapshots: %w", err)
 	}
 
-	if len(snapshotIDs) == 0 {
+	// remoteKeys is the set of metadata/<key>/ subdirectories on the
+	// destination store; failures are downgraded to a warning so a
+	// permission-denied or unreachable remote can't block a local-only
+	// remove.
+	remoteKeys, remoteErr := v.listAllRemoteSnapshotKeys()
+	if remoteErr != nil {
+		log.Warn("Could not list remote snapshots", "error", remoteErr)
+		v.UI.Warning("Could not list remote snapshots: %v.", remoteErr)
+	}
+
+	// Anything visible on the remote that doesn't correspond to a
+	// known local human ID is treated as an orphan key — handled only
+	// when --remote is in effect.
+	knownLocalKeys := make(map[string]string, len(localSnaps))
+	for _, id := range localSnaps {
+		knownLocalKeys[snapshot.RemoteSnapshotKey(id)] = id
+	}
+	var orphanRemoteKeys []string
+	for _, key := range remoteKeys {
+		if _, known := knownLocalKeys[key]; !known {
+			orphanRemoteKeys = append(orphanRemoteKeys, key)
+		}
+	}
+
+	if len(localSnaps) == 0 && len(orphanRemoteKeys) == 0 {
 		if !opts.JSON {
 			v.printlnStdout("No snapshots found")
 		}
@@ -1122,19 +1112,42 @@ func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error)
 	}
 
 	if opts.DryRun {
-		return v.handleRemoveAllDryRun(snapshotIDs, opts)
+		return v.handleRemoveAllDryRun(localSnaps, orphanRemoteKeys, opts)
 	}
 
-	return v.executeRemoveAll(snapshotIDs, opts)
+	return v.executeRemoveAll(localSnaps, orphanRemoteKeys, opts)
 }
 
-// listAllRemoteSnapshotIDs collects all unique snapshot IDs from remote storage
-func (v *Vaultik) listAllRemoteSnapshotIDs() ([]string, error) {
-	log.Info("Listing all snapshots")
+// localSnapshotIDs returns every snapshot ID present in the local
+// index database, sorted for deterministic iteration. Empty slice if
+// the database has no Repositories (e.g. tests).
+func (v *Vaultik) localSnapshotIDs() ([]string, error) {
+	if v.Repositories == nil {
+		return nil, nil
+	}
+	snaps, err := v.Repositories.Snapshots.ListRecent(v.ctx, 100000)
+	if err != nil {
+		return nil, err
+	}
+	ids := make([]string, 0, len(snaps))
+	for _, s := range snaps {
+		ids = append(ids, s.ID.String())
+	}
+	sort.Strings(ids)
+	return ids, nil
+}
+
+// listAllRemoteSnapshotKeys collects the hashed remote keys
+// (subdirectories under metadata/) currently present in the
+// destination store. Returns (nil, err) when the store cannot be
+// listed; callers must treat that as "no remote info available," not
+// fatal.
+func (v *Vaultik) listAllRemoteSnapshotKeys() ([]string, error) {
+	log.Info("Listing all remote snapshots")
 	objectCh := v.Storage.ListStream(v.ctx, "metadata/")
 
 	seen := make(map[string]bool)
-	var snapshotIDs []string
+	var keys []string
 	for object := range objectCh {
 		if object.Err != nil {
 			return nil, fmt.Errorf("listing remote snapshots: %w", object.Err)
@@ -1147,30 +1160,36 @@ func (v *Vaultik) listAllRemoteSnapshotIDs() ([]string, error) {
 				continue
 			}
 			if strings.HasSuffix(object.Key, "/") || strings.Contains(object.Key, "/manifest.json.zst") {
-				sid := parts[1]
-				if !seen[sid] {
-					seen[sid] = true
-					snapshotIDs = append(snapshotIDs, sid)
+				key := parts[1]
+				if !seen[key] {
+					seen[key] = true
+					keys = append(keys, key)
 				}
 			}
 		}
 	}
 
-	return snapshotIDs, nil
+	return keys, nil
 }
 
 // handleRemoveAllDryRun handles the dry-run mode for removing all snapshots
-func (v *Vaultik) handleRemoveAllDryRun(snapshotIDs []string, opts *RemoveOptions) (*RemoveResult, error) {
-	result := &RemoveResult{
-		DryRun:           true,
-		SnapshotsRemoved: snapshotIDs,
+func (v *Vaultik) handleRemoveAllDryRun(localSnaps, orphanRemoteKeys []string, opts *RemoveOptions) (*RemoveResult, error) {
+	result := &RemoveResult{DryRun: true}
+	result.SnapshotsRemoved = append(result.SnapshotsRemoved, localSnaps...)
+	if opts.Remote {
+		result.SnapshotsRemoved = append(result.SnapshotsRemoved, orphanRemoteKeys...)
 	}
 	if !opts.JSON {
-		v.printfStdout("Would remove %d snapshot(s):\n", len(snapshotIDs))
-		for _, id := range snapshotIDs {
+		v.printfStdout("Would remove %d local snapshot(s):\n", len(localSnaps))
+		for _, id := range localSnaps {
 			v.printfStdout("  %s\n", id)
 		}
-		if opts.Remote {
+		if opts.Remote && len(orphanRemoteKeys) > 0 {
+			v.printfStdout("Would also remove %d orphan remote snapshot key(s):\n", len(orphanRemoteKeys))
+			for _, key := range orphanRemoteKeys {
+				v.printfStdout("  %s\n", key)
+			}
+		} else if opts.Remote {
 			v.printlnStdout("Would also remove from remote storage")
 		}
 		v.printlnStdout("[Dry run - no changes made]")
@@ -1181,17 +1200,19 @@ func (v *Vaultik) handleRemoveAllDryRun(snapshotIDs []string, opts *RemoveOption
 	return result, nil
 }
 
-// executeRemoveAll removes all snapshots from local database and optionally from remote storage
-func (v *Vaultik) executeRemoveAll(snapshotIDs []string, opts *RemoveOptions) (*RemoveResult, error) {
+// executeRemoveAll deletes every local snapshot (and, with --remote,
+// every corresponding remote metadata directory plus any orphan remote
+// keys that don't match a local snapshot).
+func (v *Vaultik) executeRemoveAll(localSnaps, orphanRemoteKeys []string, opts *RemoveOptions) (*RemoveResult, error) {
 	// --all requires --force
 	if !opts.Force {
 		return nil, fmt.Errorf("--all requires --force")
 	}
 
-	log.Info("Removing all snapshots", "count", len(snapshotIDs))
+	log.Info("Removing all snapshots", "local_count", len(localSnaps), "orphan_remote_count", len(orphanRemoteKeys))
 
 	result := &RemoveResult{}
-	for _, snapshotID := range snapshotIDs {
+	for _, snapshotID := range localSnaps {
 		log.Info("Removing snapshot", "snapshot_id", snapshotID)
 
 		if err := v.deleteSnapshotFromLocalDB(snapshotID); err != nil {
@@ -1200,7 +1221,7 @@ func (v *Vaultik) executeRemoveAll(snapshotIDs []string, opts *RemoveOptions) (*
 		}
 
 		if opts.Remote {
-			if err := v.deleteSnapshotFromRemote(snapshotID); err != nil {
+			if err := v.deleteRemoteSnapshotByKey(snapshot.RemoteSnapshotKey(snapshotID)); err != nil {
 				log.Error("Failed to remove from remote", "snapshot_id", snapshotID, "error", err)
 				continue
 			}
@@ -1209,10 +1230,29 @@ func (v *Vaultik) executeRemoveAll(snapshotIDs []string, opts *RemoveOptions) (*
 		result.SnapshotsRemoved = append(result.SnapshotsRemoved, snapshotID)
 	}
 
+	if opts.Remote {
+		for _, key := range orphanRemoteKeys {
+			log.Info("Removing orphan remote snapshot", "remote_key", key)
+			if err := v.deleteRemoteSnapshotByKey(key); err != nil {
+				log.Error("Failed to remove orphan from remote", "remote_key", key, "error", err)
+				continue
+			}
+			result.SnapshotsRemoved = append(result.SnapshotsRemoved, key)
+		}
+	}
+
 	if opts.Remote {
 		result.RemoteRemoved = true
 	}
 
+	// Clean up everything that just became orphaned locally so the
+	// index database doesn't carry 39k ghost rows after a wipe.
+	if v.SnapshotManager != nil {
+		if err := v.SnapshotManager.CleanupOrphanedData(v.ctx); err != nil {
+			log.Warn("Failed to clean up orphaned local data after bulk removal", "error", err)
+		}
+	}
+
 	if opts.JSON {
 		return result, v.outputRemoveJSON(result)
 	}
@@ -1220,7 +1260,7 @@ func (v *Vaultik) executeRemoveAll(snapshotIDs []string, opts *RemoveOptions) (*
 	v.printfStdout("Removed %d snapshot(s)\n", len(result.SnapshotsRemoved))
 	if opts.Remote {
 		v.printlnStdout("Removed snapshot metadata from remote storage")
-		v.printlnStdout("\nNote: Blobs were not removed. Run 'vaultik prune' to remove orphaned blobs.")
+		v.printlnStdout("\nNote: Remote blobs were not removed. Run 'vaultik prune' to remove orphaned blobs.")
 	}
 
 	return result, nil
@@ -1249,9 +1289,13 @@ func (v *Vaultik) deleteSnapshotFromLocalDB(snapshotID string) error {
 	return nil
 }
 
-// deleteSnapshotFromRemote removes snapshot metadata files from remote storage
-func (v *Vaultik) deleteSnapshotFromRemote(snapshotID string) error {
-	prefix := fmt.Sprintf("metadata/%s/", snapshotID)
+// deleteRemoteSnapshotByKey removes everything under
+// metadata/<remoteKey>/ on the destination store. The argument is a
+// remote key (double-SHA256 derivation), not a human snapshot ID;
+// callers that have a human ID must hash via snapshot.RemoteSnapshotKey
+// first.
+func (v *Vaultik) deleteRemoteSnapshotByKey(remoteKey string) error {
+	prefix := fmt.Sprintf("metadata/%s/", remoteKey)
 	objectCh := v.Storage.ListStream(v.ctx, prefix)
 
 	var objectsToDelete []string

internal/vaultik/verify.go

@@ -106,8 +106,11 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
 
 // loadVerificationData downloads manifest, database, and blob list for verification
 func (v *Vaultik) loadVerificationData(snapshotID string, opts *VerifyOptions, result *VerifyResult) (*snapshot.Manifest, *tempDB, []snapshot.BlobInfo, error) {
+	// All remote paths use the hashed key derived from the human ID.
+	remoteKey := snapshot.RemoteSnapshotKey(snapshotID)
+
 	// Download manifest
-	manifestPath := fmt.Sprintf("metadata/%s/manifest.json.zst", snapshotID)
+	manifestPath := fmt.Sprintf("metadata/%s/manifest.json.zst", remoteKey)
 	log.Info("Downloading manifest", "path", manifestPath)
 	if !opts.JSON {
 		v.printfStdout("Downloading manifest...\n")
@@ -136,7 +139,7 @@ func (v *Vaultik) loadVerificationData(snapshotID string, opts *VerifyOptions, r
 	}
 
 	// Download and decrypt database
-	dbPath := fmt.Sprintf("metadata/%s/db.zst.age", snapshotID)
+	dbPath := fmt.Sprintf("metadata/%s/db.zst.age", remoteKey)
 	log.Info("Downloading encrypted database", "path", dbPath)
 	dbReader, err := v.Storage.Get(v.ctx, dbPath)
 	if err != nil {