Delete metadata directories a backup left without a manifest (closes #390)

A backup that stopped between uploading db.zst.age and its manifest
left that directory under metadata/ on the destination for good:
prune deleted nothing there.

The blob step of prune, which snapshot create --prune also runs, now
deletes each directory under metadata/ that has no manifest.json.zst,
that no local snapshot record claims, and all of whose objects are
older than prune_grace_period; an object listed with no time keeps its
directory, and only names that are remote keys are deleted. The prompt
and the --force output name each such directory, the --json result
counts them (metadata_dirs_deleted, metadata_dirs_failed), and a failed
delete fails the run. Every description of the grace period now
covers these directories.

Model: opus-5-5

README.md

@@ -98,11 +98,12 @@ vaultik snapshot restore <snapshot-id> ~/vaultik-restore
 
 On a destination shared between hosts or users, the prune in that cron line
 relies on `prune_grace_period` (default `24h`) to leave the blobs of a backup
-another one is still running, so it must be longer than the longest backup
-run. It does not protect a blob that backup reuses from an earlier run that
-stopped before its manifest: the prune can delete such a blob, and the new
-snapshot then cannot be restored. To not depend on the grace period, give each
-host its own prefix or directory. See [locking](#locking).
+another one is still running, and its directory under `metadata/` that has no
+manifest yet, so it must be longer than the longest backup run. It does not
+protect a blob that backup reuses from an earlier run that stopped before its
+manifest: the prune can delete such a blob, and the new snapshot then cannot
+be restored. To not depend on the grace period, give each host its own prefix
+or directory. See [locking](#locking).
 
 ## restoring on another machine
 
@@ -204,18 +205,19 @@ corrupt the local index.
 
 The lock covers one user on one machine. It does not stop another user,
 or another host, from writing to the same destination store at the same
-time. A backup uploads its blobs before the manifest that lists them, so a
-prune from elsewhere at that moment finds blobs no snapshot lists. It
-deletes such a blob only once it is older than `prune_grace_period`
-(default `24h`), which therefore must be longer than the longest backup run
-of any host or user sharing the destination. The grace period does not
-protect a blob a backup reuses from an earlier run that stopped before its
-manifest: a backup skips the upload of a blob already on the destination, so
-that blob keeps the time of its first upload, and a prune from elsewhere
-during the backup can delete it; the new snapshot then cannot be restored.
-`remote nuke` deletes everything regardless. To not depend on the grace
-period, give each host or user its own prefix (or directory) on the
-destination.
+time. A backup uploads its blobs, and then its `db.zst.age`, before the
+manifest that lists them, so a prune from elsewhere at that moment finds
+blobs no snapshot lists and a directory under `metadata/` with no manifest. It
+deletes such a blob only once it is older than `prune_grace_period` (default
+`24h`), and such a directory only once every object in it is, so the grace
+period must be longer than the longest backup run of any host or user sharing
+the destination. The grace period does not protect a blob a backup reuses
+from an earlier run that stopped before its manifest: a backup skips the
+upload of a blob already on the destination, so that blob keeps the time of
+its first upload, and a prune from elsewhere during the backup can delete it;
+the new snapshot then cannot be restored. `remote nuke` deletes everything
+regardless. To not depend on the grace period, give each host or user its own
+prefix (or directory) on the destination.
 
 Read-only commands — `info`, `snapshot list`, `snapshot verify`, and
 `remote info` — do not take the lock and are never blocked, so they run
@@ -318,9 +320,12 @@ value where a list belongs or a word where a number belongs), is refused.
   error that explains how to fix it
 * `--cron`: Silent on total success; warnings and errors are still printed
   (for crontab)
-* `--prune`: After backup, drop older snapshots of each backed-up name and
+* `--prune`: After backup, drop older snapshots of each backed-up name,
   remove orphaned blobs older than `prune_grace_period` from remote storage,
-  as `prune` does. By default keeps only the latest
+  and delete each directory under `metadata/` that has no
+  `manifest.json.zst` once every object in it is older than
+  `prune_grace_period` and its snapshot is not in the local index, as `prune`
+  does. By default keeps only the latest
   snapshot per name; use `--keep-newer-than` for a rolling window. When
   `--skip-errors` skipped a file or directory it could not read or list, the
   prune does not run and every older snapshot is kept, since one of them may
@@ -453,22 +458,29 @@ may belong to a backup still running, on this host or another sharing the
 destination store, that has not yet uploaded its manifest. The grace period
 does not protect a blob such a backup reuses from an earlier run that stopped
 before its manifest: that blob keeps the time of its first upload, so it can
-be deleted. Step (3) reads all remote manifests — network cost scales with
-the number of snapshots.
+be deleted. Step (3) also deletes each directory under `metadata/` that has
+no `manifest.json.zst`, as a backup leaves when it stops after uploading
+`db.zst.age` but before its manifest, once every object in it is older than
+`prune_grace_period` and its snapshot is not in the local index. An object the
+listing gives no time for keeps its directory, and a directory whose name is
+not a remote key is left alone. Step (3) reads all remote manifests — network
+cost scales with the number of snapshots.
 `snapshot create --prune` runs the same cleanup automatically; this is the
 manual entry point for the same work. If the destination store cannot
 be listed, or the local index holds snapshots and the destination has
 no `vaultik-destination.txt` (an unplugged volume's empty mount point),
 prune fails and deletes nothing. `snapshot purge` does the same. If a
-blob cannot be deleted, the command (and `snapshot create --prune`)
-exits non-zero.
+blob or such a directory cannot be deleted, the command (and
+`snapshot create --prune`) exits non-zero.
 * `--force`: Skip confirmation prompt
 * `--json`: Output stats as JSON. Also skips the confirmation prompt, as
   `--force` does. A blob that cannot be deleted is counted in
   `blobs_failed`, and the document is printed before the non-zero exit.
   A blob kept for being newer than `prune_grace_period`, or because the
   listing of the destination gives no time for it, is counted in
-  `blobs_too_new` and not in `blobs_found`.
+  `blobs_too_new` and not in `blobs_found`. A directory under `metadata/`
+  with no manifest is counted in `metadata_dirs_deleted` once deleted, or
+  in `metadata_dirs_failed` if it cannot be.
 
 **`info`**: Display system configuration, storage settings, encryption
 recipients, and local database statistics.
@@ -545,8 +557,9 @@ and can be deleted.
 
 A destination shared between hosts or users, each with its own local index,
 relies on `prune_grace_period` to keep one's prune from deleting the blobs of
-another's backup still running, so it must be longer than the longest backup
-run. It does not protect a blob that backup reuses from an earlier run that
+another's backup still running, or its directory under `metadata/` that has
+no manifest yet, so it must be longer than the longest backup run. It does not
+protect a blob that backup reuses from an earlier run that
 stopped before its manifest, which the prune can delete. To not depend on the
 grace period, give each host its own prefix (or directory). See
 [locking](#locking).
@@ -639,6 +652,9 @@ format does and does not protect.
 3. List all blobs in storage
 4. Delete any blob not in the referenced set that is older than
    `prune_grace_period`
+5. Delete each directory under `metadata/` with no manifest whose objects
+   are all older than `prune_grace_period`, unless its snapshot is in the
+   local index
 
 ### chunking and deduplication
 
@@ -683,7 +699,7 @@ complete annotated example also lives in
 | `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
 | `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
 | `compression_level` | `3` | zstd compression level (1-19) |
-| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it. A backup uploads its blobs before the manifest that lists them, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs at once |
+| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. A backup uploads its blobs and its `db.zst.age` before the manifest that lists them, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once |
 | `hostname` | system hostname | Hostname used in snapshot IDs |
 | `index_path` | platform data dir | Local SQLite index path |
 

config.example.yml

@@ -329,10 +329,12 @@ storage_url: "rclone://myremote/path/to/backups"
 # Default: 3
 compression_level: 5
 
-# How old a blob that no snapshot lists must be before prune deletes it
-# A backup uploads its blobs before the manifest that lists them, so on a
-# destination shared between hosts or users this must be longer than the
-# longest backup run
+# How old a blob that no snapshot lists must be before prune deletes it, and
+# how old every object in a directory under metadata/ with no manifest must
+# be before prune deletes that directory
+# A backup uploads its blobs and its db.zst.age before the manifest that
+# lists them, so on a destination shared between hosts or users this must be
+# longer than the longest backup run
 # It does not protect a blob a backup reuses from an earlier run that stopped
 # before its manifest: that blob keeps the time of its first upload, so a
 # prune can delete it

docs/DATAMODEL.md

@@ -230,6 +230,10 @@ The restore process doesn't use the local database. Instead:
    - Query blobs not referenced by any remaining snapshot
    - Delete from S3, once older than `prune_grace_period`, and from the local
      database
+2. **Delete Metadata Directories With No Manifest**
+   - Delete each directory under `metadata/` that has no `manifest.json.zst`
+     and no local snapshot record, once every object in it is older than
+     `prune_grace_period`
 
 ### 6. Incomplete Snapshot Cleanup
 

docs/REPOSTRUCTURE.md

@@ -163,10 +163,10 @@ These are known, deliberate properties of the format and the tooling, recorded s
 ## Pruning Safety
 
 The prune operation is safe because:
-1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references
+1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references. Under `metadata/` it deletes only a directory that has no `manifest.json.zst`, whose name is a remote key, and that no local snapshot record claims
 2. Manifests are unencrypted and can be read without keys
 3. If any manifest cannot be downloaded or decoded, prune deletes nothing and exits with an error, rather than treating that snapshot's blobs as unreferenced
-4. It deletes an unreferenced blob only once the blob is older than `prune_grace_period` (default `24h`). A backup uploads its blobs before the manifest that lists them, so a prune running meanwhile, on this host or another sharing the destination, leaves them as long as no backup run takes longer than the grace period. The grace period does not protect a blob a backup reuses from an earlier run that stopped before its manifest: the backup skips the upload of a blob already on the destination, so that blob keeps the time of its first upload, and a prune before the backup's manifest is written can delete it; the new snapshot then cannot be restored. `remote nuke` has no such check
+4. It deletes an unreferenced blob only once the blob is older than `prune_grace_period` (default `24h`), and a directory under `metadata/` with no manifest only once every object in it is. A backup uploads its blobs and its `db.zst.age` before the manifest that lists them, so a prune running meanwhile, on this host or another sharing the destination, leaves them as long as no backup run takes longer than the grace period. The grace period does not protect a blob a backup reuses from an earlier run that stopped before its manifest: the backup skips the upload of a blob already on the destination, so that blob keeps the time of its first upload, and a prune before the backup's manifest is written can delete it; the new snapshot then cannot be restored. `remote nuke` has no such check
 
 ## Restoration Requirements
 

internal/cli/config.go

@@ -239,8 +239,10 @@ storage_url: ""
 # Default: 3
 # compression_level: 3
 
-# How old a blob that no snapshot lists must be before prune deletes it.
-# A backup uploads its blobs before the manifest that lists them, so on a
+# How old a blob that no snapshot lists must be before prune deletes it,
+# and how old every object in a directory under metadata/ with no manifest
+# must be before prune deletes that directory. A backup uploads its blobs
+# and its db.zst.age before the manifest that lists them, so on a
 # destination shared between hosts or users this must be longer than the
 # longest backup run. It does not protect a blob a backup reuses from an
 # earlier run that stopped before its manifest: that blob keeps the time of

internal/cli/prune.go

@@ -26,6 +26,10 @@ another sharing the destination store, that has not yet uploaded its
 manifest. The grace period does not protect a blob such a backup
 reuses from an earlier run that stopped before its manifest: that
 blob keeps the time of its first upload, so it can be deleted.
+Remote cleanup also deletes each directory under metadata/ with no
+manifest, as a backup that stopped before its manifest leaves, once
+all its objects are older than prune_grace_period and its snapshot is
+not in the local index.
 
 Snapshot create --prune runs the same cleanup automatically; this
 command is the manual entry point for the same work (e.g. after a

internal/cli/snapshot.go

@@ -131,9 +131,12 @@ on Linux), then /etc/vaultik/config.yml.`,
 	cmd.Flags().BoolVar(&opts.Cron, "cron", false,
 		"Run in cron mode (silent unless warning or error)")
 	cmd.Flags().BoolVar(&opts.Prune, "prune", false,
-		"After backup, drop older snapshots of the same name and remove "+
-			"orphaned blobs older than prune_grace_period (not after a "+
-			"backup that skipped unreadable files)")
+		"After backup, drop older snapshots of the same name, remove "+
+			"orphaned blobs older than prune_grace_period, and delete each "+
+			"directory under metadata/ with no manifest once all its "+
+			"objects are older than prune_grace_period and its snapshot is "+
+			"not in the local index (not after a backup that skipped "+
+			"unreadable files)")
 	cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
 		"With --prune: keep snapshots newer than this duration "+
 			"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+

internal/config/config.go

@@ -192,10 +192,12 @@ type Config struct {
 	CompressionLevel int                       `yaml:"compression_level"`
 
 	// PruneGracePeriod is how old a blob no manifest lists must be before
-	// a prune deletes it. A backup lists its blobs in a manifest only
-	// once it has uploaded them all, so a younger one may belong to a
-	// backup still running, on this host or another sharing the
-	// destination store.
+	// a prune deletes it, and how old every object in a directory under
+	// metadata/ with no manifest must be before a prune deletes that
+	// directory. A backup writes its manifest only once it has uploaded
+	// all its blobs and its db.zst.age, so a younger blob or directory may
+	// belong to a backup still running, on this host or another sharing
+	// the destination store.
 	PruneGracePeriod time.Duration `yaml:"prune_grace_period"`
 
 	// StorageURL specifies the storage backend using a URL format.

internal/vaultik/prune.go

@@ -4,10 +4,12 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
+	"sort"
 	"strings"
 	"time"
 
 	"sneak.berlin/go/vaultik/internal/log"
+	"sneak.berlin/go/vaultik/internal/snapshot"
 	"sneak.berlin/go/vaultik/internal/storage"
 )
 
@@ -24,6 +26,11 @@ var errNukeRequiresForce = errors.New(
 // errBlobDeleteFailed reports unreferenced blobs prune could not delete.
 var errBlobDeleteFailed = errors.New("blob(s) could not be deleted")
 
+// errMetadataDirDeleteFailed reports directories under metadata/ with no
+// manifest that prune could not delete.
+var errMetadataDirDeleteFailed = errors.New(
+	"metadata director(ies) with no manifest could not be deleted")
+
 // errObjectsLeft reports objects remote nuke still finds on the backup
 // destination store after deleting.
 var errObjectsLeft = errors.New(
@@ -31,7 +38,8 @@ var errObjectsLeft = errors.New(
 
 // nukeCommandHint is the command suggested when a remote nuke could not
 // delete everything: running it again deletes what is left, which
-// `vaultik prune` does not do for snapshot metadata.
+// `vaultik prune` does not do for snapshot metadata that has its
+// manifest.
 const nukeCommandHint = "vaultik remote nuke --force"
 
 // metadataDirName is the top-level remote directory holding snapshot
@@ -169,6 +177,11 @@ type PruneBlobsResult struct {
 	// newer than the prune grace period, or of unknown age. BlobsFound
 	// does not include them.
 	BlobsTooNew int `json:"blobs_too_new"`
+	// MetadataDirsDeleted counts the directories under metadata/ with no
+	// manifest that prune deleted, and MetadataDirsFailed those it could
+	// not delete.
+	MetadataDirsDeleted int `json:"metadata_dirs_deleted"`
+	MetadataDirsFailed  int `json:"metadata_dirs_failed,omitempty"`
 }
 
 // Prune removes orphaned data from the local index database AND
@@ -200,11 +213,17 @@ func (v *Vaultik) Prune(opts *PruneOptions) error {
 	return v.PruneBlobs(opts)
 }
 
-// PruneBlobs removes unreferenced blobs from storage
+// PruneBlobs removes unreferenced blobs from storage, and each directory
+// under metadata/ that a backup left with no manifest.
 func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
 	log.Info("Starting prune operation")
 
-	allBlobsReferenced, err := v.collectReferencedBlobs()
+	metadataObjects, err := v.listMetadataObjects()
+	if err != nil {
+		return err
+	}
+
+	allBlobsReferenced, err := v.collectReferencedBlobs(metadataObjects)
 	if err != nil {
 		return err
 	}
@@ -217,6 +236,11 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
 	unreferencedBlobs, totalSize, tooNew := v.findUnreferencedBlobs(
 		allBlobs, allBlobsReferenced)
 
+	dirsWithoutManifest, err := v.findMetadataDirsWithoutManifest(metadataObjects)
+	if err != nil {
+		return err
+	}
+
 	result := &PruneBlobsResult{
 		BlobsFound:  len(unreferencedBlobs),
 		BlobsTooNew: tooNew,
@@ -224,7 +248,7 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
 
 	v.reportBlobsTooNew(tooNew, opts)
 
-	if len(unreferencedBlobs) == 0 {
+	if len(unreferencedBlobs) == 0 && len(dirsWithoutManifest) == 0 {
 		log.Info("No unreferenced blobs found")
 
 		if opts.JSON {
@@ -237,38 +261,16 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
 	}
 
 	log.Info("Found unreferenced blobs",
-		"count", len(unreferencedBlobs), "total_size", ubytes(totalSize))
+		"count", len(unreferencedBlobs), "total_size", ubytes(totalSize),
+		"metadata_dirs_without_manifest", len(dirsWithoutManifest))
 
-	if opts.Force && !opts.JSON {
-		v.UI.Infof("Found %d unreferenced blob(s) totaling %s.",
-			len(unreferencedBlobs), ubytes(totalSize))
-	}
-
-	if !opts.Force && !opts.JSON {
-		// The count, the prompt and its answer bypass the UI: they are an
-		// interactive exchange the operator must see, so --quiet does not
-		// silence them.
-		v.stdoutf("Found %d unreferenced blob(s) totaling %s.\n",
-			len(unreferencedBlobs), ubytes(totalSize))
-		v.stdoutf("\nDelete %d unreferenced blob(s)? [y/N] ", len(unreferencedBlobs))
-
-		var confirm string
-
-		_, err = v.scanStdin(&confirm)
-		if err != nil {
-			v.printlnStdout("Cancelled")
-
-			return nil //nolint:nilerr // read failure means no confirmation
-		}
-
-		if strings.ToLower(confirm) != "y" {
-			v.printlnStdout("Cancelled")
-
-			return nil
-		}
+	if !v.confirmPrune(opts, len(unreferencedBlobs), totalSize,
+		dirsWithoutManifest) {
+		return nil
 	}
 
 	v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result)
+	v.deleteMetadataDirs(dirsWithoutManifest, result)
 
 	if opts.JSON {
 		err = v.outputPruneBlobsJSON(result)
@@ -276,33 +278,140 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
 			return err
 		}
 	} else {
+		v.reportPruneDeletions(result, len(dirsWithoutManifest))
+	}
+
+	var failures []error
+
+	if result.BlobsFailed > 0 {
+		failures = append(failures,
+			fmt.Errorf("%d %w", result.BlobsFailed, errBlobDeleteFailed))
+	}
+
+	if result.MetadataDirsFailed > 0 {
+		failures = append(failures, fmt.Errorf("%d %w",
+			result.MetadataDirsFailed, errMetadataDirDeleteFailed))
+	}
+
+	return errors.Join(failures...)
+}
+
+// confirmPrune states what prune found to delete, naming each directory
+// under metadata/ by its remote key, and, unless --force or --json is
+// given, asks the operator to confirm. It reports whether to go ahead.
+func (v *Vaultik) confirmPrune(
+	opts *PruneOptions, blobCount int, blobsSize int64, dirs []string,
+) bool {
+	if opts.JSON {
+		return true
+	}
+
+	dirCount := len(dirs)
+	blobsFound := fmt.Sprintf("Found %d unreferenced blob(s) totaling %s.",
+		blobCount, ubytes(blobsSize))
+	dirsFound := fmt.Sprintf("Found %d metadata director(ies) left by "+
+		"backups that stopped before their manifest:", dirCount)
+
+	if opts.Force {
+		if blobCount > 0 {
+			v.UI.Infof("%s", blobsFound)
+		}
+
+		if dirCount > 0 {
+			v.UI.Infof("%s", dirsFound)
+
+			for _, remoteKey := range dirs {
+				v.UI.Detailf("metadata/%s/", remoteKey)
+			}
+		}
+
+		return true
+	}
+
+	// The counts, the names, the prompt and its answer bypass the UI:
+	// they are an interactive exchange the operator must see, so --quiet
+	// does not silence them.
+	if blobCount > 0 {
+		v.stdoutf("%s\n", blobsFound)
+	}
+
+	if dirCount > 0 {
+		v.stdoutf("%s\n", dirsFound)
+
+		for _, remoteKey := range dirs {
+			v.stdoutf("  metadata/%s/\n", remoteKey)
+		}
+	}
+
+	switch {
+	case dirCount == 0:
+		v.stdoutf("\nDelete %d unreferenced blob(s)? [y/N] ", blobCount)
+	case blobCount == 0:
+		v.stdoutf("\nDelete %d metadata director(ies)? [y/N] ", dirCount)
+	default:
+		v.stdoutf("\nDelete %d unreferenced blob(s) and %d metadata "+
+			"director(ies)? [y/N] ", blobCount, dirCount)
+	}
+
+	var confirm string
+
+	// A failure to read the answer means no confirmation.
+	_, err := v.scanStdin(&confirm)
+	if err != nil || strings.ToLower(confirm) != "y" {
+		v.printlnStdout("Cancelled")
+
+		return false
+	}
+
+	return true
+}
+
+// reportPruneDeletions says how many blobs and metadata directories
+// prune deleted, naming each kind only when it found some to delete.
+func (v *Vaultik) reportPruneDeletions(result *PruneBlobsResult, dirCount int) {
+	if result.BlobsFound > 0 {
 		v.UI.Infof("Deleted %d blob(s) totaling %s.",
 			result.BlobsDeleted, ubytes(result.BytesFreed))
 	}
 
-	if result.BlobsFailed > 0 {
-		return fmt.Errorf("%d %w", result.BlobsFailed, errBlobDeleteFailed)
+	if dirCount > 0 {
+		v.UI.Infof("Deleted %d metadata director(ies) with no manifest.",
+			result.MetadataDirsDeleted)
 	}
-
-	return nil
 }
 
-// collectReferencedBlobs downloads all manifests and returns the set of
-// referenced blob hashes.
+// listMetadataObjects lists every object under metadata/ on the
+// destination store.
+func (v *Vaultik) listMetadataObjects() ([]storage.ObjectInfo, error) {
+	log.Info("Listing remote snapshots")
+
+	var objects []storage.ObjectInfo
+
+	for object := range v.Storage.ListStream(v.ctx, "metadata/") {
+		if object.Err != nil {
+			return nil, fmt.Errorf("listing metadata objects: %w", object.Err)
+		}
+
+		objects = append(objects, object)
+	}
+
+	return objects, nil
+}
+
+// collectReferencedBlobs downloads every manifest among the listed
+// metadata objects and returns the set of referenced blob hashes.
 //
 // Every manifest must be read successfully. A manifest that cannot be
 // downloaded or decoded means its snapshot's blobs are unknown, so
 // treating them as unreferenced would let prune delete data a snapshot
 // still needs. Rather than risk that silent loss, any failure returns an
 // error naming the remote key and prune deletes nothing.
-func (v *Vaultik) collectReferencedBlobs() (map[string]bool, error) {
-	log.Info("Listing remote snapshots")
-	// IDs returned by listUniqueSnapshotIDs are remote keys (hashed
+func (v *Vaultik) collectReferencedBlobs(
+	metadataObjects []storage.ObjectInfo,
+) (map[string]bool, error) {
+	// IDs returned by uniqueSnapshotIDs are remote keys (hashed
 	// subdirectories under metadata/), not human snapshot IDs.
-	remoteKeys, err := v.listUniqueSnapshotIDs()
-	if err != nil {
-		return nil, fmt.Errorf("listing snapshot keys: %w", err)
-	}
+	remoteKeys := uniqueSnapshotIDs(metadataObjects)
 
 	log.Info("Found manifests in remote storage", "count", len(remoteKeys))
 
@@ -327,18 +436,14 @@ func (v *Vaultik) collectReferencedBlobs() (map[string]bool, error) {
 	return allBlobsReferenced, nil
 }
 
-// listUniqueSnapshotIDs returns deduplicated snapshot IDs from remote metadata
-func (v *Vaultik) listUniqueSnapshotIDs() ([]string, error) {
-	objectCh := v.Storage.ListStream(v.ctx, "metadata/")
+// uniqueSnapshotIDs returns deduplicated snapshot IDs from the listed
+// metadata objects.
+func uniqueSnapshotIDs(metadataObjects []storage.ObjectInfo) []string {
 	seen := make(map[string]bool)
 
 	var snapshotIDs []string
 
-	for object := range objectCh {
-		if object.Err != nil {
-			return nil, fmt.Errorf("listing metadata objects: %w", object.Err)
-		}
-
+	for _, object := range metadataObjects {
 		parts := strings.Split(object.Key, "/")
 		if len(parts) >= minSnapshotIDParts &&
 			parts[0] == metadataDirName && parts[1] != "" {
@@ -353,7 +458,7 @@ func (v *Vaultik) listUniqueSnapshotIDs() ([]string, error) {
 		}
 	}
 
-	return snapshotIDs, nil
+	return snapshotIDs
 }
 
 // listAllRemoteBlobs returns a map of all blob hashes in remote storage
@@ -427,6 +532,82 @@ func (v *Vaultik) findUnreferencedBlobs(
 	return unreferenced, totalSize, tooNew
 }
 
+// findMetadataDirsWithoutManifest returns the remote key of each
+// directory under metadata/ that has no manifest.json.zst, such as one a
+// backup leaves when its manifest upload fails after db.zst.age was
+// uploaded. It leaves out a directory a local snapshot record claims,
+// and one holding an object newer than the prune grace period, or of
+// unknown age because the listing gave no time for it: a backup still
+// running may be writing there. A name that is not a remote key is
+// never returned.
+func (v *Vaultik) findMetadataDirsWithoutManifest(
+	metadataObjects []storage.ObjectInfo,
+) ([]string, error) {
+	localIDs, err := v.localSnapshotIDs()
+	if err != nil {
+		return nil, fmt.Errorf("listing local snapshots: %w", err)
+	}
+
+	keep := make(map[string]bool, len(localIDs))
+	for _, id := range localIDs {
+		keep[snapshot.RemoteSnapshotKey(id)] = true
+	}
+
+	cutoff := time.Now().UTC().Add(-v.Config.PruneGracePeriod)
+	listed := make(map[string]bool)
+
+	for _, object := range metadataObjects {
+		parts := strings.Split(object.Key, "/")
+		if len(parts) < metadataKeyParts || !isBlobHash(parts[1]) {
+			continue
+		}
+
+		remoteKey := parts[1]
+		listed[remoteKey] = true
+
+		if parts[2] == "manifest.json.zst" ||
+			object.ModTime.IsZero() || object.ModTime.After(cutoff) {
+			keep[remoteKey] = true
+		}
+	}
+
+	var remoteKeys []string
+
+	for remoteKey := range listed {
+		if !keep[remoteKey] {
+			remoteKeys = append(remoteKeys, remoteKey)
+		}
+	}
+
+	sort.Strings(remoteKeys)
+
+	return remoteKeys, nil
+}
+
+// deleteMetadataDirs deletes everything in the given directories under
+// metadata/ and counts in the result the directories deleted and those
+// it could not delete.
+func (v *Vaultik) deleteMetadataDirs(
+	remoteKeys []string, result *PruneBlobsResult,
+) {
+	for _, remoteKey := range remoteKeys {
+		_, err := v.deleteRemoteSnapshotByKey(remoteKey)
+		if err != nil {
+			log.Error("Failed to delete metadata directory with no manifest",
+				"remote_key", remoteKey, "error", err)
+
+			result.MetadataDirsFailed++
+
+			continue
+		}
+
+		log.Info("Deleted metadata directory with no manifest",
+			"remote_key", remoteKey)
+
+		result.MetadataDirsDeleted++
+	}
+}
+
 // reportBlobsTooNew says how many unreferenced blobs prune kept for being
 // newer than the grace period or of unknown age. Under --json the
 // document carries the count instead.

internal/vaultik/prune_metadata_dir_test.go

@@ -0,0 +1,225 @@
+package vaultik_test
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"sneak.berlin/go/vaultik/internal/config"
+	"sneak.berlin/go/vaultik/internal/log"
+	"sneak.berlin/go/vaultik/internal/snapshot"
+	"sneak.berlin/go/vaultik/internal/storage"
+	"sneak.berlin/go/vaultik/internal/storage/faultstore"
+	"sneak.berlin/go/vaultik/internal/ui"
+	"sneak.berlin/go/vaultik/internal/vaultik"
+)
+
+// metadataObjectKey returns the key of the named file in the metadata
+// directory of the snapshot with the given human ID.
+func metadataObjectKey(snapshotID, name string) string {
+	return "metadata/" + snapshot.RemoteSnapshotKey(snapshotID) + "/" + name
+}
+
+// TestPruneDeletesMetadataDirWithoutManifestOlderThanGracePeriod checks
+// that prune deletes a directory under metadata/ holding only
+// db.zst.age, as a backup leaves when its manifest upload fails, once it
+// is older than the grace period, and keeps one newer than that. A
+// directory with a manifest is never deleted, however old.
+func TestPruneDeletesMetadataDirWithoutManifestOlderThanGracePeriod(
+	t *testing.T,
+) {
+	log.Initialize(log.Config{})
+	t.Parallel()
+
+	ctx := context.Background()
+	storeDir := t.TempDir()
+
+	store, err := storage.NewFileStorer(storeDir)
+	require.NoError(t, err)
+
+	const (
+		oldID      = "testhost_home_2026-01-01T00:00:00Z"
+		newID      = "testhost_home_2026-01-02T00:00:00Z"
+		completeID = "testhost_home_2026-01-03T00:00:00Z"
+	)
+
+	oldDB := metadataObjectKey(oldID, "db.zst.age")
+	newDB := metadataObjectKey(newID, "db.zst.age")
+	completeDB := metadataObjectKey(completeID, "db.zst.age")
+	completeManifest := metadataObjectKey(completeID, "manifest.json.zst")
+
+	manifest, err := snapshot.EncodeManifest(&snapshot.Manifest{
+		SnapshotID: snapshot.RemoteSnapshotKey(completeID),
+	}, 3)
+	require.NoError(t, err)
+
+	for _, key := range []string{oldDB, newDB, completeDB} {
+		require.NoError(t, store.Put(ctx, key, strings.NewReader("db")))
+	}
+
+	require.NoError(t, store.Put(ctx, completeManifest,
+		bytes.NewReader(manifest)))
+
+	gracePeriod := time.Hour
+	beforeGracePeriod := time.Now().Add(-gracePeriod - time.Minute)
+
+	for _, key := range []string{oldDB, completeDB, completeManifest} {
+		require.NoError(t, os.Chtimes(filepath.Join(storeDir, key),
+			beforeGracePeriod, beforeGracePeriod))
+	}
+
+	stdout := &bytes.Buffer{}
+	v := &vaultik.Vaultik{
+		Config:  &config.Config{PruneGracePeriod: gracePeriod},
+		Storage: store,
+		Stdout:  stdout,
+		UI:      ui.NewWithColor(stdout, false),
+	}
+	v.SetContext(ctx)
+
+	require.NoError(t, v.PruneBlobs(&vaultik.PruneOptions{JSON: true}))
+
+	_, err = store.Stat(ctx, oldDB)
+	require.ErrorIs(t, err, storage.ErrNotFound,
+		"a directory with no manifest older than the grace period "+
+			"must be deleted")
+
+	for _, key := range []string{newDB, completeDB, completeManifest} {
+		_, err = store.Stat(ctx, key)
+		require.NoError(t, err, "%s must be kept", key)
+	}
+
+	var result vaultik.PruneBlobsResult
+
+	require.NoError(t, json.Unmarshal(stdout.Bytes(), &result))
+	assert.Equal(t, 1, result.MetadataDirsDeleted)
+}
+
+// TestPruneKeepsMetadataDirClaimedUnknownAgeOrNotRemoteKey checks that
+// prune --force keeps a directory under metadata/ with no manifest when a
+// local snapshot record claims it, when the listing gives no time for an
+// object in it, or when its name is not a remote key. A fourth such
+// directory is named, deleted and reported, although there are no
+// unreferenced blobs to delete.
+func TestPruneKeepsMetadataDirClaimedUnknownAgeOrNotRemoteKey(t *testing.T) {
+	log.Initialize(log.Config{})
+	t.Parallel()
+
+	ctx := context.Background()
+	env := newListEnv(t)
+
+	const (
+		claimedID = "testhost_home_2026-01-01T00:00:00Z"
+		leftID    = "testhost_home_2026-01-03T00:00:00Z"
+	)
+
+	env.addLocal(t, claimedID, time.Now().Add(-time.Hour))
+
+	claimedDB := metadataObjectKey(claimedID, "db.zst.age")
+	unknownAgeDB := metadataObjectKey("testhost_home_2026-01-02T00:00:00Z",
+		"db.zst.age")
+	leftDB := metadataObjectKey(leftID, "db.zst.age")
+	notRemoteKeyDB := "metadata/not-a-remote-key/db.zst.age"
+
+	for _, key := range []string{claimedDB, leftDB, notRemoteKeyDB} {
+		require.NoError(t, env.store.Put(ctx, key, strings.NewReader("db")))
+	}
+
+	// Written into the mock store's map directly rather than through
+	// Put, so its listing gives no time.
+	env.store.data[unknownAgeDB] = []byte("db")
+
+	require.NoError(t, env.v.PruneBlobs(&vaultik.PruneOptions{Force: true}))
+
+	assert.True(t, env.store.hasKey(claimedDB),
+		"a directory a local snapshot record claims must be kept")
+	assert.True(t, env.store.hasKey(unknownAgeDB),
+		"a directory holding an object of unknown age must be kept")
+	assert.True(t, env.store.hasKey(notRemoteKeyDB),
+		"a directory whose name is not a remote key must be kept")
+	assert.False(t, env.store.hasKey(leftDB),
+		"a directory with no manifest, unclaimed and old, must be deleted")
+
+	output := env.stdout.String()
+
+	assert.Contains(t, output, "Found 1 metadata director(ies) left by "+
+		"backups that stopped before their manifest:")
+	assert.Contains(t, output,
+		"metadata/"+snapshot.RemoteSnapshotKey(leftID)+"/")
+	assert.NotContains(t, output, "not-a-remote-key")
+	assert.Contains(t, output,
+		"Deleted 1 metadata director(ies) with no manifest.")
+	assert.NotContains(t, output, "No unreferenced blobs to remove.")
+}
+
+// TestPruneDeclinedKeepsMetadataDirWithoutManifest checks that the prompt
+// names a directory under metadata/ with no manifest that prune is about
+// to delete, and that answering no keeps it.
+func TestPruneDeclinedKeepsMetadataDirWithoutManifest(t *testing.T) {
+	log.Initialize(log.Config{})
+	t.Parallel()
+
+	env := newListEnv(t)
+
+	const leftID = "testhost_home_2026-01-01T00:00:00Z"
+
+	leftDB := metadataObjectKey(leftID, "db.zst.age")
+	require.NoError(t, env.store.Put(context.Background(), leftDB,
+		strings.NewReader("db")))
+
+	env.v.Stdin = strings.NewReader("n\n")
+
+	require.NoError(t, env.v.PruneBlobs(&vaultik.PruneOptions{}))
+
+	assert.True(t, env.store.hasKey(leftDB),
+		"a declined prune must keep the directory")
+
+	output := env.stdout.String()
+
+	assert.Contains(t, output,
+		"  metadata/"+snapshot.RemoteSnapshotKey(leftID)+"/\n")
+	assert.Contains(t, output, "Delete 1 metadata director(ies)? [y/N] ")
+	assert.Contains(t, output, "Cancelled")
+	assert.NotContains(t, output, "Deleted")
+}
+
+// TestPruneFailedMetadataDirDeleteReturnsError checks that prune --json
+// counts a directory under metadata/ with no manifest that it cannot
+// delete, and then returns an error, so the command exits non-zero.
+func TestPruneFailedMetadataDirDeleteReturnsError(t *testing.T) {
+	log.Initialize(log.Config{})
+	t.Parallel()
+
+	env := newListEnv(t)
+
+	leftDB := metadataObjectKey("testhost_home_2026-01-01T00:00:00Z",
+		"db.zst.age")
+	require.NoError(t, env.store.Put(context.Background(), leftDB,
+		strings.NewReader("db")))
+
+	store := faultstore.New(env.store)
+	store.OnDelete = func(key string) bool {
+		return strings.HasPrefix(key, "metadata/")
+	}
+	env.v.Storage = store
+
+	err := env.v.PruneBlobs(&vaultik.PruneOptions{JSON: true})
+
+	require.ErrorContains(t, err,
+		"1 metadata director(ies) with no manifest could not be deleted")
+	assert.True(t, env.store.hasKey(leftDB))
+
+	var result vaultik.PruneBlobsResult
+
+	require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &result),
+		"stdout must hold the JSON document, got:\n%s", env.stdout.String())
+	assert.Equal(t, 1, result.MetadataDirsFailed)
+	assert.Equal(t, 0, result.MetadataDirsDeleted)
+}

internal/vaultik/snapshot.go

@@ -1301,7 +1301,7 @@ const pruneCommandHint = "vaultik prune"
 // snapshotRemoveCommandHint is the command suggested, with the
 // snapshot's ID, when a remove could not reach the destination store:
 // running it again removes the snapshot's metadata there, which
-// `vaultik prune` never does.
+// `vaultik prune` never does while the manifest is there.
 const snapshotRemoveCommandHint = "vaultik snapshot remove"
 
 // RemoveSnapshot removes a snapshot from the local index database and,
@@ -1504,7 +1504,8 @@ func (v *Vaultik) confirmRemoveSnapshot(snapshotID string, opts *RemoveOptions)
 // the local index, its removal there has already happened, so a failure
 // is only a warning telling the user to run `vaultik snapshot remove`
 // for the snapshot again once the destination store is reachable
-// (`vaultik prune` never removes snapshot metadata).
+// (`vaultik prune` never removes snapshot metadata that has its
+// manifest).
 func (v *Vaultik) removeSnapshotRemote(
 	snapshotID string, inLocalIndex bool, markerErr error,
 	opts *RemoveOptions,