Interrupted metadata export leaves a snapshot silently treated as complete but unrestorable #177

New Issue

2026-09-22T09:37:46+02:00

clawbot commented

2026-09-22 09:37:46 +02:00

Split from issue #148 (its chosen option b) and scenario 2 of issue #72.

Issue #148 closed the interrupted blob upload window: a run now deduplicates only against chunks held by an uploaded blob, and drops un-uploaded blob rows at startup. Its option b also names a second window that #148 did not close and that this issue covers.

The window

A backup's final phase (finalizeSnapshotMetadata in internal/vaultik/snapshot.go) marks the snapshot complete in the local index (CompleteSnapshot -> MarkComplete, which sets completed_at) and only then exports the snapshot metadata to the destination (ExportSnapshotMetadata: the blob manifest and the encrypted database). If the run dies after MarkComplete but before or during that export -- the "blobs uploaded, manifest never written" state, i.e. scenario 2 of issue #72 (kill -9 between the manifest write and the db.zst.age upload) -- the local index shows the snapshot as complete while the destination holds no manifest or database for it.

On the next run, PruneDatabase deletes only snapshots whose completed_at is NULL, so this one survives and is treated as complete. snapshot list shows it as local-only "drift", but nothing detects or repairs it, and only snapshot verify against the destination would surface the missing metadata.

CleanupIncompleteSnapshots already performs the correct remote check (stat the metadata key; drop the record when it is absent), but only for snapshots still marked incomplete, and it is not on the backup path (PruneDatabase is used instead). Reconciling these two is part of this work.

Definition of done

A snapshot is never silently treated as complete when its metadata was never exported. Either:
- order completion so the snapshot is marked complete only after its metadata export succeeds (an interrupted export then leaves it incomplete and the next run's prune removes it); or
- at the start of a run, detect the "complete locally, no metadata at the destination" state and repair it (re-export or drop), with snapshot list and snapshot verify reporting it honestly rather than crashing or silently ignoring it.
A test injects a fault during the metadata export (blobs uploaded, manifest/database not written), re-runs the backup, and asserts the observable end state: the destination and the local index agree, and the snapshot either restores or is honestly reported as absent -- never silently listed as a good backup. This uses the fault-injection seam from issue #72 (unmerged as of this writing); coordinate with that branch.
make check green.

Scenario 2, verbatim from issue #72

kill -9 between manifest write and db.zst.age upload. This is the dangerous window -- the destination has blobs and a manifest but no index. Assert the next run detects and repairs the state, and that snapshot list and verify both report it honestly rather than crashing or silently ignoring it.

Model: opus-4-8

Split from [issue #148](https://git.eeqj.de/sneak/vaultik/issues/148) (its chosen option b) and scenario 2 of [issue #72](https://git.eeqj.de/sneak/vaultik/issues/72). [Issue #148](https://git.eeqj.de/sneak/vaultik/issues/148) closed the interrupted *blob upload* window: a run now deduplicates only against chunks held by an uploaded blob, and drops un-uploaded blob rows at startup. Its option b also names a second window that #148 did not close and that this issue covers. ## The window A backup's final phase (`finalizeSnapshotMetadata` in `internal/vaultik/snapshot.go`) marks the snapshot complete in the local index (`CompleteSnapshot` -> `MarkComplete`, which sets `completed_at`) and only then exports the snapshot metadata to the destination (`ExportSnapshotMetadata`: the blob manifest and the encrypted database). If the run dies after `MarkComplete` but before or during that export -- the "blobs uploaded, manifest never written" state, i.e. scenario 2 of [issue #72](https://git.eeqj.de/sneak/vaultik/issues/72) (kill -9 between the manifest write and the `db.zst.age` upload) -- the local index shows the snapshot as complete while the destination holds no manifest or database for it. On the next run, `PruneDatabase` deletes only snapshots whose `completed_at` is NULL, so this one survives and is treated as complete. `snapshot list` shows it as local-only "drift", but nothing detects or repairs it, and only `snapshot verify` against the destination would surface the missing metadata. `CleanupIncompleteSnapshots` already performs the correct remote check (stat the metadata key; drop the record when it is absent), but only for snapshots still marked incomplete, and it is not on the backup path (`PruneDatabase` is used instead). Reconciling these two is part of this work. ## Definition of done 1. A snapshot is never silently treated as complete when its metadata was never exported. Either: - order completion so the snapshot is marked complete only after its metadata export succeeds (an interrupted export then leaves it incomplete and the next run's prune removes it); or - at the start of a run, detect the "complete locally, no metadata at the destination" state and repair it (re-export or drop), with `snapshot list` and `snapshot verify` reporting it honestly rather than crashing or silently ignoring it. 2. A test injects a fault during the metadata export (blobs uploaded, manifest/database not written), re-runs the backup, and asserts the observable end state: the destination and the local index agree, and the snapshot either restores or is honestly reported as absent -- never silently listed as a good backup. This uses the fault-injection seam from [issue #72](https://git.eeqj.de/sneak/vaultik/issues/72) (unmerged as of this writing); coordinate with that branch. 3. `make check` green. ## Scenario 2, verbatim from issue #72 > kill -9 between manifest write and db.zst.age upload. This is the dangerous window -- the destination has blobs and a manifest but no index. Assert the next run detects and repairs the state, and that snapshot list and verify both report it honestly rather than crashing or silently ignoring it. Model: opus-4-8

clawbot referenced this issue

2026-09-22 09:38:06 +02:00

Interrupted blob upload leaves committed chunk rows; a retry silently produces an unrestorable snapshot #148

clawbot referenced this issue

2026-09-22 09:38:46 +02:00

Trust only uploaded blobs for deduplication (closes #148) #175

clawbot referenced this issue

2026-09-22 09:39:01 +02:00

Trust only uploaded blobs for deduplication (closes #148) #175

clawbot referenced this issue from a commit

2026-09-22 10:07:39 +02:00

Trust only uploaded blobs for deduplication (closes #148)