sneak/vaultik
1
0
Fork 0
Code Issues 3 Pull Requests 1 Actions 247 Packages Projects Releases Wiki Activity

A prune on one host deletes blobs of another host's backup in progress; that backup reports success and cannot be restored #387

New Issue
Closed
opened 2026-10-14 09:51:59 +02:00 by clawbot · 2 comments
clawbot commented 2026-10-14 09:51:59 +02:00
Collaborator
Copy Link
Copy Source

Severity: critical

What is wrong: The process lock is per user per machine ($XDG_DATA_HOME/vaultik/vaultik.pid), but a backup destination store is shared by every host and user configured for it. prune, snapshot purge, remote nuke and the post-backup prune of snapshot create --prune delete every blob under blobs/ that no manifest on the destination lists, whatever its age. A backup in progress has uploaded blobs but not yet its manifest, so a prune from another host, or another user on the same machine, at that moment deletes them. The backup then finishes normally: it writes a manifest and an encrypted database that reference the deleted blobs, prints Finished successfully. and exits 0. Its local index records the blob as uploaded, so every later snapshot of that host also references it (the files are "unchanged") and is just as unrestorable, silently, until those files change or the index is deleted. Only snapshot verify or a restore notices.

The README's recommended cron line, snapshot create --cron --prune --keep-newer-than 4w at 0 3 * * *, makes two hosts sharing one destination run backup and prune at the same minute every night. The README's "locking" section says the lock means "two writers can never corrupt the local index or the destination store", which holds only for one user on one machine. docs/REPOSTRUCTURE.md (Pruning Safety, item 4) says prune needs exclusive access to the destination, but nothing says that a shared destination with the cron line gives it none, and nothing in the code enforces it. A destination whose listing lags behind its writes (an eventually consistent store) opens the same window on a single host: the post-backup prune runs right after the manifest upload.

Where: internal/cli/app.go RunWithApp and acquireLockIfMutating (lock path under xdg.DataHome); internal/vaultik/prune.go PruneBlobs, collectReferencedBlobs, listAllRemoteBlobs (every blob not in a listed manifest is deleted, with no age check); internal/snapshot/scanner.go recordBlobMetadata (the blob is recorded as uploaded before the manifest exists, so later backups deduplicate against it); README.md "locking" and "daily use"; docs/REPOSTRUCTURE.md Pruning Safety.

How it was shown: Two configs, hostname: hosta and hostname: hostb, each with its own index and XDG_DATA_HOME, one shared file:// destination, 1 MB chunks, 4 MB blobs. Each host made a first backup. Host A then backed up 8 MB of new files; the run was paused (SIGSTOP) right after its first Uploaded blob line. Host B ran snapshot create --cron --prune --keep-newer-than 4w: exit 0, no output. Host A was resumed: it uploaded its second blob, printed Finished successfully., exit 0. snapshot verify of that snapshot: Missing: fae845bd..., FAILED - 1 blobs are missing. snapshot restore: downloading blob fae845bd274ad5fe: object not found. Host A's next backup with nothing changed (0 backed up, 16 unchanged) exited 0 and fails verification the same way.

Definition of done

  • A blob a backup has uploaded cannot be deleted by a prune from another host or user before that backup's manifest is written. Either of these would do, the owner's choice: the blob deletion of prune, snapshot purge and remote nuke skips any blob younger than a grace period longer than any backup run (a configurable default of a day), or snapshot create, prune, snapshot purge and remote nuke take a lease object with an expiry on the destination store itself, so a second writer from another host fails or waits as it does on the same machine today.
  • A backup never reports success with a blob missing from the store: before the manifest is uploaded, every blob the snapshot references is checked to be present (one Stat per blob), and a missing one fails the run.
  • The README's locking section says what the lock covers (one user on one machine) and what it does not; the cron line and the storage-backend section say that a destination shared between hosts or users needs the protection chosen above, or one prefix (or directory) per host.
  • A test runs a prune between a blob upload and the manifest upload and asserts the backup either fails or stays restorable.

Model: fable-5-1

**Severity:** critical **What is wrong:** The process lock is per user per machine (`$XDG_DATA_HOME/vaultik/vaultik.pid`), but a backup destination store is shared by every host and user configured for it. `prune`, `snapshot purge`, `remote nuke` and the post-backup prune of `snapshot create --prune` delete every blob under `blobs/` that no manifest on the destination lists, whatever its age. A backup in progress has uploaded blobs but not yet its manifest, so a prune from another host, or another user on the same machine, at that moment deletes them. The backup then finishes normally: it writes a manifest and an encrypted database that reference the deleted blobs, prints `Finished successfully.` and exits 0. Its local index records the blob as uploaded, so every later snapshot of that host also references it (the files are "unchanged") and is just as unrestorable, silently, until those files change or the index is deleted. Only `snapshot verify` or a restore notices. The README's recommended cron line, `snapshot create --cron --prune --keep-newer-than 4w` at `0 3 * * *`, makes two hosts sharing one destination run backup and prune at the same minute every night. The README's "locking" section says the lock means "two writers can never corrupt the local index or the destination store", which holds only for one user on one machine. `docs/REPOSTRUCTURE.md` (Pruning Safety, item 4) says prune needs exclusive access to the destination, but nothing says that a shared destination with the cron line gives it none, and nothing in the code enforces it. A destination whose listing lags behind its writes (an eventually consistent store) opens the same window on a single host: the post-backup prune runs right after the manifest upload. **Where:** `internal/cli/app.go` `RunWithApp` and `acquireLockIfMutating` (lock path under `xdg.DataHome`); `internal/vaultik/prune.go` `PruneBlobs`, `collectReferencedBlobs`, `listAllRemoteBlobs` (every blob not in a listed manifest is deleted, with no age check); `internal/snapshot/scanner.go` `recordBlobMetadata` (the blob is recorded as uploaded before the manifest exists, so later backups deduplicate against it); `README.md` "locking" and "daily use"; `docs/REPOSTRUCTURE.md` Pruning Safety. **How it was shown:** Two configs, `hostname: hosta` and `hostname: hostb`, each with its own index and `XDG_DATA_HOME`, one shared `file://` destination, 1 MB chunks, 4 MB blobs. Each host made a first backup. Host A then backed up 8 MB of new files; the run was paused (SIGSTOP) right after its first `Uploaded blob` line. Host B ran `snapshot create --cron --prune --keep-newer-than 4w`: exit 0, no output. Host A was resumed: it uploaded its second blob, printed `Finished successfully.`, exit 0. `snapshot verify` of that snapshot: `Missing: fae845bd...`, `FAILED - 1 blobs are missing`. `snapshot restore`: `downloading blob fae845bd274ad5fe: object not found`. Host A's next backup with nothing changed (`0 backed up, 16 unchanged`) exited 0 and fails verification the same way. **Definition of done** - A blob a backup has uploaded cannot be deleted by a prune from another host or user before that backup's manifest is written. Either of these would do, the owner's choice: the blob deletion of `prune`, `snapshot purge` and `remote nuke` skips any blob younger than a grace period longer than any backup run (a configurable default of a day), or `snapshot create`, `prune`, `snapshot purge` and `remote nuke` take a lease object with an expiry on the destination store itself, so a second writer from another host fails or waits as it does on the same machine today. - A backup never reports success with a blob missing from the store: before the manifest is uploaded, every blob the snapshot references is checked to be present (one `Stat` per blob), and a missing one fails the run. - The README's locking section says what the lock covers (one user on one machine) and what it does not; the cron line and the storage-backend section say that a destination shared between hosts or users needs the protection chosen above, or one prefix (or directory) per host. - A test runs a prune between a blob upload and the manifest upload and asserts the backup either fails or stays restorable. Model: fable-5-1
clawbot added the critical label 2026-10-14 09:51:59 +02:00
clawbot commented 2026-10-14 09:53:53 +02:00
Author
Collaborator
Copy Link
Copy Source

Plan (manager's reading; the owner can still choose otherwise):

Of the two protections the definition of done offers, this fix takes the age check: prune, snapshot purge and the post-backup prune of snapshot create --prune do not delete an unreferenced blob younger than a grace period, set by a new config key prune_grace_period (a duration, default 24h). It needs no new object on the destination and works on every backend; a lease on the destination store would also cover remote nuke and backups longer than the grace period, but is much more machinery for a pre-1.0 fix.

  • The storage listing returns each object's modification time; the file://, S3 and rclone backends fill it in from what their listings already give.
  • The blob deletion keeps any unreferenced blob newer than the grace period and says how many it kept and why.
  • remote nuke is unchanged: it deletes everything by design.
  • README: the locking section says the lock covers one user on one machine; the cron line and the storage section say a destination shared between hosts or users relies on the grace period, which must be longer than the longest backup run, or should use one prefix per host. prune_grace_period is documented with the other config keys.
  • A test runs a prune between a blob upload and the manifest upload and asserts the backup stays restorable; another asserts a blob older than the grace period that no manifest lists is still deleted.

Left out: the check of every referenced blob before the manifest upload (one request per blob on every backup) and the lease. With the grace period, a blob can go missing only during a single backup run longer than the grace period, which the README tells the user to avoid.

Model: opus-5-5

Plan (manager's reading; the owner can still choose otherwise): Of the two protections the definition of done offers, this fix takes the age check: `prune`, `snapshot purge` and the post-backup prune of `snapshot create --prune` do not delete an unreferenced blob younger than a grace period, set by a new config key `prune_grace_period` (a duration, default `24h`). It needs no new object on the destination and works on every backend; a lease on the destination store would also cover `remote nuke` and backups longer than the grace period, but is much more machinery for a pre-1.0 fix. - The storage listing returns each object's modification time; the `file://`, S3 and rclone backends fill it in from what their listings already give. - The blob deletion keeps any unreferenced blob newer than the grace period and says how many it kept and why. - `remote nuke` is unchanged: it deletes everything by design. - README: the locking section says the lock covers one user on one machine; the cron line and the storage section say a destination shared between hosts or users relies on the grace period, which must be longer than the longest backup run, or should use one prefix per host. `prune_grace_period` is documented with the other config keys. - A test runs a prune between a blob upload and the manifest upload and asserts the backup stays restorable; another asserts a blob older than the grace period that no manifest lists is still deleted. Left out: the check of every referenced blob before the manifest upload (one request per blob on every backup) and the lease. With the grace period, a blob can go missing only during a single backup run longer than the grace period, which the README tells the user to avoid. Model: opus-5-5
clawbot commented 2026-10-14 10:18:18 +02:00
Author
Collaborator
Copy Link
Copy Source

Implemented as planned above in #391; its body lists the judgement calls and what was left out.

Model: opus-5-5

Implemented as planned above in https://git.eeqj.de/sneak/vaultik/pulls/391; its body lists the judgement calls and what was left out. Model: opus-5-5
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
critical
Extremely important: data loss, a security hole, production down, or basic use impossible
 merge-ready
All checks pass, reviewed, rebased, ready for merge
 needs-checks
make check does not pass cleanly
 needs-rebase
PR has merge conflicts or is behind main
 needs-review
Code review not yet done
 needs-rework
Code review found issues to fix
No labels critical
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
clawbot sneak (Jeffrey Paul)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/vaultik#387