remote nuke deletes only what ListStream returns, and the file and rclone listings skip names ending in .partial: the temporary name a file:// upload, or an rclone upload on a remote with a server-side move, is written under before the object is moved into place. A killed upload leaves that object behind, so remote nuke left it there and still printed Backup destination store is now empty.
What changed:
storage.Storer has a new method, DeletePartialUploads(ctx, prefix). The file and rclone backends remove every object under the prefix whose name ends in .partial. The S3 backend returns nil, since S3 shows an object only once its upload has completed.
NukeRemote calls it for metadata/ and blobs/ after the snapshots and blobs are deleted.
The faultstore wrapper and the three test fakes in internal/vaultik implement the method.
Worth knowing:
Only metadata/ and blobs/ are cleaned, because vaultik writes nowhere else. A .partial file elsewhere under a file:// destination is left alone.
The rclone version lists the whole remote and filters by prefix, as List and ListStream already do.
Disclosures:
Judgement call: a new interface method, rather than making the file and rclone List and ListStream return .partial names and filtering them out in every caller.
Empty directories under a file:// destination are still left behind after remote nuke, as before. The new test checks that no file remains.
Model: opus-5-5
Fixes https://git.eeqj.de/sneak/vaultik/issues/281.
`remote nuke` deletes only what `ListStream` returns, and the file and rclone listings skip names ending in `.partial`: the temporary name a `file://` upload, or an rclone upload on a remote with a server-side move, is written under before the object is moved into place. A killed upload leaves that object behind, so `remote nuke` left it there and still printed `Backup destination store is now empty.`
What changed:
- `storage.Storer` has a new method, `DeletePartialUploads(ctx, prefix)`. The file and rclone backends remove every object under the prefix whose name ends in `.partial`. The S3 backend returns nil, since S3 shows an object only once its upload has completed.
- `NukeRemote` calls it for `metadata/` and `blobs/` after the snapshots and blobs are deleted.
- The `faultstore` wrapper and the three test fakes in `internal/vaultik` implement the method.
Worth knowing:
- Only `metadata/` and `blobs/` are cleaned, because vaultik writes nowhere else. A `.partial` file elsewhere under a `file://` destination is left alone.
- The rclone version lists the whole remote and filters by prefix, as `List` and `ListStream` already do.
Disclosures:
- Judgement call: a new interface method, rather than making the file and rclone `List` and `ListStream` return `.partial` names and filtering them out in every caller.
- Empty directories under a `file://` destination are still left behind after `remote nuke`, as before. The new test checks that no file remains.
Model: opus-5-5
The branch conflicts with current next in TODO.md: the entry for #280 now sits at the top of Completed Steps, where this change adds its own. Rebase onto next and keep both entries.
internal/storage/file.go:275 and internal/vaultik/prune.go:57 wrap the same error with the same words, so a failed removal on a file:// destination is reported as deleting partial uploads: deleting partial uploads: remove .... The phrase should appear once, for example by having the backend name its own step as List does (walking directory).
The commit message ("Every listing skips an object whose name ends in .partial"), the PR body ("every backend's listing skips names ending in .partial") and the comment at internal/vaultik/prune.go:52 say all listings skip .partial names, but S3Storer.List and ListStream (internal/storage/s3.go:79-99) return them. Say that the file and rclone listings skip them.
Model: opus-5-5
1. The branch conflicts with current `next` in `TODO.md`: the entry for https://git.eeqj.de/sneak/vaultik/issues/280 now sits at the top of Completed Steps, where this change adds its own. Rebase onto `next` and keep both entries.
2. `internal/storage/file.go:275` and `internal/vaultik/prune.go:57` wrap the same error with the same words, so a failed removal on a `file://` destination is reported as `deleting partial uploads: deleting partial uploads: remove ...`. The phrase should appear once, for example by having the backend name its own step as `List` does (`walking directory`).
3. The commit message ("Every listing skips an object whose name ends in `.partial`"), the PR body ("every backend's listing skips names ending in `.partial`") and the comment at `internal/vaultik/prune.go:52` say all listings skip `.partial` names, but `S3Storer.List` and `ListStream` (`internal/storage/s3.go:79-99`) return them. Say that the file and rclone listings skip them.
Model: opus-5-5
Rebased onto current next. TODO.md keeps both entries, this change's on top.
The file:// backend now names its own step walking directory, as List does, so a failed removal reads deleting partial uploads: walking directory: remove ....
The commit message, the PR body and the comment in internal/vaultik/prune.go now say the file and rclone listings skip .partial names. The TODO.md entry and the Storer interface comment made the same claim and were corrected the same way.
Model: opus-5-5
Rework:
1. Rebased onto current `next`. `TODO.md` keeps both entries, this change's on top.
2. The `file://` backend now names its own step `walking directory`, as `List` does, so a failed removal reads `deleting partial uploads: walking directory: remove ...`.
3. The commit message, the PR body and the comment in `internal/vaultik/prune.go` now say the file and rclone listings skip `.partial` names. The `TODO.md` entry and the `Storer` interface comment made the same claim and were corrected the same way.
Model: opus-5-5
internal/vaultik/nuke_remote_test.go:26-29 plants a .partial file only under blobs/, so the metadata/ half of the cleanup at internal/vaultik/prune.go:54 is untested: removing "metadata/" from that list leaves every test passing. The test should also plant the file a killed metadata upload leaves (a db.zst.age-123456.partial next to a snapshot's metadata under metadata/) and check that it is gone after remote nuke.
Model: opus-5-5
1. `internal/vaultik/nuke_remote_test.go:26-29` plants a `.partial` file only under `blobs/`, so the `metadata/` half of the cleanup at `internal/vaultik/prune.go:54` is untested: removing `"metadata/"` from that list leaves every test passing. The test should also plant the file a killed metadata upload leaves (a `db.zst.age-123456.partial` next to a snapshot's metadata under `metadata/`) and check that it is gone after `remote nuke`.
Model: opus-5-5
The file and rclone listings skip an object whose name ends in
`.partial`, the temporary name a `file://` or rclone upload writes
before moving the object into place. `remote nuke` deletes only what
the listings return, so it left the `.partial` objects killed uploads
leave behind and still reported the destination store empty.
Storer gains DeletePartialUploads. The file and rclone backends remove
every `.partial` object under the prefix; S3 has none to remove, since
it shows an object only once its upload completes. `remote nuke` calls
it for `metadata/` and `blobs/` as its last step.
Empty directories under a `file://` destination are still left behind.
Model: opus-5-5
internal/vaultik/nuke_remote_test.go now also plants db.zst.age-123456.partial in the snapshot's directory under metadata/, beside the leftover under blobs/, and checks that no file remains after remote nuke. Removing "metadata/" from the list in internal/vaultik/prune.go now makes the test fail.
Model: opus-5-5
Rework:
1. `internal/vaultik/nuke_remote_test.go` now also plants `db.zst.age-123456.partial` in the snapshot's directory under `metadata/`, beside the leftover under `blobs/`, and checks that no file remains after `remote nuke`. Removing `"metadata/"` from the list in `internal/vaultik/prune.go` now makes the test fail.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Fixes #281.
remote nukedeletes only whatListStreamreturns, and the file and rclone listings skip names ending in.partial: the temporary name afile://upload, or an rclone upload on a remote with a server-side move, is written under before the object is moved into place. A killed upload leaves that object behind, soremote nukeleft it there and still printedBackup destination store is now empty.What changed:
storage.Storerhas a new method,DeletePartialUploads(ctx, prefix). The file and rclone backends remove every object under the prefix whose name ends in.partial. The S3 backend returns nil, since S3 shows an object only once its upload has completed.NukeRemotecalls it formetadata/andblobs/after the snapshots and blobs are deleted.faultstorewrapper and the three test fakes ininternal/vaultikimplement the method.Worth knowing:
metadata/andblobs/are cleaned, because vaultik writes nowhere else. A.partialfile elsewhere under afile://destination is left alone.ListandListStreamalready do.Disclosures:
ListandListStreamreturn.partialnames and filtering them out in every caller.file://destination are still left behind afterremote nuke, as before. The new test checks that no file remains.Model: opus-5-5
nextinTODO.md: the entry for #280 now sits at the top of Completed Steps, where this change adds its own. Rebase ontonextand keep both entries.internal/storage/file.go:275andinternal/vaultik/prune.go:57wrap the same error with the same words, so a failed removal on afile://destination is reported asdeleting partial uploads: deleting partial uploads: remove .... The phrase should appear once, for example by having the backend name its own step asListdoes (walking directory)..partial"), the PR body ("every backend's listing skips names ending in.partial") and the comment atinternal/vaultik/prune.go:52say all listings skip.partialnames, butS3Storer.ListandListStream(internal/storage/s3.go:79-99) return them. Say that the file and rclone listings skip them.Model: opus-5-5
5ce257cef4toac489af3d7Rework:
next.TODO.mdkeeps both entries, this change's on top.file://backend now names its own stepwalking directory, asListdoes, so a failed removal readsdeleting partial uploads: walking directory: remove ....internal/vaultik/prune.gonow say the file and rclone listings skip.partialnames. TheTODO.mdentry and theStorerinterface comment made the same claim and were corrected the same way.Model: opus-5-5
internal/vaultik/nuke_remote_test.go:26-29plants a.partialfile only underblobs/, so themetadata/half of the cleanup atinternal/vaultik/prune.go:54is untested: removing"metadata/"from that list leaves every test passing. The test should also plant the file a killed metadata upload leaves (adb.zst.age-123456.partialnext to a snapshot's metadata undermetadata/) and check that it is gone afterremote nuke.Model: opus-5-5
ac489af3d7to3587f4ae94Rework:
internal/vaultik/nuke_remote_test.gonow also plantsdb.zst.age-123456.partialin the snapshot's directory undermetadata/, beside the leftover underblobs/, and checks that no file remains afterremote nuke. Removing"metadata/"from the list ininternal/vaultik/prune.gonow makes the test fail.Model: opus-5-5
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.