Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
635d2c27a8 |
@@ -349,11 +349,9 @@ the backup destination store. Blobs are NOT touched — deleting blobs
|
|||||||
requires reading every remaining remote manifest (the destination store
|
requires reading every remaining remote manifest (the destination store
|
||||||
may hold snapshots this host doesn't know about), which is what
|
may hold snapshots this host doesn't know about), which is what
|
||||||
`vaultik prune` does. On success the command prints the exact `vaultik
|
`vaultik prune` does. On success the command prints the exact `vaultik
|
||||||
prune` invocation to run as a follow-up. An ID that names no snapshot in
|
prune` invocation to run as a follow-up. Local row cleanup (files,
|
||||||
the local index or on the destination store is an error; with
|
chunks, blobs the snapshot was the last referrer for) runs
|
||||||
`--local-only`, the snapshot must be in the local index. Local row
|
automatically. If the destination store is unreachable, the local-DB
|
||||||
cleanup (files, chunks, blobs the snapshot was the last referrer for)
|
|
||||||
runs automatically. If the destination store is unreachable, the local-DB
|
|
||||||
removal still completes and a warning is emitted; run `vaultik snapshot
|
removal still completes and a warning is emitted; run `vaultik snapshot
|
||||||
remove <snapshot-id>` again once the store is reachable to remove the
|
remove <snapshot-id>` again once the store is reachable to remove the
|
||||||
snapshot's metadata from it (`vaultik prune` does not). To wipe everything
|
snapshot's metadata from it (`vaultik prune` does not). To wipe everything
|
||||||
@@ -384,10 +382,12 @@ manifest on the destination store to compute the still-referenced blob
|
|||||||
set and delete any blob not in that set. Step (3) reads all remote
|
set and delete any blob not in that set. Step (3) reads all remote
|
||||||
manifests — network cost scales with the number of snapshots. `snapshot
|
manifests — network cost scales with the number of snapshots. `snapshot
|
||||||
create --prune` runs the same cleanup automatically; this is the
|
create --prune` runs the same cleanup automatically; this is the
|
||||||
manual entry point for the same work.
|
manual entry point for the same work. If a blob cannot be deleted, the
|
||||||
|
command (and `snapshot create --prune`) exits non-zero.
|
||||||
* `--force`: Skip confirmation prompt
|
* `--force`: Skip confirmation prompt
|
||||||
* `--json`: Output stats as JSON. Also skips the confirmation prompt, as
|
* `--json`: Output stats as JSON. Also skips the confirmation prompt, as
|
||||||
`--force` does.
|
`--force` does. A blob that cannot be deleted is counted in
|
||||||
|
`blobs_failed`, and the document is printed before the non-zero exit.
|
||||||
|
|
||||||
**`info`**: Display system configuration, storage settings, encryption
|
**`info`**: Display system configuration, storage settings, encryption
|
||||||
recipients, and local database statistics.
|
recipients, and local database statistics.
|
||||||
@@ -407,7 +407,8 @@ snapshot's blob count and blob size unknown, `null` in `--json`.
|
|||||||
**`remote nuke`**: Delete every snapshot's metadata and every blob from the
|
**`remote nuke`**: Delete every snapshot's metadata and every blob from the
|
||||||
backup destination store, leaving the bucket prefix empty. Destructive and
|
backup destination store, leaving the bucket prefix empty. Destructive and
|
||||||
irreversible. This is the single supported way to wipe the entire
|
irreversible. This is the single supported way to wipe the entire
|
||||||
destination store.
|
destination store. If anything cannot be deleted, the command exits
|
||||||
|
non-zero; running it again deletes what is left.
|
||||||
* `--force`: Required to confirm destruction.
|
* `--force`: Required to confirm destruction.
|
||||||
|
|
||||||
**`database delete`**: Delete the local SQLite state database file
|
**`database delete`**: Delete the local SQLite state database file
|
||||||
|
|||||||
@@ -22,15 +22,16 @@ the tag exists and is exercised; what is left is merging `next` to
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-10-13: Made `snapshot remove` fail for a snapshot that does not
|
- 2026-10-13: Made `prune` and `remote nuke` exit non-zero when a delete
|
||||||
exist ([issue #293](https://git.eeqj.de/sneak/vaultik/issues/293)).
|
fails ([issue #290](https://git.eeqj.de/sneak/vaultik/issues/290)).
|
||||||
The command never checked the ID: it reported the snapshot removed
|
`prune`, and the prune step of `snapshot create --prune`, counted a
|
||||||
from the local database and from the destination store and exited 0,
|
blob it could not delete and still exited 0; it now prints its summary
|
||||||
under `--json`, `--dry-run` and `--local-only` too. An ID that names
|
or `--json` document and then fails. `remote nuke` warned about a
|
||||||
no snapshot in the local index or on the destination store is now an
|
snapshot's metadata it could not delete, said to run `vaultik prune`,
|
||||||
error; with `--local-only`, the snapshot must be in the local index.
|
which never removes metadata, and then reported the store empty. It
|
||||||
The command now reports a removal from either side only when one
|
now fails without that report and says to run `vaultik remote nuke
|
||||||
happened, and `remote_removed` is true only when metadata was deleted.
|
--force` again, and a successful nuke no longer advises `prune` for
|
||||||
|
blobs it deletes itself.
|
||||||
|
|
||||||
- 2026-10-12: Bound the local index to the destination an `s3.*` config
|
- 2026-10-12: Bound the local index to the destination an `s3.*` config
|
||||||
names ([issue #298](https://git.eeqj.de/sneak/vaultik/issues/298)).
|
names ([issue #298](https://git.eeqj.de/sneak/vaultik/issues/298)).
|
||||||
|
|||||||
@@ -255,12 +255,8 @@ manifest (the destination store may hold snapshots this host doesn't
|
|||||||
know about), which is what 'vaultik prune' does. On success the command
|
know about), which is what 'vaultik prune' does. On success the command
|
||||||
prints the exact 'vaultik prune' invocation to run as a follow-up.
|
prints the exact 'vaultik prune' invocation to run as a follow-up.
|
||||||
|
|
||||||
An ID that names no snapshot in the local index or on the destination
|
|
||||||
store is an error.
|
|
||||||
|
|
||||||
Use --local-only to skip the remote half (e.g. when you want to forget a
|
Use --local-only to skip the remote half (e.g. when you want to forget a
|
||||||
snapshot locally without touching the destination store); the snapshot
|
snapshot locally without touching the destination store).
|
||||||
must then be in the local index.
|
|
||||||
|
|
||||||
If the remote is unreachable, the local-database removal still completes
|
If the remote is unreachable, the local-database removal still completes
|
||||||
and a warning is emitted; run 'vaultik snapshot remove <snapshot-id>' again
|
and a warning is emitted; run 'vaultik snapshot remove <snapshot-id>' again
|
||||||
|
|||||||
@@ -1,15 +1,16 @@
|
|||||||
// Package faultstore provides a storage.Storer wrapper that injects
|
// Package faultstore provides a storage.Storer wrapper that injects
|
||||||
// faults on demand, so tests can reproduce the failure modes a real
|
// faults on demand, so tests can reproduce the failure modes a real
|
||||||
// backend exhibits: an upload that fails partway, a backend that reports
|
// backend exhibits: an upload that fails partway, a backend that reports
|
||||||
// success while storing nothing, and reads that return corrupt or
|
// success while storing nothing, reads that return corrupt or truncated
|
||||||
// truncated bytes. It is the seam called for by the fault-injection
|
// bytes, and a delete the backend refuses. It is the seam called for by
|
||||||
// tests (sneak/vaultik issue 72) and is meant to be reused by future
|
// the fault-injection tests (sneak/vaultik issue 72) and is meant to be
|
||||||
// tests rather than re-implemented per case.
|
// reused by future tests rather than re-implemented per case.
|
||||||
//
|
//
|
||||||
// The wrapper delegates every method to the inner Storer. Two hooks
|
// The wrapper delegates every method to the inner Storer. Three hooks
|
||||||
// change that: OnPut decides the fate of each write, and OnGet decides
|
// change that: OnPut decides the fate of each write, OnGet decides how
|
||||||
// how each read's bytes are returned. Both are keyed by the object key,
|
// each read's bytes are returned, and OnDelete decides whether each
|
||||||
// so a test can fault only blobs, only metadata, or a single object.
|
// delete fails. All are keyed by the object key, so a test can fault
|
||||||
|
// only blobs, only metadata, or a single object.
|
||||||
package faultstore
|
package faultstore
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -25,6 +26,10 @@ import (
|
|||||||
// ErrInjectedUpload is returned by a Put the OnPut hook chose to fail.
|
// ErrInjectedUpload is returned by a Put the OnPut hook chose to fail.
|
||||||
var ErrInjectedUpload = errors.New("faultstore: injected upload failure")
|
var ErrInjectedUpload = errors.New("faultstore: injected upload failure")
|
||||||
|
|
||||||
|
// ErrInjectedDelete is returned by a Delete the OnDelete hook chose to
|
||||||
|
// fail.
|
||||||
|
var ErrInjectedDelete = errors.New("faultstore: injected delete failure")
|
||||||
|
|
||||||
// PutAction is the disposition OnPut assigns to a write.
|
// PutAction is the disposition OnPut assigns to a write.
|
||||||
type PutAction int
|
type PutAction int
|
||||||
|
|
||||||
@@ -66,6 +71,11 @@ type Storer struct {
|
|||||||
// OnGet, when set, is consulted for every Get with the object key
|
// OnGet, when set, is consulted for every Get with the object key
|
||||||
// and damages the returned bytes accordingly.
|
// and damages the returned bytes accordingly.
|
||||||
OnGet func(key string) GetFault
|
OnGet func(key string) GetFault
|
||||||
|
|
||||||
|
// OnDelete, when set, is consulted before every Delete with the
|
||||||
|
// object key. Returning true fails the delete and leaves the object
|
||||||
|
// in place.
|
||||||
|
OnDelete func(key string) bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// New wraps inner. inner must be non-nil.
|
// New wraps inner. inner must be non-nil.
|
||||||
@@ -154,8 +164,12 @@ func (f *Storer) Stat(ctx context.Context, key string) (*storage.ObjectInfo, err
|
|||||||
return f.inner.Stat(ctx, key)
|
return f.inner.Stat(ctx, key)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Delete delegates unchanged.
|
// Delete removes the object unless OnDelete fails the delete.
|
||||||
func (f *Storer) Delete(ctx context.Context, key string) error {
|
func (f *Storer) Delete(ctx context.Context, key string) error {
|
||||||
|
if f.OnDelete != nil && f.OnDelete(key) {
|
||||||
|
return fmt.Errorf("%w for %q", ErrInjectedDelete, key)
|
||||||
|
}
|
||||||
|
|
||||||
return f.inner.Delete(ctx, key)
|
return f.inner.Delete(ctx, key)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -5,12 +5,14 @@ import (
|
|||||||
"io/fs"
|
"io/fs"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||||
|
"sneak.berlin/go/vaultik/internal/storage/faultstore"
|
||||||
)
|
)
|
||||||
|
|
||||||
// TestNukeRemoteLeavesNoFiles checks that remote nuke leaves no file
|
// TestNukeRemoteLeavesNoFiles checks that remote nuke leaves no file
|
||||||
@@ -23,7 +25,7 @@ func TestNukeRemoteLeavesNoFiles(t *testing.T) {
|
|||||||
|
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
storeDir := filepath.Join(t.TempDir(), "store")
|
storeDir := filepath.Join(t.TempDir(), "store")
|
||||||
v, repos, _ := backUpToFileDestination(ctx, t, storeDir)
|
v, repos, out := backUpToFileDestination(ctx, t, storeDir)
|
||||||
|
|
||||||
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
|
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
@@ -61,4 +63,62 @@ func TestNukeRemoteLeavesNoFiles(t *testing.T) {
|
|||||||
})
|
})
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Empty(t, files)
|
assert.Empty(t, files)
|
||||||
|
|
||||||
|
// Nuke deletes the blobs itself, so advice to run prune is wrong.
|
||||||
|
assert.Contains(t, out.String(), "Backup destination store is now empty.")
|
||||||
|
assert.NotContains(t, out.String(), "vaultik prune")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestNukeRemoteFailsWhenADeleteFails checks that remote nuke returns an
|
||||||
|
// error when a snapshot's metadata or a blob cannot be deleted, does not
|
||||||
|
// report the store empty, and advises running nuke again: prune deletes
|
||||||
|
// no metadata.
|
||||||
|
func TestNukeRemoteFailsWhenADeleteFails(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, refusedPrefix := range []string{"metadata/", "blobs/"} {
|
||||||
|
t.Run(refusedPrefix, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
storeDir := filepath.Join(t.TempDir(), "store")
|
||||||
|
v, _, out := backUpToFileDestination(ctx, t, storeDir)
|
||||||
|
|
||||||
|
store := faultstore.New(v.Storage)
|
||||||
|
store.OnDelete = func(key string) bool {
|
||||||
|
return strings.HasPrefix(key, refusedPrefix)
|
||||||
|
}
|
||||||
|
v.Storage = store
|
||||||
|
|
||||||
|
require.Error(t, v.NukeRemote(true))
|
||||||
|
|
||||||
|
assert.NotContains(t, out.String(), "now empty")
|
||||||
|
assert.Contains(t, out.String(), "vaultik remote nuke --force")
|
||||||
|
assert.NotContains(t, out.String(), "vaultik prune")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved checks that a
|
||||||
|
// snapshot remote nuke cannot remove from the local index fails the
|
||||||
|
// run: nuke then leaves that snapshot's metadata on the destination
|
||||||
|
// store.
|
||||||
|
func TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
storeDir := filepath.Join(t.TempDir(), "store")
|
||||||
|
v, _, out := backUpToFileDestination(ctx, t, storeDir)
|
||||||
|
|
||||||
|
_, err := v.DB.Conn().ExecContext(ctx, `
|
||||||
|
CREATE TRIGGER refuse_snapshot_delete BEFORE DELETE ON snapshots
|
||||||
|
BEGIN SELECT RAISE(ABORT, 'simulated index error'); END`)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
require.Error(t, v.NukeRemote(true))
|
||||||
|
|
||||||
|
assert.NotContains(t, out.String(), "now empty")
|
||||||
|
assert.Contains(t, out.String(), "vaultik remote nuke --force")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -19,6 +19,14 @@ type PruneOptions struct {
|
|||||||
var errNukeRequiresForce = errors.New(
|
var errNukeRequiresForce = errors.New(
|
||||||
"nuke requires --force (this deletes ALL remote snapshots and blobs)")
|
"nuke requires --force (this deletes ALL remote snapshots and blobs)")
|
||||||
|
|
||||||
|
// errBlobDeleteFailed reports unreferenced blobs prune could not delete.
|
||||||
|
var errBlobDeleteFailed = errors.New("blob(s) could not be deleted")
|
||||||
|
|
||||||
|
// nukeCommandHint is the command suggested when a remote nuke could not
|
||||||
|
// delete everything: running it again deletes what is left, which
|
||||||
|
// `vaultik prune` does not do for snapshot metadata.
|
||||||
|
const nukeCommandHint = "vaultik remote nuke --force"
|
||||||
|
|
||||||
// metadataDirName is the top-level remote directory holding snapshot
|
// metadataDirName is the top-level remote directory holding snapshot
|
||||||
// metadata.
|
// metadata.
|
||||||
const metadataDirName = "metadata"
|
const metadataDirName = "metadata"
|
||||||
@@ -35,6 +43,22 @@ func (v *Vaultik) NukeRemote(force bool) error {
|
|||||||
return errNukeRequiresForce
|
return errNukeRequiresForce
|
||||||
}
|
}
|
||||||
|
|
||||||
|
err := v.emptyDestinationStore()
|
||||||
|
if err != nil {
|
||||||
|
v.UI.Warningf("Run '%s' again to delete what is left on the "+
|
||||||
|
"backup destination store.", nukeCommandHint)
|
||||||
|
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
v.UI.Completef("Backup destination store is now empty.")
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// emptyDestinationStore does the deleting for NukeRemote and stops at
|
||||||
|
// the first step that fails.
|
||||||
|
func (v *Vaultik) emptyDestinationStore() error {
|
||||||
v.UI.Beginf("Removing all snapshot metadata from backup destination store.")
|
v.UI.Beginf("Removing all snapshot metadata from backup destination store.")
|
||||||
|
|
||||||
_, err := v.RemoveAllSnapshots(&RemoveOptions{Force: true})
|
_, err := v.RemoveAllSnapshots(&RemoveOptions{Force: true})
|
||||||
@@ -58,8 +82,6 @@ func (v *Vaultik) NukeRemote(force bool) error {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
v.UI.Completef("Backup destination store is now empty.")
|
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -162,14 +184,17 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
|
|||||||
v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result)
|
v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result)
|
||||||
|
|
||||||
if opts.JSON {
|
if opts.JSON {
|
||||||
return v.outputPruneBlobsJSON(result)
|
err = v.outputPruneBlobsJSON(result)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
v.stdoutf("\nDeleted %d blob(s) totaling %s\n",
|
||||||
|
result.BlobsDeleted, ubytes(result.BytesFreed))
|
||||||
}
|
}
|
||||||
|
|
||||||
v.stdoutf("\nDeleted %d blob(s) totaling %s\n",
|
|
||||||
result.BlobsDeleted, ubytes(result.BytesFreed))
|
|
||||||
|
|
||||||
if result.BlobsFailed > 0 {
|
if result.BlobsFailed > 0 {
|
||||||
v.stdoutf("Failed to delete %d blob(s)\n", result.BlobsFailed)
|
return fmt.Errorf("%d %w", result.BlobsFailed, errBlobDeleteFailed)
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -0,0 +1,98 @@
|
|||||||
|
package vaultik_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
|
"sneak.berlin/go/vaultik/internal/storage/faultstore"
|
||||||
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
|
)
|
||||||
|
|
||||||
|
// refuseBlobDeletes is an OnDelete hook that fails every blob delete.
|
||||||
|
func refuseBlobDeletes(key string) bool {
|
||||||
|
return strings.HasPrefix(key, "blobs/")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestPruneBlobs_FailedDeleteReturnsError checks that prune returns an
|
||||||
|
// error, so the command exits non-zero, when an unreferenced blob cannot
|
||||||
|
// be deleted.
|
||||||
|
func TestPruneBlobs_FailedDeleteReturnsError(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
env := newListEnv(t)
|
||||||
|
|
||||||
|
// The store holds no manifest, so nothing references this blob.
|
||||||
|
addBlob(t, env.store.testStorer, testBlobHashA)
|
||||||
|
|
||||||
|
store := faultstore.New(env.store)
|
||||||
|
store.OnDelete = refuseBlobDeletes
|
||||||
|
env.v.Storage = store
|
||||||
|
|
||||||
|
err := env.v.PruneBlobs(&vaultik.PruneOptions{Force: true})
|
||||||
|
|
||||||
|
require.ErrorContains(t, err, "1 blob(s) could not be deleted")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestPruneBlobs_JSONFailedDeleteReturnsError checks that prune --json
|
||||||
|
// prints its document and then returns an error when an unreferenced
|
||||||
|
// blob cannot be deleted.
|
||||||
|
func TestPruneBlobs_JSONFailedDeleteReturnsError(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
env := newListEnv(t)
|
||||||
|
addBlob(t, env.store.testStorer, testBlobHashA)
|
||||||
|
|
||||||
|
store := faultstore.New(env.store)
|
||||||
|
store.OnDelete = refuseBlobDeletes
|
||||||
|
env.v.Storage = store
|
||||||
|
|
||||||
|
err := env.v.PruneBlobs(&vaultik.PruneOptions{JSON: true})
|
||||||
|
|
||||||
|
require.Error(t, err, "prune must fail when a blob cannot be deleted")
|
||||||
|
|
||||||
|
var result vaultik.PruneBlobsResult
|
||||||
|
|
||||||
|
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &result),
|
||||||
|
"stdout must hold the JSON document, got:\n%s", env.stdout.String())
|
||||||
|
assert.Equal(t, 1, result.BlobsFailed)
|
||||||
|
assert.Equal(t, 0, result.BlobsDeleted)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCreateSnapshotPrune_FailedBlobDeleteReturnsError checks that the
|
||||||
|
// prune step of snapshot create --prune fails the run when an
|
||||||
|
// unreferenced blob cannot be deleted.
|
||||||
|
func TestCreateSnapshotPrune_FailedBlobDeleteReturnsError(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
storeDir := filepath.Join(t.TempDir(), "store")
|
||||||
|
v, _, _ := backUpToFileDestination(ctx, t, storeDir)
|
||||||
|
|
||||||
|
// No snapshot references this blob, so the prune step deletes it.
|
||||||
|
orphan := filepath.Join(storeDir, "blobs", testBlobHashA[:2],
|
||||||
|
testBlobHashA[2:4], testBlobHashA)
|
||||||
|
require.NoError(t, os.MkdirAll(filepath.Dir(orphan), 0o750))
|
||||||
|
require.NoError(t, os.WriteFile(orphan, []byte("orphan"), 0o600))
|
||||||
|
|
||||||
|
store := faultstore.New(v.Storage)
|
||||||
|
store.OnDelete = refuseBlobDeletes
|
||||||
|
v.Storage = store
|
||||||
|
|
||||||
|
err := v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||||
|
Cron: true,
|
||||||
|
Prune: true,
|
||||||
|
})
|
||||||
|
|
||||||
|
require.Error(t, err, "the prune step must fail the run")
|
||||||
|
assert.FileExists(t, orphan)
|
||||||
|
}
|
||||||
@@ -3,21 +3,17 @@ package vaultik_test
|
|||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"database/sql"
|
|
||||||
"io"
|
"io"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/klauspost/compress/zstd"
|
"github.com/klauspost/compress/zstd"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/vaultik/internal/database"
|
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||||
"sneak.berlin/go/vaultik/internal/storage"
|
"sneak.berlin/go/vaultik/internal/storage"
|
||||||
"sneak.berlin/go/vaultik/internal/types"
|
|
||||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -218,37 +214,6 @@ func addBlob(t *testing.T, store *testStorer, hash string) {
|
|||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// newLocalIndex returns an in-memory local index holding a completed
|
|
||||||
// snapshot row for each of ids.
|
|
||||||
func newLocalIndex(t *testing.T, ids ...string) *database.Repositories {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
|
|
||||||
db, err := database.New(ctx, ":memory:")
|
|
||||||
require.NoError(t, err)
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
repos := database.NewRepositories(db)
|
|
||||||
|
|
||||||
for _, id := range ids {
|
|
||||||
completedAt := time.Now().UTC()
|
|
||||||
snap := &database.Snapshot{
|
|
||||||
ID: types.SnapshotID(id),
|
|
||||||
Hostname: snapHostname,
|
|
||||||
StartedAt: completedAt,
|
|
||||||
CompletedAt: &completedAt,
|
|
||||||
}
|
|
||||||
|
|
||||||
err = repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
|
|
||||||
return repos.Snapshots.Create(ctx, tx, snap)
|
|
||||||
})
|
|
||||||
require.NoError(t, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return repos
|
|
||||||
}
|
|
||||||
|
|
||||||
// ============================================================================
|
// ============================================================================
|
||||||
// Unit Tests for RemoveSnapshot
|
// Unit Tests for RemoveSnapshot
|
||||||
// ============================================================================
|
// ============================================================================
|
||||||
@@ -268,7 +233,6 @@ func TestRemoveSnapshot_LocalOnly_PreservesRemote(t *testing.T) {
|
|||||||
addBlob(t, store, blobA)
|
addBlob(t, store, blobA)
|
||||||
|
|
||||||
tv := vaultik.NewForTesting(store)
|
tv := vaultik.NewForTesting(store)
|
||||||
tv.Repositories = newLocalIndex(t, "snapshot-001")
|
|
||||||
|
|
||||||
opts := &vaultik.RemoveOptions{Force: true, LocalOnly: true}
|
opts := &vaultik.RemoveOptions{Force: true, LocalOnly: true}
|
||||||
result, err := tv.RemoveSnapshot("snapshot-001", opts)
|
result, err := tv.RemoveSnapshot("snapshot-001", opts)
|
||||||
@@ -304,7 +268,6 @@ func TestRemoveSnapshot_DefaultRemovesMetadataNotBlobs(t *testing.T) {
|
|||||||
addBlob(t, store, blobShared)
|
addBlob(t, store, blobShared)
|
||||||
|
|
||||||
tv := vaultik.NewForTesting(store)
|
tv := vaultik.NewForTesting(store)
|
||||||
tv.Repositories = newLocalIndex(t, "snapshot-001", "snapshot-002")
|
|
||||||
|
|
||||||
opts := &vaultik.RemoveOptions{Force: true}
|
opts := &vaultik.RemoveOptions{Force: true}
|
||||||
result, err := tv.RemoveSnapshot("snapshot-001", opts)
|
result, err := tv.RemoveSnapshot("snapshot-001", opts)
|
||||||
@@ -353,84 +316,6 @@ func TestRemoveSnapshot_DryRun(t *testing.T) {
|
|||||||
assert.Contains(t, tv.Stdout.String(), "[Dry run - no changes made]")
|
assert.Contains(t, tv.Stdout.String(), "[Dry run - no changes made]")
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestRemoveSnapshot_UnknownIDIsError checks that an ID naming no
|
|
||||||
// snapshot in the local index or on the destination store is an error
|
|
||||||
// under every option, and that nothing is reported as removed.
|
|
||||||
func TestRemoveSnapshot_UnknownIDIsError(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for name, opts := range map[string]vaultik.RemoveOptions{
|
|
||||||
"force": {Force: true},
|
|
||||||
"json": {JSON: true},
|
|
||||||
"dry-run": {Force: true, DryRun: true},
|
|
||||||
"local-only": {Force: true, LocalOnly: true},
|
|
||||||
} {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
store := newTestStorer()
|
|
||||||
addManifest(t, store, "snapshot-001", nil)
|
|
||||||
|
|
||||||
tv := vaultik.NewForTesting(store)
|
|
||||||
tv.Repositories = newLocalIndex(t, "snapshot-001")
|
|
||||||
|
|
||||||
result, err := tv.RemoveSnapshot("snapshot-999", &opts)
|
|
||||||
|
|
||||||
require.ErrorContains(t, err, "snapshot not found")
|
|
||||||
assert.False(t, result.RemoteRemoved)
|
|
||||||
assert.Empty(t, tv.Stdout.String())
|
|
||||||
assert.True(t, store.hasKey(remoteKeyPath("snapshot-001")))
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRemoveSnapshot_NoMetadataOnDestination checks that removing a
|
|
||||||
// snapshot whose metadata is not on the destination store does not
|
|
||||||
// report metadata removed from it.
|
|
||||||
func TestRemoveSnapshot_NoMetadataOnDestination(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tv := vaultik.NewForTesting(newTestStorer())
|
|
||||||
tv.Repositories = newLocalIndex(t, "snapshot-001")
|
|
||||||
|
|
||||||
result, err := tv.RemoveSnapshot("snapshot-001",
|
|
||||||
&vaultik.RemoveOptions{Force: true})
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.False(t, result.RemoteRemoved)
|
|
||||||
|
|
||||||
out := tv.Stdout.String()
|
|
||||||
assert.Contains(t, out, "Removed snapshot 'snapshot-001' from local database")
|
|
||||||
assert.NotContains(t, out, "Removed snapshot metadata from remote storage")
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRemoveSnapshot_NotInLocalIndex checks that removing a snapshot
|
|
||||||
// only the destination store holds removes its metadata there without
|
|
||||||
// reporting a removal from the local database.
|
|
||||||
func TestRemoveSnapshot_NotInLocalIndex(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
store := newTestStorer()
|
|
||||||
addManifest(t, store, "snapshot-001", nil)
|
|
||||||
|
|
||||||
tv := vaultik.NewForTesting(store)
|
|
||||||
tv.Repositories = newLocalIndex(t)
|
|
||||||
|
|
||||||
result, err := tv.RemoveSnapshot("snapshot-001",
|
|
||||||
&vaultik.RemoveOptions{Force: true})
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.True(t, result.RemoteRemoved)
|
|
||||||
assert.False(t, store.hasKey(remoteKeyPath("snapshot-001")))
|
|
||||||
|
|
||||||
out := tv.Stdout.String()
|
|
||||||
assert.NotContains(t, out, "from local database")
|
|
||||||
assert.Contains(t, out, "Removed snapshot metadata from remote storage")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRemoveAllSnapshots_RequiresForce(t *testing.T) {
|
func TestRemoveAllSnapshots_RequiresForce(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -476,7 +361,8 @@ func TestRemoveAllSnapshots_WithForce(t *testing.T) {
|
|||||||
out := tv.Stdout.String()
|
out := tv.Stdout.String()
|
||||||
assert.Contains(t, out, "Removed 2 snapshot(s)")
|
assert.Contains(t, out, "Removed 2 snapshot(s)")
|
||||||
assert.Contains(t, out, "Removed snapshot metadata from remote storage")
|
assert.Contains(t, out, "Removed snapshot metadata from remote storage")
|
||||||
assert.Contains(t, out, "vaultik prune")
|
// Remote nuke, the only caller, deletes the blobs next itself.
|
||||||
|
assert.NotContains(t, out, "vaultik prune")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRemoveAllSnapshots_DryRun(t *testing.T) {
|
func TestRemoveAllSnapshots_DryRun(t *testing.T) {
|
||||||
|
|||||||
+50
-111
@@ -23,7 +23,7 @@ var (
|
|||||||
errNoSnapshotsInConfig = errors.New("no snapshots configured")
|
errNoSnapshotsInConfig = errors.New("no snapshots configured")
|
||||||
errSnapshotVerifyFailed = errors.New("verification failed")
|
errSnapshotVerifyFailed = errors.New("verification failed")
|
||||||
errRemoveAllNeedsForce = errors.New("--all requires --force")
|
errRemoveAllNeedsForce = errors.New("--all requires --force")
|
||||||
errSnapshotNotFound = errors.New("snapshot not found")
|
errSnapshotRemoveFailed = errors.New("snapshot(s) could not be removed")
|
||||||
errInvalidTableName = errors.New("invalid table name")
|
errInvalidTableName = errors.New("invalid table name")
|
||||||
errNoAgeRecipients = errors.New(
|
errNoAgeRecipients = errors.New(
|
||||||
"creating a snapshot needs at least one public key in " +
|
"creating a snapshot needs at least one public key in " +
|
||||||
@@ -644,7 +644,7 @@ func (v *Vaultik) confirmAndExecutePurge(
|
|||||||
"snapshot_id", snapshotID, "error", err)
|
"snapshot_id", snapshotID, "error", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err = v.deleteRemoteSnapshotByKey(snapshot.RemoteSnapshotKey(snapshotID))
|
err = v.deleteRemoteSnapshotByKey(snapshot.RemoteSnapshotKey(snapshotID))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("deleting snapshot %s from remote: %w", snapshotID, err)
|
return fmt.Errorf("deleting snapshot %s from remote: %w", snapshotID, err)
|
||||||
}
|
}
|
||||||
@@ -1154,11 +1154,6 @@ func (v *Vaultik) RemoveSnapshot(
|
|||||||
return result, err
|
return result, err
|
||||||
}
|
}
|
||||||
|
|
||||||
inLocalIndex, err := v.findSnapshotToRemove(snapshotID, opts)
|
|
||||||
if err != nil {
|
|
||||||
return result, err
|
|
||||||
}
|
|
||||||
|
|
||||||
if opts.DryRun {
|
if opts.DryRun {
|
||||||
return v.removeSnapshotDryRun(result, snapshotID, opts)
|
return v.removeSnapshotDryRun(result, snapshotID, opts)
|
||||||
}
|
}
|
||||||
@@ -1167,13 +1162,11 @@ func (v *Vaultik) RemoveSnapshot(
|
|||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
if inLocalIndex {
|
log.Info("Removing snapshot from local database", "snapshot_id", snapshotID)
|
||||||
log.Info("Removing snapshot from local database", "snapshot_id", snapshotID)
|
|
||||||
|
|
||||||
err = v.deleteSnapshotFromLocalDB(snapshotID)
|
err = v.deleteSnapshotFromLocalDB(snapshotID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return result, fmt.Errorf("removing from local database: %w", err)
|
return result, fmt.Errorf("removing from local database: %w", err)
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if !opts.LocalOnly {
|
if !opts.LocalOnly {
|
||||||
@@ -1192,11 +1185,9 @@ func (v *Vaultik) RemoveSnapshot(
|
|||||||
return result, v.outputRemoveJSON(result)
|
return result, v.outputRemoveJSON(result)
|
||||||
}
|
}
|
||||||
|
|
||||||
if inLocalIndex {
|
v.stdoutf("Removed snapshot '%s' from local database\n", snapshotID)
|
||||||
v.stdoutf("Removed snapshot '%s' from local database\n", snapshotID)
|
|
||||||
}
|
|
||||||
|
|
||||||
if result.RemoteRemoved {
|
if !opts.LocalOnly && result.RemoteRemoved {
|
||||||
v.printlnStdout("Removed snapshot metadata from remote storage")
|
v.printlnStdout("Removed snapshot metadata from remote storage")
|
||||||
v.stdoutf("\nNote: The removed snapshot's blobs remain on the remote. "+
|
v.stdoutf("\nNote: The removed snapshot's blobs remain on the remote. "+
|
||||||
"Run '%s' to delete any blobs no longer referenced by any "+
|
"Run '%s' to delete any blobs no longer referenced by any "+
|
||||||
@@ -1206,45 +1197,6 @@ func (v *Vaultik) RemoveSnapshot(
|
|||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// findSnapshotToRemove reports whether the local index has a row for
|
|
||||||
// snapshotID, and returns errSnapshotNotFound when there is nothing to
|
|
||||||
// remove: no row and, unless LocalOnly is set, no metadata on the
|
|
||||||
// destination store either. The destination store is consulted only
|
|
||||||
// when the local index has no row, so a snapshot in the local index can
|
|
||||||
// still be removed from it while the destination store is unreachable.
|
|
||||||
func (v *Vaultik) findSnapshotToRemove(
|
|
||||||
snapshotID string, opts *RemoveOptions,
|
|
||||||
) (bool, error) {
|
|
||||||
if v.Repositories != nil {
|
|
||||||
snap, err := v.Repositories.Snapshots.GetByID(v.ctx, snapshotID)
|
|
||||||
if err != nil {
|
|
||||||
return false, fmt.Errorf("looking up snapshot in local index: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if snap != nil {
|
|
||||||
return true, nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if opts.LocalOnly {
|
|
||||||
return false, fmt.Errorf("%w in the local index: %s",
|
|
||||||
errSnapshotNotFound, snapshotID)
|
|
||||||
}
|
|
||||||
|
|
||||||
objects, err := v.remoteSnapshotObjects(snapshot.RemoteSnapshotKey(snapshotID))
|
|
||||||
if err != nil {
|
|
||||||
return false, fmt.Errorf("looking up snapshot on destination store: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(objects) == 0 {
|
|
||||||
return false, fmt.Errorf(
|
|
||||||
"%w in the local index or on the destination store: %s",
|
|
||||||
errSnapshotNotFound, snapshotID)
|
|
||||||
}
|
|
||||||
|
|
||||||
return false, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// removeSnapshotDryRun reports what RemoveSnapshot would do without
|
// removeSnapshotDryRun reports what RemoveSnapshot would do without
|
||||||
// making changes.
|
// making changes.
|
||||||
func (v *Vaultik) removeSnapshotDryRun(
|
func (v *Vaultik) removeSnapshotDryRun(
|
||||||
@@ -1304,14 +1256,14 @@ func (v *Vaultik) confirmRemoveSnapshot(snapshotID string, opts *RemoveOptions)
|
|||||||
// didn't finish and to run `vaultik snapshot remove` for the snapshot
|
// didn't finish and to run `vaultik snapshot remove` for the snapshot
|
||||||
// again once the destination store is reachable (`vaultik prune` never
|
// again once the destination store is reachable (`vaultik prune` never
|
||||||
// removes snapshot metadata). Returns true when the remote removal
|
// removes snapshot metadata). Returns true when the remote removal
|
||||||
// succeeded and deleted something.
|
// succeeded.
|
||||||
func (v *Vaultik) removeSnapshotRemote(snapshotID string, opts *RemoveOptions) bool {
|
func (v *Vaultik) removeSnapshotRemote(snapshotID string, opts *RemoveOptions) bool {
|
||||||
log.Info("Removing snapshot metadata from remote storage",
|
log.Info("Removing snapshot metadata from remote storage",
|
||||||
"snapshot_id", snapshotID)
|
"snapshot_id", snapshotID)
|
||||||
|
|
||||||
remoteKey := snapshot.RemoteSnapshotKey(snapshotID)
|
remoteKey := snapshot.RemoteSnapshotKey(snapshotID)
|
||||||
|
|
||||||
deleted, err := v.deleteRemoteSnapshotByKey(remoteKey)
|
err := v.deleteRemoteSnapshotByKey(remoteKey)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Warn("Could not remove snapshot metadata from remote storage; "+
|
log.Warn("Could not remove snapshot metadata from remote storage; "+
|
||||||
"run '"+snapshotRemoveCommandHint+"' with the snapshot's ID "+
|
"run '"+snapshotRemoveCommandHint+"' with the snapshot's ID "+
|
||||||
@@ -1329,7 +1281,7 @@ func (v *Vaultik) removeSnapshotRemote(snapshotID string, opts *RemoveOptions) b
|
|||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
return deleted > 0
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
// RemoveAllSnapshots removes every snapshot known to the local
|
// RemoveAllSnapshots removes every snapshot known to the local
|
||||||
@@ -1513,8 +1465,8 @@ func (v *Vaultik) handleRemoveAllDryRun(
|
|||||||
|
|
||||||
// executeRemoveAll deletes every local snapshot and, unless LocalOnly
|
// executeRemoveAll deletes every local snapshot and, unless LocalOnly
|
||||||
// is set, every corresponding remote metadata directory plus any
|
// is set, every corresponding remote metadata directory plus any
|
||||||
// orphan remote keys. Blobs are not touched; the printed prune-command
|
// orphan remote keys. Blobs are not touched. It returns an error when
|
||||||
// hint is the next step.
|
// any snapshot could not be removed.
|
||||||
func (v *Vaultik) executeRemoveAll(
|
func (v *Vaultik) executeRemoveAll(
|
||||||
localSnaps, orphanRemoteKeys []string, opts *RemoveOptions,
|
localSnaps, orphanRemoteKeys []string, opts *RemoveOptions,
|
||||||
) (*RemoveResult, error) {
|
) (*RemoveResult, error) {
|
||||||
@@ -1528,18 +1480,11 @@ func (v *Vaultik) executeRemoveAll(
|
|||||||
|
|
||||||
result := &RemoveResult{}
|
result := &RemoveResult{}
|
||||||
|
|
||||||
remoteErrors := v.removeAllLocalSnapshots(localSnaps, opts, result)
|
failures := v.removeAllLocalSnapshots(localSnaps, opts, result)
|
||||||
|
|
||||||
if !opts.LocalOnly {
|
if !opts.LocalOnly {
|
||||||
remoteErrors += v.removeAllOrphanRemotes(orphanRemoteKeys, result)
|
failures += v.removeAllOrphanRemotes(orphanRemoteKeys, result)
|
||||||
|
result.RemoteRemoved = failures == 0
|
||||||
if remoteErrors == 0 {
|
|
||||||
result.RemoteRemoved = true
|
|
||||||
} else if v.UI != nil {
|
|
||||||
v.UI.Warningf("Some remote metadata deletions failed. Run '%s' "+
|
|
||||||
"once the remote is healthy to clean up unreferenced blobs.",
|
|
||||||
pruneCommandHint)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if v.SnapshotManager != nil {
|
if v.SnapshotManager != nil {
|
||||||
@@ -1549,6 +1494,10 @@ func (v *Vaultik) executeRemoveAll(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if failures > 0 {
|
||||||
|
return result, fmt.Errorf("%d %w", failures, errSnapshotRemoveFailed)
|
||||||
|
}
|
||||||
|
|
||||||
if opts.JSON {
|
if opts.JSON {
|
||||||
return result, v.outputRemoveJSON(result)
|
return result, v.outputRemoveJSON(result)
|
||||||
}
|
}
|
||||||
@@ -1557,9 +1506,6 @@ func (v *Vaultik) executeRemoveAll(
|
|||||||
|
|
||||||
if !opts.LocalOnly && result.RemoteRemoved {
|
if !opts.LocalOnly && result.RemoteRemoved {
|
||||||
v.printlnStdout("Removed snapshot metadata from remote storage")
|
v.printlnStdout("Removed snapshot metadata from remote storage")
|
||||||
v.stdoutf("\nNote: Removed snapshots' blobs remain on the remote. "+
|
|
||||||
"Run '%s' to delete any blobs no longer referenced by any "+
|
|
||||||
"remaining remote snapshot.\n", pruneCommandHint)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return result, nil
|
return result, nil
|
||||||
@@ -1602,12 +1548,13 @@ func (v *Vaultik) deleteIncompleteSnapshot(snapshotIDStr string, id any) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// removeAllLocalSnapshots removes each known local snapshot (and, unless
|
// removeAllLocalSnapshots removes each known local snapshot (and, unless
|
||||||
// --local-only, its remote metadata), returning the number of remote
|
// --local-only, its remote metadata), returning the number of snapshots
|
||||||
// deletion failures.
|
// it could not remove. A snapshot left in the local index keeps its
|
||||||
|
// remote metadata too, since it is not an orphan remote key.
|
||||||
func (v *Vaultik) removeAllLocalSnapshots(
|
func (v *Vaultik) removeAllLocalSnapshots(
|
||||||
localSnaps []string, opts *RemoveOptions, result *RemoveResult,
|
localSnaps []string, opts *RemoveOptions, result *RemoveResult,
|
||||||
) int {
|
) int {
|
||||||
remoteErrors := 0
|
failures := 0
|
||||||
|
|
||||||
for _, snapshotID := range localSnaps {
|
for _, snapshotID := range localSnaps {
|
||||||
log.Info("Removing snapshot", "snapshot_id", snapshotID)
|
log.Info("Removing snapshot", "snapshot_id", snapshotID)
|
||||||
@@ -1617,24 +1564,26 @@ func (v *Vaultik) removeAllLocalSnapshots(
|
|||||||
log.Error("Failed to remove from local database",
|
log.Error("Failed to remove from local database",
|
||||||
"snapshot_id", snapshotID, "error", err)
|
"snapshot_id", snapshotID, "error", err)
|
||||||
|
|
||||||
|
failures++
|
||||||
|
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
if !opts.LocalOnly {
|
if !opts.LocalOnly {
|
||||||
_, err = v.deleteRemoteSnapshotByKey(
|
err = v.deleteRemoteSnapshotByKey(
|
||||||
snapshot.RemoteSnapshotKey(snapshotID))
|
snapshot.RemoteSnapshotKey(snapshotID))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Warn("Failed to remove snapshot metadata from remote",
|
log.Warn("Failed to remove snapshot metadata from remote",
|
||||||
"snapshot_id", snapshotID, "error", err)
|
"snapshot_id", snapshotID, "error", err)
|
||||||
|
|
||||||
remoteErrors++
|
failures++
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
result.SnapshotsRemoved = append(result.SnapshotsRemoved, snapshotID)
|
result.SnapshotsRemoved = append(result.SnapshotsRemoved, snapshotID)
|
||||||
}
|
}
|
||||||
|
|
||||||
return remoteErrors
|
return failures
|
||||||
}
|
}
|
||||||
|
|
||||||
// removeAllOrphanRemotes deletes remote snapshot metadata that has no
|
// removeAllOrphanRemotes deletes remote snapshot metadata that has no
|
||||||
@@ -1647,7 +1596,7 @@ func (v *Vaultik) removeAllOrphanRemotes(
|
|||||||
for _, key := range orphanRemoteKeys {
|
for _, key := range orphanRemoteKeys {
|
||||||
log.Info("Removing orphan remote snapshot", "remote_key", key)
|
log.Info("Removing orphan remote snapshot", "remote_key", key)
|
||||||
|
|
||||||
_, err := v.deleteRemoteSnapshotByKey(key)
|
err := v.deleteRemoteSnapshotByKey(key)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Warn("Failed to remove orphan from remote",
|
log.Warn("Failed to remove orphan from remote",
|
||||||
"remote_key", key, "error", err)
|
"remote_key", key, "error", err)
|
||||||
@@ -1694,44 +1643,34 @@ func (v *Vaultik) deleteSnapshotFromLocalDB(snapshotID string) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// deleteRemoteSnapshotByKey removes everything under
|
// deleteRemoteSnapshotByKey removes everything under
|
||||||
// metadata/<remoteKey>/ on the destination store and returns how many
|
// metadata/<remoteKey>/ on the destination store. The argument is a
|
||||||
// objects it deleted. The argument is a remote key (double-SHA256
|
// remote key (double-SHA256 derivation), not a human snapshot ID;
|
||||||
// derivation), not a human snapshot ID; callers that have a human ID
|
// callers that have a human ID must hash via snapshot.RemoteSnapshotKey
|
||||||
// must hash via snapshot.RemoteSnapshotKey first.
|
// first.
|
||||||
func (v *Vaultik) deleteRemoteSnapshotByKey(remoteKey string) (int, error) {
|
func (v *Vaultik) deleteRemoteSnapshotByKey(remoteKey string) error {
|
||||||
objectsToDelete, err := v.remoteSnapshotObjects(remoteKey)
|
prefix := fmt.Sprintf("metadata/%s/", remoteKey)
|
||||||
if err != nil {
|
objectCh := v.Storage.ListStream(v.ctx, prefix)
|
||||||
return 0, err
|
|
||||||
|
var objectsToDelete []string
|
||||||
|
|
||||||
|
for object := range objectCh {
|
||||||
|
if object.Err != nil {
|
||||||
|
return fmt.Errorf("listing objects: %w", object.Err)
|
||||||
|
}
|
||||||
|
|
||||||
|
objectsToDelete = append(objectsToDelete, object.Key)
|
||||||
}
|
}
|
||||||
|
|
||||||
for i, key := range objectsToDelete {
|
for _, key := range objectsToDelete {
|
||||||
err = v.Storage.Delete(v.ctx, key)
|
err := v.Storage.Delete(v.ctx, key)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return i, fmt.Errorf("removing %s: %w", key, err)
|
return fmt.Errorf("removing %s: %w", key, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
log.Debug("Deleted remote object", "key", key)
|
log.Debug("Deleted remote object", "key", key)
|
||||||
}
|
}
|
||||||
|
|
||||||
return len(objectsToDelete), nil
|
return nil
|
||||||
}
|
|
||||||
|
|
||||||
// remoteSnapshotObjects lists the keys of every object under
|
|
||||||
// metadata/<remoteKey>/ on the destination store.
|
|
||||||
func (v *Vaultik) remoteSnapshotObjects(remoteKey string) ([]string, error) {
|
|
||||||
prefix := fmt.Sprintf("metadata/%s/", remoteKey)
|
|
||||||
|
|
||||||
var keys []string
|
|
||||||
|
|
||||||
for object := range v.Storage.ListStream(v.ctx, prefix) {
|
|
||||||
if object.Err != nil {
|
|
||||||
return nil, fmt.Errorf("listing objects: %w", object.Err)
|
|
||||||
}
|
|
||||||
|
|
||||||
keys = append(keys, object.Key)
|
|
||||||
}
|
|
||||||
|
|
||||||
return keys, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// outputRemoveJSON outputs the removal result as JSON
|
// outputRemoveJSON outputs the removal result as JSON
|
||||||
|
|||||||
Reference in New Issue
Block a user