Compare commits

..
Author SHA1 Message Date
sneak c0c79cd0a2 Make prune and remote nuke fail when a delete fails (closes #290)
check / check (push) Waiting to run
prune, and the prune step of snapshot create --prune, counted a blob
it could not delete and still exited 0. It now prints its summary or
--json document and then returns an error, which replaces the
"Failed to delete N blob(s)" line.

remote nuke only warned when a snapshot's metadata could not be
deleted, advised vaultik prune, which never removes metadata, and then
reported the store empty. It now stops with an error, does not report
the store empty, and says to run vaultik remote nuke --force again. It
checks that the local index belongs to the configured destination
before deleting anything, and that error carries no such advice. A
successful nuke no longer advises prune.

Model: opus-5-5
2026-10-13 05:44:12 +00:00
45 changed files with 451 additions and 1402 deletions
+13 -17
View File
@@ -286,15 +286,13 @@ needed.
remove orphaned blobs from remote storage. By default keeps only the latest remove orphaned blobs from remote storage. By default keeps only the latest
snapshot per name; use `--keep-newer-than` for a rolling window. snapshot per name; use `--keep-newer-than` for a rolling window.
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than * `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`)
Without `--prune`, or with a value that is not a valid duration, the
command fails before the backup starts.
**`snapshot list`**: Show every snapshot known to this host — the union **`snapshot list`**: Show every snapshot known to this host — the union
of the local index and the backup destination store — with timestamps of the local index and the backup destination store — with timestamps
and three sizes per snapshot (compressed remote size; total and three sizes per snapshot (compressed remote size; total
uncompressed chunk size; size of the chunks that snapshot stored that uncompressed chunk size; size of chunks newly referenced by that
were not stored before, recorded when its backup ran). snapshot).
Listing the destination store does **not** require the age secret key, Listing the destination store does **not** require the age secret key,
so it works in vaultik's intended configuration, where the backed-up so it works in vaultik's intended configuration, where the backed-up
@@ -307,8 +305,8 @@ only the hash, so hostname and snapshot name exist solely in the local
index and in the encrypted per-snapshot database. Snapshots found only index and in the encrypted per-snapshot database. Snapshots found only
on the destination store are therefore listed as on the destination store are therefore listed as
`<remote only:<abbreviated remote key>>` and show `<remote only>` in `<remote only:<abbreviated remote key>>` and show `<remote only>` in
the uncompressed and "new chunk" columns, which come only from the the uncompressed and "new chunk" columns, which can only be computed
local index. Their timestamp and compressed size are real, from the local index. Their timestamp and compressed size are real,
read from the manifest. read from the manifest.
Snapshots in the local index with no counterpart on the destination Snapshots in the local index with no counterpart on the destination
@@ -386,10 +384,12 @@ manifest on the destination store to compute the still-referenced blob
set and delete any blob not in that set. Step (3) reads all remote set and delete any blob not in that set. Step (3) reads all remote
manifests — network cost scales with the number of snapshots. `snapshot manifests — network cost scales with the number of snapshots. `snapshot
create --prune` runs the same cleanup automatically; this is the create --prune` runs the same cleanup automatically; this is the
manual entry point for the same work. manual entry point for the same work. If a blob cannot be deleted, the
command (and `snapshot create --prune`) exits non-zero.
* `--force`: Skip confirmation prompt * `--force`: Skip confirmation prompt
* `--json`: Output stats as JSON. Also skips the confirmation prompt, as * `--json`: Output stats as JSON. Also skips the confirmation prompt, as
`--force` does. `--force` does. A blob that cannot be deleted is counted in
`blobs_failed`, and the document is printed before the non-zero exit.
**`info`**: Display system configuration, storage settings, encryption **`info`**: Display system configuration, storage settings, encryption
recipients, and local database statistics. recipients, and local database statistics.
@@ -403,15 +403,14 @@ orphaned blob figures are reported as unknown; `--json` gives them as
`null`, lists the remote key of each unreadable manifest in `null`, lists the remote key of each unreadable manifest in
`unreadable_manifests` and counts the manifests under skipped names in `unreadable_manifests` and counts the manifests under skipped names in
`skipped_manifest_count`. An unreadable manifest also leaves its `skipped_manifest_count`. An unreadable manifest also leaves its
snapshot's blob count and blob size unknown, `null` in `--json`. A file snapshot's blob count and blob size unknown, `null` in `--json`.
under `blobs/` whose name is not a blob hash is not counted as a blob,
since `prune` never deletes it.
* `--json`: Output as JSON * `--json`: Output as JSON
**`remote nuke`**: Delete every snapshot's metadata and every blob from the **`remote nuke`**: Delete every snapshot's metadata and every blob from the
backup destination store, leaving the bucket prefix empty. Destructive and backup destination store, leaving the bucket prefix empty. Destructive and
irreversible. This is the single supported way to wipe the entire irreversible. This is the single supported way to wipe the entire
destination store. destination store. If anything cannot be deleted, the command exits
non-zero; running it again deletes what is left.
* `--force`: Required to confirm destruction. * `--force`: Required to confirm destruction.
**`database delete`**: Delete the local SQLite state database file **`database delete`**: Delete the local SQLite state database file
@@ -433,10 +432,7 @@ object store. Credentials are read from `s3.access_key_id` and
`s3.secret_access_key` in the config file. `s3.secret_access_key` in the config file.
**Local filesystem** (`file:///path/to/backup`): Stores blobs and metadata on **Local filesystem** (`file:///path/to/backup`): Stores blobs and metadata on
a local or mounted filesystem. Useful for testing or backing up to a NAS. The a local or mounted filesystem. Useful for testing or backing up to a NAS.
path must be absolute; a relative one is refused when the config loads.
Percent-escapes in the path are decoded, so `my%20stick` and `my stick` name
the same directory.
**Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud **Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud
providers. Requires rclone to be configured separately (`rclone config`). providers. Requires rclone to be configured separately (`rclone config`).
+10 -7
View File
@@ -22,13 +22,16 @@ the tag exists and is exercised; what is left is merging `next` to
# Completed Steps # Completed Steps
- 2026-10-13: Stopped `remote info` counting a file under `blobs/` that - 2026-10-13: Made `prune` and `remote nuke` exit non-zero when a delete
is not a blob as an orphaned blob fails ([issue #290](https://git.eeqj.de/sneak/vaultik/issues/290)).
([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray `prune`, and the prune step of `snapshot create --prune`, counted a
file such as `.DS_Store` was reported as an orphaned blob and the blob it could not delete and still exited 0; it now prints its summary
report advised running `prune`, which skips such a file and deleted or `--json` document and then fails. `remote nuke` warned about a
nothing. `remote info` now takes its blob list from the same listing snapshot's metadata it could not delete, said to run `vaultik prune`,
`prune` uses, in the table and in `--json`. which never removes metadata, and then reported the store empty. It
now fails without that report and says to run `vaultik remote nuke
--force` again, and a successful nuke no longer advises `prune` for
blobs it deletes itself.
- 2026-10-13: Made `snapshot remove` fail for a snapshot that does not - 2026-10-13: Made `snapshot remove` fail for a snapshot that does not
exist ([issue #293](https://git.eeqj.de/sneak/vaultik/issues/293)). exist ([issue #293](https://git.eeqj.de/sneak/vaultik/issues/293)).
-1
View File
@@ -119,7 +119,6 @@ Tracks backup snapshots.
- `completed_at` (INTEGER) - Completion timestamp (NULL if in progress) - `completed_at` (INTEGER) - Completion timestamp (NULL if in progress)
- `file_count` (INTEGER) - Number of files in snapshot - `file_count` (INTEGER) - Number of files in snapshot
- `chunk_count` (INTEGER) - Number of chunks this snapshot stored that were not stored before - `chunk_count` (INTEGER) - Number of chunks this snapshot stored that were not stored before
- `new_chunk_size` (INTEGER) - Total size of those chunks before compression; `snapshot list` shows it as NEW CHUNK SIZE
- `blob_count` (INTEGER) - Number of blobs this snapshot created - `blob_count` (INTEGER) - Number of blobs this snapshot created
- `total_size` (INTEGER) - Total size of all files - `total_size` (INTEGER) - Total size of all files
- `blob_size` (INTEGER) - Total compressed size of all referenced blobs - `blob_size` (INTEGER) - Total compressed size of all referenced blobs
-11
View File
@@ -245,17 +245,6 @@ func (p *Packer) Flush(ctx context.Context) error {
return nil return nil
} }
// DiscardCurrentBlob closes and removes the temp file of the blob being
// packed, if any, and drops that blob. A caller that stops before Flush
// must call it, or the temp file is left behind. Thread-safe.
func (p *Packer) DiscardCurrentBlob() {
p.mu.Lock()
defer p.mu.Unlock()
p.cleanupTempFile()
p.currentBlob = nil
}
// FinalizeBlob finalizes the current blob being assembled. // FinalizeBlob finalizes the current blob being assembled.
// This compresses the accumulated chunks, encrypts the result, and computes // This compresses the accumulated chunks, encrypts the result, and computes
// the content-addressed hash. The finalized blob is either passed to the // the content-addressed hash. The finalized blob is either passed to the
-6
View File
@@ -251,12 +251,6 @@ func NewConfigCommand() *cobra.Command {
Use: "config", Use: "config",
Short: "Manage the configuration file", Short: "Manage the configuration file",
Long: "Commands for creating, editing, and querying the vaultik config file.", Long: "Commands for creating, editing, and querying the vaultik config file.",
// cobra checks Args only on a command that has a Run: without
// one, a mistyped subcommand printed help and exited 0.
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, _ []string) {
_ = cmd.Help()
},
} }
cmd.AddCommand(newConfigInitCommand()) cmd.AddCommand(newConfigInitCommand())
-6
View File
@@ -15,12 +15,6 @@ func NewDatabaseCommand() *cobra.Command {
Use: "database", Use: "database",
Short: "Manage the local state database", Short: "Manage the local state database",
Long: `Commands for managing the local SQLite state database.`, Long: `Commands for managing the local SQLite state database.`,
// cobra checks Args only on a command that has a Run: without
// one, a mistyped subcommand printed help and exited 0.
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, _ []string) {
_ = cmd.Help()
},
} }
cmd.AddCommand( cmd.AddCommand(
@@ -1,108 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// Flags of `snapshot create` used to build argument vectors below.
const (
flagPrune = "--prune"
flagKeepNewerThan = "--keep-newer-than"
)
// TestEntrySnapshotCreateRejectsKeepNewerThanBeforeBackup checks that a
// --keep-newer-than the post-backup prune could not use stops `snapshot
// create` before the backup: the command fails and the destination
// store stays empty. It used to be read only by the prune, after the
// snapshot was uploaded, and without --prune it was ignored.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateRejectsKeepNewerThanBeforeBackup(t *testing.T) {
for _, testCase := range []struct {
name string
args []string
}{
{
name: "decimal in a day count",
args: []string{flagPrune, flagKeepNewerThan, "1.5d"},
},
{
name: "unknown unit",
args: []string{flagPrune, flagKeepNewerThan, "4x"},
},
{
name: "without --prune",
args: []string{flagKeepNewerThan, "4w"},
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath, storeDir := writeBackupConfig(t)
code, _, _ := runEntry(t, append(
[]string{flagConfig, configPath, cmdSnapshot, cmdCreate},
testCase.args...)...)
assert.Equal(t, 1, code)
stored, err := os.ReadDir(storeDir)
require.NoError(t, err)
assert.Empty(t, stored, "nothing may be uploaded")
})
}
}
// TestEntrySnapshotCreateReportsPostBackupPruneFailure makes the
// post-backup prune fail after a good backup, by planting a manifest
// that cannot be decoded, and checks that the failure is reported as the
// prune's and not as a failed snapshot creation.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateReportsPostBackupPruneFailure(t *testing.T) {
configPath, storeDir := writeBackupConfig(t)
// A remote key is 64 lowercase hex characters. The prune reads every
// manifest on the store and fails on one it cannot decode.
plantedDir := filepath.Join(storeDir, "metadata", strings.Repeat("a", 64))
require.NoError(t, os.MkdirAll(plantedDir, 0o750))
require.NoError(t, os.WriteFile(
filepath.Join(plantedDir, "manifest.json.zst"),
[]byte("not a valid manifest"), 0o600))
code, _, stderr := runEntry(t,
flagConfig, configPath, cmdSnapshot, cmdCreate, flagPrune)
assert.Equal(t, 1, code)
assert.Contains(t, stderr, "Backup completed, but post-backup prune failed")
assert.NotContains(t, stderr, "Snapshot creation failed")
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
require.NoError(t, err)
assert.Len(t, snapshots, 2,
"the new snapshot is on the store beside the planted one")
}
// writeBackupConfig builds a hermetic config with one small file in its
// source directory and an empty destination store directory. Returns the
// config path and the store directory.
func writeBackupConfig(t *testing.T) (string, string) {
t.Helper()
configPath := writeHermeticPruneConfig(t, false)
// writeHermeticPruneConfig puts the source and store directories
// beside the config file.
dir := filepath.Dir(configPath)
sourceDir := filepath.Join(dir, "source")
require.NoError(t, os.Mkdir(sourceDir, 0o750))
require.NoError(t, os.WriteFile(
filepath.Join(sourceDir, "file.txt"), []byte("hello"), 0o600))
return configPath, filepath.Join(dir, "store")
}
@@ -1,43 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the command-word constants
import (
"testing"
"github.com/stretchr/testify/assert"
)
// TestEntryGroupRejectsUnknownSubcommand runs each group command with a
// mistyped subcommand, as a crontab line might carry one. It must fail
// with exit 1 and name the word, as `vaultik bogus` does at the top
// level, instead of printing the group's help and exiting 0 having done
// nothing. The group name alone still prints its help.
//
//nolint:paralleltest // replaces os.Args, os.Stdout and os.Stderr
func TestEntryGroupRejectsUnknownSubcommand(t *testing.T) {
for _, testCase := range []struct {
group string
mistyped string
}{
{group: cmdSnapshot, mistyped: "craete"},
{group: "config", mistyped: "inti"},
{group: cmdRemote, mistyped: "ifno"},
{group: "database", mistyped: "delte"},
{group: "completion", mistyped: "nosuchshell"},
} {
t.Run(testCase.group, func(t *testing.T) {
code, stdout, stderr := runEntry(t,
testCase.group, testCase.mistyped)
assert.Equal(t, 1, code)
assert.Empty(t, stdout)
assert.Contains(t, stderr,
`unknown command "`+testCase.mistyped+`"`)
code, stdout, _ = runEntry(t, testCase.group)
assert.Equal(t, 0, code)
assert.Contains(t, stdout,
programName+" "+testCase.group+" [command]")
})
}
}
-6
View File
@@ -18,12 +18,6 @@ func NewRemoteCommand() *cobra.Command {
Use: "remote", Use: "remote",
Short: "Remote storage management commands", Short: "Remote storage management commands",
Long: "Commands for inspecting and managing remote storage", Long: "Commands for inspecting and managing remote storage",
// cobra checks Args only on a command that has a Run: without
// one, a mistyped subcommand printed help and exited 0.
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, _ []string) {
_ = cmd.Help()
},
} }
// Add subcommands // Add subcommands
-15
View File
@@ -74,21 +74,6 @@ on the source system.`,
NewDatabaseCommand(), NewDatabaseCommand(),
) )
// cobra adds the completion command only when Execute runs, and with
// no Run, so it skipped its own cobra.NoArgs and a mistyped shell name
// printed help and exited 0. Adding it here lets it have a Run. Its
// scripts go to os.Stdout as it is now: a later SetOut does not
// redirect them.
cmd.InitDefaultCompletionCmd()
for _, sub := range cmd.Commands() {
if sub.Name() == "completion" {
sub.Run = func(completion *cobra.Command, _ []string) {
_ = completion.Help()
}
}
}
return cmd return cmd
} }
-18
View File
@@ -41,12 +41,6 @@ func NewSnapshotCommand() *cobra.Command {
Use: "snapshot", Use: "snapshot",
Short: "Snapshot management commands", Short: "Snapshot management commands",
Long: "Commands for creating, listing, and managing snapshots", Long: "Commands for creating, listing, and managing snapshots",
// cobra checks Args only on a command that has a Run: without
// one, a mistyped subcommand printed help and exited 0.
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, _ []string) {
_ = cmd.Help()
},
} }
// Add subcommands // Add subcommands
@@ -77,11 +71,6 @@ otherwise from the platform config directory (~/.config/vaultik/config.yml
on Linux), then /etc/vaultik/config.yml.`, on Linux), then /etc/vaultik/config.yml.`,
Args: cobra.ArbitraryArgs, Args: cobra.ArbitraryArgs,
RunE: func(cmd *cobra.Command, args []string) error { RunE: func(cmd *cobra.Command, args []string) error {
err := opts.Validate()
if err != nil {
return err
}
// Pass snapshot names from args // Pass snapshot names from args
opts.Snapshots = args opts.Snapshots = args
// --skip-errors is a global flag on the root command. // --skip-errors is a global flag on the root command.
@@ -108,13 +97,6 @@ on Linux), then /etc/vaultik/config.yml.`,
}, func(v *vaultik.Vaultik) error { }, func(v *vaultik.Vaultik) error {
return v.CreateSnapshot(opts) return v.CreateSnapshot(opts)
}, func(err error) { }, func(err error) {
if errors.Is(err, vaultik.ErrPostBackupPruneFailed) {
log.Error("Post-backup prune failed", "error", err)
ReportErrorf("Backup completed, but %v", err)
return
}
log.Error("Snapshot creation failed", "error", err) log.Error("Snapshot creation failed", "error", err)
ReportErrorf("Snapshot creation failed: %v", err) ReportErrorf("Snapshot creation failed: %v", err)
}) })
+2 -12
View File
@@ -61,9 +61,6 @@ var (
errBadS3PartSize = errors.New("s3.part_size must be between 5MiB and 5GiB") errBadS3PartSize = errors.New("s3.part_size must be between 5MiB and 5GiB")
errBadStorageScheme = errors.New( errBadStorageScheme = errors.New(
"storage_url must start with s3://, file://, or rclone://") "storage_url must start with s3://, file://, or rclone://")
errFileURLNotAbsolute = errors.New(
"a file:// storage_url needs an absolute path, " +
"as in file:///path/to/backup")
errStorageNotConfigured = errors.New( errStorageNotConfigured = errors.New(
"storage not configured; set storage_url or provide s3.endpoint + " + "storage not configured; set storage_url or provide s3.endpoint + " +
"s3.bucket + credentials") "s3.bucket + credentials")
@@ -464,18 +461,11 @@ func (c *Config) validateStorage() error {
// validateStorageURL validates URL-based storage configuration. File and // validateStorageURL validates URL-based storage configuration. File and
// rclone URLs need no credentials; S3 URLs require the legacy s3.* // rclone URLs need no credentials; S3 URLs require the legacy s3.*
// credential fields. A file URL must name an absolute path: a relative one // credential fields.
// would resolve against the working directory, so the destination would
// move with it.
func (c *Config) validateStorageURL() error { func (c *Config) validateStorageURL() error {
switch { switch {
case strings.HasPrefix(c.StorageURL, "file://"): case strings.HasPrefix(c.StorageURL, "file://"):
// storage.ParseStorageURL applies the same rule, but config // File storage doesn't need S3 credentials
// cannot import storage, which imports config.
if !strings.HasPrefix(c.StorageURL, "file:///") {
return errFileURLNotAbsolute
}
return nil return nil
case strings.HasPrefix(c.StorageURL, "rclone://"): case strings.HasPrefix(c.StorageURL, "rclone://"):
// Rclone storage uses rclone's own config // Rclone storage uses rclone's own config
-29
View File
@@ -337,35 +337,6 @@ func TestLoadS3PartSize(t *testing.T) {
}) })
} }
// TestLoadRejectsRelativeFileURL checks that a file:// storage_url with a
// relative path fails at load. Accepted, it would resolve against the
// working directory, so a backup run from another directory would write
// to a different destination and still succeed.
func TestLoadRejectsRelativeFileURL(t *testing.T) {
t.Parallel()
const relativeConfig = "snapshots:\n" +
" test:\n" +
" paths: [/tmp/vaultik-test-source]\n" +
"storage_url: file://dest\n"
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(relativeConfig), 0o600)
if err != nil {
t.Fatalf("write config: %v", err)
}
_, err = Load(path)
if !errors.Is(err, errFileURLNotAbsolute) {
t.Fatalf("Load() error = %v, want errFileURLNotAbsolute", err)
}
if !strings.Contains(err.Error(), "absolute path") {
t.Errorf("error %q does not say to use an absolute path", err.Error())
}
}
// TestValidateAgeRecipients checks that recipients are parsed at config load // TestValidateAgeRecipients checks that recipients are parsed at config load
// (a bad entry fails immediately, not mid-backup) and that no invalid entry — // (a bad entry fails immediately, not mid-backup) and that no invalid entry —
// least of all a pasted secret key — is echoed in the error. An empty list // least of all a pasted secret key — is echoed in the error. An empty list
+9 -12
View File
@@ -3,7 +3,6 @@ package database
import ( import (
"context" "context"
"database/sql" "database/sql"
"errors"
"fmt" "fmt"
) )
@@ -61,24 +60,22 @@ func (r *Repositories) WithTx(ctx context.Context, fn TxFunc) error {
defer func() { defer func() {
if p := recover(); p != nil { if p := recover(); p != nil {
// The panic is the failure to report, so a rollback error rollbackErr := tx.Rollback()
// is dropped. if rollbackErr != nil {
_ = tx.Rollback() Fatalf("failed to rollback transaction: %v", rollbackErr)
}
panic(p) panic(p)
} else if err != nil {
rollbackErr := tx.Rollback()
if rollbackErr != nil {
Fatalf("failed to rollback transaction: %v", rollbackErr)
}
} }
}() }()
err = fn(ctx, tx) err = fn(ctx, tx)
if err != nil { if err != nil {
// Once ctx is cancelled, database/sql rolls the transaction
// back itself and Rollback returns sql.ErrTxDone.
rollbackErr := tx.Rollback()
if rollbackErr != nil && !errors.Is(rollbackErr, sql.ErrTxDone) {
return fmt.Errorf("%w; rolling back transaction: %w",
err, rollbackErr)
}
return err return err
} }
-120
View File
@@ -4,7 +4,6 @@ import (
"context" "context"
"database/sql" "database/sql"
"errors" "errors"
"strings"
"testing" "testing"
"time" "time"
@@ -236,125 +235,6 @@ func TestRepositoriesTransactionRollback(t *testing.T) {
} }
} }
// When the context is cancelled during the body, database/sql rolls the
// transaction back itself, so WithTx's own rollback finds it already
// done. WithTx must return the body's error alone, without that rollback.
func TestRepositoriesTransactionContextCancelled(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
repos := database.NewRepositories(db)
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
file := &database.File{
Path: "/test/cancelled_file.txt",
MTime: time.Now().Truncate(time.Second),
Size: 1024,
Mode: 0644,
UID: 1000,
GID: 1000,
}
err := repos.Files.Create(ctx, tx, file)
if err != nil {
return err
}
cancel()
// database/sql rolls back from a goroutine of its own; wait
// until it has, so the test does not depend on that race.
for {
_, err = tx.ExecContext(context.WithoutCancel(ctx), "SELECT 1")
if errors.Is(err, sql.ErrTxDone) {
return ctx.Err()
}
time.Sleep(time.Millisecond)
}
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("expected context.Canceled, got: %v", err)
}
if errors.Is(err, sql.ErrTxDone) {
t.Fatalf("the already-done rollback was reported: %v", err)
}
file, err := repos.Files.GetByPath(
context.Background(), "/test/cancelled_file.txt",
)
if err != nil {
t.Fatalf("error checking for file: %v", err)
}
if file != nil {
t.Error("file should not exist after the cancelled transaction")
}
}
// A body that ends the transaction itself with ROLLBACK makes WithTx's
// own rollback fail. WithTx must return that failure together with the
// body's error.
func TestRepositoriesTransactionRollbackFails(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := database.NewRepositories(db)
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
_, err := tx.ExecContext(ctx, "ROLLBACK")
if err != nil {
return err
}
return errIntentionalRollback
})
if !errors.Is(err, errIntentionalRollback) {
t.Fatalf("expected the body's error, got: %v", err)
}
if !strings.Contains(err.Error(), "rolling back transaction") {
t.Errorf("expected the failed rollback in the error, got: %v", err)
}
}
// When the body ends the transaction itself with ROLLBACK and then
// panics, WithTx's own rollback fails; the panic must still reach the
// caller.
func TestRepositoriesTransactionPanicRollbackFails(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := database.NewRepositories(db)
defer func() {
if recover() == nil {
t.Error("expected the body's panic to reach the caller")
}
}()
_ = repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
_, err := tx.ExecContext(ctx, "ROLLBACK")
if err != nil {
return err
}
panic(errIntentionalRollback)
})
}
func TestRepositoriesReadTransaction(t *testing.T) { func TestRepositoriesReadTransaction(t *testing.T) {
t.Parallel() t.Parallel()
-1
View File
@@ -86,7 +86,6 @@ CREATE TABLE IF NOT EXISTS snapshots (
completed_at INTEGER, completed_at INTEGER,
file_count INTEGER NOT NULL DEFAULT 0, file_count INTEGER NOT NULL DEFAULT 0,
chunk_count INTEGER NOT NULL DEFAULT 0, chunk_count INTEGER NOT NULL DEFAULT 0,
new_chunk_size INTEGER NOT NULL DEFAULT 0,
blob_count INTEGER NOT NULL DEFAULT 0, blob_count INTEGER NOT NULL DEFAULT 0,
total_size INTEGER NOT NULL DEFAULT 0, total_size INTEGER NOT NULL DEFAULT 0,
blob_size INTEGER NOT NULL DEFAULT 0, blob_size INTEGER NOT NULL DEFAULT 0,
+33 -14
View File
@@ -129,7 +129,6 @@ func (r *SnapshotRepository) UpdateExtendedStats(
compressionLevel int, compressionLevel int,
uploadBytes int64, uploadBytes int64,
uploadDurationMs int64, uploadDurationMs int64,
newChunkSize int64,
) error { ) error {
compressionRatio, err := r.extendedCompressionRatio( compressionRatio, err := r.extendedCompressionRatio(
ctx, tx, snapshotID, blobUncompressedSize, ctx, tx, snapshotID, blobUncompressedSize,
@@ -144,19 +143,18 @@ func (r *SnapshotRepository) UpdateExtendedStats(
compression_ratio = ?, compression_ratio = ?,
compression_level = ?, compression_level = ?,
upload_bytes = ?, upload_bytes = ?,
upload_duration_ms = ?, upload_duration_ms = ?
new_chunk_size = ?
WHERE id = ? WHERE id = ?
` `
if tx != nil { if tx != nil {
_, err = tx.ExecContext(ctx, query, _, err = tx.ExecContext(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel, blobUncompressedSize, compressionRatio, compressionLevel,
uploadBytes, uploadDurationMs, newChunkSize, snapshotID) uploadBytes, uploadDurationMs, snapshotID)
} else { } else {
_, err = r.db.ExecWithLog(ctx, query, _, err = r.db.ExecWithLog(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel, blobUncompressedSize, compressionRatio, compressionLevel,
uploadBytes, uploadDurationMs, newChunkSize, snapshotID) uploadBytes, uploadDurationMs, snapshotID)
} }
if err != nil { if err != nil {
@@ -596,24 +594,45 @@ func (r *SnapshotRepository) GetSnapshotUncompressedChunkSize(
return totalSize, nil return totalSize, nil
} }
// GetSnapshotNewChunkSize returns the total plaintext size of the chunks the // GetSnapshotNewChunkSize returns the sum of plaintext sizes of chunks that are
// snapshot's backup stored that were not stored before, as the backup // referenced by this snapshot but not by any earlier completed snapshot known to
// recorded it. It is not derived from the snapshot's files: a file's row is // the local database. The result is the marginal uncompressed data this snapshot
// shared by every snapshot that lists it and is rewritten when the file // added to the dedup pool — i.e., the delta from prior snapshots.
// changes, so it shows only the file's latest chunks.
func (r *SnapshotRepository) GetSnapshotNewChunkSize( func (r *SnapshotRepository) GetSnapshotNewChunkSize(
ctx context.Context, snapshotID string, ctx context.Context, snapshotID string,
) (int64, error) { ) (int64, error) {
query := `SELECT new_chunk_size FROM snapshots WHERE id = ?` query := `
WITH this_snap_chunks AS (
SELECT DISTINCT fc.chunk_hash
FROM snapshot_files sf
JOIN file_chunks fc ON sf.file_id = fc.file_id
WHERE sf.snapshot_id = ?
),
prior_chunks AS (
SELECT DISTINCT fc.chunk_hash
FROM snapshots s
JOIN snapshot_files sf ON sf.snapshot_id = s.id
JOIN file_chunks fc ON fc.file_id = sf.file_id
WHERE s.completed_at IS NOT NULL
AND s.id != ?
AND s.started_at < (SELECT started_at FROM snapshots WHERE id = ?)
)
SELECT COALESCE(SUM(c.size), 0)
FROM chunks c
JOIN this_snap_chunks t ON c.chunk_hash = t.chunk_hash
WHERE c.chunk_hash NOT IN (SELECT chunk_hash FROM prior_chunks)
`
var newChunkSize int64 var totalSize int64
err := r.db.conn.QueryRowContext(ctx, query, snapshotID).Scan(&newChunkSize) err := r.db.conn.QueryRowContext(
ctx, query, snapshotID, snapshotID, snapshotID,
).Scan(&totalSize)
if err != nil { if err != nil {
return 0, fmt.Errorf("querying new chunk size: %w", err) return 0, fmt.Errorf("querying new chunk size: %w", err)
} }
return newChunkSize, nil return totalSize, nil
} }
// GetIncompleteSnapshots returns all snapshots that haven't been completed // GetIncompleteSnapshots returns all snapshots that haven't been completed
+10 -7
View File
@@ -10,8 +10,6 @@ import (
"sync" "sync"
"time" "time"
"unicode" "unicode"
"github.com/dustin/go-humanize"
) )
// groupSeparator joins an open group path to an attribute key. This // groupSeparator joins an open group path to an attribute key. This
@@ -306,13 +304,18 @@ func formatDuration(d time.Duration) string {
} }
} }
// formatBytes formats a byte count with humanize.Bytes, as every command // formatBytes formats bytes in a human-readable way
// prints one. humanize.Bytes takes no negative count, so one is printed
// as it is.
func formatBytes(b int64) string { func formatBytes(b int64) string {
if b < 0 { const unit = 1024
if b < unit {
return fmt.Sprintf("%d B", b) return fmt.Sprintf("%d B", b)
} }
return humanize.Bytes(uint64(b)) div, exp := int64(unit), 0
for n := b / unit; n >= unit; n /= unit {
div *= unit
exp++
}
return fmt.Sprintf("%.1f %cB", float64(b)/float64(div), "KMGTPE"[exp])
} }
+4 -5
View File
@@ -200,8 +200,7 @@ func TestTTYHandlerWithGroupQualifiesKeys(t *testing.T) {
func TestTTYHandlerByteFormattingSurvivesGrouping(t *testing.T) { func TestTTYHandlerByteFormattingSurvivesGrouping(t *testing.T) {
t.Parallel() t.Parallel()
// Printed as "1.0 MB", in powers of 1000 like every other byte count. const oneAndAHalfKiB = 1536
const oneMegabyte = 1_000_000
for name, testCase := range map[string]struct { for name, testCase := range map[string]struct {
derive func(*slog.Logger) *slog.Logger derive func(*slog.Logger) *slog.Logger
@@ -224,13 +223,13 @@ func TestTTYHandlerByteFormattingSurvivesGrouping(t *testing.T) {
var buf bytes.Buffer var buf bytes.Buffer
logger := slog.New(log.NewTTYHandler(&buf, debugHandlerOptions())) logger := slog.New(log.NewTTYHandler(&buf, debugHandlerOptions()))
testCase.derive(logger).Info("uploaded", "bytes", oneMegabyte) testCase.derive(logger).Info("uploaded", "bytes", oneAndAHalfKiB)
line := ansiEscape.ReplaceAllString(buf.String(), "") line := ansiEscape.ReplaceAllString(buf.String(), "")
assert.Contains(t, line, testCase.key+"=1.0 MB", assert.Contains(t, line, testCase.key+"=1.5 KB",
"a byte count must be human-readable however it is qualified") "a byte count must be human-readable however it is qualified")
assert.NotContains(t, line, strconv.Itoa(oneMegabyte), assert.NotContains(t, line, strconv.Itoa(oneAndAHalfKiB),
"the raw number must not survive the formatting") "the raw number must not survive the formatting")
}) })
} }
-47
View File
@@ -531,50 +531,3 @@ func TestExcludePatterns_UnanchoredPatternFile(t *testing.T) {
require.Equal(t, 1, result.FilesScanned, require.Equal(t, 1, result.FilesScanned,
"Unanchored pattern for file should match anywhere") "Unanchored pattern for file should match anywhere")
} }
func TestExcludePatterns_DoubleStarDirectoryAtEveryDepth(t *testing.T) {
t.Parallel()
// As in gitignore, a leading **/ also matches directly under the source dir
fs := setupAnchoredTestFS(t)
scanner, repos, cleanup := createTestScanner(t, fs, []string{"**/projectname"})
defer cleanup()
ctx := context.Background()
createSnapshotRecord(ctx, t, repos, "test-snapshot")
result, err := scanner.Scan(ctx, "/backup", "test-snapshot")
require.NoError(t, err)
// **/projectname should exclude BOTH:
// - /backup/projectname/file.txt
// - /backup/otherproject/projectname/file.txt
// Total files: 4, excluded: 2
require.Equal(t, 2, result.FilesScanned,
"**/ directory pattern should match at the top level and nested")
}
func TestExcludePatterns_DoubleStarFileAtEveryDepth(t *testing.T) {
t.Parallel()
// As in gitignore, a leading **/ also matches directly under the source dir
fs := setupAnchoredTestFS(t)
scanner, repos, cleanup := createTestScanner(t, fs, []string{"**/*.txt"})
defer cleanup()
ctx := context.Background()
createSnapshotRecord(ctx, t, repos, "test-snapshot")
result, err := scanner.Scan(ctx, "/backup", "test-snapshot")
require.NoError(t, err)
// **/*.txt should exclude ALL .txt files:
// - /backup/file.txt
// - /backup/projectname/file.txt
// - /backup/otherproject/projectname/file.txt
// Total files: 4, excluded: 3
require.Equal(t, 1, result.FilesScanned,
"**/ file pattern should match at the top level and nested")
}
+4 -21
View File
@@ -135,9 +135,7 @@ type ScannerConfig struct {
// are counted per file: BytesScanned is the size of the new and changed // are counted per file: BytesScanned is the size of the new and changed
// files, BytesSkipped that of the unchanged ones. FilesFailed counts the // files, BytesSkipped that of the unchanged ones. FilesFailed counts the
// new and changed files that phase 2 could not store; FilesScanned // new and changed files that phase 2 could not store; FilesScanned
// includes them and BytesScanned does not. ChunksCreated counts the // includes them and BytesScanned does not.
// chunks that were not already stored, and NewChunkSize is their total
// size.
type ScanResult struct { type ScanResult struct {
FilesScanned int FilesScanned int
FilesSkipped int FilesSkipped int
@@ -147,7 +145,6 @@ type ScanResult struct {
BytesSkipped int64 BytesSkipped int64
BytesDeleted int64 BytesDeleted int64
ChunksCreated int ChunksCreated int
NewChunkSize int64
BlobsCreated int BlobsCreated int
BlobsUploaded int BlobsUploaded int
BytesUploaded int64 BytesUploaded int64
@@ -1298,11 +1295,6 @@ func (s *Scanner) batchAddFilesToSnapshot(
func (s *Scanner) processPhase( func (s *Scanner) processPhase(
ctx context.Context, filesToProcess []*FileToProcess, result *ScanResult, ctx context.Context, filesToProcess []*FileToProcess, result *ScanResult,
) error { ) error {
// On an error or cancellation the blob being packed is never
// finalized, so its temp file must be removed here. After a
// successful flush there is no such blob and this does nothing.
defer s.packer.DiscardCurrentBlob()
// Calculate total bytes to process // Calculate total bytes to process
var totalBytes int64 var totalBytes int64
for _, f := range filesToProcess { for _, f := range filesToProcess {
@@ -1869,12 +1861,11 @@ func (s *Scanner) processFileStreaming(
return nil return nil
} }
// updateChunkStats counts a chunk that was not already stored, and its // updateChunkStats counts a chunk that was not already stored. The scan
// size. The scan result's file counts, BytesScanned and BytesSkipped are // result's file counts, BytesScanned and BytesSkipped are not touched
// not touched here: the scan phase counts each file once. // here: the scan phase counts each file once.
func (s *Scanner) updateChunkStats(chunkSize int64, result *ScanResult) { func (s *Scanner) updateChunkStats(chunkSize int64, result *ScanResult) {
result.ChunksCreated++ result.ChunksCreated++
result.NewChunkSize += chunkSize
if s.progress != nil { if s.progress != nil {
s.progress.GetStats().ChunksCreated.Add(1) s.progress.GetStats().ChunksCreated.Add(1)
@@ -2022,14 +2013,6 @@ func compileExcludePatterns(patterns []string) []compiledPattern {
// Remove trailing slash if present (directory indicator) // Remove trailing slash if present (directory indicator)
pattern = strings.TrimSuffix(pattern, "/") pattern = strings.TrimSuffix(pattern, "/")
// A leading "**/" demands a "/" before the name and so misses an
// entry directly under the source path. An unanchored pattern
// already matches at every depth, so drop the prefix: as in
// gitignore, "**/x" then also matches a top-level "x".
if !anchored {
pattern = strings.TrimPrefix(pattern, "**/")
}
// Compile the glob pattern // Compile the glob pattern
// For patterns without path separators, we need to match them as components // For patterns without path separators, we need to match them as components
// e.g., ".git" should match ".git" anywhere in the path // e.g., ".git" should match ".git" anywhere in the path
-57
View File
@@ -381,60 +381,3 @@ func TestScannerCancelStopsSkipErrorsRun(t *testing.T) {
}) })
} }
} }
// TestScannerRemovesBlobTempFileWhenRunStops checks that a run stopped
// part-way, by a file it cannot read or by a cancellation, removes the
// temp file of the blob it was packing.
func TestScannerRemovesBlobTempFileWhenRunStops(t *testing.T) {
t.Parallel()
// a.txt is backed up first and starts a blob; the run stops at b.txt.
const target = "/source/b.txt"
tests := []struct {
name string
newFs func(cancel context.CancelFunc) afero.Fs
}{
{
name: "file cannot be read",
newFs: func(context.CancelFunc) afero.Fs {
return &readFailFs{Fs: afero.NewMemMapFs(), target: target}
},
},
{
name: "cancelled",
newFs: func(cancel context.CancelFunc) afero.Fs {
return &cancelOnOpenFs{
Fs: afero.NewMemMapFs(), target: target, cancel: cancel,
}
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
fs := tt.newFs(cancel)
writeSkipErrorTestFile(t, fs, "/source/a.txt", "first file content")
writeSkipErrorTestFile(t, fs, target, "second file content")
_, err := runSkipErrorScan(ctx, t, fs, "/source", false, nil)
if err == nil {
t.Fatal("expected the scan to stop with an error, got nil")
}
left, err := afero.Glob(fs, filepath.Join(os.TempDir(), "vaultik-blob-*"))
if err != nil {
t.Fatalf("listing blob temp files: %v", err)
}
if len(left) != 0 {
t.Fatalf("expected no blob temp file left, got %v", left)
}
})
}
}
-2
View File
@@ -199,7 +199,6 @@ func (sm *SnapshotManager) UpdateSnapshotStatsExtended(
stats.CompressionLevel, stats.CompressionLevel,
stats.BytesUploaded, stats.BytesUploaded,
stats.UploadDurationMs, stats.UploadDurationMs,
stats.NewChunkSize,
) )
}) })
} }
@@ -907,7 +906,6 @@ type ExtendedBackupStats struct {
BlobUncompressedSize int64 // Total uncompressed size of all referenced blobs BlobUncompressedSize int64 // Total uncompressed size of all referenced blobs
CompressionLevel int // Compression level used for this snapshot CompressionLevel int // Compression level used for this snapshot
UploadDurationMs int64 // Total milliseconds spent uploading to S3 UploadDurationMs int64 // Total milliseconds spent uploading to S3
NewChunkSize int64 // Total size of the chunks counted in ChunksCreated
} }
// deleteOtherSnapshots deletes all snapshots except the current one // deleteOtherSnapshots deletes all snapshots except the current one
+23 -9
View File
@@ -1,15 +1,16 @@
// Package faultstore provides a storage.Storer wrapper that injects // Package faultstore provides a storage.Storer wrapper that injects
// faults on demand, so tests can reproduce the failure modes a real // faults on demand, so tests can reproduce the failure modes a real
// backend exhibits: an upload that fails partway, a backend that reports // backend exhibits: an upload that fails partway, a backend that reports
// success while storing nothing, and reads that return corrupt or // success while storing nothing, reads that return corrupt or truncated
// truncated bytes. It is the seam called for by the fault-injection // bytes, and a delete the backend refuses. It is the seam called for by
// tests (sneak/vaultik issue 72) and is meant to be reused by future // the fault-injection tests (sneak/vaultik issue 72) and is meant to be
// tests rather than re-implemented per case. // reused by future tests rather than re-implemented per case.
// //
// The wrapper delegates every method to the inner Storer. Two hooks // The wrapper delegates every method to the inner Storer. Three hooks
// change that: OnPut decides the fate of each write, and OnGet decides // change that: OnPut decides the fate of each write, OnGet decides how
// how each read's bytes are returned. Both are keyed by the object key, // each read's bytes are returned, and OnDelete decides whether each
// so a test can fault only blobs, only metadata, or a single object. // delete fails. All are keyed by the object key, so a test can fault
// only blobs, only metadata, or a single object.
package faultstore package faultstore
import ( import (
@@ -25,6 +26,10 @@ import (
// ErrInjectedUpload is returned by a Put the OnPut hook chose to fail. // ErrInjectedUpload is returned by a Put the OnPut hook chose to fail.
var ErrInjectedUpload = errors.New("faultstore: injected upload failure") var ErrInjectedUpload = errors.New("faultstore: injected upload failure")
// ErrInjectedDelete is returned by a Delete the OnDelete hook chose to
// fail.
var ErrInjectedDelete = errors.New("faultstore: injected delete failure")
// PutAction is the disposition OnPut assigns to a write. // PutAction is the disposition OnPut assigns to a write.
type PutAction int type PutAction int
@@ -66,6 +71,11 @@ type Storer struct {
// OnGet, when set, is consulted for every Get with the object key // OnGet, when set, is consulted for every Get with the object key
// and damages the returned bytes accordingly. // and damages the returned bytes accordingly.
OnGet func(key string) GetFault OnGet func(key string) GetFault
// OnDelete, when set, is consulted before every Delete with the
// object key. Returning true fails the delete and leaves the object
// in place.
OnDelete func(key string) bool
} }
// New wraps inner. inner must be non-nil. // New wraps inner. inner must be non-nil.
@@ -154,8 +164,12 @@ func (f *Storer) Stat(ctx context.Context, key string) (*storage.ObjectInfo, err
return f.inner.Stat(ctx, key) return f.inner.Stat(ctx, key)
} }
// Delete delegates unchanged. // Delete removes the object unless OnDelete fails the delete.
func (f *Storer) Delete(ctx context.Context, key string) error { func (f *Storer) Delete(ctx context.Context, key string) error {
if f.OnDelete != nil && f.OnDelete(key) {
return fmt.Errorf("%w for %q", ErrInjectedDelete, key)
}
return f.inner.Delete(ctx, key) return f.inner.Delete(ctx, key)
} }
+12 -26
View File
@@ -17,10 +17,8 @@ const (
// Sentinel errors for storage URL parsing. // Sentinel errors for storage URL parsing.
var ( var (
ErrEmptyStorageURL = errors.New("storage URL is empty") ErrEmptyStorageURL = errors.New("storage URL is empty")
ErrEmptyFilePath = errors.New("file URL path is empty") ErrEmptyFilePath = errors.New("file URL path is empty")
ErrRelativeFilePath = errors.New(
"file URL needs an absolute path, as in file:///path/to/backup")
ErrMissingBucket = errors.New("s3 URL missing bucket name") ErrMissingBucket = errors.New("s3 URL missing bucket name")
ErrMissingRemote = errors.New("rclone URL missing remote name") ErrMissingRemote = errors.New("rclone URL missing remote name")
ErrUnsupportedScheme = errors.New( ErrUnsupportedScheme = errors.New(
@@ -53,8 +51,17 @@ func ParseStorageURL(rawURL string) (*URL, error) {
return nil, ErrEmptyStorageURL return nil, ErrEmptyStorageURL
} }
// Handle file:// URLs
if after, ok := strings.CutPrefix(rawURL, "file://"); ok { if after, ok := strings.CutPrefix(rawURL, "file://"); ok {
return parseFileURL(after) path := after
if path == "" {
return nil, ErrEmptyFilePath
}
return &URL{
Scheme: schemeFile,
Prefix: path,
}, nil
} }
if strings.HasPrefix(rawURL, "s3://") { if strings.HasPrefix(rawURL, "s3://") {
@@ -68,27 +75,6 @@ func ParseStorageURL(rawURL string) (*URL, error) {
return nil, ErrUnsupportedScheme return nil, ErrUnsupportedScheme
} }
// parseFileURL parses the path that follows file://. The path must be
// absolute: a relative one would resolve against the working directory,
// so the destination would move with it. Percent-escapes are decoded, so
// my%20stick names the directory "my stick"; a literal space is kept.
func parseFileURL(rawPath string) (*URL, error) {
if rawPath == "" {
return nil, ErrEmptyFilePath
}
if !strings.HasPrefix(rawPath, "/") {
return nil, ErrRelativeFilePath
}
path, err := url.PathUnescape(rawPath)
if err != nil {
return nil, fmt.Errorf("invalid file URL path: %w", err)
}
return &URL{Scheme: schemeFile, Prefix: path}, nil
}
// parseS3URL parses an s3://bucket/prefix URL. It rejects credentials in // parseS3URL parses an s3://bucket/prefix URL. It rejects credentials in
// the userinfo and any query parameter other than endpoint, region and // the userinfo and any query parameter other than endpoint, region and
// ssl, so a credential-bearing URL is never stored or echoed. // ssl, so a credential-bearing URL is never stored or echoed.
+2 -16
View File
@@ -14,10 +14,7 @@ import (
func TestParseStorageURLValid(t *testing.T) { func TestParseStorageURLValid(t *testing.T) {
t.Parallel() t.Parallel()
const ( const bucket = "mybucket"
bucket = "mybucket"
fileScheme = "file"
)
cases := []struct { cases := []struct {
name string name string
@@ -27,17 +24,7 @@ func TestParseStorageURLValid(t *testing.T) {
{ {
name: "file absolute path", name: "file absolute path",
raw: "file:///var/backups/vaultik", raw: "file:///var/backups/vaultik",
want: &storage.URL{Scheme: fileScheme, Prefix: "/var/backups/vaultik"}, want: &storage.URL{Scheme: "file", Prefix: "/var/backups/vaultik"},
},
{
name: "file percent-escaped path is decoded",
raw: "file:///Volumes/my%20stick/backup",
want: &storage.URL{Scheme: fileScheme, Prefix: "/Volumes/my stick/backup"},
},
{
name: "file path with a literal space",
raw: "file:///Volumes/my stick/backup",
want: &storage.URL{Scheme: fileScheme, Prefix: "/Volumes/my stick/backup"},
}, },
{ {
name: "s3 bucket and prefix, ssl defaults on", name: "s3 bucket and prefix, ssl defaults on",
@@ -103,7 +90,6 @@ func TestParseStorageURLErrors(t *testing.T) {
}{ }{
{"empty url", "", storage.ErrEmptyStorageURL}, {"empty url", "", storage.ErrEmptyStorageURL},
{"file empty path", "file://", storage.ErrEmptyFilePath}, {"file empty path", "file://", storage.ErrEmptyFilePath},
{"file relative path", "file://dest", storage.ErrRelativeFilePath},
{"s3 missing bucket", "s3://", storage.ErrMissingBucket}, {"s3 missing bucket", "s3://", storage.ErrMissingBucket},
{"s3 missing bucket with path", "s3:///justprefix", storage.ErrMissingBucket}, {"s3 missing bucket with path", "s3:///justprefix", storage.ErrMissingBucket},
{"rclone missing remote", "rclone://", storage.ErrMissingRemote}, {"rclone missing remote", "rclone://", storage.ErrMissingRemote},
+1 -45
View File
@@ -103,8 +103,6 @@ func (v *Vaultik) FetchAndDecryptBlob(
} }
// FetchBlob downloads a blob and returns a reader for the encrypted data. // FetchBlob downloads a blob and returns a reader for the encrypted data.
// The reader is closed as soon as ctx is cancelled, which ends a read
// blocked inside it.
func (v *Vaultik) FetchBlob( func (v *Vaultik) FetchBlob(
ctx context.Context, blobHash string, ctx context.Context, blobHash string,
) (io.ReadCloser, error) { ) (io.ReadCloser, error) {
@@ -122,47 +120,5 @@ func (v *Vaultik) FetchBlob(
return nil, fmt.Errorf("downloading blob %s: %w", shortHash(blobHash), err) return nil, fmt.Errorf("downloading blob %s: %w", shortHash(blobHash), err)
} }
return newCloseOnCancelReader(ctx, rc), nil return rc, nil
}
// closeOnCancelReader closes the reader it wraps when ctx is cancelled.
// Not every backend's reader stops on cancel by itself (the file
// backend's does not), and a read stalled on a slow or hung source would
// otherwise outlast the shutdown timeout, so the process would exit
// before restore or deep verify removed its decrypted temporary files.
type closeOnCancelReader struct {
io.ReadCloser
ctx context.Context //nolint:containedctx // Read reports its error
stop func() bool
}
func newCloseOnCancelReader(
ctx context.Context, rc io.ReadCloser,
) *closeOnCancelReader {
return &closeOnCancelReader{
ReadCloser: rc,
ctx: ctx,
stop: context.AfterFunc(ctx, func() { _ = rc.Close() }),
}
}
// Read returns the context's error, not the closed reader's, once ctx
// has been cancelled.
func (r *closeOnCancelReader) Read(p []byte) (int, error) {
n, err := r.ReadCloser.Read(p)
if err != nil && r.ctx.Err() != nil {
return n, r.ctx.Err()
}
return n, err
}
// Close closes the wrapped reader, unless the cancel already closed it.
func (r *closeOnCancelReader) Close() error {
if !r.stop() {
return nil
}
return r.ReadCloser.Close()
} }
-160
View File
@@ -1,20 +1,15 @@
package vaultik_test package vaultik_test
import ( import (
"bytes"
"context" "context"
"crypto/rand"
"io" "io"
"os" "os"
"path/filepath" "path/filepath"
"slices"
"testing" "testing"
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/ui" "sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik" "sneak.berlin/go/vaultik/internal/vaultik"
) )
@@ -82,161 +77,6 @@ func TestDeepVerifyAcceptsHealthyAndRejectsCorruptBlob(t *testing.T) {
"deep verify should fail on a corrupted blob") "deep verify should fail on a corrupted blob")
} }
// TestDeepVerifyJSONCountsBlobs checks the counts that deep verify
// reports under --json: on a healthy snapshot, with a blob that cannot
// be checked, with a damaged blob, with missing blobs, and with a blob
// of the wrong size. Deep verify stops at the first damaged blob, so the
// blobs read before it are the ones verified; when blobs are missing or
// the wrong size it reads no blob contents, but counts every such blob.
func TestDeepVerifyJSONCountsBlobs(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source")
storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(32 * 1024)
maxBlobSize := int64(128 * 1024)
// Random data does not compress or deduplicate, so it fills several
// blobs.
data := make([]byte, maxBlobSize*4)
_, err := rand.Read(data)
require.NoError(t, err)
require.NoError(t, fs.MkdirAll(dataDir, 0o755))
require.NoError(t, afero.WriteFile(fs,
filepath.Join(dataDir, "data.bin"), data, 0o644))
ctx := context.Background()
cfg, storer, snapshotID := runFileStorageBackup(
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
healthy, err := deepVerifyJSON(ctx, t, cfg, storer, snapshotID)
require.NoError(t, err)
require.Equal(t, "ok", healthy.Status)
require.GreaterOrEqual(t, healthy.BlobCount, 3)
require.Equal(t, healthy.BlobCount, healthy.Verified)
require.Zero(t, healthy.Missing)
// Deep verify reads blobs in hash order, which is the sorted order of
// their storage keys.
keys, err := storer.List(ctx, "blobs/")
require.NoError(t, err)
slices.Sort(keys)
require.Len(t, keys, healthy.BlobCount)
// A blob that is present but cannot be checked is not missing: the
// check stops with the storage error.
denied := &statDeniedStorer{Storer: storer, deniedKey: keys[0]}
got, err := deepVerifyJSON(ctx, t, cfg, denied, snapshotID)
require.Error(t, err)
require.Equal(t, "failed", got.Status)
require.Zero(t, got.Missing)
require.Contains(t, got.ErrorMessage, os.ErrPermission.Error())
// Damage the last blob read, keeping its length, so every blob before
// it is read and passes first.
lastPath := filepath.Join(storeDir, keys[len(keys)-1])
blob, err := afero.ReadFile(fs, lastPath)
require.NoError(t, err)
blob[len(blob)/2] ^= 0xff
require.NoError(t, afero.WriteFile(fs, lastPath, blob, 0o600))
got, err = deepVerifyJSON(ctx, t, cfg, storer, snapshotID)
require.Error(t, err)
require.Equal(t, "failed", got.Status)
require.Equal(t, healthy.BlobCount-1, got.Verified)
require.Zero(t, got.Missing)
// Delete the first blob and the damaged last one: both must be
// counted as missing, not only the first one found.
var deletedSize int64
for _, key := range []string{keys[0], keys[len(keys)-1]} {
info, statErr := storer.Stat(ctx, key)
require.NoError(t, statErr)
deletedSize += info.Size
require.NoError(t, storer.Delete(ctx, key))
}
got, err = deepVerifyJSON(ctx, t, cfg, storer, snapshotID)
require.Error(t, err)
require.Equal(t, "failed", got.Status)
require.Equal(t, 2, got.Missing)
require.Equal(t, deletedSize, got.MissingSize)
require.Zero(t, got.Verified)
// Make a blob between the two deleted ones longer: it is counted as
// the wrong size, and both missing blobs are still counted.
grownPath := filepath.Join(storeDir, keys[1])
blob, err = afero.ReadFile(fs, grownPath)
require.NoError(t, err)
blob = append(blob, []byte("extra")...)
require.NoError(t, afero.WriteFile(fs, grownPath, blob, 0o600))
got, err = deepVerifyJSON(ctx, t, cfg, storer, snapshotID)
require.Error(t, err)
require.Equal(t, "failed", got.Status)
require.Equal(t, 1, got.Mismatched)
require.Equal(t, 2, got.Missing)
require.Zero(t, got.Verified)
}
// statDeniedStorer passes every call through to the wrapped Storer,
// except that Stat of deniedKey fails with a permission error, as it
// does for a blob that is present but cannot be read.
type statDeniedStorer struct {
storage.Storer
deniedKey string
}
func (s *statDeniedStorer) Stat(
ctx context.Context, key string,
) (*storage.ObjectInfo, error) {
if key == s.deniedKey {
return nil, os.ErrPermission
}
return s.Storer.Stat(ctx, key)
}
// deepVerifyJSON runs deep verify with --json on snapshotID and returns
// the decoded result and the error RunDeepVerify returned.
func deepVerifyJSON(
ctx context.Context, t *testing.T,
cfg *config.Config, storer storage.Storer, snapshotID string,
) (vaultik.VerifyResult, error) {
t.Helper()
var out bytes.Buffer
v := &vaultik.Vaultik{
Config: cfg,
Storage: storer,
Fs: afero.NewOsFs(),
Stdout: &out,
Stderr: io.Discard,
UI: ui.NewWithColor(io.Discard, false),
}
v.SetContext(ctx)
err := v.RunDeepVerify(snapshotID,
&vaultik.VerifyOptions{Deep: true, JSON: true})
return decodeVerifyResult(t, out.Bytes()), err
}
// corruptOneBlob flips a middle byte of the first blob file found under // corruptOneBlob flips a middle byte of the first blob file found under
// blobsDir, leaving the file length unchanged. // blobsDir, leaving the file length unchanged.
func corruptOneBlob(t *testing.T, fs afero.Fs, blobsDir string) { func corruptOneBlob(t *testing.T, fs afero.Fs, blobsDir string) {
+16
View File
@@ -89,6 +89,22 @@ type SnapshotInfo struct {
RemotePresent *bool `json:"remote_present"` RemotePresent *bool `json:"remote_present"`
} }
// formatBytes formats bytes in a human-readable format
func formatBytes(bytes int64) string {
const unit = 1024
if bytes < unit {
return fmt.Sprintf("%d B", bytes)
}
div, exp := int64(unit), 0
for n := bytes / unit; n >= unit; n /= unit {
div *= unit
exp++
}
return fmt.Sprintf("%.1f %cB", float64(bytes)/float64(div), "KMGTPE"[exp])
}
// parseSnapshotTimestamp extracts the timestamp from a snapshot ID // parseSnapshotTimestamp extracts the timestamp from a snapshot ID
// Format: hostname_snapshotname_2026-01-12T14:41:15Z // Format: hostname_snapshotname_2026-01-12T14:41:15Z
func parseSnapshotTimestamp(snapshotID string) (time.Time, error) { func parseSnapshotTimestamp(snapshotID string) (time.Time, error) {
+19 -8
View File
@@ -167,6 +167,10 @@ const (
// metadataKeyParts is the minimum "/"-separated segment count of a // metadataKeyParts is the minimum "/"-separated segment count of a
// metadata object key (metadata/<snapshot-id>/<filename>). // metadata object key (metadata/<snapshot-id>/<filename>).
metadataKeyParts = 3 metadataKeyParts = 3
// blobKeyParts is the minimum "/"-separated segment count of a blob
// object key (blobs/<aa>/<bb>/<hash>).
blobKeyParts = 4
) )
// SnapshotMetadataInfo contains information about a single snapshot's metadata // SnapshotMetadataInfo contains information about a single snapshot's metadata
@@ -450,16 +454,23 @@ func (v *Vaultik) scanRemoteBlobStorage(
v.stdoutf("Scanning blobs...\n") v.stdoutf("Scanning blobs...\n")
} }
// Use prune's listing, so a file under blobs/ that prune would never blobCh := v.Storage.ListStream(v.ctx, "blobs/")
// delete is not counted as a blob, and so not as an orphaned one. allBlobs := make(map[string]int64)
allBlobs, err := v.listAllRemoteBlobs()
if err != nil {
return err
}
for _, size := range allBlobs { for obj := range blobCh {
if obj.Err != nil {
return fmt.Errorf("listing blobs: %w", obj.Err)
}
parts := strings.Split(obj.Key, "/")
if len(parts) < blobKeyParts {
continue
}
hash := parts[3]
allBlobs[hash] = obj.Size
result.TotalBlobCount++ result.TotalBlobCount++
result.TotalBlobSize += size result.TotalBlobSize += obj.Size
} }
// A blob named only by a manifest that could not be read, or by one // A blob named only by a manifest that could not be read, or by one
-133
View File
@@ -1,133 +0,0 @@
package vaultik_test
import (
"bytes"
"context"
"maps"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/chunker"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestNewChunkSizeOfChangedFileStaysFixed covers
// https://git.eeqj.de/sneak/vaultik/issues/311. A backup whose only change
// is a modified file lists the size of the chunks that change added as its
// NEW CHUNK SIZE, and a later backup that changes the file again leaves
// every earlier snapshot's figure as it was. The file's row in the local
// index is shared by all three snapshots and rewritten by each backup.
func TestNewChunkSizeOfChangedFileStaysFixed(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src")
dbPath := filepath.Join(tempDir, "index.sqlite")
changedPath := filepath.Join(dataDir, "changed.bin")
ctx := context.Background()
files := writeFaultSourceTree(t, fs, dataDir)
cfg := changedFileConfig(dataDir, dbPath)
appendRandomBytes(t, fs, files, changedPath, appendedFileSize)
store, err := storage.NewFileStorer(filepath.Join(tempDir, "remote"))
require.NoError(t, err)
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
repos := database.NewRepositories(db)
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
require.NoError(t, backUp(v, "first"))
firstFiles := maps.Clone(files)
firstID := localSnapshotID(ctx, t, repos, "first")
appendRandomBytes(t, fs, files, changedPath, faultChunkSize)
require.NoError(t, backUp(v, "second"))
secondID := localSnapshotID(ctx, t, repos, "second")
secondAdded := sizeOfNewChunks(t, files, firstFiles)
require.Positive(t, secondAdded)
assert.Equal(t, secondAdded, listedNewChunkSizes(t, v)[secondID],
"the second snapshot must count the chunks the change added")
appendRandomBytes(t, fs, files, changedPath, faultChunkSize)
require.NoError(t, backUp(v, "third"))
sizes := listedNewChunkSizes(t, v)
assert.Equal(t, sizeOfNewChunks(t, firstFiles, nil), sizes[firstID],
"a later backup must not change the first snapshot's figure")
assert.Equal(t, secondAdded, sizes[secondID],
"a later backup must not change the second snapshot's figure")
}
// sizeOfNewChunks returns the total size of the distinct chunks of the
// files in after that no file in before has, cut the way the backup cuts
// them.
func sizeOfNewChunks(t *testing.T, after, before map[string][]byte) int64 {
t.Helper()
known := make(map[string]bool)
for _, chunk := range chunksOf(t, before) {
known[chunk.Hash] = true
}
var total int64
for _, chunk := range chunksOf(t, after) {
if !known[chunk.Hash] {
known[chunk.Hash] = true
total += chunk.Size
}
}
return total
}
// chunksOf returns the chunks of every file in files.
func chunksOf(t *testing.T, files map[string][]byte) []chunker.Chunk {
t.Helper()
c := chunker.NewChunker(faultChunkSize)
var all []chunker.Chunk
for _, content := range files {
chunks, err := c.ChunkReader(bytes.NewReader(content))
require.NoError(t, err)
all = append(all, chunks...)
}
return all
}
// listedNewChunkSizes runs `snapshot list --json` and returns each listed
// snapshot's new_chunk_size by snapshot ID.
func listedNewChunkSizes(t *testing.T, v *vaultik.Vaultik) map[string]int64 {
t.Helper()
var out bytes.Buffer
v.Stdout = &out
require.NoError(t, v.ListSnapshots(true))
sizes := make(map[string]int64)
for _, row := range decodeListJSON(t, out.String()) {
sizes[row.ID] = row.NewChunkSize
}
return sizes
}
+81 -1
View File
@@ -5,12 +5,14 @@ import (
"io/fs" "io/fs"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot" "sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
) )
// TestNukeRemoteLeavesNoFiles checks that remote nuke leaves no file // TestNukeRemoteLeavesNoFiles checks that remote nuke leaves no file
@@ -23,7 +25,7 @@ func TestNukeRemoteLeavesNoFiles(t *testing.T) {
ctx := context.Background() ctx := context.Background()
storeDir := filepath.Join(t.TempDir(), "store") storeDir := filepath.Join(t.TempDir(), "store")
v, repos, _ := backUpToFileDestination(ctx, t, storeDir) v, repos, out := backUpToFileDestination(ctx, t, storeDir)
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit) snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
require.NoError(t, err) require.NoError(t, err)
@@ -61,4 +63,82 @@ func TestNukeRemoteLeavesNoFiles(t *testing.T) {
}) })
require.NoError(t, err) require.NoError(t, err)
assert.Empty(t, files) assert.Empty(t, files)
// Nuke deletes the blobs itself, so advice to run prune is wrong.
assert.Contains(t, out.String(), "Backup destination store is now empty.")
assert.NotContains(t, out.String(), "vaultik prune")
}
// TestNukeRemoteFailsWhenADeleteFails checks that remote nuke returns an
// error when a snapshot's metadata or a blob cannot be deleted, does not
// report the store empty, and advises running nuke again: prune deletes
// no metadata.
func TestNukeRemoteFailsWhenADeleteFails(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
for _, refusedPrefix := range []string{"metadata/", "blobs/"} {
t.Run(refusedPrefix, func(t *testing.T) {
t.Parallel()
ctx := context.Background()
storeDir := filepath.Join(t.TempDir(), "store")
v, _, out := backUpToFileDestination(ctx, t, storeDir)
store := faultstore.New(v.Storage)
store.OnDelete = func(key string) bool {
return strings.HasPrefix(key, refusedPrefix)
}
v.Storage = store
require.Error(t, v.NukeRemote(true))
assert.NotContains(t, out.String(), "now empty")
assert.Contains(t, out.String(), "vaultik remote nuke --force")
assert.NotContains(t, out.String(), "vaultik prune")
})
}
}
// TestNukeRemoteRefusesAnotherDestinationWithoutRetryAdvice checks that
// remote nuke, when the local index belongs to a different destination
// than the configured one, fails without advising to run nuke again:
// that run would fail the same way.
func TestNukeRemoteRefusesAnotherDestinationWithoutRetryAdvice(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
storeDir := filepath.Join(t.TempDir(), "store")
v, _, out := backUpToFileDestination(ctx, t, storeDir)
v.Config.StorageURL = "file://" + filepath.Join(t.TempDir(), "elsewhere")
err := v.NukeRemote(true)
require.ErrorContains(t, err, "bound to a different backup destination")
assert.NotContains(t, out.String(), "vaultik remote nuke --force")
}
// TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved checks that a
// snapshot remote nuke cannot remove from the local index fails the
// run: nuke then leaves that snapshot's metadata on the destination
// store.
func TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
storeDir := filepath.Join(t.TempDir(), "store")
v, _, out := backUpToFileDestination(ctx, t, storeDir)
_, err := v.DB.Conn().ExecContext(ctx, `
CREATE TRIGGER refuse_snapshot_delete BEFORE DELETE ON snapshots
BEGIN SELECT RAISE(ABORT, 'simulated index error'); END`)
require.NoError(t, err)
require.Error(t, v.NukeRemote(true))
assert.NotContains(t, out.String(), "now empty")
assert.Contains(t, out.String(), "vaultik remote nuke --force")
} }
+39 -11
View File
@@ -19,14 +19,18 @@ type PruneOptions struct {
var errNukeRequiresForce = errors.New( var errNukeRequiresForce = errors.New(
"nuke requires --force (this deletes ALL remote snapshots and blobs)") "nuke requires --force (this deletes ALL remote snapshots and blobs)")
// errBlobDeleteFailed reports unreferenced blobs prune could not delete.
var errBlobDeleteFailed = errors.New("blob(s) could not be deleted")
// nukeCommandHint is the command suggested when a remote nuke could not
// delete everything: running it again deletes what is left, which
// `vaultik prune` does not do for snapshot metadata.
const nukeCommandHint = "vaultik remote nuke --force"
// metadataDirName is the top-level remote directory holding snapshot // metadataDirName is the top-level remote directory holding snapshot
// metadata. // metadata.
const metadataDirName = "metadata" const metadataDirName = "metadata"
// blobKeyParts is the "/"-separated segment count of a blob object key
// (blobs/<aa>/<bb>/<hash>).
const blobKeyParts = 4
// NukeRemote deletes every snapshot's metadata and every blob from remote // NukeRemote deletes every snapshot's metadata and every blob from remote
// storage, along with any object an upload cut off part-way left under a // storage, along with any object an upload cut off part-way left under a
// temporary `.partial` name. After this returns successfully the bucket // temporary `.partial` name. After this returns successfully the bucket
@@ -39,6 +43,29 @@ func (v *Vaultik) NukeRemote(force bool) error {
return errNukeRequiresForce return errNukeRequiresForce
} }
// Checked before anything is deleted: running nuke again would fail
// the same way, so the advice below must not follow this error.
err := v.EnsureStorageBinding()
if err != nil {
return err
}
err = v.emptyDestinationStore()
if err != nil {
v.UI.Warningf("Run '%s' again to delete what is left on the "+
"backup destination store.", nukeCommandHint)
return err
}
v.UI.Completef("Backup destination store is now empty.")
return nil
}
// emptyDestinationStore does the deleting for NukeRemote and stops at
// the first step that fails.
func (v *Vaultik) emptyDestinationStore() error {
v.UI.Beginf("Removing all snapshot metadata from backup destination store.") v.UI.Beginf("Removing all snapshot metadata from backup destination store.")
_, err := v.RemoveAllSnapshots(&RemoveOptions{Force: true}) _, err := v.RemoveAllSnapshots(&RemoveOptions{Force: true})
@@ -62,8 +89,6 @@ func (v *Vaultik) NukeRemote(force bool) error {
} }
} }
v.UI.Completef("Backup destination store is now empty.")
return nil return nil
} }
@@ -166,14 +191,17 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result) v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result)
if opts.JSON { if opts.JSON {
return v.outputPruneBlobsJSON(result) err = v.outputPruneBlobsJSON(result)
if err != nil {
return err
}
} else {
v.stdoutf("\nDeleted %d blob(s) totaling %s\n",
result.BlobsDeleted, ubytes(result.BytesFreed))
} }
v.stdoutf("\nDeleted %d blob(s) totaling %s\n",
result.BlobsDeleted, ubytes(result.BytesFreed))
if result.BlobsFailed > 0 { if result.BlobsFailed > 0 {
v.stdoutf("Failed to delete %d blob(s)\n", result.BlobsFailed) return fmt.Errorf("%d %w", result.BlobsFailed, errBlobDeleteFailed)
} }
return nil return nil
@@ -0,0 +1,98 @@
package vaultik_test
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// refuseBlobDeletes is an OnDelete hook that fails every blob delete.
func refuseBlobDeletes(key string) bool {
return strings.HasPrefix(key, "blobs/")
}
// TestPruneBlobs_FailedDeleteReturnsError checks that prune returns an
// error, so the command exits non-zero, when an unreferenced blob cannot
// be deleted.
func TestPruneBlobs_FailedDeleteReturnsError(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
// The store holds no manifest, so nothing references this blob.
addBlob(t, env.store.testStorer, testBlobHashA)
store := faultstore.New(env.store)
store.OnDelete = refuseBlobDeletes
env.v.Storage = store
err := env.v.PruneBlobs(&vaultik.PruneOptions{Force: true})
require.ErrorContains(t, err, "1 blob(s) could not be deleted")
}
// TestPruneBlobs_JSONFailedDeleteReturnsError checks that prune --json
// prints its document and then returns an error when an unreferenced
// blob cannot be deleted.
func TestPruneBlobs_JSONFailedDeleteReturnsError(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
addBlob(t, env.store.testStorer, testBlobHashA)
store := faultstore.New(env.store)
store.OnDelete = refuseBlobDeletes
env.v.Storage = store
err := env.v.PruneBlobs(&vaultik.PruneOptions{JSON: true})
require.Error(t, err, "prune must fail when a blob cannot be deleted")
var result vaultik.PruneBlobsResult
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &result),
"stdout must hold the JSON document, got:\n%s", env.stdout.String())
assert.Equal(t, 1, result.BlobsFailed)
assert.Equal(t, 0, result.BlobsDeleted)
}
// TestCreateSnapshotPrune_FailedBlobDeleteReturnsError checks that the
// prune step of snapshot create --prune fails the run when an
// unreferenced blob cannot be deleted.
func TestCreateSnapshotPrune_FailedBlobDeleteReturnsError(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
storeDir := filepath.Join(t.TempDir(), "store")
v, _, _ := backUpToFileDestination(ctx, t, storeDir)
// No snapshot references this blob, so the prune step deletes it.
orphan := filepath.Join(storeDir, "blobs", testBlobHashA[:2],
testBlobHashA[2:4], testBlobHashA)
require.NoError(t, os.MkdirAll(filepath.Dir(orphan), 0o750))
require.NoError(t, os.WriteFile(orphan, []byte("orphan"), 0o600))
store := faultstore.New(v.Storage)
store.OnDelete = refuseBlobDeletes
v.Storage = store
err := v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Cron: true,
Prune: true,
})
require.ErrorContains(t, err, "1 blob(s) could not be deleted")
assert.FileExists(t, orphan)
}
-33
View File
@@ -185,39 +185,6 @@ func TestRemoteInfo_SkipsNonConformingMetadataName(t *testing.T) {
assert.NotContains(t, doc, "unreadable_manifests") assert.NotContains(t, doc, "unreadable_manifests")
} }
// TestRemoteInfo_DoesNotCountNonBlobFileUnderBlobs checks that a file
// under blobs/ whose name is not a blob hash is not counted as a blob.
// prune never deletes such a file, so counting it as an orphaned blob
// would advise a prune that removes nothing.
func TestRemoteInfo_DoesNotCountNonBlobFileUnderBlobs(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
addBlob(t, env.store.testStorer, testBlobHashA)
require.NoError(t, env.store.Put(context.Background(),
"blobs/"+testBlobHashA[:2]+"/"+testBlobHashA[2:4]+"/.DS_Store",
bytes.NewReader([]byte("junk"))))
require.NoError(t, env.v.RemoteInfo(false))
text := env.stdout.String()
assert.Contains(t, text, "Total blobs on remote: 1 (")
assert.Contains(t, text, "Orphaned (unreferenced): 0 (")
assert.NotContains(t, text, "vaultik prune")
env.stdout.Reset()
require.NoError(t, env.v.RemoteInfo(true))
var doc map[string]any
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
assert.InDelta(t, 1, doc["total_blob_count"], 0)
assert.InDelta(t, 0, doc["orphaned_blob_count"], 0)
assert.InDelta(t, 0, doc["orphaned_blob_size"], 0)
}
// TestRemoteInfo_DirectoryWithoutManifestLeavesOrphansKnown checks that // TestRemoteInfo_DirectoryWithoutManifestLeavesOrphansKnown checks that
// a directory under metadata/ holding no manifest.json.zst, such as one // a directory under metadata/ holding no manifest.json.zst, such as one
// left by a backup interrupted before its manifest upload, leaves the // left by a backup interrupted before its manifest upload, leaves the
+2 -1
View File
@@ -476,7 +476,8 @@ func TestRemoveAllSnapshots_WithForce(t *testing.T) {
out := tv.Stdout.String() out := tv.Stdout.String()
assert.Contains(t, out, "Removed 2 snapshot(s)") assert.Contains(t, out, "Removed 2 snapshot(s)")
assert.Contains(t, out, "Removed snapshot metadata from remote storage") assert.Contains(t, out, "Removed snapshot metadata from remote storage")
assert.Contains(t, out, "vaultik prune") // Remote nuke, the only caller, deletes the blobs next itself.
assert.NotContains(t, out, "vaultik prune")
} }
func TestRemoveAllSnapshots_DryRun(t *testing.T) { func TestRemoveAllSnapshots_DryRun(t *testing.T) {
+1 -12
View File
@@ -309,22 +309,11 @@ func (v *Vaultik) restoreAllFiles(
return nil, err return nil, err
} }
plan, unplanned, err := newRestorePlan( plan, err := newRestorePlan(v.ctx, repos, files, chunkToBlobMap, blobIDToHash)
v.ctx, repos, files, chunkToBlobMap, blobIDToHash)
if err != nil { if err != nil {
return nil, fmt.Errorf("building restore plan: %w", err) return nil, fmt.Errorf("building restore plan: %w", err)
} }
// Ranging over files rather than the map reports them in a stable order.
for _, file := range files {
if planErr, ok := unplanned[file.ID]; ok {
err = v.handleRestoreFileError(plan, opts, result, file, file.ID, planErr)
if err != nil {
return nil, err
}
}
}
filesByID, totalBytesExpected := indexRestoreFiles(files) filesByID, totalBytesExpected := indexRestoreFiles(files)
v.UI.Beginf("Restoring %s files (%s) to %s.", v.UI.Beginf("Restoring %s files (%s) to %s.",
+2 -118
View File
@@ -142,14 +142,14 @@ func TestRestoreCleansTempDirOnInterrupt(t *testing.T) {
} }
// scratchEntries returns the vaultik blob-cache and snapshot-database // scratchEntries returns the vaultik blob-cache and snapshot-database
// scratch entries (restore's and deep verify's) currently present in dir. // scratch entries currently present in dir.
func scratchEntries(t *testing.T, dir string) []string { func scratchEntries(t *testing.T, dir string) []string {
t.Helper() t.Helper()
var matches []string var matches []string
for _, pattern := range []string{ for _, pattern := range []string{
"vaultik-blobcache-*", "vaultik-restore-*", "vaultik-verify-*", "vaultik-blobcache-*", "vaultik-restore-*",
} { } {
found, err := filepath.Glob(filepath.Join(dir, pattern)) found, err := filepath.Glob(filepath.Join(dir, pattern))
require.NoError(t, err) require.NoError(t, err)
@@ -206,119 +206,3 @@ func TestRestoreSkipErrorsCancelDuringBlobDownload(t *testing.T) {
assert.NotContains(t, out.String(), srcPath, assert.NotContains(t, out.String(), srcPath,
"the cancel was reported as a failed file") "the cancel was reported as a failed file")
} }
// stallingBlobStorer wraps a Storer so that every blob download hands
// over the first half of the blob and then stalls, ignoring the context,
// as a file:// blob on a hung USB stick or network share does. Only
// closing the returned reader ends the stall.
type stallingBlobStorer struct {
storage.Storer
once sync.Once
stalled chan struct{}
}
func newStallingBlobStorer(inner storage.Storer) *stallingBlobStorer {
return &stallingBlobStorer{Storer: inner, stalled: make(chan struct{})}
}
func (s *stallingBlobStorer) Get(
ctx context.Context, key string,
) (io.ReadCloser, error) {
rc, err := s.Storer.Get(ctx, key)
if err != nil || !strings.HasPrefix(key, "blobs/") {
return rc, err
}
blob, err := io.ReadAll(rc)
_ = rc.Close()
if err != nil {
return nil, err
}
// The writer is never closed, so once the first half has been read
// the next Read blocks until the reader is closed.
pr, pw := io.Pipe()
go func() {
_, _ = pw.Write(blob[:len(blob)/2])
s.once.Do(func() { close(s.stalled) })
}()
return pr, nil
}
// TestInterruptDuringStalledBlobReadCleansTempDir stops a restore and a
// deep verify while a blob read is stalled on a source that ignores the
// context. Stopping must end that read, so the operation unwinds and
// removes its decrypted scratch files well within the shutdown timeout
// (issue #325).
//
// Not parallel: it points TMPDIR at a private directory (via t.Setenv)
// so it can assert on exactly the scratch files the operation created.
func TestInterruptDuringStalledBlobReadCleansTempDir(t *testing.T) {
log.Initialize(log.Config{})
operations := map[string]func(v *Vaultik, snapshotID, targetDir string) error{
"restore": func(v *Vaultik, snapshotID, targetDir string) error {
return v.Restore(&RestoreOptions{
SnapshotID: snapshotID,
TargetDir: targetDir,
})
},
"deep verify": func(v *Vaultik, snapshotID, _ string) error {
return v.RunDeepVerify(snapshotID, &VerifyOptions{Deep: true})
},
}
for name, op := range operations {
t.Run(name, func(t *testing.T) {
fs := afero.NewOsFs()
root := t.TempDir()
// Random content, so the blob stays large enough to stall
// part-way through its compressed, encrypted body.
cfg, storer, snapshotID, _ := backupOneFile(context.Background(),
t, fs, root, "a.bin", randomBytes(t, 256*1024), 0o644)
scratch := filepath.Join(root, "scratch")
require.NoError(t, fs.MkdirAll(scratch, 0o755))
t.Setenv("TMPDIR", scratch)
stalling := newStallingBlobStorer(storer)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
v := newRestoreVaultik(ctx, cfg, stalling, fs)
v.cancel = cancel
var opErr error
stop := v.StartOperation(func() {
opErr = op(v, snapshotID, filepath.Join(root, "restored"))
})
select {
case <-stalling.stalled:
case <-time.After(30 * time.Second):
t.Fatal("the operation never reached a blob read")
}
require.NotEmpty(t, scratchEntries(t, scratch),
"expected decrypted scratch files to exist mid-read")
stopCtx, stopCancel := context.WithTimeout(
context.Background(), 10*time.Second)
defer stopCancel()
require.True(t, stop(stopCtx),
"stop timed out; the stalled blob read was not interrupted")
require.ErrorIs(t, opErr, context.Canceled)
require.Empty(t, scratchEntries(t, scratch),
"decrypted scratch files remained after the interrupt")
})
}
}
@@ -61,8 +61,8 @@ func craftedFile(
} }
// TestRestoreShortChunkHashInNoBlob proves a file whose short chunk hash // TestRestoreShortChunkHashInNoBlob proves a file whose short chunk hash
// has no blob_chunks row is left out of the restore plan with an error, // has no blob_chunks row fails restore planning and the chunk write with
// and fails the chunk write with an error. // an error.
func TestRestoreShortChunkHashInNoBlob(t *testing.T) { func TestRestoreShortChunkHashInNoBlob(t *testing.T) {
t.Parallel() t.Parallel()
@@ -77,11 +77,9 @@ func TestRestoreShortChunkHashInNoBlob(t *testing.T) {
chunkToBlobMap, err := v.buildChunkToBlobMap(ctx, repos) chunkToBlobMap, err := v.buildChunkToBlobMap(ctx, repos)
require.NoError(t, err) require.NoError(t, err)
plan, unplanned, err := newRestorePlan(ctx, repos, []*database.File{file}, _, err = newRestorePlan(ctx, repos, []*database.File{file},
chunkToBlobMap, map[string]string{}) chunkToBlobMap, map[string]string{})
require.NoError(t, err) require.ErrorIs(t, err, errPlanChunkMissing)
require.ErrorIs(t, unplanned[file.ID], errPlanChunkMissing)
require.False(t, plan.hasPending())
fileChunks, err := repos.FileChunks.GetByFileID(ctx, file.ID) fileChunks, err := repos.FileChunks.GetByFileID(ctx, file.ID)
require.NoError(t, err) require.NoError(t, err)
+8 -17
View File
@@ -41,25 +41,20 @@ type restorePlan struct {
// newRestorePlan builds the file→blob index for the given files. Files // newRestorePlan builds the file→blob index for the given files. Files
// whose chunks reference no blobs (symlinks, directories) start in the // whose chunks reference no blobs (symlinks, directories) start in the
// ready queue immediately. A file with a chunk that no blob holds cannot // ready queue immediately.
// be restored: it is left out of the plan and returned, by ID, with the
// reason.
func newRestorePlan( func newRestorePlan(
ctx context.Context, ctx context.Context,
repos *database.Repositories, repos *database.Repositories,
files []*database.File, files []*database.File,
chunkToBlobMap map[string]*database.BlobChunk, chunkToBlobMap map[string]*database.BlobChunk,
blobIDToHash map[string]string, blobIDToHash map[string]string,
) (*restorePlan, map[types.FileID]error, error) { ) (*restorePlan, error) {
p := &restorePlan{ p := &restorePlan{
fileBlobs: make(map[types.FileID]map[string]struct{}, len(files)), fileBlobs: make(map[types.FileID]map[string]struct{}, len(files)),
blobFiles: make(map[string]map[types.FileID]struct{}), blobFiles: make(map[string]map[types.FileID]struct{}),
ready: make([]types.FileID, 0, len(files)), ready: make([]types.FileID, 0, len(files)),
cached: make(map[string]struct{}), cached: make(map[string]struct{}),
} }
unplanned := make(map[types.FileID]error)
nextFile:
for _, f := range files { for _, f := range files {
if f.IsSymlink() || f.Mode&uint32(os.ModeDir) != 0 { if f.IsSymlink() || f.Mode&uint32(os.ModeDir) != 0 {
// No chunks to fetch — restore can run immediately. // No chunks to fetch — restore can run immediately.
@@ -71,7 +66,7 @@ nextFile:
fileChunks, err := repos.FileChunks.GetByFileID(ctx, f.ID) fileChunks, err := repos.FileChunks.GetByFileID(ctx, f.ID)
if err != nil { if err != nil {
return nil, nil, fmt.Errorf("planning %s: %w", f.Path, err) return nil, fmt.Errorf("planning %s: %w", f.Path, err)
} }
blobs := make(map[string]struct{}) blobs := make(map[string]struct{})
@@ -79,18 +74,14 @@ nextFile:
for _, fc := range fileChunks { for _, fc := range fileChunks {
bc, ok := chunkToBlobMap[fc.ChunkHash.String()] bc, ok := chunkToBlobMap[fc.ChunkHash.String()]
if !ok { if !ok {
unplanned[f.ID] = fmt.Errorf("%w: %s", return nil, fmt.Errorf("planning %s: %w: %s",
errPlanChunkMissing, shortHash(fc.ChunkHash.String())) f.Path, errPlanChunkMissing, shortHash(fc.ChunkHash.String()))
continue nextFile
} }
hash, ok := blobIDToHash[bc.BlobID.String()] hash, ok := blobIDToHash[bc.BlobID.String()]
if !ok { if !ok {
unplanned[f.ID] = fmt.Errorf("%w: %s", return nil, fmt.Errorf("planning %s: %w: %s",
errPlanBlobIDMissing, bc.BlobID) f.Path, errPlanBlobIDMissing, bc.BlobID)
continue nextFile
} }
blobs[hash] = struct{}{} blobs[hash] = struct{}{}
@@ -112,7 +103,7 @@ nextFile:
} }
} }
return p, unplanned, nil return p, nil
} }
// markBlobCached records that the named blob is now resident in the // markBlobCached records that the named blob is now resident in the
+16 -103
View File
@@ -28,8 +28,7 @@ const (
) )
// missingBlobBackup is a snapshot of single-chunk files spread over two // missingBlobBackup is a snapshot of single-chunk files spread over two
// blobs, with one of those blobs lost: deleted from the store, or missing // blobs, with one of those blobs deleted from the store.
// from the snapshot metadata.
type missingBlobBackup struct { type missingBlobBackup struct {
fs afero.Fs fs afero.Fs
cfg *config.Config cfg *config.Config
@@ -38,39 +37,20 @@ type missingBlobBackup struct {
restoreDir string restoreDir string
// files holds the original content by source path. // files holds the original content by source path.
files map[string][]byte files map[string][]byte
// lost holds the source paths whose chunk is in the lost blob. // lost holds the source paths whose chunk is in the deleted blob.
lost map[string]bool lost map[string]bool
} }
// TestRestoreSkipErrorsSkipsFilesOfMissingBlob restores with SkipErrors // TestRestoreSkipErrorsSkipsFilesOfMissingBlob restores with SkipErrors
// after one blob of a two-blob snapshot was deleted from the store. // after one blob of a two-blob snapshot was deleted. Every file stored in
// that blob must be reported as failed and left absent, every other file
// must be restored intact, and Restore must still return an error.
func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) { func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel() t.Parallel()
ctx := context.Background() ctx := context.Background()
assertRestoreSkipsLostFiles(ctx, t, backupThenDeleteOneBlob(ctx, t)) backup := backupThenDeleteOneBlob(ctx, t)
}
// TestRestoreSkipErrorsSkipsFilesWithChunkInNoBlob restores with
// SkipErrors a snapshot whose metadata lists chunks that no blob holds, as
// earlier backup defects left behind.
func TestRestoreSkipErrorsSkipsFilesWithChunkInNoBlob(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
assertRestoreSkipsLostFiles(ctx, t, backupThenDropOneBlobFromMetadata(ctx, t))
}
// assertRestoreSkipsLostFiles restores backup with SkipErrors. Every file
// stored in the lost blob must be reported as failed and left absent,
// every other file must be restored intact, and Restore must still return
// an error.
func assertRestoreSkipsLostFiles(
ctx context.Context, t *testing.T, backup *missingBlobBackup,
) {
t.Helper()
var out bytes.Buffer var out bytes.Buffer
@@ -92,14 +72,14 @@ func assertRestoreSkipsLostFiles(
if backup.lost[path] { if backup.lost[path] {
assert.Containsf(t, out.String(), path, assert.Containsf(t, out.String(), path,
"%s needs the lost blob and must be reported", path) "%s needs the deleted blob and must be reported", path)
assert.NoFileExists(t, restored) assert.NoFileExists(t, restored)
continue continue
} }
got, err := afero.ReadFile(backup.fs, restored) got, err := afero.ReadFile(backup.fs, restored)
require.NoErrorf(t, err, "%s does not need the lost blob", path) require.NoErrorf(t, err, "%s does not need the deleted blob", path)
assert.Equalf(t, content, got, "%s restored with wrong content", path) assert.Equalf(t, content, got, "%s restored with wrong content", path)
} }
} }
@@ -123,30 +103,6 @@ func TestRestoreMissingBlobAbortsWithoutSkipErrors(t *testing.T) {
require.ErrorIs(t, err, storage.ErrNotFound) require.ErrorIs(t, err, storage.ErrNotFound)
} }
// TestRestoreChunkInNoBlobAbortsWithoutSkipErrors checks that a chunk held
// by no blob still ends the restore, before any file is written, when
// SkipErrors is not set.
func TestRestoreChunkInNoBlobAbortsWithoutSkipErrors(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
backup := backupThenDropOneBlobFromMetadata(ctx, t)
v := newReaderVaultik(ctx, backup.cfg, backup.storer, nil, backup.fs)
err := v.Restore(&vaultik.RestoreOptions{
SnapshotID: backup.snapshotID,
TargetDir: backup.restoreDir,
})
require.ErrorContains(t, err, "chunk missing from blob map")
for path := range backup.files {
assert.NoFileExists(t, filepath.Join(backup.restoreDir, path))
}
}
// backupThenDeleteOneBlob backs up missingBlobFileCount single-chunk // backupThenDeleteOneBlob backs up missingBlobFileCount single-chunk
// files, then deletes from the store the blob holding the first of them. // files, then deletes from the store the blob holding the first of them.
func backupThenDeleteOneBlob( func backupThenDeleteOneBlob(
@@ -154,54 +110,6 @@ func backupThenDeleteOneBlob(
) *missingBlobBackup { ) *missingBlobBackup {
t.Helper() t.Helper()
backup, db, lostBlob := backupSingleChunkFiles(ctx, t)
require.NoError(t, db.Close())
require.NoError(t, backup.storer.Delete(ctx, fmt.Sprintf(
"blobs/%s/%s/%s", lostBlob[:2], lostBlob[2:4], lostBlob)))
return backup
}
// backupThenDropOneBlobFromMetadata backs up missingBlobFileCount
// single-chunk files, then removes the blob holding the first of them from
// the snapshot and exports the snapshot metadata again. The exported
// metadata then lists chunks that no blob holds.
func backupThenDropOneBlobFromMetadata(
ctx context.Context, t *testing.T,
) *missingBlobBackup {
t.Helper()
backup, db, lostBlob := backupSingleChunkFiles(ctx, t)
// The uploads row refers to the blob row, so it would keep the export
// from deleting it.
for _, query := range []string{
"DELETE FROM snapshot_blobs WHERE blob_hash = ?",
"DELETE FROM uploads WHERE blob_hash = ?",
} {
_, err := db.Conn().ExecContext(ctx, query, lostBlob)
require.NoError(t, err)
}
sm := newFaultSnapshotManager(
backup.fs, backup.storer, backup.cfg, database.NewRepositories(db))
require.NoError(t,
sm.ExportSnapshotMetadata(ctx, db.Path(), backup.snapshotID))
require.NoError(t, db.Close())
return backup
}
// backupSingleChunkFiles backs up missingBlobFileCount single-chunk files.
// It returns the backup, with lost naming the files stored in the same
// blob as the first of them, the still open local index, and the hash of
// that blob.
func backupSingleChunkFiles(
ctx context.Context, t *testing.T,
) (*missingBlobBackup, *database.DB, string) {
t.Helper()
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src") dataDir := filepath.Join(tempDir, "src")
@@ -235,11 +143,16 @@ func backupSingleChunkFiles(
blobOfFile[path] = blobHashOfSingleChunkFile(ctx, t, repos, path) blobOfFile[path] = blobHashOfSingleChunkFile(ctx, t, repos, path)
} }
lostBlob := blobOfFile[filepath.Join(dataDir, "file-00.bin")] require.NoError(t, db.Close())
deleted := blobOfFile[filepath.Join(dataDir, "file-00.bin")]
require.NoError(t, storer.Delete(ctx, fmt.Sprintf(
"blobs/%s/%s/%s", deleted[:2], deleted[2:4], deleted)))
lost := make(map[string]bool) lost := make(map[string]bool)
for path, hash := range blobOfFile { for path, hash := range blobOfFile {
if hash == lostBlob { if hash == deleted {
lost[path] = true lost[path] = true
} }
} }
@@ -255,7 +168,7 @@ func backupSingleChunkFiles(
restoreDir: filepath.Join(tempDir, "restored"), restoreDir: filepath.Join(tempDir, "restored"),
files: files, files: files,
lost: lost, lost: lost,
}, db, lostBlob }
} }
// blobHashOfSingleChunkFile returns the hash of the blob holding the one // blobHashOfSingleChunkFile returns the hash of the blob holding the one
+20 -51
View File
@@ -24,18 +24,13 @@ var (
errSnapshotVerifyFailed = errors.New("verification failed") errSnapshotVerifyFailed = errors.New("verification failed")
errRemoveAllNeedsForce = errors.New("--all requires --force") errRemoveAllNeedsForce = errors.New("--all requires --force")
errSnapshotNotFound = errors.New("snapshot not found") errSnapshotNotFound = errors.New("snapshot not found")
errSnapshotRemoveFailed = errors.New("snapshot(s) could not be removed")
errInvalidTableName = errors.New("invalid table name") errInvalidTableName = errors.New("invalid table name")
errNoAgeRecipients = errors.New( errNoAgeRecipients = errors.New(
"creating a snapshot needs at least one public key in " + "creating a snapshot needs at least one public key in " +
"age_recipients (generate a keypair with: age-keygen)") "age_recipients (generate a keypair with: age-keygen)")
errKeepNewerThanNeedsPrune = errors.New("--keep-newer-than requires --prune")
) )
// ErrPostBackupPruneFailed wraps a failure of the post-backup prune,
// which runs only after every snapshot was created and uploaded, so the
// caller can report it apart from a failed backup.
var ErrPostBackupPruneFailed = errors.New("post-backup prune failed")
// listRecentLimit caps how many snapshot rows are fetched from the // listRecentLimit caps how many snapshot rows are fetched from the
// local index for listing and purge operations. // local index for listing and purge operations.
const listRecentLimit = 10000 const listRecentLimit = 10000
@@ -51,26 +46,6 @@ type SnapshotCreateOptions struct {
Snapshots []string // Optional list of snapshot names to process (empty = all) Snapshots []string // Optional list of snapshot names to process (empty = all)
} }
// Validate fails when KeepNewerThan is set without Prune or is not a
// valid duration. Only the post-backup prune reads KeepNewerThan, after
// the backup, so the caller checks it before starting one.
func (opts *SnapshotCreateOptions) Validate() error {
if opts.KeepNewerThan == "" {
return nil
}
if !opts.Prune {
return errKeepNewerThanNeedsPrune
}
_, err := parseDuration(opts.KeepNewerThan)
if err != nil {
return fmt.Errorf("--keep-newer-than: %w", err)
}
return nil
}
// CreateSnapshot executes the snapshot creation operation // CreateSnapshot executes the snapshot creation operation
func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error { func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
// config.Load accepts an empty list, since listing, verifying and // config.Load accepts an empty list, since listing, verifying and
@@ -149,7 +124,7 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
if opts.Prune { if opts.Prune {
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan) err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
if err != nil { if err != nil {
return fmt.Errorf("%w: %w", ErrPostBackupPruneFailed, err) return fmt.Errorf("post-backup prune: %w", err)
} }
} }
@@ -209,7 +184,6 @@ type snapshotStats struct {
totalFiles int totalFiles int
totalBytes int64 totalBytes int64
totalChunks int totalChunks int
newChunkSize int64
totalBlobs int totalBlobs int
totalBytesSkipped int64 totalBytesSkipped int64
totalFilesSkipped int totalFilesSkipped int
@@ -342,7 +316,6 @@ func (v *Vaultik) scanAllDirectories(
stats.totalFiles += result.FilesScanned stats.totalFiles += result.FilesScanned
stats.totalBytes += result.BytesScanned stats.totalBytes += result.BytesScanned
stats.totalChunks += result.ChunksCreated stats.totalChunks += result.ChunksCreated
stats.newChunkSize += result.NewChunkSize
stats.totalBlobs += result.BlobsCreated stats.totalBlobs += result.BlobsCreated
stats.totalFilesSkipped += result.FilesSkipped stats.totalFilesSkipped += result.FilesSkipped
stats.totalFilesFailed += result.FilesFailed stats.totalFilesFailed += result.FilesFailed
@@ -405,7 +378,6 @@ func (v *Vaultik) finalizeSnapshotMetadata(
BlobUncompressedSize: stats.blobUncompressedSize, BlobUncompressedSize: stats.blobUncompressedSize,
CompressionLevel: v.Config.CompressionLevel, CompressionLevel: v.Config.CompressionLevel,
UploadDurationMs: stats.uploadDuration.Milliseconds(), UploadDurationMs: stats.uploadDuration.Milliseconds(),
NewChunkSize: stats.newChunkSize,
} }
err = v.SnapshotManager.UpdateSnapshotStatsExtended(v.ctx, snapshotID, extStats) err = v.SnapshotManager.UpdateSnapshotStatsExtended(v.ctx, snapshotID, extStats)
@@ -636,7 +608,7 @@ func (v *Vaultik) confirmAndExecutePurge(
v.stdoutf(" %s (%s, %s)\n", v.stdoutf(" %s (%s, %s)\n",
snap.ID, snap.ID,
snap.Timestamp.Format("2006-01-02 15:04:05"), snap.Timestamp.Format("2006-01-02 15:04:05"),
ubytes(snap.CompressedSize)) formatBytes(snap.CompressedSize))
} }
} }
@@ -1542,8 +1514,8 @@ func (v *Vaultik) handleRemoveAllDryRun(
// executeRemoveAll deletes every local snapshot and, unless LocalOnly // executeRemoveAll deletes every local snapshot and, unless LocalOnly
// is set, every corresponding remote metadata directory plus any // is set, every corresponding remote metadata directory plus any
// orphan remote keys. Blobs are not touched; the printed prune-command // orphan remote keys. Blobs are not touched. It returns an error when
// hint is the next step. // any snapshot could not be removed.
func (v *Vaultik) executeRemoveAll( func (v *Vaultik) executeRemoveAll(
localSnaps, orphanRemoteKeys []string, opts *RemoveOptions, localSnaps, orphanRemoteKeys []string, opts *RemoveOptions,
) (*RemoveResult, error) { ) (*RemoveResult, error) {
@@ -1557,18 +1529,11 @@ func (v *Vaultik) executeRemoveAll(
result := &RemoveResult{} result := &RemoveResult{}
remoteErrors := v.removeAllLocalSnapshots(localSnaps, opts, result) failures := v.removeAllLocalSnapshots(localSnaps, opts, result)
if !opts.LocalOnly { if !opts.LocalOnly {
remoteErrors += v.removeAllOrphanRemotes(orphanRemoteKeys, result) failures += v.removeAllOrphanRemotes(orphanRemoteKeys, result)
result.RemoteRemoved = failures == 0
if remoteErrors == 0 {
result.RemoteRemoved = true
} else if v.UI != nil {
v.UI.Warningf("Some remote metadata deletions failed. Run '%s' "+
"once the remote is healthy to clean up unreferenced blobs.",
pruneCommandHint)
}
} }
if v.SnapshotManager != nil { if v.SnapshotManager != nil {
@@ -1578,6 +1543,10 @@ func (v *Vaultik) executeRemoveAll(
} }
} }
if failures > 0 {
return result, fmt.Errorf("%d %w", failures, errSnapshotRemoveFailed)
}
if opts.JSON { if opts.JSON {
return result, v.outputRemoveJSON(result) return result, v.outputRemoveJSON(result)
} }
@@ -1586,9 +1555,6 @@ func (v *Vaultik) executeRemoveAll(
if !opts.LocalOnly && result.RemoteRemoved { if !opts.LocalOnly && result.RemoteRemoved {
v.printlnStdout("Removed snapshot metadata from remote storage") v.printlnStdout("Removed snapshot metadata from remote storage")
v.stdoutf("\nNote: Removed snapshots' blobs remain on the remote. "+
"Run '%s' to delete any blobs no longer referenced by any "+
"remaining remote snapshot.\n", pruneCommandHint)
} }
return result, nil return result, nil
@@ -1631,12 +1597,13 @@ func (v *Vaultik) deleteIncompleteSnapshot(snapshotIDStr string, id any) bool {
} }
// removeAllLocalSnapshots removes each known local snapshot (and, unless // removeAllLocalSnapshots removes each known local snapshot (and, unless
// --local-only, its remote metadata), returning the number of remote // --local-only, its remote metadata), returning the number of snapshots
// deletion failures. // it could not remove. A snapshot left in the local index keeps its
// remote metadata too, since it is not an orphan remote key.
func (v *Vaultik) removeAllLocalSnapshots( func (v *Vaultik) removeAllLocalSnapshots(
localSnaps []string, opts *RemoveOptions, result *RemoveResult, localSnaps []string, opts *RemoveOptions, result *RemoveResult,
) int { ) int {
remoteErrors := 0 failures := 0
for _, snapshotID := range localSnaps { for _, snapshotID := range localSnaps {
log.Info("Removing snapshot", "snapshot_id", snapshotID) log.Info("Removing snapshot", "snapshot_id", snapshotID)
@@ -1646,6 +1613,8 @@ func (v *Vaultik) removeAllLocalSnapshots(
log.Error("Failed to remove from local database", log.Error("Failed to remove from local database",
"snapshot_id", snapshotID, "error", err) "snapshot_id", snapshotID, "error", err)
failures++
continue continue
} }
@@ -1656,14 +1625,14 @@ func (v *Vaultik) removeAllLocalSnapshots(
log.Warn("Failed to remove snapshot metadata from remote", log.Warn("Failed to remove snapshot metadata from remote",
"snapshot_id", snapshotID, "error", err) "snapshot_id", snapshotID, "error", err)
remoteErrors++ failures++
} }
} }
result.SnapshotsRemoved = append(result.SnapshotsRemoved, snapshotID) result.SnapshotsRemoved = append(result.SnapshotsRemoved, snapshotID)
} }
return remoteErrors return failures
} }
// removeAllOrphanRemotes deletes remote snapshot metadata that has no // removeAllOrphanRemotes deletes remote snapshot metadata that has no
+3 -3
View File
@@ -553,8 +553,8 @@ func (v *Vaultik) printSnapshotTable(snapshots []SnapshotInfo) error {
if snap.LocallyTracked { if snap.LocallyTracked {
id = snap.ID.String() id = snap.ID.String()
uncompressed = ubytes(snap.UncompressedSize) uncompressed = formatBytes(snap.UncompressedSize)
newChunks = ubytes(snap.NewChunkSize) newChunks = formatBytes(snap.NewChunkSize)
} else { } else {
id = formatRemoteOnlyID(snap.RemoteKey) id = formatRemoteOnlyID(snap.RemoteKey)
uncompressed = remoteOnlyCell uncompressed = remoteOnlyCell
@@ -564,7 +564,7 @@ func (v *Vaultik) printSnapshotTable(snapshots []SnapshotInfo) error {
_, err = fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", _, err = fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n",
id, id,
snap.Timestamp.Format("2006-01-02 15:04:05"), snap.Timestamp.Format("2006-01-02 15:04:05"),
ubytes(snap.CompressedSize), formatBytes(snap.CompressedSize),
uncompressed, uncompressed,
newChunks) newChunks)
if err != nil { if err != nil {
+2 -45
View File
@@ -212,8 +212,8 @@ const (
listConfiguredName = "home" listConfiguredName = "home"
listLocalID = "testhost_home_2026-03-01T10:00:00Z" listLocalID = "testhost_home_2026-03-01T10:00:00Z"
listRemoteID = "otherhost_media_2026-03-02T11:22:33Z" listRemoteID = "otherhost_media_2026-03-02T11:22:33Z"
// fiveMegabytes is printed as "5.0 MB" by every command. // fiveMegabytes formats as "5.0 MB" through formatBytes.
fiveMegabytes = 5_000_000 fiveMegabytes = 5 * 1024 * 1024
) )
// TestListSnapshots_RemoteWithoutSecretKey is the regression guard for // TestListSnapshots_RemoteWithoutSecretKey is the regression guard for
@@ -306,48 +306,6 @@ func TestListSnapshots_RemoteOnlyRowRendering(t *testing.T) {
"are not in the local index") "are not in the local index")
} }
// TestSnapshotListAndPurgePrintSizesLikeRemoteInfo checks that the
// `snapshot list` table and the purge listing print a byte count the way
// `remote info` does: in powers of 1000, as the MB label says.
func TestSnapshotListAndPurgePrintSizesLikeRemoteInfo(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
const wantSize = "5.0 MB" // fiveMegabytes
env := newListEnv(t)
// A remote-only snapshot: `snapshot list` and `remote info` both take
// its size from the manifest.
remoteKey := env.addRemote(t, listRemoteID,
time.Date(2026, 3, 2, 11, 22, 33, 0, time.UTC))
// A locally tracked snapshot of the same size for the purge to list.
// The purge prints the snapshot's blob size, the last argument here.
start := time.Date(2026, 3, 1, 10, 0, 0, 0, time.UTC)
env.addLocal(t, listLocalID, start)
env.addRemote(t, listLocalID, start)
require.NoError(t, env.v.Repositories.Snapshots.UpdateCounts(
context.Background(), nil, listLocalID, 0, 0, 0, 0, fiveMegabytes))
require.NoError(t, env.v.RemoteInfo(false))
assert.Contains(t, findTableRow(t, env.stdout.String(), remoteKey[:16]),
wantSize, "remote info")
env.stdout.Reset()
require.NoError(t, env.v.ListSnapshots(false))
assert.Contains(t, findTableRow(t, env.stdout.String(),
"<remote only:"+remoteKey[:12]+">"), wantSize, "snapshot list")
// Without --force and with nothing on stdin the purge lists what it
// would delete, reads no confirmation, and deletes nothing.
env.stdout.Reset()
require.NoError(t, env.v.PurgeSnapshotsWithOptions(
&vaultik.SnapshotPurgeOptions{OlderThan: "1d"}))
assert.Contains(t, findTableRow(t, env.stdout.String(), listLocalID),
wantSize, "snapshot purge")
}
// findTableRow returns the single output line containing needle. // findTableRow returns the single output line containing needle.
func findTableRow(t *testing.T, out, needle string) string { func findTableRow(t *testing.T, out, needle string) string {
t.Helper() t.Helper()
@@ -487,7 +445,6 @@ type listJSONRow struct {
RemoteKey string `json:"remote_key"` RemoteKey string `json:"remote_key"`
Timestamp string `json:"timestamp"` Timestamp string `json:"timestamp"`
CompressedSize int64 `json:"compressed_size"` CompressedSize int64 `json:"compressed_size"`
NewChunkSize int64 `json:"new_chunk_size"`
LocallyTracked bool `json:"locally_tracked"` LocallyTracked bool `json:"locally_tracked"`
RemotePresent *bool `json:"remote_present"` RemotePresent *bool `json:"remote_present"`
} }
+17 -39
View File
@@ -17,7 +17,6 @@ import (
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot" "sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage"
) )
// Sentinel errors for snapshot verification failures. // Sentinel errors for snapshot verification failures.
@@ -33,9 +32,7 @@ var (
errManifestExtraBlob = errors.New("manifest contains blob not in database") errManifestExtraBlob = errors.New("manifest contains blob not in database")
errManifestMissingBlob = errors.New( errManifestMissingBlob = errors.New(
"manifest omits blob present in database") "manifest omits blob present in database")
errBlobSizeMismatch = errors.New("blob size mismatch") errBlobSizeMismatch = errors.New("blob size mismatch")
errBlobsNotInStorage = errors.New(
"blobs missing from storage or stored with the wrong size")
) )
// verifyStatusFailed is the JSON status value for a failed verification. // verifyStatusFailed is the JSON status value for a failed verification.
@@ -138,6 +135,7 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
} }
result.Status = "ok" result.Status = "ok"
result.Verified = len(dbBlobs)
if opts.JSON { if opts.JSON {
return v.outputVerifyJSON(result) return v.outputVerifyJSON(result)
@@ -292,7 +290,7 @@ func (v *Vaultik) runVerificationSteps(
v.stdoutf("Checking blob existence in remote storage...\n") v.stdoutf("Checking blob existence in remote storage...\n")
} }
err = v.verifyBlobExistenceFromDB(dbBlobs, result) err = v.verifyBlobExistenceFromDB(dbBlobs)
if err != nil { if err != nil {
return v.deepVerifyFailure(result, opts, err.Error(), err) return v.deepVerifyFailure(result, opts, err.Error(), err)
} }
@@ -303,8 +301,7 @@ func (v *Vaultik) runVerificationSteps(
len(dbBlobs), ubytes(totalSize)) len(dbBlobs), ubytes(totalSize))
} }
err = v.performDeepVerificationFromDB( err = v.performDeepVerificationFromDB(dbBlobs, tdb.db.Conn(), opts, identities)
dbBlobs, tdb.db.Conn(), opts, result, identities)
if err != nil { if err != nil {
return v.deepVerifyFailure(result, opts, err.Error(), err) return v.deepVerifyFailure(result, opts, err.Error(), err)
} }
@@ -656,14 +653,8 @@ func (v *Vaultik) verifyManifestAgainstDatabase(
return nil return nil
} }
// verifyBlobExistenceFromDB checks that every blob from the database is // verifyBlobExistenceFromDB checks that all blobs from database exist in S3
// in storage with the size the database records. It checks them all, func (v *Vaultik) verifyBlobExistenceFromDB(blobs []snapshot.BlobInfo) error {
// counting the missing and wrong-size blobs in result, and fails if it
// found any. Any storage error other than not found stops the check and
// fails with that error.
func (v *Vaultik) verifyBlobExistenceFromDB(
blobs []snapshot.BlobInfo, result *VerifyResult,
) error {
log.Info("Verifying blob existence in S3", "blob_count", len(blobs)) log.Info("Verifying blob existence in S3", "blob_count", len(blobs))
for i, blob := range blobs { for i, blob := range blobs {
@@ -676,23 +667,17 @@ func (v *Vaultik) verifyBlobExistenceFromDB(
// Construct blob path // Construct blob path
blobPath := fmt.Sprintf("blobs/%s/%s/%s", blob.Hash[:2], blob.Hash[2:4], blob.Hash) blobPath := fmt.Sprintf("blobs/%s/%s/%s", blob.Hash[:2], blob.Hash[2:4], blob.Hash)
// Check blob exists
stat, err := v.Storage.Stat(v.ctx, blobPath) stat, err := v.Storage.Stat(v.ctx, blobPath)
switch { if err != nil {
case errors.Is(err, storage.ErrNotFound): return fmt.Errorf("blob %s missing from storage: %w", blob.Hash, err)
log.Warn("Blob missing from storage", "hash", blob.Hash) }
result.Missing++ // Verify size matches
result.MissingSize += blob.CompressedSize if stat.Size != blob.CompressedSize {
case err != nil: return fmt.Errorf(
return fmt.Errorf("checking blob %s in storage: %w", "%w: blob %s: S3 has %d bytes, database has %d bytes",
blob.Hash, err) errBlobSizeMismatch, blob.Hash, stat.Size, blob.CompressedSize)
case stat.Size != blob.CompressedSize:
log.Warn("Blob has the wrong size in storage",
"hash", blob.Hash,
"storage_size", stat.Size,
"database_size", blob.CompressedSize)
result.Mismatched++
} }
// Progress update every 100 blobs // Progress update every 100 blobs
@@ -706,22 +691,16 @@ func (v *Vaultik) verifyBlobExistenceFromDB(
} }
} }
if result.Missing > 0 || result.Mismatched > 0 {
return fmt.Errorf("%w: %d missing, %d with the wrong size",
errBlobsNotInStorage, result.Missing, result.Mismatched)
}
log.Info("✓ All blobs exist in storage") log.Info("✓ All blobs exist in storage")
return nil return nil
} }
// performDeepVerificationFromDB downloads and verifies the content of // performDeepVerificationFromDB downloads and verifies the content of
// each blob using the database as source, stopping at the first blob that // each blob using the database as source.
// fails. Each blob that passes is counted in result.Verified.
func (v *Vaultik) performDeepVerificationFromDB( func (v *Vaultik) performDeepVerificationFromDB(
blobs []snapshot.BlobInfo, db *sql.DB, opts *VerifyOptions, blobs []snapshot.BlobInfo, db *sql.DB, opts *VerifyOptions,
result *VerifyResult, identities []age.Identity, identities []age.Identity,
) error { ) error {
// Calculate total bytes for ETA // Calculate total bytes for ETA
var totalBytesExpected int64 var totalBytesExpected int64
@@ -744,7 +723,6 @@ func (v *Vaultik) performDeepVerificationFromDB(
return fmt.Errorf("blob %s verification failed: %w", blobInfo.Hash, err) return fmt.Errorf("blob %s verification failed: %w", blobInfo.Hash, err)
} }
result.Verified++
bytesProcessed += blobInfo.CompressedSize bytesProcessed += blobInfo.CompressedSize
elapsed := time.Since(startTime) elapsed := time.Since(startTime)
remaining := len(blobs) - (i + 1) remaining := len(blobs) - (i + 1)