Author SHA1 Message Date
sneak 28a272903f Split the long README sentences the --prune bullet's edit touched
check / check (push) Waiting to run
The --prune bullet in README.md held sentences of 45 to 75 words that
this branch edits or reflows, and it said the missing-volume case exits
non-zero beside two cases that name their status. Each is now split
into short sentences, and that case says it exits 1.

Model: opus-5-5
2026-11-19 06:49:47 +00:00
sneak 7f3fb97012 Exit 3 when a backup or restore finished but skipped files (closes #439)
check / check (push) Waiting to run
A `snapshot create` or `snapshot restore` that --skip-errors made skip
files now exits 3, the status `restic backup` uses, so a cron job can
tell it from a run that made no backup (1). RunOperation keeps the
operation's error inside errReported so Entry can pick the status.

A run that skipped files and also found a configured path empty, missed
a mount point, or was asked for a path the snapshot lacks still exits 1:
its error wraps only that condition and names the skipped files in text.

The README gains an exit status section.

Model: opus-5-5
2026-11-19 06:38:41 +00:00
14 changed files with 249 additions and 570 deletions

No files matched your search

+37 -33
View File
@@ -169,7 +169,7 @@ vaultik [--config <path>] config init
vaultik [--config <path>] config edit
vaultik [--config <path>] config get <key>
vaultik [--config <path>] config set <key> <value>
vaultik [--config <path>] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>] [--dry-run]
vaultik [--config <path>] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>]
vaultik [--config <path>] snapshot list [--json] # alias: ls
vaultik [--config <path>] snapshot files <snapshot-id> [path] [--json]
vaultik [--config <path>] snapshot verify <snapshot-id> [--deep] [--json]
@@ -196,9 +196,10 @@ vaultik version
that cannot be restored when restoring, instead of aborting. Packing and
storage errors while creating a snapshot (which would leave a chunk recorded
but not stored) still abort the run. A run that skipped anything still
finishes but exits non-zero: `snapshot create` keeps the snapshot and counts
each file or directory it skipped as failed. A file removed during the backup
is also counted as failed but does not change the exit status.
finishes but exits `3` (see [exit status](#exit-status)): `snapshot create`
keeps the snapshot and counts each file or directory it skipped as failed. A
file removed during the backup is also counted as failed but does not change
the exit status.
### locking
@@ -271,6 +272,19 @@ Every `--json` command therefore pipes on its own, with no additional flag:
`vaultik snapshot list --json | jq .` and `vaultik prune --json | jq .` both
work as written.
### exit status
- `0`: the command succeeded.
- `1`: the command failed.
- `3`: `snapshot create` or `snapshot restore` finished, but `--skip-errors`
skipped files it could not read or restore. `restic backup` uses the same
status for this case.
- `130`: Ctrl-C, `SIGINT` or `SIGTERM` stopped the command.
A cron job can page on `1` and only log `3`. A run that skipped files and also
failed in another way exits `1`. For example, a restore that skipped files and
was asked for a path the snapshot does not hold exits `1`.
### environment variables
- `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for
@@ -347,39 +361,29 @@ In such a config, an unknown key or a value of the wrong type is refused.
local index, as `prune` does. By default keeps only the latest snapshot per
name; use `--keep-newer-than` for a rolling window. When `--skip-errors`
skipped a file or directory it could not read or list, the prune does not run
and every older snapshot is kept, since one of them may be the only one that
holds what was skipped; the command says so and exits non-zero. When the
backup found nothing in a configured path, as in the mount point of a volume
that is not mounted, the prune does not run either: every older snapshot is
kept, and the command names the path and exits non-zero, on every run until
the path holds something again. Take a path that is empty on purpose out of
the config, or remove the older snapshots by hand with `snapshot purge`. When
an earlier snapshot of the same name in the local index found a volume mounted
at or below a configured path (a directory on a different filesystem from its
parent, with an entry in it that the backup does not exclude) and this backup
does not, the prune does not run either: every older snapshot is kept, and the
command names the directory and exits non-zero, on every run until the volume
is mounted again. A backup also does not find the volume when it finds no such
entry in its directory, as when an automounted volume cannot be mounted, or
when that directory is no longer backed up, because it is excluded or its
configured path was removed. If any of this is on purpose, as for a retired
volume, remove the older snapshots by hand with `snapshot purge`. A file
removed during the backup does not stop the prune.
and every older snapshot is kept. One of them may be the only one that holds
what was skipped. The command says so and exits `3`. The prune does not run
either when the backup found nothing in a configured path, as in the mount
point of a volume that is not mounted. Every older snapshot is kept, and the
command names the path and exits `1`, on every run until the path holds
something again. Take a path that is empty on purpose out of the config, or
remove the older snapshots by hand with `snapshot purge`. The prune does not
run either when this backup does not find a volume that an earlier snapshot of
the same name in the local index found. The volume's mount point is a
directory at or below a configured path. That directory is on a different
filesystem from its parent and holds an entry the backup does not exclude.
Every older snapshot is kept, and the command names the directory and exits
`1`, on every run until the volume is mounted again. A backup also does not
find the volume when it finds no such entry in its directory, as when an
automounted volume cannot be mounted. Nor does it find the volume when that
directory is no longer backed up, because it is excluded or its configured
path was removed. If any of this is on purpose, as for a retired volume,
remove the older snapshots by hand with `snapshot purge`. A file removed
during the backup does not stop the prune.
- `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than this
duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). Without
`--prune`, or with a value that is not a valid duration, the command fails
before the backup starts.
- `--dry-run`: Scan the source paths and print what a backup would store,
without storing it. The summary gives the files examined, to back up and
unchanged, their sizes, and the count of excluded entries. An excluded
directory counts once, since the scan does not enter it. With `--verbose`, it
also prints each excluded entry and the exclude pattern that matched it. It
writes nothing to the destination and records nothing in the local index, so
it does not bind the index to the destination. Like the other commands that
read the local index, it creates an empty one if there is none. When the local
index holds an incomplete snapshot from an interrupted backup, it says so: the
next backup drops that snapshot first and may back up more files than the dry
run counts. It cannot be combined with `--prune`.
- Fails before it writes anything when a `file://` destination directory does
not exist, or when the local index holds snapshots and the destination answers
that it has no `vaultik-destination.txt`. When that look-up fails for any
+11 -10
View File
@@ -39,10 +39,10 @@ const (
// hold the PID lock so that at most one runs at a time.
mutating lockMode = iota
// readOnly commands (info, snapshot list, snapshot files, snapshot
// verify, remote info, snapshot restore, snapshot create --dry-run)
// do not write the local index or the remote store, so they run
// without the lock and are never blocked by a running mutator.
// restore writes only to the target directory it is given.
// verify, remote info, snapshot restore) do not write the local index
// or the remote store, so they run without the lock and are never
// blocked by a running mutator. restore writes only to the target
// directory it is given.
readOnly
)
@@ -198,7 +198,8 @@ func RunApp(ctx context.Context, app *fx.App) error {
// (or, under `snapshot verify --json`, put in its document). Entry
// turns it into a non-zero exit status without printing anything
// further, so the error line is not doubled. It flows up from
// RunOperation through cobra to Entry.
// RunOperation through cobra to Entry, wrapping the operation's error
// so that Entry can choose the exit status from it.
var errReported = errors.New("operation failed")
// errInterrupted marks an operation that SIGINT or SIGTERM stopped
@@ -232,8 +233,8 @@ func RunOperation(
) error {
var (
mu sync.Mutex
finished bool // op returned before any interrupt cancelled it
failed bool // op finished with an error
finished bool // op returned before any interrupt cancelled it
opErr error // what op returned, when it finished
)
opts.Invokes = append(opts.Invokes,
@@ -258,7 +259,7 @@ func RunOperation(
mu.Lock()
finished = true
failed = err != nil
opErr = err
mu.Unlock()
}
@@ -303,8 +304,8 @@ func RunOperation(
switch {
case !finished:
return errInterrupted
case failed:
return errReported
case opErr != nil:
return fmt.Errorf("%w: %w", errReported, opErr)
default:
return nil
}
+16 -2
View File
@@ -9,6 +9,7 @@ import (
"sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// shortCommitLen is the number of git commit hash characters shown in
@@ -20,6 +21,13 @@ const shortCommitLen = 12
// shell reports for a command stopped by Ctrl-C.
const exitCodeInterrupted = 130
// exitCodeSkippedFiles is the exit status of a `snapshot create` or
// `snapshot restore` that finished but skipped files under
// --skip-errors. It is the status `restic backup` uses for a snapshot
// that could not read some files, so a cron job can tell such a run
// from one that made no backup.
const exitCodeSkippedFiles = 3
// Entry is the main entry point for the CLI application.
// It first calls globals.FillUnsetFromBuildInfo, which fills in the
// version, commit and commit date that the linker flags did not set.
@@ -30,8 +38,9 @@ const exitCodeInterrupted = 130
// The banner goes to stderr because stdout carries only the output the
// user asked for, such as a completion script or a `config get` value.
//
// It returns the process exit code (0 on success, 130 when interrupted,
// 1 on any other error) rather than calling os.Exit, so that main's
// It returns the process exit code (0 on success, 3 when files were
// skipped, 130 when interrupted, 1 on any other error) rather than
// calling os.Exit, so that main's
// deferred profile writers run before the process ends. See run in
// cmd/vaultik/main.go.
func Entry() int {
@@ -58,6 +67,11 @@ func Entry() int {
return exitCodeInterrupted
}
if errors.Is(err, vaultik.ErrFilesUnreadable) ||
errors.Is(err, vaultik.ErrFilesFailedRestore) {
return exitCodeSkippedFiles
}
return 1
}
@@ -0,0 +1,83 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const flagSkipErrors = "--skip-errors"
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
// under --skip-errors, a source directory that does not exist, which the
// walk skips. The command exits with exitCodeSkippedFiles and reports
// the backup as completed, not as a failed snapshot creation. See
// https://git.eeqj.de/sneak/vaultik/issues/310 and
// https://git.eeqj.de/sneak/vaultik/issues/439.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
// writeBackupConfig would create the source directory.
configPath := writeHermeticPruneConfig(t, false)
code, stdout, stderr := runEntry(t,
flagConfig, configPath, flagSkipErrors, cmdSnapshot, cmdCreate)
assert.Equal(t, exitCodeSkippedFiles, code)
assert.Contains(t, stdout,
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
assert.NotContains(t, stdout, "Finished successfully.")
assert.Contains(t, stderr,
"Backup completed, but 1 file(s) could not be read")
assert.NotContains(t, stderr, "Snapshot creation failed")
}
// TestEntrySnapshotRestoreSkipErrorsExitStatus backs up, deletes the
// blobs from the destination, and restores under --skip-errors, so every
// file is skipped. A restore that only skipped files exits with
// exitCodeSkippedFiles. One that was also asked for a path the snapshot
// does not hold exits 1. See
// https://git.eeqj.de/sneak/vaultik/issues/439.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotRestoreSkipErrorsExitStatus(t *testing.T) {
for _, testCase := range []struct {
name string
missingPaths []string
want int
}{
{name: "skipped files only", want: exitCodeSkippedFiles},
{
name: "and a path that matches nothing",
missingPaths: []string{"/no/such/path"},
want: 1,
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath, storeDir := writeBackupConfigWithPrivateKey(t)
code, _, _ := runEntry(t,
flagConfig, configPath, cmdSnapshot, cmdCreate)
require.Equal(t, 0, code)
require.NoError(t, os.RemoveAll(filepath.Join(storeDir, blobsDir)))
// writeBackupConfig puts the source directory beside the
// config file.
sourceDir := filepath.Join(filepath.Dir(configPath), "source")
args := append([]string{
flagConfig, configPath, flagSkipErrors, cmdSnapshot,
cmdRestore, onlySnapshotID(t, configPath), t.TempDir(),
sourceDir,
}, testCase.missingPaths...)
code, _, stderr := runEntry(t, args...)
assert.Equal(t, testCase.want, code)
assert.Contains(t, stderr, "1 file(s) failed to restore")
})
}
}
@@ -1,89 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"context"
"io"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/ui"
)
// TestEntrySnapshotCreateDryRun runs `snapshot create --dry-run
// --verbose` on a source directory that holds one kept file, one file an
// unanchored pattern excludes and one directory an anchored pattern
// excludes. The summary counts the kept file and two excluded entries,
// the excluded ones are listed with their patterns, and nothing is
// written to the destination or recorded in the local index. See
// https://git.eeqj.de/sneak/vaultik/issues/436.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateDryRun(t *testing.T) {
configPath, storeDir := writeBackupConfig(t)
sourceDir := filepath.Join(filepath.Dir(configPath), "source")
cacheDir := filepath.Join(sourceDir, "cache")
require.NoError(t, os.WriteFile(
filepath.Join(sourceDir, "debug.log"), []byte("log"), 0o600))
require.NoError(t, os.Mkdir(cacheDir, 0o750))
for _, name := range []string{"a", "b"} {
require.NoError(t, os.WriteFile(
filepath.Join(cacheDir, name), []byte(name), 0o600))
}
out := ui.NewWithColor(io.Discard, false)
require.NoError(t, writeConfigSet(out, configPath, "exclude.0", "*.log"))
require.NoError(t, writeConfigSet(out, configPath, "exclude.1", "/cache"))
code, stdout, stderr := runEntry(t, flagConfig, configPath, "--verbose",
cmdSnapshot, cmdCreate, "--dry-run")
require.Equal(t, 0, code, stderr)
assert.Contains(t, stdout, "Excluded "+filepath.Join(sourceDir, "debug.log")+
`: matches exclude pattern "*.log".`)
assert.Contains(t, stdout, "Excluded "+cacheDir+
`: matches exclude pattern "/cache".`)
assert.NotContains(t, stdout, filepath.Join(cacheDir, "a"))
assert.Contains(t, stdout, "Files: 1 examined, 1 to back up, 0 unchanged.")
assert.Contains(t, stdout, "Excluded: 2 entries.")
entries, err := os.ReadDir(storeDir)
require.NoError(t, err)
assert.Empty(t, entries, "a dry run writes nothing to the destination")
ctx := context.Background()
db, err := database.New(ctx,
filepath.Join(filepath.Dir(configPath), "index.sqlite"))
require.NoError(t, err)
defer func() { require.NoError(t, db.Close()) }()
repos := database.NewRepositories(db)
snapshots, err := repos.Snapshots.ListRecent(ctx, 1)
require.NoError(t, err)
assert.Empty(t, snapshots, "a dry run records no snapshot")
binding, err := repos.LocalMeta.Get(ctx, database.LocalMetaKeyStorageURL)
require.NoError(t, err)
assert.Empty(t, binding, "a dry run does not bind the local index")
}
// TestEntrySnapshotCreateDryRunRefusesPrune checks that --dry-run with
// --prune fails before anything runs.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateDryRunRefusesPrune(t *testing.T) {
configPath, _ := writeBackupConfig(t)
code, _, stderr := runEntry(t, flagConfig, configPath,
cmdSnapshot, cmdCreate, "--dry-run", "--prune")
assert.NotEqual(t, 0, code)
assert.Contains(t, stderr, "--dry-run cannot be used with --prune")
}
@@ -1,30 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"testing"
"github.com/stretchr/testify/assert"
)
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
// under --skip-errors, a source directory that does not exist, which the
// walk skips. The command exits non-zero and reports the backup as
// completed, not as a failed snapshot creation. See
// https://git.eeqj.de/sneak/vaultik/issues/310.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
// writeBackupConfig would create the source directory.
configPath := writeHermeticPruneConfig(t, false)
code, stdout, stderr := runEntry(t,
flagConfig, configPath, "--skip-errors", cmdSnapshot, cmdCreate)
assert.Equal(t, 1, code)
assert.Contains(t, stdout,
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
assert.NotContains(t, stdout, "Finished successfully.")
assert.Contains(t, stderr,
"Backup completed, but 1 file(s) could not be read")
assert.NotContains(t, stderr, "Snapshot creation failed")
}
+19 -35
View File
@@ -99,13 +99,6 @@ on Linux), then /etc/vaultik/config.yml.`,
// Use the backup functionality from cli package
rootFlags := GetRootFlags()
opts.ListExcluded = rootFlags.Verbose
// A dry run writes nothing, so it does not take the lock.
mode := mutating
if opts.DryRun {
mode = readOnly
}
// --cron suppression is wired through v.UI by setupGlobals.
return RunOperation(cmd.Context(), AppOptions{
@@ -116,11 +109,28 @@ on Linux), then /etc/vaultik/config.yml.`,
Cron: opts.Cron,
Quiet: rootFlags.Quiet,
},
Mode: mode,
Mode: mutating,
}, func(v *vaultik.Vaultik) error {
return v.CreateSnapshot(opts)
}, func(err error) {
reportSnapshotCreateFailure(opts.DryRun, err)
if errors.Is(err, vaultik.ErrPostBackupPruneFailed) {
log.Info("Post-backup prune failed", "error", err)
ReportErrorf("Backup completed, but %v", err)
return
}
if errors.Is(err, vaultik.ErrFilesUnreadable) ||
errors.Is(err, vaultik.ErrSourcePathEmpty) ||
errors.Is(err, vaultik.ErrVolumeNotMounted) {
log.Info("Backup completed with an error", "error", err)
ReportErrorf("Backup completed, but %v", err)
return
}
log.Info("Snapshot creation failed", "error", err)
ReportErrorf("Snapshot creation failed: %v", err)
})
},
}
@@ -139,36 +149,10 @@ on Linux), then /etc/vaultik/config.yml.`,
"With --prune: keep snapshots newer than this duration "+
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
"instead of only the latest")
cmd.Flags().BoolVar(&opts.DryRun, "dry-run", false,
"Scan the snapshot source paths and print what a backup would store, "+
"writing nothing to the local index or the backup destination; "+
"with --verbose, also print each excluded entry and the exclude "+
"pattern that matched it")
return cmd
}
// reportSnapshotCreateFailure shows the user the error `snapshot create`
// failed with. dryRun says whether it was given --dry-run.
func reportSnapshotCreateFailure(dryRun bool, err error) {
switch {
case dryRun:
log.Info("Dry run failed", "error", err)
ReportErrorf("Dry run failed: %v", err)
case errors.Is(err, vaultik.ErrPostBackupPruneFailed):
log.Info("Post-backup prune failed", "error", err)
ReportErrorf("Backup completed, but %v", err)
case errors.Is(err, vaultik.ErrFilesUnreadable) ||
errors.Is(err, vaultik.ErrSourcePathEmpty) ||
errors.Is(err, vaultik.ErrVolumeNotMounted):
log.Info("Backup completed with an error", "error", err)
ReportErrorf("Backup completed, but %v", err)
default:
log.Info("Snapshot creation failed", "error", err)
ReportErrorf("Snapshot creation failed: %v", err)
}
}
// newSnapshotListCommand creates the 'snapshot list' subcommand
func newSnapshotListCommand() *cobra.Command {
var jsonOutput bool
+13 -103
View File
@@ -71,10 +71,7 @@ type Scanner struct {
// indexAndLockFiles holds the paths of vaultik's own working files,
// which the scan treats as excluded (ScannerConfig.IndexAndLockFiles).
indexAndLockFiles map[string]struct{}
// listExcluded prints each excluded entry and why it is excluded
// (DryRun).
listExcluded bool
progress *ProgressReporter
progress *ProgressReporter
// skipErrors skips files that cannot be opened or read (logged loudly);
// packer, database, encryption, and upload errors still abort the run.
skipErrors bool
@@ -193,11 +190,6 @@ type ScanResult struct {
// itself when it is a directory. It is zero for an empty directory,
// such as the mount point of a volume that is not mounted.
EntriesFound int
// EntriesExcluded counts the entries the walk found that an exclude
// pattern matches or that are vaultik's own files. An excluded
// directory counts once: the walk does not enter it.
EntriesExcluded int
}
// NewScanner creates a new scanner instance
@@ -368,42 +360,6 @@ func (s *Scanner) Scan(
return result, nil
}
// DryRun walks path as Scan does and counts what a backup would store,
// without storing it. It reads the local index to tell unchanged files
// from new and changed ones, and writes nothing to the local index or
// the backup destination store. With listExcluded it prints each
// excluded entry and the exclude pattern that matched it.
func (s *Scanner) DryRun(
ctx context.Context, path string, listExcluded bool,
) (*ScanResult, error) {
s.currentSourcePath = path
s.listExcluded = listExcluded
result := &ScanResult{
StartTime: time.Now().UTC(),
}
knownFiles, err := s.loadDatabaseState(ctx, path)
if err != nil {
return nil, err
}
existingFiles := make(map[string]struct{})
_, err = s.scanPhase(ctx, path, result, existingFiles, knownFiles)
if err != nil {
return nil, fmt.Errorf("scan phase failed: %w", err)
}
err = s.detectDeletedFiles(ctx, path, existingFiles, result)
if err != nil {
return nil, fmt.Errorf("detecting deleted files: %w", err)
}
result.EndTime = time.Now().UTC()
return result, nil
}
// GetProgress returns the progress reporter for this scanner, or nil when
// progress is off. Scan neither starts nor stops it: the caller does,
// once for all the paths it scans, because a second Stop panics.
@@ -988,8 +944,12 @@ func (s *Scanner) scanPhase(
// or cannot be looked up is neither counted nor reported. For
// directories, skip the entire subtree. info is nil when the
// walk could not look the entry up.
if excluded, walkErr := s.skipExcluded(filePath, path, info, result); excluded {
return walkErr
if s.shouldExclude(filePath, path) {
if info != nil && info.IsDir() {
return filepath.SkipDir
}
return nil
}
// The source path is in no directory listing, so its absence is
@@ -1056,32 +1016,6 @@ func (s *Scanner) scanPhase(
}, nil
}
// skipExcluded reports whether the walk skips filePath, under rootPath,
// because it is excluded, and counts it if so. The error is what the walk
// function returns for it: filepath.SkipDir for a directory, so the walk
// does not enter it, and nil otherwise. info is nil when the walk could
// not look the entry up.
func (s *Scanner) skipExcluded(
filePath, rootPath string, info os.FileInfo, result *ScanResult,
) (bool, error) {
pattern, excluded := s.excludingPattern(filePath, rootPath)
if !excluded {
return false, nil
}
result.EntriesExcluded++
if s.listExcluded {
s.printExcluded(filePath, pattern)
}
if info != nil && info.IsDir() {
return true, filepath.SkipDir
}
return true, nil
}
// countNonDirectories returns how many of files are not directories.
func countNonDirectories(files map[string]*database.File) int64 {
var count int64
@@ -2417,32 +2351,23 @@ func compileExcludePatterns(patterns []string) ([]compiledPattern, error) {
// filePath is the full path to the file
// rootPath is the root of the backup source directory
func (s *Scanner) shouldExclude(filePath, rootPath string) bool {
_, excluded := s.excludingPattern(filePath, rootPath)
return excluded
}
// excludingPattern reports whether shouldExclude excludes filePath and
// returns the exclude pattern, as configured, that matches it. The
// pattern is "" for one of vaultik's own working files.
func (s *Scanner) excludingPattern(filePath, rootPath string) (string, bool) {
if _, ok := s.indexAndLockFiles[filePath]; ok {
return "", true
return true
}
if len(s.compiledExclude) == 0 {
return "", false
return false
}
// Get the relative path from root
relPath, err := filepath.Rel(rootPath, filePath)
if err != nil {
return "", false
return false
}
// Never exclude the root directory itself
if relPath == "." {
return "", false
return false
}
// Normalize path separators
@@ -2451,26 +2376,11 @@ func (s *Scanner) excludingPattern(filePath, rootPath string) (string, bool) {
// Check each pattern
for _, cp := range s.compiledExclude {
if cp.matches(relPath) {
return cp.original, true
return true
}
}
return "", false
}
// printExcluded prints that the walk excluded filePath and why: pattern
// is the exclude pattern that matched it, "" for one of vaultik's own
// working files.
func (s *Scanner) printExcluded(filePath, pattern string) {
if pattern == "" {
s.ui.Infof("Excluded %s: vaultik's own index or lock file.",
s.ui.Path(filePath))
return
}
s.ui.Infof("Excluded %s: matches exclude pattern %q.",
s.ui.Path(filePath), pattern)
return false
}
// matches reports whether the compiled pattern matches the given
+19 -8
View File
@@ -25,9 +25,13 @@ import (
"sneak.berlin/go/vaultik/internal/types"
)
// ErrFilesFailedRestore is returned when --skip-errors skipped files the
// restore could not write. Every other file was restored, so the caller
// can report it apart from a failed restore.
var ErrFilesFailedRestore = errors.New("file(s) failed to restore")
// Sentinel errors for restore failures.
var (
errFilesFailedRestore = errors.New("file(s) failed to restore")
errPathsNotInSnapshot = errors.New("requested path(s) not in the snapshot")
errFilesFailedVerify = errors.New("files failed verification")
errDecryptionKeyRequired = errors.New(
@@ -201,9 +205,9 @@ func (v *Vaultik) Restore(opts *RestoreOptions) error {
}
// finishRestore prints the restore summary and the post-restore
// warnings, runs optional verification, and converts any failed-file
// count into an error. Otherwise it returns unmatchedErr, the error for
// the requested paths that matched nothing (nil when there are none).
// warnings, runs optional verification, and returns the error the
// restore ends with: one for the files --skip-errors skipped, one for
// unmatchedErr, the requested paths that matched nothing, or nil.
func (v *Vaultik) finishRestore(
repos *database.Repositories,
files []*database.File,
@@ -250,11 +254,18 @@ func (v *Vaultik) finishRestore(
}
}
if result.FilesFailed > 0 {
return fmt.Errorf("%d %w", result.FilesFailed, errFilesFailedRestore)
switch {
case result.FilesFailed > 0 && unmatchedErr != nil:
// A requested path that matched nothing outweighs skipped files:
// the error wraps only unmatchedErr, so the command does not exit
// as a restore that only skipped files.
return fmt.Errorf("%w; %d %s",
unmatchedErr, result.FilesFailed, ErrFilesFailedRestore.Error())
case result.FilesFailed > 0:
return fmt.Errorf("%d %w", result.FilesFailed, ErrFilesFailedRestore)
default:
return unmatchedErr
}
return unmatchedErr
}
// warnIfOwnershipNotPreserved warns once if some entry restored under
+1 -1
View File
@@ -80,7 +80,7 @@ func TestRestoreFreesBlobsNoPendingFileNeeds(t *testing.T) {
TargetDir: restoreDir,
SkipErrors: true,
})
require.ErrorIs(t, err, errFilesFailedRestore)
require.ErrorIs(t, err, ErrFilesFailedRestore)
requireRestoredFiles(t, fs, restoreDir, originals)
assert.LessOrEqualf(t, peak, 1,
+41 -54
View File
@@ -33,7 +33,6 @@ var (
"creating a snapshot needs at least one public key in " +
"age_recipients (generate a keypair with: age-keygen)")
errKeepNewerThanNeedsPrune = errors.New("--keep-newer-than requires --prune")
errDryRunWithPrune = errors.New("--dry-run cannot be used with --prune")
)
// ErrPostBackupPruneFailed wraps a failure of the post-backup prune,
@@ -50,8 +49,7 @@ var ErrFilesUnreadable = errors.New("file(s) could not be read")
// prune because the scan found nothing in a configured path of a
// snapshot, as in the mount point of a volume that is not mounted. The
// snapshots are complete and kept, so the caller can report it apart
// from a failed backup. A run that also skipped unreadable files
// returns ErrFilesUnreadable instead.
// from a failed backup.
var ErrSourcePathEmpty = errors.New("source path(s) were empty")
// ErrVolumeNotMounted is returned when --prune skipped the post-backup
@@ -60,8 +58,8 @@ var ErrSourcePathEmpty = errors.New("source path(s) were empty")
// mounted there is not mounted, the run found no entry in it, or the
// directory is no longer backed up. The snapshots are complete and
// kept, so the caller can report it apart from a failed backup. A run
// that also skipped unreadable files or found a configured path empty
// returns ErrFilesUnreadable or ErrSourcePathEmpty instead.
// that also found a configured path empty returns ErrSourcePathEmpty
// instead.
var ErrVolumeNotMounted = errors.New(
"mount point(s) an earlier snapshot recorded were not found")
@@ -78,24 +76,12 @@ type SnapshotCreateOptions struct {
KeepNewerThan string
SkipErrors bool // Skip file read errors (log them loudly but continue)
Snapshots []string // Optional list of snapshot names to process (empty = all)
// DryRun scans the snapshot source paths and prints what a backup
// would store, writing nothing to the local index or the backup
// destination store.
DryRun bool
// ListExcluded makes a dry run print each excluded entry and the
// exclude pattern that matched it. A backup ignores it.
ListExcluded bool
}
// Validate fails when DryRun is set with Prune, or when KeepNewerThan is
// set without Prune or is not a valid duration. Only the post-backup
// prune reads KeepNewerThan, after the backup, so the caller checks it
// before starting one.
// Validate fails when KeepNewerThan is set without Prune or is not a
// valid duration. Only the post-backup prune reads KeepNewerThan, after
// the backup, so the caller checks it before starting one.
func (opts *SnapshotCreateOptions) Validate() error {
if opts.DryRun && opts.Prune {
return errDryRunWithPrune
}
if opts.KeepNewerThan == "" {
return nil
}
@@ -120,10 +106,6 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
return errNoAgeRecipients
}
if opts.DryRun {
return v.dryRunSnapshots(opts)
}
overallStartTime := time.Now()
log.Info("Starting snapshot creation",
@@ -260,15 +242,29 @@ func (v *Vaultik) skipPostBackupPrune(
if filesUnreadable > 0 {
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
"could not be read; older snapshots are kept.", filesUnreadable)
}
var err error
switch {
case len(emptyPaths) > 0:
err = fmt.Errorf("%d %w", len(emptyPaths), ErrSourcePathEmpty)
case len(unmountedPaths) > 0:
err = fmt.Errorf("%d %w", len(unmountedPaths), ErrVolumeNotMounted)
default:
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
}
if len(emptyPaths) > 0 {
return fmt.Errorf("%d %w", len(emptyPaths), ErrSourcePathEmpty)
// An empty path or a missing volume can mean a whole volume is not
// in the backup, which outweighs skipped files: the error wraps only
// the former, so the command does not exit as a backup that only
// skipped files.
if filesUnreadable > 0 {
return fmt.Errorf("%w; %d %s",
err, filesUnreadable, ErrFilesUnreadable.Error())
}
return fmt.Errorf("%d %w", len(unmountedPaths), ErrVolumeNotMounted)
return err
}
// finishCreateSnapshot ends a run whose snapshots are all complete and
@@ -375,9 +371,6 @@ type snapshotStats struct {
// could not be read or listed.
totalFilesUnreadable int
// The entries the scan excluded; an excluded directory counts once.
totalEntriesExcluded int
// The configured paths in which the scan found nothing.
emptyPaths []string
@@ -391,29 +384,6 @@ type snapshotStats struct {
blobUncompressedSize int64
}
// add adds the result of scanning the configured path dir to the totals.
func (stats *snapshotStats) add(dir string, result *snapshot.ScanResult) {
stats.totalFiles += result.FilesScanned
stats.totalBytes += result.BytesScanned
stats.totalChunks += result.ChunksCreated
stats.newChunkSize += result.NewChunkSize
stats.totalBlobs += result.BlobsCreated
stats.totalFilesSkipped += result.FilesSkipped
stats.totalFilesFailed += result.FilesFailed
stats.totalFilesUnreadable += result.FilesUnreadable
stats.totalBytesSkipped += result.BytesSkipped
stats.totalFilesDeleted += result.FilesDeleted
stats.totalBytesDeleted += result.BytesDeleted
stats.totalBlobsUploaded += result.BlobsUploaded
stats.totalBytesUploaded += result.BytesUploaded
stats.uploadDuration += result.UploadDuration
stats.totalEntriesExcluded += result.EntriesExcluded
if result.EntriesFound == 0 {
stats.emptyPaths = append(stats.emptyPaths, dir)
}
}
// createNamedSnapshot creates a single named snapshot and returns the
// statistics of its scan.
func (v *Vaultik) createNamedSnapshot(
@@ -607,7 +577,24 @@ func (v *Vaultik) scanAllDirectories(
return nil, fmt.Errorf("failed to scan %s: %w", dir, err)
}
stats.add(dir, result)
stats.totalFiles += result.FilesScanned
stats.totalBytes += result.BytesScanned
stats.totalChunks += result.ChunksCreated
stats.newChunkSize += result.NewChunkSize
stats.totalBlobs += result.BlobsCreated
stats.totalFilesSkipped += result.FilesSkipped
stats.totalFilesFailed += result.FilesFailed
stats.totalFilesUnreadable += result.FilesUnreadable
stats.totalBytesSkipped += result.BytesSkipped
stats.totalFilesDeleted += result.FilesDeleted
stats.totalBytesDeleted += result.BytesDeleted
stats.totalBlobsUploaded += result.BlobsUploaded
stats.totalBytesUploaded += result.BytesUploaded
stats.uploadDuration += result.UploadDuration
if result.EntriesFound == 0 {
stats.emptyPaths = append(stats.emptyPaths, dir)
}
log.Info("Directory scan complete",
"path", dir,
-133
View File
@@ -1,133 +0,0 @@
package vaultik
import (
"fmt"
"sneak.berlin/go/vaultik/internal/snapshot"
)
// dryRunSnapshots scans the source paths of each snapshot named in
// opts, or of every configured one, and prints what a backup would
// store. It writes nothing to the local index or the backup destination
// store. It first makes the read-only checks a backup makes, so it fails
// where a backup would: a local index bound to a different destination,
// or a destination that answers it has no marker while the local index
// holds snapshots stored there.
func (v *Vaultik) dryRunSnapshots(opts *SnapshotCreateOptions) error {
err := v.checkStorageBinding()
if err != nil {
return err
}
err = v.checkDestinationMarker()
if err != nil {
return err
}
// A backup first drops incomplete snapshots and the file records only
// they list (PruneDatabase), so it backs up those files again; the
// dry run counts them as unchanged.
incomplete, err := v.Repositories.Snapshots.GetIncompleteSnapshots(v.ctx)
if err != nil {
return fmt.Errorf("getting incomplete snapshots: %w", err)
}
if len(incomplete) > 0 {
v.UI.Noticef("The local index holds %s incomplete snapshot(s) from "+
"an interrupted backup. The next backup drops them first and may "+
"back up more files than this dry run counts.",
v.UI.Count(len(incomplete)))
}
snapshotNames, err := v.snapshotNamesToCreate(opts.Snapshots)
if err != nil {
return err
}
for _, snapName := range snapshotNames {
err = v.dryRunSnapshot(opts, snapName)
if err != nil {
return err
}
}
return nil
}
// dryRunSnapshot scans the source paths of the snapshot named snapName
// and prints what a backup of it would store.
func (v *Vaultik) dryRunSnapshot(
opts *SnapshotCreateOptions, snapName string,
) error {
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
if err != nil {
return err
}
previousSnapshotID, err := v.previousSnapshotID(snapName)
if err != nil {
return err
}
scanner := v.ScannerFactory(snapshot.ScannerParams{
UI: v.UI,
Fs: v.Fs,
Exclude: v.Config.GetExcludes(snapName),
SkipErrors: opts.SkipErrors,
PreviousSnapshotID: previousSnapshotID,
})
v.UI.Beginf("Dry run of snapshot %s.", snapName)
stats := &snapshotStats{}
for i, dir := range resolvedDirs {
v.UI.Beginf("Enumerating snapshot source files in %s (%d of %d).",
v.UI.Path(dir), i+1, len(resolvedDirs))
result, err := scanner.DryRun(v.ctx, dir, opts.ListExcluded)
if err != nil {
return fmt.Errorf("failed to scan %s: %w", dir, err)
}
stats.add(dir, result)
}
v.printDryRunSummary(snapName, stats)
return nil
}
// printDryRunSummary prints what a backup of the snapshot named snapName
// would store, in the terms of printSnapshotSummary.
func (v *Vaultik) printDryRunSummary(snapName string, stats *snapshotStats) {
filesToBackUp := stats.totalFiles - stats.totalFilesSkipped -
stats.totalFilesFailed
v.UI.Completef("Dry run of snapshot %s complete; nothing was stored.",
snapName)
filesMsg := fmt.Sprintf("Files: %s examined, %s to back up, %s unchanged",
v.UI.Count(stats.totalFiles),
v.UI.Count(filesToBackUp),
v.UI.Count(stats.totalFilesSkipped))
if stats.totalFilesFailed > 0 {
filesMsg += fmt.Sprintf(", %s failed", v.UI.Count(stats.totalFilesFailed))
}
if stats.totalFilesDeleted > 0 {
filesMsg += fmt.Sprintf(", %s deleted", v.UI.Count(stats.totalFilesDeleted))
}
v.UI.Detailf("%s.", filesMsg)
dataMsg := fmt.Sprintf("Data: %s total (%s to back up)",
v.UI.Size(stats.totalBytes+stats.totalBytesSkipped),
v.UI.Size(stats.totalBytes))
if stats.totalBytesDeleted > 0 {
dataMsg += fmt.Sprintf(", %s deleted", v.UI.Size(stats.totalBytesDeleted))
}
v.UI.Detailf("%s.", dataMsg)
v.UI.Detailf("Excluded: %s entries.", v.UI.Count(stats.totalEntriesExcluded))
}
-69
View File
@@ -1,69 +0,0 @@
package vaultik_test
import (
"context"
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// dryRunSnapshotName is the newSummaryEnv snapshot the dry run test
// backs up and then dry-runs.
const dryRunSnapshotName = "first"
// TestDryRunCountsWhatABackupWouldStore runs a dry run after a backup:
// the unchanged files count as unchanged, a rewritten one as to back up,
// and the dry run records no snapshot. See
// https://git.eeqj.de/sneak/vaultik/issues/436.
func TestDryRunCountsWhatABackupWouldStore(t *testing.T) {
t.Parallel()
env := newSummaryEnv(t)
env.backUp(t, dryRunSnapshotName, false)
dryRun := func() string {
env.out.Reset()
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
DryRun: true,
Snapshots: []string{dryRunSnapshotName},
}))
return env.out.String()
}
assert.Contains(t, dryRun(),
"Files: 3 examined, 0 to back up, 3 unchanged.")
require.NoError(t, os.WriteFile(env.aPath, []byte("rewritten"), 0o600))
assert.Contains(t, dryRun(),
"Files: 3 examined, 1 to back up, 2 unchanged.")
// localSnapshotID fails unless exactly one snapshot has the name.
localSnapshotID(context.Background(), t, env.repos, dryRunSnapshotName)
}
// TestDryRunNamesIncompleteSnapshots checks that a dry run says so when
// the local index holds an incomplete snapshot, which the next backup
// drops before it scans.
func TestDryRunNamesIncompleteSnapshots(t *testing.T) {
t.Parallel()
env := newSummaryEnv(t)
env.backUp(t, dryRunSnapshotName, false)
_, err := env.v.SnapshotManager.CreateSnapshotWithName(
context.Background(), testHostname, "split", "v", "g")
require.NoError(t, err)
env.out.Reset()
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
DryRun: true,
Snapshots: []string{dryRunSnapshotName},
}))
assert.Contains(t, env.out.String(),
"The local index holds 1 incomplete snapshot(s)")
}
+9 -3
View File
@@ -176,8 +176,11 @@ func TestBackupThroughSymlinkWhoseTargetIsGoneDoesNotPrune(t *testing.T) {
// A backup under --skip-errors --prune that both skipped a file it could
// not read and found nothing in a configured path names the empty path
// too when it skips the prune. See
// https://git.eeqj.de/sneak/vaultik/issues/388.
// too when it skips the prune. It ends with the empty path's error, which
// names the unreadable file too but does not wrap ErrFilesUnreadable, so
// the command does not exit as a backup that only skipped files. See
// https://git.eeqj.de/sneak/vaultik/issues/388 and
// https://git.eeqj.de/sneak/vaultik/issues/439.
func TestBackupWithEmptySourcePathAndUnreadableFileNamesBoth(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
@@ -207,7 +210,10 @@ func TestBackupWithEmptySourcePathAndUnreadableFileNamesBoth(t *testing.T) {
SkipErrors: true,
Snapshots: []string{snapshotName},
})
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
require.ErrorIs(t, err, vaultik.ErrSourcePathEmpty)
require.NotErrorIs(t, err, vaultik.ErrFilesUnreadable)
require.EqualError(t, err,
"1 source path(s) were empty; 1 file(s) could not be read")
assert.Contains(t, env.out.String(),
"Skipped the post-backup prune because 1 file(s) could not be read")
assert.Contains(t, env.out.String(),