Compare commits

..
Author SHA1 Message Date
clawbot d71be0b4c8 Retry a failed blob upload five times, 3 seconds apart (closes #476)
check / check (push) Waiting to run
A blob upload that failed, such as one cut off by a lost connection,
ended the backup at once. It is now tried again from the start of the
blob up to five times, 3 seconds apart, with a warning before each
retry; an interrupt still stops it at once. The panic in the report,
"close of closed channel" from the progress reporter, was already fixed
on next by the commit for issue 253; a test now covers a failing upload
with the progress reporter on.

Model: opus-5-5
2026-11-19 09:28:57 +00:00
sneak 4605eae024 next: integrate accumulated work into main (#203)
check / check (push) Waiting to run
Reviewed-on: #203
2026-11-19 10:27:45 +01:00
clawbot 5b09271751 Report no device number for a zeroed stat (closes #462)
check / check (push) Waiting to run
deviceNumberFromStat returned device 0 and true for a stat whose device
and inode numbers are both zero. afero's unreleased master gives such a
stat for every in-memory file; the pinned v1.15.0 gives none. It now
returns false for it, as its comment says, and a test covers both
in-memory shapes.

Model: opus-5-5
2026-11-19 09:53:18 +01:00
clawbot 9fedcbeb0a Refuse a relative path on an rclone remote of type local (closes #465)
check / check (push) Waiting to run
rclone resolves a relative path on its local backend against the working
directory, so rclone://localfs/srv/backup wrote under whatever directory
vaultik ran from. Opening such a destination now fails with an error that
shows the absolute form, rclone://localfs//srv/backup. Other remotes keep
relative paths, which rclone resolves on the remote. The README says how
to write an absolute rclone path.

Model: opus-5-5
2026-11-19 09:24:16 +01:00
14 changed files with 496 additions and 531 deletions

No files matched your search

+10 -13
View File
@@ -169,7 +169,7 @@ vaultik [--config <path>] config init
vaultik [--config <path>] config edit
vaultik [--config <path>] config get <key>
vaultik [--config <path>] config set <key> <value>
vaultik [--config <path>] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>] [--dry-run]
vaultik [--config <path>] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>]
vaultik [--config <path>] snapshot list [--json] # alias: ls
vaultik [--config <path>] snapshot files <snapshot-id> [path] [--json]
vaultik [--config <path>] snapshot verify <snapshot-id> [--deep] [--json]
@@ -369,17 +369,6 @@ In such a config, an unknown key or a value of the wrong type is refused.
duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). Without
`--prune`, or with a value that is not a valid duration, the command fails
before the backup starts.
- `--dry-run`: Scan the source paths and print what a backup would store,
without storing it. The summary gives the files examined, to back up and
unchanged, their sizes, and the count of excluded entries. An excluded
directory counts once, since the scan does not enter it. With `--verbose`, it
also prints each excluded entry and the exclude pattern that matched it. It
writes nothing to the destination and records nothing in the local index, so
it does not bind the index to the destination. Like the other commands that
read the local index, it creates an empty one if there is none. When the local
index holds an incomplete snapshot from an interrupted backup, it says so: the
next backup drops that snapshot first and may back up more files than the dry
run counts. It cannot be combined with `--prune`.
- Fails before it writes anything when a `file://` destination directory does
not exist, or when the local index holds snapshots and the destination answers
that it has no `vaultik-destination.txt`. When that look-up fails for any
@@ -613,7 +602,11 @@ only `vaultik-destination.txt` stops such a backup, and only while the local
index holds snapshots.
**Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud
providers. Requires rclone to be configured separately (`rclone config`).
providers. Requires rclone to be configured separately (`rclone config`). The
path after the remote name is relative: rclone resolves it on the remote, for
example against the login directory on sftp. A second slash makes it absolute,
as in `rclone://remote//srv/backup`. A remote of type `local` needs an absolute
path; a relative one is refused.
An upload cut off part-way leaves nothing under the object's name on S3, which
shows an object only once its upload has completed, and on the local filesystem
@@ -625,6 +618,10 @@ being written, a killed upload can leave a truncated object under its name,
which a later backup takes for complete. A leftover `.partial` file is ignored
and can be deleted.
A blob upload that fails, such as one cut off by a lost connection, is tried
again up to five times, three seconds apart. Each new try is announced with a
warning. If the last try fails too, the backup stops with the upload's error.
A destination shared between hosts or users, each with its own local index,
relies on `prune_grace_period` to keep one's prune from deleting the blobs of
another's backup still running, or its directory under `metadata/` that has no
+4 -4
View File
@@ -39,10 +39,10 @@ const (
// hold the PID lock so that at most one runs at a time.
mutating lockMode = iota
// readOnly commands (info, snapshot list, snapshot files, snapshot
// verify, remote info, snapshot restore, snapshot create --dry-run)
// do not write the local index or the remote store, so they run
// without the lock and are never blocked by a running mutator.
// restore writes only to the target directory it is given.
// verify, remote info, snapshot restore) do not write the local index
// or the remote store, so they run without the lock and are never
// blocked by a running mutator. restore writes only to the target
// directory it is given.
readOnly
)
@@ -1,89 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"context"
"io"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/ui"
)
// TestEntrySnapshotCreateDryRun runs `snapshot create --dry-run
// --verbose` on a source directory that holds one kept file, one file an
// unanchored pattern excludes and one directory an anchored pattern
// excludes. The summary counts the kept file and two excluded entries,
// the excluded ones are listed with their patterns, and nothing is
// written to the destination or recorded in the local index. See
// https://git.eeqj.de/sneak/vaultik/issues/436.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateDryRun(t *testing.T) {
configPath, storeDir := writeBackupConfig(t)
sourceDir := filepath.Join(filepath.Dir(configPath), "source")
cacheDir := filepath.Join(sourceDir, "cache")
require.NoError(t, os.WriteFile(
filepath.Join(sourceDir, "debug.log"), []byte("log"), 0o600))
require.NoError(t, os.Mkdir(cacheDir, 0o750))
for _, name := range []string{"a", "b"} {
require.NoError(t, os.WriteFile(
filepath.Join(cacheDir, name), []byte(name), 0o600))
}
out := ui.NewWithColor(io.Discard, false)
require.NoError(t, writeConfigSet(out, configPath, "exclude.0", "*.log"))
require.NoError(t, writeConfigSet(out, configPath, "exclude.1", "/cache"))
code, stdout, stderr := runEntry(t, flagConfig, configPath, "--verbose",
cmdSnapshot, cmdCreate, "--dry-run")
require.Equal(t, 0, code, stderr)
assert.Contains(t, stdout, "Excluded "+filepath.Join(sourceDir, "debug.log")+
`: matches exclude pattern "*.log".`)
assert.Contains(t, stdout, "Excluded "+cacheDir+
`: matches exclude pattern "/cache".`)
assert.NotContains(t, stdout, filepath.Join(cacheDir, "a"))
assert.Contains(t, stdout, "Files: 1 examined, 1 to back up, 0 unchanged.")
assert.Contains(t, stdout, "Excluded: 2 entries.")
entries, err := os.ReadDir(storeDir)
require.NoError(t, err)
assert.Empty(t, entries, "a dry run writes nothing to the destination")
ctx := context.Background()
db, err := database.New(ctx,
filepath.Join(filepath.Dir(configPath), "index.sqlite"))
require.NoError(t, err)
defer func() { require.NoError(t, db.Close()) }()
repos := database.NewRepositories(db)
snapshots, err := repos.Snapshots.ListRecent(ctx, 1)
require.NoError(t, err)
assert.Empty(t, snapshots, "a dry run records no snapshot")
binding, err := repos.LocalMeta.Get(ctx, database.LocalMetaKeyStorageURL)
require.NoError(t, err)
assert.Empty(t, binding, "a dry run does not bind the local index")
}
// TestEntrySnapshotCreateDryRunRefusesPrune checks that --dry-run with
// --prune fails before anything runs.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateDryRunRefusesPrune(t *testing.T) {
configPath, _ := writeBackupConfig(t)
code, _, stderr := runEntry(t, flagConfig, configPath,
cmdSnapshot, cmdCreate, "--dry-run", "--prune")
assert.NotEqual(t, 0, code)
assert.Contains(t, stderr, "--dry-run cannot be used with --prune")
}
+19 -35
View File
@@ -99,13 +99,6 @@ on Linux), then /etc/vaultik/config.yml.`,
// Use the backup functionality from cli package
rootFlags := GetRootFlags()
opts.ListExcluded = rootFlags.Verbose
// A dry run writes nothing, so it does not take the lock.
mode := mutating
if opts.DryRun {
mode = readOnly
}
// --cron suppression is wired through v.UI by setupGlobals.
return RunOperation(cmd.Context(), AppOptions{
@@ -116,11 +109,28 @@ on Linux), then /etc/vaultik/config.yml.`,
Cron: opts.Cron,
Quiet: rootFlags.Quiet,
},
Mode: mode,
Mode: mutating,
}, func(v *vaultik.Vaultik) error {
return v.CreateSnapshot(opts)
}, func(err error) {
reportSnapshotCreateFailure(opts.DryRun, err)
if errors.Is(err, vaultik.ErrPostBackupPruneFailed) {
log.Info("Post-backup prune failed", "error", err)
ReportErrorf("Backup completed, but %v", err)
return
}
if errors.Is(err, vaultik.ErrFilesUnreadable) ||
errors.Is(err, vaultik.ErrSourcePathEmpty) ||
errors.Is(err, vaultik.ErrVolumeNotMounted) {
log.Info("Backup completed with an error", "error", err)
ReportErrorf("Backup completed, but %v", err)
return
}
log.Info("Snapshot creation failed", "error", err)
ReportErrorf("Snapshot creation failed: %v", err)
})
},
}
@@ -139,36 +149,10 @@ on Linux), then /etc/vaultik/config.yml.`,
"With --prune: keep snapshots newer than this duration "+
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
"instead of only the latest")
cmd.Flags().BoolVar(&opts.DryRun, "dry-run", false,
"Scan the snapshot source paths and print what a backup would store, "+
"writing nothing to the local index or the backup destination; "+
"with --verbose, also print each excluded entry and the exclude "+
"pattern that matched it")
return cmd
}
// reportSnapshotCreateFailure shows the user the error `snapshot create`
// failed with. dryRun says whether it was given --dry-run.
func reportSnapshotCreateFailure(dryRun bool, err error) {
switch {
case dryRun:
log.Info("Dry run failed", "error", err)
ReportErrorf("Dry run failed: %v", err)
case errors.Is(err, vaultik.ErrPostBackupPruneFailed):
log.Info("Post-backup prune failed", "error", err)
ReportErrorf("Backup completed, but %v", err)
case errors.Is(err, vaultik.ErrFilesUnreadable) ||
errors.Is(err, vaultik.ErrSourcePathEmpty) ||
errors.Is(err, vaultik.ErrVolumeNotMounted):
log.Info("Backup completed with an error", "error", err)
ReportErrorf("Backup completed, but %v", err)
default:
log.Info("Snapshot creation failed", "error", err)
ReportErrorf("Snapshot creation failed: %v", err)
}
}
// newSnapshotListCommand creates the 'snapshot list' subcommand
func newSnapshotListCommand() *cobra.Command {
var jsonOutput bool
+154
View File
@@ -0,0 +1,154 @@
//nolint:testpackage // exercises the unexported uploadBlobIfNeeded
package snapshot
import (
"bytes"
"context"
"io"
"path/filepath"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blob"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
"sneak.berlin/go/vaultik/internal/ui"
)
// retryTestBlobSize is larger than what a failing faultstore Put reads,
// so a new try that did not rewind the blob would store a short object.
const retryTestBlobSize = 4096
// retryTestBlobKey is where the retry tests upload their blob.
const retryTestBlobKey = "blobs/retry"
// uploadRetryTestBlob uploads a retryTestBlobSize blob through store,
// waiting delay between tries, and returns the blob's bytes, the warnings
// the upload printed and its error.
func uploadRetryTestBlob(
ctx context.Context, store storage.Storer, delay time.Duration,
) ([]byte, int, error) {
data := bytes.Repeat([]byte("0123456789abcdef"), retryTestBlobSize/16)
output := ui.NewWithColor(io.Discard, false)
s := &Scanner{storage: store, ui: output, blobUploadRetryDelay: delay}
_, _, err := s.uploadBlobIfNeeded(ctx, retryTestBlobKey, &blob.WithReader{
FinishedBlob: &blob.FinishedBlob{
Hash: "retry",
Compressed: int64(len(data)),
},
Reader: bytes.NewReader(data),
}, time.Now(), &ScanResult{})
return data, output.WarningCount(), err
}
// A blob upload that fails is tried again from the start of the blob,
// and each failure is a warning. See
// https://git.eeqj.de/sneak/vaultik/issues/476.
func TestFailedBlobUploadIsTriedAgain(t *testing.T) {
t.Parallel()
inner, err := storage.NewFileStorer(t.TempDir())
require.NoError(t, err)
puts := 0
store := faultstore.New(inner)
store.OnPut = func(string) faultstore.PutAction {
puts++
if puts <= 2 {
return faultstore.PutFail
}
return faultstore.PutNormal
}
data, warnings, err := uploadRetryTestBlob(context.Background(), store, 0)
require.NoError(t, err)
require.Equal(t, 3, puts)
require.Equal(t, 2, warnings)
rc, err := inner.Get(context.Background(), retryTestBlobKey)
require.NoError(t, err)
defer func() { _ = rc.Close() }()
stored, err := io.ReadAll(rc)
require.NoError(t, err)
require.Equal(t, data, stored)
}
// A blob upload that keeps failing is tried blobUploadRetries more times,
// BlobUploadRetryDelay apart, and then the backup stops with its error.
func TestBlobUploadStopsAfterRetries(t *testing.T) {
t.Parallel()
inner, err := storage.NewFileStorer(t.TempDir())
require.NoError(t, err)
puts := 0
store := faultstore.New(inner)
store.OnPut = func(string) faultstore.PutAction {
puts++
return faultstore.PutFail
}
const delay = 10 * time.Millisecond
start := time.Now()
_, warnings, err := uploadRetryTestBlob(context.Background(), store, delay)
require.ErrorIs(t, err, faultstore.ErrInjectedUpload)
require.Equal(t, 1+blobUploadRetries, puts)
require.Equal(t, blobUploadRetries, warnings)
require.GreaterOrEqual(t, time.Since(start), blobUploadRetries*delay)
}
// An upload stopped by Ctrl-C or SIGTERM is not tried again.
func TestInterruptedBlobUploadIsNotTriedAgain(t *testing.T) {
t.Parallel()
inner, err := storage.NewFileStorer(t.TempDir())
require.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
puts := 0
store := faultstore.New(inner)
store.OnPut = func(string) faultstore.PutAction {
puts++
cancel()
return faultstore.PutFail
}
_, warnings, err := uploadRetryTestBlob(ctx, store, 0)
require.Error(t, err)
require.Equal(t, 1, puts)
require.Zero(t, warnings)
}
// Backups wait BlobUploadRetryDelay before each new try.
func TestBackupScannerWaitsBetweenBlobUploadRetries(t *testing.T) {
t.Parallel()
cfg := &config.Config{
AgeRecipients: []string{testAgeRecipient},
ChunkSize: config.Size(1024 * 1024),
BlobSizeLimit: config.Size(4 * 1024 * 1024),
IndexPath: filepath.Join(t.TempDir(), "index.sqlite"),
}
scanner := provideScannerFactory(cfg, nil, nil)(ScannerParams{
Fs: afero.NewMemMapFs(),
})
require.NotNil(t, scanner)
require.Equal(t, BlobUploadRetryDelay, scanner.blobUploadRetryDelay)
}
+35
View File
@@ -4,8 +4,10 @@ package snapshot
import (
"os"
"path/filepath"
"syscall"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -38,3 +40,36 @@ func TestDeviceNumberFromStat(t *testing.T) {
// /dev is a different filesystem from the temp dir on Linux, macOS and in Docker.
assert.NotEqual(t, deviceOf(dir), deviceOf("/dev"))
}
// fileInfoWithStat is a file's info whose Sys returns stat.
type fileInfoWithStat struct {
os.FileInfo
stat *syscall.Stat_t
}
func (i fileInfoWithStat) Sys() any { return i.stat }
// The in-memory test filesystem has no device numbers. The pinned afero
// gives no stat; its unreleased master gives a stat holding only uid, gid
// and size. See https://git.eeqj.de/sneak/vaultik/issues/462.
func TestDeviceNumberFromStatInMemory(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, "/file.txt", []byte("data"), 0o600))
info, err := fs.Stat("/file.txt")
require.NoError(t, err)
_, ok := deviceNumberFromStat(info)
assert.False(t, ok, "device number read from a file info with no stat")
zeroed := fileInfoWithStat{
FileInfo: info,
stat: &syscall.Stat_t{Uid: 1000, Gid: 1000, Size: 4},
}
_, ok = deviceNumberFromStat(zeroed)
assert.False(t, ok, "device number read from a stat with no device or inode number")
}
+15 -14
View File
@@ -54,20 +54,21 @@ func provideScannerFactory(
}
return NewScanner(ScannerConfig{
FS: params.Fs,
ChunkSize: cfg.ChunkSize.Int64(),
Repositories: repos,
Storage: storer,
MaxBlobSize: cfg.BlobSizeLimit.Int64(),
CompressionLevel: cfg.CompressionLevel,
AgeRecipients: cfg.AgeRecipients,
EnableProgress: params.EnableProgress,
UI: params.UI,
Exclude: excludes,
IndexAndLockFiles: indexAndLockFiles(cfg.IndexPath, pidlock.Dir()),
SkipErrors: params.SkipErrors,
PreviousSnapshotID: params.PreviousSnapshotID,
DeviceNumber: params.DeviceNumber,
FS: params.Fs,
ChunkSize: cfg.ChunkSize.Int64(),
Repositories: repos,
Storage: storer,
MaxBlobSize: cfg.BlobSizeLimit.Int64(),
CompressionLevel: cfg.CompressionLevel,
AgeRecipients: cfg.AgeRecipients,
EnableProgress: params.EnableProgress,
UI: params.UI,
Exclude: excludes,
IndexAndLockFiles: indexAndLockFiles(cfg.IndexPath, pidlock.Dir()),
SkipErrors: params.SkipErrors,
PreviousSnapshotID: params.PreviousSnapshotID,
BlobUploadRetryDelay: BlobUploadRetryDelay,
DeviceNumber: params.DeviceNumber,
})
}
}
+88 -127
View File
@@ -71,13 +71,12 @@ type Scanner struct {
// indexAndLockFiles holds the paths of vaultik's own working files,
// which the scan treats as excluded (ScannerConfig.IndexAndLockFiles).
indexAndLockFiles map[string]struct{}
// listExcluded prints each excluded entry and why it is excluded
// (DryRun).
listExcluded bool
progress *ProgressReporter
progress *ProgressReporter
// skipErrors skips files that cannot be opened or read (logged loudly);
// packer, database, encryption, and upload errors still abort the run.
skipErrors bool
// blobUploadRetryDelay is ScannerConfig.BlobUploadRetryDelay.
blobUploadRetryDelay time.Duration
// ui is the user-facing output; never nil (defaults to a discarding writer).
ui *ui.Writer
// deviceNumber returns the number of the device that holds a file;
@@ -126,8 +125,16 @@ const (
// snapshotAssocBatchSize is how many file IDs are associated with the
// snapshot per transaction.
snapshotAssocBatchSize = 1000
// blobUploadRetries is how many more times a failed blob upload is
// tried before the backup stops with the error.
blobUploadRetries = 5
)
// BlobUploadRetryDelay is the wait before each new try of a failed blob
// upload.
const BlobUploadRetryDelay = 3 * time.Second
// ScannerConfig contains configuration for the scanner
type ScannerConfig struct {
FS afero.Fs
@@ -151,6 +158,10 @@ type ScannerConfig struct {
// PreviousSnapshotID is the most recent completed snapshot of the
// same name, "" for none.
PreviousSnapshotID string
// BlobUploadRetryDelay is the wait before each new try of a failed
// blob upload. Backups use snapshot.BlobUploadRetryDelay; tests leave
// it zero to try again at once.
BlobUploadRetryDelay time.Duration
// DeviceNumber returns the number of the device that holds a file,
// and false when there is none to read. nil reads it from the file's
// stat; tests replace it, since the in-memory test filesystem has none.
@@ -193,11 +204,6 @@ type ScanResult struct {
// itself when it is a directory. It is zero for an empty directory,
// such as the mount point of a volume that is not mounted.
EntriesFound int
// EntriesExcluded counts the entries the walk found that an exclude
// pattern matches or that are vaultik's own files. An excluded
// directory counts once: the walk does not enter it.
EntriesExcluded int
}
// NewScanner creates a new scanner instance
@@ -253,23 +259,24 @@ func NewScanner(cfg ScannerConfig) *Scanner {
}
return &Scanner{
fs: cfg.FS,
chunker: chunker.NewChunker(cfg.ChunkSize),
packer: packer,
repos: cfg.Repositories,
storage: cfg.Storage,
maxBlobSize: cfg.MaxBlobSize,
compressionLevel: cfg.CompressionLevel,
ageRecipient: strings.Join(cfg.AgeRecipients, ","),
exclude: cfg.Exclude,
compiledExclude: compiledExclude,
indexAndLockFiles: indexAndLockFiles,
progress: progress,
skipErrors: cfg.SkipErrors,
previousSnapshotID: cfg.PreviousSnapshotID,
ui: uiw,
deviceNumber: deviceNumber,
pendingChunkHashes: make(map[string]struct{}),
fs: cfg.FS,
chunker: chunker.NewChunker(cfg.ChunkSize),
packer: packer,
repos: cfg.Repositories,
storage: cfg.Storage,
maxBlobSize: cfg.MaxBlobSize,
compressionLevel: cfg.CompressionLevel,
ageRecipient: strings.Join(cfg.AgeRecipients, ","),
exclude: cfg.Exclude,
compiledExclude: compiledExclude,
indexAndLockFiles: indexAndLockFiles,
progress: progress,
skipErrors: cfg.SkipErrors,
previousSnapshotID: cfg.PreviousSnapshotID,
blobUploadRetryDelay: cfg.BlobUploadRetryDelay,
ui: uiw,
deviceNumber: deviceNumber,
pendingChunkHashes: make(map[string]struct{}),
}
}
@@ -368,42 +375,6 @@ func (s *Scanner) Scan(
return result, nil
}
// DryRun walks path as Scan does and counts what a backup would store,
// without storing it. It reads the local index to tell unchanged files
// from new and changed ones, and writes nothing to the local index or
// the backup destination store. With listExcluded it prints each
// excluded entry and the exclude pattern that matched it.
func (s *Scanner) DryRun(
ctx context.Context, path string, listExcluded bool,
) (*ScanResult, error) {
s.currentSourcePath = path
s.listExcluded = listExcluded
result := &ScanResult{
StartTime: time.Now().UTC(),
}
knownFiles, err := s.loadDatabaseState(ctx, path)
if err != nil {
return nil, err
}
existingFiles := make(map[string]struct{})
_, err = s.scanPhase(ctx, path, result, existingFiles, knownFiles)
if err != nil {
return nil, fmt.Errorf("scan phase failed: %w", err)
}
err = s.detectDeletedFiles(ctx, path, existingFiles, result)
if err != nil {
return nil, fmt.Errorf("detecting deleted files: %w", err)
}
result.EndTime = time.Now().UTC()
return result, nil
}
// GetProgress returns the progress reporter for this scanner, or nil when
// progress is off. Scan neither starts nor stops it: the caller does,
// once for all the paths it scans, because a second Stop panics.
@@ -988,8 +959,12 @@ func (s *Scanner) scanPhase(
// or cannot be looked up is neither counted nor reported. For
// directories, skip the entire subtree. info is nil when the
// walk could not look the entry up.
if excluded, walkErr := s.skipExcluded(filePath, path, info, result); excluded {
return walkErr
if s.shouldExclude(filePath, path) {
if info != nil && info.IsDir() {
return filepath.SkipDir
}
return nil
}
// The source path is in no directory listing, so its absence is
@@ -1056,32 +1031,6 @@ func (s *Scanner) scanPhase(
}, nil
}
// skipExcluded reports whether the walk skips filePath, under rootPath,
// because it is excluded, and counts it if so. The error is what the walk
// function returns for it: filepath.SkipDir for a directory, so the walk
// does not enter it, and nil otherwise. info is nil when the walk could
// not look the entry up.
func (s *Scanner) skipExcluded(
filePath, rootPath string, info os.FileInfo, result *ScanResult,
) (bool, error) {
pattern, excluded := s.excludingPattern(filePath, rootPath)
if !excluded {
return false, nil
}
result.EntriesExcluded++
if s.listExcluded {
s.printExcluded(filePath, pattern)
}
if info != nil && info.IsDir() {
return true, filepath.SkipDir
}
return true, nil
}
// countNonDirectories returns how many of files are not directories.
func countNonDirectories(files map[string]*database.File) int64 {
var count int64
@@ -1504,11 +1453,12 @@ func (s *Scanner) deviceNumberAt(path string) (uint64, bool) {
}
// deviceNumberFromStat returns the number of the device that holds the
// file info describes, and false when info carries no stat, as from the
// in-memory test filesystem.
// file info describes, and false when info carries no stat from a real
// filesystem. The in-memory test filesystem gives either no stat or one
// whose device and inode numbers are both zero, which no real file has.
func deviceNumberFromStat(info os.FileInfo) (uint64, bool) {
stat, ok := info.Sys().(*syscall.Stat_t)
if !ok {
if !ok || (stat.Dev == 0 && stat.Ino == 0) {
return 0, false
}
@@ -1941,10 +1891,7 @@ func (s *Scanner) uploadBlobIfNeeded(
s.ui.Hex(finishedBlob.Hash), s.ui.Size(finishedBlob.Compressed),
s.ui.Path(destination))
progressCallback := s.makeUploadProgressCallback(ctx, finishedBlob, startTime)
err = s.storage.PutWithProgress(ctx, blobPath, blobWithReader.Reader,
finishedBlob.Compressed, progressCallback)
err = s.putBlob(ctx, blobPath, blobWithReader)
if err != nil {
// The record is at info level: the returned error ends the backup
// and is printed as its ERROR: line. Only an interrupt cancels
@@ -1988,6 +1935,44 @@ func (s *Scanner) uploadBlobIfNeeded(
return finishedBlob.Compressed, false, nil
}
// putBlob uploads the blob to blobPath. After a failed upload it waits
// s.blobUploadRetryDelay and uploads the whole blob again, up to
// blobUploadRetries times, and then returns the last error. An interrupt
// stops it at once.
func (s *Scanner) putBlob(
ctx context.Context, blobPath string, blobWithReader *blob.WithReader,
) error {
finishedBlob := blobWithReader.FinishedBlob
for retries := 0; ; retries++ {
_, err := blobWithReader.Reader.Seek(0, io.SeekStart)
if err != nil {
return fmt.Errorf("rewinding blob for upload: %w", err)
}
progressCallback := s.makeUploadProgressCallback(
ctx, finishedBlob, time.Now().UTC())
err = s.storage.PutWithProgress(ctx, blobPath, blobWithReader.Reader,
finishedBlob.Compressed, progressCallback)
if err == nil || ctx.Err() != nil || retries == blobUploadRetries {
return err
}
log.Info("Blob upload failed, trying again",
"hash", finishedBlob.Hash, "retry", retries+1, "error", err)
s.ui.Warningf("Upload of blob %s failed: %v. Trying again in %s (retry %d of %d).",
s.ui.Hex(finishedBlob.Hash), err,
s.ui.Duration(s.blobUploadRetryDelay), retries+1, blobUploadRetries)
select {
case <-ctx.Done():
return err
case <-time.After(s.blobUploadRetryDelay):
}
}
}
// makeUploadProgressCallback creates a progress callback for blob uploads.
// It updates the live progress reporter ~twice/sec for ETAs and prints a
// human-readable status line to s.output at most every 15 seconds.
@@ -2417,32 +2402,23 @@ func compileExcludePatterns(patterns []string) ([]compiledPattern, error) {
// filePath is the full path to the file
// rootPath is the root of the backup source directory
func (s *Scanner) shouldExclude(filePath, rootPath string) bool {
_, excluded := s.excludingPattern(filePath, rootPath)
return excluded
}
// excludingPattern reports whether shouldExclude excludes filePath and
// returns the exclude pattern, as configured, that matches it. The
// pattern is "" for one of vaultik's own working files.
func (s *Scanner) excludingPattern(filePath, rootPath string) (string, bool) {
if _, ok := s.indexAndLockFiles[filePath]; ok {
return "", true
return true
}
if len(s.compiledExclude) == 0 {
return "", false
return false
}
// Get the relative path from root
relPath, err := filepath.Rel(rootPath, filePath)
if err != nil {
return "", false
return false
}
// Never exclude the root directory itself
if relPath == "." {
return "", false
return false
}
// Normalize path separators
@@ -2451,26 +2427,11 @@ func (s *Scanner) excludingPattern(filePath, rootPath string) (string, bool) {
// Check each pattern
for _, cp := range s.compiledExclude {
if cp.matches(relPath) {
return cp.original, true
return true
}
}
return "", false
}
// printExcluded prints that the walk excluded filePath and why: pattern
// is the exclude pattern that matched it, "" for one of vaultik's own
// working files.
func (s *Scanner) printExcluded(filePath, pattern string) {
if pattern == "" {
s.ui.Infof("Excluded %s: vaultik's own index or lock file.",
s.ui.Path(filePath))
return
}
s.ui.Infof("Excluded %s: matches exclude pattern %q.",
s.ui.Path(filePath), pattern)
return false
}
// matches reports whether the compiled pattern matches the given
+23
View File
@@ -8,6 +8,7 @@ import (
"fmt"
"io"
"os"
"path/filepath"
"strings"
"time"
@@ -23,6 +24,15 @@ import (
// ErrRemoteNotFound is returned when an rclone remote is not configured.
var ErrRemoteNotFound = errors.New("rclone remote not found in config")
// ErrRcloneRelativeLocalPath is returned when the path on an rclone remote
// of type local is relative. rclone would resolve it against the working
// directory, so the destination would move with it.
var ErrRcloneRelativeLocalPath = errors.New(
"the path on an rclone remote of type local must be absolute")
// rcloneLocalBackend is the type of rclone's local disk backend.
const rcloneLocalBackend = "local"
// RcloneStorer implements Storer using rclone's filesystem abstraction.
// This allows vaultik to use any of rclone's 70+ supported storage providers.
type RcloneStorer struct {
@@ -53,6 +63,19 @@ func NewRcloneStorer(ctx context.Context, remote, path string) (*RcloneStorer, e
rclonePath += path
}
// A remote rclone cannot find is reported by fs.NewFs below. Only the
// backend of what fs.ParseRemote returns is needed here.
backend, _, _, _, err := fs.ParseRemote(rclonePath) //nolint:dogsled
if err == nil && backend.Name == rcloneLocalBackend && !filepath.IsAbs(path) {
example := path
if example == "" {
example = "path/to/backup"
}
return nil, fmt.Errorf("%w: use rclone://%s//%s",
ErrRcloneRelativeLocalPath, remote, example)
}
// On S3, Swift and some other remotes rclone keeps a time of its own in
// each object's metadata, which costs one request per object to read.
// rclone's use_server_modtime setting makes ModTime the time the
+73
View File
@@ -436,6 +436,79 @@ func TestNewRcloneStorerReadsRcloneConfigEnv(t *testing.T) {
}
}
// useRcloneConfig points rclone at a config file holding contents for the
// rest of the test.
func useRcloneConfig(t *testing.T, contents string) {
t.Helper()
previous := config.GetConfigPath()
t.Cleanup(func() { _ = config.SetConfigPath(previous) })
confFile := filepath.Join(t.TempDir(), "rclone.conf")
err := os.WriteFile(confFile, []byte(contents), 0o600)
if err != nil {
t.Fatalf("writing rclone config: %v", err)
}
t.Setenv("RCLONE_CONFIG", confFile)
}
// TestNewRcloneStorerRefusesRelativeLocalPath checks that a relative path
// on a remote of type local is refused, with the absolute form in the
// error, and that nothing is created under the working directory. rclone
// would otherwise resolve the path against the working directory.
//
//nolint:paralleltest // changes the working directory and the rclone config
func TestNewRcloneStorerRefusesRelativeLocalPath(t *testing.T) {
useRcloneConfig(t, "[localfs]\ntype = local\n")
workdir := t.TempDir()
t.Chdir(workdir)
for path, wantInError := range map[string]string{
"srv/backup": "rclone://localfs//srv/backup",
"": "rclone://localfs//path/to/backup",
} {
_, err := storage.NewRcloneStorer(context.Background(), "localfs", path)
if !errors.Is(err, storage.ErrRcloneRelativeLocalPath) {
t.Errorf("path %q: error = %v, want ErrRcloneRelativeLocalPath",
path, err)
continue
}
if !strings.Contains(err.Error(), wantInError) {
t.Errorf("path %q: error %q does not show %q",
path, err, wantInError)
}
}
entries, err := os.ReadDir(workdir)
if err != nil {
t.Fatalf("reading working directory: %v", err)
}
if len(entries) != 0 {
t.Errorf("working directory holds %d entries, want none", len(entries))
}
}
// TestNewRcloneStorerKeepsRelativePathOnOtherRemotes checks that a relative
// path is accepted on a remote not of type local, where rclone resolves it
// on the remote itself, as sftp does against the login directory.
//
//nolint:paralleltest // changes the process-global rclone config
func TestNewRcloneStorerKeepsRelativePathOnOtherRemotes(t *testing.T) {
useRcloneConfig(t, "[mem]\ntype = memory\n")
_, err := storage.NewRcloneStorer(context.Background(), "mem", "backups")
if err != nil {
t.Errorf("NewRcloneStorer: %v", err)
}
}
// TestRcloneListStreamOnS3MakesNoRequestPerObject checks that listing an
// rclone S3 remote takes each object's time from the listing itself. By
// default rclone reads the time it keeps in each object's metadata, one
+21 -47
View File
@@ -33,7 +33,6 @@ var (
"creating a snapshot needs at least one public key in " +
"age_recipients (generate a keypair with: age-keygen)")
errKeepNewerThanNeedsPrune = errors.New("--keep-newer-than requires --prune")
errDryRunWithPrune = errors.New("--dry-run cannot be used with --prune")
)
// ErrPostBackupPruneFailed wraps a failure of the post-backup prune,
@@ -78,24 +77,12 @@ type SnapshotCreateOptions struct {
KeepNewerThan string
SkipErrors bool // Skip file read errors (log them loudly but continue)
Snapshots []string // Optional list of snapshot names to process (empty = all)
// DryRun scans the snapshot source paths and prints what a backup
// would store, writing nothing to the local index or the backup
// destination store.
DryRun bool
// ListExcluded makes a dry run print each excluded entry and the
// exclude pattern that matched it. A backup ignores it.
ListExcluded bool
}
// Validate fails when DryRun is set with Prune, or when KeepNewerThan is
// set without Prune or is not a valid duration. Only the post-backup
// prune reads KeepNewerThan, after the backup, so the caller checks it
// before starting one.
// Validate fails when KeepNewerThan is set without Prune or is not a
// valid duration. Only the post-backup prune reads KeepNewerThan, after
// the backup, so the caller checks it before starting one.
func (opts *SnapshotCreateOptions) Validate() error {
if opts.DryRun && opts.Prune {
return errDryRunWithPrune
}
if opts.KeepNewerThan == "" {
return nil
}
@@ -120,10 +107,6 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
return errNoAgeRecipients
}
if opts.DryRun {
return v.dryRunSnapshots(opts)
}
overallStartTime := time.Now()
log.Info("Starting snapshot creation",
@@ -375,9 +358,6 @@ type snapshotStats struct {
// could not be read or listed.
totalFilesUnreadable int
// The entries the scan excluded; an excluded directory counts once.
totalEntriesExcluded int
// The configured paths in which the scan found nothing.
emptyPaths []string
@@ -391,29 +371,6 @@ type snapshotStats struct {
blobUncompressedSize int64
}
// add adds the result of scanning the configured path dir to the totals.
func (stats *snapshotStats) add(dir string, result *snapshot.ScanResult) {
stats.totalFiles += result.FilesScanned
stats.totalBytes += result.BytesScanned
stats.totalChunks += result.ChunksCreated
stats.newChunkSize += result.NewChunkSize
stats.totalBlobs += result.BlobsCreated
stats.totalFilesSkipped += result.FilesSkipped
stats.totalFilesFailed += result.FilesFailed
stats.totalFilesUnreadable += result.FilesUnreadable
stats.totalBytesSkipped += result.BytesSkipped
stats.totalFilesDeleted += result.FilesDeleted
stats.totalBytesDeleted += result.BytesDeleted
stats.totalBlobsUploaded += result.BlobsUploaded
stats.totalBytesUploaded += result.BytesUploaded
stats.uploadDuration += result.UploadDuration
stats.totalEntriesExcluded += result.EntriesExcluded
if result.EntriesFound == 0 {
stats.emptyPaths = append(stats.emptyPaths, dir)
}
}
// createNamedSnapshot creates a single named snapshot and returns the
// statistics of its scan.
func (v *Vaultik) createNamedSnapshot(
@@ -607,7 +564,24 @@ func (v *Vaultik) scanAllDirectories(
return nil, fmt.Errorf("failed to scan %s: %w", dir, err)
}
stats.add(dir, result)
stats.totalFiles += result.FilesScanned
stats.totalBytes += result.BytesScanned
stats.totalChunks += result.ChunksCreated
stats.newChunkSize += result.NewChunkSize
stats.totalBlobs += result.BlobsCreated
stats.totalFilesSkipped += result.FilesSkipped
stats.totalFilesFailed += result.FilesFailed
stats.totalFilesUnreadable += result.FilesUnreadable
stats.totalBytesSkipped += result.BytesSkipped
stats.totalFilesDeleted += result.FilesDeleted
stats.totalBytesDeleted += result.BytesDeleted
stats.totalBlobsUploaded += result.BlobsUploaded
stats.totalBytesUploaded += result.BytesUploaded
stats.uploadDuration += result.UploadDuration
if result.EntriesFound == 0 {
stats.emptyPaths = append(stats.emptyPaths, dir)
}
log.Info("Directory scan complete",
"path", dir,
-133
View File
@@ -1,133 +0,0 @@
package vaultik
import (
"fmt"
"sneak.berlin/go/vaultik/internal/snapshot"
)
// dryRunSnapshots scans the source paths of each snapshot named in
// opts, or of every configured one, and prints what a backup would
// store. It writes nothing to the local index or the backup destination
// store. It first makes the read-only checks a backup makes, so it fails
// where a backup would: a local index bound to a different destination,
// or a destination that answers it has no marker while the local index
// holds snapshots stored there.
func (v *Vaultik) dryRunSnapshots(opts *SnapshotCreateOptions) error {
err := v.checkStorageBinding()
if err != nil {
return err
}
err = v.checkDestinationMarker()
if err != nil {
return err
}
// A backup first drops incomplete snapshots and the file records only
// they list (PruneDatabase), so it backs up those files again; the
// dry run counts them as unchanged.
incomplete, err := v.Repositories.Snapshots.GetIncompleteSnapshots(v.ctx)
if err != nil {
return fmt.Errorf("getting incomplete snapshots: %w", err)
}
if len(incomplete) > 0 {
v.UI.Noticef("The local index holds %s incomplete snapshot(s) from "+
"an interrupted backup. The next backup drops them first and may "+
"back up more files than this dry run counts.",
v.UI.Count(len(incomplete)))
}
snapshotNames, err := v.snapshotNamesToCreate(opts.Snapshots)
if err != nil {
return err
}
for _, snapName := range snapshotNames {
err = v.dryRunSnapshot(opts, snapName)
if err != nil {
return err
}
}
return nil
}
// dryRunSnapshot scans the source paths of the snapshot named snapName
// and prints what a backup of it would store.
func (v *Vaultik) dryRunSnapshot(
opts *SnapshotCreateOptions, snapName string,
) error {
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
if err != nil {
return err
}
previousSnapshotID, err := v.previousSnapshotID(snapName)
if err != nil {
return err
}
scanner := v.ScannerFactory(snapshot.ScannerParams{
UI: v.UI,
Fs: v.Fs,
Exclude: v.Config.GetExcludes(snapName),
SkipErrors: opts.SkipErrors,
PreviousSnapshotID: previousSnapshotID,
})
v.UI.Beginf("Dry run of snapshot %s.", snapName)
stats := &snapshotStats{}
for i, dir := range resolvedDirs {
v.UI.Beginf("Enumerating snapshot source files in %s (%d of %d).",
v.UI.Path(dir), i+1, len(resolvedDirs))
result, err := scanner.DryRun(v.ctx, dir, opts.ListExcluded)
if err != nil {
return fmt.Errorf("failed to scan %s: %w", dir, err)
}
stats.add(dir, result)
}
v.printDryRunSummary(snapName, stats)
return nil
}
// printDryRunSummary prints what a backup of the snapshot named snapName
// would store, in the terms of printSnapshotSummary.
func (v *Vaultik) printDryRunSummary(snapName string, stats *snapshotStats) {
filesToBackUp := stats.totalFiles - stats.totalFilesSkipped -
stats.totalFilesFailed
v.UI.Completef("Dry run of snapshot %s complete; nothing was stored.",
snapName)
filesMsg := fmt.Sprintf("Files: %s examined, %s to back up, %s unchanged",
v.UI.Count(stats.totalFiles),
v.UI.Count(filesToBackUp),
v.UI.Count(stats.totalFilesSkipped))
if stats.totalFilesFailed > 0 {
filesMsg += fmt.Sprintf(", %s failed", v.UI.Count(stats.totalFilesFailed))
}
if stats.totalFilesDeleted > 0 {
filesMsg += fmt.Sprintf(", %s deleted", v.UI.Count(stats.totalFilesDeleted))
}
v.UI.Detailf("%s.", filesMsg)
dataMsg := fmt.Sprintf("Data: %s total (%s to back up)",
v.UI.Size(stats.totalBytes+stats.totalBytesSkipped),
v.UI.Size(stats.totalBytes))
if stats.totalBytesDeleted > 0 {
dataMsg += fmt.Sprintf(", %s deleted", v.UI.Size(stats.totalBytesDeleted))
}
v.UI.Detailf("%s.", dataMsg)
v.UI.Detailf("Excluded: %s entries.", v.UI.Count(stats.totalEntriesExcluded))
}
-69
View File
@@ -1,69 +0,0 @@
package vaultik_test
import (
"context"
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// dryRunSnapshotName is the newSummaryEnv snapshot the dry run test
// backs up and then dry-runs.
const dryRunSnapshotName = "first"
// TestDryRunCountsWhatABackupWouldStore runs a dry run after a backup:
// the unchanged files count as unchanged, a rewritten one as to back up,
// and the dry run records no snapshot. See
// https://git.eeqj.de/sneak/vaultik/issues/436.
func TestDryRunCountsWhatABackupWouldStore(t *testing.T) {
t.Parallel()
env := newSummaryEnv(t)
env.backUp(t, dryRunSnapshotName, false)
dryRun := func() string {
env.out.Reset()
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
DryRun: true,
Snapshots: []string{dryRunSnapshotName},
}))
return env.out.String()
}
assert.Contains(t, dryRun(),
"Files: 3 examined, 0 to back up, 3 unchanged.")
require.NoError(t, os.WriteFile(env.aPath, []byte("rewritten"), 0o600))
assert.Contains(t, dryRun(),
"Files: 3 examined, 1 to back up, 2 unchanged.")
// localSnapshotID fails unless exactly one snapshot has the name.
localSnapshotID(context.Background(), t, env.repos, dryRunSnapshotName)
}
// TestDryRunNamesIncompleteSnapshots checks that a dry run says so when
// the local index holds an incomplete snapshot, which the next backup
// drops before it scans.
func TestDryRunNamesIncompleteSnapshots(t *testing.T) {
t.Parallel()
env := newSummaryEnv(t)
env.backUp(t, dryRunSnapshotName, false)
_, err := env.v.SnapshotManager.CreateSnapshotWithName(
context.Background(), testHostname, "split", "v", "g")
require.NoError(t, err)
env.out.Reset()
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
DryRun: true,
Snapshots: []string{dryRunSnapshotName},
}))
assert.Contains(t, env.out.String(),
"The local index holds 1 incomplete snapshot(s)")
}
@@ -4,6 +4,7 @@ import (
"context"
"maps"
"path/filepath"
"strings"
"testing"
"github.com/spf13/afero"
@@ -12,6 +13,7 @@ import (
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
"sneak.berlin/go/vaultik/internal/vaultik"
)
@@ -68,3 +70,55 @@ func TestBackupWithoutCronOfTwoPathSnapshotRestoresBothPaths(t *testing.T) {
assertRestoredTree(t, fs, restoreDir, files)
}
// A backup without --cron whose blob uploads keep failing stops with the
// upload's error, and stops the progress reporter once. See
// https://git.eeqj.de/sneak/vaultik/issues/476.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupWithoutCronStopsWithBlobUploadError(t *testing.T) {
log.Initialize(log.Config{})
const snapshotName = "data"
fs := afero.NewOsFs()
tempDir := t.TempDir()
firstDir := filepath.Join(tempDir, "first")
secondDir := filepath.Join(tempDir, "second")
dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background()
writeFaultSourceTree(t, fs, firstDir)
writeFaultSourceTree(t, fs, secondDir)
cfg := faultTestConfig()
cfg.IndexPath = dbPath
cfg.ChunkSize = config.Size(faultChunkSize)
cfg.Snapshots = map[string]config.SnapshotConfig{
snapshotName: {Paths: []string{firstDir, secondDir}},
}
inner, err := storage.NewFileStorer(t.TempDir())
require.NoError(t, err)
store := faultstore.New(inner)
store.OnPut = func(key string) faultstore.PutAction {
if strings.HasPrefix(key, "blobs/") {
return faultstore.PutFail
}
return faultstore.PutNormal
}
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
v := newBackupVaultik(ctx, cfg, store, database.NewRepositories(db), db, fs)
err = v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Snapshots: []string{snapshotName},
})
require.ErrorIs(t, err, faultstore.ErrInjectedUpload)
}