Compare commits

..
Author SHA1 Message Date
sneak bf614c58ac Document where vaultik reads the rclone config (closes #469)
check / check (push) Waiting to run
The rclone section now says vaultik reads rclone's own config file,
which needs no rclone tool once it exists, and that RCLONE_CONFIG names
another file. RCLONE_CONFIG joins the environment-variable list, and
restoring on another machine says the new host needs that file.

Model: opus-5-5
2026-11-19 08:39:28 +00:00
6 changed files with 23 additions and 311 deletions

No files matched your search

+16 -9
View File
@@ -127,6 +127,11 @@ vaultik config set s3.secret_access_key "..."
vaultik snapshot list vaultik snapshot list
``` ```
For an rclone destination, the storage credentials are in rclone's config file
instead. Copy that file to the new host, or recreate it with `rclone config`,
and set `storage_url` to the original `rclone://remote/path`. See
[storage backends](#storage-backends).
`snapshot list` reads the destination store without the private key. A snapshot `snapshot list` reads the destination store without the private key. A snapshot
that is not in this host's (empty) local index is shown as remote-only: its row that is not in this host's (empty) local index is shown as remote-only: its row
is identified by `<remote only:...>` rather than by a `hostname_name_timestamp` is identified by `<remote only:...>` rather than by a `hostname_name_timestamp`
@@ -281,6 +286,8 @@ work as written.
the key is not typed into your shell history. the key is not typed into your shell history.
- `VAULTIK_CONFIG`: Path to config file (overridden by `--config`) - `VAULTIK_CONFIG`: Path to config file (overridden by `--config`)
- `VAULTIK_INDEX_PATH`: Override local SQLite index path - `VAULTIK_INDEX_PATH`: Override local SQLite index path
- `RCLONE_CONFIG`: Path to the rclone config file used by an `rclone://`
destination (default `~/.config/rclone/rclone.conf`)
- `VAULTIK_CPUPROFILE`: Write a CPU profile to this path for the duration of the - `VAULTIK_CPUPROFILE`: Write a CPU profile to this path for the duration of the
run (development/debugging) run (development/debugging)
- `VAULTIK_MEMPROFILE`: Write a heap profile to this path when the run exits - `VAULTIK_MEMPROFILE`: Write a heap profile to this path when the run exits
@@ -602,11 +609,15 @@ only `vaultik-destination.txt` stops such a backup, and only while the local
index holds snapshots. index holds snapshots.
**Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud **Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud
providers. Requires rclone to be configured separately (`rclone config`). The providers. `remote` is the name of a remote in rclone's config file, which holds
path after the remote name is relative: rclone resolves it on the remote, for that remote's credentials. vaultik reads the same file the `rclone` tool reads,
example against the login directory on sftp. A second slash makes it absolute, by default `~/.config/rclone/rclone.conf`. Create it with `rclone config`, or
as in `rclone://remote//srv/backup`. A remote of type `local` needs an absolute copy it from another machine; vaultik does not need the `rclone` tool once the
path; a relative one is refused. file exists. The `RCLONE_CONFIG` environment variable names a different file.
The path after the remote name is relative: rclone resolves it on the remote,
for example against the login directory on sftp. A second slash makes it
absolute, as in `rclone://remote//srv/backup`. A remote of type `local` needs an
absolute path; a relative one is refused.
An upload cut off part-way leaves nothing under the object's name on S3, which An upload cut off part-way leaves nothing under the object's name on S3, which
shows an object only once its upload has completed, and on the local filesystem shows an object only once its upload has completed, and on the local filesystem
@@ -618,10 +629,6 @@ being written, a killed upload can leave a truncated object under its name,
which a later backup takes for complete. A leftover `.partial` file is ignored which a later backup takes for complete. A leftover `.partial` file is ignored
and can be deleted. and can be deleted.
A blob upload that fails, such as one cut off by a lost connection, is tried
again up to five times, three seconds apart. Each new try is announced with a
warning. If the last try fails too, the backup stops with the upload's error.
A destination shared between hosts or users, each with its own local index, A destination shared between hosts or users, each with its own local index,
relies on `prune_grace_period` to keep one's prune from deleting the blobs of relies on `prune_grace_period` to keep one's prune from deleting the blobs of
another's backup still running, or its directory under `metadata/` that has no another's backup still running, or its directory under `metadata/` that has no
-154
View File
@@ -1,154 +0,0 @@
//nolint:testpackage // exercises the unexported uploadBlobIfNeeded
package snapshot
import (
"bytes"
"context"
"io"
"path/filepath"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blob"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
"sneak.berlin/go/vaultik/internal/ui"
)
// retryTestBlobSize is larger than what a failing faultstore Put reads,
// so a new try that did not rewind the blob would store a short object.
const retryTestBlobSize = 4096
// retryTestBlobKey is where the retry tests upload their blob.
const retryTestBlobKey = "blobs/retry"
// uploadRetryTestBlob uploads a retryTestBlobSize blob through store,
// waiting delay between tries, and returns the blob's bytes, the warnings
// the upload printed and its error.
func uploadRetryTestBlob(
ctx context.Context, store storage.Storer, delay time.Duration,
) ([]byte, int, error) {
data := bytes.Repeat([]byte("0123456789abcdef"), retryTestBlobSize/16)
output := ui.NewWithColor(io.Discard, false)
s := &Scanner{storage: store, ui: output, blobUploadRetryDelay: delay}
_, _, err := s.uploadBlobIfNeeded(ctx, retryTestBlobKey, &blob.WithReader{
FinishedBlob: &blob.FinishedBlob{
Hash: "retry",
Compressed: int64(len(data)),
},
Reader: bytes.NewReader(data),
}, time.Now(), &ScanResult{})
return data, output.WarningCount(), err
}
// A blob upload that fails is tried again from the start of the blob,
// and each failure is a warning. See
// https://git.eeqj.de/sneak/vaultik/issues/476.
func TestFailedBlobUploadIsTriedAgain(t *testing.T) {
t.Parallel()
inner, err := storage.NewFileStorer(t.TempDir())
require.NoError(t, err)
puts := 0
store := faultstore.New(inner)
store.OnPut = func(string) faultstore.PutAction {
puts++
if puts <= 2 {
return faultstore.PutFail
}
return faultstore.PutNormal
}
data, warnings, err := uploadRetryTestBlob(context.Background(), store, 0)
require.NoError(t, err)
require.Equal(t, 3, puts)
require.Equal(t, 2, warnings)
rc, err := inner.Get(context.Background(), retryTestBlobKey)
require.NoError(t, err)
defer func() { _ = rc.Close() }()
stored, err := io.ReadAll(rc)
require.NoError(t, err)
require.Equal(t, data, stored)
}
// A blob upload that keeps failing is tried blobUploadRetries more times,
// BlobUploadRetryDelay apart, and then the backup stops with its error.
func TestBlobUploadStopsAfterRetries(t *testing.T) {
t.Parallel()
inner, err := storage.NewFileStorer(t.TempDir())
require.NoError(t, err)
puts := 0
store := faultstore.New(inner)
store.OnPut = func(string) faultstore.PutAction {
puts++
return faultstore.PutFail
}
const delay = 10 * time.Millisecond
start := time.Now()
_, warnings, err := uploadRetryTestBlob(context.Background(), store, delay)
require.ErrorIs(t, err, faultstore.ErrInjectedUpload)
require.Equal(t, 1+blobUploadRetries, puts)
require.Equal(t, blobUploadRetries, warnings)
require.GreaterOrEqual(t, time.Since(start), blobUploadRetries*delay)
}
// An upload stopped by Ctrl-C or SIGTERM is not tried again.
func TestInterruptedBlobUploadIsNotTriedAgain(t *testing.T) {
t.Parallel()
inner, err := storage.NewFileStorer(t.TempDir())
require.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
puts := 0
store := faultstore.New(inner)
store.OnPut = func(string) faultstore.PutAction {
puts++
cancel()
return faultstore.PutFail
}
_, warnings, err := uploadRetryTestBlob(ctx, store, 0)
require.Error(t, err)
require.Equal(t, 1, puts)
require.Zero(t, warnings)
}
// Backups wait BlobUploadRetryDelay before each new try.
func TestBackupScannerWaitsBetweenBlobUploadRetries(t *testing.T) {
t.Parallel()
cfg := &config.Config{
AgeRecipients: []string{testAgeRecipient},
ChunkSize: config.Size(1024 * 1024),
BlobSizeLimit: config.Size(4 * 1024 * 1024),
IndexPath: filepath.Join(t.TempDir(), "index.sqlite"),
}
scanner := provideScannerFactory(cfg, nil, nil)(ScannerParams{
Fs: afero.NewMemMapFs(),
})
require.NotNil(t, scanner)
require.Equal(t, BlobUploadRetryDelay, scanner.blobUploadRetryDelay)
}
-35
View File
@@ -4,10 +4,8 @@ package snapshot
import ( import (
"os" "os"
"path/filepath" "path/filepath"
"syscall"
"testing" "testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@@ -40,36 +38,3 @@ func TestDeviceNumberFromStat(t *testing.T) {
// /dev is a different filesystem from the temp dir on Linux, macOS and in Docker. // /dev is a different filesystem from the temp dir on Linux, macOS and in Docker.
assert.NotEqual(t, deviceOf(dir), deviceOf("/dev")) assert.NotEqual(t, deviceOf(dir), deviceOf("/dev"))
} }
// fileInfoWithStat is a file's info whose Sys returns stat.
type fileInfoWithStat struct {
os.FileInfo
stat *syscall.Stat_t
}
func (i fileInfoWithStat) Sys() any { return i.stat }
// The in-memory test filesystem has no device numbers. The pinned afero
// gives no stat; its unreleased master gives a stat holding only uid, gid
// and size. See https://git.eeqj.de/sneak/vaultik/issues/462.
func TestDeviceNumberFromStatInMemory(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, "/file.txt", []byte("data"), 0o600))
info, err := fs.Stat("/file.txt")
require.NoError(t, err)
_, ok := deviceNumberFromStat(info)
assert.False(t, ok, "device number read from a file info with no stat")
zeroed := fileInfoWithStat{
FileInfo: info,
stat: &syscall.Stat_t{Uid: 1000, Gid: 1000, Size: 4},
}
_, ok = deviceNumberFromStat(zeroed)
assert.False(t, ok, "device number read from a stat with no device or inode number")
}
-1
View File
@@ -67,7 +67,6 @@ func provideScannerFactory(
IndexAndLockFiles: indexAndLockFiles(cfg.IndexPath, pidlock.Dir()), IndexAndLockFiles: indexAndLockFiles(cfg.IndexPath, pidlock.Dir()),
SkipErrors: params.SkipErrors, SkipErrors: params.SkipErrors,
PreviousSnapshotID: params.PreviousSnapshotID, PreviousSnapshotID: params.PreviousSnapshotID,
BlobUploadRetryDelay: BlobUploadRetryDelay,
DeviceNumber: params.DeviceNumber, DeviceNumber: params.DeviceNumber,
}) })
} }
+7 -58
View File
@@ -75,8 +75,6 @@ type Scanner struct {
// skipErrors skips files that cannot be opened or read (logged loudly); // skipErrors skips files that cannot be opened or read (logged loudly);
// packer, database, encryption, and upload errors still abort the run. // packer, database, encryption, and upload errors still abort the run.
skipErrors bool skipErrors bool
// blobUploadRetryDelay is ScannerConfig.BlobUploadRetryDelay.
blobUploadRetryDelay time.Duration
// ui is the user-facing output; never nil (defaults to a discarding writer). // ui is the user-facing output; never nil (defaults to a discarding writer).
ui *ui.Writer ui *ui.Writer
// deviceNumber returns the number of the device that holds a file; // deviceNumber returns the number of the device that holds a file;
@@ -125,16 +123,8 @@ const (
// snapshotAssocBatchSize is how many file IDs are associated with the // snapshotAssocBatchSize is how many file IDs are associated with the
// snapshot per transaction. // snapshot per transaction.
snapshotAssocBatchSize = 1000 snapshotAssocBatchSize = 1000
// blobUploadRetries is how many more times a failed blob upload is
// tried before the backup stops with the error.
blobUploadRetries = 5
) )
// BlobUploadRetryDelay is the wait before each new try of a failed blob
// upload.
const BlobUploadRetryDelay = 3 * time.Second
// ScannerConfig contains configuration for the scanner // ScannerConfig contains configuration for the scanner
type ScannerConfig struct { type ScannerConfig struct {
FS afero.Fs FS afero.Fs
@@ -158,10 +148,6 @@ type ScannerConfig struct {
// PreviousSnapshotID is the most recent completed snapshot of the // PreviousSnapshotID is the most recent completed snapshot of the
// same name, "" for none. // same name, "" for none.
PreviousSnapshotID string PreviousSnapshotID string
// BlobUploadRetryDelay is the wait before each new try of a failed
// blob upload. Backups use snapshot.BlobUploadRetryDelay; tests leave
// it zero to try again at once.
BlobUploadRetryDelay time.Duration
// DeviceNumber returns the number of the device that holds a file, // DeviceNumber returns the number of the device that holds a file,
// and false when there is none to read. nil reads it from the file's // and false when there is none to read. nil reads it from the file's
// stat; tests replace it, since the in-memory test filesystem has none. // stat; tests replace it, since the in-memory test filesystem has none.
@@ -273,7 +259,6 @@ func NewScanner(cfg ScannerConfig) *Scanner {
progress: progress, progress: progress,
skipErrors: cfg.SkipErrors, skipErrors: cfg.SkipErrors,
previousSnapshotID: cfg.PreviousSnapshotID, previousSnapshotID: cfg.PreviousSnapshotID,
blobUploadRetryDelay: cfg.BlobUploadRetryDelay,
ui: uiw, ui: uiw,
deviceNumber: deviceNumber, deviceNumber: deviceNumber,
pendingChunkHashes: make(map[string]struct{}), pendingChunkHashes: make(map[string]struct{}),
@@ -1453,12 +1438,11 @@ func (s *Scanner) deviceNumberAt(path string) (uint64, bool) {
} }
// deviceNumberFromStat returns the number of the device that holds the // deviceNumberFromStat returns the number of the device that holds the
// file info describes, and false when info carries no stat from a real // file info describes, and false when info carries no stat, as from the
// filesystem. The in-memory test filesystem gives either no stat or one // in-memory test filesystem.
// whose device and inode numbers are both zero, which no real file has.
func deviceNumberFromStat(info os.FileInfo) (uint64, bool) { func deviceNumberFromStat(info os.FileInfo) (uint64, bool) {
stat, ok := info.Sys().(*syscall.Stat_t) stat, ok := info.Sys().(*syscall.Stat_t)
if !ok || (stat.Dev == 0 && stat.Ino == 0) { if !ok {
return 0, false return 0, false
} }
@@ -1891,7 +1875,10 @@ func (s *Scanner) uploadBlobIfNeeded(
s.ui.Hex(finishedBlob.Hash), s.ui.Size(finishedBlob.Compressed), s.ui.Hex(finishedBlob.Hash), s.ui.Size(finishedBlob.Compressed),
s.ui.Path(destination)) s.ui.Path(destination))
err = s.putBlob(ctx, blobPath, blobWithReader) progressCallback := s.makeUploadProgressCallback(ctx, finishedBlob, startTime)
err = s.storage.PutWithProgress(ctx, blobPath, blobWithReader.Reader,
finishedBlob.Compressed, progressCallback)
if err != nil { if err != nil {
// The record is at info level: the returned error ends the backup // The record is at info level: the returned error ends the backup
// and is printed as its ERROR: line. Only an interrupt cancels // and is printed as its ERROR: line. Only an interrupt cancels
@@ -1935,44 +1922,6 @@ func (s *Scanner) uploadBlobIfNeeded(
return finishedBlob.Compressed, false, nil return finishedBlob.Compressed, false, nil
} }
// putBlob uploads the blob to blobPath. After a failed upload it waits
// s.blobUploadRetryDelay and uploads the whole blob again, up to
// blobUploadRetries times, and then returns the last error. An interrupt
// stops it at once.
func (s *Scanner) putBlob(
ctx context.Context, blobPath string, blobWithReader *blob.WithReader,
) error {
finishedBlob := blobWithReader.FinishedBlob
for retries := 0; ; retries++ {
_, err := blobWithReader.Reader.Seek(0, io.SeekStart)
if err != nil {
return fmt.Errorf("rewinding blob for upload: %w", err)
}
progressCallback := s.makeUploadProgressCallback(
ctx, finishedBlob, time.Now().UTC())
err = s.storage.PutWithProgress(ctx, blobPath, blobWithReader.Reader,
finishedBlob.Compressed, progressCallback)
if err == nil || ctx.Err() != nil || retries == blobUploadRetries {
return err
}
log.Info("Blob upload failed, trying again",
"hash", finishedBlob.Hash, "retry", retries+1, "error", err)
s.ui.Warningf("Upload of blob %s failed: %v. Trying again in %s (retry %d of %d).",
s.ui.Hex(finishedBlob.Hash), err,
s.ui.Duration(s.blobUploadRetryDelay), retries+1, blobUploadRetries)
select {
case <-ctx.Done():
return err
case <-time.After(s.blobUploadRetryDelay):
}
}
}
// makeUploadProgressCallback creates a progress callback for blob uploads. // makeUploadProgressCallback creates a progress callback for blob uploads.
// It updates the live progress reporter ~twice/sec for ETAs and prints a // It updates the live progress reporter ~twice/sec for ETAs and prints a
// human-readable status line to s.output at most every 15 seconds. // human-readable status line to s.output at most every 15 seconds.
@@ -4,7 +4,6 @@ import (
"context" "context"
"maps" "maps"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"github.com/spf13/afero" "github.com/spf13/afero"
@@ -13,7 +12,6 @@ import (
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage" "sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
"sneak.berlin/go/vaultik/internal/vaultik" "sneak.berlin/go/vaultik/internal/vaultik"
) )
@@ -70,55 +68,3 @@ func TestBackupWithoutCronOfTwoPathSnapshotRestoresBothPaths(t *testing.T) {
assertRestoredTree(t, fs, restoreDir, files) assertRestoredTree(t, fs, restoreDir, files)
} }
// A backup without --cron whose blob uploads keep failing stops with the
// upload's error, and stops the progress reporter once. See
// https://git.eeqj.de/sneak/vaultik/issues/476.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupWithoutCronStopsWithBlobUploadError(t *testing.T) {
log.Initialize(log.Config{})
const snapshotName = "data"
fs := afero.NewOsFs()
tempDir := t.TempDir()
firstDir := filepath.Join(tempDir, "first")
secondDir := filepath.Join(tempDir, "second")
dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background()
writeFaultSourceTree(t, fs, firstDir)
writeFaultSourceTree(t, fs, secondDir)
cfg := faultTestConfig()
cfg.IndexPath = dbPath
cfg.ChunkSize = config.Size(faultChunkSize)
cfg.Snapshots = map[string]config.SnapshotConfig{
snapshotName: {Paths: []string{firstDir, secondDir}},
}
inner, err := storage.NewFileStorer(t.TempDir())
require.NoError(t, err)
store := faultstore.New(inner)
store.OnPut = func(key string) faultstore.PutAction {
if strings.HasPrefix(key, "blobs/") {
return faultstore.PutFail
}
return faultstore.PutNormal
}
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
v := newBackupVaultik(ctx, cfg, store, database.NewRepositories(db), db, fs)
err = v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Snapshots: []string{snapshotName},
})
require.ErrorIs(t, err, faultstore.ErrInjectedUpload)
}