1 Commits
Author SHA1 Message Date
sneak e5b08b8638 Exit 3 when a backup or restore finished but skipped files (closes #439)
check / check (push) Waiting to run
A `snapshot create` or `snapshot restore` that --skip-errors made skip
files now exits 3, the status `restic backup` uses, so a cron job can
tell it from a run that made no backup (1). RunOperation keeps the
operation's error inside errReported so Entry can pick the status.

A run that skipped files and also found a configured path empty, missed
a mount point, or was asked for a path the snapshot lacks still exits 1:
its error wraps only that condition and names the skipped files in text.

The README gains an exit status section.

Model: opus-5-5
2026-11-19 13:09:34 +00:00
17 changed files with 305 additions and 197 deletions

No files matched your search

+36 -22
View File
@@ -197,9 +197,10 @@ vaultik --version
that cannot be restored when restoring, instead of aborting. Packing and
storage errors while creating a snapshot (which would leave a chunk recorded
but not stored) still abort the run. A run that skipped anything still
finishes but exits non-zero: `snapshot create` keeps the snapshot and counts
each file or directory it skipped as failed. A file removed during the backup
is also counted as failed but does not change the exit status.
finishes but exits `3` (see [exit status](#exit-status)): `snapshot create`
keeps the snapshot and counts each file or directory it skipped as failed. A
file removed during the backup is also counted as failed but does not change
the exit status.
### locking
@@ -272,6 +273,19 @@ Every `--json` command therefore pipes on its own, with no additional flag:
`vaultik snapshot list --json | jq .` and `vaultik prune --json | jq .` both
work as written.
### exit status
- `0`: the command succeeded.
- `1`: the command failed.
- `3`: `snapshot create` or `snapshot restore` finished, but `--skip-errors`
skipped files it could not read or restore. `restic backup` uses the same
status for this case.
- `130`: Ctrl-C, `SIGINT` or `SIGTERM` stopped the command.
A cron job can page on `1` and only log `3`. A run that skipped files and also
failed in another way exits `1`. For example, a restore that skipped files and
was asked for a path the snapshot does not hold exits `1`.
### environment variables
- `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for
@@ -345,25 +359,25 @@ In such a config, an unknown key or a value of the wrong type is refused.
delete from the destination store what `prune` deletes there. By default keeps
only the latest snapshot per name; use `--keep-newer-than` for a rolling
window. When `--skip-errors` skipped a file or directory it could not read or
list, the prune does not run and every older snapshot is kept, since one of
them may be the only one that holds what was skipped; the command says so and
exits non-zero. When the backup found nothing in a configured path, as in the
mount point of a volume that is not mounted, the prune does not run either:
every older snapshot is kept, and the command names the path and exits
non-zero, on every run until the path holds something again. Take a path that
is empty on purpose out of the config, or remove the older snapshots by hand
with `snapshot purge`. When an earlier snapshot of the same name in the local
index found a volume mounted at or below a configured path (a directory on a
different filesystem from its parent, with an entry in it that the backup does
not exclude) and this backup does not, the prune does not run either: every
older snapshot is kept, and the command names the directory and exits
non-zero, on every run until the volume is mounted again. A backup also does
not find the volume when it finds no such entry in its directory, as when an
automounted volume cannot be mounted, or when that directory is no longer
backed up, because it is excluded or its configured path was removed. If any
of this is on purpose, as for a retired volume, remove the older snapshots by
hand with `snapshot purge`. A file removed during the backup does not stop the
prune.
list, the prune does not run and every older snapshot is kept. One of them may
be the only one that holds what was skipped. The command says so and exits
`3`. The prune does not run either when the backup found nothing in a
configured path, as in the mount point of a volume that is not mounted. Every
older snapshot is kept, and the command names the path and exits `1`, on every
run until the path holds something again. Take a path that is empty on purpose
out of the config, or remove the older snapshots by hand with
`snapshot purge`. The prune does not run either when this backup does not find
a volume that an earlier snapshot of the same name in the local index found.
The volume's mount point is a directory at or below a configured path. That
directory is on a different filesystem from its parent and holds an entry the
backup does not exclude. Every older snapshot is kept, and the command names
the directory and exits `1`, on every run until the volume is mounted again. A
backup also does not find the volume when it finds no such entry in its
directory, as when an automounted volume cannot be mounted. Nor does it find
the volume when that directory is no longer backed up, because it is excluded
or its configured path was removed. If any of this is on purpose, as for a
retired volume, remove the older snapshots by hand with `snapshot purge`. A
file removed during the backup does not stop the prune.
- `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than this
duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). Without
`--prune`, or with a value that is not a valid duration, the command fails
+6 -5
View File
@@ -198,7 +198,8 @@ func RunApp(ctx context.Context, app *fx.App) error {
// (or, under `snapshot verify --json`, put in its document). Entry
// turns it into a non-zero exit status without printing anything
// further, so the error line is not doubled. It flows up from
// RunOperation through cobra to Entry.
// RunOperation through cobra to Entry, wrapping the operation's error
// so that Entry can choose the exit status from it.
var errReported = errors.New("operation failed")
// errInterrupted marks an operation that SIGINT or SIGTERM stopped
@@ -233,7 +234,7 @@ func RunOperation(
var (
mu sync.Mutex
finished bool // op returned before any interrupt cancelled it
failed bool // op finished with an error
opErr error // what op returned, when it finished
)
opts.Invokes = append(opts.Invokes,
@@ -258,7 +259,7 @@ func RunOperation(
mu.Lock()
finished = true
failed = err != nil
opErr = err
mu.Unlock()
}
@@ -303,8 +304,8 @@ func RunOperation(
switch {
case !finished:
return errInterrupted
case failed:
return errReported
case opErr != nil:
return fmt.Errorf("%w: %w", errReported, opErr)
default:
return nil
}
-7
View File
@@ -282,13 +282,6 @@ func TestConfigSetKeepsFileThatWouldNotLoad(t *testing.T) {
storageURLKey, testS3URL,
"s3.secret_access_key is required for s3:// URLs; " +
"set s3.access_key_id and s3.secret_access_key first"},
// rclone:// URLs that no command could open.
{configWithoutRecipients, storageURLKey, "rclone:///backups",
"rclone URL missing remote name"},
{configWithoutRecipients, storageURLKey, "rclone://dav,user=x/backups",
"rclone URL must not carry parameters in the remote name"},
{configWithoutRecipients, storageURLKey, "rclone://dav/backups?x=1",
`unknown query parameter in storage URL: "x"`},
// With its reference filled in, the config loads.
{levelReference, "blob_size_limit", "1MB",
"blob_size_limit must be at least"},
+16 -2
View File
@@ -9,6 +9,7 @@ import (
"sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// shortCommitLen is the number of git commit hash characters shown in
@@ -20,6 +21,13 @@ const shortCommitLen = 12
// shell reports for a command stopped by Ctrl-C.
const exitCodeInterrupted = 130
// exitCodeSkippedFiles is the exit status of a `snapshot create` or
// `snapshot restore` that finished but skipped files under
// --skip-errors. It is the status `restic backup` uses for a snapshot
// that could not read some files, so a cron job can tell such a run
// from one that made no backup.
const exitCodeSkippedFiles = 3
// Entry is the main entry point for the CLI application.
// It first calls globals.FillUnsetFromBuildInfo, which fills in the
// version, commit and commit date that the linker flags did not set.
@@ -30,8 +38,9 @@ const exitCodeInterrupted = 130
// The banner goes to stderr because stdout carries only the output the
// user asked for, such as a completion script or a `config get` value.
//
// It returns the process exit code (0 on success, 130 when interrupted,
// 1 on any other error) rather than calling os.Exit, so that main's
// It returns the process exit code (0 on success, 3 when files were
// skipped, 130 when interrupted, 1 on any other error) rather than
// calling os.Exit, so that main's
// deferred profile writers run before the process ends. See run in
// cmd/vaultik/main.go.
func Entry() int {
@@ -58,6 +67,11 @@ func Entry() int {
return exitCodeInterrupted
}
if errors.Is(err, vaultik.ErrFilesUnreadable) ||
errors.Is(err, vaultik.ErrFilesFailedRestore) {
return exitCodeSkippedFiles
}
return 1
}
@@ -0,0 +1,83 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const flagSkipErrors = "--skip-errors"
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
// under --skip-errors, a source directory that does not exist, which the
// walk skips. The command exits with exitCodeSkippedFiles and reports
// the backup as completed, not as a failed snapshot creation. See
// https://git.eeqj.de/sneak/vaultik/issues/310 and
// https://git.eeqj.de/sneak/vaultik/issues/439.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
// writeBackupConfig would create the source directory.
configPath := writeHermeticPruneConfig(t, false)
code, stdout, stderr := runEntry(t,
flagConfig, configPath, flagSkipErrors, cmdSnapshot, cmdCreate)
assert.Equal(t, exitCodeSkippedFiles, code)
assert.Contains(t, stdout,
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
assert.NotContains(t, stdout, "Finished successfully.")
assert.Contains(t, stderr,
"Backup completed, but 1 file(s) could not be read")
assert.NotContains(t, stderr, "Snapshot creation failed")
}
// TestEntrySnapshotRestoreSkipErrorsExitStatus backs up, deletes the
// blobs from the destination, and restores under --skip-errors, so every
// file is skipped. A restore that only skipped files exits with
// exitCodeSkippedFiles. One that was also asked for a path the snapshot
// does not hold exits 1. See
// https://git.eeqj.de/sneak/vaultik/issues/439.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotRestoreSkipErrorsExitStatus(t *testing.T) {
for _, testCase := range []struct {
name string
missingPaths []string
want int
}{
{name: "skipped files only", want: exitCodeSkippedFiles},
{
name: "and a path that matches nothing",
missingPaths: []string{"/no/such/path"},
want: 1,
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath, storeDir := writeBackupConfigWithPrivateKey(t)
code, _, _ := runEntry(t,
flagConfig, configPath, cmdSnapshot, cmdCreate)
require.Equal(t, 0, code)
require.NoError(t, os.RemoveAll(filepath.Join(storeDir, blobsDir)))
// writeBackupConfig puts the source directory beside the
// config file.
sourceDir := filepath.Join(filepath.Dir(configPath), "source")
args := append([]string{
flagConfig, configPath, flagSkipErrors, cmdSnapshot,
cmdRestore, onlySnapshotID(t, configPath), t.TempDir(),
sourceDir,
}, testCase.missingPaths...)
code, _, stderr := runEntry(t, args...)
assert.Equal(t, testCase.want, code)
assert.Contains(t, stderr, "1 file(s) failed to restore")
})
}
}
@@ -1,30 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"testing"
"github.com/stretchr/testify/assert"
)
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
// under --skip-errors, a source directory that does not exist, which the
// walk skips. The command exits non-zero and reports the backup as
// completed, not as a failed snapshot creation. See
// https://git.eeqj.de/sneak/vaultik/issues/310.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
// writeBackupConfig would create the source directory.
configPath := writeHermeticPruneConfig(t, false)
code, stdout, stderr := runEntry(t,
flagConfig, configPath, "--skip-errors", cmdSnapshot, cmdCreate)
assert.Equal(t, 1, code)
assert.Contains(t, stdout,
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
assert.NotContains(t, stdout, "Finished successfully.")
assert.Contains(t, stderr,
"Backup completed, but 1 file(s) could not be read")
assert.NotContains(t, stderr, "Snapshot creation failed")
}
+24 -11
View File
@@ -21,7 +21,6 @@ import (
"gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/chunker"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storageurl"
)
const appName = "vaultik"
@@ -66,6 +65,11 @@ var (
"emit (chunk_size times the FastCDC size spread)")
errBadCompression = errors.New("compression_level must be between 1 and 19")
errBadS3PartSize = errors.New("s3.part_size must be between 5MiB and 5GiB")
errBadStorageScheme = errors.New(
"storage_url must start with s3://, file://, or rclone://")
errFileURLNotAbsolute = errors.New(
"a file:// storage_url needs an absolute path, " +
"as in file:///path/to/backup")
errIndexPathNotAbsolute = errors.New(
"index_path and VAULTIK_INDEX_PATH need an absolute path, " +
"as in ~/.local/share/vaultik/index.sqlite")
@@ -554,20 +558,26 @@ func (c *Config) validateStorage() error {
return nil
}
// validateStorageURL checks storage_url with the same parser that opens
// the destination, so a value every command would refuse fails at load.
// File and rclone URLs need no credentials; S3 URLs require the legacy
// s3.* credential fields.
// validateStorageURL validates URL-based storage configuration. File and
// rclone URLs need no credentials; S3 URLs require the legacy s3.*
// credential fields. A file URL must name an absolute path: a relative one
// would resolve against the working directory, so the destination would
// move with it.
func (c *Config) validateStorageURL() error {
parsed, err := storageurl.Parse(c.StorageURL)
if err != nil {
return fmt.Errorf("invalid storage_url: %w", err)
switch {
case strings.HasPrefix(c.StorageURL, "file://"):
// storage.ParseStorageURL applies the same rule, but config
// cannot import storage, which imports config.
if !strings.HasPrefix(c.StorageURL, "file:///") {
return errFileURLNotAbsolute
}
if parsed.Scheme != storageurl.SchemeS3 {
return nil
}
case strings.HasPrefix(c.StorageURL, "rclone://"):
// Rclone storage uses rclone's own config
return nil
case strings.HasPrefix(c.StorageURL, "s3://"):
// S3 storage needs credentials
if c.S3.AccessKeyID == "" {
return fmt.Errorf("%w for s3:// URLs", ErrS3KeyIDRequired)
}
@@ -577,6 +587,9 @@ func (c *Config) validateStorageURL() error {
}
return nil
default:
return errBadStorageScheme
}
}
// Module exports the config module for fx dependency injection.
+2 -3
View File
@@ -10,7 +10,6 @@ import (
"sneak.berlin/go/vaultik/internal/chunker"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storageurl"
)
const (
@@ -503,8 +502,8 @@ func TestLoadRejectsRelativeFileURL(t *testing.T) {
}
_, err = Load(path)
if !errors.Is(err, storageurl.ErrRelativeFilePath) {
t.Fatalf("Load() error = %v, want storageurl.ErrRelativeFilePath", err)
if !errors.Is(err, errFileURLNotAbsolute) {
t.Fatalf("Load() error = %v, want errFileURLNotAbsolute", err)
}
if !strings.Contains(err.Error(), "absolute path") {
+1 -2
View File
@@ -9,7 +9,6 @@ import (
"strings"
"github.com/spf13/afero"
"sneak.berlin/go/vaultik/internal/storageurl"
)
// FileStorer implements Storer using the local filesystem.
@@ -302,7 +301,7 @@ func (f *FileStorer) ListPartialUploads(
// Info returns human-readable storage location information.
func (f *FileStorer) Info() Info {
return Info{
Type: storageurl.SchemeFile,
Type: schemeFile,
Location: f.basePath,
}
}
+5 -11
View File
@@ -2,20 +2,14 @@ package storage
import (
"context"
"errors"
"fmt"
"strings"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/s3"
"sneak.berlin/go/vaultik/internal/storageurl"
)
// ErrUnsupportedStorage is returned for a parsed storage URL whose scheme
// has no storage backend.
var ErrUnsupportedStorage = errors.New("unsupported storage scheme")
// defaultS3Region is used when neither the URL nor the config specify one.
const defaultS3Region = "us-east-1"
@@ -46,19 +40,19 @@ func NewStorer(cfg *config.Config) (Storer, error) {
//nolint:ireturn // factory intentionally returns the Storer interface
func storerFromURL(rawURL string, cfg *config.Config) (Storer, error) {
parsed, err := storageurl.Parse(rawURL)
parsed, err := ParseStorageURL(rawURL)
if err != nil {
return nil, fmt.Errorf("parsing storage URL: %w", err)
}
switch parsed.Scheme {
case storageurl.SchemeFile:
case schemeFile:
return NewFileStorer(parsed.Prefix)
case storageurl.SchemeS3:
case schemeS3:
return storerFromParsedS3URL(parsed, cfg)
case storageurl.SchemeRclone:
case schemeRclone:
return NewRcloneStorer(
context.Background(), parsed.RcloneRemote, parsed.Prefix)
@@ -71,7 +65,7 @@ func storerFromURL(rawURL string, cfg *config.Config) (Storer, error) {
// filling endpoint protocol and region defaults from the config.
//
//nolint:ireturn // factory intentionally returns the Storer interface
func storerFromParsedS3URL(parsed *storageurl.URL, cfg *config.Config) (Storer, error) {
func storerFromParsedS3URL(parsed *URL, cfg *config.Config) (Storer, error) {
// Build endpoint URL
endpoint := parsed.Endpoint
if endpoint == "" {
+1 -2
View File
@@ -16,7 +16,6 @@ import (
"github.com/rclone/rclone/fs/config"
"github.com/rclone/rclone/fs/config/configfile"
"github.com/rclone/rclone/fs/operations"
"sneak.berlin/go/vaultik/internal/storageurl"
// Import all rclone backends
_ "github.com/rclone/rclone/backend/all"
@@ -289,7 +288,7 @@ func (r *RcloneStorer) Info() Info {
}
return Info{
Type: storageurl.SchemeRclone,
Type: schemeRclone,
Location: location,
}
}
@@ -1,7 +1,4 @@
// Package storageurl parses the storage_url config value. The config
// package uses it to refuse a bad value when the config loads; the storage
// package uses it to open the destination.
package storageurl
package storage
import (
"errors"
@@ -13,9 +10,9 @@ import (
// Storage URL scheme names.
const (
SchemeFile = "file"
SchemeS3 = "s3"
SchemeRclone = "rclone"
schemeFile = "file"
schemeS3 = "s3"
schemeRclone = "rclone"
)
// Sentinel errors for storage URL parsing.
@@ -28,6 +25,7 @@ var (
ErrMissingRemote = errors.New("rclone URL missing remote name")
ErrUnsupportedScheme = errors.New(
"unsupported URL scheme: must start with s3://, file://, or rclone://")
ErrUnsupportedStorage = errors.New("unsupported storage scheme")
ErrURLCredentials = errors.New(
"storage URL must not carry credentials; " +
"set s3.access_key_id and s3.secret_access_key in the config instead")
@@ -51,12 +49,12 @@ type URL struct {
RcloneRemote string // rclone remote name (for rclone:// URLs)
}
// Parse parses a storage URL string.
// ParseStorageURL parses a storage URL string.
// Supported formats:
// - s3://bucket/prefix?endpoint=host&region=us-east-1&ssl=true
// - file:///absolute/path/to/backup
// - rclone://remote/path/to/backups
func Parse(rawURL string) (*URL, error) {
func ParseStorageURL(rawURL string) (*URL, error) {
if rawURL == "" {
return nil, ErrEmptyStorageURL
}
@@ -94,7 +92,7 @@ func parseFileURL(rawPath string) (*URL, error) {
return nil, fmt.Errorf("invalid file URL path: %w", err)
}
return &URL{Scheme: SchemeFile, Prefix: path}, nil
return &URL{Scheme: schemeFile, Prefix: path}, nil
}
// parseS3URL parses an s3://bucket/prefix URL. It rejects credentials in
@@ -137,7 +135,7 @@ func parseS3URL(rawURL string) (*URL, error) {
}
return &URL{
Scheme: SchemeS3,
Scheme: schemeS3,
Bucket: bucket,
Prefix: strings.TrimPrefix(u.Path, "/"),
Endpoint: query.Get("endpoint"),
@@ -191,7 +189,7 @@ func parseRcloneURL(rawURL string) (*URL, error) {
}
return &URL{
Scheme: SchemeRclone,
Scheme: schemeRclone,
Prefix: strings.TrimPrefix(u.Path, "/"),
RcloneRemote: remote,
}, nil
@@ -228,9 +226,9 @@ func wrapParseError(err error) error {
// String returns a human-readable representation of the storage URL.
func (u *URL) String() string {
switch u.Scheme {
case SchemeFile:
case schemeFile:
return "file://" + u.Prefix
case SchemeS3:
case schemeS3:
endpoint := u.Endpoint
if endpoint == "" {
endpoint = "s3.amazonaws.com"
@@ -241,7 +239,7 @@ func (u *URL) String() string {
}
return fmt.Sprintf("s3://%s (endpoint: %s)", u.Bucket, endpoint)
case SchemeRclone:
case schemeRclone:
if u.Prefix != "" {
return fmt.Sprintf("rclone://%s/%s", u.RcloneRemote, u.Prefix)
}
@@ -1,4 +1,4 @@
package storageurl_test
package storage_test
import (
"errors"
@@ -6,12 +6,12 @@ import (
"strings"
"testing"
"sneak.berlin/go/vaultik/internal/storageurl"
"sneak.berlin/go/vaultik/internal/storage"
)
// TestParseValid checks that each supported scheme parses into the
// expected fields, since those fields decide which backend is built.
func TestParseValid(t *testing.T) {
// TestParseStorageURLValid checks that each supported scheme parses into
// the expected fields, since those fields decide which backend is built.
func TestParseStorageURLValid(t *testing.T) {
t.Parallel()
const (
@@ -22,27 +22,27 @@ func TestParseValid(t *testing.T) {
cases := []struct {
name string
raw string
want *storageurl.URL
want *storage.URL
}{
{
name: "file absolute path",
raw: "file:///var/backups/vaultik",
want: &storageurl.URL{Scheme: fileScheme, Prefix: "/var/backups/vaultik"},
want: &storage.URL{Scheme: fileScheme, Prefix: "/var/backups/vaultik"},
},
{
name: "file percent-escaped path is decoded",
raw: "file:///Volumes/my%20stick/backup",
want: &storageurl.URL{Scheme: fileScheme, Prefix: "/Volumes/my stick/backup"},
want: &storage.URL{Scheme: fileScheme, Prefix: "/Volumes/my stick/backup"},
},
{
name: "file path with a literal space",
raw: "file:///Volumes/my stick/backup",
want: &storageurl.URL{Scheme: fileScheme, Prefix: "/Volumes/my stick/backup"},
want: &storage.URL{Scheme: fileScheme, Prefix: "/Volumes/my stick/backup"},
},
{
name: "s3 bucket and prefix, ssl defaults on",
raw: "s3://mybucket/backups/host",
want: &storageurl.URL{
want: &storage.URL{
Scheme: "s3", Bucket: bucket,
Prefix: "backups/host", UseSSL: true,
},
@@ -50,12 +50,12 @@ func TestParseValid(t *testing.T) {
{
name: "s3 bucket only",
raw: "s3://mybucket",
want: &storageurl.URL{Scheme: "s3", Bucket: bucket, UseSSL: true},
want: &storage.URL{Scheme: "s3", Bucket: bucket, UseSSL: true},
},
{
name: "s3 with endpoint, region, ssl off",
raw: "s3://mybucket?endpoint=minio.example.com&region=us-west-2&ssl=false",
want: &storageurl.URL{
want: &storage.URL{
Scheme: "s3", Bucket: bucket,
Endpoint: "minio.example.com", Region: "us-west-2", UseSSL: false,
},
@@ -63,14 +63,14 @@ func TestParseValid(t *testing.T) {
{
name: "rclone remote and path",
raw: "rclone://gdrive/backups/host",
want: &storageurl.URL{
want: &storage.URL{
Scheme: "rclone", RcloneRemote: "gdrive", Prefix: "backups/host",
},
},
{
name: "rclone remote only",
raw: "rclone://gdrive",
want: &storageurl.URL{Scheme: "rclone", RcloneRemote: "gdrive"},
want: &storage.URL{Scheme: "rclone", RcloneRemote: "gdrive"},
},
}
@@ -78,22 +78,22 @@ func TestParseValid(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got, err := storageurl.Parse(tc.raw)
got, err := storage.ParseStorageURL(tc.raw)
if err != nil {
t.Fatalf("Parse(%q) returned error: %v", tc.raw, err)
t.Fatalf("ParseStorageURL(%q) returned error: %v", tc.raw, err)
}
if !reflect.DeepEqual(got, tc.want) {
t.Errorf("Parse(%q) = %+v, want %+v", tc.raw, got, tc.want)
t.Errorf("ParseStorageURL(%q) = %+v, want %+v", tc.raw, got, tc.want)
}
})
}
}
// TestParseErrors checks that empty, missing, and unknown-scheme inputs
// fail with the documented sentinel errors instead of parsing to a wrong
// destination.
func TestParseErrors(t *testing.T) {
// TestParseStorageURLErrors checks that empty, missing, and unknown-scheme
// inputs fail with the documented sentinel errors instead of parsing to a
// wrong destination.
func TestParseStorageURLErrors(t *testing.T) {
t.Parallel()
cases := []struct {
@@ -101,33 +101,34 @@ func TestParseErrors(t *testing.T) {
raw string
wantErr error
}{
{"empty url", "", storageurl.ErrEmptyStorageURL},
{"file empty path", "file://", storageurl.ErrEmptyFilePath},
{"file relative path", "file://dest", storageurl.ErrRelativeFilePath},
{"s3 missing bucket", "s3://", storageurl.ErrMissingBucket},
{"s3 missing bucket with path", "s3:///justprefix", storageurl.ErrMissingBucket},
{"rclone missing remote", "rclone://", storageurl.ErrMissingRemote},
{"unknown scheme", "gs://bucket/x", storageurl.ErrUnsupportedScheme},
{"no scheme", "/local/path", storageurl.ErrUnsupportedScheme},
{"empty url", "", storage.ErrEmptyStorageURL},
{"file empty path", "file://", storage.ErrEmptyFilePath},
{"file relative path", "file://dest", storage.ErrRelativeFilePath},
{"s3 missing bucket", "s3://", storage.ErrMissingBucket},
{"s3 missing bucket with path", "s3:///justprefix", storage.ErrMissingBucket},
{"rclone missing remote", "rclone://", storage.ErrMissingRemote},
{"unknown scheme", "gs://bucket/x", storage.ErrUnsupportedScheme},
{"no scheme", "/local/path", storage.ErrUnsupportedScheme},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := storageurl.Parse(tc.raw)
_, err := storage.ParseStorageURL(tc.raw)
if !errors.Is(err, tc.wantErr) {
t.Errorf("Parse(%q) error = %v, want %v",
t.Errorf("ParseStorageURL(%q) error = %v, want %v",
tc.raw, err, tc.wantErr)
}
})
}
}
// TestParseRejectsCredentials checks that a URL carrying credentials in
// its userinfo or in an unknown query parameter is rejected, and that the
// error never echoes the secret-bearing URL back into logs or output.
func TestParseRejectsCredentials(t *testing.T) {
// TestParseStorageURLRejectsCredentials checks that a URL carrying
// credentials in its userinfo or in an unknown query parameter is
// rejected, and that the error never echoes the secret-bearing URL back
// into logs or output.
func TestParseStorageURLRejectsCredentials(t *testing.T) {
t.Parallel()
// Split so the literals never form a "user:pass@" URL pattern that
@@ -146,44 +147,44 @@ func TestParseRejectsCredentials(t *testing.T) {
{
name: "s3 userinfo",
raw: "s3://" + key + ":" + secret + "@mybucket/prefix",
wantErr: storageurl.ErrURLCredentials,
wantErr: storage.ErrURLCredentials,
secrets: []string{key, secret, "mybucket"},
},
{
name: "s3 userinfo in endpoint",
raw: "s3://mybucket?endpoint=https://" +
key + ":" + secret + "@minio.example.com",
wantErr: storageurl.ErrURLCredentials,
wantErr: storage.ErrURLCredentials,
secrets: []string{key, secret},
},
{
name: "s3 userinfo in endpoint without scheme",
raw: "s3://mybucket?endpoint=" + key + ":" + secret + "@minio.example.com",
wantErr: storageurl.ErrURLCredentials,
wantErr: storage.ErrURLCredentials,
secrets: []string{key, secret},
},
{
name: "s3 unknown query param",
raw: "s3://mybucket?access_key=" + key + "&secret=" + secret,
wantErr: storageurl.ErrURLUnknownParam,
wantErr: storage.ErrURLUnknownParam,
secrets: []string{key, secret},
},
{
name: "s3 misspelt endpoint",
raw: "s3://mybucket?endpiont=minio.example.com",
wantErr: storageurl.ErrURLUnknownParam,
wantErr: storage.ErrURLUnknownParam,
secrets: nil,
},
{
name: "rclone userinfo",
raw: "rclone://user:" + secret + "@gdrive/backups",
wantErr: storageurl.ErrURLCredentials,
wantErr: storage.ErrURLCredentials,
secrets: []string{secret},
},
{
name: "rclone query param",
raw: "rclone://gdrive/backups?token=" + secret,
wantErr: storageurl.ErrURLUnknownParam,
wantErr: storage.ErrURLUnknownParam,
secrets: []string{secret},
},
}
@@ -192,9 +193,9 @@ func TestParseRejectsCredentials(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := storageurl.Parse(tc.raw)
_, err := storage.ParseStorageURL(tc.raw)
if !errors.Is(err, tc.wantErr) {
t.Fatalf("Parse(%q) error = %v, want %v",
t.Fatalf("ParseStorageURL(%q) error = %v, want %v",
tc.raw, err, tc.wantErr)
}
@@ -215,11 +216,11 @@ func TestParseRejectsCredentials(t *testing.T) {
}
}
// TestParseRejectsRcloneRemoteParams checks that an rclone remote name
// carrying parameters (rclone's "remote,param=value" form, which can hold
// a credential) is rejected without echoing the value, including the URLs
// that url.Parse itself refuses.
func TestParseRejectsRcloneRemoteParams(t *testing.T) {
// TestParseStorageURLRejectsRcloneRemoteParams checks that an rclone
// remote name carrying parameters (rclone's "remote,param=value" form,
// which can hold a credential) is rejected without echoing the value,
// including the URLs that url.Parse itself refuses.
func TestParseStorageURLRejectsRcloneRemoteParams(t *testing.T) {
t.Parallel()
cases := []struct {
@@ -248,10 +249,10 @@ func TestParseRejectsRcloneRemoteParams(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := storageurl.Parse(tc.raw)
if !errors.Is(err, storageurl.ErrRcloneRemoteParams) {
t.Fatalf("Parse(%q) error = %v, want %v",
tc.raw, err, storageurl.ErrRcloneRemoteParams)
_, err := storage.ParseStorageURL(tc.raw)
if !errors.Is(err, storage.ErrRcloneRemoteParams) {
t.Fatalf("ParseStorageURL(%q) error = %v, want %v",
tc.raw, err, storage.ErrRcloneRemoteParams)
}
for _, secret := range tc.secrets {
@@ -263,10 +264,10 @@ func TestParseRejectsRcloneRemoteParams(t *testing.T) {
}
}
// TestParseRejectsUndecodableQuery checks that an s3 or rclone URL whose
// query cannot be decoded is rejected, not read with that part of the
// query left out, and that the error does not echo it.
func TestParseRejectsUndecodableQuery(t *testing.T) {
// TestParseStorageURLRejectsUndecodableQuery checks that an s3 or rclone
// URL whose query cannot be decoded is rejected, not read with that part
// of the query left out, and that the error does not echo it.
func TestParseStorageURLRejectsUndecodableQuery(t *testing.T) {
t.Parallel()
const (
@@ -302,14 +303,14 @@ func TestParseRejectsUndecodableQuery(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := storageurl.Parse(tc.raw)
_, err := storage.ParseStorageURL(tc.raw)
if err == nil {
t.Fatalf("Parse(%q) returned no error", tc.raw)
t.Fatalf("ParseStorageURL(%q) returned no error", tc.raw)
}
if !errors.Is(err, storageurl.ErrURLBadQuery) {
t.Errorf("Parse(%q) error = %v, want %v",
tc.raw, err, storageurl.ErrURLBadQuery)
if !errors.Is(err, storage.ErrURLBadQuery) {
t.Errorf("ParseStorageURL(%q) error = %v, want %v",
tc.raw, err, storage.ErrURLBadQuery)
}
for _, s := range []string{key, secret, "minio.example.com"} {
@@ -321,17 +322,17 @@ func TestParseRejectsUndecodableQuery(t *testing.T) {
}
}
// TestParseFailureHidesURL checks that when url.Parse itself fails, the
// wrapped error carries only the inner cause, not the *url.Error whose
// text embeds the raw (possibly credential-bearing) URL.
func TestParseFailureHidesURL(t *testing.T) {
// TestParseStorageURLParseFailureHidesURL checks that when url.Parse
// itself fails, the wrapped error carries only the inner cause, not the
// *url.Error whose text embeds the raw (possibly credential-bearing) URL.
func TestParseStorageURLParseFailureHidesURL(t *testing.T) {
t.Parallel()
const raw = "s3://mybucket/%zz"
_, err := storageurl.Parse(raw)
_, err := storage.ParseStorageURL(raw)
if err == nil {
t.Fatalf("Parse(%q) returned no error", raw)
t.Fatalf("ParseStorageURL(%q) returned no error", raw)
}
if strings.Contains(err.Error(), "mybucket") {
+19 -8
View File
@@ -25,9 +25,13 @@ import (
"sneak.berlin/go/vaultik/internal/types"
)
// ErrFilesFailedRestore is returned when --skip-errors skipped files the
// restore could not write. Every other file was restored, so the caller
// can report it apart from a failed restore.
var ErrFilesFailedRestore = errors.New("file(s) failed to restore")
// Sentinel errors for restore failures.
var (
errFilesFailedRestore = errors.New("file(s) failed to restore")
errPathsNotInSnapshot = errors.New("requested path(s) not in the snapshot")
errFilesFailedVerify = errors.New("files failed verification")
errDecryptionKeyRequired = errors.New(
@@ -201,9 +205,9 @@ func (v *Vaultik) Restore(opts *RestoreOptions) error {
}
// finishRestore prints the restore summary and the post-restore
// warnings, runs optional verification, and converts any failed-file
// count into an error. Otherwise it returns unmatchedErr, the error for
// the requested paths that matched nothing (nil when there are none).
// warnings, runs optional verification, and returns the error the
// restore ends with: one for the files --skip-errors skipped, one for
// unmatchedErr, the requested paths that matched nothing, or nil.
func (v *Vaultik) finishRestore(
repos *database.Repositories,
files []*database.File,
@@ -250,12 +254,19 @@ func (v *Vaultik) finishRestore(
}
}
if result.FilesFailed > 0 {
return fmt.Errorf("%d %w", result.FilesFailed, errFilesFailedRestore)
}
switch {
case result.FilesFailed > 0 && unmatchedErr != nil:
// A requested path that matched nothing outweighs skipped files:
// the error wraps only unmatchedErr, so the command does not exit
// as a restore that only skipped files.
return fmt.Errorf("%w; %d %s",
unmatchedErr, result.FilesFailed, ErrFilesFailedRestore.Error())
case result.FilesFailed > 0:
return fmt.Errorf("%d %w", result.FilesFailed, ErrFilesFailedRestore)
default:
return unmatchedErr
}
}
// warnIfOwnershipNotPreserved warns once if some entry restored under
// targetDir has another owner or group on disk than the snapshot
+1 -1
View File
@@ -80,7 +80,7 @@ func TestRestoreFreesBlobsNoPendingFileNeeds(t *testing.T) {
TargetDir: restoreDir,
SkipErrors: true,
})
require.ErrorIs(t, err, errFilesFailedRestore)
require.ErrorIs(t, err, ErrFilesFailedRestore)
requireRestoredFiles(t, fs, restoreDir, originals)
assert.LessOrEqualf(t, peak, 1,
+20 -7
View File
@@ -49,8 +49,7 @@ var ErrFilesUnreadable = errors.New("file(s) could not be read")
// prune because the scan found nothing in a configured path of a
// snapshot, as in the mount point of a volume that is not mounted. The
// snapshots are complete and kept, so the caller can report it apart
// from a failed backup. A run that also skipped unreadable files
// returns ErrFilesUnreadable instead.
// from a failed backup.
var ErrSourcePathEmpty = errors.New("source path(s) were empty")
// ErrVolumeNotMounted is returned when --prune skipped the post-backup
@@ -59,8 +58,8 @@ var ErrSourcePathEmpty = errors.New("source path(s) were empty")
// mounted there is not mounted, the run found no entry in it, or the
// directory is no longer backed up. The snapshots are complete and
// kept, so the caller can report it apart from a failed backup. A run
// that also skipped unreadable files or found a configured path empty
// returns ErrFilesUnreadable or ErrSourcePathEmpty instead.
// that also found a configured path empty returns ErrSourcePathEmpty
// instead.
var ErrVolumeNotMounted = errors.New(
"mount point(s) an earlier snapshot recorded were not found")
@@ -243,15 +242,29 @@ func (v *Vaultik) skipPostBackupPrune(
if filesUnreadable > 0 {
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
"could not be read; older snapshots are kept.", filesUnreadable)
}
var err error
switch {
case len(emptyPaths) > 0:
err = fmt.Errorf("%d %w", len(emptyPaths), ErrSourcePathEmpty)
case len(unmountedPaths) > 0:
err = fmt.Errorf("%d %w", len(unmountedPaths), ErrVolumeNotMounted)
default:
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
}
if len(emptyPaths) > 0 {
return fmt.Errorf("%d %w", len(emptyPaths), ErrSourcePathEmpty)
// An empty path or a missing volume can mean a whole volume is not
// in the backup, which outweighs skipped files: the error wraps only
// the former, so the command does not exit as a backup that only
// skipped files.
if filesUnreadable > 0 {
return fmt.Errorf("%w; %d %s",
err, filesUnreadable, ErrFilesUnreadable.Error())
}
return fmt.Errorf("%d %w", len(unmountedPaths), ErrVolumeNotMounted)
return err
}
// finishCreateSnapshot ends a run whose snapshots are all complete and
+9 -3
View File
@@ -176,8 +176,11 @@ func TestBackupThroughSymlinkWhoseTargetIsGoneDoesNotPrune(t *testing.T) {
// A backup under --skip-errors --prune that both skipped a file it could
// not read and found nothing in a configured path names the empty path
// too when it skips the prune. See
// https://git.eeqj.de/sneak/vaultik/issues/388.
// too when it skips the prune. It ends with the empty path's error, which
// names the unreadable file too but does not wrap ErrFilesUnreadable, so
// the command does not exit as a backup that only skipped files. See
// https://git.eeqj.de/sneak/vaultik/issues/388 and
// https://git.eeqj.de/sneak/vaultik/issues/439.
func TestBackupWithEmptySourcePathAndUnreadableFileNamesBoth(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
@@ -207,7 +210,10 @@ func TestBackupWithEmptySourcePathAndUnreadableFileNamesBoth(t *testing.T) {
SkipErrors: true,
Snapshots: []string{snapshotName},
})
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
require.ErrorIs(t, err, vaultik.ErrSourcePathEmpty)
require.NotErrorIs(t, err, vaultik.ErrFilesUnreadable)
require.EqualError(t, err,
"1 source path(s) were empty; 1 file(s) could not be read")
assert.Contains(t, env.out.String(),
"Skipped the post-backup prune because 1 file(s) could not be read")
assert.Contains(t, env.out.String(),