Compare commits
16
Commits
ba343c30e2
...
next
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5ca0d46932 | ||
|
|
6a8bb2f389 | ||
|
|
5f1f8c4f66 | ||
|
|
e8ab13b526 | ||
|
|
f6df500ba2 | ||
|
|
7aae6e4ff3 | ||
|
|
d6eed26482 | ||
|
|
cdcbe8adfd | ||
|
|
1a478ef275 | ||
|
|
d8d13d1c8b | ||
|
|
028a166256 | ||
|
|
c07db21b45 | ||
|
|
1df1d23094 | ||
|
|
0d0c76fa24 | ||
|
|
88a2aa7953 | ||
|
|
098ed80990 |
No files matched your search
@@ -313,12 +313,10 @@ when the next path element is a number (so
|
|||||||
`vaultik config set age_recipients.0 age1...` works on a file without
|
`vaultik config set age_recipients.0 age1...` works on a file without
|
||||||
`age_recipients`), otherwise as a map. If the config loads, a change after which
|
`age_recipients`), otherwise as a map. If the config loads, a change after which
|
||||||
it would not, such as `vaultik config set compression_level 99`, is refused and
|
it would not, such as `vaultik config set compression_level 99`, is refused and
|
||||||
the file is left unchanged. A config that does not load, such as the one
|
the file is left unchanged. A config that does not load is checked one setting
|
||||||
`config init` writes before `storage_url` is set or one with a `${ENV:...}`
|
at a time. Examples are the file `config init` writes before `storage_url` is
|
||||||
reference to a variable not set in this shell, is checked one setting at a time:
|
set, and one with a `${ENV:...}` reference to a variable not set in this shell.
|
||||||
only a change after which the changed setting cannot be read, such as one giving
|
In such a config, an unknown key or a value of the wrong type is refused.
|
||||||
it the wrong type (a single value where a list belongs or a word where a number
|
|
||||||
belongs), is refused.
|
|
||||||
|
|
||||||
**`snapshot create`**: Perform incremental backup of configured snapshots.
|
**`snapshot create`**: Perform incremental backup of configured snapshots.
|
||||||
|
|
||||||
@@ -1041,20 +1039,37 @@ is `git describe --tags --always --dirty`, which `script/version` runs for the
|
|||||||
A `docker build .` of a clone, with no build arguments, runs the same
|
A `docker build .` of a clone, with no build arguments, runs the same
|
||||||
`git describe` (without `--dirty`) on the `.git` in its build context, so it
|
`git describe` (without `--dirty`) on the `.git` in its build context, so it
|
||||||
stamps the same value for a clean commit; the build fails if the context carries
|
stamps the same value for a clean commit; the build fails if the context carries
|
||||||
`.git` and no version, commit or commit date comes out. A binary built without
|
`.git` and no version, commit or commit date comes out. `make` outside a git
|
||||||
git metadata reports `dev`, or `unknown` when `script/docker` or
|
checkout and `docker build .` without `.git` both stamp `dev`. `script/docker`
|
||||||
`script/cibuild` built it outside a git checkout.
|
and `script/cibuild` stamp `unknown` when they run outside a git checkout.
|
||||||
|
|
||||||
|
A binary built by plain `go install` or `go build` carries no stamped version.
|
||||||
|
It reports the version Go recorded in it instead:
|
||||||
|
|
||||||
|
- a `go install` of the tag `v1.0.0` → `v1.0.0`.
|
||||||
|
- a `go build` of an untagged commit → a Go pseudo-version, such as
|
||||||
|
`v1.0.1-0.20261110231226-0123456789ab`.
|
||||||
|
- a build where Go recorded no version → `dev`.
|
||||||
|
|
||||||
`goreleaser` stamps a release binary with the tag minus its leading `v`, so the
|
`goreleaser` stamps a release binary with the tag minus its leading `v`, so the
|
||||||
tag `v1.0.0` produces `vaultik 1.0.0`, matching the archive name
|
tag `v1.0.0` produces `vaultik 1.0.0`, matching the archive name
|
||||||
`vaultik_1.0.0_linux_amd64.tar.gz`. `goreleaser --snapshot` stamps
|
`vaultik_1.0.0_linux_amd64.tar.gz`. `goreleaser --snapshot` stamps
|
||||||
`dev-<12 chars of the commit sha>` rather than inventing the next patch number.
|
`dev-<12 chars of the commit sha>` rather than inventing the next patch number.
|
||||||
`vaultik version` calls a build a development build when its version is `dev`,
|
|
||||||
`unknown`, `dev-<sha>`, the short commit sha or `<tag>-<N>-g<short sha>`, with
|
`vaultik version` calls a build a development build when its version is one of
|
||||||
or without `-dirty`; only a plain tag, such as `v1.0.0` or `1.0.0`, is a
|
these:
|
||||||
release. If `script/version` cannot be run at all, `make` stops with an error
|
|
||||||
instead of building an unversioned binary, and a binary that somehow carries an
|
- `dev` or `unknown`.
|
||||||
empty version string still reports itself as a development build.
|
- `dev-<sha>`.
|
||||||
|
- the short commit sha.
|
||||||
|
- `<tag>-<N>-g<short sha>`.
|
||||||
|
- a Go pseudo-version.
|
||||||
|
- any version with a `-dirty` or `+dirty` suffix.
|
||||||
|
|
||||||
|
Only a plain tag, such as `v1.0.0` or `1.0.0`, is a release. If `script/version`
|
||||||
|
cannot be run at all, `make` stops with an error instead of building an
|
||||||
|
unversioned binary. A binary stamped with an empty version reports the version
|
||||||
|
Go recorded in it; if Go recorded none, it is a development build.
|
||||||
|
|
||||||
### cutting a release
|
### cutting a release
|
||||||
|
|
||||||
|
|||||||
@@ -295,6 +295,7 @@ storage_url: "rclone://myremote/path/to/backups"
|
|||||||
|
|
||||||
# Path to local SQLite index database
|
# Path to local SQLite index database
|
||||||
# This database tracks file state for incremental backups
|
# This database tracks file state for incremental backups
|
||||||
|
# It must be an absolute path; ~ is expanded
|
||||||
# Default: the platform data directory, e.g.
|
# Default: the platform data directory, e.g.
|
||||||
# macOS: ~/Library/Application Support/vaultik/index.sqlite
|
# macOS: ~/Library/Application Support/vaultik/index.sqlite
|
||||||
# Linux: ~/.local/share/vaultik/index.sqlite
|
# Linux: ~/.local/share/vaultik/index.sqlite
|
||||||
|
|||||||
+3
-1205
File diff suppressed because it is too large.
Load diff
@@ -24,6 +24,7 @@ require (
|
|||||||
github.com/stretchr/testify v1.11.1
|
github.com/stretchr/testify v1.11.1
|
||||||
go.uber.org/dig v1.19.0
|
go.uber.org/dig v1.19.0
|
||||||
go.uber.org/fx v1.24.0
|
go.uber.org/fx v1.24.0
|
||||||
|
golang.org/x/mod v0.29.0
|
||||||
golang.org/x/sync v0.18.0
|
golang.org/x/sync v0.18.0
|
||||||
golang.org/x/sys v0.38.0
|
golang.org/x/sys v0.38.0
|
||||||
golang.org/x/term v0.37.0
|
golang.org/x/term v0.37.0
|
||||||
|
|||||||
@@ -2,8 +2,10 @@ package blobgen_test
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"testing"
|
"testing"
|
||||||
|
"testing/iotest"
|
||||||
|
|
||||||
"filippo.io/age"
|
"filippo.io/age"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
@@ -120,6 +122,38 @@ func TestNewReaderTrailingAndGarbage(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestNewReaderGarbageErrorQuotesNoBytes checks that lines of text in
|
||||||
|
// place of a blob fail with ErrNotAgeStream, and that the error carries
|
||||||
|
// none of their bytes. The lines end in a newline because age's own error
|
||||||
|
// quotes the data's first line only when it reads a whole line.
|
||||||
|
func TestNewReaderGarbageErrorQuotesNoBytes(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
id, _ := makeIdentity(t)
|
||||||
|
garbage := bytes.Repeat([]byte("garbage-bytes-\n"), 100)
|
||||||
|
|
||||||
|
_, err := blobgen.NewReader(bytes.NewReader(garbage), id)
|
||||||
|
|
||||||
|
require.ErrorIs(t, err, blobgen.ErrNotAgeStream)
|
||||||
|
assert.NotContains(t, err.Error(), "garbage-bytes-")
|
||||||
|
}
|
||||||
|
|
||||||
|
// errConnectionReset stands for a storage read that fails.
|
||||||
|
var errConnectionReset = errors.New("connection reset")
|
||||||
|
|
||||||
|
// TestNewReaderReadFailure checks that a failed read of the age header is
|
||||||
|
// returned as that read error, not as ErrNotAgeStream.
|
||||||
|
func TestNewReaderReadFailure(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
id, _ := makeIdentity(t)
|
||||||
|
|
||||||
|
_, err := blobgen.NewReader(iotest.ErrReader(errConnectionReset), id)
|
||||||
|
|
||||||
|
require.ErrorIs(t, err, errConnectionReset)
|
||||||
|
assert.NotErrorIs(t, err, blobgen.ErrNotAgeStream)
|
||||||
|
}
|
||||||
|
|
||||||
// TestNewWriterInvalidLevel covers the rejected end of issue case 9: an
|
// TestNewWriterInvalidLevel covers the rejected end of issue case 9: an
|
||||||
// out-of-range compression level errors and writes nothing to the destination.
|
// out-of-range compression level errors and writes nothing to the destination.
|
||||||
func TestNewWriterInvalidLevel(t *testing.T) {
|
func TestNewWriterInvalidLevel(t *testing.T) {
|
||||||
|
|||||||
@@ -11,6 +11,11 @@ import (
|
|||||||
"github.com/klauspost/compress/zstd"
|
"github.com/klauspost/compress/zstd"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// ErrNotAgeStream is returned by NewReader when the data cannot be opened
|
||||||
|
// as an age stream, for example because its header is malformed. It
|
||||||
|
// replaces age's own error, which can quote the data's first bytes.
|
||||||
|
var ErrNotAgeStream = errors.New("not a valid age stream")
|
||||||
|
|
||||||
// Reader wraps decompression and decryption with SHA256 verification
|
// Reader wraps decompression and decryption with SHA256 verification
|
||||||
type Reader struct {
|
type Reader struct {
|
||||||
reader io.Reader
|
reader io.Reader
|
||||||
@@ -24,11 +29,25 @@ type Reader struct {
|
|||||||
// NewReader creates a new Reader that decrypts, decompresses, and verifies
|
// NewReader creates a new Reader that decrypts, decompresses, and verifies
|
||||||
// data. Every supplied identity is offered to age.Decrypt, so a blob
|
// data. Every supplied identity is offered to age.Decrypt, so a blob
|
||||||
// encrypted to any one of them can be read.
|
// encrypted to any one of them can be read.
|
||||||
|
//
|
||||||
|
// A failure to read r is returned as that error, and a blob encrypted to
|
||||||
|
// none of the identities as age.NoIdentityMatchError. Any other failure
|
||||||
|
// to open the age stream is ErrNotAgeStream.
|
||||||
func NewReader(r io.Reader, identities ...age.Identity) (*Reader, error) {
|
func NewReader(r io.Reader, identities ...age.Identity) (*Reader, error) {
|
||||||
// Create decryption reader
|
src := &readErrorRecorder{reader: r}
|
||||||
decReader, err := age.Decrypt(r, identities...)
|
|
||||||
|
decReader, err := age.Decrypt(src, identities...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
var noMatch *age.NoIdentityMatchError
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case src.err != nil:
|
||||||
|
return nil, fmt.Errorf("reading age header: %w", src.err)
|
||||||
|
case errors.As(err, &noMatch):
|
||||||
return nil, fmt.Errorf("creating decryption reader: %w", err)
|
return nil, fmt.Errorf("creating decryption reader: %w", err)
|
||||||
|
default:
|
||||||
|
return nil, ErrNotAgeStream
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Create decompression reader
|
// Create decompression reader
|
||||||
@@ -52,6 +71,23 @@ func NewReader(r io.Reader, identities ...age.Identity) (*Reader, error) {
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// readErrorRecorder passes reads through to reader and keeps the first
|
||||||
|
// error other than io.EOF, so NewReader can tell a failed read from bad
|
||||||
|
// data.
|
||||||
|
type readErrorRecorder struct {
|
||||||
|
reader io.Reader
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *readErrorRecorder) Read(p []byte) (int, error) {
|
||||||
|
n, err := r.reader.Read(p)
|
||||||
|
if err != nil && !errors.Is(err, io.EOF) && r.err == nil {
|
||||||
|
r.err = err
|
||||||
|
}
|
||||||
|
|
||||||
|
return n, err
|
||||||
|
}
|
||||||
|
|
||||||
// Read implements io.Reader
|
// Read implements io.Reader
|
||||||
func (r *Reader) Read(p []byte) (int, error) {
|
func (r *Reader) Read(p []byte) (int, error) {
|
||||||
n, err := r.teeReader.Read(p)
|
n, err := r.teeReader.Read(p)
|
||||||
|
|||||||
+2
-5
@@ -7,11 +7,9 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"path/filepath"
|
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/adrg/xdg"
|
|
||||||
"github.com/spf13/cobra"
|
"github.com/spf13/cobra"
|
||||||
"go.uber.org/dig"
|
"go.uber.org/dig"
|
||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
@@ -339,7 +337,7 @@ func runVaultikApp(
|
|||||||
},
|
},
|
||||||
Mode: mode,
|
Mode: mode,
|
||||||
}, op, func(err error) {
|
}, op, func(err error) {
|
||||||
log.Error(failMsg, "error", err)
|
log.Info(failMsg, "error", err)
|
||||||
ReportErrorf("%s: %v", failMsg, err)
|
ReportErrorf("%s: %v", failMsg, err)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -351,8 +349,7 @@ func runVaultikApp(
|
|||||||
// only one runs at a time; a read-only command runs without it and is not
|
// only one runs at a time; a read-only command runs without it and is not
|
||||||
// blocked while a mutator holds the lock (opts.Mode).
|
// blocked while a mutator holds the lock (opts.Mode).
|
||||||
func RunWithApp(ctx context.Context, opts AppOptions) error {
|
func RunWithApp(ctx context.Context, opts AppOptions) error {
|
||||||
release, err := acquireLockIfMutating(opts.Mode,
|
release, err := acquireLockIfMutating(opts.Mode, pidlock.Dir())
|
||||||
filepath.Join(xdg.DataHome, "vaultik"))
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|||||||
+38
-40
@@ -49,8 +49,14 @@ var (
|
|||||||
errWouldNotLoad = errors.New(
|
errWouldNotLoad = errors.New(
|
||||||
"vaultik would not load the config after this change, " +
|
"vaultik would not load the config after this change, " +
|
||||||
"so the file was not changed")
|
"so the file was not changed")
|
||||||
|
errSetS3KeysFirst = errors.New(
|
||||||
|
"set s3.access_key_id and s3.secret_access_key first, " +
|
||||||
|
"then storage_url")
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// storageURLKey is the config key that selects the storage backend.
|
||||||
|
const storageURLKey = "storage_url"
|
||||||
|
|
||||||
// macOS is the value of runtime.GOOS on macOS.
|
// macOS is the value of runtime.GOOS on macOS.
|
||||||
const macOS = "darwin"
|
const macOS = "darwin"
|
||||||
|
|
||||||
@@ -276,7 +282,8 @@ storage_url: ""
|
|||||||
# Hostname used in snapshot IDs. Default: system hostname.
|
# Hostname used in snapshot IDs. Default: system hostname.
|
||||||
# hostname: myserver
|
# hostname: myserver
|
||||||
|
|
||||||
# Path to the local SQLite index database.
|
# Path to the local SQLite index database. It must be an absolute path;
|
||||||
|
# ~ is expanded.
|
||||||
# Default: the platform data directory, e.g.
|
# Default: the platform data directory, e.g.
|
||||||
# macOS: ~/Library/Application Support/vaultik/index.sqlite
|
# macOS: ~/Library/Application Support/vaultik/index.sqlite
|
||||||
# Linux: ~/.local/share/vaultik/index.sqlite
|
# Linux: ~/.local/share/vaultik/index.sqlite
|
||||||
@@ -443,11 +450,10 @@ number, otherwise as a map.
|
|||||||
|
|
||||||
If the config loads, a change after which it would not, such as
|
If the config loads, a change after which it would not, such as
|
||||||
compression_level 99, is refused and the file is left unchanged. A
|
compression_level 99, is refused and the file is left unchanged. A
|
||||||
config that does not load, such as the one config init writes before
|
config that does not load is checked one setting at a time. Examples
|
||||||
storage_url is set or one with a ${ENV:...} reference to a variable
|
are the file config init writes before storage_url is set, and one
|
||||||
not set here, is checked one setting at a time: only a change after
|
with a ${ENV:...} reference to a variable not set in this shell. In
|
||||||
which the changed setting cannot be read, such as one giving it the
|
such a config, an unknown key or a value of the wrong type is refused.
|
||||||
wrong type, is refused.
|
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
vaultik config set storage_url "file:///mnt/backups"
|
vaultik config set storage_url "file:///mnt/backups"
|
||||||
@@ -484,6 +490,13 @@ func writeConfigSet(out *ui.Writer, path, key, value string) error {
|
|||||||
|
|
||||||
keys := strings.Split(key, ".")
|
keys := strings.Split(key, ".")
|
||||||
|
|
||||||
|
if keys[0] != config.EnvSection {
|
||||||
|
_, err = config.SettingType(keys)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
err = yamlPathSet(root, keys, value)
|
err = yamlPathSet(root, keys, value)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -506,7 +519,7 @@ func writeConfigSet(out *ui.Writer, path, key, value string) error {
|
|||||||
|
|
||||||
err = checkConfigChange(oldData, data, settingData)
|
err = checkConfigChange(oldData, data, settingData)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("%w: %w", errWouldNotLoad, err)
|
return wouldNotLoadError(key, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
err = os.WriteFile(path, data, configFileMode)
|
err = os.WriteFile(path, data, configFileMode)
|
||||||
@@ -530,6 +543,21 @@ func writeConfigSet(out *ui.Writer, path, key, value string) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// wouldNotLoadError returns the refusal of a change to key that vaultik
|
||||||
|
// would not load for reason. When storage_url moves to s3:// before the S3
|
||||||
|
// credentials are set, it says to set them first: they can be set while
|
||||||
|
// the old storage_url is still in place.
|
||||||
|
func wouldNotLoadError(key string, reason error) error {
|
||||||
|
if key == storageURLKey &&
|
||||||
|
(errors.Is(reason, config.ErrS3KeyIDRequired) ||
|
||||||
|
errors.Is(reason, config.ErrS3SecretRequired)) {
|
||||||
|
return fmt.Errorf("%w: %w; %w", errWouldNotLoad, reason,
|
||||||
|
errSetS3KeysFirst)
|
||||||
|
}
|
||||||
|
|
||||||
|
return fmt.Errorf("%w: %w", errWouldNotLoad, reason)
|
||||||
|
}
|
||||||
|
|
||||||
// checkConfigChange returns the reason vaultik would not load newData, the
|
// checkConfigChange returns the reason vaultik would not load newData, the
|
||||||
// config after a change, if the change brings that failure in. If oldData,
|
// config after a change, if the change brings that failure in. If oldData,
|
||||||
// the config before it, loads, the new one must load too. A config that
|
// the config before it, loads, the new one must load too. A config that
|
||||||
@@ -739,41 +767,11 @@ func yamlPathSet(root *yaml.Node, keys []string, value string) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// configKeyIsString reports whether the dotted key names a string in
|
// configKeyIsString reports whether the dotted key names a string in
|
||||||
// config.Config, following the fields' yaml tags, as s3.access_key_id and
|
// config.Config, as s3.access_key_id and snapshots.home.exclude.0 do.
|
||||||
// snapshots.home.exclude.0 do.
|
|
||||||
func configKeyIsString(keys []string) bool {
|
func configKeyIsString(keys []string) bool {
|
||||||
typ := reflect.TypeFor[config.Config]()
|
typ, err := config.SettingType(keys)
|
||||||
|
|
||||||
for _, key := range keys {
|
return err == nil && typ.Kind() == reflect.String
|
||||||
switch {
|
|
||||||
case typ.Kind() == reflect.Map || typ.Kind() == reflect.Slice:
|
|
||||||
// The key is a snapshot name or a list index.
|
|
||||||
typ = typ.Elem()
|
|
||||||
case typ.Kind() == reflect.Struct:
|
|
||||||
field, ok := yamlField(typ, key)
|
|
||||||
if !ok {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
typ = field.Type
|
|
||||||
default:
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return typ.Kind() == reflect.String
|
|
||||||
}
|
|
||||||
|
|
||||||
// yamlField returns the field of struct type typ whose yaml tag names key.
|
|
||||||
func yamlField(typ reflect.Type, key string) (reflect.StructField, bool) {
|
|
||||||
for field := range typ.Fields() {
|
|
||||||
name, _, _ := strings.Cut(field.Tag.Get("yaml"), ",")
|
|
||||||
if name == key {
|
|
||||||
return field, true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return reflect.StructField{}, false
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// yamlSetInMapping resolves (creating if needed) the value node for the
|
// yamlSetInMapping resolves (creating if needed) the value node for the
|
||||||
|
|||||||
+137
-2
@@ -2,6 +2,7 @@ package cli //nolint:testpackage // exercises unexported yamlPathGet/yamlPathSet
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
|
"errors"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strconv"
|
"strconv"
|
||||||
@@ -152,6 +153,21 @@ snapshots:
|
|||||||
- "~"
|
- "~"
|
||||||
`
|
`
|
||||||
|
|
||||||
|
// testS3URL is an S3 storage_url, which needs credentials to load.
|
||||||
|
const testS3URL = "s3://bucket/prefix?endpoint=https://s3.example.com"
|
||||||
|
|
||||||
|
// configWithS3Keys loads: its storage_url is testS3URL and both S3 keys are
|
||||||
|
// set.
|
||||||
|
const configWithS3Keys = storageURLKey + ": " + testS3URL + `
|
||||||
|
s3:
|
||||||
|
access_key_id: AKIAEXAMPLE
|
||||||
|
secret_access_key: example-secret
|
||||||
|
snapshots:
|
||||||
|
home:
|
||||||
|
paths:
|
||||||
|
- "~"
|
||||||
|
`
|
||||||
|
|
||||||
// configWithUnsetReference loads only where VAULTIK_TEST_UNSET_SECRET is
|
// configWithUnsetReference loads only where VAULTIK_TEST_UNSET_SECRET is
|
||||||
// set, as a backup job's environment might set it. No test sets it.
|
// set, as a backup job's environment might set it. No test sets it.
|
||||||
const configWithUnsetReference = configWithoutRecipients + `s3:
|
const configWithUnsetReference = configWithoutRecipients + `s3:
|
||||||
@@ -225,7 +241,7 @@ func TestConfigSetIndexUnderMissingKey(t *testing.T) {
|
|||||||
// change after which the config would not load, with an error giving the
|
// change after which the config would not load, with an error giving the
|
||||||
// reason, and leaves the file as it was. On the file `config init` writes,
|
// reason, and leaves the file as it was. On the file `config init` writes,
|
||||||
// which does not load until storage_url is set, and on one that cannot be
|
// which does not load until storage_url is set, and on one that cannot be
|
||||||
// read here, only a value of the wrong type is refused.
|
// read here, only an unknown key or a value of the wrong type is refused.
|
||||||
func TestConfigSetKeepsFileThatWouldNotLoad(t *testing.T) {
|
func TestConfigSetKeepsFileThatWouldNotLoad(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -237,6 +253,14 @@ func TestConfigSetKeepsFileThatWouldNotLoad(t *testing.T) {
|
|||||||
value string
|
value string
|
||||||
reason string
|
reason string
|
||||||
}{
|
}{
|
||||||
|
// Keys that no setting has, on a config that loads and on one
|
||||||
|
// that does not.
|
||||||
|
{configWithoutRecipients, "nosuchkey", "1",
|
||||||
|
`unknown config key "nosuchkey"`},
|
||||||
|
{configWithoutRecipients, "snapshots.home.exlude", "/tmp",
|
||||||
|
`unknown config key "snapshots.home.exlude"`},
|
||||||
|
{defaultConfigText(macOS), "compression_lvl", "9",
|
||||||
|
`unknown config key "compression_lvl"`},
|
||||||
// Index 1 of the empty list the missing key gives.
|
// Index 1 of the empty list the missing key gives.
|
||||||
{configWithoutRecipients, "age_recipients.1", testRecipient,
|
{configWithoutRecipients, "age_recipients.1", testRecipient,
|
||||||
"index out of range"},
|
"index out of range"},
|
||||||
@@ -244,12 +268,20 @@ func TestConfigSetKeepsFileThatWouldNotLoad(t *testing.T) {
|
|||||||
{configWithoutRecipients, "snapshots.home.paths", "/srv",
|
{configWithoutRecipients, "snapshots.home.paths", "/srv",
|
||||||
"into []string"},
|
"into []string"},
|
||||||
{configWithoutRecipients, "chunk_size", "fast",
|
{configWithoutRecipients, "chunk_size", "fast",
|
||||||
"invalid size format"},
|
`chunk_size: "fast" is not a size`},
|
||||||
// Values of the right type that fail validation.
|
// Values of the right type that fail validation.
|
||||||
{configWithoutRecipients, "compression_level", "99",
|
{configWithoutRecipients, "compression_level", "99",
|
||||||
"compression_level must be between 1 and 19"},
|
"compression_level must be between 1 and 19"},
|
||||||
{configWithoutRecipients, "chunk_size", "1KB",
|
{configWithoutRecipients, "chunk_size", "1KB",
|
||||||
"chunk_size must be at least 1MB"},
|
"chunk_size must be at least 1MB"},
|
||||||
|
// An s3:// URL before its credentials names them.
|
||||||
|
{configWithoutRecipients, storageURLKey, testS3URL,
|
||||||
|
"s3.access_key_id is required for s3:// URLs; " +
|
||||||
|
"set s3.access_key_id and s3.secret_access_key first"},
|
||||||
|
{configWithoutRecipients + "s3:\n access_key_id: AKIAEXAMPLE\n",
|
||||||
|
storageURLKey, testS3URL,
|
||||||
|
"s3.secret_access_key is required for s3:// URLs; " +
|
||||||
|
"set s3.access_key_id and s3.secret_access_key first"},
|
||||||
// With its reference filled in, the config loads.
|
// With its reference filled in, the config loads.
|
||||||
{levelReference, "blob_size_limit", "1MB",
|
{levelReference, "blob_size_limit", "1MB",
|
||||||
"blob_size_limit must be at least"},
|
"blob_size_limit must be at least"},
|
||||||
@@ -335,6 +367,109 @@ func TestConfigSetRefusesOnlyItsOwnFailure(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestConfigSetS3KeysBeforeStorageURL checks that the order the refusal
|
||||||
|
// gives works on a config that loads with a file:// storage_url.
|
||||||
|
func TestConfigSetS3KeysBeforeStorageURL(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
path := filepath.Join(t.TempDir(), "config.yml")
|
||||||
|
|
||||||
|
err := os.WriteFile(path, []byte(configWithoutRecipients), configFileMode)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("write config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, set := range [][2]string{
|
||||||
|
{"s3.access_key_id", "AKIAEXAMPLE"},
|
||||||
|
{"s3.secret_access_key", "example-secret"},
|
||||||
|
{storageURLKey, testS3URL},
|
||||||
|
} {
|
||||||
|
err = writeConfigSet(ui.NewWithColor(&bytes.Buffer{}, false),
|
||||||
|
path, set[0], set[1])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("config set %s: %v", set[0], err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := os.ReadFile(path) //nolint:gosec // G304: test-controlled path
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(string(data), storageURLKey+": "+testS3URL) {
|
||||||
|
t.Errorf("config set storage_url wrote:\n%s", data)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestConfigSetRefusesClearingS3Key checks that clearing s3.access_key_id on
|
||||||
|
// a config with an s3:// storage_url is refused without saying to set the
|
||||||
|
// S3 keys first. That advice is only for a change to storage_url.
|
||||||
|
func TestConfigSetRefusesClearingS3Key(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const key = "s3.access_key_id"
|
||||||
|
|
||||||
|
path := filepath.Join(t.TempDir(), "config.yml")
|
||||||
|
|
||||||
|
err := os.WriteFile(path, []byte(configWithS3Keys), configFileMode)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("write config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = writeConfigSet(ui.NewWithColor(&bytes.Buffer{}, false), path, key, "")
|
||||||
|
if !errors.Is(err, errWouldNotLoad) {
|
||||||
|
t.Errorf("clearing %s gives error %v, want one saying %q",
|
||||||
|
key, err, errWouldNotLoad)
|
||||||
|
}
|
||||||
|
|
||||||
|
if errors.Is(err, errSetS3KeysFirst) {
|
||||||
|
t.Errorf("clearing %s gives error %v, want one not saying %q",
|
||||||
|
key, err, errSetS3KeysFirst)
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := os.ReadFile(path) //nolint:gosec // G304: test-controlled path
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if string(data) != configWithS3Keys {
|
||||||
|
t.Errorf("clearing %s changed the file to:\n%s", key, data)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestConfigSetWritesEnvSection checks that config set writes a key under
|
||||||
|
// the top-level env section, whose keys are not settings, on a config that
|
||||||
|
// loads.
|
||||||
|
func TestConfigSetWritesEnvSection(t *testing.T) {
|
||||||
|
const name = "VAULTIK_TEST_CONFIG_SET_ENV"
|
||||||
|
|
||||||
|
// Loading the changed config sets the variable; t.Setenv restores it
|
||||||
|
// after the test.
|
||||||
|
t.Setenv(name, "")
|
||||||
|
|
||||||
|
path := filepath.Join(t.TempDir(), "config.yml")
|
||||||
|
|
||||||
|
err := os.WriteFile(path, []byte(configWithoutRecipients), configFileMode)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("write config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = writeConfigSet(ui.NewWithColor(&bytes.Buffer{}, false),
|
||||||
|
path, "env."+name, "value")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("config set env.%s: %v", name, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := os.ReadFile(path) //nolint:gosec // G304: test-controlled path
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(string(data), "env:\n "+name+": value\n") {
|
||||||
|
t.Errorf("config set env.%s wrote:\n%s", name, data)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestConfigSetStringLooksLikeNumber sets string settings to values that
|
// TestConfigSetStringLooksLikeNumber sets string settings to values that
|
||||||
// YAML reads as numbers or booleans when they are unquoted, and checks that
|
// YAML reads as numbers or booleans when they are unquoted, and checks that
|
||||||
// config.Load returns each one unchanged.
|
// config.Load returns each one unchanged.
|
||||||
|
|||||||
+12
-12
@@ -6,7 +6,9 @@ import (
|
|||||||
|
|
||||||
"github.com/spf13/cobra"
|
"github.com/spf13/cobra"
|
||||||
"sneak.berlin/go/vaultik/internal/config"
|
"sneak.berlin/go/vaultik/internal/config"
|
||||||
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
)
|
)
|
||||||
|
|
||||||
// NewDatabaseCommand creates the database command group
|
// NewDatabaseCommand creates the database command group
|
||||||
@@ -90,26 +92,24 @@ Use --force to skip the confirmation prompt.`,
|
|||||||
|
|
||||||
var confirm string
|
var confirm string
|
||||||
|
|
||||||
_, err = fmt.Scanln(&confirm)
|
// A failure to read the answer means no confirmation.
|
||||||
|
_, err = fmt.Fscanln(cmd.InOrStdin(), &confirm)
|
||||||
if err != nil || confirm != "yes" {
|
if err != nil || confirm != "yes" {
|
||||||
_, _ = fmt.Fprintln(w, "Aborted.")
|
_, _ = fmt.Fprintln(w, "Aborted.")
|
||||||
|
|
||||||
//nolint:nilerr // a failed/aborted confirmation is a clean abort
|
return vaultik.ErrNotConfirmed
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Delete the database file
|
// Delete the database file and its side files. A side file
|
||||||
err = os.Remove(dbPath)
|
// may not exist, so only failing to delete the database file
|
||||||
if err != nil {
|
// is an error.
|
||||||
|
for _, indexFile := range database.IndexFiles(dbPath) {
|
||||||
|
err = os.Remove(indexFile)
|
||||||
|
if err != nil && indexFile == dbPath {
|
||||||
return fmt.Errorf("failed to delete database: %w", err)
|
return fmt.Errorf("failed to delete database: %w", err)
|
||||||
}
|
}
|
||||||
|
}
|
||||||
// Also delete WAL and SHM files if they exist
|
|
||||||
walPath := dbPath + "-wal"
|
|
||||||
shmPath := dbPath + "-shm"
|
|
||||||
_ = os.Remove(walPath) // Ignore errors - files may not exist
|
|
||||||
_ = os.Remove(shmPath)
|
|
||||||
|
|
||||||
out.Infof("Local state database deleted: %s.", dbPath)
|
out.Infof("Local state database deleted: %s.", dbPath)
|
||||||
log.Info("Local state database deleted", "path", dbPath)
|
log.Info("Local state database deleted", "path", dbPath)
|
||||||
|
|||||||
@@ -0,0 +1,75 @@
|
|||||||
|
package cli //nolint:testpackage // sets the unexported rootFlags directly
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestDatabaseDeleteDeclinedIsError checks that answering anything but
|
||||||
|
// "yes" keeps the database and returns ErrNotConfirmed, so the command
|
||||||
|
// exits non-zero.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // mutates the process-global rootFlags
|
||||||
|
func TestDatabaseDeleteDeclinedIsError(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
dbPath, cfgPath := seedDatabaseDeleteConfig(t, dir)
|
||||||
|
|
||||||
|
setRootFlags(t, RootFlags{ConfigPath: cfgPath})
|
||||||
|
|
||||||
|
var out bytes.Buffer
|
||||||
|
|
||||||
|
cmd := newDatabaseDeleteCommand()
|
||||||
|
cmd.SetOut(&out)
|
||||||
|
cmd.SetErr(io.Discard)
|
||||||
|
cmd.SetIn(strings.NewReader("no\n"))
|
||||||
|
cmd.SetArgs(nil)
|
||||||
|
|
||||||
|
err := cmd.Execute()
|
||||||
|
if !errors.Is(err, vaultik.ErrNotConfirmed) {
|
||||||
|
t.Errorf("declined database delete returned %v, want %v",
|
||||||
|
err, vaultik.ErrNotConfirmed)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = os.Stat(dbPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("declined database delete removed the file: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(out.String(), "Aborted.") {
|
||||||
|
t.Errorf("declined database delete printed %q, want Aborted.",
|
||||||
|
out.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestDatabaseDeleteRemovesSideFiles checks that the side files SQLite
|
||||||
|
// keeps beside the database file are deleted with it.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // mutates the process-global rootFlags
|
||||||
|
func TestDatabaseDeleteRemovesSideFiles(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
dbPath, cfgPath := seedDatabaseDeleteConfig(t, dir)
|
||||||
|
|
||||||
|
indexFiles := database.IndexFiles(dbPath)
|
||||||
|
for _, indexFile := range indexFiles {
|
||||||
|
seedFile(t, dir, filepath.Base(indexFile), "data")
|
||||||
|
}
|
||||||
|
|
||||||
|
setRootFlags(t, RootFlags{ConfigPath: cfgPath})
|
||||||
|
|
||||||
|
mustExecute(t, newDatabaseDeleteCommand(), "--force")
|
||||||
|
|
||||||
|
for _, indexFile := range indexFiles {
|
||||||
|
_, err := os.Stat(indexFile)
|
||||||
|
if !os.IsNotExist(err) {
|
||||||
|
t.Errorf("database delete left %s: stat err = %v", indexFile, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -21,6 +21,8 @@ const shortCommitLen = 12
|
|||||||
const exitCodeInterrupted = 130
|
const exitCodeInterrupted = 130
|
||||||
|
|
||||||
// Entry is the main entry point for the CLI application.
|
// Entry is the main entry point for the CLI application.
|
||||||
|
// It first calls globals.FillUnsetFromBuildInfo, which fills in the
|
||||||
|
// version, commit and commit date that the linker flags did not set.
|
||||||
// It prints the startup banner to stderr (unless a banner-suppressing
|
// It prints the startup banner to stderr (unless a banner-suppressing
|
||||||
// flag is present in os.Args — see bannerSuppressedInArgs), executes the
|
// flag is present in os.Args — see bannerSuppressedInArgs), executes the
|
||||||
// root cobra command, and routes any returned error through the
|
// root cobra command, and routes any returned error through the
|
||||||
@@ -33,6 +35,7 @@ const exitCodeInterrupted = 130
|
|||||||
// deferred profile writers run before the process ends. See run in
|
// deferred profile writers run before the process ends. See run in
|
||||||
// cmd/vaultik/main.go.
|
// cmd/vaultik/main.go.
|
||||||
func Entry() int {
|
func Entry() int {
|
||||||
|
globals.FillUnsetFromBuildInfo()
|
||||||
emitStartupBanner(os.Args[1:], os.Stderr)
|
emitStartupBanner(os.Args[1:], os.Stderr)
|
||||||
|
|
||||||
rootCmd := NewRootCommand()
|
rootCmd := NewRootCommand()
|
||||||
@@ -82,6 +85,10 @@ func emitStartupBanner(args []string, w io.Writer) {
|
|||||||
// 🛑 ERROR: format. Use it from goroutine error paths (where returning
|
// 🛑 ERROR: format. Use it from goroutine error paths (where returning
|
||||||
// an error to cobra isn't an option) and anywhere else a CLI command
|
// an error to cobra isn't an option) and anywhere else a CLI command
|
||||||
// must surface a failure outside the normal RunE return path.
|
// must surface a failure outside the normal RunE return path.
|
||||||
|
//
|
||||||
|
// This line is the one report of the failure. A caller that also logs
|
||||||
|
// the failure logs it at info level, so the record appears only under
|
||||||
|
// --verbose or --debug and the error is not printed twice.
|
||||||
func ReportErrorf(format string, args ...any) {
|
func ReportErrorf(format string, args ...any) {
|
||||||
ui.New(os.Stderr).Errorf(format, args...)
|
ui.New(os.Stderr).Errorf(format, args...)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,205 @@
|
|||||||
|
package cli //nolint:testpackage // shares the fixtures and capture helpers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"filippo.io/age"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
|
"sneak.berlin/go/vaultik/internal/ui"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// errorLinePrefix starts the final line that reports a failed command.
|
||||||
|
errorLinePrefix = "ERROR: "
|
||||||
|
|
||||||
|
flagVerbose = "--verbose"
|
||||||
|
cmdRestore = "restore"
|
||||||
|
|
||||||
|
// blobsDir is the directory of a destination store that holds the
|
||||||
|
// blobs.
|
||||||
|
blobsDir = "blobs"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestEntryFailureIsPrintedOnce runs commands that fail after startup
|
||||||
|
// against a destination they cannot use. `info` is not among them: it
|
||||||
|
// cannot fail after startup.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||||
|
func TestEntryFailureIsPrintedOnce(t *testing.T) {
|
||||||
|
for _, command := range [][]string{
|
||||||
|
{cmdSnapshot, cmdCreate},
|
||||||
|
{cmdSnapshot, cmdRemove, someSnapshotID},
|
||||||
|
{cmdSnapshot, cmdVerify, someSnapshotID},
|
||||||
|
{cmdRemote, cmdInfo},
|
||||||
|
{cmdPrune, flagJSON},
|
||||||
|
} {
|
||||||
|
for _, verbose := range []bool{false, true} {
|
||||||
|
name := fmt.Sprintf("%s verbose=%t", strings.Join(command, " "), verbose)
|
||||||
|
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
requireErrorPrintedOnce(t, verbose, append([]string{flagConfig,
|
||||||
|
writeUnusableDestinationConfig(t)}, command...)...)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestEntryBlobUploadFailureIsPrintedOnce backs up to a destination
|
||||||
|
// whose blobs directory is a file, so the blob upload fails.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||||
|
func TestEntryBlobUploadFailureIsPrintedOnce(t *testing.T) {
|
||||||
|
for _, verbose := range []bool{false, true} {
|
||||||
|
t.Run(fmt.Sprintf("verbose=%t", verbose), func(t *testing.T) {
|
||||||
|
configPath, storeDir := writeBackupConfig(t)
|
||||||
|
|
||||||
|
require.NoError(t, os.WriteFile(
|
||||||
|
filepath.Join(storeDir, blobsDir), nil, 0o600))
|
||||||
|
|
||||||
|
requireErrorPrintedOnce(t, verbose,
|
||||||
|
flagConfig, configPath, cmdSnapshot, cmdCreate)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestEntryBlobDownloadFailureIsPrintedOnce backs up, deletes the blobs
|
||||||
|
// from the destination, and restores, so the blob download fails.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||||
|
func TestEntryBlobDownloadFailureIsPrintedOnce(t *testing.T) {
|
||||||
|
for _, verbose := range []bool{false, true} {
|
||||||
|
t.Run(fmt.Sprintf("verbose=%t", verbose), func(t *testing.T) {
|
||||||
|
configPath, storeDir := writeBackupConfigWithPrivateKey(t)
|
||||||
|
|
||||||
|
code, _, _ := runEntry(t,
|
||||||
|
flagConfig, configPath, cmdSnapshot, cmdCreate)
|
||||||
|
require.Equal(t, 0, code)
|
||||||
|
|
||||||
|
require.NoError(t, os.RemoveAll(filepath.Join(storeDir, blobsDir)))
|
||||||
|
|
||||||
|
requireErrorPrintedOnce(t, verbose, flagConfig, configPath,
|
||||||
|
cmdSnapshot, cmdRestore, onlySnapshotID(t, configPath), t.TempDir())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// requireErrorPrintedOnce runs Entry with args, adding --verbose when
|
||||||
|
// verbose is set. The command must fail, and stderr must carry its error
|
||||||
|
// once, as the final ERROR: line. A log record carrying the same error
|
||||||
|
// appears only under --verbose.
|
||||||
|
func requireErrorPrintedOnce(t *testing.T, verbose bool, args ...string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if verbose {
|
||||||
|
args = append(args, flagVerbose)
|
||||||
|
}
|
||||||
|
|
||||||
|
code, _, stderr := runEntry(t, args...)
|
||||||
|
require.Equal(t, 1, code)
|
||||||
|
|
||||||
|
errorLine, records := failureOutput(t, stderr)
|
||||||
|
|
||||||
|
if !verbose {
|
||||||
|
assert.Zero(t, records,
|
||||||
|
"without --verbose no log record repeats the error:\n%s", stderr)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
assert.NotZero(t, records,
|
||||||
|
"--verbose must keep the log record of %q", errorLine)
|
||||||
|
}
|
||||||
|
|
||||||
|
// failureOutput requires stderr to hold exactly one ERROR: line and
|
||||||
|
// returns it, with the number of log records whose error attribute is
|
||||||
|
// the error that line reports. stderr is a pipe here, so every log
|
||||||
|
// record is a JSON object on its own line.
|
||||||
|
func failureOutput(t *testing.T, stderr string) (string, int) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var (
|
||||||
|
errorLines []string
|
||||||
|
records []map[string]any
|
||||||
|
)
|
||||||
|
|
||||||
|
for line := range strings.Lines(stderr) {
|
||||||
|
var record map[string]any
|
||||||
|
if json.Unmarshal([]byte(line), &record) == nil {
|
||||||
|
records = append(records, record)
|
||||||
|
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.Contains(line, errorLinePrefix) {
|
||||||
|
errorLines = append(errorLines, strings.TrimSpace(line))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
require.Len(t, errorLines, 1,
|
||||||
|
"the error must be printed once as an ERROR: line:\n%s", stderr)
|
||||||
|
|
||||||
|
// An error can run over several lines; the ERROR: line holds the
|
||||||
|
// first of them.
|
||||||
|
repeats := 0
|
||||||
|
|
||||||
|
for _, record := range records {
|
||||||
|
failure, ok := record["error"].(string)
|
||||||
|
firstLine, _, _ := strings.Cut(failure, "\n")
|
||||||
|
|
||||||
|
if ok && strings.Contains(errorLines[0], firstLine) {
|
||||||
|
repeats++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return errorLines[0], repeats
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeBackupConfigWithPrivateKey builds what writeBackupConfig builds,
|
||||||
|
// with a generated key pair in place of its recipient. The config also
|
||||||
|
// holds the private key, so that restore can decrypt the backup. Returns
|
||||||
|
// the config path and the store directory.
|
||||||
|
func writeBackupConfigWithPrivateKey(t *testing.T) (string, string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
configPath, storeDir := writeBackupConfig(t)
|
||||||
|
|
||||||
|
identity, err := age.GenerateX25519Identity()
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
out := ui.NewWithColor(io.Discard, false)
|
||||||
|
|
||||||
|
require.NoError(t, writeConfigSet(out, configPath,
|
||||||
|
"age_recipients.0", identity.Recipient().String()))
|
||||||
|
require.NoError(t, writeConfigSet(out, configPath,
|
||||||
|
"age_secret_key", identity.String()))
|
||||||
|
|
||||||
|
return configPath, storeDir
|
||||||
|
}
|
||||||
|
|
||||||
|
// onlySnapshotID returns the ID of the one snapshot in the local index
|
||||||
|
// that writeBackupConfig puts beside the config file.
|
||||||
|
func onlySnapshotID(t *testing.T, configPath string) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
db, err := database.New(ctx,
|
||||||
|
filepath.Join(filepath.Dir(configPath), "index.sqlite"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
defer func() { require.NoError(t, db.Close()) }()
|
||||||
|
|
||||||
|
backedUp, err := database.NewRepositories(db).Snapshots.GetOnlySnapshot(ctx)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
return backedUp.ID.String()
|
||||||
|
}
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/adrg/xdg"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestEntrySnapshotCreateSkipsIndexAndLockFiles backs up a directory
|
||||||
|
// that holds vaultik's index and PID file beside one other file, with no
|
||||||
|
// exclude pattern set. The summary counts only the other file. See
|
||||||
|
// https://git.eeqj.de/sneak/vaultik/issues/421.
|
||||||
|
//
|
||||||
|
// Not parallel: it replaces os.Args, os.Stdout, os.Stderr and the xdg
|
||||||
|
// globals.
|
||||||
|
func TestEntrySnapshotCreateSkipsIndexAndLockFiles(t *testing.T) {
|
||||||
|
for _, testCase := range []struct {
|
||||||
|
name string
|
||||||
|
// linked sets index_path to a symlink outside the backed-up
|
||||||
|
// directory that points to the index inside it.
|
||||||
|
linked bool
|
||||||
|
}{
|
||||||
|
{name: "index in the backed-up directory"},
|
||||||
|
{name: "index_path links into the backed-up directory", linked: true},
|
||||||
|
} {
|
||||||
|
t.Run(testCase.name, func(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
configPath := filepath.Join(dir, "config.yml")
|
||||||
|
sourceDir := filepath.Join(dir, "source")
|
||||||
|
storeDir := filepath.Join(dir, "store")
|
||||||
|
|
||||||
|
indexPath := filepath.Join(sourceDir, "index.sqlite")
|
||||||
|
|
||||||
|
if testCase.linked {
|
||||||
|
link := filepath.Join(dir, "index.sqlite")
|
||||||
|
require.NoError(t, os.Symlink(indexPath, link))
|
||||||
|
|
||||||
|
indexPath = link
|
||||||
|
}
|
||||||
|
|
||||||
|
contents := fmt.Sprintf(hermeticConfig,
|
||||||
|
sourceDir, storeDir, indexPath)
|
||||||
|
require.NoError(t,
|
||||||
|
os.WriteFile(configPath, []byte(contents), configFileMode))
|
||||||
|
require.NoError(t, os.Mkdir(storeDir, 0o750))
|
||||||
|
require.NoError(t, os.Mkdir(sourceDir, 0o750))
|
||||||
|
require.NoError(t, os.WriteFile(
|
||||||
|
filepath.Join(sourceDir, "file.txt"), []byte("hello"), 0o600))
|
||||||
|
|
||||||
|
// The PID file goes in a directory under XDG_DATA_HOME, which
|
||||||
|
// xdg reads at package init.
|
||||||
|
t.Setenv("XDG_DATA_HOME", filepath.Join(sourceDir, "data"))
|
||||||
|
xdg.Reload()
|
||||||
|
t.Cleanup(xdg.Reload)
|
||||||
|
|
||||||
|
code, stdout, _ := runEntry(t,
|
||||||
|
flagConfig, configPath, cmdSnapshot, cmdCreate)
|
||||||
|
|
||||||
|
require.Equal(t, 0, code)
|
||||||
|
assert.Contains(t, stdout,
|
||||||
|
"Files: 1 examined, 1 backed up, 0 unchanged.")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -61,8 +61,8 @@ func TestEntrySnapshotCreateRejectsKeepNewerThanBeforeBackup(t *testing.T) {
|
|||||||
|
|
||||||
// TestEntrySnapshotCreateReportsPostBackupPruneFailure makes the
|
// TestEntrySnapshotCreateReportsPostBackupPruneFailure makes the
|
||||||
// post-backup prune fail after a good backup, by planting a manifest
|
// post-backup prune fail after a good backup, by planting a manifest
|
||||||
// that cannot be decoded, and checks that the failure is reported as the
|
// that cannot be decoded, and checks that the failure is reported once,
|
||||||
// prune's and not as a failed snapshot creation.
|
// as the prune's and not as a failed snapshot creation.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||||
func TestEntrySnapshotCreateReportsPostBackupPruneFailure(t *testing.T) {
|
func TestEntrySnapshotCreateReportsPostBackupPruneFailure(t *testing.T) {
|
||||||
@@ -83,6 +83,9 @@ func TestEntrySnapshotCreateReportsPostBackupPruneFailure(t *testing.T) {
|
|||||||
assert.Contains(t, stderr, "Backup completed, but post-backup prune failed")
|
assert.Contains(t, stderr, "Backup completed, but post-backup prune failed")
|
||||||
assert.NotContains(t, stderr, "Snapshot creation failed")
|
assert.NotContains(t, stderr, "Snapshot creation failed")
|
||||||
|
|
||||||
|
_, repeats := failureOutput(t, stderr)
|
||||||
|
assert.Zero(t, repeats, "no log record may repeat the error:\n%s", stderr)
|
||||||
|
|
||||||
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
|
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Len(t, snapshots, 2,
|
assert.Len(t, snapshots, 2,
|
||||||
@@ -93,7 +96,8 @@ func TestEntrySnapshotCreateReportsPostBackupPruneFailure(t *testing.T) {
|
|||||||
// empties the source directory, as a volume that is not mounted leaves
|
// empties the source directory, as a volume that is not mounted leaves
|
||||||
// its mount point, and runs `snapshot create --prune`. The command exits
|
// its mount point, and runs `snapshot create --prune`. The command exits
|
||||||
// non-zero and reports the backup as completed, not as a failed snapshot
|
// non-zero and reports the backup as completed, not as a failed snapshot
|
||||||
// creation. See https://git.eeqj.de/sneak/vaultik/issues/388.
|
// creation. The error is printed once. See
|
||||||
|
// https://git.eeqj.de/sneak/vaultik/issues/388.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||||
func TestEntrySnapshotCreatePruneExitsNonZeroWhenSourceIsEmpty(t *testing.T) {
|
func TestEntrySnapshotCreatePruneExitsNonZeroWhenSourceIsEmpty(t *testing.T) {
|
||||||
@@ -114,6 +118,9 @@ func TestEntrySnapshotCreatePruneExitsNonZeroWhenSourceIsEmpty(t *testing.T) {
|
|||||||
"Backup completed, but 1 source path(s) were empty")
|
"Backup completed, but 1 source path(s) were empty")
|
||||||
assert.NotContains(t, stderr, "Snapshot creation failed")
|
assert.NotContains(t, stderr, "Snapshot creation failed")
|
||||||
|
|
||||||
|
_, repeats := failureOutput(t, stderr)
|
||||||
|
assert.Zero(t, repeats, "no log record may repeat the error:\n%s", stderr)
|
||||||
|
|
||||||
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
|
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Len(t, snapshots, 2, "the older snapshot is kept")
|
assert.Len(t, snapshots, 2, "the older snapshot is kept")
|
||||||
@@ -124,7 +131,8 @@ func TestEntrySnapshotCreatePruneExitsNonZeroWhenSourceIsEmpty(t *testing.T) {
|
|||||||
// directory as a mount point, and runs `snapshot create --prune`, which
|
// directory as a mount point, and runs `snapshot create --prune`, which
|
||||||
// does not find it as one. The command exits non-zero, reports the
|
// does not find it as one. The command exits non-zero, reports the
|
||||||
// backup as completed, not as a failed snapshot creation, and keeps the
|
// backup as completed, not as a failed snapshot creation, and keeps the
|
||||||
// older snapshot. See https://git.eeqj.de/sneak/vaultik/issues/395.
|
// older snapshot. The error is printed once. See
|
||||||
|
// https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||||
func TestEntrySnapshotCreatePruneExitsNonZeroWhenMountPointIsMissing(
|
func TestEntrySnapshotCreatePruneExitsNonZeroWhenMountPointIsMissing(
|
||||||
@@ -159,6 +167,9 @@ func TestEntrySnapshotCreatePruneExitsNonZeroWhenMountPointIsMissing(
|
|||||||
"an earlier snapshot recorded were not found")
|
"an earlier snapshot recorded were not found")
|
||||||
assert.NotContains(t, stderr, "Snapshot creation failed")
|
assert.NotContains(t, stderr, "Snapshot creation failed")
|
||||||
|
|
||||||
|
_, repeats := failureOutput(t, stderr)
|
||||||
|
assert.Zero(t, repeats, "no log record may repeat the error:\n%s", stderr)
|
||||||
|
|
||||||
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
|
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Len(t, snapshots, 2, "the older snapshot is kept")
|
assert.Len(t, snapshots, 2, "the older snapshot is kept")
|
||||||
|
|||||||
@@ -39,7 +39,7 @@ func NewInfoCommand() *cobra.Command {
|
|||||||
}, func(v *vaultik.Vaultik) error {
|
}, func(v *vaultik.Vaultik) error {
|
||||||
return v.ShowInfo()
|
return v.ShowInfo()
|
||||||
}, func(err error) {
|
}, func(err error) {
|
||||||
log.Error("Failed to show info", "error", err)
|
log.Info("Failed to show info", "error", err)
|
||||||
ReportErrorf("Failed to show info: %v", err)
|
ReportErrorf("Failed to show info: %v", err)
|
||||||
})
|
})
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -60,7 +60,7 @@ referenced.`,
|
|||||||
}, func(v *vaultik.Vaultik) error {
|
}, func(v *vaultik.Vaultik) error {
|
||||||
return v.Prune(opts)
|
return v.Prune(opts)
|
||||||
}, func(err error) {
|
}, func(err error) {
|
||||||
log.Error("Prune operation failed", "error", err)
|
log.Info("Prune operation failed", "error", err)
|
||||||
ReportErrorf("Prune failed: %v", err)
|
ReportErrorf("Prune failed: %v", err)
|
||||||
})
|
})
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -98,7 +98,7 @@ func newRemoteInfoCommand() *cobra.Command {
|
|||||||
}, func(v *vaultik.Vaultik) error {
|
}, func(v *vaultik.Vaultik) error {
|
||||||
return v.RemoteInfo(jsonOutput)
|
return v.RemoteInfo(jsonOutput)
|
||||||
}, func(err error) {
|
}, func(err error) {
|
||||||
log.Error("Failed to get remote info", "error", err)
|
log.Info("Failed to get remote info", "error", err)
|
||||||
ReportErrorf("Failed to get remote info: %v", err)
|
ReportErrorf("Failed to get remote info: %v", err)
|
||||||
})
|
})
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -109,7 +109,7 @@ on Linux), then /etc/vaultik/config.yml.`,
|
|||||||
return v.CreateSnapshot(opts)
|
return v.CreateSnapshot(opts)
|
||||||
}, func(err error) {
|
}, func(err error) {
|
||||||
if errors.Is(err, vaultik.ErrPostBackupPruneFailed) {
|
if errors.Is(err, vaultik.ErrPostBackupPruneFailed) {
|
||||||
log.Error("Post-backup prune failed", "error", err)
|
log.Info("Post-backup prune failed", "error", err)
|
||||||
ReportErrorf("Backup completed, but %v", err)
|
ReportErrorf("Backup completed, but %v", err)
|
||||||
|
|
||||||
return
|
return
|
||||||
@@ -118,13 +118,13 @@ on Linux), then /etc/vaultik/config.yml.`,
|
|||||||
if errors.Is(err, vaultik.ErrFilesUnreadable) ||
|
if errors.Is(err, vaultik.ErrFilesUnreadable) ||
|
||||||
errors.Is(err, vaultik.ErrSourcePathEmpty) ||
|
errors.Is(err, vaultik.ErrSourcePathEmpty) ||
|
||||||
errors.Is(err, vaultik.ErrVolumeNotMounted) {
|
errors.Is(err, vaultik.ErrVolumeNotMounted) {
|
||||||
log.Error("Backup completed with an error", "error", err)
|
log.Info("Backup completed with an error", "error", err)
|
||||||
ReportErrorf("Backup completed, but %v", err)
|
ReportErrorf("Backup completed, but %v", err)
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
log.Error("Snapshot creation failed", "error", err)
|
log.Info("Snapshot creation failed", "error", err)
|
||||||
ReportErrorf("Snapshot creation failed: %v", err)
|
ReportErrorf("Snapshot creation failed: %v", err)
|
||||||
})
|
})
|
||||||
},
|
},
|
||||||
@@ -258,7 +258,7 @@ func newSnapshotVerifyCommand() *cobra.Command {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
log.Error("Verification failed", "error", err)
|
log.Info("Verification failed", "error", err)
|
||||||
ReportErrorf("Verification failed: %v", err)
|
ReportErrorf("Verification failed: %v", err)
|
||||||
})
|
})
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -104,7 +104,7 @@ func runRestore(cmd *cobra.Command, args []string, opts *RestoreOptions) error {
|
|||||||
SkipErrors: rootFlags.SkipErrors,
|
SkipErrors: rootFlags.SkipErrors,
|
||||||
})
|
})
|
||||||
}, func(err error) {
|
}, func(err error) {
|
||||||
log.Error("Restore operation failed", "error", err)
|
log.Info("Restore operation failed", "error", err)
|
||||||
ReportErrorf("Restore failed: %v", err)
|
ReportErrorf("Restore failed: %v", err)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
+72
-15
@@ -9,6 +9,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
@@ -69,14 +70,22 @@ var (
|
|||||||
errFileURLNotAbsolute = errors.New(
|
errFileURLNotAbsolute = errors.New(
|
||||||
"a file:// storage_url needs an absolute path, " +
|
"a file:// storage_url needs an absolute path, " +
|
||||||
"as in file:///path/to/backup")
|
"as in file:///path/to/backup")
|
||||||
|
errIndexPathNotAbsolute = errors.New(
|
||||||
|
"index_path and VAULTIK_INDEX_PATH need an absolute path, " +
|
||||||
|
"as in ~/.local/share/vaultik/index.sqlite")
|
||||||
errNegativePruneGracePeriod = errors.New(
|
errNegativePruneGracePeriod = errors.New(
|
||||||
"prune_grace_period must not be negative")
|
"prune_grace_period must not be negative")
|
||||||
errStorageNotConfigured = errors.New(
|
errStorageNotConfigured = errors.New(
|
||||||
"storage not configured; set storage_url or provide s3.endpoint + " +
|
"storage not configured; set storage_url or provide s3.endpoint + " +
|
||||||
"s3.bucket + credentials")
|
"s3.bucket + credentials")
|
||||||
errS3BucketRequired = errors.New("s3.bucket is required (or set storage_url)")
|
errS3BucketRequired = errors.New("s3.bucket is required (or set storage_url)")
|
||||||
errS3KeyIDRequired = errors.New("s3.access_key_id is required")
|
)
|
||||||
errS3SecretRequired = errors.New("s3.secret_access_key is required")
|
|
||||||
|
// Validate returns these when S3 storage is configured without its
|
||||||
|
// credentials.
|
||||||
|
var (
|
||||||
|
ErrS3KeyIDRequired = errors.New("s3.access_key_id is required")
|
||||||
|
ErrS3SecretRequired = errors.New("s3.secret_access_key is required")
|
||||||
)
|
)
|
||||||
|
|
||||||
// expandTilde expands ~ at the start of a path to the user's home directory.
|
// expandTilde expands ~ at the start of a path to the user's home directory.
|
||||||
@@ -286,15 +295,39 @@ func Load(path string) (*Config, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Parse reads YAML config data as Load does, without validating it. It
|
// Parse reads YAML config data as Load does, without validating it. It
|
||||||
// applies default values for optional fields, fills in ${...} references
|
// refuses a key that no setting has and a size setting whose value is not a
|
||||||
// using smartconfig, and expands ~ in paths.
|
// size, naming the setting. It applies default values for optional
|
||||||
|
// fields, fills in ${...} references using smartconfig, and expands ~ in
|
||||||
|
// paths.
|
||||||
func Parse(data []byte) (*Config, error) {
|
func Parse(data []byte) (*Config, error) {
|
||||||
|
// smartconfig drops line numbers, so keys are checked in the data as
|
||||||
|
// written.
|
||||||
|
var root yaml.Node
|
||||||
|
|
||||||
|
err := yaml.Unmarshal(data, &root)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to parse config: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = checkKeys(&root, reflect.TypeFor[Config](), "",
|
||||||
|
map[*yaml.Node]bool{}, map[nodeAndType]bool{})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
// Load config using smartconfig for interpolation
|
// Load config using smartconfig for interpolation
|
||||||
sc, err := smartconfig.NewFromReader(bytes.NewReader(data))
|
sc, err := smartconfig.NewFromReader(bytes.NewReader(data))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to load config file: %w", err)
|
return nil, fmt.Errorf("failed to load config file: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A ${...} reference can bring in keys that are not in the file, and
|
||||||
|
// is the value of a size setting only once it is filled in.
|
||||||
|
err = checkFilledConfig(sc.Data())
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
cfg := &Config{
|
cfg := &Config{
|
||||||
// Set defaults
|
// Set defaults
|
||||||
BlobSizeLimit: defaultBlobSizeLimit,
|
BlobSizeLimit: defaultBlobSizeLimit,
|
||||||
@@ -363,7 +396,11 @@ func Parse(data []byte) (*Config, error) {
|
|||||||
// never the value. An empty list is accepted, because only snapshot
|
// never the value. An empty list is accepted, because only snapshot
|
||||||
// create needs a recipient and it checks for one itself
|
// create needs a recipient and it checks for one itself
|
||||||
// - At least one snapshot must be configured with at least one path
|
// - At least one snapshot must be configured with at least one path
|
||||||
|
// - Every exclude pattern, global or in a snapshot, must compile; errors
|
||||||
|
// name the setting and quote the pattern
|
||||||
// - Storage must be configured (either storage_url or s3.* fields)
|
// - Storage must be configured (either storage_url or s3.* fields)
|
||||||
|
// - The index path must be absolute: a relative one would resolve against
|
||||||
|
// the working directory, so each directory would get its own index
|
||||||
// - Chunk size must be at least 1MB
|
// - Chunk size must be at least 1MB
|
||||||
// - Blob size limit must be at least the largest chunk the chunker can emit
|
// - Blob size limit must be at least the largest chunk the chunker can emit
|
||||||
// (chunk_size times chunker.ChunkSizeSpread), so a single-chunk blob never
|
// (chunk_size times chunker.ChunkSizeSpread), so a single-chunk blob never
|
||||||
@@ -381,22 +418,26 @@ func (c *Config) Validate() error {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(c.Snapshots) == 0 {
|
err := c.validateSnapshots()
|
||||||
return errNoSnapshots
|
if err != nil {
|
||||||
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
for name, snap := range c.Snapshots {
|
err = c.validateExcludes()
|
||||||
if len(snap.Paths) == 0 {
|
if err != nil {
|
||||||
return fmt.Errorf("%w: %q", errSnapshotNoPaths, name)
|
return err
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Validate storage configuration
|
// Validate storage configuration
|
||||||
err := c.validateStorage()
|
err = c.validateStorage()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if !filepath.IsAbs(c.IndexPath) {
|
||||||
|
return fmt.Errorf("%w: %q", errIndexPathNotAbsolute, c.IndexPath)
|
||||||
|
}
|
||||||
|
|
||||||
if c.ChunkSize.Int64() < minChunkSize {
|
if c.ChunkSize.Int64() < minChunkSize {
|
||||||
return errChunkSizeTooSmall
|
return errChunkSizeTooSmall
|
||||||
}
|
}
|
||||||
@@ -427,6 +468,22 @@ func (c *Config) Validate() error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// validateSnapshots refuses a config with no snapshot or with a snapshot
|
||||||
|
// that has no path.
|
||||||
|
func (c *Config) validateSnapshots() error {
|
||||||
|
if len(c.Snapshots) == 0 {
|
||||||
|
return errNoSnapshots
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, snap := range c.Snapshots {
|
||||||
|
if len(snap.Paths) == 0 {
|
||||||
|
return fmt.Errorf("%w: %q", errSnapshotNoPaths, name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// validateAgeRecipient parses one age_recipients entry with the age library
|
// validateAgeRecipient parses one age_recipients entry with the age library
|
||||||
// and returns a value-free error on failure. A recipient string can be
|
// and returns a value-free error on failure. A recipient string can be
|
||||||
// sensitive (an operator may paste a secret key by mistake), so neither the
|
// sensitive (an operator may paste a secret key by mistake), so neither the
|
||||||
@@ -490,11 +547,11 @@ func (c *Config) validateStorage() error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if c.S3.AccessKeyID == "" {
|
if c.S3.AccessKeyID == "" {
|
||||||
return errS3KeyIDRequired
|
return ErrS3KeyIDRequired
|
||||||
}
|
}
|
||||||
|
|
||||||
if c.S3.SecretAccessKey == "" {
|
if c.S3.SecretAccessKey == "" {
|
||||||
return errS3SecretRequired
|
return ErrS3SecretRequired
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
@@ -521,11 +578,11 @@ func (c *Config) validateStorageURL() error {
|
|||||||
case strings.HasPrefix(c.StorageURL, "s3://"):
|
case strings.HasPrefix(c.StorageURL, "s3://"):
|
||||||
// S3 storage needs credentials
|
// S3 storage needs credentials
|
||||||
if c.S3.AccessKeyID == "" {
|
if c.S3.AccessKeyID == "" {
|
||||||
return fmt.Errorf("%w for s3:// URLs", errS3KeyIDRequired)
|
return fmt.Errorf("%w for s3:// URLs", ErrS3KeyIDRequired)
|
||||||
}
|
}
|
||||||
|
|
||||||
if c.S3.SecretAccessKey == "" {
|
if c.S3.SecretAccessKey == "" {
|
||||||
return fmt.Errorf("%w for s3:// URLs", errS3SecretRequired)
|
return fmt.Errorf("%w for s3:// URLs", ErrS3SecretRequired)
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -165,6 +165,7 @@ func TestValidateBlobSizeLimit(t *testing.T) {
|
|||||||
AgeRecipients: []string{testSneakAgePublicKey},
|
AgeRecipients: []string{testSneakAgePublicKey},
|
||||||
Snapshots: map[string]SnapshotConfig{"test": {Paths: []string{"/tmp/src"}}},
|
Snapshots: map[string]SnapshotConfig{"test": {Paths: []string{"/tmp/src"}}},
|
||||||
StorageURL: "file:///tmp/vaultik-test-store",
|
StorageURL: "file:///tmp/vaultik-test-store",
|
||||||
|
IndexPath: "/tmp/vaultik-test-index.sqlite",
|
||||||
ChunkSize: chunkSize,
|
ChunkSize: chunkSize,
|
||||||
BlobSizeLimit: blobLimit,
|
BlobSizeLimit: blobLimit,
|
||||||
CompressionLevel: 3,
|
CompressionLevel: 3,
|
||||||
@@ -510,6 +511,65 @@ func TestLoadRejectsRelativeFileURL(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestLoadIndexPath checks that a relative index_path fails at load, naming
|
||||||
|
// the value. Accepted, it would resolve against the working directory, so a
|
||||||
|
// backup run from another directory would start from an empty index. A path
|
||||||
|
// starting with ~ is expanded and loads.
|
||||||
|
func TestLoadIndexPath(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
home, err := os.UserHomeDir()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("home directory: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
load := func(t *testing.T, indexPath string) (*Config, error) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
data := "snapshots:\n" +
|
||||||
|
" test:\n" +
|
||||||
|
" paths: [/tmp/vaultik-test-source]\n" +
|
||||||
|
"storage_url: file:///tmp/vaultik-test-storage\n" +
|
||||||
|
"index_path: " + indexPath + "\n"
|
||||||
|
|
||||||
|
path := filepath.Join(t.TempDir(), "config.yml")
|
||||||
|
|
||||||
|
err := os.WriteFile(path, []byte(data), 0o600)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("write config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Load(path)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Run("relative path is rejected", func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, err := load(t, "relative/index.sqlite")
|
||||||
|
if !errors.Is(err, errIndexPathNotAbsolute) {
|
||||||
|
t.Fatalf("Load() error = %v, want errIndexPathNotAbsolute", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(err.Error(), `"relative/index.sqlite"`) {
|
||||||
|
t.Errorf("error %q does not name the value", err.Error())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("path starting with ~ is expanded", func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cfg, err := load(t, "~/vaultik/index.sqlite")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := filepath.Join(home, "vaultik", "index.sqlite")
|
||||||
|
if cfg.IndexPath != want {
|
||||||
|
t.Errorf("IndexPath = %q, want %q", cfg.IndexPath, want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
// TestValidateAgeRecipients checks that recipients are parsed at config load
|
// TestValidateAgeRecipients checks that recipients are parsed at config load
|
||||||
// (a bad entry fails immediately, not mid-backup) and that no invalid entry —
|
// (a bad entry fails immediately, not mid-backup) and that no invalid entry —
|
||||||
// least of all a pasted secret key — is echoed in the error. An empty list
|
// least of all a pasted secret key — is echoed in the error. An empty list
|
||||||
@@ -522,6 +582,7 @@ func TestValidateAgeRecipients(t *testing.T) {
|
|||||||
AgeRecipients: recipients,
|
AgeRecipients: recipients,
|
||||||
Snapshots: map[string]SnapshotConfig{"test": {Paths: []string{"/tmp/src"}}},
|
Snapshots: map[string]SnapshotConfig{"test": {Paths: []string{"/tmp/src"}}},
|
||||||
StorageURL: "file:///tmp/vaultik-test-store",
|
StorageURL: "file:///tmp/vaultik-test-store",
|
||||||
|
IndexPath: "/tmp/vaultik-test-index.sqlite",
|
||||||
ChunkSize: Size(10 * 1024 * 1024),
|
ChunkSize: Size(10 * 1024 * 1024),
|
||||||
BlobSizeLimit: Size(10 * 1024 * 1024 * 1024),
|
BlobSizeLimit: Size(10 * 1024 * 1024 * 1024),
|
||||||
CompressionLevel: 3,
|
CompressionLevel: 3,
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
package config
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"sneak.berlin/go/vaultik/internal/exclude"
|
||||||
|
)
|
||||||
|
|
||||||
|
// validateExcludes refuses an exclude pattern, global or in a snapshot,
|
||||||
|
// that does not compile, naming the setting and quoting the pattern.
|
||||||
|
func (c *Config) validateExcludes() error {
|
||||||
|
err := validateExcludeList("exclude", c.Exclude)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, name := range c.SnapshotNames() {
|
||||||
|
err = validateExcludeList("snapshots."+name+".exclude",
|
||||||
|
c.Snapshots[name].Exclude)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// validateExcludeList refuses the first pattern in patterns that does not
|
||||||
|
// compile. setting is the name of the list, as in "exclude".
|
||||||
|
func validateExcludeList(setting string, patterns []string) error {
|
||||||
|
for i, pattern := range patterns {
|
||||||
|
_, _, err := exclude.Compile(pattern)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("%s[%d]: %w", setting, i, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
package config_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"sneak.berlin/go/vaultik/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestLoadRefusesBadExcludePattern checks that an exclude pattern that does
|
||||||
|
// not compile, global or in a snapshot, fails at load with the setting and
|
||||||
|
// the pattern named, rather than the backup storing what it meant to leave
|
||||||
|
// out.
|
||||||
|
func TestLoadRefusesBadExcludePattern(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const storage = "storage_url: file:///tmp/vaultik-test-storage\n"
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
config string
|
||||||
|
wantErr string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "global",
|
||||||
|
config: "exclude: ['*.tmp', '.cache', 'build[']\n" +
|
||||||
|
"snapshots:\n" +
|
||||||
|
" home:\n" +
|
||||||
|
" paths: [/tmp/vaultik-test-source]\n" + storage,
|
||||||
|
wantErr: `exclude[2]: "build[" is not a valid exclude pattern`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "snapshot",
|
||||||
|
config: "snapshots:\n" +
|
||||||
|
" home:\n" +
|
||||||
|
" paths: [/tmp/vaultik-test-source]\n" +
|
||||||
|
" exclude: ['/a/**/x[']\n" + storage,
|
||||||
|
wantErr: `snapshots.home.exclude[0]: "/a/**/x[" is not a valid exclude pattern`,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
path := filepath.Join(t.TempDir(), "config.yml")
|
||||||
|
|
||||||
|
err := os.WriteFile(path, []byte(tt.config), 0o600)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("write config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = config.Load(path)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("Load() returned nil, want an error containing %q", tt.wantErr)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||||
|
t.Fatalf("Load() error = %q, want it to contain %q", err, tt.wantErr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,220 @@
|
|||||||
|
package config
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"reflect"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gopkg.in/yaml.v3"
|
||||||
|
)
|
||||||
|
|
||||||
|
// yamlMergeTag is the tag YAML gives a merge key (<<), which adds the keys
|
||||||
|
// of the map it names to the map it sits in.
|
||||||
|
const yamlMergeTag = "!!merge"
|
||||||
|
|
||||||
|
// EnvSection is the top-level key of a map of environment variables that
|
||||||
|
// smartconfig sets while it loads the config. It is not a vaultik setting,
|
||||||
|
// so its keys are not checked.
|
||||||
|
const EnvSection = "env"
|
||||||
|
|
||||||
|
var (
|
||||||
|
errUnknownKey = errors.New("unknown config key")
|
||||||
|
errAnchorContainsItself = errors.New("anchor contains itself")
|
||||||
|
)
|
||||||
|
|
||||||
|
// nodeAndType is a node of a parsed config and the setting type it is
|
||||||
|
// checked against.
|
||||||
|
type nodeAndType struct {
|
||||||
|
node *yaml.Node
|
||||||
|
typ reflect.Type
|
||||||
|
}
|
||||||
|
|
||||||
|
// SettingType returns the type of the setting that the dotted key names in
|
||||||
|
// Config, following the fields' yaml tags, as for s3.bucket or
|
||||||
|
// snapshots.home.exclude.0. A snapshot name or a list index may be any
|
||||||
|
// word. It returns an unknown key error when no setting has that key.
|
||||||
|
func SettingType(keys []string) (reflect.Type, error) {
|
||||||
|
typ := reflect.TypeFor[Config]()
|
||||||
|
|
||||||
|
for _, key := range keys {
|
||||||
|
switch {
|
||||||
|
case typ.Kind() == reflect.Map || typ.Kind() == reflect.Slice:
|
||||||
|
typ = typ.Elem()
|
||||||
|
case typ.Kind() == reflect.Struct:
|
||||||
|
field, ok := yamlField(typ, key)
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("%w %q", errUnknownKey, strings.Join(keys, "."))
|
||||||
|
}
|
||||||
|
|
||||||
|
typ = field.Type
|
||||||
|
default:
|
||||||
|
return nil, fmt.Errorf("%w %q", errUnknownKey, strings.Join(keys, "."))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return typ, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// yamlField returns the field of struct type typ whose yaml tag names key.
|
||||||
|
func yamlField(typ reflect.Type, key string) (reflect.StructField, bool) {
|
||||||
|
for field := range typ.Fields() {
|
||||||
|
name, _, _ := strings.Cut(field.Tag.Get("yaml"), ",")
|
||||||
|
if name == key && name != "-" {
|
||||||
|
return field, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return reflect.StructField{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkKeys returns an error naming the first key in node, a parsed config
|
||||||
|
// or part of one at the dotted path, that no setting in typ has. The error
|
||||||
|
// gives the key's line in the file, when it has one. A value of the wrong
|
||||||
|
// type is left for the YAML decoder to report. following holds the aliases
|
||||||
|
// the walk is inside. Meeting one of them again means its anchor contains
|
||||||
|
// itself, which is an error. checked holds each anchored node the walk has
|
||||||
|
// checked completely, with the type it was checked against. An alias to
|
||||||
|
// that node is not followed again for that type.
|
||||||
|
func checkKeys(node *yaml.Node, typ reflect.Type, path string,
|
||||||
|
following map[*yaml.Node]bool, checked map[nodeAndType]bool,
|
||||||
|
) error {
|
||||||
|
switch node.Kind {
|
||||||
|
case yaml.DocumentNode:
|
||||||
|
for _, child := range node.Content {
|
||||||
|
err := checkKeys(child, typ, path, following, checked)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case yaml.AliasNode:
|
||||||
|
if following[node] {
|
||||||
|
return fmt.Errorf("%w: %q on line %d",
|
||||||
|
errAnchorContainsItself, node.Value, node.Line)
|
||||||
|
}
|
||||||
|
|
||||||
|
target := nodeAndType{node.Alias, typ}
|
||||||
|
if checked[target] {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
following[node] = true
|
||||||
|
defer delete(following, node)
|
||||||
|
|
||||||
|
err := checkKeys(node.Alias, typ, path, following, checked)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
checked[target] = true
|
||||||
|
case yaml.SequenceNode:
|
||||||
|
if typ.Kind() != reflect.Slice {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, child := range node.Content {
|
||||||
|
err := checkKeys(child, typ.Elem(), joinKey(path, strconv.Itoa(i)),
|
||||||
|
following, checked)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case yaml.MappingNode:
|
||||||
|
return checkMappingKeys(node, typ, path, following, checked)
|
||||||
|
case yaml.ScalarNode:
|
||||||
|
// A scalar has no keys.
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkFilledConfig checks data, the config after smartconfig has filled in
|
||||||
|
// its ${...} references: its keys with checkKeys, then its size settings
|
||||||
|
// with checkSizes. A key that a reference brings in is not in the file, and
|
||||||
|
// the copy that Encode makes has no line numbers.
|
||||||
|
func checkFilledConfig(data map[string]any) error {
|
||||||
|
var filled yaml.Node
|
||||||
|
|
||||||
|
err := filled.Encode(data)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to marshal config data: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = checkKeys(&filled, reflect.TypeFor[Config](), "",
|
||||||
|
map[*yaml.Node]bool{}, map[nodeAndType]bool{})
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
return checkSizes(&filled, nil)
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkMappingKeys is checkKeys for a map.
|
||||||
|
func checkMappingKeys(node *yaml.Node, typ reflect.Type, path string,
|
||||||
|
following map[*yaml.Node]bool, checked map[nodeAndType]bool,
|
||||||
|
) error {
|
||||||
|
for i := 0; i+1 < len(node.Content); i += 2 {
|
||||||
|
key, value := node.Content[i], node.Content[i+1]
|
||||||
|
|
||||||
|
if key.Tag == yamlMergeTag {
|
||||||
|
// The value is one map or a list of maps.
|
||||||
|
merged := []*yaml.Node{value}
|
||||||
|
if value.Kind == yaml.SequenceNode {
|
||||||
|
merged = value.Content
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, m := range merged {
|
||||||
|
err := checkKeys(m, typ, path, following, checked)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
keyPath := joinKey(path, key.Value)
|
||||||
|
if keyPath == EnvSection {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
var valueType reflect.Type
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case typ.Kind() == reflect.Map:
|
||||||
|
valueType = typ.Elem()
|
||||||
|
case typ.Kind() == reflect.Struct:
|
||||||
|
field, ok := yamlField(typ, key.Value)
|
||||||
|
if !ok && key.Line == 0 {
|
||||||
|
// A key that a ${...} reference brought in has no line.
|
||||||
|
return fmt.Errorf("%w %q", errUnknownKey, keyPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !ok {
|
||||||
|
return fmt.Errorf("%w %q on line %d",
|
||||||
|
errUnknownKey, keyPath, key.Line)
|
||||||
|
}
|
||||||
|
|
||||||
|
valueType = field.Type
|
||||||
|
default:
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
err := checkKeys(value, valueType, keyPath, following, checked)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// joinKey appends key to the dotted path.
|
||||||
|
func joinKey(path, key string) string {
|
||||||
|
if path == "" {
|
||||||
|
return key
|
||||||
|
}
|
||||||
|
|
||||||
|
return path + "." + key
|
||||||
|
}
|
||||||
@@ -0,0 +1,163 @@
|
|||||||
|
package config //nolint:testpackage // checks the unexported errUnknownKey
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"slices"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// keysTestConfig loads; each case below adds one line to it.
|
||||||
|
const keysTestConfig = "storage_url: file:///tmp/vaultik-test-storage\n" +
|
||||||
|
"s3:\n" +
|
||||||
|
" region: us-east-1\n" +
|
||||||
|
"snapshots:\n" +
|
||||||
|
" test:\n" +
|
||||||
|
" paths: [/tmp/vaultik-test-source]\n"
|
||||||
|
|
||||||
|
// TestParseRefusesUnknownKey checks that a key no setting has, at any
|
||||||
|
// depth, fails the parse with an error naming the key and its line. A key
|
||||||
|
// that a ${...} reference brings in is not in the file, so its error gives
|
||||||
|
// no line.
|
||||||
|
func TestParseRefusesUnknownKey(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
snapshotPath := filepath.Join(t.TempDir(), "snapshot.yml")
|
||||||
|
|
||||||
|
err := os.WriteFile(snapshotPath, []byte("other:\n"+
|
||||||
|
" paths: [/tmp/vaultik-test-source]\n"+
|
||||||
|
" exlude: [/tmp/cache]\n"), 0o600)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("write snapshot: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
yaml string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{"top level", keysTestConfig + "compression_lvl: 9\n",
|
||||||
|
`unknown config key "compression_lvl" on line 7`},
|
||||||
|
{"in a snapshot", keysTestConfig + " exlude: [/tmp/cache]\n",
|
||||||
|
`unknown config key "snapshots.test.exlude" on line 7`},
|
||||||
|
{"in s3", strings.Replace(keysTestConfig, " region:",
|
||||||
|
" bukcet: b\n region:", 1),
|
||||||
|
`unknown config key "s3.bukcet" on line 3`},
|
||||||
|
{"from a reference",
|
||||||
|
keysTestConfig + " other: ${YAML:" + snapshotPath + ":other}\n",
|
||||||
|
`unknown config key "snapshots.other.exlude"`},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, err := Parse([]byte(tt.yaml))
|
||||||
|
if !errors.Is(err, errUnknownKey) {
|
||||||
|
t.Fatalf("Parse() error = %v, want errUnknownKey", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err.Error() != tt.want {
|
||||||
|
t.Errorf("Parse() error = %q, want %q", err.Error(), tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestParseRefusesAnchorThatContainsItself checks that a YAML merge key (<<)
|
||||||
|
// naming the anchor of the map it sits in fails the parse with an error.
|
||||||
|
func TestParseRefusesAnchorThatContainsItself(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
data := strings.Replace(keysTestConfig, " test:", " test: &test", 1) +
|
||||||
|
" <<: *test\n"
|
||||||
|
|
||||||
|
_, err := Parse([]byte(data))
|
||||||
|
if !errors.Is(err, errAnchorContainsItself) {
|
||||||
|
t.Fatalf("Parse() error = %v, want errAnchorContainsItself", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := `anchor contains itself: "test" on line 7`
|
||||||
|
if err.Error() != want {
|
||||||
|
t.Errorf("Parse() error = %q, want %q", err.Error(), want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestParseRefusesExcessiveAliasing checks that Parse returns an error for
|
||||||
|
// a file whose merge keys (<<) fan out. Each of eight snapshots merges the
|
||||||
|
// one before it ten times. A key check that followed every alias would walk
|
||||||
|
// the first snapshot over 10^8 times and run past the package's test
|
||||||
|
// timeout.
|
||||||
|
func TestParseRefusesExcessiveAliasing(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const snapshots, merges = 8, 10
|
||||||
|
|
||||||
|
var data strings.Builder
|
||||||
|
|
||||||
|
data.WriteString(strings.Replace(keysTestConfig, " test:", " test: &s0", 1))
|
||||||
|
|
||||||
|
for i := 1; i <= snapshots; i++ {
|
||||||
|
aliases := slices.Repeat([]string{"*s" + strconv.Itoa(i-1)}, merges)
|
||||||
|
fmt.Fprintf(&data, " s%d: &s%d\n <<: [%s]\n",
|
||||||
|
i, i, strings.Join(aliases, ", "))
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := Parse([]byte(data.String()))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Parse() returned no error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestParseAcceptsKnownKeys checks that a config with a ${...} reference
|
||||||
|
// and a YAML merge key (<<) still parses.
|
||||||
|
func TestParseAcceptsKnownKeys(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
levelPath := filepath.Join(t.TempDir(), "level")
|
||||||
|
|
||||||
|
err := os.WriteFile(levelPath, []byte("9"), 0o600)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("write level: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
data := strings.Replace(keysTestConfig, " test:", " test: &test", 1) +
|
||||||
|
" other:\n" +
|
||||||
|
" <<: *test\n" +
|
||||||
|
" exclude: [/tmp/vaultik-test-source/cache]\n" +
|
||||||
|
"compression_level: ${FILE:" + levelPath + "}\n"
|
||||||
|
|
||||||
|
cfg, err := Parse([]byte(data))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cfg.CompressionLevel != 9 {
|
||||||
|
t.Errorf("compression_level = %d, want 9", cfg.CompressionLevel)
|
||||||
|
}
|
||||||
|
|
||||||
|
other := cfg.Snapshots["other"]
|
||||||
|
if len(other.Paths) != 1 || len(other.Exclude) != 1 {
|
||||||
|
t.Errorf("snapshot other = %+v, want the paths of test and one exclude",
|
||||||
|
other)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestParseAcceptsEnvSection checks that a config with a top-level env
|
||||||
|
// section still parses. smartconfig sets the environment variables it
|
||||||
|
// lists, so its keys are not settings.
|
||||||
|
func TestParseAcceptsEnvSection(t *testing.T) {
|
||||||
|
const name = "VAULTIK_TEST_ENV_SECTION"
|
||||||
|
|
||||||
|
// Parse sets the variable; t.Setenv restores it after the test.
|
||||||
|
t.Setenv(name, "")
|
||||||
|
|
||||||
|
_, err := Parse([]byte(keysTestConfig + "env:\n " + name + ": set\n"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
+77
-11
@@ -4,13 +4,20 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"math"
|
"math"
|
||||||
|
"reflect"
|
||||||
|
"slices"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
|
||||||
"github.com/dustin/go-humanize"
|
"github.com/dustin/go-humanize"
|
||||||
|
"gopkg.in/yaml.v3"
|
||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
errSizeType = errors.New("size must be a number or string")
|
errSizeType = errors.New("size must be a number or string")
|
||||||
errSizeTooLarge = errors.New("size exceeds maximum supported value")
|
errSizeTooLarge = errors.New("size exceeds maximum supported value")
|
||||||
|
errNotASize = errors.New(
|
||||||
|
"is not a size; use a number of bytes or a unit such as 10MB")
|
||||||
)
|
)
|
||||||
|
|
||||||
// Size represents a byte size that can be specified in configuration files.
|
// Size represents a byte size that can be specified in configuration files.
|
||||||
@@ -32,6 +39,15 @@ func (s *Size) UnmarshalYAML(unmarshal func(any) error) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A number too large for an int64 decodes as a float64. Its text can be
|
||||||
|
// 1e+20, which is not a size string.
|
||||||
|
var floatVal float64
|
||||||
|
|
||||||
|
err = unmarshal(&floatVal)
|
||||||
|
if err == nil && floatVal > math.MaxInt64 {
|
||||||
|
return errSizeTooLarge
|
||||||
|
}
|
||||||
|
|
||||||
// Try to unmarshal as string
|
// Try to unmarshal as string
|
||||||
var strVal string
|
var strVal string
|
||||||
|
|
||||||
@@ -40,17 +56,66 @@ func (s *Size) UnmarshalYAML(unmarshal func(any) error) error {
|
|||||||
return errSizeType
|
return errSizeType
|
||||||
}
|
}
|
||||||
|
|
||||||
// Parse the string using go-humanize
|
size, err := ParseSize(strVal)
|
||||||
bytes, err := humanize.ParseBytes(strVal)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("invalid size format: %w", err)
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
if bytes > math.MaxInt64 {
|
*s = size
|
||||||
return fmt.Errorf("%w: %s", errSizeTooLarge, strVal)
|
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
*s = Size(bytes)
|
// checkSizes returns an error naming the first size setting in node whose
|
||||||
|
// value is not a size. node is the config, or the part of it at the dotted
|
||||||
|
// keys, after smartconfig has filled in its ${...} references.
|
||||||
|
func checkSizes(node *yaml.Node, keys []string) error {
|
||||||
|
typ, err := SettingType(keys)
|
||||||
|
if err != nil {
|
||||||
|
// Unknown keys are refused before this check, so keys is in one of
|
||||||
|
// two places. The env section holds no settings. A value of the
|
||||||
|
// wrong shape, such as a map given for hostname or inside exclude,
|
||||||
|
// is reported later by the YAML decoder.
|
||||||
|
return nil //nolint:nilerr // not an error here
|
||||||
|
}
|
||||||
|
|
||||||
|
if typ == reflect.TypeFor[Size]() {
|
||||||
|
var size Size
|
||||||
|
|
||||||
|
err = node.Decode(&size)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("%s: %w", strings.Join(keys, "."), err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
switch node.Kind {
|
||||||
|
case yaml.DocumentNode:
|
||||||
|
for _, child := range node.Content {
|
||||||
|
err = checkSizes(child, keys)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case yaml.MappingNode:
|
||||||
|
for i := 0; i+1 < len(node.Content); i += 2 {
|
||||||
|
err = checkSizes(node.Content[i+1],
|
||||||
|
slices.Concat(keys, []string{node.Content[i].Value}))
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case yaml.SequenceNode:
|
||||||
|
for i, child := range node.Content {
|
||||||
|
err = checkSizes(child, slices.Concat(keys, []string{strconv.Itoa(i)}))
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case yaml.ScalarNode, yaml.AliasNode:
|
||||||
|
// A scalar holds no settings, and the filled config has no aliases.
|
||||||
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -70,16 +135,17 @@ func (s Size) String() string {
|
|||||||
return humanize.Bytes(uint64(s))
|
return humanize.Bytes(uint64(s))
|
||||||
}
|
}
|
||||||
|
|
||||||
// ParseSize parses a size string into a Size value
|
// ParseSize parses a size string, such as "10MB", into a Size value. An
|
||||||
|
// error quotes s.
|
||||||
func ParseSize(s string) (Size, error) {
|
func ParseSize(s string) (Size, error) {
|
||||||
bytes, err := humanize.ParseBytes(s)
|
bytes, err := humanize.ParseBigBytes(s)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, fmt.Errorf("invalid size format: %w", err)
|
return 0, fmt.Errorf("%q %w", s, errNotASize)
|
||||||
}
|
}
|
||||||
|
|
||||||
if bytes > math.MaxInt64 {
|
if !bytes.IsInt64() {
|
||||||
return 0, fmt.Errorf("%w: %s", errSizeTooLarge, s)
|
return 0, fmt.Errorf("%w: %s", errSizeTooLarge, s)
|
||||||
}
|
}
|
||||||
|
|
||||||
return Size(bytes), nil
|
return Size(bytes.Int64()), nil
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
package config //nolint:testpackage // checks errNotASize and errSizeTooLarge
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestParseRefusesValueThatIsNotASize checks that a size setting whose
|
||||||
|
// value is not a size fails the parse with an error naming the setting and
|
||||||
|
// quoting the value, also when a ${...} reference fills the value in.
|
||||||
|
func TestParseRefusesValueThatIsNotASize(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
sizePath := filepath.Join(t.TempDir(), "size")
|
||||||
|
|
||||||
|
err := os.WriteFile(sizePath, []byte("lots"), 0o600)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("write size: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
yaml string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{"chunk_size", keysTestConfig + "chunk_size: banana\n",
|
||||||
|
`chunk_size: "banana" is not a size; ` +
|
||||||
|
"use a number of bytes or a unit such as 10MB"},
|
||||||
|
{"blob_size_limit", keysTestConfig + "blob_size_limit: 10XB\n",
|
||||||
|
`blob_size_limit: "10XB" is not a size; ` +
|
||||||
|
"use a number of bytes or a unit such as 10MB"},
|
||||||
|
{"s3.part_size", strings.Replace(keysTestConfig, " region:",
|
||||||
|
" part_size: big\n region:", 1),
|
||||||
|
`s3.part_size: "big" is not a size; ` +
|
||||||
|
"use a number of bytes or a unit such as 10MB"},
|
||||||
|
{"from a reference",
|
||||||
|
keysTestConfig + "chunk_size: ${FILE:" + sizePath + "}\n",
|
||||||
|
`chunk_size: "lots" is not a size; ` +
|
||||||
|
"use a number of bytes or a unit such as 10MB"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, err := Parse([]byte(tt.yaml))
|
||||||
|
if !errors.Is(err, errNotASize) {
|
||||||
|
t.Fatalf("Parse() error = %v, want errNotASize", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err.Error() != tt.want {
|
||||||
|
t.Errorf("Parse() error = %q, want %q", err.Error(), tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestParseRefusesSizeTooLarge checks that a size setting whose value is
|
||||||
|
// too large for a size fails the parse with an error naming the setting.
|
||||||
|
// YAML reads the second value as a float.
|
||||||
|
func TestParseRefusesSizeTooLarge(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
value string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{"100EB", "chunk_size: size exceeds maximum supported value: 100EB"},
|
||||||
|
{"99999999999999999999",
|
||||||
|
"chunk_size: size exceeds maximum supported value"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.value, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, err := Parse([]byte(keysTestConfig + "chunk_size: " + tt.value + "\n"))
|
||||||
|
if !errors.Is(err, errSizeTooLarge) {
|
||||||
|
t.Fatalf("Parse() error = %v, want errSizeTooLarge", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err.Error() != tt.want {
|
||||||
|
t.Errorf("Parse() error = %q, want %q", err.Error(), tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestParseAcceptsSizes checks that size settings given as a number of
|
||||||
|
// bytes, as a unit, and through a ${...} reference still parse.
|
||||||
|
func TestParseAcceptsSizes(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
sizePath := filepath.Join(t.TempDir(), "size")
|
||||||
|
|
||||||
|
err := os.WriteFile(sizePath, []byte("20MB"), 0o600)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("write size: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
data := strings.Replace(keysTestConfig, " region:",
|
||||||
|
" part_size: 6000000\n region:", 1) +
|
||||||
|
"chunk_size: ${FILE:" + sizePath + "}\n" +
|
||||||
|
"blob_size_limit: 1GB\n"
|
||||||
|
|
||||||
|
cfg, err := Parse([]byte(data))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cfg.ChunkSize != 20_000_000 || cfg.BlobSizeLimit != 1_000_000_000 ||
|
||||||
|
cfg.S3.PartSize != 6_000_000 {
|
||||||
|
t.Errorf("chunk_size, blob_size_limit, s3.part_size = %d, %d, %d, "+
|
||||||
|
"want 20000000, 1000000000, 6000000",
|
||||||
|
cfg.ChunkSize, cfg.BlobSizeLimit, cfg.S3.PartSize)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -20,6 +20,17 @@ const indexDirPerm = 0o700
|
|||||||
// backed-up path and chunk hash and must stay private.
|
// backed-up path and chunk hash and must stay private.
|
||||||
const indexFilePerm = 0o600
|
const indexFilePerm = 0o600
|
||||||
|
|
||||||
|
// IndexFiles returns the path of the index at indexPath and the paths of
|
||||||
|
// the side files SQLite keeps beside it while it writes.
|
||||||
|
func IndexFiles(indexPath string) []string {
|
||||||
|
return []string{
|
||||||
|
indexPath,
|
||||||
|
indexPath + "-wal",
|
||||||
|
indexPath + "-shm",
|
||||||
|
indexPath + "-journal",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ensureIndexFileMode makes the index file owner-only before the SQLite
|
// ensureIndexFileMode makes the index file owner-only before the SQLite
|
||||||
// driver opens it: it creates the file 0600 if absent, or chmods an existing
|
// driver opens it: it creates the file 0600 if absent, or chmods an existing
|
||||||
// one to 0600. Doing this first matters because SQLite creates its -wal and
|
// one to 0600. Doing this first matters because SQLite creates its -wal and
|
||||||
|
|||||||
@@ -0,0 +1,77 @@
|
|||||||
|
// Package exclude compiles exclude patterns into globs.
|
||||||
|
package exclude
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/gobwas/glob"
|
||||||
|
)
|
||||||
|
|
||||||
|
var errBadExcludePattern = errors.New("is not a valid exclude pattern")
|
||||||
|
|
||||||
|
// Compile compiles one exclude pattern into the globs that match it.
|
||||||
|
// anchored reports whether the pattern matches only from the root of the
|
||||||
|
// source directory. An empty pattern yields no globs. Config.Validate and
|
||||||
|
// the scanner both compile patterns with it, so a pattern that loads is one
|
||||||
|
// the scanner can use.
|
||||||
|
func Compile(pattern string) ([]glob.Glob, bool, error) {
|
||||||
|
if pattern == "" {
|
||||||
|
return nil, false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// A leading "/" anchors the pattern to the source directory.
|
||||||
|
trimmed, anchored := strings.CutPrefix(pattern, "/")
|
||||||
|
|
||||||
|
// A trailing "/" marks a directory and is not matched.
|
||||||
|
trimmed = strings.TrimSuffix(trimmed, "/")
|
||||||
|
|
||||||
|
// A leading "**/" demands a "/" before the name and so misses an
|
||||||
|
// entry directly under the source path. It means "at every depth",
|
||||||
|
// which is what an unanchored pattern already does, so drop the
|
||||||
|
// prefix and match unanchored: as in gitignore, "**/x" and
|
||||||
|
// "/**/x" then also match a top-level "x".
|
||||||
|
if rest, found := strings.CutPrefix(trimmed, "**/"); found {
|
||||||
|
trimmed = rest
|
||||||
|
anchored = false
|
||||||
|
}
|
||||||
|
|
||||||
|
// Likewise "/**/" inside a pattern demands two slashes, so
|
||||||
|
// "a/**/*.log" misses "a/x.log". As in gitignore it stands for
|
||||||
|
// zero or more directories, so also compile the pattern with it
|
||||||
|
// made a single "/". The glob library's {a,b} is no help: with a
|
||||||
|
// "**" inside it in mid-pattern, it misses paths it should match.
|
||||||
|
var globs []glob.Glob
|
||||||
|
|
||||||
|
for _, variant := range doubleStarVariants(trimmed) {
|
||||||
|
g, err := glob.Compile(variant, '/')
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, fmt.Errorf("%q %w: %w",
|
||||||
|
pattern, errBadExcludePattern, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
globs = append(globs, g)
|
||||||
|
}
|
||||||
|
|
||||||
|
return globs, anchored, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// doubleStarVariants returns pattern once for every way of keeping each
|
||||||
|
// "/**/" in it or making it a single "/".
|
||||||
|
func doubleStarVariants(pattern string) []string {
|
||||||
|
const doubleStar = "/**/"
|
||||||
|
|
||||||
|
before, after, found := strings.Cut(pattern, doubleStar)
|
||||||
|
if !found {
|
||||||
|
return []string{pattern}
|
||||||
|
}
|
||||||
|
|
||||||
|
var variants []string
|
||||||
|
|
||||||
|
for _, rest := range doubleStarVariants(after) {
|
||||||
|
variants = append(variants, before+doubleStar+rest, before+"/"+rest)
|
||||||
|
}
|
||||||
|
|
||||||
|
return variants
|
||||||
|
}
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
package exclude_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/gobwas/glob"
|
||||||
|
"sneak.berlin/go/vaultik/internal/exclude"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestCompile checks the error, the anchoring and the globs that Compile
|
||||||
|
// returns for one exclude pattern.
|
||||||
|
func TestCompile(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
pattern string
|
||||||
|
wantErr string
|
||||||
|
wantAnchored bool
|
||||||
|
// wantMatches lists paths that one of the globs must match. When
|
||||||
|
// it is empty, Compile must return no globs.
|
||||||
|
wantMatches []string
|
||||||
|
}{
|
||||||
|
{name: "does not compile", pattern: "build[",
|
||||||
|
wantErr: `"build[" is not a valid exclude pattern`},
|
||||||
|
{name: "empty", pattern: ""},
|
||||||
|
{name: "leading slash anchors", pattern: "/build",
|
||||||
|
wantAnchored: true, wantMatches: []string{"build"}},
|
||||||
|
{name: "double star after the slash clears anchoring",
|
||||||
|
pattern: "/**/build", wantMatches: []string{"build"}},
|
||||||
|
{name: "double star in the middle", pattern: "a/**/*.log",
|
||||||
|
wantMatches: []string{"a/x.log", "a/b/x.log"}},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
globs, anchored, err := exclude.Compile(tt.pattern)
|
||||||
|
if tt.wantErr == "" && err != nil {
|
||||||
|
t.Fatalf("Compile(%q) error = %v, want none", tt.pattern, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if tt.wantErr != "" &&
|
||||||
|
(err == nil || !strings.HasPrefix(err.Error(), tt.wantErr)) {
|
||||||
|
t.Fatalf("Compile(%q) error = %v, want one starting with %q",
|
||||||
|
tt.pattern, err, tt.wantErr)
|
||||||
|
}
|
||||||
|
|
||||||
|
if anchored != tt.wantAnchored {
|
||||||
|
t.Errorf("Compile(%q) anchored = %t, want %t",
|
||||||
|
tt.pattern, anchored, tt.wantAnchored)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(tt.wantMatches) == 0 && len(globs) != 0 {
|
||||||
|
t.Errorf("Compile(%q) returned %d globs, want none",
|
||||||
|
tt.pattern, len(globs))
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, path := range tt.wantMatches {
|
||||||
|
matched := slices.ContainsFunc(globs, func(g glob.Glob) bool {
|
||||||
|
return g.Match(path)
|
||||||
|
})
|
||||||
|
if !matched {
|
||||||
|
t.Errorf("Compile(%q) globs do not match %q",
|
||||||
|
tt.pattern, path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,19 +1,24 @@
|
|||||||
// Package globals holds application-wide metadata (name, version,
|
// Package globals holds application-wide metadata (name, version,
|
||||||
// commit) that is populated at build time via linker flags.
|
// commit) that is populated at build time via linker flags, or from the
|
||||||
|
// build info Go records in the binary when the linker flags set none.
|
||||||
package globals
|
package globals
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"regexp"
|
"regexp"
|
||||||
|
"runtime/debug"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"golang.org/x/mod/module"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Appname is the application name, populated from main().
|
// Appname is the application name, populated from main().
|
||||||
var Appname = "vaultik" //nolint:gochecknoglobals // set via -ldflags at build time
|
var Appname = "vaultik" //nolint:gochecknoglobals // set via -ldflags at build time
|
||||||
|
|
||||||
// DevVersion is the version a binary reports when it was built without
|
// DevVersion is the version a binary reports when it was built without
|
||||||
// git metadata: script/version emits it outside a git checkout, and an
|
// git metadata: script/version emits it outside a git checkout, and a
|
||||||
// unstamped `go build` keeps it. goreleaser's snapshot template stamps
|
// binary keeps it when neither its linker flags nor its Go build info
|
||||||
|
// name a version. goreleaser's snapshot template stamps
|
||||||
// it followed by "-" and the commit it was built from. It is
|
// it followed by "-" and the commit it was built from. It is
|
||||||
// deliberately not a number.
|
// deliberately not a number.
|
||||||
const DevVersion = "dev"
|
const DevVersion = "dev"
|
||||||
@@ -32,6 +37,54 @@ var Commit = Unknown //nolint:gochecknoglobals // set via -ldflags at build time
|
|||||||
// CommitDate is the ISO-8601 date of the commit, populated from main().
|
// CommitDate is the ISO-8601 date of the commit, populated from main().
|
||||||
var CommitDate = Unknown //nolint:gochecknoglobals // set via -ldflags at build time
|
var CommitDate = Unknown //nolint:gochecknoglobals // set via -ldflags at build time
|
||||||
|
|
||||||
|
// develVersion is the main module version Go records when it cannot
|
||||||
|
// tell which version it built.
|
||||||
|
const develVersion = "(devel)"
|
||||||
|
|
||||||
|
// FillUnsetFromBuildInfo applies FillUnset to Version, Commit and
|
||||||
|
// CommitDate, with the build info of the running binary.
|
||||||
|
func FillUnsetFromBuildInfo() {
|
||||||
|
info, ok := debug.ReadBuildInfo()
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
Version, Commit, CommitDate = FillUnset(
|
||||||
|
info, Version, Commit, CommitDate)
|
||||||
|
}
|
||||||
|
|
||||||
|
// FillUnset returns version, commit and commitDate. Each one that
|
||||||
|
// still holds its default ("dev" or empty for the version, "unknown"
|
||||||
|
// for the others) is replaced by the value Go recorded in info. A
|
||||||
|
// `go install` of a tag sets no linker flags, but Go records the tag
|
||||||
|
// as the main module's version. Go records the commit and its time
|
||||||
|
// only when it builds from a git checkout. The time is cut to a date,
|
||||||
|
// as the Makefile stamps it.
|
||||||
|
func FillUnset(
|
||||||
|
info *debug.BuildInfo, version, commit, commitDate string,
|
||||||
|
) (string, string, string) {
|
||||||
|
if (version == "" || version == DevVersion) &&
|
||||||
|
info.Main.Version != "" && info.Main.Version != develVersion {
|
||||||
|
version = info.Main.Version
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, s := range info.Settings {
|
||||||
|
switch s.Key {
|
||||||
|
case "vcs.revision":
|
||||||
|
if commit == Unknown {
|
||||||
|
commit = s.Value
|
||||||
|
}
|
||||||
|
case "vcs.time":
|
||||||
|
t, err := time.Parse(time.RFC3339, s.Value)
|
||||||
|
if commitDate == Unknown && err == nil {
|
||||||
|
commitDate = t.UTC().Format(time.DateOnly)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return version, commit, commitDate
|
||||||
|
}
|
||||||
|
|
||||||
// Author identifies the upstream author of vaultik.
|
// Author identifies the upstream author of vaultik.
|
||||||
const Author = "Jeffrey Paul <sneak@sneak.berlin>"
|
const Author = "Jeffrey Paul <sneak@sneak.berlin>"
|
||||||
|
|
||||||
@@ -68,8 +121,10 @@ func New() (*Globals, error) {
|
|||||||
// a release. "dev" and goreleaser's snapshot "dev-<sha>" count, and so
|
// a release. "dev" and goreleaser's snapshot "dev-<sha>" count, and so
|
||||||
// does what `git describe --tags --always --dirty` gives a make or
|
// does what `git describe --tags --always --dirty` gives a make or
|
||||||
// docker build of an untagged commit: the bare short commit, or
|
// docker build of an untagged commit: the bare short commit, or
|
||||||
// tag-N-gHASH on a commit after a tag. Any version ending in "-dirty"
|
// tag-N-gHASH on a commit after a tag. A Go pseudo-version counts: Go
|
||||||
// counts, a modified checkout of a tag ("v1.0.0-dirty") included.
|
// records one for a build of an untagged commit without linker flags.
|
||||||
|
// Any version ending in "-dirty" or "+dirty" counts, a modified
|
||||||
|
// checkout of a tag ("v1.0.0-dirty", "v1.0.0+dirty") included.
|
||||||
// A plain tag such as "v1.0.0" or "1.0.0" is a release.
|
// A plain tag such as "v1.0.0" or "1.0.0" is a release.
|
||||||
//
|
//
|
||||||
// The empty string counts too. Nothing that knows its version reports
|
// The empty string counts too. Nothing that knows its version reports
|
||||||
@@ -82,7 +137,8 @@ func New() (*Globals, error) {
|
|||||||
func IsDevVersion(v string) bool {
|
func IsDevVersion(v string) bool {
|
||||||
if v == "" || v == Unknown || v == DevVersion ||
|
if v == "" || v == Unknown || v == DevVersion ||
|
||||||
strings.HasPrefix(v, DevVersion+"-") ||
|
strings.HasPrefix(v, DevVersion+"-") ||
|
||||||
strings.HasSuffix(v, "-dirty") {
|
strings.HasSuffix(v, "-dirty") || strings.HasSuffix(v, "+dirty") ||
|
||||||
|
module.IsPseudoVersion(v) {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package globals_test
|
package globals_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"runtime/debug"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"sneak.berlin/go/vaultik/internal/globals"
|
"sneak.berlin/go/vaultik/internal/globals"
|
||||||
@@ -59,12 +60,20 @@ func TestIsDevVersion(t *testing.T) {
|
|||||||
{"1.0.0-rc.1-12-g877eb2f", true},
|
{"1.0.0-rc.1-12-g877eb2f", true},
|
||||||
// A tagged commit with uncommitted changes is not that tag.
|
// A tagged commit with uncommitted changes is not that tag.
|
||||||
{"v1.0.0-dirty", true},
|
{"v1.0.0-dirty", true},
|
||||||
|
// What Go records for a build without linker flags: a
|
||||||
|
// pseudo-version for an untagged commit, "+dirty" for a
|
||||||
|
// modified checkout.
|
||||||
|
{"v1.0.0-rc3.0.20261111001226-abcdefabcdef", true},
|
||||||
|
{"v0.0.0-20261111001226-abcdefabcdef", true},
|
||||||
|
{"v1.0.1-0.20261111001226-abcdefabcdef", true},
|
||||||
|
{"v1.0.0+dirty", true},
|
||||||
// What a tagged build produces (goreleaser's .Version strips
|
// What a tagged build produces (goreleaser's .Version strips
|
||||||
// the leading "v"; script/version keeps it).
|
// the leading "v"; script/version keeps it).
|
||||||
{"1.0.0", false},
|
{"1.0.0", false},
|
||||||
{"0.1.0", false},
|
{"0.1.0", false},
|
||||||
{"1.0.0-rc.1", false},
|
{"1.0.0-rc.1", false},
|
||||||
{"v1.0.0", false},
|
{"v1.0.0", false},
|
||||||
|
{"v1.0.0-rc3", false},
|
||||||
// A release must not be mistaken for a dev build just because
|
// A release must not be mistaken for a dev build just because
|
||||||
// the string happens to contain "dev".
|
// the string happens to contain "dev".
|
||||||
{"1.0.0-dev", false},
|
{"1.0.0-dev", false},
|
||||||
@@ -86,10 +95,9 @@ func TestIsDevVersion(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestDefaultVersionIsDev pins the linker-flag contract: an unstamped
|
// TestDefaultVersionIsDev pins the default: a binary whose linker flags
|
||||||
// binary (no -ldflags at all, which is what `go build ./...` and `go
|
// and build info name no version must report itself as a development
|
||||||
// install` produce) must report itself as a development build rather
|
// build rather than as some default release number.
|
||||||
// than as some default release number.
|
|
||||||
func TestDefaultVersionIsDev(t *testing.T) {
|
func TestDefaultVersionIsDev(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -98,3 +106,104 @@ func TestDefaultVersionIsDev(t *testing.T) {
|
|||||||
globals.DevVersion)
|
globals.DevVersion)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// buildInfo returns build info with the given main module version and,
|
||||||
|
// when revision is not empty, the commit settings Go records for a
|
||||||
|
// build from a git checkout.
|
||||||
|
func buildInfo(version, revision string) *debug.BuildInfo {
|
||||||
|
info := &debug.BuildInfo{
|
||||||
|
Main: debug.Module{Path: "sneak.berlin/go/vaultik", Version: version},
|
||||||
|
}
|
||||||
|
|
||||||
|
if revision != "" {
|
||||||
|
info.Settings = []debug.BuildSetting{
|
||||||
|
{Key: "vcs.revision", Value: revision},
|
||||||
|
{Key: "vcs.time", Value: "2026-11-10T23:12:26Z"},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return info
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFillUnset covers a binary from `go install` of a tag, which
|
||||||
|
// carries no linker flags: it must report the tag. Values the linker
|
||||||
|
// flags set are kept.
|
||||||
|
func TestFillUnset(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const (
|
||||||
|
tag = "v1.0.0-rc3"
|
||||||
|
rev = "0123456789abcdef0123456789abcdef01234567"
|
||||||
|
)
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
info *debug.BuildInfo
|
||||||
|
version, commit, date string
|
||||||
|
wantVersion, wantCommit string
|
||||||
|
wantDate string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "go install of a tag",
|
||||||
|
info: buildInfo(tag, ""),
|
||||||
|
version: globals.DevVersion,
|
||||||
|
commit: globals.Unknown,
|
||||||
|
date: globals.Unknown,
|
||||||
|
wantVersion: tag,
|
||||||
|
wantCommit: globals.Unknown,
|
||||||
|
wantDate: globals.Unknown,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty version",
|
||||||
|
info: buildInfo(tag, ""),
|
||||||
|
version: "",
|
||||||
|
commit: globals.Unknown,
|
||||||
|
date: globals.Unknown,
|
||||||
|
wantVersion: tag,
|
||||||
|
wantCommit: globals.Unknown,
|
||||||
|
wantDate: globals.Unknown,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "go build in a git checkout",
|
||||||
|
info: buildInfo(tag, rev),
|
||||||
|
version: globals.DevVersion,
|
||||||
|
commit: globals.Unknown,
|
||||||
|
date: globals.Unknown,
|
||||||
|
wantVersion: tag,
|
||||||
|
wantCommit: rev,
|
||||||
|
wantDate: "2026-11-10",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "Go could not tell the version",
|
||||||
|
info: buildInfo("(devel)", ""),
|
||||||
|
version: globals.DevVersion,
|
||||||
|
commit: globals.Unknown,
|
||||||
|
date: globals.Unknown,
|
||||||
|
wantVersion: globals.DevVersion,
|
||||||
|
wantCommit: globals.Unknown,
|
||||||
|
wantDate: globals.Unknown,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "linker flags set",
|
||||||
|
info: buildInfo(tag, rev),
|
||||||
|
version: "v1.0.0-rc2-4-gabcdef1",
|
||||||
|
commit: "abcdef1",
|
||||||
|
date: "2026-11-01",
|
||||||
|
wantVersion: "v1.0.0-rc2-4-gabcdef1",
|
||||||
|
wantCommit: "abcdef1",
|
||||||
|
wantDate: "2026-11-01",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range cases {
|
||||||
|
version, commit, date := globals.FillUnset(
|
||||||
|
tc.info, tc.version, tc.commit, tc.date)
|
||||||
|
|
||||||
|
if version != tc.wantVersion || commit != tc.wantCommit ||
|
||||||
|
date != tc.wantDate {
|
||||||
|
t.Errorf("%s: got %q, %q, %q; want %q, %q, %q", tc.name,
|
||||||
|
version, commit, date,
|
||||||
|
tc.wantVersion, tc.wantCommit, tc.wantDate)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
|
"github.com/adrg/xdg"
|
||||||
"golang.org/x/sys/unix"
|
"golang.org/x/sys/unix"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -29,8 +30,20 @@ const (
|
|||||||
lockDirPerm = 0o700
|
lockDirPerm = 0o700
|
||||||
// pidFilePerm is the mode for the PID file (owner-only).
|
// pidFilePerm is the mode for the PID file (owner-only).
|
||||||
pidFilePerm = 0o600
|
pidFilePerm = 0o600
|
||||||
|
// pidFileName is the name of the PID file in the lock directory.
|
||||||
|
pidFileName = "vaultik.pid"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Dir returns the directory that holds vaultik's PID file.
|
||||||
|
func Dir() string {
|
||||||
|
return filepath.Join(xdg.DataHome, "vaultik")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Path returns the path of the PID file that Acquire uses in lockDir.
|
||||||
|
func Path(lockDir string) string {
|
||||||
|
return filepath.Join(lockDir, pidFileName)
|
||||||
|
}
|
||||||
|
|
||||||
// Acquire attempts to acquire a PID lock in the specified directory.
|
// Acquire attempts to acquire a PID lock in the specified directory.
|
||||||
// If another process holds the lock, it returns ErrAlreadyRunning with
|
// If another process holds the lock, it returns ErrAlreadyRunning with
|
||||||
// that process's PID. On success, it writes the current PID to the lock
|
// that process's PID. On success, it writes the current PID to the lock
|
||||||
@@ -42,7 +55,7 @@ func Acquire(lockDir string) (*Lock, error) {
|
|||||||
return nil, fmt.Errorf("creating lock directory: %w", err)
|
return nil, fmt.Errorf("creating lock directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
lockPath := filepath.Join(lockDir, "vaultik.pid")
|
lockPath := Path(lockDir)
|
||||||
|
|
||||||
// No O_TRUNC: the file may hold the PID of the process that has the
|
// No O_TRUNC: the file may hold the PID of the process that has the
|
||||||
// lock, which the error below reports.
|
// lock, which the error below reports.
|
||||||
|
|||||||
@@ -0,0 +1,65 @@
|
|||||||
|
package snapshot_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/spf13/afero"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
|
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A backup of the directory that holds vaultik's index and PID file
|
||||||
|
// stores neither, with no exclude pattern set, and still stores the
|
||||||
|
// files beside them. See https://git.eeqj.de/sneak/vaultik/issues/421.
|
||||||
|
func TestScanSkipsIndexAndLockFiles(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const dataDir = "/home/user/.local/share/vaultik"
|
||||||
|
|
||||||
|
indexAndLockFiles := append(database.IndexFiles(dataDir+"/index.sqlite"),
|
||||||
|
dataDir+"/vaultik.pid")
|
||||||
|
|
||||||
|
fs := afero.NewMemMapFs()
|
||||||
|
for _, p := range append(indexAndLockFiles, dataDir+"/notes.txt") {
|
||||||
|
require.NoError(t, afero.WriteFile(fs, p, []byte("data"), 0o600))
|
||||||
|
}
|
||||||
|
|
||||||
|
db, err := database.NewTestDB()
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
defer func() { _ = db.Close() }()
|
||||||
|
|
||||||
|
repos := database.NewRepositories(db)
|
||||||
|
|
||||||
|
scanner := snapshot.NewScanner(snapshot.ScannerConfig{
|
||||||
|
FS: fs,
|
||||||
|
ChunkSize: 64 * 1024,
|
||||||
|
Repositories: repos,
|
||||||
|
MaxBlobSize: 1024 * 1024,
|
||||||
|
CompressionLevel: 3,
|
||||||
|
AgeRecipients: []string{
|
||||||
|
"age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p"},
|
||||||
|
IndexAndLockFiles: indexAndLockFiles,
|
||||||
|
})
|
||||||
|
require.NotNil(t, scanner)
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
createSnapshotRecord(ctx, t, repos, "test-snapshot")
|
||||||
|
|
||||||
|
result, err := scanner.Scan(ctx, "/home/user", "test-snapshot")
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
require.Equal(t, 1, result.FilesScanned)
|
||||||
|
|
||||||
|
stored, err := repos.Files.GetByPath(ctx, dataDir+"/notes.txt")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NotNil(t, stored, "a file beside the index is stored")
|
||||||
|
|
||||||
|
for _, p := range indexAndLockFiles {
|
||||||
|
f, err := repos.Files.GetByPath(ctx, p)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Nil(t, f, "%s is not stored", p)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -2,11 +2,13 @@ package snapshot
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"os"
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
|
||||||
"github.com/spf13/afero"
|
"github.com/spf13/afero"
|
||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
"sneak.berlin/go/vaultik/internal/config"
|
"sneak.berlin/go/vaultik/internal/config"
|
||||||
"sneak.berlin/go/vaultik/internal/database"
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
|
"sneak.berlin/go/vaultik/internal/pidlock"
|
||||||
"sneak.berlin/go/vaultik/internal/storage"
|
"sneak.berlin/go/vaultik/internal/storage"
|
||||||
"sneak.berlin/go/vaultik/internal/ui"
|
"sneak.berlin/go/vaultik/internal/ui"
|
||||||
)
|
)
|
||||||
@@ -62,9 +64,52 @@ func provideScannerFactory(
|
|||||||
EnableProgress: params.EnableProgress,
|
EnableProgress: params.EnableProgress,
|
||||||
UI: params.UI,
|
UI: params.UI,
|
||||||
Exclude: excludes,
|
Exclude: excludes,
|
||||||
|
IndexAndLockFiles: indexAndLockFiles(cfg.IndexPath, pidlock.Dir()),
|
||||||
SkipErrors: params.SkipErrors,
|
SkipErrors: params.SkipErrors,
|
||||||
PreviousSnapshotID: params.PreviousSnapshotID,
|
PreviousSnapshotID: params.PreviousSnapshotID,
|
||||||
DeviceNumber: params.DeviceNumber,
|
DeviceNumber: params.DeviceNumber,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// indexAndLockFiles returns the paths of the index at indexPath, its side
|
||||||
|
// files and the PID file in lockDir. Symlinks in their directories are
|
||||||
|
// resolved, as they are in the snapshot's source paths, so the scan finds
|
||||||
|
// them under the same names. If indexPath is a symlink, the file it points
|
||||||
|
// to and that file's side files are returned too.
|
||||||
|
func indexAndLockFiles(indexPath, lockDir string) []string {
|
||||||
|
paths := append(database.IndexFiles(indexPath), pidlock.Path(lockDir))
|
||||||
|
for i, p := range paths {
|
||||||
|
paths[i] = resolveParentDir(p)
|
||||||
|
}
|
||||||
|
|
||||||
|
// SQLite writes the file a symlinked index points to and keeps the
|
||||||
|
// side files beside that file. The database package creates the index
|
||||||
|
// before the scanner is built, so the link resolves.
|
||||||
|
index := resolveParentDir(indexPath)
|
||||||
|
|
||||||
|
target, err := filepath.EvalSymlinks(index)
|
||||||
|
if err == nil && target != index {
|
||||||
|
paths = append(paths, database.IndexFiles(target)...)
|
||||||
|
}
|
||||||
|
|
||||||
|
return paths
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolveParentDir returns the absolute form of path with the symlinks in
|
||||||
|
// its parent directory resolved. If the parent directory cannot be
|
||||||
|
// resolved, it returns the absolute form unresolved. If path has no
|
||||||
|
// absolute form, it returns path.
|
||||||
|
func resolveParentDir(path string) string {
|
||||||
|
abs, err := filepath.Abs(path)
|
||||||
|
if err != nil {
|
||||||
|
return path
|
||||||
|
}
|
||||||
|
|
||||||
|
dir, err := filepath.EvalSymlinks(filepath.Dir(abs))
|
||||||
|
if err != nil {
|
||||||
|
return abs
|
||||||
|
}
|
||||||
|
|
||||||
|
return filepath.Join(dir, filepath.Base(abs))
|
||||||
|
}
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
//nolint:testpackage // exercises the unexported indexAndLockFiles helper
|
||||||
|
package snapshot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The scan walks source paths with symlinks resolved, so vaultik's own
|
||||||
|
// files are named the same way, or a symlinked home would back them up.
|
||||||
|
func TestIndexAndLockFilesResolvesSymlinkedDirectory(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
realDir, err := filepath.EvalSymlinks(t.TempDir())
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
link := filepath.Join(t.TempDir(), "link")
|
||||||
|
require.NoError(t, os.Symlink(realDir, link))
|
||||||
|
|
||||||
|
paths := indexAndLockFiles(filepath.Join(link, "index.sqlite"), link)
|
||||||
|
|
||||||
|
require.Equal(t, []string{
|
||||||
|
filepath.Join(realDir, "index.sqlite"),
|
||||||
|
filepath.Join(realDir, "index.sqlite-wal"),
|
||||||
|
filepath.Join(realDir, "index.sqlite-shm"),
|
||||||
|
filepath.Join(realDir, "index.sqlite-journal"),
|
||||||
|
filepath.Join(realDir, "vaultik.pid"),
|
||||||
|
}, paths)
|
||||||
|
}
|
||||||
@@ -20,6 +20,7 @@ import (
|
|||||||
"sneak.berlin/go/vaultik/internal/blob"
|
"sneak.berlin/go/vaultik/internal/blob"
|
||||||
"sneak.berlin/go/vaultik/internal/chunker"
|
"sneak.berlin/go/vaultik/internal/chunker"
|
||||||
"sneak.berlin/go/vaultik/internal/database"
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
|
"sneak.berlin/go/vaultik/internal/exclude"
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
"sneak.berlin/go/vaultik/internal/storage"
|
"sneak.berlin/go/vaultik/internal/storage"
|
||||||
"sneak.berlin/go/vaultik/internal/types"
|
"sneak.berlin/go/vaultik/internal/types"
|
||||||
@@ -67,6 +68,9 @@ type Scanner struct {
|
|||||||
currentSourcePath string
|
currentSourcePath string
|
||||||
exclude []string // Glob patterns for files/directories to exclude
|
exclude []string // Glob patterns for files/directories to exclude
|
||||||
compiledExclude []compiledPattern // Compiled glob patterns
|
compiledExclude []compiledPattern // Compiled glob patterns
|
||||||
|
// indexAndLockFiles holds the paths of vaultik's own working files,
|
||||||
|
// which the scan treats as excluded (ScannerConfig.IndexAndLockFiles).
|
||||||
|
indexAndLockFiles map[string]struct{}
|
||||||
progress *ProgressReporter
|
progress *ProgressReporter
|
||||||
// skipErrors skips files that cannot be opened or read (logged loudly);
|
// skipErrors skips files that cannot be opened or read (logged loudly);
|
||||||
// packer, database, encryption, and upload errors still abort the run.
|
// packer, database, encryption, and upload errors still abort the run.
|
||||||
@@ -133,6 +137,11 @@ type ScannerConfig struct {
|
|||||||
EnableProgress bool // Enable the live progress reporter (ETAs, throughput)
|
EnableProgress bool // Enable the live progress reporter (ETAs, throughput)
|
||||||
UI *ui.Writer // Where user-facing scanner messages go; nil = discard
|
UI *ui.Writer // Where user-facing scanner messages go; nil = discard
|
||||||
Exclude []string // Glob patterns for files/directories to exclude
|
Exclude []string // Glob patterns for files/directories to exclude
|
||||||
|
// IndexAndLockFiles are the absolute paths of vaultik's own working
|
||||||
|
// files: the index, its side files and the PID file. The scan treats
|
||||||
|
// them as excluded, whatever Exclude says, since a copy taken while
|
||||||
|
// they are being written is torn.
|
||||||
|
IndexAndLockFiles []string
|
||||||
// SkipErrors skips files that cannot be opened or read (log loudly but
|
// SkipErrors skips files that cannot be opened or read (log loudly but
|
||||||
// continue); packer, database, encryption, and upload errors still abort.
|
// continue); packer, database, encryption, and upload errors still abort.
|
||||||
SkipErrors bool
|
SkipErrors bool
|
||||||
@@ -213,8 +222,12 @@ func NewScanner(cfg ScannerConfig) *Scanner {
|
|||||||
progress = NewProgressReporter()
|
progress = NewProgressReporter()
|
||||||
}
|
}
|
||||||
|
|
||||||
// Compile exclude patterns
|
compiledExclude, err := compileExcludePatterns(cfg.Exclude)
|
||||||
compiledExclude := compileExcludePatterns(cfg.Exclude)
|
if err != nil {
|
||||||
|
log.Error("Failed to compile exclude patterns", "error", err)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
uiw := cfg.UI
|
uiw := cfg.UI
|
||||||
if uiw == nil {
|
if uiw == nil {
|
||||||
@@ -226,6 +239,11 @@ func NewScanner(cfg ScannerConfig) *Scanner {
|
|||||||
deviceNumber = deviceNumberFromStat
|
deviceNumber = deviceNumberFromStat
|
||||||
}
|
}
|
||||||
|
|
||||||
|
indexAndLockFiles := make(map[string]struct{}, len(cfg.IndexAndLockFiles))
|
||||||
|
for _, p := range cfg.IndexAndLockFiles {
|
||||||
|
indexAndLockFiles[filepath.Clean(p)] = struct{}{}
|
||||||
|
}
|
||||||
|
|
||||||
return &Scanner{
|
return &Scanner{
|
||||||
fs: cfg.FS,
|
fs: cfg.FS,
|
||||||
chunker: chunker.NewChunker(cfg.ChunkSize),
|
chunker: chunker.NewChunker(cfg.ChunkSize),
|
||||||
@@ -237,6 +255,7 @@ func NewScanner(cfg ScannerConfig) *Scanner {
|
|||||||
ageRecipient: strings.Join(cfg.AgeRecipients, ","),
|
ageRecipient: strings.Join(cfg.AgeRecipients, ","),
|
||||||
exclude: cfg.Exclude,
|
exclude: cfg.Exclude,
|
||||||
compiledExclude: compiledExclude,
|
compiledExclude: compiledExclude,
|
||||||
|
indexAndLockFiles: indexAndLockFiles,
|
||||||
progress: progress,
|
progress: progress,
|
||||||
skipErrors: cfg.SkipErrors,
|
skipErrors: cfg.SkipErrors,
|
||||||
previousSnapshotID: cfg.PreviousSnapshotID,
|
previousSnapshotID: cfg.PreviousSnapshotID,
|
||||||
@@ -1861,7 +1880,13 @@ func (s *Scanner) uploadBlobIfNeeded(
|
|||||||
err = s.storage.PutWithProgress(ctx, blobPath, blobWithReader.Reader,
|
err = s.storage.PutWithProgress(ctx, blobPath, blobWithReader.Reader,
|
||||||
finishedBlob.Compressed, progressCallback)
|
finishedBlob.Compressed, progressCallback)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Error("Failed to upload blob", "hash", finishedBlob.Hash, "error", err)
|
// The record is at info level: the returned error ends the backup
|
||||||
|
// and is printed as its ERROR: line. Only an interrupt cancels
|
||||||
|
// ctx, and the user is told about it, so an upload it stopped is
|
||||||
|
// not logged as a storage failure.
|
||||||
|
if ctx.Err() == nil {
|
||||||
|
log.Info("Failed to upload blob", "hash", finishedBlob.Hash, "error", err)
|
||||||
|
}
|
||||||
|
|
||||||
return 0, false, fmt.Errorf("uploading blob to storage: %w", err)
|
return 0, false, fmt.Errorf("uploading blob to storage: %w", err)
|
||||||
}
|
}
|
||||||
@@ -2248,8 +2273,9 @@ func (s *Scanner) detectDeletedFiles(
|
|||||||
|
|
||||||
// excludedOrInExcludedDir reports whether filePath, which is at or
|
// excludedOrInExcludedDir reports whether filePath, which is at or
|
||||||
// beneath rootPath, or any directory between rootPath and it, matches an
|
// beneath rootPath, or any directory between rootPath and it, matches an
|
||||||
// exclude pattern. A pattern such as */.local/share/Trash matches the
|
// exclude pattern or is one of vaultik's own files. A pattern such as
|
||||||
// directory but not the files inside it, which the walk skips too.
|
// */.local/share/Trash matches the directory but not the files inside it,
|
||||||
|
// which the walk skips too.
|
||||||
func (s *Scanner) excludedOrInExcludedDir(filePath, rootPath string) bool {
|
func (s *Scanner) excludedOrInExcludedDir(filePath, rootPath string) bool {
|
||||||
for p := filePath; len(p) > len(rootPath); p = filepath.Dir(p) {
|
for p := filePath; len(p) > len(rootPath); p = filepath.Dir(p) {
|
||||||
if s.shouldExclude(p, rootPath) {
|
if s.shouldExclude(p, rootPath) {
|
||||||
@@ -2297,51 +2323,18 @@ func wrapPermissionError(path string, err error) error {
|
|||||||
"with --skip-errors to continue past unreadable files)", path, err)
|
"with --skip-errors to continue past unreadable files)", path, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// compileExcludePatterns compiles the exclude patterns into glob matchers
|
// compileExcludePatterns compiles the exclude patterns into glob matchers.
|
||||||
func compileExcludePatterns(patterns []string) []compiledPattern {
|
// Config.Validate has already refused a pattern that does not compile.
|
||||||
|
func compileExcludePatterns(patterns []string) ([]compiledPattern, error) {
|
||||||
var compiled []compiledPattern
|
var compiled []compiledPattern
|
||||||
|
|
||||||
for _, p := range patterns {
|
for _, p := range patterns {
|
||||||
if p == "" {
|
globs, anchored, err := exclude.Compile(p)
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
// Check if pattern is anchored (starts with /)
|
|
||||||
anchored := strings.HasPrefix(p, "/")
|
|
||||||
|
|
||||||
pattern := p
|
|
||||||
if anchored {
|
|
||||||
pattern = p[1:] // Remove leading /
|
|
||||||
}
|
|
||||||
|
|
||||||
// Remove trailing slash if present (directory indicator)
|
|
||||||
pattern = strings.TrimSuffix(pattern, "/")
|
|
||||||
|
|
||||||
// A leading "**/" demands a "/" before the name and so misses an
|
|
||||||
// entry directly under the source path. It means "at every depth",
|
|
||||||
// which is what an unanchored pattern already does, so drop the
|
|
||||||
// prefix and match unanchored: as in gitignore, "**/x" and
|
|
||||||
// "/**/x" then also match a top-level "x".
|
|
||||||
if rest, found := strings.CutPrefix(pattern, "**/"); found {
|
|
||||||
pattern = rest
|
|
||||||
anchored = false
|
|
||||||
}
|
|
||||||
|
|
||||||
// Likewise "/**/" inside a pattern demands two slashes, so
|
|
||||||
// "a/**/*.log" misses "a/x.log". As in gitignore it stands for
|
|
||||||
// zero or more directories, so also compile the pattern with it
|
|
||||||
// made a single "/". The glob library's {a,b} is no help: with a
|
|
||||||
// "**" inside it in mid-pattern, it misses paths it should match.
|
|
||||||
for _, variant := range doubleStarVariants(pattern) {
|
|
||||||
// For patterns without path separators, we need to match them as components
|
|
||||||
// e.g., ".git" should match ".git" anywhere in the path
|
|
||||||
g, err := glob.Compile(variant, '/')
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Warn("Invalid exclude pattern, skipping", "pattern", p, "error", err)
|
return nil, err
|
||||||
|
|
||||||
break
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
for _, g := range globs {
|
||||||
compiled = append(compiled, compiledPattern{
|
compiled = append(compiled, compiledPattern{
|
||||||
pattern: g,
|
pattern: g,
|
||||||
anchored: anchored,
|
anchored: anchored,
|
||||||
@@ -2350,32 +2343,18 @@ func compileExcludePatterns(patterns []string) []compiledPattern {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return compiled
|
return compiled, nil
|
||||||
}
|
|
||||||
|
|
||||||
// doubleStarVariants returns pattern once for every way of keeping each
|
|
||||||
// "/**/" in it or making it a single "/".
|
|
||||||
func doubleStarVariants(pattern string) []string {
|
|
||||||
const doubleStar = "/**/"
|
|
||||||
|
|
||||||
before, after, found := strings.Cut(pattern, doubleStar)
|
|
||||||
if !found {
|
|
||||||
return []string{pattern}
|
|
||||||
}
|
|
||||||
|
|
||||||
var variants []string
|
|
||||||
|
|
||||||
for _, rest := range doubleStarVariants(after) {
|
|
||||||
variants = append(variants, before+doubleStar+rest, before+"/"+rest)
|
|
||||||
}
|
|
||||||
|
|
||||||
return variants
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// shouldExclude checks if a path should be excluded based on exclude patterns
|
// shouldExclude checks if a path should be excluded based on exclude patterns
|
||||||
|
// or because it is one of vaultik's own working files.
|
||||||
// filePath is the full path to the file
|
// filePath is the full path to the file
|
||||||
// rootPath is the root of the backup source directory
|
// rootPath is the root of the backup source directory
|
||||||
func (s *Scanner) shouldExclude(filePath, rootPath string) bool {
|
func (s *Scanner) shouldExclude(filePath, rootPath string) bool {
|
||||||
|
if _, ok := s.indexAndLockFiles[filePath]; ok {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
if len(s.compiledExclude) == 0 {
|
if len(s.compiledExclude) == 0 {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,135 @@
|
|||||||
|
//nolint:testpackage // exercises the unexported uploadBlobIfNeeded
|
||||||
|
package snapshot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/vaultik/internal/blob"
|
||||||
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
|
"sneak.berlin/go/vaultik/internal/storage"
|
||||||
|
"sneak.berlin/go/vaultik/internal/storage/faultstore"
|
||||||
|
"sneak.berlin/go/vaultik/internal/ui"
|
||||||
|
)
|
||||||
|
|
||||||
|
const uploadFailedLogMessage = "Failed to upload blob"
|
||||||
|
|
||||||
|
// uploadTestBlob uploads a small blob through store.
|
||||||
|
func uploadTestBlob(ctx context.Context, store storage.Storer) error {
|
||||||
|
data := []byte("blob data")
|
||||||
|
s := &Scanner{storage: store, ui: ui.NewWithColor(io.Discard, false)}
|
||||||
|
|
||||||
|
_, _, err := s.uploadBlobIfNeeded(ctx, "blobs/test", &blob.WithReader{
|
||||||
|
FinishedBlob: &blob.FinishedBlob{
|
||||||
|
Hash: "test",
|
||||||
|
Compressed: int64(len(data)),
|
||||||
|
},
|
||||||
|
Reader: bytes.NewReader(data),
|
||||||
|
}, time.Now(), &ScanResult{})
|
||||||
|
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// captureLog runs fn and returns what the logger wrote to stderr
|
||||||
|
// meanwhile, at the --verbose level. The logger writes to the os.Stderr
|
||||||
|
// it finds when it is initialized, so os.Stderr is pointed at a file
|
||||||
|
// first. Not parallel-safe: os.Stderr and the logger are process-global.
|
||||||
|
func captureLog(t *testing.T, fn func()) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
stderrPath := filepath.Join(t.TempDir(), "stderr")
|
||||||
|
|
||||||
|
stderrFile, err := os.Create(stderrPath) //nolint:gosec // G304: test temp path
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("creating stderr file: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
previous := os.Stderr
|
||||||
|
os.Stderr = stderrFile
|
||||||
|
|
||||||
|
log.Initialize(log.Config{Verbose: true})
|
||||||
|
|
||||||
|
fn()
|
||||||
|
|
||||||
|
os.Stderr = previous
|
||||||
|
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
|
||||||
|
_ = stderrFile.Close()
|
||||||
|
|
||||||
|
captured, err := os.ReadFile(stderrPath) //nolint:gosec // G304: test temp path
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("reading stderr file: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return string(captured)
|
||||||
|
}
|
||||||
|
|
||||||
|
// An upload stopped by Ctrl-C or SIGTERM is not logged as an error; the
|
||||||
|
// interrupted message is all the user sees. See
|
||||||
|
// https://git.eeqj.de/sneak/vaultik/issues/434.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // captureLog replaces os.Stderr
|
||||||
|
func TestInterruptedUploadIsNotLoggedAsError(t *testing.T) {
|
||||||
|
inner, err := storage.NewFileStorer(t.TempDir())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
t.Cleanup(cancel)
|
||||||
|
|
||||||
|
// The user interrupts mid-upload. The injected error does not wrap
|
||||||
|
// context.Canceled, so only ctx shows why the upload failed.
|
||||||
|
store := faultstore.New(inner)
|
||||||
|
store.OnPut = func(string) faultstore.PutAction {
|
||||||
|
cancel()
|
||||||
|
|
||||||
|
return faultstore.PutFail
|
||||||
|
}
|
||||||
|
|
||||||
|
var uploadErr error
|
||||||
|
|
||||||
|
stderr := captureLog(t, func() { uploadErr = uploadTestBlob(ctx, store) })
|
||||||
|
|
||||||
|
if uploadErr == nil {
|
||||||
|
t.Fatal("expected the upload to fail")
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.Contains(stderr, uploadFailedLogMessage) {
|
||||||
|
t.Errorf("interrupted upload logged as a failure: %q", stderr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// An upload that fails for any other reason is still logged.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // captureLog replaces os.Stderr
|
||||||
|
func TestFailedUploadIsLogged(t *testing.T) {
|
||||||
|
inner, err := storage.NewFileStorer(t.TempDir())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
store := faultstore.New(inner)
|
||||||
|
store.OnPut = func(string) faultstore.PutAction { return faultstore.PutFail }
|
||||||
|
|
||||||
|
var uploadErr error
|
||||||
|
|
||||||
|
stderr := captureLog(t, func() {
|
||||||
|
uploadErr = uploadTestBlob(context.Background(), store)
|
||||||
|
})
|
||||||
|
|
||||||
|
if uploadErr == nil {
|
||||||
|
t.Fatal("expected the upload to fail")
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(stderr, uploadFailedLogMessage) {
|
||||||
|
t.Errorf("failed upload not logged: %q", stderr)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -22,6 +22,10 @@ var errBlobHashMismatch = errors.New("blob hash mismatch")
|
|||||||
var errBlobNotFullyRead = errors.New(
|
var errBlobNotFullyRead = errors.New(
|
||||||
"blob closed before fully read; hash not verified")
|
"blob closed before fully read; hash not verified")
|
||||||
|
|
||||||
|
// errWrongPrivateKeyForBlob is returned when the configured key parses
|
||||||
|
// but is not one of the keys a blob was encrypted to.
|
||||||
|
var errWrongPrivateKeyForBlob = errors.New("cannot decrypt this blob")
|
||||||
|
|
||||||
// hashVerifyReader wraps a blobgen.Reader and verifies the double-SHA-256 hash
|
// hashVerifyReader wraps a blobgen.Reader and verifies the double-SHA-256 hash
|
||||||
// of decrypted plaintext when Close is called. It reuses the hash that
|
// of decrypted plaintext when Close is called. It reuses the hash that
|
||||||
// blobgen.Reader already computes internally via its TeeReader, avoiding
|
// blobgen.Reader already computes internally via its TeeReader, avoiding
|
||||||
@@ -87,10 +91,14 @@ func (v *Vaultik) FetchAndDecryptBlob(
|
|||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
reader, err := blobgen.NewReader(rc, identities...)
|
reader, err := v.newBlobReader(rc, identities)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = rc.Close()
|
_ = rc.Close()
|
||||||
|
|
||||||
|
if errors.Is(err, errWrongPrivateKeyForBlob) {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
return nil, fmt.Errorf("creating blob reader: %w", err)
|
return nil, fmt.Errorf("creating blob reader: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -102,6 +110,28 @@ func (v *Vaultik) FetchAndDecryptBlob(
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// newBlobReader opens the decrypting reader for a blob. When none of the
|
||||||
|
// identities can decrypt it, the error says in plain words that the
|
||||||
|
// configured private key is not one of the keys the blob was encrypted
|
||||||
|
// to and names where the key was configured.
|
||||||
|
func (v *Vaultik) newBlobReader(
|
||||||
|
r io.Reader, identities []age.Identity,
|
||||||
|
) (*blobgen.Reader, error) {
|
||||||
|
blobReader, err := blobgen.NewReader(r, identities...)
|
||||||
|
|
||||||
|
if _, ok := errors.AsType[*age.NoIdentityMatchError](err); ok {
|
||||||
|
return nil, fmt.Errorf(
|
||||||
|
"%w: the private key in %s is not one of the keys the blob was encrypted to",
|
||||||
|
errWrongPrivateKeyForBlob, v.Config.AgeSecretKeySourceName())
|
||||||
|
}
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return blobReader, nil
|
||||||
|
}
|
||||||
|
|
||||||
// FetchBlob downloads a blob and returns a reader for the encrypted data.
|
// FetchBlob downloads a blob and returns a reader for the encrypted data.
|
||||||
// The reader is closed as soon as ctx is cancelled, which ends a read
|
// The reader is closed as soon as ctx is cancelled, which ends a read
|
||||||
// blocked inside it. A storage error is returned as is: restore and deep
|
// blocked inside it. A storage error is returned as is: restore and deep
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ func (v *Vaultik) checkDestinationMarker() error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Config is nil only in tests, which then record no binding (see
|
// Config is nil only in tests, which then record no binding (see
|
||||||
// EnsureStorageBinding).
|
// checkStorageBinding).
|
||||||
if v.Config != nil {
|
if v.Config != nil {
|
||||||
stored, configured, err := v.storageBinding()
|
stored, configured, err := v.storageBinding()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -83,11 +83,19 @@ func (v *Vaultik) checkDestinationMarker() error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// markDestination checks the backup destination store with
|
// markDestination refuses a local index bound to a different
|
||||||
// checkDestinationMarker and then writes the marker, so a first backup
|
// destination, checks the backup destination store for its marker,
|
||||||
// marks its destination and a later one writes the same text again.
|
// writes the marker, and then records the binding. A first backup marks
|
||||||
|
// its destination and a later one writes the same text again. The
|
||||||
|
// binding is recorded last, so a first backup that cannot write to the
|
||||||
|
// destination leaves the local index unbound.
|
||||||
func (v *Vaultik) markDestination() error {
|
func (v *Vaultik) markDestination() error {
|
||||||
err := v.checkDestinationMarker()
|
err := v.checkStorageBinding()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
err = v.checkDestinationMarker()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -99,5 +107,5 @@ func (v *Vaultik) markDestination() error {
|
|||||||
destinationMarkerKey, err)
|
destinationMarkerKey, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return v.EnsureStorageBinding()
|
||||||
}
|
}
|
||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/spf13/afero"
|
"github.com/spf13/afero"
|
||||||
@@ -24,19 +25,28 @@ import (
|
|||||||
// that is not plugged in, cannot be listed. It is not an empty store, so
|
// that is not plugged in, cannot be listed. It is not an empty store, so
|
||||||
// no command may conclude from it that the local snapshots are gone.
|
// no command may conclude from it that the local snapshots are gone.
|
||||||
|
|
||||||
// newFileDestinationVaultik returns a Vaultik that backs up the
|
// newFileDestinationVaultik writes the fault source tree and returns a
|
||||||
// snapshot named "first" to a file:// destination at storeDir.
|
// Vaultik that backs it up to a file:// destination at storeDir.
|
||||||
func newFileDestinationVaultik(
|
func newFileDestinationVaultik(
|
||||||
ctx context.Context, t *testing.T, storeDir string,
|
ctx context.Context, t *testing.T, storeDir string,
|
||||||
) (*vaultik.Vaultik, *database.Repositories) {
|
) (*vaultik.Vaultik, *database.Repositories) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
osFs := afero.NewOsFs()
|
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
dataDir := filepath.Join(tempDir, "src")
|
dataDir := filepath.Join(tempDir, "src")
|
||||||
dbPath := filepath.Join(tempDir, "index.sqlite")
|
writeFaultSourceTree(t, afero.NewOsFs(), dataDir)
|
||||||
|
|
||||||
writeFaultSourceTree(t, osFs, dataDir)
|
return openFileDestinationVaultik(ctx, t, storeDir, dataDir,
|
||||||
|
filepath.Join(tempDir, "index.sqlite"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// openFileDestinationVaultik returns a Vaultik that backs up dataDir to a
|
||||||
|
// file:// destination at storeDir, with its local index at dbPath. The
|
||||||
|
// index is closed when the test ends.
|
||||||
|
func openFileDestinationVaultik(
|
||||||
|
ctx context.Context, t *testing.T, storeDir, dataDir, dbPath string,
|
||||||
|
) (*vaultik.Vaultik, *database.Repositories) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
store, err := storage.NewFileStorer(storeDir)
|
store, err := storage.NewFileStorer(storeDir)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
@@ -48,22 +58,84 @@ func newFileDestinationVaultik(
|
|||||||
repos := database.NewRepositories(db)
|
repos := database.NewRepositories(db)
|
||||||
cfg := changedFileConfig(dataDir, dbPath)
|
cfg := changedFileConfig(dataDir, dbPath)
|
||||||
|
|
||||||
return newBackupVaultik(ctx, cfg, store, repos, db, osFs), repos
|
return newBackupVaultik(ctx, cfg, store, repos, db, afero.NewOsFs()), repos
|
||||||
|
}
|
||||||
|
|
||||||
|
// firstBackup is one backup of the snapshot named "first" to a file://
|
||||||
|
// destination. dir holds its source tree in src, its store in store and
|
||||||
|
// its closed local index in index/index.sqlite. The first test that
|
||||||
|
// needs it makes it, and TestMain removes it. Every test that copies it
|
||||||
|
// backs up the same source tree, so no test may change that tree.
|
||||||
|
//
|
||||||
|
//nolint:gochecknoglobals // made once and shared by the tests that copy it
|
||||||
|
var firstBackup struct {
|
||||||
|
once sync.Once
|
||||||
|
dir string
|
||||||
|
ready bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestMain removes firstBackup once every test has run.
|
||||||
|
func TestMain(m *testing.M) {
|
||||||
|
code := m.Run()
|
||||||
|
|
||||||
|
_ = os.RemoveAll(firstBackup.dir)
|
||||||
|
|
||||||
|
os.Exit(code)
|
||||||
|
}
|
||||||
|
|
||||||
|
// makeFirstBackup writes the fault source tree and backs it up as the
|
||||||
|
// snapshot named "first", all in a new directory that it records in
|
||||||
|
// firstBackup.
|
||||||
|
func makeFirstBackup(ctx context.Context, t *testing.T) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
dir, err := os.MkdirTemp("", "vaultik-first-backup-")
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
firstBackup.dir = dir
|
||||||
|
storeDir := filepath.Join(dir, "store")
|
||||||
|
dataDir := filepath.Join(dir, "src")
|
||||||
|
indexDir := filepath.Join(dir, "index")
|
||||||
|
|
||||||
|
require.NoError(t, os.Mkdir(storeDir, 0o750))
|
||||||
|
require.NoError(t, os.Mkdir(indexDir, 0o750))
|
||||||
|
writeFaultSourceTree(t, afero.NewOsFs(), dataDir)
|
||||||
|
|
||||||
|
v, _ := openFileDestinationVaultik(ctx, t, storeDir, dataDir,
|
||||||
|
filepath.Join(indexDir, "index.sqlite"))
|
||||||
|
require.NoError(t, backUp(v, "first"))
|
||||||
|
require.NoError(t, v.DB.Close())
|
||||||
|
|
||||||
|
firstBackup.ready = true
|
||||||
}
|
}
|
||||||
|
|
||||||
// backUpToFileDestination creates the directory storeDir, as a user must
|
// backUpToFileDestination creates the directory storeDir, as a user must
|
||||||
// before the first backup, and backs up the snapshot named "first" to a
|
// before the first backup, and returns a Vaultik whose file://
|
||||||
// file:// destination there. Everything the returned Vaultik prints
|
// destination there holds a backup of the snapshot named "first". The
|
||||||
// after the backup goes to the returned buffer.
|
// store and the local index are copies of firstBackup's, which is faster
|
||||||
|
// than backing up in every test. The returned Vaultik backs up a source
|
||||||
|
// tree shared with every other caller, so no test may change that tree.
|
||||||
|
// Everything the returned Vaultik prints after the backup goes to the
|
||||||
|
// returned buffer.
|
||||||
func backUpToFileDestination(
|
func backUpToFileDestination(
|
||||||
ctx context.Context, t *testing.T, storeDir string,
|
ctx context.Context, t *testing.T, storeDir string,
|
||||||
) (*vaultik.Vaultik, *database.Repositories, *bytes.Buffer) {
|
) (*vaultik.Vaultik, *database.Repositories, *bytes.Buffer) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
require.NoError(t, os.MkdirAll(storeDir, 0o750))
|
firstBackup.once.Do(func() { makeFirstBackup(ctx, t) })
|
||||||
|
require.True(t, firstBackup.ready, "another test failed to make firstBackup")
|
||||||
|
|
||||||
v, repos := newFileDestinationVaultik(ctx, t, storeDir)
|
require.NoError(t, os.MkdirAll(storeDir, 0o750))
|
||||||
require.NoError(t, backUp(v, "first"))
|
require.NoError(t, os.CopyFS(storeDir,
|
||||||
|
os.DirFS(filepath.Join(firstBackup.dir, "store"))))
|
||||||
|
|
||||||
|
indexDir := t.TempDir()
|
||||||
|
require.NoError(t, os.CopyFS(indexDir,
|
||||||
|
os.DirFS(filepath.Join(firstBackup.dir, "index"))))
|
||||||
|
|
||||||
|
v, repos := openFileDestinationVaultik(ctx, t, storeDir,
|
||||||
|
filepath.Join(firstBackup.dir, "src"),
|
||||||
|
filepath.Join(indexDir, "index.sqlite"))
|
||||||
|
|
||||||
out := &bytes.Buffer{}
|
out := &bytes.Buffer{}
|
||||||
v.Stdout = out
|
v.Stdout = out
|
||||||
|
|||||||
@@ -68,7 +68,7 @@ func (v *Vaultik) NukeRemote(force bool) error {
|
|||||||
// the same way, so the advice below must not follow these errors.
|
// the same way, so the advice below must not follow these errors.
|
||||||
// The listing also fails on a destination that answers that it has
|
// The listing also fails on a destination that answers that it has
|
||||||
// no marker while the local index holds snapshots.
|
// no marker while the local index holds snapshots.
|
||||||
err := v.EnsureStorageBinding()
|
err := v.checkStorageBinding()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -192,7 +192,7 @@ type PruneBlobsResult struct {
|
|||||||
// PruneDatabase or PruneBlobs individually unless it specifically wants
|
// PruneDatabase or PruneBlobs individually unless it specifically wants
|
||||||
// one half.
|
// one half.
|
||||||
func (v *Vaultik) Prune(opts *PruneOptions) error {
|
func (v *Vaultik) Prune(opts *PruneOptions) error {
|
||||||
err := v.EnsureStorageBinding()
|
err := v.checkStorageBinding()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -267,7 +267,7 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
|
|||||||
|
|
||||||
if !v.confirmPrune(opts, len(unreferencedBlobs), totalSize,
|
if !v.confirmPrune(opts, len(unreferencedBlobs), totalSize,
|
||||||
dirsWithoutManifest) {
|
dirsWithoutManifest) {
|
||||||
return nil
|
return ErrNotConfirmed
|
||||||
}
|
}
|
||||||
|
|
||||||
v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result)
|
v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result)
|
||||||
|
|||||||
@@ -176,7 +176,8 @@ func TestPruneDeclinedKeepsMetadataDirWithoutManifest(t *testing.T) {
|
|||||||
|
|
||||||
env.v.Stdin = strings.NewReader("n\n")
|
env.v.Stdin = strings.NewReader("n\n")
|
||||||
|
|
||||||
require.NoError(t, env.v.PruneBlobs(&vaultik.PruneOptions{}))
|
require.ErrorIs(t, env.v.PruneBlobs(&vaultik.PruneOptions{}),
|
||||||
|
vaultik.ErrNotConfirmed)
|
||||||
|
|
||||||
assert.True(t, env.store.hasKey(leftDB),
|
assert.True(t, env.store.hasKey(leftDB),
|
||||||
"a declined prune must keep the directory")
|
"a declined prune must keep the directory")
|
||||||
|
|||||||
@@ -54,9 +54,10 @@ func TestPurgeSnapshots_QuietPromptListsSnapshots(t *testing.T) {
|
|||||||
v.UI.SetQuiet(true)
|
v.UI.SetQuiet(true)
|
||||||
|
|
||||||
// Stdin is empty, so the prompt reads no answer and cancels.
|
// Stdin is empty, so the prompt reads no answer and cancels.
|
||||||
require.NoError(t, v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||||
KeepLatest: true,
|
KeepLatest: true,
|
||||||
}))
|
})
|
||||||
|
require.ErrorIs(t, err, vaultik.ErrNotConfirmed)
|
||||||
|
|
||||||
assert.Contains(t, out.String(), snapHomeT0)
|
assert.Contains(t, out.String(), snapHomeT0)
|
||||||
assert.Contains(t, out.String(), "Delete 1 snapshot(s)? [y/N]")
|
assert.Contains(t, out.String(), "Delete 1 snapshot(s)? [y/N]")
|
||||||
@@ -77,7 +78,8 @@ func TestPruneBlobs_QuietPromptShowsBlobTotal(t *testing.T) {
|
|||||||
addBlob(t, env.store.testStorer, testBlobHashA)
|
addBlob(t, env.store.testStorer, testBlobHashA)
|
||||||
|
|
||||||
// Stdin is empty, so the prompt reads no answer and cancels.
|
// Stdin is empty, so the prompt reads no answer and cancels.
|
||||||
require.NoError(t, env.v.PruneBlobs(&vaultik.PruneOptions{}))
|
require.ErrorIs(t, env.v.PruneBlobs(&vaultik.PruneOptions{}),
|
||||||
|
vaultik.ErrNotConfirmed)
|
||||||
|
|
||||||
assert.Contains(t, env.stdout.String(),
|
assert.Contains(t, env.stdout.String(),
|
||||||
"Found 1 unreferenced blob(s) totaling")
|
"Found 1 unreferenced blob(s) totaling")
|
||||||
|
|||||||
@@ -337,6 +337,27 @@ func TestRemoveSnapshot_DefaultRemovesMetadataNotBlobs(t *testing.T) {
|
|||||||
assert.Contains(t, out, "vaultik prune")
|
assert.Contains(t, out, "vaultik prune")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestRemoveSnapshot_DeclinedIsError checks that answering no to the
|
||||||
|
// prompt keeps the snapshot and returns ErrNotConfirmed, so the command
|
||||||
|
// exits non-zero.
|
||||||
|
func TestRemoveSnapshot_DeclinedIsError(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
store := newTestStorer()
|
||||||
|
addManifest(t, store, "snapshot-001", nil)
|
||||||
|
|
||||||
|
tv := vaultik.NewForTesting(store)
|
||||||
|
tv.Repositories = newLocalIndex(t, "snapshot-001")
|
||||||
|
tv.Stdin.WriteString("n\n")
|
||||||
|
|
||||||
|
_, err := tv.RemoveSnapshot("snapshot-001", &vaultik.RemoveOptions{})
|
||||||
|
|
||||||
|
require.ErrorIs(t, err, vaultik.ErrNotConfirmed)
|
||||||
|
assert.True(t, store.hasKey(remoteKeyPath("snapshot-001")))
|
||||||
|
assert.Contains(t, tv.Stdout.String(), "Cancelled")
|
||||||
|
}
|
||||||
|
|
||||||
func TestRemoveSnapshot_DryRun(t *testing.T) {
|
func TestRemoveSnapshot_DryRun(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|||||||
+99
-28
@@ -40,6 +40,9 @@ var (
|
|||||||
// which is secret, so the message is safe to print and log.
|
// which is secret, so the message is safe to print and log.
|
||||||
errInvalidAgeSecretKey = errors.New(
|
errInvalidAgeSecretKey = errors.New(
|
||||||
"configured age secret key holds no usable age identity")
|
"configured age secret key holds no usable age identity")
|
||||||
|
// errWrongPrivateKey is returned when the configured key parses but
|
||||||
|
// is not one of the keys the snapshot was encrypted to.
|
||||||
|
errWrongPrivateKey = errors.New("cannot decrypt this snapshot")
|
||||||
errBlobMissingFromIndex = errors.New("blob hash missing from blob index")
|
errBlobMissingFromIndex = errors.New("blob hash missing from blob index")
|
||||||
errChunkNotInAnyBlob = errors.New("chunk not found in any blob")
|
errChunkNotInAnyBlob = errors.New("chunk not found in any blob")
|
||||||
errBlobIDNotInHashIndex = errors.New("blob id missing from hash index")
|
errBlobIDNotInHashIndex = errors.New("blob id missing from hash index")
|
||||||
@@ -106,7 +109,12 @@ type RestoreResult struct {
|
|||||||
BytesDownloaded int64
|
BytesDownloaded int64
|
||||||
Duration time.Duration
|
Duration time.Duration
|
||||||
FilesFailed int
|
FilesFailed int
|
||||||
FailedFiles []string // Paths of files that failed to restore
|
FailedFiles []string // Paths of files that failed to restore for their own reason
|
||||||
|
// Blobs that could not be downloaded, and the paths of the files
|
||||||
|
// skipped because they need one of them. Those files are counted in
|
||||||
|
// FilesFailed but not listed in the summary.
|
||||||
|
BlobsFailed int
|
||||||
|
FilesNeedingFailedBlob []string
|
||||||
// Verification results (only populated if Verify option is set)
|
// Verification results (only populated if Verify option is set)
|
||||||
FilesVerified int
|
FilesVerified int
|
||||||
BytesVerified int64
|
BytesVerified int64
|
||||||
@@ -128,10 +136,19 @@ func (v *Vaultik) Restore(opts *RestoreOptions) error {
|
|||||||
"paths", opts.Paths,
|
"paths", opts.Paths,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
remoteKey, err := v.resolveSnapshotRemoteKey(opts.SnapshotID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
// Step 1: Download and decrypt the snapshot metadata database
|
// Step 1: Download and decrypt the snapshot metadata database
|
||||||
log.Info("Downloading snapshot metadata...")
|
log.Info("Downloading snapshot metadata...")
|
||||||
|
|
||||||
tempDB, tempDir, err := v.downloadSnapshotDB(opts.SnapshotID, identities)
|
tempDB, tempDir, err := v.downloadSnapshotDB(opts.SnapshotID, remoteKey, identities)
|
||||||
|
if errors.Is(err, errWrongPrivateKey) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("downloading snapshot database: %w", err)
|
return fmt.Errorf("downloading snapshot database: %w", err)
|
||||||
}
|
}
|
||||||
@@ -226,6 +243,11 @@ func (v *Vaultik) finishRestore(
|
|||||||
if result.FilesFailed > 0 {
|
if result.FilesFailed > 0 {
|
||||||
v.UI.Warningf("%d file(s) failed to restore:", result.FilesFailed)
|
v.UI.Warningf("%d file(s) failed to restore:", result.FilesFailed)
|
||||||
|
|
||||||
|
if result.BlobsFailed > 0 {
|
||||||
|
v.UI.Detailf("%d of them need a blob that could not be read (%d blob(s) in all).",
|
||||||
|
len(result.FilesNeedingFailedBlob), result.BlobsFailed)
|
||||||
|
}
|
||||||
|
|
||||||
for _, path := range result.FailedFiles {
|
for _, path := range result.FailedFiles {
|
||||||
v.UI.Detailf("%s", v.UI.Path(path))
|
v.UI.Detailf("%s", v.UI.Path(path))
|
||||||
}
|
}
|
||||||
@@ -323,6 +345,28 @@ func (v *Vaultik) restoreIdentities() ([]age.Identity, error) {
|
|||||||
return identities, nil
|
return identities, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// newSnapshotDBReader opens the decrypting reader for a snapshot
|
||||||
|
// database. When none of the identities can decrypt it, the error says
|
||||||
|
// in plain words that the configured private key is the wrong one and
|
||||||
|
// names where it was configured.
|
||||||
|
func (v *Vaultik) newSnapshotDBReader(
|
||||||
|
r io.Reader, identities []age.Identity,
|
||||||
|
) (*blobgen.Reader, error) {
|
||||||
|
blobReader, err := blobgen.NewReader(r, identities...)
|
||||||
|
|
||||||
|
if _, ok := errors.AsType[*age.NoIdentityMatchError](err); ok {
|
||||||
|
return nil, fmt.Errorf(
|
||||||
|
"%w: the private key in %s is not one of the keys the backup was encrypted to",
|
||||||
|
errWrongPrivateKey, v.Config.AgeSecretKeySourceName())
|
||||||
|
}
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return blobReader, nil
|
||||||
|
}
|
||||||
|
|
||||||
// restoreAllFiles processes files in blob-locality order: drain every
|
// restoreAllFiles processes files in blob-locality order: drain every
|
||||||
// file whose blob set is on disk, download the missing blobs for the
|
// file whose blob set is on disk, download the missing blobs for the
|
||||||
// pending file with the smallest uncached count, repeat. This keeps
|
// pending file with the smallest uncached count, repeat. This keeps
|
||||||
@@ -494,8 +538,8 @@ func (v *Vaultik) runRestoreLoop(
|
|||||||
// Returns false when nothing is pending download (the caller stops).
|
// Returns false when nothing is pending download (the caller stops).
|
||||||
//
|
//
|
||||||
// A blob that cannot be downloaded is reported through
|
// A blob that cannot be downloaded is reported through
|
||||||
// handleRestoreFileError for every pending file that references it, so
|
// handleRestoreBlobError, so it aborts the restore unless --skip-errors
|
||||||
// it aborts the restore unless --skip-errors is set.
|
// is set.
|
||||||
func (s *restoreSession) downloadNextBlobSet(
|
func (s *restoreSession) downloadNextBlobSet(
|
||||||
plan *restorePlan, filesByID map[types.FileID]*database.File,
|
plan *restorePlan, filesByID map[types.FileID]*database.File,
|
||||||
) (bool, error) {
|
) (bool, error) {
|
||||||
@@ -528,12 +572,11 @@ func (s *restoreSession) downloadNextBlobSet(
|
|||||||
return false, err
|
return false, err
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, fileID := range plan.filesReferencingBlob(hash) {
|
blobErr := s.v.handleRestoreBlobError(
|
||||||
fileErr := s.v.handleRestoreFileError(
|
plan, s.opts, s.result, filesByID,
|
||||||
plan, s.opts, s.result, filesByID[fileID], fileID, err)
|
plan.filesReferencingBlob(hash), err)
|
||||||
if fileErr != nil {
|
if blobErr != nil {
|
||||||
return false, fileErr
|
return false, blobErr
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// next is among the failed files, so the rest of its blob set
|
// next is among the failed files, so the rest of its blob set
|
||||||
@@ -624,19 +667,19 @@ func (v *Vaultik) restoreProgressTick(
|
|||||||
|
|
||||||
// handleRestoreFileError records a per-file restore failure: fatal unless
|
// handleRestoreFileError records a per-file restore failure: fatal unless
|
||||||
// --skip-errors is set, in which case the file is counted as failed and
|
// --skip-errors is set, in which case the file is counted as failed and
|
||||||
// dropped from the plan.
|
// dropped from the plan. A fatal failure is not logged here: the returned
|
||||||
|
// error is printed as the command's ERROR: line.
|
||||||
func (v *Vaultik) handleRestoreFileError(
|
func (v *Vaultik) handleRestoreFileError(
|
||||||
plan *restorePlan, opts *RestoreOptions, result *RestoreResult,
|
plan *restorePlan, opts *RestoreOptions, result *RestoreResult,
|
||||||
file *database.File, fileID types.FileID, err error,
|
file *database.File, fileID types.FileID, err error,
|
||||||
) error {
|
) error {
|
||||||
log.Error("Failed to restore file", "path", file.Path, "error", err)
|
|
||||||
|
|
||||||
if !opts.SkipErrors {
|
if !opts.SkipErrors {
|
||||||
return fmt.Errorf(
|
return fmt.Errorf(
|
||||||
"restoring %s: %w (pass --skip-errors to continue past "+
|
"restoring %s: %w (pass --skip-errors to continue past "+
|
||||||
"restore failures)", file.Path, err)
|
"restore failures)", file.Path, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
log.Error("Failed to restore file", "path", file.Path, "error", err)
|
||||||
v.UI.Errorf("Failed to restore %s: %v. Skipping (--skip-errors).",
|
v.UI.Errorf("Failed to restore %s: %v. Skipping (--skip-errors).",
|
||||||
v.UI.Path(file.Path.String()), err)
|
v.UI.Path(file.Path.String()), err)
|
||||||
|
|
||||||
@@ -648,6 +691,41 @@ func (v *Vaultik) handleRestoreFileError(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// handleRestoreBlobError records a blob that could not be downloaded.
|
||||||
|
// fileIDs are the pending files that need it, never empty. Without
|
||||||
|
// --skip-errors the first of them fails the restore. With it, the blob is
|
||||||
|
// reported once, and its files are counted as failed and dropped from
|
||||||
|
// the plan without being reported one by one.
|
||||||
|
func (v *Vaultik) handleRestoreBlobError(
|
||||||
|
plan *restorePlan, opts *RestoreOptions, result *RestoreResult,
|
||||||
|
filesByID map[types.FileID]*database.File, fileIDs []types.FileID,
|
||||||
|
err error,
|
||||||
|
) error {
|
||||||
|
if !opts.SkipErrors {
|
||||||
|
return v.handleRestoreFileError(
|
||||||
|
plan, opts, result, filesByID[fileIDs[0]], fileIDs[0], err)
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Error("Failed to restore files", "files", len(fileIDs), "error", err)
|
||||||
|
v.UI.Errorf("Failed to restore %d file(s): %v. Skipping them (--skip-errors).",
|
||||||
|
len(fileIDs), err)
|
||||||
|
|
||||||
|
result.BlobsFailed++
|
||||||
|
|
||||||
|
for _, fileID := range fileIDs {
|
||||||
|
path := filesByID[fileID].Path.String()
|
||||||
|
|
||||||
|
log.Debug("Skipped file that needs a failed blob", "path", path)
|
||||||
|
|
||||||
|
result.FilesFailed++
|
||||||
|
result.FilesNeedingFailedBlob = append(result.FilesNeedingFailedBlob, path)
|
||||||
|
|
||||||
|
plan.finishFile(fileID)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// printRestoreProgress emits a periodic restore-phase status line via
|
// printRestoreProgress emits a periodic restore-phase status line via
|
||||||
// the UI writer, mirroring scanner.printProcessingProgress so the two
|
// the UI writer, mirroring scanner.printProcessingProgress so the two
|
||||||
// long-running commands have the same on-screen rhythm.
|
// long-running commands have the same on-screen rhythm.
|
||||||
@@ -719,8 +797,8 @@ func (v *Vaultik) handleRestoreVerification(
|
|||||||
) error {
|
) error {
|
||||||
// A file that failed to restore was not written and is already
|
// A file that failed to restore was not written and is already
|
||||||
// reported as such, so it is not verified.
|
// reported as such, so it is not verified.
|
||||||
failedToRestore := make(map[string]bool, len(result.FailedFiles))
|
failedToRestore := make(map[string]bool, result.FilesFailed)
|
||||||
for _, path := range result.FailedFiles {
|
for _, path := range slices.Concat(result.FailedFiles, result.FilesNeedingFailedBlob) {
|
||||||
failedToRestore[path] = true
|
failedToRestore[path] = true
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -754,19 +832,12 @@ func (v *Vaultik) handleRestoreVerification(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// downloadSnapshotDB downloads and decrypts the snapshot metadata
|
// downloadSnapshotDB downloads and decrypts the metadata database of the
|
||||||
// database. The identifier is resolved to the snapshot's remote key: a
|
// snapshot at remoteKey, which resolveSnapshotRemoteKey returned for
|
||||||
// human ID is hashed, and a remote key (or its abbreviation, as printed
|
// snapshotID.
|
||||||
// for a remote-only snapshot) is used as-is, so a host with no local
|
|
||||||
// index can restore the snapshots it can only see on the store.
|
|
||||||
func (v *Vaultik) downloadSnapshotDB(
|
func (v *Vaultik) downloadSnapshotDB(
|
||||||
snapshotID string, identities []age.Identity,
|
snapshotID, remoteKey string, identities []age.Identity,
|
||||||
) (*database.DB, string, error) {
|
) (*database.DB, string, error) {
|
||||||
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
|
|
||||||
if err != nil {
|
|
||||||
return nil, "", err
|
|
||||||
}
|
|
||||||
|
|
||||||
// Download encrypted database from storage
|
// Download encrypted database from storage
|
||||||
dbKey := fmt.Sprintf("metadata/%s/db.zst.age", remoteKey)
|
dbKey := fmt.Sprintf("metadata/%s/db.zst.age", remoteKey)
|
||||||
|
|
||||||
@@ -783,9 +854,9 @@ func (v *Vaultik) downloadSnapshotDB(
|
|||||||
// the plaintext to a temp file. Neither the encrypted bytes nor the
|
// the plaintext to a temp file. Neither the encrypted bytes nor the
|
||||||
// decrypted database is ever held whole in memory; a snapshot database
|
// decrypted database is ever held whole in memory; a snapshot database
|
||||||
// can be large.
|
// can be large.
|
||||||
blobReader, err := blobgen.NewReader(reader, identities...)
|
blobReader, err := v.newSnapshotDBReader(reader, identities)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, "", fmt.Errorf("creating decryption reader: %w", err)
|
return nil, "", err
|
||||||
}
|
}
|
||||||
|
|
||||||
defer func() { _ = blobReader.Close() }()
|
defer func() { _ = blobReader.Close() }()
|
||||||
|
|||||||
@@ -46,11 +46,11 @@ import (
|
|||||||
// slice out a few KB — currently the dominant cost in restore.
|
// slice out a few KB — currently the dominant cost in restore.
|
||||||
//
|
//
|
||||||
// The test deliberately constructs an adversarial scenario: three
|
// The test deliberately constructs an adversarial scenario: three
|
||||||
// blobs A/B/C of ~6 MB each, nine files distributed across them, and
|
// blobs A/B/C of five 64 KiB chunks each, nine files distributed across
|
||||||
// path-ordered names that interleave the blobs (a1, b1, c1, a2, b2,
|
// them, and path-ordered names that interleave the blobs (a1, b1, c1,
|
||||||
// c2, …) so naive path-order processing would touch every blob before
|
// a2, b2, c2, …) so naive path-order processing would touch every blob
|
||||||
// finishing any of them.
|
// before finishing any of them.
|
||||||
// localitySource is one 1 MiB fixture file used by the locality test.
|
// localitySource is one 64 KiB fixture file used by the locality test.
|
||||||
type localitySource struct {
|
type localitySource struct {
|
||||||
path string
|
path string
|
||||||
data []byte
|
data []byte
|
||||||
@@ -65,10 +65,10 @@ type localityCopy struct {
|
|||||||
|
|
||||||
// localitySourceBytes is the size of each source file of the locality
|
// localitySourceBytes is the size of each source file of the locality
|
||||||
// fixture.
|
// fixture.
|
||||||
const localitySourceBytes = 1024 * 1024
|
const localitySourceBytes = 64 * 1024
|
||||||
|
|
||||||
// buildLocalityFixture writes the adversarial source layout described
|
// buildLocalityFixture writes the adversarial source layout described
|
||||||
// in TestRestoreLocalityAndReadAt: 15 sources of 1 MiB each (which the
|
// in TestRestoreLocalityAndReadAt: 15 sources of 64 KiB each (which the
|
||||||
// backup packs into 3 blobs of 5 chunks) plus 9 interleaved-name copies
|
// backup packs into 3 blobs of 5 chunks) plus 9 interleaved-name copies
|
||||||
// (3 per blob group).
|
// (3 per blob group).
|
||||||
func buildLocalityFixture(
|
func buildLocalityFixture(
|
||||||
|
|||||||
@@ -51,27 +51,72 @@ func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
assertRestoreSkipsLostFiles(ctx, t, backupThenDeleteOneBlob(ctx, t))
|
backup := backupThenDeleteOneBlob(ctx, t)
|
||||||
|
out := assertRestoreSkipsLostFiles(ctx, t, backup)
|
||||||
|
|
||||||
|
assertLostFilesCountedNotListed(t, backup, out)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRestoreSkipErrorsReportsUnreadableBlobOnce restores with SkipErrors
|
||||||
|
// after one blob of a two-blob snapshot was overwritten with other bytes
|
||||||
|
// of the same size. The blob must be reported once, by its hash, as not a
|
||||||
|
// valid age stream, without quoting its bytes, and its files must be
|
||||||
|
// counted, not listed.
|
||||||
|
func TestRestoreSkipErrorsReportsUnreadableBlobOnce(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
backup, lostBlob := backupThenOverwriteOneBlob(ctx, t)
|
||||||
|
out := assertRestoreSkipsLostFiles(ctx, t, backup)
|
||||||
|
|
||||||
|
assert.Equal(t, 1, strings.Count(out, "not a valid age stream"), out)
|
||||||
|
assert.Contains(t, out, lostBlob[:16])
|
||||||
|
assert.NotContains(t, out, overwriteBlobPattern)
|
||||||
|
assertLostFilesCountedNotListed(t, backup, out)
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestRestoreSkipErrorsSkipsFilesWithChunkInNoBlob restores with
|
// TestRestoreSkipErrorsSkipsFilesWithChunkInNoBlob restores with
|
||||||
// SkipErrors a snapshot whose metadata lists chunks that no blob holds, as
|
// SkipErrors a snapshot whose metadata lists chunks that no blob holds, as
|
||||||
// earlier backup defects left behind.
|
// earlier backup defects left behind. Each such file fails for its own
|
||||||
|
// reason, so each is listed.
|
||||||
func TestRestoreSkipErrorsSkipsFilesWithChunkInNoBlob(t *testing.T) {
|
func TestRestoreSkipErrorsSkipsFilesWithChunkInNoBlob(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
assertRestoreSkipsLostFiles(ctx, t, backupThenDropOneBlobFromMetadata(ctx, t))
|
backup := backupThenDropOneBlobFromMetadata(ctx, t)
|
||||||
|
out := assertRestoreSkipsLostFiles(ctx, t, backup)
|
||||||
|
|
||||||
|
for path := range backup.lost {
|
||||||
|
assert.Containsf(t, out, path, "%s must be reported", path)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// assertRestoreSkipsLostFiles restores backup with SkipErrors. Every file
|
// assertLostFilesCountedNotListed checks that the restore output out
|
||||||
// stored in the lost blob must be reported as failed and left absent,
|
// counts the files of the lost blob and names none of them.
|
||||||
// every other file must be restored intact, and Restore must still return
|
func assertLostFilesCountedNotListed(
|
||||||
// an error.
|
t *testing.T, backup *missingBlobBackup, out string,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
assert.Contains(t, out, fmt.Sprintf(
|
||||||
|
"%d of them need a blob that could not be read (1 blob(s) in all)",
|
||||||
|
len(backup.lost)))
|
||||||
|
|
||||||
|
for path := range backup.lost {
|
||||||
|
assert.NotContainsf(t, out, path,
|
||||||
|
"%s needs the lost blob and must be counted, not listed", path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertRestoreSkipsLostFiles restores backup with SkipErrors and returns
|
||||||
|
// what it printed. Every file stored in the lost blob must be left
|
||||||
|
// absent, every other file must be restored intact, and Restore must
|
||||||
|
// still return an error.
|
||||||
func assertRestoreSkipsLostFiles(
|
func assertRestoreSkipsLostFiles(
|
||||||
ctx context.Context, t *testing.T, backup *missingBlobBackup,
|
ctx context.Context, t *testing.T, backup *missingBlobBackup,
|
||||||
) {
|
) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
@@ -93,8 +138,6 @@ func assertRestoreSkipsLostFiles(
|
|||||||
restored := filepath.Join(backup.restoreDir, path)
|
restored := filepath.Join(backup.restoreDir, path)
|
||||||
|
|
||||||
if backup.lost[path] {
|
if backup.lost[path] {
|
||||||
assert.Containsf(t, out.String(), path,
|
|
||||||
"%s needs the lost blob and must be reported", path)
|
|
||||||
assert.NoFileExists(t, restored)
|
assert.NoFileExists(t, restored)
|
||||||
|
|
||||||
continue
|
continue
|
||||||
@@ -104,6 +147,8 @@ func assertRestoreSkipsLostFiles(
|
|||||||
require.NoErrorf(t, err, "%s does not need the lost blob", path)
|
require.NoErrorf(t, err, "%s does not need the lost blob", path)
|
||||||
assert.Equalf(t, content, got, "%s restored with wrong content", path)
|
assert.Equalf(t, content, got, "%s restored with wrong content", path)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return out.String()
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestRestoreSkipErrorsVerifyChecksOnlyRestoredFiles restores with
|
// TestRestoreSkipErrorsVerifyChecksOnlyRestoredFiles restores with
|
||||||
@@ -144,9 +189,9 @@ func TestRestoreSkipErrorsVerifyChecksOnlyRestoredFiles(t *testing.T) {
|
|||||||
// TestRestoreSkipErrorsVerifyReportsFailuresApart restores with
|
// TestRestoreSkipErrorsVerifyReportsFailuresApart restores with
|
||||||
// SkipErrors and Verify after one blob of a two-blob snapshot was
|
// SkipErrors and Verify after one blob of a two-blob snapshot was
|
||||||
// deleted, while one file that does restore is written one byte too long.
|
// deleted, while one file that does restore is written one byte too long.
|
||||||
// The restore warning must list only the files of the deleted blob, the
|
// The restore warning must count the files of the deleted blob without
|
||||||
// verification summary only the long file, each path once, and Restore
|
// listing them, the verification summary must list only the long file,
|
||||||
// must return an error.
|
// once, and Restore must return an error.
|
||||||
func TestRestoreSkipErrorsVerifyReportsFailuresApart(t *testing.T) {
|
func TestRestoreSkipErrorsVerifyReportsFailuresApart(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -186,9 +231,9 @@ func TestRestoreSkipErrorsVerifyReportsFailuresApart(t *testing.T) {
|
|||||||
"Verification failed: 1 files did not match")
|
"Verification failed: 1 files did not match")
|
||||||
require.Truef(t, found, "no verification summary in:\n%s", out.String())
|
require.Truef(t, found, "no verification summary in:\n%s", out.String())
|
||||||
|
|
||||||
|
assertLostFilesCountedNotListed(t, backup, warning)
|
||||||
|
|
||||||
for path := range backup.lost {
|
for path := range backup.lost {
|
||||||
assert.Equalf(t, 1, strings.Count(warning, path),
|
|
||||||
"%s must be listed once as failed to restore", path)
|
|
||||||
assert.NotContainsf(t, summary, path,
|
assert.NotContainsf(t, summary, path,
|
||||||
"%s was not restored and must not fail verification", path)
|
"%s was not restored and must not fail verification", path)
|
||||||
}
|
}
|
||||||
@@ -292,6 +337,35 @@ func backupThenDeleteOneBlob(
|
|||||||
return backup
|
return backup
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// overwriteBlobPattern repeats, each time followed by a newline, through
|
||||||
|
// the bytes that backupThenOverwriteOneBlob writes over a blob. age's own
|
||||||
|
// error quotes the data's first line only when it reads a whole line.
|
||||||
|
const overwriteBlobPattern = "overwritten-blob-bytes-"
|
||||||
|
|
||||||
|
// backupThenOverwriteOneBlob backs up missingBlobFileCount single-chunk
|
||||||
|
// files, then overwrites the blob holding the first of them with as many
|
||||||
|
// bytes of overwriteBlobPattern lines. It also returns that blob's hash.
|
||||||
|
func backupThenOverwriteOneBlob(
|
||||||
|
ctx context.Context, t *testing.T,
|
||||||
|
) (*missingBlobBackup, string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
backup, db, lostBlob := backupSingleChunkFiles(ctx, t)
|
||||||
|
require.NoError(t, db.Close())
|
||||||
|
|
||||||
|
key := fmt.Sprintf("blobs/%s/%s/%s", lostBlob[:2], lostBlob[2:4], lostBlob)
|
||||||
|
|
||||||
|
info, err := backup.storer.Stat(ctx, key)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
line := overwriteBlobPattern + "\n"
|
||||||
|
pattern := bytes.Repeat([]byte(line), int(info.Size)/len(line)+1)
|
||||||
|
require.NoError(t, backup.storer.Put(
|
||||||
|
ctx, key, bytes.NewReader(pattern[:info.Size])))
|
||||||
|
|
||||||
|
return backup, lostBlob
|
||||||
|
}
|
||||||
|
|
||||||
// backupThenDropOneBlobFromMetadata backs up missingBlobFileCount
|
// backupThenDropOneBlobFromMetadata backs up missingBlobFileCount
|
||||||
// single-chunk files, then removes the blob holding the first of them from
|
// single-chunk files, then removes the blob holding the first of them from
|
||||||
// the snapshot and exports the snapshot metadata again. The exported
|
// the snapshot and exports the snapshot metadata again. The exported
|
||||||
|
|||||||
@@ -27,10 +27,10 @@ import (
|
|||||||
// TestRestoreSweeperEvictsBlobs exercises the reference-counted blob
|
// TestRestoreSweeperEvictsBlobs exercises the reference-counted blob
|
||||||
// disk cache eviction during restore.
|
// disk cache eviction during restore.
|
||||||
//
|
//
|
||||||
// The scenario: 30 unique 1 MB random files plus 10 duplicates of those
|
// The scenario: 30 unique 128 KiB random files plus 10 duplicates of
|
||||||
// (40 files total, 30 MB of unique content) get backed up with a 10 MB
|
// those (40 files total, 3.75 MiB of unique content) get backed up with
|
||||||
// blob_size_limit. After backup the snapshot's encrypted blobs are
|
// a 1 MiB blob_size_limit. After backup the snapshot's encrypted blobs
|
||||||
// restored through Vaultik.Restore, and per-key Get counts on the
|
// are restored through Vaultik.Restore, and per-key Get counts on the
|
||||||
// storage layer are recorded. Each blob in the snapshot MUST be
|
// storage layer are recorded. Each blob in the snapshot MUST be
|
||||||
// downloaded exactly once — re-downloads would mean the sweeper either
|
// downloaded exactly once — re-downloads would mean the sweeper either
|
||||||
// evicted a blob that was still needed (LRU regression) or that the
|
// evicted a blob that was still needed (LRU regression) or that the
|
||||||
@@ -41,7 +41,7 @@ import (
|
|||||||
// original AND every duplicate referencing its chunks have been
|
// original AND every duplicate referencing its chunks have been
|
||||||
// restored.
|
// restored.
|
||||||
// buildSweeperFixture writes the source layout for the sweeper test: 30
|
// buildSweeperFixture writes the source layout for the sweeper test: 30
|
||||||
// unique 1 MB random files plus 10 duplicates of a random subset. The
|
// unique 128 KiB random files plus 10 duplicates of a random subset. The
|
||||||
// PRNG seed is fixed so failures are reproducible; the entropy is what
|
// PRNG seed is fixed so failures are reproducible; the entropy is what
|
||||||
// matters here — the FastCDC chunker needs realistic-looking data to
|
// matters here — the FastCDC chunker needs realistic-looking data to
|
||||||
// pick chunk boundaries naturally. Returns the expected content by
|
// pick chunk boundaries naturally. Returns the expected content by
|
||||||
@@ -51,7 +51,7 @@ func buildSweeperFixture(
|
|||||||
) map[string][]byte {
|
) map[string][]byte {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
const fileSize = 1 * 1024 * 1024
|
const fileSize = 128 * 1024
|
||||||
|
|
||||||
rng := rand.New(rand.NewSource(42)) //nolint:gosec // G404: fixture data only
|
rng := rand.New(rand.NewSource(42)) //nolint:gosec // G404: fixture data only
|
||||||
|
|
||||||
@@ -159,7 +159,7 @@ func runSweeperBackup(
|
|||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
require.Equal(t, uniqueFiles+duplicateFiles, scanResult.FilesScanned)
|
require.Equal(t, uniqueFiles+duplicateFiles, scanResult.FilesScanned)
|
||||||
require.Greater(t, scanResult.BlobsCreated, 1,
|
require.Greater(t, scanResult.BlobsCreated, 1,
|
||||||
"30 MB of unique data at 10 MB blob size should yield multiple blobs")
|
"3.75 MiB of unique data at 1 MiB blob size should yield multiple blobs")
|
||||||
|
|
||||||
require.NoError(t, sm.CompleteSnapshot(ctx, snapshotID))
|
require.NoError(t, sm.CompleteSnapshot(ctx, snapshotID))
|
||||||
require.NoError(t, sm.ExportSnapshotMetadata(ctx, dbPath, snapshotID))
|
require.NoError(t, sm.ExportSnapshotMetadata(ctx, dbPath, snapshotID))
|
||||||
@@ -199,8 +199,8 @@ func TestRestoreSweeperEvictsBlobs(t *testing.T) {
|
|||||||
|
|
||||||
expected := buildSweeperFixture(t, fs, dataDir, uniqueFiles, duplicateFiles)
|
expected := buildSweeperFixture(t, fs, dataDir, uniqueFiles, duplicateFiles)
|
||||||
|
|
||||||
chunkSize := int64(64 * 1024)
|
chunkSize := int64(16 * 1024)
|
||||||
maxBlobSize := int64(10 * 1024 * 1024)
|
maxBlobSize := int64(1024 * 1024)
|
||||||
|
|
||||||
storer, err := storage.NewFileStorer(storeDir)
|
storer, err := storage.NewFileStorer(storeDir)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|||||||
@@ -0,0 +1,161 @@
|
|||||||
|
package vaultik_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"path"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"filippo.io/age"
|
||||||
|
"github.com/spf13/afero"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/vaultik/internal/blobgen"
|
||||||
|
"sneak.berlin/go/vaultik/internal/config"
|
||||||
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
|
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||||
|
"sneak.berlin/go/vaultik/internal/ui"
|
||||||
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestWrongPrivateKeyError checks that restore and deep verify, given a
|
||||||
|
// valid private key the backup was not encrypted to, fail with one plain
|
||||||
|
// sentence that names where the key was configured.
|
||||||
|
func TestWrongPrivateKeyError(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
store := NewMockStorer()
|
||||||
|
|
||||||
|
const snapshotID = "test-host_2026-01-01T00:00:00Z"
|
||||||
|
|
||||||
|
remoteKey := snapshot.RemoteSnapshotKey(snapshotID)
|
||||||
|
|
||||||
|
manifest, err := snapshot.EncodeManifest(
|
||||||
|
&snapshot.Manifest{SnapshotID: remoteKey}, 3)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, store.Put(ctx,
|
||||||
|
"metadata/"+remoteKey+"/manifest.json.zst", bytes.NewReader(manifest)))
|
||||||
|
|
||||||
|
// The database is encrypted to the test recipient only.
|
||||||
|
var db bytes.Buffer
|
||||||
|
|
||||||
|
writer, err := blobgen.NewWriter(&db, 1, []string{testAgePublicKey})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
_, err = writer.Write([]byte("snapshot database"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, writer.Close())
|
||||||
|
require.NoError(t, store.Put(ctx,
|
||||||
|
"metadata/"+remoteKey+"/db.zst.age", &db))
|
||||||
|
|
||||||
|
otherKey, err := age.GenerateX25519Identity()
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
v := &vaultik.Vaultik{
|
||||||
|
Config: &config.Config{AgeSecretKey: otherKey.String()},
|
||||||
|
Storage: store,
|
||||||
|
Fs: afero.NewMemMapFs(),
|
||||||
|
Stdout: io.Discard,
|
||||||
|
Stderr: io.Discard,
|
||||||
|
UI: ui.NewWithColor(io.Discard, false),
|
||||||
|
}
|
||||||
|
v.SetContext(ctx)
|
||||||
|
|
||||||
|
// A Config built directly has no recorded source, so the error names
|
||||||
|
// the config-file field.
|
||||||
|
const want = "cannot decrypt this snapshot: the private key in " +
|
||||||
|
"age_secret_key is not one of the keys the backup was encrypted to"
|
||||||
|
|
||||||
|
err = v.Restore(&vaultik.RestoreOptions{
|
||||||
|
SnapshotID: snapshotID,
|
||||||
|
TargetDir: t.TempDir(),
|
||||||
|
})
|
||||||
|
require.EqualError(t, err, want)
|
||||||
|
|
||||||
|
err = v.RunDeepVerify(snapshotID, &vaultik.VerifyOptions{Deep: true})
|
||||||
|
require.EqualError(t, err, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestWrongPrivateKeyForBlobError backs up a file, then backs it up again
|
||||||
|
// from an empty local index with a second recipient added. The second
|
||||||
|
// snapshot reuses the stored blob, which stays encrypted to the first
|
||||||
|
// recipient only. With only the added key configured, restore and deep
|
||||||
|
// verify read the database but fail on the blob with one plain sentence
|
||||||
|
// that names where the key was configured.
|
||||||
|
func TestWrongPrivateKeyForBlobError(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
fs := afero.NewOsFs()
|
||||||
|
tempDir := t.TempDir()
|
||||||
|
dataDir := filepath.Join(tempDir, "source")
|
||||||
|
dataPath := filepath.Join(dataDir, "data.bin")
|
||||||
|
storeDir := t.TempDir()
|
||||||
|
secondDBPath := filepath.Join(tempDir, "second.sqlite")
|
||||||
|
|
||||||
|
chunkSize := int64(64 * 1024)
|
||||||
|
maxBlobSize := int64(512 * 1024)
|
||||||
|
|
||||||
|
require.NoError(t, fs.MkdirAll(dataDir, 0o755))
|
||||||
|
require.NoError(t, afero.WriteFile(fs, dataPath,
|
||||||
|
bytesPattern("key-", int(chunkSize)), 0o644))
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
runFileStorageBackup(ctx, t, fs, dataDir, storeDir,
|
||||||
|
filepath.Join(tempDir, "first.sqlite"), chunkSize, maxBlobSize)
|
||||||
|
|
||||||
|
env := setupDedupBackupEnv(
|
||||||
|
ctx, t, fs, storeDir, secondDBPath, chunkSize, maxBlobSize)
|
||||||
|
|
||||||
|
defer func() { _ = env.db.Close() }()
|
||||||
|
|
||||||
|
addedKey, err := age.GenerateX25519Identity()
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
env.cfg.AgeRecipients = append(env.cfg.AgeRecipients,
|
||||||
|
addedKey.Recipient().String())
|
||||||
|
|
||||||
|
snapshotID, result := runDedupSnapshot(ctx, t, env.sm, env.makeScanner(),
|
||||||
|
env.cfg.Hostname, "second", dataDir, secondDBPath)
|
||||||
|
require.Zero(t, result.BlobsUploaded,
|
||||||
|
"the second backup should reuse the stored blob")
|
||||||
|
|
||||||
|
require.NoError(t, env.db.Close())
|
||||||
|
|
||||||
|
blobKeys, err := env.storer.List(ctx, "blobs/")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, blobKeys, 1)
|
||||||
|
|
||||||
|
blobHash := path.Base(blobKeys[0])
|
||||||
|
|
||||||
|
env.cfg.AgeSecretKey = addedKey.String()
|
||||||
|
|
||||||
|
v := &vaultik.Vaultik{
|
||||||
|
Config: env.cfg,
|
||||||
|
Storage: env.storer,
|
||||||
|
Fs: fs,
|
||||||
|
Stdout: io.Discard,
|
||||||
|
Stderr: io.Discard,
|
||||||
|
UI: ui.NewWithColor(io.Discard, false),
|
||||||
|
}
|
||||||
|
v.SetContext(ctx)
|
||||||
|
|
||||||
|
const sentence = "cannot decrypt this blob: the private key in " +
|
||||||
|
"age_secret_key is not one of the keys the blob was encrypted to"
|
||||||
|
|
||||||
|
// Restore names the blob by the first 16 characters of its hash.
|
||||||
|
err = v.Restore(&vaultik.RestoreOptions{
|
||||||
|
SnapshotID: snapshotID,
|
||||||
|
TargetDir: t.TempDir(),
|
||||||
|
})
|
||||||
|
require.EqualError(t, err, "restoring "+dataPath+": downloading blob "+
|
||||||
|
blobHash[:16]+": "+sentence+
|
||||||
|
" (pass --skip-errors to continue past restore failures)")
|
||||||
|
|
||||||
|
err = v.RunDeepVerify(snapshotID, &vaultik.VerifyOptions{Deep: true})
|
||||||
|
require.EqualError(t, err,
|
||||||
|
"blob "+blobHash+" verification failed: "+sentence)
|
||||||
|
}
|
||||||
@@ -115,14 +115,9 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
|||||||
"index_path", v.Config.IndexPath,
|
"index_path", v.Config.IndexPath,
|
||||||
)
|
)
|
||||||
|
|
||||||
err := v.EnsureStorageBinding()
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
// Before PruneDatabase: it drops incomplete snapshots, which could
|
// Before PruneDatabase: it drops incomplete snapshots, which could
|
||||||
// leave the local index without one and so skip the marker check.
|
// leave the local index without one and so skip the marker check.
|
||||||
err = v.markDestination()
|
err := v.markDestination()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -749,7 +744,7 @@ type SnapshotPurgeOptions struct {
|
|||||||
// snapshot name, not the latest globally. This prevents `home` and `system`
|
// snapshot name, not the latest globally. This prevents `home` and `system`
|
||||||
// snapshots from cannibalizing each other.
|
// snapshots from cannibalizing each other.
|
||||||
func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
|
func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
|
||||||
err := v.EnsureStorageBinding()
|
err := v.checkStorageBinding()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -881,17 +876,12 @@ func (v *Vaultik) confirmAndExecutePurge(
|
|||||||
|
|
||||||
var confirm string
|
var confirm string
|
||||||
|
|
||||||
|
// A failure to read the answer means no confirmation.
|
||||||
_, err := v.scanStdin(&confirm)
|
_, err := v.scanStdin(&confirm)
|
||||||
if err != nil {
|
if err != nil || strings.ToLower(confirm) != "y" {
|
||||||
v.printlnStdout("Cancelled")
|
v.printlnStdout("Cancelled")
|
||||||
|
|
||||||
return nil //nolint:nilerr // treat EOF or read error as "no"
|
return ErrNotConfirmed
|
||||||
}
|
|
||||||
|
|
||||||
if strings.ToLower(confirm) != "y" {
|
|
||||||
v.printlnStdout("Cancelled")
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
} else if !quiet {
|
} else if !quiet {
|
||||||
v.UI.Infof("The following snapshots will be deleted:")
|
v.UI.Infof("The following snapshots will be deleted:")
|
||||||
@@ -969,25 +959,22 @@ func (v *Vaultik) VerifySnapshotWithOptions(
|
|||||||
Mode: "shallow",
|
Mode: "shallow",
|
||||||
}
|
}
|
||||||
|
|
||||||
v.printVerifyHeader(snapshotID, opts)
|
// The key is kept so we can also check for the snapshot's encrypted
|
||||||
|
// database below.
|
||||||
// Resolve the identifier to the snapshot's remote key. A human ID is
|
|
||||||
// hashed; a remote key (or its abbreviation, as printed for a
|
|
||||||
// remote-only snapshot) is used as-is, so a host with no local index
|
|
||||||
// can verify a snapshot it can only see on the store. The key is kept
|
|
||||||
// so we can also check for the snapshot's encrypted database below.
|
|
||||||
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
|
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if opts.JSON {
|
if opts.JSON {
|
||||||
result.Status = verifyStatusFailed
|
result.Status = verifyStatusFailed
|
||||||
result.ErrorMessage = fmt.Sprintf("resolving snapshot identifier: %v", err)
|
result.ErrorMessage = err.Error()
|
||||||
|
|
||||||
return v.outputVerifyJSON(result)
|
return v.outputVerifyJSON(result)
|
||||||
}
|
}
|
||||||
|
|
||||||
return fmt.Errorf("resolving snapshot identifier: %w", err)
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
v.printVerifyHeader(snapshotID, opts)
|
||||||
|
|
||||||
manifest, err := v.downloadManifestByKey(remoteKey)
|
manifest, err := v.downloadManifestByKey(remoteKey)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if opts.JSON {
|
if opts.JSON {
|
||||||
@@ -1285,7 +1272,7 @@ func (v *Vaultik) outputVerifyJSON(result *VerifyResult) error {
|
|||||||
// jq` fail (issue #108). The narration is duplicated as log records,
|
// jq` fail (issue #108). The narration is duplicated as log records,
|
||||||
// which go to stderr and so cannot corrupt the document.
|
// which go to stderr and so cannot corrupt the document.
|
||||||
func (v *Vaultik) CleanupLocalSnapshots(opts *PruneOptions) error {
|
func (v *Vaultik) CleanupLocalSnapshots(opts *PruneOptions) error {
|
||||||
err := v.EnsureStorageBinding()
|
err := v.checkStorageBinding()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -1475,12 +1462,12 @@ func (v *Vaultik) RemoveSnapshot(
|
|||||||
SnapshotID: snapshotID,
|
SnapshotID: snapshotID,
|
||||||
}
|
}
|
||||||
|
|
||||||
err := v.EnsureStorageBinding()
|
err := v.checkStorageBinding()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return result, err
|
return result, err
|
||||||
}
|
}
|
||||||
|
|
||||||
inLocalIndex, err := v.findSnapshotToRemove(snapshotID, opts)
|
inLocalIndex, err := v.checkSnapshotExists(snapshotID, opts.LocalOnly)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return result, err
|
return result, err
|
||||||
}
|
}
|
||||||
@@ -1490,7 +1477,7 @@ func (v *Vaultik) RemoveSnapshot(
|
|||||||
}
|
}
|
||||||
|
|
||||||
if !v.confirmRemoveSnapshot(snapshotID, opts) {
|
if !v.confirmRemoveSnapshot(snapshotID, opts) {
|
||||||
return result, nil
|
return result, ErrNotConfirmed
|
||||||
}
|
}
|
||||||
|
|
||||||
// Checked before the local record is deleted: it may be the last
|
// Checked before the local record is deleted: it may be the last
|
||||||
@@ -1543,17 +1530,20 @@ func (v *Vaultik) RemoveSnapshot(
|
|||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// findSnapshotToRemove reports whether the local index has a row for
|
// checkSnapshotExists returns errSnapshotNotFound when the local index
|
||||||
// snapshotID, and returns errSnapshotNotFound when there is nothing to
|
// has no row for snapshotID and, unless localOnly is set, the
|
||||||
// remove: no row and, unless LocalOnly is set, no metadata on the
|
// destination store has no metadata for it either. inLocalIndex reports
|
||||||
// destination store either. The destination store is consulted only
|
// whether the local index has that row. The destination store is
|
||||||
// when the local index has no row, so a snapshot in the local index can
|
// consulted only when the local index has no row, so a snapshot in the
|
||||||
// still be removed from it while the destination store is unreachable,
|
// local index can still be removed from it while the destination store
|
||||||
// and only after checkDestinationMarker passes, so an unplugged volume's
|
// is unreachable, and only after checkDestinationMarker passes, so an
|
||||||
// empty mount point is not taken for a store without the snapshot.
|
// unplugged volume's empty mount point is not taken for a store without
|
||||||
func (v *Vaultik) findSnapshotToRemove(
|
// the snapshot.
|
||||||
snapshotID string, opts *RemoveOptions,
|
//
|
||||||
) (bool, error) {
|
//nolint:nonamedreturns // the result name says what the bool means
|
||||||
|
func (v *Vaultik) checkSnapshotExists(
|
||||||
|
snapshotID string, localOnly bool,
|
||||||
|
) (inLocalIndex bool, err error) {
|
||||||
if v.Repositories != nil {
|
if v.Repositories != nil {
|
||||||
snap, err := v.Repositories.Snapshots.GetByID(v.ctx, snapshotID)
|
snap, err := v.Repositories.Snapshots.GetByID(v.ctx, snapshotID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1565,12 +1555,12 @@ func (v *Vaultik) findSnapshotToRemove(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if opts.LocalOnly {
|
if localOnly {
|
||||||
return false, fmt.Errorf("%w in the local index: %s",
|
return false, fmt.Errorf("%w in the local index: %s",
|
||||||
errSnapshotNotFound, snapshotID)
|
errSnapshotNotFound, snapshotID)
|
||||||
}
|
}
|
||||||
|
|
||||||
err := v.checkDestinationMarker()
|
err = v.checkDestinationMarker()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false, err
|
return false, err
|
||||||
}
|
}
|
||||||
@@ -1712,7 +1702,7 @@ func (v *Vaultik) removeSnapshotRemote(
|
|||||||
// before it removes anything when the destination store cannot be
|
// before it removes anything when the destination store cannot be
|
||||||
// listed or is not present by checkDestinationMarker.
|
// listed or is not present by checkDestinationMarker.
|
||||||
func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error) {
|
func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error) {
|
||||||
err := v.EnsureStorageBinding()
|
err := v.checkStorageBinding()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -28,8 +28,9 @@ var (
|
|||||||
// Two forms are accepted, matching the two things a host can know:
|
// Two forms are accepted, matching the two things a host can know:
|
||||||
//
|
//
|
||||||
// - A human snapshot ID (hostname_name_timestamp), which a host holding
|
// - A human snapshot ID (hostname_name_timestamp), which a host holding
|
||||||
// the local index has. It is hashed to its remote key; the store is
|
// the local index has. It is hashed to its remote key. An ID that
|
||||||
// not consulted.
|
// names no snapshot in the local index or on the destination store
|
||||||
|
// is an error (see checkSnapshotExists).
|
||||||
// - A remote key, or the leading part of one, which is all a host with
|
// - A remote key, or the leading part of one, which is all a host with
|
||||||
// no local index can know — it is exactly what `snapshot list` prints
|
// no local index can know — it is exactly what `snapshot list` prints
|
||||||
// for a remote-only snapshot (see formatRemoteOnlyID). It is resolved
|
// for a remote-only snapshot (see formatRemoteOnlyID). It is resolved
|
||||||
@@ -41,6 +42,11 @@ var (
|
|||||||
// RFC3339 timestamp).
|
// RFC3339 timestamp).
|
||||||
func (v *Vaultik) resolveSnapshotRemoteKey(identifier string) (string, error) {
|
func (v *Vaultik) resolveSnapshotRemoteKey(identifier string) (string, error) {
|
||||||
if !isRemoteKeyOrPrefix(identifier) {
|
if !isRemoteKeyOrPrefix(identifier) {
|
||||||
|
_, err := v.checkSnapshotExists(identifier, false)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
|
||||||
return snapshot.RemoteSnapshotKey(identifier), nil
|
return snapshot.RemoteSnapshotKey(identifier), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -187,6 +187,17 @@ func (e *listEnv) addRemoteRawTimestamp(
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
remoteKey := snapshot.RemoteSnapshotKey(snapshotID)
|
remoteKey := snapshot.RemoteSnapshotKey(snapshotID)
|
||||||
|
e.putRemoteManifest(t, remoteKey,
|
||||||
|
encodeRemoteManifest(t, remoteKey, timestamp))
|
||||||
|
|
||||||
|
return remoteKey
|
||||||
|
}
|
||||||
|
|
||||||
|
// encodeRemoteManifest returns the encoded manifest of a one-blob
|
||||||
|
// snapshot whose remote key is remoteKey.
|
||||||
|
func encodeRemoteManifest(t *testing.T, remoteKey, timestamp string) []byte {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
manifest := &snapshot.Manifest{
|
manifest := &snapshot.Manifest{
|
||||||
// Note: the hashed key, never the human ID. That is precisely
|
// Note: the hashed key, never the human ID. That is precisely
|
||||||
// why a remote-only snapshot cannot be named.
|
// why a remote-only snapshot cannot be named.
|
||||||
@@ -202,11 +213,17 @@ func (e *listEnv) addRemoteRawTimestamp(
|
|||||||
data, err := snapshot.EncodeManifest(manifest, 3)
|
data, err := snapshot.EncodeManifest(manifest, 3)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
err = e.store.Put(context.Background(),
|
return data
|
||||||
|
}
|
||||||
|
|
||||||
|
// putRemoteManifest stores an encoded manifest on the destination store
|
||||||
|
// under remoteKey.
|
||||||
|
func (e *listEnv) putRemoteManifest(t *testing.T, remoteKey string, data []byte) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
err := e.store.Put(context.Background(),
|
||||||
"metadata/"+remoteKey+"/manifest.json.zst", bytes.NewReader(data))
|
"metadata/"+remoteKey+"/manifest.json.zst", bytes.NewReader(data))
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
return remoteKey
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Fixtures shared across the listing tests.
|
// Fixtures shared across the listing tests.
|
||||||
@@ -345,8 +362,9 @@ func TestSnapshotListAndPurgePrintSizesLikeRemoteInfo(t *testing.T) {
|
|||||||
// Without --force and with nothing on stdin the purge lists what it
|
// Without --force and with nothing on stdin the purge lists what it
|
||||||
// would delete, reads no confirmation, and deletes nothing.
|
// would delete, reads no confirmation, and deletes nothing.
|
||||||
env.stdout.Reset()
|
env.stdout.Reset()
|
||||||
require.NoError(t, env.v.PurgeSnapshotsWithOptions(
|
require.ErrorIs(t, env.v.PurgeSnapshotsWithOptions(
|
||||||
&vaultik.SnapshotPurgeOptions{OlderThan: "1d"}))
|
&vaultik.SnapshotPurgeOptions{OlderThan: "1d"}),
|
||||||
|
vaultik.ErrNotConfirmed)
|
||||||
assert.Contains(t, findTableRow(t, env.stdout.String(), listLocalID),
|
assert.Contains(t, findTableRow(t, env.stdout.String(), listLocalID),
|
||||||
wantSize, "snapshot purge")
|
wantSize, "snapshot purge")
|
||||||
}
|
}
|
||||||
@@ -791,9 +809,17 @@ func TestListSnapshots_JSONReportsTruncation(t *testing.T) {
|
|||||||
|
|
||||||
timestamp := time.Date(2026, 3, 2, 11, 22, 33, 0, time.UTC)
|
timestamp := time.Date(2026, 3, 2, 11, 22, 33, 0, time.UTC)
|
||||||
|
|
||||||
// One past the cap, so exactly one snapshot is omitted.
|
// One past the cap, so exactly one snapshot is omitted. The listing
|
||||||
|
// names a snapshot by the key its manifest is stored under, never by
|
||||||
|
// the manifest's own ID. So all of them share one encoded manifest:
|
||||||
|
// encoding 1001 took seconds.
|
||||||
|
data := encodeRemoteManifest(t,
|
||||||
|
snapshot.RemoteSnapshotKey("otherhost_bulk"),
|
||||||
|
timestamp.Format(time.RFC3339))
|
||||||
|
|
||||||
for i := range maxRemoteOnlyRowsForTest + 1 {
|
for i := range maxRemoteOnlyRowsForTest + 1 {
|
||||||
env.addRemote(t, fmt.Sprintf("otherhost_bulk_%04d", i), timestamp)
|
env.putRemoteManifest(t, snapshot.RemoteSnapshotKey(
|
||||||
|
fmt.Sprintf("otherhost_bulk_%04d", i)), data)
|
||||||
}
|
}
|
||||||
|
|
||||||
stderr := captureProcessStderr(t, func() {
|
stderr := captureProcessStderr(t, func() {
|
||||||
|
|||||||
@@ -13,35 +13,30 @@ import (
|
|||||||
var errStorageBindingMismatch = errors.New(
|
var errStorageBindingMismatch = errors.New(
|
||||||
"local index is bound to a different backup destination")
|
"local index is bound to a different backup destination")
|
||||||
|
|
||||||
// EnsureStorageBinding guarantees that the local index database is
|
// checkStorageBinding refuses when the local index database is bound to
|
||||||
// bound to the currently-configured storage destination. Every mutating
|
// a different destination than the configured one, with an error that
|
||||||
// command must call this before touching either the local index or the
|
// tells the user how to recover (revert the config, or run `vaultik
|
||||||
// destination store, because the two live in lockstep: the local index
|
// database delete`). Every mutating command calls it before touching
|
||||||
// records which chunks/blobs already exist at the destination, and
|
// either the local index or the destination store, because the two live
|
||||||
// mismatched destination + local index produces silent corruption (the
|
// in lockstep: the local index records which chunks/blobs already exist
|
||||||
// scanner sees "known" chunks and skips uploads, then writes snapshots
|
// at the destination, and a mismatched pair produces silent corruption
|
||||||
// whose manifests reference blobs that aren't on the new destination).
|
// (the scanner sees "known" chunks and skips uploads, then writes
|
||||||
|
// snapshots whose manifests reference blobs that aren't on the new
|
||||||
|
// destination).
|
||||||
//
|
//
|
||||||
// Behaviour:
|
// It records nothing. Only a backup binds the local index, through
|
||||||
// - On first use (empty stored value), record the configured
|
// EnsureStorageBinding, and it does so before adding anything to it. An
|
||||||
// destination (storage_url, or the s3.* endpoint, bucket and
|
// unbound local index therefore holds nothing from any destination.
|
||||||
// prefix) and log the binding.
|
|
||||||
// - When the stored value matches the configured destination, do
|
|
||||||
// nothing and return nil.
|
|
||||||
// - When the two differ, refuse with an error that tells the user
|
|
||||||
// how to recover (revert the config, or run `vaultik database
|
|
||||||
// delete` to discard the local index and rebuild against the new
|
|
||||||
// destination on the next backup).
|
|
||||||
//
|
//
|
||||||
// Read-only inspection commands (remote info, snapshot list, etc.)
|
// Read-only inspection commands (remote info, snapshot list, etc.)
|
||||||
// deliberately don't call this: they can be run against a bare
|
// deliberately don't call this: they can be run against a bare
|
||||||
// destination store without any binding state.
|
// destination store without any binding state.
|
||||||
func (v *Vaultik) EnsureStorageBinding() error {
|
func (v *Vaultik) checkStorageBinding() error {
|
||||||
if v.Repositories == nil || v.Config == nil {
|
if v.Repositories == nil || v.Config == nil {
|
||||||
// NewForTesting builds a Vaultik with no DB or config;
|
// NewForTesting builds a Vaultik with no DB or config;
|
||||||
// there's nothing to bind and no binding to check. Callers
|
// there's no binding to check. Callers exercising the bind
|
||||||
// exercising the bind path use a real DB and populate Config
|
// path use a real DB and populate Config explicitly (see
|
||||||
// explicitly (see storage_bind_test.go).
|
// storage_bind_test.go).
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -50,6 +45,24 @@ func (v *Vaultik) EnsureStorageBinding() error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if stored == "" || stored == configured {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return fmt.Errorf("%w\n%s",
|
||||||
|
errStorageBindingMismatch, buildBindingMismatchMessage(stored, configured))
|
||||||
|
}
|
||||||
|
|
||||||
|
// EnsureStorageBinding records the configured destination (storage_url,
|
||||||
|
// or the s3.* endpoint, bucket and prefix) as the binding of an unbound
|
||||||
|
// local index. It refuses as checkStorageBinding does when the local
|
||||||
|
// index is bound to a different destination.
|
||||||
|
func (v *Vaultik) EnsureStorageBinding() error {
|
||||||
|
stored, configured, err := v.storageBinding()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
if stored == "" {
|
if stored == "" {
|
||||||
err = v.Repositories.LocalMeta.Set(
|
err = v.Repositories.LocalMeta.Set(
|
||||||
v.ctx, database.LocalMetaKeyStorageURL, configured)
|
v.ctx, database.LocalMetaKeyStorageURL, configured)
|
||||||
@@ -62,14 +75,14 @@ func (v *Vaultik) EnsureStorageBinding() error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
if stored == configured {
|
if stored != configured {
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return fmt.Errorf("%w\n%s",
|
return fmt.Errorf("%w\n%s",
|
||||||
errStorageBindingMismatch, buildBindingMismatchMessage(stored, configured))
|
errStorageBindingMismatch, buildBindingMismatchMessage(stored, configured))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// storageBinding returns the destination the local index is bound to
|
// storageBinding returns the destination the local index is bound to
|
||||||
// ("" when no binding is recorded yet), then the configured destination
|
// ("" when no binding is recorded yet), then the configured destination
|
||||||
// written the same way. It records nothing, so a read-only command can
|
// written the same way. It records nothing, so a read-only command can
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package vaultik_test
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"path/filepath"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
@@ -9,9 +10,126 @@ import (
|
|||||||
"sneak.berlin/go/vaultik/internal/config"
|
"sneak.berlin/go/vaultik/internal/config"
|
||||||
"sneak.berlin/go/vaultik/internal/database"
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
|
"sneak.berlin/go/vaultik/internal/storage"
|
||||||
|
"sneak.berlin/go/vaultik/internal/storage/faultstore"
|
||||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// storageBindingOf returns the destination the local index is bound to,
|
||||||
|
// or "" when it is not bound.
|
||||||
|
func storageBindingOf(
|
||||||
|
ctx context.Context, t *testing.T, repos *database.Repositories,
|
||||||
|
) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
got, err := repos.LocalMeta.Get(ctx, database.LocalMetaKeyStorageURL)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
return got
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFailedFirstBackupLeavesIndexUnbound checks that a first backup
|
||||||
|
// the destination refuses, as a mistyped bucket would, leaves the local
|
||||||
|
// index unbound, so a backup with the corrected destination succeeds.
|
||||||
|
func TestFailedFirstBackupLeavesIndexUnbound(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
storeDir := t.TempDir()
|
||||||
|
v, repos := newFileDestinationVaultik(ctx, t, storeDir)
|
||||||
|
working := v.Storage
|
||||||
|
|
||||||
|
refusing := faultstore.New(working)
|
||||||
|
refusing.OnPut = func(string) faultstore.PutAction { return faultstore.PutFail }
|
||||||
|
refusing.OnStat = func(string) bool { return true }
|
||||||
|
v.Storage = refusing
|
||||||
|
v.Config.StorageURL = "file://" + filepath.Join(t.TempDir(), "typo")
|
||||||
|
|
||||||
|
require.ErrorIs(t, backUp(v, "first"), faultstore.ErrInjectedUpload)
|
||||||
|
assert.Empty(t, storageBindingOf(ctx, t, repos))
|
||||||
|
|
||||||
|
v.Storage = working
|
||||||
|
v.Config.StorageURL = "file://" + storeDir
|
||||||
|
|
||||||
|
require.NoError(t, backUp(v, "first"))
|
||||||
|
assert.Equal(t, "file://"+storeDir, storageBindingOf(ctx, t, repos))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBackupToOtherDestinationWritesNoMarker checks that a backup
|
||||||
|
// refuses a local index bound to a different destination before it
|
||||||
|
// writes the marker to the configured one.
|
||||||
|
func TestBackupToOtherDestinationWritesNoMarker(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
firstDir := t.TempDir()
|
||||||
|
v, _ := newFileDestinationVaultik(ctx, t, firstDir)
|
||||||
|
v.Config.StorageURL = "file://" + firstDir
|
||||||
|
|
||||||
|
require.NoError(t, backUp(v, "first"))
|
||||||
|
|
||||||
|
secondDir := t.TempDir()
|
||||||
|
secondStore, err := storage.NewFileStorer(secondDir)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
v.Storage = secondStore
|
||||||
|
v.Config.StorageURL = "file://" + secondDir
|
||||||
|
|
||||||
|
require.ErrorContains(t, backUp(v, "first"),
|
||||||
|
"local index is bound to a different backup destination")
|
||||||
|
assert.NoFileExists(t, filepath.Join(secondDir, markerFile))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCommandsOtherThanBackupLeaveIndexUnbound checks that commands
|
||||||
|
// other than a backup do not bind an unbound local index, whether or not
|
||||||
|
// they fail. Each runs on a fresh local index.
|
||||||
|
func TestCommandsOtherThanBackupLeaveIndexUnbound(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
run func(v *vaultik.Vaultik) error
|
||||||
|
}{
|
||||||
|
{"prune", func(v *vaultik.Vaultik) error {
|
||||||
|
return v.Prune(&vaultik.PruneOptions{Force: true})
|
||||||
|
}},
|
||||||
|
{"snapshot purge", func(v *vaultik.Vaultik) error {
|
||||||
|
return v.PurgeSnapshotsWithOptions(
|
||||||
|
&vaultik.SnapshotPurgeOptions{KeepLatest: true, Force: true})
|
||||||
|
}},
|
||||||
|
{"snapshot remove", func(v *vaultik.Vaultik) error {
|
||||||
|
_, err := v.RemoveSnapshot("host_first_2026-01-02T03:04:05Z",
|
||||||
|
&vaultik.RemoveOptions{Force: true})
|
||||||
|
|
||||||
|
return err
|
||||||
|
}},
|
||||||
|
{"snapshot remove --all", func(v *vaultik.Vaultik) error {
|
||||||
|
_, err := v.RemoveAllSnapshots(&vaultik.RemoveOptions{Force: true})
|
||||||
|
|
||||||
|
return err
|
||||||
|
}},
|
||||||
|
{"remote nuke", func(v *vaultik.Vaultik) error {
|
||||||
|
return v.NukeRemote(true)
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
v, repos := newFileDestinationVaultik(ctx, t, t.TempDir())
|
||||||
|
|
||||||
|
_ = tt.run(v)
|
||||||
|
|
||||||
|
assert.Empty(t, storageBindingOf(ctx, t, repos))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// buildBindTestVaultik returns a minimal Vaultik wired with a real
|
// buildBindTestVaultik returns a minimal Vaultik wired with a real
|
||||||
// in-memory DB and a config carrying the given StorageURL. Enough to
|
// in-memory DB and a config carrying the given StorageURL. Enough to
|
||||||
// exercise EnsureStorageBinding without spinning up storage or fx.
|
// exercise EnsureStorageBinding without spinning up storage or fx.
|
||||||
|
|||||||
@@ -0,0 +1,62 @@
|
|||||||
|
package vaultik_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/vaultik/internal/config"
|
||||||
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestUnknownSnapshotID checks that restore and verify, given an ID that
|
||||||
|
// names no snapshot in the local index or on the destination store, say
|
||||||
|
// so in the words `snapshot remove` uses and print nothing else.
|
||||||
|
func TestUnknownSnapshotID(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const unknownID = "host1_home_2026-01-01T00:00:00Z"
|
||||||
|
|
||||||
|
commands := map[string]func(v *vaultik.Vaultik) error{
|
||||||
|
"restore": func(v *vaultik.Vaultik) error {
|
||||||
|
return v.Restore(&vaultik.RestoreOptions{
|
||||||
|
SnapshotID: unknownID,
|
||||||
|
TargetDir: t.TempDir(),
|
||||||
|
})
|
||||||
|
},
|
||||||
|
"verify": func(v *vaultik.Vaultik) error {
|
||||||
|
return v.VerifySnapshotWithOptions(unknownID, &vaultik.VerifyOptions{})
|
||||||
|
},
|
||||||
|
"verify --deep": func(v *vaultik.Vaultik) error {
|
||||||
|
return v.VerifySnapshotWithOptions(unknownID,
|
||||||
|
&vaultik.VerifyOptions{Deep: true})
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, run := range commands {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
store := newTestStorer()
|
||||||
|
addManifest(t, store, "snapshot-001", nil)
|
||||||
|
|
||||||
|
// A backup wrote to this destination, so it carries the marker.
|
||||||
|
require.NoError(t, store.Put(context.Background(), markerFile,
|
||||||
|
bytes.NewReader(nil)))
|
||||||
|
|
||||||
|
tv := vaultik.NewForTesting(store)
|
||||||
|
tv.Config = &config.Config{AgeSecretKey: testAgeSecretKey}
|
||||||
|
tv.Repositories = newLocalIndex(t, "snapshot-001")
|
||||||
|
|
||||||
|
err := run(tv.Vaultik)
|
||||||
|
|
||||||
|
require.EqualError(t, err, "snapshot not found in the local "+
|
||||||
|
"index or on the destination store: "+unknownID)
|
||||||
|
assert.Empty(t, tv.Stdout.String())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -5,6 +5,7 @@ package vaultik
|
|||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
@@ -184,6 +185,12 @@ func (v *Vaultik) printlnStdout(args ...any) {
|
|||||||
_, _ = fmt.Fprintln(v.Stdout, args...)
|
_, _ = fmt.Fprintln(v.Stdout, args...)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ErrNotConfirmed is returned when a confirmation prompt gets any answer
|
||||||
|
// but the one it asks for (`y` or `Y`, or `yes` for `database delete`), or
|
||||||
|
// stdin ends first. Nothing the prompt asked about is deleted, and the
|
||||||
|
// command exits non-zero.
|
||||||
|
var ErrNotConfirmed = errors.New("deletion not confirmed")
|
||||||
|
|
||||||
// scanStdin reads a line of input from stdin.
|
// scanStdin reads a line of input from stdin.
|
||||||
func (v *Vaultik) scanStdin(a ...any) (int, error) {
|
func (v *Vaultik) scanStdin(a ...any) (int, error) {
|
||||||
return fmt.Fscanln(v.Stdin, a...)
|
return fmt.Fscanln(v.Stdin, a...)
|
||||||
|
|||||||
+21
-16
@@ -120,12 +120,17 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
|
|||||||
|
|
||||||
log.Info("Starting snapshot verification", "snapshot_id", snapshotID, "mode", "deep")
|
log.Info("Starting snapshot verification", "snapshot_id", snapshotID, "mode", "deep")
|
||||||
|
|
||||||
|
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
|
||||||
|
if err != nil {
|
||||||
|
return v.deepVerifyFailure(result, opts, err.Error(), err)
|
||||||
|
}
|
||||||
|
|
||||||
if !opts.JSON && !v.UI.Quiet() {
|
if !opts.JSON && !v.UI.Quiet() {
|
||||||
v.stdoutf("Deep verification of snapshot: %s\n\n", snapshotID)
|
v.stdoutf("Deep verification of snapshot: %s\n\n", snapshotID)
|
||||||
}
|
}
|
||||||
|
|
||||||
manifest, tempDB, dbBlobs, err := v.loadVerificationData(
|
manifest, tempDB, dbBlobs, err := v.loadVerificationData(
|
||||||
snapshotID, opts, result, identities)
|
snapshotID, remoteKey, opts, result, identities)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -170,21 +175,13 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// loadVerificationData downloads manifest, database, and blob list for verification
|
// loadVerificationData downloads manifest, database, and blob list for
|
||||||
|
// verification of the snapshot at remoteKey, which
|
||||||
|
// resolveSnapshotRemoteKey returned for snapshotID.
|
||||||
func (v *Vaultik) loadVerificationData(
|
func (v *Vaultik) loadVerificationData(
|
||||||
snapshotID string, opts *VerifyOptions, result *VerifyResult,
|
snapshotID, remoteKey string, opts *VerifyOptions, result *VerifyResult,
|
||||||
identities []age.Identity,
|
identities []age.Identity,
|
||||||
) (*snapshot.Manifest, *tempDB, []snapshot.BlobInfo, error) {
|
) (*snapshot.Manifest, *tempDB, []snapshot.BlobInfo, error) {
|
||||||
// Resolve the identifier to the snapshot's remote key. A human ID is
|
|
||||||
// hashed; a remote key (or its abbreviation, as printed for a
|
|
||||||
// remote-only snapshot) is used as-is, so a host with no local index
|
|
||||||
// can verify a snapshot it can only see on the store.
|
|
||||||
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
|
|
||||||
if err != nil {
|
|
||||||
return nil, nil, nil, v.deepVerifyFailure(result, opts,
|
|
||||||
fmt.Sprintf("resolving snapshot identifier: %v", err), err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Download manifest. downloadManifestByKey is the single reader for
|
// Download manifest. downloadManifestByKey is the single reader for
|
||||||
// remote manifests; see its doc comment.
|
// remote manifests; see its doc comment.
|
||||||
log.Info("Downloading manifest", "remote_key", remoteKey)
|
log.Info("Downloading manifest", "remote_key", remoteKey)
|
||||||
@@ -267,6 +264,10 @@ func (v *Vaultik) downloadVerifiedSnapshotDB(
|
|||||||
defer func() { _ = dbReader.Close() }()
|
defer func() { _ = dbReader.Close() }()
|
||||||
|
|
||||||
tdb, err := v.decryptAndLoadDatabase(dbReader, identities)
|
tdb, err := v.decryptAndLoadDatabase(dbReader, identities)
|
||||||
|
if errors.Is(err, errWrongPrivateKey) {
|
||||||
|
return nil, v.deepVerifyFailure(result, opts, err.Error(), err)
|
||||||
|
}
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, v.deepVerifyFailure(result, opts,
|
return nil, v.deepVerifyFailure(result, opts,
|
||||||
fmt.Sprintf("failed to decrypt database: %v", err),
|
fmt.Sprintf("failed to decrypt database: %v", err),
|
||||||
@@ -351,9 +352,9 @@ func (v *Vaultik) decryptAndLoadDatabase(
|
|||||||
reader io.ReadCloser, identities []age.Identity,
|
reader io.ReadCloser, identities []age.Identity,
|
||||||
) (*tempDB, error) {
|
) (*tempDB, error) {
|
||||||
// Decrypt and decompress through the shared blobgen reader.
|
// Decrypt and decompress through the shared blobgen reader.
|
||||||
blobReader, err := blobgen.NewReader(reader, identities...)
|
blobReader, err := v.newSnapshotDBReader(reader, identities)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to create decryption reader: %w", err)
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
defer func() { _ = blobReader.Close() }()
|
defer func() { _ = blobReader.Close() }()
|
||||||
@@ -426,7 +427,11 @@ func (v *Vaultik) verifyBlob(
|
|||||||
// the plaintext as it is read. A blob's hash — its remote name — is the
|
// the plaintext as it is read. A blob's hash — its remote name — is the
|
||||||
// double SHA-256 of that plaintext (see blobgen.DoubleSHA256), not of the
|
// double SHA-256 of that plaintext (see blobgen.DoubleSHA256), not of the
|
||||||
// encrypted bytes.
|
// encrypted bytes.
|
||||||
blobReader, err := blobgen.NewReader(reader, identities...)
|
blobReader, err := v.newBlobReader(reader, identities)
|
||||||
|
if errors.Is(err, errWrongPrivateKeyForBlob) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to create blob reader: %w", err)
|
return fmt.Errorf("failed to create blob reader: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
+2
-2
@@ -11,8 +11,8 @@
|
|||||||
# when no tag is reachable, each with a "-dirty" suffix when tracked
|
# when no tag is reachable, each with a "-dirty" suffix when tracked
|
||||||
# files have uncommitted changes. Untracked files are ignored: a stray
|
# files have uncommitted changes. Untracked files are ignored: a stray
|
||||||
# scratch file does not change what was compiled. Outside a git checkout
|
# scratch file does not change what was compiled. Outside a git checkout
|
||||||
# (release tarball, `go install`), or in one with no commits, it is
|
# (such as an unpacked release tarball), or in one with no commits, it
|
||||||
# "dev".
|
# is "dev".
|
||||||
#
|
#
|
||||||
# Nothing here ever invents a version number.
|
# Nothing here ever invents a version number.
|
||||||
set -eu
|
set -eu
|
||||||
|
|||||||
Reference in new issue
Block a user