Compare commits
1
Commits
next
..
fcc9f34b83
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fcc9f34b83 |
@@ -390,13 +390,7 @@ recipients, and local database statistics.
|
||||
|
||||
**`remote info`**: Show storage backend type and location plus detailed
|
||||
remote storage inventory: per-snapshot metadata sizes, blob counts, and
|
||||
orphaned blob detection. A name under `metadata/` that is not a remote
|
||||
key is skipped with a warning and is not printed. If a listed
|
||||
`manifest.json.zst` cannot be read, or sits under a skipped name, the
|
||||
orphaned blob figures are reported as unknown; `--json` gives them as
|
||||
`null`, lists the remote key of each unreadable manifest in
|
||||
`unreadable_manifests` and counts the manifests under skipped names in
|
||||
`skipped_manifest_count`.
|
||||
orphaned blob detection.
|
||||
* `--json`: Output as JSON
|
||||
|
||||
**`remote nuke`**: Delete every snapshot's metadata and every blob from the
|
||||
|
||||
@@ -22,72 +22,6 @@ the tag exists and is exercised; what is left is merging `next` to
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-07: Cut the time the `internal/vaultik` and `internal/database`
|
||||
tests take ([issue #235](https://git.eeqj.de/sneak/vaultik/issues/235)).
|
||||
Most of the `internal/vaultik` time went to 24 tests that ran one at a
|
||||
time only because they call `log.Initialize`; they now call it before
|
||||
`t.Parallel()`, as the package's other tests do. `TestLargeDatasets`
|
||||
committed each of its 1,500 inserts on its own and now makes them in
|
||||
one transaction. `TestDedupOnlySnapshotRestores` gives its second
|
||||
backup its own snapshot name instead of sleeping past the one-second
|
||||
timestamp in the snapshot ID.
|
||||
|
||||
- 2026-10-07: Made two messages say only what is true
|
||||
([issue #240](https://git.eeqj.de/sneak/vaultik/issues/240)). A config
|
||||
file that others can read was warned about as containing S3
|
||||
credentials even when it set none, as a `file://` config does. The
|
||||
warning now says the file may contain S3 credentials only when
|
||||
`s3.access_key_id` or `s3.secret_access_key` is set, since either may
|
||||
come from a `${...}` reference rather than the file, and otherwise
|
||||
says the file is readable by others. `snapshot purge` against a
|
||||
destination store it could not list gave an error with
|
||||
`listing remote snapshots:` in it twice; the prefix now appears once.
|
||||
|
||||
- 2026-10-07: Made `s3.part_size` set the multipart upload part size
|
||||
([issue #232](https://git.eeqj.de/sneak/vaultik/issues/232)). It was
|
||||
loaded and defaulted but never passed to the S3 client, whose uploader
|
||||
used a fixed 10MiB part. It now reaches the uploader for `storage_url`
|
||||
and for the `s3.*` fields, and a part size S3 refuses, below 5MiB or
|
||||
above 5GiB, `0` included, fails at config load. A blob too large for
|
||||
S3's limit of 10,000 parts at the configured size is uploaded in larger
|
||||
parts. The docs gave the default as `5MB`, which the config file reads
|
||||
as 5,000,000 bytes, below the minimum; they now say `5MiB`.
|
||||
|
||||
- 2026-10-07: Made per-name retention work when the hostname contains `_`
|
||||
([issue #230](https://git.eeqj.de/sneak/vaultik/issues/230)). A
|
||||
snapshot ID is `hostname_name_timestamp`, and the name was read as
|
||||
everything between the first and the last `_`, so with
|
||||
`hostname: my_host` the name `home` came out as `host_home`.
|
||||
`snapshot purge --keep-latest --snapshot home` then printed "No
|
||||
snapshots to delete", and `snapshot create --prune` purged nothing
|
||||
without a message. The name is now read using the hostname the
|
||||
`snapshots` table stores with each snapshot, cut at its first `.` as it
|
||||
is in the ID.
|
||||
|
||||
- 2026-10-07: Made `remote info` stop reporting a snapshot's blobs as
|
||||
orphaned when its manifest cannot be read, and stop printing raw
|
||||
names from under `metadata/`
|
||||
([issue #228](https://git.eeqj.de/sneak/vaultik/issues/228)). A
|
||||
manifest it failed to read was skipped, so that snapshot's blobs were
|
||||
counted as orphaned and the report advised running `vaultik prune`.
|
||||
The orphan figures are now unknown in that case, with no prune
|
||||
advice, and `--json` gives them as `null` with the unreadable remote
|
||||
keys in `unreadable_manifests`. A name under `metadata/` that is not
|
||||
64 lowercase hex characters is now skipped with a warning instead of
|
||||
being printed, control characters included. A manifest under a
|
||||
skipped name is then not read either, so it also leaves the orphan
|
||||
figures unknown, and `--json` counts such manifests in
|
||||
`skipped_manifest_count`. A directory with no manifest in it, as left
|
||||
by an interrupted backup, leaves the figures known.
|
||||
|
||||
- 2026-10-07: Made `config set` keep a string that looks like a number
|
||||
([issue #229](https://git.eeqj.de/sneak/vaultik/issues/229)). It wrote
|
||||
every value unquoted, and `config.Load` reads the file through untyped
|
||||
YAML, so an access key `00112233` loaded as `38043` and a hostname `007`
|
||||
as `7`. A value for a string setting in `config.Config` is now tagged as
|
||||
a YAML string, which the file quotes wherever YAML would read a number or
|
||||
a boolean; other settings are still written unquoted.
|
||||
|
||||
- 2026-10-07: Made a backup notice a file rewritten with its size
|
||||
unchanged and a new mtime in the same second as the one in the index
|
||||
([issue #226](https://git.eeqj.de/sneak/vaultik/issues/226)). The
|
||||
|
||||
+4
-5
@@ -287,11 +287,10 @@ storage_url: "rclone://myremote/path/to/backups"
|
||||
# #use_ssl: true
|
||||
#
|
||||
# # Part size for multipart uploads
|
||||
# # Minimum 5MiB, maximum 5GiB; affects memory usage during upload
|
||||
# # A blob too large for 10,000 parts of this size gets larger parts
|
||||
# # Supports: 10MB, 16MiB, 100MiB, etc. (5MB is below the minimum)
|
||||
# # Default: 5MiB
|
||||
# #part_size: 5MiB
|
||||
# # Minimum 5MB, affects memory usage during upload
|
||||
# # Supports: 5MB, 10M, 100MiB, etc.
|
||||
# # Default: 5MB
|
||||
# #part_size: 5MB
|
||||
|
||||
# Path to local SQLite index database
|
||||
# This database tracks file state for incremental backups
|
||||
|
||||
+1
-56
@@ -7,14 +7,11 @@ import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
"gopkg.in/yaml.v3"
|
||||
"sneak.berlin/go/vaultik/internal/config"
|
||||
"sneak.berlin/go/vaultik/internal/ui"
|
||||
)
|
||||
|
||||
@@ -34,9 +31,6 @@ const configDirMode = 0o755
|
||||
// yaml.Marshal's 4-space default.
|
||||
const configYAMLIndent = 2
|
||||
|
||||
// yamlStringTag is YAML's tag for a string scalar.
|
||||
const yamlStringTag = "!!str"
|
||||
|
||||
var (
|
||||
errConfigExists = errors.New("config file already exists")
|
||||
errEmptyConfig = errors.New("empty config file")
|
||||
@@ -205,7 +199,7 @@ storage_url: ""
|
||||
# access_key_id: YOUR_ACCESS_KEY
|
||||
# secret_access_key: YOUR_SECRET_KEY
|
||||
# # region: us-east-1 # Default: us-east-1
|
||||
# # part_size: 5MiB # Upload part size, 5MiB to 5GiB. Default: 5MiB
|
||||
# # part_size: 5MB # Multipart upload part size. Default: 5MB
|
||||
# # For the s3:// form, disable TLS with ?ssl=false in the URL, not use_ssl.
|
||||
|
||||
# ─── OPTIONAL ────────────────────────────────────────────────────────────────
|
||||
@@ -589,58 +583,9 @@ func yamlPathSet(root *yaml.Node, keys []string, value string) error {
|
||||
}
|
||||
}
|
||||
|
||||
// config.Load reads the file through untyped YAML, which turns an
|
||||
// unquoted 00112233 into the number 38043 and 1e5 into 100000. Tagging
|
||||
// a string setting as a string makes the encoder quote such a value.
|
||||
// Other settings stay unquoted, so compression_level 9 is a number.
|
||||
// The encoder refuses to write a value that is not valid UTF-8 as a
|
||||
// string. Left untagged, such a value is written as base64 !!binary and
|
||||
// loads back unchanged.
|
||||
if configKeyIsString(keys) && utf8.ValidString(value) {
|
||||
node.Tag = yamlStringTag
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// configKeyIsString reports whether the dotted key names a string in
|
||||
// config.Config, following the fields' yaml tags, as s3.access_key_id and
|
||||
// snapshots.home.exclude.0 do.
|
||||
func configKeyIsString(keys []string) bool {
|
||||
typ := reflect.TypeFor[config.Config]()
|
||||
|
||||
for _, key := range keys {
|
||||
switch {
|
||||
case typ.Kind() == reflect.Map || typ.Kind() == reflect.Slice:
|
||||
// The key is a snapshot name or a list index.
|
||||
typ = typ.Elem()
|
||||
case typ.Kind() == reflect.Struct:
|
||||
field, ok := yamlField(typ, key)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
|
||||
typ = field.Type
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
return typ.Kind() == reflect.String
|
||||
}
|
||||
|
||||
// yamlField returns the field of struct type typ whose yaml tag names key.
|
||||
func yamlField(typ reflect.Type, key string) (reflect.StructField, bool) {
|
||||
for field := range typ.Fields() {
|
||||
name, _, _ := strings.Cut(field.Tag.Get("yaml"), ",")
|
||||
if name == key {
|
||||
return field, true
|
||||
}
|
||||
}
|
||||
|
||||
return reflect.StructField{}, false
|
||||
}
|
||||
|
||||
// yamlSetInMapping resolves (creating if needed) the value node for key
|
||||
// within a mapping node, setting it to value when it is the final path
|
||||
// element, and returns the node to descend into.
|
||||
|
||||
@@ -4,7 +4,6 @@ import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -95,110 +94,6 @@ func TestConfigSetRecipientOnFreshConfig(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestConfigSetStringLooksLikeNumber sets string settings to values that
|
||||
// YAML reads as numbers or booleans when they are unquoted, and checks that
|
||||
// config.Load returns each one unchanged.
|
||||
func TestConfigSetStringLooksLikeNumber(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
key string
|
||||
value string
|
||||
field func(cfg *config.Config) string
|
||||
}{
|
||||
{"s3.access_key_id", "00112233",
|
||||
func(cfg *config.Config) string { return cfg.S3.AccessKeyID }},
|
||||
{"s3.secret_access_key", "12345678901234567890123456789012",
|
||||
func(cfg *config.Config) string { return cfg.S3.SecretAccessKey }},
|
||||
{"hostname", "007",
|
||||
func(cfg *config.Config) string { return cfg.Hostname }},
|
||||
{"s3.prefix", "1e5",
|
||||
func(cfg *config.Config) string { return cfg.S3.Prefix }},
|
||||
{"s3.bucket", "true",
|
||||
func(cfg *config.Config) string { return cfg.S3.Bucket }},
|
||||
{"s3.region", "FALSE",
|
||||
func(cfg *config.Config) string { return cfg.S3.Region }},
|
||||
{"snapshots.home.exclude.0", "1.10",
|
||||
func(cfg *config.Config) string { return cfg.Snapshots["home"].Exclude[0] }},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.key+"="+tt.value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := loadAfterConfigSet(t, tt.key, tt.value)
|
||||
|
||||
got := tt.field(cfg)
|
||||
if got != tt.value {
|
||||
t.Errorf("%s = %q after config set %q", tt.key, got, tt.value)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestConfigSetNonUTF8Path checks that config set still accepts a value that
|
||||
// is not valid UTF-8, such as a path with a Latin-1 file name, and that
|
||||
// config.Load returns it unchanged.
|
||||
func TestConfigSetNonUTF8Path(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const dir = "/srv/caf\xe9"
|
||||
|
||||
cfg := loadAfterConfigSet(t, "snapshots.home.paths.0", dir)
|
||||
|
||||
got := cfg.Snapshots["home"].Paths[0]
|
||||
if got != dir {
|
||||
t.Errorf("snapshots.home.paths.0 = %q, want %q", got, dir)
|
||||
}
|
||||
}
|
||||
|
||||
// TestConfigSetNumberStaysNumber checks that a number set for an integer
|
||||
// setting is still read as a number, not as a quoted string.
|
||||
func TestConfigSetNumberStaysNumber(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const level = 9
|
||||
|
||||
cfg := loadAfterConfigSet(t, "compression_level", strconv.Itoa(level))
|
||||
|
||||
if cfg.CompressionLevel != level {
|
||||
t.Errorf("compression_level = %d, want %d", cfg.CompressionLevel, level)
|
||||
}
|
||||
}
|
||||
|
||||
// loadAfterConfigSet writes the file `config init` writes, sets storage_url
|
||||
// to a local directory so that the file passes validation, applies
|
||||
// `config set key value` and returns what config.Load reads back.
|
||||
func loadAfterConfigSet(t *testing.T, key, value string) *config.Config {
|
||||
t.Helper()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "config.yml")
|
||||
|
||||
err := os.WriteFile(path, []byte(defaultConfigTemplate), configFileMode)
|
||||
if err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
|
||||
out := ui.NewWithColor(&bytes.Buffer{}, false)
|
||||
|
||||
err = writeConfigSet(out, path, "storage_url", "file:///mnt/backups")
|
||||
if err != nil {
|
||||
t.Fatalf("config set storage_url: %v", err)
|
||||
}
|
||||
|
||||
err = writeConfigSet(out, path, key, value)
|
||||
if err != nil {
|
||||
t.Fatalf("config set %s: %v", key, err)
|
||||
}
|
||||
|
||||
cfg, err := config.Load(path)
|
||||
if err != nil {
|
||||
t.Fatalf("config.Load: %v", err)
|
||||
}
|
||||
|
||||
return cfg
|
||||
}
|
||||
|
||||
const testYAML = `# top comment
|
||||
compression_level: 3
|
||||
age_recipients:
|
||||
|
||||
@@ -33,14 +33,11 @@ const secretKeyPrefix = "AGE-SECRET-KEY-"
|
||||
const (
|
||||
defaultBlobSizeLimit = Size(10 * 1024 * 1024 * 1024) // 10GB
|
||||
defaultChunkSize = Size(10 * 1024 * 1024) // 10MB
|
||||
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MiB
|
||||
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MB
|
||||
defaultCompressionLevel = 3
|
||||
minChunkSize = 1024 * 1024 // 1MB
|
||||
minCompressionLevel = 1
|
||||
maxCompressionLevel = 19
|
||||
// S3 accepts a multipart upload part from 5MiB to 5GiB.
|
||||
minS3PartSize = 5 * 1024 * 1024
|
||||
maxS3PartSize = 5 * 1024 * 1024 * 1024
|
||||
)
|
||||
|
||||
// Sentinel validation errors.
|
||||
@@ -58,7 +55,6 @@ var (
|
||||
"blob_size_limit must be at least the largest chunk the chunker can " +
|
||||
"emit (chunk_size times the FastCDC size spread)")
|
||||
errBadCompression = errors.New("compression_level must be between 1 and 19")
|
||||
errBadS3PartSize = errors.New("s3.part_size must be between 5MiB and 5GiB")
|
||||
errBadStorageScheme = errors.New(
|
||||
"storage_url must start with s3://, file://, or rclone://")
|
||||
errStorageNotConfigured = errors.New(
|
||||
@@ -248,7 +244,6 @@ func Load(path string) (*Config, error) {
|
||||
ChunkSize: defaultChunkSize,
|
||||
IndexPath: filepath.Join(xdg.DataHome, appName, "index.sqlite"),
|
||||
CompressionLevel: defaultCompressionLevel,
|
||||
S3: S3Config{PartSize: defaultS3PartSize},
|
||||
}
|
||||
|
||||
// Convert smartconfig data to YAML then unmarshal
|
||||
@@ -299,13 +294,17 @@ func Load(path string) (*Config, error) {
|
||||
cfg.S3.Region = "us-east-1"
|
||||
}
|
||||
|
||||
if cfg.S3.PartSize == 0 {
|
||||
cfg.S3.PartSize = defaultS3PartSize
|
||||
}
|
||||
|
||||
// Check config file permissions (warn if world or group readable)
|
||||
//nolint:gosec // G703: config path is operator-supplied by design
|
||||
info, statErr := os.Stat(path)
|
||||
if statErr == nil {
|
||||
mode := info.Mode().Perm()
|
||||
if mode&0044 != 0 { // group or world readable
|
||||
log.Warn(cfg.readableByOthersWarning(),
|
||||
log.Warn("Config file has insecure permissions (contains S3 credentials)",
|
||||
"path", path,
|
||||
"mode", fmt.Sprintf("%04o", mode),
|
||||
"recommendation", "chmod 600 "+path)
|
||||
@@ -333,7 +332,6 @@ func Load(path string) (*Config, error) {
|
||||
// (chunk_size times chunker.ChunkSizeSpread), so a single-chunk blob never
|
||||
// exceeds the configured limit
|
||||
// - Compression level must be between 1 and 19
|
||||
// - S3 part size must be between 5MiB and 5GiB, the part sizes S3 accepts
|
||||
//
|
||||
// Returns an error describing the first validation failure encountered.
|
||||
func (c *Config) Validate() error {
|
||||
@@ -378,11 +376,6 @@ func (c *Config) Validate() error {
|
||||
return errBadCompression
|
||||
}
|
||||
|
||||
if c.S3.PartSize.Int64() < minS3PartSize ||
|
||||
c.S3.PartSize.Int64() > maxS3PartSize {
|
||||
return errBadS3PartSize
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -418,18 +411,6 @@ func (c *Config) setAgeSecretKey() {
|
||||
}
|
||||
}
|
||||
|
||||
// readableByOthersWarning is the warning Load logs when others can read
|
||||
// the config file. It says "may contain" because the S3 credentials are
|
||||
// seen only after smartconfig has replaced any ${...} reference in the
|
||||
// file with its value, so a set credential need not be in the file.
|
||||
func (c *Config) readableByOthersWarning() string {
|
||||
if c.S3.AccessKeyID != "" || c.S3.SecretAccessKey != "" {
|
||||
return "Config file is readable by others and may contain S3 credentials"
|
||||
}
|
||||
|
||||
return "Config file is readable by others"
|
||||
}
|
||||
|
||||
// validateStorage validates storage configuration.
|
||||
// If StorageURL is set, it takes precedence. S3 URLs require credentials.
|
||||
// File URLs don't require any S3 configuration.
|
||||
|
||||
@@ -8,7 +8,6 @@ import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/chunker"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -167,7 +166,6 @@ func TestValidateBlobSizeLimit(t *testing.T) {
|
||||
ChunkSize: chunkSize,
|
||||
BlobSizeLimit: blobLimit,
|
||||
CompressionLevel: 3,
|
||||
S3: S3Config{PartSize: defaultS3PartSize},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -223,120 +221,6 @@ func TestValidateBlobSizeLimit(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestValidateS3PartSize checks that s3.part_size is held to the part sizes
|
||||
// S3 accepts, 5MiB to 5GiB, by changing only the part size of the test
|
||||
// config. "5MB" in the config file is 5,000,000 bytes, below the minimum.
|
||||
func TestValidateS3PartSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
base, err := Load(os.Getenv("VAULTIK_CONFIG"))
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to load config: %v", err)
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
partSize Size
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "5MB is rejected",
|
||||
partSize: 5_000_000,
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "one byte below 5MiB is rejected",
|
||||
partSize: minS3PartSize - 1,
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "5MiB is accepted",
|
||||
partSize: minS3PartSize,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "5GiB is accepted",
|
||||
partSize: maxS3PartSize,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "one byte above 5GiB is rejected",
|
||||
partSize: maxS3PartSize + 1,
|
||||
wantErr: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := *base
|
||||
cfg.S3.PartSize = tt.partSize
|
||||
|
||||
err := cfg.Validate()
|
||||
if tt.wantErr {
|
||||
if !errors.Is(err, errBadS3PartSize) {
|
||||
t.Fatalf("Validate() error = %v, want errBadS3PartSize", err)
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
t.Fatalf("Validate() unexpected error: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoadS3PartSize checks that a config file without s3.part_size loads
|
||||
// with the 5MiB default, and that an explicit 0 fails at load like any other
|
||||
// part size S3 refuses.
|
||||
func TestLoadS3PartSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const withoutPartSize = "snapshots:\n" +
|
||||
" test:\n" +
|
||||
" paths: [/tmp/vaultik-test-source]\n" +
|
||||
"storage_url: file:///tmp/vaultik-test-storage\n"
|
||||
|
||||
writeConfig := func(t *testing.T, text string) string {
|
||||
t.Helper()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "config.yml")
|
||||
|
||||
err := os.WriteFile(path, []byte(text), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
|
||||
return path
|
||||
}
|
||||
|
||||
t.Run("absent loads as 5MiB", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg, err := Load(writeConfig(t, withoutPartSize))
|
||||
if err != nil {
|
||||
t.Fatalf("Load() unexpected error: %v", err)
|
||||
}
|
||||
|
||||
if cfg.S3.PartSize != defaultS3PartSize {
|
||||
t.Errorf("s3.part_size = %d, want %d",
|
||||
cfg.S3.PartSize, defaultS3PartSize)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("0 is rejected", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := Load(writeConfig(t, withoutPartSize+"s3:\n part_size: 0\n"))
|
||||
if !errors.Is(err, errBadS3PartSize) {
|
||||
t.Fatalf("Load() error = %v, want errBadS3PartSize", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestValidateAgeRecipients checks that recipients are parsed at config load
|
||||
// (a bad entry fails immediately, not mid-backup) and that no invalid entry —
|
||||
// least of all a pasted secret key — is echoed in the error. An empty list
|
||||
@@ -352,7 +236,6 @@ func TestValidateAgeRecipients(t *testing.T) {
|
||||
ChunkSize: Size(10 * 1024 * 1024),
|
||||
BlobSizeLimit: Size(10 * 1024 * 1024 * 1024),
|
||||
CompressionLevel: 3,
|
||||
S3: S3Config{PartSize: defaultS3PartSize},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -460,125 +343,3 @@ func TestAgeSecretKeySourceName(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// loadReadableConfig writes configYAML to a file that others can read,
|
||||
// loads it, and returns what the logger wrote to stderr meanwhile. The
|
||||
// logger writes to the os.Stderr it finds when it is initialized, so
|
||||
// os.Stderr is pointed at a file first. Not parallel-safe: os.Stderr and
|
||||
// the logger are process-global.
|
||||
func loadReadableConfig(t *testing.T, configYAML string) string {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
configPath := filepath.Join(dir, "config.yml")
|
||||
stderrPath := filepath.Join(dir, "stderr")
|
||||
|
||||
err := os.WriteFile(configPath, []byte(configYAML), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("writing config: %v", err)
|
||||
}
|
||||
|
||||
//nolint:gosec // G302: the test needs a config file others can read
|
||||
err = os.Chmod(configPath, 0o644)
|
||||
if err != nil {
|
||||
t.Fatalf("chmod config: %v", err)
|
||||
}
|
||||
|
||||
stderrFile, err := os.Create(stderrPath) //nolint:gosec // G304: test temp path
|
||||
if err != nil {
|
||||
t.Fatalf("creating stderr file: %v", err)
|
||||
}
|
||||
|
||||
previous := os.Stderr
|
||||
os.Stderr = stderrFile
|
||||
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
_, loadErr := Load(configPath)
|
||||
|
||||
os.Stderr = previous
|
||||
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
_ = stderrFile.Close()
|
||||
|
||||
if loadErr != nil {
|
||||
t.Fatalf("Load() error = %v", loadErr)
|
||||
}
|
||||
|
||||
captured, err := os.ReadFile(stderrPath) //nolint:gosec // G304: test temp path
|
||||
if err != nil {
|
||||
t.Fatalf("reading stderr file: %v", err)
|
||||
}
|
||||
|
||||
return string(captured)
|
||||
}
|
||||
|
||||
// TestLoadWarnsReadableConfigWithoutS3Credentials checks that a config
|
||||
// file others can read, holding no S3 credentials, is warned about
|
||||
// without a claim that it holds them.
|
||||
//
|
||||
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
|
||||
func TestLoadWarnsReadableConfigWithoutS3Credentials(t *testing.T) {
|
||||
stderr := loadReadableConfig(t, `
|
||||
storage_url: file:///var/backups/vaultik
|
||||
snapshots:
|
||||
home:
|
||||
paths:
|
||||
- /home
|
||||
`)
|
||||
|
||||
if !strings.Contains(stderr, "Config file is readable by others") {
|
||||
t.Errorf("expected a warning that the file is readable by others, got %q",
|
||||
stderr)
|
||||
}
|
||||
|
||||
if strings.Contains(stderr, "S3 credentials") {
|
||||
t.Errorf("warning names S3 credentials the file does not set: %q", stderr)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoadWarnsReadableConfigWithS3Credentials checks that a config file
|
||||
// others can read and that sets S3 credentials, as values or as ${ENV:...}
|
||||
// references, is warned about as one that may contain them.
|
||||
//
|
||||
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
|
||||
func TestLoadWarnsReadableConfigWithS3Credentials(t *testing.T) {
|
||||
t.Setenv("VAULTIK_TEST_ACCESS_KEY_ID", "test-access-key")
|
||||
t.Setenv("VAULTIK_TEST_SECRET_ACCESS_KEY", "test-secret-key")
|
||||
|
||||
configs := map[string]string{
|
||||
"values": `
|
||||
storage_url: s3://bucket/prefix?endpoint=s3.example.com
|
||||
s3:
|
||||
access_key_id: test-access-key
|
||||
secret_access_key: test-secret-key
|
||||
snapshots:
|
||||
home:
|
||||
paths:
|
||||
- /home
|
||||
`,
|
||||
"references": `
|
||||
storage_url: s3://bucket/prefix?endpoint=s3.example.com
|
||||
s3:
|
||||
access_key_id: ${ENV:VAULTIK_TEST_ACCESS_KEY_ID}
|
||||
secret_access_key: ${ENV:VAULTIK_TEST_SECRET_ACCESS_KEY}
|
||||
snapshots:
|
||||
home:
|
||||
paths:
|
||||
- /home
|
||||
`,
|
||||
}
|
||||
|
||||
for name, configYAML := range configs {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
stderr := loadReadableConfig(t, configYAML)
|
||||
|
||||
if !strings.Contains(stderr,
|
||||
"Config file is readable by others and may contain S3 credentials") {
|
||||
t.Errorf("expected a warning naming the S3 credentials, got %q",
|
||||
stderr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,7 +3,6 @@ package database
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
@@ -368,7 +367,7 @@ func verifyBlobNullUploadTS(
|
||||
}
|
||||
|
||||
// createLargeDatasetFiles creates fileCount files and adds every other
|
||||
// one to the snapshot, in one transaction as a backup writes them.
|
||||
// one to the snapshot.
|
||||
func createLargeDatasetFiles(
|
||||
t *testing.T,
|
||||
repos *Repositories,
|
||||
@@ -377,38 +376,31 @@ func createLargeDatasetFiles(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
ctx := context.Background()
|
||||
start := time.Now()
|
||||
|
||||
err := repos.WithTx(context.Background(),
|
||||
func(ctx context.Context, tx *sql.Tx) error {
|
||||
for i := range fileCount {
|
||||
file := &File{
|
||||
Path: types.FilePath(fmt.Sprintf("/large/file%05d.txt", i)),
|
||||
MTime: time.Now(),
|
||||
Size: int64(i * 1024),
|
||||
Mode: 0644,
|
||||
UID: uint32(1000 + (i % 10)),
|
||||
GID: uint32(1000 + (i % 10)),
|
||||
}
|
||||
for i := range fileCount {
|
||||
file := &File{
|
||||
Path: types.FilePath(fmt.Sprintf("/large/file%05d.txt", i)),
|
||||
MTime: time.Now(),
|
||||
Size: int64(i * 1024),
|
||||
Mode: 0644,
|
||||
UID: uint32(1000 + (i % 10)),
|
||||
GID: uint32(1000 + (i % 10)),
|
||||
}
|
||||
|
||||
err := repos.Files.Create(ctx, tx, file)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating file %d: %w", i, err)
|
||||
}
|
||||
err := repos.Files.Create(ctx, nil, file)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create file %d: %v", i, err)
|
||||
}
|
||||
|
||||
// Add half to snapshot
|
||||
if i%2 == 0 {
|
||||
err = repos.Snapshots.AddFileByID(ctx, tx, snapshotID, file.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
// Add half to snapshot
|
||||
if i%2 == 0 {
|
||||
err = repos.Snapshots.AddFileByID(ctx, nil, snapshotID, file.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
t.Logf("Created %d files in %v", fileCount, time.Since(start))
|
||||
|
||||
+5
-23
@@ -27,12 +27,10 @@ type Client struct {
|
||||
bucket string
|
||||
prefix string
|
||||
endpoint string
|
||||
partSize int64
|
||||
}
|
||||
|
||||
// Config contains S3 client configuration.
|
||||
// All fields are required except Prefix, which defaults to an empty string,
|
||||
// and PartSize, where zero means the SDK default of 5 MiB.
|
||||
// All fields are required except Prefix, which defaults to an empty string.
|
||||
// A non-empty Prefix is joined to every key with one "/", whether or not
|
||||
// it ends with one.
|
||||
// The Endpoint field should include the protocol (http:// or https://).
|
||||
@@ -43,9 +41,6 @@ type Config struct {
|
||||
AccessKeyID string
|
||||
SecretAccessKey string
|
||||
Region string
|
||||
// PartSize is the size in bytes of each part of a multipart upload.
|
||||
// An upload too large for S3's limit of 10,000 parts gets larger parts.
|
||||
PartSize int64
|
||||
}
|
||||
|
||||
// nopLogger is a logger that discards all output.
|
||||
@@ -95,7 +90,6 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) {
|
||||
bucket: cfg.Bucket,
|
||||
prefix: prefix,
|
||||
endpoint: cfg.Endpoint,
|
||||
partSize: cfg.PartSize,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -129,9 +123,12 @@ func (c *Client) PutObjectWithProgress(
|
||||
) error {
|
||||
fullKey := c.prefix + key
|
||||
|
||||
// uploadPartSize is 10MB for better progress granularity.
|
||||
const uploadPartSize = 10 * 1024 * 1024
|
||||
|
||||
// Create an uploader with the S3 client
|
||||
uploader := manager.NewUploader(c.s3Client, func(u *manager.Uploader) {
|
||||
u.PartSize = uploadPartSize(c.partSize, size)
|
||||
u.PartSize = uploadPartSize
|
||||
})
|
||||
|
||||
// Create a progress reader that tracks upload progress
|
||||
@@ -152,21 +149,6 @@ func (c *Client) PutObjectWithProgress(
|
||||
return err
|
||||
}
|
||||
|
||||
// uploadPartSize returns the part size for an upload of size bytes: the
|
||||
// configured part size (the SDK default when zero), raised where needed so
|
||||
// the upload fits in S3's limit of 10,000 parts. The uploader cannot raise
|
||||
// it itself, because it cannot seek the progress reader to learn its size.
|
||||
func uploadPartSize(configured, size int64) int64 {
|
||||
if configured == 0 {
|
||||
configured = manager.DefaultUploadPartSize
|
||||
}
|
||||
|
||||
maxParts := int64(manager.MaxUploadParts)
|
||||
smallestThatFits := (size + maxParts - 1) / maxParts // rounded up
|
||||
|
||||
return max(configured, smallestThatFits)
|
||||
}
|
||||
|
||||
// GetObject downloads an object from S3 with the specified key.
|
||||
// The key is automatically prefixed with the configured prefix.
|
||||
// Returns a ReadCloser containing the object data. The caller must
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
package s3
|
||||
|
||||
import "testing"
|
||||
|
||||
// TestUploadPartSize checks that an upload too large for 10,000 parts of the
|
||||
// configured size gets parts just large enough to fit in 10,000.
|
||||
func TestUploadPartSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const mib = 1024 * 1024
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
configured int64
|
||||
size int64
|
||||
want int64
|
||||
}{
|
||||
{
|
||||
name: "an upload that fits keeps the configured size",
|
||||
configured: 5 * mib,
|
||||
size: 10 * 1024 * mib,
|
||||
want: 5 * mib,
|
||||
},
|
||||
{
|
||||
name: "exactly 10,000 parts keeps the configured size",
|
||||
configured: 6 * mib,
|
||||
size: 10_000 * 6 * mib,
|
||||
want: 6 * mib,
|
||||
},
|
||||
{
|
||||
name: "one byte more than 10,000 parts adds a byte to each",
|
||||
configured: 6 * mib,
|
||||
size: 10_000*6*mib + 1,
|
||||
want: 6*mib + 1,
|
||||
},
|
||||
{
|
||||
name: "zero means the SDK default of 5MiB",
|
||||
configured: 0,
|
||||
size: 1,
|
||||
want: 5 * mib,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got := uploadPartSize(tt.configured, tt.size)
|
||||
if got != tt.want {
|
||||
t.Errorf("uploadPartSize(%d, %d) = %d, want %d",
|
||||
tt.configured, tt.size, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -28,7 +28,6 @@ func provideClient(lc fx.Lifecycle, cfg *config.Config) (*Client, error) {
|
||||
AccessKeyID: cfg.S3.AccessKeyID,
|
||||
SecretAccessKey: cfg.S3.SecretAccessKey,
|
||||
Region: cfg.S3.Region,
|
||||
PartSize: cfg.S3.PartSize.Int64(),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
||||
@@ -105,21 +105,17 @@ func (sm *SnapshotManager) CreateSnapshot(
|
||||
return sm.CreateSnapshotWithName(ctx, hostname, "", version, gitRevision)
|
||||
}
|
||||
|
||||
// ShortHostname returns hostname up to its first dot. A snapshot ID starts
|
||||
// with this form, while the snapshots table stores the full hostname.
|
||||
func ShortHostname(hostname string) string {
|
||||
short, _, _ := strings.Cut(hostname, ".")
|
||||
|
||||
return short
|
||||
}
|
||||
|
||||
// CreateSnapshotWithName creates a new snapshot record with an optional
|
||||
// snapshot name. The snapshot ID format is: hostname_name_timestamp or
|
||||
// hostname_timestamp if name is empty.
|
||||
func (sm *SnapshotManager) CreateSnapshotWithName(
|
||||
ctx context.Context, hostname, name, version, gitRevision string,
|
||||
) (string, error) {
|
||||
shortHostname := ShortHostname(hostname)
|
||||
// Use short hostname (strip domain if present)
|
||||
shortHostname := hostname
|
||||
if before, _, ok := strings.Cut(hostname, "."); ok {
|
||||
shortHostname = before
|
||||
}
|
||||
|
||||
// Build snapshot ID with optional name
|
||||
timestamp := time.Now().UTC().Format("2006-01-02T15:04:05Z")
|
||||
|
||||
@@ -99,7 +99,6 @@ func storerFromParsedS3URL(parsed *URL, cfg *config.Config) (Storer, error) {
|
||||
AccessKeyID: cfg.S3.AccessKeyID,
|
||||
SecretAccessKey: cfg.S3.SecretAccessKey,
|
||||
Region: region,
|
||||
PartSize: cfg.S3.PartSize.Int64(),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("creating S3 client: %w", err)
|
||||
@@ -135,7 +134,6 @@ func storerFromLegacyS3Config(cfg *config.Config) (Storer, error) {
|
||||
AccessKeyID: cfg.S3.AccessKeyID,
|
||||
SecretAccessKey: cfg.S3.SecretAccessKey,
|
||||
Region: region,
|
||||
PartSize: cfg.S3.PartSize.Int64(),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("creating S3 client: %w", err)
|
||||
|
||||
@@ -1,14 +1,11 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"github.com/johannesboyne/gofakes3"
|
||||
@@ -22,13 +19,6 @@ import (
|
||||
// s3TestBucket is the bucket created for each in-process S3 server.
|
||||
const s3TestBucket = "test-bucket"
|
||||
|
||||
// Credentials for the tests that build a storer from a config.Config. The
|
||||
// in-process S3 server accepts any.
|
||||
const (
|
||||
s3TestAccessKeyID = "key"
|
||||
s3TestSecretAccessKey = "secret"
|
||||
)
|
||||
|
||||
// newS3Storer builds an s3:// backend backed by a fresh in-process
|
||||
// S3 server. It reuses the same in-memory S3 harness (gofakes3 + s3mem
|
||||
// over httptest) that internal/s3 and the not-found regression test use,
|
||||
@@ -138,8 +128,8 @@ func TestS3URLPrefixKeyLayout(t *testing.T) {
|
||||
storer, err := storage.NewStorer(&config.Config{
|
||||
StorageURL: storageURL + "?endpoint=" + srv.URL,
|
||||
S3: config.S3Config{
|
||||
AccessKeyID: s3TestAccessKeyID,
|
||||
SecretAccessKey: s3TestSecretAccessKey,
|
||||
AccessKeyID: "key",
|
||||
SecretAccessKey: "secret",
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
@@ -181,90 +171,6 @@ func TestS3URLPrefixKeyLayout(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestS3UploadUsesConfiguredPartSize checks that s3.part_size reaches the
|
||||
// multipart uploader, through storage_url and through the s3.* fields. An
|
||||
// object three parts long must arrive as three parts; at the SDK's default
|
||||
// of 5 MiB it would arrive as four.
|
||||
func TestS3UploadUsesConfiguredPartSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
partSize = 6 * 1024 * 1024
|
||||
wantParts = 3
|
||||
)
|
||||
|
||||
backend := s3mem.New()
|
||||
|
||||
err := backend.CreateBucket(s3TestBucket)
|
||||
if err != nil {
|
||||
t.Fatalf("create bucket: %v", err)
|
||||
}
|
||||
|
||||
// Every part of a multipart upload is one request with a partNumber.
|
||||
var parts atomic.Int32
|
||||
|
||||
fake := gofakes3.New(backend).Server()
|
||||
srv := httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Query().Has("partNumber") {
|
||||
parts.Add(1)
|
||||
}
|
||||
|
||||
fake.ServeHTTP(w, r)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
cfg *config.Config
|
||||
}{
|
||||
{
|
||||
name: "storage_url",
|
||||
cfg: &config.Config{
|
||||
StorageURL: "s3://" + s3TestBucket + "?endpoint=" + srv.URL,
|
||||
S3: config.S3Config{
|
||||
AccessKeyID: s3TestAccessKeyID,
|
||||
SecretAccessKey: s3TestSecretAccessKey,
|
||||
PartSize: partSize,
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "s3.endpoint",
|
||||
cfg: &config.Config{
|
||||
S3: config.S3Config{
|
||||
Endpoint: srv.URL,
|
||||
Bucket: s3TestBucket,
|
||||
AccessKeyID: s3TestAccessKeyID,
|
||||
SecretAccessKey: s3TestSecretAccessKey,
|
||||
PartSize: partSize,
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
parts.Store(0)
|
||||
|
||||
storer, err := storage.NewStorer(tc.cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: NewStorer: %v", tc.name, err)
|
||||
}
|
||||
|
||||
data := bytes.NewReader(make([]byte, wantParts*partSize))
|
||||
|
||||
err = storer.PutWithProgress(
|
||||
context.Background(), "blob", data, data.Size(), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: PutWithProgress: %v", tc.name, err)
|
||||
}
|
||||
|
||||
if got := parts.Load(); got != wantParts {
|
||||
t.Errorf("%s: uploaded in %d parts, want %d", tc.name, got, wantParts)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// listStreamKeys returns the keys ListStream yields under a prefix, and
|
||||
// fails the test on a listing error.
|
||||
func listStreamKeys(t *testing.T, s storage.Storer, prefix string) []string {
|
||||
|
||||
@@ -130,9 +130,10 @@ func assertThirdSnapshotRestores(
|
||||
// up, and that snapshot is removed. The first snapshot keeps the file row,
|
||||
// which now lists the appended content's chunks, while removal drops the
|
||||
// blob that held them.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupAfterRemovingNewestSnapshotRestoresChangedFile(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -177,9 +178,10 @@ func TestBackupAfterRemovingNewestSnapshotRestoresChangedFile(t *testing.T) {
|
||||
// The next run's prune drops that incomplete snapshot and its blob, while
|
||||
// the first snapshot keeps the file row, which now lists the appended
|
||||
// content's chunks.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupAfterInterruptedRunRestoresChangedFile(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
|
||||
@@ -38,9 +38,11 @@ import (
|
||||
// (https://git.eeqj.de/sneak/vaultik/issues/130) and is not re-tested
|
||||
// here; these tests target the layers above the backend.
|
||||
//
|
||||
// log.Initialize replaces the package-global logger that a running
|
||||
// backup or restore reads, so each test calls it before t.Parallel,
|
||||
// while no parallel test is running yet.
|
||||
// The tests run serially, not with t.Parallel: each calls
|
||||
// log.Initialize, which replaces the package-global logger, and a
|
||||
// backup or restore running concurrently reads that same logger. Under
|
||||
// -race the two collide. Running one at a time is the same choice
|
||||
// prune_count_test.go already makes for the same reason.
|
||||
|
||||
const (
|
||||
faultChunkSize = int64(64 * 1024)
|
||||
@@ -163,19 +165,17 @@ func newReaderVaultik(
|
||||
// Scenario 3: a stored blob's bytes are flipped before restore reads
|
||||
// them. Restore must fail loudly, and no file must be left on the
|
||||
// restore target holding corrupt content.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRestoreRejectsCorruptBlob(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
assertRestoreRejectsDamagedBlob(t, faultstore.GetCorrupt, "corrupt")
|
||||
}
|
||||
|
||||
// Scenario 4: a stored blob is truncated before restore reads it. Same
|
||||
// contract as the corrupt case.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRestoreRejectsTruncatedBlob(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
assertRestoreRejectsDamagedBlob(t, faultstore.GetTruncate, "truncated")
|
||||
}
|
||||
|
||||
@@ -188,6 +188,7 @@ func assertRestoreRejectsDamagedBlob(
|
||||
t *testing.T, fault faultstore.GetFault, name string,
|
||||
) {
|
||||
t.Helper()
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -231,9 +232,10 @@ func assertRestoreRejectsDamagedBlob(
|
||||
|
||||
// Scenario 6: the backend accepts blob uploads and reports success but
|
||||
// stores nothing. verify --deep must catch it.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestDeepVerifyCatchesLyingBackend(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -283,9 +285,10 @@ func TestDeepVerifyCatchesLyingBackend(t *testing.T) {
|
||||
// Scenario 1a: a blob upload fails partway through. The interrupted run
|
||||
// must not record the blob as uploaded, must not reference it from the
|
||||
// snapshot, and must leave no blob object at the destination.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestInterruptedBlobUploadRecordsNoUploadedBlob(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -358,9 +361,10 @@ func TestInterruptedBlobUploadRecordsNoUploadedBlob(t *testing.T) {
|
||||
// chunks in a blob that was actually uploaded, so the retry re-chunks and
|
||||
// re-uploads the affected data instead of silently referencing data that
|
||||
// never reached storage.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -422,9 +426,10 @@ func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) {
|
||||
// covered by TestBackupCompletesOnlyAfterMetadataExport
|
||||
// (https://git.eeqj.de/sneak/vaultik/issues/177); this test exercises the
|
||||
// lower-level export path in isolation.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -504,9 +509,10 @@ func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
|
||||
// destination. Rerunning the backup must then prune the incomplete
|
||||
// snapshot, produce a snapshot whose destination metadata and local index
|
||||
// agree, and restore. See https://git.eeqj.de/sneak/vaultik/issues/177.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupCompletesOnlyAfterMetadataExport(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -679,9 +685,10 @@ func faultScannerFactory(
|
||||
// Scenario 5: the restore target runs out of space mid-file. Restore
|
||||
// must fail with an out-of-space error, and must not leave a truncated
|
||||
// file at the target path presenting as a complete restore.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRestoreReportsDiskFull(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
osFS := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
|
||||
+18
-23
@@ -9,7 +9,6 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/dustin/go-humanize"
|
||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||
"sneak.berlin/go/vaultik/internal/types"
|
||||
)
|
||||
|
||||
@@ -47,9 +46,12 @@ const (
|
||||
year = 365 * day
|
||||
)
|
||||
|
||||
// A snapshot ID split on "_" has at least a hostname and a trailing
|
||||
// timestamp.
|
||||
const minSnapshotIDParts = 2
|
||||
// Snapshot IDs split on "_" into hostname, optional name parts, and a
|
||||
// trailing timestamp.
|
||||
const (
|
||||
minSnapshotIDParts = 2
|
||||
minSnapshotIDNameParts = 3
|
||||
)
|
||||
|
||||
// SnapshotInfo contains information about a snapshot.
|
||||
//
|
||||
@@ -119,27 +121,20 @@ func parseSnapshotTimestamp(snapshotID string) (time.Time, error) {
|
||||
return timestamp.UTC(), nil
|
||||
}
|
||||
|
||||
// parseSnapshotName extracts the snapshot name from a snapshot ID of the
|
||||
// form hostname_name_timestamp, given the hostname stored with that
|
||||
// snapshot. The hostname and the name may both contain underscores, so the
|
||||
// name is what is left after removing the short hostname and its "_" from
|
||||
// the front and the last "_" and the timestamp from the end. Returns "" for
|
||||
// an ID with no name (hostname_timestamp), and for an ID that does not start
|
||||
// with that hostname, which CreateSnapshotWithName never writes.
|
||||
func parseSnapshotName(snapshotID, hostname string) string {
|
||||
prefix := snapshot.ShortHostname(hostname) + "_"
|
||||
|
||||
rest, ok := strings.CutPrefix(snapshotID, prefix)
|
||||
if !ok {
|
||||
// parseSnapshotName extracts the snapshot name from a snapshot ID.
|
||||
// Format: hostname_snapshotname_timestamp — the middle part(s) between hostname
|
||||
// and the RFC3339 timestamp are the snapshot name (may contain underscores).
|
||||
// Returns the snapshot name, or empty string if the ID is malformed.
|
||||
func parseSnapshotName(snapshotID string) string {
|
||||
parts := strings.Split(snapshotID, "_")
|
||||
if len(parts) < minSnapshotIDNameParts {
|
||||
// Format: hostname_timestamp — no snapshot name
|
||||
return ""
|
||||
}
|
||||
|
||||
end := strings.LastIndex(rest, "_")
|
||||
if end < 0 {
|
||||
return ""
|
||||
}
|
||||
|
||||
return rest[:end]
|
||||
// Format: hostname_name_timestamp — middle parts are the name.
|
||||
// The last part is the RFC3339 timestamp, the first part is the hostname,
|
||||
// everything in between is the snapshot name (which may itself contain underscores).
|
||||
return strings.Join(parts[1:len(parts)-1], "_")
|
||||
}
|
||||
|
||||
// parseDuration parses a duration string with support for human-friendly units:
|
||||
|
||||
@@ -11,55 +11,33 @@ func TestParseSnapshotName(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
snapshotID string
|
||||
hostname string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "standard format with name",
|
||||
snapshotID: "myhost_home_2026-01-12T14:41:15Z",
|
||||
hostname: "myhost",
|
||||
want: "home",
|
||||
},
|
||||
{
|
||||
name: "standard format with different name",
|
||||
snapshotID: "server1_system_2026-02-15T09:30:00Z",
|
||||
hostname: "server1",
|
||||
want: "system",
|
||||
},
|
||||
{
|
||||
name: "name with underscores",
|
||||
snapshotID: "myhost_my_special_backup_2026-03-01T00:00:00Z",
|
||||
hostname: "myhost",
|
||||
want: "my_special_backup",
|
||||
},
|
||||
{
|
||||
name: "hostname with underscores",
|
||||
snapshotID: "my_host_docs_2026-03-01T00:00:00Z",
|
||||
hostname: "my_host",
|
||||
want: "docs",
|
||||
},
|
||||
{
|
||||
name: "stored hostname with domain",
|
||||
snapshotID: "my_host_mail_2026-03-01T00:00:00Z",
|
||||
hostname: "my_host.example.com",
|
||||
want: "mail",
|
||||
},
|
||||
{
|
||||
name: "no name",
|
||||
snapshotID: "my_host_2026-03-01T00:00:00Z",
|
||||
hostname: "my_host",
|
||||
want: "",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got := parseSnapshotName(tt.snapshotID, tt.hostname)
|
||||
got := parseSnapshotName(tt.snapshotID)
|
||||
if got != tt.want {
|
||||
t.Errorf("parseSnapshotName(%q, %q) = %q, want %q",
|
||||
tt.snapshotID, tt.hostname, got, tt.want)
|
||||
t.Errorf("parseSnapshotName(%q) = %q, want %q",
|
||||
tt.snapshotID, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
+24
-108
@@ -183,11 +183,6 @@ type SnapshotMetadataInfo struct {
|
||||
TotalSize int64 `json:"total_size"`
|
||||
BlobCount int `json:"blob_count"`
|
||||
BlobsSize int64 `json:"blobs_size"`
|
||||
|
||||
// Set when the listing holds this snapshot's manifest.json.zst. A
|
||||
// backup interrupted before its manifest upload leaves a directory
|
||||
// without one, which prune does not treat as a snapshot.
|
||||
hasManifest bool
|
||||
}
|
||||
|
||||
// RemoteInfoResult contains all remote storage information
|
||||
@@ -211,20 +206,9 @@ type RemoteInfoResult struct {
|
||||
ReferencedBlobCount int `json:"referenced_blob_count"`
|
||||
ReferencedBlobSize int64 `json:"referenced_blob_size"`
|
||||
|
||||
// Orphaned blobs. Both stay nil (null in the JSON) when a manifest
|
||||
// was listed but not read, since that snapshot's blobs would be
|
||||
// counted as orphaned.
|
||||
OrphanedBlobCount *int `json:"orphaned_blob_count"`
|
||||
OrphanedBlobSize *int64 `json:"orphaned_blob_size"`
|
||||
|
||||
// Remote key of each snapshot whose manifest could not be read
|
||||
UnreadableManifests []string `json:"unreadable_manifests,omitempty"`
|
||||
|
||||
// Number of manifests not read because the name above them under
|
||||
// metadata/ is not a remote key. The names themselves are not
|
||||
// reported: they come from the destination store and may hold
|
||||
// control characters.
|
||||
SkippedManifestCount int `json:"skipped_manifest_count,omitempty"`
|
||||
// Orphaned blobs
|
||||
OrphanedBlobCount int `json:"orphaned_blob_count"`
|
||||
OrphanedBlobSize int64 `json:"orphaned_blob_size"`
|
||||
}
|
||||
|
||||
// RemoteInfo displays information about remote storage
|
||||
@@ -250,28 +234,16 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
|
||||
v.stdoutf("Scanning snapshot metadata...\n")
|
||||
}
|
||||
|
||||
snapshotMetadata, snapshotIDs, skippedManifestCount, err := v.collectSnapshotMetadata()
|
||||
snapshotMetadata, snapshotIDs, err := v.collectSnapshotMetadata()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
result.SkippedManifestCount = skippedManifestCount
|
||||
|
||||
if showText {
|
||||
manifestCount := 0
|
||||
|
||||
for _, info := range snapshotMetadata {
|
||||
if info.hasManifest {
|
||||
manifestCount++
|
||||
}
|
||||
}
|
||||
|
||||
v.stdoutf("Downloading %d manifest(s)...\n", manifestCount)
|
||||
v.stdoutf("Downloading %d manifest(s)...\n", len(snapshotIDs))
|
||||
}
|
||||
|
||||
referencedBlobs, unreadableManifests := v.collectReferencedBlobsFromManifests(
|
||||
snapshotIDs, snapshotMetadata)
|
||||
result.UnreadableManifests = unreadableManifests
|
||||
referencedBlobs := v.collectReferencedBlobsFromManifests(snapshotIDs, snapshotMetadata)
|
||||
|
||||
v.populateRemoteInfoResult(result, snapshotMetadata, snapshotIDs, referencedBlobs)
|
||||
|
||||
@@ -284,7 +256,7 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
|
||||
"snapshots", result.TotalMetadataCount,
|
||||
"total_blobs", result.TotalBlobCount,
|
||||
"referenced_blobs", result.ReferencedBlobCount,
|
||||
"unreadable_manifests", len(result.UnreadableManifests))
|
||||
"orphaned_blobs", result.OrphanedBlobCount)
|
||||
|
||||
if jsonOutput {
|
||||
enc := json.NewEncoder(v.Stdout)
|
||||
@@ -301,18 +273,16 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
|
||||
}
|
||||
|
||||
// collectSnapshotMetadata scans remote metadata and returns
|
||||
// per-snapshot info, sorted IDs and the number of manifests it skipped
|
||||
// because the name above them is not a remote key.
|
||||
// per-snapshot info and sorted IDs.
|
||||
func (v *Vaultik) collectSnapshotMetadata() (
|
||||
map[string]*SnapshotMetadataInfo, []string, int, error,
|
||||
map[string]*SnapshotMetadataInfo, []string, error,
|
||||
) {
|
||||
snapshotMetadata := make(map[string]*SnapshotMetadataInfo)
|
||||
skippedManifestCount := 0
|
||||
|
||||
metadataCh := v.Storage.ListStream(v.ctx, "metadata/")
|
||||
for obj := range metadataCh {
|
||||
if obj.Err != nil {
|
||||
return nil, nil, 0, fmt.Errorf("listing metadata: %w", obj.Err)
|
||||
return nil, nil, fmt.Errorf("listing metadata: %w", obj.Err)
|
||||
}
|
||||
|
||||
parts := strings.Split(obj.Key, "/")
|
||||
@@ -321,22 +291,6 @@ func (v *Vaultik) collectSnapshotMetadata() (
|
||||
}
|
||||
|
||||
snapshotID := parts[1]
|
||||
filename := parts[2]
|
||||
isManifest := filename == "manifest.json.zst"
|
||||
|
||||
// The name comes from the destination store, which is not
|
||||
// trusted, and is printed in the report. Accept it only in the
|
||||
// form of a remote key.
|
||||
if !isBlobHash(snapshotID) {
|
||||
log.Warn("Skipping non-conforming key under metadata/",
|
||||
"key", obj.Key)
|
||||
|
||||
if isManifest {
|
||||
skippedManifestCount++
|
||||
}
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if _, exists := snapshotMetadata[snapshotID]; !exists {
|
||||
snapshotMetadata[snapshotID] = &SnapshotMetadataInfo{SnapshotID: snapshotID}
|
||||
@@ -344,10 +298,7 @@ func (v *Vaultik) collectSnapshotMetadata() (
|
||||
|
||||
info := snapshotMetadata[snapshotID]
|
||||
|
||||
if isManifest {
|
||||
info.hasManifest = true
|
||||
}
|
||||
|
||||
filename := parts[2]
|
||||
if strings.HasPrefix(filename, "manifest") {
|
||||
info.ManifestSize = obj.Size
|
||||
} else if strings.HasPrefix(filename, "db") {
|
||||
@@ -364,25 +315,17 @@ func (v *Vaultik) collectSnapshotMetadata() (
|
||||
|
||||
sort.Strings(snapshotIDs)
|
||||
|
||||
return snapshotMetadata, snapshotIDs, skippedManifestCount, nil
|
||||
return snapshotMetadata, snapshotIDs, nil
|
||||
}
|
||||
|
||||
// collectReferencedBlobsFromManifests downloads the listed manifests
|
||||
// and returns referenced blob hashes with sizes, and the remote keys
|
||||
// of the manifests it could not read.
|
||||
// collectReferencedBlobsFromManifests downloads manifests and returns
|
||||
// referenced blob hashes with sizes.
|
||||
func (v *Vaultik) collectReferencedBlobsFromManifests(
|
||||
snapshotIDs []string, snapshotMetadata map[string]*SnapshotMetadataInfo,
|
||||
) (map[string]int64, []string) {
|
||||
) map[string]int64 {
|
||||
referencedBlobs := make(map[string]int64)
|
||||
|
||||
var unreadable []string
|
||||
|
||||
for _, snapshotID := range snapshotIDs {
|
||||
info := snapshotMetadata[snapshotID]
|
||||
if !info.hasManifest {
|
||||
continue
|
||||
}
|
||||
|
||||
// snapshotIDs here are remote keys, taken straight from the
|
||||
// metadata/ listing. downloadManifestByKey is the single reader
|
||||
// for remote manifests; see its doc comment.
|
||||
@@ -390,11 +333,10 @@ func (v *Vaultik) collectReferencedBlobsFromManifests(
|
||||
if err != nil {
|
||||
log.Warn("Failed to read manifest", "snapshot", snapshotID, "error", err)
|
||||
|
||||
unreadable = append(unreadable, snapshotID)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
info := snapshotMetadata[snapshotID]
|
||||
info.BlobCount = manifest.BlobCount
|
||||
|
||||
var blobsSize int64
|
||||
@@ -407,7 +349,7 @@ func (v *Vaultik) collectReferencedBlobsFromManifests(
|
||||
info.BlobsSize = blobsSize
|
||||
}
|
||||
|
||||
return referencedBlobs, unreadable
|
||||
return referencedBlobs
|
||||
}
|
||||
|
||||
// populateRemoteInfoResult fills in the result's snapshot and
|
||||
@@ -436,9 +378,8 @@ func (v *Vaultik) populateRemoteInfoResult(
|
||||
}
|
||||
|
||||
// scanRemoteBlobStorage lists all blobs on remote and computes orphan
|
||||
// stats when every listed manifest was read. showText is true only
|
||||
// when the human report is being printed (not --json, not --quiet),
|
||||
// gating the progress line.
|
||||
// stats. showText is true only when the human report is being printed
|
||||
// (not --json, not --quiet), gating the progress line.
|
||||
func (v *Vaultik) scanRemoteBlobStorage(
|
||||
result *RemoteInfoResult, referencedBlobs map[string]int64, showText bool,
|
||||
) error {
|
||||
@@ -465,28 +406,13 @@ func (v *Vaultik) scanRemoteBlobStorage(
|
||||
result.TotalBlobSize += obj.Size
|
||||
}
|
||||
|
||||
// A blob named only by a manifest that could not be read, or by one
|
||||
// under a skipped name, would be counted as orphaned, so the orphan
|
||||
// figures stay unknown.
|
||||
if len(result.UnreadableManifests) > 0 || result.SkippedManifestCount > 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
var (
|
||||
orphanedCount int
|
||||
orphanedSize int64
|
||||
)
|
||||
|
||||
for hash, size := range allBlobs {
|
||||
if _, referenced := referencedBlobs[hash]; !referenced {
|
||||
orphanedCount++
|
||||
orphanedSize += size
|
||||
result.OrphanedBlobCount++
|
||||
result.OrphanedBlobSize += size
|
||||
}
|
||||
}
|
||||
|
||||
result.OrphanedBlobCount = &orphanedCount
|
||||
result.OrphanedBlobSize = &orphanedSize
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -539,21 +465,11 @@ func (v *Vaultik) printRemoteInfoTable(result *RemoteInfoResult) {
|
||||
v.stdoutf("Referenced by snapshots: %s (%s)\n",
|
||||
humanize.Comma(int64(result.ReferencedBlobCount)),
|
||||
ubytes(result.ReferencedBlobSize))
|
||||
|
||||
if result.OrphanedBlobCount == nil {
|
||||
v.stdoutf("Orphaned (unreferenced): unknown "+
|
||||
"(%d manifest(s) could not be read, "+
|
||||
"%d manifest(s) under a non-conforming name skipped)\n",
|
||||
len(result.UnreadableManifests), result.SkippedManifestCount)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
v.stdoutf("Orphaned (unreferenced): %s (%s)\n",
|
||||
humanize.Comma(int64(*result.OrphanedBlobCount)),
|
||||
ubytes(*result.OrphanedBlobSize))
|
||||
humanize.Comma(int64(result.OrphanedBlobCount)),
|
||||
ubytes(result.OrphanedBlobSize))
|
||||
|
||||
if *result.OrphanedBlobCount > 0 {
|
||||
if result.OrphanedBlobCount > 0 {
|
||||
v.stdoutf("\nRun 'vaultik prune' to remove orphaned blobs.\n")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -928,17 +928,17 @@ func setupDedupBackupEnv(
|
||||
}
|
||||
}
|
||||
|
||||
// runDedupSnapshot creates a snapshot with the given name, scans dataDir
|
||||
// into it, completes it, and exports its metadata, returning the snapshot
|
||||
// ID and scan result.
|
||||
// runDedupSnapshot creates a "dedup" snapshot, scans dataDir into it,
|
||||
// completes it, and exports its metadata, returning the snapshot ID and
|
||||
// scan result.
|
||||
func runDedupSnapshot(
|
||||
ctx context.Context, t *testing.T,
|
||||
sm *snapshot.SnapshotManager, scanner *snapshot.Scanner,
|
||||
hostname, name, dataDir, dbPath string,
|
||||
hostname, dataDir, dbPath string,
|
||||
) (string, *snapshot.ScanResult) {
|
||||
t.Helper()
|
||||
|
||||
id, err := sm.CreateSnapshotWithName(ctx, hostname, name, "v", "g")
|
||||
id, err := sm.CreateSnapshotWithName(ctx, hostname, "dedup", "v", "g")
|
||||
require.NoError(t, err)
|
||||
|
||||
result, err := scanner.Scan(ctx, dataDir, id)
|
||||
@@ -980,15 +980,16 @@ func TestDedupOnlySnapshotRestores(t *testing.T) {
|
||||
|
||||
// First snapshot — uploads all blobs.
|
||||
_, r1 := runDedupSnapshot(ctx, t, sm, makeScanner(),
|
||||
cfg.Hostname, "first", dataDir, dbPath)
|
||||
cfg.Hostname, dataDir, dbPath)
|
||||
require.Positive(t, r1.BlobsCreated,
|
||||
"first snapshot should upload at least one blob")
|
||||
|
||||
// Second snapshot — same data, every chunk dedups. Its own name gives
|
||||
// it a different snapshot ID without waiting for the one-second
|
||||
// timestamp in the ID to tick over.
|
||||
// Second snapshot — same data, every chunk dedups. Sleep past the
|
||||
// second-precision timestamp so the snapshot IDs differ.
|
||||
time.Sleep(1100 * time.Millisecond)
|
||||
|
||||
id2, r2 := runDedupSnapshot(ctx, t, sm, makeScanner(),
|
||||
cfg.Hostname, "second", dataDir, dbPath)
|
||||
cfg.Hostname, dataDir, dbPath)
|
||||
require.Equal(t, 0, r2.BlobsCreated,
|
||||
"second snapshot should upload zero new blobs (fully dedup'd)")
|
||||
|
||||
|
||||
@@ -6,7 +6,6 @@ import (
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/spf13/afero"
|
||||
@@ -78,9 +77,10 @@ func backUpThenUnplug(
|
||||
// TestFirstBackupCreatesDestinationDirectory checks that a first backup
|
||||
// to a destination directory that does not exist yet creates it, and
|
||||
// that the destination can be listed afterwards.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestFirstBackupCreatesDestinationDirectory(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
storeDir := filepath.Join(t.TempDir(), "volume", "backup")
|
||||
@@ -95,9 +95,10 @@ func TestFirstBackupCreatesDestinationDirectory(t *testing.T) {
|
||||
// TestListSnapshotsWarnsWhenDestinationMissing checks that snapshot list
|
||||
// warns and shows the local index alone, without reporting the local
|
||||
// snapshot as missing from the destination.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestListSnapshotsWarnsWhenDestinationMissing(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
v, repos, out := backUpThenUnplug(ctx, t)
|
||||
@@ -114,9 +115,10 @@ func TestListSnapshotsWarnsWhenDestinationMissing(t *testing.T) {
|
||||
// TestRemoveSnapshotWarnsWhenDestinationMissing checks that snapshot
|
||||
// remove warns that the metadata could not be removed from the
|
||||
// destination, instead of reporting that it was.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRemoveSnapshotWarnsWhenDestinationMissing(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
v, repos, out := backUpThenUnplug(ctx, t)
|
||||
@@ -135,9 +137,10 @@ func TestRemoveSnapshotWarnsWhenDestinationMissing(t *testing.T) {
|
||||
// TestPruneKeepsLocalRecordsWhenDestinationMissing checks that prune
|
||||
// fails on a destination it cannot list and deletes no local snapshot
|
||||
// record.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
v, repos, _ := backUpThenUnplug(ctx, t)
|
||||
@@ -150,22 +153,3 @@ func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, snapshots, 1, "prune must delete no local snapshot record")
|
||||
}
|
||||
|
||||
// TestPurgeSaysListingFailedOnceWhenDestinationMissing checks that
|
||||
// snapshot purge fails on a destination it cannot list, with an error
|
||||
// that says "listing remote snapshots" once.
|
||||
func TestPurgeSaysListingFailedOnceWhenDestinationMissing(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
v, _, _ := backUpThenUnplug(ctx, t)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
Force: true,
|
||||
})
|
||||
require.ErrorIs(t, err, fs.ErrNotExist)
|
||||
assert.Equal(t, 1, strings.Count(err.Error(), "listing remote snapshots"),
|
||||
err.Error())
|
||||
}
|
||||
|
||||
@@ -14,9 +14,10 @@ import (
|
||||
// the discarded-error bug: getTableCount for a table its query cannot
|
||||
// resolve must not silently become 0. A count that could not be read is
|
||||
// reported as unknown, which a reader can tell apart from an empty table.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestTableCountForReportSurfacesReadFailure(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ func setupConsistencyTest(
|
||||
completedAt := startedAt.Add(5 * time.Minute)
|
||||
snap := &database.Snapshot{
|
||||
ID: types.SnapshotID(id),
|
||||
Hostname: snapHostname,
|
||||
Hostname: testHostname,
|
||||
VaultikVersion: testLabel,
|
||||
StartedAt: startedAt,
|
||||
CompletedAt: &completedAt,
|
||||
|
||||
@@ -17,10 +17,8 @@ import (
|
||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||
)
|
||||
|
||||
// Snapshot IDs reused across the purge tests, and the hostname they were
|
||||
// taken on.
|
||||
// Snapshot IDs reused across the purge tests.
|
||||
const (
|
||||
snapHostname = "testhost"
|
||||
snapSystemT0 = "testhost_system_2026-01-01T00:00:00Z"
|
||||
snapHomeT0 = "testhost_home_2026-01-01T00:00:00Z"
|
||||
snapHomeT1 = "testhost_home_2026-01-01T01:00:00Z"
|
||||
@@ -28,12 +26,9 @@ const (
|
||||
)
|
||||
|
||||
// setupPurgeTest creates a Vaultik instance with an in-memory database and mock
|
||||
// storage pre-populated with the given snapshot IDs, all taken on hostname.
|
||||
// Each snapshot is marked as completed. Remote metadata stubs are created so
|
||||
// syncWithRemote keeps them.
|
||||
func setupPurgeTest(
|
||||
t *testing.T, hostname string, snapshotIDs []string,
|
||||
) *vaultik.Vaultik {
|
||||
// storage pre-populated with the given snapshot IDs. Each snapshot is marked as
|
||||
// completed. Remote metadata stubs are created so syncWithRemote keeps them.
|
||||
func setupPurgeTest(t *testing.T, snapshotIDs []string) *vaultik.Vaultik {
|
||||
t.Helper()
|
||||
|
||||
ctx := context.Background()
|
||||
@@ -56,7 +51,7 @@ func setupPurgeTest(
|
||||
completedAt := startedAt.Add(5 * time.Minute)
|
||||
snap := &database.Snapshot{
|
||||
ID: types.SnapshotID(id),
|
||||
Hostname: types.Hostname(hostname),
|
||||
Hostname: "testhost",
|
||||
VaultikVersion: testLabel,
|
||||
StartedAt: startedAt,
|
||||
CompletedAt: &completedAt,
|
||||
@@ -125,7 +120,7 @@ func TestPurgeKeepLatest_PerName(t *testing.T) {
|
||||
"testhost_system_2026-01-01T04:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -153,7 +148,7 @@ func TestPurgeKeepLatest_SingleName(t *testing.T) {
|
||||
"testhost_home_2026-01-01T02:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -181,7 +176,7 @@ func TestPurgeKeepLatest_WithNameFilter(t *testing.T) {
|
||||
"testhost_home_2026-01-01T04:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -203,7 +198,7 @@ func TestPurgeKeepLatest_NoSnapshots(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
v := setupPurgeTest(t, snapHostname, nil)
|
||||
v := setupPurgeTest(t, nil)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -221,7 +216,7 @@ func TestPurgeKeepLatest_NameFilterNoMatch(t *testing.T) {
|
||||
"testhost_system_2026-01-01T01:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -248,7 +243,7 @@ func TestPurgeOlderThan_WithNameFilter(t *testing.T) {
|
||||
snapHomeT0,
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
|
||||
// Purge only "home" snapshots older than 365 days
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
@@ -282,7 +277,7 @@ func TestPurgeKeepLatest_ThreeNames(t *testing.T) {
|
||||
"testhost_home_2026-01-01T06:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -296,28 +291,3 @@ func TestPurgeKeepLatest_ThreeNames(t *testing.T) {
|
||||
assert.Contains(t, remaining, "testhost_system_2026-01-01T04:00:00Z")
|
||||
assert.Contains(t, remaining, "testhost_media_2026-01-01T05:00:00Z")
|
||||
}
|
||||
|
||||
// A hostname may contain underscores, so the snapshot name cannot be found
|
||||
// by splitting the ID at them. A purge by name must still select "docs".
|
||||
func TestPurgeKeepLatest_HostnameWithUnderscore(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
system = "my_host_system_2026-01-01T00:00:00Z"
|
||||
docsT1 = "my_host_docs_2026-01-01T01:00:00Z"
|
||||
docsT2 = "my_host_docs_2026-01-01T02:00:00Z"
|
||||
)
|
||||
|
||||
v := setupPurgeTest(t, "my_host", []string{system, docsT1, docsT2})
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
Force: true,
|
||||
Names: []string{"docs"},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.ElementsMatch(t, []string{system, docsT2},
|
||||
listRemainingSnapshots(t, v))
|
||||
}
|
||||
|
||||
@@ -1,159 +0,0 @@
|
||||
package vaultik_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||
)
|
||||
|
||||
// testBlobHashB is a blob that the manifest written by addRemote does
|
||||
// not reference.
|
||||
const testBlobHashB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" +
|
||||
"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
|
||||
|
||||
// TestRemoteInfo_UnreadableManifestLeavesOrphansUnknown checks that a
|
||||
// manifest remote info cannot read makes the orphan figures unknown. A
|
||||
// blob referenced only by that snapshot would otherwise be counted as
|
||||
// orphaned, and the report would advise running prune.
|
||||
func TestRemoteInfo_UnreadableManifestLeavesOrphansUnknown(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newListEnv(t)
|
||||
|
||||
// The readable manifest references blob A only.
|
||||
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
|
||||
addBlob(t, env.store.testStorer, testBlobHashA)
|
||||
addBlob(t, env.store.testStorer, testBlobHashB)
|
||||
|
||||
// With every manifest readable, blob B is orphaned.
|
||||
require.NoError(t, env.v.RemoteInfo(true))
|
||||
|
||||
var doc map[string]any
|
||||
|
||||
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
|
||||
assert.InDelta(t, 1, doc["orphaned_blob_count"], 0)
|
||||
|
||||
// A second snapshot whose manifest cannot be decoded. Blob B may be
|
||||
// one of its blobs.
|
||||
unreadableKey := snapshot.RemoteSnapshotKey(listLocalID)
|
||||
require.NoError(t, env.store.Put(context.Background(),
|
||||
"metadata/"+unreadableKey+"/manifest.json.zst",
|
||||
bytes.NewReader([]byte("not a valid manifest"))))
|
||||
|
||||
env.stdout.Reset()
|
||||
require.NoError(t, env.v.RemoteInfo(false))
|
||||
|
||||
text := env.stdout.String()
|
||||
assert.Contains(t, text, "Orphaned (unreferenced): unknown "+
|
||||
"(1 manifest(s) could not be read, "+
|
||||
"0 manifest(s) under a non-conforming name skipped)")
|
||||
assert.NotContains(t, text, "vaultik prune")
|
||||
|
||||
env.stdout.Reset()
|
||||
require.NoError(t, env.v.RemoteInfo(true))
|
||||
|
||||
doc = nil
|
||||
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
|
||||
assert.Contains(t, doc, "orphaned_blob_count")
|
||||
assert.Nil(t, doc["orphaned_blob_count"])
|
||||
assert.Contains(t, doc, "orphaned_blob_size")
|
||||
assert.Nil(t, doc["orphaned_blob_size"])
|
||||
assert.Equal(t, []any{unreadableKey}, doc["unreadable_manifests"])
|
||||
}
|
||||
|
||||
// TestRemoteInfo_SkipsNonConformingMetadataName checks that a directory
|
||||
// under metadata/ whose name is not a remote key is left out of the
|
||||
// report, and that the orphan figures are unknown when it holds a
|
||||
// manifest. The name comes from the destination store; printed raw, its
|
||||
// control characters would reach the terminal. Its manifest is not
|
||||
// read, so a blob only it references would otherwise be counted as
|
||||
// orphaned.
|
||||
func TestRemoteInfo_SkipsNonConformingMetadataName(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newListEnv(t)
|
||||
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
|
||||
addBlob(t, env.store.testStorer, testBlobHashA)
|
||||
addBlob(t, env.store.testStorer, testBlobHashB)
|
||||
require.NoError(t, env.store.Put(context.Background(),
|
||||
"metadata/\x1b[31mred/manifest.json.zst",
|
||||
bytes.NewReader([]byte("not a valid manifest"))))
|
||||
|
||||
require.NoError(t, env.v.RemoteInfo(false))
|
||||
|
||||
text := env.stdout.String()
|
||||
assert.NotContains(t, text, "\x1b")
|
||||
assert.NotContains(t, text, "31mred")
|
||||
assert.Contains(t, text, "Total (1 snapshots)")
|
||||
assert.Contains(t, text, "Orphaned (unreferenced): unknown "+
|
||||
"(0 manifest(s) could not be read, "+
|
||||
"1 manifest(s) under a non-conforming name skipped)")
|
||||
assert.NotContains(t, text, "vaultik prune")
|
||||
|
||||
env.stdout.Reset()
|
||||
require.NoError(t, env.v.RemoteInfo(true))
|
||||
|
||||
out := env.stdout.String()
|
||||
assert.NotContains(t, out, "31mred")
|
||||
|
||||
var doc map[string]any
|
||||
|
||||
require.NoError(t, json.Unmarshal([]byte(out), &doc))
|
||||
assert.Contains(t, doc, "orphaned_blob_count")
|
||||
assert.Nil(t, doc["orphaned_blob_count"])
|
||||
assert.Contains(t, doc, "orphaned_blob_size")
|
||||
assert.Nil(t, doc["orphaned_blob_size"])
|
||||
assert.InDelta(t, 1, doc["skipped_manifest_count"], 0)
|
||||
assert.NotContains(t, doc, "unreadable_manifests")
|
||||
}
|
||||
|
||||
// TestRemoteInfo_DirectoryWithoutManifestLeavesOrphansKnown checks that
|
||||
// a directory under metadata/ holding no manifest.json.zst, such as one
|
||||
// left by a backup interrupted before its manifest upload, leaves the
|
||||
// orphan figures known. prune does not treat such a directory as a
|
||||
// snapshot and deletes the blobs the report lists as orphaned.
|
||||
func TestRemoteInfo_DirectoryWithoutManifestLeavesOrphansKnown(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newListEnv(t)
|
||||
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
|
||||
addBlob(t, env.store.testStorer, testBlobHashA)
|
||||
addBlob(t, env.store.testStorer, testBlobHashB)
|
||||
|
||||
// One directory under a remote key and one under a non-conforming
|
||||
// name, each holding only a database.
|
||||
names := []string{snapshot.RemoteSnapshotKey(listLocalID), "\x1b[31mred"}
|
||||
for _, name := range names {
|
||||
require.NoError(t, env.store.Put(context.Background(),
|
||||
"metadata/"+name+"/db.zst.age",
|
||||
bytes.NewReader([]byte("not a valid database"))))
|
||||
}
|
||||
|
||||
require.NoError(t, env.v.RemoteInfo(false))
|
||||
|
||||
text := env.stdout.String()
|
||||
assert.NotContains(t, text, "\x1b")
|
||||
assert.Contains(t, text, "Downloading 1 manifest(s)...")
|
||||
assert.Contains(t, text, "Orphaned (unreferenced): 1 (")
|
||||
assert.Contains(t, text, "Run 'vaultik prune' to remove orphaned blobs.")
|
||||
|
||||
env.stdout.Reset()
|
||||
require.NoError(t, env.v.RemoteInfo(true))
|
||||
|
||||
var doc map[string]any
|
||||
|
||||
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
|
||||
assert.InDelta(t, 1, doc["orphaned_blob_count"], 0)
|
||||
assert.NotContains(t, doc, "unreadable_manifests")
|
||||
assert.NotContains(t, doc, "skipped_manifest_count")
|
||||
}
|
||||
@@ -164,9 +164,10 @@ func scratchEntries(t *testing.T, dir string) []string {
|
||||
// restore while a blob download is in progress. The download fails only
|
||||
// because of the cancel, so Restore must return context.Canceled without
|
||||
// reporting the file that needs the blob as failed.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRestoreSkipErrorsCancelDuringBlobDownload(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
|
||||
@@ -45,10 +45,9 @@ type missingBlobBackup struct {
|
||||
// after one blob of a two-blob snapshot was deleted. Every file stored in
|
||||
// that blob must be reported as failed and left absent, every other file
|
||||
// must be restored intact, and Restore must still return an error.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
backup := backupThenDeleteOneBlob(ctx, t)
|
||||
|
||||
@@ -86,10 +85,9 @@ func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
|
||||
|
||||
// TestRestoreMissingBlobAbortsWithoutSkipErrors checks that a deleted blob
|
||||
// still ends the restore with an error when SkipErrors is not set.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRestoreMissingBlobAbortsWithoutSkipErrors(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
backup := backupThenDeleteOneBlob(ctx, t)
|
||||
|
||||
@@ -109,6 +107,7 @@ func backupThenDeleteOneBlob(
|
||||
ctx context.Context, t *testing.T,
|
||||
) *missingBlobBackup {
|
||||
t.Helper()
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
|
||||
@@ -18,9 +18,10 @@ import (
|
||||
// A file rewritten with its size unchanged and a new mtime in the same
|
||||
// second as the mtime the index holds must still be backed up. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/226.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupOfSameSecondRewriteRestoresNewContent(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
|
||||
@@ -14,7 +14,6 @@ import (
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||
"sneak.berlin/go/vaultik/internal/types"
|
||||
)
|
||||
|
||||
// Sentinel errors for snapshot management.
|
||||
@@ -496,23 +495,19 @@ func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
|
||||
nameFilter[n] = struct{}{}
|
||||
}
|
||||
|
||||
// Collect completed snapshots and their names, applying the name filter.
|
||||
// Collect completed snapshots, applying the name filter.
|
||||
snapshots := make([]SnapshotInfo, 0, len(dbSnapshots))
|
||||
names := make(map[types.SnapshotID]string, len(dbSnapshots))
|
||||
|
||||
for _, s := range dbSnapshots {
|
||||
if s.CompletedAt == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
name := parseSnapshotName(s.ID.String(), s.Hostname.String())
|
||||
if len(nameFilter) > 0 {
|
||||
if _, ok := nameFilter[name]; !ok {
|
||||
if _, ok := nameFilter[parseSnapshotName(s.ID.String())]; !ok {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
names[s.ID] = name
|
||||
snapshots = append(snapshots, SnapshotInfo{
|
||||
ID: s.ID,
|
||||
Timestamp: s.StartedAt,
|
||||
@@ -525,7 +520,7 @@ func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
|
||||
return snapshots[i].Timestamp.After(snapshots[j].Timestamp)
|
||||
})
|
||||
|
||||
toDelete, err := selectSnapshotsToPurge(snapshots, names, opts)
|
||||
toDelete, err := selectSnapshotsToPurge(snapshots, opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -543,11 +538,9 @@ func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
|
||||
|
||||
// selectSnapshotsToPurge applies the purge retention criteria to the
|
||||
// newest-first sorted snapshot list and returns the deletion
|
||||
// candidates. names maps each snapshot's ID to its snapshot name.
|
||||
// candidates.
|
||||
func selectSnapshotsToPurge(
|
||||
snapshots []SnapshotInfo,
|
||||
names map[types.SnapshotID]string,
|
||||
opts *SnapshotPurgeOptions,
|
||||
snapshots []SnapshotInfo, opts *SnapshotPurgeOptions,
|
||||
) ([]SnapshotInfo, error) {
|
||||
var toDelete []SnapshotInfo
|
||||
|
||||
@@ -558,7 +551,7 @@ func selectSnapshotsToPurge(
|
||||
seen := make(map[string]bool)
|
||||
|
||||
for _, snap := range snapshots {
|
||||
name := names[snap.ID]
|
||||
name := parseSnapshotName(snap.ID.String())
|
||||
if seen[name] {
|
||||
toDelete = append(toDelete, snap)
|
||||
|
||||
@@ -1052,7 +1045,7 @@ func (v *Vaultik) syncWithRemote() error {
|
||||
// every local snapshot record (issue #160).
|
||||
remoteKeys, err := v.listAllRemoteSnapshotKeys()
|
||||
if err != nil {
|
||||
return err
|
||||
return fmt.Errorf("listing remote snapshots: %w", err)
|
||||
}
|
||||
|
||||
remoteKeySet := make(map[string]bool, len(remoteKeys))
|
||||
|
||||
@@ -54,6 +54,8 @@ type summaryEnv struct {
|
||||
func newSummaryEnv(t *testing.T) *summaryEnv {
|
||||
t.Helper()
|
||||
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
srcDir := filepath.Join(tempDir, "src")
|
||||
@@ -191,10 +193,9 @@ func (e *summaryEnv) dataLine(total, backedUp int64) string {
|
||||
// A first backup stores copy.bin's chunks while backing up a.bin, so
|
||||
// copy.bin's chunks are deduplicated within the run. Each file and byte
|
||||
// is still counted once.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestSnapshotSummaryFirstRun(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newSummaryEnv(t)
|
||||
|
||||
summary := env.backUp(t, "first", false)
|
||||
@@ -218,10 +219,9 @@ func TestSnapshotSummaryFirstRun(t *testing.T) {
|
||||
// An incremental backup where a.bin's mtime changed but its content did
|
||||
// not: a.bin is backed up again and every one of its chunks is already
|
||||
// stored.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestSnapshotSummaryIncrementalRunWithDeduplicatedChunks(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newSummaryEnv(t)
|
||||
|
||||
env.backUp(t, "first", false)
|
||||
@@ -251,10 +251,9 @@ func TestSnapshotSummaryIncrementalRunWithDeduplicatedChunks(t *testing.T) {
|
||||
// Under --cron the progress reporter is off; the upload figures must
|
||||
// still reach the summary and the snapshots row. The snapshot has two
|
||||
// paths, each backed up by its own scan.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestSnapshotSummaryCronRunRecordsUploads(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newSummaryEnv(t)
|
||||
|
||||
summary := env.backUp(t, "split", true)
|
||||
|
||||
@@ -18,9 +18,10 @@ import (
|
||||
// A backup without --cron runs the progress reporter while one scanner
|
||||
// scans each path of the snapshot in turn. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/253.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupWithoutCronOfTwoPathSnapshotRestoresBothPaths(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
const snapshotName = "data"
|
||||
|
||||
|
||||
+1
-1
@@ -19,7 +19,7 @@ s3:
|
||||
secret_access_key: test-secret-key
|
||||
region: us-east-1
|
||||
use_ssl: true
|
||||
part_size: 5242880 # 5MiB
|
||||
part_size: 5242880 # 5MB
|
||||
index_path: /tmp/vaultik-test.sqlite
|
||||
chunk_size: 10MB
|
||||
blob_size_limit: 10GB
|
||||
|
||||
Reference in New Issue
Block a user