Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b130c0ceb2 | ||
|
|
8b22ae8d42 | ||
|
|
3fc8a8f2f4 | ||
|
|
7696f83258 | ||
|
|
5d1118d143 | ||
|
|
b57ce2277d | ||
|
|
8496404d8b | ||
|
|
cdc60c4dfa | ||
|
|
49eed7a3e5 | ||
|
|
85d4ef118d |
@@ -200,8 +200,10 @@ local index or the destination store. `config`, `database delete`,
|
||||
### stdout and stderr
|
||||
|
||||
Log output — everything from `--verbose` and `--debug`, and every
|
||||
warning and error the logger emits — goes to **stderr**. stdout carries
|
||||
the output you asked for: tables, and the documents produced by `--json`.
|
||||
warning and error the logger emits — goes to **stderr**, and so does the
|
||||
startup banner. stdout carries the output you asked for: tables, the
|
||||
documents produced by `--json`, `config get` values, and completion
|
||||
scripts.
|
||||
|
||||
This means `vaultik snapshot list --verbose > out.txt` captures the
|
||||
listing and leaves the diagnostics on your terminal. To capture both,
|
||||
@@ -350,8 +352,9 @@ may hold snapshots this host doesn't know about), which is what
|
||||
prune` invocation to run as a follow-up. Local row cleanup (files,
|
||||
chunks, blobs the snapshot was the last referrer for) runs
|
||||
automatically. If the destination store is unreachable, the local-DB
|
||||
removal still completes and a warning is emitted; rerun `vaultik prune`
|
||||
once the store is reachable to finish remote cleanup. To wipe everything
|
||||
removal still completes and a warning is emitted; run `vaultik snapshot
|
||||
remove <snapshot-id>` again once the store is reachable to remove the
|
||||
snapshot's metadata from it (`vaultik prune` does not). To wipe everything
|
||||
on the destination in one go, use `vaultik remote nuke --force`.
|
||||
* `--local-only`: Skip remote cleanup; only touch the local index
|
||||
* `--dry-run`: Show what would be deleted without deleting
|
||||
@@ -387,7 +390,13 @@ recipients, and local database statistics.
|
||||
|
||||
**`remote info`**: Show storage backend type and location plus detailed
|
||||
remote storage inventory: per-snapshot metadata sizes, blob counts, and
|
||||
orphaned blob detection.
|
||||
orphaned blob detection. A name under `metadata/` that is not a remote
|
||||
key is skipped with a warning and is not printed. If a listed
|
||||
`manifest.json.zst` cannot be read, or sits under a skipped name, the
|
||||
orphaned blob figures are reported as unknown; `--json` gives them as
|
||||
`null`, lists the remote key of each unreadable manifest in
|
||||
`unreadable_manifests` and counts the manifests under skipped names in
|
||||
`skipped_manifest_count`.
|
||||
* `--json`: Output as JSON
|
||||
|
||||
**`remote nuke`**: Delete every snapshot's metadata and every blob from the
|
||||
@@ -645,8 +654,8 @@ Work planned after 1.0. Loosely ordered by priority.
|
||||
## output style
|
||||
|
||||
Every command's user-facing output is governed by `internal/ui`, in one
|
||||
of two ways. Color is enabled when stdout is a TTY and the `NO_COLOR`
|
||||
environment variable is unset (https://no-color.org/).
|
||||
of two ways. Color is enabled when the stream written to is a TTY and
|
||||
the `NO_COLOR` environment variable is unset (https://no-color.org/).
|
||||
|
||||
* **Status, progress, warnings, and errors** go through the `internal/ui`
|
||||
message methods below: marker-prefixed, colored on a TTY, and — except
|
||||
@@ -656,24 +665,26 @@ environment variable is unset (https://no-color.org/).
|
||||
`config init`, `config set`, and `database delete`.
|
||||
* **The data a command exists to produce** is written plain, with no
|
||||
marker and no color, because a marker would corrupt a table or a
|
||||
parsed document. This covers the `version`, `info`, and `remote info`
|
||||
reports, the `snapshot list` table, `config get` values, and every
|
||||
`--json` document. `--quiet` silences the human reports and tables
|
||||
(`version`, `info`, `remote info`, `snapshot list`) but never the
|
||||
machine-consumed `config get` value or the `--json` documents, which a
|
||||
script depends on. The `database delete` confirmation prompt is also
|
||||
written this way and always shown: it is an interactive exchange the
|
||||
operator must see.
|
||||
parsed document. This covers the `version`, `info`, `remote info` and
|
||||
`snapshot verify` reports, the `snapshot list` table, `config get`
|
||||
values, and every `--json` document. `--quiet` silences the human
|
||||
reports and tables (`version`, `info`, `remote info`, `snapshot
|
||||
verify`, `snapshot list`) but never the machine-consumed `config get`
|
||||
value or the `--json` documents, which a script depends on. The
|
||||
`database delete` confirmation prompt is also written this way and
|
||||
always shown: it is an interactive exchange the operator must see.
|
||||
|
||||
`internal/ui` writes to stdout; it is the output the user asked for.
|
||||
Structured log records are a different thing and go through
|
||||
`internal/log`, which writes to stderr (see "stdout and stderr" above).
|
||||
`internal/ui` writes to stdout; it is the output the user asked for. The
|
||||
exceptions are the startup banner and the error a failed command ends
|
||||
with, which go to stderr. Structured log records are a different thing
|
||||
and go through `internal/log`, which writes to stderr (see "stdout and
|
||||
stderr" above).
|
||||
|
||||
Message classes:
|
||||
|
||||
| Class | Marker | Alignment | Use for |
|
||||
|-------|--------|-----------|---------|
|
||||
| Banner | none | column 0 | The startup line printed once per invocation |
|
||||
| Banner | none | column 0 | The startup line printed once per invocation, on stderr |
|
||||
| Begin | `》` (white) | column 0 | An operation is about to start (present-continuous verb) |
|
||||
| Complete | `》` (green) | column 0 | An operation just finished (past-tense verb) |
|
||||
| Info | `》` (white) | column 0 | Neutral status update |
|
||||
|
||||
@@ -22,6 +22,93 @@ the tag exists and is exercised; what is left is merging `next` to
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-07: Made two messages say only what is true
|
||||
([issue #240](https://git.eeqj.de/sneak/vaultik/issues/240)). A config
|
||||
file that others can read was warned about as containing S3
|
||||
credentials even when it set none, as a `file://` config does. The
|
||||
warning now says the file may contain S3 credentials only when
|
||||
`s3.access_key_id` or `s3.secret_access_key` is set, since either may
|
||||
come from a `${...}` reference rather than the file, and otherwise
|
||||
says the file is readable by others. `snapshot purge` against a
|
||||
destination store it could not list gave an error with
|
||||
`listing remote snapshots:` in it twice; the prefix now appears once.
|
||||
|
||||
- 2026-10-07: Made `s3.part_size` set the multipart upload part size
|
||||
([issue #232](https://git.eeqj.de/sneak/vaultik/issues/232)). It was
|
||||
loaded and defaulted but never passed to the S3 client, whose uploader
|
||||
used a fixed 10MiB part. It now reaches the uploader for `storage_url`
|
||||
and for the `s3.*` fields, and a part size S3 refuses, below 5MiB or
|
||||
above 5GiB, `0` included, fails at config load. A blob too large for
|
||||
S3's limit of 10,000 parts at the configured size is uploaded in larger
|
||||
parts. The docs gave the default as `5MB`, which the config file reads
|
||||
as 5,000,000 bytes, below the minimum; they now say `5MiB`.
|
||||
|
||||
- 2026-10-07: Made per-name retention work when the hostname contains `_`
|
||||
([issue #230](https://git.eeqj.de/sneak/vaultik/issues/230)). A
|
||||
snapshot ID is `hostname_name_timestamp`, and the name was read as
|
||||
everything between the first and the last `_`, so with
|
||||
`hostname: my_host` the name `home` came out as `host_home`.
|
||||
`snapshot purge --keep-latest --snapshot home` then printed "No
|
||||
snapshots to delete", and `snapshot create --prune` purged nothing
|
||||
without a message. The name is now read using the hostname the
|
||||
`snapshots` table stores with each snapshot, cut at its first `.` as it
|
||||
is in the ID.
|
||||
|
||||
- 2026-10-07: Made `remote info` stop reporting a snapshot's blobs as
|
||||
orphaned when its manifest cannot be read, and stop printing raw
|
||||
names from under `metadata/`
|
||||
([issue #228](https://git.eeqj.de/sneak/vaultik/issues/228)). A
|
||||
manifest it failed to read was skipped, so that snapshot's blobs were
|
||||
counted as orphaned and the report advised running `vaultik prune`.
|
||||
The orphan figures are now unknown in that case, with no prune
|
||||
advice, and `--json` gives them as `null` with the unreadable remote
|
||||
keys in `unreadable_manifests`. A name under `metadata/` that is not
|
||||
64 lowercase hex characters is now skipped with a warning instead of
|
||||
being printed, control characters included. A manifest under a
|
||||
skipped name is then not read either, so it also leaves the orphan
|
||||
figures unknown, and `--json` counts such manifests in
|
||||
`skipped_manifest_count`. A directory with no manifest in it, as left
|
||||
by an interrupted backup, leaves the figures known.
|
||||
|
||||
- 2026-10-07: Made `config set` keep a string that looks like a number
|
||||
([issue #229](https://git.eeqj.de/sneak/vaultik/issues/229)). It wrote
|
||||
every value unquoted, and `config.Load` reads the file through untyped
|
||||
YAML, so an access key `00112233` loaded as `38043` and a hostname `007`
|
||||
as `7`. A value for a string setting in `config.Config` is now tagged as
|
||||
a YAML string, which the file quotes wherever YAML would read a number or
|
||||
a boolean; other settings are still written unquoted.
|
||||
|
||||
- 2026-10-07: Made a backup notice a file rewritten with its size
|
||||
unchanged and a new mtime in the same second as the one in the index
|
||||
([issue #226](https://git.eeqj.de/sneak/vaultik/issues/226)). The
|
||||
`files` table held mtime in whole seconds and the scanner compared
|
||||
whole seconds, so every later snapshot kept the old content. A new
|
||||
`mtime_nsec` column holds the nanoseconds within the second that
|
||||
`mtime` holds, and the scanner compares the full mtime. A local index
|
||||
created before the change lacks the column and is rebuilt with
|
||||
`vaultik database delete` and a full backup.
|
||||
|
||||
- 2026-10-07: Made taking the process-wide lock atomic
|
||||
([issue #227](https://git.eeqj.de/sneak/vaultik/issues/227)). The lock
|
||||
read `vaultik.pid`, checked whether that PID was alive and then wrote
|
||||
its own, so two writers started together could both pass the check and
|
||||
both run. It is now an `flock` on `vaultik.pid`, held until the run
|
||||
ends; the kernel drops it when the process exits, so a crash leaves no
|
||||
lock behind. A clean exit now empties the file instead of deleting it,
|
||||
because deleting it would let two later runs each lock a different
|
||||
file.
|
||||
|
||||
- 2026-10-06: Made `snapshot remove --json` write only its document to
|
||||
stdout when the destination store cannot be reached
|
||||
([issue #251](https://git.eeqj.de/sneak/vaultik/issues/251)). Its
|
||||
warning that the snapshot's metadata was left on the destination store
|
||||
went to stdout ahead of the document, breaking `| jq` on a command that
|
||||
exited 0. Under `--json` the warning now reaches stderr only, through
|
||||
the logger. The warning, the README and the command's help said
|
||||
`vaultik prune` would finish the cleanup, but `prune` never removes
|
||||
snapshot metadata; they now say to run `vaultik snapshot remove` for the
|
||||
snapshot again once the destination store is reachable.
|
||||
|
||||
- 2026-10-06: Made the backup summary and the `snapshots` row count each
|
||||
file, byte and upload once
|
||||
([issue #225](https://git.eeqj.de/sneak/vaultik/issues/225)). The
|
||||
@@ -35,6 +122,15 @@ the tag exists and is exercised; what is left is merging `next` to
|
||||
the snapshot references, and `docs/DATAMODEL.md` now says
|
||||
`chunk_count` and `blob_count` count what the run added.
|
||||
|
||||
- 2026-10-06: Made command output follow the README's stdout and stderr
|
||||
rules ([issue #224](https://git.eeqj.de/sneak/vaultik/issues/224)). The
|
||||
startup banner went to stdout, so a `completion` script or a
|
||||
`config get` value started with it; the banner now goes to stderr. A
|
||||
failing `remote info`, `prune` or `snapshot remove` under `--json`
|
||||
printed nothing on either stream, and now reports its error on stderr.
|
||||
`snapshot verify --quiet` printed its whole report; it now prints
|
||||
none, and a failure still reaches stderr with the same exit status.
|
||||
|
||||
- 2026-10-06: Made a backup without `--cron` of a snapshot with two or
|
||||
more `paths` complete instead of panicking with `close of closed
|
||||
channel` ([issue #253](https://git.eeqj.de/sneak/vaultik/issues/253)).
|
||||
|
||||
+5
-4
@@ -287,10 +287,11 @@ storage_url: "rclone://myremote/path/to/backups"
|
||||
# #use_ssl: true
|
||||
#
|
||||
# # Part size for multipart uploads
|
||||
# # Minimum 5MB, affects memory usage during upload
|
||||
# # Supports: 5MB, 10M, 100MiB, etc.
|
||||
# # Default: 5MB
|
||||
# #part_size: 5MB
|
||||
# # Minimum 5MiB, maximum 5GiB; affects memory usage during upload
|
||||
# # A blob too large for 10,000 parts of this size gets larger parts
|
||||
# # Supports: 10MB, 16MiB, 100MiB, etc. (5MB is below the minimum)
|
||||
# # Default: 5MiB
|
||||
# #part_size: 5MiB
|
||||
|
||||
# Path to local SQLite index database
|
||||
# This database tracks file state for incremental backups
|
||||
|
||||
+2
-1
@@ -36,7 +36,8 @@ Stores metadata about files in the filesystem being backed up.
|
||||
**Columns:**
|
||||
- `id` (TEXT PRIMARY KEY) - UUID for the file record
|
||||
- `path` (TEXT NOT NULL UNIQUE) - Absolute file path
|
||||
- `mtime` (INTEGER NOT NULL) - Modification time as Unix timestamp
|
||||
- `mtime` (INTEGER NOT NULL) - Modification time, whole seconds since the Unix epoch
|
||||
- `mtime_nsec` (INTEGER NOT NULL) - Nanoseconds within that second, 0 to 999999999
|
||||
- `size` (INTEGER NOT NULL) - File size in bytes
|
||||
- `mode` (INTEGER NOT NULL) - Unix file permissions and type
|
||||
- `uid` (INTEGER NOT NULL) - User ID of file owner
|
||||
|
||||
+13
-18
@@ -200,10 +200,10 @@ func RunApp(ctx context.Context, app *fx.App) error {
|
||||
}
|
||||
|
||||
// errReported marks a failure the operation has already shown the user
|
||||
// (and deliberately withheld under --json). Entry turns it into a
|
||||
// non-zero exit status without printing anything further, so the error
|
||||
// line is not doubled. It flows up from RunOperation through cobra to
|
||||
// Entry.
|
||||
// (or, under `snapshot verify --json`, put in its document). Entry
|
||||
// turns it into a non-zero exit status without printing anything
|
||||
// further, so the error line is not doubled. It flows up from
|
||||
// RunOperation through cobra to Entry.
|
||||
var errReported = errors.New("operation failed")
|
||||
|
||||
// RunOperation runs op against the Vaultik instance inside the fx app
|
||||
@@ -220,10 +220,10 @@ var errReported = errors.New("operation failed")
|
||||
// interrupt OnStop cancels op and waits for the goroutine to return, so
|
||||
// op's cleanup (removing decrypted scratch files) runs before the
|
||||
// process exits; the wait is bounded by shutdownTimeout. report is
|
||||
// called with a non-canceled failure so the caller can log it (and
|
||||
// suppress it under --json) before it becomes errReported. A context
|
||||
// cancellation is the interrupt path, not a failure: it is neither
|
||||
// reported nor counted as one.
|
||||
// called with a non-canceled failure so the caller can show it to the
|
||||
// user before it becomes errReported. A context cancellation is the
|
||||
// interrupt path, not a failure: it is neither reported nor counted as
|
||||
// one.
|
||||
func RunOperation(
|
||||
ctx context.Context, opts AppOptions,
|
||||
op func(v *vaultik.Vaultik) error, report func(err error),
|
||||
@@ -293,13 +293,12 @@ func RunOperation(
|
||||
// runVaultikApp runs the standard single-operation command lifecycle
|
||||
// shared by the snapshot list/purge/remove and remote nuke subcommands:
|
||||
// resolve the config, then run op against the Vaultik instance through
|
||||
// RunOperation, reporting a failure prefixed with failMsg (suppressed
|
||||
// while suppressErrors is true, e.g. under --json). mode says whether the
|
||||
// command takes the PID lock. jsonOutput marks a command whose stdout is a
|
||||
// JSON document: it quiets the UI but, unlike Quiet, leaves the stderr log
|
||||
// level alone.
|
||||
// RunOperation, reporting a failure prefixed with failMsg on stderr. mode
|
||||
// says whether the command takes the PID lock. jsonOutput marks a command
|
||||
// whose stdout is a JSON document: it quiets the UI but, unlike Quiet,
|
||||
// leaves the stderr log level alone.
|
||||
func runVaultikApp(
|
||||
cmd *cobra.Command, mode lockMode, jsonOutput, suppressErrors bool,
|
||||
cmd *cobra.Command, mode lockMode, jsonOutput bool,
|
||||
failMsg string, op func(v *vaultik.Vaultik) error,
|
||||
) error {
|
||||
configPath, err := ResolveConfigPath()
|
||||
@@ -319,10 +318,6 @@ func runVaultikApp(
|
||||
},
|
||||
Mode: mode,
|
||||
}, op, func(err error) {
|
||||
if suppressErrors {
|
||||
return
|
||||
}
|
||||
|
||||
log.Error(failMsg, "error", err)
|
||||
ReportErrorf("%s: %v", failMsg, err)
|
||||
})
|
||||
|
||||
+56
-1
@@ -7,11 +7,14 @@ import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
"gopkg.in/yaml.v3"
|
||||
"sneak.berlin/go/vaultik/internal/config"
|
||||
"sneak.berlin/go/vaultik/internal/ui"
|
||||
)
|
||||
|
||||
@@ -31,6 +34,9 @@ const configDirMode = 0o755
|
||||
// yaml.Marshal's 4-space default.
|
||||
const configYAMLIndent = 2
|
||||
|
||||
// yamlStringTag is YAML's tag for a string scalar.
|
||||
const yamlStringTag = "!!str"
|
||||
|
||||
var (
|
||||
errConfigExists = errors.New("config file already exists")
|
||||
errEmptyConfig = errors.New("empty config file")
|
||||
@@ -199,7 +205,7 @@ storage_url: ""
|
||||
# access_key_id: YOUR_ACCESS_KEY
|
||||
# secret_access_key: YOUR_SECRET_KEY
|
||||
# # region: us-east-1 # Default: us-east-1
|
||||
# # part_size: 5MB # Multipart upload part size. Default: 5MB
|
||||
# # part_size: 5MiB # Upload part size, 5MiB to 5GiB. Default: 5MiB
|
||||
# # For the s3:// form, disable TLS with ?ssl=false in the URL, not use_ssl.
|
||||
|
||||
# ─── OPTIONAL ────────────────────────────────────────────────────────────────
|
||||
@@ -583,9 +589,58 @@ func yamlPathSet(root *yaml.Node, keys []string, value string) error {
|
||||
}
|
||||
}
|
||||
|
||||
// config.Load reads the file through untyped YAML, which turns an
|
||||
// unquoted 00112233 into the number 38043 and 1e5 into 100000. Tagging
|
||||
// a string setting as a string makes the encoder quote such a value.
|
||||
// Other settings stay unquoted, so compression_level 9 is a number.
|
||||
// The encoder refuses to write a value that is not valid UTF-8 as a
|
||||
// string. Left untagged, such a value is written as base64 !!binary and
|
||||
// loads back unchanged.
|
||||
if configKeyIsString(keys) && utf8.ValidString(value) {
|
||||
node.Tag = yamlStringTag
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// configKeyIsString reports whether the dotted key names a string in
|
||||
// config.Config, following the fields' yaml tags, as s3.access_key_id and
|
||||
// snapshots.home.exclude.0 do.
|
||||
func configKeyIsString(keys []string) bool {
|
||||
typ := reflect.TypeFor[config.Config]()
|
||||
|
||||
for _, key := range keys {
|
||||
switch {
|
||||
case typ.Kind() == reflect.Map || typ.Kind() == reflect.Slice:
|
||||
// The key is a snapshot name or a list index.
|
||||
typ = typ.Elem()
|
||||
case typ.Kind() == reflect.Struct:
|
||||
field, ok := yamlField(typ, key)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
|
||||
typ = field.Type
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
return typ.Kind() == reflect.String
|
||||
}
|
||||
|
||||
// yamlField returns the field of struct type typ whose yaml tag names key.
|
||||
func yamlField(typ reflect.Type, key string) (reflect.StructField, bool) {
|
||||
for field := range typ.Fields() {
|
||||
name, _, _ := strings.Cut(field.Tag.Get("yaml"), ",")
|
||||
if name == key {
|
||||
return field, true
|
||||
}
|
||||
}
|
||||
|
||||
return reflect.StructField{}, false
|
||||
}
|
||||
|
||||
// yamlSetInMapping resolves (creating if needed) the value node for key
|
||||
// within a mapping node, setting it to value when it is the final path
|
||||
// element, and returns the node to descend into.
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -94,6 +95,110 @@ func TestConfigSetRecipientOnFreshConfig(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestConfigSetStringLooksLikeNumber sets string settings to values that
|
||||
// YAML reads as numbers or booleans when they are unquoted, and checks that
|
||||
// config.Load returns each one unchanged.
|
||||
func TestConfigSetStringLooksLikeNumber(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
key string
|
||||
value string
|
||||
field func(cfg *config.Config) string
|
||||
}{
|
||||
{"s3.access_key_id", "00112233",
|
||||
func(cfg *config.Config) string { return cfg.S3.AccessKeyID }},
|
||||
{"s3.secret_access_key", "12345678901234567890123456789012",
|
||||
func(cfg *config.Config) string { return cfg.S3.SecretAccessKey }},
|
||||
{"hostname", "007",
|
||||
func(cfg *config.Config) string { return cfg.Hostname }},
|
||||
{"s3.prefix", "1e5",
|
||||
func(cfg *config.Config) string { return cfg.S3.Prefix }},
|
||||
{"s3.bucket", "true",
|
||||
func(cfg *config.Config) string { return cfg.S3.Bucket }},
|
||||
{"s3.region", "FALSE",
|
||||
func(cfg *config.Config) string { return cfg.S3.Region }},
|
||||
{"snapshots.home.exclude.0", "1.10",
|
||||
func(cfg *config.Config) string { return cfg.Snapshots["home"].Exclude[0] }},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.key+"="+tt.value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := loadAfterConfigSet(t, tt.key, tt.value)
|
||||
|
||||
got := tt.field(cfg)
|
||||
if got != tt.value {
|
||||
t.Errorf("%s = %q after config set %q", tt.key, got, tt.value)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestConfigSetNonUTF8Path checks that config set still accepts a value that
|
||||
// is not valid UTF-8, such as a path with a Latin-1 file name, and that
|
||||
// config.Load returns it unchanged.
|
||||
func TestConfigSetNonUTF8Path(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const dir = "/srv/caf\xe9"
|
||||
|
||||
cfg := loadAfterConfigSet(t, "snapshots.home.paths.0", dir)
|
||||
|
||||
got := cfg.Snapshots["home"].Paths[0]
|
||||
if got != dir {
|
||||
t.Errorf("snapshots.home.paths.0 = %q, want %q", got, dir)
|
||||
}
|
||||
}
|
||||
|
||||
// TestConfigSetNumberStaysNumber checks that a number set for an integer
|
||||
// setting is still read as a number, not as a quoted string.
|
||||
func TestConfigSetNumberStaysNumber(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const level = 9
|
||||
|
||||
cfg := loadAfterConfigSet(t, "compression_level", strconv.Itoa(level))
|
||||
|
||||
if cfg.CompressionLevel != level {
|
||||
t.Errorf("compression_level = %d, want %d", cfg.CompressionLevel, level)
|
||||
}
|
||||
}
|
||||
|
||||
// loadAfterConfigSet writes the file `config init` writes, sets storage_url
|
||||
// to a local directory so that the file passes validation, applies
|
||||
// `config set key value` and returns what config.Load reads back.
|
||||
func loadAfterConfigSet(t *testing.T, key, value string) *config.Config {
|
||||
t.Helper()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "config.yml")
|
||||
|
||||
err := os.WriteFile(path, []byte(defaultConfigTemplate), configFileMode)
|
||||
if err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
|
||||
out := ui.NewWithColor(&bytes.Buffer{}, false)
|
||||
|
||||
err = writeConfigSet(out, path, "storage_url", "file:///mnt/backups")
|
||||
if err != nil {
|
||||
t.Fatalf("config set storage_url: %v", err)
|
||||
}
|
||||
|
||||
err = writeConfigSet(out, path, key, value)
|
||||
if err != nil {
|
||||
t.Fatalf("config set %s: %v", key, err)
|
||||
}
|
||||
|
||||
cfg, err := config.Load(path)
|
||||
if err != nil {
|
||||
t.Fatalf("config.Load: %v", err)
|
||||
}
|
||||
|
||||
return cfg
|
||||
}
|
||||
|
||||
const testYAML = `# top comment
|
||||
compression_level: 3
|
||||
age_recipients:
|
||||
|
||||
+10
-9
@@ -16,16 +16,18 @@ import (
|
||||
const shortCommitLen = 12
|
||||
|
||||
// Entry is the main entry point for the CLI application.
|
||||
// It prints the startup banner to stdout (unless a banner-suppressing
|
||||
// It prints the startup banner to stderr (unless a banner-suppressing
|
||||
// flag is present in os.Args — see bannerSuppressedInArgs), executes the
|
||||
// root cobra command, and routes any returned error through the
|
||||
// ui.Writer so the user sees a properly formatted "🛑 ERROR:" line.
|
||||
// The banner goes to stderr because stdout carries only the output the
|
||||
// user asked for, such as a completion script or a `config get` value.
|
||||
//
|
||||
// It returns the process exit code (0 on success, 1 on error) rather
|
||||
// than calling os.Exit, so that main's deferred profile writers run
|
||||
// before the process ends. See run in cmd/vaultik/main.go.
|
||||
func Entry() int {
|
||||
emitStartupBanner(os.Args[1:], os.Stdout)
|
||||
emitStartupBanner(os.Args[1:], os.Stderr)
|
||||
|
||||
rootCmd := NewRootCommand()
|
||||
rootCmd.SilenceErrors = true
|
||||
@@ -33,8 +35,9 @@ func Entry() int {
|
||||
err := rootCmd.Execute()
|
||||
if err != nil {
|
||||
// An operation that ran inside the fx app has already reported
|
||||
// its own failure (and suppressed it under --json); errReported
|
||||
// says so. Printing it again here would double the error line.
|
||||
// its own failure (`snapshot verify --json` puts it in the
|
||||
// document instead); errReported says so. Printing it again
|
||||
// here would double the error line.
|
||||
// Every other error — bad arguments, a config that would not
|
||||
// load — reaches Entry unreported, so it is shown here.
|
||||
if !errors.Is(err, errReported) {
|
||||
@@ -49,9 +52,8 @@ func Entry() int {
|
||||
|
||||
// emitStartupBanner writes the startup banner to w unless args (the
|
||||
// argument vector with the program name already stripped) contains a
|
||||
// flag that suppresses it. Split out of Entry so that the decision — the
|
||||
// only thing standing between a --json invocation and a parseable
|
||||
// stdout — is reachable from a test without running the whole CLI.
|
||||
// flag that suppresses it. Split out of Entry so that the decision is
|
||||
// reachable from a test without running the whole CLI.
|
||||
func emitStartupBanner(args []string, w io.Writer) {
|
||||
if bannerSuppressedInArgs(args) {
|
||||
return
|
||||
@@ -86,8 +88,7 @@ func ReportErrorf(format string, args ...any) {
|
||||
// --json is a subcommand flag rather than a persistent one, but so is
|
||||
// --cron (it exists only on `snapshot create`), so this adds no new
|
||||
// class of imprecision. The only cost of a false positive is a missing
|
||||
// decorative banner; the cost of a false negative is a corrupt document
|
||||
// on stdout, so the scan errs deliberately in that direction.
|
||||
// decorative banner.
|
||||
func bannerSuppressedInArgs(args []string) bool {
|
||||
for _, a := range args {
|
||||
if a == "--" {
|
||||
|
||||
@@ -36,23 +36,14 @@ const (
|
||||
// strips it before scanning, so it has to be present.
|
||||
programName = "vaultik"
|
||||
|
||||
// someSnapshotID is any snapshot identifier: these tests never run
|
||||
// the command, so it only has to occupy the positional argument.
|
||||
// someSnapshotID only fills the positional argument; no test needs
|
||||
// the snapshot to exist.
|
||||
someSnapshotID = "host_2026-01-01T00:00:00Z"
|
||||
)
|
||||
|
||||
// placeholderJSONDocument stands in for whatever document a --json
|
||||
// command writes to stdout. `snapshot list --json` with no snapshots
|
||||
// prints exactly this; the other --json commands print an object rather
|
||||
// than an array, but this test is not about their shape. It is about
|
||||
// what is on stdout *before* them, which is the same for all of them
|
||||
// because Entry prints the banner before cobra has parsed anything and
|
||||
// therefore before it can know which command is running.
|
||||
const placeholderJSONDocument = "[]\n"
|
||||
|
||||
// jsonArgumentVectors are the argument vectors of every --json
|
||||
// invocation the CLI accepts, with the program name stripped exactly as
|
||||
// Entry strips it. Each one must leave stdout untouched by the banner.
|
||||
// Entry strips it. Each one must suppress the banner.
|
||||
//
|
||||
//nolint:gochecknoglobals // read-only test fixture shared by two tests
|
||||
var jsonArgumentVectors = map[string][]string{
|
||||
@@ -74,39 +65,23 @@ var jsonArgumentVectors = map[string][]string{
|
||||
},
|
||||
}
|
||||
|
||||
// TestJSONInvocationStdoutIsExactlyOneDocument is the CLI-layer
|
||||
// regression guard for issue #106: `vaultik snapshot list --json | jq`
|
||||
// must work with no other flags.
|
||||
//
|
||||
// internal/vaultik's TestListSnapshots_JSONStdoutIsOnlyTheDocument
|
||||
// guards the same contract one layer down, but it calls the library
|
||||
// function directly and so cannot see Entry, which is where the
|
||||
// contamination was: the startup banner is written to stdout before
|
||||
// cobra parses anything, and the suppression scan did not know about
|
||||
// --json. The two banner lines and the blank line landed ahead of the
|
||||
// document and `jq` refused the result.
|
||||
//
|
||||
// The document is a constant here because this test is about the
|
||||
// argument vectors, one per --json command; the one that runs a real
|
||||
// command end to end is TestEntryJSONStdoutIsExactlyOneDocument below.
|
||||
func TestJSONInvocationStdoutIsExactlyOneDocument(t *testing.T) {
|
||||
// TestJSONInvocationSuppressesBanner checks that every --json
|
||||
// invocation suppresses the startup banner, as the README says --json
|
||||
// does along with --quiet and --cron. The scan is over the raw argument
|
||||
// vector, so each position and spelling of --json is listed.
|
||||
func TestJSONInvocationSuppressesBanner(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for name, argv := range jsonArgumentVectors {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var banner bytes.Buffer
|
||||
|
||||
emitStartupBanner(argv, &stdout)
|
||||
emitStartupBanner(argv, &banner)
|
||||
|
||||
require.Empty(t, stdout.String(),
|
||||
"nothing may reach stdout ahead of a --json document")
|
||||
|
||||
_, err := stdout.WriteString(placeholderJSONDocument)
|
||||
require.NoError(t, err)
|
||||
|
||||
requireExactlyOneJSONDocument(t, stdout.String())
|
||||
assert.Empty(t, banner.String(),
|
||||
"--json suppresses the banner")
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -127,11 +102,11 @@ func TestBannerStillPrintedWithoutSuppressingFlag(t *testing.T) {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var banner bytes.Buffer
|
||||
|
||||
emitStartupBanner(argv, &stdout)
|
||||
emitStartupBanner(argv, &banner)
|
||||
|
||||
assert.Contains(t, stdout.String(), "starting up at",
|
||||
assert.Contains(t, banner.String(), "starting up at",
|
||||
"the banner belongs on invocations that did not opt out")
|
||||
})
|
||||
}
|
||||
@@ -247,9 +222,7 @@ func TestEntryJSONStdoutIsExactlyOneDocument(t *testing.T) {
|
||||
// captureProcessStdout redirects the process's own stdout to a pipe for
|
||||
// the duration of fn and returns what was written to it. The redirection
|
||||
// has to be at the file-descriptor level rather than through an injected
|
||||
// writer, because the banner and the JSON encoder reach os.Stdout
|
||||
// independently and the point of the test is that both land in the same
|
||||
// place.
|
||||
// writer, because the commands Entry runs reach os.Stdout directly.
|
||||
//
|
||||
// Not parallel-safe: os.Stdout is process-global.
|
||||
func captureProcessStdout(t *testing.T, fn func()) string {
|
||||
|
||||
@@ -15,8 +15,8 @@ import (
|
||||
// run, so a failing command must come back with a non-zero code rather
|
||||
// than ending the process here.
|
||||
//
|
||||
// Stdout is captured only to keep the banner and command output off the
|
||||
// test log; the assertion is on the returned code.
|
||||
// Stdout and stderr are captured only to keep the banner and command
|
||||
// output off the test log; the assertion is on the returned code.
|
||||
//
|
||||
//nolint:paralleltest // replaces os.Args and rootFlags
|
||||
func TestEntryReturnsStatusCode(t *testing.T) {
|
||||
@@ -50,7 +50,7 @@ func TestEntryReturnsStatusCode(t *testing.T) {
|
||||
|
||||
var code int
|
||||
|
||||
_ = captureProcessStdout(t, func() { code = Entry() })
|
||||
_, _ = captureProcessStdoutAndStderr(t, func() { code = Entry() })
|
||||
|
||||
assert.Equal(t, testCase.want, code)
|
||||
})
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
package cli //nolint:testpackage // shares hermeticConfig and the capture helpers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/adrg/xdg"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/database"
|
||||
)
|
||||
|
||||
// TestEntryCompletionStdoutIsTheScript runs `vaultik completion bash`,
|
||||
// whose stdout the README tells the user to source. The script has to
|
||||
// start on the first line.
|
||||
//
|
||||
//nolint:paralleltest // replaces os.Args, os.Stdout and os.Stderr
|
||||
func TestEntryCompletionStdoutIsTheScript(t *testing.T) {
|
||||
code, stdout, _ := runEntry(t, "completion", "bash")
|
||||
|
||||
require.Equal(t, 0, code)
|
||||
|
||||
firstLine, _, _ := strings.Cut(stdout, "\n")
|
||||
assert.True(t, strings.HasPrefix(firstLine, "# bash completion"),
|
||||
"the first line of stdout must be the script's, got %q", firstLine)
|
||||
}
|
||||
|
||||
// TestEntryConfigGetStdoutIsTheValue runs `vaultik config get`, whose
|
||||
// stdout a script reads as the value and nothing else.
|
||||
//
|
||||
//nolint:paralleltest // replaces os.Args, os.Stdout and os.Stderr
|
||||
func TestEntryConfigGetStdoutIsTheValue(t *testing.T) {
|
||||
configPath := filepath.Join(t.TempDir(), "config.yml")
|
||||
require.NoError(t, os.WriteFile(configPath,
|
||||
[]byte("hostname: test-host\n"), configFileMode))
|
||||
|
||||
code, stdout, _ := runEntry(t,
|
||||
flagConfig, configPath, "config", "get", "hostname")
|
||||
|
||||
require.Equal(t, 0, code)
|
||||
assert.Equal(t, "test-host\n", stdout)
|
||||
}
|
||||
|
||||
// TestEntryJSONFailureIsReportedOnStderr runs each --json command that
|
||||
// writes no document when it fails, against a destination it cannot
|
||||
// use. The error must reach stderr, and stdout must stay empty.
|
||||
//
|
||||
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||
func TestEntryJSONFailureIsReportedOnStderr(t *testing.T) {
|
||||
for _, testCase := range []struct {
|
||||
name string
|
||||
args []string
|
||||
wantOnStderr string
|
||||
}{
|
||||
{
|
||||
name: "remote info",
|
||||
args: []string{cmdRemote, cmdInfo, flagJSON},
|
||||
wantOnStderr: "Failed to get remote info",
|
||||
},
|
||||
{
|
||||
name: "prune",
|
||||
args: []string{cmdPrune, flagJSON},
|
||||
wantOnStderr: "Prune failed",
|
||||
},
|
||||
{
|
||||
name: "snapshot remove",
|
||||
args: []string{cmdSnapshot, cmdRemove, someSnapshotID, flagJSON},
|
||||
wantOnStderr: "Failed to remove snapshot",
|
||||
},
|
||||
} {
|
||||
t.Run(testCase.name, func(t *testing.T) {
|
||||
configPath := writeUnusableDestinationConfig(t)
|
||||
|
||||
code, stdout, stderr := runEntry(t,
|
||||
append([]string{flagConfig, configPath}, testCase.args...)...)
|
||||
|
||||
assert.Equal(t, 1, code)
|
||||
assert.Empty(t, stdout,
|
||||
"a failed --json command has no document to write")
|
||||
assert.Contains(t, stderr, testCase.wantOnStderr,
|
||||
"the failure must be reported on stderr")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestEntrySnapshotRemoveJSONWarningIsOnStderr runs `snapshot remove
|
||||
// --json` on a snapshot in the local index, against a destination
|
||||
// directory that does not exist. The command removes the snapshot from
|
||||
// the local index and still exits 0. Its stdout must hold the document
|
||||
// alone, with the warning about the destination store on stderr: the
|
||||
// command to run again once it is reachable, and the snapshot's ID in
|
||||
// the record's snapshot_id field.
|
||||
//
|
||||
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||
func TestEntrySnapshotRemoveJSONWarningIsOnStderr(t *testing.T) {
|
||||
configPath, indexPath := writeMissingDestinationConfig(t)
|
||||
seedStaleSnapshotRecord(t, indexPath)
|
||||
|
||||
code, stdout, stderr := runEntry(t, flagConfig, configPath,
|
||||
cmdSnapshot, cmdRemove, stalePruneSnapshotID, flagJSON)
|
||||
|
||||
require.Equal(t, 0, code)
|
||||
requireExactlyOneJSONDocument(t, stdout)
|
||||
|
||||
// stderr is a pipe here, so the logger writes one JSON record a line.
|
||||
var warning map[string]any
|
||||
|
||||
for line := range strings.Lines(stderr) {
|
||||
if strings.Contains(line,
|
||||
"Could not remove snapshot metadata from remote storage") {
|
||||
require.NoError(t, json.Unmarshal([]byte(line), &warning))
|
||||
}
|
||||
}
|
||||
|
||||
require.NotNil(t, warning, "the warning must reach stderr")
|
||||
assert.Contains(t, warning[slog.MessageKey],
|
||||
"run 'vaultik snapshot remove' with the snapshot's ID again")
|
||||
assert.Equal(t, stalePruneSnapshotID, warning["snapshot_id"])
|
||||
}
|
||||
|
||||
// writeMissingDestinationConfig builds a config whose destination
|
||||
// directory does not exist. Returns the config path and the path of
|
||||
// its local index, which is not created here.
|
||||
func writeMissingDestinationConfig(t *testing.T) (string, string) {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
configPath := filepath.Join(dir, "config.yml")
|
||||
indexPath := filepath.Join(dir, "index.sqlite")
|
||||
|
||||
contents := fmt.Sprintf(hermeticConfig,
|
||||
filepath.Join(dir, "source"),
|
||||
filepath.Join(dir, "missing-store"),
|
||||
indexPath)
|
||||
|
||||
require.NoError(t,
|
||||
os.WriteFile(configPath, []byte(contents), configFileMode))
|
||||
|
||||
// The PID lock lives under xdg.DataHome, which xdg resolves at
|
||||
// package init; point it at the temp dir so the test neither
|
||||
// touches nor collides with the real one.
|
||||
t.Setenv("XDG_DATA_HOME", filepath.Join(dir, "data"))
|
||||
xdg.Reload()
|
||||
t.Cleanup(xdg.Reload)
|
||||
|
||||
return configPath, indexPath
|
||||
}
|
||||
|
||||
// writeUnusableDestinationConfig builds a config whose destination
|
||||
// directory does not exist, which fails `remote info`, and whose local
|
||||
// index is bound to another destination, which fails `prune` and
|
||||
// `snapshot remove` (a missing destination alone only makes `snapshot
|
||||
// remove` warn). Returns the config path.
|
||||
func writeUnusableDestinationConfig(t *testing.T) string {
|
||||
t.Helper()
|
||||
|
||||
configPath, indexPath := writeMissingDestinationConfig(t)
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
db, err := database.New(ctx, indexPath)
|
||||
require.NoError(t, err)
|
||||
|
||||
defer func() { require.NoError(t, db.Close()) }()
|
||||
|
||||
require.NoError(t, database.NewRepositories(db).LocalMeta.Set(ctx,
|
||||
database.LocalMetaKeyStorageURL, "file://"+t.TempDir()))
|
||||
|
||||
return configPath
|
||||
}
|
||||
|
||||
// runEntry runs Entry with args after the program name and returns its
|
||||
// exit code and what it wrote to stdout and stderr.
|
||||
//
|
||||
// Not parallel-safe: it replaces os.Args, os.Stdout and os.Stderr.
|
||||
func runEntry(t *testing.T, args ...string) (int, string, string) {
|
||||
t.Helper()
|
||||
|
||||
previousArgs := os.Args
|
||||
|
||||
t.Cleanup(func() {
|
||||
os.Args = previousArgs
|
||||
rootFlags = RootFlags{}
|
||||
})
|
||||
|
||||
os.Args = append([]string{programName}, args...)
|
||||
|
||||
var code int
|
||||
|
||||
stdout, stderr := captureProcessStdoutAndStderr(t,
|
||||
func() { code = Entry() })
|
||||
|
||||
return code, stdout, stderr
|
||||
}
|
||||
@@ -48,10 +48,6 @@ work (e.g. after a crashed backup or to reclaim storage).`,
|
||||
}, func(v *vaultik.Vaultik) error {
|
||||
return v.Prune(opts)
|
||||
}, func(err error) {
|
||||
if opts.JSON {
|
||||
return
|
||||
}
|
||||
|
||||
log.Error("Prune operation failed", "error", err)
|
||||
ReportErrorf("Prune failed: %v", err)
|
||||
})
|
||||
|
||||
@@ -45,7 +45,7 @@ This is destructive and irreversible. Requires --force.`,
|
||||
return errNukeNeedsForce
|
||||
}
|
||||
|
||||
return runVaultikApp(cmd, mutating, false, false, "Remote nuke failed",
|
||||
return runVaultikApp(cmd, mutating, false, "Remote nuke failed",
|
||||
func(v *vaultik.Vaultik) error {
|
||||
return v.NukeRemote(true)
|
||||
})
|
||||
@@ -92,10 +92,6 @@ func newRemoteInfoCommand() *cobra.Command {
|
||||
}, func(v *vaultik.Vaultik) error {
|
||||
return v.RemoteInfo(jsonOutput)
|
||||
}, func(err error) {
|
||||
if jsonOutput {
|
||||
return
|
||||
}
|
||||
|
||||
log.Error("Failed to get remote info", "error", err)
|
||||
ReportErrorf("Failed to get remote info: %v", err)
|
||||
})
|
||||
|
||||
@@ -126,7 +126,7 @@ func newSnapshotListCommand() *cobra.Command {
|
||||
Long: "Lists all snapshots with their ID, timestamp, and compressed size",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return runVaultikApp(cmd, readOnly, false, false,
|
||||
return runVaultikApp(cmd, readOnly, false,
|
||||
"Failed to list snapshots",
|
||||
func(v *vaultik.Vaultik) error {
|
||||
return v.ListSnapshots(jsonOutput)
|
||||
@@ -162,7 +162,7 @@ restrict the operation to specific snapshot names.`,
|
||||
return errPurgeCriteriaBoth
|
||||
}
|
||||
|
||||
return runVaultikApp(cmd, mutating, false, false,
|
||||
return runVaultikApp(cmd, mutating, false,
|
||||
"Failed to purge snapshots",
|
||||
func(v *vaultik.Vaultik) error {
|
||||
return v.PurgeSnapshotsWithOptions(opts)
|
||||
@@ -258,14 +258,15 @@ Use --local-only to skip the remote half (e.g. when you want to forget a
|
||||
snapshot locally without touching the destination store).
|
||||
|
||||
If the remote is unreachable, the local-database removal still completes
|
||||
and a warning is emitted; rerun 'vaultik prune' once the destination store
|
||||
is reachable to finish remote cleanup.
|
||||
and a warning is emitted; run 'vaultik snapshot remove <snapshot-id>' again
|
||||
once the destination store is reachable to remove the snapshot's metadata
|
||||
from it ('vaultik prune' does not).
|
||||
|
||||
To wipe the entire destination store and start over, use 'vaultik remote
|
||||
nuke --force' — it is the single supported entry point for that.`,
|
||||
Args: requireSnapshotIDArg,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
return runVaultikApp(cmd, mutating, opts.JSON, opts.JSON,
|
||||
return runVaultikApp(cmd, mutating, opts.JSON,
|
||||
"Failed to remove snapshot",
|
||||
func(v *vaultik.Vaultik) error {
|
||||
_, err := v.RemoveSnapshot(args[0], opts)
|
||||
|
||||
@@ -33,11 +33,14 @@ const secretKeyPrefix = "AGE-SECRET-KEY-"
|
||||
const (
|
||||
defaultBlobSizeLimit = Size(10 * 1024 * 1024 * 1024) // 10GB
|
||||
defaultChunkSize = Size(10 * 1024 * 1024) // 10MB
|
||||
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MB
|
||||
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MiB
|
||||
defaultCompressionLevel = 3
|
||||
minChunkSize = 1024 * 1024 // 1MB
|
||||
minCompressionLevel = 1
|
||||
maxCompressionLevel = 19
|
||||
// S3 accepts a multipart upload part from 5MiB to 5GiB.
|
||||
minS3PartSize = 5 * 1024 * 1024
|
||||
maxS3PartSize = 5 * 1024 * 1024 * 1024
|
||||
)
|
||||
|
||||
// Sentinel validation errors.
|
||||
@@ -55,6 +58,7 @@ var (
|
||||
"blob_size_limit must be at least the largest chunk the chunker can " +
|
||||
"emit (chunk_size times the FastCDC size spread)")
|
||||
errBadCompression = errors.New("compression_level must be between 1 and 19")
|
||||
errBadS3PartSize = errors.New("s3.part_size must be between 5MiB and 5GiB")
|
||||
errBadStorageScheme = errors.New(
|
||||
"storage_url must start with s3://, file://, or rclone://")
|
||||
errStorageNotConfigured = errors.New(
|
||||
@@ -244,6 +248,7 @@ func Load(path string) (*Config, error) {
|
||||
ChunkSize: defaultChunkSize,
|
||||
IndexPath: filepath.Join(xdg.DataHome, appName, "index.sqlite"),
|
||||
CompressionLevel: defaultCompressionLevel,
|
||||
S3: S3Config{PartSize: defaultS3PartSize},
|
||||
}
|
||||
|
||||
// Convert smartconfig data to YAML then unmarshal
|
||||
@@ -294,17 +299,13 @@ func Load(path string) (*Config, error) {
|
||||
cfg.S3.Region = "us-east-1"
|
||||
}
|
||||
|
||||
if cfg.S3.PartSize == 0 {
|
||||
cfg.S3.PartSize = defaultS3PartSize
|
||||
}
|
||||
|
||||
// Check config file permissions (warn if world or group readable)
|
||||
//nolint:gosec // G703: config path is operator-supplied by design
|
||||
info, statErr := os.Stat(path)
|
||||
if statErr == nil {
|
||||
mode := info.Mode().Perm()
|
||||
if mode&0044 != 0 { // group or world readable
|
||||
log.Warn("Config file has insecure permissions (contains S3 credentials)",
|
||||
log.Warn(cfg.readableByOthersWarning(),
|
||||
"path", path,
|
||||
"mode", fmt.Sprintf("%04o", mode),
|
||||
"recommendation", "chmod 600 "+path)
|
||||
@@ -332,6 +333,7 @@ func Load(path string) (*Config, error) {
|
||||
// (chunk_size times chunker.ChunkSizeSpread), so a single-chunk blob never
|
||||
// exceeds the configured limit
|
||||
// - Compression level must be between 1 and 19
|
||||
// - S3 part size must be between 5MiB and 5GiB, the part sizes S3 accepts
|
||||
//
|
||||
// Returns an error describing the first validation failure encountered.
|
||||
func (c *Config) Validate() error {
|
||||
@@ -376,6 +378,11 @@ func (c *Config) Validate() error {
|
||||
return errBadCompression
|
||||
}
|
||||
|
||||
if c.S3.PartSize.Int64() < minS3PartSize ||
|
||||
c.S3.PartSize.Int64() > maxS3PartSize {
|
||||
return errBadS3PartSize
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -411,6 +418,18 @@ func (c *Config) setAgeSecretKey() {
|
||||
}
|
||||
}
|
||||
|
||||
// readableByOthersWarning is the warning Load logs when others can read
|
||||
// the config file. It says "may contain" because the S3 credentials are
|
||||
// seen only after smartconfig has replaced any ${...} reference in the
|
||||
// file with its value, so a set credential need not be in the file.
|
||||
func (c *Config) readableByOthersWarning() string {
|
||||
if c.S3.AccessKeyID != "" || c.S3.SecretAccessKey != "" {
|
||||
return "Config file is readable by others and may contain S3 credentials"
|
||||
}
|
||||
|
||||
return "Config file is readable by others"
|
||||
}
|
||||
|
||||
// validateStorage validates storage configuration.
|
||||
// If StorageURL is set, it takes precedence. S3 URLs require credentials.
|
||||
// File URLs don't require any S3 configuration.
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/chunker"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -166,6 +167,7 @@ func TestValidateBlobSizeLimit(t *testing.T) {
|
||||
ChunkSize: chunkSize,
|
||||
BlobSizeLimit: blobLimit,
|
||||
CompressionLevel: 3,
|
||||
S3: S3Config{PartSize: defaultS3PartSize},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -221,6 +223,120 @@ func TestValidateBlobSizeLimit(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestValidateS3PartSize checks that s3.part_size is held to the part sizes
|
||||
// S3 accepts, 5MiB to 5GiB, by changing only the part size of the test
|
||||
// config. "5MB" in the config file is 5,000,000 bytes, below the minimum.
|
||||
func TestValidateS3PartSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
base, err := Load(os.Getenv("VAULTIK_CONFIG"))
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to load config: %v", err)
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
partSize Size
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "5MB is rejected",
|
||||
partSize: 5_000_000,
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "one byte below 5MiB is rejected",
|
||||
partSize: minS3PartSize - 1,
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "5MiB is accepted",
|
||||
partSize: minS3PartSize,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "5GiB is accepted",
|
||||
partSize: maxS3PartSize,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "one byte above 5GiB is rejected",
|
||||
partSize: maxS3PartSize + 1,
|
||||
wantErr: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := *base
|
||||
cfg.S3.PartSize = tt.partSize
|
||||
|
||||
err := cfg.Validate()
|
||||
if tt.wantErr {
|
||||
if !errors.Is(err, errBadS3PartSize) {
|
||||
t.Fatalf("Validate() error = %v, want errBadS3PartSize", err)
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
t.Fatalf("Validate() unexpected error: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoadS3PartSize checks that a config file without s3.part_size loads
|
||||
// with the 5MiB default, and that an explicit 0 fails at load like any other
|
||||
// part size S3 refuses.
|
||||
func TestLoadS3PartSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const withoutPartSize = "snapshots:\n" +
|
||||
" test:\n" +
|
||||
" paths: [/tmp/vaultik-test-source]\n" +
|
||||
"storage_url: file:///tmp/vaultik-test-storage\n"
|
||||
|
||||
writeConfig := func(t *testing.T, text string) string {
|
||||
t.Helper()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "config.yml")
|
||||
|
||||
err := os.WriteFile(path, []byte(text), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
|
||||
return path
|
||||
}
|
||||
|
||||
t.Run("absent loads as 5MiB", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg, err := Load(writeConfig(t, withoutPartSize))
|
||||
if err != nil {
|
||||
t.Fatalf("Load() unexpected error: %v", err)
|
||||
}
|
||||
|
||||
if cfg.S3.PartSize != defaultS3PartSize {
|
||||
t.Errorf("s3.part_size = %d, want %d",
|
||||
cfg.S3.PartSize, defaultS3PartSize)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("0 is rejected", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := Load(writeConfig(t, withoutPartSize+"s3:\n part_size: 0\n"))
|
||||
if !errors.Is(err, errBadS3PartSize) {
|
||||
t.Fatalf("Load() error = %v, want errBadS3PartSize", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestValidateAgeRecipients checks that recipients are parsed at config load
|
||||
// (a bad entry fails immediately, not mid-backup) and that no invalid entry —
|
||||
// least of all a pasted secret key — is echoed in the error. An empty list
|
||||
@@ -236,6 +352,7 @@ func TestValidateAgeRecipients(t *testing.T) {
|
||||
ChunkSize: Size(10 * 1024 * 1024),
|
||||
BlobSizeLimit: Size(10 * 1024 * 1024 * 1024),
|
||||
CompressionLevel: 3,
|
||||
S3: S3Config{PartSize: defaultS3PartSize},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -343,3 +460,125 @@ func TestAgeSecretKeySourceName(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// loadReadableConfig writes configYAML to a file that others can read,
|
||||
// loads it, and returns what the logger wrote to stderr meanwhile. The
|
||||
// logger writes to the os.Stderr it finds when it is initialized, so
|
||||
// os.Stderr is pointed at a file first. Not parallel-safe: os.Stderr and
|
||||
// the logger are process-global.
|
||||
func loadReadableConfig(t *testing.T, configYAML string) string {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
configPath := filepath.Join(dir, "config.yml")
|
||||
stderrPath := filepath.Join(dir, "stderr")
|
||||
|
||||
err := os.WriteFile(configPath, []byte(configYAML), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("writing config: %v", err)
|
||||
}
|
||||
|
||||
//nolint:gosec // G302: the test needs a config file others can read
|
||||
err = os.Chmod(configPath, 0o644)
|
||||
if err != nil {
|
||||
t.Fatalf("chmod config: %v", err)
|
||||
}
|
||||
|
||||
stderrFile, err := os.Create(stderrPath) //nolint:gosec // G304: test temp path
|
||||
if err != nil {
|
||||
t.Fatalf("creating stderr file: %v", err)
|
||||
}
|
||||
|
||||
previous := os.Stderr
|
||||
os.Stderr = stderrFile
|
||||
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
_, loadErr := Load(configPath)
|
||||
|
||||
os.Stderr = previous
|
||||
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
_ = stderrFile.Close()
|
||||
|
||||
if loadErr != nil {
|
||||
t.Fatalf("Load() error = %v", loadErr)
|
||||
}
|
||||
|
||||
captured, err := os.ReadFile(stderrPath) //nolint:gosec // G304: test temp path
|
||||
if err != nil {
|
||||
t.Fatalf("reading stderr file: %v", err)
|
||||
}
|
||||
|
||||
return string(captured)
|
||||
}
|
||||
|
||||
// TestLoadWarnsReadableConfigWithoutS3Credentials checks that a config
|
||||
// file others can read, holding no S3 credentials, is warned about
|
||||
// without a claim that it holds them.
|
||||
//
|
||||
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
|
||||
func TestLoadWarnsReadableConfigWithoutS3Credentials(t *testing.T) {
|
||||
stderr := loadReadableConfig(t, `
|
||||
storage_url: file:///var/backups/vaultik
|
||||
snapshots:
|
||||
home:
|
||||
paths:
|
||||
- /home
|
||||
`)
|
||||
|
||||
if !strings.Contains(stderr, "Config file is readable by others") {
|
||||
t.Errorf("expected a warning that the file is readable by others, got %q",
|
||||
stderr)
|
||||
}
|
||||
|
||||
if strings.Contains(stderr, "S3 credentials") {
|
||||
t.Errorf("warning names S3 credentials the file does not set: %q", stderr)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoadWarnsReadableConfigWithS3Credentials checks that a config file
|
||||
// others can read and that sets S3 credentials, as values or as ${ENV:...}
|
||||
// references, is warned about as one that may contain them.
|
||||
//
|
||||
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
|
||||
func TestLoadWarnsReadableConfigWithS3Credentials(t *testing.T) {
|
||||
t.Setenv("VAULTIK_TEST_ACCESS_KEY_ID", "test-access-key")
|
||||
t.Setenv("VAULTIK_TEST_SECRET_ACCESS_KEY", "test-secret-key")
|
||||
|
||||
configs := map[string]string{
|
||||
"values": `
|
||||
storage_url: s3://bucket/prefix?endpoint=s3.example.com
|
||||
s3:
|
||||
access_key_id: test-access-key
|
||||
secret_access_key: test-secret-key
|
||||
snapshots:
|
||||
home:
|
||||
paths:
|
||||
- /home
|
||||
`,
|
||||
"references": `
|
||||
storage_url: s3://bucket/prefix?endpoint=s3.example.com
|
||||
s3:
|
||||
access_key_id: ${ENV:VAULTIK_TEST_ACCESS_KEY_ID}
|
||||
secret_access_key: ${ENV:VAULTIK_TEST_SECRET_ACCESS_KEY}
|
||||
snapshots:
|
||||
home:
|
||||
paths:
|
||||
- /home
|
||||
`,
|
||||
}
|
||||
|
||||
for name, configYAML := range configs {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
stderr := loadReadableConfig(t, configYAML)
|
||||
|
||||
if !strings.Contains(stderr,
|
||||
"Config file is readable by others and may contain S3 credentials") {
|
||||
t.Errorf("expected a warning naming the S3 credentials, got %q",
|
||||
stderr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
+28
-19
@@ -33,11 +33,13 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
|
||||
}
|
||||
|
||||
query := `
|
||||
INSERT INTO files (id, path, source_path, mtime, size, mode, uid, gid, link_target)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
INSERT INTO files
|
||||
(id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(path) DO UPDATE SET
|
||||
source_path = excluded.source_path,
|
||||
mtime = excluded.mtime,
|
||||
mtime_nsec = excluded.mtime_nsec,
|
||||
size = excluded.size,
|
||||
mode = excluded.mode,
|
||||
uid = excluded.uid,
|
||||
@@ -54,16 +56,19 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
|
||||
if tx != nil {
|
||||
LogSQL("Execute", query,
|
||||
file.ID.String(), file.Path.String(), file.SourcePath.String(),
|
||||
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
|
||||
file.MTime.Unix(), file.MTime.Nanosecond(),
|
||||
file.Size, file.Mode, file.UID, file.GID,
|
||||
file.LinkTarget.String())
|
||||
err = tx.QueryRowContext(ctx, query,
|
||||
file.ID.String(), file.Path.String(), file.SourcePath.String(),
|
||||
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
|
||||
file.MTime.Unix(), file.MTime.Nanosecond(),
|
||||
file.Size, file.Mode, file.UID, file.GID,
|
||||
file.LinkTarget.String()).Scan(&idStr)
|
||||
} else {
|
||||
err = r.db.QueryRowWithLog(ctx, query,
|
||||
file.ID.String(), file.Path.String(), file.SourcePath.String(),
|
||||
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
|
||||
file.MTime.Unix(), file.MTime.Nanosecond(),
|
||||
file.Size, file.Mode, file.UID, file.GID,
|
||||
file.LinkTarget.String()).Scan(&idStr)
|
||||
}
|
||||
|
||||
@@ -84,7 +89,7 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
|
||||
// in the index.
|
||||
func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, error) {
|
||||
query := `
|
||||
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
|
||||
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
|
||||
FROM files
|
||||
WHERE path = ?
|
||||
`
|
||||
@@ -104,7 +109,7 @@ func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, err
|
||||
// GetByID retrieves a file by its UUID
|
||||
func (r *FileRepository) GetByID(ctx context.Context, id types.FileID) (*File, error) {
|
||||
query := `
|
||||
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
|
||||
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
|
||||
FROM files
|
||||
WHERE id = ?
|
||||
`
|
||||
@@ -127,7 +132,7 @@ func (r *FileRepository) GetByPathTx(
|
||||
ctx context.Context, tx *sql.Tx, path string,
|
||||
) (*File, error) {
|
||||
query := `
|
||||
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
|
||||
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
|
||||
FROM files
|
||||
WHERE path = ?
|
||||
`
|
||||
@@ -158,13 +163,14 @@ func (r *FileRepository) ListModifiedSince(
|
||||
ctx context.Context, since time.Time,
|
||||
) ([]*File, error) {
|
||||
query := `
|
||||
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
|
||||
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
|
||||
FROM files
|
||||
WHERE mtime >= ?
|
||||
WHERE (mtime, mtime_nsec) >= (?, ?)
|
||||
ORDER BY path
|
||||
`
|
||||
|
||||
rows, err := r.db.conn.QueryContext(ctx, query, since.Unix())
|
||||
rows, err := r.db.conn.QueryContext(ctx, query,
|
||||
since.Unix(), since.Nanosecond())
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("querying files: %w", err)
|
||||
}
|
||||
@@ -239,7 +245,7 @@ func (r *FileRepository) ListUnderPath(
|
||||
|
||||
// LIKE would ignore ASCII case and treat _ and % in path as wildcards.
|
||||
query := `
|
||||
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
|
||||
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
|
||||
FROM files
|
||||
WHERE path = ? OR substr(path, 1, length(?)) = ?
|
||||
ORDER BY path
|
||||
@@ -324,7 +330,7 @@ func (r *FileRepository) ListIDsWithChunksNotInUploadedBlobs(
|
||||
// ListAll returns all files in the database
|
||||
func (r *FileRepository) ListAll(ctx context.Context) ([]*File, error) {
|
||||
query := `
|
||||
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
|
||||
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
|
||||
FROM files
|
||||
ORDER BY path
|
||||
`
|
||||
@@ -365,7 +371,7 @@ func (r *FileRepository) CreateBatch(
|
||||
}
|
||||
|
||||
// Each files row binds this many SQL variables.
|
||||
const fileCols = 9
|
||||
const fileCols = 10
|
||||
|
||||
// Batch at 100 rows to be safe with SQLite's variable limit.
|
||||
const batchSize = 100
|
||||
@@ -376,7 +382,7 @@ func (r *FileRepository) CreateBatch(
|
||||
batch := files[i:end]
|
||||
|
||||
query := `INSERT INTO files
|
||||
(id, path, source_path, mtime, size, mode, uid, gid, link_target)
|
||||
(id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target)
|
||||
VALUES `
|
||||
|
||||
args := make([]any, 0, len(batch)*fileCols)
|
||||
@@ -388,11 +394,12 @@ func (r *FileRepository) CreateBatch(
|
||||
querySb325.WriteString(", ")
|
||||
}
|
||||
|
||||
querySb325.WriteString("(?, ?, ?, ?, ?, ?, ?, ?, ?)")
|
||||
querySb325.WriteString("(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
|
||||
|
||||
args = append(args,
|
||||
f.ID.String(), f.Path.String(), f.SourcePath.String(),
|
||||
f.MTime.Unix(), f.Size, f.Mode, f.UID, f.GID,
|
||||
f.MTime.Unix(), f.MTime.Nanosecond(),
|
||||
f.Size, f.Mode, f.UID, f.GID,
|
||||
f.LinkTarget.String())
|
||||
}
|
||||
|
||||
@@ -401,6 +408,7 @@ func (r *FileRepository) CreateBatch(
|
||||
query += ` ON CONFLICT(path) DO UPDATE SET
|
||||
source_path = excluded.source_path,
|
||||
mtime = excluded.mtime,
|
||||
mtime_nsec = excluded.mtime_nsec,
|
||||
size = excluded.size,
|
||||
mode = excluded.mode,
|
||||
uid = excluded.uid,
|
||||
@@ -460,7 +468,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
|
||||
var (
|
||||
file File
|
||||
idStr, pathStr, sourcePathStr string
|
||||
mtimeUnix int64
|
||||
mtimeUnix, mtimeNsec int64
|
||||
linkTarget sql.NullString
|
||||
)
|
||||
|
||||
@@ -469,6 +477,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
|
||||
&pathStr,
|
||||
&sourcePathStr,
|
||||
&mtimeUnix,
|
||||
&mtimeNsec,
|
||||
&file.Size,
|
||||
&file.Mode,
|
||||
&file.UID,
|
||||
@@ -487,7 +496,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
|
||||
file.Path = types.FilePath(pathStr)
|
||||
file.SourcePath = types.SourcePath(sourcePathStr)
|
||||
|
||||
file.MTime = time.Unix(mtimeUnix, 0).UTC()
|
||||
file.MTime = time.Unix(mtimeUnix, mtimeNsec).UTC()
|
||||
if linkTarget.Valid {
|
||||
file.LinkTarget = types.FilePath(linkTarget.String)
|
||||
}
|
||||
|
||||
@@ -252,6 +252,115 @@ func TestFileRepositorySymlink(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// An mtime after 2262 or before 1678 does not fit in int64 nanoseconds
|
||||
// since the epoch, and must still come back from the database unchanged.
|
||||
func TestFileRepositoryMTimeOutsideInt64NanosecondRange(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db, cleanup := setupTestDB(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
repo := database.NewFileRepository(db)
|
||||
|
||||
mtimes := []time.Time{
|
||||
time.Date(2300, time.January, 1, 0, 0, 0, 123456789, time.UTC),
|
||||
time.Date(1601, time.January, 1, 0, 0, 0, 987654321, time.UTC),
|
||||
}
|
||||
|
||||
for _, mtime := range mtimes {
|
||||
created := &database.File{
|
||||
Path: types.FilePath("/created-" + mtime.Format(time.RFC3339Nano)),
|
||||
MTime: mtime,
|
||||
}
|
||||
|
||||
err := repo.Create(ctx, nil, created)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create file: %v", err)
|
||||
}
|
||||
|
||||
batched := &database.File{
|
||||
ID: types.NewFileID(),
|
||||
Path: types.FilePath("/batched-" + mtime.Format(time.RFC3339Nano)),
|
||||
MTime: mtime,
|
||||
}
|
||||
|
||||
err = repo.CreateBatch(ctx, nil, []*database.File{batched})
|
||||
if err != nil {
|
||||
t.Fatalf("failed to batch create file: %v", err)
|
||||
}
|
||||
|
||||
for _, path := range []types.FilePath{created.Path, batched.Path} {
|
||||
retrieved, err := repo.GetByPath(ctx, path.String())
|
||||
if err != nil {
|
||||
t.Fatalf("failed to get file: %v", err)
|
||||
}
|
||||
|
||||
if !retrieved.MTime.Equal(mtime) {
|
||||
t.Errorf("%s: mtime got %v, want %v",
|
||||
path, retrieved.MTime, mtime)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A file already in the index and rewritten within the same second must get
|
||||
// its new nanoseconds stored, through both Create and CreateBatch.
|
||||
func TestFileRepositoryUpsertMTimeInSameSecond(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db, cleanup := setupTestDB(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
repo := database.NewFileRepository(db)
|
||||
|
||||
indexed := time.Date(2026, time.October, 7, 12, 0, 0, 100000000, time.UTC)
|
||||
rewritten := indexed.Add(800 * time.Millisecond)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
upsert func(file *database.File) error
|
||||
}{
|
||||
{"Create", func(file *database.File) error {
|
||||
return repo.Create(ctx, nil, file)
|
||||
}},
|
||||
{"CreateBatch", func(file *database.File) error {
|
||||
return repo.CreateBatch(ctx, nil, []*database.File{file})
|
||||
}},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
file := &database.File{
|
||||
ID: types.NewFileID(),
|
||||
Path: types.FilePath("/" + tt.name),
|
||||
MTime: indexed,
|
||||
}
|
||||
|
||||
err := tt.upsert(file)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: failed to create file: %v", tt.name, err)
|
||||
}
|
||||
|
||||
file.MTime = rewritten
|
||||
|
||||
err = tt.upsert(file)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: failed to update file: %v", tt.name, err)
|
||||
}
|
||||
|
||||
retrieved, err := repo.GetByPath(ctx, file.Path.String())
|
||||
if err != nil {
|
||||
t.Fatalf("%s: failed to get file: %v", tt.name, err)
|
||||
}
|
||||
|
||||
if !retrieved.MTime.Equal(rewritten) {
|
||||
t.Errorf("%s: mtime got %v, want %v",
|
||||
tt.name, retrieved.MTime, rewritten)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileRepositoryTransaction(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -606,8 +606,7 @@ func TestTimezoneHandling(t *testing.T) {
|
||||
t.Skip("timezone not available")
|
||||
}
|
||||
|
||||
// Use Truncate to remove sub-second precision since we store as Unix timestamps
|
||||
nyTime := time.Now().In(loc).Truncate(time.Second)
|
||||
nyTime := time.Now().In(loc)
|
||||
file := &File{
|
||||
Path: "/timezone-test.txt",
|
||||
MTime: nyTime,
|
||||
|
||||
@@ -6,7 +6,8 @@ CREATE TABLE IF NOT EXISTS files (
|
||||
id TEXT PRIMARY KEY, -- UUID
|
||||
path TEXT NOT NULL UNIQUE,
|
||||
source_path TEXT NOT NULL DEFAULT '', -- The source directory this file came from (for restore path stripping)
|
||||
mtime INTEGER NOT NULL,
|
||||
mtime INTEGER NOT NULL, -- whole seconds since the Unix epoch
|
||||
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
|
||||
size INTEGER NOT NULL,
|
||||
mode INTEGER NOT NULL,
|
||||
uid INTEGER NOT NULL,
|
||||
|
||||
@@ -158,19 +158,3 @@ type UploadStats struct {
|
||||
MinDurationMs int64
|
||||
MaxDurationMs int64
|
||||
}
|
||||
|
||||
// GetCountBySnapshot returns the count of uploads for a specific snapshot
|
||||
func (r *UploadRepository) GetCountBySnapshot(
|
||||
ctx context.Context, snapshotID string,
|
||||
) (int64, error) {
|
||||
query := `SELECT COUNT(*) FROM uploads WHERE snapshot_id = ?`
|
||||
|
||||
var count int64
|
||||
|
||||
err := r.conn.QueryRowContext(ctx, query, snapshotID).Scan(&count)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return count, nil
|
||||
}
|
||||
|
||||
+67
-52
@@ -10,15 +10,18 @@ import (
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
// ErrAlreadyRunning indicates another vaultik instance is running.
|
||||
var ErrAlreadyRunning = errors.New("another vaultik instance is already running")
|
||||
|
||||
// Lock represents an acquired PID lock.
|
||||
// Lock represents an acquired PID lock: an flock(2) on the PID file,
|
||||
// held while the file stays open. The kernel drops it when the process
|
||||
// exits, however it exits, so a crashed run never leaves the lock held.
|
||||
type Lock struct {
|
||||
path string
|
||||
file *os.File
|
||||
}
|
||||
|
||||
const (
|
||||
@@ -29,10 +32,9 @@ const (
|
||||
)
|
||||
|
||||
// Acquire attempts to acquire a PID lock in the specified directory.
|
||||
// If the lock file exists and the process is still running, it returns
|
||||
// ErrAlreadyRunning with details about the existing process.
|
||||
// On success, it writes the current PID to the lock file and returns
|
||||
// a Lock that must be released with Release().
|
||||
// If another process holds the lock, it returns ErrAlreadyRunning with
|
||||
// that process's PID. On success, it writes the current PID to the lock
|
||||
// file and returns a Lock that must be released with Release().
|
||||
func Acquire(lockDir string) (*Lock, error) {
|
||||
// Ensure lock directory exists
|
||||
err := os.MkdirAll(lockDir, lockDirPerm)
|
||||
@@ -42,56 +44,82 @@ func Acquire(lockDir string) (*Lock, error) {
|
||||
|
||||
lockPath := filepath.Join(lockDir, "vaultik.pid")
|
||||
|
||||
// Check for existing lock
|
||||
existingPID, err := readPIDFile(lockPath)
|
||||
if err == nil {
|
||||
// Lock file exists, check if process is running
|
||||
if isProcessRunning(existingPID) {
|
||||
return nil, fmt.Errorf("%w (PID %d)", ErrAlreadyRunning, existingPID)
|
||||
}
|
||||
// Process is not running, stale lock file - we can take over
|
||||
}
|
||||
|
||||
// Write our PID
|
||||
pid := os.Getpid()
|
||||
|
||||
err = os.WriteFile(lockPath, []byte(strconv.Itoa(pid)), pidFilePerm)
|
||||
// No O_TRUNC: the file may hold the PID of the process that has the
|
||||
// lock, which the error below reports.
|
||||
file, err := os.OpenFile( //nolint:gosec // G304: path is our own lock file
|
||||
lockPath, os.O_RDWR|os.O_CREATE, pidFilePerm)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("writing PID file: %w", err)
|
||||
return nil, fmt.Errorf("opening PID file: %w", err)
|
||||
}
|
||||
|
||||
return &Lock{path: lockPath}, nil
|
||||
err = unix.Flock(int(file.Fd()), unix.LOCK_EX|unix.LOCK_NB)
|
||||
if err != nil {
|
||||
_ = file.Close()
|
||||
|
||||
if errors.Is(err, unix.EWOULDBLOCK) {
|
||||
return nil, alreadyRunningError(lockPath)
|
||||
}
|
||||
|
||||
return nil, fmt.Errorf("locking PID file: %w", err)
|
||||
}
|
||||
|
||||
err = writePID(file)
|
||||
if err != nil {
|
||||
_ = file.Close()
|
||||
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &Lock{file: file}, nil
|
||||
}
|
||||
|
||||
// Release removes the PID lock file.
|
||||
// Release empties the PID file and closes it, which drops the lock.
|
||||
// It is safe to call Release multiple times.
|
||||
func (l *Lock) Release() error {
|
||||
if l == nil || l.path == "" {
|
||||
if l == nil || l.file == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Verify we still own the lock (our PID is in the file)
|
||||
existingPID, err := readPIDFile(l.path)
|
||||
file := l.file
|
||||
l.file = nil
|
||||
|
||||
// Do not remove the file here. A process that opened it a moment
|
||||
// earlier could then lock the removed file while another creates and
|
||||
// locks a new one, and both would run.
|
||||
truncateErr := file.Truncate(0)
|
||||
closeErr := file.Close()
|
||||
|
||||
return errors.Join(truncateErr, closeErr)
|
||||
}
|
||||
|
||||
// writePID replaces the contents of the locked PID file with the current
|
||||
// PID.
|
||||
func writePID(file *os.File) error {
|
||||
err := file.Truncate(0)
|
||||
if err != nil {
|
||||
// File already gone or unreadable - that's fine
|
||||
return nil //nolint:nilerr // unreadable lock file means nothing to release
|
||||
return fmt.Errorf("truncating PID file: %w", err)
|
||||
}
|
||||
|
||||
if existingPID != os.Getpid() {
|
||||
// Someone else wrote to our lock file - don't remove it
|
||||
return nil
|
||||
_, err = file.WriteAt([]byte(strconv.Itoa(os.Getpid())), 0)
|
||||
if err != nil {
|
||||
return fmt.Errorf("writing PID file: %w", err)
|
||||
}
|
||||
|
||||
err = os.Remove(l.path)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("removing PID file: %w", err)
|
||||
}
|
||||
|
||||
l.path = "" // Prevent double-release
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// alreadyRunningError reports that another process holds the lock,
|
||||
// naming its PID when the file holds one. The holder writes its PID just
|
||||
// after it locks, so the file can briefly be empty.
|
||||
func alreadyRunningError(lockPath string) error {
|
||||
pid, err := readPIDFile(lockPath)
|
||||
if err != nil {
|
||||
return ErrAlreadyRunning
|
||||
}
|
||||
|
||||
return fmt.Errorf("%w (PID %d)", ErrAlreadyRunning, pid)
|
||||
}
|
||||
|
||||
// readPIDFile reads and parses the PID from a lock file.
|
||||
func readPIDFile(path string) (int, error) {
|
||||
data, err := os.ReadFile(path) //nolint:gosec // G304: path is our own lock file
|
||||
@@ -106,16 +134,3 @@ func readPIDFile(path string) (int, error) {
|
||||
|
||||
return pid, nil
|
||||
}
|
||||
|
||||
// isProcessRunning checks if a process with the given PID is running.
|
||||
func isProcessRunning(pid int) bool {
|
||||
process, err := os.FindProcess(pid)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
|
||||
// On Unix, FindProcess always succeeds. We need to send signal 0 to check.
|
||||
err = process.Signal(syscall.Signal(0))
|
||||
|
||||
return err == nil
|
||||
}
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
@@ -33,9 +34,10 @@ func TestAcquireAndRelease(t *testing.T) {
|
||||
err = lock.Release()
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify PID file is gone
|
||||
_, err = os.Stat(pidPath)
|
||||
assert.True(t, os.IsNotExist(err))
|
||||
// Verify PID file is empty
|
||||
data, err = os.ReadFile(pidPath) //nolint:gosec // G304: test's own temp file
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, data)
|
||||
}
|
||||
|
||||
func TestAcquireBlocksSecondInstance(t *testing.T) {
|
||||
@@ -55,6 +57,64 @@ func TestAcquireBlocksSecondInstance(t *testing.T) {
|
||||
lock2, err := pidlock.Acquire(tmpDir)
|
||||
require.ErrorIs(t, err, pidlock.ErrAlreadyRunning)
|
||||
assert.Nil(t, lock2)
|
||||
|
||||
// Once the first lock is released, the next Acquire succeeds
|
||||
require.NoError(t, lock1.Release())
|
||||
|
||||
lock3, err := pidlock.Acquire(tmpDir)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, lock3.Release())
|
||||
}
|
||||
|
||||
// TestConcurrentAcquireAdmitsOne starts many Acquire calls at the same
|
||||
// moment, as two cron entries firing together would, and checks that
|
||||
// exactly one of them gets the lock.
|
||||
func TestConcurrentAcquireAdmitsOne(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const callers = 50
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
start := make(chan struct{})
|
||||
|
||||
var (
|
||||
mu sync.Mutex
|
||||
acquired []*pidlock.Lock
|
||||
failures []error
|
||||
wg sync.WaitGroup
|
||||
)
|
||||
|
||||
for range callers {
|
||||
wg.Go(func() {
|
||||
<-start
|
||||
|
||||
lock, err := pidlock.Acquire(tmpDir)
|
||||
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
|
||||
if err != nil {
|
||||
failures = append(failures, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
acquired = append(acquired, lock)
|
||||
})
|
||||
}
|
||||
|
||||
close(start)
|
||||
wg.Wait()
|
||||
|
||||
for _, lock := range acquired {
|
||||
require.NoError(t, lock.Release())
|
||||
}
|
||||
|
||||
assert.Len(t, acquired, 1, "exactly one caller should hold the lock")
|
||||
|
||||
for _, err := range failures {
|
||||
require.ErrorIs(t, err, pidlock.ErrAlreadyRunning)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAcquireWithStaleLock(t *testing.T) {
|
||||
|
||||
+23
-5
@@ -27,10 +27,12 @@ type Client struct {
|
||||
bucket string
|
||||
prefix string
|
||||
endpoint string
|
||||
partSize int64
|
||||
}
|
||||
|
||||
// Config contains S3 client configuration.
|
||||
// All fields are required except Prefix, which defaults to an empty string.
|
||||
// All fields are required except Prefix, which defaults to an empty string,
|
||||
// and PartSize, where zero means the SDK default of 5 MiB.
|
||||
// A non-empty Prefix is joined to every key with one "/", whether or not
|
||||
// it ends with one.
|
||||
// The Endpoint field should include the protocol (http:// or https://).
|
||||
@@ -41,6 +43,9 @@ type Config struct {
|
||||
AccessKeyID string
|
||||
SecretAccessKey string
|
||||
Region string
|
||||
// PartSize is the size in bytes of each part of a multipart upload.
|
||||
// An upload too large for S3's limit of 10,000 parts gets larger parts.
|
||||
PartSize int64
|
||||
}
|
||||
|
||||
// nopLogger is a logger that discards all output.
|
||||
@@ -90,6 +95,7 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) {
|
||||
bucket: cfg.Bucket,
|
||||
prefix: prefix,
|
||||
endpoint: cfg.Endpoint,
|
||||
partSize: cfg.PartSize,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -123,12 +129,9 @@ func (c *Client) PutObjectWithProgress(
|
||||
) error {
|
||||
fullKey := c.prefix + key
|
||||
|
||||
// uploadPartSize is 10MB for better progress granularity.
|
||||
const uploadPartSize = 10 * 1024 * 1024
|
||||
|
||||
// Create an uploader with the S3 client
|
||||
uploader := manager.NewUploader(c.s3Client, func(u *manager.Uploader) {
|
||||
u.PartSize = uploadPartSize
|
||||
u.PartSize = uploadPartSize(c.partSize, size)
|
||||
})
|
||||
|
||||
// Create a progress reader that tracks upload progress
|
||||
@@ -149,6 +152,21 @@ func (c *Client) PutObjectWithProgress(
|
||||
return err
|
||||
}
|
||||
|
||||
// uploadPartSize returns the part size for an upload of size bytes: the
|
||||
// configured part size (the SDK default when zero), raised where needed so
|
||||
// the upload fits in S3's limit of 10,000 parts. The uploader cannot raise
|
||||
// it itself, because it cannot seek the progress reader to learn its size.
|
||||
func uploadPartSize(configured, size int64) int64 {
|
||||
if configured == 0 {
|
||||
configured = manager.DefaultUploadPartSize
|
||||
}
|
||||
|
||||
maxParts := int64(manager.MaxUploadParts)
|
||||
smallestThatFits := (size + maxParts - 1) / maxParts // rounded up
|
||||
|
||||
return max(configured, smallestThatFits)
|
||||
}
|
||||
|
||||
// GetObject downloads an object from S3 with the specified key.
|
||||
// The key is automatically prefixed with the configured prefix.
|
||||
// Returns a ReadCloser containing the object data. The caller must
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
package s3
|
||||
|
||||
import "testing"
|
||||
|
||||
// TestUploadPartSize checks that an upload too large for 10,000 parts of the
|
||||
// configured size gets parts just large enough to fit in 10,000.
|
||||
func TestUploadPartSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const mib = 1024 * 1024
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
configured int64
|
||||
size int64
|
||||
want int64
|
||||
}{
|
||||
{
|
||||
name: "an upload that fits keeps the configured size",
|
||||
configured: 5 * mib,
|
||||
size: 10 * 1024 * mib,
|
||||
want: 5 * mib,
|
||||
},
|
||||
{
|
||||
name: "exactly 10,000 parts keeps the configured size",
|
||||
configured: 6 * mib,
|
||||
size: 10_000 * 6 * mib,
|
||||
want: 6 * mib,
|
||||
},
|
||||
{
|
||||
name: "one byte more than 10,000 parts adds a byte to each",
|
||||
configured: 6 * mib,
|
||||
size: 10_000*6*mib + 1,
|
||||
want: 6*mib + 1,
|
||||
},
|
||||
{
|
||||
name: "zero means the SDK default of 5MiB",
|
||||
configured: 0,
|
||||
size: 1,
|
||||
want: 5 * mib,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got := uploadPartSize(tt.configured, tt.size)
|
||||
if got != tt.want {
|
||||
t.Errorf("uploadPartSize(%d, %d) = %d, want %d",
|
||||
tt.configured, tt.size, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -28,6 +28,7 @@ func provideClient(lc fx.Lifecycle, cfg *config.Config) (*Client, error) {
|
||||
AccessKeyID: cfg.S3.AccessKeyID,
|
||||
SecretAccessKey: cfg.S3.SecretAccessKey,
|
||||
Region: cfg.S3.Region,
|
||||
PartSize: cfg.S3.PartSize.Int64(),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
||||
@@ -1221,7 +1221,7 @@ func (s *Scanner) checkFileInMemory(
|
||||
|
||||
// Check if file has changed
|
||||
if existingFile.Size != file.Size ||
|
||||
existingFile.MTime.Unix() != file.MTime.Unix() ||
|
||||
!existingFile.MTime.Equal(file.MTime) ||
|
||||
existingFile.Mode != file.Mode ||
|
||||
existingFile.UID != file.UID ||
|
||||
existingFile.GID != file.GID {
|
||||
@@ -1829,8 +1829,9 @@ func (s *Scanner) processFileStreaming(
|
||||
return nil
|
||||
}
|
||||
|
||||
// updateChunkStats counts a chunk that was not already stored. File and
|
||||
// byte counts are not touched here: the scan phase counts each file once.
|
||||
// updateChunkStats counts a chunk that was not already stored. The scan
|
||||
// result's file counts, BytesScanned and BytesSkipped are not touched
|
||||
// here: the scan phase counts each file once.
|
||||
func (s *Scanner) updateChunkStats(chunkSize int64, result *ScanResult) {
|
||||
result.ChunksCreated++
|
||||
|
||||
|
||||
@@ -105,17 +105,21 @@ func (sm *SnapshotManager) CreateSnapshot(
|
||||
return sm.CreateSnapshotWithName(ctx, hostname, "", version, gitRevision)
|
||||
}
|
||||
|
||||
// ShortHostname returns hostname up to its first dot. A snapshot ID starts
|
||||
// with this form, while the snapshots table stores the full hostname.
|
||||
func ShortHostname(hostname string) string {
|
||||
short, _, _ := strings.Cut(hostname, ".")
|
||||
|
||||
return short
|
||||
}
|
||||
|
||||
// CreateSnapshotWithName creates a new snapshot record with an optional
|
||||
// snapshot name. The snapshot ID format is: hostname_name_timestamp or
|
||||
// hostname_timestamp if name is empty.
|
||||
func (sm *SnapshotManager) CreateSnapshotWithName(
|
||||
ctx context.Context, hostname, name, version, gitRevision string,
|
||||
) (string, error) {
|
||||
// Use short hostname (strip domain if present)
|
||||
shortHostname := hostname
|
||||
if before, _, ok := strings.Cut(hostname, "."); ok {
|
||||
shortHostname = before
|
||||
}
|
||||
shortHostname := ShortHostname(hostname)
|
||||
|
||||
// Build snapshot ID with optional name
|
||||
timestamp := time.Now().UTC().Format("2006-01-02T15:04:05Z")
|
||||
|
||||
@@ -99,6 +99,7 @@ func storerFromParsedS3URL(parsed *URL, cfg *config.Config) (Storer, error) {
|
||||
AccessKeyID: cfg.S3.AccessKeyID,
|
||||
SecretAccessKey: cfg.S3.SecretAccessKey,
|
||||
Region: region,
|
||||
PartSize: cfg.S3.PartSize.Int64(),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("creating S3 client: %w", err)
|
||||
@@ -134,6 +135,7 @@ func storerFromLegacyS3Config(cfg *config.Config) (Storer, error) {
|
||||
AccessKeyID: cfg.S3.AccessKeyID,
|
||||
SecretAccessKey: cfg.S3.SecretAccessKey,
|
||||
Region: region,
|
||||
PartSize: cfg.S3.PartSize.Int64(),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("creating S3 client: %w", err)
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"github.com/johannesboyne/gofakes3"
|
||||
@@ -19,6 +22,13 @@ import (
|
||||
// s3TestBucket is the bucket created for each in-process S3 server.
|
||||
const s3TestBucket = "test-bucket"
|
||||
|
||||
// Credentials for the tests that build a storer from a config.Config. The
|
||||
// in-process S3 server accepts any.
|
||||
const (
|
||||
s3TestAccessKeyID = "key"
|
||||
s3TestSecretAccessKey = "secret"
|
||||
)
|
||||
|
||||
// newS3Storer builds an s3:// backend backed by a fresh in-process
|
||||
// S3 server. It reuses the same in-memory S3 harness (gofakes3 + s3mem
|
||||
// over httptest) that internal/s3 and the not-found regression test use,
|
||||
@@ -128,8 +138,8 @@ func TestS3URLPrefixKeyLayout(t *testing.T) {
|
||||
storer, err := storage.NewStorer(&config.Config{
|
||||
StorageURL: storageURL + "?endpoint=" + srv.URL,
|
||||
S3: config.S3Config{
|
||||
AccessKeyID: "key",
|
||||
SecretAccessKey: "secret",
|
||||
AccessKeyID: s3TestAccessKeyID,
|
||||
SecretAccessKey: s3TestSecretAccessKey,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
@@ -171,6 +181,90 @@ func TestS3URLPrefixKeyLayout(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestS3UploadUsesConfiguredPartSize checks that s3.part_size reaches the
|
||||
// multipart uploader, through storage_url and through the s3.* fields. An
|
||||
// object three parts long must arrive as three parts; at the SDK's default
|
||||
// of 5 MiB it would arrive as four.
|
||||
func TestS3UploadUsesConfiguredPartSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
partSize = 6 * 1024 * 1024
|
||||
wantParts = 3
|
||||
)
|
||||
|
||||
backend := s3mem.New()
|
||||
|
||||
err := backend.CreateBucket(s3TestBucket)
|
||||
if err != nil {
|
||||
t.Fatalf("create bucket: %v", err)
|
||||
}
|
||||
|
||||
// Every part of a multipart upload is one request with a partNumber.
|
||||
var parts atomic.Int32
|
||||
|
||||
fake := gofakes3.New(backend).Server()
|
||||
srv := httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Query().Has("partNumber") {
|
||||
parts.Add(1)
|
||||
}
|
||||
|
||||
fake.ServeHTTP(w, r)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
cfg *config.Config
|
||||
}{
|
||||
{
|
||||
name: "storage_url",
|
||||
cfg: &config.Config{
|
||||
StorageURL: "s3://" + s3TestBucket + "?endpoint=" + srv.URL,
|
||||
S3: config.S3Config{
|
||||
AccessKeyID: s3TestAccessKeyID,
|
||||
SecretAccessKey: s3TestSecretAccessKey,
|
||||
PartSize: partSize,
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "s3.endpoint",
|
||||
cfg: &config.Config{
|
||||
S3: config.S3Config{
|
||||
Endpoint: srv.URL,
|
||||
Bucket: s3TestBucket,
|
||||
AccessKeyID: s3TestAccessKeyID,
|
||||
SecretAccessKey: s3TestSecretAccessKey,
|
||||
PartSize: partSize,
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
parts.Store(0)
|
||||
|
||||
storer, err := storage.NewStorer(tc.cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: NewStorer: %v", tc.name, err)
|
||||
}
|
||||
|
||||
data := bytes.NewReader(make([]byte, wantParts*partSize))
|
||||
|
||||
err = storer.PutWithProgress(
|
||||
context.Background(), "blob", data, data.Size(), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: PutWithProgress: %v", tc.name, err)
|
||||
}
|
||||
|
||||
if got := parts.Load(); got != wantParts {
|
||||
t.Errorf("%s: uploaded in %d parts, want %d", tc.name, got, wantParts)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// listStreamKeys returns the keys ListStream yields under a prefix, and
|
||||
// fails the test on a listing error.
|
||||
func listStreamKeys(t *testing.T, s storage.Storer, prefix string) []string {
|
||||
|
||||
+23
-18
@@ -9,6 +9,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/dustin/go-humanize"
|
||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||
"sneak.berlin/go/vaultik/internal/types"
|
||||
)
|
||||
|
||||
@@ -46,12 +47,9 @@ const (
|
||||
year = 365 * day
|
||||
)
|
||||
|
||||
// Snapshot IDs split on "_" into hostname, optional name parts, and a
|
||||
// trailing timestamp.
|
||||
const (
|
||||
minSnapshotIDParts = 2
|
||||
minSnapshotIDNameParts = 3
|
||||
)
|
||||
// A snapshot ID split on "_" has at least a hostname and a trailing
|
||||
// timestamp.
|
||||
const minSnapshotIDParts = 2
|
||||
|
||||
// SnapshotInfo contains information about a snapshot.
|
||||
//
|
||||
@@ -121,20 +119,27 @@ func parseSnapshotTimestamp(snapshotID string) (time.Time, error) {
|
||||
return timestamp.UTC(), nil
|
||||
}
|
||||
|
||||
// parseSnapshotName extracts the snapshot name from a snapshot ID.
|
||||
// Format: hostname_snapshotname_timestamp — the middle part(s) between hostname
|
||||
// and the RFC3339 timestamp are the snapshot name (may contain underscores).
|
||||
// Returns the snapshot name, or empty string if the ID is malformed.
|
||||
func parseSnapshotName(snapshotID string) string {
|
||||
parts := strings.Split(snapshotID, "_")
|
||||
if len(parts) < minSnapshotIDNameParts {
|
||||
// Format: hostname_timestamp — no snapshot name
|
||||
// parseSnapshotName extracts the snapshot name from a snapshot ID of the
|
||||
// form hostname_name_timestamp, given the hostname stored with that
|
||||
// snapshot. The hostname and the name may both contain underscores, so the
|
||||
// name is what is left after removing the short hostname and its "_" from
|
||||
// the front and the last "_" and the timestamp from the end. Returns "" for
|
||||
// an ID with no name (hostname_timestamp), and for an ID that does not start
|
||||
// with that hostname, which CreateSnapshotWithName never writes.
|
||||
func parseSnapshotName(snapshotID, hostname string) string {
|
||||
prefix := snapshot.ShortHostname(hostname) + "_"
|
||||
|
||||
rest, ok := strings.CutPrefix(snapshotID, prefix)
|
||||
if !ok {
|
||||
return ""
|
||||
}
|
||||
// Format: hostname_name_timestamp — middle parts are the name.
|
||||
// The last part is the RFC3339 timestamp, the first part is the hostname,
|
||||
// everything in between is the snapshot name (which may itself contain underscores).
|
||||
return strings.Join(parts[1:len(parts)-1], "_")
|
||||
|
||||
end := strings.LastIndex(rest, "_")
|
||||
if end < 0 {
|
||||
return ""
|
||||
}
|
||||
|
||||
return rest[:end]
|
||||
}
|
||||
|
||||
// parseDuration parses a duration string with support for human-friendly units:
|
||||
|
||||
@@ -11,33 +11,55 @@ func TestParseSnapshotName(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
snapshotID string
|
||||
hostname string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "standard format with name",
|
||||
snapshotID: "myhost_home_2026-01-12T14:41:15Z",
|
||||
hostname: "myhost",
|
||||
want: "home",
|
||||
},
|
||||
{
|
||||
name: "standard format with different name",
|
||||
snapshotID: "server1_system_2026-02-15T09:30:00Z",
|
||||
hostname: "server1",
|
||||
want: "system",
|
||||
},
|
||||
{
|
||||
name: "name with underscores",
|
||||
snapshotID: "myhost_my_special_backup_2026-03-01T00:00:00Z",
|
||||
hostname: "myhost",
|
||||
want: "my_special_backup",
|
||||
},
|
||||
{
|
||||
name: "hostname with underscores",
|
||||
snapshotID: "my_host_docs_2026-03-01T00:00:00Z",
|
||||
hostname: "my_host",
|
||||
want: "docs",
|
||||
},
|
||||
{
|
||||
name: "stored hostname with domain",
|
||||
snapshotID: "my_host_mail_2026-03-01T00:00:00Z",
|
||||
hostname: "my_host.example.com",
|
||||
want: "mail",
|
||||
},
|
||||
{
|
||||
name: "no name",
|
||||
snapshotID: "my_host_2026-03-01T00:00:00Z",
|
||||
hostname: "my_host",
|
||||
want: "",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got := parseSnapshotName(tt.snapshotID)
|
||||
got := parseSnapshotName(tt.snapshotID, tt.hostname)
|
||||
if got != tt.want {
|
||||
t.Errorf("parseSnapshotName(%q) = %q, want %q",
|
||||
tt.snapshotID, got, tt.want)
|
||||
t.Errorf("parseSnapshotName(%q, %q) = %q, want %q",
|
||||
tt.snapshotID, tt.hostname, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
+109
-25
@@ -183,6 +183,11 @@ type SnapshotMetadataInfo struct {
|
||||
TotalSize int64 `json:"total_size"`
|
||||
BlobCount int `json:"blob_count"`
|
||||
BlobsSize int64 `json:"blobs_size"`
|
||||
|
||||
// Set when the listing holds this snapshot's manifest.json.zst. A
|
||||
// backup interrupted before its manifest upload leaves a directory
|
||||
// without one, which prune does not treat as a snapshot.
|
||||
hasManifest bool
|
||||
}
|
||||
|
||||
// RemoteInfoResult contains all remote storage information
|
||||
@@ -206,9 +211,20 @@ type RemoteInfoResult struct {
|
||||
ReferencedBlobCount int `json:"referenced_blob_count"`
|
||||
ReferencedBlobSize int64 `json:"referenced_blob_size"`
|
||||
|
||||
// Orphaned blobs
|
||||
OrphanedBlobCount int `json:"orphaned_blob_count"`
|
||||
OrphanedBlobSize int64 `json:"orphaned_blob_size"`
|
||||
// Orphaned blobs. Both stay nil (null in the JSON) when a manifest
|
||||
// was listed but not read, since that snapshot's blobs would be
|
||||
// counted as orphaned.
|
||||
OrphanedBlobCount *int `json:"orphaned_blob_count"`
|
||||
OrphanedBlobSize *int64 `json:"orphaned_blob_size"`
|
||||
|
||||
// Remote key of each snapshot whose manifest could not be read
|
||||
UnreadableManifests []string `json:"unreadable_manifests,omitempty"`
|
||||
|
||||
// Number of manifests not read because the name above them under
|
||||
// metadata/ is not a remote key. The names themselves are not
|
||||
// reported: they come from the destination store and may hold
|
||||
// control characters.
|
||||
SkippedManifestCount int `json:"skipped_manifest_count,omitempty"`
|
||||
}
|
||||
|
||||
// RemoteInfo displays information about remote storage
|
||||
@@ -234,16 +250,28 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
|
||||
v.stdoutf("Scanning snapshot metadata...\n")
|
||||
}
|
||||
|
||||
snapshotMetadata, snapshotIDs, err := v.collectSnapshotMetadata()
|
||||
snapshotMetadata, snapshotIDs, skippedManifestCount, err := v.collectSnapshotMetadata()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
result.SkippedManifestCount = skippedManifestCount
|
||||
|
||||
if showText {
|
||||
v.stdoutf("Downloading %d manifest(s)...\n", len(snapshotIDs))
|
||||
manifestCount := 0
|
||||
|
||||
for _, info := range snapshotMetadata {
|
||||
if info.hasManifest {
|
||||
manifestCount++
|
||||
}
|
||||
}
|
||||
|
||||
v.stdoutf("Downloading %d manifest(s)...\n", manifestCount)
|
||||
}
|
||||
|
||||
referencedBlobs := v.collectReferencedBlobsFromManifests(snapshotIDs, snapshotMetadata)
|
||||
referencedBlobs, unreadableManifests := v.collectReferencedBlobsFromManifests(
|
||||
snapshotIDs, snapshotMetadata)
|
||||
result.UnreadableManifests = unreadableManifests
|
||||
|
||||
v.populateRemoteInfoResult(result, snapshotMetadata, snapshotIDs, referencedBlobs)
|
||||
|
||||
@@ -256,7 +284,7 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
|
||||
"snapshots", result.TotalMetadataCount,
|
||||
"total_blobs", result.TotalBlobCount,
|
||||
"referenced_blobs", result.ReferencedBlobCount,
|
||||
"orphaned_blobs", result.OrphanedBlobCount)
|
||||
"unreadable_manifests", len(result.UnreadableManifests))
|
||||
|
||||
if jsonOutput {
|
||||
enc := json.NewEncoder(v.Stdout)
|
||||
@@ -273,16 +301,18 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
|
||||
}
|
||||
|
||||
// collectSnapshotMetadata scans remote metadata and returns
|
||||
// per-snapshot info and sorted IDs.
|
||||
// per-snapshot info, sorted IDs and the number of manifests it skipped
|
||||
// because the name above them is not a remote key.
|
||||
func (v *Vaultik) collectSnapshotMetadata() (
|
||||
map[string]*SnapshotMetadataInfo, []string, error,
|
||||
map[string]*SnapshotMetadataInfo, []string, int, error,
|
||||
) {
|
||||
snapshotMetadata := make(map[string]*SnapshotMetadataInfo)
|
||||
skippedManifestCount := 0
|
||||
|
||||
metadataCh := v.Storage.ListStream(v.ctx, "metadata/")
|
||||
for obj := range metadataCh {
|
||||
if obj.Err != nil {
|
||||
return nil, nil, fmt.Errorf("listing metadata: %w", obj.Err)
|
||||
return nil, nil, 0, fmt.Errorf("listing metadata: %w", obj.Err)
|
||||
}
|
||||
|
||||
parts := strings.Split(obj.Key, "/")
|
||||
@@ -291,6 +321,22 @@ func (v *Vaultik) collectSnapshotMetadata() (
|
||||
}
|
||||
|
||||
snapshotID := parts[1]
|
||||
filename := parts[2]
|
||||
isManifest := filename == "manifest.json.zst"
|
||||
|
||||
// The name comes from the destination store, which is not
|
||||
// trusted, and is printed in the report. Accept it only in the
|
||||
// form of a remote key.
|
||||
if !isBlobHash(snapshotID) {
|
||||
log.Warn("Skipping non-conforming key under metadata/",
|
||||
"key", obj.Key)
|
||||
|
||||
if isManifest {
|
||||
skippedManifestCount++
|
||||
}
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if _, exists := snapshotMetadata[snapshotID]; !exists {
|
||||
snapshotMetadata[snapshotID] = &SnapshotMetadataInfo{SnapshotID: snapshotID}
|
||||
@@ -298,7 +344,10 @@ func (v *Vaultik) collectSnapshotMetadata() (
|
||||
|
||||
info := snapshotMetadata[snapshotID]
|
||||
|
||||
filename := parts[2]
|
||||
if isManifest {
|
||||
info.hasManifest = true
|
||||
}
|
||||
|
||||
if strings.HasPrefix(filename, "manifest") {
|
||||
info.ManifestSize = obj.Size
|
||||
} else if strings.HasPrefix(filename, "db") {
|
||||
@@ -315,17 +364,25 @@ func (v *Vaultik) collectSnapshotMetadata() (
|
||||
|
||||
sort.Strings(snapshotIDs)
|
||||
|
||||
return snapshotMetadata, snapshotIDs, nil
|
||||
return snapshotMetadata, snapshotIDs, skippedManifestCount, nil
|
||||
}
|
||||
|
||||
// collectReferencedBlobsFromManifests downloads manifests and returns
|
||||
// referenced blob hashes with sizes.
|
||||
// collectReferencedBlobsFromManifests downloads the listed manifests
|
||||
// and returns referenced blob hashes with sizes, and the remote keys
|
||||
// of the manifests it could not read.
|
||||
func (v *Vaultik) collectReferencedBlobsFromManifests(
|
||||
snapshotIDs []string, snapshotMetadata map[string]*SnapshotMetadataInfo,
|
||||
) map[string]int64 {
|
||||
) (map[string]int64, []string) {
|
||||
referencedBlobs := make(map[string]int64)
|
||||
|
||||
var unreadable []string
|
||||
|
||||
for _, snapshotID := range snapshotIDs {
|
||||
info := snapshotMetadata[snapshotID]
|
||||
if !info.hasManifest {
|
||||
continue
|
||||
}
|
||||
|
||||
// snapshotIDs here are remote keys, taken straight from the
|
||||
// metadata/ listing. downloadManifestByKey is the single reader
|
||||
// for remote manifests; see its doc comment.
|
||||
@@ -333,10 +390,11 @@ func (v *Vaultik) collectReferencedBlobsFromManifests(
|
||||
if err != nil {
|
||||
log.Warn("Failed to read manifest", "snapshot", snapshotID, "error", err)
|
||||
|
||||
unreadable = append(unreadable, snapshotID)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
info := snapshotMetadata[snapshotID]
|
||||
info.BlobCount = manifest.BlobCount
|
||||
|
||||
var blobsSize int64
|
||||
@@ -349,7 +407,7 @@ func (v *Vaultik) collectReferencedBlobsFromManifests(
|
||||
info.BlobsSize = blobsSize
|
||||
}
|
||||
|
||||
return referencedBlobs
|
||||
return referencedBlobs, unreadable
|
||||
}
|
||||
|
||||
// populateRemoteInfoResult fills in the result's snapshot and
|
||||
@@ -378,8 +436,9 @@ func (v *Vaultik) populateRemoteInfoResult(
|
||||
}
|
||||
|
||||
// scanRemoteBlobStorage lists all blobs on remote and computes orphan
|
||||
// stats. showText is true only when the human report is being printed
|
||||
// (not --json, not --quiet), gating the progress line.
|
||||
// stats when every listed manifest was read. showText is true only
|
||||
// when the human report is being printed (not --json, not --quiet),
|
||||
// gating the progress line.
|
||||
func (v *Vaultik) scanRemoteBlobStorage(
|
||||
result *RemoteInfoResult, referencedBlobs map[string]int64, showText bool,
|
||||
) error {
|
||||
@@ -406,13 +465,28 @@ func (v *Vaultik) scanRemoteBlobStorage(
|
||||
result.TotalBlobSize += obj.Size
|
||||
}
|
||||
|
||||
// A blob named only by a manifest that could not be read, or by one
|
||||
// under a skipped name, would be counted as orphaned, so the orphan
|
||||
// figures stay unknown.
|
||||
if len(result.UnreadableManifests) > 0 || result.SkippedManifestCount > 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
var (
|
||||
orphanedCount int
|
||||
orphanedSize int64
|
||||
)
|
||||
|
||||
for hash, size := range allBlobs {
|
||||
if _, referenced := referencedBlobs[hash]; !referenced {
|
||||
result.OrphanedBlobCount++
|
||||
result.OrphanedBlobSize += size
|
||||
orphanedCount++
|
||||
orphanedSize += size
|
||||
}
|
||||
}
|
||||
|
||||
result.OrphanedBlobCount = &orphanedCount
|
||||
result.OrphanedBlobSize = &orphanedSize
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -465,11 +539,21 @@ func (v *Vaultik) printRemoteInfoTable(result *RemoteInfoResult) {
|
||||
v.stdoutf("Referenced by snapshots: %s (%s)\n",
|
||||
humanize.Comma(int64(result.ReferencedBlobCount)),
|
||||
ubytes(result.ReferencedBlobSize))
|
||||
v.stdoutf("Orphaned (unreferenced): %s (%s)\n",
|
||||
humanize.Comma(int64(result.OrphanedBlobCount)),
|
||||
ubytes(result.OrphanedBlobSize))
|
||||
|
||||
if result.OrphanedBlobCount > 0 {
|
||||
if result.OrphanedBlobCount == nil {
|
||||
v.stdoutf("Orphaned (unreferenced): unknown "+
|
||||
"(%d manifest(s) could not be read, "+
|
||||
"%d manifest(s) under a non-conforming name skipped)\n",
|
||||
len(result.UnreadableManifests), result.SkippedManifestCount)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
v.stdoutf("Orphaned (unreferenced): %s (%s)\n",
|
||||
humanize.Comma(int64(*result.OrphanedBlobCount)),
|
||||
ubytes(*result.OrphanedBlobSize))
|
||||
|
||||
if *result.OrphanedBlobCount > 0 {
|
||||
v.stdoutf("\nRun 'vaultik prune' to remove orphaned blobs.\n")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/spf13/afero"
|
||||
@@ -153,3 +154,23 @@ func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, snapshots, 1, "prune must delete no local snapshot record")
|
||||
}
|
||||
|
||||
// TestPurgeSaysListingFailedOnceWhenDestinationMissing checks that
|
||||
// snapshot purge fails on a destination it cannot list, with an error
|
||||
// that says "listing remote snapshots" once.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestPurgeSaysListingFailedOnceWhenDestinationMissing(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
ctx := context.Background()
|
||||
v, _, _ := backUpThenUnplug(ctx, t)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
Force: true,
|
||||
})
|
||||
require.ErrorIs(t, err, fs.ErrNotExist)
|
||||
assert.Equal(t, 1, strings.Count(err.Error(), "listing remote snapshots"),
|
||||
err.Error())
|
||||
}
|
||||
|
||||
@@ -42,7 +42,7 @@ func setupConsistencyTest(
|
||||
completedAt := startedAt.Add(5 * time.Minute)
|
||||
snap := &database.Snapshot{
|
||||
ID: types.SnapshotID(id),
|
||||
Hostname: testHostname,
|
||||
Hostname: snapHostname,
|
||||
VaultikVersion: testLabel,
|
||||
StartedAt: startedAt,
|
||||
CompletedAt: &completedAt,
|
||||
|
||||
@@ -17,8 +17,10 @@ import (
|
||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||
)
|
||||
|
||||
// Snapshot IDs reused across the purge tests.
|
||||
// Snapshot IDs reused across the purge tests, and the hostname they were
|
||||
// taken on.
|
||||
const (
|
||||
snapHostname = "testhost"
|
||||
snapSystemT0 = "testhost_system_2026-01-01T00:00:00Z"
|
||||
snapHomeT0 = "testhost_home_2026-01-01T00:00:00Z"
|
||||
snapHomeT1 = "testhost_home_2026-01-01T01:00:00Z"
|
||||
@@ -26,9 +28,12 @@ const (
|
||||
)
|
||||
|
||||
// setupPurgeTest creates a Vaultik instance with an in-memory database and mock
|
||||
// storage pre-populated with the given snapshot IDs. Each snapshot is marked as
|
||||
// completed. Remote metadata stubs are created so syncWithRemote keeps them.
|
||||
func setupPurgeTest(t *testing.T, snapshotIDs []string) *vaultik.Vaultik {
|
||||
// storage pre-populated with the given snapshot IDs, all taken on hostname.
|
||||
// Each snapshot is marked as completed. Remote metadata stubs are created so
|
||||
// syncWithRemote keeps them.
|
||||
func setupPurgeTest(
|
||||
t *testing.T, hostname string, snapshotIDs []string,
|
||||
) *vaultik.Vaultik {
|
||||
t.Helper()
|
||||
|
||||
ctx := context.Background()
|
||||
@@ -51,7 +56,7 @@ func setupPurgeTest(t *testing.T, snapshotIDs []string) *vaultik.Vaultik {
|
||||
completedAt := startedAt.Add(5 * time.Minute)
|
||||
snap := &database.Snapshot{
|
||||
ID: types.SnapshotID(id),
|
||||
Hostname: "testhost",
|
||||
Hostname: types.Hostname(hostname),
|
||||
VaultikVersion: testLabel,
|
||||
StartedAt: startedAt,
|
||||
CompletedAt: &completedAt,
|
||||
@@ -120,7 +125,7 @@ func TestPurgeKeepLatest_PerName(t *testing.T) {
|
||||
"testhost_system_2026-01-01T04:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -148,7 +153,7 @@ func TestPurgeKeepLatest_SingleName(t *testing.T) {
|
||||
"testhost_home_2026-01-01T02:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -176,7 +181,7 @@ func TestPurgeKeepLatest_WithNameFilter(t *testing.T) {
|
||||
"testhost_home_2026-01-01T04:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -198,7 +203,7 @@ func TestPurgeKeepLatest_NoSnapshots(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
v := setupPurgeTest(t, nil)
|
||||
v := setupPurgeTest(t, snapHostname, nil)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -216,7 +221,7 @@ func TestPurgeKeepLatest_NameFilterNoMatch(t *testing.T) {
|
||||
"testhost_system_2026-01-01T01:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -243,7 +248,7 @@ func TestPurgeOlderThan_WithNameFilter(t *testing.T) {
|
||||
snapHomeT0,
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
|
||||
// Purge only "home" snapshots older than 365 days
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
@@ -277,7 +282,7 @@ func TestPurgeKeepLatest_ThreeNames(t *testing.T) {
|
||||
"testhost_home_2026-01-01T06:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -291,3 +296,28 @@ func TestPurgeKeepLatest_ThreeNames(t *testing.T) {
|
||||
assert.Contains(t, remaining, "testhost_system_2026-01-01T04:00:00Z")
|
||||
assert.Contains(t, remaining, "testhost_media_2026-01-01T05:00:00Z")
|
||||
}
|
||||
|
||||
// A hostname may contain underscores, so the snapshot name cannot be found
|
||||
// by splitting the ID at them. A purge by name must still select "docs".
|
||||
func TestPurgeKeepLatest_HostnameWithUnderscore(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
system = "my_host_system_2026-01-01T00:00:00Z"
|
||||
docsT1 = "my_host_docs_2026-01-01T01:00:00Z"
|
||||
docsT2 = "my_host_docs_2026-01-01T02:00:00Z"
|
||||
)
|
||||
|
||||
v := setupPurgeTest(t, "my_host", []string{system, docsT1, docsT2})
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
Force: true,
|
||||
Names: []string{"docs"},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.ElementsMatch(t, []string{system, docsT2},
|
||||
listRemainingSnapshots(t, v))
|
||||
}
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
package vaultik_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||
)
|
||||
|
||||
// testBlobHashB is a blob that the manifest written by addRemote does
|
||||
// not reference.
|
||||
const testBlobHashB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" +
|
||||
"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
|
||||
|
||||
// TestRemoteInfo_UnreadableManifestLeavesOrphansUnknown checks that a
|
||||
// manifest remote info cannot read makes the orphan figures unknown. A
|
||||
// blob referenced only by that snapshot would otherwise be counted as
|
||||
// orphaned, and the report would advise running prune.
|
||||
func TestRemoteInfo_UnreadableManifestLeavesOrphansUnknown(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newListEnv(t)
|
||||
|
||||
// The readable manifest references blob A only.
|
||||
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
|
||||
addBlob(t, env.store.testStorer, testBlobHashA)
|
||||
addBlob(t, env.store.testStorer, testBlobHashB)
|
||||
|
||||
// With every manifest readable, blob B is orphaned.
|
||||
require.NoError(t, env.v.RemoteInfo(true))
|
||||
|
||||
var doc map[string]any
|
||||
|
||||
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
|
||||
assert.InDelta(t, 1, doc["orphaned_blob_count"], 0)
|
||||
|
||||
// A second snapshot whose manifest cannot be decoded. Blob B may be
|
||||
// one of its blobs.
|
||||
unreadableKey := snapshot.RemoteSnapshotKey(listLocalID)
|
||||
require.NoError(t, env.store.Put(context.Background(),
|
||||
"metadata/"+unreadableKey+"/manifest.json.zst",
|
||||
bytes.NewReader([]byte("not a valid manifest"))))
|
||||
|
||||
env.stdout.Reset()
|
||||
require.NoError(t, env.v.RemoteInfo(false))
|
||||
|
||||
text := env.stdout.String()
|
||||
assert.Contains(t, text, "Orphaned (unreferenced): unknown "+
|
||||
"(1 manifest(s) could not be read, "+
|
||||
"0 manifest(s) under a non-conforming name skipped)")
|
||||
assert.NotContains(t, text, "vaultik prune")
|
||||
|
||||
env.stdout.Reset()
|
||||
require.NoError(t, env.v.RemoteInfo(true))
|
||||
|
||||
doc = nil
|
||||
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
|
||||
assert.Contains(t, doc, "orphaned_blob_count")
|
||||
assert.Nil(t, doc["orphaned_blob_count"])
|
||||
assert.Contains(t, doc, "orphaned_blob_size")
|
||||
assert.Nil(t, doc["orphaned_blob_size"])
|
||||
assert.Equal(t, []any{unreadableKey}, doc["unreadable_manifests"])
|
||||
}
|
||||
|
||||
// TestRemoteInfo_SkipsNonConformingMetadataName checks that a directory
|
||||
// under metadata/ whose name is not a remote key is left out of the
|
||||
// report, and that the orphan figures are unknown when it holds a
|
||||
// manifest. The name comes from the destination store; printed raw, its
|
||||
// control characters would reach the terminal. Its manifest is not
|
||||
// read, so a blob only it references would otherwise be counted as
|
||||
// orphaned.
|
||||
func TestRemoteInfo_SkipsNonConformingMetadataName(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newListEnv(t)
|
||||
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
|
||||
addBlob(t, env.store.testStorer, testBlobHashA)
|
||||
addBlob(t, env.store.testStorer, testBlobHashB)
|
||||
require.NoError(t, env.store.Put(context.Background(),
|
||||
"metadata/\x1b[31mred/manifest.json.zst",
|
||||
bytes.NewReader([]byte("not a valid manifest"))))
|
||||
|
||||
require.NoError(t, env.v.RemoteInfo(false))
|
||||
|
||||
text := env.stdout.String()
|
||||
assert.NotContains(t, text, "\x1b")
|
||||
assert.NotContains(t, text, "31mred")
|
||||
assert.Contains(t, text, "Total (1 snapshots)")
|
||||
assert.Contains(t, text, "Orphaned (unreferenced): unknown "+
|
||||
"(0 manifest(s) could not be read, "+
|
||||
"1 manifest(s) under a non-conforming name skipped)")
|
||||
assert.NotContains(t, text, "vaultik prune")
|
||||
|
||||
env.stdout.Reset()
|
||||
require.NoError(t, env.v.RemoteInfo(true))
|
||||
|
||||
out := env.stdout.String()
|
||||
assert.NotContains(t, out, "31mred")
|
||||
|
||||
var doc map[string]any
|
||||
|
||||
require.NoError(t, json.Unmarshal([]byte(out), &doc))
|
||||
assert.Contains(t, doc, "orphaned_blob_count")
|
||||
assert.Nil(t, doc["orphaned_blob_count"])
|
||||
assert.Contains(t, doc, "orphaned_blob_size")
|
||||
assert.Nil(t, doc["orphaned_blob_size"])
|
||||
assert.InDelta(t, 1, doc["skipped_manifest_count"], 0)
|
||||
assert.NotContains(t, doc, "unreadable_manifests")
|
||||
}
|
||||
|
||||
// TestRemoteInfo_DirectoryWithoutManifestLeavesOrphansKnown checks that
|
||||
// a directory under metadata/ holding no manifest.json.zst, such as one
|
||||
// left by a backup interrupted before its manifest upload, leaves the
|
||||
// orphan figures known. prune does not treat such a directory as a
|
||||
// snapshot and deletes the blobs the report lists as orphaned.
|
||||
func TestRemoteInfo_DirectoryWithoutManifestLeavesOrphansKnown(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newListEnv(t)
|
||||
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
|
||||
addBlob(t, env.store.testStorer, testBlobHashA)
|
||||
addBlob(t, env.store.testStorer, testBlobHashB)
|
||||
|
||||
// One directory under a remote key and one under a non-conforming
|
||||
// name, each holding only a database.
|
||||
names := []string{snapshot.RemoteSnapshotKey(listLocalID), "\x1b[31mred"}
|
||||
for _, name := range names {
|
||||
require.NoError(t, env.store.Put(context.Background(),
|
||||
"metadata/"+name+"/db.zst.age",
|
||||
bytes.NewReader([]byte("not a valid database"))))
|
||||
}
|
||||
|
||||
require.NoError(t, env.v.RemoteInfo(false))
|
||||
|
||||
text := env.stdout.String()
|
||||
assert.NotContains(t, text, "\x1b")
|
||||
assert.Contains(t, text, "Downloading 1 manifest(s)...")
|
||||
assert.Contains(t, text, "Orphaned (unreferenced): 1 (")
|
||||
assert.Contains(t, text, "Run 'vaultik prune' to remove orphaned blobs.")
|
||||
|
||||
env.stdout.Reset()
|
||||
require.NoError(t, env.v.RemoteInfo(true))
|
||||
|
||||
var doc map[string]any
|
||||
|
||||
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
|
||||
assert.InDelta(t, 1, doc["orphaned_blob_count"], 0)
|
||||
assert.NotContains(t, doc, "unreadable_manifests")
|
||||
assert.NotContains(t, doc, "skipped_manifest_count")
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
package vaultik_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/afero"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/database"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||
)
|
||||
|
||||
// A file rewritten with its size unchanged and a new mtime in the same
|
||||
// second as the mtime the index holds must still be backed up. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/226.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupOfSameSecondRewriteRestoresNewContent(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
dataDir := filepath.Join(tempDir, "src")
|
||||
storeDir := filepath.Join(tempDir, "remote")
|
||||
restoreDir := filepath.Join(tempDir, "restored")
|
||||
dbPath := filepath.Join(tempDir, "index.sqlite")
|
||||
rewrittenPath := filepath.Join(dataDir, "small.txt")
|
||||
|
||||
ctx := context.Background()
|
||||
files := writeFaultSourceTree(t, fs, dataDir)
|
||||
cfg := changedFileConfig(dataDir, dbPath)
|
||||
|
||||
firstMTime := time.Date(2026, time.January, 2, 3, 4, 5, 0, time.UTC).
|
||||
Add(100 * time.Millisecond)
|
||||
secondMTime := firstMTime.Add(800 * time.Millisecond)
|
||||
|
||||
require.NoError(t, fs.Chtimes(rewrittenPath, firstMTime, firstMTime))
|
||||
|
||||
store, err := storage.NewFileStorer(storeDir)
|
||||
require.NoError(t, err)
|
||||
|
||||
db, err := database.New(ctx, dbPath)
|
||||
require.NoError(t, err)
|
||||
|
||||
repos := database.NewRepositories(db)
|
||||
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
|
||||
|
||||
require.NoError(t, backUp(v, "first"))
|
||||
|
||||
// Upper-casing ASCII text keeps its size.
|
||||
files[rewrittenPath] = bytes.ToUpper(files[rewrittenPath])
|
||||
require.NoError(t, afero.WriteFile(fs, rewrittenPath, files[rewrittenPath], 0o644))
|
||||
require.NoError(t, fs.Chtimes(rewrittenPath, secondMTime, secondMTime))
|
||||
|
||||
require.NoError(t, backUp(v, "second"))
|
||||
|
||||
id := localSnapshotID(ctx, t, repos, "second")
|
||||
require.NoError(t, db.Close())
|
||||
|
||||
reader := newReaderVaultik(ctx, cfg, store, nil, fs)
|
||||
require.NoError(t, reader.Restore(&vaultik.RestoreOptions{
|
||||
SnapshotID: id,
|
||||
TargetDir: restoreDir,
|
||||
Verify: true,
|
||||
}))
|
||||
|
||||
assertRestoredTree(t, fs, restoreDir, files)
|
||||
}
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||
"sneak.berlin/go/vaultik/internal/types"
|
||||
)
|
||||
|
||||
// Sentinel errors for snapshot management.
|
||||
@@ -495,19 +496,23 @@ func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
|
||||
nameFilter[n] = struct{}{}
|
||||
}
|
||||
|
||||
// Collect completed snapshots, applying the name filter.
|
||||
// Collect completed snapshots and their names, applying the name filter.
|
||||
snapshots := make([]SnapshotInfo, 0, len(dbSnapshots))
|
||||
names := make(map[types.SnapshotID]string, len(dbSnapshots))
|
||||
|
||||
for _, s := range dbSnapshots {
|
||||
if s.CompletedAt == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
name := parseSnapshotName(s.ID.String(), s.Hostname.String())
|
||||
if len(nameFilter) > 0 {
|
||||
if _, ok := nameFilter[parseSnapshotName(s.ID.String())]; !ok {
|
||||
if _, ok := nameFilter[name]; !ok {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
names[s.ID] = name
|
||||
snapshots = append(snapshots, SnapshotInfo{
|
||||
ID: s.ID,
|
||||
Timestamp: s.StartedAt,
|
||||
@@ -520,7 +525,7 @@ func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
|
||||
return snapshots[i].Timestamp.After(snapshots[j].Timestamp)
|
||||
})
|
||||
|
||||
toDelete, err := selectSnapshotsToPurge(snapshots, opts)
|
||||
toDelete, err := selectSnapshotsToPurge(snapshots, names, opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -538,9 +543,11 @@ func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
|
||||
|
||||
// selectSnapshotsToPurge applies the purge retention criteria to the
|
||||
// newest-first sorted snapshot list and returns the deletion
|
||||
// candidates.
|
||||
// candidates. names maps each snapshot's ID to its snapshot name.
|
||||
func selectSnapshotsToPurge(
|
||||
snapshots []SnapshotInfo, opts *SnapshotPurgeOptions,
|
||||
snapshots []SnapshotInfo,
|
||||
names map[types.SnapshotID]string,
|
||||
opts *SnapshotPurgeOptions,
|
||||
) ([]SnapshotInfo, error) {
|
||||
var toDelete []SnapshotInfo
|
||||
|
||||
@@ -551,7 +558,7 @@ func selectSnapshotsToPurge(
|
||||
seen := make(map[string]bool)
|
||||
|
||||
for _, snap := range snapshots {
|
||||
name := parseSnapshotName(snap.ID.String())
|
||||
name := names[snap.ID]
|
||||
if seen[name] {
|
||||
toDelete = append(toDelete, snap)
|
||||
|
||||
@@ -711,7 +718,7 @@ func (v *Vaultik) VerifySnapshotWithOptions(
|
||||
result.BlobCount = manifest.BlobCount
|
||||
result.TotalSize = manifest.TotalCompressedSize
|
||||
|
||||
if !opts.JSON {
|
||||
if !opts.JSON && !v.UI.Quiet() {
|
||||
v.stdoutf("Snapshot information:\n")
|
||||
v.stdoutf(" Blob count: %d\n", manifest.BlobCount)
|
||||
v.stdoutf(" Total size: %s\n", ubytes(manifest.TotalCompressedSize))
|
||||
@@ -766,7 +773,7 @@ func (v *Vaultik) printVerifyHeader(snapshotID string, opts *VerifyOptions) {
|
||||
snapshotTime = t
|
||||
}
|
||||
|
||||
if !opts.JSON {
|
||||
if !opts.JSON && !v.UI.Quiet() {
|
||||
v.stdoutf("Verifying snapshot %s\n", snapshotID)
|
||||
|
||||
if !snapshotTime.IsZero() {
|
||||
@@ -806,7 +813,7 @@ func (v *Vaultik) verifyManifestBlobs(
|
||||
stat, err := v.Storage.Stat(v.ctx, blobPath)
|
||||
switch {
|
||||
case err != nil:
|
||||
if !opts.JSON {
|
||||
if !opts.JSON && !v.UI.Quiet() {
|
||||
v.stdoutf(" Missing: %s (%s)\n",
|
||||
blob.Hash, ubytes(blob.CompressedSize))
|
||||
}
|
||||
@@ -814,7 +821,7 @@ func (v *Vaultik) verifyManifestBlobs(
|
||||
missing++
|
||||
missingSize += blob.CompressedSize
|
||||
case stat.Size != blob.CompressedSize:
|
||||
if !opts.JSON {
|
||||
if !opts.JSON && !v.UI.Quiet() {
|
||||
v.stdoutf(" Wrong size: %s (store has %s, manifest lists %s)\n",
|
||||
blob.Hash, ubytes(stat.Size), ubytes(blob.CompressedSize))
|
||||
}
|
||||
@@ -846,6 +853,22 @@ func (v *Vaultik) formatVerifyResult(
|
||||
return v.outputVerifyJSON(result)
|
||||
}
|
||||
|
||||
// Under --quiet a failure is still returned, and the cli layer
|
||||
// prints it on stderr.
|
||||
if !v.UI.Quiet() {
|
||||
v.printVerifySummary(result, failure)
|
||||
}
|
||||
|
||||
if failure != "" {
|
||||
return fmt.Errorf("%w: %s", errSnapshotVerifyFailed, failure)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// printVerifySummary prints the counts and the status line that end the
|
||||
// human-readable shallow verify report. failure is empty when it passed.
|
||||
func (v *Vaultik) printVerifySummary(result *VerifyResult, failure string) {
|
||||
v.stdoutf("\nVerification complete:\n")
|
||||
v.stdoutf(" Present with listed size: %d blobs\n", result.Verified)
|
||||
|
||||
@@ -867,14 +890,12 @@ func (v *Vaultik) formatVerifyResult(
|
||||
if failure != "" {
|
||||
v.stdoutf("FAILED - %s\n", failure)
|
||||
|
||||
return fmt.Errorf("%w: %s", errSnapshotVerifyFailed, failure)
|
||||
return
|
||||
}
|
||||
|
||||
// Report only what was actually checked: presence and size, not contents.
|
||||
v.stdoutf("OK - all %d blobs listed in the manifest are present with the "+
|
||||
"listed size; contents not checked (use --deep)\n", result.Verified)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// shallowVerifyFailure returns a human-readable description of everything
|
||||
@@ -1031,7 +1052,7 @@ func (v *Vaultik) syncWithRemote() error {
|
||||
// every local snapshot record (issue #160).
|
||||
remoteKeys, err := v.listAllRemoteSnapshotKeys()
|
||||
if err != nil {
|
||||
return fmt.Errorf("listing remote snapshots: %w", err)
|
||||
return err
|
||||
}
|
||||
|
||||
remoteKeySet := make(map[string]bool, len(remoteKeys))
|
||||
@@ -1096,6 +1117,12 @@ type RemoveResult struct {
|
||||
// just-removed snapshot left behind on the destination store.
|
||||
const pruneCommandHint = "vaultik prune"
|
||||
|
||||
// snapshotRemoveCommandHint is the command suggested, with the
|
||||
// snapshot's ID, when a remove could not reach the destination store:
|
||||
// running it again removes the snapshot's metadata there, which
|
||||
// `vaultik prune` never does.
|
||||
const snapshotRemoveCommandHint = "vaultik snapshot remove"
|
||||
|
||||
// RemoveSnapshot removes a snapshot from the local index database and,
|
||||
// unless LocalOnly is set, also strips the snapshot's metadata from the
|
||||
// destination store. Blobs are NOT touched: removing a snapshot's
|
||||
@@ -1132,7 +1159,7 @@ func (v *Vaultik) RemoveSnapshot(
|
||||
}
|
||||
|
||||
if !opts.LocalOnly {
|
||||
result.RemoteRemoved = v.removeSnapshotRemote(snapshotID)
|
||||
result.RemoteRemoved = v.removeSnapshotRemote(snapshotID, opts)
|
||||
}
|
||||
|
||||
if v.SnapshotManager != nil {
|
||||
@@ -1215,9 +1242,11 @@ func (v *Vaultik) confirmRemoveSnapshot(snapshotID string, opts *RemoveOptions)
|
||||
// removeSnapshotRemote strips the snapshot's metadata from the
|
||||
// destination store, warning and proceeding on failure: the local-DB
|
||||
// removal has already happened, so the user is told the remote half
|
||||
// didn't finish and can retry with `vaultik prune` once the destination
|
||||
// store is reachable. Returns true when the remote removal succeeded.
|
||||
func (v *Vaultik) removeSnapshotRemote(snapshotID string) bool {
|
||||
// didn't finish and to run `vaultik snapshot remove` for the snapshot
|
||||
// again once the destination store is reachable (`vaultik prune` never
|
||||
// removes snapshot metadata). Returns true when the remote removal
|
||||
// succeeded.
|
||||
func (v *Vaultik) removeSnapshotRemote(snapshotID string, opts *RemoveOptions) bool {
|
||||
log.Info("Removing snapshot metadata from remote storage",
|
||||
"snapshot_id", snapshotID)
|
||||
|
||||
@@ -1225,13 +1254,17 @@ func (v *Vaultik) removeSnapshotRemote(snapshotID string) bool {
|
||||
|
||||
err := v.deleteRemoteSnapshotByKey(remoteKey)
|
||||
if err != nil {
|
||||
log.Warn("Could not remove snapshot metadata from remote storage",
|
||||
"error", err)
|
||||
log.Warn("Could not remove snapshot metadata from remote storage; "+
|
||||
"run '"+snapshotRemoveCommandHint+"' with the snapshot's ID "+
|
||||
"again once the remote is reachable",
|
||||
"snapshot_id", snapshotID, "error", err)
|
||||
|
||||
if v.UI != nil {
|
||||
// The UI writes to stdout, which under --json holds only the
|
||||
// document; the log record above is the warning on stderr.
|
||||
if v.UI != nil && !opts.JSON {
|
||||
v.UI.Warningf("Could not remove snapshot metadata from remote: "+
|
||||
"%v. Run '%s' once the remote is reachable to finish cleanup.",
|
||||
err, pruneCommandHint)
|
||||
"%v. Run '%s %s' again once the remote is reachable.",
|
||||
err, snapshotRemoveCommandHint, snapshotID)
|
||||
}
|
||||
|
||||
return false
|
||||
|
||||
+15
-12
@@ -103,7 +103,7 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
|
||||
|
||||
log.Info("Starting snapshot verification", "snapshot_id", snapshotID, "mode", "deep")
|
||||
|
||||
if !opts.JSON {
|
||||
if !opts.JSON && !v.UI.Quiet() {
|
||||
v.stdoutf("Deep verification of snapshot: %s\n\n", snapshotID)
|
||||
}
|
||||
|
||||
@@ -143,10 +143,13 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
|
||||
|
||||
log.Info("✓ Verification completed successfully",
|
||||
"snapshot_id", snapshotID, "mode", "deep", "blobs_verified", len(dbBlobs))
|
||||
v.stdoutf("\n✓ Verification completed successfully\n")
|
||||
v.stdoutf(" Snapshot: %s\n", snapshotID)
|
||||
v.stdoutf(" Blobs verified: %d\n", len(dbBlobs))
|
||||
v.stdoutf(" Total size: %s\n", ubytes(totalSize))
|
||||
|
||||
if !v.UI.Quiet() {
|
||||
v.stdoutf("\n✓ Verification completed successfully\n")
|
||||
v.stdoutf(" Snapshot: %s\n", snapshotID)
|
||||
v.stdoutf(" Blobs verified: %d\n", len(dbBlobs))
|
||||
v.stdoutf(" Total size: %s\n", ubytes(totalSize))
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -170,7 +173,7 @@ func (v *Vaultik) loadVerificationData(
|
||||
// remote manifests; see its doc comment.
|
||||
log.Info("Downloading manifest", "remote_key", remoteKey)
|
||||
|
||||
if !opts.JSON {
|
||||
if !opts.JSON && !v.UI.Quiet() {
|
||||
v.stdoutf("Downloading manifest...\n")
|
||||
}
|
||||
|
||||
@@ -185,7 +188,7 @@ func (v *Vaultik) loadVerificationData(
|
||||
"manifest_blob_count", manifest.BlobCount,
|
||||
"manifest_total_size", ubytes(manifest.TotalCompressedSize))
|
||||
|
||||
if !opts.JSON {
|
||||
if !opts.JSON && !v.UI.Quiet() {
|
||||
v.stdoutf("Manifest loaded: %d blobs (%s)\n",
|
||||
manifest.BlobCount, ubytes(manifest.TotalCompressedSize))
|
||||
v.stdoutf("Downloading and decrypting database...\n")
|
||||
@@ -215,7 +218,7 @@ func (v *Vaultik) loadVerificationData(
|
||||
"db_blob_count", len(dbBlobs),
|
||||
"db_total_size", ubytes(dbTotalSize))
|
||||
|
||||
if !opts.JSON {
|
||||
if !opts.JSON && !v.UI.Quiet() {
|
||||
v.stdoutf("Database loaded: %d blobs (%s)\n",
|
||||
len(dbBlobs), ubytes(dbTotalSize))
|
||||
}
|
||||
@@ -273,7 +276,7 @@ func (v *Vaultik) runVerificationSteps(
|
||||
totalSize int64,
|
||||
identities []age.Identity,
|
||||
) error {
|
||||
if !opts.JSON {
|
||||
if !opts.JSON && !v.UI.Quiet() {
|
||||
v.stdoutf("Verifying manifest against database...\n")
|
||||
}
|
||||
|
||||
@@ -282,7 +285,7 @@ func (v *Vaultik) runVerificationSteps(
|
||||
return v.deepVerifyFailure(result, opts, err.Error(), err)
|
||||
}
|
||||
|
||||
if !opts.JSON {
|
||||
if !opts.JSON && !v.UI.Quiet() {
|
||||
v.stdoutf("Manifest verified.\n")
|
||||
v.stdoutf("Checking blob existence in remote storage...\n")
|
||||
}
|
||||
@@ -292,7 +295,7 @@ func (v *Vaultik) runVerificationSteps(
|
||||
return v.deepVerifyFailure(result, opts, err.Error(), err)
|
||||
}
|
||||
|
||||
if !opts.JSON {
|
||||
if !opts.JSON && !v.UI.Quiet() {
|
||||
v.stdoutf("All blobs exist.\n")
|
||||
v.stdoutf("Downloading and verifying blob contents (%d blobs, %s)...\n",
|
||||
len(dbBlobs), ubytes(totalSize))
|
||||
@@ -748,7 +751,7 @@ func (v *Vaultik) performDeepVerificationFromDB(
|
||||
"eta", eta.Round(time.Second),
|
||||
)
|
||||
|
||||
if !opts.JSON {
|
||||
if !opts.JSON && !v.UI.Quiet() {
|
||||
v.stdoutf(" Verified %d/%d blobs (%d remaining) - %s/%s - elapsed %s, eta %s\n",
|
||||
i+1, len(blobs), remaining,
|
||||
ubytes(bytesProcessed),
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
package vaultik_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/spf13/afero"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||
"sneak.berlin/go/vaultik/internal/ui"
|
||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||
)
|
||||
|
||||
// TestVerify_QuietSuppressesReport is the --quiet contract for
|
||||
// `snapshot verify`: neither shallow nor deep verify writes its report,
|
||||
// a failed verify still returns its error (which the cli layer prints
|
||||
// on stderr), and the --json document still emits.
|
||||
func TestVerify_QuietSuppressesReport(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
|
||||
dataDir := filepath.Join(tempDir, "source")
|
||||
storeDir := filepath.Join(tempDir, "remote")
|
||||
dbPath := filepath.Join(tempDir, "index.sqlite")
|
||||
|
||||
chunkSize := int64(32 * 1024)
|
||||
maxBlobSize := int64(128 * 1024)
|
||||
|
||||
require.NoError(t, fs.MkdirAll(dataDir, 0o755))
|
||||
require.NoError(t, afero.WriteFile(fs,
|
||||
filepath.Join(dataDir, "data.bin"),
|
||||
bytesPattern("quiet-", int(maxBlobSize*2)), 0o644))
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
cfg, storer, snapshotID := runFileStorageBackup(
|
||||
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
|
||||
|
||||
// The UI writes to the same buffer as Stdout, as both write to the
|
||||
// process's stdout in production.
|
||||
var stdout bytes.Buffer
|
||||
|
||||
newQuietVerifier := func() *vaultik.Vaultik {
|
||||
v := &vaultik.Vaultik{
|
||||
Config: cfg,
|
||||
Storage: storer,
|
||||
Fs: fs,
|
||||
Stdout: &stdout,
|
||||
Stderr: io.Discard,
|
||||
UI: ui.NewWithColor(&stdout, false),
|
||||
}
|
||||
v.SetContext(ctx)
|
||||
v.UI.SetQuiet(true)
|
||||
|
||||
return v
|
||||
}
|
||||
|
||||
require.NoError(t, newQuietVerifier().VerifySnapshotWithOptions(
|
||||
snapshotID, &vaultik.VerifyOptions{}))
|
||||
require.Empty(t, stdout.String(),
|
||||
"shallow verify must write no report under --quiet")
|
||||
|
||||
require.NoError(t, newQuietVerifier().VerifySnapshotWithOptions(
|
||||
snapshotID, &vaultik.VerifyOptions{Deep: true}))
|
||||
require.Empty(t, stdout.String(),
|
||||
"deep verify must write no report under --quiet")
|
||||
|
||||
require.NoError(t, newQuietVerifier().VerifySnapshotWithOptions(
|
||||
snapshotID, &vaultik.VerifyOptions{JSON: true}))
|
||||
require.Equal(t, "ok", decodeVerifyResult(t, stdout.Bytes()).Status,
|
||||
"the --json document must still emit under --quiet")
|
||||
|
||||
// A snapshot without its encrypted database fails shallow verify. A
|
||||
// failed report also lists each missing blob and each blob of the
|
||||
// wrong size, so remove one blob and grow another.
|
||||
require.NoError(t, os.Remove(filepath.Join(storeDir, "metadata",
|
||||
snapshot.RemoteSnapshotKey(snapshotID), "db.zst.age")))
|
||||
|
||||
blobFiles, err := filepath.Glob(
|
||||
filepath.Join(storeDir, "blobs", "*", "*", "*"))
|
||||
require.NoError(t, err)
|
||||
require.GreaterOrEqual(t, len(blobFiles), 2,
|
||||
"the snapshot must span two blobs, one to remove and one to grow")
|
||||
require.NoError(t, os.Remove(blobFiles[0]))
|
||||
growOneBlob(t, fs, filepath.Join(storeDir, "blobs"))
|
||||
|
||||
stdout.Reset()
|
||||
|
||||
require.Error(t, newQuietVerifier().VerifySnapshotWithOptions(
|
||||
snapshotID, &vaultik.VerifyOptions{}),
|
||||
"--quiet must not change the outcome of a failed verify")
|
||||
require.Empty(t, stdout.String(),
|
||||
"a failed verify must write no report under --quiet")
|
||||
}
|
||||
+1
-1
@@ -19,7 +19,7 @@ s3:
|
||||
secret_access_key: test-secret-key
|
||||
region: us-east-1
|
||||
use_ssl: true
|
||||
part_size: 5242880 # 5MB
|
||||
part_size: 5242880 # 5MiB
|
||||
index_path: /tmp/vaultik-test.sqlite
|
||||
chunk_size: 10MB
|
||||
blob_size_limit: 10GB
|
||||
|
||||
Reference in New Issue
Block a user