Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3543f45ba8 | ||
|
|
c355ef4d25 | ||
|
|
5927e1aa3d | ||
|
|
9ca962969a | ||
|
|
89ebfc78e2 | ||
|
|
07ef3a1c78 | ||
|
|
c423d13191 |
+5
-5
@@ -72,11 +72,11 @@ RUN [ -n "$CHECK_EPOCH" ] || exit 1
|
||||
# running, and exits 0 reporting `0 issues.` on a tree the real config
|
||||
# fails. Demonstrated on this repo at this pin, recorded on
|
||||
# https://git.eeqj.de/sneak/vaultik/pulls/114: with a planted
|
||||
# over-length line, `script/lint` exits 1 naming the `lll` finding with
|
||||
# `linters:` and exits 0 with `linterz:`. A set-but-ineffective config
|
||||
# quietly falling back to defaults is precisely the false-green class
|
||||
# this gate exists to eliminate, so it must not sit in the gate's own
|
||||
# configuration.
|
||||
# over-length line, `script/lint` exits 1 naming the `revive` finding
|
||||
# with `linters:` and exits 0 with `linterz:`. A set-but-ineffective
|
||||
# config quietly falling back to defaults is precisely the false-green
|
||||
# class this gate exists to eliminate, so it must not sit in the gate's
|
||||
# own configuration.
|
||||
#
|
||||
# `config verify` catches it, and it does so OFFLINE at this pinned
|
||||
# version -- verified, not assumed. Under `docker run --network none`
|
||||
|
||||
@@ -251,7 +251,8 @@ local index alone, and still exits zero.
|
||||
per-snapshot-name (`--keep-latest` keeps the latest of each name, not the
|
||||
latest globally).
|
||||
* `--keep-latest`: Keep only the most recent snapshot of each name
|
||||
* `--older-than <duration>`: Remove snapshots older than duration (e.g. `30d`, `6m`, `1y`)
|
||||
* `--older-than <duration>`: Remove snapshots older than duration (e.g. `30d`,
|
||||
`4w`, `6mo`, `1y`; `m` is minutes, `mo` is months)
|
||||
* `--snapshot <name>`: Restrict to specific snapshot names (repeat for multiple)
|
||||
* `--force`: Skip confirmation prompt
|
||||
|
||||
|
||||
@@ -25,6 +25,34 @@ release" is exactly the contradiction
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-09-21: Stopped `prune` from reporting a failed row count as 0
|
||||
([issue #96](https://git.eeqj.de/sneak/vaultik/issues/96)). The seven
|
||||
`getTableCount` reads in `PruneDatabase` discarded their error, so a
|
||||
query that could not run became a plausible `0` and the before/after
|
||||
delta computed from it looked like real work. Each read now logs at
|
||||
warn on failure and renders as `unknown`, never `0`, so an empty table
|
||||
is distinguishable from one that could not be queried. The counts have
|
||||
no `--json` representation — under `--json` the summary is suppressed
|
||||
entirely — so nothing there can show a false `0`.
|
||||
|
||||
- 2026-09-21: Made the s3 storage backend report a missing object as
|
||||
`storage.ErrNotFound`, like the `file` and `rclone` backends and as the
|
||||
`Storer` interface documents. `S3Storer.Get` and `Stat` returned the raw
|
||||
AWS SDK error, so `errors.Is(err, storage.ErrNotFound)` was false on s3
|
||||
and callers branched differently per backend. Added a small `s3.IsNotFound`
|
||||
helper (reused by `HeadObject`) and a test that a missing key maps to
|
||||
`ErrNotFound`
|
||||
([issue #129](https://git.eeqj.de/sneak/vaultik/issues/129)).
|
||||
- 2026-09-21: Fixed `verify --deep` reporting healthy snapshots as
|
||||
corrupt. Its final blob-integrity check hashed the encrypted
|
||||
downloaded bytes with a single SHA256 and compared that to the blob
|
||||
ID, which is the double SHA256 of the plaintext, so the two could
|
||||
never match. It now hashes the decompressed plaintext and compares the
|
||||
double SHA256. Added a test that backs up a real snapshot, deep-verifies
|
||||
it, then flips a byte in one stored blob and confirms deep verification
|
||||
then fails
|
||||
([issue #131](https://git.eeqj.de/sneak/vaultik/issues/131)).
|
||||
|
||||
- 2026-09-21: Made `snapshot create` VACUUM the per-snapshot metadata
|
||||
database through the `modernc.org/sqlite` driver instead of shelling
|
||||
out to the external `sqlite` command-line binary (issue #120). A
|
||||
@@ -50,6 +78,24 @@ release" is exactly the contradiction
|
||||
keeps that exact compiler from auto-switching. Bumping Go now touches
|
||||
`go.mod`, the checksum, and the `Dockerfile` `golang` digest together.
|
||||
|
||||
- 2026-09-21: Collapsed the two duration parsers into one and fixed the
|
||||
`--older-than` months example
|
||||
([issue #123](https://git.eeqj.de/sneak/vaultik/issues/123)). Two
|
||||
functions named `parseDuration` existed with different grammars;
|
||||
`snapshot purge --older-than` and `--keep-newer-than` both already went
|
||||
through the one in `internal/vaultik`, while the richer copy in
|
||||
`internal/cli/duration.go` was reachable only from its own test. Kept
|
||||
the live-path parser and deleted the unused one, so no flag's accepted
|
||||
grammar changes. The trap the issue was filed over: `README.md`
|
||||
documented `6m` as the months example for `--older-than`, but `m` is
|
||||
minutes, so the documented command deleted every snapshot older than
|
||||
six minutes on a destructive flag. Corrected the doc to `6mo` and put
|
||||
both flags' help text on one example list that states `m` is minutes
|
||||
and `mo` is months. The surviving parser now rejects negatives, which
|
||||
it previously accepted (`-5h`) or silently made positive (`-5d`).
|
||||
Table-driven tests cover every unit, `6m` as six minutes, `6mo` as 180
|
||||
days, and rejection of a bare number, an unknown unit, and a negative.
|
||||
|
||||
- 2026-08-10: Moved every lint run into its own container, as a build
|
||||
step ([issue #113](https://git.eeqj.de/sneak/vaultik/issues/113)).
|
||||
New root `Dockerfile.lint`, built by `script/lint`, runs
|
||||
@@ -78,10 +124,11 @@ release" is exactly the contradiction
|
||||
into each check command, and a fresh `$(date +%s%N)$$` per invocation
|
||||
computed as a bare assignment. `cmd/vaultik/lintdocker_test.go`
|
||||
parses both Dockerfiles and both scripts and fails if any part of
|
||||
that is dropped, because every way of losing it is silent. Its
|
||||
host-lint assertion is structural — no script runs `golangci-lint`
|
||||
except through `docker` — rather than a search for the one retired
|
||||
variable name, which nothing could ever reintroduce.
|
||||
that is dropped, because every way of losing it is silent. No test
|
||||
asserts that no script runs the host linter: `script/lint` is the one
|
||||
lint entry point and runs `golangci-lint` only inside the container,
|
||||
and keeping it that way is a review matter, not something a test
|
||||
proves.
|
||||
|
||||
The product `Dockerfile` lost its lint stage rather than gaining a
|
||||
second linter pin: `make lint` is now `docker build`, so the stage
|
||||
|
||||
@@ -28,6 +28,11 @@ import (
|
||||
// -- that a real finding actually fails the build -- is verified by
|
||||
// hand against a deliberately broken tree, recorded on the pull
|
||||
// request.
|
||||
//
|
||||
// One property is deliberately NOT tested here: that no script runs the
|
||||
// linter on the host. script/lint is the only lint entry point, and it
|
||||
// runs golangci-lint only inside the container; keeping it that way is a
|
||||
// review matter, not something a test in this file establishes.
|
||||
|
||||
// The files under guard, relative to the repository root.
|
||||
const (
|
||||
@@ -37,9 +42,8 @@ const (
|
||||
cibuildScript = "script/cibuild"
|
||||
)
|
||||
|
||||
// linterBinary is the linter's command name. Every occurrence of it in
|
||||
// executable shell in this repo must be inside a docker invocation; see
|
||||
// TestNoHostLintPathRemains.
|
||||
// linterBinary is the linter's command name, used to locate the
|
||||
// config-verify and lint steps in Dockerfile.lint.
|
||||
const linterBinary = "golangci-lint"
|
||||
|
||||
// checkEpochARG is the declaration, with no default value. A default
|
||||
@@ -219,90 +223,6 @@ func TestCibuildBuildsBothDockerfilesWithFreshEpochs(t *testing.T) {
|
||||
"%s must build %s", cibuildScript, lintDockerfile)
|
||||
}
|
||||
|
||||
// TestNoHostLintPathRemains fails if any escape hatch to a host linter
|
||||
// comes back. The owner's ruling is that every lint run happens inside
|
||||
// a container; a PATH binary that happens to match the pinned version
|
||||
// is a different build reached by a different code path, and admitting
|
||||
// it is what lets a local pass disagree with CI.
|
||||
//
|
||||
// This asserts the PROPERTY -- no script invokes the linter except
|
||||
// through docker -- rather than the absence of any particular variable
|
||||
// name. An earlier version of this test looked only for the literal
|
||||
// VAULTIK_LINT_IN_CONTAINER, the name of the hatch that was removed
|
||||
// alongside it, so nothing could ever trip it again: a hatch under any
|
||||
// other name left it passing. A structural test that passes on a broken
|
||||
// tree is worse than no test, because it is what a later reader trusts
|
||||
// instead of re-deriving the invariant.
|
||||
//
|
||||
// script/lint-fix is not exempted. It is the one script that runs the
|
||||
// linter as a container rather than as a build step, but it still runs
|
||||
// it in one, so the same property holds of it.
|
||||
func TestNoHostLintPathRemains(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
root := repoRoot(t)
|
||||
|
||||
entries, err := os.ReadDir(filepath.Join(root, "script"))
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, entries, "no scripts found to scan")
|
||||
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() {
|
||||
continue
|
||||
}
|
||||
|
||||
name := filepath.Join("script", entry.Name())
|
||||
for _, line := range shellCode(readRepoFile(t, name)) {
|
||||
assertLinterIsContainerised(t, name, line)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// assertLinterIsContainerised fails if the line runs the linter without
|
||||
// handing it to docker first. Position matters: docker has to come
|
||||
// before the binary, or the line is running the host linter and merely
|
||||
// mentioning docker afterwards.
|
||||
func assertLinterIsContainerised(t *testing.T, name, line string) {
|
||||
t.Helper()
|
||||
|
||||
at := strings.Index(line, linterBinary)
|
||||
if at < 0 {
|
||||
return
|
||||
}
|
||||
|
||||
docker := strings.Index(line, "docker")
|
||||
|
||||
assert.True(t, docker >= 0 && docker < at,
|
||||
"%s runs %s on the host; every lint run happens in a container"+
|
||||
" (line: %s)", name, linterBinary, line)
|
||||
}
|
||||
|
||||
// TestShellCodeSeesCodeAndNotProse keeps the scanner above honest. It
|
||||
// has to ignore comments and here-document bodies, because script/lint
|
||||
// and script/bootstrap both NAME golangci-lint in prose -- in comments,
|
||||
// and in the error text they print -- precisely to say that the host
|
||||
// binary is never used. A scanner that went blind, by over-eager
|
||||
// stripping or by failing to join continuation lines, would make
|
||||
// TestNoHostLintPathRemains pass on everything.
|
||||
func TestShellCodeSeesCodeAndNotProse(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
script := strings.Join([]string{
|
||||
"#!/bin/sh",
|
||||
"# a comment naming golangci-lint",
|
||||
"cat >&2 <<EOF",
|
||||
"prose naming golangci-lint, printed not executed",
|
||||
"EOF",
|
||||
"docker run --rm \\",
|
||||
" \"$image\" \\",
|
||||
" golangci-lint run ./...",
|
||||
}, "\n")
|
||||
|
||||
assert.Equal(t,
|
||||
[]string{"cat >&2 <<EOF", `docker run --rm "$image" golangci-lint run ./...`},
|
||||
shellCode(script))
|
||||
}
|
||||
|
||||
// assertEpochExpandedInto fails unless some instruction runs the named
|
||||
// command with the epoch expanded into it. Expansion, not mere
|
||||
// declaration: an ARG that no instruction references is not guaranteed
|
||||
@@ -407,70 +327,6 @@ func indexContaining(found []string, want string) int {
|
||||
return -1
|
||||
}
|
||||
|
||||
// shellCode returns a POSIX shell script's executable lines: comments
|
||||
// dropped, here-document bodies dropped, and backslash continuations
|
||||
// joined so a multi-line command is a single string. Whitespace is
|
||||
// collapsed, as it is for Dockerfile instructions.
|
||||
//
|
||||
// Both exclusions are load-bearing rather than tidiness. The scripts
|
||||
// name golangci-lint in prose to state that the host binary is never
|
||||
// used, and joining continuations is what lets the one legitimate
|
||||
// container invocation -- script/lint-fix's `docker run`, whose linter
|
||||
// command sits several lines below the word `docker` -- be recognised
|
||||
// as containerised.
|
||||
func shellCode(contents string) []string {
|
||||
var (
|
||||
out []string
|
||||
joined string
|
||||
terminate string
|
||||
)
|
||||
|
||||
for line := range strings.SplitSeq(contents, "\n") {
|
||||
trimmed := strings.TrimSpace(line)
|
||||
|
||||
if terminate != "" {
|
||||
if trimmed == terminate {
|
||||
terminate = ""
|
||||
}
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if joined == "" && (trimmed == "" || strings.HasPrefix(trimmed, "#")) {
|
||||
continue
|
||||
}
|
||||
|
||||
joined += strings.TrimSuffix(trimmed, `\`) + " "
|
||||
if strings.HasSuffix(trimmed, `\`) {
|
||||
continue
|
||||
}
|
||||
|
||||
joined = strings.Join(strings.Fields(joined), " ")
|
||||
terminate = heredocTerminator(joined)
|
||||
|
||||
out = append(out, joined)
|
||||
joined = ""
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
// heredocTerminator returns the terminator of the here-document a
|
||||
// command opens, or "" if it opens none. Only the first on a line is
|
||||
// recognised; nothing in script/ opens two.
|
||||
func heredocTerminator(line string) string {
|
||||
_, after, opens := strings.Cut(line, "<<")
|
||||
if !opens {
|
||||
return ""
|
||||
}
|
||||
|
||||
// `<<-` strips leading tabs from the body; the terminator word is
|
||||
// the same either way, and callers compare against trimmed lines.
|
||||
word, _, _ := strings.Cut(strings.TrimPrefix(after, "-"), " ")
|
||||
|
||||
return strings.Trim(word, `'"`)
|
||||
}
|
||||
|
||||
// readRepoFile reads a file by its path relative to the repository
|
||||
// root.
|
||||
func readRepoFile(t *testing.T, name string) string {
|
||||
|
||||
@@ -1,126 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Approximate lengths of the extended calendar units accepted by
|
||||
// parseDuration.
|
||||
const (
|
||||
durationDay = 24 * time.Hour
|
||||
durationWeek = 7 * durationDay
|
||||
durationMonth = 30 * durationDay
|
||||
durationYear = 365 * durationDay
|
||||
)
|
||||
|
||||
var (
|
||||
errNegativeDuration = errors.New("negative durations are not supported")
|
||||
errInvalidDuration = errors.New("invalid duration format")
|
||||
errUnknownTimeUnit = errors.New("unknown time unit")
|
||||
)
|
||||
|
||||
// parseDuration parses duration strings. Supports standard Go duration format
|
||||
// (e.g., "3h30m", "1h45m30s") as well as extended units:
|
||||
// - d: days (e.g., "30d", "7d")
|
||||
// - w: weeks (e.g., "2w", "4w")
|
||||
// - mo: months (30 days) (e.g., "6mo", "1mo")
|
||||
// - y: years (365 days) (e.g., "1y", "2y")
|
||||
//
|
||||
// Can combine units: "1y6mo", "2w3d", "1d12h30m"
|
||||
func parseDuration(s string) (time.Duration, error) {
|
||||
// First try standard Go duration parsing
|
||||
d, err := time.ParseDuration(s)
|
||||
if err == nil {
|
||||
return d, nil
|
||||
}
|
||||
|
||||
// Extended duration parsing
|
||||
// Check for negative values
|
||||
if strings.HasPrefix(strings.TrimSpace(s), "-") {
|
||||
return 0, errNegativeDuration
|
||||
}
|
||||
|
||||
// Pattern matches: number + unit, repeated
|
||||
re := regexp.MustCompile(`(\d+(?:\.\d+)?)\s*([a-zA-Z]+)`)
|
||||
matches := re.FindAllStringSubmatch(s, -1)
|
||||
|
||||
if len(matches) == 0 {
|
||||
return 0, fmt.Errorf("%w: %q", errInvalidDuration, s)
|
||||
}
|
||||
|
||||
var total time.Duration
|
||||
|
||||
for _, match := range matches {
|
||||
valueStr := match[1]
|
||||
unit := strings.ToLower(match[2])
|
||||
|
||||
value, err := strconv.ParseFloat(valueStr, 64)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("invalid number %q: %w", valueStr, err)
|
||||
}
|
||||
|
||||
d, err := durationForUnit(value, unit)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
total += d
|
||||
}
|
||||
|
||||
return total, nil
|
||||
}
|
||||
|
||||
// durationForUnit converts a value with a (case-normalized) unit suffix
|
||||
// into a time.Duration, accepting Go's standard units plus the extended
|
||||
// calendar units.
|
||||
func durationForUnit(value float64, unit string) (time.Duration, error) {
|
||||
switch unit {
|
||||
// Standard time units
|
||||
case "ns", "nanosecond", "nanoseconds":
|
||||
return time.Duration(value), nil
|
||||
case "us", "µs", "microsecond", "microseconds":
|
||||
return time.Duration(value * float64(time.Microsecond)), nil
|
||||
case "ms", "millisecond", "milliseconds":
|
||||
return time.Duration(value * float64(time.Millisecond)), nil
|
||||
case "s", "sec", "second", "seconds":
|
||||
return time.Duration(value * float64(time.Second)), nil
|
||||
case "m", "min", "minute", "minutes":
|
||||
return time.Duration(value * float64(time.Minute)), nil
|
||||
case "h", "hr", "hour", "hours":
|
||||
return time.Duration(value * float64(time.Hour)), nil
|
||||
// Extended units
|
||||
case "d", "day", "days":
|
||||
return time.Duration(value * float64(durationDay)), nil
|
||||
case "w", "week", "weeks":
|
||||
return time.Duration(value * float64(durationWeek)), nil
|
||||
case "mo", "month", "months":
|
||||
// Using 30 days as approximation
|
||||
return time.Duration(value * float64(durationMonth)), nil
|
||||
case "y", "year", "years":
|
||||
// Using 365 days as approximation
|
||||
return time.Duration(value * float64(durationYear)), nil
|
||||
default:
|
||||
// Try parsing as standard Go duration unit
|
||||
testStr := "1" + unit
|
||||
|
||||
_, err := time.ParseDuration(testStr)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("%w: %q", errUnknownTimeUnit, unit)
|
||||
}
|
||||
|
||||
// It's a valid Go duration unit, parse the full value
|
||||
fullStr := fmt.Sprintf("%g%s", value, unit)
|
||||
|
||||
d, err := time.ParseDuration(fullStr)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("invalid duration %q: %w", fullStr, err)
|
||||
}
|
||||
|
||||
return d, nil
|
||||
}
|
||||
}
|
||||
@@ -1,299 +0,0 @@
|
||||
package cli //nolint:testpackage // needs access to unexported parseDuration
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
type parseDurationCase struct {
|
||||
name string
|
||||
input string
|
||||
expected time.Duration
|
||||
wantErr bool
|
||||
}
|
||||
|
||||
// runParseDurationCases executes a table of parseDuration cases as
|
||||
// parallel subtests.
|
||||
func runParseDurationCases(t *testing.T, tests []parseDurationCase) {
|
||||
t.Helper()
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, err := parseDuration(tt.input)
|
||||
|
||||
if tt.wantErr {
|
||||
require.Error(t, err, "expected error for input %q", tt.input)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
require.NoError(t, err, "unexpected error for input %q", tt.input)
|
||||
assert.Equal(t, tt.expected, got, "duration mismatch for input %q", tt.input)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseDurationStandard(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
runParseDurationCases(t, []parseDurationCase{
|
||||
{
|
||||
name: "standard seconds",
|
||||
input: "30s",
|
||||
expected: 30 * time.Second,
|
||||
},
|
||||
{
|
||||
name: "standard minutes",
|
||||
input: "45m",
|
||||
expected: 45 * time.Minute,
|
||||
},
|
||||
{
|
||||
name: "standard hours",
|
||||
input: "2h",
|
||||
expected: 2 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "standard combined",
|
||||
input: "3h30m",
|
||||
expected: 3*time.Hour + 30*time.Minute,
|
||||
},
|
||||
{
|
||||
name: "standard complex",
|
||||
input: "1h45m30s",
|
||||
expected: 1*time.Hour + 45*time.Minute + 30*time.Second,
|
||||
},
|
||||
{
|
||||
name: "standard with milliseconds",
|
||||
input: "1s500ms",
|
||||
expected: 1*time.Second + 500*time.Millisecond,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
func TestParseDurationExtendedUnits(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
runParseDurationCases(t, []parseDurationCase{
|
||||
// Extended units - days
|
||||
{
|
||||
name: "single day",
|
||||
input: "1d",
|
||||
expected: 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "multiple days",
|
||||
input: "7d",
|
||||
expected: 7 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "fractional days",
|
||||
input: "1.5d",
|
||||
expected: 36 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "days spelled out",
|
||||
input: "3days",
|
||||
expected: 3 * 24 * time.Hour,
|
||||
},
|
||||
// Extended units - weeks
|
||||
{
|
||||
name: "single week",
|
||||
input: "1w",
|
||||
expected: 7 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "multiple weeks",
|
||||
input: "4w",
|
||||
expected: 4 * 7 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "weeks spelled out",
|
||||
input: "2weeks",
|
||||
expected: 2 * 7 * 24 * time.Hour,
|
||||
},
|
||||
// Extended units - months
|
||||
{
|
||||
name: "single month",
|
||||
input: "1mo",
|
||||
expected: 30 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "multiple months",
|
||||
input: "6mo",
|
||||
expected: 6 * 30 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "months spelled out",
|
||||
input: "3months",
|
||||
expected: 3 * 30 * 24 * time.Hour,
|
||||
},
|
||||
// Extended units - years
|
||||
{
|
||||
name: "single year",
|
||||
input: "1y",
|
||||
expected: 365 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "multiple years",
|
||||
input: "2y",
|
||||
expected: 2 * 365 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "years spelled out",
|
||||
input: "1year",
|
||||
expected: 365 * 24 * time.Hour,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
func TestParseDurationCombinedAndErrors(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
runParseDurationCases(t, []parseDurationCase{
|
||||
// Combined extended units
|
||||
{
|
||||
name: "weeks and days",
|
||||
input: "2w3d",
|
||||
expected: 2*7*24*time.Hour + 3*24*time.Hour,
|
||||
},
|
||||
{
|
||||
name: "years and months",
|
||||
input: "1y6mo",
|
||||
expected: 365*24*time.Hour + 6*30*24*time.Hour,
|
||||
},
|
||||
{
|
||||
name: "days and hours",
|
||||
input: "1d12h",
|
||||
expected: 24*time.Hour + 12*time.Hour,
|
||||
},
|
||||
{
|
||||
name: "complex combination",
|
||||
input: "1y2mo3w4d5h6m7s",
|
||||
expected: 365*24*time.Hour + 2*30*24*time.Hour +
|
||||
3*7*24*time.Hour + 4*24*time.Hour +
|
||||
5*time.Hour + 6*time.Minute + 7*time.Second,
|
||||
},
|
||||
{
|
||||
name: "with spaces",
|
||||
input: "1d 12h 30m",
|
||||
expected: 24*time.Hour + 12*time.Hour + 30*time.Minute,
|
||||
},
|
||||
// Edge cases
|
||||
{
|
||||
name: "zero duration",
|
||||
input: "0s",
|
||||
expected: 0,
|
||||
},
|
||||
{
|
||||
name: "large duration",
|
||||
input: "10y",
|
||||
expected: 10 * 365 * 24 * time.Hour,
|
||||
},
|
||||
// Error cases
|
||||
{
|
||||
name: "empty string",
|
||||
input: "",
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "invalid format",
|
||||
input: "abc",
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "unknown unit",
|
||||
input: "5x",
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "invalid number",
|
||||
input: "xyzd",
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "negative not supported",
|
||||
input: "-5d",
|
||||
wantErr: true,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
func TestParseDurationSpecialCases(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Test that standard Go durations work exactly as expected
|
||||
standardDurations := []string{
|
||||
"300ms",
|
||||
"1.5h",
|
||||
"2h45m",
|
||||
"72h",
|
||||
"1us",
|
||||
"1µs",
|
||||
"1ns",
|
||||
}
|
||||
|
||||
for _, d := range standardDurations {
|
||||
expected, err := time.ParseDuration(d)
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := parseDuration(d)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, expected, got, "standard duration %q should parse identically", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseDurationRealWorldExamples(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Test real-world snapshot purge scenarios
|
||||
tests := []struct {
|
||||
description string
|
||||
input string
|
||||
olderThan time.Duration
|
||||
}{
|
||||
{
|
||||
description: "keep snapshots from last 30 days",
|
||||
input: "30d",
|
||||
olderThan: 30 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
description: "keep snapshots from last 6 months",
|
||||
input: "6mo",
|
||||
olderThan: 6 * 30 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
description: "keep snapshots from last year",
|
||||
input: "1y",
|
||||
olderThan: 365 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
description: "keep snapshots from last week and a half",
|
||||
input: "1w3d",
|
||||
olderThan: 10 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
description: "keep snapshots from last 90 days",
|
||||
input: "90d",
|
||||
olderThan: 90 * 24 * time.Hour,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.description, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, err := parseDuration(tt.input)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, tt.olderThan, got)
|
||||
|
||||
// Verify the duration makes sense for snapshot purging
|
||||
assert.Greater(t, got, time.Hour,
|
||||
"snapshot purge duration should be at least an hour")
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -141,7 +141,8 @@ specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
|
||||
"orphaned blobs")
|
||||
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
||||
"With --prune: keep snapshots newer than this duration "+
|
||||
"(e.g. 4w, 30d, 6mo) instead of only the latest")
|
||||
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
||||
"instead of only the latest")
|
||||
|
||||
return cmd
|
||||
}
|
||||
@@ -204,7 +205,8 @@ restrict the operation to specific snapshot names.`,
|
||||
cmd.Flags().BoolVar(&opts.KeepLatest, "keep-latest", false,
|
||||
"Keep only the latest snapshot of each name")
|
||||
cmd.Flags().StringVar(&opts.OlderThan, "older-than", "",
|
||||
"Remove snapshots older than duration (e.g., 30d, 6m, 1y)")
|
||||
"Remove snapshots older than duration "+
|
||||
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months)")
|
||||
cmd.Flags().BoolVar(&opts.Force, "force", false, "Skip confirmation prompt")
|
||||
cmd.Flags().StringArrayVar(&opts.Names, "snapshot", nil,
|
||||
"Restrict to snapshots with these names (repeat for multiple)")
|
||||
|
||||
+13
-5
@@ -219,11 +219,7 @@ func (c *Client) HeadObject(ctx context.Context, key string) (bool, error) {
|
||||
Key: aws.String(fullKey),
|
||||
})
|
||||
if err != nil {
|
||||
var (
|
||||
notFound *s3types.NotFound
|
||||
noSuchKey *s3types.NoSuchKey
|
||||
)
|
||||
if errors.As(err, ¬Found) || errors.As(err, &noSuchKey) {
|
||||
if IsNotFound(err) {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
@@ -233,6 +229,18 @@ func (c *Client) HeadObject(ctx context.Context, key string) (bool, error) {
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// IsNotFound reports whether err indicates that an object does not exist.
|
||||
// Head and Get requests surface a missing object as different SDK types,
|
||||
// so both are checked here.
|
||||
func IsNotFound(err error) bool {
|
||||
var (
|
||||
notFound *s3types.NotFound
|
||||
noSuchKey *s3types.NoSuchKey
|
||||
)
|
||||
|
||||
return errors.As(err, ¬Found) || errors.As(err, &noSuchKey)
|
||||
}
|
||||
|
||||
// ObjectInfo contains information about an S3 object.
|
||||
// It is used by ListObjectsStream to return object metadata
|
||||
// along with any errors encountered during listing.
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"reflect"
|
||||
"sort"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
)
|
||||
|
||||
// runStorerConformance is the shared Storer contract. Every backend that
|
||||
// can run in-process is expected to pass it: TestFileStorer runs it against
|
||||
// file://, TestS3Storer against s3://. A new backend inherits this coverage
|
||||
// by passing its own constructor, so the contract is defined once.
|
||||
//
|
||||
// It exercises the public Storer interface: round-trip, stat, list with
|
||||
// prefix filtering, overwrite, delete, delete-of-missing, and not-found on
|
||||
// Get and Stat. Each section takes its own fresh backend instance, so the
|
||||
// order of sections never matters and no section sees another's objects.
|
||||
func runStorerConformance(t *testing.T, newStorer func(*testing.T) storage.Storer) {
|
||||
t.Helper()
|
||||
|
||||
conformanceRoundTrip(t, newStorer(t))
|
||||
conformanceOverwrite(t, newStorer(t))
|
||||
conformanceList(t, newStorer(t))
|
||||
conformanceDelete(t, newStorer(t))
|
||||
conformanceNotFound(t, newStorer(t))
|
||||
}
|
||||
|
||||
// conformanceRoundTrip stores a nested key, then reads it back and stats it.
|
||||
func conformanceRoundTrip(t *testing.T, s storage.Storer) {
|
||||
t.Helper()
|
||||
|
||||
ctx := context.Background()
|
||||
key := "blobs/aa/bb/object.bin"
|
||||
want := []byte("round-trip payload")
|
||||
|
||||
err := s.Put(ctx, key, bytes.NewReader(want))
|
||||
if err != nil {
|
||||
t.Fatalf("Put: %v", err)
|
||||
}
|
||||
|
||||
got := getBytes(t, s, key)
|
||||
if !bytes.Equal(got, want) {
|
||||
t.Errorf("Get returned %q, want %q", got, want)
|
||||
}
|
||||
|
||||
info, err := s.Stat(ctx, key)
|
||||
if err != nil {
|
||||
t.Fatalf("Stat: %v", err)
|
||||
}
|
||||
|
||||
if info.Key != key {
|
||||
t.Errorf("Stat key = %q, want %q", info.Key, key)
|
||||
}
|
||||
|
||||
if info.Size != int64(len(want)) {
|
||||
t.Errorf("Stat size = %d, want %d", info.Size, len(want))
|
||||
}
|
||||
}
|
||||
|
||||
// conformanceOverwrite checks that a second Put replaces the first.
|
||||
func conformanceOverwrite(t *testing.T, s storage.Storer) {
|
||||
t.Helper()
|
||||
|
||||
ctx := context.Background()
|
||||
key := "meta/snapshot.json"
|
||||
|
||||
err := s.Put(ctx, key, bytes.NewReader([]byte("first")))
|
||||
if err != nil {
|
||||
t.Fatalf("first Put: %v", err)
|
||||
}
|
||||
|
||||
want := []byte("second and longer payload")
|
||||
|
||||
err = s.Put(ctx, key, bytes.NewReader(want))
|
||||
if err != nil {
|
||||
t.Fatalf("second Put: %v", err)
|
||||
}
|
||||
|
||||
got := getBytes(t, s, key)
|
||||
if !bytes.Equal(got, want) {
|
||||
t.Errorf("after overwrite Get returned %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// conformanceList checks prefix filtering and the empty result for a
|
||||
// prefix that matches nothing.
|
||||
func conformanceList(t *testing.T, s storage.Storer) {
|
||||
t.Helper()
|
||||
|
||||
ctx := context.Background()
|
||||
keys := []string{"blobs/aa/one", "blobs/bb/two", "meta/three"}
|
||||
|
||||
for _, k := range keys {
|
||||
err := s.Put(ctx, k, bytes.NewReader([]byte("data")))
|
||||
if err != nil {
|
||||
t.Fatalf("Put %q: %v", k, err)
|
||||
}
|
||||
}
|
||||
|
||||
if got := listSorted(t, s, ""); !reflect.DeepEqual(got, keys) {
|
||||
t.Errorf("List(\"\") = %v, want %v", got, keys)
|
||||
}
|
||||
|
||||
wantBlobs := []string{"blobs/aa/one", "blobs/bb/two"}
|
||||
if got := listSorted(t, s, "blobs/"); !reflect.DeepEqual(got, wantBlobs) {
|
||||
t.Errorf("List(\"blobs/\") = %v, want %v", got, wantBlobs)
|
||||
}
|
||||
|
||||
if got := listSorted(t, s, "absent/"); len(got) != 0 {
|
||||
t.Errorf("List(\"absent/\") = %v, want empty", got)
|
||||
}
|
||||
}
|
||||
|
||||
// conformanceDelete checks that Delete removes an object and that deleting
|
||||
// a missing key is not an error.
|
||||
func conformanceDelete(t *testing.T, s storage.Storer) {
|
||||
t.Helper()
|
||||
|
||||
ctx := context.Background()
|
||||
key := "blobs/cc/gone.bin"
|
||||
|
||||
err := s.Put(ctx, key, bytes.NewReader([]byte("temporary")))
|
||||
if err != nil {
|
||||
t.Fatalf("Put: %v", err)
|
||||
}
|
||||
|
||||
err = s.Delete(ctx, key)
|
||||
if err != nil {
|
||||
t.Fatalf("Delete: %v", err)
|
||||
}
|
||||
|
||||
_, err = s.Get(ctx, key)
|
||||
if !errors.Is(err, storage.ErrNotFound) {
|
||||
t.Errorf("Get after Delete error = %v, want ErrNotFound", err)
|
||||
}
|
||||
|
||||
err = s.Delete(ctx, key)
|
||||
if err != nil {
|
||||
t.Errorf("Delete of missing key = %v, want nil", err)
|
||||
}
|
||||
}
|
||||
|
||||
// conformanceNotFound checks Get and Stat on an absent key.
|
||||
func conformanceNotFound(t *testing.T, s storage.Storer) {
|
||||
t.Helper()
|
||||
|
||||
ctx := context.Background()
|
||||
key := "never/written"
|
||||
|
||||
_, err := s.Get(ctx, key)
|
||||
if !errors.Is(err, storage.ErrNotFound) {
|
||||
t.Errorf("Get error = %v, want ErrNotFound", err)
|
||||
}
|
||||
|
||||
_, err = s.Stat(ctx, key)
|
||||
if !errors.Is(err, storage.ErrNotFound) {
|
||||
t.Errorf("Stat error = %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
// getBytes reads a key fully and closes the reader.
|
||||
func getBytes(t *testing.T, s storage.Storer, key string) []byte {
|
||||
t.Helper()
|
||||
|
||||
rc, err := s.Get(context.Background(), key)
|
||||
if err != nil {
|
||||
t.Fatalf("Get %q: %v", key, err)
|
||||
}
|
||||
|
||||
defer func() { _ = rc.Close() }()
|
||||
|
||||
data, err := io.ReadAll(rc)
|
||||
if err != nil {
|
||||
t.Fatalf("read %q: %v", key, err)
|
||||
}
|
||||
|
||||
return data
|
||||
}
|
||||
|
||||
// listSorted returns the keys under a prefix in a stable order.
|
||||
func listSorted(t *testing.T, s storage.Storer, prefix string) []string {
|
||||
t.Helper()
|
||||
|
||||
keys, err := s.List(context.Background(), prefix)
|
||||
if err != nil {
|
||||
t.Fatalf("List %q: %v", prefix, err)
|
||||
}
|
||||
|
||||
sort.Strings(keys)
|
||||
|
||||
return keys
|
||||
}
|
||||
+79
-54
@@ -46,31 +46,18 @@ func (f *FileStorer) SetFilesystem(fs afero.Fs) {
|
||||
// storage base path.
|
||||
const storageDirPerm = 0o755
|
||||
|
||||
// tempSuffix marks a partially written object. writeAtomic streams into a
|
||||
// temp file carrying this suffix and only renames it onto the real key once
|
||||
// the whole object is on disk, so an interrupted write can never leave a
|
||||
// truncated object at the key a later run would Stat and trust as a complete
|
||||
// blob. List and ListStream skip these files, so a leftover from an
|
||||
// interrupted write is never listed or trusted as a blob; it is otherwise
|
||||
// harmless and is overwritten when the same key is written again.
|
||||
const tempSuffix = ".partial"
|
||||
|
||||
// Put stores data at the specified key.
|
||||
func (f *FileStorer) Put(_ context.Context, key string, data io.Reader) error {
|
||||
path := f.fullPath(key)
|
||||
|
||||
// Create parent directories
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
err := f.fs.MkdirAll(dir, storageDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating directories: %w", err)
|
||||
}
|
||||
|
||||
file, err := f.fs.Create(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating file: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = file.Close() }()
|
||||
|
||||
_, err = io.Copy(file, data)
|
||||
if err != nil {
|
||||
return fmt.Errorf("writing file: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
return f.writeAtomic(key, data, nil)
|
||||
}
|
||||
|
||||
// PutWithProgress stores data with progress reporting.
|
||||
@@ -78,35 +65,7 @@ func (f *FileStorer) PutWithProgress(
|
||||
_ context.Context, key string, data io.Reader,
|
||||
_ int64, progress ProgressCallback,
|
||||
) error {
|
||||
path := f.fullPath(key)
|
||||
|
||||
// Create parent directories
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
err := f.fs.MkdirAll(dir, storageDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating directories: %w", err)
|
||||
}
|
||||
|
||||
file, err := f.fs.Create(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating file: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = file.Close() }()
|
||||
|
||||
// Wrap with progress tracking
|
||||
pw := &progressWriter{
|
||||
writer: file,
|
||||
callback: progress,
|
||||
}
|
||||
|
||||
_, err = io.Copy(pw, data)
|
||||
if err != nil {
|
||||
return fmt.Errorf("writing file: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
return f.writeAtomic(key, data, progress)
|
||||
}
|
||||
|
||||
// Get retrieves data from the specified key.
|
||||
@@ -188,7 +147,7 @@ func (f *FileStorer) List(ctx context.Context, prefix string) ([]string, error)
|
||||
default:
|
||||
}
|
||||
|
||||
if !info.IsDir() {
|
||||
if !info.IsDir() && !strings.HasSuffix(info.Name(), tempSuffix) {
|
||||
// Convert back to key (relative path from basePath)
|
||||
relPath, err := filepath.Rel(f.basePath, path)
|
||||
if err != nil {
|
||||
@@ -245,7 +204,7 @@ func (f *FileStorer) ListStream(ctx context.Context, prefix string) <-chan Objec
|
||||
return nil //nolint:nilerr // continue walking despite errors
|
||||
}
|
||||
|
||||
if !info.IsDir() {
|
||||
if !info.IsDir() && !strings.HasSuffix(info.Name(), tempSuffix) {
|
||||
relPath, err := filepath.Rel(f.basePath, path)
|
||||
if err != nil {
|
||||
ch <- ObjectInfo{Err: fmt.Errorf("computing relative path: %w", err)}
|
||||
@@ -275,6 +234,72 @@ func (f *FileStorer) Info() Info {
|
||||
}
|
||||
}
|
||||
|
||||
// writeAtomic streams data into a temp file in the destination directory,
|
||||
// fsyncs it, and renames it onto the final key. The key therefore appears
|
||||
// only once the whole object has been durably written; a failure part-way
|
||||
// leaves a temp file (removed here on the failing path) rather than a
|
||||
// truncated object at the key.
|
||||
func (f *FileStorer) writeAtomic(
|
||||
key string, data io.Reader, progress ProgressCallback,
|
||||
) error {
|
||||
path := f.fullPath(key)
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
err := f.fs.MkdirAll(dir, storageDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating directories: %w", err)
|
||||
}
|
||||
|
||||
tmp, err := afero.TempFile(f.fs, dir, filepath.Base(path)+"-*"+tempSuffix)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating temp file: %w", err)
|
||||
}
|
||||
|
||||
tmpPath := tmp.Name()
|
||||
|
||||
// Remove the temp file unless the rename below claims it. On the success
|
||||
// path renamed is true, so the deferred Close and Remove are harmless
|
||||
// no-ops on a name that no longer exists.
|
||||
renamed := false
|
||||
|
||||
defer func() {
|
||||
_ = tmp.Close()
|
||||
|
||||
if !renamed {
|
||||
_ = f.fs.Remove(tmpPath)
|
||||
}
|
||||
}()
|
||||
|
||||
var w io.Writer = tmp
|
||||
if progress != nil {
|
||||
w = &progressWriter{writer: tmp, callback: progress}
|
||||
}
|
||||
|
||||
_, err = io.Copy(w, data)
|
||||
if err != nil {
|
||||
return fmt.Errorf("writing file: %w", err)
|
||||
}
|
||||
|
||||
err = tmp.Sync()
|
||||
if err != nil {
|
||||
return fmt.Errorf("syncing temp file: %w", err)
|
||||
}
|
||||
|
||||
err = tmp.Close()
|
||||
if err != nil {
|
||||
return fmt.Errorf("closing temp file: %w", err)
|
||||
}
|
||||
|
||||
err = f.fs.Rename(tmpPath, path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("renaming temp file: %w", err)
|
||||
}
|
||||
|
||||
renamed = true
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// fullPath returns the full filesystem path for a key.
|
||||
func (f *FileStorer) fullPath(key string) string {
|
||||
return filepath.Join(f.basePath, key)
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
)
|
||||
|
||||
// errStreamInterrupted stands in for an upload cut off mid-stream.
|
||||
var errStreamInterrupted = errors.New("connection reset mid-upload")
|
||||
|
||||
// failingReader yields its data once, then fails.
|
||||
type failingReader struct {
|
||||
data []byte
|
||||
done bool
|
||||
}
|
||||
|
||||
func (r *failingReader) Read(p []byte) (int, error) {
|
||||
if r.done {
|
||||
return 0, errStreamInterrupted
|
||||
}
|
||||
|
||||
n := copy(p, r.data)
|
||||
r.done = true
|
||||
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// TestFileStorer_InterruptedWriteLeavesNoTrustedObject checks that a write
|
||||
// cut off mid-stream leaves nothing at the destination key, so a later run
|
||||
// cannot Stat a truncated object and trust it as a complete blob.
|
||||
func TestFileStorer_InterruptedWriteLeavesNoTrustedObject(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f, err := storage.NewFileStorer(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStorer: %v", err)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
key := "blobs/aa/bb/aabbccddeeff"
|
||||
|
||||
err = f.PutWithProgress(ctx, key, &failingReader{data: []byte("partial")}, 4096, nil)
|
||||
if err == nil {
|
||||
t.Fatal("expected the interrupted write to fail, got nil")
|
||||
}
|
||||
|
||||
_, err = f.Stat(ctx, key)
|
||||
if !errors.Is(err, storage.ErrNotFound) {
|
||||
t.Fatalf("expected key absent after interrupted write, got Stat err %v", err)
|
||||
}
|
||||
|
||||
keys, err := f.List(ctx, "blobs/")
|
||||
if err != nil {
|
||||
t.Fatalf("List: %v", err)
|
||||
}
|
||||
|
||||
if len(keys) != 0 {
|
||||
t.Fatalf("expected no keys listed after interrupted write, got %v", keys)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFileStorer_ListSkipsPartialFiles checks that a leftover temp file (the
|
||||
// storage layer names them with a ".partial" suffix) is never surfaced as a
|
||||
// key by List or ListStream.
|
||||
func TestFileStorer_ListSkipsPartialFiles(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
base := t.TempDir()
|
||||
|
||||
f, err := storage.NewFileStorer(base)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStorer: %v", err)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
realKey := "blobs/aa/bb/aabbccddeeff"
|
||||
|
||||
err = f.Put(ctx, realKey, strings.NewReader("blob-bytes"))
|
||||
if err != nil {
|
||||
t.Fatalf("Put: %v", err)
|
||||
}
|
||||
|
||||
// A stray temp file, as an interrupted write would leave behind.
|
||||
leftover := filepath.Join(base, "blobs/aa/bb/aabbccddeeff-123456.partial")
|
||||
|
||||
err = os.WriteFile(leftover, []byte("half"), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("writing leftover temp file: %v", err)
|
||||
}
|
||||
|
||||
keys, err := f.List(ctx, "blobs/")
|
||||
if err != nil {
|
||||
t.Fatalf("List: %v", err)
|
||||
}
|
||||
|
||||
if len(keys) != 1 || keys[0] != realKey {
|
||||
t.Fatalf("List should return only the real key, got %v", keys)
|
||||
}
|
||||
|
||||
var streamed []string
|
||||
|
||||
for obj := range f.ListStream(ctx, "blobs/") {
|
||||
if obj.Err != nil {
|
||||
t.Fatalf("ListStream: %v", obj.Err)
|
||||
}
|
||||
|
||||
streamed = append(streamed, obj.Key)
|
||||
}
|
||||
|
||||
if len(streamed) != 1 || streamed[0] != realKey {
|
||||
t.Fatalf("ListStream should return only the real key, got %v", streamed)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
)
|
||||
|
||||
// newFileStorer builds a file:// backend rooted at a fresh temp directory.
|
||||
//
|
||||
//nolint:ireturn // conformance runs against the Storer interface by design
|
||||
func newFileStorer(t *testing.T) storage.Storer {
|
||||
t.Helper()
|
||||
|
||||
s, err := storage.NewFileStorer(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStorer: %v", err)
|
||||
}
|
||||
|
||||
return s
|
||||
}
|
||||
|
||||
// TestFileStorer runs the shared Storer contract against the file:// backend.
|
||||
func TestFileStorer(t *testing.T) {
|
||||
t.Parallel()
|
||||
runStorerConformance(t, newFileStorer)
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
)
|
||||
|
||||
// The rclone backend is a thin adapter over the rclone library: it turns a
|
||||
// (remote, path) pair into rclone's "remote:path" string, hands it to
|
||||
// rclone, and maps rclone's own results back to the Storer interface. What
|
||||
// can be tested in-process, without a configured remote or network, is that
|
||||
// adapter layer — how the arguments are shaped and how construction errors
|
||||
// are reported. The data-plane operations (Put/Get/List/Delete) are rclone's
|
||||
// own, exercised against a real provider (drive, s3-via-rclone, ...), which
|
||||
// needs a configured remote with credentials and network access and so is
|
||||
// out of reach of a unit test. The shared Storer conformance suite therefore
|
||||
// runs against the in-process file and s3 backends; the rclone backend
|
||||
// inherits that contract once a remote is configured.
|
||||
//
|
||||
// These tests use rclone's ":local:" on-the-fly backend, which addresses the
|
||||
// local filesystem directly without any configured remote, so construction
|
||||
// runs entirely in-process.
|
||||
|
||||
// TestNewRcloneStorerConstruction checks that a valid remote constructs a
|
||||
// backend and that Info() reports the shaped "remote:path" location.
|
||||
//
|
||||
//nolint:paralleltest // NewRcloneStorer installs the process-global rclone config
|
||||
func TestNewRcloneStorerConstruction(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
s, err := storage.NewRcloneStorer(context.Background(), ":local", dir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewRcloneStorer: %v", err)
|
||||
}
|
||||
|
||||
// Info().Location is the "remote:path" string the adapter builds from
|
||||
// its two arguments, so asserting it confirms the argument shaping.
|
||||
want := ":local:" + dir
|
||||
if got := s.Info().Location; got != want {
|
||||
t.Errorf("Info().Location = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewRcloneStorerUnknownRemote checks that a remote that is not in the
|
||||
// rclone config fails construction with the ErrRemoteNotFound sentinel,
|
||||
// rather than silently returning a backend pointed nowhere.
|
||||
//
|
||||
//nolint:paralleltest // NewRcloneStorer installs the process-global rclone config
|
||||
func TestNewRcloneStorerUnknownRemote(t *testing.T) {
|
||||
_, err := storage.NewRcloneStorer(
|
||||
context.Background(), "vaultik-no-such-remote", "path")
|
||||
if !errors.Is(err, storage.ErrRemoteNotFound) {
|
||||
t.Errorf("NewRcloneStorer error = %v, want ErrRemoteNotFound", err)
|
||||
}
|
||||
}
|
||||
+16
-1
@@ -38,14 +38,29 @@ func (s *S3Storer) PutWithProgress(
|
||||
}
|
||||
|
||||
// Get retrieves data from the specified key.
|
||||
// Returns ErrNotFound if the object does not exist.
|
||||
func (s *S3Storer) Get(ctx context.Context, key string) (io.ReadCloser, error) {
|
||||
return s.client.GetObject(ctx, key)
|
||||
rc, err := s.client.GetObject(ctx, key)
|
||||
if err != nil {
|
||||
if s3.IsNotFound(err) {
|
||||
return nil, fmt.Errorf("get %q: %w", key, ErrNotFound)
|
||||
}
|
||||
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return rc, nil
|
||||
}
|
||||
|
||||
// Stat returns metadata about an object without retrieving its contents.
|
||||
// Returns ErrNotFound if the object does not exist.
|
||||
func (s *S3Storer) Stat(ctx context.Context, key string) (*ObjectInfo, error) {
|
||||
info, err := s.client.StatObject(ctx, key)
|
||||
if err != nil {
|
||||
if s3.IsNotFound(err) {
|
||||
return nil, fmt.Errorf("stat %q: %w", key, ErrNotFound)
|
||||
}
|
||||
|
||||
return nil, err
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/johannesboyne/gofakes3"
|
||||
"github.com/johannesboyne/gofakes3/backend/s3mem"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/s3"
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
)
|
||||
|
||||
// s3TestBucket is the bucket created for each in-process S3 server.
|
||||
const s3TestBucket = "test-bucket"
|
||||
|
||||
// newS3Storer builds an s3:// backend backed by a fresh in-process
|
||||
// S3 server. It reuses the same in-memory S3 harness (gofakes3 + s3mem
|
||||
// over httptest) that internal/s3 and the not-found regression test use,
|
||||
// so no new mock or dependency is introduced. Each call gets its own
|
||||
// server, bucket, and client, so the conformance suite's per-section
|
||||
// instances stay isolated.
|
||||
//
|
||||
//nolint:ireturn // conformance runs against the Storer interface by design
|
||||
func newS3Storer(t *testing.T) storage.Storer {
|
||||
t.Helper()
|
||||
|
||||
backend := s3mem.New()
|
||||
|
||||
err := backend.CreateBucket(s3TestBucket)
|
||||
if err != nil {
|
||||
t.Fatalf("create bucket: %v", err)
|
||||
}
|
||||
|
||||
srv := httptest.NewServer(gofakes3.New(backend).Server())
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
client, err := s3.NewClient(context.Background(), s3.Config{
|
||||
Endpoint: srv.URL,
|
||||
Bucket: s3TestBucket,
|
||||
AccessKeyID: "test",
|
||||
SecretAccessKey: "test",
|
||||
Region: "us-east-1",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("new client: %v", err)
|
||||
}
|
||||
|
||||
return storage.NewS3Storer(client)
|
||||
}
|
||||
|
||||
// TestS3Storer runs the shared Storer contract against the s3:// backend,
|
||||
// so it is held to the same round-trip, list, delete, and not-found
|
||||
// behaviour as the file:// backend.
|
||||
func TestS3Storer(t *testing.T) {
|
||||
t.Parallel()
|
||||
runStorerConformance(t, newS3Storer)
|
||||
}
|
||||
|
||||
// TestS3StorerMissingKeyMapsToErrNotFound pins the specific contract that a
|
||||
// missing object surfaces as storage.ErrNotFound rather than the raw AWS SDK
|
||||
// error. Without the mapping, errors.Is(err, storage.ErrNotFound) is false on
|
||||
// s3 and callers would branch differently per backend.
|
||||
func TestS3StorerMissingKeyMapsToErrNotFound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
storer := newS3Storer(t)
|
||||
ctx := context.Background()
|
||||
|
||||
_, err := storer.Get(ctx, "does-not-exist")
|
||||
if !errors.Is(err, storage.ErrNotFound) {
|
||||
t.Errorf("Get on missing key: got %v, want ErrNotFound", err)
|
||||
}
|
||||
|
||||
_, err = storer.Stat(ctx, "does-not-exist")
|
||||
if !errors.Is(err, storage.ErrNotFound) {
|
||||
t.Errorf("Stat on missing key: got %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
)
|
||||
|
||||
// TestParseStorageURLValid checks that each supported scheme parses into
|
||||
// the expected fields, since those fields decide which backend is built.
|
||||
func TestParseStorageURLValid(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const bucket = "mybucket"
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
raw string
|
||||
want *storage.URL
|
||||
}{
|
||||
{
|
||||
name: "file absolute path",
|
||||
raw: "file:///var/backups/vaultik",
|
||||
want: &storage.URL{Scheme: "file", Prefix: "/var/backups/vaultik"},
|
||||
},
|
||||
{
|
||||
name: "s3 bucket and prefix, ssl defaults on",
|
||||
raw: "s3://mybucket/backups/host",
|
||||
want: &storage.URL{
|
||||
Scheme: "s3", Bucket: bucket,
|
||||
Prefix: "backups/host", UseSSL: true,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "s3 bucket only",
|
||||
raw: "s3://mybucket",
|
||||
want: &storage.URL{Scheme: "s3", Bucket: bucket, UseSSL: true},
|
||||
},
|
||||
{
|
||||
name: "s3 with endpoint, region, ssl off",
|
||||
raw: "s3://mybucket?endpoint=minio.example.com®ion=us-west-2&ssl=false",
|
||||
want: &storage.URL{
|
||||
Scheme: "s3", Bucket: bucket,
|
||||
Endpoint: "minio.example.com", Region: "us-west-2", UseSSL: false,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "rclone remote and path",
|
||||
raw: "rclone://gdrive/backups/host",
|
||||
want: &storage.URL{
|
||||
Scheme: "rclone", RcloneRemote: "gdrive", Prefix: "backups/host",
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "rclone remote only",
|
||||
raw: "rclone://gdrive",
|
||||
want: &storage.URL{Scheme: "rclone", RcloneRemote: "gdrive"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, err := storage.ParseStorageURL(tc.raw)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseStorageURL(%q) returned error: %v", tc.raw, err)
|
||||
}
|
||||
|
||||
if !reflect.DeepEqual(got, tc.want) {
|
||||
t.Errorf("ParseStorageURL(%q) = %+v, want %+v", tc.raw, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestParseStorageURLErrors checks that empty, missing, and unknown-scheme
|
||||
// inputs fail with the documented sentinel errors instead of parsing to a
|
||||
// wrong destination.
|
||||
func TestParseStorageURLErrors(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
raw string
|
||||
wantErr error
|
||||
}{
|
||||
{"empty url", "", storage.ErrEmptyStorageURL},
|
||||
{"file empty path", "file://", storage.ErrEmptyFilePath},
|
||||
{"s3 missing bucket", "s3://", storage.ErrMissingBucket},
|
||||
{"s3 missing bucket with path", "s3:///justprefix", storage.ErrMissingBucket},
|
||||
{"rclone missing remote", "rclone://", storage.ErrMissingRemote},
|
||||
{"unknown scheme", "gs://bucket/x", storage.ErrUnsupportedScheme},
|
||||
{"no scheme", "/local/path", storage.ErrUnsupportedScheme},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := storage.ParseStorageURL(tc.raw)
|
||||
if !errors.Is(err, tc.wantErr) {
|
||||
t.Errorf("ParseStorageURL(%q) error = %v, want %v",
|
||||
tc.raw, err, tc.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
package vaultik_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/spf13/afero"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
"sneak.berlin/go/vaultik/internal/ui"
|
||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||
)
|
||||
|
||||
// TestDeepVerifyAcceptsHealthyAndRejectsCorruptBlob backs up a real
|
||||
// snapshot with the on-disk storage backend, runs deep verification on
|
||||
// it, then flips a byte inside one stored blob and runs deep
|
||||
// verification again. A healthy snapshot must pass; a corrupted blob
|
||||
// must fail. The healthy case is the regression guard: deep
|
||||
// verification used to hash the encrypted blob bytes and compare them
|
||||
// to the blob's ID (the double SHA256 of the plaintext), so it reported
|
||||
// every healthy blob as corrupt.
|
||||
func TestDeepVerifyAcceptsHealthyAndRejectsCorruptBlob(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
|
||||
dataDir := filepath.Join(tempDir, "source")
|
||||
storeDir := filepath.Join(tempDir, "remote")
|
||||
dbPath := filepath.Join(tempDir, "index.sqlite")
|
||||
|
||||
chunkSize := int64(64 * 1024)
|
||||
maxBlobSize := int64(512 * 1024)
|
||||
|
||||
// One file large enough to span several chunks within a single blob.
|
||||
require.NoError(t, fs.MkdirAll(dataDir, 0o755))
|
||||
require.NoError(t, afero.WriteFile(fs,
|
||||
filepath.Join(dataDir, "data.bin"),
|
||||
bytesPattern("deep-", int(chunkSize*3)), 0o644))
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// runFileStorageBackup writes a real snapshot to storeDir and closes
|
||||
// the source index, so verification runs from remote bytes only.
|
||||
cfg, storer, snapshotID := runFileStorageBackup(
|
||||
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
|
||||
|
||||
newVerifier := func() *vaultik.Vaultik {
|
||||
v := &vaultik.Vaultik{
|
||||
Config: cfg,
|
||||
Storage: storer,
|
||||
Fs: fs,
|
||||
Stdout: io.Discard,
|
||||
Stderr: io.Discard,
|
||||
UI: ui.NewWithColor(io.Discard, false),
|
||||
}
|
||||
v.SetContext(ctx)
|
||||
|
||||
return v
|
||||
}
|
||||
|
||||
require.NoError(t,
|
||||
newVerifier().RunDeepVerify(snapshotID, &vaultik.VerifyOptions{Deep: true}),
|
||||
"deep verify should pass on a healthy snapshot")
|
||||
|
||||
// Flip a byte inside one blob without changing its length, so the
|
||||
// blob-existence and size checks still pass and verification reaches
|
||||
// the blob-content stage.
|
||||
corruptOneBlob(t, fs, filepath.Join(storeDir, "blobs"))
|
||||
|
||||
require.Error(t,
|
||||
newVerifier().RunDeepVerify(snapshotID, &vaultik.VerifyOptions{Deep: true}),
|
||||
"deep verify should fail on a corrupted blob")
|
||||
}
|
||||
|
||||
// corruptOneBlob flips a middle byte of the first blob file found under
|
||||
// blobsDir, leaving the file length unchanged.
|
||||
func corruptOneBlob(t *testing.T, fs afero.Fs, blobsDir string) {
|
||||
t.Helper()
|
||||
|
||||
var blobPath string
|
||||
|
||||
err := afero.Walk(fs, blobsDir,
|
||||
func(path string, info os.FileInfo, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if blobPath == "" && !info.IsDir() {
|
||||
blobPath = path
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, blobPath, "expected at least one blob on disk")
|
||||
|
||||
data, err := afero.ReadFile(fs, blobPath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, data)
|
||||
|
||||
data[len(data)/2] ^= 0xff
|
||||
require.NoError(t, afero.WriteFile(fs, blobPath, data, 0o644))
|
||||
}
|
||||
@@ -33,8 +33,9 @@ func ubytes(n int64) string {
|
||||
var (
|
||||
errMalformedSnapshotID = errors.New(
|
||||
"invalid snapshot ID format: expected hostname_snapshotname_timestamp")
|
||||
errInvalidDuration = errors.New("invalid duration")
|
||||
errUnknownTimeUnit = errors.New("unknown time unit")
|
||||
errInvalidDuration = errors.New("invalid duration")
|
||||
errUnknownTimeUnit = errors.New("unknown time unit")
|
||||
errNegativeDuration = errors.New("negative durations are not supported")
|
||||
)
|
||||
|
||||
// Time-unit lengths used by parseDuration.
|
||||
@@ -138,8 +139,13 @@ func parseSnapshotName(snapshotID string) string {
|
||||
|
||||
// parseDuration parses a duration string with support for human-friendly units:
|
||||
// d/day/days, w/week/weeks, mo/month/months, y/year/years, plus standard Go
|
||||
// duration units (h, m, s).
|
||||
// duration units. Following Go, m is minutes and mo is months. A bare number,
|
||||
// an unknown unit, and a negative value are all rejected.
|
||||
func parseDuration(s string) (time.Duration, error) {
|
||||
if strings.HasPrefix(strings.TrimSpace(s), "-") {
|
||||
return 0, errNegativeDuration
|
||||
}
|
||||
|
||||
d, err := time.ParseDuration(s)
|
||||
if err == nil {
|
||||
return d, nil
|
||||
|
||||
@@ -51,13 +51,32 @@ func TestParseDuration(t *testing.T) {
|
||||
want time.Duration
|
||||
err bool
|
||||
}{
|
||||
{"30d", 30 * 24 * time.Hour, false},
|
||||
{"4w", 4 * 7 * 24 * time.Hour, false},
|
||||
{"6mo", 6 * 30 * 24 * time.Hour, false},
|
||||
{"1y", 365 * 24 * time.Hour, false},
|
||||
{"2w3d", 2*7*24*time.Hour + 3*24*time.Hour, false},
|
||||
{"1h", time.Hour, false},
|
||||
// Go units, including the m-is-minutes / mo-is-months distinction
|
||||
// that this parser exists to keep straight.
|
||||
{"10ns", 10 * time.Nanosecond, false},
|
||||
{"10us", 10 * time.Microsecond, false},
|
||||
{"500ms", 500 * time.Millisecond, false},
|
||||
{"30s", 30 * time.Second, false},
|
||||
{"6m", 6 * time.Minute, false},
|
||||
{"1h", time.Hour, false},
|
||||
// Extended calendar units.
|
||||
{"30d", 30 * 24 * time.Hour, false},
|
||||
{"3days", 3 * 24 * time.Hour, false},
|
||||
{"4w", 4 * 7 * 24 * time.Hour, false},
|
||||
{"2weeks", 2 * 7 * 24 * time.Hour, false},
|
||||
{"6mo", 180 * 24 * time.Hour, false},
|
||||
{"1month", 30 * 24 * time.Hour, false},
|
||||
{"1y", 365 * 24 * time.Hour, false},
|
||||
{"2years", 2 * 365 * 24 * time.Hour, false},
|
||||
// Combined units.
|
||||
{"2w3d", 2*7*24*time.Hour + 3*24*time.Hour, false},
|
||||
{"1y6mo", 365*24*time.Hour + 180*24*time.Hour, false},
|
||||
// Rejected inputs.
|
||||
{"6", 0, true}, // bare number, no unit
|
||||
{"5x", 0, true}, // unknown unit
|
||||
{"-5d", 0, true}, // negative, extended unit
|
||||
{"-5h", 0, true}, // negative, Go unit
|
||||
{"", 0, true}, // empty
|
||||
{"garbage", 0, true},
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
package vaultik //nolint:testpackage // exercises unexported count helpers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/database"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
)
|
||||
|
||||
// TestTableCountForReportSurfacesReadFailure is the regression guard for
|
||||
// the discarded-error bug: getTableCount for a table its query cannot
|
||||
// resolve must not silently become 0. A count that could not be read is
|
||||
// reported as unknown, which a reader can tell apart from an empty table.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestTableCountForReportSurfacesReadFailure(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
db, err := database.New(ctx, ":memory:")
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
v := &Vaultik{DB: db}
|
||||
v.SetContext(ctx)
|
||||
|
||||
// A table present in the schema reads as a real count.
|
||||
blobs := v.tableCountForReport("blobs")
|
||||
require.NotNil(t, blobs, "an existing table must read as a real count")
|
||||
assert.Equal(t, int64(0), *blobs)
|
||||
|
||||
// A syntactically valid name the sanitizer accepts but whose table
|
||||
// the query cannot resolve is the exact shape #96 describes: a
|
||||
// would-be loud failure that used to be discarded into a 0.
|
||||
_, err = v.getTableCount("snapshots_missing")
|
||||
require.Error(t, err, "a query against a nonexistent table must fail")
|
||||
|
||||
missing := v.tableCountForReport("snapshots_missing")
|
||||
assert.Nil(t, missing, "a failed read is unknown, not a count")
|
||||
|
||||
// The rendered count for a failed read must say unknown, never 0.
|
||||
assert.Equal(t, countUnknown, countText(missing))
|
||||
assert.NotEqual(t, "0", countText(missing))
|
||||
}
|
||||
|
||||
// TestCountTextDistinguishesEmptyFromUnknown pins the distinction the
|
||||
// output has to preserve: 0 means the table was empty, "unknown" means
|
||||
// the count could not be read.
|
||||
func TestCountTextDistinguishesEmptyFromUnknown(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
zero := int64(0)
|
||||
seven := int64(7)
|
||||
|
||||
assert.Equal(t, "0", countText(&zero))
|
||||
assert.Equal(t, "7", countText(&seven))
|
||||
assert.Equal(t, countUnknown, countText(nil))
|
||||
}
|
||||
|
||||
// TestCountDiffUnknownWhenEitherSideUnknown checks that a delta computed
|
||||
// from an unreadable count is itself unknown rather than a plausible
|
||||
// number.
|
||||
func TestCountDiffUnknownWhenEitherSideUnknown(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
before := int64(10)
|
||||
after := int64(3)
|
||||
|
||||
require.NotNil(t, countDiff(&before, &after))
|
||||
assert.Equal(t, int64(7), *countDiff(&before, &after))
|
||||
|
||||
assert.Nil(t, countDiff(nil, &after), "unknown before yields unknown delta")
|
||||
assert.Nil(t, countDiff(&before, nil), "unknown after yields unknown delta")
|
||||
assert.Nil(t, countDiff(nil, nil))
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -1540,12 +1541,17 @@ func (v *Vaultik) outputRemoveJSON(result *RemoveResult) error {
|
||||
return encoder.Encode(result)
|
||||
}
|
||||
|
||||
// PruneResult contains statistics about the prune operation
|
||||
// PruneResult contains statistics about the prune operation.
|
||||
// SnapshotsDeleted counts snapshots actually deleted. FilesDeleted,
|
||||
// ChunksDeleted, and BlobsDeleted are derived from before/after row
|
||||
// counts of the local index; each is nil when a count could not be read,
|
||||
// so an unreadable count is reported as unknown rather than silently
|
||||
// as 0.
|
||||
type PruneResult struct {
|
||||
SnapshotsDeleted int64
|
||||
FilesDeleted int64
|
||||
ChunksDeleted int64
|
||||
BlobsDeleted int64
|
||||
FilesDeleted *int64
|
||||
ChunksDeleted *int64
|
||||
BlobsDeleted *int64
|
||||
}
|
||||
|
||||
// PruneDatabase removes incomplete snapshots and orphaned files, chunks,
|
||||
@@ -1560,7 +1566,7 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
|
||||
result := &PruneResult{}
|
||||
|
||||
// Snapshot counts before deletion of incompletes.
|
||||
snapshotCountBefore, _ := v.getTableCount("snapshots")
|
||||
snapshotCountBefore := v.tableCountForReport("snapshots")
|
||||
|
||||
// First, delete any incomplete snapshots
|
||||
incompleteSnapshots, err := v.Repositories.Snapshots.GetIncompleteSnapshots(v.ctx)
|
||||
@@ -1575,9 +1581,9 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
|
||||
}
|
||||
|
||||
// Get counts before cleanup for reporting
|
||||
fileCountBefore, _ := v.getTableCount("files")
|
||||
chunkCountBefore, _ := v.getTableCount("chunks")
|
||||
blobCountBefore, _ := v.getTableCount("blobs")
|
||||
fileCountBefore := v.tableCountForReport("files")
|
||||
chunkCountBefore := v.tableCountForReport("chunks")
|
||||
blobCountBefore := v.tableCountForReport("blobs")
|
||||
|
||||
// Run the cleanup
|
||||
err = v.SnapshotManager.CleanupOrphanedData(v.ctx)
|
||||
@@ -1586,36 +1592,83 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
|
||||
}
|
||||
|
||||
// Get counts after cleanup
|
||||
fileCountAfter, _ := v.getTableCount("files")
|
||||
chunkCountAfter, _ := v.getTableCount("chunks")
|
||||
blobCountAfter, _ := v.getTableCount("blobs")
|
||||
fileCountAfter := v.tableCountForReport("files")
|
||||
chunkCountAfter := v.tableCountForReport("chunks")
|
||||
blobCountAfter := v.tableCountForReport("blobs")
|
||||
|
||||
result.FilesDeleted = fileCountBefore - fileCountAfter
|
||||
result.ChunksDeleted = chunkCountBefore - chunkCountAfter
|
||||
result.BlobsDeleted = blobCountBefore - blobCountAfter
|
||||
result.FilesDeleted = countDiff(fileCountBefore, fileCountAfter)
|
||||
result.ChunksDeleted = countDiff(chunkCountBefore, chunkCountAfter)
|
||||
result.BlobsDeleted = countDiff(blobCountBefore, blobCountAfter)
|
||||
|
||||
log.Info("Local database prune complete",
|
||||
"incomplete_snapshots", result.SnapshotsDeleted,
|
||||
"orphaned_files", result.FilesDeleted,
|
||||
"orphaned_chunks", result.ChunksDeleted,
|
||||
"orphaned_blobs", result.BlobsDeleted,
|
||||
"orphaned_files", countText(result.FilesDeleted),
|
||||
"orphaned_chunks", countText(result.ChunksDeleted),
|
||||
"orphaned_blobs", countText(result.BlobsDeleted),
|
||||
)
|
||||
|
||||
snapshotCountAfter := snapshotCountBefore - result.SnapshotsDeleted
|
||||
// Snapshots remaining after removing the incomplete ones; unknown if
|
||||
// the pre-prune snapshot count could not be read.
|
||||
snapshotsRemain := countDiff(snapshotCountBefore, &result.SnapshotsDeleted)
|
||||
|
||||
v.UI.Completef("Pruned local index database.")
|
||||
v.UI.Detailf("Incomplete snapshots: %d removed (%d remain).",
|
||||
result.SnapshotsDeleted, snapshotCountAfter)
|
||||
v.UI.Detailf("Orphaned files: %d removed (%d remain).",
|
||||
result.FilesDeleted, fileCountAfter)
|
||||
v.UI.Detailf("Orphaned chunks: %d removed (%d remain).",
|
||||
result.ChunksDeleted, chunkCountAfter)
|
||||
v.UI.Detailf("Orphaned blobs: %d removed (%d remain).",
|
||||
result.BlobsDeleted, blobCountAfter)
|
||||
v.UI.Detailf("Incomplete snapshots: %s removed (%s remain).",
|
||||
countText(&result.SnapshotsDeleted), countText(snapshotsRemain))
|
||||
v.UI.Detailf("Orphaned files: %s removed (%s remain).",
|
||||
countText(result.FilesDeleted), countText(fileCountAfter))
|
||||
v.UI.Detailf("Orphaned chunks: %s removed (%s remain).",
|
||||
countText(result.ChunksDeleted), countText(chunkCountAfter))
|
||||
v.UI.Detailf("Orphaned blobs: %s removed (%s remain).",
|
||||
countText(result.BlobsDeleted), countText(blobCountAfter))
|
||||
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// countUnknown is what a count reads as when its query could not be run,
|
||||
// distinct from "0", which means the table really was empty.
|
||||
const countUnknown = "unknown"
|
||||
|
||||
// tableCountForReport returns the row count of a table for the prune
|
||||
// summary, or nil if the count could not be read. A read failure is
|
||||
// logged at warn — visible even under --json, which routes warnings to
|
||||
// stderr — and then rendered as unknown rather than silently becoming 0,
|
||||
// so a broken query is a visible failure instead of a plausible wrong
|
||||
// number.
|
||||
func (v *Vaultik) tableCountForReport(tableName string) *int64 {
|
||||
count, err := v.getTableCount(tableName)
|
||||
if err != nil {
|
||||
log.Warn("could not read table row count for prune summary",
|
||||
"table", tableName, "error", err)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
return &count
|
||||
}
|
||||
|
||||
// countDiff returns before-after, or nil if either count is unknown so
|
||||
// that an unreadable count does not collapse into a plausible delta.
|
||||
func countDiff(before, after *int64) *int64 {
|
||||
if before == nil || after == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
diff := *before - *after
|
||||
|
||||
return &diff
|
||||
}
|
||||
|
||||
// countText renders a count that may be unknown: nil (the read failed)
|
||||
// becomes "unknown", never "0", so a reader can tell an empty table from
|
||||
// one that could not be queried.
|
||||
func countText(count *int64) string {
|
||||
if count == nil {
|
||||
return countUnknown
|
||||
}
|
||||
|
||||
return strconv.FormatInt(*count, 10)
|
||||
}
|
||||
|
||||
// validTableNameRe matches table names containing only lowercase
|
||||
// alphanumeric characters and underscores.
|
||||
var validTableNameRe = regexp.MustCompile(`^[a-z0-9_]+$`)
|
||||
|
||||
+19
-14
@@ -344,12 +344,8 @@ func (v *Vaultik) verifyBlob(blobInfo snapshot.BlobInfo, db *sql.DB) error {
|
||||
return fmt.Errorf("failed to get decryptor: %w", err)
|
||||
}
|
||||
|
||||
// Hash the encrypted blob data as it streams through to decryption
|
||||
blobHasher := sha256.New()
|
||||
teeReader := io.TeeReader(reader, blobHasher)
|
||||
|
||||
// Decrypt blob (reading through teeReader to hash encrypted data)
|
||||
decryptedReader, err := decryptor.DecryptStream(teeReader)
|
||||
// Decrypt blob
|
||||
decryptedReader, err := decryptor.DecryptStream(reader)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to decrypt: %w", err)
|
||||
}
|
||||
@@ -361,12 +357,19 @@ func (v *Vaultik) verifyBlob(blobInfo snapshot.BlobInfo, db *sql.DB) error {
|
||||
}
|
||||
defer decompressor.Close()
|
||||
|
||||
chunkCount, err := v.verifyBlobChunks(db, blobInfo.Hash, decompressor)
|
||||
// A blob's hash — its remote name — is the double SHA256 of its
|
||||
// decompressed plaintext (see blobgen.Writer.Sum256), not of the
|
||||
// encrypted bytes. Hash the plaintext as chunk verification streams
|
||||
// it, then compare on completion.
|
||||
plaintextHasher := sha256.New()
|
||||
hashedStream := io.TeeReader(decompressor, plaintextHasher)
|
||||
|
||||
chunkCount, err := v.verifyBlobChunks(db, blobInfo.Hash, hashedStream)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = v.verifyBlobFinalIntegrity(decompressor, blobHasher, blobInfo.Hash)
|
||||
err = v.verifyBlobFinalIntegrity(hashedStream, plaintextHasher, blobInfo.Hash)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -470,14 +473,13 @@ func (v *Vaultik) verifyBlobChunks(
|
||||
}
|
||||
|
||||
// verifyBlobFinalIntegrity checks that no trailing data exists in the
|
||||
// decompressed stream and that the encrypted blob hash matches the
|
||||
// expected value.
|
||||
// decompressed stream and that the blob hash matches the expected value.
|
||||
func (v *Vaultik) verifyBlobFinalIntegrity(
|
||||
decompressor io.Reader, blobHasher hash.Hash, expectedHash string,
|
||||
plaintext io.Reader, plaintextHasher hash.Hash, expectedHash string,
|
||||
) error {
|
||||
// Verify no remaining data in blob - if the chunk list is accurate,
|
||||
// the blob should be fully consumed.
|
||||
remaining, err := io.Copy(io.Discard, decompressor)
|
||||
remaining, err := io.Copy(io.Discard, plaintext)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to check for remaining blob data: %w", err)
|
||||
}
|
||||
@@ -486,8 +488,11 @@ func (v *Vaultik) verifyBlobFinalIntegrity(
|
||||
return fmt.Errorf("%w: %d bytes", errTrailingBlobData, remaining)
|
||||
}
|
||||
|
||||
// Verify blob hash matches the encrypted data we downloaded
|
||||
calculatedBlobHash := hex.EncodeToString(blobHasher.Sum(nil))
|
||||
// The blob hash is the double SHA256 of its plaintext content.
|
||||
firstHash := plaintextHasher.Sum(nil)
|
||||
secondHash := sha256.Sum256(firstHash)
|
||||
calculatedBlobHash := hex.EncodeToString(secondHash[:])
|
||||
|
||||
if calculatedBlobHash != expectedHash {
|
||||
return fmt.Errorf("%w: calculated %s, expected %s",
|
||||
errBlobHashMismatch, calculatedBlobHash, expectedHash)
|
||||
|
||||
Reference in New Issue
Block a user