Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a99f9877d6 |
+24
-3
@@ -20,8 +20,6 @@
|
|||||||
# golang:1.26.1-alpine, 2026-03-17
|
# golang:1.26.1-alpine, 2026-03-17
|
||||||
FROM golang:1.26.1-alpine@sha256:2389ebfa5b7f43eeafbd6be0c3700cc46690ef842ad962f6c5bd6be49ed82039 AS builder
|
FROM golang:1.26.1-alpine@sha256:2389ebfa5b7f43eeafbd6be0c3700cc46690ef842ad962f6c5bd6be49ed82039 AS builder
|
||||||
|
|
||||||
ARG VERSION=dev
|
|
||||||
|
|
||||||
# Build tooling: make, plus a C toolchain because `go test -race` needs cgo.
|
# Build tooling: make, plus a C toolchain because `go test -race` needs cgo.
|
||||||
# The sqlite driver is pure Go (modernc.org/sqlite), so no sqlite library or
|
# The sqlite driver is pure Go (modernc.org/sqlite), so no sqlite library or
|
||||||
# CLI is required.
|
# CLI is required.
|
||||||
@@ -66,8 +64,31 @@ RUN [ -n "$CHECK_EPOCH" ] || exit 1
|
|||||||
RUN echo "check epoch: ${CHECK_EPOCH}" && make fmt-check
|
RUN echo "check epoch: ${CHECK_EPOCH}" && make fmt-check
|
||||||
RUN echo "check epoch: ${CHECK_EPOCH}" && make test
|
RUN echo "check epoch: ${CHECK_EPOCH}" && make test
|
||||||
|
|
||||||
|
# Version, commit and build date are computed on the host by
|
||||||
|
# script/docker and script/cibuild (where .git exists) and passed in as
|
||||||
|
# build args. The build context excludes .git (see .dockerignore), so
|
||||||
|
# the build cannot derive them itself: it used to try, with `git
|
||||||
|
# rev-parse` inside this stage, and always got "unknown". VERSION comes
|
||||||
|
# from script/version, the source of truth shared with the Makefile, so
|
||||||
|
# it carries the same tag / dev-<sha> / -dirty rules and a Docker image
|
||||||
|
# reports the same string a local build of the same tree would.
|
||||||
|
#
|
||||||
|
# The defaults are the fallback for a bare `docker build .` that passes
|
||||||
|
# none of them: an unset arg would otherwise stamp an empty string and
|
||||||
|
# produce an image that cannot report its own version, commit or date.
|
||||||
|
# They match what an out-of-git build reports elsewhere.
|
||||||
|
#
|
||||||
|
# These ARGs sit here, after the checks, rather than at the top of the
|
||||||
|
# stage: every commit changes their values, and a value change
|
||||||
|
# invalidates all layers below the ARG. Declared up top they would bust
|
||||||
|
# `go mod download`; here they only rekey this build layer, which the
|
||||||
|
# COPY of the sources above already rebuilds on any change anyway.
|
||||||
|
ARG VERSION=dev
|
||||||
|
ARG COMMIT=unknown
|
||||||
|
ARG COMMIT_DATE=unknown
|
||||||
|
|
||||||
# Build (pure Go, no CGO required since we use modernc.org/sqlite)
|
# Build (pure Go, no CGO required since we use modernc.org/sqlite)
|
||||||
RUN CGO_ENABLED=0 go build -ldflags "-X 'sneak.berlin/go/vaultik/internal/globals.Version=${VERSION}' -X 'sneak.berlin/go/vaultik/internal/globals.Commit=$(git rev-parse HEAD 2>/dev/null || echo unknown)' -X 'sneak.berlin/go/vaultik/internal/globals.CommitDate=$(git show -s --format=%cs HEAD 2>/dev/null || echo unknown)'" -o /vaultik ./cmd/vaultik
|
RUN CGO_ENABLED=0 go build -ldflags "-X 'sneak.berlin/go/vaultik/internal/globals.Version=${VERSION}' -X 'sneak.berlin/go/vaultik/internal/globals.Commit=${COMMIT}' -X 'sneak.berlin/go/vaultik/internal/globals.CommitDate=${COMMIT_DATE}'" -o /vaultik ./cmd/vaultik
|
||||||
|
|
||||||
# Runtime stage
|
# Runtime stage
|
||||||
# alpine:3.21, 2026-02-25
|
# alpine:3.21, 2026-02-25
|
||||||
|
|||||||
@@ -25,27 +25,6 @@ release" is exactly the contradiction
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-09-21: Stopped `prune` from reporting a failed row count as 0
|
|
||||||
([issue #96](https://git.eeqj.de/sneak/vaultik/issues/96)). The seven
|
|
||||||
`getTableCount` reads in `PruneDatabase` discarded their error, so a
|
|
||||||
query that could not run became a plausible `0` and the before/after
|
|
||||||
delta computed from it looked like real work. Each read now logs at
|
|
||||||
warn on failure and renders as `unknown`, never `0`, so an empty table
|
|
||||||
is distinguishable from one that could not be queried. The counts have
|
|
||||||
no `--json` representation — under `--json` the summary is suppressed
|
|
||||||
entirely — so nothing there can show a false `0`.
|
|
||||||
|
|
||||||
- 2026-09-21: Stopped `--json` from silencing stderr diagnostics
|
|
||||||
([issue #112](https://git.eeqj.de/sneak/vaultik/issues/112)). `--json`
|
|
||||||
used to be folded into `Quiet`, which pinned the log level to `WARN`,
|
|
||||||
so `prune --json` gave a machine consumer no record of the local index
|
|
||||||
rows it deleted even under `--verbose`. `--json` now quiets only the
|
|
||||||
stdout UI (the JSON document must stay clean, per
|
|
||||||
[issue #108](https://git.eeqj.de/sneak/vaultik/issues/108)); the stderr
|
|
||||||
log level follows `--verbose`/`--debug` again. The coupling was
|
|
||||||
removed the same way for `snapshot verify`, `snapshot remove`, and
|
|
||||||
`remote info`, which carried it for the same outdated reason.
|
|
||||||
|
|
||||||
- 2026-09-21: Made the s3 storage backend report a missing object as
|
- 2026-09-21: Made the s3 storage backend report a missing object as
|
||||||
`storage.ErrNotFound`, like the `file` and `rclone` backends and as the
|
`storage.ErrNotFound`, like the `file` and `rclone` backends and as the
|
||||||
`Storer` interface documents. `S3Storer.Get` and `Stat` returned the raw
|
`Storer` interface documents. `S3Storer.Get` and `Stat` returned the raw
|
||||||
@@ -54,6 +33,7 @@ release" is exactly the contradiction
|
|||||||
helper (reused by `HeadObject`) and a test that a missing key maps to
|
helper (reused by `HeadObject`) and a test that a missing key maps to
|
||||||
`ErrNotFound`
|
`ErrNotFound`
|
||||||
([issue #129](https://git.eeqj.de/sneak/vaultik/issues/129)).
|
([issue #129](https://git.eeqj.de/sneak/vaultik/issues/129)).
|
||||||
|
|
||||||
- 2026-09-21: Fixed `verify --deep` reporting healthy snapshots as
|
- 2026-09-21: Fixed `verify --deep` reporting healthy snapshots as
|
||||||
corrupt. Its final blob-integrity check hashed the encrypted
|
corrupt. Its final blob-integrity check hashed the encrypted
|
||||||
downloaded bytes with a single SHA256 and compared that to the blob
|
downloaded bytes with a single SHA256 and compared that to the blob
|
||||||
|
|||||||
@@ -0,0 +1,102 @@
|
|||||||
|
package main_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// This file guards the version stamping of the product image (issue
|
||||||
|
// #75). The failure it protects against is silent: the image still
|
||||||
|
// builds and runs, but `vaultik version` inside it reports "commit:
|
||||||
|
// unknown", so an operator cannot tell which source produced a given
|
||||||
|
// backup. .dockerignore excludes .git, so the build cannot derive the
|
||||||
|
// commit itself; the values must be computed on the host and passed in.
|
||||||
|
//
|
||||||
|
// These are parses of the committed files, for the same reason the lint
|
||||||
|
// guards next door are: shelling out to docker would nest a build
|
||||||
|
// inside `make test`. That `vaultik version` in the built image really
|
||||||
|
// prints the host's version is verified by hand and recorded on the
|
||||||
|
// pull request.
|
||||||
|
|
||||||
|
// dockerScript is script/docker, relative to the repository root.
|
||||||
|
const dockerScript = "script/docker"
|
||||||
|
|
||||||
|
// versionArgs are the ldflag targets the build stamps and, matching
|
||||||
|
// them, the build args the host must supply. The names line up so the
|
||||||
|
// same list checks both files.
|
||||||
|
func versionArgs() []string {
|
||||||
|
return []string{"VERSION", "COMMIT", "COMMIT_DATE"}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestProductDockerfileTakesVersionAsBuildArgs fails unless the build
|
||||||
|
// declares each version arg and stamps it into the binary by ldflag
|
||||||
|
// reference, rather than computing it in the container.
|
||||||
|
func TestProductDockerfileTakesVersionAsBuildArgs(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
found := instructions(t, productDockerfile)
|
||||||
|
|
||||||
|
for _, arg := range versionArgs() {
|
||||||
|
require.GreaterOrEqual(t, indexOf(found, "ARG "+arg), 0,
|
||||||
|
"%s must declare `ARG %s` so the host can pass it in",
|
||||||
|
productDockerfile, arg)
|
||||||
|
|
||||||
|
assertLdflagReferences(t, found, arg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestProductDockerfileDoesNotDeriveVersionItself is the anti-regression
|
||||||
|
// for the original defect: the container ran `git rev-parse`, but .git
|
||||||
|
// is not in the build context, so it always resolved to "unknown". No
|
||||||
|
// git command may reach into a build that cannot see the history.
|
||||||
|
func TestProductDockerfileDoesNotDeriveVersionItself(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
text := instructionText(readRepoFile(t, productDockerfile))
|
||||||
|
|
||||||
|
assert.NotContains(t, text, "git ",
|
||||||
|
"%s must not run git: .git is excluded from the build context, so"+
|
||||||
|
" any value it derives is wrong. Pass version, commit and date"+
|
||||||
|
" in as build args instead.", productDockerfile)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestDockerScriptComputesVersionOnTheHost fails unless script/docker
|
||||||
|
// derives each value where .git exists and passes it as a build arg,
|
||||||
|
// with VERSION coming from script/version so a Docker build reports the
|
||||||
|
// same string a local build of the same tree would.
|
||||||
|
func TestDockerScriptComputesVersionOnTheHost(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
script := readRepoFile(t, dockerScript)
|
||||||
|
|
||||||
|
for _, arg := range versionArgs() {
|
||||||
|
assert.Contains(t, script, "--build-arg "+arg+"=",
|
||||||
|
"%s must pass --build-arg %s to the build", dockerScript, arg)
|
||||||
|
}
|
||||||
|
|
||||||
|
assert.Contains(t, script, "/version",
|
||||||
|
"%s must take VERSION from script/version, the source of truth"+
|
||||||
|
" shared with the Makefile", dockerScript)
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertLdflagReferences fails unless some build instruction stamps the
|
||||||
|
// named variable from the ARG (a ${arg} reference), not from a value
|
||||||
|
// computed inside the container.
|
||||||
|
func assertLdflagReferences(t *testing.T, found []string, arg string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
for _, instruction := range found {
|
||||||
|
if strings.HasPrefix(instruction, "RUN ") &&
|
||||||
|
strings.Contains(instruction, "go build") &&
|
||||||
|
strings.Contains(instruction, "${"+arg+"}") {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
assert.Fail(t, "version arg is declared but never stamped",
|
||||||
|
"the go build in %s must reference ${%s} in its ldflags, or the"+
|
||||||
|
" arg is passed and discarded", productDockerfile, arg)
|
||||||
|
}
|
||||||
@@ -304,10 +304,14 @@ func instructionText(contents string) string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// indexOf returns the position of the first instruction equal to, or
|
// indexOf returns the position of the first instruction equal to, or
|
||||||
// beginning with, want; -1 if there is none.
|
// beginning with, want; -1 if there is none. An `ARG NAME=default`
|
||||||
|
// counts as beginning with `ARG NAME`, so a declared arg is found
|
||||||
|
// whether or not it carries a default.
|
||||||
func indexOf(found []string, want string) int {
|
func indexOf(found []string, want string) int {
|
||||||
for i, instruction := range found {
|
for i, instruction := range found {
|
||||||
if instruction == want || strings.HasPrefix(instruction, want+" ") {
|
if instruction == want ||
|
||||||
|
strings.HasPrefix(instruction, want+" ") ||
|
||||||
|
strings.HasPrefix(instruction, want+"=") {
|
||||||
return i
|
return i
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+11
-1
@@ -10,6 +10,16 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
|
os.Exit(run())
|
||||||
|
}
|
||||||
|
|
||||||
|
// run sets up optional profiling, runs the CLI, and returns the process
|
||||||
|
// exit code. os.Exit lives in main so it fires only after run's deferred
|
||||||
|
// profile writers have flushed. cli.Entry returns a status code rather
|
||||||
|
// than calling os.Exit itself: an os.Exit from inside it would skip
|
||||||
|
// these defers and truncate the profile of a failing command -- exactly
|
||||||
|
// the command one most often wants to profile.
|
||||||
|
func run() int {
|
||||||
// CPU profiling: set VAULTIK_CPUPROFILE=/path/to/cpu.prof
|
// CPU profiling: set VAULTIK_CPUPROFILE=/path/to/cpu.prof
|
||||||
if cpuProfile := os.Getenv("VAULTIK_CPUPROFILE"); cpuProfile != "" {
|
if cpuProfile := os.Getenv("VAULTIK_CPUPROFILE"); cpuProfile != "" {
|
||||||
f, err := os.Create(cpuProfile) //nolint:gosec // G304: operator-set path
|
f, err := os.Create(cpuProfile) //nolint:gosec // G304: operator-set path
|
||||||
@@ -46,5 +56,5 @@ func main() {
|
|||||||
}()
|
}()
|
||||||
}
|
}
|
||||||
|
|
||||||
cli.Entry()
|
return cli.Entry()
|
||||||
}
|
}
|
||||||
|
|||||||
+90
-46
@@ -11,6 +11,7 @@ import (
|
|||||||
"os/signal"
|
"os/signal"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
"syscall"
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -48,11 +49,6 @@ type AppOptions struct {
|
|||||||
// silenced — per the documented convention that --quiet suppresses
|
// silenced — per the documented convention that --quiet suppresses
|
||||||
// non-error output only. The startup banner is printed by Entry
|
// non-error output only. The startup banner is printed by Entry
|
||||||
// before cobra parses arguments, gated by the same arg-level check.
|
// before cobra parses arguments, gated by the same arg-level check.
|
||||||
//
|
|
||||||
// --json quiets the UI here too, because stdout then carries a JSON
|
|
||||||
// document and human narration would corrupt it. Unlike Quiet it does
|
|
||||||
// not lower the stderr log level (issue #112), so --verbose/--debug
|
|
||||||
// still surface diagnostics alongside the document.
|
|
||||||
func setupGlobals(
|
func setupGlobals(
|
||||||
lc fx.Lifecycle, g *globals.Globals, v *vaultik.Vaultik, opts log.Options,
|
lc fx.Lifecycle, g *globals.Globals, v *vaultik.Vaultik, opts log.Options,
|
||||||
) {
|
) {
|
||||||
@@ -60,7 +56,7 @@ func setupGlobals(
|
|||||||
OnStart: func(_ context.Context) error {
|
OnStart: func(_ context.Context) error {
|
||||||
g.StartTime = time.Now().UTC()
|
g.StartTime = time.Now().UTC()
|
||||||
|
|
||||||
if opts.Cron || opts.Quiet || opts.JSON {
|
if opts.Cron || opts.Quiet {
|
||||||
v.UI.SetQuiet(true)
|
v.UI.SetQuiet(true)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -201,54 +197,54 @@ func RunApp(ctx context.Context, app *fx.App) error {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// runVaultikApp runs the standard single-operation command lifecycle
|
// errReported marks a failure the operation has already shown the user
|
||||||
// shared by the list/purge/verify/remove/remote-info subcommands:
|
// (and deliberately withheld under --json). Entry turns it into a
|
||||||
// resolve the config, start the fx app, run op against the Vaultik
|
// non-zero exit status without printing anything further, so the error
|
||||||
// instance in a goroutine, report a failure prefixed with failMsg
|
// line is not doubled. It flows up from RunOperation through cobra to
|
||||||
// (suppressed while suppressErrors is true, e.g. under --json), then
|
// Entry.
|
||||||
// trigger shutdown. The operation is cancelled when the app stops.
|
var errReported = errors.New("operation failed")
|
||||||
// jsonOutput marks a command whose stdout is a JSON document: it quiets
|
|
||||||
// the UI but, unlike Quiet, leaves the stderr log level alone.
|
// RunOperation runs op against the Vaultik instance inside the fx app
|
||||||
func runVaultikApp(
|
// and turns a failure into a returned error rather than an os.Exit from
|
||||||
cmd *cobra.Command, jsonOutput, suppressErrors bool,
|
// within the goroutine. An os.Exit there skipped main's deferred
|
||||||
failMsg string, op func(v *vaultik.Vaultik) error,
|
// profile writers -- so profiling a failing command yielded a truncated
|
||||||
|
// profile (issue #75) -- and RunWithApp's PID-lock release, and denied
|
||||||
|
// the app any graceful shutdown; returning the error to the top runs
|
||||||
|
// all three.
|
||||||
|
//
|
||||||
|
// op runs in a goroutine so OnStart returns promptly and an interrupt
|
||||||
|
// can still cancel through OnStop; when it finishes, success or failure,
|
||||||
|
// it triggers shutdown, which is what lets RunWithApp return. report is
|
||||||
|
// called with a non-canceled failure so the caller can log it (and
|
||||||
|
// suppress it under --json) before it becomes errReported. A context
|
||||||
|
// cancellation is the interrupt path, not a failure: it is neither
|
||||||
|
// reported nor counted as one.
|
||||||
|
func RunOperation(
|
||||||
|
ctx context.Context, opts AppOptions,
|
||||||
|
op func(v *vaultik.Vaultik) error, report func(err error),
|
||||||
) error {
|
) error {
|
||||||
configPath, err := ResolveConfigPath()
|
var (
|
||||||
if err != nil {
|
mu sync.Mutex
|
||||||
return err
|
failed bool
|
||||||
}
|
)
|
||||||
|
|
||||||
rootFlags := GetRootFlags()
|
opts.Invokes = append(opts.Invokes,
|
||||||
|
|
||||||
return RunWithApp(cmd.Context(), AppOptions{
|
|
||||||
ConfigPath: configPath,
|
|
||||||
LogOptions: log.Options{
|
|
||||||
Verbose: rootFlags.Verbose,
|
|
||||||
Debug: rootFlags.Debug,
|
|
||||||
Quiet: rootFlags.Quiet,
|
|
||||||
JSON: jsonOutput,
|
|
||||||
},
|
|
||||||
Modules: []fx.Option{},
|
|
||||||
Invokes: []fx.Option{
|
|
||||||
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
|
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
|
||||||
lc.Append(fx.Hook{
|
lc.Append(fx.Hook{
|
||||||
OnStart: func(_ context.Context) error {
|
OnStart: func(_ context.Context) error {
|
||||||
go func() {
|
go func() {
|
||||||
err := op(v)
|
err := op(v)
|
||||||
if err != nil {
|
if err != nil && !errors.Is(err, context.Canceled) {
|
||||||
if !errors.Is(err, context.Canceled) {
|
report(err)
|
||||||
if !suppressErrors {
|
|
||||||
log.Error(failMsg, "error", err)
|
mu.Lock()
|
||||||
ReportErrorf("%s: %v", failMsg, err)
|
failed = true
|
||||||
|
mu.Unlock()
|
||||||
}
|
}
|
||||||
|
|
||||||
os.Exit(1)
|
stopErr := v.Shutdowner.Shutdown()
|
||||||
}
|
if stopErr != nil {
|
||||||
}
|
log.Error("Failed to shutdown", "error", stopErr)
|
||||||
|
|
||||||
err = v.Shutdowner.Shutdown()
|
|
||||||
if err != nil {
|
|
||||||
log.Error("Failed to shutdown", "error", err)
|
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
|
|
||||||
@@ -260,8 +256,56 @@ func runVaultikApp(
|
|||||||
return nil
|
return nil
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
}),
|
}))
|
||||||
|
|
||||||
|
err := RunWithApp(ctx, opts)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// The goroutine sets failed before triggering the shutdown that lets
|
||||||
|
// RunWithApp return, so the write is in place by the time we read it.
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
|
||||||
|
if failed {
|
||||||
|
return errReported
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// runVaultikApp runs the standard single-operation command lifecycle
|
||||||
|
// shared by the list/purge/verify/remove/remote-info subcommands:
|
||||||
|
// resolve the config, then run op against the Vaultik instance through
|
||||||
|
// RunOperation, reporting a failure prefixed with failMsg (suppressed
|
||||||
|
// while suppressErrors is true, e.g. under --json). extraQuiet is OR-ed
|
||||||
|
// into LogOptions.Quiet (e.g. --json output modes).
|
||||||
|
func runVaultikApp(
|
||||||
|
cmd *cobra.Command, extraQuiet, suppressErrors bool,
|
||||||
|
failMsg string, op func(v *vaultik.Vaultik) error,
|
||||||
|
) error {
|
||||||
|
configPath, err := ResolveConfigPath()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
rootFlags := GetRootFlags()
|
||||||
|
|
||||||
|
return RunOperation(cmd.Context(), AppOptions{
|
||||||
|
ConfigPath: configPath,
|
||||||
|
LogOptions: log.Options{
|
||||||
|
Verbose: rootFlags.Verbose,
|
||||||
|
Debug: rootFlags.Debug,
|
||||||
|
Quiet: rootFlags.Quiet || extraQuiet,
|
||||||
},
|
},
|
||||||
|
}, op, func(err error) {
|
||||||
|
if suppressErrors {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Error(failMsg, "error", err)
|
||||||
|
ReportErrorf("%s: %v", failMsg, err)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+17
-2
@@ -1,6 +1,7 @@
|
|||||||
package cli
|
package cli
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -19,7 +20,11 @@ const shortCommitLen = 12
|
|||||||
// flag is present in os.Args — see bannerSuppressedInArgs), executes the
|
// flag is present in os.Args — see bannerSuppressedInArgs), executes the
|
||||||
// root cobra command, and routes any returned error through the
|
// root cobra command, and routes any returned error through the
|
||||||
// ui.Writer so the user sees a properly formatted "🛑 ERROR:" line.
|
// ui.Writer so the user sees a properly formatted "🛑 ERROR:" line.
|
||||||
func Entry() {
|
//
|
||||||
|
// It returns the process exit code (0 on success, 1 on error) rather
|
||||||
|
// than calling os.Exit, so that main's deferred profile writers run
|
||||||
|
// before the process ends. See run in cmd/vaultik/main.go.
|
||||||
|
func Entry() int {
|
||||||
emitStartupBanner(os.Args[1:], os.Stdout)
|
emitStartupBanner(os.Args[1:], os.Stdout)
|
||||||
|
|
||||||
rootCmd := NewRootCommand()
|
rootCmd := NewRootCommand()
|
||||||
@@ -27,9 +32,19 @@ func Entry() {
|
|||||||
|
|
||||||
err := rootCmd.Execute()
|
err := rootCmd.Execute()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
// An operation that ran inside the fx app has already reported
|
||||||
|
// its own failure (and suppressed it under --json); errReported
|
||||||
|
// says so. Printing it again here would double the error line.
|
||||||
|
// Every other error — bad arguments, a config that would not
|
||||||
|
// load — reaches Entry unreported, so it is shown here.
|
||||||
|
if !errors.Is(err, errReported) {
|
||||||
ReportErrorf("%s", err.Error())
|
ReportErrorf("%s", err.Error())
|
||||||
os.Exit(1)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
// emitStartupBanner writes the startup banner to w unless args (the
|
// emitStartupBanner writes the startup banner to w unless args (the
|
||||||
|
|||||||
@@ -230,7 +230,7 @@ func TestEntryJSONStdoutIsExactlyOneDocument(t *testing.T) {
|
|||||||
programName, flagConfig, configPath, cmdSnapshot, cmdList, flagJSON,
|
programName, flagConfig, configPath, cmdSnapshot, cmdList, flagJSON,
|
||||||
}
|
}
|
||||||
|
|
||||||
stdout := captureProcessStdout(t, Entry)
|
stdout := captureProcessStdout(t, func() { _ = Entry() })
|
||||||
|
|
||||||
requireExactlyOneJSONDocument(t, stdout)
|
requireExactlyOneJSONDocument(t, stdout)
|
||||||
|
|
||||||
|
|||||||
@@ -1,139 +0,0 @@
|
|||||||
package cli //nolint:testpackage // shares the prune fixtures and capture helpers
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"io"
|
|
||||||
"os"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
|
||||||
|
|
||||||
// staleRecordLogMessage is the local-cleanup audit line CleanupLocalSnapshots
|
|
||||||
// logs for each stale record. It is exactly the signal issue #112 says a
|
|
||||||
// machine consumer lost under --json: gated off stdout, and pinned below
|
|
||||||
// the log level on stderr because --json used to force Quiet.
|
|
||||||
const staleRecordLogMessage = "Removing stale local snapshot record"
|
|
||||||
|
|
||||||
// TestEntryPruneJSONStderrHonoursVerbosity is the end-to-end regression
|
|
||||||
// guard for issue #112. Under --json the log level must still follow
|
|
||||||
// --verbose/--debug rather than being pinned to WARN, so the
|
|
||||||
// local-cleanup records reach stderr under --verbose while stdout stays
|
|
||||||
// exactly one JSON document; without --verbose they stay below the
|
|
||||||
// level, as they do without --json.
|
|
||||||
//
|
|
||||||
// Both halves are asserted together on the same run, because the fix has
|
|
||||||
// to keep the document clean (issue #108) while freeing stderr.
|
|
||||||
//
|
|
||||||
// Not parallel: it replaces os.Args, os.Stdout, os.Stderr and the xdg
|
|
||||||
// globals.
|
|
||||||
//
|
|
||||||
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
|
||||||
func TestEntryPruneJSONStderrHonoursVerbosity(t *testing.T) {
|
|
||||||
for _, testCase := range []struct {
|
|
||||||
name string
|
|
||||||
verbose bool
|
|
||||||
wantOnStderr bool
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
name: "verbose json surfaces the cleanup record on stderr",
|
|
||||||
verbose: true,
|
|
||||||
wantOnStderr: true,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "json alone keeps the cleanup record below the level",
|
|
||||||
verbose: false,
|
|
||||||
wantOnStderr: false,
|
|
||||||
},
|
|
||||||
} {
|
|
||||||
t.Run(testCase.name, func(t *testing.T) {
|
|
||||||
configPath := writeHermeticPruneConfig(t, true)
|
|
||||||
|
|
||||||
previousArgs := os.Args
|
|
||||||
|
|
||||||
t.Cleanup(func() {
|
|
||||||
os.Args = previousArgs
|
|
||||||
rootFlags = RootFlags{}
|
|
||||||
})
|
|
||||||
|
|
||||||
args := []string{
|
|
||||||
programName, flagConfig, configPath, cmdPrune, flagJSON,
|
|
||||||
}
|
|
||||||
if testCase.verbose {
|
|
||||||
args = append(args, "--verbose")
|
|
||||||
}
|
|
||||||
|
|
||||||
os.Args = args
|
|
||||||
|
|
||||||
stdout, stderr := captureProcessStdoutAndStderr(t, Entry)
|
|
||||||
|
|
||||||
// The document stays clean in both cases: freeing stderr must
|
|
||||||
// not regress issue #108.
|
|
||||||
requireExactlyOneJSONDocument(t, stdout)
|
|
||||||
|
|
||||||
if testCase.wantOnStderr {
|
|
||||||
assert.Contains(t, stderr, staleRecordLogMessage,
|
|
||||||
"--verbose --json must emit the cleanup record on stderr")
|
|
||||||
assert.Contains(t, stderr, stalePruneSnapshotID,
|
|
||||||
"the record must name the snapshot it removed")
|
|
||||||
} else {
|
|
||||||
assert.NotContains(t, stderr, staleRecordLogMessage,
|
|
||||||
"without --verbose the record stays below the log level")
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// captureProcessStdoutAndStderr redirects both of the process's own
|
|
||||||
// standard streams to pipes for the duration of fn and returns what was
|
|
||||||
// written to each. The redirection is at the file-descriptor level
|
|
||||||
// because the logger binds os.Stderr when it initializes inside fn, and
|
|
||||||
// the JSON document reaches os.Stdout independently; the point is to see
|
|
||||||
// where each actually lands.
|
|
||||||
//
|
|
||||||
// Not parallel-safe: os.Stdout and os.Stderr are process-global.
|
|
||||||
func captureProcessStdoutAndStderr(t *testing.T, fn func()) (string, string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
outReader, outWriter, err := os.Pipe()
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
errReader, errWriter, err := os.Pipe()
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
previousOut, previousErr := os.Stdout, os.Stderr
|
|
||||||
os.Stdout, os.Stderr = outWriter, errWriter
|
|
||||||
|
|
||||||
capturedOut := drain(outReader)
|
|
||||||
capturedErr := drain(errReader)
|
|
||||||
|
|
||||||
fn()
|
|
||||||
|
|
||||||
os.Stdout, os.Stderr = previousOut, previousErr
|
|
||||||
|
|
||||||
require.NoError(t, outWriter.Close())
|
|
||||||
require.NoError(t, errWriter.Close())
|
|
||||||
|
|
||||||
out, errOut := <-capturedOut, <-capturedErr
|
|
||||||
|
|
||||||
require.NoError(t, outReader.Close())
|
|
||||||
require.NoError(t, errReader.Close())
|
|
||||||
|
|
||||||
return out, errOut
|
|
||||||
}
|
|
||||||
|
|
||||||
// drain copies a reader to a string on a goroutine and delivers the
|
|
||||||
// result once the writer end is closed.
|
|
||||||
func drain(reader io.Reader) <-chan string {
|
|
||||||
captured := make(chan string, 1)
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
var buf bytes.Buffer
|
|
||||||
|
|
||||||
_, _ = io.Copy(&buf, reader)
|
|
||||||
captured <- buf.String()
|
|
||||||
}()
|
|
||||||
|
|
||||||
return captured
|
|
||||||
}
|
|
||||||
@@ -81,7 +81,7 @@ func TestEntryPruneJSONStdoutIsExactlyOneDocument(t *testing.T) {
|
|||||||
programName, flagConfig, configPath, cmdPrune, flagJSON,
|
programName, flagConfig, configPath, cmdPrune, flagJSON,
|
||||||
}
|
}
|
||||||
|
|
||||||
stdout := captureProcessStdout(t, Entry)
|
stdout := captureProcessStdout(t, func() { _ = Entry() })
|
||||||
|
|
||||||
requireExactlyOneJSONDocument(t, stdout)
|
requireExactlyOneJSONDocument(t, stdout)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,58 @@
|
|||||||
|
package cli //nolint:testpackage // shares programName and the capture helpers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestEntryReturnsStatusCode pins the contract main() relies on for
|
||||||
|
// issue #75: Entry reports success or failure through its return value
|
||||||
|
// and never calls os.Exit. An os.Exit from inside Entry would skip
|
||||||
|
// main's deferred profile writers and truncate the profile of a failing
|
||||||
|
// command. main turns this code into os.Exit only after those defers
|
||||||
|
// run, so a failing command must come back with a non-zero code rather
|
||||||
|
// than ending the process here.
|
||||||
|
//
|
||||||
|
// Stdout is captured only to keep the banner and command output off the
|
||||||
|
// test log; the assertion is on the returned code.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // replaces os.Args and rootFlags
|
||||||
|
func TestEntryReturnsStatusCode(t *testing.T) {
|
||||||
|
for _, testCase := range []struct {
|
||||||
|
name string
|
||||||
|
args []string
|
||||||
|
want int
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
// version is self-contained: it needs no config and no
|
||||||
|
// destination store, so it exercises the success path.
|
||||||
|
name: "successful command returns zero",
|
||||||
|
args: []string{programName, "version"},
|
||||||
|
want: 0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "unknown command returns one",
|
||||||
|
args: []string{programName, "no-such-command"},
|
||||||
|
want: 1,
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(testCase.name, func(t *testing.T) {
|
||||||
|
previousArgs := os.Args
|
||||||
|
|
||||||
|
t.Cleanup(func() {
|
||||||
|
os.Args = previousArgs
|
||||||
|
rootFlags = RootFlags{}
|
||||||
|
})
|
||||||
|
|
||||||
|
os.Args = testCase.args
|
||||||
|
|
||||||
|
var code int
|
||||||
|
|
||||||
|
_ = captureProcessStdout(t, func() { code = Entry() })
|
||||||
|
|
||||||
|
assert.Equal(t, testCase.want, code)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
+4
-35
@@ -1,12 +1,7 @@
|
|||||||
package cli
|
package cli
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"os"
|
|
||||||
|
|
||||||
"github.com/spf13/cobra"
|
"github.com/spf13/cobra"
|
||||||
"go.uber.org/fx"
|
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
)
|
)
|
||||||
@@ -33,44 +28,18 @@ func NewInfoCommand() *cobra.Command {
|
|||||||
// Use the app framework
|
// Use the app framework
|
||||||
rootFlags := GetRootFlags()
|
rootFlags := GetRootFlags()
|
||||||
|
|
||||||
return RunWithApp(cmd.Context(), AppOptions{
|
return RunOperation(cmd.Context(), AppOptions{
|
||||||
ConfigPath: configPath,
|
ConfigPath: configPath,
|
||||||
LogOptions: log.Options{
|
LogOptions: log.Options{
|
||||||
Verbose: rootFlags.Verbose,
|
Verbose: rootFlags.Verbose,
|
||||||
Debug: rootFlags.Debug,
|
Debug: rootFlags.Debug,
|
||||||
Quiet: rootFlags.Quiet,
|
Quiet: rootFlags.Quiet,
|
||||||
},
|
},
|
||||||
Modules: []fx.Option{},
|
}, func(v *vaultik.Vaultik) error {
|
||||||
Invokes: []fx.Option{
|
return v.ShowInfo()
|
||||||
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
|
}, func(err error) {
|
||||||
lc.Append(fx.Hook{
|
|
||||||
OnStart: func(_ context.Context) error {
|
|
||||||
go func() {
|
|
||||||
err := v.ShowInfo()
|
|
||||||
if err != nil {
|
|
||||||
if !errors.Is(err, context.Canceled) {
|
|
||||||
log.Error("Failed to show info", "error", err)
|
log.Error("Failed to show info", "error", err)
|
||||||
ReportErrorf("Failed to show info: %v", err)
|
ReportErrorf("Failed to show info: %v", err)
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err = v.Shutdowner.Shutdown()
|
|
||||||
if err != nil {
|
|
||||||
log.Error("Failed to shutdown", "error", err)
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
},
|
|
||||||
OnStop: func(_ context.Context) error {
|
|
||||||
v.Cancel()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}),
|
|
||||||
},
|
|
||||||
})
|
})
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|||||||
+9
-44
@@ -1,12 +1,7 @@
|
|||||||
package cli
|
package cli
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"os"
|
|
||||||
|
|
||||||
"github.com/spf13/cobra"
|
"github.com/spf13/cobra"
|
||||||
"go.uber.org/fx"
|
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
)
|
)
|
||||||
@@ -41,52 +36,22 @@ work (e.g. after a crashed backup or to reclaim storage).`,
|
|||||||
// Use the app framework like other commands
|
// Use the app framework like other commands
|
||||||
rootFlags := GetRootFlags()
|
rootFlags := GetRootFlags()
|
||||||
|
|
||||||
return RunWithApp(cmd.Context(), AppOptions{
|
return RunOperation(cmd.Context(), AppOptions{
|
||||||
ConfigPath: configPath,
|
ConfigPath: configPath,
|
||||||
LogOptions: log.Options{
|
LogOptions: log.Options{
|
||||||
Verbose: rootFlags.Verbose,
|
Verbose: rootFlags.Verbose,
|
||||||
Debug: rootFlags.Debug,
|
Debug: rootFlags.Debug,
|
||||||
Quiet: rootFlags.Quiet,
|
Quiet: rootFlags.Quiet || opts.JSON,
|
||||||
JSON: opts.JSON,
|
|
||||||
},
|
},
|
||||||
Modules: []fx.Option{},
|
}, func(v *vaultik.Vaultik) error {
|
||||||
Invokes: []fx.Option{
|
return v.Prune(opts)
|
||||||
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
|
}, func(err error) {
|
||||||
lc.Append(fx.Hook{
|
if opts.JSON {
|
||||||
OnStart: func(_ context.Context) error {
|
return
|
||||||
// Start the prune operation in a goroutine
|
}
|
||||||
go func() {
|
|
||||||
// Run the prune operation
|
|
||||||
err := v.Prune(opts)
|
|
||||||
if err != nil {
|
|
||||||
if !errors.Is(err, context.Canceled) {
|
|
||||||
if !opts.JSON {
|
|
||||||
log.Error("Prune operation failed", "error", err)
|
log.Error("Prune operation failed", "error", err)
|
||||||
ReportErrorf("Prune failed: %v", err)
|
ReportErrorf("Prune failed: %v", err)
|
||||||
}
|
|
||||||
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Shutdown the app when prune completes
|
|
||||||
err = v.Shutdowner.Shutdown()
|
|
||||||
if err != nil {
|
|
||||||
log.Error("Failed to shutdown", "error", err)
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
},
|
|
||||||
OnStop: func(_ context.Context) error {
|
|
||||||
log.Debug("Stopping prune operation")
|
|
||||||
v.Cancel()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}),
|
|
||||||
},
|
|
||||||
})
|
})
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|||||||
+9
-38
@@ -1,12 +1,9 @@
|
|||||||
package cli
|
package cli
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
|
||||||
"errors"
|
"errors"
|
||||||
"os"
|
|
||||||
|
|
||||||
"github.com/spf13/cobra"
|
"github.com/spf13/cobra"
|
||||||
"go.uber.org/fx"
|
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
)
|
)
|
||||||
@@ -83,48 +80,22 @@ func newRemoteInfoCommand() *cobra.Command {
|
|||||||
|
|
||||||
rootFlags := GetRootFlags()
|
rootFlags := GetRootFlags()
|
||||||
|
|
||||||
return RunWithApp(cmd.Context(), AppOptions{
|
return RunOperation(cmd.Context(), AppOptions{
|
||||||
ConfigPath: configPath,
|
ConfigPath: configPath,
|
||||||
LogOptions: log.Options{
|
LogOptions: log.Options{
|
||||||
Verbose: rootFlags.Verbose,
|
Verbose: rootFlags.Verbose,
|
||||||
Debug: rootFlags.Debug,
|
Debug: rootFlags.Debug,
|
||||||
Quiet: rootFlags.Quiet,
|
Quiet: rootFlags.Quiet || jsonOutput,
|
||||||
JSON: jsonOutput,
|
|
||||||
},
|
},
|
||||||
Modules: []fx.Option{},
|
}, func(v *vaultik.Vaultik) error {
|
||||||
Invokes: []fx.Option{
|
return v.RemoteInfo(jsonOutput)
|
||||||
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
|
}, func(err error) {
|
||||||
lc.Append(fx.Hook{
|
if jsonOutput {
|
||||||
OnStart: func(_ context.Context) error {
|
return
|
||||||
go func() {
|
}
|
||||||
err := v.RemoteInfo(jsonOutput)
|
|
||||||
if err != nil {
|
|
||||||
if !errors.Is(err, context.Canceled) {
|
|
||||||
if !jsonOutput {
|
|
||||||
log.Error("Failed to get remote info", "error", err)
|
log.Error("Failed to get remote info", "error", err)
|
||||||
ReportErrorf("Failed to get remote info: %v", err)
|
ReportErrorf("Failed to get remote info: %v", err)
|
||||||
}
|
|
||||||
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err = v.Shutdowner.Shutdown()
|
|
||||||
if err != nil {
|
|
||||||
log.Error("Failed to shutdown", "error", err)
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
},
|
|
||||||
OnStop: func(_ context.Context) error {
|
|
||||||
v.Cancel()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}),
|
|
||||||
},
|
|
||||||
})
|
})
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|||||||
+14
-73
@@ -1,13 +1,10 @@
|
|||||||
package cli
|
package cli
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
|
||||||
|
|
||||||
"github.com/spf13/cobra"
|
"github.com/spf13/cobra"
|
||||||
"go.uber.org/fx"
|
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
)
|
)
|
||||||
@@ -86,7 +83,8 @@ specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
|
|||||||
// Use the backup functionality from cli package
|
// Use the backup functionality from cli package
|
||||||
rootFlags := GetRootFlags()
|
rootFlags := GetRootFlags()
|
||||||
|
|
||||||
return RunWithApp(cmd.Context(), AppOptions{
|
// --cron suppression is wired through v.UI by setupGlobals.
|
||||||
|
return RunOperation(cmd.Context(), AppOptions{
|
||||||
ConfigPath: configPath,
|
ConfigPath: configPath,
|
||||||
LogOptions: log.Options{
|
LogOptions: log.Options{
|
||||||
Verbose: rootFlags.Verbose,
|
Verbose: rootFlags.Verbose,
|
||||||
@@ -94,42 +92,11 @@ specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
|
|||||||
Cron: opts.Cron,
|
Cron: opts.Cron,
|
||||||
Quiet: rootFlags.Quiet,
|
Quiet: rootFlags.Quiet,
|
||||||
},
|
},
|
||||||
Modules: []fx.Option{},
|
}, func(v *vaultik.Vaultik) error {
|
||||||
Invokes: []fx.Option{
|
return v.CreateSnapshot(opts)
|
||||||
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
|
}, func(err error) {
|
||||||
lc.Append(fx.Hook{
|
|
||||||
OnStart: func(_ context.Context) error {
|
|
||||||
// Start the snapshot creation in a goroutine
|
|
||||||
go func() {
|
|
||||||
// --cron suppression is wired through v.UI by setupGlobals.
|
|
||||||
err := v.CreateSnapshot(opts)
|
|
||||||
if err != nil {
|
|
||||||
if !errors.Is(err, context.Canceled) {
|
|
||||||
log.Error("Snapshot creation failed", "error", err)
|
log.Error("Snapshot creation failed", "error", err)
|
||||||
ReportErrorf("Snapshot creation failed: %v", err)
|
ReportErrorf("Snapshot creation failed: %v", err)
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Shutdown the app when snapshot completes
|
|
||||||
err = v.Shutdowner.Shutdown()
|
|
||||||
if err != nil {
|
|
||||||
log.Error("Failed to shutdown", "error", err)
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
},
|
|
||||||
OnStop: func(_ context.Context) error {
|
|
||||||
log.Debug("Stopping snapshot creation")
|
|
||||||
// Cancel the Vaultik context
|
|
||||||
v.Cancel()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}),
|
|
||||||
},
|
|
||||||
})
|
})
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -234,48 +201,22 @@ func newSnapshotVerifyCommand() *cobra.Command {
|
|||||||
|
|
||||||
rootFlags := GetRootFlags()
|
rootFlags := GetRootFlags()
|
||||||
|
|
||||||
return RunWithApp(cmd.Context(), AppOptions{
|
return RunOperation(cmd.Context(), AppOptions{
|
||||||
ConfigPath: configPath,
|
ConfigPath: configPath,
|
||||||
LogOptions: log.Options{
|
LogOptions: log.Options{
|
||||||
Verbose: rootFlags.Verbose,
|
Verbose: rootFlags.Verbose,
|
||||||
Debug: rootFlags.Debug,
|
Debug: rootFlags.Debug,
|
||||||
Quiet: rootFlags.Quiet,
|
Quiet: rootFlags.Quiet || opts.JSON,
|
||||||
JSON: opts.JSON,
|
|
||||||
},
|
},
|
||||||
Modules: []fx.Option{},
|
}, func(v *vaultik.Vaultik) error {
|
||||||
Invokes: []fx.Option{
|
return v.VerifySnapshotWithOptions(snapshotID, opts)
|
||||||
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
|
}, func(err error) {
|
||||||
lc.Append(fx.Hook{
|
if opts.JSON {
|
||||||
OnStart: func(_ context.Context) error {
|
return
|
||||||
go func() {
|
}
|
||||||
err := v.VerifySnapshotWithOptions(snapshotID, opts)
|
|
||||||
if err != nil {
|
|
||||||
if !errors.Is(err, context.Canceled) {
|
|
||||||
if !opts.JSON {
|
|
||||||
log.Error("Verification failed", "error", err)
|
log.Error("Verification failed", "error", err)
|
||||||
ReportErrorf("Verification failed: %v", err)
|
ReportErrorf("Verification failed: %v", err)
|
||||||
}
|
|
||||||
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err = v.Shutdowner.Shutdown()
|
|
||||||
if err != nil {
|
|
||||||
log.Error("Failed to shutdown", "error", err)
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
},
|
|
||||||
OnStop: func(_ context.Context) error {
|
|
||||||
v.Cancel()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}),
|
|
||||||
},
|
|
||||||
})
|
})
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,16 +1,8 @@
|
|||||||
package cli
|
package cli
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"os"
|
|
||||||
|
|
||||||
"github.com/spf13/cobra"
|
"github.com/spf13/cobra"
|
||||||
"go.uber.org/fx"
|
|
||||||
"sneak.berlin/go/vaultik/internal/config"
|
|
||||||
"sneak.berlin/go/vaultik/internal/globals"
|
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
"sneak.berlin/go/vaultik/internal/storage"
|
|
||||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -25,15 +17,6 @@ type RestoreOptions struct {
|
|||||||
Verify bool // Verify restored files after restore
|
Verify bool // Verify restored files after restore
|
||||||
}
|
}
|
||||||
|
|
||||||
// RestoreApp contains all dependencies needed for restore
|
|
||||||
type RestoreApp struct {
|
|
||||||
Globals *globals.Globals
|
|
||||||
Config *config.Config
|
|
||||||
Storage storage.Storer
|
|
||||||
Vaultik *vaultik.Vaultik
|
|
||||||
Shutdowner fx.Shutdowner
|
|
||||||
}
|
|
||||||
|
|
||||||
// newSnapshotRestoreCommand creates the 'snapshot restore' subcommand
|
// newSnapshotRestoreCommand creates the 'snapshot restore' subcommand
|
||||||
func newSnapshotRestoreCommand() *cobra.Command {
|
func newSnapshotRestoreCommand() *cobra.Command {
|
||||||
opts := &RestoreOptions{}
|
opts := &RestoreOptions{}
|
||||||
@@ -77,7 +60,8 @@ Examples:
|
|||||||
return cmd
|
return cmd
|
||||||
}
|
}
|
||||||
|
|
||||||
// runRestore parses arguments and runs the restore operation through the app framework
|
// runRestore parses arguments and runs the restore operation through the
|
||||||
|
// app framework.
|
||||||
func runRestore(cmd *cobra.Command, args []string, opts *RestoreOptions) error {
|
func runRestore(cmd *cobra.Command, args []string, opts *RestoreOptions) error {
|
||||||
snapshotID := args[0]
|
snapshotID := args[0]
|
||||||
|
|
||||||
@@ -86,87 +70,30 @@ func runRestore(cmd *cobra.Command, args []string, opts *RestoreOptions) error {
|
|||||||
opts.Paths = args[restoreMinArgs:]
|
opts.Paths = args[restoreMinArgs:]
|
||||||
}
|
}
|
||||||
|
|
||||||
// Use unified config resolution
|
|
||||||
configPath, err := ResolveConfigPath()
|
configPath, err := ResolveConfigPath()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Use the app framework like other commands
|
|
||||||
rootFlags := GetRootFlags()
|
rootFlags := GetRootFlags()
|
||||||
|
|
||||||
return RunWithApp(cmd.Context(), AppOptions{
|
return RunOperation(cmd.Context(), AppOptions{
|
||||||
ConfigPath: configPath,
|
ConfigPath: configPath,
|
||||||
LogOptions: log.Options{
|
LogOptions: log.Options{
|
||||||
Verbose: rootFlags.Verbose,
|
Verbose: rootFlags.Verbose,
|
||||||
Debug: rootFlags.Debug,
|
Debug: rootFlags.Debug,
|
||||||
Quiet: rootFlags.Quiet,
|
Quiet: rootFlags.Quiet,
|
||||||
},
|
},
|
||||||
Modules: buildRestoreModules(),
|
}, func(v *vaultik.Vaultik) error {
|
||||||
Invokes: buildRestoreInvokes(snapshotID, opts),
|
return v.Restore(&vaultik.RestoreOptions{
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// buildRestoreModules returns the fx.Options for dependency injection in restore
|
|
||||||
func buildRestoreModules() []fx.Option {
|
|
||||||
return []fx.Option{
|
|
||||||
fx.Provide(fx.Annotate(
|
|
||||||
func(g *globals.Globals, cfg *config.Config,
|
|
||||||
storer storage.Storer, v *vaultik.Vaultik, shutdowner fx.Shutdowner) *RestoreApp {
|
|
||||||
return &RestoreApp{
|
|
||||||
Globals: g,
|
|
||||||
Config: cfg,
|
|
||||||
Storage: storer,
|
|
||||||
Vaultik: v,
|
|
||||||
Shutdowner: shutdowner,
|
|
||||||
}
|
|
||||||
},
|
|
||||||
)),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// buildRestoreInvokes returns the fx.Options that wire up the restore lifecycle
|
|
||||||
func buildRestoreInvokes(snapshotID string, opts *RestoreOptions) []fx.Option {
|
|
||||||
return []fx.Option{
|
|
||||||
fx.Invoke(func(app *RestoreApp, lc fx.Lifecycle) {
|
|
||||||
lc.Append(fx.Hook{
|
|
||||||
OnStart: func(_ context.Context) error {
|
|
||||||
// Start the restore operation in a goroutine
|
|
||||||
go func() {
|
|
||||||
// Run the restore operation
|
|
||||||
restoreOpts := &vaultik.RestoreOptions{
|
|
||||||
SnapshotID: snapshotID,
|
SnapshotID: snapshotID,
|
||||||
TargetDir: opts.TargetDir,
|
TargetDir: opts.TargetDir,
|
||||||
Paths: opts.Paths,
|
Paths: opts.Paths,
|
||||||
Verify: opts.Verify,
|
Verify: opts.Verify,
|
||||||
SkipErrors: GetRootFlags().SkipErrors,
|
SkipErrors: rootFlags.SkipErrors,
|
||||||
}
|
})
|
||||||
|
}, func(err error) {
|
||||||
err := app.Vaultik.Restore(restoreOpts)
|
|
||||||
if err != nil {
|
|
||||||
if !errors.Is(err, context.Canceled) {
|
|
||||||
log.Error("Restore operation failed", "error", err)
|
log.Error("Restore operation failed", "error", err)
|
||||||
ReportErrorf("Restore failed: %v", err)
|
ReportErrorf("Restore failed: %v", err)
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Shutdown the app when restore completes
|
|
||||||
err = app.Shutdowner.Shutdown()
|
|
||||||
if err != nil {
|
|
||||||
log.Error("Failed to shutdown", "error", err)
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
},
|
|
||||||
OnStop: func(_ context.Context) error {
|
|
||||||
log.Debug("Stopping restore operation")
|
|
||||||
app.Vaultik.Cancel()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
},
|
|
||||||
})
|
})
|
||||||
}),
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
+1
-15
@@ -14,18 +14,8 @@ var Module = fx.Module("log",
|
|||||||
)
|
)
|
||||||
|
|
||||||
// New creates a new logger configuration from provided options.
|
// New creates a new logger configuration from provided options.
|
||||||
//
|
|
||||||
// JSON is intentionally not carried into Config: a command emitting a
|
|
||||||
// JSON document on stdout must keep its stderr log level under
|
|
||||||
// --verbose/--debug, so --json must not lower it (issue #112). JSON
|
|
||||||
// silences the stdout UI in setupGlobals instead.
|
|
||||||
func New(opts Options) Config {
|
func New(opts Options) Config {
|
||||||
return Config{
|
return Config(opts)
|
||||||
Verbose: opts.Verbose,
|
|
||||||
Debug: opts.Debug,
|
|
||||||
Cron: opts.Cron,
|
|
||||||
Quiet: opts.Quiet,
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Options are provided by the CLI.
|
// Options are provided by the CLI.
|
||||||
@@ -34,8 +24,4 @@ type Options struct {
|
|||||||
Debug bool
|
Debug bool
|
||||||
Cron bool
|
Cron bool
|
||||||
Quiet bool
|
Quiet bool
|
||||||
// JSON marks a command whose stdout carries a machine-readable
|
|
||||||
// document. It silences the human UI on stdout (see setupGlobals),
|
|
||||||
// but unlike Quiet it leaves the stderr log level alone.
|
|
||||||
JSON bool
|
|
||||||
}
|
}
|
||||||
|
|||||||
+54
-79
@@ -46,18 +46,31 @@ func (f *FileStorer) SetFilesystem(fs afero.Fs) {
|
|||||||
// storage base path.
|
// storage base path.
|
||||||
const storageDirPerm = 0o755
|
const storageDirPerm = 0o755
|
||||||
|
|
||||||
// tempSuffix marks a partially written object. writeAtomic streams into a
|
|
||||||
// temp file carrying this suffix and only renames it onto the real key once
|
|
||||||
// the whole object is on disk, so an interrupted write can never leave a
|
|
||||||
// truncated object at the key a later run would Stat and trust as a complete
|
|
||||||
// blob. List and ListStream skip these files, so a leftover from an
|
|
||||||
// interrupted write is never listed or trusted as a blob; it is otherwise
|
|
||||||
// harmless and is overwritten when the same key is written again.
|
|
||||||
const tempSuffix = ".partial"
|
|
||||||
|
|
||||||
// Put stores data at the specified key.
|
// Put stores data at the specified key.
|
||||||
func (f *FileStorer) Put(_ context.Context, key string, data io.Reader) error {
|
func (f *FileStorer) Put(_ context.Context, key string, data io.Reader) error {
|
||||||
return f.writeAtomic(key, data, nil)
|
path := f.fullPath(key)
|
||||||
|
|
||||||
|
// Create parent directories
|
||||||
|
dir := filepath.Dir(path)
|
||||||
|
|
||||||
|
err := f.fs.MkdirAll(dir, storageDirPerm)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("creating directories: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
file, err := f.fs.Create(path)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("creating file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = file.Close() }()
|
||||||
|
|
||||||
|
_, err = io.Copy(file, data)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("writing file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// PutWithProgress stores data with progress reporting.
|
// PutWithProgress stores data with progress reporting.
|
||||||
@@ -65,7 +78,35 @@ func (f *FileStorer) PutWithProgress(
|
|||||||
_ context.Context, key string, data io.Reader,
|
_ context.Context, key string, data io.Reader,
|
||||||
_ int64, progress ProgressCallback,
|
_ int64, progress ProgressCallback,
|
||||||
) error {
|
) error {
|
||||||
return f.writeAtomic(key, data, progress)
|
path := f.fullPath(key)
|
||||||
|
|
||||||
|
// Create parent directories
|
||||||
|
dir := filepath.Dir(path)
|
||||||
|
|
||||||
|
err := f.fs.MkdirAll(dir, storageDirPerm)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("creating directories: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
file, err := f.fs.Create(path)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("creating file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = file.Close() }()
|
||||||
|
|
||||||
|
// Wrap with progress tracking
|
||||||
|
pw := &progressWriter{
|
||||||
|
writer: file,
|
||||||
|
callback: progress,
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = io.Copy(pw, data)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("writing file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Get retrieves data from the specified key.
|
// Get retrieves data from the specified key.
|
||||||
@@ -147,7 +188,7 @@ func (f *FileStorer) List(ctx context.Context, prefix string) ([]string, error)
|
|||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
if !info.IsDir() && !strings.HasSuffix(info.Name(), tempSuffix) {
|
if !info.IsDir() {
|
||||||
// Convert back to key (relative path from basePath)
|
// Convert back to key (relative path from basePath)
|
||||||
relPath, err := filepath.Rel(f.basePath, path)
|
relPath, err := filepath.Rel(f.basePath, path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -204,7 +245,7 @@ func (f *FileStorer) ListStream(ctx context.Context, prefix string) <-chan Objec
|
|||||||
return nil //nolint:nilerr // continue walking despite errors
|
return nil //nolint:nilerr // continue walking despite errors
|
||||||
}
|
}
|
||||||
|
|
||||||
if !info.IsDir() && !strings.HasSuffix(info.Name(), tempSuffix) {
|
if !info.IsDir() {
|
||||||
relPath, err := filepath.Rel(f.basePath, path)
|
relPath, err := filepath.Rel(f.basePath, path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
ch <- ObjectInfo{Err: fmt.Errorf("computing relative path: %w", err)}
|
ch <- ObjectInfo{Err: fmt.Errorf("computing relative path: %w", err)}
|
||||||
@@ -234,72 +275,6 @@ func (f *FileStorer) Info() Info {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// writeAtomic streams data into a temp file in the destination directory,
|
|
||||||
// fsyncs it, and renames it onto the final key. The key therefore appears
|
|
||||||
// only once the whole object has been durably written; a failure part-way
|
|
||||||
// leaves a temp file (removed here on the failing path) rather than a
|
|
||||||
// truncated object at the key.
|
|
||||||
func (f *FileStorer) writeAtomic(
|
|
||||||
key string, data io.Reader, progress ProgressCallback,
|
|
||||||
) error {
|
|
||||||
path := f.fullPath(key)
|
|
||||||
dir := filepath.Dir(path)
|
|
||||||
|
|
||||||
err := f.fs.MkdirAll(dir, storageDirPerm)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("creating directories: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
tmp, err := afero.TempFile(f.fs, dir, filepath.Base(path)+"-*"+tempSuffix)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("creating temp file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
tmpPath := tmp.Name()
|
|
||||||
|
|
||||||
// Remove the temp file unless the rename below claims it. On the success
|
|
||||||
// path renamed is true, so the deferred Close and Remove are harmless
|
|
||||||
// no-ops on a name that no longer exists.
|
|
||||||
renamed := false
|
|
||||||
|
|
||||||
defer func() {
|
|
||||||
_ = tmp.Close()
|
|
||||||
|
|
||||||
if !renamed {
|
|
||||||
_ = f.fs.Remove(tmpPath)
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
var w io.Writer = tmp
|
|
||||||
if progress != nil {
|
|
||||||
w = &progressWriter{writer: tmp, callback: progress}
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = io.Copy(w, data)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("writing file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = tmp.Sync()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("syncing temp file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = tmp.Close()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("closing temp file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = f.fs.Rename(tmpPath, path)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("renaming temp file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
renamed = true
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// fullPath returns the full filesystem path for a key.
|
// fullPath returns the full filesystem path for a key.
|
||||||
func (f *FileStorer) fullPath(key string) string {
|
func (f *FileStorer) fullPath(key string) string {
|
||||||
return filepath.Join(f.basePath, key)
|
return filepath.Join(f.basePath, key)
|
||||||
|
|||||||
@@ -1,119 +0,0 @@
|
|||||||
package storage_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"sneak.berlin/go/vaultik/internal/storage"
|
|
||||||
)
|
|
||||||
|
|
||||||
// errStreamInterrupted stands in for an upload cut off mid-stream.
|
|
||||||
var errStreamInterrupted = errors.New("connection reset mid-upload")
|
|
||||||
|
|
||||||
// failingReader yields its data once, then fails.
|
|
||||||
type failingReader struct {
|
|
||||||
data []byte
|
|
||||||
done bool
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *failingReader) Read(p []byte) (int, error) {
|
|
||||||
if r.done {
|
|
||||||
return 0, errStreamInterrupted
|
|
||||||
}
|
|
||||||
|
|
||||||
n := copy(p, r.data)
|
|
||||||
r.done = true
|
|
||||||
|
|
||||||
return n, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFileStorer_InterruptedWriteLeavesNoTrustedObject checks that a write
|
|
||||||
// cut off mid-stream leaves nothing at the destination key, so a later run
|
|
||||||
// cannot Stat a truncated object and trust it as a complete blob.
|
|
||||||
func TestFileStorer_InterruptedWriteLeavesNoTrustedObject(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
f, err := storage.NewFileStorer(t.TempDir())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("NewFileStorer: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
key := "blobs/aa/bb/aabbccddeeff"
|
|
||||||
|
|
||||||
err = f.PutWithProgress(ctx, key, &failingReader{data: []byte("partial")}, 4096, nil)
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("expected the interrupted write to fail, got nil")
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = f.Stat(ctx, key)
|
|
||||||
if !errors.Is(err, storage.ErrNotFound) {
|
|
||||||
t.Fatalf("expected key absent after interrupted write, got Stat err %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
keys, err := f.List(ctx, "blobs/")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("List: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(keys) != 0 {
|
|
||||||
t.Fatalf("expected no keys listed after interrupted write, got %v", keys)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFileStorer_ListSkipsPartialFiles checks that a leftover temp file (the
|
|
||||||
// storage layer names them with a ".partial" suffix) is never surfaced as a
|
|
||||||
// key by List or ListStream.
|
|
||||||
func TestFileStorer_ListSkipsPartialFiles(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
base := t.TempDir()
|
|
||||||
|
|
||||||
f, err := storage.NewFileStorer(base)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("NewFileStorer: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
realKey := "blobs/aa/bb/aabbccddeeff"
|
|
||||||
|
|
||||||
err = f.Put(ctx, realKey, strings.NewReader("blob-bytes"))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Put: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A stray temp file, as an interrupted write would leave behind.
|
|
||||||
leftover := filepath.Join(base, "blobs/aa/bb/aabbccddeeff-123456.partial")
|
|
||||||
|
|
||||||
err = os.WriteFile(leftover, []byte("half"), 0o600)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("writing leftover temp file: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
keys, err := f.List(ctx, "blobs/")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("List: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(keys) != 1 || keys[0] != realKey {
|
|
||||||
t.Fatalf("List should return only the real key, got %v", keys)
|
|
||||||
}
|
|
||||||
|
|
||||||
var streamed []string
|
|
||||||
|
|
||||||
for obj := range f.ListStream(ctx, "blobs/") {
|
|
||||||
if obj.Err != nil {
|
|
||||||
t.Fatalf("ListStream: %v", obj.Err)
|
|
||||||
}
|
|
||||||
|
|
||||||
streamed = append(streamed, obj.Key)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(streamed) != 1 || streamed[0] != realKey {
|
|
||||||
t.Fatalf("ListStream should return only the real key, got %v", streamed)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,79 +0,0 @@
|
|||||||
package vaultik //nolint:testpackage // exercises unexported count helpers
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
"sneak.berlin/go/vaultik/internal/database"
|
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestTableCountForReportSurfacesReadFailure is the regression guard for
|
|
||||||
// the discarded-error bug: getTableCount for a table its query cannot
|
|
||||||
// resolve must not silently become 0. A count that could not be read is
|
|
||||||
// reported as unknown, which a reader can tell apart from an empty table.
|
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestTableCountForReportSurfacesReadFailure(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
|
|
||||||
db, err := database.New(ctx, ":memory:")
|
|
||||||
require.NoError(t, err)
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
v := &Vaultik{DB: db}
|
|
||||||
v.SetContext(ctx)
|
|
||||||
|
|
||||||
// A table present in the schema reads as a real count.
|
|
||||||
blobs := v.tableCountForReport("blobs")
|
|
||||||
require.NotNil(t, blobs, "an existing table must read as a real count")
|
|
||||||
assert.Equal(t, int64(0), *blobs)
|
|
||||||
|
|
||||||
// A syntactically valid name the sanitizer accepts but whose table
|
|
||||||
// the query cannot resolve is the exact shape #96 describes: a
|
|
||||||
// would-be loud failure that used to be discarded into a 0.
|
|
||||||
_, err = v.getTableCount("snapshots_missing")
|
|
||||||
require.Error(t, err, "a query against a nonexistent table must fail")
|
|
||||||
|
|
||||||
missing := v.tableCountForReport("snapshots_missing")
|
|
||||||
assert.Nil(t, missing, "a failed read is unknown, not a count")
|
|
||||||
|
|
||||||
// The rendered count for a failed read must say unknown, never 0.
|
|
||||||
assert.Equal(t, countUnknown, countText(missing))
|
|
||||||
assert.NotEqual(t, "0", countText(missing))
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestCountTextDistinguishesEmptyFromUnknown pins the distinction the
|
|
||||||
// output has to preserve: 0 means the table was empty, "unknown" means
|
|
||||||
// the count could not be read.
|
|
||||||
func TestCountTextDistinguishesEmptyFromUnknown(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
zero := int64(0)
|
|
||||||
seven := int64(7)
|
|
||||||
|
|
||||||
assert.Equal(t, "0", countText(&zero))
|
|
||||||
assert.Equal(t, "7", countText(&seven))
|
|
||||||
assert.Equal(t, countUnknown, countText(nil))
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestCountDiffUnknownWhenEitherSideUnknown checks that a delta computed
|
|
||||||
// from an unreadable count is itself unknown rather than a plausible
|
|
||||||
// number.
|
|
||||||
func TestCountDiffUnknownWhenEitherSideUnknown(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
before := int64(10)
|
|
||||||
after := int64(3)
|
|
||||||
|
|
||||||
require.NotNil(t, countDiff(&before, &after))
|
|
||||||
assert.Equal(t, int64(7), *countDiff(&before, &after))
|
|
||||||
|
|
||||||
assert.Nil(t, countDiff(nil, &after), "unknown before yields unknown delta")
|
|
||||||
assert.Nil(t, countDiff(&before, nil), "unknown after yields unknown delta")
|
|
||||||
assert.Nil(t, countDiff(nil, nil))
|
|
||||||
}
|
|
||||||
@@ -8,7 +8,6 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"regexp"
|
"regexp"
|
||||||
"sort"
|
"sort"
|
||||||
"strconv"
|
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -1541,17 +1540,12 @@ func (v *Vaultik) outputRemoveJSON(result *RemoveResult) error {
|
|||||||
return encoder.Encode(result)
|
return encoder.Encode(result)
|
||||||
}
|
}
|
||||||
|
|
||||||
// PruneResult contains statistics about the prune operation.
|
// PruneResult contains statistics about the prune operation
|
||||||
// SnapshotsDeleted counts snapshots actually deleted. FilesDeleted,
|
|
||||||
// ChunksDeleted, and BlobsDeleted are derived from before/after row
|
|
||||||
// counts of the local index; each is nil when a count could not be read,
|
|
||||||
// so an unreadable count is reported as unknown rather than silently
|
|
||||||
// as 0.
|
|
||||||
type PruneResult struct {
|
type PruneResult struct {
|
||||||
SnapshotsDeleted int64
|
SnapshotsDeleted int64
|
||||||
FilesDeleted *int64
|
FilesDeleted int64
|
||||||
ChunksDeleted *int64
|
ChunksDeleted int64
|
||||||
BlobsDeleted *int64
|
BlobsDeleted int64
|
||||||
}
|
}
|
||||||
|
|
||||||
// PruneDatabase removes incomplete snapshots and orphaned files, chunks,
|
// PruneDatabase removes incomplete snapshots and orphaned files, chunks,
|
||||||
@@ -1566,7 +1560,7 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
|
|||||||
result := &PruneResult{}
|
result := &PruneResult{}
|
||||||
|
|
||||||
// Snapshot counts before deletion of incompletes.
|
// Snapshot counts before deletion of incompletes.
|
||||||
snapshotCountBefore := v.tableCountForReport("snapshots")
|
snapshotCountBefore, _ := v.getTableCount("snapshots")
|
||||||
|
|
||||||
// First, delete any incomplete snapshots
|
// First, delete any incomplete snapshots
|
||||||
incompleteSnapshots, err := v.Repositories.Snapshots.GetIncompleteSnapshots(v.ctx)
|
incompleteSnapshots, err := v.Repositories.Snapshots.GetIncompleteSnapshots(v.ctx)
|
||||||
@@ -1581,9 +1575,9 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Get counts before cleanup for reporting
|
// Get counts before cleanup for reporting
|
||||||
fileCountBefore := v.tableCountForReport("files")
|
fileCountBefore, _ := v.getTableCount("files")
|
||||||
chunkCountBefore := v.tableCountForReport("chunks")
|
chunkCountBefore, _ := v.getTableCount("chunks")
|
||||||
blobCountBefore := v.tableCountForReport("blobs")
|
blobCountBefore, _ := v.getTableCount("blobs")
|
||||||
|
|
||||||
// Run the cleanup
|
// Run the cleanup
|
||||||
err = v.SnapshotManager.CleanupOrphanedData(v.ctx)
|
err = v.SnapshotManager.CleanupOrphanedData(v.ctx)
|
||||||
@@ -1592,83 +1586,36 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Get counts after cleanup
|
// Get counts after cleanup
|
||||||
fileCountAfter := v.tableCountForReport("files")
|
fileCountAfter, _ := v.getTableCount("files")
|
||||||
chunkCountAfter := v.tableCountForReport("chunks")
|
chunkCountAfter, _ := v.getTableCount("chunks")
|
||||||
blobCountAfter := v.tableCountForReport("blobs")
|
blobCountAfter, _ := v.getTableCount("blobs")
|
||||||
|
|
||||||
result.FilesDeleted = countDiff(fileCountBefore, fileCountAfter)
|
result.FilesDeleted = fileCountBefore - fileCountAfter
|
||||||
result.ChunksDeleted = countDiff(chunkCountBefore, chunkCountAfter)
|
result.ChunksDeleted = chunkCountBefore - chunkCountAfter
|
||||||
result.BlobsDeleted = countDiff(blobCountBefore, blobCountAfter)
|
result.BlobsDeleted = blobCountBefore - blobCountAfter
|
||||||
|
|
||||||
log.Info("Local database prune complete",
|
log.Info("Local database prune complete",
|
||||||
"incomplete_snapshots", result.SnapshotsDeleted,
|
"incomplete_snapshots", result.SnapshotsDeleted,
|
||||||
"orphaned_files", countText(result.FilesDeleted),
|
"orphaned_files", result.FilesDeleted,
|
||||||
"orphaned_chunks", countText(result.ChunksDeleted),
|
"orphaned_chunks", result.ChunksDeleted,
|
||||||
"orphaned_blobs", countText(result.BlobsDeleted),
|
"orphaned_blobs", result.BlobsDeleted,
|
||||||
)
|
)
|
||||||
|
|
||||||
// Snapshots remaining after removing the incomplete ones; unknown if
|
snapshotCountAfter := snapshotCountBefore - result.SnapshotsDeleted
|
||||||
// the pre-prune snapshot count could not be read.
|
|
||||||
snapshotsRemain := countDiff(snapshotCountBefore, &result.SnapshotsDeleted)
|
|
||||||
|
|
||||||
v.UI.Completef("Pruned local index database.")
|
v.UI.Completef("Pruned local index database.")
|
||||||
v.UI.Detailf("Incomplete snapshots: %s removed (%s remain).",
|
v.UI.Detailf("Incomplete snapshots: %d removed (%d remain).",
|
||||||
countText(&result.SnapshotsDeleted), countText(snapshotsRemain))
|
result.SnapshotsDeleted, snapshotCountAfter)
|
||||||
v.UI.Detailf("Orphaned files: %s removed (%s remain).",
|
v.UI.Detailf("Orphaned files: %d removed (%d remain).",
|
||||||
countText(result.FilesDeleted), countText(fileCountAfter))
|
result.FilesDeleted, fileCountAfter)
|
||||||
v.UI.Detailf("Orphaned chunks: %s removed (%s remain).",
|
v.UI.Detailf("Orphaned chunks: %d removed (%d remain).",
|
||||||
countText(result.ChunksDeleted), countText(chunkCountAfter))
|
result.ChunksDeleted, chunkCountAfter)
|
||||||
v.UI.Detailf("Orphaned blobs: %s removed (%s remain).",
|
v.UI.Detailf("Orphaned blobs: %d removed (%d remain).",
|
||||||
countText(result.BlobsDeleted), countText(blobCountAfter))
|
result.BlobsDeleted, blobCountAfter)
|
||||||
|
|
||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// countUnknown is what a count reads as when its query could not be run,
|
|
||||||
// distinct from "0", which means the table really was empty.
|
|
||||||
const countUnknown = "unknown"
|
|
||||||
|
|
||||||
// tableCountForReport returns the row count of a table for the prune
|
|
||||||
// summary, or nil if the count could not be read. A read failure is
|
|
||||||
// logged at warn — visible even under --json, which routes warnings to
|
|
||||||
// stderr — and then rendered as unknown rather than silently becoming 0,
|
|
||||||
// so a broken query is a visible failure instead of a plausible wrong
|
|
||||||
// number.
|
|
||||||
func (v *Vaultik) tableCountForReport(tableName string) *int64 {
|
|
||||||
count, err := v.getTableCount(tableName)
|
|
||||||
if err != nil {
|
|
||||||
log.Warn("could not read table row count for prune summary",
|
|
||||||
"table", tableName, "error", err)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return &count
|
|
||||||
}
|
|
||||||
|
|
||||||
// countDiff returns before-after, or nil if either count is unknown so
|
|
||||||
// that an unreadable count does not collapse into a plausible delta.
|
|
||||||
func countDiff(before, after *int64) *int64 {
|
|
||||||
if before == nil || after == nil {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
diff := *before - *after
|
|
||||||
|
|
||||||
return &diff
|
|
||||||
}
|
|
||||||
|
|
||||||
// countText renders a count that may be unknown: nil (the read failed)
|
|
||||||
// becomes "unknown", never "0", so a reader can tell an empty table from
|
|
||||||
// one that could not be queried.
|
|
||||||
func countText(count *int64) string {
|
|
||||||
if count == nil {
|
|
||||||
return countUnknown
|
|
||||||
}
|
|
||||||
|
|
||||||
return strconv.FormatInt(*count, 10)
|
|
||||||
}
|
|
||||||
|
|
||||||
// validTableNameRe matches table names containing only lowercase
|
// validTableNameRe matches table names containing only lowercase
|
||||||
// alphanumeric characters and underscores.
|
// alphanumeric characters and underscores.
|
||||||
var validTableNameRe = regexp.MustCompile(`^[a-z0-9_]+$`)
|
var validTableNameRe = regexp.MustCompile(`^[a-z0-9_]+$`)
|
||||||
|
|||||||
+16
-1
@@ -56,8 +56,23 @@ main() {
|
|||||||
docker build --output=type=cacheonly \
|
docker build --output=type=cacheonly \
|
||||||
--build-arg CHECK_EPOCH="$epoch" -f Dockerfile.lint .
|
--build-arg CHECK_EPOCH="$epoch" -f Dockerfile.lint .
|
||||||
|
|
||||||
|
# Version, commit and build date are computed here on the host, the
|
||||||
|
# same way script/docker does, and passed into the product build so
|
||||||
|
# the CI-built image reports its real source. The build context
|
||||||
|
# excludes .git (see .dockerignore), so the build cannot derive them
|
||||||
|
# itself; without these it would stamp the Dockerfile's dev/unknown
|
||||||
|
# fallbacks. VERSION comes from script/version, the source of truth
|
||||||
|
# shared with the Makefile.
|
||||||
|
version="$("$ROOT/script/version")"
|
||||||
|
commit="$(git rev-parse HEAD 2>/dev/null || echo unknown)"
|
||||||
|
commit_date="$(git show -s --format=%cs HEAD 2>/dev/null || echo unknown)"
|
||||||
|
|
||||||
epoch="$(date +%s%N)$$"
|
epoch="$(date +%s%N)$$"
|
||||||
docker build --build-arg CHECK_EPOCH="$epoch" .
|
docker build --build-arg CHECK_EPOCH="$epoch" \
|
||||||
|
--build-arg VERSION="$version" \
|
||||||
|
--build-arg COMMIT="$commit" \
|
||||||
|
--build-arg COMMIT_DATE="$commit_date" \
|
||||||
|
.
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
@@ -24,7 +24,24 @@ main() {
|
|||||||
# whether the tree is clean. The Dockerfile now refuses to build
|
# whether the tree is clean. The Dockerfile now refuses to build
|
||||||
# without a non-empty value, so this is required, not optional.
|
# without a non-empty value, so this is required, not optional.
|
||||||
epoch="$(date +%s%N)$$"
|
epoch="$(date +%s%N)$$"
|
||||||
|
|
||||||
|
# Version, commit and build date are computed here on the host,
|
||||||
|
# where .git exists, and passed into the build. The build context
|
||||||
|
# excludes .git (see .dockerignore), so the container cannot derive
|
||||||
|
# them itself -- it used to try and always got "unknown", giving
|
||||||
|
# every image a "commit: unknown" it could not be traced from.
|
||||||
|
# VERSION comes from script/version, the source of truth shared with
|
||||||
|
# the Makefile, so a Docker build reports the same string (tag,
|
||||||
|
# dev-<sha>, or a -dirty variant) that a local build of the same
|
||||||
|
# tree would.
|
||||||
|
version="$("$SCRIPT_DIR/version")"
|
||||||
|
commit="$(git rev-parse HEAD 2>/dev/null || echo unknown)"
|
||||||
|
commit_date="$(git show -s --format=%cs HEAD 2>/dev/null || echo unknown)"
|
||||||
|
|
||||||
docker build --build-arg CHECK_EPOCH="$epoch" \
|
docker build --build-arg CHECK_EPOCH="$epoch" \
|
||||||
|
--build-arg VERSION="$version" \
|
||||||
|
--build-arg COMMIT="$commit" \
|
||||||
|
--build-arg COMMIT_DATE="$commit_date" \
|
||||||
-t "$("$SCRIPT_DIR/projectname")" .
|
-t "$("$SCRIPT_DIR/projectname")" .
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user