Author SHA1 Message Date
sneak 311756d452 Quiet only the stdout UI under --json, not the log level (closes #112)
check / check (pull_request) Failing after 1s
--json was folded into Quiet, which pinned the stderr log level to WARN.
So `prune --json` gave a machine consumer no record of the local index
rows it deleted, even under --verbose: the audit records were gated off
stdout and pinned below the level on stderr.

--json now quiets only the stdout UI, keeping the JSON document clean
(issue #108); the stderr log level follows --verbose/--debug again, since
diagnostics have gone to stderr since #82. The same coupling is removed
for `snapshot verify`, `snapshot remove`, and `remote info`, which
carried it for the same outdated reason.

A test asserts `--verbose prune --json` emits the cleanup record on
stderr while stdout stays exactly one document, and that --json alone
keeps it below the level.

model: claude-opus-4-8
2026-09-21 19:43:56 +00:00
36 changed files with 569 additions and 1560 deletions
+3 -8
View File
@@ -104,12 +104,7 @@ Version: 2025-06-08
13. Pre-1.0: NEVER write database migrations. There are no live databases 13. Pre-1.0: NEVER write database migrations. There are no live databases
anywhere — every user's local index can be rebuilt from a fresh full anywhere — every user's local index can be rebuilt from a fresh full
backup. To change the schema, edit `internal/database/schema/001.sql` backup. When the schema changes, just change `schema.sql` (and any code
(and any code that touches the affected tables) directly; do not add new that touches the affected tables). The local index is disposable until
numbered schema files. Those numbered files and the `schema_migrations` 1.0 ships and is tagged.
table they populate only bootstrap a fresh database — they are not an
upgrade path. The local index is disposable until 1.0 ships and is
tagged; once 1.0 is tagged that clause expires and the question of
upgrading existing indexes returns. See [`docs/DATAMODEL.md`](docs/DATAMODEL.md)
for the full explanation.
+3 -24
View File
@@ -20,6 +20,8 @@
# golang:1.26.1-alpine, 2026-03-17 # golang:1.26.1-alpine, 2026-03-17
FROM golang:1.26.1-alpine@sha256:2389ebfa5b7f43eeafbd6be0c3700cc46690ef842ad962f6c5bd6be49ed82039 AS builder FROM golang:1.26.1-alpine@sha256:2389ebfa5b7f43eeafbd6be0c3700cc46690ef842ad962f6c5bd6be49ed82039 AS builder
ARG VERSION=dev
# Build tooling: make, plus a C toolchain because `go test -race` needs cgo. # Build tooling: make, plus a C toolchain because `go test -race` needs cgo.
# The sqlite driver is pure Go (modernc.org/sqlite), so no sqlite library or # The sqlite driver is pure Go (modernc.org/sqlite), so no sqlite library or
# CLI is required. # CLI is required.
@@ -64,31 +66,8 @@ RUN [ -n "$CHECK_EPOCH" ] || exit 1
RUN echo "check epoch: ${CHECK_EPOCH}" && make fmt-check RUN echo "check epoch: ${CHECK_EPOCH}" && make fmt-check
RUN echo "check epoch: ${CHECK_EPOCH}" && make test RUN echo "check epoch: ${CHECK_EPOCH}" && make test
# Version, commit and build date are computed on the host by
# script/docker and script/cibuild (where .git exists) and passed in as
# build args. The build context excludes .git (see .dockerignore), so
# the build cannot derive them itself: it used to try, with `git
# rev-parse` inside this stage, and always got "unknown". VERSION comes
# from script/version, the source of truth shared with the Makefile, so
# it carries the same tag / dev-<sha> / -dirty rules and a Docker image
# reports the same string a local build of the same tree would.
#
# The defaults are the fallback for a bare `docker build .` that passes
# none of them: an unset arg would otherwise stamp an empty string and
# produce an image that cannot report its own version, commit or date.
# They match what an out-of-git build reports elsewhere.
#
# These ARGs sit here, after the checks, rather than at the top of the
# stage: every commit changes their values, and a value change
# invalidates all layers below the ARG. Declared up top they would bust
# `go mod download`; here they only rekey this build layer, which the
# COPY of the sources above already rebuilds on any change anyway.
ARG VERSION=dev
ARG COMMIT=unknown
ARG COMMIT_DATE=unknown
# Build (pure Go, no CGO required since we use modernc.org/sqlite) # Build (pure Go, no CGO required since we use modernc.org/sqlite)
RUN CGO_ENABLED=0 go build -ldflags "-X 'sneak.berlin/go/vaultik/internal/globals.Version=${VERSION}' -X 'sneak.berlin/go/vaultik/internal/globals.Commit=${COMMIT}' -X 'sneak.berlin/go/vaultik/internal/globals.CommitDate=${COMMIT_DATE}'" -o /vaultik ./cmd/vaultik RUN CGO_ENABLED=0 go build -ldflags "-X 'sneak.berlin/go/vaultik/internal/globals.Version=${VERSION}' -X 'sneak.berlin/go/vaultik/internal/globals.Commit=$(git rev-parse HEAD 2>/dev/null || echo unknown)' -X 'sneak.berlin/go/vaultik/internal/globals.CommitDate=$(git show -s --format=%cs HEAD 2>/dev/null || echo unknown)'" -o /vaultik ./cmd/vaultik
# Runtime stage # Runtime stage
# alpine:3.21, 2026-02-25 # alpine:3.21, 2026-02-25
+11 -70
View File
@@ -84,57 +84,6 @@ VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot restore <snapshot-i
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w # 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
``` ```
## restoring on another machine
Restoring on a host that never ran the backup — a replacement machine
after the original is gone — is the case vaultik is built for. That host
needs only three things: the `vaultik` binary, the age **private** key,
and the storage credentials for the destination. It does **not** need the
local index, the original config file, or the original hostname.
```sh
# install
go install sneak.berlin/go/vaultik/cmd/vaultik@latest
# create a config and point it at the ORIGINAL backup destination
vaultik config init
vaultik config set storage_url "s3://bucket/prefix?endpoint=https://s3.example.com"
vaultik config set s3.access_key_id "..."
vaultik config set s3.secret_access_key "..."
# see what is on the destination store
vaultik snapshot list
```
`snapshot list` reads the destination store without the private key. A
snapshot that is not in this host's (empty) local index is shown as
remote-only: its row is identified by `<remote only:...>` rather than by
a `hostname_name_timestamp` name, because the name lives only in the
local index and the encrypted database and cannot be recovered from the
store. Its timestamp and compressed size are real. (See the `snapshot
list` description under [command details](#command-details) for the full
explanation.)
Use that remote key — the hex printed inside `<remote only:...>`, or the
full `remote_key` from `snapshot list --json` — to restore and verify:
```sh
# restore everything to /tmp/restored, then check every restored file's
# chunk hashes
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' \
vaultik snapshot restore --verify <remote-key> /tmp/restored
# optionally, deep-verify the snapshot against the store (downloads and
# cryptographically checks every blob)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' \
vaultik snapshot verify --deep <remote-key>
```
`age_recipients` (the public key) is not needed to restore — only the
private key in `VAULTIK_AGE_SECRET_KEY`. Both the abbreviated key printed
in the table and the full 64-character key from `--json` are accepted; a
leading part of the key is enough as long as it is unambiguous.
--- ---
## cli ## cli
@@ -296,8 +245,6 @@ local index alone, and still exits zero.
* Default (shallow): checks that all blobs referenced in the manifest exist in storage * Default (shallow): checks that all blobs referenced in the manifest exist in storage
* `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the * `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the
encrypted metadata database encrypted metadata database
* Accepts the same identifiers as `snapshot restore`: a snapshot ID, or a
remote-only snapshot's remote key (or an unambiguous leading part of it)
* `--json`: Output results as JSON * `--json`: Output results as JSON
**`snapshot purge`**: Remove old snapshots based on criteria. Retention is **`snapshot purge`**: Remove old snapshots based on criteria. Retention is
@@ -328,10 +275,6 @@ on the destination in one go, use `vaultik remote nuke --force`.
**`snapshot restore`**: Restore files from a backup snapshot. **`snapshot restore`**: Restore files from a backup snapshot.
* Requires `VAULTIK_AGE_SECRET_KEY` environment variable * Requires `VAULTIK_AGE_SECRET_KEY` environment variable
* Accepts a snapshot ID, or — for a snapshot only on the destination
store — its remote key (or an unambiguous leading part of it) as shown
by `snapshot list`. See
[restoring on another machine](#restoring-on-another-machine).
* Optional path arguments to restore specific files/directories (default: all) * Optional path arguments to restore specific files/directories (default: all)
* Preserves file permissions, timestamps, ownership (ownership requires root), * Preserves file permissions, timestamps, ownership (ownership requires root),
symlinks, and empty directories symlinks, and empty directories
@@ -514,13 +457,9 @@ Key fields:
sequentially. Restore speed is bound by single-stream throughput. sequentially. Restore speed is bound by single-stream throughput.
* **Device nodes, named pipes, and sockets are silently skipped.** Only * **Device nodes, named pipes, and sockets are silently skipped.** Only
regular files, directories, and symlinks are backed up. regular files, directories, and symlinks are backed up.
* **No upgrade path between versions.** There is no supported way to carry * **No database migrations.** If the local SQLite schema changes between
an existing local index across a schema change; if the local SQLite versions, delete the local database (`vaultik database delete`) and run
schema changes between versions, delete the local database (`vaultik a full backup. Remote storage is unaffected.
database delete`) and run a full backup. Remote storage is unaffected.
(The binary does embed numbered schema files and a `schema_migrations`
table to bootstrap a fresh database — see [`docs/DATAMODEL.md`](docs/DATAMODEL.md)
— but that is not an upgrade path.)
* **Files that change during backup may be inconsistent.** There is no * **Files that change during backup may be inconsistent.** There is no
filesystem snapshot or freeze. If a file is modified between the scan filesystem snapshot or freeze. If a file is modified between the scan
and chunk phases, the backed-up copy may reflect a partial write. and chunk phases, the backed-up copy may reflect a partial write.
@@ -586,12 +525,14 @@ priority.
### infrastructure ### infrastructure
* **Cross-version schema upgrades.** There is no upgrade path between * **Cross-machine restore documentation.** The "restore from
released versions — pre-1.0 schema changes are handled by `vaultik another host" workflow works but isn't documented as a
database delete` plus a full re-scan (see first-class operation in this README. Worth a dedicated section
[`docs/DATAMODEL.md`](docs/DATAMODEL.md)). Post-1.0 we'll need a once it's settled.
migration story to keep existing index databases usable across * **Schema migrations.** Currently nonexistent — pre-1.0 schema
upgrades. changes are handled by `vaultik database delete` plus a full
re-scan. Post-1.0 we'll need a migration story to keep existing
index databases usable across upgrades.
* **Storage backend coverage tests.** S3, file://, and rclone:// * **Storage backend coverage tests.** S3, file://, and rclone://
all share the Storer interface but the rclone path is the least all share the Storer interface but the rclone path is the least
exercised in CI. exercised in CI.
+11 -13
View File
@@ -25,19 +25,6 @@ release" is exactly the contradiction
# Completed Steps # Completed Steps
- 2026-09-21: Stopped an interrupted blob upload from making a later
backup deduplicate against data that was never stored
([issue #148](https://git.eeqj.de/sneak/vaultik/issues/148)). The
packer commits a blob's `chunks`, `blob_chunks`, and `blobs` rows
before the upload is attempted, so a failed upload left chunk rows
behind and the next run skipped re-uploading them, producing a
snapshot that reported success but could not be restored. A run now
deduplicates only against chunks held by a blob whose `uploaded_ts` is
set, and at startup drops any un-uploaded blob rows (and the chunks
they orphan) so the affected data is re-chunked and re-uploaded. Blobs
recorded with no remote backend are marked uploaded so this invariant
holds uniformly.
- 2026-09-21: Stopped `prune` from reporting a failed row count as 0 - 2026-09-21: Stopped `prune` from reporting a failed row count as 0
([issue #96](https://git.eeqj.de/sneak/vaultik/issues/96)). The seven ([issue #96](https://git.eeqj.de/sneak/vaultik/issues/96)). The seven
`getTableCount` reads in `PruneDatabase` discarded their error, so a `getTableCount` reads in `PruneDatabase` discarded their error, so a
@@ -48,6 +35,17 @@ release" is exactly the contradiction
no `--json` representation — under `--json` the summary is suppressed no `--json` representation — under `--json` the summary is suppressed
entirely — so nothing there can show a false `0`. entirely — so nothing there can show a false `0`.
- 2026-09-21: Stopped `--json` from silencing stderr diagnostics
([issue #112](https://git.eeqj.de/sneak/vaultik/issues/112)). `--json`
used to be folded into `Quiet`, which pinned the log level to `WARN`,
so `prune --json` gave a machine consumer no record of the local index
rows it deleted even under `--verbose`. `--json` now quiets only the
stdout UI (the JSON document must stay clean, per
[issue #108](https://git.eeqj.de/sneak/vaultik/issues/108)); the stderr
log level follows `--verbose`/`--debug` again. The coupling was
removed the same way for `snapshot verify`, `snapshot remove`, and
`remote info`, which carried it for the same outdated reason.
- 2026-09-21: Made the s3 storage backend report a missing object as - 2026-09-21: Made the s3 storage backend report a missing object as
`storage.ErrNotFound`, like the `file` and `rclone` backends and as the `storage.ErrNotFound`, like the `file` and `rclone` backends and as the
`Storer` interface documents. `S3Storer.Get` and `Stat` returned the raw `Storer` interface documents. `S3Storer.Get` and `Stat` returned the raw
-102
View File
@@ -1,102 +0,0 @@
package main_test
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// This file guards the version stamping of the product image (issue
// #75). The failure it protects against is silent: the image still
// builds and runs, but `vaultik version` inside it reports "commit:
// unknown", so an operator cannot tell which source produced a given
// backup. .dockerignore excludes .git, so the build cannot derive the
// commit itself; the values must be computed on the host and passed in.
//
// These are parses of the committed files, for the same reason the lint
// guards next door are: shelling out to docker would nest a build
// inside `make test`. That `vaultik version` in the built image really
// prints the host's version is verified by hand and recorded on the
// pull request.
// dockerScript is script/docker, relative to the repository root.
const dockerScript = "script/docker"
// versionArgs are the ldflag targets the build stamps and, matching
// them, the build args the host must supply. The names line up so the
// same list checks both files.
func versionArgs() []string {
return []string{"VERSION", "COMMIT", "COMMIT_DATE"}
}
// TestProductDockerfileTakesVersionAsBuildArgs fails unless the build
// declares each version arg and stamps it into the binary by ldflag
// reference, rather than computing it in the container.
func TestProductDockerfileTakesVersionAsBuildArgs(t *testing.T) {
t.Parallel()
found := instructions(t, productDockerfile)
for _, arg := range versionArgs() {
require.GreaterOrEqual(t, indexOf(found, "ARG "+arg), 0,
"%s must declare `ARG %s` so the host can pass it in",
productDockerfile, arg)
assertLdflagReferences(t, found, arg)
}
}
// TestProductDockerfileDoesNotDeriveVersionItself is the anti-regression
// for the original defect: the container ran `git rev-parse`, but .git
// is not in the build context, so it always resolved to "unknown". No
// git command may reach into a build that cannot see the history.
func TestProductDockerfileDoesNotDeriveVersionItself(t *testing.T) {
t.Parallel()
text := instructionText(readRepoFile(t, productDockerfile))
assert.NotContains(t, text, "git ",
"%s must not run git: .git is excluded from the build context, so"+
" any value it derives is wrong. Pass version, commit and date"+
" in as build args instead.", productDockerfile)
}
// TestDockerScriptComputesVersionOnTheHost fails unless script/docker
// derives each value where .git exists and passes it as a build arg,
// with VERSION coming from script/version so a Docker build reports the
// same string a local build of the same tree would.
func TestDockerScriptComputesVersionOnTheHost(t *testing.T) {
t.Parallel()
script := readRepoFile(t, dockerScript)
for _, arg := range versionArgs() {
assert.Contains(t, script, "--build-arg "+arg+"=",
"%s must pass --build-arg %s to the build", dockerScript, arg)
}
assert.Contains(t, script, "/version",
"%s must take VERSION from script/version, the source of truth"+
" shared with the Makefile", dockerScript)
}
// assertLdflagReferences fails unless some build instruction stamps the
// named variable from the ARG (a ${arg} reference), not from a value
// computed inside the container.
func assertLdflagReferences(t *testing.T, found []string, arg string) {
t.Helper()
for _, instruction := range found {
if strings.HasPrefix(instruction, "RUN ") &&
strings.Contains(instruction, "go build") &&
strings.Contains(instruction, "${"+arg+"}") {
return
}
}
assert.Fail(t, "version arg is declared but never stamped",
"the go build in %s must reference ${%s} in its ldflags, or the"+
" arg is passed and discarded", productDockerfile, arg)
}
+2 -6
View File
@@ -304,14 +304,10 @@ func instructionText(contents string) string {
} }
// indexOf returns the position of the first instruction equal to, or // indexOf returns the position of the first instruction equal to, or
// beginning with, want; -1 if there is none. An `ARG NAME=default` // beginning with, want; -1 if there is none.
// counts as beginning with `ARG NAME`, so a declared arg is found
// whether or not it carries a default.
func indexOf(found []string, want string) int { func indexOf(found []string, want string) int {
for i, instruction := range found { for i, instruction := range found {
if instruction == want || if instruction == want || strings.HasPrefix(instruction, want+" ") {
strings.HasPrefix(instruction, want+" ") ||
strings.HasPrefix(instruction, want+"=") {
return i return i
} }
} }
+1 -11
View File
@@ -10,16 +10,6 @@ import (
) )
func main() { func main() {
os.Exit(run())
}
// run sets up optional profiling, runs the CLI, and returns the process
// exit code. os.Exit lives in main so it fires only after run's deferred
// profile writers have flushed. cli.Entry returns a status code rather
// than calling os.Exit itself: an os.Exit from inside it would skip
// these defers and truncate the profile of a failing command -- exactly
// the command one most often wants to profile.
func run() int {
// CPU profiling: set VAULTIK_CPUPROFILE=/path/to/cpu.prof // CPU profiling: set VAULTIK_CPUPROFILE=/path/to/cpu.prof
if cpuProfile := os.Getenv("VAULTIK_CPUPROFILE"); cpuProfile != "" { if cpuProfile := os.Getenv("VAULTIK_CPUPROFILE"); cpuProfile != "" {
f, err := os.Create(cpuProfile) //nolint:gosec // G304: operator-set path f, err := os.Create(cpuProfile) //nolint:gosec // G304: operator-set path
@@ -56,5 +46,5 @@ func run() int {
}() }()
} }
return cli.Entry() cli.Entry()
} }
+5 -24
View File
@@ -5,30 +5,11 @@
Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication. Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication.
**Important Notes:** **Important Notes:**
- **No Migration Support (pre-1.0)**: Vaultik does not support database schema
This section is the authoritative explanation of the schema/migration story; migrations. The local index is treated as disposable — if the schema changes,
other documents (the README and `AGENTS.md`) link here. delete the local SQLite database (`vaultik database delete`) and run a full
backup. The remote storage is unaffected; the new index will re-deduplicate
- **No upgrade path between versions (pre-1.0)**: Vaultik has no supported way to against existing remote blobs.
carry an existing local index across a schema change. The index is disposable
— if the on-disk schema changes between versions, delete the local SQLite
database (`vaultik database delete`) and run a full backup. Remote storage is
unaffected; the new index re-deduplicates against existing remote blobs. This
is the standing project policy, and it is separate from the schema bootstrap
described next.
- **Schema bootstrap**: a fresh database is populated from numbered SQL files
embedded in the binary under `internal/database/schema/`. `000.sql` creates the
`schema_migrations` table; `001.sql` creates the application tables. On opening
a database the code applies each numbered file that has not yet run and records
its version in `schema_migrations`. This bootstraps a new database; it does not
upgrade an existing one between released versions.
- **Changing the schema (pre-1.0)**: edit `internal/database/schema/001.sql` (and
the code that touches the affected tables) directly. Do not add new numbered
files — there is no installed base to migrate.
- **Disposability expires at 1.0**: the index is treated as disposable only until
1.0 ships and is tagged. Once 1.0 is tagged that clause expires and the
question of upgrading existing indexes returns. It is deliberately left open
here.
- **Version Compatibility**: In rare cases, you may need to use the same version - **Version Compatibility**: In rare cases, you may need to use the same version
of Vaultik to restore a backup as was used to create it. This ensures of Vaultik to restore a backup as was used to create it. This ensures
compatibility with the metadata format stored in S3. compatibility with the metadata format stored in S3.
+46 -90
View File
@@ -11,7 +11,6 @@ import (
"os/signal" "os/signal"
"path/filepath" "path/filepath"
"strings" "strings"
"sync"
"syscall" "syscall"
"time" "time"
@@ -49,6 +48,11 @@ type AppOptions struct {
// silenced — per the documented convention that --quiet suppresses // silenced — per the documented convention that --quiet suppresses
// non-error output only. The startup banner is printed by Entry // non-error output only. The startup banner is printed by Entry
// before cobra parses arguments, gated by the same arg-level check. // before cobra parses arguments, gated by the same arg-level check.
//
// --json quiets the UI here too, because stdout then carries a JSON
// document and human narration would corrupt it. Unlike Quiet it does
// not lower the stderr log level (issue #112), so --verbose/--debug
// still surface diagnostics alongside the document.
func setupGlobals( func setupGlobals(
lc fx.Lifecycle, g *globals.Globals, v *vaultik.Vaultik, opts log.Options, lc fx.Lifecycle, g *globals.Globals, v *vaultik.Vaultik, opts log.Options,
) { ) {
@@ -56,7 +60,7 @@ func setupGlobals(
OnStart: func(_ context.Context) error { OnStart: func(_ context.Context) error {
g.StartTime = time.Now().UTC() g.StartTime = time.Now().UTC()
if opts.Cron || opts.Quiet { if opts.Cron || opts.Quiet || opts.JSON {
v.UI.SetQuiet(true) v.UI.SetQuiet(true)
} }
@@ -197,54 +201,54 @@ func RunApp(ctx context.Context, app *fx.App) error {
} }
} }
// errReported marks a failure the operation has already shown the user // runVaultikApp runs the standard single-operation command lifecycle
// (and deliberately withheld under --json). Entry turns it into a // shared by the list/purge/verify/remove/remote-info subcommands:
// non-zero exit status without printing anything further, so the error // resolve the config, start the fx app, run op against the Vaultik
// line is not doubled. It flows up from RunOperation through cobra to // instance in a goroutine, report a failure prefixed with failMsg
// Entry. // (suppressed while suppressErrors is true, e.g. under --json), then
var errReported = errors.New("operation failed") // trigger shutdown. The operation is cancelled when the app stops.
// jsonOutput marks a command whose stdout is a JSON document: it quiets
// RunOperation runs op against the Vaultik instance inside the fx app // the UI but, unlike Quiet, leaves the stderr log level alone.
// and turns a failure into a returned error rather than an os.Exit from func runVaultikApp(
// within the goroutine. An os.Exit there skipped main's deferred cmd *cobra.Command, jsonOutput, suppressErrors bool,
// profile writers -- so profiling a failing command yielded a truncated failMsg string, op func(v *vaultik.Vaultik) error,
// profile (issue #75) -- and RunWithApp's PID-lock release, and denied
// the app any graceful shutdown; returning the error to the top runs
// all three.
//
// op runs in a goroutine so OnStart returns promptly and an interrupt
// can still cancel through OnStop; when it finishes, success or failure,
// it triggers shutdown, which is what lets RunWithApp return. report is
// called with a non-canceled failure so the caller can log it (and
// suppress it under --json) before it becomes errReported. A context
// cancellation is the interrupt path, not a failure: it is neither
// reported nor counted as one.
func RunOperation(
ctx context.Context, opts AppOptions,
op func(v *vaultik.Vaultik) error, report func(err error),
) error { ) error {
var ( configPath, err := ResolveConfigPath()
mu sync.Mutex if err != nil {
failed bool return err
) }
opts.Invokes = append(opts.Invokes, rootFlags := GetRootFlags()
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
JSON: jsonOutput,
},
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) { fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{ lc.Append(fx.Hook{
OnStart: func(_ context.Context) error { OnStart: func(_ context.Context) error {
go func() { go func() {
err := op(v) err := op(v)
if err != nil && !errors.Is(err, context.Canceled) { if err != nil {
report(err) if !errors.Is(err, context.Canceled) {
if !suppressErrors {
mu.Lock() log.Error(failMsg, "error", err)
failed = true ReportErrorf("%s: %v", failMsg, err)
mu.Unlock()
} }
stopErr := v.Shutdowner.Shutdown() os.Exit(1)
if stopErr != nil { }
log.Error("Failed to shutdown", "error", stopErr) }
err = v.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
} }
}() }()
@@ -256,56 +260,8 @@ func RunOperation(
return nil return nil
}, },
}) })
})) }),
err := RunWithApp(ctx, opts)
if err != nil {
return err
}
// The goroutine sets failed before triggering the shutdown that lets
// RunWithApp return, so the write is in place by the time we read it.
mu.Lock()
defer mu.Unlock()
if failed {
return errReported
}
return nil
}
// runVaultikApp runs the standard single-operation command lifecycle
// shared by the list/purge/verify/remove/remote-info subcommands:
// resolve the config, then run op against the Vaultik instance through
// RunOperation, reporting a failure prefixed with failMsg (suppressed
// while suppressErrors is true, e.g. under --json). extraQuiet is OR-ed
// into LogOptions.Quiet (e.g. --json output modes).
func runVaultikApp(
cmd *cobra.Command, extraQuiet, suppressErrors bool,
failMsg string, op func(v *vaultik.Vaultik) error,
) error {
configPath, err := ResolveConfigPath()
if err != nil {
return err
}
rootFlags := GetRootFlags()
return RunOperation(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet || extraQuiet,
}, },
}, op, func(err error) {
if suppressErrors {
return
}
log.Error(failMsg, "error", err)
ReportErrorf("%s: %v", failMsg, err)
}) })
} }
+2 -17
View File
@@ -1,7 +1,6 @@
package cli package cli
import ( import (
"errors"
"io" "io"
"os" "os"
"strings" "strings"
@@ -20,11 +19,7 @@ const shortCommitLen = 12
// flag is present in os.Args — see bannerSuppressedInArgs), executes the // flag is present in os.Args — see bannerSuppressedInArgs), executes the
// root cobra command, and routes any returned error through the // root cobra command, and routes any returned error through the
// ui.Writer so the user sees a properly formatted "🛑 ERROR:" line. // ui.Writer so the user sees a properly formatted "🛑 ERROR:" line.
// func Entry() {
// It returns the process exit code (0 on success, 1 on error) rather
// than calling os.Exit, so that main's deferred profile writers run
// before the process ends. See run in cmd/vaultik/main.go.
func Entry() int {
emitStartupBanner(os.Args[1:], os.Stdout) emitStartupBanner(os.Args[1:], os.Stdout)
rootCmd := NewRootCommand() rootCmd := NewRootCommand()
@@ -32,19 +27,9 @@ func Entry() int {
err := rootCmd.Execute() err := rootCmd.Execute()
if err != nil { if err != nil {
// An operation that ran inside the fx app has already reported
// its own failure (and suppressed it under --json); errReported
// says so. Printing it again here would double the error line.
// Every other error — bad arguments, a config that would not
// load — reaches Entry unreported, so it is shown here.
if !errors.Is(err, errReported) {
ReportErrorf("%s", err.Error()) ReportErrorf("%s", err.Error())
os.Exit(1)
} }
return 1
}
return 0
} }
// emitStartupBanner writes the startup banner to w unless args (the // emitStartupBanner writes the startup banner to w unless args (the
+1 -1
View File
@@ -230,7 +230,7 @@ func TestEntryJSONStdoutIsExactlyOneDocument(t *testing.T) {
programName, flagConfig, configPath, cmdSnapshot, cmdList, flagJSON, programName, flagConfig, configPath, cmdSnapshot, cmdList, flagJSON,
} }
stdout := captureProcessStdout(t, func() { _ = Entry() }) stdout := captureProcessStdout(t, Entry)
requireExactlyOneJSONDocument(t, stdout) requireExactlyOneJSONDocument(t, stdout)
@@ -0,0 +1,139 @@
package cli //nolint:testpackage // shares the prune fixtures and capture helpers
import (
"bytes"
"io"
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// staleRecordLogMessage is the local-cleanup audit line CleanupLocalSnapshots
// logs for each stale record. It is exactly the signal issue #112 says a
// machine consumer lost under --json: gated off stdout, and pinned below
// the log level on stderr because --json used to force Quiet.
const staleRecordLogMessage = "Removing stale local snapshot record"
// TestEntryPruneJSONStderrHonoursVerbosity is the end-to-end regression
// guard for issue #112. Under --json the log level must still follow
// --verbose/--debug rather than being pinned to WARN, so the
// local-cleanup records reach stderr under --verbose while stdout stays
// exactly one JSON document; without --verbose they stay below the
// level, as they do without --json.
//
// Both halves are asserted together on the same run, because the fix has
// to keep the document clean (issue #108) while freeing stderr.
//
// Not parallel: it replaces os.Args, os.Stdout, os.Stderr and the xdg
// globals.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntryPruneJSONStderrHonoursVerbosity(t *testing.T) {
for _, testCase := range []struct {
name string
verbose bool
wantOnStderr bool
}{
{
name: "verbose json surfaces the cleanup record on stderr",
verbose: true,
wantOnStderr: true,
},
{
name: "json alone keeps the cleanup record below the level",
verbose: false,
wantOnStderr: false,
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath := writeHermeticPruneConfig(t, true)
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
args := []string{
programName, flagConfig, configPath, cmdPrune, flagJSON,
}
if testCase.verbose {
args = append(args, "--verbose")
}
os.Args = args
stdout, stderr := captureProcessStdoutAndStderr(t, Entry)
// The document stays clean in both cases: freeing stderr must
// not regress issue #108.
requireExactlyOneJSONDocument(t, stdout)
if testCase.wantOnStderr {
assert.Contains(t, stderr, staleRecordLogMessage,
"--verbose --json must emit the cleanup record on stderr")
assert.Contains(t, stderr, stalePruneSnapshotID,
"the record must name the snapshot it removed")
} else {
assert.NotContains(t, stderr, staleRecordLogMessage,
"without --verbose the record stays below the log level")
}
})
}
}
// captureProcessStdoutAndStderr redirects both of the process's own
// standard streams to pipes for the duration of fn and returns what was
// written to each. The redirection is at the file-descriptor level
// because the logger binds os.Stderr when it initializes inside fn, and
// the JSON document reaches os.Stdout independently; the point is to see
// where each actually lands.
//
// Not parallel-safe: os.Stdout and os.Stderr are process-global.
func captureProcessStdoutAndStderr(t *testing.T, fn func()) (string, string) {
t.Helper()
outReader, outWriter, err := os.Pipe()
require.NoError(t, err)
errReader, errWriter, err := os.Pipe()
require.NoError(t, err)
previousOut, previousErr := os.Stdout, os.Stderr
os.Stdout, os.Stderr = outWriter, errWriter
capturedOut := drain(outReader)
capturedErr := drain(errReader)
fn()
os.Stdout, os.Stderr = previousOut, previousErr
require.NoError(t, outWriter.Close())
require.NoError(t, errWriter.Close())
out, errOut := <-capturedOut, <-capturedErr
require.NoError(t, outReader.Close())
require.NoError(t, errReader.Close())
return out, errOut
}
// drain copies a reader to a string on a goroutine and delivers the
// result once the writer end is closed.
func drain(reader io.Reader) <-chan string {
captured := make(chan string, 1)
go func() {
var buf bytes.Buffer
_, _ = io.Copy(&buf, reader)
captured <- buf.String()
}()
return captured
}
+1 -1
View File
@@ -81,7 +81,7 @@ func TestEntryPruneJSONStdoutIsExactlyOneDocument(t *testing.T) {
programName, flagConfig, configPath, cmdPrune, flagJSON, programName, flagConfig, configPath, cmdPrune, flagJSON,
} }
stdout := captureProcessStdout(t, func() { _ = Entry() }) stdout := captureProcessStdout(t, Entry)
requireExactlyOneJSONDocument(t, stdout) requireExactlyOneJSONDocument(t, stdout)
-58
View File
@@ -1,58 +0,0 @@
package cli //nolint:testpackage // shares programName and the capture helpers
import (
"os"
"testing"
"github.com/stretchr/testify/assert"
)
// TestEntryReturnsStatusCode pins the contract main() relies on for
// issue #75: Entry reports success or failure through its return value
// and never calls os.Exit. An os.Exit from inside Entry would skip
// main's deferred profile writers and truncate the profile of a failing
// command. main turns this code into os.Exit only after those defers
// run, so a failing command must come back with a non-zero code rather
// than ending the process here.
//
// Stdout is captured only to keep the banner and command output off the
// test log; the assertion is on the returned code.
//
//nolint:paralleltest // replaces os.Args and rootFlags
func TestEntryReturnsStatusCode(t *testing.T) {
for _, testCase := range []struct {
name string
args []string
want int
}{
{
// version is self-contained: it needs no config and no
// destination store, so it exercises the success path.
name: "successful command returns zero",
args: []string{programName, "version"},
want: 0,
},
{
name: "unknown command returns one",
args: []string{programName, "no-such-command"},
want: 1,
},
} {
t.Run(testCase.name, func(t *testing.T) {
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
os.Args = testCase.args
var code int
_ = captureProcessStdout(t, func() { code = Entry() })
assert.Equal(t, testCase.want, code)
})
}
}
+35 -4
View File
@@ -1,7 +1,12 @@
package cli package cli
import ( import (
"context"
"errors"
"os"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik" "sneak.berlin/go/vaultik/internal/vaultik"
) )
@@ -28,18 +33,44 @@ func NewInfoCommand() *cobra.Command {
// Use the app framework // Use the app framework
rootFlags := GetRootFlags() rootFlags := GetRootFlags()
return RunOperation(cmd.Context(), AppOptions{ return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath, ConfigPath: configPath,
LogOptions: log.Options{ LogOptions: log.Options{
Verbose: rootFlags.Verbose, Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug, Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet, Quiet: rootFlags.Quiet,
}, },
}, func(v *vaultik.Vaultik) error { Modules: []fx.Option{},
return v.ShowInfo() Invokes: []fx.Option{
}, func(err error) { fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
go func() {
err := v.ShowInfo()
if err != nil {
if !errors.Is(err, context.Canceled) {
log.Error("Failed to show info", "error", err) log.Error("Failed to show info", "error", err)
ReportErrorf("Failed to show info: %v", err) ReportErrorf("Failed to show info: %v", err)
os.Exit(1)
}
}
err = v.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(_ context.Context) error {
v.Cancel()
return nil
},
})
}),
},
}) })
}, },
} }
+44 -9
View File
@@ -1,7 +1,12 @@
package cli package cli
import ( import (
"context"
"errors"
"os"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik" "sneak.berlin/go/vaultik/internal/vaultik"
) )
@@ -36,22 +41,52 @@ work (e.g. after a crashed backup or to reclaim storage).`,
// Use the app framework like other commands // Use the app framework like other commands
rootFlags := GetRootFlags() rootFlags := GetRootFlags()
return RunOperation(cmd.Context(), AppOptions{ return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath, ConfigPath: configPath,
LogOptions: log.Options{ LogOptions: log.Options{
Verbose: rootFlags.Verbose, Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug, Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet || opts.JSON, Quiet: rootFlags.Quiet,
JSON: opts.JSON,
}, },
}, func(v *vaultik.Vaultik) error { Modules: []fx.Option{},
return v.Prune(opts) Invokes: []fx.Option{
}, func(err error) { fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
if opts.JSON { lc.Append(fx.Hook{
return OnStart: func(_ context.Context) error {
} // Start the prune operation in a goroutine
go func() {
// Run the prune operation
err := v.Prune(opts)
if err != nil {
if !errors.Is(err, context.Canceled) {
if !opts.JSON {
log.Error("Prune operation failed", "error", err) log.Error("Prune operation failed", "error", err)
ReportErrorf("Prune failed: %v", err) ReportErrorf("Prune failed: %v", err)
}
os.Exit(1)
}
}
// Shutdown the app when prune completes
err = v.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(_ context.Context) error {
log.Debug("Stopping prune operation")
v.Cancel()
return nil
},
})
}),
},
}) })
}, },
} }
+38 -9
View File
@@ -1,9 +1,12 @@
package cli package cli
import ( import (
"context"
"errors" "errors"
"os"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik" "sneak.berlin/go/vaultik/internal/vaultik"
) )
@@ -80,22 +83,48 @@ func newRemoteInfoCommand() *cobra.Command {
rootFlags := GetRootFlags() rootFlags := GetRootFlags()
return RunOperation(cmd.Context(), AppOptions{ return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath, ConfigPath: configPath,
LogOptions: log.Options{ LogOptions: log.Options{
Verbose: rootFlags.Verbose, Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug, Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet || jsonOutput, Quiet: rootFlags.Quiet,
JSON: jsonOutput,
}, },
}, func(v *vaultik.Vaultik) error { Modules: []fx.Option{},
return v.RemoteInfo(jsonOutput) Invokes: []fx.Option{
}, func(err error) { fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
if jsonOutput { lc.Append(fx.Hook{
return OnStart: func(_ context.Context) error {
} go func() {
err := v.RemoteInfo(jsonOutput)
if err != nil {
if !errors.Is(err, context.Canceled) {
if !jsonOutput {
log.Error("Failed to get remote info", "error", err) log.Error("Failed to get remote info", "error", err)
ReportErrorf("Failed to get remote info: %v", err) ReportErrorf("Failed to get remote info: %v", err)
}
os.Exit(1)
}
}
err = v.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(_ context.Context) error {
v.Cancel()
return nil
},
})
}),
},
}) })
}, },
} }
+74 -18
View File
@@ -1,10 +1,13 @@
package cli package cli
import ( import (
"context"
"errors" "errors"
"fmt" "fmt"
"os"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik" "sneak.berlin/go/vaultik/internal/vaultik"
) )
@@ -83,8 +86,7 @@ specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
// Use the backup functionality from cli package // Use the backup functionality from cli package
rootFlags := GetRootFlags() rootFlags := GetRootFlags()
// --cron suppression is wired through v.UI by setupGlobals. return RunWithApp(cmd.Context(), AppOptions{
return RunOperation(cmd.Context(), AppOptions{
ConfigPath: configPath, ConfigPath: configPath,
LogOptions: log.Options{ LogOptions: log.Options{
Verbose: rootFlags.Verbose, Verbose: rootFlags.Verbose,
@@ -92,11 +94,42 @@ specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
Cron: opts.Cron, Cron: opts.Cron,
Quiet: rootFlags.Quiet, Quiet: rootFlags.Quiet,
}, },
}, func(v *vaultik.Vaultik) error { Modules: []fx.Option{},
return v.CreateSnapshot(opts) Invokes: []fx.Option{
}, func(err error) { fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
// Start the snapshot creation in a goroutine
go func() {
// --cron suppression is wired through v.UI by setupGlobals.
err := v.CreateSnapshot(opts)
if err != nil {
if !errors.Is(err, context.Canceled) {
log.Error("Snapshot creation failed", "error", err) log.Error("Snapshot creation failed", "error", err)
ReportErrorf("Snapshot creation failed: %v", err) ReportErrorf("Snapshot creation failed: %v", err)
os.Exit(1)
}
}
// Shutdown the app when snapshot completes
err = v.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(_ context.Context) error {
log.Debug("Stopping snapshot creation")
// Cancel the Vaultik context
v.Cancel()
return nil
},
})
}),
},
}) })
}, },
} }
@@ -188,10 +221,7 @@ func newSnapshotVerifyCommand() *cobra.Command {
cmd := &cobra.Command{ cmd := &cobra.Command{
Use: "verify <snapshot-id>", Use: "verify <snapshot-id>",
Short: "Verify snapshot integrity", Short: "Verify snapshot integrity",
Long: "Verifies that all blobs referenced in a snapshot exist.\n\n" + Long: "Verifies that all blobs referenced in a snapshot exist",
"The snapshot may be named by its ID or, on a host with no local\n" +
"index, by the remote key that 'snapshot list' prints for a\n" +
"remote-only snapshot (an unambiguous leading part is enough).",
Args: requireSnapshotIDArg, Args: requireSnapshotIDArg,
RunE: func(cmd *cobra.Command, args []string) error { RunE: func(cmd *cobra.Command, args []string) error {
snapshotID := args[0] snapshotID := args[0]
@@ -204,22 +234,48 @@ func newSnapshotVerifyCommand() *cobra.Command {
rootFlags := GetRootFlags() rootFlags := GetRootFlags()
return RunOperation(cmd.Context(), AppOptions{ return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath, ConfigPath: configPath,
LogOptions: log.Options{ LogOptions: log.Options{
Verbose: rootFlags.Verbose, Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug, Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet || opts.JSON, Quiet: rootFlags.Quiet,
JSON: opts.JSON,
}, },
}, func(v *vaultik.Vaultik) error { Modules: []fx.Option{},
return v.VerifySnapshotWithOptions(snapshotID, opts) Invokes: []fx.Option{
}, func(err error) { fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
if opts.JSON { lc.Append(fx.Hook{
return OnStart: func(_ context.Context) error {
} go func() {
err := v.VerifySnapshotWithOptions(snapshotID, opts)
if err != nil {
if !errors.Is(err, context.Canceled) {
if !opts.JSON {
log.Error("Verification failed", "error", err) log.Error("Verification failed", "error", err)
ReportErrorf("Verification failed: %v", err) ReportErrorf("Verification failed: %v", err)
}
os.Exit(1)
}
}
err = v.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(_ context.Context) error {
v.Cancel()
return nil
},
})
}),
},
}) })
}, },
} }
+81 -12
View File
@@ -1,8 +1,16 @@
package cli package cli
import ( import (
"context"
"errors"
"os"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/vaultik" "sneak.berlin/go/vaultik/internal/vaultik"
) )
@@ -17,6 +25,15 @@ type RestoreOptions struct {
Verify bool // Verify restored files after restore Verify bool // Verify restored files after restore
} }
// RestoreApp contains all dependencies needed for restore
type RestoreApp struct {
Globals *globals.Globals
Config *config.Config
Storage storage.Storer
Vaultik *vaultik.Vaultik
Shutdowner fx.Shutdowner
}
// newSnapshotRestoreCommand creates the 'snapshot restore' subcommand // newSnapshotRestoreCommand creates the 'snapshot restore' subcommand
func newSnapshotRestoreCommand() *cobra.Command { func newSnapshotRestoreCommand() *cobra.Command {
opts := &RestoreOptions{} opts := &RestoreOptions{}
@@ -31,10 +48,6 @@ target directory.
If no paths are specified, all files are restored. If no paths are specified, all files are restored.
If paths are specified, only matching files/directories are restored. If paths are specified, only matching files/directories are restored.
The snapshot may be named by its ID or, when restoring on a host with no
local index, by the remote key that 'snapshot list' prints for a
remote-only snapshot (an unambiguous leading part is enough).
Requires the VAULTIK_AGE_SECRET_KEY environment variable to be set with Requires the VAULTIK_AGE_SECRET_KEY environment variable to be set with
the age private key. the age private key.
@@ -64,8 +77,7 @@ Examples:
return cmd return cmd
} }
// runRestore parses arguments and runs the restore operation through the // runRestore parses arguments and runs the restore operation through the app framework
// app framework.
func runRestore(cmd *cobra.Command, args []string, opts *RestoreOptions) error { func runRestore(cmd *cobra.Command, args []string, opts *RestoreOptions) error {
snapshotID := args[0] snapshotID := args[0]
@@ -74,30 +86,87 @@ func runRestore(cmd *cobra.Command, args []string, opts *RestoreOptions) error {
opts.Paths = args[restoreMinArgs:] opts.Paths = args[restoreMinArgs:]
} }
// Use unified config resolution
configPath, err := ResolveConfigPath() configPath, err := ResolveConfigPath()
if err != nil { if err != nil {
return err return err
} }
// Use the app framework like other commands
rootFlags := GetRootFlags() rootFlags := GetRootFlags()
return RunOperation(cmd.Context(), AppOptions{ return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath, ConfigPath: configPath,
LogOptions: log.Options{ LogOptions: log.Options{
Verbose: rootFlags.Verbose, Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug, Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet, Quiet: rootFlags.Quiet,
}, },
}, func(v *vaultik.Vaultik) error { Modules: buildRestoreModules(),
return v.Restore(&vaultik.RestoreOptions{ Invokes: buildRestoreInvokes(snapshotID, opts),
})
}
// buildRestoreModules returns the fx.Options for dependency injection in restore
func buildRestoreModules() []fx.Option {
return []fx.Option{
fx.Provide(fx.Annotate(
func(g *globals.Globals, cfg *config.Config,
storer storage.Storer, v *vaultik.Vaultik, shutdowner fx.Shutdowner) *RestoreApp {
return &RestoreApp{
Globals: g,
Config: cfg,
Storage: storer,
Vaultik: v,
Shutdowner: shutdowner,
}
},
)),
}
}
// buildRestoreInvokes returns the fx.Options that wire up the restore lifecycle
func buildRestoreInvokes(snapshotID string, opts *RestoreOptions) []fx.Option {
return []fx.Option{
fx.Invoke(func(app *RestoreApp, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
// Start the restore operation in a goroutine
go func() {
// Run the restore operation
restoreOpts := &vaultik.RestoreOptions{
SnapshotID: snapshotID, SnapshotID: snapshotID,
TargetDir: opts.TargetDir, TargetDir: opts.TargetDir,
Paths: opts.Paths, Paths: opts.Paths,
Verify: opts.Verify, Verify: opts.Verify,
SkipErrors: rootFlags.SkipErrors, SkipErrors: GetRootFlags().SkipErrors,
}) }
}, func(err error) {
err := app.Vaultik.Restore(restoreOpts)
if err != nil {
if !errors.Is(err, context.Canceled) {
log.Error("Restore operation failed", "error", err) log.Error("Restore operation failed", "error", err)
ReportErrorf("Restore failed: %v", err) ReportErrorf("Restore failed: %v", err)
os.Exit(1)
}
}
// Shutdown the app when restore completes
err = app.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(_ context.Context) error {
log.Debug("Stopping restore operation")
app.Vaultik.Cancel()
return nil
},
}) })
}),
}
} }
-24
View File
@@ -208,30 +208,6 @@ func (r *BlobRepository) DeleteOrphaned(ctx context.Context) error {
return nil return nil
} }
// DeleteUnuploaded deletes blob rows whose upload never completed
// (uploaded_ts IS NULL) and returns how many were removed. Their
// blob_chunks rows are removed by the ON DELETE CASCADE foreign key.
// A blob is only ever attached to a snapshot once its upload has been
// recorded, so an un-uploaded blob is never referenced by a completed
// snapshot: dropping it discards chunk rows that point at data which
// was never stored remotely, so the affected content is re-chunked and
// re-uploaded on the next run.
func (r *BlobRepository) DeleteUnuploaded(ctx context.Context) (int64, error) {
query := `DELETE FROM blobs WHERE uploaded_ts IS NULL`
result, err := r.db.ExecWithLog(ctx, query)
if err != nil {
return 0, fmt.Errorf("deleting un-uploaded blobs: %w", err)
}
rowsAffected, _ := result.RowsAffected()
if rowsAffected > 0 {
log.Debug("Deleted un-uploaded blobs", "count", rowsAffected)
}
return rowsAffected, nil
}
// getOne fetches a single blob row matched on the given column, or // getOne fetches a single blob row matched on the given column, or
// (nil, nil) when no row matches. // (nil, nil) when no row matches.
func (r *BlobRepository) getOne( func (r *BlobRepository) getOne(
+2 -22
View File
@@ -7,32 +7,12 @@ import (
// List returns every chunk in the index, ordered by chunk hash. // List returns every chunk in the index, ordered by chunk hash.
func (r *ChunkRepository) List(ctx context.Context) ([]*Chunk, error) { func (r *ChunkRepository) List(ctx context.Context) ([]*Chunk, error) {
return r.list(ctx, ` query := `
SELECT chunk_hash, size SELECT chunk_hash, size
FROM chunks FROM chunks
ORDER BY chunk_hash ORDER BY chunk_hash
`) `
}
// ListInUploadedBlobs returns the chunks that are stored in a blob whose
// upload has completed (uploaded_ts set), ordered by chunk hash. These
// are the only chunks a backup may safely deduplicate against: a chunk
// recorded solely in a blob that was never uploaded refers to data that
// is not in remote storage, so trusting it would silently drop that data
// from later snapshots.
func (r *ChunkRepository) ListInUploadedBlobs(ctx context.Context) ([]*Chunk, error) {
return r.list(ctx, `
SELECT DISTINCT c.chunk_hash, c.size
FROM chunks c
JOIN blob_chunks bc ON c.chunk_hash = bc.chunk_hash
JOIN blobs b ON bc.blob_id = b.id
WHERE b.uploaded_ts IS NOT NULL
ORDER BY c.chunk_hash
`)
}
// list runs a chunk-selecting query and scans the (chunk_hash, size) rows.
func (r *ChunkRepository) list(ctx context.Context, query string) ([]*Chunk, error) {
rows, err := r.db.conn.QueryContext(ctx, query) rows, err := r.db.conn.QueryContext(ctx, query)
if err != nil { if err != nil {
return nil, fmt.Errorf("querying chunks: %w", err) return nil, fmt.Errorf("querying chunks: %w", err)
+15 -1
View File
@@ -14,8 +14,18 @@ var Module = fx.Module("log",
) )
// New creates a new logger configuration from provided options. // New creates a new logger configuration from provided options.
//
// JSON is intentionally not carried into Config: a command emitting a
// JSON document on stdout must keep its stderr log level under
// --verbose/--debug, so --json must not lower it (issue #112). JSON
// silences the stdout UI in setupGlobals instead.
func New(opts Options) Config { func New(opts Options) Config {
return Config(opts) return Config{
Verbose: opts.Verbose,
Debug: opts.Debug,
Cron: opts.Cron,
Quiet: opts.Quiet,
}
} }
// Options are provided by the CLI. // Options are provided by the CLI.
@@ -24,4 +34,8 @@ type Options struct {
Debug bool Debug bool
Cron bool Cron bool
Quiet bool Quiet bool
// JSON marks a command whose stdout carries a machine-readable
// document. It silences the human UI on stdout (see setupGlobals),
// but unlike Quiet it leaves the stderr log level alone.
JSON bool
} }
+5 -57
View File
@@ -220,14 +220,7 @@ func (s *Scanner) Scan(
defer s.progress.Stop() defer s.progress.Stop()
} }
// Phase 0: Repair any state left by an interrupted previous run, then // Phase 0: Load known files and chunks from database into memory for fast lookup
// load known files and chunks from the database into memory for fast
// lookup.
err := s.repairInterruptedBlobs(ctx)
if err != nil {
return nil, err
}
knownFiles, err := s.loadDatabaseState(ctx, path) knownFiles, err := s.loadDatabaseState(ctx, path)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -324,38 +317,6 @@ func (s *Scanner) loadDatabaseState(
return knownFiles, nil return knownFiles, nil
} }
// repairInterruptedBlobs discards blob rows left by a previous run whose
// upload never completed. Such a blob has its chunks, blob_chunks, and
// blobs rows committed to the local index before the upload is attempted,
// so a crash or dropped connection mid-upload leaves them behind while the
// data never reaches remote storage. Deduplicating against those chunks on
// a later run would produce a snapshot that reports success but cannot be
// restored. Dropping the un-uploaded blobs (their blob_chunks cascade) and
// then any chunks left unreferenced forces the affected data to be
// re-chunked and re-uploaded this run. A blob is attached to a snapshot
// only once its upload is recorded, so this never touches a completed
// snapshot's data.
func (s *Scanner) repairInterruptedBlobs(ctx context.Context) error {
removed, err := s.repos.Blobs.DeleteUnuploaded(ctx)
if err != nil {
return fmt.Errorf("removing un-uploaded blob records: %w", err)
}
if removed == 0 {
return nil
}
log.Warn("Discarded blob records from an interrupted previous run; "+
"their data will be re-uploaded", "blobs", removed)
err = s.repos.Chunks.DeleteOrphaned(ctx)
if err != nil {
return fmt.Errorf("removing orphaned chunks: %w", err)
}
return nil
}
// summarizeScanPhase calculates total size to process, updates progress tracking, // summarizeScanPhase calculates total size to process, updates progress tracking,
// and prints the scan phase summary with file counts and sizes // and prints the scan phase summary with file counts and sizes
func (s *Scanner) summarizeScanPhase( func (s *Scanner) summarizeScanPhase(
@@ -431,14 +392,11 @@ func (s *Scanner) loadKnownFiles(
return result, nil return result, nil
} }
// loadKnownChunks loads the chunk hashes safe to deduplicate against into // loadKnownChunks loads all known chunk hashes from the database into a
// an in-memory map for fast lookup, avoiding per-chunk database queries // map for fast lookup. This avoids per-chunk database queries during file
// during file processing. Only chunks held by a blob whose upload // processing.
// completed are loaded: a chunk left behind by an interrupted upload
// refers to data that never reached remote storage, and deduplicating
// against it would silently produce an unrestorable snapshot.
func (s *Scanner) loadKnownChunks(ctx context.Context) error { func (s *Scanner) loadKnownChunks(ctx context.Context) error {
chunks, err := s.repos.Chunks.ListInUploadedBlobs(ctx) chunks, err := s.repos.Chunks.List(ctx)
if err != nil { if err != nil {
return fmt.Errorf("listing chunks: %w", err) return fmt.Errorf("listing chunks: %w", err)
} }
@@ -1443,17 +1401,7 @@ func (s *Scanner) finalizeProcessPhase(ctx context.Context, result *ScanResult)
return fmt.Errorf("parsing blob ID: %w", err) return fmt.Errorf("parsing blob ID: %w", err)
} }
// With no remote backend the blob's lifecycle ends here, so
// mark it uploaded in the same transaction that attaches it to
// the snapshot. This keeps the invariant that any blob a
// snapshot references has uploaded_ts set, so deduplication and
// interrupted-run repair treat these blobs as trustworthy.
err = s.repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error { err = s.repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
err := s.repos.Blobs.UpdateUploaded(ctx, tx, b.ID)
if err != nil {
return fmt.Errorf("marking blob uploaded: %w", err)
}
return s.repos.Snapshots.AddBlob(ctx, tx, s.snapshotID, blobID, return s.repos.Snapshots.AddBlob(ctx, tx, s.snapshotID, blobID,
types.BlobHash(b.Hash)) types.BlobHash(b.Hash))
}) })
-198
View File
@@ -1,198 +0,0 @@
package storage_test
import (
"bytes"
"context"
"errors"
"io"
"reflect"
"sort"
"testing"
"sneak.berlin/go/vaultik/internal/storage"
)
// runStorerConformance is the shared Storer contract. Every backend that
// can run in-process is expected to pass it: TestFileStorer runs it against
// file://, TestS3Storer against s3://. A new backend inherits this coverage
// by passing its own constructor, so the contract is defined once.
//
// It exercises the public Storer interface: round-trip, stat, list with
// prefix filtering, overwrite, delete, delete-of-missing, and not-found on
// Get and Stat. Each section takes its own fresh backend instance, so the
// order of sections never matters and no section sees another's objects.
func runStorerConformance(t *testing.T, newStorer func(*testing.T) storage.Storer) {
t.Helper()
conformanceRoundTrip(t, newStorer(t))
conformanceOverwrite(t, newStorer(t))
conformanceList(t, newStorer(t))
conformanceDelete(t, newStorer(t))
conformanceNotFound(t, newStorer(t))
}
// conformanceRoundTrip stores a nested key, then reads it back and stats it.
func conformanceRoundTrip(t *testing.T, s storage.Storer) {
t.Helper()
ctx := context.Background()
key := "blobs/aa/bb/object.bin"
want := []byte("round-trip payload")
err := s.Put(ctx, key, bytes.NewReader(want))
if err != nil {
t.Fatalf("Put: %v", err)
}
got := getBytes(t, s, key)
if !bytes.Equal(got, want) {
t.Errorf("Get returned %q, want %q", got, want)
}
info, err := s.Stat(ctx, key)
if err != nil {
t.Fatalf("Stat: %v", err)
}
if info.Key != key {
t.Errorf("Stat key = %q, want %q", info.Key, key)
}
if info.Size != int64(len(want)) {
t.Errorf("Stat size = %d, want %d", info.Size, len(want))
}
}
// conformanceOverwrite checks that a second Put replaces the first.
func conformanceOverwrite(t *testing.T, s storage.Storer) {
t.Helper()
ctx := context.Background()
key := "meta/snapshot.json"
err := s.Put(ctx, key, bytes.NewReader([]byte("first")))
if err != nil {
t.Fatalf("first Put: %v", err)
}
want := []byte("second and longer payload")
err = s.Put(ctx, key, bytes.NewReader(want))
if err != nil {
t.Fatalf("second Put: %v", err)
}
got := getBytes(t, s, key)
if !bytes.Equal(got, want) {
t.Errorf("after overwrite Get returned %q, want %q", got, want)
}
}
// conformanceList checks prefix filtering and the empty result for a
// prefix that matches nothing.
func conformanceList(t *testing.T, s storage.Storer) {
t.Helper()
ctx := context.Background()
keys := []string{"blobs/aa/one", "blobs/bb/two", "meta/three"}
for _, k := range keys {
err := s.Put(ctx, k, bytes.NewReader([]byte("data")))
if err != nil {
t.Fatalf("Put %q: %v", k, err)
}
}
if got := listSorted(t, s, ""); !reflect.DeepEqual(got, keys) {
t.Errorf("List(\"\") = %v, want %v", got, keys)
}
wantBlobs := []string{"blobs/aa/one", "blobs/bb/two"}
if got := listSorted(t, s, "blobs/"); !reflect.DeepEqual(got, wantBlobs) {
t.Errorf("List(\"blobs/\") = %v, want %v", got, wantBlobs)
}
if got := listSorted(t, s, "absent/"); len(got) != 0 {
t.Errorf("List(\"absent/\") = %v, want empty", got)
}
}
// conformanceDelete checks that Delete removes an object and that deleting
// a missing key is not an error.
func conformanceDelete(t *testing.T, s storage.Storer) {
t.Helper()
ctx := context.Background()
key := "blobs/cc/gone.bin"
err := s.Put(ctx, key, bytes.NewReader([]byte("temporary")))
if err != nil {
t.Fatalf("Put: %v", err)
}
err = s.Delete(ctx, key)
if err != nil {
t.Fatalf("Delete: %v", err)
}
_, err = s.Get(ctx, key)
if !errors.Is(err, storage.ErrNotFound) {
t.Errorf("Get after Delete error = %v, want ErrNotFound", err)
}
err = s.Delete(ctx, key)
if err != nil {
t.Errorf("Delete of missing key = %v, want nil", err)
}
}
// conformanceNotFound checks Get and Stat on an absent key.
func conformanceNotFound(t *testing.T, s storage.Storer) {
t.Helper()
ctx := context.Background()
key := "never/written"
_, err := s.Get(ctx, key)
if !errors.Is(err, storage.ErrNotFound) {
t.Errorf("Get error = %v, want ErrNotFound", err)
}
_, err = s.Stat(ctx, key)
if !errors.Is(err, storage.ErrNotFound) {
t.Errorf("Stat error = %v, want ErrNotFound", err)
}
}
// getBytes reads a key fully and closes the reader.
func getBytes(t *testing.T, s storage.Storer, key string) []byte {
t.Helper()
rc, err := s.Get(context.Background(), key)
if err != nil {
t.Fatalf("Get %q: %v", key, err)
}
defer func() { _ = rc.Close() }()
data, err := io.ReadAll(rc)
if err != nil {
t.Fatalf("read %q: %v", key, err)
}
return data
}
// listSorted returns the keys under a prefix in a stable order.
func listSorted(t *testing.T, s storage.Storer, prefix string) []string {
t.Helper()
keys, err := s.List(context.Background(), prefix)
if err != nil {
t.Fatalf("List %q: %v", prefix, err)
}
sort.Strings(keys)
return keys
}
-27
View File
@@ -1,27 +0,0 @@
package storage_test
import (
"testing"
"sneak.berlin/go/vaultik/internal/storage"
)
// newFileStorer builds a file:// backend rooted at a fresh temp directory.
//
//nolint:ireturn // conformance runs against the Storer interface by design
func newFileStorer(t *testing.T) storage.Storer {
t.Helper()
s, err := storage.NewFileStorer(t.TempDir())
if err != nil {
t.Fatalf("NewFileStorer: %v", err)
}
return s
}
// TestFileStorer runs the shared Storer contract against the file:// backend.
func TestFileStorer(t *testing.T) {
t.Parallel()
runStorerConformance(t, newFileStorer)
}
-58
View File
@@ -1,58 +0,0 @@
package storage_test
import (
"context"
"errors"
"testing"
"sneak.berlin/go/vaultik/internal/storage"
)
// The rclone backend is a thin adapter over the rclone library: it turns a
// (remote, path) pair into rclone's "remote:path" string, hands it to
// rclone, and maps rclone's own results back to the Storer interface. What
// can be tested in-process, without a configured remote or network, is that
// adapter layer — how the arguments are shaped and how construction errors
// are reported. The data-plane operations (Put/Get/List/Delete) are rclone's
// own, exercised against a real provider (drive, s3-via-rclone, ...), which
// needs a configured remote with credentials and network access and so is
// out of reach of a unit test. The shared Storer conformance suite therefore
// runs against the in-process file and s3 backends; the rclone backend
// inherits that contract once a remote is configured.
//
// These tests use rclone's ":local:" on-the-fly backend, which addresses the
// local filesystem directly without any configured remote, so construction
// runs entirely in-process.
// TestNewRcloneStorerConstruction checks that a valid remote constructs a
// backend and that Info() reports the shaped "remote:path" location.
//
//nolint:paralleltest // NewRcloneStorer installs the process-global rclone config
func TestNewRcloneStorerConstruction(t *testing.T) {
dir := t.TempDir()
s, err := storage.NewRcloneStorer(context.Background(), ":local", dir)
if err != nil {
t.Fatalf("NewRcloneStorer: %v", err)
}
// Info().Location is the "remote:path" string the adapter builds from
// its two arguments, so asserting it confirms the argument shaping.
want := ":local:" + dir
if got := s.Info().Location; got != want {
t.Errorf("Info().Location = %q, want %q", got, want)
}
}
// TestNewRcloneStorerUnknownRemote checks that a remote that is not in the
// rclone config fails construction with the ErrRemoteNotFound sentinel,
// rather than silently returning a backend pointed nowhere.
//
//nolint:paralleltest // NewRcloneStorer installs the process-global rclone config
func TestNewRcloneStorerUnknownRemote(t *testing.T) {
_, err := storage.NewRcloneStorer(
context.Background(), "vaultik-no-such-remote", "path")
if !errors.Is(err, storage.ErrRemoteNotFound) {
t.Errorf("NewRcloneStorer error = %v, want ErrRemoteNotFound", err)
}
}
+14 -36
View File
@@ -13,23 +13,18 @@ import (
"sneak.berlin/go/vaultik/internal/storage" "sneak.berlin/go/vaultik/internal/storage"
) )
// s3TestBucket is the bucket created for each in-process S3 server. // TestS3StorerMissingKeyMapsToErrNotFound verifies that the s3 backend reports
const s3TestBucket = "test-bucket" // a missing object as storage.ErrNotFound, matching the file and rclone
// backends and the Storer contract. Without the mapping, Get and Stat leak the
// newS3Storer builds an s3:// backend backed by a fresh in-process // raw SDK error and errors.Is(err, storage.ErrNotFound) is false.
// S3 server. It reuses the same in-memory S3 harness (gofakes3 + s3mem
// over httptest) that internal/s3 and the not-found regression test use,
// so no new mock or dependency is introduced. Each call gets its own
// server, bucket, and client, so the conformance suite's per-section
// instances stay isolated.
// //
//nolint:ireturn // conformance runs against the Storer interface by design //nolint:paralleltest // shares an in-process S3 server via t.Cleanup
func newS3Storer(t *testing.T) storage.Storer { func TestS3StorerMissingKeyMapsToErrNotFound(t *testing.T) {
t.Helper() const bucket = "test-bucket"
backend := s3mem.New() backend := s3mem.New()
err := backend.CreateBucket(s3TestBucket) err := backend.CreateBucket(bucket)
if err != nil { if err != nil {
t.Fatalf("create bucket: %v", err) t.Fatalf("create bucket: %v", err)
} }
@@ -37,9 +32,11 @@ func newS3Storer(t *testing.T) storage.Storer {
srv := httptest.NewServer(gofakes3.New(backend).Server()) srv := httptest.NewServer(gofakes3.New(backend).Server())
t.Cleanup(srv.Close) t.Cleanup(srv.Close)
client, err := s3.NewClient(context.Background(), s3.Config{ ctx := context.Background()
client, err := s3.NewClient(ctx, s3.Config{
Endpoint: srv.URL, Endpoint: srv.URL,
Bucket: s3TestBucket, Bucket: bucket,
AccessKeyID: "test", AccessKeyID: "test",
SecretAccessKey: "test", SecretAccessKey: "test",
Region: "us-east-1", Region: "us-east-1",
@@ -48,28 +45,9 @@ func newS3Storer(t *testing.T) storage.Storer {
t.Fatalf("new client: %v", err) t.Fatalf("new client: %v", err)
} }
return storage.NewS3Storer(client) storer := storage.NewS3Storer(client)
}
// TestS3Storer runs the shared Storer contract against the s3:// backend, _, err = storer.Get(ctx, "does-not-exist")
// so it is held to the same round-trip, list, delete, and not-found
// behaviour as the file:// backend.
func TestS3Storer(t *testing.T) {
t.Parallel()
runStorerConformance(t, newS3Storer)
}
// TestS3StorerMissingKeyMapsToErrNotFound pins the specific contract that a
// missing object surfaces as storage.ErrNotFound rather than the raw AWS SDK
// error. Without the mapping, errors.Is(err, storage.ErrNotFound) is false on
// s3 and callers would branch differently per backend.
func TestS3StorerMissingKeyMapsToErrNotFound(t *testing.T) {
t.Parallel()
storer := newS3Storer(t)
ctx := context.Background()
_, err := storer.Get(ctx, "does-not-exist")
if !errors.Is(err, storage.ErrNotFound) { if !errors.Is(err, storage.ErrNotFound) {
t.Errorf("Get on missing key: got %v, want ErrNotFound", err) t.Errorf("Get on missing key: got %v, want ErrNotFound", err)
} }
-110
View File
@@ -1,110 +0,0 @@
package storage_test
import (
"errors"
"reflect"
"testing"
"sneak.berlin/go/vaultik/internal/storage"
)
// TestParseStorageURLValid checks that each supported scheme parses into
// the expected fields, since those fields decide which backend is built.
func TestParseStorageURLValid(t *testing.T) {
t.Parallel()
const bucket = "mybucket"
cases := []struct {
name string
raw string
want *storage.URL
}{
{
name: "file absolute path",
raw: "file:///var/backups/vaultik",
want: &storage.URL{Scheme: "file", Prefix: "/var/backups/vaultik"},
},
{
name: "s3 bucket and prefix, ssl defaults on",
raw: "s3://mybucket/backups/host",
want: &storage.URL{
Scheme: "s3", Bucket: bucket,
Prefix: "backups/host", UseSSL: true,
},
},
{
name: "s3 bucket only",
raw: "s3://mybucket",
want: &storage.URL{Scheme: "s3", Bucket: bucket, UseSSL: true},
},
{
name: "s3 with endpoint, region, ssl off",
raw: "s3://mybucket?endpoint=minio.example.com&region=us-west-2&ssl=false",
want: &storage.URL{
Scheme: "s3", Bucket: bucket,
Endpoint: "minio.example.com", Region: "us-west-2", UseSSL: false,
},
},
{
name: "rclone remote and path",
raw: "rclone://gdrive/backups/host",
want: &storage.URL{
Scheme: "rclone", RcloneRemote: "gdrive", Prefix: "backups/host",
},
},
{
name: "rclone remote only",
raw: "rclone://gdrive",
want: &storage.URL{Scheme: "rclone", RcloneRemote: "gdrive"},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got, err := storage.ParseStorageURL(tc.raw)
if err != nil {
t.Fatalf("ParseStorageURL(%q) returned error: %v", tc.raw, err)
}
if !reflect.DeepEqual(got, tc.want) {
t.Errorf("ParseStorageURL(%q) = %+v, want %+v", tc.raw, got, tc.want)
}
})
}
}
// TestParseStorageURLErrors checks that empty, missing, and unknown-scheme
// inputs fail with the documented sentinel errors instead of parsing to a
// wrong destination.
func TestParseStorageURLErrors(t *testing.T) {
t.Parallel()
cases := []struct {
name string
raw string
wantErr error
}{
{"empty url", "", storage.ErrEmptyStorageURL},
{"file empty path", "file://", storage.ErrEmptyFilePath},
{"s3 missing bucket", "s3://", storage.ErrMissingBucket},
{"s3 missing bucket with path", "s3:///justprefix", storage.ErrMissingBucket},
{"rclone missing remote", "rclone://", storage.ErrMissingRemote},
{"unknown scheme", "gs://bucket/x", storage.ErrUnsupportedScheme},
{"no scheme", "/local/path", storage.ErrUnsupportedScheme},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := storage.ParseStorageURL(tc.raw)
if !errors.Is(err, tc.wantErr) {
t.Errorf("ParseStorageURL(%q) error = %v, want %v",
tc.raw, err, tc.wantErr)
}
})
}
}
-198
View File
@@ -1,198 +0,0 @@
package vaultik //nolint:testpackage // constructs Vaultik with unexported fields
import (
"bytes"
"context"
"errors"
"io"
"path/filepath"
"strings"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/ui"
)
// errUploadInterrupted stands in for a dropped connection or kill -9
// partway through a blob upload.
var errUploadInterrupted = errors.New("simulated interrupted blob upload")
// interruptedBlobStorer wraps a real Storer but fails every blob upload,
// modelling a run that dies mid-blob after the packer has already
// committed the blob's chunk rows to the local index.
type interruptedBlobStorer struct {
storage.Storer
}
func (s *interruptedBlobStorer) PutWithProgress(
ctx context.Context, key string, reader io.Reader,
size int64, cb storage.ProgressCallback,
) error {
if strings.HasPrefix(key, "blobs/") {
return errUploadInterrupted
}
return s.Storer.PutWithProgress(ctx, key, reader, size, cb)
}
func (s *interruptedBlobStorer) Put(
ctx context.Context, key string, reader io.Reader,
) error {
if strings.HasPrefix(key, "blobs/") {
return errUploadInterrupted
}
return s.Storer.Put(ctx, key, reader)
}
// TestBackupRetryAfterInterruptedUploadIsRestorable reproduces the
// silent-data-loss defect in
// https://git.eeqj.de/sneak/vaultik/issues/148: a blob upload is
// interrupted, leaving chunk rows in the local index for data that never
// reached storage. The retry run reuses the same index. Before the fix it
// deduplicated against those orphaned chunks, uploaded nothing for them,
// and produced a snapshot that reported success but could not be
// restored. The retried snapshot must instead restore byte-for-byte.
func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
require.NoError(t, fs.MkdirAll(dataDir, 0o755))
// Random content forces real chunks; small blobs guarantee at least
// one blob is finalized (and its upload attempted) during the run.
sources := map[string][]byte{
"a.bin": randomBytes(t, 128*1024),
"b.bin": randomBytes(t, 128*1024),
"c.bin": randomBytes(t, 128*1024),
}
for name, data := range sources {
require.NoError(t, afero.WriteFile(
fs, filepath.Join(dataDir, name), data, 0o644))
}
cfg := &config.Config{
AgeRecipients: []string{"age1ezrjmfpwsc95svdg0y54mums3zevgzu0x0ecq2" +
"f7tp8a05gl0sjq9q9wjg"},
AgeSecretKey: "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKU" +
"T68TXSFPK7APHXA2QS2NJA5",
CompressionLevel: 3,
Hostname: "test-host",
ChunkSize: config.Size(16 * 1024),
BlobSizeLimit: config.Size(64 * 1024),
}
working, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
ctx := context.Background()
// Run 1: the upload is interrupted, so the scan fails but leaves the
// interrupted blob's chunk rows committed in the index.
_, err = runBackup(ctx, fs, cfg, &interruptedBlobStorer{Storer: working},
dataDir, dbPath, "interrupted")
require.Error(t, err, "interrupted upload must fail the run")
// Run 2: retry on the same index with a working backend. This must
// succeed and produce a fully restorable snapshot.
snapshotID, err := runBackup(ctx, fs, cfg, working, dataDir, dbPath, "retry")
require.NoError(t, err, "retry backup must succeed")
v := &Vaultik{
Config: cfg,
Storage: working,
Fs: fs,
Stdout: io.Discard,
Stderr: io.Discard,
UI: ui.NewWithColor(io.Discard, false),
}
v.SetContext(ctx)
require.NoError(t, v.Restore(&RestoreOptions{
SnapshotID: snapshotID,
TargetDir: restoreDir,
}), "the retried snapshot must be restorable")
for name, data := range sources {
restored := filepath.Join(restoreDir, dataDir, name)
got, err := afero.ReadFile(fs, restored)
require.NoErrorf(t, err, "restored file missing: %s", name)
require.Truef(t, bytes.Equal(got, data),
"restored bytes differ from original for %s", name)
}
}
// runBackup performs one backup of dataDir into a fresh snapshot on the
// index at dbPath, returning the snapshot ID. When the scan succeeds it
// also completes and exports the snapshot metadata so the result can be
// restored. The index database is always closed before returning.
func runBackup(
ctx context.Context,
fs afero.Fs,
cfg *config.Config,
storer storage.Storer,
dataDir, dbPath, name string,
) (string, error) {
db, err := database.New(ctx, dbPath)
if err != nil {
return "", err
}
defer func() { _ = db.Close() }()
repos := database.NewRepositories(db)
sm := snapshot.NewSnapshotManager(snapshot.SnapshotManagerParams{
Repos: repos,
Storage: storer,
Config: cfg,
})
sm.SetFilesystem(fs)
scanner := snapshot.NewScanner(snapshot.ScannerConfig{
FS: fs,
Storage: storer,
ChunkSize: cfg.ChunkSize.Int64(),
MaxBlobSize: cfg.BlobSizeLimit.Int64(),
CompressionLevel: cfg.CompressionLevel,
AgeRecipients: cfg.AgeRecipients,
Repositories: repos,
})
snapshotID, err := sm.CreateSnapshotWithName(
ctx, cfg.Hostname, name, "test-version", "test-git")
if err != nil {
return "", err
}
_, err = scanner.Scan(ctx, dataDir, snapshotID)
if err != nil {
return snapshotID, err
}
err = sm.CompleteSnapshot(ctx, snapshotID)
if err != nil {
return snapshotID, err
}
err = sm.ExportSnapshotMetadata(ctx, dbPath, snapshotID)
if err != nil {
return snapshotID, err
}
return snapshotID, nil
}
+5 -10
View File
@@ -18,6 +18,7 @@ import (
"sneak.berlin/go/vaultik/internal/blobgen" "sneak.berlin/go/vaultik/internal/blobgen"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/types" "sneak.berlin/go/vaultik/internal/types"
) )
@@ -576,20 +577,14 @@ func (v *Vaultik) handleRestoreVerification(
} }
// downloadSnapshotDB downloads and decrypts the snapshot metadata // downloadSnapshotDB downloads and decrypts the snapshot metadata
// database. The identifier is resolved to the snapshot's remote key: a // database. The snapshotID is the human ID; we hash it to the remote
// human ID is hashed, and a remote key (or its abbreviation, as printed // key for the storage path.
// for a remote-only snapshot) is used as-is, so a host with no local
// index can restore the snapshots it can only see on the store.
func (v *Vaultik) downloadSnapshotDB( func (v *Vaultik) downloadSnapshotDB(
snapshotID string, identity age.Identity, snapshotID string, identity age.Identity,
) (*database.DB, error) { ) (*database.DB, error) {
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
if err != nil {
return nil, err
}
// Download encrypted database from storage // Download encrypted database from storage
dbKey := fmt.Sprintf("metadata/%s/db.zst.age", remoteKey) dbKey := fmt.Sprintf("metadata/%s/db.zst.age",
snapshot.RemoteSnapshotKey(snapshotID))
reader, err := v.Storage.Get(v.ctx, dbKey) reader, err := v.Storage.Get(v.ctx, dbKey)
if err != nil { if err != nil {
@@ -1,167 +0,0 @@
package vaultik_test
import (
"bytes"
"context"
"io"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestRestoreOnAnotherMachine proves the disaster-recovery path: a host
// that has only the vaultik binary, the age secret key, and the storage
// credentials — no local index, a different hostname, and no
// age_recipients configured — can list, restore, and verify a snapshot
// straight from the destination store.
//
// The backup half writes a snapshot with one index and hostname. The
// restore half throws that index away entirely: a fresh, empty index and
// a config that shares nothing with the original but the storage location
// and the secret key. If restore or verify needed the original local
// index — or the human snapshot ID that only that index holds — this test
// could not run, because the recovery host can know neither.
func TestRestoreOnAnotherMachine(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(64 * 1024)
maxBlobSize := int64(512 * 1024)
sourceFiles := writeRecoverySourceTree(t, fs, dataDir, chunkSize)
ctx := context.Background()
// Backup host: one index, hostname test-host, age_recipients set.
// runFileStorageBackup closes the index before returning, so nothing
// below can lean on it.
_, storer, originalID := runFileStorageBackup(
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
// Recovery host: a fresh empty index, a different hostname, and no
// age_recipients — only the secret key and the same storage location.
recovery, stdout := newRecoveryHost(ctx, t, fs, storer)
// The recovery index really is empty. This is the assertion that makes
// the test a guard against restore quietly depending on the original
// index: if it did, an empty index would make restore fail.
localSnaps, err := recovery.Repositories.Snapshots.ListRecent(ctx, 100)
require.NoError(t, err)
require.Empty(t, localSnaps, "recovery host must start with no local index")
// List: the snapshot shows up as remote-only, identified by its remote
// key, with no recoverable human ID.
require.NoError(t, recovery.ListSnapshots(true))
rows := decodeListJSON(t, stdout.String())
require.Len(t, rows, 1)
remote := rows[0]
assert.False(t, remote.LocallyTracked, "snapshot must be remote-only here")
assert.Empty(t, remote.ID, "the human ID is unknown to the recovery host")
require.Len(t, remote.RemoteKey, 64)
assert.Equal(t, snapshot.RemoteSnapshotKey(originalID), remote.RemoteKey,
"the listed key is the hashed snapshot ID")
// Restore driven by the abbreviated identifier the table prints (the
// first 12 hex of the remote key), then deep-verify from the store
// keyed by the full remote key. Both are what a recovery host can know.
require.NoError(t, recovery.Restore(&vaultik.RestoreOptions{
SnapshotID: remote.RemoteKey[:12],
TargetDir: restoreDir,
Verify: true,
}))
require.NoError(t, recovery.RunDeepVerify(
remote.RemoteKey, &vaultik.VerifyOptions{Deep: true}))
assertRestoredTreeMatches(t, fs, restoreDir, sourceFiles)
}
// writeRecoverySourceTree writes a small source tree spanning several
// chunks (so restore reassembles real multi-chunk files) and returns the
// content keyed by absolute path.
func writeRecoverySourceTree(
t *testing.T, fs afero.Fs, dataDir string, chunkSize int64,
) map[string][]byte {
t.Helper()
sourceFiles := map[string][]byte{
filepath.Join(dataDir, "notes.txt"): []byte("recover me"),
filepath.Join(dataDir, "sub", "big.bin"): bytesPattern("big-", int(chunkSize*3)),
filepath.Join(dataDir, "sub", "small.bin"): bytesPattern("small-", 128),
}
for path, content := range sourceFiles {
require.NoError(t, fs.MkdirAll(filepath.Dir(path), 0o755))
require.NoError(t, afero.WriteFile(fs, path, content, 0o644))
}
return sourceFiles
}
// newRecoveryHost builds the Vaultik a replacement machine would run: an
// empty in-memory index, a hostname different from the backup host, no
// age_recipients, and only the secret key plus the shared storer. It
// returns the instance and the buffer its stdout is wired to.
func newRecoveryHost(
ctx context.Context, t *testing.T, fs afero.Fs, storer storage.Storer,
) (*vaultik.Vaultik, *bytes.Buffer) {
t.Helper()
recoveryDB, err := database.New(ctx, ":memory:")
require.NoError(t, err)
t.Cleanup(func() { _ = recoveryDB.Close() })
stdout := &bytes.Buffer{}
recovery := &vaultik.Vaultik{
Config: &config.Config{
AgeSecretKey: testAgeSecretKey,
Hostname: "recovery-host",
},
Storage: storer,
Fs: fs,
Repositories: database.NewRepositories(recoveryDB),
DB: recoveryDB,
Stdout: stdout,
Stderr: io.Discard,
UI: ui.NewWithColor(io.Discard, false),
}
recovery.SetContext(ctx)
return recovery, stdout
}
// assertRestoredTreeMatches byte-compares every restored file against its
// source content.
func assertRestoredTreeMatches(
t *testing.T, fs afero.Fs, restoreDir string, sourceFiles map[string][]byte,
) {
t.Helper()
for origPath, expected := range sourceFiles {
restored := filepath.Join(restoreDir, origPath)
got, err := afero.ReadFile(fs, restored)
require.NoErrorf(t, err, "restored file missing: %s", restored)
require.Truef(t, bytes.Equal(got, expected),
"byte mismatch for %s", origPath)
}
}
+3 -5
View File
@@ -670,11 +670,9 @@ func (v *Vaultik) VerifySnapshotWithOptions(
v.printVerifyHeader(snapshotID, opts) v.printVerifyHeader(snapshotID, opts)
// Resolve the identifier to the snapshot's remote key and download the // Download and parse manifest. The caller supplies a human
// manifest. A human ID is hashed; a remote key (or its abbreviation, // snapshot ID; we hash it to address remote storage.
// as printed for a remote-only snapshot) is used as-is, so a host with manifest, err := v.downloadManifestByKey(snapshot.RemoteSnapshotKey(snapshotID))
// no local index can verify a snapshot it can only see on the store.
manifest, err := v.resolveAndDownloadManifest(snapshotID)
if err != nil { if err != nil {
if opts.JSON { if opts.JSON {
result.Status = verifyStatusFailed result.Status = verifyStatusFailed
-101
View File
@@ -1,101 +0,0 @@
package vaultik
import (
"errors"
"fmt"
"strings"
"sneak.berlin/go/vaultik/internal/snapshot"
)
// remoteKeyHexLen is the length of a full remote snapshot key: a SHA256
// digest rendered as lowercase hex.
const remoteKeyHexLen = 64
// Sentinel errors for resolving a snapshot identifier against the store.
var (
errSnapshotKeyNotFound = errors.New(
"no snapshot on the destination store matches this identifier")
errSnapshotKeyAmbiguous = errors.New(
"identifier matches more than one snapshot on the destination store")
)
// resolveSnapshotRemoteKey turns a snapshot identifier supplied on the
// command line into the remote key that names the snapshot's metadata
// directory on the destination store. Every remote path a restore or
// verify reads is built from that key.
//
// Two forms are accepted, matching the two things a host can know:
//
// - A human snapshot ID (hostname_name_timestamp), which a host holding
// the local index has. It is hashed to its remote key; the store is
// not consulted.
// - A remote key, or the leading part of one, which is all a host with
// no local index can know — it is exactly what `snapshot list` prints
// for a remote-only snapshot (see formatRemoteOnlyID). It is resolved
// against the destination store's metadata listing; an identifier that
// matches no snapshot, or more than one, is an error.
//
// The two are told apart by shape: a remote key is lowercase hex, and a
// human snapshot ID never is (it carries a hostname, underscores, and an
// RFC3339 timestamp).
func (v *Vaultik) resolveSnapshotRemoteKey(identifier string) (string, error) {
if !isRemoteKeyOrPrefix(identifier) {
return snapshot.RemoteSnapshotKey(identifier), nil
}
keys, err := v.listAllRemoteSnapshotKeys()
if err != nil {
return "", fmt.Errorf(
"listing destination store to resolve %q: %w", identifier, err)
}
var matches []string
for _, key := range keys {
if strings.HasPrefix(key, identifier) {
matches = append(matches, key)
}
}
switch len(matches) {
case 1:
return matches[0], nil
case 0:
return "", fmt.Errorf("%w: %s", errSnapshotKeyNotFound, identifier)
default:
return "", fmt.Errorf("%w: %s (%d matches)",
errSnapshotKeyAmbiguous, identifier, len(matches))
}
}
// resolveAndDownloadManifest resolves a snapshot identifier to its remote
// key (see resolveSnapshotRemoteKey) and downloads that snapshot's
// manifest.
func (v *Vaultik) resolveAndDownloadManifest(
identifier string,
) (*snapshot.Manifest, error) {
remoteKey, err := v.resolveSnapshotRemoteKey(identifier)
if err != nil {
return nil, err
}
return v.downloadManifestByKey(remoteKey)
}
// isRemoteKeyOrPrefix reports whether s is a full remote key or the
// leading part of one: 1 to 64 lowercase hex characters. A human snapshot
// ID is never all hex, so this shape test is enough to tell the two apart.
func isRemoteKeyOrPrefix(s string) bool {
if s == "" || len(s) > remoteKeyHexLen {
return false
}
for _, r := range s {
if (r < '0' || r > '9') && (r < 'a' || r > 'f') {
return false
}
}
return true
}
+9 -18
View File
@@ -138,15 +138,8 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
func (v *Vaultik) loadVerificationData( func (v *Vaultik) loadVerificationData(
snapshotID string, opts *VerifyOptions, result *VerifyResult, snapshotID string, opts *VerifyOptions, result *VerifyResult,
) (*snapshot.Manifest, *tempDB, []snapshot.BlobInfo, error) { ) (*snapshot.Manifest, *tempDB, []snapshot.BlobInfo, error) {
// Resolve the identifier to the snapshot's remote key. A human ID is // All remote paths use the hashed key derived from the human ID.
// hashed; a remote key (or its abbreviation, as printed for a remoteKey := snapshot.RemoteSnapshotKey(snapshotID)
// remote-only snapshot) is used as-is, so a host with no local index
// can verify a snapshot it can only see on the store.
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
if err != nil {
return nil, nil, nil, v.deepVerifyFailure(result, opts,
fmt.Sprintf("resolving snapshot identifier: %v", err), err)
}
// Download manifest. downloadManifestByKey is the single reader for // Download manifest. downloadManifestByKey is the single reader for
// remote manifests; see its doc comment. // remote manifests; see its doc comment.
@@ -193,7 +186,7 @@ func (v *Vaultik) loadVerificationData(
fmt.Errorf("failed to decrypt database: %w", err)) fmt.Errorf("failed to decrypt database: %w", err))
} }
dbBlobs, err := v.getBlobsFromDatabase(tdb.DB) dbBlobs, err := v.getBlobsFromDatabase(snapshotID, tdb.DB)
if err != nil { if err != nil {
_ = tdb.Close() _ = tdb.Close()
@@ -508,21 +501,19 @@ func (v *Vaultik) verifyBlobFinalIntegrity(
return nil return nil
} }
// getBlobsFromDatabase gets all blobs for the snapshot from the database. // getBlobsFromDatabase gets all blobs for the snapshot from the database
// func (v *Vaultik) getBlobsFromDatabase(
// The exported per-snapshot database holds exactly one snapshot's data snapshotID string, db *sql.DB,
// (see cleanSnapshotDB), so every row in snapshot_blobs belongs to it. ) ([]snapshot.BlobInfo, error) {
// We select them directly rather than filtering by the human snapshot ID,
// which a host restoring from the store alone does not have.
func (v *Vaultik) getBlobsFromDatabase(db *sql.DB) ([]snapshot.BlobInfo, error) {
query := ` query := `
SELECT b.blob_hash, b.compressed_size SELECT b.blob_hash, b.compressed_size
FROM snapshot_blobs sb FROM snapshot_blobs sb
JOIN blobs b ON sb.blob_hash = b.blob_hash JOIN blobs b ON sb.blob_hash = b.blob_hash
WHERE sb.snapshot_id = ?
ORDER BY b.blob_hash ORDER BY b.blob_hash
` `
rows, err := db.QueryContext(v.ctx, query) rows, err := db.QueryContext(v.ctx, query, snapshotID)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to query snapshot blobs: %w", err) return nil, fmt.Errorf("failed to query snapshot blobs: %w", err)
} }
+1 -16
View File
@@ -56,23 +56,8 @@ main() {
docker build --output=type=cacheonly \ docker build --output=type=cacheonly \
--build-arg CHECK_EPOCH="$epoch" -f Dockerfile.lint . --build-arg CHECK_EPOCH="$epoch" -f Dockerfile.lint .
# Version, commit and build date are computed here on the host, the
# same way script/docker does, and passed into the product build so
# the CI-built image reports its real source. The build context
# excludes .git (see .dockerignore), so the build cannot derive them
# itself; without these it would stamp the Dockerfile's dev/unknown
# fallbacks. VERSION comes from script/version, the source of truth
# shared with the Makefile.
version="$("$ROOT/script/version")"
commit="$(git rev-parse HEAD 2>/dev/null || echo unknown)"
commit_date="$(git show -s --format=%cs HEAD 2>/dev/null || echo unknown)"
epoch="$(date +%s%N)$$" epoch="$(date +%s%N)$$"
docker build --build-arg CHECK_EPOCH="$epoch" \ docker build --build-arg CHECK_EPOCH="$epoch" .
--build-arg VERSION="$version" \
--build-arg COMMIT="$commit" \
--build-arg COMMIT_DATE="$commit_date" \
.
} }
main "$@" main "$@"
-17
View File
@@ -24,24 +24,7 @@ main() {
# whether the tree is clean. The Dockerfile now refuses to build # whether the tree is clean. The Dockerfile now refuses to build
# without a non-empty value, so this is required, not optional. # without a non-empty value, so this is required, not optional.
epoch="$(date +%s%N)$$" epoch="$(date +%s%N)$$"
# Version, commit and build date are computed here on the host,
# where .git exists, and passed into the build. The build context
# excludes .git (see .dockerignore), so the container cannot derive
# them itself -- it used to try and always got "unknown", giving
# every image a "commit: unknown" it could not be traced from.
# VERSION comes from script/version, the source of truth shared with
# the Makefile, so a Docker build reports the same string (tag,
# dev-<sha>, or a -dirty variant) that a local build of the same
# tree would.
version="$("$SCRIPT_DIR/version")"
commit="$(git rev-parse HEAD 2>/dev/null || echo unknown)"
commit_date="$(git show -s --format=%cs HEAD 2>/dev/null || echo unknown)"
docker build --build-arg CHECK_EPOCH="$epoch" \ docker build --build-arg CHECK_EPOCH="$epoch" \
--build-arg VERSION="$version" \
--build-arg COMMIT="$commit" \
--build-arg COMMIT_DATE="$commit_date" \
-t "$("$SCRIPT_DIR/projectname")" . -t "$("$SCRIPT_DIR/projectname")" .
} }