Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f3dc6e78a7 |
+1
-7
@@ -366,17 +366,11 @@ bucket/
|
||||
│ └── {full-hash} # Compressed+encrypted blob
|
||||
│
|
||||
└── metadata/
|
||||
└── {remote-key}/
|
||||
└── {snapshot-id}/
|
||||
├── db.zst.age # Encrypted binary SQLite database
|
||||
└── manifest.json.zst # Blob list (for pruning/verification)
|
||||
```
|
||||
|
||||
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
|
||||
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
|
||||
written to the store as a directory name. See
|
||||
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
|
||||
derivation and a worked example.
|
||||
|
||||
## Thread Safety
|
||||
|
||||
- `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`.
|
||||
|
||||
+3
-5
@@ -22,10 +22,8 @@ FROM golang:1.26.1-alpine@sha256:2389ebfa5b7f43eeafbd6be0c3700cc46690ef842ad962f
|
||||
|
||||
ARG VERSION=dev
|
||||
|
||||
# Build tooling: make, plus a C toolchain because `go test -race` needs cgo.
|
||||
# The sqlite driver is pure Go (modernc.org/sqlite), so no sqlite library or
|
||||
# CLI is required.
|
||||
RUN apk add --no-cache make build-base
|
||||
# Install build dependencies for CGO (mattn/go-sqlite3) and sqlite3 CLI (tests)
|
||||
RUN apk add --no-cache make build-base sqlite
|
||||
|
||||
WORKDIR /src
|
||||
|
||||
@@ -73,7 +71,7 @@ RUN CGO_ENABLED=0 go build -ldflags "-X 'sneak.berlin/go/vaultik/internal/global
|
||||
# alpine:3.21, 2026-02-25
|
||||
FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709
|
||||
|
||||
RUN apk add --no-cache ca-certificates
|
||||
RUN apk add --no-cache ca-certificates sqlite
|
||||
|
||||
# Copy binary from builder
|
||||
COPY --from=builder /vaultik /usr/local/bin/vaultik
|
||||
|
||||
+5
-5
@@ -72,11 +72,11 @@ RUN [ -n "$CHECK_EPOCH" ] || exit 1
|
||||
# running, and exits 0 reporting `0 issues.` on a tree the real config
|
||||
# fails. Demonstrated on this repo at this pin, recorded on
|
||||
# https://git.eeqj.de/sneak/vaultik/pulls/114: with a planted
|
||||
# over-length line, `script/lint` exits 1 naming the `revive` finding
|
||||
# with `linters:` and exits 0 with `linterz:`. A set-but-ineffective
|
||||
# config quietly falling back to defaults is precisely the false-green
|
||||
# class this gate exists to eliminate, so it must not sit in the gate's
|
||||
# own configuration.
|
||||
# over-length line, `script/lint` exits 1 naming the `lll` finding with
|
||||
# `linters:` and exits 0 with `linterz:`. A set-but-ineffective config
|
||||
# quietly falling back to defaults is precisely the false-green class
|
||||
# this gate exists to eliminate, so it must not sit in the gate's own
|
||||
# configuration.
|
||||
#
|
||||
# `config verify` catches it, and it does so OFFLINE at this pinned
|
||||
# version -- verified, not assumed. Under `docker run --network none`
|
||||
|
||||
@@ -251,8 +251,7 @@ local index alone, and still exits zero.
|
||||
per-snapshot-name (`--keep-latest` keeps the latest of each name, not the
|
||||
latest globally).
|
||||
* `--keep-latest`: Keep only the most recent snapshot of each name
|
||||
* `--older-than <duration>`: Remove snapshots older than duration (e.g. `30d`,
|
||||
`4w`, `6mo`, `1y`; `m` is minutes, `mo` is months)
|
||||
* `--older-than <duration>`: Remove snapshots older than duration (e.g. `30d`, `6m`, `1y`)
|
||||
* `--snapshot <name>`: Restrict to specific snapshot names (repeat for multiple)
|
||||
* `--force`: Skip confirmation prompt
|
||||
|
||||
@@ -345,7 +344,7 @@ both are set.
|
||||
├── blobs/
|
||||
│ └── <aa>/<bb>/<full_blob_hash>
|
||||
└── metadata/
|
||||
└── <remote-key>/
|
||||
└── <snapshot_id>/
|
||||
├── db.zst.age # Encrypted binary SQLite database
|
||||
└── manifest.json.zst # Unencrypted blob list (for pruning)
|
||||
```
|
||||
@@ -356,18 +355,8 @@ both are set.
|
||||
* `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling
|
||||
pruning without the private key
|
||||
|
||||
Snapshot IDs follow the human-readable format
|
||||
`<hostname>_<snapshot-name>_<RFC3339-timestamp>` (e.g.
|
||||
`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the
|
||||
destination store in plaintext. Each snapshot's metadata directory is named
|
||||
with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a listing
|
||||
of the store reveals no hostname or snapshot name. The backup time is not
|
||||
hidden: manifest.json.zst carries a plaintext timestamp, and object
|
||||
modification times are visible at the storage layer regardless. For example,
|
||||
`server1_home_2025-06-01T12:00:00Z` is stored under
|
||||
`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`.
|
||||
See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
|
||||
derivation.
|
||||
Snapshot IDs follow the format `<hostname>_<snapshot-name>_<RFC3339-timestamp>`
|
||||
(e.g. `server1_home_2025-06-01T12:00:00Z`).
|
||||
|
||||
### data flow
|
||||
|
||||
@@ -384,7 +373,7 @@ derivation.
|
||||
|
||||
**restore:**
|
||||
|
||||
1. Download and decrypt `metadata/<remote-key>/db.zst.age`
|
||||
1. Download and decrypt `metadata/<snapshot_id>/db.zst.age`
|
||||
2. Open the binary SQLite database
|
||||
3. Query files (optionally filtered by paths)
|
||||
4. Download and decrypt required blobs
|
||||
@@ -614,6 +603,7 @@ regardless of color setting (emoji are not color).
|
||||
and the pre-commit hook both run it. A `golangci-lint` installed on
|
||||
`PATH` is not a substitute and is never used on a host, whatever its
|
||||
version.
|
||||
* `sqlite3` CLI, which the test suite shells out to
|
||||
* S3-compatible object storage (or local filesystem, or rclone remote)
|
||||
|
||||
## development workflow
|
||||
@@ -644,8 +634,8 @@ standard: normalized scripts in `script/` are the entrypoints for the
|
||||
development workflow, and the Makefile targets are thin shims that call
|
||||
them. We provide:
|
||||
|
||||
* `script/bootstrap` — install all development dependencies (go, Go
|
||||
module download). It deliberately does not install `golangci-lint`;
|
||||
* `script/bootstrap` — install all development dependencies (go, sqlite3,
|
||||
Go module download). It deliberately does not install `golangci-lint`;
|
||||
see `script/lint` below.
|
||||
* `script/setup` — make a fresh clone ready for development: runs
|
||||
`script/bootstrap`, then `script/install-precommit`
|
||||
|
||||
@@ -25,54 +25,6 @@ release" is exactly the contradiction
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-09-21: Stopped `prune` from reporting a failed row count as 0
|
||||
([issue #96](https://git.eeqj.de/sneak/vaultik/issues/96)). The seven
|
||||
`getTableCount` reads in `PruneDatabase` discarded their error, so a
|
||||
query that could not run became a plausible `0` and the before/after
|
||||
delta computed from it looked like real work. Each read now logs at
|
||||
warn on failure and renders as `unknown`, never `0`, so an empty table
|
||||
is distinguishable from one that could not be queried. The counts have
|
||||
no `--json` representation — under `--json` the summary is suppressed
|
||||
entirely — so nothing there can show a false `0`.
|
||||
|
||||
- 2026-09-21: Stopped `--json` from silencing stderr diagnostics
|
||||
([issue #112](https://git.eeqj.de/sneak/vaultik/issues/112)). `--json`
|
||||
used to be folded into `Quiet`, which pinned the log level to `WARN`,
|
||||
so `prune --json` gave a machine consumer no record of the local index
|
||||
rows it deleted even under `--verbose`. `--json` now quiets only the
|
||||
stdout UI (the JSON document must stay clean, per
|
||||
[issue #108](https://git.eeqj.de/sneak/vaultik/issues/108)); the stderr
|
||||
log level follows `--verbose`/`--debug` again. The coupling was
|
||||
removed the same way for `snapshot verify`, `snapshot remove`, and
|
||||
`remote info`, which carried it for the same outdated reason.
|
||||
|
||||
- 2026-09-21: Made the s3 storage backend report a missing object as
|
||||
`storage.ErrNotFound`, like the `file` and `rclone` backends and as the
|
||||
`Storer` interface documents. `S3Storer.Get` and `Stat` returned the raw
|
||||
AWS SDK error, so `errors.Is(err, storage.ErrNotFound)` was false on s3
|
||||
and callers branched differently per backend. Added a small `s3.IsNotFound`
|
||||
helper (reused by `HeadObject`) and a test that a missing key maps to
|
||||
`ErrNotFound`
|
||||
([issue #129](https://git.eeqj.de/sneak/vaultik/issues/129)).
|
||||
- 2026-09-21: Fixed `verify --deep` reporting healthy snapshots as
|
||||
corrupt. Its final blob-integrity check hashed the encrypted
|
||||
downloaded bytes with a single SHA256 and compared that to the blob
|
||||
ID, which is the double SHA256 of the plaintext, so the two could
|
||||
never match. It now hashes the decompressed plaintext and compares the
|
||||
double SHA256. Added a test that backs up a real snapshot, deep-verifies
|
||||
it, then flips a byte in one stored blob and confirms deep verification
|
||||
then fails
|
||||
([issue #131](https://git.eeqj.de/sneak/vaultik/issues/131)).
|
||||
|
||||
- 2026-09-21: Made `snapshot create` VACUUM the per-snapshot metadata
|
||||
database through the `modernc.org/sqlite` driver instead of shelling
|
||||
out to the external `sqlite` command-line binary (issue #120). A
|
||||
backup no longer needs that binary on `PATH`, so `make check` passes
|
||||
on a stock `go install` host; `script/bootstrap` and the `Dockerfile`
|
||||
(both the test-build and the shipped runtime stage) no longer install
|
||||
it, and a new test asserts the uploaded database keeps no pages from
|
||||
deleted rows. Dropped the now-false note on the 2026-08-07 entry below
|
||||
that said bootstrap installs it.
|
||||
- 2026-09-21: Made `.gitea/workflows/check.yml` run on pushes to `main`
|
||||
and `next` and on pull requests against either, so unit PRs (whose
|
||||
base is `next`) and `next` itself get a CI run instead of relying on a
|
||||
@@ -89,24 +41,6 @@ release" is exactly the contradiction
|
||||
keeps that exact compiler from auto-switching. Bumping Go now touches
|
||||
`go.mod`, the checksum, and the `Dockerfile` `golang` digest together.
|
||||
|
||||
- 2026-09-21: Collapsed the two duration parsers into one and fixed the
|
||||
`--older-than` months example
|
||||
([issue #123](https://git.eeqj.de/sneak/vaultik/issues/123)). Two
|
||||
functions named `parseDuration` existed with different grammars;
|
||||
`snapshot purge --older-than` and `--keep-newer-than` both already went
|
||||
through the one in `internal/vaultik`, while the richer copy in
|
||||
`internal/cli/duration.go` was reachable only from its own test. Kept
|
||||
the live-path parser and deleted the unused one, so no flag's accepted
|
||||
grammar changes. The trap the issue was filed over: `README.md`
|
||||
documented `6m` as the months example for `--older-than`, but `m` is
|
||||
minutes, so the documented command deleted every snapshot older than
|
||||
six minutes on a destructive flag. Corrected the doc to `6mo` and put
|
||||
both flags' help text on one example list that states `m` is minutes
|
||||
and `mo` is months. The surviving parser now rejects negatives, which
|
||||
it previously accepted (`-5h`) or silently made positive (`-5d`).
|
||||
Table-driven tests cover every unit, `6m` as six minutes, `6mo` as 180
|
||||
days, and rejection of a bare number, an unknown unit, and a negative.
|
||||
|
||||
- 2026-08-10: Moved every lint run into its own container, as a build
|
||||
step ([issue #113](https://git.eeqj.de/sneak/vaultik/issues/113)).
|
||||
New root `Dockerfile.lint`, built by `script/lint`, runs
|
||||
@@ -135,11 +69,10 @@ release" is exactly the contradiction
|
||||
into each check command, and a fresh `$(date +%s%N)$$` per invocation
|
||||
computed as a bare assignment. `cmd/vaultik/lintdocker_test.go`
|
||||
parses both Dockerfiles and both scripts and fails if any part of
|
||||
that is dropped, because every way of losing it is silent. No test
|
||||
asserts that no script runs the host linter: `script/lint` is the one
|
||||
lint entry point and runs `golangci-lint` only inside the container,
|
||||
and keeping it that way is a review matter, not something a test
|
||||
proves.
|
||||
that is dropped, because every way of losing it is silent. Its
|
||||
host-lint assertion is structural — no script runs `golangci-lint`
|
||||
except through `docker` — rather than a search for the one retired
|
||||
variable name, which nothing could ever reintroduce.
|
||||
|
||||
The product `Dockerfile` lost its lint stage rather than gaining a
|
||||
second linter pin: `make lint` is now `docker build`, so the stage
|
||||
@@ -606,7 +539,7 @@ release" is exactly the contradiction
|
||||
was green was wrong.
|
||||
- 2026-08-07: Added the standard `.golangci.yml` and `.editorconfig`
|
||||
(issue #59); lint findings under the new config are tracked in issue
|
||||
#61.
|
||||
#61. `script/bootstrap` now installs sqlite3 (needed by tests).
|
||||
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
|
||||
Makefile shims, README Entrypoints section
|
||||
- 2026-07-02: Consolidated CLI verbs, retired overlapping commands; bound
|
||||
|
||||
@@ -28,11 +28,6 @@ import (
|
||||
// -- that a real finding actually fails the build -- is verified by
|
||||
// hand against a deliberately broken tree, recorded on the pull
|
||||
// request.
|
||||
//
|
||||
// One property is deliberately NOT tested here: that no script runs the
|
||||
// linter on the host. script/lint is the only lint entry point, and it
|
||||
// runs golangci-lint only inside the container; keeping it that way is a
|
||||
// review matter, not something a test in this file establishes.
|
||||
|
||||
// The files under guard, relative to the repository root.
|
||||
const (
|
||||
@@ -42,8 +37,9 @@ const (
|
||||
cibuildScript = "script/cibuild"
|
||||
)
|
||||
|
||||
// linterBinary is the linter's command name, used to locate the
|
||||
// config-verify and lint steps in Dockerfile.lint.
|
||||
// linterBinary is the linter's command name. Every occurrence of it in
|
||||
// executable shell in this repo must be inside a docker invocation; see
|
||||
// TestNoHostLintPathRemains.
|
||||
const linterBinary = "golangci-lint"
|
||||
|
||||
// checkEpochARG is the declaration, with no default value. A default
|
||||
@@ -223,6 +219,90 @@ func TestCibuildBuildsBothDockerfilesWithFreshEpochs(t *testing.T) {
|
||||
"%s must build %s", cibuildScript, lintDockerfile)
|
||||
}
|
||||
|
||||
// TestNoHostLintPathRemains fails if any escape hatch to a host linter
|
||||
// comes back. The owner's ruling is that every lint run happens inside
|
||||
// a container; a PATH binary that happens to match the pinned version
|
||||
// is a different build reached by a different code path, and admitting
|
||||
// it is what lets a local pass disagree with CI.
|
||||
//
|
||||
// This asserts the PROPERTY -- no script invokes the linter except
|
||||
// through docker -- rather than the absence of any particular variable
|
||||
// name. An earlier version of this test looked only for the literal
|
||||
// VAULTIK_LINT_IN_CONTAINER, the name of the hatch that was removed
|
||||
// alongside it, so nothing could ever trip it again: a hatch under any
|
||||
// other name left it passing. A structural test that passes on a broken
|
||||
// tree is worse than no test, because it is what a later reader trusts
|
||||
// instead of re-deriving the invariant.
|
||||
//
|
||||
// script/lint-fix is not exempted. It is the one script that runs the
|
||||
// linter as a container rather than as a build step, but it still runs
|
||||
// it in one, so the same property holds of it.
|
||||
func TestNoHostLintPathRemains(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
root := repoRoot(t)
|
||||
|
||||
entries, err := os.ReadDir(filepath.Join(root, "script"))
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, entries, "no scripts found to scan")
|
||||
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() {
|
||||
continue
|
||||
}
|
||||
|
||||
name := filepath.Join("script", entry.Name())
|
||||
for _, line := range shellCode(readRepoFile(t, name)) {
|
||||
assertLinterIsContainerised(t, name, line)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// assertLinterIsContainerised fails if the line runs the linter without
|
||||
// handing it to docker first. Position matters: docker has to come
|
||||
// before the binary, or the line is running the host linter and merely
|
||||
// mentioning docker afterwards.
|
||||
func assertLinterIsContainerised(t *testing.T, name, line string) {
|
||||
t.Helper()
|
||||
|
||||
at := strings.Index(line, linterBinary)
|
||||
if at < 0 {
|
||||
return
|
||||
}
|
||||
|
||||
docker := strings.Index(line, "docker")
|
||||
|
||||
assert.True(t, docker >= 0 && docker < at,
|
||||
"%s runs %s on the host; every lint run happens in a container"+
|
||||
" (line: %s)", name, linterBinary, line)
|
||||
}
|
||||
|
||||
// TestShellCodeSeesCodeAndNotProse keeps the scanner above honest. It
|
||||
// has to ignore comments and here-document bodies, because script/lint
|
||||
// and script/bootstrap both NAME golangci-lint in prose -- in comments,
|
||||
// and in the error text they print -- precisely to say that the host
|
||||
// binary is never used. A scanner that went blind, by over-eager
|
||||
// stripping or by failing to join continuation lines, would make
|
||||
// TestNoHostLintPathRemains pass on everything.
|
||||
func TestShellCodeSeesCodeAndNotProse(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
script := strings.Join([]string{
|
||||
"#!/bin/sh",
|
||||
"# a comment naming golangci-lint",
|
||||
"cat >&2 <<EOF",
|
||||
"prose naming golangci-lint, printed not executed",
|
||||
"EOF",
|
||||
"docker run --rm \\",
|
||||
" \"$image\" \\",
|
||||
" golangci-lint run ./...",
|
||||
}, "\n")
|
||||
|
||||
assert.Equal(t,
|
||||
[]string{"cat >&2 <<EOF", `docker run --rm "$image" golangci-lint run ./...`},
|
||||
shellCode(script))
|
||||
}
|
||||
|
||||
// assertEpochExpandedInto fails unless some instruction runs the named
|
||||
// command with the epoch expanded into it. Expansion, not mere
|
||||
// declaration: an ARG that no instruction references is not guaranteed
|
||||
@@ -327,6 +407,70 @@ func indexContaining(found []string, want string) int {
|
||||
return -1
|
||||
}
|
||||
|
||||
// shellCode returns a POSIX shell script's executable lines: comments
|
||||
// dropped, here-document bodies dropped, and backslash continuations
|
||||
// joined so a multi-line command is a single string. Whitespace is
|
||||
// collapsed, as it is for Dockerfile instructions.
|
||||
//
|
||||
// Both exclusions are load-bearing rather than tidiness. The scripts
|
||||
// name golangci-lint in prose to state that the host binary is never
|
||||
// used, and joining continuations is what lets the one legitimate
|
||||
// container invocation -- script/lint-fix's `docker run`, whose linter
|
||||
// command sits several lines below the word `docker` -- be recognised
|
||||
// as containerised.
|
||||
func shellCode(contents string) []string {
|
||||
var (
|
||||
out []string
|
||||
joined string
|
||||
terminate string
|
||||
)
|
||||
|
||||
for line := range strings.SplitSeq(contents, "\n") {
|
||||
trimmed := strings.TrimSpace(line)
|
||||
|
||||
if terminate != "" {
|
||||
if trimmed == terminate {
|
||||
terminate = ""
|
||||
}
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if joined == "" && (trimmed == "" || strings.HasPrefix(trimmed, "#")) {
|
||||
continue
|
||||
}
|
||||
|
||||
joined += strings.TrimSuffix(trimmed, `\`) + " "
|
||||
if strings.HasSuffix(trimmed, `\`) {
|
||||
continue
|
||||
}
|
||||
|
||||
joined = strings.Join(strings.Fields(joined), " ")
|
||||
terminate = heredocTerminator(joined)
|
||||
|
||||
out = append(out, joined)
|
||||
joined = ""
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
// heredocTerminator returns the terminator of the here-document a
|
||||
// command opens, or "" if it opens none. Only the first on a line is
|
||||
// recognised; nothing in script/ opens two.
|
||||
func heredocTerminator(line string) string {
|
||||
_, after, opens := strings.Cut(line, "<<")
|
||||
if !opens {
|
||||
return ""
|
||||
}
|
||||
|
||||
// `<<-` strips leading tabs from the body; the terminator word is
|
||||
// the same either way, and callers compare against trimmed lines.
|
||||
word, _, _ := strings.Cut(strings.TrimPrefix(after, "-"), " ")
|
||||
|
||||
return strings.Trim(word, `'"`)
|
||||
}
|
||||
|
||||
// readRepoFile reads a file by its path relative to the repository
|
||||
// root.
|
||||
func readRepoFile(t *testing.T, name string) string {
|
||||
|
||||
+3
-5
@@ -192,12 +192,10 @@ Tracks blob upload metrics.
|
||||
After a snapshot is completed:
|
||||
1. Copy database to temporary file
|
||||
2. Clean temporary database to contain only current snapshot data
|
||||
3. VACUUM the trimmed database so deleted rows leave no pages behind
|
||||
3. Export to SQL dump using sqlite3
|
||||
4. Compress with zstd and encrypt with age
|
||||
5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
|
||||
6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst`
|
||||
|
||||
The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation).
|
||||
5. Upload to S3 as `metadata/{snapshot-id}/db.zst.age`
|
||||
6. Generate blob manifest and upload as `metadata/{snapshot-id}/manifest.json.zst`
|
||||
|
||||
### 4. Restore Process
|
||||
|
||||
|
||||
+17
-37
@@ -17,13 +17,11 @@ Vaultik stores all backup data in an S3-compatible object store. The repository
|
||||
│ └── <hash[2:4]>/
|
||||
│ └── <full-hash>
|
||||
└── metadata/
|
||||
└── <remote-key>/
|
||||
└── <snapshot-id>/
|
||||
├── db.zst.age
|
||||
└── manifest.json.zst
|
||||
```
|
||||
|
||||
The metadata subdirectory is named with the **remote key**, a one-way hash of the snapshot ID, not with the human-readable snapshot ID itself. See [Remote Key Derivation](#remote-key-derivation).
|
||||
|
||||
## Blobs Directory (`blobs/`)
|
||||
|
||||
### Structure
|
||||
@@ -42,11 +40,9 @@ Blobs contain the actual file data from backups and must be encrypted for securi
|
||||
|
||||
## Metadata Directory (`metadata/`)
|
||||
|
||||
Each snapshot has its own subdirectory. The directory is **not** named with the human-readable snapshot ID; it is named with the remote key — a one-way hash of that ID. The human ID is never written to the destination store as a directory name (see [Remote Key Derivation](#remote-key-derivation)).
|
||||
Each snapshot has its own subdirectory named with the snapshot ID.
|
||||
|
||||
### Snapshot ID Format
|
||||
|
||||
The human-readable snapshot ID is used in CLI arguments, log lines, and the local database. It is not written to the destination store.
|
||||
- **Format**: `<hostname>_<snapshot-name>_<RFC3339>` (or `<hostname>_<RFC3339>` if no
|
||||
name was specified)
|
||||
- **Example**: `laptop_home_2024-01-15T14:30:52Z`
|
||||
@@ -55,19 +51,6 @@ The human-readable snapshot ID is used in CLI arguments, log lines, and the loca
|
||||
- Snapshot name from the configured `snapshots:` map (optional)
|
||||
- RFC3339 UTC timestamp
|
||||
|
||||
This ID reveals the hostname, the configured snapshot name, and the backup time, so it is never used as the on-disk directory name — the remote key is used instead.
|
||||
|
||||
### Remote Key Derivation
|
||||
|
||||
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string with no structure a remote observer can reverse. Implemented in `internal/snapshot/remotekey.go`.
|
||||
|
||||
Worked example:
|
||||
- Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
|
||||
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
|
||||
- Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
|
||||
|
||||
Because the hash is one-way, a listing of the destination store reveals neither the hostname nor the snapshot name of any backup. The same remote key is stored in the manifest's `snapshot_id` field.
|
||||
|
||||
### Files in Each Snapshot Directory
|
||||
|
||||
#### `db.zst.age` - Encrypted Database
|
||||
@@ -85,17 +68,16 @@ Because the hash is one-way, a listing of the destination store reveals neither
|
||||
- **Structure**:
|
||||
```json
|
||||
{
|
||||
"snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
|
||||
"timestamp": "2025-06-01T12:00:00Z",
|
||||
"snapshot_id": "laptop_home_2024-01-15T14:30:52Z",
|
||||
"timestamp": "2024-01-15T14:30:52Z",
|
||||
"blob_count": 42,
|
||||
"total_compressed_size": 1048576,
|
||||
"blobs": [
|
||||
{ "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 24576 },
|
||||
{ "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 32768 }
|
||||
"cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678",
|
||||
"deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678",
|
||||
...
|
||||
]
|
||||
}
|
||||
```
|
||||
`snapshot_id` is the remote key (a hash), not the human ID; `timestamp` is written in the clear.
|
||||
|
||||
### Why Manifest is Unencrypted
|
||||
The manifest must be readable without the private key to enable:
|
||||
@@ -104,7 +86,7 @@ The manifest must be readable without the private key to enable:
|
||||
3. **Verification** - Checking blob existence without decryption
|
||||
4. **Cross-snapshot deduplication analysis** - Finding shared blobs between snapshots
|
||||
|
||||
The manifest contains the remote key, the backup timestamp, the blob count and total compressed size, and each blob's hash and compressed size. It contains no file names, paths, or other decrypted metadata.
|
||||
The manifest only contains blob hashes, not file names or any other sensitive information.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
@@ -114,21 +96,19 @@ The manifest contains the remote key, the backup timestamp, the blob count and t
|
||||
- **File-to-chunk mappings** (in db.zst.age)
|
||||
|
||||
### What's Not Encrypted
|
||||
- **The remote key** — directory names and the manifest `snapshot_id`, a one-way hash of the snapshot ID (see [Remote Key Derivation](#remote-key-derivation))
|
||||
- **The backup timestamp** (in manifest.json.zst)
|
||||
- **Blob hashes and their compressed sizes** (in manifest.json.zst)
|
||||
- **Blob count and total compressed size per snapshot** (in manifest.json.zst)
|
||||
- **Blob hashes** (in manifest.json.zst)
|
||||
- **Snapshot IDs** (directory names)
|
||||
- **Blob count per snapshot** (in manifest.json.zst)
|
||||
|
||||
### Privacy Implications
|
||||
From the unencrypted data, an observer of the destination store can determine:
|
||||
- **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear
|
||||
- How many blobs each snapshot references, and the total compressed size
|
||||
- The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns)
|
||||
|
||||
Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless.
|
||||
From the unencrypted data, an observer can determine:
|
||||
- When backups were taken (from snapshot IDs)
|
||||
- Which hostname created backups (from snapshot IDs)
|
||||
- How many blobs each snapshot references
|
||||
- Which blobs are shared between snapshots (deduplication patterns)
|
||||
- The size of each encrypted blob
|
||||
|
||||
An observer cannot determine:
|
||||
- The hostname or snapshot name of any backup (the directory name and the manifest `snapshot_id` are one-way hashes of the human ID)
|
||||
- File names or paths
|
||||
- File contents
|
||||
- File permissions or ownership
|
||||
|
||||
+4
-11
@@ -48,11 +48,6 @@ type AppOptions struct {
|
||||
// silenced — per the documented convention that --quiet suppresses
|
||||
// non-error output only. The startup banner is printed by Entry
|
||||
// before cobra parses arguments, gated by the same arg-level check.
|
||||
//
|
||||
// --json quiets the UI here too, because stdout then carries a JSON
|
||||
// document and human narration would corrupt it. Unlike Quiet it does
|
||||
// not lower the stderr log level (issue #112), so --verbose/--debug
|
||||
// still surface diagnostics alongside the document.
|
||||
func setupGlobals(
|
||||
lc fx.Lifecycle, g *globals.Globals, v *vaultik.Vaultik, opts log.Options,
|
||||
) {
|
||||
@@ -60,7 +55,7 @@ func setupGlobals(
|
||||
OnStart: func(_ context.Context) error {
|
||||
g.StartTime = time.Now().UTC()
|
||||
|
||||
if opts.Cron || opts.Quiet || opts.JSON {
|
||||
if opts.Cron || opts.Quiet {
|
||||
v.UI.SetQuiet(true)
|
||||
}
|
||||
|
||||
@@ -207,10 +202,9 @@ func RunApp(ctx context.Context, app *fx.App) error {
|
||||
// instance in a goroutine, report a failure prefixed with failMsg
|
||||
// (suppressed while suppressErrors is true, e.g. under --json), then
|
||||
// trigger shutdown. The operation is cancelled when the app stops.
|
||||
// jsonOutput marks a command whose stdout is a JSON document: it quiets
|
||||
// the UI but, unlike Quiet, leaves the stderr log level alone.
|
||||
// extraQuiet is OR-ed into LogOptions.Quiet (e.g. --json output modes).
|
||||
func runVaultikApp(
|
||||
cmd *cobra.Command, jsonOutput, suppressErrors bool,
|
||||
cmd *cobra.Command, extraQuiet, suppressErrors bool,
|
||||
failMsg string, op func(v *vaultik.Vaultik) error,
|
||||
) error {
|
||||
configPath, err := ResolveConfigPath()
|
||||
@@ -225,8 +219,7 @@ func runVaultikApp(
|
||||
LogOptions: log.Options{
|
||||
Verbose: rootFlags.Verbose,
|
||||
Debug: rootFlags.Debug,
|
||||
Quiet: rootFlags.Quiet,
|
||||
JSON: jsonOutput,
|
||||
Quiet: rootFlags.Quiet || extraQuiet,
|
||||
},
|
||||
Modules: []fx.Option{},
|
||||
Invokes: []fx.Option{
|
||||
|
||||
+1
-30
@@ -1,7 +1,6 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
@@ -25,11 +24,6 @@ const configSetArgs = 2
|
||||
// parent config dirs (e.g. ~/.config) are conventionally traversable.
|
||||
const configDirMode = 0o755
|
||||
|
||||
// configYAMLIndent matches the 2-space indentation of defaultConfigTemplate,
|
||||
// so `config set` writes the file back with the same indentation rather than
|
||||
// yaml.Marshal's 4-space default.
|
||||
const configYAMLIndent = 2
|
||||
|
||||
var (
|
||||
errConfigExists = errors.New("config file already exists")
|
||||
errEmptyConfig = errors.New("empty config file")
|
||||
@@ -387,7 +381,7 @@ Examples:
|
||||
return err
|
||||
}
|
||||
|
||||
out, err := marshalConfigYAML(root)
|
||||
out, err := yaml.Marshal(root)
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshaling config: %w", err)
|
||||
}
|
||||
@@ -411,29 +405,6 @@ Examples:
|
||||
}
|
||||
}
|
||||
|
||||
// marshalConfigYAML renders a config document tree with 2-space indentation,
|
||||
// matching defaultConfigTemplate. yaml.Marshal defaults to 4 spaces, which
|
||||
// would reindent the whole file on the first `config set` despite the promise
|
||||
// to preserve formatting.
|
||||
func marshalConfigYAML(root *yaml.Node) ([]byte, error) {
|
||||
var buf bytes.Buffer
|
||||
|
||||
enc := yaml.NewEncoder(&buf)
|
||||
enc.SetIndent(configYAMLIndent)
|
||||
|
||||
err := enc.Encode(root)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
err = enc.Close()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return buf.Bytes(), nil
|
||||
}
|
||||
|
||||
// loadYAMLFile parses a YAML file into a yaml.Node document tree,
|
||||
// which preserves comments and ordering for round-tripping.
|
||||
func loadYAMLFile(path string) (*yaml.Node, error) {
|
||||
|
||||
@@ -188,47 +188,6 @@ func TestYAMLPathSet(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestConfigSetPreservesFormatting asserts the `config set` write path
|
||||
// (marshalConfigYAML) round-trips a 2-space-indented file without reindenting
|
||||
// it to yaml.Marshal's 4-space default, and keeps comments.
|
||||
func TestConfigSetPreservesFormatting(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
root := parseTestYAML(t)
|
||||
|
||||
err := yamlPathSet(root, splitPath("s3.bucket"), "newbucket")
|
||||
if err != nil {
|
||||
t.Fatalf("set s3.bucket: %v", err)
|
||||
}
|
||||
|
||||
out, err := marshalConfigYAML(root)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal: %v", err)
|
||||
}
|
||||
|
||||
text := string(out)
|
||||
|
||||
for _, want := range []string{"# top comment", "# inline comment"} {
|
||||
if !contains(text, want) {
|
||||
t.Errorf("round-tripped YAML dropped comment %q:\n%s", want, text)
|
||||
}
|
||||
}
|
||||
|
||||
// Nested map keys stay at 2-space indent; the bug reindented them to 4.
|
||||
if !contains(text, "\n bucket: newbucket") {
|
||||
t.Errorf("expected 2-space indent for s3.bucket, got:\n%s", text)
|
||||
}
|
||||
|
||||
if contains(text, "\n bucket:") {
|
||||
t.Errorf("s3.bucket reindented to 4 spaces:\n%s", text)
|
||||
}
|
||||
|
||||
// Sequence items under a key also stay at 2 spaces.
|
||||
if !contains(text, "\n - age1aaa") {
|
||||
t.Errorf("expected 2-space indent for sequence item, got:\n%s", text)
|
||||
}
|
||||
}
|
||||
|
||||
func splitPath(s string) []string {
|
||||
return strings.Split(s, ".")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Approximate lengths of the extended calendar units accepted by
|
||||
// parseDuration.
|
||||
const (
|
||||
durationDay = 24 * time.Hour
|
||||
durationWeek = 7 * durationDay
|
||||
durationMonth = 30 * durationDay
|
||||
durationYear = 365 * durationDay
|
||||
)
|
||||
|
||||
var (
|
||||
errNegativeDuration = errors.New("negative durations are not supported")
|
||||
errInvalidDuration = errors.New("invalid duration format")
|
||||
errUnknownTimeUnit = errors.New("unknown time unit")
|
||||
)
|
||||
|
||||
// parseDuration parses duration strings. Supports standard Go duration format
|
||||
// (e.g., "3h30m", "1h45m30s") as well as extended units:
|
||||
// - d: days (e.g., "30d", "7d")
|
||||
// - w: weeks (e.g., "2w", "4w")
|
||||
// - mo: months (30 days) (e.g., "6mo", "1mo")
|
||||
// - y: years (365 days) (e.g., "1y", "2y")
|
||||
//
|
||||
// Can combine units: "1y6mo", "2w3d", "1d12h30m"
|
||||
func parseDuration(s string) (time.Duration, error) {
|
||||
// First try standard Go duration parsing
|
||||
d, err := time.ParseDuration(s)
|
||||
if err == nil {
|
||||
return d, nil
|
||||
}
|
||||
|
||||
// Extended duration parsing
|
||||
// Check for negative values
|
||||
if strings.HasPrefix(strings.TrimSpace(s), "-") {
|
||||
return 0, errNegativeDuration
|
||||
}
|
||||
|
||||
// Pattern matches: number + unit, repeated
|
||||
re := regexp.MustCompile(`(\d+(?:\.\d+)?)\s*([a-zA-Z]+)`)
|
||||
matches := re.FindAllStringSubmatch(s, -1)
|
||||
|
||||
if len(matches) == 0 {
|
||||
return 0, fmt.Errorf("%w: %q", errInvalidDuration, s)
|
||||
}
|
||||
|
||||
var total time.Duration
|
||||
|
||||
for _, match := range matches {
|
||||
valueStr := match[1]
|
||||
unit := strings.ToLower(match[2])
|
||||
|
||||
value, err := strconv.ParseFloat(valueStr, 64)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("invalid number %q: %w", valueStr, err)
|
||||
}
|
||||
|
||||
d, err := durationForUnit(value, unit)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
total += d
|
||||
}
|
||||
|
||||
return total, nil
|
||||
}
|
||||
|
||||
// durationForUnit converts a value with a (case-normalized) unit suffix
|
||||
// into a time.Duration, accepting Go's standard units plus the extended
|
||||
// calendar units.
|
||||
func durationForUnit(value float64, unit string) (time.Duration, error) {
|
||||
switch unit {
|
||||
// Standard time units
|
||||
case "ns", "nanosecond", "nanoseconds":
|
||||
return time.Duration(value), nil
|
||||
case "us", "µs", "microsecond", "microseconds":
|
||||
return time.Duration(value * float64(time.Microsecond)), nil
|
||||
case "ms", "millisecond", "milliseconds":
|
||||
return time.Duration(value * float64(time.Millisecond)), nil
|
||||
case "s", "sec", "second", "seconds":
|
||||
return time.Duration(value * float64(time.Second)), nil
|
||||
case "m", "min", "minute", "minutes":
|
||||
return time.Duration(value * float64(time.Minute)), nil
|
||||
case "h", "hr", "hour", "hours":
|
||||
return time.Duration(value * float64(time.Hour)), nil
|
||||
// Extended units
|
||||
case "d", "day", "days":
|
||||
return time.Duration(value * float64(durationDay)), nil
|
||||
case "w", "week", "weeks":
|
||||
return time.Duration(value * float64(durationWeek)), nil
|
||||
case "mo", "month", "months":
|
||||
// Using 30 days as approximation
|
||||
return time.Duration(value * float64(durationMonth)), nil
|
||||
case "y", "year", "years":
|
||||
// Using 365 days as approximation
|
||||
return time.Duration(value * float64(durationYear)), nil
|
||||
default:
|
||||
// Try parsing as standard Go duration unit
|
||||
testStr := "1" + unit
|
||||
|
||||
_, err := time.ParseDuration(testStr)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("%w: %q", errUnknownTimeUnit, unit)
|
||||
}
|
||||
|
||||
// It's a valid Go duration unit, parse the full value
|
||||
fullStr := fmt.Sprintf("%g%s", value, unit)
|
||||
|
||||
d, err := time.ParseDuration(fullStr)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("invalid duration %q: %w", fullStr, err)
|
||||
}
|
||||
|
||||
return d, nil
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,299 @@
|
||||
package cli //nolint:testpackage // needs access to unexported parseDuration
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
type parseDurationCase struct {
|
||||
name string
|
||||
input string
|
||||
expected time.Duration
|
||||
wantErr bool
|
||||
}
|
||||
|
||||
// runParseDurationCases executes a table of parseDuration cases as
|
||||
// parallel subtests.
|
||||
func runParseDurationCases(t *testing.T, tests []parseDurationCase) {
|
||||
t.Helper()
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, err := parseDuration(tt.input)
|
||||
|
||||
if tt.wantErr {
|
||||
require.Error(t, err, "expected error for input %q", tt.input)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
require.NoError(t, err, "unexpected error for input %q", tt.input)
|
||||
assert.Equal(t, tt.expected, got, "duration mismatch for input %q", tt.input)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseDurationStandard(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
runParseDurationCases(t, []parseDurationCase{
|
||||
{
|
||||
name: "standard seconds",
|
||||
input: "30s",
|
||||
expected: 30 * time.Second,
|
||||
},
|
||||
{
|
||||
name: "standard minutes",
|
||||
input: "45m",
|
||||
expected: 45 * time.Minute,
|
||||
},
|
||||
{
|
||||
name: "standard hours",
|
||||
input: "2h",
|
||||
expected: 2 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "standard combined",
|
||||
input: "3h30m",
|
||||
expected: 3*time.Hour + 30*time.Minute,
|
||||
},
|
||||
{
|
||||
name: "standard complex",
|
||||
input: "1h45m30s",
|
||||
expected: 1*time.Hour + 45*time.Minute + 30*time.Second,
|
||||
},
|
||||
{
|
||||
name: "standard with milliseconds",
|
||||
input: "1s500ms",
|
||||
expected: 1*time.Second + 500*time.Millisecond,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
func TestParseDurationExtendedUnits(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
runParseDurationCases(t, []parseDurationCase{
|
||||
// Extended units - days
|
||||
{
|
||||
name: "single day",
|
||||
input: "1d",
|
||||
expected: 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "multiple days",
|
||||
input: "7d",
|
||||
expected: 7 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "fractional days",
|
||||
input: "1.5d",
|
||||
expected: 36 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "days spelled out",
|
||||
input: "3days",
|
||||
expected: 3 * 24 * time.Hour,
|
||||
},
|
||||
// Extended units - weeks
|
||||
{
|
||||
name: "single week",
|
||||
input: "1w",
|
||||
expected: 7 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "multiple weeks",
|
||||
input: "4w",
|
||||
expected: 4 * 7 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "weeks spelled out",
|
||||
input: "2weeks",
|
||||
expected: 2 * 7 * 24 * time.Hour,
|
||||
},
|
||||
// Extended units - months
|
||||
{
|
||||
name: "single month",
|
||||
input: "1mo",
|
||||
expected: 30 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "multiple months",
|
||||
input: "6mo",
|
||||
expected: 6 * 30 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "months spelled out",
|
||||
input: "3months",
|
||||
expected: 3 * 30 * 24 * time.Hour,
|
||||
},
|
||||
// Extended units - years
|
||||
{
|
||||
name: "single year",
|
||||
input: "1y",
|
||||
expected: 365 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "multiple years",
|
||||
input: "2y",
|
||||
expected: 2 * 365 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "years spelled out",
|
||||
input: "1year",
|
||||
expected: 365 * 24 * time.Hour,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
func TestParseDurationCombinedAndErrors(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
runParseDurationCases(t, []parseDurationCase{
|
||||
// Combined extended units
|
||||
{
|
||||
name: "weeks and days",
|
||||
input: "2w3d",
|
||||
expected: 2*7*24*time.Hour + 3*24*time.Hour,
|
||||
},
|
||||
{
|
||||
name: "years and months",
|
||||
input: "1y6mo",
|
||||
expected: 365*24*time.Hour + 6*30*24*time.Hour,
|
||||
},
|
||||
{
|
||||
name: "days and hours",
|
||||
input: "1d12h",
|
||||
expected: 24*time.Hour + 12*time.Hour,
|
||||
},
|
||||
{
|
||||
name: "complex combination",
|
||||
input: "1y2mo3w4d5h6m7s",
|
||||
expected: 365*24*time.Hour + 2*30*24*time.Hour +
|
||||
3*7*24*time.Hour + 4*24*time.Hour +
|
||||
5*time.Hour + 6*time.Minute + 7*time.Second,
|
||||
},
|
||||
{
|
||||
name: "with spaces",
|
||||
input: "1d 12h 30m",
|
||||
expected: 24*time.Hour + 12*time.Hour + 30*time.Minute,
|
||||
},
|
||||
// Edge cases
|
||||
{
|
||||
name: "zero duration",
|
||||
input: "0s",
|
||||
expected: 0,
|
||||
},
|
||||
{
|
||||
name: "large duration",
|
||||
input: "10y",
|
||||
expected: 10 * 365 * 24 * time.Hour,
|
||||
},
|
||||
// Error cases
|
||||
{
|
||||
name: "empty string",
|
||||
input: "",
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "invalid format",
|
||||
input: "abc",
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "unknown unit",
|
||||
input: "5x",
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "invalid number",
|
||||
input: "xyzd",
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "negative not supported",
|
||||
input: "-5d",
|
||||
wantErr: true,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
func TestParseDurationSpecialCases(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Test that standard Go durations work exactly as expected
|
||||
standardDurations := []string{
|
||||
"300ms",
|
||||
"1.5h",
|
||||
"2h45m",
|
||||
"72h",
|
||||
"1us",
|
||||
"1µs",
|
||||
"1ns",
|
||||
}
|
||||
|
||||
for _, d := range standardDurations {
|
||||
expected, err := time.ParseDuration(d)
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := parseDuration(d)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, expected, got, "standard duration %q should parse identically", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseDurationRealWorldExamples(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Test real-world snapshot purge scenarios
|
||||
tests := []struct {
|
||||
description string
|
||||
input string
|
||||
olderThan time.Duration
|
||||
}{
|
||||
{
|
||||
description: "keep snapshots from last 30 days",
|
||||
input: "30d",
|
||||
olderThan: 30 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
description: "keep snapshots from last 6 months",
|
||||
input: "6mo",
|
||||
olderThan: 6 * 30 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
description: "keep snapshots from last year",
|
||||
input: "1y",
|
||||
olderThan: 365 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
description: "keep snapshots from last week and a half",
|
||||
input: "1w3d",
|
||||
olderThan: 10 * 24 * time.Hour,
|
||||
},
|
||||
{
|
||||
description: "keep snapshots from last 90 days",
|
||||
input: "90d",
|
||||
olderThan: 90 * 24 * time.Hour,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.description, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, err := parseDuration(tt.input)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, tt.olderThan, got)
|
||||
|
||||
// Verify the duration makes sense for snapshot purging
|
||||
assert.Greater(t, got, time.Hour,
|
||||
"snapshot purge duration should be at least an hour")
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1,139 +0,0 @@
|
||||
package cli //nolint:testpackage // shares the prune fixtures and capture helpers
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"io"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// staleRecordLogMessage is the local-cleanup audit line CleanupLocalSnapshots
|
||||
// logs for each stale record. It is exactly the signal issue #112 says a
|
||||
// machine consumer lost under --json: gated off stdout, and pinned below
|
||||
// the log level on stderr because --json used to force Quiet.
|
||||
const staleRecordLogMessage = "Removing stale local snapshot record"
|
||||
|
||||
// TestEntryPruneJSONStderrHonoursVerbosity is the end-to-end regression
|
||||
// guard for issue #112. Under --json the log level must still follow
|
||||
// --verbose/--debug rather than being pinned to WARN, so the
|
||||
// local-cleanup records reach stderr under --verbose while stdout stays
|
||||
// exactly one JSON document; without --verbose they stay below the
|
||||
// level, as they do without --json.
|
||||
//
|
||||
// Both halves are asserted together on the same run, because the fix has
|
||||
// to keep the document clean (issue #108) while freeing stderr.
|
||||
//
|
||||
// Not parallel: it replaces os.Args, os.Stdout, os.Stderr and the xdg
|
||||
// globals.
|
||||
//
|
||||
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||
func TestEntryPruneJSONStderrHonoursVerbosity(t *testing.T) {
|
||||
for _, testCase := range []struct {
|
||||
name string
|
||||
verbose bool
|
||||
wantOnStderr bool
|
||||
}{
|
||||
{
|
||||
name: "verbose json surfaces the cleanup record on stderr",
|
||||
verbose: true,
|
||||
wantOnStderr: true,
|
||||
},
|
||||
{
|
||||
name: "json alone keeps the cleanup record below the level",
|
||||
verbose: false,
|
||||
wantOnStderr: false,
|
||||
},
|
||||
} {
|
||||
t.Run(testCase.name, func(t *testing.T) {
|
||||
configPath := writeHermeticPruneConfig(t, true)
|
||||
|
||||
previousArgs := os.Args
|
||||
|
||||
t.Cleanup(func() {
|
||||
os.Args = previousArgs
|
||||
rootFlags = RootFlags{}
|
||||
})
|
||||
|
||||
args := []string{
|
||||
programName, flagConfig, configPath, cmdPrune, flagJSON,
|
||||
}
|
||||
if testCase.verbose {
|
||||
args = append(args, "--verbose")
|
||||
}
|
||||
|
||||
os.Args = args
|
||||
|
||||
stdout, stderr := captureProcessStdoutAndStderr(t, Entry)
|
||||
|
||||
// The document stays clean in both cases: freeing stderr must
|
||||
// not regress issue #108.
|
||||
requireExactlyOneJSONDocument(t, stdout)
|
||||
|
||||
if testCase.wantOnStderr {
|
||||
assert.Contains(t, stderr, staleRecordLogMessage,
|
||||
"--verbose --json must emit the cleanup record on stderr")
|
||||
assert.Contains(t, stderr, stalePruneSnapshotID,
|
||||
"the record must name the snapshot it removed")
|
||||
} else {
|
||||
assert.NotContains(t, stderr, staleRecordLogMessage,
|
||||
"without --verbose the record stays below the log level")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// captureProcessStdoutAndStderr redirects both of the process's own
|
||||
// standard streams to pipes for the duration of fn and returns what was
|
||||
// written to each. The redirection is at the file-descriptor level
|
||||
// because the logger binds os.Stderr when it initializes inside fn, and
|
||||
// the JSON document reaches os.Stdout independently; the point is to see
|
||||
// where each actually lands.
|
||||
//
|
||||
// Not parallel-safe: os.Stdout and os.Stderr are process-global.
|
||||
func captureProcessStdoutAndStderr(t *testing.T, fn func()) (string, string) {
|
||||
t.Helper()
|
||||
|
||||
outReader, outWriter, err := os.Pipe()
|
||||
require.NoError(t, err)
|
||||
|
||||
errReader, errWriter, err := os.Pipe()
|
||||
require.NoError(t, err)
|
||||
|
||||
previousOut, previousErr := os.Stdout, os.Stderr
|
||||
os.Stdout, os.Stderr = outWriter, errWriter
|
||||
|
||||
capturedOut := drain(outReader)
|
||||
capturedErr := drain(errReader)
|
||||
|
||||
fn()
|
||||
|
||||
os.Stdout, os.Stderr = previousOut, previousErr
|
||||
|
||||
require.NoError(t, outWriter.Close())
|
||||
require.NoError(t, errWriter.Close())
|
||||
|
||||
out, errOut := <-capturedOut, <-capturedErr
|
||||
|
||||
require.NoError(t, outReader.Close())
|
||||
require.NoError(t, errReader.Close())
|
||||
|
||||
return out, errOut
|
||||
}
|
||||
|
||||
// drain copies a reader to a string on a goroutine and delivers the
|
||||
// result once the writer end is closed.
|
||||
func drain(reader io.Reader) <-chan string {
|
||||
captured := make(chan string, 1)
|
||||
|
||||
go func() {
|
||||
var buf bytes.Buffer
|
||||
|
||||
_, _ = io.Copy(&buf, reader)
|
||||
captured <- buf.String()
|
||||
}()
|
||||
|
||||
return captured
|
||||
}
|
||||
@@ -46,8 +46,7 @@ work (e.g. after a crashed backup or to reclaim storage).`,
|
||||
LogOptions: log.Options{
|
||||
Verbose: rootFlags.Verbose,
|
||||
Debug: rootFlags.Debug,
|
||||
Quiet: rootFlags.Quiet,
|
||||
JSON: opts.JSON,
|
||||
Quiet: rootFlags.Quiet || opts.JSON,
|
||||
},
|
||||
Modules: []fx.Option{},
|
||||
Invokes: []fx.Option{
|
||||
|
||||
@@ -88,8 +88,7 @@ func newRemoteInfoCommand() *cobra.Command {
|
||||
LogOptions: log.Options{
|
||||
Verbose: rootFlags.Verbose,
|
||||
Debug: rootFlags.Debug,
|
||||
Quiet: rootFlags.Quiet,
|
||||
JSON: jsonOutput,
|
||||
Quiet: rootFlags.Quiet || jsonOutput,
|
||||
},
|
||||
Modules: []fx.Option{},
|
||||
Invokes: []fx.Option{
|
||||
|
||||
@@ -141,8 +141,7 @@ specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
|
||||
"orphaned blobs")
|
||||
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
||||
"With --prune: keep snapshots newer than this duration "+
|
||||
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
||||
"instead of only the latest")
|
||||
"(e.g. 4w, 30d, 6mo) instead of only the latest")
|
||||
|
||||
return cmd
|
||||
}
|
||||
@@ -205,8 +204,7 @@ restrict the operation to specific snapshot names.`,
|
||||
cmd.Flags().BoolVar(&opts.KeepLatest, "keep-latest", false,
|
||||
"Keep only the latest snapshot of each name")
|
||||
cmd.Flags().StringVar(&opts.OlderThan, "older-than", "",
|
||||
"Remove snapshots older than duration "+
|
||||
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months)")
|
||||
"Remove snapshots older than duration (e.g., 30d, 6m, 1y)")
|
||||
cmd.Flags().BoolVar(&opts.Force, "force", false, "Skip confirmation prompt")
|
||||
cmd.Flags().StringArrayVar(&opts.Names, "snapshot", nil,
|
||||
"Restrict to snapshots with these names (repeat for multiple)")
|
||||
@@ -239,8 +237,7 @@ func newSnapshotVerifyCommand() *cobra.Command {
|
||||
LogOptions: log.Options{
|
||||
Verbose: rootFlags.Verbose,
|
||||
Debug: rootFlags.Debug,
|
||||
Quiet: rootFlags.Quiet,
|
||||
JSON: opts.JSON,
|
||||
Quiet: rootFlags.Quiet || opts.JSON,
|
||||
},
|
||||
Modules: []fx.Option{},
|
||||
Invokes: []fx.Option{
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
package cli
|
||||
|
||||
import "time"
|
||||
|
||||
// SnapshotInfo represents snapshot information for listing
|
||||
//
|
||||
//nolint:tagliatelle // snake_case is the established output format
|
||||
type SnapshotInfo struct {
|
||||
ID string `json:"id"`
|
||||
Timestamp time.Time `json:"timestamp"`
|
||||
CompressedSize int64 `json:"compressed_size"`
|
||||
}
|
||||
+1
-15
@@ -14,18 +14,8 @@ var Module = fx.Module("log",
|
||||
)
|
||||
|
||||
// New creates a new logger configuration from provided options.
|
||||
//
|
||||
// JSON is intentionally not carried into Config: a command emitting a
|
||||
// JSON document on stdout must keep its stderr log level under
|
||||
// --verbose/--debug, so --json must not lower it (issue #112). JSON
|
||||
// silences the stdout UI in setupGlobals instead.
|
||||
func New(opts Options) Config {
|
||||
return Config{
|
||||
Verbose: opts.Verbose,
|
||||
Debug: opts.Debug,
|
||||
Cron: opts.Cron,
|
||||
Quiet: opts.Quiet,
|
||||
}
|
||||
return Config(opts)
|
||||
}
|
||||
|
||||
// Options are provided by the CLI.
|
||||
@@ -34,8 +24,4 @@ type Options struct {
|
||||
Debug bool
|
||||
Cron bool
|
||||
Quiet bool
|
||||
// JSON marks a command whose stdout carries a machine-readable
|
||||
// document. It silences the human UI on stdout (see setupGlobals),
|
||||
// but unlike Quiet it leaves the stderr log level alone.
|
||||
JSON bool
|
||||
}
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
// Package models defines shared value types describing files, chunks,
|
||||
// blobs, and snapshots as they move through the backup pipeline.
|
||||
package models
|
||||
|
||||
import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// FileInfo represents a file in the backup system
|
||||
type FileInfo struct {
|
||||
Path string
|
||||
MTime time.Time
|
||||
Size int64
|
||||
}
|
||||
|
||||
// ChunkInfo represents a content-addressed chunk
|
||||
type ChunkInfo struct {
|
||||
Hash string // SHA256 hash
|
||||
Size int64
|
||||
Offset int64 // Offset within source file
|
||||
}
|
||||
|
||||
// ChunkRef represents a reference to a chunk in a blob or file
|
||||
type ChunkRef struct {
|
||||
ChunkHash string
|
||||
Offset int64
|
||||
Length int64
|
||||
}
|
||||
|
||||
// BlobInfo represents an encrypted blob containing multiple chunks
|
||||
type BlobInfo struct {
|
||||
Hash string // SHA256 hash of the blob content (content-addressable)
|
||||
CreatedAt time.Time
|
||||
Size int64
|
||||
ChunkCount int
|
||||
}
|
||||
|
||||
// Snapshot represents a backup snapshot
|
||||
type Snapshot struct {
|
||||
ID string // ISO8601 timestamp
|
||||
Hostname string
|
||||
Version string
|
||||
CreatedAt time.Time
|
||||
FileCount int64
|
||||
ChunkCount int64
|
||||
BlobCount int64
|
||||
TotalSize int64
|
||||
MetadataSize int64
|
||||
}
|
||||
|
||||
// SnapshotMetadata contains the full metadata for a snapshot
|
||||
type SnapshotMetadata struct {
|
||||
Snapshot *Snapshot
|
||||
Files map[string]*FileInfo
|
||||
Chunks map[string]*ChunkInfo
|
||||
Blobs map[string]*BlobInfo
|
||||
FileChunks map[string][]*ChunkRef // path -> chunks
|
||||
BlobChunks map[string][]*ChunkRef // blob hash -> chunks
|
||||
}
|
||||
|
||||
// Chunk represents a data chunk for processing
|
||||
type Chunk struct {
|
||||
Data []byte
|
||||
Hash string
|
||||
Offset int64
|
||||
Length int64
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
package models_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/models"
|
||||
)
|
||||
|
||||
// TestModelsCompilation ensures all model types can be instantiated
|
||||
func TestModelsCompilation(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// This test primarily serves as a compilation test
|
||||
// to ensure all types are properly defined
|
||||
|
||||
// Test FileInfo
|
||||
fi := &models.FileInfo{
|
||||
Path: "/test/file.txt",
|
||||
MTime: time.Now(),
|
||||
Size: 1024,
|
||||
}
|
||||
if fi.Path != "/test/file.txt" {
|
||||
t.Errorf("FileInfo.Path not set correctly")
|
||||
}
|
||||
|
||||
// Test ChunkInfo
|
||||
ci := &models.ChunkInfo{
|
||||
Hash: "abc123",
|
||||
Size: 512,
|
||||
Offset: 0,
|
||||
}
|
||||
if ci.Hash != "abc123" {
|
||||
t.Errorf("ChunkInfo.Hash not set correctly")
|
||||
}
|
||||
|
||||
// Test BlobInfo
|
||||
bi := &models.BlobInfo{
|
||||
Hash: "blob123",
|
||||
CreatedAt: time.Now(),
|
||||
Size: 1024,
|
||||
ChunkCount: 2,
|
||||
}
|
||||
if bi.Hash != "blob123" {
|
||||
t.Errorf("BlobInfo.Hash not set correctly")
|
||||
}
|
||||
|
||||
// Test Snapshot
|
||||
s := &models.Snapshot{
|
||||
ID: "2024-01-01T00:00:00Z",
|
||||
Hostname: "test-host",
|
||||
Version: "1.0.0",
|
||||
CreatedAt: time.Now(),
|
||||
}
|
||||
if s.ID != "2024-01-01T00:00:00Z" {
|
||||
t.Errorf("Snapshot.ID not set correctly")
|
||||
}
|
||||
}
|
||||
+5
-13
@@ -219,7 +219,11 @@ func (c *Client) HeadObject(ctx context.Context, key string) (bool, error) {
|
||||
Key: aws.String(fullKey),
|
||||
})
|
||||
if err != nil {
|
||||
if IsNotFound(err) {
|
||||
var (
|
||||
notFound *s3types.NotFound
|
||||
noSuchKey *s3types.NoSuchKey
|
||||
)
|
||||
if errors.As(err, ¬Found) || errors.As(err, &noSuchKey) {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
@@ -229,18 +233,6 @@ func (c *Client) HeadObject(ctx context.Context, key string) (bool, error) {
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// IsNotFound reports whether err indicates that an object does not exist.
|
||||
// Head and Get requests surface a missing object as different SDK types,
|
||||
// so both are checked here.
|
||||
func IsNotFound(err error) bool {
|
||||
var (
|
||||
notFound *s3types.NotFound
|
||||
noSuchKey *s3types.NoSuchKey
|
||||
)
|
||||
|
||||
return errors.As(err, ¬Found) || errors.As(err, &noSuchKey)
|
||||
}
|
||||
|
||||
// ObjectInfo contains information about an S3 object.
|
||||
// It is used by ListObjectsStream to return object metadata
|
||||
// along with any errors encountered during listing.
|
||||
|
||||
@@ -22,9 +22,8 @@ const remoteKeyPrefix = "vaultik|"
|
||||
//
|
||||
// - the "metadata/<remote-key>/..." subdirectory on the storage
|
||||
// backend so a directory listing of the bucket / file:// dest
|
||||
// doesn't reveal hostnames or configured snapshot names. (The
|
||||
// backup time is not hidden: the manifest.json.zst inside that
|
||||
// directory carries a plaintext RFC3339 timestamp.)
|
||||
// doesn't reveal hostnames, configured snapshot names, or backup
|
||||
// timestamps;
|
||||
// - the `snapshot_id` field of the unencrypted manifest.json.zst
|
||||
// for the same reason;
|
||||
// - any code path that needs to translate a known local snapshot ID
|
||||
|
||||
@@ -44,6 +44,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -668,31 +669,14 @@ func (sm *SnapshotManager) collectCleanupStats(
|
||||
|
||||
// vacuumDatabase runs VACUUM on the database to remove deleted data and compact
|
||||
// This is critical for security - ensures no stale/deleted data pages are uploaded
|
||||
//
|
||||
// VACUUM runs through the modernc.org/sqlite driver, on a freshly opened
|
||||
// connection with no transaction in flight (VACUUM cannot run inside one).
|
||||
// The database opens in WAL mode, so VACUUM's rewrite lands in the WAL; the
|
||||
// checkpoint on Close flushes it into the main file, which is the file we
|
||||
// then compress and upload.
|
||||
func (sm *SnapshotManager) vacuumDatabase(ctx context.Context, dbPath string) error {
|
||||
log.Debug("Running VACUUM on database", "path", dbPath)
|
||||
//nolint:gosec // G204: fixed argv; dbPath is our own temp file path
|
||||
cmd := exec.CommandContext(ctx, "sqlite3", dbPath, "VACUUM;")
|
||||
|
||||
db, err := database.New(ctx, dbPath)
|
||||
output, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
return fmt.Errorf("opening database for VACUUM: %w", err)
|
||||
}
|
||||
|
||||
defer func() {
|
||||
cerr := db.Close()
|
||||
if cerr != nil {
|
||||
log.Debug("Failed to close database after VACUUM",
|
||||
"path", dbPath, "error", cerr)
|
||||
}
|
||||
}()
|
||||
|
||||
_, err = db.ExecWithLog(ctx, "VACUUM")
|
||||
if err != nil {
|
||||
return fmt.Errorf("running VACUUM: %w", err)
|
||||
return fmt.Errorf("running VACUUM: %w (output: %s)", err, string(output))
|
||||
}
|
||||
|
||||
return nil
|
||||
@@ -856,10 +840,8 @@ func (sm *SnapshotManager) generateBlobManifest(
|
||||
}
|
||||
|
||||
// Create manifest. SnapshotID in the unencrypted manifest is the
|
||||
// double-SHA256 remote key (see RemoteSnapshotKey), not the human ID,
|
||||
// so neither this field nor the directory name reveals the hostname or
|
||||
// snapshot name. Timestamp below is written in the clear, so the backup
|
||||
// time is observable to anyone who can read the manifest.
|
||||
// double-SHA256 remote key, not the human ID, so the public bytes
|
||||
// don't reveal hostname/snapshot-name/timestamp metadata.
|
||||
manifest := &Manifest{
|
||||
SnapshotID: RemoteSnapshotKey(snapshotID),
|
||||
Timestamp: time.Now().UTC().Format(time.RFC3339),
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
package snapshot
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"database/sql"
|
||||
"io"
|
||||
@@ -97,97 +96,6 @@ func verifyCleanedDB(
|
||||
}
|
||||
}
|
||||
|
||||
// TestVacuumDatabaseRemovesDeletedData proves the export path uploads a
|
||||
// compacted database: after rows carrying a recognizable marker are deleted
|
||||
// and vacuumDatabase runs, no page holding that marker survives in the file
|
||||
// on disk (the file compressFile later reads for upload).
|
||||
func TestVacuumDatabaseRemovesDeletedData(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
fs := afero.NewOsFs()
|
||||
|
||||
tempDir := t.TempDir()
|
||||
dbPath := filepath.Join(tempDir, "snapshot.db")
|
||||
|
||||
db, err := database.New(ctx, dbPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create database: %v", err)
|
||||
}
|
||||
|
||||
// A marker distinctive enough that its presence in the raw file can only
|
||||
// come from the rows inserted below.
|
||||
marker := []byte("VACUUM_PROBE_DEADBEEF_DELETED_ROW")
|
||||
payload := bytes.Repeat(marker, 128) // ~4 KiB per row
|
||||
|
||||
_, err = db.Conn().ExecContext(ctx,
|
||||
"CREATE TABLE vacuum_probe (id INTEGER PRIMARY KEY, payload BLOB)")
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create probe table: %v", err)
|
||||
}
|
||||
|
||||
for range 512 {
|
||||
_, err = db.Conn().ExecContext(ctx,
|
||||
"INSERT INTO vacuum_probe (payload) VALUES (?)", payload)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to insert probe row: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
_, err = db.Conn().ExecContext(ctx, "DELETE FROM vacuum_probe")
|
||||
if err != nil {
|
||||
t.Fatalf("failed to delete probe rows: %v", err)
|
||||
}
|
||||
|
||||
// Close so the deletes reach the main file, mirroring the state
|
||||
// prepareExportDB hands to vacuumDatabase.
|
||||
err = db.Close()
|
||||
if err != nil {
|
||||
t.Fatalf("failed to close database: %v", err)
|
||||
}
|
||||
|
||||
beforeInfo, err := fs.Stat(dbPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to stat database before vacuum: %v", err)
|
||||
}
|
||||
|
||||
beforeBytes, err := afero.ReadFile(fs, dbPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read database before vacuum: %v", err)
|
||||
}
|
||||
|
||||
if !bytes.Contains(beforeBytes, marker) {
|
||||
t.Fatalf("expected deleted-row data to linger before vacuum")
|
||||
}
|
||||
|
||||
sm := &SnapshotManager{fs: fs}
|
||||
|
||||
err = sm.vacuumDatabase(ctx, dbPath)
|
||||
if err != nil {
|
||||
t.Fatalf("vacuumDatabase failed: %v", err)
|
||||
}
|
||||
|
||||
afterBytes, err := afero.ReadFile(fs, dbPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read database after vacuum: %v", err)
|
||||
}
|
||||
|
||||
if bytes.Contains(afterBytes, marker) {
|
||||
t.Fatalf("deleted-row data survived vacuum in the uploaded file")
|
||||
}
|
||||
|
||||
afterInfo, err := fs.Stat(dbPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to stat database after vacuum: %v", err)
|
||||
}
|
||||
|
||||
if afterInfo.Size() >= beforeInfo.Size() {
|
||||
t.Fatalf("expected vacuum to shrink the file: before=%d after=%d",
|
||||
beforeInfo.Size(), afterInfo.Size())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanSnapshotDBEmptySnapshot(t *testing.T) {
|
||||
// Initialize logger
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
+54
-79
@@ -46,18 +46,31 @@ func (f *FileStorer) SetFilesystem(fs afero.Fs) {
|
||||
// storage base path.
|
||||
const storageDirPerm = 0o755
|
||||
|
||||
// tempSuffix marks a partially written object. writeAtomic streams into a
|
||||
// temp file carrying this suffix and only renames it onto the real key once
|
||||
// the whole object is on disk, so an interrupted write can never leave a
|
||||
// truncated object at the key a later run would Stat and trust as a complete
|
||||
// blob. List and ListStream skip these files, so a leftover from an
|
||||
// interrupted write is never listed or trusted as a blob; it is otherwise
|
||||
// harmless and is overwritten when the same key is written again.
|
||||
const tempSuffix = ".partial"
|
||||
|
||||
// Put stores data at the specified key.
|
||||
func (f *FileStorer) Put(_ context.Context, key string, data io.Reader) error {
|
||||
return f.writeAtomic(key, data, nil)
|
||||
path := f.fullPath(key)
|
||||
|
||||
// Create parent directories
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
err := f.fs.MkdirAll(dir, storageDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating directories: %w", err)
|
||||
}
|
||||
|
||||
file, err := f.fs.Create(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating file: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = file.Close() }()
|
||||
|
||||
_, err = io.Copy(file, data)
|
||||
if err != nil {
|
||||
return fmt.Errorf("writing file: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// PutWithProgress stores data with progress reporting.
|
||||
@@ -65,7 +78,35 @@ func (f *FileStorer) PutWithProgress(
|
||||
_ context.Context, key string, data io.Reader,
|
||||
_ int64, progress ProgressCallback,
|
||||
) error {
|
||||
return f.writeAtomic(key, data, progress)
|
||||
path := f.fullPath(key)
|
||||
|
||||
// Create parent directories
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
err := f.fs.MkdirAll(dir, storageDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating directories: %w", err)
|
||||
}
|
||||
|
||||
file, err := f.fs.Create(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating file: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = file.Close() }()
|
||||
|
||||
// Wrap with progress tracking
|
||||
pw := &progressWriter{
|
||||
writer: file,
|
||||
callback: progress,
|
||||
}
|
||||
|
||||
_, err = io.Copy(pw, data)
|
||||
if err != nil {
|
||||
return fmt.Errorf("writing file: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Get retrieves data from the specified key.
|
||||
@@ -147,7 +188,7 @@ func (f *FileStorer) List(ctx context.Context, prefix string) ([]string, error)
|
||||
default:
|
||||
}
|
||||
|
||||
if !info.IsDir() && !strings.HasSuffix(info.Name(), tempSuffix) {
|
||||
if !info.IsDir() {
|
||||
// Convert back to key (relative path from basePath)
|
||||
relPath, err := filepath.Rel(f.basePath, path)
|
||||
if err != nil {
|
||||
@@ -204,7 +245,7 @@ func (f *FileStorer) ListStream(ctx context.Context, prefix string) <-chan Objec
|
||||
return nil //nolint:nilerr // continue walking despite errors
|
||||
}
|
||||
|
||||
if !info.IsDir() && !strings.HasSuffix(info.Name(), tempSuffix) {
|
||||
if !info.IsDir() {
|
||||
relPath, err := filepath.Rel(f.basePath, path)
|
||||
if err != nil {
|
||||
ch <- ObjectInfo{Err: fmt.Errorf("computing relative path: %w", err)}
|
||||
@@ -234,72 +275,6 @@ func (f *FileStorer) Info() Info {
|
||||
}
|
||||
}
|
||||
|
||||
// writeAtomic streams data into a temp file in the destination directory,
|
||||
// fsyncs it, and renames it onto the final key. The key therefore appears
|
||||
// only once the whole object has been durably written; a failure part-way
|
||||
// leaves a temp file (removed here on the failing path) rather than a
|
||||
// truncated object at the key.
|
||||
func (f *FileStorer) writeAtomic(
|
||||
key string, data io.Reader, progress ProgressCallback,
|
||||
) error {
|
||||
path := f.fullPath(key)
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
err := f.fs.MkdirAll(dir, storageDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating directories: %w", err)
|
||||
}
|
||||
|
||||
tmp, err := afero.TempFile(f.fs, dir, filepath.Base(path)+"-*"+tempSuffix)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating temp file: %w", err)
|
||||
}
|
||||
|
||||
tmpPath := tmp.Name()
|
||||
|
||||
// Remove the temp file unless the rename below claims it. On the success
|
||||
// path renamed is true, so the deferred Close and Remove are harmless
|
||||
// no-ops on a name that no longer exists.
|
||||
renamed := false
|
||||
|
||||
defer func() {
|
||||
_ = tmp.Close()
|
||||
|
||||
if !renamed {
|
||||
_ = f.fs.Remove(tmpPath)
|
||||
}
|
||||
}()
|
||||
|
||||
var w io.Writer = tmp
|
||||
if progress != nil {
|
||||
w = &progressWriter{writer: tmp, callback: progress}
|
||||
}
|
||||
|
||||
_, err = io.Copy(w, data)
|
||||
if err != nil {
|
||||
return fmt.Errorf("writing file: %w", err)
|
||||
}
|
||||
|
||||
err = tmp.Sync()
|
||||
if err != nil {
|
||||
return fmt.Errorf("syncing temp file: %w", err)
|
||||
}
|
||||
|
||||
err = tmp.Close()
|
||||
if err != nil {
|
||||
return fmt.Errorf("closing temp file: %w", err)
|
||||
}
|
||||
|
||||
err = f.fs.Rename(tmpPath, path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("renaming temp file: %w", err)
|
||||
}
|
||||
|
||||
renamed = true
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// fullPath returns the full filesystem path for a key.
|
||||
func (f *FileStorer) fullPath(key string) string {
|
||||
return filepath.Join(f.basePath, key)
|
||||
|
||||
@@ -1,119 +0,0 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
)
|
||||
|
||||
// errStreamInterrupted stands in for an upload cut off mid-stream.
|
||||
var errStreamInterrupted = errors.New("connection reset mid-upload")
|
||||
|
||||
// failingReader yields its data once, then fails.
|
||||
type failingReader struct {
|
||||
data []byte
|
||||
done bool
|
||||
}
|
||||
|
||||
func (r *failingReader) Read(p []byte) (int, error) {
|
||||
if r.done {
|
||||
return 0, errStreamInterrupted
|
||||
}
|
||||
|
||||
n := copy(p, r.data)
|
||||
r.done = true
|
||||
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// TestFileStorer_InterruptedWriteLeavesNoTrustedObject checks that a write
|
||||
// cut off mid-stream leaves nothing at the destination key, so a later run
|
||||
// cannot Stat a truncated object and trust it as a complete blob.
|
||||
func TestFileStorer_InterruptedWriteLeavesNoTrustedObject(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f, err := storage.NewFileStorer(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStorer: %v", err)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
key := "blobs/aa/bb/aabbccddeeff"
|
||||
|
||||
err = f.PutWithProgress(ctx, key, &failingReader{data: []byte("partial")}, 4096, nil)
|
||||
if err == nil {
|
||||
t.Fatal("expected the interrupted write to fail, got nil")
|
||||
}
|
||||
|
||||
_, err = f.Stat(ctx, key)
|
||||
if !errors.Is(err, storage.ErrNotFound) {
|
||||
t.Fatalf("expected key absent after interrupted write, got Stat err %v", err)
|
||||
}
|
||||
|
||||
keys, err := f.List(ctx, "blobs/")
|
||||
if err != nil {
|
||||
t.Fatalf("List: %v", err)
|
||||
}
|
||||
|
||||
if len(keys) != 0 {
|
||||
t.Fatalf("expected no keys listed after interrupted write, got %v", keys)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFileStorer_ListSkipsPartialFiles checks that a leftover temp file (the
|
||||
// storage layer names them with a ".partial" suffix) is never surfaced as a
|
||||
// key by List or ListStream.
|
||||
func TestFileStorer_ListSkipsPartialFiles(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
base := t.TempDir()
|
||||
|
||||
f, err := storage.NewFileStorer(base)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStorer: %v", err)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
realKey := "blobs/aa/bb/aabbccddeeff"
|
||||
|
||||
err = f.Put(ctx, realKey, strings.NewReader("blob-bytes"))
|
||||
if err != nil {
|
||||
t.Fatalf("Put: %v", err)
|
||||
}
|
||||
|
||||
// A stray temp file, as an interrupted write would leave behind.
|
||||
leftover := filepath.Join(base, "blobs/aa/bb/aabbccddeeff-123456.partial")
|
||||
|
||||
err = os.WriteFile(leftover, []byte("half"), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("writing leftover temp file: %v", err)
|
||||
}
|
||||
|
||||
keys, err := f.List(ctx, "blobs/")
|
||||
if err != nil {
|
||||
t.Fatalf("List: %v", err)
|
||||
}
|
||||
|
||||
if len(keys) != 1 || keys[0] != realKey {
|
||||
t.Fatalf("List should return only the real key, got %v", keys)
|
||||
}
|
||||
|
||||
var streamed []string
|
||||
|
||||
for obj := range f.ListStream(ctx, "blobs/") {
|
||||
if obj.Err != nil {
|
||||
t.Fatalf("ListStream: %v", obj.Err)
|
||||
}
|
||||
|
||||
streamed = append(streamed, obj.Key)
|
||||
}
|
||||
|
||||
if len(streamed) != 1 || streamed[0] != realKey {
|
||||
t.Fatalf("ListStream should return only the real key, got %v", streamed)
|
||||
}
|
||||
}
|
||||
+1
-16
@@ -38,29 +38,14 @@ func (s *S3Storer) PutWithProgress(
|
||||
}
|
||||
|
||||
// Get retrieves data from the specified key.
|
||||
// Returns ErrNotFound if the object does not exist.
|
||||
func (s *S3Storer) Get(ctx context.Context, key string) (io.ReadCloser, error) {
|
||||
rc, err := s.client.GetObject(ctx, key)
|
||||
if err != nil {
|
||||
if s3.IsNotFound(err) {
|
||||
return nil, fmt.Errorf("get %q: %w", key, ErrNotFound)
|
||||
}
|
||||
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return rc, nil
|
||||
return s.client.GetObject(ctx, key)
|
||||
}
|
||||
|
||||
// Stat returns metadata about an object without retrieving its contents.
|
||||
// Returns ErrNotFound if the object does not exist.
|
||||
func (s *S3Storer) Stat(ctx context.Context, key string) (*ObjectInfo, error) {
|
||||
info, err := s.client.StatObject(ctx, key)
|
||||
if err != nil {
|
||||
if s3.IsNotFound(err) {
|
||||
return nil, fmt.Errorf("stat %q: %w", key, ErrNotFound)
|
||||
}
|
||||
|
||||
return nil, err
|
||||
}
|
||||
|
||||
|
||||
@@ -1,59 +0,0 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/johannesboyne/gofakes3"
|
||||
"github.com/johannesboyne/gofakes3/backend/s3mem"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/s3"
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
)
|
||||
|
||||
// TestS3StorerMissingKeyMapsToErrNotFound verifies that the s3 backend reports
|
||||
// a missing object as storage.ErrNotFound, matching the file and rclone
|
||||
// backends and the Storer contract. Without the mapping, Get and Stat leak the
|
||||
// raw SDK error and errors.Is(err, storage.ErrNotFound) is false.
|
||||
//
|
||||
//nolint:paralleltest // shares an in-process S3 server via t.Cleanup
|
||||
func TestS3StorerMissingKeyMapsToErrNotFound(t *testing.T) {
|
||||
const bucket = "test-bucket"
|
||||
|
||||
backend := s3mem.New()
|
||||
|
||||
err := backend.CreateBucket(bucket)
|
||||
if err != nil {
|
||||
t.Fatalf("create bucket: %v", err)
|
||||
}
|
||||
|
||||
srv := httptest.NewServer(gofakes3.New(backend).Server())
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
client, err := s3.NewClient(ctx, s3.Config{
|
||||
Endpoint: srv.URL,
|
||||
Bucket: bucket,
|
||||
AccessKeyID: "test",
|
||||
SecretAccessKey: "test",
|
||||
Region: "us-east-1",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("new client: %v", err)
|
||||
}
|
||||
|
||||
storer := storage.NewS3Storer(client)
|
||||
|
||||
_, err = storer.Get(ctx, "does-not-exist")
|
||||
if !errors.Is(err, storage.ErrNotFound) {
|
||||
t.Errorf("Get on missing key: got %v, want ErrNotFound", err)
|
||||
}
|
||||
|
||||
_, err = storer.Stat(ctx, "does-not-exist")
|
||||
if !errors.Is(err, storage.ErrNotFound) {
|
||||
t.Errorf("Stat on missing key: got %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
@@ -1,108 +0,0 @@
|
||||
package vaultik_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/spf13/afero"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
"sneak.berlin/go/vaultik/internal/ui"
|
||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||
)
|
||||
|
||||
// TestDeepVerifyAcceptsHealthyAndRejectsCorruptBlob backs up a real
|
||||
// snapshot with the on-disk storage backend, runs deep verification on
|
||||
// it, then flips a byte inside one stored blob and runs deep
|
||||
// verification again. A healthy snapshot must pass; a corrupted blob
|
||||
// must fail. The healthy case is the regression guard: deep
|
||||
// verification used to hash the encrypted blob bytes and compare them
|
||||
// to the blob's ID (the double SHA256 of the plaintext), so it reported
|
||||
// every healthy blob as corrupt.
|
||||
func TestDeepVerifyAcceptsHealthyAndRejectsCorruptBlob(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
|
||||
dataDir := filepath.Join(tempDir, "source")
|
||||
storeDir := filepath.Join(tempDir, "remote")
|
||||
dbPath := filepath.Join(tempDir, "index.sqlite")
|
||||
|
||||
chunkSize := int64(64 * 1024)
|
||||
maxBlobSize := int64(512 * 1024)
|
||||
|
||||
// One file large enough to span several chunks within a single blob.
|
||||
require.NoError(t, fs.MkdirAll(dataDir, 0o755))
|
||||
require.NoError(t, afero.WriteFile(fs,
|
||||
filepath.Join(dataDir, "data.bin"),
|
||||
bytesPattern("deep-", int(chunkSize*3)), 0o644))
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// runFileStorageBackup writes a real snapshot to storeDir and closes
|
||||
// the source index, so verification runs from remote bytes only.
|
||||
cfg, storer, snapshotID := runFileStorageBackup(
|
||||
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
|
||||
|
||||
newVerifier := func() *vaultik.Vaultik {
|
||||
v := &vaultik.Vaultik{
|
||||
Config: cfg,
|
||||
Storage: storer,
|
||||
Fs: fs,
|
||||
Stdout: io.Discard,
|
||||
Stderr: io.Discard,
|
||||
UI: ui.NewWithColor(io.Discard, false),
|
||||
}
|
||||
v.SetContext(ctx)
|
||||
|
||||
return v
|
||||
}
|
||||
|
||||
require.NoError(t,
|
||||
newVerifier().RunDeepVerify(snapshotID, &vaultik.VerifyOptions{Deep: true}),
|
||||
"deep verify should pass on a healthy snapshot")
|
||||
|
||||
// Flip a byte inside one blob without changing its length, so the
|
||||
// blob-existence and size checks still pass and verification reaches
|
||||
// the blob-content stage.
|
||||
corruptOneBlob(t, fs, filepath.Join(storeDir, "blobs"))
|
||||
|
||||
require.Error(t,
|
||||
newVerifier().RunDeepVerify(snapshotID, &vaultik.VerifyOptions{Deep: true}),
|
||||
"deep verify should fail on a corrupted blob")
|
||||
}
|
||||
|
||||
// corruptOneBlob flips a middle byte of the first blob file found under
|
||||
// blobsDir, leaving the file length unchanged.
|
||||
func corruptOneBlob(t *testing.T, fs afero.Fs, blobsDir string) {
|
||||
t.Helper()
|
||||
|
||||
var blobPath string
|
||||
|
||||
err := afero.Walk(fs, blobsDir,
|
||||
func(path string, info os.FileInfo, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if blobPath == "" && !info.IsDir() {
|
||||
blobPath = path
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, blobPath, "expected at least one blob on disk")
|
||||
|
||||
data, err := afero.ReadFile(fs, blobPath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, data)
|
||||
|
||||
data[len(data)/2] ^= 0xff
|
||||
require.NoError(t, afero.WriteFile(fs, blobPath, data, 0o644))
|
||||
}
|
||||
@@ -35,7 +35,6 @@ var (
|
||||
"invalid snapshot ID format: expected hostname_snapshotname_timestamp")
|
||||
errInvalidDuration = errors.New("invalid duration")
|
||||
errUnknownTimeUnit = errors.New("unknown time unit")
|
||||
errNegativeDuration = errors.New("negative durations are not supported")
|
||||
)
|
||||
|
||||
// Time-unit lengths used by parseDuration.
|
||||
@@ -139,13 +138,8 @@ func parseSnapshotName(snapshotID string) string {
|
||||
|
||||
// parseDuration parses a duration string with support for human-friendly units:
|
||||
// d/day/days, w/week/weeks, mo/month/months, y/year/years, plus standard Go
|
||||
// duration units. Following Go, m is minutes and mo is months. A bare number,
|
||||
// an unknown unit, and a negative value are all rejected.
|
||||
// duration units (h, m, s).
|
||||
func parseDuration(s string) (time.Duration, error) {
|
||||
if strings.HasPrefix(strings.TrimSpace(s), "-") {
|
||||
return 0, errNegativeDuration
|
||||
}
|
||||
|
||||
d, err := time.ParseDuration(s)
|
||||
if err == nil {
|
||||
return d, nil
|
||||
|
||||
@@ -51,32 +51,13 @@ func TestParseDuration(t *testing.T) {
|
||||
want time.Duration
|
||||
err bool
|
||||
}{
|
||||
// Go units, including the m-is-minutes / mo-is-months distinction
|
||||
// that this parser exists to keep straight.
|
||||
{"10ns", 10 * time.Nanosecond, false},
|
||||
{"10us", 10 * time.Microsecond, false},
|
||||
{"500ms", 500 * time.Millisecond, false},
|
||||
{"30s", 30 * time.Second, false},
|
||||
{"6m", 6 * time.Minute, false},
|
||||
{"1h", time.Hour, false},
|
||||
// Extended calendar units.
|
||||
{"30d", 30 * 24 * time.Hour, false},
|
||||
{"3days", 3 * 24 * time.Hour, false},
|
||||
{"4w", 4 * 7 * 24 * time.Hour, false},
|
||||
{"2weeks", 2 * 7 * 24 * time.Hour, false},
|
||||
{"6mo", 180 * 24 * time.Hour, false},
|
||||
{"1month", 30 * 24 * time.Hour, false},
|
||||
{"6mo", 6 * 30 * 24 * time.Hour, false},
|
||||
{"1y", 365 * 24 * time.Hour, false},
|
||||
{"2years", 2 * 365 * 24 * time.Hour, false},
|
||||
// Combined units.
|
||||
{"2w3d", 2*7*24*time.Hour + 3*24*time.Hour, false},
|
||||
{"1y6mo", 365*24*time.Hour + 180*24*time.Hour, false},
|
||||
// Rejected inputs.
|
||||
{"6", 0, true}, // bare number, no unit
|
||||
{"5x", 0, true}, // unknown unit
|
||||
{"-5d", 0, true}, // negative, extended unit
|
||||
{"-5h", 0, true}, // negative, Go unit
|
||||
{"", 0, true}, // empty
|
||||
{"1h", time.Hour, false},
|
||||
{"30s", 30 * time.Second, false},
|
||||
{"garbage", 0, true},
|
||||
}
|
||||
|
||||
|
||||
@@ -1,79 +0,0 @@
|
||||
package vaultik //nolint:testpackage // exercises unexported count helpers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/database"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
)
|
||||
|
||||
// TestTableCountForReportSurfacesReadFailure is the regression guard for
|
||||
// the discarded-error bug: getTableCount for a table its query cannot
|
||||
// resolve must not silently become 0. A count that could not be read is
|
||||
// reported as unknown, which a reader can tell apart from an empty table.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestTableCountForReportSurfacesReadFailure(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
db, err := database.New(ctx, ":memory:")
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
v := &Vaultik{DB: db}
|
||||
v.SetContext(ctx)
|
||||
|
||||
// A table present in the schema reads as a real count.
|
||||
blobs := v.tableCountForReport("blobs")
|
||||
require.NotNil(t, blobs, "an existing table must read as a real count")
|
||||
assert.Equal(t, int64(0), *blobs)
|
||||
|
||||
// A syntactically valid name the sanitizer accepts but whose table
|
||||
// the query cannot resolve is the exact shape #96 describes: a
|
||||
// would-be loud failure that used to be discarded into a 0.
|
||||
_, err = v.getTableCount("snapshots_missing")
|
||||
require.Error(t, err, "a query against a nonexistent table must fail")
|
||||
|
||||
missing := v.tableCountForReport("snapshots_missing")
|
||||
assert.Nil(t, missing, "a failed read is unknown, not a count")
|
||||
|
||||
// The rendered count for a failed read must say unknown, never 0.
|
||||
assert.Equal(t, countUnknown, countText(missing))
|
||||
assert.NotEqual(t, "0", countText(missing))
|
||||
}
|
||||
|
||||
// TestCountTextDistinguishesEmptyFromUnknown pins the distinction the
|
||||
// output has to preserve: 0 means the table was empty, "unknown" means
|
||||
// the count could not be read.
|
||||
func TestCountTextDistinguishesEmptyFromUnknown(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
zero := int64(0)
|
||||
seven := int64(7)
|
||||
|
||||
assert.Equal(t, "0", countText(&zero))
|
||||
assert.Equal(t, "7", countText(&seven))
|
||||
assert.Equal(t, countUnknown, countText(nil))
|
||||
}
|
||||
|
||||
// TestCountDiffUnknownWhenEitherSideUnknown checks that a delta computed
|
||||
// from an unreadable count is itself unknown rather than a plausible
|
||||
// number.
|
||||
func TestCountDiffUnknownWhenEitherSideUnknown(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
before := int64(10)
|
||||
after := int64(3)
|
||||
|
||||
require.NotNil(t, countDiff(&before, &after))
|
||||
assert.Equal(t, int64(7), *countDiff(&before, &after))
|
||||
|
||||
assert.Nil(t, countDiff(nil, &after), "unknown before yields unknown delta")
|
||||
assert.Nil(t, countDiff(&before, nil), "unknown after yields unknown delta")
|
||||
assert.Nil(t, countDiff(nil, nil))
|
||||
}
|
||||
@@ -8,7 +8,6 @@ import (
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -1541,17 +1540,12 @@ func (v *Vaultik) outputRemoveJSON(result *RemoveResult) error {
|
||||
return encoder.Encode(result)
|
||||
}
|
||||
|
||||
// PruneResult contains statistics about the prune operation.
|
||||
// SnapshotsDeleted counts snapshots actually deleted. FilesDeleted,
|
||||
// ChunksDeleted, and BlobsDeleted are derived from before/after row
|
||||
// counts of the local index; each is nil when a count could not be read,
|
||||
// so an unreadable count is reported as unknown rather than silently
|
||||
// as 0.
|
||||
// PruneResult contains statistics about the prune operation
|
||||
type PruneResult struct {
|
||||
SnapshotsDeleted int64
|
||||
FilesDeleted *int64
|
||||
ChunksDeleted *int64
|
||||
BlobsDeleted *int64
|
||||
FilesDeleted int64
|
||||
ChunksDeleted int64
|
||||
BlobsDeleted int64
|
||||
}
|
||||
|
||||
// PruneDatabase removes incomplete snapshots and orphaned files, chunks,
|
||||
@@ -1566,7 +1560,7 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
|
||||
result := &PruneResult{}
|
||||
|
||||
// Snapshot counts before deletion of incompletes.
|
||||
snapshotCountBefore := v.tableCountForReport("snapshots")
|
||||
snapshotCountBefore, _ := v.getTableCount("snapshots")
|
||||
|
||||
// First, delete any incomplete snapshots
|
||||
incompleteSnapshots, err := v.Repositories.Snapshots.GetIncompleteSnapshots(v.ctx)
|
||||
@@ -1581,9 +1575,9 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
|
||||
}
|
||||
|
||||
// Get counts before cleanup for reporting
|
||||
fileCountBefore := v.tableCountForReport("files")
|
||||
chunkCountBefore := v.tableCountForReport("chunks")
|
||||
blobCountBefore := v.tableCountForReport("blobs")
|
||||
fileCountBefore, _ := v.getTableCount("files")
|
||||
chunkCountBefore, _ := v.getTableCount("chunks")
|
||||
blobCountBefore, _ := v.getTableCount("blobs")
|
||||
|
||||
// Run the cleanup
|
||||
err = v.SnapshotManager.CleanupOrphanedData(v.ctx)
|
||||
@@ -1592,83 +1586,36 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
|
||||
}
|
||||
|
||||
// Get counts after cleanup
|
||||
fileCountAfter := v.tableCountForReport("files")
|
||||
chunkCountAfter := v.tableCountForReport("chunks")
|
||||
blobCountAfter := v.tableCountForReport("blobs")
|
||||
fileCountAfter, _ := v.getTableCount("files")
|
||||
chunkCountAfter, _ := v.getTableCount("chunks")
|
||||
blobCountAfter, _ := v.getTableCount("blobs")
|
||||
|
||||
result.FilesDeleted = countDiff(fileCountBefore, fileCountAfter)
|
||||
result.ChunksDeleted = countDiff(chunkCountBefore, chunkCountAfter)
|
||||
result.BlobsDeleted = countDiff(blobCountBefore, blobCountAfter)
|
||||
result.FilesDeleted = fileCountBefore - fileCountAfter
|
||||
result.ChunksDeleted = chunkCountBefore - chunkCountAfter
|
||||
result.BlobsDeleted = blobCountBefore - blobCountAfter
|
||||
|
||||
log.Info("Local database prune complete",
|
||||
"incomplete_snapshots", result.SnapshotsDeleted,
|
||||
"orphaned_files", countText(result.FilesDeleted),
|
||||
"orphaned_chunks", countText(result.ChunksDeleted),
|
||||
"orphaned_blobs", countText(result.BlobsDeleted),
|
||||
"orphaned_files", result.FilesDeleted,
|
||||
"orphaned_chunks", result.ChunksDeleted,
|
||||
"orphaned_blobs", result.BlobsDeleted,
|
||||
)
|
||||
|
||||
// Snapshots remaining after removing the incomplete ones; unknown if
|
||||
// the pre-prune snapshot count could not be read.
|
||||
snapshotsRemain := countDiff(snapshotCountBefore, &result.SnapshotsDeleted)
|
||||
snapshotCountAfter := snapshotCountBefore - result.SnapshotsDeleted
|
||||
|
||||
v.UI.Completef("Pruned local index database.")
|
||||
v.UI.Detailf("Incomplete snapshots: %s removed (%s remain).",
|
||||
countText(&result.SnapshotsDeleted), countText(snapshotsRemain))
|
||||
v.UI.Detailf("Orphaned files: %s removed (%s remain).",
|
||||
countText(result.FilesDeleted), countText(fileCountAfter))
|
||||
v.UI.Detailf("Orphaned chunks: %s removed (%s remain).",
|
||||
countText(result.ChunksDeleted), countText(chunkCountAfter))
|
||||
v.UI.Detailf("Orphaned blobs: %s removed (%s remain).",
|
||||
countText(result.BlobsDeleted), countText(blobCountAfter))
|
||||
v.UI.Detailf("Incomplete snapshots: %d removed (%d remain).",
|
||||
result.SnapshotsDeleted, snapshotCountAfter)
|
||||
v.UI.Detailf("Orphaned files: %d removed (%d remain).",
|
||||
result.FilesDeleted, fileCountAfter)
|
||||
v.UI.Detailf("Orphaned chunks: %d removed (%d remain).",
|
||||
result.ChunksDeleted, chunkCountAfter)
|
||||
v.UI.Detailf("Orphaned blobs: %d removed (%d remain).",
|
||||
result.BlobsDeleted, blobCountAfter)
|
||||
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// countUnknown is what a count reads as when its query could not be run,
|
||||
// distinct from "0", which means the table really was empty.
|
||||
const countUnknown = "unknown"
|
||||
|
||||
// tableCountForReport returns the row count of a table for the prune
|
||||
// summary, or nil if the count could not be read. A read failure is
|
||||
// logged at warn — visible even under --json, which routes warnings to
|
||||
// stderr — and then rendered as unknown rather than silently becoming 0,
|
||||
// so a broken query is a visible failure instead of a plausible wrong
|
||||
// number.
|
||||
func (v *Vaultik) tableCountForReport(tableName string) *int64 {
|
||||
count, err := v.getTableCount(tableName)
|
||||
if err != nil {
|
||||
log.Warn("could not read table row count for prune summary",
|
||||
"table", tableName, "error", err)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
return &count
|
||||
}
|
||||
|
||||
// countDiff returns before-after, or nil if either count is unknown so
|
||||
// that an unreadable count does not collapse into a plausible delta.
|
||||
func countDiff(before, after *int64) *int64 {
|
||||
if before == nil || after == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
diff := *before - *after
|
||||
|
||||
return &diff
|
||||
}
|
||||
|
||||
// countText renders a count that may be unknown: nil (the read failed)
|
||||
// becomes "unknown", never "0", so a reader can tell an empty table from
|
||||
// one that could not be queried.
|
||||
func countText(count *int64) string {
|
||||
if count == nil {
|
||||
return countUnknown
|
||||
}
|
||||
|
||||
return strconv.FormatInt(*count, 10)
|
||||
}
|
||||
|
||||
// validTableNameRe matches table names containing only lowercase
|
||||
// alphanumeric characters and underscores.
|
||||
var validTableNameRe = regexp.MustCompile(`^[a-z0-9_]+$`)
|
||||
|
||||
+14
-19
@@ -344,8 +344,12 @@ func (v *Vaultik) verifyBlob(blobInfo snapshot.BlobInfo, db *sql.DB) error {
|
||||
return fmt.Errorf("failed to get decryptor: %w", err)
|
||||
}
|
||||
|
||||
// Decrypt blob
|
||||
decryptedReader, err := decryptor.DecryptStream(reader)
|
||||
// Hash the encrypted blob data as it streams through to decryption
|
||||
blobHasher := sha256.New()
|
||||
teeReader := io.TeeReader(reader, blobHasher)
|
||||
|
||||
// Decrypt blob (reading through teeReader to hash encrypted data)
|
||||
decryptedReader, err := decryptor.DecryptStream(teeReader)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to decrypt: %w", err)
|
||||
}
|
||||
@@ -357,19 +361,12 @@ func (v *Vaultik) verifyBlob(blobInfo snapshot.BlobInfo, db *sql.DB) error {
|
||||
}
|
||||
defer decompressor.Close()
|
||||
|
||||
// A blob's hash — its remote name — is the double SHA256 of its
|
||||
// decompressed plaintext (see blobgen.Writer.Sum256), not of the
|
||||
// encrypted bytes. Hash the plaintext as chunk verification streams
|
||||
// it, then compare on completion.
|
||||
plaintextHasher := sha256.New()
|
||||
hashedStream := io.TeeReader(decompressor, plaintextHasher)
|
||||
|
||||
chunkCount, err := v.verifyBlobChunks(db, blobInfo.Hash, hashedStream)
|
||||
chunkCount, err := v.verifyBlobChunks(db, blobInfo.Hash, decompressor)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = v.verifyBlobFinalIntegrity(hashedStream, plaintextHasher, blobInfo.Hash)
|
||||
err = v.verifyBlobFinalIntegrity(decompressor, blobHasher, blobInfo.Hash)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -473,13 +470,14 @@ func (v *Vaultik) verifyBlobChunks(
|
||||
}
|
||||
|
||||
// verifyBlobFinalIntegrity checks that no trailing data exists in the
|
||||
// decompressed stream and that the blob hash matches the expected value.
|
||||
// decompressed stream and that the encrypted blob hash matches the
|
||||
// expected value.
|
||||
func (v *Vaultik) verifyBlobFinalIntegrity(
|
||||
plaintext io.Reader, plaintextHasher hash.Hash, expectedHash string,
|
||||
decompressor io.Reader, blobHasher hash.Hash, expectedHash string,
|
||||
) error {
|
||||
// Verify no remaining data in blob - if the chunk list is accurate,
|
||||
// the blob should be fully consumed.
|
||||
remaining, err := io.Copy(io.Discard, plaintext)
|
||||
remaining, err := io.Copy(io.Discard, decompressor)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to check for remaining blob data: %w", err)
|
||||
}
|
||||
@@ -488,11 +486,8 @@ func (v *Vaultik) verifyBlobFinalIntegrity(
|
||||
return fmt.Errorf("%w: %d bytes", errTrailingBlobData, remaining)
|
||||
}
|
||||
|
||||
// The blob hash is the double SHA256 of its plaintext content.
|
||||
firstHash := plaintextHasher.Sum(nil)
|
||||
secondHash := sha256.Sum256(firstHash)
|
||||
calculatedBlobHash := hex.EncodeToString(secondHash[:])
|
||||
|
||||
// Verify blob hash matches the encrypted data we downloaded
|
||||
calculatedBlobHash := hex.EncodeToString(blobHasher.Sum(nil))
|
||||
if calculatedBlobHash != expectedHash {
|
||||
return fmt.Errorf("%w: calculated %s, expected %s",
|
||||
errBlobHashMismatch, calculatedBlobHash, expectedHash)
|
||||
|
||||
@@ -114,6 +114,9 @@ main() {
|
||||
# from CI. Nothing on the host is ever used as a linter, at any
|
||||
# version, so installing one here would buy nothing.
|
||||
|
||||
# sqlite3 CLI: the test suite shells out to it (VACUUM).
|
||||
if missing sqlite3; then pkg_install sqlite sqlite3 sqlite sqlite; fi
|
||||
|
||||
# goreleaser, at the version pinned by script/install-goreleaser and
|
||||
# verified against a hardcoded sha256. Package managers are not used
|
||||
# for it: they ship whatever version they happen to carry, and the
|
||||
|
||||
+2
-3
@@ -27,9 +27,8 @@ set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
|
||||
# Go 1.26.1, 2026-09-21. Checksums are the sha256 values go.dev publishes
|
||||
# for each archive at https://go.dev/dl/ (also in its ?mode=json
|
||||
# manifest).
|
||||
# Go 1.26.1. Checksums are the sha256 values go.dev publishes for each
|
||||
# archive at https://go.dev/dl/ (also in its ?mode=json manifest).
|
||||
GO_VERSION="1.26.1"
|
||||
SHA256_LINUX_AMD64="031f088e5d955bab8657ede27ad4e3bc5b7c1ba281f05f245bcc304f327c987a"
|
||||
SHA256_LINUX_ARM64="a290581cfe4fe28ddd737dde3095f3dbeb7f2e4065cab4eae44dfc53b760c2f7"
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
# Vaultik test configuration
|
||||
hostname: test-host
|
||||
index_path: /tmp/vaultik-test/index.db
|
||||
source_dirs:
|
||||
- /tmp/vaultik-test/source
|
||||
|
||||
# S3 configuration
|
||||
s3:
|
||||
endpoint: http://localhost:19000 # gofakes3 test endpoint
|
||||
bucket: test-bucket
|
||||
prefix: test-
|
||||
access_key_id: test-key
|
||||
secret_access_key: test-secret
|
||||
region: us-east-1
|
||||
|
||||
# Chunking configuration
|
||||
chunk_size: 65536 # 64KB average chunk size
|
||||
min_chunk_size: 32768 # 32KB minimum
|
||||
max_chunk_size: 131072 # 128KB maximum
|
||||
blob_size: 1048576 # 1MB blobs for testing
|
||||
|
||||
# Compression
|
||||
compression_level: 3
|
||||
|
||||
# Encryption
|
||||
# age_recipients:
|
||||
# - age1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3mw88h
|
||||
@@ -0,0 +1,24 @@
|
||||
age_recipients:
|
||||
- age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj # sneak's long term age key
|
||||
- age1ezrjmfpwsc95svdg0y54mums3zevgzu0x0ecq2f7tp8a05gl0sjq9q9wjg # insecure integration test key
|
||||
source_dirs:
|
||||
- /tmp/vaultik-test-source
|
||||
exclude:
|
||||
- '*.log'
|
||||
- '*.tmp'
|
||||
- '.git'
|
||||
- 'node_modules'
|
||||
s3:
|
||||
endpoint: http://ber1app1.local:3900/
|
||||
bucket: vaultik-integration-test
|
||||
prefix: test-host/
|
||||
access_key_id: GKbc8e6d35fdf50847f155aca5
|
||||
secret_access_key: 217046bee47c050301e3cc13e3cba1a8a943cf5f37f8c7979c349c5254441d18
|
||||
region: us-east-1
|
||||
use_ssl: false
|
||||
part_size: 5242880 # 5MB
|
||||
index_path: /tmp/vaultik-integration-test.sqlite
|
||||
chunk_size: 10MB
|
||||
blob_size_limit: 10GB
|
||||
compression_level: 3
|
||||
hostname: test-host
|
||||
Reference in New Issue
Block a user