sneak/vaultik
1
0
Fork 0
Code Issues 12 Pull Requests 2 Actions 25 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/vaultik:6dc134a50ea1fa1f9ed6cf791dcbc8e3f6ab0301
Branches Tags
sneak/vaultik:main sneak/vaultik:snapshot-remove-json-warning-on-stderr sneak/vaultik:next
..
compare: sneak/vaultik:snapshot-remove-json-warning-on-stderr
Branches Tags
sneak/vaultik:snapshot-remove-json-warning-on-stderr sneak/vaultik:next sneak/vaultik:main
17 Commits
6dc134a50e .. snapshot-remove-json-warning-on-stderr
Author SHA1 Message Date
sneak 012a161b61 Write only the document to stdout from snapshot remove --json (closes #251)
check / check (push) Waiting to run
Details 
When the destination store cannot be reached, `snapshot remove` still
removes the snapshot from the local index and warns. The warning went
through the UI, which writes to stdout, so under `--json` it landed
ahead of the document and `| jq` failed on a command that exited 0.
Under `--json` the UI warning is now skipped, and the logger's warning
on stderr carries the follow-up.

That follow-up, in the warning, the README and the command's help, said
`vaultik prune` would finish the cleanup, but `prune` never removes
snapshot metadata. They now say to run `vaultik snapshot remove` for the
snapshot again once the destination store is reachable.

The new test removes a snapshot from the local index against a missing
destination directory and checks both streams.

Model: opus-5-5
 2026-10-07 01:09:40 +00:00
clawbot 49eed7a3e5 Count each file, byte and upload once in backup statistics (closes #225)
check / check (push) Waiting to run
Details 
The scanner added a changed file's bytes again for each new chunk and
counted a file as unchanged for each chunk already stored. It now
counts files and bytes once, in the scan phase, and counts its own
uploads, so a --cron run, which has no progress reporter, records
them. The blob count no longer adds earlier paths' blobs again.

The snapshots row now stores the size of all files in total_size and
the referenced blobs' sizes in blob_size, blob_uncompressed_size and
compression_ratio, as docs/DATAMODEL.md says. Those sizes come from
one query, and a failed query fails the snapshot. DATAMODEL.md now
says chunk_count and blob_count count what the run added.

Removed UpdateSnapshotStats and GetCountBySnapshot, which nothing
calls any more.

Model: opus-5-5
 2026-10-07 02:59:27 +02:00
clawbot 85d4ef118d Keep command output to the README's stdout and stderr rules (closes #224)
check / check (push) Waiting to run
Details 
The startup banner moves from stdout to stderr, so a `completion`
script, a `config get` value and the hidden `__complete` command print
only their own output. `--quiet`, `--cron` and `--json` still suppress
it.

A failing `remote info`, `prune` or `snapshot remove` under `--json`
now reports its error on stderr. Their reporters returned early under
`--json`, so the failure reached neither stream.

`snapshot verify --quiet` writes no report. A failure is still
returned and printed on stderr, with the same exit status.

Judgement call: the banner's stream, posted on the issue for the owner.

Model: opus-5-5
 2026-10-07 01:12:15 +02:00
clawbot b06f992152 Start the progress reporter once per snapshot, not once per path (closes #253)
check / check (push) Waiting to run
Details 
A backup without --cron of a snapshot with two or more paths panicked
with "close of closed channel". Scan runs once per path, and it started
the progress reporter and deferred its Stop each time; Stop closes the
reporter's signal channel, so the second path's Stop panicked.
scanAllDirectories now starts the reporter before the first path and
stops it after the last, and Scan no longer starts or stops it. A new
test backs up a two-path snapshot with the reporter on and restores
both paths.

Model: opus-5-5
 2026-10-07 00:12:10 +02:00
clawbot 5d685f03ce List the files under a path by path, not by string prefix (closes #223)
check / check (push) Waiting to run
Details 
FileRepository.ListByPrefix matched with SQL LIKE: a plain string
prefix that ignores ASCII case and treats _ and % as wildcards.
Restoring /home/u/doc also restored doc2, DOC and doc.txt.bak, and a
backup counted the files of a longer sibling path as deleted. It is
now ListUnderPath, which returns the file at the path and every file
whose path starts with the path plus a slash, compared exactly. A
trailing slash is ignored, so "/" still lists every file.

Three tests relied on string-prefix matching and now name a full path
or call ListAll.

ListIDsWithChunksNotInUploadedBlobs keeps its LIKE: it only adds file
IDs the scan never looks up.

Model: opus-5-5
 2026-10-06 22:12:16 +02:00
clawbot f59086c0e5 Return an error, not a panic, on a malformed snapshot database (closes #231)
check / check (push) Waiting to run
Details 
Restore cut chunk hashes from the snapshot database to 16 characters
for its error messages, so a shorter hash panicked. Those messages now
use shortHash. Under --verify, a file_chunks row with no chunks row was
dereferenced, and the chunk size from the database was allocated in
one piece, so a negative or huge size panicked. A missing row is now an
error, a negative size is rejected, and each chunk is hashed by
streaming it from the restored file.

A restored file shorter than its chunks now fails verify as a short
read instead of an unexpected EOF.

Model: opus-5-5
 2026-10-06 21:16:28 +02:00
clawbot d276d891da Join the S3 prefix to every key with one slash (closes #222)
check / check (push) Waiting to run
Details 
The S3 client built each key as prefix + key, and the URL parser keeps
the prefix as written, so s3://bucket/p stored p + "blobs/..." with no
slash while s3://bucket/p/ stored p/blobs/.... A recovery host that wrote
the URL the other way found no snapshots.

NewClient now strips trailing slashes from the prefix and adds one back
when anything is left, giving the README layout for both URL forms; an
empty prefix stays at the bucket root. The s3.prefix config setting
goes through the same client and gets the same join.

A new test writes through each URL shape against an in-process S3
server, checks the key in the bucket, and lists through both List and
ListStream, which every snapshot listing uses.

Model: opus-5-5
 2026-10-06 19:46:09 +02:00
clawbot 5aa5ba5544 Apply restored owners, modes and times in an order that keeps them (closes #219)
check / check (push) Waiting to run
Details 
A directory got its stored mode and mtime before its contents were
written, so a read-only directory came back without its files and a
non-empty one carried the time of the restore. Directories are now
created owner-only (0700) and get their stored owner, mode and mtime
after the restore loop, each before its parent, skipping any whose
place a symlink has since taken. A file's mode is now applied after its
chown, which on Linux clears setuid and setgid. A symlink gets its
stored owner (as root) and mtime on the link itself, through
golang.org/x/sys/unix, now a direct dependency.

An interrupted restore leaves its directories at 0700.

Model: opus-5-5
 2026-10-06 18:29:09 +02:00
clawbot 315b6483b8 Mark github.com/spf13/pflag as a direct dependency in go.mod (closes #246)
check / check (push) Successful in 12m34s
Details 
internal/cli/snapshot_restore_test.go imports github.com/spf13/pflag
directly, but go.mod still marked it // indirect. script/precommit runs
go mod tidy and fails when that changes go.mod, so the pre-commit hook
stopped every commit. This is the go mod tidy output: pflag moves to the
direct require block, and go.sum does not change. script/cibuild does
not run the tidy, which is why the gate stayed green.

Model: opus-5-5
 2026-10-06 16:59:26 +02:00
clawbot 14fc4c9893 Load a config that has no age recipient (closes #221)
check / check (push) Successful in 13m47s
Details 
The README's steps for restoring on another machine failed at the first
command: `config init` wrote a placeholder recipient, and `config.Load`
rejects any recipient that does not parse. `config init` now writes an
empty `age_recipients` list, `config.Load` accepts an empty list, and
`snapshot create` refuses to start without a recipient. A malformed
recipient is still rejected at load.

The recovery-host test now builds its config with `config init` and
`config set` and reads it through `config.Load`, so it imports
`internal/cli`. On a fresh file, `config set age_recipients.0` writes the
list in flow style (`[age1...]`).

Model: opus-5-5
 2026-10-06 15:46:13 +02:00
clawbot 66c80a70e3 Skip files of an unreadable blob under restore --skip-errors (closes #218)
check / check (push) Successful in 12m34s
Details 
A blob that failed to download ended `snapshot restore` even with
--skip-errors, after restoring whichever files came first. The download
error now goes through the same per-file handling as any other restore
error, once for every pending file that references the blob. With
--skip-errors those files are reported as failed, the rest are
restored, and the command still exits non-zero. Without the flag the
restore still aborts; the error now also names one affected file and
suggests --skip-errors. A cancelled restore still ends at once. The
--skip-errors help text and README now limit the packing and storage
caveat to snapshot creation.

Model: opus-5-5
 2026-10-06 14:46:05 +02:00
clawbot 81f83b29f6 Re-vendor the canonical files from sneak/prompts at dd4027b (closes #213)
check / check (push) Successful in 12m47s
Details 
Linting and testing become the lint and test phases of the Dockerfile,
and the build stage depends on both. Dockerfile.lint, CHECK_EPOCH and
the tests that checked them are removed. Every docker build in script/
passes --no-cache, and script/cibuild runs script/bootstrap first. A
host without Go gets the go.mod version from script/install-go in
.tool/go, which bootstrap, the Makefile, fmt, fmt-check, precommit and
release add to PATH; fmt-check skips .tool. The image takes its version
from the VERSION build arg or git describe, dev without .git. This
repo's own entries follow the canonical content in .gitignore and
.editorconfig. The golangci-lint v2.14.0 findings are fixed. The rules
in CLAUDE.md move into AGENTS.md. IsDevVersion counts "unknown".

Model: opus-5-5
 2026-10-06 12:46:13 +02:00
clawbot c4adb72d80 Run the local index in WAL mode with a busy timeout (closes #217)
check / check (push) Successful in 5m10s
Details
check / check (pull_request) Successful in 5m38s
Details 
The connection settings were passed as `_journal_mode=`-style
parameters, which the SQLite driver drops without an error, so the
index ran in rollback-journal mode with no busy timeout. `snapshot
list` or `info` reading during a backup could make the backup's next
write fail with "database is locked". Both open paths now pass
`_pragma=` parameters; foreign keys moved there too.

With WAL on, rows committed to the open index can still be in the
-wal file, which a copy of the main file misses. The metadata export
now copies the index with VACUUM INTO, into an empty 0600 file.

The retry after a failed open no longer claims a TRUNCATE recovery; it
retries with the same settings.

Model: opus-5-5
 2026-10-06 11:29:18 +02:00
clawbot 4a167e153a Record the real uid and gid of backed-up files (closes #216)
check / check (push) Successful in 4m58s
Details
check / check (pull_request) Successful in 6m21s
Details 
The scanner read uid and gid by asserting the stat result to an
interface with Uid() and Gid() methods. *syscall.Stat_t has Uid and Gid
fields, not methods, so the assertion never matched and every file,
directory and symlink was stored as 0:0; a restore as root then gave
everything to root. The scanner now reads the fields of
*syscall.Stat_t.

The first backup after this change re-reads every file not owned by
root, because its stored uid and gid no longer match the disk.

When the tests run as root, as in the Docker build, the new test
compares 0 with 0 and cannot catch the defect; a non-root run does.

Model: opus-5-5
 2026-10-06 09:46:16 +02:00
clawbot ea72697992 List a missing file:// destination directory as an error (closes #220)
check / check (push) Successful in 5m53s
Details
check / check (pull_request) Successful in 4m53s
Details 
The file backend listed a destination directory that does not exist as
an empty store. With the volume unplugged, snapshot list reported every
local snapshot as missing from the store, snapshot remove said it had
removed metadata it never reached, and prune dropped every local
snapshot record. List and ListStream now fail when the destination
directory is missing, so those commands take their existing path for a
store that cannot be listed. A missing prefix under an existing
directory is still an empty listing, and a first backup still creates
the directory.

Three tests listed a file:// destination nothing had created; they now
create it.

Model: opus-5-5
 2026-10-06 08:46:17 +02:00
clawbot 713be502bd Reject a duration with characters outside its parts (closes #215)
check / check (push) Successful in 7m28s
Details
check / check (pull_request) Successful in 5m39s
Details 
parseDuration fell back to an unanchored search for number-and-unit
pieces when time.ParseDuration failed, and skipped everything in
between. 1.0y became 0, so `snapshot create --prune --keep-newer-than
1.0y` deleted every snapshot of the backed-up names, the new one
included. 2.1w became one week and 1,5y five years. The fallback now
requires the whole input to be whole-number-and-unit parts with nothing
between them. A bare number is rejected before time.ParseDuration sees
it, since Go reads 0 and +0 as zero with no unit.

Judgement call: a space between number and unit (`30 days`) was
accepted and is now an error, matching Go's own units.

Model: opus-5-5
 2026-10-06 06:12:07 +02:00
clawbot 35cf985c18 Re-chunk a known file whose chunks no uploaded blob holds (closes #214)
check / check (push) Successful in 6m53s
Details
check / check (pull_request) Successful in 6m17s
Details 
File rows are shared by every snapshot and updated in place, while a
blob row is deleted once no snapshot references it. Removing the newest
snapshot, or the prune after an interrupted run, could drop the only
blob holding a changed file's current chunks while an older snapshot
kept the file row. The next backup compared metadata only, skipped the
file, and completed a snapshot that could not restore it.

The scanner now loads the IDs of known files that list a chunk no
uploaded blob holds and re-chunks them even when their metadata is
unchanged.

The tests append to a file, so the file keeps its first chunk in a blob
the first snapshot still references. Each backup run gets its own
snapshot name, so the second-precision snapshot IDs differ without
sleeping.

Model: opus-5-5
 2026-10-06 04:46:16 +02:00
74 changed files with 3469 additions and 610 deletions
Expand all files Collapse all files
.editorconfig
+3
View File
@@ -10,3 +10,6 @@ insert_final_newline = true
[Makefile]
indent_style = tab
[*.go]
indent_style = tab
.gitea/workflows/release.yml
+6 -5
View File
@@ -16,11 +16,12 @@ jobs:
fetch-depth: 0
# goreleaser is not a compiler: it shells out to `go` for the
# `before:` hook and for every one of the four cross-compiles.
# Nothing else in this repo puts a Go toolchain on the runner --
# check.yml runs script/cibuild, which does all of its work inside
# the digest-pinned Dockerfile images -- so without this step the
# release either fails at the before-hook or, worse, ships binaries
# built by whatever Go the runner happens to carry.
# Without this step the release either fails at the before-hook or,
# worse, ships binaries built by whatever Go the runner happens to
# carry. check.yml's runner gets the same Go through
# script/bootstrap, which calls script/install-go, and uses it only
# for `go mod download` and gofmt; it compiles inside the
# digest-pinned Dockerfile images.
#
# actions/setup-go would pin the action by commit sha, but the Go
# tarball it downloads at runtime is verified against no value in
.gitignore
+15
View File
@@ -45,3 +45,18 @@ node_modules/
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# Go build and test output.
*.log
*.out
*.test
coverage.html
# This repo's own host-built artifacts.
/vaultik
/dist/
/.tool/
# Local configs for development; they hold storage credentials.
local-config.yaml
dev-config.yaml
AGENTS.md
-1
View File
@@ -128,4 +128,3 @@ Version: 2025-06-08
18. For estimates: backing up over 2.5Gbit/s ethernet to an S3 server
backed by 2000MB/sec SSD takes about 4 seconds per gigabyte.
ARCHITECTURE.md
+3 -3
View File
@@ -335,10 +335,10 @@ CreateSnapshot(opts)
│ │
│ └─► Accumulate statistics
│
├─► SnapshotManager.UpdateSnapshotStatsExtended()
│
├─► SnapshotManager.PopulateSnapshotBlobs() // record referenced blobs
│
├─► SnapshotManager.UpdateSnapshotStatsExtended()
│
├─► SnapshotManager.ExportSnapshotMetadata()
│ │
│ ├─► Copy database to temp file
@@ -353,7 +353,7 @@ CreateSnapshot(opts)
## Deduplication Strategy
1. **File-level**: Files unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid)
1. **File-level**: Files unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded.
Dockerfile
+15 -7
View File
@@ -40,20 +40,28 @@ RUN go mod download
COPY . .
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout. The commit and
# its date always come from that .git, and are "unknown" without one.
# `git describe --tags --always` on the .git in the build context. The
# commit and its date always come from that .git. With .git present, a
# version that is still empty, dev or unknown, or a commit or date that
# is unknown, fails the build: git is missing or could not read the
# checkout, as when .git is a file pointing outside the context. A
# context without .git, such as a source export, stamps "dev" and an
# "unknown" commit and date.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
RUN VERSION="${VERSION:-$(git describe --tags --always || echo dev)}"; \
commit="$(git rev-parse HEAD || echo unknown)"; \
commit_date="$(git show -s --format=%cs HEAD || echo unknown)"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
if [ "$commit" = unknown ] || [ "$commit_date" = unknown ]; then \
echo "commit is '$commit' and its date '$commit_date'" \
"although .git is present" >&2; \
exit 1; \
fi; \
fi; \
commit="$(git rev-parse HEAD || echo unknown)"; \
commit_date="$(git show -s --format=%cs HEAD || echo unknown)"; \
globals=sneak.berlin/go/vaultik/internal/globals; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X ${globals}.Version=${VERSION} \
Makefile
+3
View File
@@ -1,5 +1,8 @@
.PHONY: all bootstrap setup check test lint lint-fix fmt fmt-check build clean deps test-coverage local install release release-snapshot docker hooks
# Where script/bootstrap installs Go when the host has none.
export PATH := $(PATH):$(CURDIR)/.tool/go/bin
# Version number, derived from git by script/version (`git describe
# --tags --always --dirty`). This used to be a hardcoded
# constant, which meant every local build claimed to be a release that
README.md
+48 -34
View File
@@ -40,7 +40,8 @@ Features:
* modern encryption ([age](https://age-encryption.org/), X25519 + ChaCha20-Poly1305)
* content-defined chunking with deduplication (FastCDC)
* incremental backups (only changed files are re-chunked)
* incremental backups (a file is re-chunked only when it changed or a
chunk it lists is held by no uploaded blob)
* multithreaded zstd compression at configurable levels
* content-addressed immutable storage
* local state tracking in SQLite (enables write-only incremental backups)
@@ -177,7 +178,7 @@ vaultik version
* `--verbose`, `-v`: Enable verbose output (on stderr — see below)
* `--debug`: Enable debug output (on stderr — see below)
* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner)
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors (which would leave a chunk recorded but not stored) still abort the run.
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run.
### locking
@@ -199,8 +200,10 @@ local index or the destination store. `config`, `database delete`,
### stdout and stderr
Log output — everything from `--verbose` and `--debug`, and every
warning and error the logger emits — goes to **stderr**. stdout carries
the output you asked for: tables, and the documents produced by `--json`.
warning and error the logger emits — goes to **stderr**, and so does the
startup banner. stdout carries the output you asked for: tables, the
documents produced by `--json`, `config get` values, and completion
scripts.
This means `vaultik snapshot list --verbose > out.txt` captures the
listing and leaves the diagnostics on your terminal. To capture both,
@@ -349,8 +352,9 @@ may hold snapshots this host doesn't know about), which is what
prune` invocation to run as a follow-up. Local row cleanup (files,
chunks, blobs the snapshot was the last referrer for) runs
automatically. If the destination store is unreachable, the local-DB
removal still completes and a warning is emitted; rerun `vaultik prune`
once the store is reachable to finish remote cleanup. To wipe everything
removal still completes and a warning is emitted; run `vaultik snapshot
remove <snapshot-id>` again once the store is reachable to remove the
snapshot's metadata from it (`vaultik prune` does not). To wipe everything
on the destination in one go, use `vaultik remote nuke --force`.
* `--local-only`: Skip remote cleanup; only touch the local index
* `--dry-run`: Show what would be deleted without deleting
@@ -499,8 +503,9 @@ format does and does not protect.
* Content-defined chunking using the FastCDC algorithm
* Average chunk size: configurable (default 10MB)
* Deduplication at file level (unchanged files skipped) and chunk level
(identical chunks across files stored once)
* Deduplication at file level (unchanged files skipped, unless a chunk
the file lists is held by no uploaded blob) and chunk level (identical
chunks across files stored once)
* Multiple chunks packed into blobs to reduce object count
### encryption
@@ -528,7 +533,7 @@ complete annotated example also lives in
| Field | Default | Description |
|-------|---------|-------------|
| `age_recipients` | (required) | Age public keys for encryption |
| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty |
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
| `snapshots` | (required) | Named snapshot definitions with paths and excludes |
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
@@ -643,8 +648,8 @@ Work planned after 1.0. Loosely ordered by priority.
## output style
Every command's user-facing output is governed by `internal/ui`, in one
of two ways. Color is enabled when stdout is a TTY and the `NO_COLOR`
environment variable is unset (https://no-color.org/).
of two ways. Color is enabled when the stream written to is a TTY and
the `NO_COLOR` environment variable is unset (https://no-color.org/).
* **Status, progress, warnings, and errors** go through the `internal/ui`
message methods below: marker-prefixed, colored on a TTY, and — except
@@ -654,24 +659,26 @@ environment variable is unset (https://no-color.org/).
`config init`, `config set`, and `database delete`.
* **The data a command exists to produce** is written plain, with no
marker and no color, because a marker would corrupt a table or a
parsed document. This covers the `version`, `info`, and `remote info`
reports, the `snapshot list` table, `config get` values, and every
`--json` document. `--quiet` silences the human reports and tables
(`version`, `info`, `remote info`, `snapshot list`) but never the
machine-consumed `config get` value or the `--json` documents, which a
script depends on. The `database delete` confirmation prompt is also
written this way and always shown: it is an interactive exchange the
operator must see.
parsed document. This covers the `version`, `info`, `remote info` and
`snapshot verify` reports, the `snapshot list` table, `config get`
values, and every `--json` document. `--quiet` silences the human
reports and tables (`version`, `info`, `remote info`, `snapshot
verify`, `snapshot list`) but never the machine-consumed `config get`
value or the `--json` documents, which a script depends on. The
`database delete` confirmation prompt is also written this way and
always shown: it is an interactive exchange the operator must see.
`internal/ui` writes to stdout; it is the output the user asked for.
Structured log records are a different thing and go through
`internal/log`, which writes to stderr (see "stdout and stderr" above).
`internal/ui` writes to stdout; it is the output the user asked for. The
exceptions are the startup banner and the error a failed command ends
with, which go to stderr. Structured log records are a different thing
and go through `internal/log`, which writes to stderr (see "stdout and
stderr" above).
Message classes:
| Class | Marker | Alignment | Use for |
|-------|--------|-----------|---------|
| Banner | none | column 0 | The startup line printed once per invocation |
| Banner | none | column 0 | The startup line printed once per invocation, on stderr |
| Begin | `》` (white) | column 0 | An operation is about to start (present-continuous verb) |
| Complete | `》` (green) | column 0 | An operation just finished (past-tense verb) |
| Info | `》` (white) | column 0 | Neutral status update |
@@ -761,9 +768,13 @@ standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call
them. We provide:
* `script/bootstrap` — install all development dependencies (go, Go
module download). It deliberately does not install `golangci-lint`;
see `script/lint` below.
* `script/bootstrap` — install all development dependencies (Go, Go
module download). A host without Go gets the `go.mod` version through
`script/install-go`, in `.tool/go`, which `script/bootstrap` itself,
the `Makefile`, `script/fmt`, `script/fmt-check`, `script/precommit`
and `script/release` add to their `PATH`. It
deliberately does not install `golangci-lint`; see `script/lint`
below.
* `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`
* `script/projectname` — print the project name (used for the Docker
@@ -778,12 +789,14 @@ them. We provide:
`script/bootstrap` insists on.
* `script/install-go` — install the Go toolchain named by `go.mod`'s
`go` directive into `.tool/go` from a sha256-verified `go.dev`
archive, and put it on `PATH`. Idempotent. Called only by the release
workflow, which needs a host Go for `goreleaser` to shell out to;
nothing else on the release runner does. `actions/setup-go` is not
used because it verifies the downloaded toolchain against no value in
this repo. Bumping Go edits `go.mod`, the checksum in this script, and
the two `golang` digests in the `Dockerfile` together.
archive, for Linux or macOS on amd64 or arm64. Idempotent. On a CI
runner it also puts `.tool/go/bin` on `PATH` for the steps that
follow. Called by the release workflow, which needs a host Go for
`goreleaser` to shell out to, and by `script/bootstrap` on a host
without Go. `actions/setup-go` is not used because it verifies the
downloaded toolchain against no value in this repo. Bumping Go edits
`go.mod`, the checksums in this script, and the two `golang` digests
in the `Dockerfile` together.
* `script/release` — cross-compile and publish the release artifacts
with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose
version is not the pinned one, on the same reasoning as `script/lint`.
@@ -811,7 +824,7 @@ them. We provide:
find out whether the tree is clean.
* `script/fmt` — format all code (writes)
* `script/fmt-check` — check formatting (read-only). It runs `gofmt` on
the host.
the host, over every Go file outside `.tool`.
* `script/check` — run `script/test`, `script/lint`, and
`script/fmt-check`.
* `script/docker` — build the Docker image tagged via
@@ -852,7 +865,8 @@ file. It is `git describe --tags --always --dirty`, which
A `docker build .` of a clone, with no build arguments, runs the same
`git describe` (without `--dirty`) on the `.git` in its build context,
so it stamps the same value for a clean commit; the build fails if the
context carries `.git` and no version comes out. A binary built without
context carries `.git` and no version, commit or commit date comes out.
A binary built without
git metadata reports `dev`, or `unknown` when `script/docker` or
`script/cibuild` built it outside a git checkout.
TODO.md
+149
View File
@@ -22,6 +22,110 @@ the tag exists and is exercised; what is left is merging `next` to
# Completed Steps
- 2026-10-06: Made `snapshot remove --json` write only its document to
stdout when the destination store cannot be reached
([issue #251](https://git.eeqj.de/sneak/vaultik/issues/251)). Its
warning that the snapshot's metadata was left on the destination store
went to stdout ahead of the document, breaking `| jq` on a command that
exited 0. Under `--json` the warning now reaches stderr only, through
the logger. The warning, the README and the command's help said
`vaultik prune` would finish the cleanup, but `prune` never removes
snapshot metadata; they now say to run `vaultik snapshot remove` for the
snapshot again once the destination store is reachable.
- 2026-10-06: Made the backup summary and the `snapshots` row count each
file, byte and upload once
([issue #225](https://git.eeqj.de/sneak/vaultik/issues/225)). The
scanner added a file's bytes again for each new chunk and counted a
file as unchanged for each chunk already stored, so a first backup
reported twice its size and "backed up" could go negative. Upload
figures came from the progress reporter, which `--cron` turns off, and
`blob_count` counted earlier paths' blobs again for each later path.
The scanner now counts uploads itself; `blob_size`,
`blob_uncompressed_size` and `compression_ratio` describe the blobs
the snapshot references, and `docs/DATAMODEL.md` now says
`chunk_count` and `blob_count` count what the run added.
- 2026-10-06: Made command output follow the README's stdout and stderr
rules ([issue #224](https://git.eeqj.de/sneak/vaultik/issues/224)). The
startup banner went to stdout, so a `completion` script or a
`config get` value started with it; the banner now goes to stderr. A
failing `remote info`, `prune` or `snapshot remove` under `--json`
printed nothing on either stream, and now reports its error on stderr.
`snapshot verify --quiet` printed its whole report; it now prints
none, and a failure still reaches stderr with the same exit status.
- 2026-10-06: Made a backup without `--cron` of a snapshot with two or
more `paths` complete instead of panicking with `close of closed
channel` ([issue #253](https://git.eeqj.de/sneak/vaultik/issues/253)).
`Scan` runs once per path and started and stopped the progress
reporter each time, and a second stop panics. The reporter is now
started and stopped once per snapshot, around the scans of all its
paths.
- 2026-10-06: Made a restore path argument select only that path and
what is beneath it
([issue #223](https://git.eeqj.de/sneak/vaultik/issues/223)). The
lookup matched with SQL `LIKE`, so `/home/u/doc` also restored
`doc2`, `DOC` and `doc.txt.bak`, and a `_` or `%` in the path acted
as a wildcard. A backup used the same lookup to load the known files
of each configured path, so files of a longer sibling path were
counted as deleted. `FileRepository.ListUnderPath`, which replaces
`ListByPrefix`, returns the file at the path and every file whose
path starts with the path plus `/`, compared case-sensitively.
- 2026-10-06: Made restore return an error instead of panicking on a
malformed snapshot database
([issue #231](https://git.eeqj.de/sneak/vaultik/issues/231)). A chunk
hash shorter than 16 characters crashed the error message naming it,
and `--verify` dereferenced a missing `chunks` row and allocated
whatever chunk size the database gave. Those messages now go through
`shortHash`, a missing row is an error, and `--verify` rejects a
negative size and hashes each chunk as a stream.
- 2026-10-06: Made `s3://bucket/prefix` and `s3://bucket/prefix/` the same
destination ([issue #222](https://git.eeqj.de/sneak/vaultik/issues/222)).
The S3 client put the prefix directly in front of each key, so a prefix
without a trailing slash stored `prefixblobs/...`. A non-empty prefix is
now joined to every key with one `/`, giving the README's
`<bucket>/<prefix>/blobs/...` layout. The `s3.prefix` config setting goes
through the same client and gets the same join.
- 2026-10-06: Made restore apply owners, modes and times in an order
that keeps them
([issue #219](https://git.eeqj.de/sneak/vaultik/issues/219)). A
directory got its stored mode and mtime before its contents were
written, so a read-only directory came back without its files and a
non-empty one carried the time of the restore. Directories are now
created owner-only and get their stored owner, mode and mtime once the
restore loop is done, each before its parent. A file's mode is applied
after its chown, which on Linux clears setuid and setgid, and a
symlink gets its stored owner (as root) and mtime on the link itself.
- 2026-10-06: Made `go.mod` what `go mod tidy` writes, so the
pre-commit hook no longer stops every commit
([issue #246](https://git.eeqj.de/sneak/vaultik/issues/246)). A test
in `internal/cli` imports `github.com/spf13/pflag` directly, but
`go.mod` still marked it `// indirect`, and `script/precommit` fails
whenever the tidy changes `go.mod`. It is now in the direct `require`
block.
- 2026-10-06: Made the README's steps for restoring on another machine
work ([issue #221](https://git.eeqj.de/sneak/vaultik/issues/221)).
`config init` wrote a placeholder recipient that `config.Load`
rejects, so every command on the new machine failed before reaching
the store. The file now has an empty `age_recipients` list,
`config.Load` accepts an empty list, and `snapshot create` refuses to
run without a recipient.
- 2026-10-06: Made `snapshot restore --skip-errors` skip the files that
need a blob it cannot download
([issue #218](https://git.eeqj.de/sneak/vaultik/issues/218)). A missing
or damaged blob ended the restore even with the flag, after restoring
whichever files happened to come first. Every file that needs such a
blob is now reported as failed, the rest are restored, and the command
still exits non-zero. Without the flag the blob error still aborts.
- 2026-10-06: Re-vendored the canonical files from `sneak/prompts` at
`dd4027b` ([issue #213](https://git.eeqj.de/sneak/vaultik/issues/213)).
Linting and testing are now the `lint` and `test` phases of the
@@ -31,6 +135,51 @@ the tag exists and is exercised; what is left is merging `next` to
is v2.14.0, with its new findings fixed in the code, and the rules in
`CLAUDE.md` now live in `AGENTS.md`.
- 2026-10-06: Made the local index actually run in WAL mode with a busy
timeout ([issue #217](https://git.eeqj.de/sneak/vaultik/issues/217)).
The connection settings were written in a form the SQLite driver
ignores, so the index ran without either and `snapshot list` or `info`
during a backup could make the backup's next write fail with
`database is locked`. They are now `_pragma=` parameters, and the
metadata export copies the open index with `VACUUM INTO`, because a
copy of the file alone misses rows still in the `-wal` file.
- 2026-10-06: Made a backup record the real uid and gid of files,
directories and symlinks
([issue #216](https://git.eeqj.de/sneak/vaultik/issues/216)). The
scanner asked the stat result for `Uid()` and `Gid()` methods, which
`*syscall.Stat_t` does not have, so every entry was stored as `0:0`
and a restore as root gave every file to root. It now reads the
`Uid` and `Gid` fields of `*syscall.Stat_t`.
- 2026-10-06: Made a `file://` destination whose directory is missing
count as one that cannot be listed
([issue #220](https://git.eeqj.de/sneak/vaultik/issues/220)). The file
backend listed a missing directory as an empty store, so with the
quickstart's USB stick unplugged `snapshot list` reported every local
snapshot as missing from the store, `snapshot remove` claimed to have
removed metadata it never reached, and `prune` dropped every local
snapshot record. Listing a missing destination directory is now an
error; a first backup still creates the directory.
- 2026-10-06: Made `--older-than` and `--keep-newer-than` reject a
duration with characters outside its number-and-unit parts
([issue #215](https://git.eeqj.de/sneak/vaultik/issues/215)). The
parser picked out the parts it recognised and skipped the rest, so
`1.0y` became zero and `--prune --keep-newer-than 1.0y` deleted every
snapshot of the backed-up names, the new one included. `1.0y`,
`1,5y`, `30 days`, `x7d` and a bare `0` are now errors; decimals still
work in Go units such as `1.5h`.
- 2026-10-06: Made a backup re-chunk a known file that lists a chunk no
uploaded blob holds
([issue #214](https://git.eeqj.de/sneak/vaultik/issues/214)). File
rows are shared by every snapshot and updated in place, so removing or
pruning the only snapshot that referenced a changed file's current
blob left an older snapshot keeping a row that matched the disk while
no blob held its chunks. The next backup skipped the file and
completed a snapshot that could not restore it.
- 2026-09-22: Routed the last direct-to-stdout command output through
`internal/ui`
([issue #149](https://git.eeqj.de/sneak/vaultik/issues/149)). The
cmd/vaultik/dockerversion_test.go
+12 -6
View File
@@ -45,9 +45,11 @@ func TestProductDockerfileTakesVersionAsBuildArg(t *testing.T) {
// TestProductDockerfileDerivesVersionFromGit fails unless a build given
// no VERSION, such as a plain `docker build .` of a clone, takes it from
// `git describe` of the .git in its context, and fails rather than
// stamp "dev" when that .git yields no version. The commit and its date
// come from the same .git.
// `git describe` of the .git in its context, stamps "dev" when the
// context has no .git, and fails rather than stamp "dev" when that .git
// yields no version. The commit and its date come from the same .git,
// and the build fails rather than stamp them "unknown" when it is
// present.
func TestProductDockerfileDerivesVersionFromGit(t *testing.T) {
t.Parallel()
@@ -56,9 +58,9 @@ func TestProductDockerfileDerivesVersionFromGit(t *testing.T) {
buildAt := indexContaining(found, "go build")
require.GreaterOrEqual(t, buildAt, 0, "%s must build", productDockerfile)
assert.Contains(t, found[buildAt], "git describe --tags --always",
"%s must derive the version from git when no VERSION is given",
productDockerfile)
assert.Contains(t, found[buildAt], "git describe --tags --always || echo dev",
"%s must derive the version from git when no VERSION is given,"+
" and stamp dev when the context has no .git", productDockerfile)
assert.Contains(t, found[buildAt], "[ -e .git ]",
"%s must fail when the context carries .git but yields no version",
productDockerfile)
@@ -66,6 +68,10 @@ func TestProductDockerfileDerivesVersionFromGit(t *testing.T) {
"%s must stamp the commit from git", productDockerfile)
assert.Contains(t, found[buildAt], "git show -s --format=%cs HEAD",
"%s must stamp the commit date from git", productDockerfile)
assert.Contains(t, found[buildAt],
`[ "$commit" = unknown ] || [ "$commit_date" = unknown ]`,
"%s must fail when the context carries .git but yields no commit"+
" or date", productDockerfile)
}
// TestDockerScriptComputesVersionOnTheHost fails unless script/docker
config.example.yml
+2 -1
View File
@@ -3,7 +3,8 @@
# Copy this file and uncomment/modify the values you need
# Age recipient public keys for encryption
# This is REQUIRED - backups are encrypted to these public keys
# Backups are encrypted to these public keys. snapshot create needs at least
# one; listing, verifying and restoring do not
# Generate with: age-keygen | grep "public key"
age_recipients:
- age1cj2k2addawy294f6k2gr2mf9gps9r3syplryxca3nvxj3daqm96qfp84tz
docs/DATAMODEL.md
+5 -4
View File
@@ -117,10 +117,10 @@ Tracks backup snapshots.
- `started_at` (INTEGER) - Start timestamp
- `completed_at` (INTEGER) - Completion timestamp (NULL if in progress)
- `file_count` (INTEGER) - Number of files in snapshot
- `chunk_count` (INTEGER) - Number of unique chunks
- `blob_count` (INTEGER) - Number of blobs referenced
- `chunk_count` (INTEGER) - Number of chunks this snapshot stored that were not stored before
- `blob_count` (INTEGER) - Number of blobs this snapshot created
- `total_size` (INTEGER) - Total size of all files
- `blob_size` (INTEGER) - Total size of all blobs (compressed)
- `blob_size` (INTEGER) - Total compressed size of all referenced blobs
- `blob_uncompressed_size` (INTEGER) - Total uncompressed size of all referenced blobs
- `compression_ratio` (REAL) - Compression ratio achieved
- `compression_level` (INTEGER) - Compression level used for this snapshot
@@ -195,6 +195,7 @@ Tracks blob upload metrics.
1. **Change Detection**
- `SELECT * FROM files WHERE path = ?` - Get previous file metadata
- Compare mtime, size, mode to detect changes
- Re-chunk a file that lists a chunk no uploaded blob holds, even when its metadata is unchanged
- Skip unchanged files but still add to `snapshot_files`
2. **Chunk Reuse**
@@ -279,7 +280,7 @@ This ensures consistency, especially important for operations like:
3. **Batch Operations**: Where possible, operations are batched within transactions
4. **Write-Ahead Logging**: SQLite WAL mode is enabled for better concurrency
4. **Write-Ahead Logging**: The local index runs in SQLite WAL mode with a 10-second busy timeout, so a read-only command such as `snapshot list` can read it while a backup writes to it. Committed rows can sit in the `-wal` file beside the index until a checkpoint, so the metadata export copies the index through SQLite (`VACUUM INTO`), not as a file
## Data Integrity
go.mod
+2 -2
View File
@@ -20,9 +20,11 @@ require (
github.com/rclone/rclone v1.72.1
github.com/spf13/afero v1.15.0
github.com/spf13/cobra v1.10.1
github.com/spf13/pflag v1.0.10
github.com/stretchr/testify v1.11.1
go.uber.org/fx v1.24.0
golang.org/x/sync v0.18.0
golang.org/x/sys v0.38.0
golang.org/x/term v0.37.0
gopkg.in/yaml.v3 v3.0.1
modernc.org/sqlite v1.38.0
@@ -225,7 +227,6 @@ require (
github.com/smarty/assertions v1.16.0 // indirect
github.com/sony/gobreaker v1.0.0 // indirect
github.com/spacemonkeygo/monkit/v3 v3.0.25-0.20251022131615-eb24eb109368 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/t3rm1n4l/go-mega v0.0.0-20251031123324-a804aaa87491 // indirect
github.com/tidwall/gjson v1.18.0 // indirect
github.com/tidwall/match v1.1.1 // indirect
@@ -263,7 +264,6 @@ require (
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 // indirect
golang.org/x/net v0.47.0 // indirect
golang.org/x/oauth2 v0.33.0 // indirect
golang.org/x/sys v0.38.0 // indirect
golang.org/x/text v0.31.0 // indirect
golang.org/x/time v0.14.0 // indirect
golang.org/x/tools v0.38.0 // indirect
internal/cli/app.go
+13 -18
View File
@@ -200,10 +200,10 @@ func RunApp(ctx context.Context, app *fx.App) error {
}
// errReported marks a failure the operation has already shown the user
// (and deliberately withheld under --json). Entry turns it into a
// non-zero exit status without printing anything further, so the error
// line is not doubled. It flows up from RunOperation through cobra to
// Entry.
// (or, under `snapshot verify --json`, put in its document). Entry
// turns it into a non-zero exit status without printing anything
// further, so the error line is not doubled. It flows up from
// RunOperation through cobra to Entry.
var errReported = errors.New("operation failed")
// RunOperation runs op against the Vaultik instance inside the fx app
@@ -220,10 +220,10 @@ var errReported = errors.New("operation failed")
// interrupt OnStop cancels op and waits for the goroutine to return, so
// op's cleanup (removing decrypted scratch files) runs before the
// process exits; the wait is bounded by shutdownTimeout. report is
// called with a non-canceled failure so the caller can log it (and
// suppress it under --json) before it becomes errReported. A context
// cancellation is the interrupt path, not a failure: it is neither
// reported nor counted as one.
// called with a non-canceled failure so the caller can show it to the
// user before it becomes errReported. A context cancellation is the
// interrupt path, not a failure: it is neither reported nor counted as
// one.
func RunOperation(
ctx context.Context, opts AppOptions,
op func(v *vaultik.Vaultik) error, report func(err error),
@@ -293,13 +293,12 @@ func RunOperation(
// runVaultikApp runs the standard single-operation command lifecycle
// shared by the snapshot list/purge/remove and remote nuke subcommands:
// resolve the config, then run op against the Vaultik instance through
// RunOperation, reporting a failure prefixed with failMsg (suppressed
// while suppressErrors is true, e.g. under --json). mode says whether the
// command takes the PID lock. jsonOutput marks a command whose stdout is a
// JSON document: it quiets the UI but, unlike Quiet, leaves the stderr log
// level alone.
// RunOperation, reporting a failure prefixed with failMsg on stderr. mode
// says whether the command takes the PID lock. jsonOutput marks a command
// whose stdout is a JSON document: it quiets the UI but, unlike Quiet,
// leaves the stderr log level alone.
func runVaultikApp(
cmd *cobra.Command, mode lockMode, jsonOutput, suppressErrors bool,
cmd *cobra.Command, mode lockMode, jsonOutput bool,
failMsg string, op func(v *vaultik.Vaultik) error,
) error {
configPath, err := ResolveConfigPath()
@@ -319,10 +318,6 @@ func runVaultikApp(
},
Mode: mode,
}, op, func(err error) {
if suppressErrors {
return
}
log.Error(failMsg, "error", err)
ReportErrorf("%s: %v", failMsg, err)
})
internal/cli/config.go
+7 -4
View File
@@ -45,16 +45,19 @@ const defaultConfigTemplate = `# vaultik configuration
# ─── REQUIRED ────────────────────────────────────────────────────────────────
# Age recipient public keys for encryption.
# Age recipient public keys for encryption. snapshot create needs at least
# one; listing, verifying and restoring do not, so a machine that only
# restores can leave this empty.
# Backups are encrypted to ALL listed recipients; any one of the corresponding
# private keys can decrypt. Adding a recipient later does not re-encrypt data
# already stored: deduplicated chunks and existing blobs stay encrypted to the
# earlier recipients, so a newly added key cannot restore them on its own (see
# docs/REPOSTRUCTURE.md, Accepted Risks). Generate a keypair with:
# docs/REPOSTRUCTURE.md, Accepted Risks). Generate a keypair and add its
# public key with:
# age-keygen -o vaultik_backup_private_key.txt
# grep 'public key' vaultik_backup_private_key.txt
age_recipients:
- age1REPLACE_WITH_YOUR_PUBLIC_KEY
# vaultik config set age_recipients.0 age1...
age_recipients: []
# Named snapshots. Each snapshot backs up one or more paths and can have its
# own exclude patterns in addition to the global excludes below.
internal/cli/config_test.go
+41 -2
View File
@@ -24,8 +24,10 @@ func TestDefaultConfigTemplateParses(t *testing.T) {
t.Fatalf("default config template is not valid YAML: %v", err)
}
if len(cfg.AgeRecipients) != 1 {
t.Errorf("expected 1 placeholder age recipient, got %d", len(cfg.AgeRecipients))
// A placeholder recipient would fail config.Load, so the template
// leaves the list empty.
if len(cfg.AgeRecipients) != 0 {
t.Errorf("expected no age recipients, got %d", len(cfg.AgeRecipients))
}
home, ok := cfg.Snapshots["home"]
@@ -55,6 +57,43 @@ func TestDefaultConfigTemplateParses(t *testing.T) {
}
}
// TestConfigSetRecipientOnFreshConfig follows the README quickstart: on the
// file `config init` writes, `config set age_recipients.0` and
// `config set storage_url` give a config that loads with that recipient.
func TestConfigSetRecipientOnFreshConfig(t *testing.T) {
t.Parallel()
const recipient = "age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj"
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(defaultConfigTemplate), configFileMode)
if err != nil {
t.Fatalf("write config: %v", err)
}
out := ui.NewWithColor(&bytes.Buffer{}, false)
err = writeConfigSet(out, path, "age_recipients.0", recipient)
if err != nil {
t.Fatalf("config set age_recipients.0: %v", err)
}
err = writeConfigSet(out, path, "storage_url", "file:///mnt/backups")
if err != nil {
t.Fatalf("config set storage_url: %v", err)
}
cfg, err := config.Load(path)
if err != nil {
t.Fatalf("config.Load: %v", err)
}
if len(cfg.AgeRecipients) != 1 || cfg.AgeRecipients[0] != recipient {
t.Errorf("age_recipients = %v, want [%s]", cfg.AgeRecipients, recipient)
}
}
const testYAML = `# top comment
compression_level: 3
age_recipients:
internal/cli/entry.go
+10 -9
View File
@@ -16,16 +16,18 @@ import (
const shortCommitLen = 12
// Entry is the main entry point for the CLI application.
// It prints the startup banner to stdout (unless a banner-suppressing
// It prints the startup banner to stderr (unless a banner-suppressing
// flag is present in os.Args — see bannerSuppressedInArgs), executes the
// root cobra command, and routes any returned error through the
// ui.Writer so the user sees a properly formatted "🛑 ERROR:" line.
// The banner goes to stderr because stdout carries only the output the
// user asked for, such as a completion script or a `config get` value.
//
// It returns the process exit code (0 on success, 1 on error) rather
// than calling os.Exit, so that main's deferred profile writers run
// before the process ends. See run in cmd/vaultik/main.go.
func Entry() int {
emitStartupBanner(os.Args[1:], os.Stdout)
emitStartupBanner(os.Args[1:], os.Stderr)
rootCmd := NewRootCommand()
rootCmd.SilenceErrors = true
@@ -33,8 +35,9 @@ func Entry() int {
err := rootCmd.Execute()
if err != nil {
// An operation that ran inside the fx app has already reported
// its own failure (and suppressed it under --json); errReported
// says so. Printing it again here would double the error line.
// its own failure (`snapshot verify --json` puts it in the
// document instead); errReported says so. Printing it again
// here would double the error line.
// Every other error — bad arguments, a config that would not
// load — reaches Entry unreported, so it is shown here.
if !errors.Is(err, errReported) {
@@ -49,9 +52,8 @@ func Entry() int {
// emitStartupBanner writes the startup banner to w unless args (the
// argument vector with the program name already stripped) contains a
// flag that suppresses it. Split out of Entry so that the decision — the
// only thing standing between a --json invocation and a parseable
// stdout — is reachable from a test without running the whole CLI.
// flag that suppresses it. Split out of Entry so that the decision is
// reachable from a test without running the whole CLI.
func emitStartupBanner(args []string, w io.Writer) {
if bannerSuppressedInArgs(args) {
return
@@ -86,8 +88,7 @@ func ReportErrorf(format string, args ...any) {
// --json is a subcommand flag rather than a persistent one, but so is
// --cron (it exists only on `snapshot create`), so this adds no new
// class of imprecision. The only cost of a false positive is a missing
// decorative banner; the cost of a false negative is a corrupt document
// on stdout, so the scan errs deliberately in that direction.
// decorative banner.
func bannerSuppressedInArgs(args []string) bool {
for _, a := range args {
if a == "--" {
internal/cli/entry_banner_test.go
+25 -49
View File
@@ -36,23 +36,14 @@ const (
// strips it before scanning, so it has to be present.
programName = "vaultik"
// someSnapshotID is any snapshot identifier: these tests never run
// the command, so it only has to occupy the positional argument.
// someSnapshotID only fills the positional argument; no test needs
// the snapshot to exist.
someSnapshotID = "host_2026-01-01T00:00:00Z"
)
// placeholderJSONDocument stands in for whatever document a --json
// command writes to stdout. `snapshot list --json` with no snapshots
// prints exactly this; the other --json commands print an object rather
// than an array, but this test is not about their shape. It is about
// what is on stdout *before* them, which is the same for all of them
// because Entry prints the banner before cobra has parsed anything and
// therefore before it can know which command is running.
const placeholderJSONDocument = "[]\n"
// jsonArgumentVectors are the argument vectors of every --json
// invocation the CLI accepts, with the program name stripped exactly as
// Entry strips it. Each one must leave stdout untouched by the banner.
// Entry strips it. Each one must suppress the banner.
//
//nolint:gochecknoglobals // read-only test fixture shared by two tests
var jsonArgumentVectors = map[string][]string{
@@ -74,39 +65,23 @@ var jsonArgumentVectors = map[string][]string{
},
}
// TestJSONInvocationStdoutIsExactlyOneDocument is the CLI-layer
// regression guard for issue #106: `vaultik snapshot list --json | jq`
// must work with no other flags.
//
// internal/vaultik's TestListSnapshots_JSONStdoutIsOnlyTheDocument
// guards the same contract one layer down, but it calls the library
// function directly and so cannot see Entry, which is where the
// contamination was: the startup banner is written to stdout before
// cobra parses anything, and the suppression scan did not know about
// --json. The two banner lines and the blank line landed ahead of the
// document and `jq` refused the result.
//
// The document is a constant here because this test is about the
// argument vectors, one per --json command; the one that runs a real
// command end to end is TestEntryJSONStdoutIsExactlyOneDocument below.
func TestJSONInvocationStdoutIsExactlyOneDocument(t *testing.T) {
// TestJSONInvocationSuppressesBanner checks that every --json
// invocation suppresses the startup banner, as the README says --json
// does along with --quiet and --cron. The scan is over the raw argument
// vector, so each position and spelling of --json is listed.
func TestJSONInvocationSuppressesBanner(t *testing.T) {
t.Parallel()
for name, argv := range jsonArgumentVectors {
t.Run(name, func(t *testing.T) {
t.Parallel()
var stdout bytes.Buffer
var banner bytes.Buffer
emitStartupBanner(argv, &stdout)
emitStartupBanner(argv, &banner)
require.Empty(t, stdout.String(),
"nothing may reach stdout ahead of a --json document")
_, err := stdout.WriteString(placeholderJSONDocument)
require.NoError(t, err)
requireExactlyOneJSONDocument(t, stdout.String())
assert.Empty(t, banner.String(),
"--json suppresses the banner")
})
}
}
@@ -127,11 +102,11 @@ func TestBannerStillPrintedWithoutSuppressingFlag(t *testing.T) {
t.Run(name, func(t *testing.T) {
t.Parallel()
var stdout bytes.Buffer
var banner bytes.Buffer
emitStartupBanner(argv, &stdout)
emitStartupBanner(argv, &banner)
assert.Contains(t, stdout.String(), "starting up at",
assert.Contains(t, banner.String(), "starting up at",
"the banner belongs on invocations that did not opt out")
})
}
@@ -172,9 +147,9 @@ func TestBannerSuppressedInArgs(t *testing.T) {
// hermeticConfig is a complete, valid config that needs no network and
// no credentials: file:// storage is exempt from the S3 credential
// checks, and FileStorer over a directory that does not exist lists
// zero objects without erroring. Chunk, blob and compression settings
// are filled in by config.Load.
// checks. A test that lists the destination must create its directory
// first, because listing a directory that does not exist is an error.
// Chunk, blob and compression settings are filled in by config.Load.
const hermeticConfig = `age_recipients:
- age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj
snapshots:
@@ -196,22 +171,25 @@ hostname: test-host
// `snapshot list` is the command chosen because it is the only --json
// command that reaches its document without a populated destination
// store: it reads the local index, streams `metadata/` (empty here),
// and treats a barren destination as an empty list rather than a
// failure.
// and treats an empty destination directory as an empty list rather
// than a failure.
//
// Not parallel: it replaces os.Args, os.Stdout and the xdg globals.
func TestEntryJSONStdoutIsExactlyOneDocument(t *testing.T) {
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
storeDir := filepath.Join(dir, "store")
contents := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"),
filepath.Join(dir, "store"),
storeDir,
filepath.Join(dir, "index.sqlite"))
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
require.NoError(t, os.Mkdir(storeDir, 0o750))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
@@ -244,9 +222,7 @@ func TestEntryJSONStdoutIsExactlyOneDocument(t *testing.T) {
// captureProcessStdout redirects the process's own stdout to a pipe for
// the duration of fn and returns what was written to it. The redirection
// has to be at the file-descriptor level rather than through an injected
// writer, because the banner and the JSON encoder reach os.Stdout
// independently and the point of the test is that both land in the same
// place.
// writer, because the commands Entry runs reach os.Stdout directly.
//
// Not parallel-safe: os.Stdout is process-global.
func captureProcessStdout(t *testing.T, fn func()) string {
internal/cli/entry_prune_json_test.go
+10 -5
View File
@@ -98,25 +98,30 @@ func TestEntryPruneJSONStdoutIsExactlyOneDocument(t *testing.T) {
}
}
// writeHermeticPruneConfig builds a config over a temp directory and, if
// seedStale is set, creates the index database up front with one
// snapshot record that has no counterpart on the destination store.
// Returns the config path.
// writeHermeticPruneConfig builds a config over a temp directory with an
// empty destination directory and, if seedStale is set, creates the
// index database up front with one snapshot record that has no
// counterpart on the destination store. Returns the config path.
func writeHermeticPruneConfig(t *testing.T, seedStale bool) string {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
indexPath := filepath.Join(dir, "index.sqlite")
storeDir := filepath.Join(dir, "store")
contents := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"),
filepath.Join(dir, "store"),
storeDir,
indexPath)
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
// prune fails on a destination directory that does not exist, so the
// empty store is created here rather than left to a first backup.
require.NoError(t, os.Mkdir(storeDir, 0o750))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
internal/cli/entry_status_test.go
+3 -3
View File
@@ -15,8 +15,8 @@ import (
// run, so a failing command must come back with a non-zero code rather
// than ending the process here.
//
// Stdout is captured only to keep the banner and command output off the
// test log; the assertion is on the returned code.
// Stdout and stderr are captured only to keep the banner and command
// output off the test log; the assertion is on the returned code.
//
//nolint:paralleltest // replaces os.Args and rootFlags
func TestEntryReturnsStatusCode(t *testing.T) {
@@ -50,7 +50,7 @@ func TestEntryReturnsStatusCode(t *testing.T) {
var code int
_ = captureProcessStdout(t, func() { code = Entry() })
_, _ = captureProcessStdoutAndStderr(t, func() { code = Entry() })
assert.Equal(t, testCase.want, code)
})
internal/cli/entry_stdout_stderr_test.go
+186
View File
@@ -0,0 +1,186 @@
package cli //nolint:testpackage // shares hermeticConfig and the capture helpers
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/adrg/xdg"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
)
// TestEntryCompletionStdoutIsTheScript runs `vaultik completion bash`,
// whose stdout the README tells the user to source. The script has to
// start on the first line.
//
//nolint:paralleltest // replaces os.Args, os.Stdout and os.Stderr
func TestEntryCompletionStdoutIsTheScript(t *testing.T) {
code, stdout, _ := runEntry(t, "completion", "bash")
require.Equal(t, 0, code)
firstLine, _, _ := strings.Cut(stdout, "\n")
assert.True(t, strings.HasPrefix(firstLine, "# bash completion"),
"the first line of stdout must be the script's, got %q", firstLine)
}
// TestEntryConfigGetStdoutIsTheValue runs `vaultik config get`, whose
// stdout a script reads as the value and nothing else.
//
//nolint:paralleltest // replaces os.Args, os.Stdout and os.Stderr
func TestEntryConfigGetStdoutIsTheValue(t *testing.T) {
configPath := filepath.Join(t.TempDir(), "config.yml")
require.NoError(t, os.WriteFile(configPath,
[]byte("hostname: test-host\n"), configFileMode))
code, stdout, _ := runEntry(t,
flagConfig, configPath, "config", "get", "hostname")
require.Equal(t, 0, code)
assert.Equal(t, "test-host\n", stdout)
}
// TestEntryJSONFailureIsReportedOnStderr runs each --json command that
// writes no document when it fails, against a destination it cannot
// use. The error must reach stderr, and stdout must stay empty.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntryJSONFailureIsReportedOnStderr(t *testing.T) {
for _, testCase := range []struct {
name string
args []string
wantOnStderr string
}{
{
name: "remote info",
args: []string{cmdRemote, cmdInfo, flagJSON},
wantOnStderr: "Failed to get remote info",
},
{
name: "prune",
args: []string{cmdPrune, flagJSON},
wantOnStderr: "Prune failed",
},
{
name: "snapshot remove",
args: []string{cmdSnapshot, cmdRemove, someSnapshotID, flagJSON},
wantOnStderr: "Failed to remove snapshot",
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath := writeUnusableDestinationConfig(t)
code, stdout, stderr := runEntry(t,
append([]string{flagConfig, configPath}, testCase.args...)...)
assert.Equal(t, 1, code)
assert.Empty(t, stdout,
"a failed --json command has no document to write")
assert.Contains(t, stderr, testCase.wantOnStderr,
"the failure must be reported on stderr")
})
}
}
// TestEntrySnapshotRemoveJSONWarningIsOnStderr runs `snapshot remove
// --json` on a snapshot in the local index, against a destination
// directory that does not exist. The command removes the snapshot from
// the local index and still exits 0. Its stdout must hold the document
// alone, with the warning about the destination store, and the command
// to run again once it is reachable, on stderr.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotRemoveJSONWarningIsOnStderr(t *testing.T) {
configPath, indexPath := writeMissingDestinationConfig(t)
seedStaleSnapshotRecord(t, indexPath)
code, stdout, stderr := runEntry(t, flagConfig, configPath,
cmdSnapshot, cmdRemove, stalePruneSnapshotID, flagJSON)
require.Equal(t, 0, code)
requireExactlyOneJSONDocument(t, stdout)
assert.Contains(t, stderr,
"Could not remove snapshot metadata from remote storage")
assert.Contains(t, stderr,
"run 'vaultik snapshot remove "+stalePruneSnapshotID+"' again")
}
// writeMissingDestinationConfig builds a config whose destination
// directory does not exist. Returns the config path and the path of
// its local index, which is not created here.
func writeMissingDestinationConfig(t *testing.T) (string, string) {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
indexPath := filepath.Join(dir, "index.sqlite")
contents := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"),
filepath.Join(dir, "missing-store"),
indexPath)
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
t.Setenv("XDG_DATA_HOME", filepath.Join(dir, "data"))
xdg.Reload()
t.Cleanup(xdg.Reload)
return configPath, indexPath
}
// writeUnusableDestinationConfig builds a config whose destination
// directory does not exist, which fails `remote info`, and whose local
// index is bound to another destination, which fails `prune` and
// `snapshot remove` (a missing destination alone only makes `snapshot
// remove` warn). Returns the config path.
func writeUnusableDestinationConfig(t *testing.T) string {
t.Helper()
configPath, indexPath := writeMissingDestinationConfig(t)
ctx := context.Background()
db, err := database.New(ctx, indexPath)
require.NoError(t, err)
defer func() { require.NoError(t, db.Close()) }()
require.NoError(t, database.NewRepositories(db).LocalMeta.Set(ctx,
database.LocalMetaKeyStorageURL, "file://"+t.TempDir()))
return configPath
}
// runEntry runs Entry with args after the program name and returns its
// exit code and what it wrote to stdout and stderr.
//
// Not parallel-safe: it replaces os.Args, os.Stdout and os.Stderr.
func runEntry(t *testing.T, args ...string) (int, string, string) {
t.Helper()
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
os.Args = append([]string{programName}, args...)
var code int
stdout, stderr := captureProcessStdoutAndStderr(t,
func() { code = Entry() })
return code, stdout, stderr
}
internal/cli/prune.go
-4
View File
@@ -48,10 +48,6 @@ work (e.g. after a crashed backup or to reclaim storage).`,
}, func(v *vaultik.Vaultik) error {
return v.Prune(opts)
}, func(err error) {
if opts.JSON {
return
}
log.Error("Prune operation failed", "error", err)
ReportErrorf("Prune failed: %v", err)
})
internal/cli/remote.go
+1 -5
View File
@@ -45,7 +45,7 @@ This is destructive and irreversible. Requires --force.`,
return errNukeNeedsForce
}
return runVaultikApp(cmd, mutating, false, false, "Remote nuke failed",
return runVaultikApp(cmd, mutating, false, "Remote nuke failed",
func(v *vaultik.Vaultik) error {
return v.NukeRemote(true)
})
@@ -92,10 +92,6 @@ func newRemoteInfoCommand() *cobra.Command {
}, func(v *vaultik.Vaultik) error {
return v.RemoteInfo(jsonOutput)
}, func(err error) {
if jsonOutput {
return
}
log.Error("Failed to get remote info", "error", err)
ReportErrorf("Failed to get remote info: %v", err)
})
internal/cli/root.go
+2 -1
View File
@@ -60,7 +60,8 @@ on the source system.`,
cmd.PersistentFlags().BoolVar(&rootFlags.SkipErrors, "skip-errors", false,
"Skip files that cannot be read when creating a snapshot, or "+
"that cannot be restored when restoring, instead of aborting "+
"(packing and storage errors still abort)")
"(packing and storage errors while creating a snapshot still "+
"abort)")
// Add subcommands
cmd.AddCommand(
internal/cli/snapshot.go
+6 -5
View File
@@ -126,7 +126,7 @@ func newSnapshotListCommand() *cobra.Command {
Long: "Lists all snapshots with their ID, timestamp, and compressed size",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
return runVaultikApp(cmd, readOnly, false, false,
return runVaultikApp(cmd, readOnly, false,
"Failed to list snapshots",
func(v *vaultik.Vaultik) error {
return v.ListSnapshots(jsonOutput)
@@ -162,7 +162,7 @@ restrict the operation to specific snapshot names.`,
return errPurgeCriteriaBoth
}
return runVaultikApp(cmd, mutating, false, false,
return runVaultikApp(cmd, mutating, false,
"Failed to purge snapshots",
func(v *vaultik.Vaultik) error {
return v.PurgeSnapshotsWithOptions(opts)
@@ -258,14 +258,15 @@ Use --local-only to skip the remote half (e.g. when you want to forget a
snapshot locally without touching the destination store).
If the remote is unreachable, the local-database removal still completes
and a warning is emitted; rerun 'vaultik prune' once the destination store
is reachable to finish remote cleanup.
and a warning is emitted; run 'vaultik snapshot remove <snapshot-id>' again
once the destination store is reachable to remove the snapshot's metadata
from it ('vaultik prune' does not).
To wipe the entire destination store and start over, use 'vaultik remote
nuke --force' — it is the single supported entry point for that.`,
Args: requireSnapshotIDArg,
RunE: func(cmd *cobra.Command, args []string) error {
return runVaultikApp(cmd, mutating, opts.JSON, opts.JSON,
return runVaultikApp(cmd, mutating, opts.JSON,
"Failed to remove snapshot",
func(v *vaultik.Vaultik) error {
_, err := v.RemoveSnapshot(args[0], opts)
internal/config/config.go
+5 -10
View File
@@ -42,9 +42,7 @@ const (
// Sentinel validation errors.
var (
errNoConfigPath = errors.New("config path not provided")
errNoAgeRecipients = errors.New(
"at least one age_recipient is required (generate with: age-keygen)")
errNoConfigPath = errors.New("config path not provided")
errRecipientIsSecretKey = errors.New(
"an age secret key was given where a public key (age1...) belongs")
errRecipientNotX25519 = errors.New(
@@ -323,9 +321,10 @@ func Load(path string) (*Config, error) {
// Validate checks if the configuration is valid and complete.
// It ensures all required fields are present and have valid values:
// - At least one age recipient must be specified, and every recipient must
// parse as an X25519 age1... public key (so a bad entry fails at load, not
// mid-backup); errors name the position, never the value
// - Every age recipient must parse as an X25519 age1... public key (so a
// bad entry fails at load, not mid-backup); errors name the position,
// never the value. An empty list is accepted, because only snapshot
// create needs a recipient and it checks for one itself
// - At least one snapshot must be configured with at least one path
// - Storage must be configured (either storage_url or s3.* fields)
// - Chunk size must be at least 1MB
@@ -336,10 +335,6 @@ func Load(path string) (*Config, error) {
//
// Returns an error describing the first validation failure encountered.
func (c *Config) Validate() error {
if len(c.AgeRecipients) == 0 {
return errNoAgeRecipients
}
for i, recipient := range c.AgeRecipients {
err := validateAgeRecipient(recipient)
if err != nil {
internal/config/config_test.go
+8 -2
View File
@@ -223,7 +223,8 @@ func TestValidateBlobSizeLimit(t *testing.T) {
// TestValidateAgeRecipients checks that recipients are parsed at config load
// (a bad entry fails immediately, not mid-backup) and that no invalid entry —
// least of all a pasted secret key — is echoed in the error.
// least of all a pasted secret key — is echoed in the error. An empty list
// loads, because only snapshot create needs a recipient.
func TestValidateAgeRecipients(t *testing.T) {
t.Parallel()
@@ -244,7 +245,12 @@ func TestValidateAgeRecipients(t *testing.T) {
wantErr bool
}{
{
name: "config init placeholder is rejected",
name: "no recipients is accepted",
recipients: nil,
wantErr: false,
},
{
name: "placeholder recipient is rejected",
recipients: []string{"age1REPLACE_WITH_YOUR_PUBLIC_KEY"},
wantErr: true,
},
internal/database/database.go
+39 -44
View File
@@ -42,6 +42,10 @@ var schemaFS embed.FS
// table itself. It is applied before the normal migration loop.
const bootstrapVersion = 0
// busyTimeoutMs is how long a connection to the index waits for another
// connection's lock before failing with "database is locked".
const busyTimeoutMs = 10000
// DB represents the Vaultik local index database connection.
// It uses SQLite to track file metadata, content-defined chunks, and blob associations.
// The database enables incremental backups by detecting changed files and
@@ -94,10 +98,27 @@ func ParseMigrationVersion(filename string) (int, error) {
return version, nil
}
// indexDSN returns the driver DSN that opens the database at path. The
// driver runs each _pragma parameter on every connection it opens and drops
// any parameter it does not know without an error, so a setting written in
// another form silently does nothing. In WAL mode one connection can write
// while others read; the busy timeout makes a connection wait for a lock
// instead of failing at once.
func indexDSN(path string) string {
return fmt.Sprintf(
"%s?_pragma=busy_timeout(%d)&_pragma=journal_mode(WAL)"+
"&_pragma=synchronous(NORMAL)&_pragma=foreign_keys(1)",
path, busyTimeoutMs)
}
// New creates a new database connection at the specified path.
// It creates the schema if needed and configures SQLite with WAL mode for
// better concurrency. SQLite handles crash recovery automatically when
// opening a database with journal/WAL files present.
// It creates the schema if needed. Every connection runs in WAL mode with
// a busy timeout and foreign keys on (see indexDSN), so a read-only command
// can read the index while a backup writes to it. Committed rows can sit in
// the -wal file beside the database until a checkpoint, so a copy of the
// database file alone may miss them.
// SQLite handles crash recovery automatically when opening a database with
// journal/WAL files present.
// The path parameter can be a file path for persistent storage or ":memory:"
// for an in-memory database (useful for testing).
func New(ctx context.Context, path string) (*DB, error) {
@@ -110,11 +131,7 @@ func New(ctx context.Context, path string) (*DB, error) {
// First attempt with standard WAL mode
log.Debug("Attempting to open database with WAL mode", "path", path)
conn, err := sql.Open(
"sqlite",
path+"?_journal_mode=WAL&_synchronous=NORMAL&_busy_timeout=10000"+
"&_locking_mode=NORMAL&_foreign_keys=ON",
)
conn, err := sql.Open("sqlite", indexDSN(path))
if err == nil {
configureConnPool(conn)
@@ -134,8 +151,8 @@ func New(ctx context.Context, path string) (*DB, error) {
_ = conn.Close()
}
// If first attempt failed, try with TRUNCATE mode to clear any locks
return openWithRecovery(ctx, path)
// If the first attempt failed, try once more
return retryOpen(ctx, path)
}
// configureConnPool serializes all database access through one connection.
@@ -147,18 +164,12 @@ func configureConnPool(conn *sql.DB) {
conn.SetMaxIdleConns(1)
}
// finishOpen enables foreign keys, wraps the connection, and applies any
// pending migrations. On migration failure the connection is closed.
// finishOpen wraps the connection and applies any pending migrations. On
// migration failure the connection is closed.
func finishOpen(ctx context.Context, conn *sql.DB, path string) (*DB, error) {
// Enable foreign keys explicitly
_, err := conn.ExecContext(ctx, "PRAGMA foreign_keys = ON")
if err != nil {
log.Warn("Failed to enable foreign keys", "path", path, "error", err)
}
db := &DB{conn: conn, path: path}
err = applyMigrations(ctx, conn)
err := applyMigrations(ctx, conn)
if err != nil {
_ = conn.Close()
@@ -168,21 +179,15 @@ func finishOpen(ctx context.Context, conn *sql.DB, path string) (*DB, error) {
return db, nil
}
// openWithRecovery retries opening the database in TRUNCATE journal mode to
// clear stale locks, then switches back to WAL mode.
func openWithRecovery(ctx context.Context, path string) (*DB, error) {
log.Info(
"Database appears locked, attempting recovery with TRUNCATE mode",
"path", path,
)
// retryOpen makes a second attempt to open the database, with the same
// settings, after the first attempt failed, for example because another
// process held a lock for longer than the busy timeout.
func retryOpen(ctx context.Context, path string) (*DB, error) {
log.Info("Database appears locked, retrying open", "path", path)
conn, err := sql.Open(
"sqlite",
path+"?_journal_mode=TRUNCATE&_synchronous=NORMAL&_busy_timeout=10000"+
"&_foreign_keys=ON",
)
conn, err := sql.Open("sqlite", indexDSN(path))
if err != nil {
return nil, fmt.Errorf("opening database in recovery mode: %w", err)
return nil, fmt.Errorf("opening database on retry: %w", err)
}
configureConnPool(conn)
@@ -190,28 +195,18 @@ func openWithRecovery(ctx context.Context, path string) (*DB, error) {
err = conn.PingContext(ctx)
if err != nil {
log.Debug(
"Failed to ping database in recovery mode, closing",
"Failed to ping database on retry, closing",
"path", path, "error", err,
)
_ = conn.Close()
return nil, fmt.Errorf(
"database still locked after recovery attempt: %w",
"database still locked on retry: %w",
err,
)
}
log.Debug("Database opened in TRUNCATE mode", "path", path)
// Switch back to WAL mode
log.Debug("Switching database back to WAL mode", "path", path)
_, err = conn.ExecContext(ctx, "PRAGMA journal_mode=WAL")
if err != nil {
log.Warn("Failed to switch back to WAL mode", "path", path, "error", err)
}
db, err := finishOpen(ctx, conn, path)
if err != nil {
return nil, err
internal/database/database_test.go
+83
View File
@@ -120,6 +120,89 @@ func TestDatabaseConcurrentAccess(t *testing.T) {
}
}
// TestNewSetsJournalModeAndBusyTimeout checks that the connection settings
// New passes reach SQLite. The driver drops a setting it does not recognise
// without an error, so only reading the value back shows it took effect.
func TestNewSetsJournalModeAndBusyTimeout(t *testing.T) {
t.Parallel()
ctx := context.Background()
db, err := New(ctx, filepath.Join(t.TempDir(), "index.db"))
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
var journalMode string
err = db.conn.QueryRowContext(ctx, "PRAGMA journal_mode").Scan(&journalMode)
if err != nil {
t.Fatalf("reading journal_mode: %v", err)
}
if journalMode != "wal" {
t.Errorf("journal_mode = %q, want %q", journalMode, "wal")
}
var busyTimeout int
err = db.conn.QueryRowContext(ctx, "PRAGMA busy_timeout").Scan(&busyTimeout)
if err != nil {
t.Fatalf("reading busy_timeout: %v", err)
}
if busyTimeout != busyTimeoutMs {
t.Errorf("busy_timeout = %d, want %d", busyTimeout, busyTimeoutMs)
}
}
// TestNewWriteSucceedsWhileAnotherHandleReads opens the same index twice,
// as a read-only command does while a backup runs, and checks that a write
// on one handle commits while the other is in the middle of a read.
func TestNewWriteSucceedsWhileAnotherHandleReads(t *testing.T) {
t.Parallel()
ctx := context.Background()
dbPath := filepath.Join(t.TempDir(), "index.db")
reader, err := New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to open reading handle: %v", err)
}
defer func() { _ = reader.Close() }()
writer, err := New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to open writing handle: %v", err)
}
defer func() { _ = writer.Close() }()
// The read lock taken by the SELECT is held until the transaction ends.
readTx, err := reader.BeginTx(ctx, nil)
if err != nil {
t.Fatalf("beginning read transaction: %v", err)
}
defer func() { _ = readTx.Rollback() }()
var count int
err = readTx.QueryRowContext(ctx, "SELECT COUNT(*) FROM chunks").Scan(&count)
if err != nil {
t.Fatalf("reading chunks: %v", err)
}
_, err = writer.ExecWithLog(ctx,
"INSERT INTO chunks (chunk_hash, size) VALUES (?, ?)", "hash", 1024)
if err != nil {
t.Fatalf("write while another handle reads: %v", err)
}
}
func TestParseMigrationVersion(t *testing.T) {
t.Parallel()
internal/database/files.go
+61 -6
View File
@@ -228,19 +228,24 @@ func (r *FileRepository) DeleteByID(
return nil
}
// ListByPrefix returns all files whose path starts with prefix, ordered by
// path.
func (r *FileRepository) ListByPrefix(
ctx context.Context, prefix string,
// ListUnderPath returns the file at path and every file beneath it,
// ordered by path. Paths are compared case-sensitively, and a trailing
// slash on path is ignored, so "/" lists every file.
func (r *FileRepository) ListUnderPath(
ctx context.Context, path string,
) ([]*File, error) {
path = strings.TrimRight(path, "/")
dirPrefix := path + "/"
// LIKE would ignore ASCII case and treat _ and % in path as wildcards.
query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
FROM files
WHERE path LIKE ? || '%'
WHERE path = ? OR substr(path, 1, length(?)) = ?
ORDER BY path
`
rows, err := r.db.conn.QueryContext(ctx, query, prefix)
rows, err := r.db.conn.QueryContext(ctx, query, path, dirPrefix, dirPrefix)
if err != nil {
return nil, fmt.Errorf("querying files: %w", err)
}
@@ -266,6 +271,56 @@ func (r *FileRepository) ListByPrefix(
return files, rows.Err()
}
// ListIDsWithChunksNotInUploadedBlobs returns the IDs of the files whose
// path starts with prefix and that list at least one chunk held by no
// blob whose upload has completed (uploaded_ts set). A new snapshot
// cannot reference such a chunk, so a backup must not treat the file as
// unchanged even when its metadata matches the file on disk.
func (r *FileRepository) ListIDsWithChunksNotInUploadedBlobs(
ctx context.Context, prefix string,
) ([]types.FileID, error) {
query := `
SELECT DISTINCT f.id
FROM files f
JOIN file_chunks fc ON fc.file_id = f.id
WHERE f.path LIKE ? || '%'
AND NOT EXISTS (
SELECT 1
FROM blob_chunks bc
JOIN blobs b ON bc.blob_id = b.id
WHERE bc.chunk_hash = fc.chunk_hash
AND b.uploaded_ts IS NOT NULL
)
`
rows, err := r.db.conn.QueryContext(ctx, query, prefix)
if err != nil {
return nil, fmt.Errorf("querying files: %w", err)
}
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var ids []types.FileID
for rows.Next() {
var id types.FileID
err := rows.Scan(&id)
if err != nil {
return nil, fmt.Errorf("scanning file ID: %w", err)
}
ids = append(ids, id)
}
return ids, rows.Err()
}
// ListAll returns all files in the database
func (r *FileRepository) ListAll(ctx context.Context) ([]*File, error) {
query := `
internal/database/files_test.go
+78
View File
@@ -5,10 +5,12 @@ import (
"database/sql"
"errors"
"os"
"slices"
"testing"
"time"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
)
// errTestRollback is the sentinel returned from transaction bodies to
@@ -134,6 +136,82 @@ func TestFileRepositoryListDelete(t *testing.T) {
}
}
func TestFileRepositoryListUnderPath(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewFileRepository(db)
const (
docDir = "/home/u/doc"
docFile = "/home/u/doc/a.txt"
)
// In path order, so the root case can expect all of them as listed.
paths := []string{
"/home/u/50%/x.txt",
"/home/u/50percent/y.txt",
"/home/u/DOC/c.txt",
"/home/u/a_b/x.txt",
"/home/u/axb/y.txt",
docDir,
"/home/u/doc.txt.bak",
docFile,
"/home/u/doc/sub/b.txt",
"/home/u/doc2/b.txt",
}
for _, path := range paths {
err := repo.Create(ctx, nil, &database.File{
Path: types.FilePath(path),
MTime: time.Now().Truncate(time.Second),
Mode: 0644,
})
if err != nil {
t.Fatalf("failed to create %s: %v", path, err)
}
}
docTree := []string{docDir, docFile, "/home/u/doc/sub/b.txt"}
tests := []struct {
name string
path string
want []string
}{
{"directory", docDir, docTree},
{"directory with trailing slash", docDir + "/", docTree},
{"directory differing only in case", "/home/u/DOC",
[]string{"/home/u/DOC/c.txt"}},
{"file", docFile, []string{docFile}},
{"underscore is literal", "/home/u/a_b",
[]string{"/home/u/a_b/x.txt"}},
{"percent is literal", "/home/u/50%",
[]string{"/home/u/50%/x.txt"}},
{"root", "/", paths},
}
for _, tt := range tests {
files, err := repo.ListUnderPath(ctx, tt.path)
if err != nil {
t.Fatalf("%s: failed to list files: %v", tt.name, err)
}
got := make([]string, 0, len(files))
for _, f := range files {
got = append(got, f.Path.String())
}
if !slices.Equal(got, tt.want) {
t.Errorf("%s: listing %q got %q, want %q",
tt.name, tt.path, got, tt.want)
}
}
}
func TestFileRepositorySymlink(t *testing.T) {
t.Parallel()
internal/database/models.go
+2 -2
View File
@@ -99,8 +99,8 @@ type Snapshot struct {
StartedAt time.Time
CompletedAt *time.Time // nil if still in progress
FileCount int64
ChunkCount int64
BlobCount int64
ChunkCount int64 // Chunks this snapshot stored that were not stored before
BlobCount int64 // Blobs this snapshot created
TotalSize int64 // Total size of all referenced files
// BlobSize is the total size of all referenced blobs (compressed and
internal/database/repository_comprehensive_test.go
+2 -2
View File
@@ -824,7 +824,7 @@ func TestTransactionIsolation(t *testing.T) {
}
// Verify the file was not created (transaction rolled back)
files, err := repos.Files.ListByPrefix(ctx, "/tx-test")
files, err := repos.Files.ListUnderPath(ctx, "/tx-test.txt")
if err != nil {
t.Fatal(err)
}
@@ -916,7 +916,7 @@ func TestConcurrentOrphanedCleanup(t *testing.T) {
}
// Verify correct files were deleted
files, err := repos.Files.ListByPrefix(ctx, "/concurrent-")
files, err := repos.Files.ListAll(ctx)
if err != nil {
t.Fatal(err)
}
internal/database/repository_debug_test.go
+1 -1
View File
@@ -147,7 +147,7 @@ func TestOrphanedFileCleanupDebug(t *testing.T) {
t.Logf("Files count after cleanup: %d", count)
// List remaining files
files, err := repos.Files.ListByPrefix(ctx, "/")
files, err := repos.Files.ListUnderPath(ctx, "/")
if err != nil {
t.Fatal(err)
}
internal/database/repository_edge_cases_test.go
+4 -4
View File
@@ -442,12 +442,12 @@ func TestLargeDatasets(t *testing.T) {
createLargeDatasetFiles(t, repos, snapshot.ID.String(), fileCount)
})
// Test ListByPrefix performance
// Test ListUnderPath performance
//nolint:paralleltest // phases share one database and are order-dependent
t.Run("list by prefix performance", func(t *testing.T) {
t.Run("list under path performance", func(t *testing.T) {
start := time.Now()
files, err := repos.Files.ListByPrefix(ctx, "/large/")
files, err := repos.Files.ListUnderPath(ctx, "/large/")
if err != nil {
t.Fatal(err)
}
@@ -472,7 +472,7 @@ func TestLargeDatasets(t *testing.T) {
t.Logf("Cleaned up orphaned files in %v", time.Since(start))
// Verify correct number remain
files, err := repos.Files.ListByPrefix(ctx, "/large/")
files, err := repos.Files.ListUnderPath(ctx, "/large/")
if err != nil {
t.Fatal(err)
}
internal/database/snapshots.go
+28 -3
View File
@@ -127,6 +127,7 @@ func (r *SnapshotRepository) UpdateExtendedStats(
snapshotID string,
blobUncompressedSize int64,
compressionLevel int,
uploadBytes int64,
uploadDurationMs int64,
) error {
compressionRatio, err := r.extendedCompressionRatio(
@@ -141,7 +142,7 @@ func (r *SnapshotRepository) UpdateExtendedStats(
SET blob_uncompressed_size = ?,
compression_ratio = ?,
compression_level = ?,
upload_bytes = blob_size,
upload_bytes = ?,
upload_duration_ms = ?
WHERE id = ?
`
@@ -149,11 +150,11 @@ func (r *SnapshotRepository) UpdateExtendedStats(
if tx != nil {
_, err = tx.ExecContext(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel,
uploadDurationMs, snapshotID)
uploadBytes, uploadDurationMs, snapshotID)
} else {
_, err = r.db.ExecWithLog(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel,
uploadDurationMs, snapshotID)
uploadBytes, uploadDurationMs, snapshotID)
}
if err != nil {
@@ -543,6 +544,30 @@ func (r *SnapshotRepository) GetSnapshotTotalCompressedSize(
return totalSize, nil
}
// GetSnapshotBlobSizes returns the total compressed and uncompressed sizes
// of all blobs referenced by a snapshot.
func (r *SnapshotRepository) GetSnapshotBlobSizes(
ctx context.Context, snapshotID string,
) (int64, int64, error) {
query := `
SELECT COALESCE(SUM(b.compressed_size), 0),
COALESCE(SUM(b.uncompressed_size), 0)
FROM snapshot_blobs sb
JOIN blobs b ON sb.blob_hash = b.blob_hash
WHERE sb.snapshot_id = ?
`
var compressed, uncompressed int64
err := r.db.conn.QueryRowContext(ctx, query, snapshotID).Scan(
&compressed, &uncompressed)
if err != nil {
return 0, 0, fmt.Errorf("querying snapshot blob sizes: %w", err)
}
return compressed, uncompressed, nil
}
// GetSnapshotUncompressedChunkSize returns the sum of plaintext sizes of all unique
// chunks referenced by a snapshot (via snapshot_files → file_chunks → chunks).
func (r *SnapshotRepository) GetSnapshotUncompressedChunkSize(
internal/database/snapshots_test.go
+59
View File
@@ -145,6 +145,65 @@ func TestSnapshotRepositoryUpdateCounts(t *testing.T) {
}
}
// GetSnapshotBlobSizes totals the blobs the snapshot references, and only
// those.
func TestSnapshotRepositoryGetSnapshotBlobSizes(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := database.NewRepositories(db)
snapshot := &database.Snapshot{
ID: "2024-01-03T12:00:00Z",
Hostname: testHostname,
VaultikVersion: testVersion,
StartedAt: time.Now().Truncate(time.Second),
}
err := repos.Snapshots.Create(ctx, nil, snapshot)
if err != nil {
t.Fatalf("failed to create snapshot: %v", err)
}
blobs := []*database.Blob{
{Hash: "referenced-1", CompressedSize: 10, UncompressedSize: 100},
{Hash: "referenced-2", CompressedSize: 20, UncompressedSize: 200},
{Hash: "unreferenced", CompressedSize: 40, UncompressedSize: 400},
}
for _, blob := range blobs {
blob.ID = types.NewBlobID()
blob.CreatedTS = time.Now().Truncate(time.Second)
err = repos.Blobs.Create(ctx, nil, blob)
if err != nil {
t.Fatalf("failed to create blob %s: %v", blob.Hash, err)
}
}
for _, blob := range blobs[:2] {
err = repos.Snapshots.AddBlob(ctx, nil, snapshot.ID.String(),
blob.ID, blob.Hash)
if err != nil {
t.Fatalf("failed to add blob %s to snapshot: %v", blob.Hash, err)
}
}
compressed, uncompressed, err := repos.Snapshots.GetSnapshotBlobSizes(
ctx, snapshot.ID.String())
if err != nil {
t.Fatalf("failed to get snapshot blob sizes: %v", err)
}
if compressed != 30 || uncompressed != 300 {
t.Errorf("blob sizes: got %d and %d, want 30 and 300",
compressed, uncompressed)
}
}
func TestSnapshotRepositoryListRecent(t *testing.T) {
t.Parallel()
internal/database/uploads.go
-16
View File
@@ -158,19 +158,3 @@ type UploadStats struct {
MinDurationMs int64
MaxDurationMs int64
}
// GetCountBySnapshot returns the count of uploads for a specific snapshot
func (r *UploadRepository) GetCountBySnapshot(
ctx context.Context, snapshotID string,
) (int64, error) {
query := `SELECT COUNT(*) FROM uploads WHERE snapshot_id = ?`
var count int64
err := r.conn.QueryRowContext(ctx, query, snapshotID).Scan(&count)
if err != nil {
return 0, err
}
return count, nil
}
internal/globals/globals.go
+10 -6
View File
@@ -18,14 +18,19 @@ var Appname = "vaultik" //nolint:gochecknoglobals // set via -ldflags at build t
// deliberately not a number.
const DevVersion = "dev"
// Unknown is what Commit and CommitDate hold when the build did not
// stamp them, and the version script/docker and script/cibuild stamp
// when the host has no git checkout.
const Unknown = "unknown"
// Version is the application version, populated from main().
var Version = DevVersion //nolint:gochecknoglobals // set via -ldflags at build time
// Commit is the git commit hash, populated from main().
var Commit = "unknown" //nolint:gochecknoglobals // set via -ldflags at build time
var Commit = Unknown //nolint:gochecknoglobals // set via -ldflags at build time
// CommitDate is the ISO-8601 date of the commit, populated from main().
var CommitDate = "unknown" //nolint:gochecknoglobals // set via -ldflags at build time
var CommitDate = Unknown //nolint:gochecknoglobals // set via -ldflags at build time
// Author identifies the upstream author of vaultik.
const Author = "Jeffrey Paul <sneak@sneak.berlin>"
@@ -72,11 +77,10 @@ func New() (*Globals, error) {
// safe reading of "we could not establish that this is a release" is
// that it is not one. The Makefile refuses to build at all in that
// case; this is the second line of defence, for a binary linked by
// something other than the Makefile. "unknown" counts for the same
// reason: script/docker and script/cibuild stamp it when the host has
// no git checkout.
// something other than the Makefile. Unknown counts for the same
// reason.
func IsDevVersion(v string) bool {
if v == "" || v == "unknown" || v == DevVersion ||
if v == "" || v == Unknown || v == DevVersion ||
strings.HasPrefix(v, DevVersion+"-") ||
strings.HasSuffix(v, "-dirty") {
return true
internal/s3/client.go
+11 -1
View File
@@ -6,6 +6,7 @@ import (
"context"
"errors"
"io"
"strings"
"sync/atomic"
"github.com/aws/aws-sdk-go-v2/aws"
@@ -30,6 +31,8 @@ type Client struct {
// Config contains S3 client configuration.
// All fields are required except Prefix, which defaults to an empty string.
// A non-empty Prefix is joined to every key with one "/", whether or not
// it ends with one.
// The Endpoint field should include the protocol (http:// or https://).
type Config struct {
Endpoint string
@@ -75,10 +78,17 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) {
s3Client := s3.NewFromConfig(awsCfg, s3Opts)
// Every method below builds a key as prefix + key, so the prefix
// must carry its own trailing "/".
prefix := strings.TrimRight(cfg.Prefix, "/")
if prefix != "" {
prefix += "/"
}
return &Client{
s3Client: s3Client,
bucket: cfg.Bucket,
prefix: cfg.Prefix,
prefix: prefix,
endpoint: cfg.Endpoint,
}, nil
}
internal/snapshot/backup_test.go
+1 -1
View File
@@ -71,7 +71,7 @@ func verifyBackupFiles(
) {
t.Helper()
files, err := repos.Files.ListByPrefix(ctx, "")
files, err := repos.Files.ListAll(ctx)
if err != nil {
t.Fatalf("Failed to list files: %v", err)
}
internal/snapshot/export_perm_test.go
+16 -8
View File
@@ -1,40 +1,48 @@
//nolint:testpackage // exercises the unexported copyFile helper
//nolint:testpackage // exercises the unexported copyDatabase helper
package snapshot
import (
"context"
"os"
"path/filepath"
"syscall"
"testing"
"github.com/spf13/afero"
"sneak.berlin/go/vaultik/internal/database"
)
// TestCopyFileExportCopyMode verifies that the exported snapshot database
// copy is created owner-only (0600), even under a lenient 022 umask that
// would otherwise leave a fresh file world-readable.
// TestCopyDatabaseExportCopyMode verifies that the exported snapshot
// database copy is created owner-only (0600), even under a lenient 022
// umask that would otherwise leave a fresh file world-readable.
//
//nolint:paralleltest // syscall.Umask is process-global; parallel tests would clash
func TestCopyFileExportCopyMode(t *testing.T) {
func TestCopyDatabaseExportCopyMode(t *testing.T) {
restore := syscall.Umask(0o022)
defer syscall.Umask(restore)
ctx := context.Background()
dir := t.TempDir()
src := filepath.Join(dir, "index.sqlite")
err := os.WriteFile(src, []byte("index data"), 0o600)
db, err := database.New(ctx, src)
if err != nil {
t.Fatalf("creating source index: %v", err)
}
err = db.Close()
if err != nil {
t.Fatalf("closing source index: %v", err)
}
dst := filepath.Join(dir, "snapshot.db")
sm := &SnapshotManager{fs: afero.NewOsFs()}
err = sm.copyFile(src, dst)
err = sm.copyDatabase(ctx, src, dst)
if err != nil {
t.Fatalf("copyFile: %v", err)
t.Fatalf("copyDatabase: %v", err)
}
info, err := os.Stat(dst)
internal/snapshot/progress.go
-4
View File
@@ -66,7 +66,6 @@ type ProgressStats struct {
BlobsCreated atomic.Int64
BlobsUploaded atomic.Int64
BytesUploaded atomic.Int64
UploadDurationMs atomic.Int64 // Total milliseconds spent uploading
CurrentFile atomic.Value // stores string
TotalSize atomic.Int64 // Total size to process (set after scan phase)
TotalFiles atomic.Int64 // Total files to process in phase 2
@@ -231,9 +230,6 @@ func (pr *ProgressReporter) ReportUploadComplete(
// Clear current upload
pr.stats.CurrentUpload.Store((*UploadInfo)(nil))
// Add to total upload duration
pr.stats.UploadDurationMs.Add(duration.Milliseconds())
// Calculate speed
if duration < time.Millisecond {
duration = time.Millisecond
internal/snapshot/scanner.go
+105 -99
View File
@@ -11,6 +11,7 @@ import (
"runtime"
"strings"
"sync"
"syscall"
"time"
"github.com/dustin/go-humanize"
@@ -73,6 +74,11 @@ type Scanner struct {
knownChunks map[string]struct{}
knownChunksMu sync.RWMutex
// filesToRechunk holds the IDs of known files that list a chunk no
// uploaded blob holds; they are re-chunked even when their metadata
// is unchanged.
filesToRechunk map[types.FileID]struct{}
// Pending chunk hashes - chunks that have been added to packer but not
// yet committed to DB. When a blob finalizes, the committed chunks are
// removed from this set.
@@ -86,9 +92,6 @@ type Scanner struct {
// Mutex for coordinating blob creation
packerMu sync.Mutex // Blocks chunk production during blob creation
// Context for cancellation
scanCtx context.Context //nolint:containedctx // set per-Scan for packer callbacks
}
// Periodic status output intervals and thresholds for the scan and
@@ -128,18 +131,23 @@ type ScannerConfig struct {
SkipErrors bool
}
// ScanResult contains the results of a scan operation
// ScanResult contains the results of a scan operation. Files and bytes
// are counted per file: BytesScanned is the size of the new and changed
// files, BytesSkipped that of the unchanged ones.
type ScanResult struct {
FilesScanned int
FilesSkipped int
FilesDeleted int
BytesScanned int64
BytesSkipped int64
BytesDeleted int64
ChunksCreated int
BlobsCreated int
StartTime time.Time
EndTime time.Time
FilesScanned int
FilesSkipped int
FilesDeleted int
BytesScanned int64
BytesSkipped int64
BytesDeleted int64
ChunksCreated int
BlobsCreated int
BlobsUploaded int
BytesUploaded int64
UploadDuration time.Duration
StartTime time.Time
EndTime time.Time
}
// NewScanner creates a new scanner instance
@@ -205,7 +213,6 @@ func (s *Scanner) Scan(
s.snapshotID = snapshotID
// Store source path for file records (used during restore)
s.currentSourcePath = path
s.scanCtx = ctx
result := &ScanResult{
StartTime: time.Now().UTC(),
}
@@ -213,17 +220,13 @@ func (s *Scanner) Scan(
// Set blob handler for concurrent upload
if s.storage != nil {
log.Debug("Setting blob handler for storage uploads")
s.packer.SetBlobHandler(s.handleBlobReady)
s.packer.SetBlobHandler(func(blobWithReader *blob.WithReader) error {
return s.handleBlobReady(ctx, blobWithReader, result)
})
} else {
log.Debug("No storage configured, blobs will not be uploaded")
}
// Start progress reporting if enabled
if s.progress != nil {
s.progress.Start()
defer s.progress.Stop()
}
// Phase 0: Repair any state left by an interrupted previous run, then
// load known files and chunks from the database into memory for fast
// lookup.
@@ -288,13 +291,14 @@ func (s *Scanner) Scan(
log.Info("Phase 2/3: Skipping (no files need processing, metadata-only snapshot)")
}
// Finalize result with blob statistics
s.finalizeScanResult(ctx, result)
result.EndTime = time.Now().UTC()
return result, nil
}
// GetProgress returns the progress reporter for this scanner
// GetProgress returns the progress reporter for this scanner, or nil when
// progress is off. Scan neither starts nor stops it: the caller does,
// once for all the paths it scans, because a second Stop panics.
func (s *Scanner) GetProgress() *ProgressReporter {
return s.progress
}
@@ -325,9 +329,42 @@ func (s *Scanner) loadDatabaseState(
s.ui.Completef("Loaded %s known chunks from local index database.",
s.ui.Count(len(s.knownChunks)))
err = s.loadFilesToRechunk(ctx, path)
if err != nil {
return nil, fmt.Errorf("loading files to re-chunk: %w", err)
}
return knownFiles, nil
}
// loadFilesToRechunk loads the IDs of known files under path that list a
// chunk no uploaded blob holds. A file row is shared by every snapshot
// that lists the file and is updated in place when the file changes,
// while a blob row is deleted once no snapshot references it. Removing
// the only snapshot that references a changed file's current blob
// therefore leaves an older snapshot keeping a file row whose metadata
// matches the disk while no blob the local index records as uploaded
// holds its chunks, so a new snapshot cannot reference them. The dropped
// blob can still be in remote storage until prune removes it.
func (s *Scanner) loadFilesToRechunk(ctx context.Context, path string) error {
ids, err := s.repos.Files.ListIDsWithChunksNotInUploadedBlobs(ctx, path)
if err != nil {
return fmt.Errorf("listing files: %w", err)
}
s.filesToRechunk = make(map[types.FileID]struct{}, len(ids))
for _, id := range ids {
s.filesToRechunk[id] = struct{}{}
}
if len(ids) > 0 {
log.Info("Re-chunking known files whose chunks are not all "+
"in uploaded blobs", "files", len(ids))
}
return nil
}
// repairInterruptedBlobs discards blob rows left by a previous run whose
// upload never completed. Such a blob has its chunks, blob_chunks, and
// blobs rows committed to the local index before the upload is attempted,
@@ -396,35 +433,16 @@ func (s *Scanner) summarizeScanPhase(
s.ui.Completef("%s.", msg)
}
// finalizeScanResult populates final blob statistics in the scan result
// by querying the packer and database for blob/upload counts
func (s *Scanner) finalizeScanResult(ctx context.Context, result *ScanResult) {
blobs := s.packer.GetFinishedBlobs()
result.BlobsCreated += len(blobs)
// Query database for actual blob count created during this snapshot
// The database is authoritative, especially for concurrent blob uploads
// We count uploads rather than all snapshot_blobs to get only NEW blobs
if s.snapshotID != "" {
uploadCount, err := s.repos.Uploads.GetCountBySnapshot(ctx, s.snapshotID)
if err != nil {
log.Warn("Failed to query upload count from database", "error", err)
} else {
result.BlobsCreated = int(uploadCount)
}
}
result.EndTime = time.Now().UTC()
}
// loadKnownFiles loads all known files from the database into a map for fast lookup
// This avoids per-file database queries during the scan phase
// loadKnownFiles loads the known files at and beneath path from the
// database into a map for fast lookup. Every loaded file the scan does
// not find is counted as deleted. This avoids per-file database queries
// during the scan phase.
func (s *Scanner) loadKnownFiles(
ctx context.Context, path string,
) (map[string]*database.File, error) {
files, err := s.repos.Files.ListByPrefix(ctx, path)
files, err := s.repos.Files.ListUnderPath(ctx, path)
if err != nil {
return nil, fmt.Errorf("listing files by prefix: %w", err)
return nil, fmt.Errorf("listing files under %s: %w", path, err)
}
result := make(map[string]*database.File, len(files))
@@ -1104,12 +1122,9 @@ func (s *Scanner) buildSymlinkEntry(path string, info os.FileInfo) *database.Fil
}
var uid, gid uint32
if stat, ok := info.Sys().(interface {
Uid() uint32
Gid() uint32
}); ok {
uid = stat.Uid()
gid = stat.Gid()
if stat, ok := info.Sys().(*syscall.Stat_t); ok {
uid = stat.Uid
gid = stat.Gid
}
return &database.File{
@@ -1128,12 +1143,9 @@ func (s *Scanner) buildSymlinkEntry(path string, info os.FileInfo) *database.Fil
// buildDirectoryEntry creates a File record for a directory.
func (s *Scanner) buildDirectoryEntry(path string, info os.FileInfo) *database.File {
var uid, gid uint32
if stat, ok := info.Sys().(interface {
Uid() uint32
Gid() uint32
}); ok {
uid = stat.Uid()
gid = stat.Gid()
if stat, ok := info.Sys().(*syscall.Stat_t); ok {
uid = stat.Uid
gid = stat.Gid
}
return &database.File{
@@ -1166,16 +1178,10 @@ func (s *Scanner) recordNonRegularFile(ctx context.Context, ftp *FileToProcess)
func (s *Scanner) checkFileInMemory(
path string, info os.FileInfo, knownFiles map[string]*database.File,
) (*database.File, bool) {
// Get file stats
stat, ok := info.Sys().(interface {
Uid() uint32
Gid() uint32
})
var uid, gid uint32
if ok {
uid = stat.Uid()
gid = stat.Gid()
if stat, ok := info.Sys().(*syscall.Stat_t); ok {
uid = stat.Uid
gid = stat.Gid
}
// Check against in-memory map first to get existing ID if available
@@ -1208,6 +1214,11 @@ func (s *Scanner) checkFileInMemory(
return file, true
}
// No uploaded blob holds one of its chunks (see loadFilesToRechunk)
if _, rechunk := s.filesToRechunk[fileID]; rechunk {
return file, true
}
// Check if file has changed
if existingFile.Size != file.Size ||
existingFile.MTime.Unix() != file.MTime.Unix() ||
@@ -1481,24 +1492,24 @@ func (s *Scanner) finalizeProcessPhase(ctx context.Context, result *ScanResult)
}
// handleBlobReady is called by the packer when a blob is finalized
func (s *Scanner) handleBlobReady(blobWithReader *blob.WithReader) error {
func (s *Scanner) handleBlobReady(
ctx context.Context, blobWithReader *blob.WithReader, result *ScanResult,
) error {
startTime := time.Now().UTC()
finishedBlob := blobWithReader.FinishedBlob
result.BlobsCreated++
if s.progress != nil {
s.progress.ReportUploadStart(finishedBlob.Hash, finishedBlob.Compressed)
s.progress.GetStats().BlobsCreated.Add(1)
}
ctx := s.scanCtx
if ctx == nil {
ctx = context.Background()
}
blobPath := fmt.Sprintf("blobs/%s/%s/%s",
finishedBlob.Hash[:2], finishedBlob.Hash[2:4], finishedBlob.Hash)
blobExists, err := s.uploadBlobIfNeeded(ctx, blobPath, blobWithReader, startTime)
blobExists, err := s.uploadBlobIfNeeded(
ctx, blobPath, blobWithReader, startTime, result)
if err != nil {
s.cleanupBlobTempFile(blobWithReader)
@@ -1533,6 +1544,7 @@ func (s *Scanner) uploadBlobIfNeeded(
blobPath string,
blobWithReader *blob.WithReader,
startTime time.Time,
result *ScanResult,
) (bool, error) {
finishedBlob := blobWithReader.FinishedBlob
@@ -1568,6 +1580,10 @@ func (s *Scanner) uploadBlobIfNeeded(
uploadDuration := time.Since(startTime)
uploadSpeedBps := float64(finishedBlob.Compressed) / uploadDuration.Seconds()
result.BlobsUploaded++
result.BytesUploaded += finishedBlob.Compressed
result.UploadDuration += uploadDuration
s.ui.Completef("Uploaded blob %s (%s) in %s at %s.",
s.ui.Hex(finishedBlob.Hash),
s.ui.Size(finishedBlob.Compressed),
@@ -1783,9 +1799,9 @@ func (s *Scanner) processFileStreaming(
size: chunk.Size,
})
s.updateChunkStats(chunkExists, chunk.Size, result)
if !chunkExists {
s.updateChunkStats(chunk.Size, result)
err := s.addChunkToPacker(ctx, chunk)
if err != nil {
// Mark as a packer error so --skip-errors cannot swallow it:
@@ -1813,26 +1829,16 @@ func (s *Scanner) processFileStreaming(
return nil
}
// updateChunkStats updates scan result and progress stats for a processed chunk
func (s *Scanner) updateChunkStats(
chunkExists bool, chunkSize int64, result *ScanResult,
) {
if chunkExists {
result.FilesSkipped++
// updateChunkStats counts a chunk that was not already stored. The scan
// result's file counts, BytesScanned and BytesSkipped are not touched
// here: the scan phase counts each file once.
func (s *Scanner) updateChunkStats(chunkSize int64, result *ScanResult) {
result.ChunksCreated++
result.BytesSkipped += chunkSize
if s.progress != nil {
s.progress.GetStats().BytesSkipped.Add(chunkSize)
}
} else {
result.ChunksCreated++
result.BytesScanned += chunkSize
if s.progress != nil {
s.progress.GetStats().ChunksCreated.Add(1)
s.progress.GetStats().BytesProcessed.Add(chunkSize)
s.progress.UpdateChunkingActivity()
}
if s.progress != nil {
s.progress.GetStats().ChunksCreated.Add(1)
s.progress.GetStats().BytesProcessed.Add(chunkSize)
s.progress.UpdateChunkingActivity()
}
}
internal/snapshot/scanner_test.go
+78 -1
View File
@@ -71,7 +71,7 @@ func verifySimpleScanDatabase(
t.Helper()
// Verify files in database - includes regular files and directories
files, err := repos.Files.ListByPrefix(ctx, "/source")
files, err := repos.Files.ListUnderPath(ctx, "/source")
if err != nil {
t.Fatalf("failed to list files: %v", err)
}
@@ -307,3 +307,80 @@ func TestScannerLargeFile(t *testing.T) {
}
}
}
// TestScannerRecordsOwnership backs up real files on disk and checks that
// the uid and gid of a file, a directory and a symlink are recorded.
// When the tests run as root both sides are 0, so only a run as another
// user can catch ownership recorded as 0.
func TestScannerRecordsOwnership(t *testing.T) {
t.Parallel()
sourceDir := t.TempDir()
filePath := filepath.Join(sourceDir, "file.txt")
dirPath := filepath.Join(sourceDir, "subdir")
linkPath := filepath.Join(sourceDir, "link")
err := os.WriteFile(filePath, []byte("owned"), 0o600)
if err != nil {
t.Fatal(err)
}
err = os.Mkdir(dirPath, 0o700)
if err != nil {
t.Fatal(err)
}
err = os.Symlink("file.txt", linkPath)
if err != nil {
t.Fatal(err)
}
db, err := database.NewTestDB()
if err != nil {
t.Fatalf("failed to create test database: %v", err)
}
defer func() {
err := db.Close()
if err != nil {
t.Errorf("failed to close database: %v", err)
}
}()
repos := database.NewRepositories(db)
scanner := snapshot.NewScanner(snapshot.ScannerConfig{
FS: afero.NewOsFs(),
ChunkSize: int64(1024 * 16),
Repositories: repos,
MaxBlobSize: int64(1024 * 1024),
CompressionLevel: 3,
AgeRecipients: []string{testAgePublicKey},
})
ctx := context.Background()
snapshotID := "test-snapshot-ownership"
createTestSnapshotRecord(ctx, t, repos, snapshotID)
_, err = scanner.Scan(ctx, sourceDir, snapshotID)
if err != nil {
t.Fatalf("scan failed: %v", err)
}
for _, path := range []string{filePath, dirPath, linkPath} {
file, err := repos.Files.GetByPath(ctx, path)
if err != nil {
t.Fatalf("failed to get %s: %v", path, err)
}
if file == nil {
t.Fatalf("%s was not recorded", path)
}
if int(file.UID) != os.Getuid() || int(file.GID) != os.Getgid() {
t.Errorf("%s recorded as uid %d gid %d, want uid %d gid %d",
path, file.UID, file.GID, os.Getuid(), os.Getgid())
}
}
}
internal/snapshot/snapshot.go
+38 -61
View File
@@ -154,26 +154,6 @@ func (sm *SnapshotManager) CreateSnapshotWithName(
return snapshotID, nil
}
// UpdateSnapshotStats updates the statistics for a snapshot during backup
func (sm *SnapshotManager) UpdateSnapshotStats(
ctx context.Context, snapshotID string, stats BackupStats,
) error {
err := sm.repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
return sm.repos.Snapshots.UpdateCounts(ctx, tx, snapshotID,
int64(stats.FilesScanned),
int64(stats.ChunksCreated),
int64(stats.BlobsCreated),
stats.BytesScanned,
stats.BytesUploaded,
)
})
if err != nil {
return fmt.Errorf("updating snapshot stats: %w", err)
}
return nil
}
// UpdateSnapshotStatsExtended updates snapshot statistics with extended metrics.
// This includes compression level, uncompressed blob size, and upload duration.
func (sm *SnapshotManager) UpdateSnapshotStatsExtended(
@@ -185,8 +165,8 @@ func (sm *SnapshotManager) UpdateSnapshotStatsExtended(
int64(stats.FilesScanned),
int64(stats.ChunksCreated),
int64(stats.BlobsCreated),
stats.BytesScanned,
stats.BytesUploaded,
stats.TotalSize,
stats.BlobSize,
)
if err != nil {
return err
@@ -196,6 +176,7 @@ func (sm *SnapshotManager) UpdateSnapshotStatsExtended(
return sm.repos.Snapshots.UpdateExtendedStats(ctx, tx, snapshotID,
stats.BlobUncompressedSize,
stats.CompressionLevel,
stats.BytesUploaded,
stats.UploadDurationMs,
)
})
@@ -386,12 +367,12 @@ func (sm *SnapshotManager) prepareExportDB(
ctx context.Context, dbPath, snapshotID, tempDir string,
) ([]byte, string, error) {
// Step 1: Copy database to temp file
// The main database should be closed at this point
// The main database is still open here, so it is copied through SQLite
tempDBPath := filepath.Join(tempDir, "snapshot.db")
log.Debug("Copying database to temporary location",
"source", dbPath, "destination", tempDBPath)
err := sm.copyFile(dbPath, tempDBPath)
err := sm.copyDatabase(ctx, dbPath, tempDBPath)
if err != nil {
return nil, "", fmt.Errorf("copying database: %w", err)
}
@@ -648,9 +629,10 @@ func (sm *SnapshotManager) collectCleanupStats(
//
// VACUUM runs through the modernc.org/sqlite driver, on a freshly opened
// connection with no transaction in flight (VACUUM cannot run inside one).
// The database opens in WAL mode, so VACUUM's rewrite lands in the WAL; the
// checkpoint on Close flushes it into the main file, which is the file we
// then compress and upload.
// database.New opens the file in WAL mode, so VACUUM's rewrite lands in the
// -wal file. This is the only connection to the file, so closing it
// checkpoints the rewrite into the main file and removes the -wal file; the
// main file is the one compressFile then compresses and uploads.
func (sm *SnapshotManager) vacuumDatabase(ctx context.Context, dbPath string) error {
log.Debug("Running VACUUM on database", "path", dbPath)
@@ -744,26 +726,15 @@ func (sm *SnapshotManager) compressFile(inputPath, outputPath string) error {
// user; it holds the same private index data as the local index file.
const exportCopyPerm = 0o600
// copyFile copies a file from src to dst. The destination is the exported
// snapshot database, so it is created owner-only rather than with the
// umask-dependent default.
func (sm *SnapshotManager) copyFile(src, dst string) error {
log.Debug("Opening source file for copy", "path", src)
sourceFile, err := sm.fs.Open(src)
if err != nil {
return err
}
defer func() {
log.Debug("Closing source file", "path", src)
err := sourceFile.Close()
if err != nil {
log.Debug("Failed to close source file", "path", src, "error", err)
}
}()
// copyDatabase copies the database at src to dst with VACUUM INTO. It reads
// through SQLite, so the copy holds rows committed to src that are still in
// its -wal file, which a copy of the file alone would miss. The destination
// is the exported snapshot database, so it is created empty and owner-only
// first rather than with the umask-dependent default: VACUUM INTO writes
// into an existing empty file and keeps its mode.
func (sm *SnapshotManager) copyDatabase(
ctx context.Context, src, dst string,
) error {
log.Debug("Creating destination file", "path", dst)
destFile, err := sm.fs.OpenFile(
@@ -773,23 +744,28 @@ func (sm *SnapshotManager) copyFile(src, dst string) error {
return err
}
defer func() {
log.Debug("Closing destination file", "path", dst)
err := destFile.Close()
if err != nil {
log.Debug("Failed to close destination file", "path", dst, "error", err)
}
}()
log.Debug("Copying file data")
n, err := io.Copy(destFile, sourceFile)
err = destFile.Close()
if err != nil {
return err
}
log.Debug("File copy complete", "bytes_copied", n)
db, err := database.New(ctx, src)
if err != nil {
return fmt.Errorf("opening database to copy: %w", err)
}
defer func() {
cerr := db.Close()
if cerr != nil {
log.Debug("Failed to close database after copy",
"path", src, "error", cerr)
}
}()
_, err = db.ExecWithLog(ctx, "VACUUM INTO ?", dst)
if err != nil {
return fmt.Errorf("running VACUUM INTO: %w", err)
}
return nil
}
@@ -895,7 +871,7 @@ func (sm *SnapshotManager) getFileSize(path string) int64 {
// BackupStats contains statistics from a backup operation
type BackupStats struct {
FilesScanned int
BytesScanned int64
TotalSize int64 // Total size of all files examined
ChunksCreated int
BlobsCreated int
BytesUploaded int64
@@ -905,6 +881,7 @@ type BackupStats struct {
type ExtendedBackupStats struct {
BackupStats
BlobSize int64 // Total compressed size of all referenced blobs
BlobUncompressedSize int64 // Total uncompressed size of all referenced blobs
CompressionLevel int // Compression level used for this snapshot
UploadDurationMs int64 // Total milliseconds spent uploading to S3
internal/snapshot/snapshot_test.go
+71
View File
@@ -188,6 +188,77 @@ func TestVacuumDatabaseRemovesDeletedData(t *testing.T) {
}
}
// TestPrepareExportDBKeepsRowsCommittedToOpenIndex exports from an index
// that is still open, as a backup does. A row committed there can still be
// in the index's -wal file, and the export must hold it all the same.
func TestPrepareExportDBKeepsRowsCommittedToOpenIndex(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
fs := afero.NewOsFs()
dbPath := filepath.Join(t.TempDir(), "index.sqlite")
db, err := database.New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
repos := database.NewRepositories(db)
snapshot := &database.Snapshot{ID: "open-index-snapshot", Hostname: "test-host"}
err = repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
return repos.Snapshots.Create(ctx, tx, snapshot)
})
if err != nil {
t.Fatalf("failed to create snapshot: %v", err)
}
sm := &SnapshotManager{
config: &config.Config{
CompressionLevel: 3,
AgeRecipients: []string{testAgeRecipient},
},
fs: fs,
}
_, tempDBPath, err := sm.prepareExportDB(
ctx, dbPath, snapshot.ID.String(), t.TempDir())
if err != nil {
t.Fatalf("prepareExportDB failed: %v", err)
}
// Only the main database file is compressed and uploaded, so open a
// copy of that file alone.
uploadedPath := filepath.Join(t.TempDir(), "uploaded.db")
err = copyFile(fs, tempDBPath, uploadedPath)
if err != nil {
t.Fatalf("failed to copy exported database: %v", err)
}
exported, err := database.OpenReadOnly(ctx, uploadedPath)
if err != nil {
t.Fatalf("failed to open exported database: %v", err)
}
defer func() { _ = exported.Close() }()
got, err := database.NewRepositories(exported).Snapshots.GetByID(
ctx, snapshot.ID.String())
if err != nil {
t.Fatalf("failed to read snapshot from export: %v", err)
}
if got == nil {
t.Fatal("exported database is missing the snapshot row")
}
}
func TestCleanSnapshotDBEmptySnapshot(t *testing.T) {
// Initialize logger
log.Initialize(log.Config{})
internal/storage/file.go
+23 -8
View File
@@ -22,11 +22,11 @@ type FileStorer struct {
//
// Construction is intentionally cheap and does not touch the filesystem.
// The basePath is recorded; the directory is created lazily on first
// write. Reads (Get/Stat/List) tolerate a missing basePath — a missing
// or unmounted destination during `snapshot list` should NOT block the
// command, it should degrade to "no remote snapshots reachable" with a
// warning. Write operations (Put/PutWithProgress) call MkdirAll for the
// write. Write operations (Put/PutWithProgress) call MkdirAll for the
// per-blob parent directory, which also covers basePath on first use.
// Get and Stat report a key under a missing basePath as ErrNotFound.
// List and ListStream fail on a missing basePath, because listing it as
// an empty store would make `prune` drop every local snapshot record.
//
// Uses the real OS filesystem by default; call SetFilesystem to
// override for testing.
@@ -119,13 +119,20 @@ func (f *FileStorer) Delete(_ context.Context, key string) error {
return nil
}
// List returns all keys with the given prefix.
// List returns all keys with the given prefix. It fails when the
// destination directory is missing; a missing prefix under it is an
// empty listing.
func (f *FileStorer) List(ctx context.Context, prefix string) ([]string, error) {
var keys []string
_, err := f.fs.Stat(f.basePath)
if err != nil {
return nil, fmt.Errorf("checking destination directory: %w", err)
}
basePath := f.fullPath(prefix)
// Check if base path exists
// Check if the prefix exists
exists, err := afero.Exists(f.fs, basePath)
if err != nil {
return nil, fmt.Errorf("checking path: %w", err)
@@ -167,16 +174,24 @@ func (f *FileStorer) List(ctx context.Context, prefix string) ([]string, error)
return keys, nil
}
// ListStream returns a channel of ObjectInfo for large result sets.
// ListStream returns a channel of ObjectInfo for large result sets. Like
// List, it sends an error when the destination directory is missing.
func (f *FileStorer) ListStream(ctx context.Context, prefix string) <-chan ObjectInfo {
ch := make(chan ObjectInfo)
go func() {
defer close(ch)
_, err := f.fs.Stat(f.basePath)
if err != nil {
ch <- ObjectInfo{Err: fmt.Errorf("checking destination directory: %w", err)}
return
}
basePath := f.fullPath(prefix)
// Check if base path exists
// Check if the prefix exists
exists, err := afero.Exists(f.fs, basePath)
if err != nil {
ch <- ObjectInfo{Err: fmt.Errorf("checking path: %w", err)}
internal/storage/file_backend_test.go
+35
View File
@@ -1,6 +1,10 @@
package storage_test
import (
"context"
"errors"
"io/fs"
"path/filepath"
"testing"
"sneak.berlin/go/vaultik/internal/storage"
@@ -25,3 +29,34 @@ func TestFileStorer(t *testing.T) {
t.Parallel()
runStorerConformance(t, newFileStorer)
}
// TestFileStorerListMissingDestination checks that List and ListStream
// fail when the destination directory does not exist, instead of
// reporting an empty store.
func TestFileStorerListMissingDestination(t *testing.T) {
t.Parallel()
ctx := context.Background()
s, err := storage.NewFileStorer(filepath.Join(t.TempDir(), "unmounted"))
if err != nil {
t.Fatalf("NewFileStorer: %v", err)
}
keys, err := s.List(ctx, "metadata/")
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("List = %v, %v; want a not-exist error", keys, err)
}
var streamErr error
for object := range s.ListStream(ctx, "metadata/") {
if object.Err != nil {
streamErr = object.Err
}
}
if !errors.Is(streamErr, fs.ErrNotExist) {
t.Errorf("ListStream error = %v, want a not-exist error", streamErr)
}
}
internal/storage/s3_test.go
+109
View File
@@ -4,11 +4,14 @@ import (
"context"
"errors"
"net/http/httptest"
"slices"
"strings"
"testing"
"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/backend/s3mem"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/s3"
"sneak.berlin/go/vaultik/internal/storage"
)
@@ -79,3 +82,109 @@ func TestS3StorerMissingKeyMapsToErrNotFound(t *testing.T) {
t.Errorf("Stat on missing key: got %v, want ErrNotFound", err)
}
}
// TestS3URLPrefixKeyLayout pins the bucket keys an s3:// URL reads and
// writes: the README's remote storage layout, with the prefix joined to
// each key by one "/". s3://b/p and s3://b/p/ must be the same
// destination, or a host that writes the URL the other way finds no
// snapshots. The listed object is put straight into the bucket, as
// another host would have written it. Both List and ListStream are
// checked: ListStream is what every snapshot listing goes through.
func TestS3URLPrefixKeyLayout(t *testing.T) {
t.Parallel()
const (
blobKey = "blobs/aa/bb/aabbccdd"
listPrefix = "metadata/"
manifestKey = listPrefix + "snap/manifest.json.zst"
manifestBody = "manifest"
)
cases := []struct {
urlPath string // URL path after the bucket name
keyPrefix string // what every key in the bucket must start with
}{
{urlPath: "/p", keyPrefix: "p/"},
{urlPath: "/p/", keyPrefix: "p/"},
{urlPath: "", keyPrefix: ""},
}
for _, tc := range cases {
storageURL := "s3://" + s3TestBucket + tc.urlPath
t.Run(storageURL, func(t *testing.T) {
t.Parallel()
backend := s3mem.New()
err := backend.CreateBucket(s3TestBucket)
if err != nil {
t.Fatalf("create bucket: %v", err)
}
srv := httptest.NewServer(gofakes3.New(backend).Server())
t.Cleanup(srv.Close)
storer, err := storage.NewStorer(&config.Config{
StorageURL: storageURL + "?endpoint=" + srv.URL,
S3: config.S3Config{
AccessKeyID: "key",
SecretAccessKey: "secret",
},
})
if err != nil {
t.Fatalf("NewStorer: %v", err)
}
ctx := context.Background()
err = storer.Put(ctx, blobKey, strings.NewReader("blob"))
if err != nil {
t.Fatalf("Put: %v", err)
}
_, err = backend.HeadObject(s3TestBucket, tc.keyPrefix+blobKey)
if err != nil {
t.Errorf("blob not stored at %q: %v", tc.keyPrefix+blobKey, err)
}
_, err = backend.PutObject(s3TestBucket, tc.keyPrefix+manifestKey,
nil, strings.NewReader(manifestBody), int64(len(manifestBody)))
if err != nil {
t.Fatalf("seed manifest: %v", err)
}
keys, err := storer.List(ctx, listPrefix)
if err != nil {
t.Fatalf("List: %v", err)
}
if !slices.Equal(keys, []string{manifestKey}) {
t.Errorf("List(%q) = %q, want [%q]", listPrefix, keys, manifestKey)
}
streamed := listStreamKeys(t, storer, listPrefix)
if !slices.Equal(streamed, []string{manifestKey}) {
t.Errorf("ListStream(%q) = %q, want [%q]", listPrefix, streamed, manifestKey)
}
})
}
}
// listStreamKeys returns the keys ListStream yields under a prefix, and
// fails the test on a listing error.
func listStreamKeys(t *testing.T, s storage.Storer, prefix string) []string {
t.Helper()
var keys []string
for obj := range s.ListStream(context.Background(), prefix) {
if obj.Err != nil {
t.Fatalf("ListStream %q: %v", prefix, obj.Err)
}
keys = append(keys, obj.Key)
}
return keys
}
internal/vaultik/changed_file_restore_test.go
+234
View File
@@ -0,0 +1,234 @@
package vaultik_test
import (
"context"
"crypto/rand"
"path/filepath"
"slices"
"strings"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/chunker"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// These tests cover https://git.eeqj.de/sneak/vaultik/issues/214: once
// the only snapshot referencing a changed file's current blob is
// dropped, an older snapshot still keeps the file's row, and the next
// backup must re-chunk the file instead of treating it as unchanged.
//
// Each backup run uses its own snapshot name over the same source
// directory, so the snapshot IDs differ without waiting for their
// one-second timestamps to tick over. File rows are keyed by path, so
// the runs share them exactly as runs of one snapshot name would.
// changedFileConfig returns a full-backup config with the snapshot names
// "first", "second" and "third", all backing up dataDir.
func changedFileConfig(dataDir, dbPath string) *config.Config {
source := config.SnapshotConfig{Paths: []string{dataDir}}
cfg := faultTestConfig()
cfg.IndexPath = dbPath
cfg.ChunkSize = config.Size(faultChunkSize)
cfg.Snapshots = map[string]config.SnapshotConfig{
"first": source,
"second": source,
"third": source,
}
return cfg
}
// backUp runs a full backup of the named snapshot.
func backUp(v *vaultik.Vaultik, name string) error {
return v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Cron: true,
Snapshots: []string{name},
})
}
// appendedFileSize is the size of the file before the tests append to it.
// It is twice the largest chunk the chunker cuts, so the file's first
// chunk ends before the appended bytes and stays in a blob the first
// snapshot still references, while its new chunks are only in the blob
// that gets dropped.
const appendedFileSize = 2 * chunker.ChunkSizeSpread * faultChunkSize
// appendRandomBytes appends n random bytes to the file at path, creating
// the file if it does not exist, and records the new content in files.
// Random content gives the chunker real cut points.
func appendRandomBytes(
t *testing.T, fs afero.Fs, files map[string][]byte, path string, n int64,
) {
t.Helper()
added := make([]byte, n)
_, err := rand.Read(added)
require.NoError(t, err)
content := slices.Concat(files[path], added)
require.NoError(t, afero.WriteFile(fs, path, content, 0o644))
files[path] = content
}
// localSnapshotID returns the ID of the one snapshot in the local index
// that was backed up under name.
func localSnapshotID(
ctx context.Context, t *testing.T,
repos *database.Repositories, name string,
) string {
t.Helper()
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
require.NoError(t, err)
var ids []string
for _, s := range snapshots {
if strings.Contains(s.ID.String(), "_"+name+"_") {
ids = append(ids, s.ID.String())
}
}
require.Lenf(t, ids, 1, "expected one local snapshot named %q", name)
return ids[0]
}
// assertThirdSnapshotRestores closes the local index, then restores the
// snapshot named "third" from store alone and byte-compares every file
// against files.
func assertThirdSnapshotRestores(
ctx context.Context, t *testing.T, cfg *config.Config,
store storage.Storer, repos *database.Repositories, db *database.DB,
fs afero.Fs, restoreDir string, files map[string][]byte,
) {
t.Helper()
id := localSnapshotID(ctx, t, repos, "third")
require.NoError(t, db.Close())
reader := newReaderVaultik(ctx, cfg, store, nil, fs)
require.NoError(t, reader.Restore(&vaultik.RestoreOptions{
SnapshotID: id,
TargetDir: restoreDir,
Verify: true,
}), "the backup after the drop must restore the changed file")
assertRestoredTree(t, fs, restoreDir, files)
}
// Trigger 1: bytes are appended to the file, a second snapshot backs it
// up, and that snapshot is removed. The first snapshot keeps the file row,
// which now lists the appended content's chunks, while removal drops the
// blob that held them.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupAfterRemovingNewestSnapshotRestoresChangedFile(t *testing.T) {
log.Initialize(log.Config{})
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
changedPath := filepath.Join(dataDir, "appended.bin")
ctx := context.Background()
files := writeFaultSourceTree(t, fs, dataDir)
cfg := changedFileConfig(dataDir, dbPath)
appendRandomBytes(t, fs, files, changedPath, appendedFileSize)
store, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
repos := database.NewRepositories(db)
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
require.NoError(t, backUp(v, "first"))
appendRandomBytes(t, fs, files, changedPath, faultChunkSize)
require.NoError(t, backUp(v, "second"))
_, err = v.RemoveSnapshot(localSnapshotID(ctx, t, repos, "second"),
&vaultik.RemoveOptions{Force: true})
require.NoError(t, err)
require.NoError(t, backUp(v, "third"))
assertThirdSnapshotRestores(
ctx, t, cfg, store, repos, db, fs, restoreDir, files)
}
// Trigger 2: bytes are appended to the file and the run that backs it up
// is interrupted at the manifest upload, after its blobs were uploaded.
// The next run's prune drops that incomplete snapshot and its blob, while
// the first snapshot keeps the file row, which now lists the appended
// content's chunks.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupAfterInterruptedRunRestoresChangedFile(t *testing.T) {
log.Initialize(log.Config{})
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
changedPath := filepath.Join(dataDir, "appended.bin")
ctx := context.Background()
files := writeFaultSourceTree(t, fs, dataDir)
cfg := changedFileConfig(dataDir, dbPath)
appendRandomBytes(t, fs, files, changedPath, appendedFileSize)
inner, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
failManifest := false
store := faultstore.New(inner)
store.OnPut = func(key string) faultstore.PutAction {
if failManifest && strings.HasSuffix(key, "manifest.json.zst") {
return faultstore.PutFail
}
return faultstore.PutNormal
}
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
repos := database.NewRepositories(db)
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
require.NoError(t, backUp(v, "first"))
appendRandomBytes(t, fs, files, changedPath, faultChunkSize)
failManifest = true
require.Error(t, backUp(v, "second"),
"the run must fail when its manifest upload fails")
failManifest = false
require.NoError(t, backUp(v, "third"))
assertThirdSnapshotRestores(
ctx, t, cfg, inner, repos, db, fs, restoreDir, files)
}
internal/vaultik/fault_injection_test.go
+4
View File
@@ -301,6 +301,10 @@ func TestInterruptedBlobUploadRecordsNoUploadedBlob(t *testing.T) {
writeFaultSourceTree(t, fs, dataDir)
// No upload succeeds, so nothing creates the destination directory.
// It must exist for the listing below to show that no blob survived.
require.NoError(t, os.Mkdir(storeDir, 0o750))
inner, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
internal/vaultik/helpers.go
+15 -5
View File
@@ -140,24 +140,34 @@ func parseSnapshotName(snapshotID string) string {
// parseDuration parses a duration string with support for human-friendly units:
// d/day/days, w/week/weeks, mo/month/months, y/year/years, plus standard Go
// duration units. Following Go, m is minutes and mo is months. A bare number,
// an unknown unit, and a negative value are all rejected.
// an unknown unit, and a negative value are all rejected. Outside the Go
// units the input must be whole numbers each followed directly by a unit
// (2w3d), so 1.0y and 30 days are errors; decimals work only in Go units
// (1.5h).
func parseDuration(s string) (time.Duration, error) {
if strings.HasPrefix(strings.TrimSpace(s), "-") {
return 0, errNegativeDuration
}
// A bare number has no unit, but time.ParseDuration accepts 0, which
// would put the cutoff at now and select every snapshot.
_, err := strconv.ParseFloat(s, 64)
if err == nil {
return 0, fmt.Errorf("%w: %q", errInvalidDuration, s)
}
d, err := time.ParseDuration(s)
if err == nil {
return d, nil
}
re := regexp.MustCompile(`(\d+)\s*([a-zA-Z]+)`)
matches := re.FindAllStringSubmatch(s, -1)
if len(matches) == 0 {
if !regexp.MustCompile(`^(\d+[a-zA-Z]+)+$`).MatchString(s) {
return 0, fmt.Errorf("%w: %q", errInvalidDuration, s)
}
re := regexp.MustCompile(`(\d+)([a-zA-Z]+)`)
matches := re.FindAllStringSubmatch(s, -1)
var total time.Duration
for _, match := range matches {
internal/vaultik/helpers_test.go
+11
View File
@@ -59,6 +59,7 @@ func TestParseDuration(t *testing.T) {
{"30s", 30 * time.Second, false},
{"6m", 6 * time.Minute, false},
{"1h", time.Hour, false},
{"1.5h", 90 * time.Minute, false},
// Extended calendar units.
{"30d", 30 * 24 * time.Hour, false},
{"3days", 3 * 24 * time.Hour, false},
@@ -73,11 +74,21 @@ func TestParseDuration(t *testing.T) {
{"1y6mo", 365*24*time.Hour + 180*24*time.Hour, false},
// Rejected inputs.
{"6", 0, true}, // bare number, no unit
{"0", 0, true}, // bare number; time.ParseDuration accepts it
{"+0", 0, true}, // bare number; time.ParseDuration accepts it
{"5x", 0, true}, // unknown unit
{"-5d", 0, true}, // negative, extended unit
{"-5h", 0, true}, // negative, Go unit
{"", 0, true}, // empty
{"garbage", 0, true},
// Characters outside the number-and-unit parts must not be
// skipped: 1.0y read as 0y would select every snapshot.
{"1.0y", 0, true},
{"2.1w", 0, true},
{"1,5y", 0, true},
{"30 days", 0, true},
{"30 days ago", 0, true},
{"x7d", 0, true},
}
for _, tt := range tests {
internal/vaultik/integration_test.go
+1 -1
View File
@@ -249,7 +249,7 @@ func verifyEndToEndBackupState(
assert.Positive(t, blobUploads, "Should upload at least one blob")
// Verify files in database
files, err := repos.Files.ListByPrefix(ctx, "/home/user")
files, err := repos.Files.ListUnderPath(ctx, "/home/user")
require.NoError(t, err)
// Count only regular files (not directories)
regularFiles := 0
internal/vaultik/missing_destination_test.go
+155
View File
@@ -0,0 +1,155 @@
package vaultik_test
import (
"bytes"
"context"
"io/fs"
"os"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// These tests cover https://git.eeqj.de/sneak/vaultik/issues/220: a
// file:// destination whose directory is missing, such as a USB stick
// that is not plugged in, cannot be listed. It is not an empty store, so
// no command may conclude from it that the local snapshots are gone.
// backUpToFileDestination backs up the snapshot named "first" to a
// file:// destination at storeDir, which need not exist yet. Everything
// the returned Vaultik prints after the backup goes to the returned
// buffer.
func backUpToFileDestination(
ctx context.Context, t *testing.T, storeDir string,
) (*vaultik.Vaultik, *database.Repositories, *bytes.Buffer) {
t.Helper()
osFs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src")
dbPath := filepath.Join(tempDir, "index.sqlite")
writeFaultSourceTree(t, osFs, dataDir)
store, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
repos := database.NewRepositories(db)
cfg := changedFileConfig(dataDir, dbPath)
v := newBackupVaultik(ctx, cfg, store, repos, db, osFs)
require.NoError(t, backUp(v, "first"))
out := &bytes.Buffer{}
v.Stdout = out
v.UI = ui.NewWithColor(out, false)
return v, repos, out
}
// backUpThenUnplug backs up to a file:// destination, then moves the
// destination directory away, as unplugging the volume it lives on would.
func backUpThenUnplug(
ctx context.Context, t *testing.T,
) (*vaultik.Vaultik, *database.Repositories, *bytes.Buffer) {
t.Helper()
storeDir := filepath.Join(t.TempDir(), "usbstick")
v, repos, out := backUpToFileDestination(ctx, t, storeDir)
require.NoError(t, os.Rename(storeDir, storeDir+"-unplugged"))
return v, repos, out
}
// TestFirstBackupCreatesDestinationDirectory checks that a first backup
// to a destination directory that does not exist yet creates it, and
// that the destination can be listed afterwards.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestFirstBackupCreatesDestinationDirectory(t *testing.T) {
log.Initialize(log.Config{})
ctx := context.Background()
storeDir := filepath.Join(t.TempDir(), "volume", "backup")
v, _, out := backUpToFileDestination(ctx, t, storeDir)
require.NoError(t, v.ListSnapshots(false))
assert.NotContains(t, out.String(), "Could not list backup destination store")
assert.NotContains(t, out.String(), "not found in backup destination store")
}
// TestListSnapshotsWarnsWhenDestinationMissing checks that snapshot list
// warns and shows the local index alone, without reporting the local
// snapshot as missing from the destination.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestListSnapshotsWarnsWhenDestinationMissing(t *testing.T) {
log.Initialize(log.Config{})
ctx := context.Background()
v, repos, out := backUpThenUnplug(ctx, t)
id := localSnapshotID(ctx, t, repos, "first")
require.NoError(t, v.ListSnapshots(false))
assert.Contains(t, out.String(), "Could not list backup destination store")
assert.Contains(t, out.String(), "Showing snapshots from the local index only.")
assert.Contains(t, out.String(), id)
assert.NotContains(t, out.String(), "not found in backup destination store")
}
// TestRemoveSnapshotWarnsWhenDestinationMissing checks that snapshot
// remove warns that the metadata could not be removed from the
// destination, instead of reporting that it was.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestRemoveSnapshotWarnsWhenDestinationMissing(t *testing.T) {
log.Initialize(log.Config{})
ctx := context.Background()
v, repos, out := backUpThenUnplug(ctx, t)
result, err := v.RemoveSnapshot(localSnapshotID(ctx, t, repos, "first"),
&vaultik.RemoveOptions{Force: true})
require.NoError(t, err)
assert.False(t, result.RemoteRemoved)
assert.Contains(t, out.String(),
"Could not remove snapshot metadata from remote")
assert.NotContains(t, out.String(),
"Removed snapshot metadata from remote storage")
}
// TestPruneKeepsLocalRecordsWhenDestinationMissing checks that prune
// fails on a destination it cannot list and deletes no local snapshot
// record.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) {
log.Initialize(log.Config{})
ctx := context.Background()
v, repos, _ := backUpThenUnplug(ctx, t)
err := v.Prune(&vaultik.PruneOptions{Force: true})
require.ErrorIs(t, err, fs.ErrNotExist)
require.ErrorContains(t, err, "listing remote snapshots")
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
require.NoError(t, err)
assert.Len(t, snapshots, 1, "prune must delete no local snapshot record")
}
internal/vaultik/restore.go
+142 -37
View File
@@ -10,11 +10,13 @@ import (
"math"
"os"
"path/filepath"
"slices"
"strings"
"time"
"filippo.io/age"
"github.com/spf13/afero"
"golang.org/x/sys/unix"
"sneak.berlin/go/vaultik/internal/blobgen"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
@@ -40,6 +42,7 @@ var (
errChunkNotInAnyBlob = errors.New("chunk not found in any blob")
errBlobIDNotInHashIndex = errors.New("blob id missing from hash index")
errShortChunkRead = errors.New("short read")
errChunkRowMissing = errors.New("chunk has no row in the chunks table")
errRestorePathEscapesTarget = errors.New(
"refusing to restore path outside the target directory")
errTrailingRestoreData = errors.New(
@@ -63,6 +66,12 @@ const snapshotDBFilename = "snapshot.db"
// directories themselves get their stored mode).
const restoreDirMode = 0o755
// restoreDirCreateMode is the owner-only mode a directory from the
// snapshot is created with during restore, so its contents can be written
// whatever its stored mode. The stored mode is applied after the restore
// loop, by applyDirectoryMetadata.
const restoreDirCreateMode = 0o700
// restoreFileMode is the restrictive mode a regular file is created with
// during restore. Content is written while the file holds this mode; the
// stored mode is applied only after the file is fully written and closed,
@@ -332,6 +341,8 @@ func (v *Vaultik) restoreAllFiles(
return nil, err
}
session.applyDirectoryMetadata()
return result, nil
}
@@ -355,7 +366,7 @@ func (v *Vaultik) runRestoreLoop(
fileID, ready := plan.popReady()
if !ready {
downloaded, err := session.downloadNextBlobSet(plan)
downloaded, err := session.downloadNextBlobSet(plan, filesByID)
if err != nil {
return err
}
@@ -411,7 +422,13 @@ func (v *Vaultik) runRestoreLoop(
// blob set and downloads its blobs; after each blob lands, the plan
// moves any pending file whose set just emptied onto the ready queue.
// Returns false when nothing is pending download (the caller stops).
func (s *restoreSession) downloadNextBlobSet(plan *restorePlan) (bool, error) {
//
// A blob that cannot be downloaded is reported through
// handleRestoreFileError for every pending file that references it, so
// it aborts the restore unless --skip-errors is set.
func (s *restoreSession) downloadNextBlobSet(
plan *restorePlan, filesByID map[types.FileID]*database.File,
) (bool, error) {
s.sweeper.sweep()
next, ok := plan.pickNextDownload()
@@ -433,7 +450,25 @@ func (s *restoreSession) downloadNextBlobSet(plan *restorePlan) (bool, error) {
err := s.downloadBlobToCache(hash, blob.CompressedSize, blob.UncompressedSize)
if err != nil {
return false, fmt.Errorf("downloading blob %s: %w", shortHash(hash), err)
err = fmt.Errorf("downloading blob %s: %w", shortHash(hash), err)
// On cancel the error says nothing about the blob, so it ends
// the restore instead of failing the files that need it.
if s.ctx.Err() != nil {
return false, err
}
for _, fileID := range plan.filesReferencingBlob(hash) {
fileErr := s.v.handleRestoreFileError(
plan, s.opts, s.result, filesByID[fileID], fileID, err)
if fileErr != nil {
return false, fileErr
}
}
// next is among the failed files, so the rest of its blob set
// is left for any other file that still needs it.
return true, nil
}
s.result.BlobsDownloaded++
@@ -795,10 +830,9 @@ func (v *Vaultik) getFilesToRestore(
// Normalize the filter path
filter = filepath.Clean(filter)
// Get files with this prefix
files, err := repos.Files.ListByPrefix(ctx, filter)
files, err := repos.Files.ListUnderPath(ctx, filter)
if err != nil {
return nil, fmt.Errorf("listing files with prefix %s: %w", filter, err)
return nil, fmt.Errorf("listing files under %s: %w", filter, err)
}
for _, file := range files {
@@ -877,6 +911,9 @@ type restoreSession struct {
// the call entirely as non-root and emit one warning at the end
// of the restore explaining that ownership was not preserved.
runningAsRoot bool
// directories holds every restored directory, for
// applyDirectoryMetadata to finish after the restore loop.
directories []*database.File
}
// containedRestorePath resolves rel — a path read from the snapshot
@@ -983,6 +1020,8 @@ func (s *restoreSession) restoreSymlink(file *database.File, targetPath string)
if err != nil {
return fmt.Errorf("creating symlink: %w", err)
}
s.applySymlinkMetadata(file, targetPath)
} else {
log.Debug("Symlink creation not supported on this filesystem",
"path", file.Path, "target", file.LinkTarget)
@@ -995,35 +1034,90 @@ func (s *restoreSession) restoreSymlink(file *database.File, targetPath string)
return nil
}
// restoreDirectory restores a directory with its permissions, mtime,
// and (on real filesystems, with sufficient privileges) ownership.
// restoreDirectory creates a directory with restoreDirCreateMode. Its
// stored mode, owner and mtime are applied after the restore loop by
// applyDirectoryMetadata: a read-only stored mode would block writing
// its contents, and writing them changes its mtime.
func (s *restoreSession) restoreDirectory(
file *database.File, targetPath string,
) error {
err := s.v.Fs.MkdirAll(targetPath, os.FileMode(file.Mode))
err := s.v.Fs.MkdirAll(targetPath, restoreDirCreateMode)
if err != nil {
return fmt.Errorf("creating directory: %w", err)
}
// MkdirAll applies the process umask, so chmod to the exact stored
// mode. A failure here is non-fatal.
err = s.v.Fs.Chmod(targetPath, os.FileMode(file.Mode))
if err != nil {
log.Debug("Failed to set permissions", "path", targetPath, "error", err)
}
s.applyFileMetadata(file, targetPath)
s.directories = append(s.directories, file)
s.result.FilesRestored++
return nil
}
// applyDirectoryMetadata applies the stored owner, mtime and mode to every
// restored directory, each before its parent, so a parent whose stored
// mode denies search does not block its children. Failures are logged at
// debug level and do not abort the restore.
func (s *restoreSession) applyDirectoryMetadata() {
// A path sorts after its parent's, so reverse order puts every
// directory before its parent.
slices.SortFunc(s.directories, func(a, b *database.File) int {
return strings.Compare(b.Path.String(), a.Path.String())
})
for _, dir := range s.directories {
targetPath, err := containedRestorePath(
s.v.Fs, s.opts.TargetDir, dir.Path.String())
if err != nil {
log.Debug("Failed to set directory metadata",
"path", dir.Path, "error", err)
continue
}
// A later entry can have put a symlink in the directory's place,
// for example one stored as "/d/" next to the directory "/d". The
// calls below follow symlinks, so they would change its target.
info, err := lstatIfPossible(s.v.Fs, targetPath)
if err != nil || !info.IsDir() {
log.Debug("Not setting directory metadata: no longer a directory",
"path", targetPath, "error", err)
continue
}
s.applyFileMetadata(dir, targetPath)
err = s.v.Fs.Chmod(targetPath, os.FileMode(dir.Mode))
if err != nil {
log.Debug("Failed to set permissions", "path", targetPath, "error", err)
}
}
}
// applySymlinkMetadata applies ownership (when running as root) and mtime
// to a restored symlink itself; os.Chown and Chtimes would follow it.
// Failures are logged at debug level and do not abort the restore.
func (s *restoreSession) applySymlinkMetadata(file *database.File, targetPath string) {
if s.runningAsRoot {
err := os.Lchown(targetPath, int(file.UID), int(file.GID))
if err != nil {
log.Debug("Failed to set ownership", "path", targetPath, "error", err)
}
}
mtime := unix.NsecToTimeval(file.MTime.UnixNano())
err := unix.Lutimes(targetPath, []unix.Timeval{mtime, mtime})
if err != nil {
log.Debug("Failed to set mtime", "path", targetPath, "error", err)
}
}
// applyFileMetadata applies ownership (when running as root on a real
// filesystem) and mtime to a restored path. Permission mode is applied
// separately by each caller, with different failure handling, so it is
// not touched here. Failures are logged at debug level and do not abort
// the restore.
// filesystem) and mtime to a restored path. The caller applies the mode
// afterwards: on Linux a chown clears the setuid and setgid bits of a
// regular file. Failures are logged at debug level and do not abort the
// restore.
func (s *restoreSession) applyFileMetadata(file *database.File, targetPath string) {
if s.runningAsRoot {
if _, ok := s.v.Fs.(*afero.OsFs); ok {
@@ -1114,8 +1208,8 @@ func (s *restoreSession) restoreRegularFile(
return fmt.Errorf("closing output file: %w", err)
}
s.applyRestoredFileMode(file, targetPath)
s.applyFileMetadata(file, targetPath)
s.applyRestoredFileMode(file, targetPath)
s.result.FilesRestored++
s.result.BytesRestored += bytesWritten
@@ -1173,7 +1267,7 @@ func (s *restoreSession) writeFileChunks(
blobChunk, ok := s.chunkToBlobMap[chunkHashStr]
if !ok {
return bytesWritten, timings, fmt.Errorf(
"%w: %s", errChunkNotInAnyBlob, chunkHashStr[:16])
"%w: %s", errChunkNotInAnyBlob, shortHash(chunkHashStr))
}
blobHash, ok := s.blobIDToHash[blobChunk.BlobID.String()]
@@ -1190,7 +1284,7 @@ func (s *restoreSession) writeFileChunks(
if err != nil {
return bytesWritten, timings, fmt.Errorf(
"reading chunk %s from cached blob %s: %w",
fc.ChunkHash[:16], blobHash[:16], err)
shortHash(chunkHashStr), shortHash(blobHash), err)
}
t0 = time.Now()
@@ -1388,33 +1482,44 @@ func (v *Vaultik) verifyFile(
chunk, err := repos.Chunks.GetByHash(ctx, fc.ChunkHash.String())
if err != nil {
return bytesVerified, fmt.Errorf("getting chunk %s: %w",
fc.ChunkHash.String()[:16], err)
shortHash(fc.ChunkHash.String()), err)
}
// Read chunk data from file
chunkData := make([]byte, chunk.Size)
n, err := io.ReadFull(f, chunkData)
if err != nil {
return bytesVerified, fmt.Errorf("reading chunk data: %w", err)
if chunk == nil {
return bytesVerified, fmt.Errorf("%w: %s",
errChunkRowMissing, shortHash(fc.ChunkHash.String()))
}
if int64(n) != chunk.Size {
// chunk.Size comes from the snapshot database, which is not
// trusted: reject a negative size, and hash the chunk by
// streaming it rather than allocating that many bytes.
if chunk.Size < 0 {
return bytesVerified, fmt.Errorf("%w: chunk %d size %d",
errNegativeChunkLength, fc.Idx, chunk.Size)
}
hasher := sha256.New()
n, err := io.CopyN(hasher, f, chunk.Size)
if errors.Is(err, io.EOF) {
return bytesVerified, fmt.Errorf("%w: expected %d bytes, got %d",
errShortChunkRead, chunk.Size, n)
}
// Calculate hash and compare
hash := sha256.Sum256(chunkData)
actualHash := hex.EncodeToString(hash[:])
if err != nil {
return bytesVerified, fmt.Errorf("reading chunk data: %w", err)
}
actualHash := hex.EncodeToString(hasher.Sum(nil))
expectedHash := fc.ChunkHash.String()
if actualHash != expectedHash {
return bytesVerified, fmt.Errorf("%w: chunk %d: expected %s, got %s",
errChunkHashMismatch, fc.Idx, expectedHash[:16], actualHash[:16])
errChunkHashMismatch, fc.Idx,
shortHash(expectedHash), shortHash(actualHash))
}
bytesVerified += int64(n)
bytesVerified += n
}
// The stored chunks account for the whole file, so the reader must
internal/vaultik/restore_another_machine_test.go
+45 -12
View File
@@ -10,6 +10,7 @@ import (
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/cli"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
@@ -27,10 +28,12 @@ import (
//
// The backup half writes a snapshot with one index and hostname. The
// restore half throws that index away entirely: a fresh, empty index and
// a config that shares nothing with the original but the storage location
// and the secret key. If restore or verify needed the original local
// index — or the human snapshot ID that only that index holds — this test
// could not run, because the recovery host can know neither.
// a config written by `config init` and `config set storage_url`, as in
// the README's steps for restoring on another machine, which shares
// nothing with the original but the storage location and the secret key.
// If restore or verify needed the original local index — or
// the human snapshot ID that only that index holds — this test could not
// run, because the recovery host can know neither.
func TestRestoreOnAnotherMachine(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
@@ -58,7 +61,12 @@ func TestRestoreOnAnotherMachine(t *testing.T) {
// Recovery host: a fresh empty index, a different hostname, and no
// age_recipients — only the secret key and the same storage location.
recovery, stdout := newRecoveryHost(ctx, t, fs, storer)
configPath := filepath.Join(tempDir, "config.yml")
runVaultikCommand(t, "--config", configPath, "config", "init")
runVaultikCommand(t, "--config", configPath,
"config", "set", "storage_url", "file://"+storeDir)
recovery, stdout := newRecoveryHost(ctx, t, fs, storer, configPath)
// The recovery index really is empty. This is the assertion that makes
// the test a guard against restore quietly depending on the original
@@ -93,6 +101,10 @@ func TestRestoreOnAnotherMachine(t *testing.T) {
remote.RemoteKey, &vaultik.VerifyOptions{Deep: true}))
assertRestoredTreeMatches(t, fs, restoreDir, sourceFiles)
// With no public key configured, a backup must refuse to start.
err = recovery.CreateSnapshot(&vaultik.SnapshotCreateOptions{Cron: true})
require.ErrorContains(t, err, "age_recipients")
}
// writeRecoverySourceTree writes a small source tree spanning several
@@ -118,14 +130,24 @@ func writeRecoverySourceTree(
}
// newRecoveryHost builds the Vaultik a replacement machine would run: an
// empty in-memory index, a hostname different from the backup host, no
// age_recipients, and only the secret key plus the shared storer. It
// returns the instance and the buffer its stdout is wired to.
// empty in-memory index, a hostname different from the backup host, and
// only the secret key plus the shared storer. Its config is read from
// configPath by config.Load, as every command reads it. It returns the
// instance and the buffer its stdout is wired to.
func newRecoveryHost(
ctx context.Context, t *testing.T, fs afero.Fs, storer storage.Storer,
configPath string,
) (*vaultik.Vaultik, *bytes.Buffer) {
t.Helper()
cfg, err := config.Load(configPath)
require.NoError(t, err)
// Set directly rather than through VAULTIK_AGE_SECRET_KEY, which a
// parallel test cannot change.
cfg.AgeSecretKey = testAgeSecretKey
cfg.Hostname = "recovery-host"
recoveryDB, err := database.New(ctx, ":memory:")
require.NoError(t, err)
t.Cleanup(func() { _ = recoveryDB.Close() })
@@ -133,10 +155,7 @@ func newRecoveryHost(
stdout := &bytes.Buffer{}
recovery := &vaultik.Vaultik{
Config: &config.Config{
AgeSecretKey: testAgeSecretKey,
Hostname: "recovery-host",
},
Config: cfg,
Storage: storer,
Fs: fs,
Repositories: database.NewRepositories(recoveryDB),
@@ -150,6 +169,20 @@ func newRecoveryHost(
return recovery, stdout
}
// runVaultikCommand runs one vaultik command line in-process and fails the
// test if it returns an error. The command writes the cli package's global
// flag variables, so it must not be called from two tests that run at the
// same time.
func runVaultikCommand(t *testing.T, args ...string) {
t.Helper()
cmd := cli.NewRootCommand()
cmd.SetArgs(args)
cmd.SetOut(io.Discard)
cmd.SetErr(io.Discard)
require.NoError(t, cmd.Execute())
}
// assertRestoredTreeMatches byte-compares every restored file against its
// source content.
func assertRestoredTreeMatches(
internal/vaultik/restore_interrupt_test.go
+50
View File
@@ -1,6 +1,7 @@
package vaultik //nolint:testpackage // sets ctx/cancel and inspects scratch files
import (
"bytes"
"context"
"io"
"path/filepath"
@@ -11,6 +12,7 @@ import (
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
@@ -157,3 +159,51 @@ func scratchEntries(t *testing.T, dir string) []string {
return matches
}
// TestRestoreSkipErrorsCancelDuringBlobDownload cancels a SkipErrors
// restore while a blob download is in progress. The download fails only
// because of the cancel, so Restore must return context.Canceled without
// reporting the file that needs the blob as failed.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestRestoreSkipErrorsCancelDuringBlobDownload(t *testing.T) {
log.Initialize(log.Config{})
fs := afero.NewOsFs()
tempDir := t.TempDir()
cfg, storer, snapshotID, srcPath := backupOneFile(context.Background(),
t, fs, tempDir, "a.txt", []byte("hello vaultik"), 0o644)
gate := newBlockingBlobStorer(storer)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
var out bytes.Buffer
v := newRestoreVaultik(ctx, cfg, gate, fs)
v.UI = ui.NewWithColor(&out, false)
restoreErr := make(chan error, 1)
go func() {
restoreErr <- v.Restore(&RestoreOptions{
SnapshotID: snapshotID,
TargetDir: filepath.Join(tempDir, "restored"),
SkipErrors: true,
})
}()
select {
case <-gate.entered:
case <-time.After(30 * time.Second):
t.Fatal("restore never reached the blob-download phase")
}
cancel()
require.ErrorIs(t, <-restoreErr, context.Canceled)
assert.NotContains(t, out.String(), srcPath,
"the cancel was reported as a failed file")
}
internal/vaultik/restore_malformed_db_test.go
+221
View File
@@ -0,0 +1,221 @@
package vaultik //nolint:testpackage // drives unexported restore and verify steps
import (
"context"
"math"
"path/filepath"
"strings"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
)
// These tests feed restore and --verify a snapshot database written by
// hand, as a damaged or hostile store could serve one. Each malformed row
// must end in an error, not a panic.
// shortChunkHash is shorter than the hash prefix that error messages print.
const shortChunkHash = "abc"
// restoredFileContent is the content of the restored file under verify.
const restoredFileContent = "xyz"
// craftedSnapshotDB opens an empty snapshot database in a temp directory.
func craftedSnapshotDB(t *testing.T) (*database.DB, *database.Repositories) {
t.Helper()
db, err := database.New(context.Background(),
filepath.Join(t.TempDir(), "snapshot.db"))
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
return db, database.NewRepositories(db)
}
// craftedFile adds a regular file whose only chunk has the given hash.
// Adding the chunks row, if any, is left to the caller.
func craftedFile(
t *testing.T, repos *database.Repositories, chunkHash string,
) *database.File {
t.Helper()
ctx := context.Background()
file := &database.File{
Path: "/src/f",
MTime: time.Now().UTC(),
Size: int64(len(restoredFileContent)),
Mode: 0o644,
}
require.NoError(t, repos.Files.Create(ctx, nil, file))
require.NoError(t, repos.FileChunks.Create(ctx, nil, &database.FileChunk{
FileID: file.ID,
ChunkHash: types.ChunkHash(chunkHash),
}))
return file
}
// TestRestoreShortChunkHashInNoBlob proves a file whose short chunk hash
// has no blob_chunks row fails restore planning and the chunk write with
// an error.
func TestRestoreShortChunkHashInNoBlob(t *testing.T) {
t.Parallel()
ctx := context.Background()
_, repos := craftedSnapshotDB(t)
require.NoError(t, repos.Chunks.Create(ctx, nil,
&database.Chunk{ChunkHash: shortChunkHash, Size: 3}))
file := craftedFile(t, repos, shortChunkHash)
v := NewForTesting(nil)
chunkToBlobMap, err := v.buildChunkToBlobMap(ctx, repos)
require.NoError(t, err)
_, err = newRestorePlan(ctx, repos, []*database.File{file},
chunkToBlobMap, map[string]string{})
require.ErrorIs(t, err, errPlanChunkMissing)
fileChunks, err := repos.FileChunks.GetByFileID(ctx, file.ID)
require.NoError(t, err)
out, err := afero.NewMemMapFs().Create("out")
require.NoError(t, err)
session := &restoreSession{
v: v.Vaultik, ctx: ctx, chunkToBlobMap: chunkToBlobMap,
}
_, _, err = session.writeFileChunks(out, fileChunks)
require.ErrorIs(t, err, errChunkNotInAnyBlob)
}
// TestRestoreShortChunkHashReadPastBlobEnd proves a short chunk hash
// whose blob_chunks row reads past the end of its blob fails the chunk
// write with an error.
func TestRestoreShortChunkHashReadPastBlobEnd(t *testing.T) {
t.Parallel()
ctx := context.Background()
_, repos := craftedSnapshotDB(t)
blobHash := strings.Repeat("b", blobHashHexLen)
blob := &database.Blob{
ID: types.NewBlobID(),
Hash: types.BlobHash(blobHash),
CreatedTS: time.Now().UTC(),
}
require.NoError(t, repos.Blobs.Create(ctx, nil, blob))
require.NoError(t, repos.Chunks.Create(ctx, nil,
&database.Chunk{ChunkHash: shortChunkHash, Size: 3}))
require.NoError(t, repos.BlobChunks.Create(ctx, nil, &database.BlobChunk{
BlobID: blob.ID,
ChunkHash: shortChunkHash,
Length: 100,
}))
file := craftedFile(t, repos, shortChunkHash)
cache, err := newBlobDiskCache(1 << 20)
require.NoError(t, err)
t.Cleanup(func() { _ = cache.Close() })
require.NoError(t, cache.Put(blobHash, []byte("abc")))
v := NewForTesting(nil)
chunkToBlobMap, err := v.buildChunkToBlobMap(ctx, repos)
require.NoError(t, err)
_, blobIDToHash, err := v.buildBlobIndexes(repos)
require.NoError(t, err)
fileChunks, err := repos.FileChunks.GetByFileID(ctx, file.ID)
require.NoError(t, err)
out, err := afero.NewMemMapFs().Create("out")
require.NoError(t, err)
session := &restoreSession{
v: v.Vaultik,
ctx: ctx,
chunkToBlobMap: chunkToBlobMap,
blobIDToHash: blobIDToHash,
blobCache: cache,
}
_, _, err = session.writeFileChunks(out, fileChunks)
require.ErrorIs(t, err, errCacheReadBeyondBlob)
}
// TestVerifyFileMalformedChunkRow proves --verify returns an error for a
// chunk with no chunks row, a short chunk hash, and a chunk size from the
// database that is negative or larger than the restored file.
func TestVerifyFileMalformedChunkRow(t *testing.T) {
t.Parallel()
fullHash := types.ChunkHash(strings.Repeat("c", blobHashHexLen))
tests := []struct {
name string
hash types.ChunkHash
chunk *database.Chunk // nil adds no chunks row
want error
}{
{
name: "missing chunk row",
hash: fullHash,
want: errChunkRowMissing,
},
{
name: "short hash",
hash: shortChunkHash,
chunk: &database.Chunk{ChunkHash: shortChunkHash, Size: 3},
want: errChunkHashMismatch,
},
{
name: "size larger than the file",
hash: fullHash,
chunk: &database.Chunk{ChunkHash: fullHash, Size: math.MaxInt64},
want: errShortChunkRead,
},
{
name: "negative size",
hash: fullHash,
chunk: &database.Chunk{ChunkHash: fullHash, Size: -1},
want: errNegativeChunkLength,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
ctx := context.Background()
db, repos := craftedSnapshotDB(t)
if tt.chunk == nil {
// A crafted database need not satisfy its foreign keys.
_, err := db.Conn().ExecContext(ctx, "PRAGMA foreign_keys = OFF")
require.NoError(t, err)
} else {
require.NoError(t, repos.Chunks.Create(ctx, nil, tt.chunk))
}
file := craftedFile(t, repos, tt.hash.String())
v := NewForTesting(nil)
v.Fs = afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(v.Fs, "/restore/f",
[]byte(restoredFileContent), 0o600))
_, err := v.verifyFile(ctx, repos, file, "/restore/f")
require.ErrorIs(t, err, tt.want)
})
}
}
internal/vaultik/restore_metadata_test.go
+349
View File
@@ -0,0 +1,349 @@
package vaultik //nolint:testpackage // drives unexported restore internals
import (
"context"
"os"
"path/filepath"
"syscall"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/types"
)
// These tests check that restore applies each entry's owner, mode and
// mtime in an order that keeps them.
const (
readOnlyDirMode = uint32(os.ModeDir | 0o555)
unsearchableMode = uint32(os.ModeDir | 0o600)
plainDirMode = uint32(os.ModeDir | 0o755)
plainFileMode = uint32(0o644)
setuidFileMode = uint32(os.ModeSetuid | 0o755)
otherOwnerID = uint32(4321)
writableTestMode = 0o755
symlinkTargetPath = "/nonexistent/target"
memTargetDir = "/restore"
// Owner bits a normal user needs on a directory to create an entry
// in it, and to change an entry in it.
ownerWriteAndSearch = os.FileMode(0o300)
ownerSearch = os.FileMode(0o100)
)
// normalUserFs refuses what the kernel refuses a normal user. make test
// runs as root, which a read-only or unsearchable directory does not
// stop, so without it the tests below could not fail there. Creating an
// entry needs owner write and search on the directory holding it;
// changing an entry's mode or times needs owner search. Only that one
// directory is checked, not every ancestor.
type normalUserFs struct {
afero.Fs
}
//nolint:ireturn // afero.Fs.OpenFile is defined to return the interface
func (fs normalUserFs) OpenFile(
name string, flag int, perm os.FileMode,
) (afero.File, error) {
if flag&os.O_CREATE != 0 {
err := fs.checkParent(name, ownerWriteAndSearch)
if err != nil {
return nil, err
}
}
return fs.Fs.OpenFile(name, flag, perm)
}
func (fs normalUserFs) MkdirAll(path string, perm os.FileMode) error {
_, err := fs.Stat(path)
if err != nil {
err = fs.checkParent(path, ownerWriteAndSearch)
if err != nil {
return err
}
}
return fs.Fs.MkdirAll(path, perm)
}
func (fs normalUserFs) Chmod(name string, mode os.FileMode) error {
err := fs.checkParent(name, ownerSearch)
if err != nil {
return err
}
return fs.Fs.Chmod(name, mode)
}
func (fs normalUserFs) Chtimes(name string, atime, mtime time.Time) error {
err := fs.checkParent(name, ownerSearch)
if err != nil {
return err
}
return fs.Fs.Chtimes(name, atime, mtime)
}
// checkParent returns a permission error when the directory holding name
// exists and its owner bits lack any of need.
func (fs normalUserFs) checkParent(name string, need os.FileMode) error {
info, err := fs.Stat(filepath.Dir(name))
if err == nil && info.Mode().Perm()&need != need {
return &os.PathError{Op: "access", Path: name, Err: os.ErrPermission}
}
return nil
}
// TestRestoreFillsReadOnlyDirectory checks that a read-only directory
// still receives the entries inside it, and ends with its stored mode.
func TestRestoreFillsReadOnlyDirectory(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
rows, repos := makeFiles(ctx, t, []*database.File{
{Path: "/ro", Mode: readOnlyDirMode},
{Path: "/ro/file", Mode: plainFileMode},
{Path: "/ro/sub", Mode: readOnlyDirMode},
{Path: "/ro/sub/file", Mode: plainFileMode},
})
fs := afero.NewMemMapFs()
v := newContainmentVaultik(ctx, normalUserFs{Fs: fs})
_, err := v.restoreAllFiles(rows, repos,
&RestoreOptions{TargetDir: memTargetDir}, nil, nil)
require.NoError(t, err)
for _, path := range []string{"/ro/file", "/ro/sub/file"} {
_, err := fs.Stat(filepath.Join(memTargetDir, path))
require.NoErrorf(t, err, "file inside a read-only directory: %s", path)
}
for _, dir := range []string{"/ro", "/ro/sub"} {
info, err := fs.Stat(filepath.Join(memTargetDir, dir))
require.NoError(t, err)
assert.Equalf(t, os.FileMode(0o555), info.Mode().Perm(), "mode of %s", dir)
}
}
// TestRestoreKeepsNonEmptyDirectoryMTime checks that a directory keeps
// its stored mtime although entries were written into it. It runs on the
// real filesystem, where writing an entry changes its directory's mtime.
func TestRestoreKeepsNonEmptyDirectoryMTime(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
targetDir := t.TempDir()
mtime := time.Date(2001, time.February, 3, 4, 5, 6, 0, time.UTC)
rows, repos := makeFiles(ctx, t, []*database.File{
{Path: "/dir", Mode: plainDirMode, MTime: mtime},
{Path: "/dir/file", Mode: plainFileMode, MTime: mtime},
})
v := newContainmentVaultik(ctx, afero.NewOsFs())
_, err := v.restoreAllFiles(rows, repos,
&RestoreOptions{TargetDir: targetDir}, nil, nil)
require.NoError(t, err)
info, err := os.Stat(filepath.Join(targetDir, "dir"))
require.NoError(t, err)
assert.Truef(t, info.ModTime().Equal(mtime),
"directory mtime is %s, stored %s", info.ModTime(), mtime)
}
// TestRestoreFinishesChildBeforeUnsearchableParent checks that a
// directory inside one whose stored mode denies search still gets its
// own stored mode and mtime.
func TestRestoreFinishesChildBeforeUnsearchableParent(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
mtime := time.Date(2001, time.February, 3, 4, 5, 6, 0, time.UTC)
rows, repos := makeFiles(ctx, t, []*database.File{
{Path: "/locked", Mode: unsearchableMode, MTime: mtime},
{Path: "/locked/sub", Mode: plainDirMode, MTime: mtime},
})
fs := afero.NewMemMapFs()
v := newContainmentVaultik(ctx, normalUserFs{Fs: fs})
_, err := v.restoreAllFiles(rows, repos,
&RestoreOptions{TargetDir: memTargetDir}, nil, nil)
require.NoError(t, err)
info, err := fs.Stat(filepath.Join(memTargetDir, "locked"))
require.NoError(t, err)
assert.Equal(t, os.FileMode(0o600), info.Mode().Perm())
info, err = fs.Stat(filepath.Join(memTargetDir, "locked", "sub"))
require.NoError(t, err)
assert.Equal(t, os.FileMode(0o755), info.Mode().Perm())
assert.Truef(t, info.ModTime().Equal(mtime),
"mtime of sub is %s, stored %s", info.ModTime(), mtime)
}
// TestRestoreLeavesSymlinkedDirectoryTargetAlone checks that a directory
// whose place a later entry takes with a symlink does not hand its stored
// owner, mode and mtime to whatever the symlink points at. "/d" and "/d/"
// are different stored paths for the same place on disk.
func TestRestoreLeavesSymlinkedDirectoryTargetAlone(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
tempDir := t.TempDir()
targetDir := filepath.Join(tempDir, "target")
outsideDir := filepath.Join(tempDir, "outside")
require.NoError(t, os.Mkdir(outsideDir, writableTestMode))
before, err := os.Stat(outsideDir)
require.NoError(t, err)
mtime := time.Date(2001, time.February, 3, 4, 5, 6, 0, time.UTC)
rows, repos := makeFiles(ctx, t, []*database.File{
{
Path: "/d",
Mode: readOnlyDirMode,
UID: otherOwnerID,
GID: otherOwnerID,
MTime: mtime,
},
{Path: "/d/", LinkTarget: types.FilePath(outsideDir), MTime: mtime},
})
v := newContainmentVaultik(ctx, afero.NewOsFs())
_, err = v.restoreAllFiles(rows, repos,
&RestoreOptions{TargetDir: targetDir}, nil, nil)
require.NoError(t, err)
after, err := os.Stat(outsideDir)
require.NoError(t, err)
assert.Equal(t, before.Mode(), after.Mode())
assert.Truef(t, after.ModTime().Equal(before.ModTime()),
"mtime changed from %s to %s", before.ModTime(), after.ModTime())
beforeOwner, ok := before.Sys().(*syscall.Stat_t)
require.True(t, ok)
afterOwner, ok := after.Sys().(*syscall.Stat_t)
require.True(t, ok)
assert.Equal(t, beforeOwner.Uid, afterOwner.Uid)
assert.Equal(t, beforeOwner.Gid, afterOwner.Gid)
}
// TestRestoreKeepsSetuidThroughChown checks that a setuid file keeps the
// bit when restore changes its owner. Linux clears setuid on any chown of
// a regular file, even one to its current owner, so the file is recorded
// with the current user as owner and the session is told it runs as
// root: the chown then needs no privilege.
func TestRestoreKeepsSetuidThroughChown(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
targetDir := t.TempDir()
info, err := os.Stat(targetDir)
require.NoError(t, err)
owner, ok := info.Sys().(*syscall.Stat_t)
require.True(t, ok)
rows, repos := makeFiles(ctx, t, []*database.File{{
Path: "/suid",
Mode: setuidFileMode,
UID: owner.Uid,
GID: owner.Gid,
MTime: time.Date(2001, time.February, 3, 4, 5, 6, 0, time.UTC),
}})
session := &restoreSession{
v: newContainmentVaultik(ctx, afero.NewOsFs()),
ctx: ctx,
repos: repos,
opts: &RestoreOptions{TargetDir: targetDir},
result: &RestoreResult{},
runningAsRoot: true,
}
require.NoError(t, session.restoreFile(rows[0]))
info, err = os.Stat(filepath.Join(targetDir, "suid"))
require.NoError(t, err)
assert.Equal(t, os.FileMode(setuidFileMode),
info.Mode()&(os.ModeSetuid|os.ModePerm))
}
// TestRestoreSetsSymlinkMTime checks that a restored symlink gets its
// stored mtime on the link itself. The link dangles, so a call that
// follows it would fail.
func TestRestoreSetsSymlinkMTime(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
targetDir := t.TempDir()
mtime := time.Date(2001, time.February, 3, 4, 5, 6, 0, time.UTC)
rows, repos := makeFiles(ctx, t, []*database.File{{
Path: "/link",
LinkTarget: types.FilePath(symlinkTargetPath),
MTime: mtime,
}})
v := newContainmentVaultik(ctx, afero.NewOsFs())
_, err := v.restoreAllFiles(rows, repos,
&RestoreOptions{TargetDir: targetDir}, nil, nil)
require.NoError(t, err)
info, err := os.Lstat(filepath.Join(targetDir, "link"))
require.NoError(t, err)
assert.Truef(t, info.ModTime().Equal(mtime),
"symlink mtime is %s, stored %s", info.ModTime(), mtime)
}
// TestRestoreSetsSymlinkOwnerAsRoot checks that a symlink restored as
// root gets its stored owner on the link itself.
func TestRestoreSetsSymlinkOwnerAsRoot(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
if os.Geteuid() != 0 {
t.Skip("giving a file to another user needs root")
}
ctx := context.Background()
targetDir := t.TempDir()
rows, repos := makeFiles(ctx, t, []*database.File{{
Path: "/link",
LinkTarget: types.FilePath(symlinkTargetPath),
UID: otherOwnerID,
GID: otherOwnerID,
MTime: time.Date(2001, time.February, 3, 4, 5, 6, 0, time.UTC),
}})
v := newContainmentVaultik(ctx, afero.NewOsFs())
_, err := v.restoreAllFiles(rows, repos,
&RestoreOptions{TargetDir: targetDir}, nil, nil)
require.NoError(t, err)
info, err := os.Lstat(filepath.Join(targetDir, "link"))
require.NoError(t, err)
owner, ok := info.Sys().(*syscall.Stat_t)
require.True(t, ok)
assert.Equal(t, otherOwnerID, owner.Uid)
assert.Equal(t, otherOwnerID, owner.Gid)
}
internal/vaultik/restore_plan.go
+15 -1
View File
@@ -75,7 +75,7 @@ func newRestorePlan(
bc, ok := chunkToBlobMap[fc.ChunkHash.String()]
if !ok {
return nil, fmt.Errorf("planning %s: %w: %s",
f.Path, errPlanChunkMissing, fc.ChunkHash.String()[:16])
f.Path, errPlanChunkMissing, shortHash(fc.ChunkHash.String()))
}
hash, ok := blobIDToHash[bc.BlobID.String()]
@@ -214,6 +214,20 @@ func (p *restorePlan) blobsNeeded(fileID types.FileID) []string {
return out
}
// filesReferencingBlob returns the pending files that reference the
// named blob, in any order. The result is a copy, so the caller may
// finishFile each of them while ranging over it.
func (p *restorePlan) filesReferencingBlob(blobHash string) []types.FileID {
files := p.blobFiles[blobHash]
out := make([]types.FileID, 0, len(files))
for id := range files {
out = append(out, id)
}
return out
}
// hasPending reports whether any unfinished files remain.
func (p *restorePlan) hasPending() bool {
return len(p.fileBlobs) > 0
internal/vaultik/restore_skip_errors_test.go
+194
View File
@@ -0,0 +1,194 @@
package vaultik_test
import (
"bytes"
"context"
"fmt"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// A file no larger than the chunker's minimum chunk size (a quarter of
// faultChunkSize) is stored as one chunk, so it lives in exactly one blob.
// More of them than fit in faultMaxBlobSize make the snapshot span two
// blobs.
const (
missingBlobFileBytes = int(faultChunkSize / 4)
missingBlobFileCount = 20
)
// missingBlobBackup is a snapshot of single-chunk files spread over two
// blobs, with one of those blobs deleted from the store.
type missingBlobBackup struct {
fs afero.Fs
cfg *config.Config
storer storage.Storer
snapshotID string
restoreDir string
// files holds the original content by source path.
files map[string][]byte
// lost holds the source paths whose chunk is in the deleted blob.
lost map[string]bool
}
// TestRestoreSkipErrorsSkipsFilesOfMissingBlob restores with SkipErrors
// after one blob of a two-blob snapshot was deleted. Every file stored in
// that blob must be reported as failed and left absent, every other file
// must be restored intact, and Restore must still return an error.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
ctx := context.Background()
backup := backupThenDeleteOneBlob(ctx, t)
var out bytes.Buffer
v := newReaderVaultik(ctx, backup.cfg, backup.storer, nil, backup.fs)
v.UI = ui.NewWithColor(&out, false)
err := v.Restore(&vaultik.RestoreOptions{
SnapshotID: backup.snapshotID,
TargetDir: backup.restoreDir,
SkipErrors: true,
})
require.Error(t, err, "restore must fail when files were skipped")
assert.Contains(t, err.Error(),
fmt.Sprintf("%d file(s) failed to restore", len(backup.lost)))
for path, content := range backup.files {
restored := filepath.Join(backup.restoreDir, path)
if backup.lost[path] {
assert.Containsf(t, out.String(), path,
"%s needs the deleted blob and must be reported", path)
assert.NoFileExists(t, restored)
continue
}
got, err := afero.ReadFile(backup.fs, restored)
require.NoErrorf(t, err, "%s does not need the deleted blob", path)
assert.Equalf(t, content, got, "%s restored with wrong content", path)
}
}
// TestRestoreMissingBlobAbortsWithoutSkipErrors checks that a deleted blob
// still ends the restore with an error when SkipErrors is not set.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestRestoreMissingBlobAbortsWithoutSkipErrors(t *testing.T) {
ctx := context.Background()
backup := backupThenDeleteOneBlob(ctx, t)
v := newReaderVaultik(ctx, backup.cfg, backup.storer, nil, backup.fs)
err := v.Restore(&vaultik.RestoreOptions{
SnapshotID: backup.snapshotID,
TargetDir: backup.restoreDir,
})
require.ErrorIs(t, err, storage.ErrNotFound)
}
// backupThenDeleteOneBlob backs up missingBlobFileCount single-chunk
// files, then deletes from the store the blob holding the first of them.
func backupThenDeleteOneBlob(
ctx context.Context, t *testing.T,
) *missingBlobBackup {
t.Helper()
log.Initialize(log.Config{})
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src")
dbPath := filepath.Join(tempDir, "index.sqlite")
cfg := faultTestConfig()
require.NoError(t, fs.MkdirAll(dataDir, 0o755))
files := make(map[string][]byte, missingBlobFileCount)
for i := range missingBlobFileCount {
path := filepath.Join(dataDir, fmt.Sprintf("file-%02d.bin", i))
files[path] = bytesPattern(
fmt.Sprintf("file-%02d-", i), missingBlobFileBytes)
require.NoError(t, afero.WriteFile(fs, path, files[path], 0o644))
}
storer, err := storage.NewFileStorer(filepath.Join(tempDir, "remote"))
require.NoError(t, err)
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
repos := database.NewRepositories(db)
id := fullFaultBackup(
ctx, t, fs, storer, cfg, repos, dataDir, dbPath, "missingblob")
blobOfFile := make(map[string]string, len(files))
for path := range files {
blobOfFile[path] = blobHashOfSingleChunkFile(ctx, t, repos, path)
}
require.NoError(t, db.Close())
deleted := blobOfFile[filepath.Join(dataDir, "file-00.bin")]
require.NoError(t, storer.Delete(ctx, fmt.Sprintf(
"blobs/%s/%s/%s", deleted[:2], deleted[2:4], deleted)))
lost := make(map[string]bool)
for path, hash := range blobOfFile {
if hash == deleted {
lost[path] = true
}
}
require.Less(t, len(lost), len(files),
"the snapshot must span more than one blob")
return &missingBlobBackup{
fs: fs,
cfg: cfg,
storer: storer,
snapshotID: id,
restoreDir: filepath.Join(tempDir, "restored"),
files: files,
lost: lost,
}
}
// blobHashOfSingleChunkFile returns the hash of the blob holding the one
// chunk of the file at path, as recorded in the local index.
func blobHashOfSingleChunkFile(
ctx context.Context, t *testing.T,
repos *database.Repositories, path string,
) string {
t.Helper()
chunks, err := repos.FileChunks.GetByPath(ctx, path)
require.NoError(t, err)
require.Lenf(t, chunks, 1, "%s must be a single chunk", path)
blobChunk, err := repos.BlobChunks.GetByChunkHash(
ctx, chunks[0].ChunkHash.String())
require.NoError(t, err)
require.NotNilf(t, blobChunk, "chunk of %s is in no blob", path)
blob, err := repos.Blobs.GetByID(ctx, blobChunk.BlobID.String())
require.NoError(t, err)
return blob.Hash.String()
}
internal/vaultik/snapshot.go
+81 -67
View File
@@ -23,6 +23,9 @@ var (
errSnapshotVerifyFailed = errors.New("verification failed")
errRemoveAllNeedsForce = errors.New("--all requires --force")
errInvalidTableName = errors.New("invalid table name")
errNoAgeRecipients = errors.New(
"creating a snapshot needs at least one public key in " +
"age_recipients (generate a keypair with: age-keygen)")
)
// listRecentLimit caps how many snapshot rows are fetched from the
@@ -42,6 +45,12 @@ type SnapshotCreateOptions struct {
// CreateSnapshot executes the snapshot creation operation
func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
// config.Load accepts an empty list, since listing, verifying and
// restoring need no public key.
if len(v.Config.AgeRecipients) == 0 {
return errNoAgeRecipients
}
overallStartTime := time.Now()
log.Info("Starting snapshot creation",
@@ -180,6 +189,11 @@ type snapshotStats struct {
totalBytesUploaded int64
totalBlobsUploaded int
uploadDuration time.Duration
// The sizes of all blobs the snapshot references, set by
// finalizeSnapshotMetadata once snapshot_blobs is populated.
blobSize int64
blobUncompressedSize int64
}
// createNamedSnapshot creates a single named snapshot
@@ -219,8 +233,6 @@ func (v *Vaultik) createNamedSnapshot(
return err
}
v.collectUploadStats(scanner, stats)
err = v.finalizeSnapshotMetadata(snapshotID, stats)
if err != nil {
return err
@@ -272,6 +284,11 @@ func (v *Vaultik) resolveSnapshotPaths(snapName string) ([]string, error) {
func (v *Vaultik) scanAllDirectories(
scanner *snapshot.Scanner, resolvedDirs []string, snapshotID string,
) (*snapshotStats, error) {
if progress := scanner.GetProgress(); progress != nil {
progress.Start()
defer progress.Stop()
}
stats := &snapshotStats{}
for i, dir := range resolvedDirs {
@@ -300,6 +317,9 @@ func (v *Vaultik) scanAllDirectories(
stats.totalBytesSkipped += result.BytesSkipped
stats.totalFilesDeleted += result.FilesDeleted
stats.totalBytesDeleted += result.BytesDeleted
stats.totalBlobsUploaded += result.BlobsUploaded
stats.totalBytesUploaded += result.BytesUploaded
stats.uploadDuration += result.UploadDuration
log.Info("Directory scan complete",
"path", dir,
@@ -315,18 +335,6 @@ func (v *Vaultik) scanAllDirectories(
return stats, nil
}
// collectUploadStats gathers upload statistics from the scanner's
// progress reporter.
func (v *Vaultik) collectUploadStats(scanner *snapshot.Scanner, stats *snapshotStats) {
if s := scanner.GetProgress(); s != nil {
progressStats := s.GetStats()
stats.totalBytesUploaded = progressStats.BytesUploaded.Load()
stats.totalBlobsUploaded = int(progressStats.BlobsUploaded.Load())
stats.uploadDuration = time.Duration(
progressStats.UploadDurationMs.Load()) * time.Millisecond
}
}
// finalizeSnapshotMetadata updates stats, exports metadata, and only then
// marks the snapshot complete. Recording completion last is deliberate: an
// export interrupted by a crash leaves the snapshot incomplete rather than
@@ -336,31 +344,39 @@ func (v *Vaultik) collectUploadStats(scanner *snapshot.Scanner, stats *snapshotS
func (v *Vaultik) finalizeSnapshotMetadata(
snapshotID string, stats *snapshotStats,
) error {
// snapshot_blobs must be populated before the blob sizes below, which
// total the snapshot's blobs, and before the export, which builds the
// manifest and the trimmed metadata database from it.
err := v.SnapshotManager.PopulateSnapshotBlobs(v.ctx, snapshotID)
if err != nil {
return fmt.Errorf("populating snapshot blobs: %w", err)
}
stats.blobSize, stats.blobUncompressedSize, err =
v.Repositories.Snapshots.GetSnapshotBlobSizes(v.ctx, snapshotID)
if err != nil {
return fmt.Errorf("getting snapshot blob sizes: %w", err)
}
extStats := snapshot.ExtendedBackupStats{
BackupStats: snapshot.BackupStats{
FilesScanned: stats.totalFiles,
BytesScanned: stats.totalBytes,
TotalSize: stats.totalBytes + stats.totalBytesSkipped,
ChunksCreated: stats.totalChunks,
BlobsCreated: stats.totalBlobs,
BytesUploaded: stats.totalBytesUploaded,
},
BlobUncompressedSize: 0,
BlobSize: stats.blobSize,
BlobUncompressedSize: stats.blobUncompressedSize,
CompressionLevel: v.Config.CompressionLevel,
UploadDurationMs: stats.uploadDuration.Milliseconds(),
}
err := v.SnapshotManager.UpdateSnapshotStatsExtended(v.ctx, snapshotID, extStats)
err = v.SnapshotManager.UpdateSnapshotStatsExtended(v.ctx, snapshotID, extStats)
if err != nil {
return fmt.Errorf("updating snapshot stats: %w", err)
}
// snapshot_blobs must be populated before the export, which builds the
// manifest and the trimmed metadata database from it.
err = v.SnapshotManager.PopulateSnapshotBlobs(v.ctx, snapshotID)
if err != nil {
return fmt.Errorf("populating snapshot blobs: %w", err)
}
err = v.SnapshotManager.ExportSnapshotMetadata(
v.ctx, v.Config.IndexPath, snapshotID)
if err != nil {
@@ -395,12 +411,10 @@ func (v *Vaultik) printSnapshotSummary(
totalFilesChanged := stats.totalFiles - stats.totalFilesSkipped
totalBytesAll := stats.totalBytes + stats.totalBytesSkipped
// Get total blob sizes from database
compressedSize, uncompressedSize := v.getSnapshotBlobSizes(snapshotID)
var compressionRatio float64
if uncompressedSize > 0 {
compressionRatio = float64(compressedSize) / float64(uncompressedSize)
if stats.blobUncompressedSize > 0 {
compressionRatio = float64(stats.blobSize) /
float64(stats.blobUncompressedSize)
} else {
compressionRatio = 1.0
}
@@ -428,8 +442,8 @@ func (v *Vaultik) printSnapshotSummary(
if stats.totalBlobsUploaded > 0 {
v.UI.Detailf("Storage: %s compressed from %s (%.2fx ratio).",
v.UI.Size(compressedSize),
v.UI.Size(uncompressedSize),
v.UI.Size(stats.blobSize),
v.UI.Size(stats.blobUncompressedSize),
compressionRatio)
v.UI.Detailf("Upload: %d blobs, %s in %s (%s).",
stats.totalBlobsUploaded,
@@ -441,27 +455,6 @@ func (v *Vaultik) printSnapshotSummary(
v.UI.Detailf("Snapshot create duration: %s.", v.UI.Duration(snapshotDuration))
}
// getSnapshotBlobSizes returns total compressed and uncompressed blob
// sizes for a snapshot.
func (v *Vaultik) getSnapshotBlobSizes(snapshotID string) (int64, int64) {
var compressed, uncompressed int64
blobHashes, err := v.Repositories.Snapshots.GetBlobHashes(v.ctx, snapshotID)
if err != nil {
return 0, 0
}
for _, hash := range blobHashes {
blob, err := v.Repositories.Blobs.GetByHash(v.ctx, hash)
if err == nil && blob != nil {
compressed += blob.CompressedSize
uncompressed += blob.UncompressedSize
}
}
return compressed, uncompressed
}
// SnapshotPurgeOptions contains options for the snapshot purge command.
type SnapshotPurgeOptions struct {
KeepLatest bool // Keep only the most recent snapshot per name
@@ -718,7 +711,7 @@ func (v *Vaultik) VerifySnapshotWithOptions(
result.BlobCount = manifest.BlobCount
result.TotalSize = manifest.TotalCompressedSize
if !opts.JSON {
if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Snapshot information:\n")
v.stdoutf(" Blob count: %d\n", manifest.BlobCount)
v.stdoutf(" Total size: %s\n", ubytes(manifest.TotalCompressedSize))
@@ -773,7 +766,7 @@ func (v *Vaultik) printVerifyHeader(snapshotID string, opts *VerifyOptions) {
snapshotTime = t
}
if !opts.JSON {
if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Verifying snapshot %s\n", snapshotID)
if !snapshotTime.IsZero() {
@@ -813,7 +806,7 @@ func (v *Vaultik) verifyManifestBlobs(
stat, err := v.Storage.Stat(v.ctx, blobPath)
switch {
case err != nil:
if !opts.JSON {
if !opts.JSON && !v.UI.Quiet() {
v.stdoutf(" Missing: %s (%s)\n",
blob.Hash, ubytes(blob.CompressedSize))
}
@@ -821,7 +814,7 @@ func (v *Vaultik) verifyManifestBlobs(
missing++
missingSize += blob.CompressedSize
case stat.Size != blob.CompressedSize:
if !opts.JSON {
if !opts.JSON && !v.UI.Quiet() {
v.stdoutf(" Wrong size: %s (store has %s, manifest lists %s)\n",
blob.Hash, ubytes(stat.Size), ubytes(blob.CompressedSize))
}
@@ -853,6 +846,22 @@ func (v *Vaultik) formatVerifyResult(
return v.outputVerifyJSON(result)
}
// Under --quiet a failure is still returned, and the cli layer
// prints it on stderr.
if !v.UI.Quiet() {
v.printVerifySummary(result, failure)
}
if failure != "" {
return fmt.Errorf("%w: %s", errSnapshotVerifyFailed, failure)
}
return nil
}
// printVerifySummary prints the counts and the status line that end the
// human-readable shallow verify report. failure is empty when it passed.
func (v *Vaultik) printVerifySummary(result *VerifyResult, failure string) {
v.stdoutf("\nVerification complete:\n")
v.stdoutf(" Present with listed size: %d blobs\n", result.Verified)
@@ -874,14 +883,12 @@ func (v *Vaultik) formatVerifyResult(
if failure != "" {
v.stdoutf("FAILED - %s\n", failure)
return fmt.Errorf("%w: %s", errSnapshotVerifyFailed, failure)
return
}
// Report only what was actually checked: presence and size, not contents.
v.stdoutf("OK - all %d blobs listed in the manifest are present with the "+
"listed size; contents not checked (use --deep)\n", result.Verified)
return nil
}
// shallowVerifyFailure returns a human-readable description of everything
@@ -1139,7 +1146,7 @@ func (v *Vaultik) RemoveSnapshot(
}
if !opts.LocalOnly {
result.RemoteRemoved = v.removeSnapshotRemote(snapshotID)
result.RemoteRemoved = v.removeSnapshotRemote(snapshotID, opts)
}
if v.SnapshotManager != nil {
@@ -1222,9 +1229,11 @@ func (v *Vaultik) confirmRemoveSnapshot(snapshotID string, opts *RemoveOptions)
// removeSnapshotRemote strips the snapshot's metadata from the
// destination store, warning and proceeding on failure: the local-DB
// removal has already happened, so the user is told the remote half
// didn't finish and can retry with `vaultik prune` once the destination
// store is reachable. Returns true when the remote removal succeeded.
func (v *Vaultik) removeSnapshotRemote(snapshotID string) bool {
// didn't finish and to run `vaultik snapshot remove` for the snapshot
// again once the destination store is reachable (`vaultik prune` never
// removes snapshot metadata). Returns true when the remote removal
// succeeded.
func (v *Vaultik) removeSnapshotRemote(snapshotID string, opts *RemoveOptions) bool {
log.Info("Removing snapshot metadata from remote storage",
"snapshot_id", snapshotID)
@@ -1232,13 +1241,18 @@ func (v *Vaultik) removeSnapshotRemote(snapshotID string) bool {
err := v.deleteRemoteSnapshotByKey(remoteKey)
if err != nil {
log.Warn("Could not remove snapshot metadata from remote storage",
removeCommand := "vaultik snapshot remove " + snapshotID
log.Warn("Could not remove snapshot metadata from remote storage; "+
"run '"+removeCommand+"' again once the remote is reachable",
"error", err)
if v.UI != nil {
// The UI writes to stdout, which under --json holds only the
// document; the log record above is the warning on stderr.
if v.UI != nil && !opts.JSON {
v.UI.Warningf("Could not remove snapshot metadata from remote: "+
"%v. Run '%s' once the remote is reachable to finish cleanup.",
err, pruneCommandHint)
"%v. Run '%s' again once the remote is reachable.",
err, removeCommand)
}
return false
internal/vaultik/snapshot_summary_test.go
+285
View File
@@ -0,0 +1,285 @@
package vaultik_test
import (
"bytes"
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// These tests cover https://git.eeqj.de/sneak/vaultik/issues/225: the
// summary printed after a backup, and the statistics stored in the
// snapshots table, count each file, byte and upload once, and a --cron
// run records its uploads.
// summaryUploadDelay slows every blob upload, so a run's upload time is
// at least this long per blob even on a local store.
const summaryUploadDelay = 20 * time.Millisecond
// summaryEnv is a backup setup whose user-facing output is kept in out.
type summaryEnv struct {
v *vaultik.Vaultik
db *database.DB
repos *database.Repositories
out *bytes.Buffer
// aPath is a.bin, whose content copy.bin repeats; aSize is its size
// and totalSize the size of all three source files.
aPath string
aSize int64
totalSize int64
}
// newSummaryEnv writes src/one/a.bin, src/one/small.txt and
// src/two/copy.bin, a copy of a.bin. Every chunk of copy.bin is therefore
// already stored by the time the backup reaches it.
//
// The snapshot names "first" and "second" back up src; "split" backs up
// src/one and src/two as two paths.
func newSummaryEnv(t *testing.T) *summaryEnv {
t.Helper()
log.Initialize(log.Config{})
fs := afero.NewOsFs()
tempDir := t.TempDir()
srcDir := filepath.Join(tempDir, "src")
dirOne := filepath.Join(srcDir, "one")
dirTwo := filepath.Join(srcDir, "two")
dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background()
aContent := bytesPattern("a-", int(3*faultChunkSize))
smallContent := []byte("hello vaultik")
files := map[string][]byte{
filepath.Join(dirOne, "a.bin"): aContent,
filepath.Join(dirOne, "small.txt"): smallContent,
filepath.Join(dirTwo, "copy.bin"): aContent,
}
for path, content := range files {
require.NoError(t, fs.MkdirAll(filepath.Dir(path), 0o755))
require.NoError(t, afero.WriteFile(fs, path, content, 0o644))
}
cfg := faultTestConfig()
cfg.IndexPath = dbPath
cfg.ChunkSize = config.Size(faultChunkSize)
cfg.Snapshots = map[string]config.SnapshotConfig{
"first": {Paths: []string{srcDir}},
"second": {Paths: []string{srcDir}},
"split": {Paths: []string{dirOne, dirTwo}},
}
inner, err := storage.NewFileStorer(filepath.Join(tempDir, "remote"))
require.NoError(t, err)
store := faultstore.New(inner)
store.OnPut = func(key string) faultstore.PutAction {
if strings.HasPrefix(key, "blobs/") {
time.Sleep(summaryUploadDelay)
}
return faultstore.PutNormal
}
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
repos := database.NewRepositories(db)
out := &bytes.Buffer{}
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
v.UI = ui.NewWithColor(out, false)
return &summaryEnv{
v: v,
db: db,
repos: repos,
out: out,
aPath: filepath.Join(dirOne, "a.bin"),
aSize: int64(len(aContent)),
totalSize: int64(2*len(aContent) + len(smallContent)),
}
}
// backUp runs a backup of the named snapshot and returns its output.
func (e *summaryEnv) backUp(t *testing.T, name string, cron bool) string {
t.Helper()
e.out.Reset()
require.NoError(t, e.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Cron: cron,
Snapshots: []string{name},
}))
return e.out.String()
}
// snapshot returns the local snapshots row of the snapshot named name.
func (e *summaryEnv) snapshot(t *testing.T, name string) *database.Snapshot {
t.Helper()
ctx := context.Background()
snap, err := e.repos.Snapshots.GetByID(ctx,
localSnapshotID(ctx, t, e.repos, name))
require.NoError(t, err)
require.NotNil(t, snap)
return snap
}
// uploads returns how many blobs the snapshot uploaded and their
// total size, as recorded in the uploads table.
func (e *summaryEnv) uploads(t *testing.T, snapshotID string) (int64, int64) {
t.Helper()
var count, size int64
err := e.db.Conn().QueryRowContext(context.Background(), `
SELECT COUNT(*), COALESCE(SUM(size), 0)
FROM uploads WHERE snapshot_id = ?`, snapshotID).Scan(&count, &size)
require.NoError(t, err)
return count, size
}
// referencedBlobSizes returns the compressed and uncompressed sizes of
// all blobs the snapshot references.
func (e *summaryEnv) referencedBlobSizes(
t *testing.T, snapshotID string,
) (int64, int64) {
t.Helper()
var compressed, uncompressed int64
err := e.db.Conn().QueryRowContext(context.Background(), `
SELECT COALESCE(SUM(b.compressed_size), 0),
COALESCE(SUM(b.uncompressed_size), 0)
FROM snapshot_blobs sb JOIN blobs b ON b.blob_hash = sb.blob_hash
WHERE sb.snapshot_id = ?`, snapshotID).Scan(&compressed, &uncompressed)
require.NoError(t, err)
return compressed, uncompressed
}
// filesLine returns the summary's line of file counts.
func filesLine(examined, backedUp, unchanged int) string {
return fmt.Sprintf("Files: %d examined, %d backed up, %d unchanged.",
examined, backedUp, unchanged)
}
// dataLine returns the summary's line of byte counts.
func (e *summaryEnv) dataLine(total, backedUp int64) string {
return fmt.Sprintf("Data: %s total (%s backed up).",
e.v.UI.Size(total), e.v.UI.Size(backedUp))
}
// A first backup stores copy.bin's chunks while backing up a.bin, so
// copy.bin's chunks are deduplicated within the run. Each file and byte
// is still counted once.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestSnapshotSummaryFirstRun(t *testing.T) {
env := newSummaryEnv(t)
summary := env.backUp(t, "first", false)
assert.Contains(t, summary, filesLine(3, 3, 0))
assert.Contains(t, summary, env.dataLine(env.totalSize, env.totalSize))
snap := env.snapshot(t, "first")
uploadCount, uploadBytes := env.uploads(t, snap.ID.String())
require.Positive(t, uploadCount)
assert.Contains(t, summary, fmt.Sprintf("Upload: %d blobs, %s in ",
uploadCount, env.v.UI.Size(uploadBytes)))
assert.Equal(t, int64(3), snap.FileCount)
assert.Equal(t, env.totalSize, snap.TotalSize)
assert.Equal(t, uploadCount, snap.BlobCount)
assert.Equal(t, uploadBytes, snap.UploadBytes)
}
// An incremental backup where a.bin's mtime changed but its content did
// not: a.bin is backed up again and every one of its chunks is already
// stored.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestSnapshotSummaryIncrementalRunWithDeduplicatedChunks(t *testing.T) {
env := newSummaryEnv(t)
env.backUp(t, "first", false)
later := time.Now().Add(time.Hour)
require.NoError(t, os.Chtimes(env.aPath, later, later))
summary := env.backUp(t, "second", false)
assert.Contains(t, summary, filesLine(3, 1, 2))
assert.Contains(t, summary, env.dataLine(env.totalSize, env.aSize))
assert.NotContains(t, summary, "Upload:")
snap := env.snapshot(t, "second")
compressed, uncompressed := env.referencedBlobSizes(t, snap.ID.String())
require.Positive(t, compressed)
assert.Equal(t, env.totalSize, snap.TotalSize)
assert.Zero(t, snap.ChunkCount)
assert.Zero(t, snap.BlobCount)
assert.Zero(t, snap.UploadBytes)
assert.Equal(t, compressed, snap.BlobSize,
"blob_size must total the blobs the snapshot references")
assert.Equal(t, uncompressed, snap.BlobUncompressedSize)
}
// Under --cron the progress reporter is off; the upload figures must
// still reach the summary and the snapshots row. The snapshot has two
// paths, each backed up by its own scan.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestSnapshotSummaryCronRunRecordsUploads(t *testing.T) {
env := newSummaryEnv(t)
summary := env.backUp(t, "split", true)
snap := env.snapshot(t, "split")
uploadCount, uploadBytes := env.uploads(t, snap.ID.String())
require.Positive(t, uploadCount)
assert.Contains(t, summary, filesLine(3, 3, 0))
assert.Contains(t, summary, env.dataLine(env.totalSize, env.totalSize))
assert.Contains(t, summary, fmt.Sprintf("Upload: %d blobs, %s in ",
uploadCount, env.v.UI.Size(uploadBytes)))
assert.Equal(t, env.totalSize, snap.TotalSize)
assert.Equal(t, uploadCount, snap.BlobCount,
"blob_count must count each blob once, however many paths the "+
"snapshot has")
assert.Equal(t, uploadBytes, snap.UploadBytes)
assert.GreaterOrEqual(t, snap.UploadDurationMs,
uploadCount*summaryUploadDelay.Milliseconds())
compressed, uncompressed := env.referencedBlobSizes(t, snap.ID.String())
require.Positive(t, uncompressed)
assert.Equal(t, compressed, snap.BlobSize)
assert.Equal(t, uncompressed, snap.BlobUncompressedSize)
assert.InDelta(t, float64(compressed)/float64(uncompressed),
snap.CompressionRatio, 1e-9)
}
internal/vaultik/two_path_snapshot_test.go
+71
View File
@@ -0,0 +1,71 @@
package vaultik_test
import (
"context"
"maps"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// A backup without --cron runs the progress reporter while one scanner
// scans each path of the snapshot in turn. See
// https://git.eeqj.de/sneak/vaultik/issues/253.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupWithoutCronOfTwoPathSnapshotRestoresBothPaths(t *testing.T) {
log.Initialize(log.Config{})
const snapshotName = "data"
fs := afero.NewOsFs()
tempDir := t.TempDir()
firstDir := filepath.Join(tempDir, "first")
secondDir := filepath.Join(tempDir, "second")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background()
files := writeFaultSourceTree(t, fs, firstDir)
maps.Copy(files, writeFaultSourceTree(t, fs, secondDir))
cfg := faultTestConfig()
cfg.IndexPath = dbPath
cfg.ChunkSize = config.Size(faultChunkSize)
cfg.Snapshots = map[string]config.SnapshotConfig{
snapshotName: {Paths: []string{firstDir, secondDir}},
}
store, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
repos := database.NewRepositories(db)
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
require.NoError(t, v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Snapshots: []string{snapshotName},
}))
id := localSnapshotID(ctx, t, repos, snapshotName)
require.NoError(t, db.Close())
reader := newReaderVaultik(ctx, cfg, store, nil, fs)
require.NoError(t, reader.Restore(&vaultik.RestoreOptions{
SnapshotID: id,
TargetDir: restoreDir,
Verify: true,
}))
assertRestoredTree(t, fs, restoreDir, files)
}
internal/vaultik/verify.go
+15 -12
View File
@@ -103,7 +103,7 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
log.Info("Starting snapshot verification", "snapshot_id", snapshotID, "mode", "deep")
if !opts.JSON {
if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Deep verification of snapshot: %s\n\n", snapshotID)
}
@@ -143,10 +143,13 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
log.Info("✓ Verification completed successfully",
"snapshot_id", snapshotID, "mode", "deep", "blobs_verified", len(dbBlobs))
v.stdoutf("\n✓ Verification completed successfully\n")
v.stdoutf(" Snapshot: %s\n", snapshotID)
v.stdoutf(" Blobs verified: %d\n", len(dbBlobs))
v.stdoutf(" Total size: %s\n", ubytes(totalSize))
if !v.UI.Quiet() {
v.stdoutf("\n✓ Verification completed successfully\n")
v.stdoutf(" Snapshot: %s\n", snapshotID)
v.stdoutf(" Blobs verified: %d\n", len(dbBlobs))
v.stdoutf(" Total size: %s\n", ubytes(totalSize))
}
return nil
}
@@ -170,7 +173,7 @@ func (v *Vaultik) loadVerificationData(
// remote manifests; see its doc comment.
log.Info("Downloading manifest", "remote_key", remoteKey)
if !opts.JSON {
if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Downloading manifest...\n")
}
@@ -185,7 +188,7 @@ func (v *Vaultik) loadVerificationData(
"manifest_blob_count", manifest.BlobCount,
"manifest_total_size", ubytes(manifest.TotalCompressedSize))
if !opts.JSON {
if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Manifest loaded: %d blobs (%s)\n",
manifest.BlobCount, ubytes(manifest.TotalCompressedSize))
v.stdoutf("Downloading and decrypting database...\n")
@@ -215,7 +218,7 @@ func (v *Vaultik) loadVerificationData(
"db_blob_count", len(dbBlobs),
"db_total_size", ubytes(dbTotalSize))
if !opts.JSON {
if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Database loaded: %d blobs (%s)\n",
len(dbBlobs), ubytes(dbTotalSize))
}
@@ -273,7 +276,7 @@ func (v *Vaultik) runVerificationSteps(
totalSize int64,
identities []age.Identity,
) error {
if !opts.JSON {
if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Verifying manifest against database...\n")
}
@@ -282,7 +285,7 @@ func (v *Vaultik) runVerificationSteps(
return v.deepVerifyFailure(result, opts, err.Error(), err)
}
if !opts.JSON {
if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Manifest verified.\n")
v.stdoutf("Checking blob existence in remote storage...\n")
}
@@ -292,7 +295,7 @@ func (v *Vaultik) runVerificationSteps(
return v.deepVerifyFailure(result, opts, err.Error(), err)
}
if !opts.JSON {
if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("All blobs exist.\n")
v.stdoutf("Downloading and verifying blob contents (%d blobs, %s)...\n",
len(dbBlobs), ubytes(totalSize))
@@ -748,7 +751,7 @@ func (v *Vaultik) performDeepVerificationFromDB(
"eta", eta.Round(time.Second),
)
if !opts.JSON {
if !opts.JSON && !v.UI.Quiet() {
v.stdoutf(" Verified %d/%d blobs (%d remaining) - %s/%s - elapsed %s, eta %s\n",
i+1, len(blobs), remaining,
ubytes(bytesProcessed),
internal/vaultik/verify_quiet_test.go
+102
View File
@@ -0,0 +1,102 @@
package vaultik_test
import (
"bytes"
"context"
"io"
"os"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestVerify_QuietSuppressesReport is the --quiet contract for
// `snapshot verify`: neither shallow nor deep verify writes its report,
// a failed verify still returns its error (which the cli layer prints
// on stderr), and the --json document still emits.
func TestVerify_QuietSuppressesReport(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source")
storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(32 * 1024)
maxBlobSize := int64(128 * 1024)
require.NoError(t, fs.MkdirAll(dataDir, 0o755))
require.NoError(t, afero.WriteFile(fs,
filepath.Join(dataDir, "data.bin"),
bytesPattern("quiet-", int(maxBlobSize*2)), 0o644))
ctx := context.Background()
cfg, storer, snapshotID := runFileStorageBackup(
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
// The UI writes to the same buffer as Stdout, as both write to the
// process's stdout in production.
var stdout bytes.Buffer
newQuietVerifier := func() *vaultik.Vaultik {
v := &vaultik.Vaultik{
Config: cfg,
Storage: storer,
Fs: fs,
Stdout: &stdout,
Stderr: io.Discard,
UI: ui.NewWithColor(&stdout, false),
}
v.SetContext(ctx)
v.UI.SetQuiet(true)
return v
}
require.NoError(t, newQuietVerifier().VerifySnapshotWithOptions(
snapshotID, &vaultik.VerifyOptions{}))
require.Empty(t, stdout.String(),
"shallow verify must write no report under --quiet")
require.NoError(t, newQuietVerifier().VerifySnapshotWithOptions(
snapshotID, &vaultik.VerifyOptions{Deep: true}))
require.Empty(t, stdout.String(),
"deep verify must write no report under --quiet")
require.NoError(t, newQuietVerifier().VerifySnapshotWithOptions(
snapshotID, &vaultik.VerifyOptions{JSON: true}))
require.Equal(t, "ok", decodeVerifyResult(t, stdout.Bytes()).Status,
"the --json document must still emit under --quiet")
// A snapshot without its encrypted database fails shallow verify. A
// failed report also lists each missing blob and each blob of the
// wrong size, so remove one blob and grow another.
require.NoError(t, os.Remove(filepath.Join(storeDir, "metadata",
snapshot.RemoteSnapshotKey(snapshotID), "db.zst.age")))
blobFiles, err := filepath.Glob(
filepath.Join(storeDir, "blobs", "*", "*", "*"))
require.NoError(t, err)
require.GreaterOrEqual(t, len(blobFiles), 2,
"the snapshot must span two blobs, one to remove and one to grow")
require.NoError(t, os.Remove(blobFiles[0]))
growOneBlob(t, fs, filepath.Join(storeDir, "blobs"))
stdout.Reset()
require.Error(t, newQuietVerifier().VerifySnapshotWithOptions(
snapshotID, &vaultik.VerifyOptions{}),
"--quiet must not change the outcome of a failed verify")
require.Empty(t, stdout.String(),
"a failed verify must write no report under --quiet")
}
script/bootstrap
+9 -8
View File
@@ -38,12 +38,7 @@ pkg_install() {
detect_pkgmgr
case "$PKGMGR" in
nix) nix-env -iA "nixpkgs.$1" ;;
# A fresh image, such as the CI runner's, has no package lists,
# so apt-get install finds nothing until they are fetched.
apt)
$SUDO apt-get update
$SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2"
;;
apt) $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2" ;;
brew) brew install "$3" ;;
apk) apk add --no-cache "$4" ;;
esac
@@ -109,8 +104,14 @@ main() {
if missing git; then pkg_install git git git git; fi
if missing make; then pkg_install gnumake make make make; fi
# Go toolchain
if missing go; then pkg_install go golang go go; fi
# Go toolchain: the host's own, or else the version go.mod names,
# hash-verified, in .tool/go. That directory is not on the caller's
# PATH; this script, the Makefile, script/fmt, script/fmt-check,
# script/precommit and script/release add it to theirs.
if missing go; then
"$ROOT/script/install-go"
PATH="$PATH:$ROOT/.tool/go/bin"
fi
# golangci-lint is deliberately NOT installed: script/lint lints by
# building the lint phase of the Dockerfile, whose digest-pinned
script/fmt
+2
View File
@@ -6,6 +6,8 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
# Where script/bootstrap installs Go when the host has none.
PATH="$PATH:$ROOT/.tool/go/bin"
go fmt ./...
}
script/fmt-check
+8 -1
View File
@@ -7,7 +7,14 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
unformatted="$(gofmt -l .)"
# Where script/bootstrap installs Go when the host has none.
PATH="$PATH:$ROOT/.tool/go/bin"
# .tool/go holds that toolchain's own sources, which are not ours.
if ! unformatted="$(find . -path ./.tool -prune -o -name '*.go' -type f \
-exec gofmt -l {} +)"; then
echo "fmt-check: gofmt failed" >&2
exit 1
fi
if [ -n "$unformatted" ]; then
echo "Files not formatted:" >&2
echo "$unformatted" >&2
script/install-go
+22 -19
View File
@@ -4,13 +4,13 @@
# own extension to scripts-to-rule-them-all. Idempotent: exits at once
# when the pinned toolchain is already installed.
#
# Only .gitea/workflows/release.yml calls this. goreleaser is not a
# .gitea/workflows/release.yml calls this, and so does script/bootstrap
# when the host has no Go, as on the check runner. goreleaser is not a
# compiler: it shells out to `go` for the `before:` hook and for every
# one of the four cross-compiles, so the release runner needs a Go
# toolchain on PATH. check.yml compiles nothing on the host: its one
# host use of Go is gofmt in script/fmt-check, with whatever Go
# script/bootstrap finds or installs. So this is the release path's only
# host Go, and per REPO_POLICIES.md it must be pinned by hash.
# toolchain on PATH. The check runner compiles nothing on the host; it
# uses this Go only for bootstrap's `go mod download` and for gofmt in
# script/fmt-check. Per REPO_POLICIES.md a host Go is pinned by hash.
# actions/setup-go exposes no checksum input, so Go is installed the way
# script/install-goreleaser installs goreleaser: download the exact
# archive from go.dev and refuse it unless its sha256 matches the value
@@ -19,12 +19,11 @@
# The version is go.mod's `go` directive, the single source of truth for
# the toolchain. GO_VERSION below MUST equal it, and this script fails
# when they disagree -- so bumping Go is one reviewed change touching
# go.mod, the checksum here, and the Dockerfile's two golang digests
# go.mod, the checksums here, and the Dockerfile's two golang digests
# together.
#
# Linux only, because that is what the release runner is. A darwin dev
# building a snapshot uses their own Go; supporting an OS means adding
# its checksums.
# Linux and macOS, each on amd64 and arm64: the four archives whose
# checksums are committed below.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
@@ -35,6 +34,8 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
GO_VERSION="1.26.1"
SHA256_LINUX_AMD64="031f088e5d955bab8657ede27ad4e3bc5b7c1ba281f05f245bcc304f327c987a"
SHA256_LINUX_ARM64="a290581cfe4fe28ddd737dde3095f3dbeb7f2e4065cab4eae44dfc53b760c2f7"
SHA256_DARWIN_AMD64="65773dab2f8cc4cd23d93ba6d0a805de150ca0b78378879292be0b903b8cdd08"
SHA256_DARWIN_ARM64="353df43a7811ce284c8938b5f3c7df40b7bfb6f56cb165b150bc40b5e2dd541f"
GOROOT_DIR="$ROOT/.tool/go"
GOCMD="$GOROOT_DIR/bin/go"
@@ -103,26 +104,28 @@ main() {
arch="$(uname -m)"
case "$os" in
Linux) os="linux" ;;
Darwin) os="darwin" ;;
*)
echo "install-go: unsupported OS $os (release runner is Linux)" >&2
echo "install-go: unsupported OS $os" >&2
exit 1
;;
esac
case "$arch" in
x86_64 | amd64)
arch="amd64"
sum="$SHA256_LINUX_AMD64"
;;
arm64 | aarch64)
arch="arm64"
sum="$SHA256_LINUX_ARM64"
;;
x86_64 | amd64) arch="amd64" ;;
arm64 | aarch64) arch="arm64" ;;
*)
echo "install-go: no pinned checksum for architecture $arch" >&2
echo "install-go: unsupported architecture $arch" >&2
exit 1
;;
esac
case "${os}-${arch}" in
linux-amd64) sum="$SHA256_LINUX_AMD64" ;;
linux-arm64) sum="$SHA256_LINUX_ARM64" ;;
darwin-amd64) sum="$SHA256_DARWIN_AMD64" ;;
darwin-arm64) sum="$SHA256_DARWIN_ARM64" ;;
esac
archive="go${GO_VERSION}.${os}-${arch}.tar.gz"
url="https://go.dev/dl/${archive}"
script/precommit
+2
View File
@@ -9,6 +9,8 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
# Where script/bootstrap installs Go when the host has none.
PATH="$PATH:$ROOT/.tool/go/bin"
go mod tidy
go fmt ./...
git diff --exit-code -- go.mod go.sum || {
script/release
+2
View File
@@ -44,6 +44,8 @@ resolve_goreleaser() {
main() {
cd "$ROOT"
# Where script/bootstrap installs Go when the host has none.
PATH="$PATH:$ROOT/.tool/go/bin"
if ! bin="$(resolve_goreleaser)"; then
cat >&2 <<EOF