sneak/vaultik
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

base: sneak/vaultik:24f0dabf5103fdf56f0a0a08d957fb113fa3e7a4
Branches Tags
sneak/vaultik:main sneak/vaultik:next
..
compare: sneak/vaultik:next
Branches Tags
sneak/vaultik:next sneak/vaultik:main
70 Commits
24f0dabf51 .. next
Author SHA1 Message Date
clawbot fa56165fe4 Stop a cancelled backup at once under --skip-errors (closes #286)
check / check (push) Canceled after 0s
Details 
After Ctrl-C or SIGTERM, phase 2 of a --skip-errors backup treated the
cancellation error from each remaining file like an unreadable file: it
opened the file, printed an error line, counted it as failed and moved
on to the next. The processing loop now checks for cancellation before
each file, and an error from a file once the run is cancelled stops the
run instead of being skipped. The loop check is what stops a run
cancelled while an empty file is open, since an empty file has no
chunks and nothing reads the context while it is backed up.

The --skip-errors test helper now takes the scan's context.

Model: opus-5-5
 2026-10-08 15:12:07 +02:00
clawbot 34f61fb408 Make remote nuke delete leftover .partial uploads (closes #281)
check / check (push) Canceled after 0s
Details 
The file and rclone listings skip an object whose name ends in
`.partial`, the temporary name a `file://` or rclone upload writes
before moving the object into place. `remote nuke` deletes only what
the listings return, so it left the `.partial` objects killed uploads
leave behind and still reported the destination store empty.

Storer gains DeletePartialUploads. The file and rclone backends remove
every `.partial` object under the prefix; S3 has none to remove, since
it shows an object only once its upload completes. `remote nuke` calls
it for `metadata/` and `blobs/` as its last step.

Empty directories under a `file://` destination are still left behind.

Model: opus-5-5
 2026-10-08 14:17:20 +02:00
clawbot 05bf73c48a Never skip a local index error for a directory or symlink (closes #284)
check / check (push) Canceled after 0s
Details 
Under --skip-errors, an error recording a directory or symlink in the
local index was skipped like an unreadable file. The snapshot completed
without the entry, and a restore did not recreate it.

processFileWithErrorHandling now records a directory or symlink itself
and returns any error, which stops the backup with or without the flag.
The skip sees only a regular file's errors, so countFailedFile no longer
checks for a directory.

The test injects the error with a SQLite trigger that refuses the
directory's files row.

Model: opus-5-5
 2026-10-08 12:59:26 +02:00
clawbot 79a73fa122 Count a file a backup could not store as failed (closes #280)
check / check (push) Canceled after 0s
Details 
A file that phase 1 of a backup counted and phase 2 could not open,
because it was unreadable under --skip-errors or removed in between,
was added to the unchanged count while its size stayed in
BytesScanned. The summary showed it as unchanged with its bytes backed
up, and the snapshots row's file_count and total_size included it.

The scanner now counts such a file in FilesFailed and takes its size
out of BytesScanned. The summary's files line adds "N failed", and
file_count leaves the file out.

A directory phase 2 cannot record is not counted as failed, since
phase 1 counts no directories; that case has no test.

Model: opus-5-5
 2026-10-08 10:12:10 +02:00
clawbot 0d0368df81 Report unknown blob figures for a snapshot whose manifest cannot be read (closes #272)
check / check (push) Canceled after 0s
Details 
When remote info could not read a snapshot's manifest, the orphan
figures were unknown but the snapshot's row still gave 0 blobs and 0 B,
in the table and in --json. The row's blob count and blob size are now
unknown, and null in --json.

A directory with no manifest, as an interrupted backup leaves, still
shows 0: the orphan figures count its blobs as orphaned, so it
references none.

The constant holding the "unknown" text is renamed from countUnknown to
unknownText, since it now also stands for a size.

Model: opus-5-5
 2026-10-08 08:46:11 +02:00
clawbot c06c4d201b Keep the progress line of a multi-path snapshot within 100% (closes #271)
check / check (push) Canceled after 0s
Details 
One progress reporter spans every path of a snapshot. Its counts of
files and bytes processed run across all the paths, while the totals
they were divided by were reset to each path's own, so a second path
smaller than the first showed more than 100% and an ETA of `unknown`.
The totals now add up over the paths scanned so far. The rate behind
the ETA restarts when each path's scan phase ends, so that phase does
not lower the rate while the path is processed; until then the rate is
the previous path's. SetTotalSize is renamed AddTotalSize because it
now adds.

The issue says the ETA went negative; it was computed negative and
printed as `unknown`.

Model: opus-5-5
 2026-10-08 07:29:11 +02:00
clawbot 70f008a21a Document that --json skips the confirmation prompt (closes #268)
check / check (push) Canceled after 0s
Details 
snapshot remove and prune delete without asking under --json, because a
prompt on stdout would break the JSON document. Their --json help and
README entries named only --force as skipping the prompt. Both now say
--json skips it too. One test checks the --json help of both commands;
another runs prune with --json, no --force and empty stdin, and checks
that the unreferenced blob is deleted and stdout holds only the JSON
document.

Model: opus-5-5
 2026-10-08 04:59:30 +02:00
clawbot fce253fa39 Give a second snapshot create in the same second its own ID (closes #270)
check / check (push) Canceled after 0s
Details 
The timestamp in a snapshot ID is in whole seconds, so a `snapshot
create` that started in the same second as the previous run of that
snapshot name got the same ID, and inserting its row failed with
`UNIQUE constraint failed: snapshots.id`. CreateSnapshotWithName now
looks the ID up in the local index first and, if it is taken, waits a
second and takes a new timestamp. The ID format is unchanged.

Only the local index is checked. That is where the insert fails, and
the process lock serializes runs, so nothing takes the ID between the
lookup and the insert.

Model: opus-5-5
 2026-10-08 03:29:07 +02:00
clawbot e459a66099 Stop a backup on a symlink whose target cannot be read (closes #269)
check / check (push) Canceled after 0s
Details 
When readlink failed, the scanner logged at debug level and left the
symlink out of the snapshot, and the run reported success even without
--skip-errors. The error now goes through the same handling as any
other entry the walk cannot read: the run aborts, or with --skip-errors
the symlink is skipped with the usual "Failed to access" error line.

A symlink removed between the walk's lstat and the readlink also
aborts the run, as an entry that vanishes during the walk already does.

Model: opus-5-5
 2026-10-08 00:12:13 +02:00
clawbot 32c4a46b49 Write rclone uploads under a temporary name and move them into place (closes #266)
check / check (push) Canceled after 0s
Details 
The rclone backend wrote each object straight to its key, so killing an
upload to a local or sftp remote left a truncated object there that the
next backup trusted.

On every remote with a server-side move, an object is now written under
a name ending in `.partial` and moved onto its key with rclone's
operations.Move, which removes an object already at the key first;
drive, dropbox and others will not move onto one. Listings skip
`.partial` names. Rclone's own copy also requires the PartialUploads
flag; this does not, because hdfs shows a file while it is written
without setting it. Remotes without a move are written in place.

Model: opus-5-5
 2026-10-07 22:59:26 +02:00
clawbot b5389a62b5 Exit 130 and say so when a command is interrupted (closes #267)
check / check (push) Canceled after 0s
Details 
Ctrl-C or SIGTERM during snapshot create, restore or verify exited 0
with no error line (1, also silent, under snapshot verify --json), so
an unfinished --cron backup looked like a success. RunOperation now
records whether op returned while the Vaultik context was still live;
a run where it had not by the time RunWithApp returned is interrupted,
whatever op returned. It used to look for context.Canceled in op's
error, which verify --json does not return. Entry prints "interrupted
before the command finished" on stderr for it and returns 130, under
--cron and --json too.

SIGTERM also gives 130, as the issue asks, not 143.
The test does not cover an op still running when the 30s shutdown
timeout ends.

Model: opus-5-5
 2026-10-07 21:59:27 +02:00
clawbot d87202fb70 Correct doc and help sentences that are false about the code (closes #233)
check / check (push) Canceled after 0s
Details 
A blob is written whole to a temporary file and uploaded once finished,
not streamed to storage. The README, ARCHITECTURE.md and
config.example.yml now say a backup needs free temporary space for each
blob (twice that for an rclone destination that cannot stream uploads)
and for the metadata export's copies of the local index, in $TMPDIR, or
partly in /var/tmp when TMPDIR is unset. Also corrected: the snapshot ID
format, what restore reads and how incomplete snapshots are removed in
docs/DATAMODEL.md, what source_path holds, the index_path default, the
config search order in the snapshot create help, what snapshot remove
cleans up in the prune help, how the release installs Go, and the
script/release and script/fmt-check comments.

Model: opus-5-5
 2026-10-07 17:59:27 +02:00
clawbot e161343eac Speed up the vaultik and database package tests (closes #235)
check / check (push) Canceled after 0s
Details 
In the Dockerfile test phase, internal/vaultik spent 23.6s of 31.9s in
36 tests run one at a time. 24 of them were serial only because they
call log.Initialize; they now call it before t.Parallel(), so the
logger is replaced before any parallel test runs. The 12 still serial
change the umask, TMPDIR, os.Stderr or time.Local.

TestLargeDatasets took 8.4s of the 9.8s internal/database run by
committing each of its 1,500 inserts on its own; it now makes them in
one transaction. TestDedupOnlySnapshotRestores gives its second backup
its own snapshot name instead of sleeping 1.1s for a new snapshot ID.

Model: opus-5-5
 2026-10-07 17:12:07 +02:00
clawbot d53202eb86 Fix two misleading messages (closes #240)
check / check (push) Canceled after 0s
Details 
The warning for a config file that others can read always said the
file contained S3 credentials, so a file:// config with none got a
false claim. When s3.access_key_id or s3.secret_access_key is set it
now says the file may contain them, because Load sees the values only
after smartconfig has replaced any ${...} reference, so a set
credential need not be in the file. Otherwise it says the file is
readable by others.

snapshot purge wrapped the listing error, which already starts with
"listing remote snapshots:", in that prefix a second time.
syncWithRemote now returns it unwrapped, as CleanupLocalSnapshots
does.

Model: opus-5-5
 2026-10-07 15:29:07 +02:00
clawbot 8b22ae8d42 Pass s3.part_size to the multipart uploader (closes #232)
check / check (push) Canceled after 0s
Details 
s3.part_size was loaded and defaulted but never reached the S3 client,
whose uploader used a fixed 10MiB part. The client now takes the part
size from the config, for storage_url and for the s3.* fields, and
config load rejects a value below 5MiB or above 5GiB, an explicit 0
included. A blob too large for S3's limit of 10,000 parts at that size
is uploaded in larger parts, since the uploader cannot learn the size of
the reader it is given. The docs gave the default as 5MB, which the
config file reads as 5,000,000 bytes, below the minimum; they now say
5MiB.

Judgement call: the 5GiB maximum is enforced along with the 5MiB
minimum the issue names.

Model: opus-5-5
 2026-10-07 14:29:10 +02:00
clawbot 3fc8a8f2f4 Read the snapshot name using the stored hostname (closes #230)
check / check (push) Canceled after 0s
Details 
A snapshot ID is hostname_name_timestamp, and purge took the name to be
everything between the first and the last underscore. With a hostname
such as my_host the name home came out as host_home, so
`snapshot purge --keep-latest --snapshot home` found nothing to delete
and `snapshot create --prune` purged nothing without a message. The name
is now read by removing the hostname stored with the snapshot, in the
short form the ID uses, so both may contain underscores. This was chosen
over rejecting underscores in `hostname` when the config loads, which
would also stop restores on such a host.

The purge consistency test stored a hostname that did not match its
snapshot IDs; it now matches, as it always does in production.

Model: opus-5-5
 2026-10-07 12:12:08 +02:00
clawbot 7696f83258 Leave remote info orphan figures unknown when a manifest is unreadable (closes #228)
check / check (push) Canceled after 0s
Details 
When a manifest could not be read, remote info skipped it, counted that
snapshot's blobs as orphaned and advised running prune. The orphan
figures are now unknown in that case, with no prune advice; --json gives
them as null and lists the unreadable remote keys in
unreadable_manifests. Only a listed manifest.json.zst is read, so a
directory without one, as an interrupted backup leaves, keeps the
figures known.

Names under metadata/ were used unchecked and printed raw. A name that
is not a remote key is now skipped with a warning. A manifest under it
is not read either, so it also leaves the figures unknown; --json counts
such manifests in skipped_manifest_count.

Model: opus-5-5
 2026-10-07 10:59:26 +02:00
clawbot 5d1118d143 Quote a string setting that YAML would read as a number (closes #229)
check / check (push) Canceled after 0s
Details 
config set wrote every value as an unquoted YAML scalar, and config.Load
reads the file through untyped YAML, so an access key 00112233 loaded as
38043 and a hostname 007 as 7.

config set now looks the key up in config.Config by the fields' yaml
tags. A string setting is tagged !!str, which the encoder quotes
wherever YAML would read a number or a boolean. Other settings stay
unquoted, so compression_level 9 is still a number. A value that is not
valid UTF-8 stays untagged and is written as !!binary, which loads back
unchanged.

Judgement call: the type comes from reflection over config.Config.

Model: opus-5-5
 2026-10-07 09:29:19 +02:00
clawbot b57ce2277d Store and compare file mtimes to the nanosecond (closes #226)
check / check (push) Canceled after 0s
Details 
The files table held mtime in whole seconds and the scanner compared
whole seconds. A file rewritten with its size unchanged and a new mtime
in the same second as the indexed one was treated as unchanged, and
every later snapshot restored the old content. A new mtime_nsec column
now holds the nanoseconds within the second that mtime holds, and the
scanner compares the full mtime.

A local index created before this change lacks the column and is
rebuilt with `vaultik database delete` and a full backup. A snapshot
made before it cannot be restored by this version.

Model: opus-5-5
 2026-10-07 07:12:12 +02:00
clawbot 8496404d8b Make the process-wide lock atomic with flock (closes #227)
check / check (push) Canceled after 0s
Details 
Acquire read vaultik.pid, checked whether that PID was alive, then
wrote its own, so two writers started together could both pass the
check and both run. The lock is now an flock on vaultik.pid, held
while the file stays open; the kernel drops it when the process exits,
so the stale-PID check is gone.

Release empties the file instead of deleting it. Deleting it would let
a process that opened the old file a moment earlier lock it while
another creates and locks a new one.

The new concurrent test fails against the old code only when the race
is hit, not on every run; against the fix it cannot admit two callers.

Model: opus-5-5
 2026-10-07 06:12:10 +02:00
clawbot cdc60c4dfa Write only the document to stdout from snapshot remove --json (closes #251)
check / check (push) Canceled after 0s
Details 
When the destination store cannot be reached, `snapshot remove` still
removes the snapshot from the local index and warns. The warning went
through the UI, which writes to stdout, so under `--json` it landed
ahead of the document and `| jq` failed on a command that exited 0.
Under `--json` the UI warning is now skipped; the logger's warning on
stderr carries the follow-up, with the snapshot ID as a field.

The follow-up, in the warning, the README and the command's help, said
`vaultik prune` would finish the cleanup, but `prune` never removes
snapshot metadata. They now say to run `vaultik snapshot remove` for the
snapshot again once the destination store is reachable.

Model: opus-5-5
 2026-10-07 04:46:13 +02:00
clawbot 49eed7a3e5 Count each file, byte and upload once in backup statistics (closes #225)
check / check (push) Canceled after 0s
Details 
The scanner added a changed file's bytes again for each new chunk and
counted a file as unchanged for each chunk already stored. It now
counts files and bytes once, in the scan phase, and counts its own
uploads, so a --cron run, which has no progress reporter, records
them. The blob count no longer adds earlier paths' blobs again.

The snapshots row now stores the size of all files in total_size and
the referenced blobs' sizes in blob_size, blob_uncompressed_size and
compression_ratio, as docs/DATAMODEL.md says. Those sizes come from
one query, and a failed query fails the snapshot. DATAMODEL.md now
says chunk_count and blob_count count what the run added.

Removed UpdateSnapshotStats and GetCountBySnapshot, which nothing
calls any more.

Model: opus-5-5
 2026-10-07 02:59:27 +02:00
clawbot 85d4ef118d Keep command output to the README's stdout and stderr rules (closes #224)
check / check (push) Successful in 16m49s
Details 
The startup banner moves from stdout to stderr, so a `completion`
script, a `config get` value and the hidden `__complete` command print
only their own output. `--quiet`, `--cron` and `--json` still suppress
it.

A failing `remote info`, `prune` or `snapshot remove` under `--json`
now reports its error on stderr. Their reporters returned early under
`--json`, so the failure reached neither stream.

`snapshot verify --quiet` writes no report. A failure is still
returned and printed on stderr, with the same exit status.

Judgement call: the banner's stream, posted on the issue for the owner.

Model: opus-5-5
 2026-10-07 01:12:15 +02:00
clawbot b06f992152 Start the progress reporter once per snapshot, not once per path (closes #253)
check / check (push) Successful in 19m54s
Details 
A backup without --cron of a snapshot with two or more paths panicked
with "close of closed channel". Scan runs once per path, and it started
the progress reporter and deferred its Stop each time; Stop closes the
reporter's signal channel, so the second path's Stop panicked.
scanAllDirectories now starts the reporter before the first path and
stops it after the last, and Scan no longer starts or stops it. A new
test backs up a two-path snapshot with the reporter on and restores
both paths.

Model: opus-5-5
 2026-10-07 00:12:10 +02:00
clawbot 5d685f03ce List the files under a path by path, not by string prefix (closes #223)
check / check (push) Successful in 12m16s
Details 
FileRepository.ListByPrefix matched with SQL LIKE: a plain string
prefix that ignores ASCII case and treats _ and % as wildcards.
Restoring /home/u/doc also restored doc2, DOC and doc.txt.bak, and a
backup counted the files of a longer sibling path as deleted. It is
now ListUnderPath, which returns the file at the path and every file
whose path starts with the path plus a slash, compared exactly. A
trailing slash is ignored, so "/" still lists every file.

Three tests relied on string-prefix matching and now name a full path
or call ListAll.

ListIDsWithChunksNotInUploadedBlobs keeps its LIKE: it only adds file
IDs the scan never looks up.

Model: opus-5-5
 2026-10-06 22:12:16 +02:00
clawbot f59086c0e5 Return an error, not a panic, on a malformed snapshot database (closes #231)
check / check (push) Successful in 13m39s
Details 
Restore cut chunk hashes from the snapshot database to 16 characters
for its error messages, so a shorter hash panicked. Those messages now
use shortHash. Under --verify, a file_chunks row with no chunks row was
dereferenced, and the chunk size from the database was allocated in
one piece, so a negative or huge size panicked. A missing row is now an
error, a negative size is rejected, and each chunk is hashed by
streaming it from the restored file.

A restored file shorter than its chunks now fails verify as a short
read instead of an unexpected EOF.

Model: opus-5-5
 2026-10-06 21:16:28 +02:00
clawbot d276d891da Join the S3 prefix to every key with one slash (closes #222)
check / check (push) Successful in 10m59s
Details 
The S3 client built each key as prefix + key, and the URL parser keeps
the prefix as written, so s3://bucket/p stored p + "blobs/..." with no
slash while s3://bucket/p/ stored p/blobs/.... A recovery host that wrote
the URL the other way found no snapshots.

NewClient now strips trailing slashes from the prefix and adds one back
when anything is left, giving the README layout for both URL forms; an
empty prefix stays at the bucket root. The s3.prefix config setting
goes through the same client and gets the same join.

A new test writes through each URL shape against an in-process S3
server, checks the key in the bucket, and lists through both List and
ListStream, which every snapshot listing uses.

Model: opus-5-5
 2026-10-06 19:46:09 +02:00
clawbot 5aa5ba5544 Apply restored owners, modes and times in an order that keeps them (closes #219)
check / check (push) Successful in 11m53s
Details 
A directory got its stored mode and mtime before its contents were
written, so a read-only directory came back without its files and a
non-empty one carried the time of the restore. Directories are now
created owner-only (0700) and get their stored owner, mode and mtime
after the restore loop, each before its parent, skipping any whose
place a symlink has since taken. A file's mode is now applied after its
chown, which on Linux clears setuid and setgid. A symlink gets its
stored owner (as root) and mtime on the link itself, through
golang.org/x/sys/unix, now a direct dependency.

An interrupted restore leaves its directories at 0700.

Model: opus-5-5
 2026-10-06 18:29:09 +02:00
clawbot 315b6483b8 Mark github.com/spf13/pflag as a direct dependency in go.mod (closes #246)
check / check (push) Successful in 12m34s
Details 
internal/cli/snapshot_restore_test.go imports github.com/spf13/pflag
directly, but go.mod still marked it // indirect. script/precommit runs
go mod tidy and fails when that changes go.mod, so the pre-commit hook
stopped every commit. This is the go mod tidy output: pflag moves to the
direct require block, and go.sum does not change. script/cibuild does
not run the tidy, which is why the gate stayed green.

Model: opus-5-5
 2026-10-06 16:59:26 +02:00
clawbot 14fc4c9893 Load a config that has no age recipient (closes #221)
check / check (push) Successful in 13m47s
Details 
The README's steps for restoring on another machine failed at the first
command: `config init` wrote a placeholder recipient, and `config.Load`
rejects any recipient that does not parse. `config init` now writes an
empty `age_recipients` list, `config.Load` accepts an empty list, and
`snapshot create` refuses to start without a recipient. A malformed
recipient is still rejected at load.

The recovery-host test now builds its config with `config init` and
`config set` and reads it through `config.Load`, so it imports
`internal/cli`. On a fresh file, `config set age_recipients.0` writes the
list in flow style (`[age1...]`).

Model: opus-5-5
 2026-10-06 15:46:13 +02:00
clawbot 66c80a70e3 Skip files of an unreadable blob under restore --skip-errors (closes #218)
check / check (push) Successful in 12m34s
Details 
A blob that failed to download ended `snapshot restore` even with
--skip-errors, after restoring whichever files came first. The download
error now goes through the same per-file handling as any other restore
error, once for every pending file that references the blob. With
--skip-errors those files are reported as failed, the rest are
restored, and the command still exits non-zero. Without the flag the
restore still aborts; the error now also names one affected file and
suggests --skip-errors. A cancelled restore still ends at once. The
--skip-errors help text and README now limit the packing and storage
caveat to snapshot creation.

Model: opus-5-5
 2026-10-06 14:46:05 +02:00
clawbot 81f83b29f6 Re-vendor the canonical files from sneak/prompts at dd4027b (closes #213)
check / check (push) Successful in 12m47s
Details 
Linting and testing become the lint and test phases of the Dockerfile,
and the build stage depends on both. Dockerfile.lint, CHECK_EPOCH and
the tests that checked them are removed. Every docker build in script/
passes --no-cache, and script/cibuild runs script/bootstrap first. A
host without Go gets the go.mod version from script/install-go in
.tool/go, which bootstrap, the Makefile, fmt, fmt-check, precommit and
release add to PATH; fmt-check skips .tool. The image takes its version
from the VERSION build arg or git describe, dev without .git. This
repo's own entries follow the canonical content in .gitignore and
.editorconfig. The golangci-lint v2.14.0 findings are fixed. The rules
in CLAUDE.md move into AGENTS.md. IsDevVersion counts "unknown".

Model: opus-5-5
 2026-10-06 12:46:13 +02:00
clawbot c4adb72d80 Run the local index in WAL mode with a busy timeout (closes #217)
check / check (push) Successful in 5m10s
Details
check / check (pull_request) Successful in 5m38s
Details 
The connection settings were passed as `_journal_mode=`-style
parameters, which the SQLite driver drops without an error, so the
index ran in rollback-journal mode with no busy timeout. `snapshot
list` or `info` reading during a backup could make the backup's next
write fail with "database is locked". Both open paths now pass
`_pragma=` parameters; foreign keys moved there too.

With WAL on, rows committed to the open index can still be in the
-wal file, which a copy of the main file misses. The metadata export
now copies the index with VACUUM INTO, into an empty 0600 file.

The retry after a failed open no longer claims a TRUNCATE recovery; it
retries with the same settings.

Model: opus-5-5
 2026-10-06 11:29:18 +02:00
clawbot 4a167e153a Record the real uid and gid of backed-up files (closes #216)
check / check (push) Successful in 4m58s
Details
check / check (pull_request) Successful in 6m21s
Details 
The scanner read uid and gid by asserting the stat result to an
interface with Uid() and Gid() methods. *syscall.Stat_t has Uid and Gid
fields, not methods, so the assertion never matched and every file,
directory and symlink was stored as 0:0; a restore as root then gave
everything to root. The scanner now reads the fields of
*syscall.Stat_t.

The first backup after this change re-reads every file not owned by
root, because its stored uid and gid no longer match the disk.

When the tests run as root, as in the Docker build, the new test
compares 0 with 0 and cannot catch the defect; a non-root run does.

Model: opus-5-5
 2026-10-06 09:46:16 +02:00
clawbot ea72697992 List a missing file:// destination directory as an error (closes #220)
check / check (push) Successful in 5m53s
Details
check / check (pull_request) Successful in 4m53s
Details 
The file backend listed a destination directory that does not exist as
an empty store. With the volume unplugged, snapshot list reported every
local snapshot as missing from the store, snapshot remove said it had
removed metadata it never reached, and prune dropped every local
snapshot record. List and ListStream now fail when the destination
directory is missing, so those commands take their existing path for a
store that cannot be listed. A missing prefix under an existing
directory is still an empty listing, and a first backup still creates
the directory.

Three tests listed a file:// destination nothing had created; they now
create it.

Model: opus-5-5
 2026-10-06 08:46:17 +02:00
clawbot 713be502bd Reject a duration with characters outside its parts (closes #215)
check / check (push) Successful in 7m28s
Details
check / check (pull_request) Successful in 5m39s
Details 
parseDuration fell back to an unanchored search for number-and-unit
pieces when time.ParseDuration failed, and skipped everything in
between. 1.0y became 0, so `snapshot create --prune --keep-newer-than
1.0y` deleted every snapshot of the backed-up names, the new one
included. 2.1w became one week and 1,5y five years. The fallback now
requires the whole input to be whole-number-and-unit parts with nothing
between them. A bare number is rejected before time.ParseDuration sees
it, since Go reads 0 and +0 as zero with no unit.

Judgement call: a space between number and unit (`30 days`) was
accepted and is now an error, matching Go's own units.

Model: opus-5-5
 2026-10-06 06:12:07 +02:00
clawbot 35cf985c18 Re-chunk a known file whose chunks no uploaded blob holds (closes #214)
check / check (push) Successful in 6m53s
Details
check / check (pull_request) Successful in 6m17s
Details 
File rows are shared by every snapshot and updated in place, while a
blob row is deleted once no snapshot references it. Removing the newest
snapshot, or the prune after an interrupted run, could drop the only
blob holding a changed file's current chunks while an older snapshot
kept the file row. The next backup compared metadata only, skipped the
file, and completed a snapshot that could not restore it.

The scanner now loads the IDs of known files that list a chunk no
uploaded blob holds and re-chunks them even when their metadata is
unchanged.

The tests append to a file, so the file keeps its first chunk in a blob
the first snapshot still references. Each backup run gets its own
snapshot name, so the second-precision snapshot IDs differ without
sleeping.

Model: opus-5-5
 2026-10-06 04:46:16 +02:00
clawbot 070090124a Stamp the tag or short commit in a plain docker build (closes #211)
check / check (push) Successful in 3m35s
Details
check / check (pull_request) Successful in 3m16s
Details 
A plain `docker build .` stamped `dev`: `.dockerignore` left out `.git`
and the Dockerfile defaulted VERSION to `dev`. `.dockerignore` now
sends `.git` without `.git/config`. Given no build arguments, the
builder stamps `git describe --tags --always` and the commit and date
from git, and fails if `.git` is present but yields no version. The
empty CHECK_EPOCH refusal is gone so the plain build succeeds.
`script/version` now prints `git describe --tags --always --dirty`, so
make, the scripts and a plain build agree. `vaultik version` treats
the short commit, tag-N-gHASH forms and any version ending in `-dirty`
as development builds, so they keep the development-build notice.

Model: opus-5-5
 2026-10-02 10:04:14 +02:00
clawbot 584444b619 List only after-1.0 work in the README roadmap (closes #208)
check / check (push) Successful in 3m35s
Details
check / check (pull_request) Successful in 3m36s
Details 
The README roadmap and the TODO.md Next Step still described finished
1.0 work as remaining. The roadmap now lists only work planned after
1.0. Its security item says the code was reviewed before 1.0, every bug
found was fixed, and the accepted risks are listed; an outside audit
stays as after-1.0 work. The error-condition item is gone because every
failure case it listed has a fault-injection test. Daemon mode is added.
TODO.md says the 1.0 work is complete on next and that merging and
tagging are the owner's.

Judgement call: dropped the human-readable size flags item; no command
flag takes a raw-integer size.

Model: opus-5-5
 2026-10-01 21:41:41 +02:00
clawbot b30e79ee45 Run the disk-full restore test again (closes #207)
check / check (push) Successful in 4m52s
Details
check / check (pull_request) Successful in 5m19s
Details 
TestRestoreReportsDiskFull was skipped pending
#163, which is closed. The skip
and its "skipped until" wording are removed.

Restore is unchanged. With the skip removed the test failed because
restore succeeded: its simulated full disk capped only Create, but
restore now opens each file with OpenFile, so nothing was capped. It now
caps OpenFile, and only for files under the restore target: restore also
writes the decrypted metadata database under $TMPDIR through the same
filesystem, and capping that would fail the restore before any file
reached the target.

Judgement call: the test was corrected, not restore; both assertions are
unchanged.

Model: opus-5-5
 2026-10-01 20:24:30 +02:00
sneak d886a9026f Merge branch 'main' into next
check / check (push) Successful in 4m8s
Details
check / check (pull_request) Successful in 2m51s
Details
2026-09-29 03:02:24 +02:00
clawbot 6e1f499048 Document that migrations are supported and none are added before 1.0 (closes #68)
check / check (push) Successful in 3m6s
Details
check / check (pull_request) Successful in 3m8s
Details 
The docs now say vaultik supports migrations. The numbered files in `internal/database/schema/` are migrations: `schema_migrations` records which have run, and opening a database applies any that have not. None are added before 1.0 because nothing is installed anywhere yet, so a schema change edits `001.sql` directly. After 1.0 each change is a new numbered file, and an existing local database is migrated when vaultik is updated.

`docs/DATAMODEL.md` owns the explanation. The README caveat and roadmap entry and `AGENTS.md` policy 13 link to it. This replaces the wording from #146, which said there was no upgrade path.

Disclosure: `CLAUDE.md` line 33, the owner's file, changes from "do not need to support migrations" to "do not add migrations before 1.0".

Model: opus-5-5
 2026-09-28 20:21:38 +02:00
sneak 15e6506e5d next: integrate accumulated work into main (#114)
check / check (push) Successful in 2m35s
Details 
Reviewed-on: #114
 2026-09-23 13:03:02 +02:00
clawbot d24f5dc33c Adopt the canonical golangci-lint config (closes #90)
check / check (push) Successful in 3m9s
Details
check / check (pull_request) Successful in 1m29s
Details 
The lint config is now the canonical file from `prompts`, which replaces the deprecated `gomodguard` with `gomodguard_v2`, so lint prints no deprecation warnings. It also turns on the `depguard` `test-support` rule. The one difference from canonical is that the deny list names vaultik's own test-only package `internal/storage/faultstore`, so shipped code cannot import it. The new config found nothing to fix in the source.

Issues and PRs that pin the old `.golangci.yml` sha256 as an untouched-file check need the new one: `7122fcf0dd0ea57441374f98ebd98bb3da23decb67f9209fee5175170838fbd1`.

Model: opus-5-5
 2026-09-23 02:14:40 +02:00
clawbot eed117fe25 Route direct-stdout command output through internal/ui (closes #149)
check / check (pull_request) Successful in 1m27s
Details
check / check (push) Successful in 3m18s
Details 
version, info, remote info, config, and database delete wrote plain text straight to stdout, so they were unstyled and ignored --quiet. Output is now governed by internal/ui in two buckets. Status lines and confirmations (config init, config set, the database delete prompt) go through the ui message methods and are silenced by --quiet. The data a command exists to produce is written plain -- the version/info/remote-info reports, the snapshot list table, config get values, and the --json documents -- and is never suppressed, since a script depends on it and a marker would corrupt a table or document. The database delete confirmation prompt is always shown. Pure-cli commands reach ui through a small commandUI helper.

Model: opus-4-8
 2026-09-22 20:28:50 +02:00
clawbot dd7a610c23 Mark a snapshot complete only after its metadata export succeeds (closes #177)
check / check (pull_request) Successful in 1m42s
Details
check / check (push) Successful in 3m35s
Details 
finalizeSnapshotMetadata marked the snapshot complete and then exported its metadata. A crash after completion but before/during the export left the local index showing the snapshot complete while the destination had no manifest or database, and PruneDatabase (which drops only NULL completed_at rows) kept it: a silently unrestorable snapshot.

Reorder so completion is recorded last. CompleteSnapshot is split into PopulateSnapshotBlobs (before the export) and MarkSnapshotComplete (after it). An interrupted export now leaves the snapshot incomplete, so the next run PruneDatabase drops it and re-backs-up the data; the reverse tiny window leaves a restorable snapshot the index reports honestly as remote-only. Update REPOSTRUCTURE.md guarantee 4 and the ARCHITECTURE.md flow. Add a fault-injection test driving the full create path.

Model: opus-4-8
 2026-09-22 20:00:41 +02:00
clawbot 1548c0f933 Correct the security claims in docs and comments, and record the accepted risks (closes #171)
check / check (pull_request) Successful in 1m58s
Details
check / check (push) Successful in 2m49s
Details 
Docs and comments only; no behaviour change. Corrects ten overclaims the security review found: snapshot names are hashed but the hash uses no secret, so a guessed hostname and name can be confirmed; a blob is named by hex(SHA256(SHA256(uncompressed contents))), stated once in docs/REPOSTRUCTURE.md and referenced elsewhere; double hashing does not hide known content (blob packing does); age uses ChaCha20-Poly1305, not XChaCha20; encryption is required, not optional; a snapshot is marked complete before its metadata is uploaded; the export comment now matches its only caller; deep verify detects corruption, not authorship; adding a recipient does not reach existing data; restore examples target a user-owned directory.

Adds an Accepted Risks subsection under Security Considerations with the seven documented risks, cross-referenced from the README.

Model: opus-4-8
 2026-09-22 19:28:31 +02:00
clawbot c3bec7d3aa Reject a metadata database truncated to the age header and nonce (closes #152)
check / check (push) Successful in 2m5s
Details
check / check (pull_request) Successful in 2m28s
Details 
An object holding just the age header and its 16-byte nonce decrypts without error: the truncated read surfaces as io.ErrUnexpectedEOF at the age layer, which the zstd decoder maps to a clean EOF at frame start. blobgen then reported zero bytes and no error, so a truncated stream was indistinguishable from a valid empty one, and the metadata database export slipped through -- restore built a fresh schema on the empty file and reported success.

blobgen.Reader.Read now, on EOF, reads once more from the age reader and surfaces io.ErrUnexpectedEOF unless that read is (0, io.EOF), the state a genuine end leaves. downloadSnapshotDB additionally rejects a zero-length decrypted database before any schema is built.

Model: opus-4-8
 2026-09-22 18:11:57 +02:00
clawbot 1244c9e48d Bound download expansion and escape control chars on the terminal (closes #164)
check / check (pull_request) Successful in 1m47s
Details
check / check (push) Successful in 3m11s
Details 
Objects fetched from the store are untrusted; several decode paths let one expand or print without limit.

- blobgen.LimitReader errors past a byte cap (not io.LimitReader silent EOF). DecodeManifest reads through caps on both compressed input and decompressed output, far above any real manifest, so json.Decode cannot buffer a compressible bomb. FetchAndDecryptBlob bounds decompression to the blob recorded uncompressed_size (not the restoring host blob_size_limit).
- downloadSnapshotDB streams straight from storage to its temp file with io.Copy, replacing two ReadAll calls that held the whole database twice.
- FetchBlob drops the per-blob Stat round-trip, its expectedSize parameter and returned size, all of which only fed a debug log.
- TTYHandler and ui.Writer escape control characters in messages, attribute keys/values, and rendered identifiers/paths before colour codes are applied, so a crafted value cannot drive the terminal.

Model: opus-4-8
 2026-09-22 17:00:35 +02:00
clawbot 82c51a5337 Validate blob hashes, offsets and lengths from the destination (closes #155)
check / check (pull_request) Successful in 1m22s
Details
check / check (push) Successful in 3m30s
Details 
A blob hash read back from the downloaded snapshot database or the store listing was trusted unchecked. A hostile remote could set a hash such as "aa/../../etc" and have a decrypted blob written outside the cache directory, or feed a short or negative value that panicked a command.

blobDiskCache.path now refuses any key with a path separator, and ReadAt rejects a negative offset or length, bounding so a sum cannot overflow past the check. A new isBlobHash helper gates FetchBlob, shallow and deep verify, and restore: buildBlobIndexes rejects every hash from the snapshot database before any fetch. The blobs/ and metadata/ listings skip a non-conforming name, and short-hash prefixes in log and error text go through a panic-safe shortHash helper.

Model: opus-4-8
 2026-09-22 16:11:28 +02:00
clawbot d88ed64489 Parse the age identity key once and accept every identity in it (closes #165)
check / check (pull_request) Successful in 2m41s
Details
check / check (push) Successful in 2m56s
Details 
Restore and verify --deep now parse the configured age secret key a single time through a new helper that uses age.ParseIdentities and hands every identity to age.Decrypt. A key file with several identities (a whole age-keygen file) is fully accepted, so a blob encrypted to any of its recipients decrypts, not just the first.

The helper is the first step of both commands, so a missing or unparseable key fails before anything is downloaded. Its error names the config source and never echoes the key value. config.extractAgeSecretKey and its silent fallback are removed; the key is stored raw and parsed only where decryption happens. README, the restore help, and the missing-key error now read the key from a file with \$(cat ...) rather than typed literally, keeping it out of shell history.

Model: opus-4-8
 2026-09-22 15:45:27 +02:00
clawbot bd9656dbd4 Reject a decrypted snapshot database that is not the requested one (closes #156)
check / check (push) Successful in 1m26s
Details
check / check (pull_request) Successful in 2m51s
Details 
Restore and deep verify downloaded and decrypted metadata/<key>/db.zst.age by object name alone. age decryption proves the database is readable, not that it is the snapshot that was asked for: an attacker who swaps in another valid db.zst.age could redirect the operation, and deep verify with a swapped database plus an empty manifest reported success with zero blobs verified.

After the database is opened, both paths now confirm its identity: an exported per-snapshot database holds one snapshot row, and a snapshot remote key derives from that row ID, so the database is the requested one exactly when its sole snapshot hashes back to the remote key fetched. The shared check lives in verifySnapshotDBIdentity, backed by a new SnapshotRepository.GetOnlySnapshot.

Model: opus-4-8
 2026-09-22 15:01:02 +02:00
clawbot 7e611b95db Check blob sizes and the database in shallow verify (closes #169)
check / check (push) Successful in 1m21s
Details
check / check (pull_request) Successful in 2m37s
Details 
Shallow snapshot verify only checked that each blob object existed and then reported "All blobs verified", overstating what it did.

It now compares each blob stored size against the manifest compressed_size, using the same comparison as the deep path, and checks that the snapshot encrypted database (db.zst.age) is present. A blob of the wrong size no longer counts as verified. The final line reports only what was checked: presence and size, not contents.

The README verify description and the CLI short/long text are corrected to match. Removed the now-unused resolveAndDownloadManifest helper and errBlobsMissing sentinel.

Model: opus-4-8
 2026-09-22 14:28:44 +02:00
clawbot 4f27608560 Add negative and boundary tests for blobgen and types (closes #170)
check / check (push) Successful in 1m22s
Details
check / check (pull_request) Successful in 1m18s
Details 
Test-only. internal/blobgen and internal/types had no negative or boundary coverage. Adds, in package blobgen_test: Writer-to-Reader round trips at the 64 KiB age-segment edges for random and compressible data, checking plaintext, byte counts and the reader/writer hashes by decrypting; a wrong-identity open; truncation and single-byte corruption of a multi-segment blob at every region; trailing bytes, empty input and garbage; rejected and accepted compression levels; nil, empty and invalid recipients; and a failing destination. In package types_test: Value/Scan round trips, NULL, wrong-type and malformed Scan, Parse and IsZero for FileID and BlobID.

The "cut right after the age header and nonce" truncation is excluded: it reads as valid and empty today and belongs to #152.

Model: opus-4-8
 2026-09-22 14:28:34 +02:00
clawbot ae6aaaa388 Wait for the interrupted operation to clean up before exit (closes #159)
check / check (pull_request) Successful in 1m22s
Details
check / check (push) Successful in 3m13s
Details 
On SIGINT/SIGTERM the process could exit before the interrupted command cleanup defers ran, leaving decrypted data in the temp directory (the blob cache and the decrypted snapshot database).

RunApp now mirrors fx run sequence: start, block on app.Wait(), then app.Stop(), returning only after Stop completes. fx delivers both an OS interrupt and the finished operation Shutdowner.Shutdown() on one channel. Stop runs the OnStop hooks; the operation hook cancels the command and waits for its goroutine to return (bounded by shutdownTimeout) before exit. The old code returned as soon as app.Done fired, without Stop, so a real interrupt unwound to os.Exit while cleanup still ran. Restore loops check the context between chunks and blobs so the wait ends promptly. A cli test drives RunApp through the OnStop hook.

Model: opus-4-8
 2026-09-22 14:00:49 +02:00
clawbot f788668287 Remove the unused crypto path and write the blob-ID hash step once (closes #151)
check / check (push) Successful in 1m21s
Details
check / check (pull_request) Successful in 2m39s
Details 
Production encryption and decryption already run through blobgen; the crypto package (Encryptor, Decryptor, UpdateRecipients, the fx Module) and Vaultik.GetEncryptor/GetDecryptor had no production caller. Delete crypto and route verify --deep through the same blobgen reader restore uses, parsing the age key once.

The second blob-ID hash step is now one exported blobgen.DoubleSHA256; Writer.Sum256 (the double hash) becomes Writer.ContentID so it no longer collides with Reader.Sum256 (the single plaintext hash). Also delete the never-adopted internal/types newtypes and the uncalled CleanupIncompleteSnapshots and its now-dead deleteSnapshot caller, and correct ARCHITECTURE.md. No production behavior changes.

Model: opus-4-8
 2026-09-22 13:46:02 +02:00
clawbot 238ce3985f Scrub example config of real credentials and internal hosts (closes #172)
check / check (push) Successful in 1m20s
Details
check / check (pull_request) Successful in 2m41s
Details 
config.example.yml carried a real-looking 20-char S3 access key id and 40-char secret, a private-address http:// endpoint, and a storage_url naming an internal rclone remote and pool path. Replace them with the same neutral placeholders the config init template uses: YOUR_ACCESS_KEY / YOUR_SECRET_KEY, a https://s3.example.com endpoint, a mybucket bucket, and rclone://myremote/path/to/backups. No behavior or other keys change.

The credentials live in the commented-out s3 block, which the loader never parses, so the new test reads the file raw text to assert the placeholders are present and no http:// endpoint remains, and also loads it to confirm the active storage_url still parses.

Model: opus-4-8
 2026-09-22 13:45:52 +02:00
clawbot 548a7ae156 Give the local index and its export copy an explicit 0600 mode (closes #168)
check / check (pull_request) Successful in 1m23s
Details
check / check (push) Successful in 2m57s
Details 
The local index lists every backed-up path and chunk hash, but its file mode was left to the SQLite driver and the umask, so under a typical 022 umask a fresh index (and its -wal/-shm side files) landed world-readable. The snapshot export copied the index to snapshot.db with a permissive create as well.

provideDatabase now calls ensureIndexFileMode before opening the driver: it creates the index 0600 if missing and chmods an existing one to 0600. Doing this before the driver opens the file matters because SQLite gives its -wal and -shm files the mode of the main database file. The export copy is now created 0600. Tests under umask 022 cover a fresh index, an existing 0644 index, and the export copy.

Model: opus-4-8
 2026-09-22 13:12:07 +02:00
clawbot 3a58377127 Parse age_recipients at config load and never echo the entry (closes #153)
check / check (push) Successful in 1m23s
Details
check / check (pull_request) Successful in 1m18s
Details 
Config.Validate now parses every age_recipients entry with age.ParseX25519Recipient, so a bad recipient fails at config load instead of deep in a backup after the snapshot row and tree walk. On failure the error names the position (age_recipients[N]) and never the value: a recipient string can itself be a secret key an operator pasted by mistake, and age's own error quotes its input. An entry starting with AGE-SECRET-KEY- gets a specific message.

The remaining parse sites (blobgen.NewWriter, crypto NewEncryptor and UpdateRecipients), reachable by callers that skip config.Load, likewise drop the value and age's wrapped error, naming only the position.

Model: opus-4-8
 2026-09-22 13:01:00 +02:00
clawbot a6434de57f Open the downloaded snapshot database read-only, on a private temp dir (closes #162)
check / check (pull_request) Successful in 1m21s
Details
check / check (push) Successful in 3m4s
Details 
Restore and deep verify used to open the decrypted snapshot database read-write through the local-index constructor, which applied migrations against whatever the file carried, and left the decrypted file in the shared temp directory. A forged file could redefine what restore queries return, and an interrupted open left decrypted metadata on disk.

Add database.OpenReadOnly: opens the file read-only (mode=ro) with query_only and trusted_schema=OFF, never applies schema files, and refuses a file whose schema carries a trigger, view or virtual table or lacks an expected table. Restore and deep verify now both use it, each inside its own private (0700) temp directory removed on every return path. pickNextDownload returns (FileID, bool) so a genuine nil-UUID file is not mistaken for "nothing left".

Model: opus-4-8
 2026-09-22 12:45:53 +02:00
clawbot b4654f8e52 Abort the run when packing fails, even under --skip-errors (closes #161)
check / check (push) Successful in 1m22s
Details
check / check (pull_request) Successful in 3m2s
Details 
A chunk is registered as pending (known, scanner-pending, packer pending-row) before it is packed. Under --skip-errors the scanner skipped a file on any processing error, including a failure inside addChunkToPacker (packing, database, encryption, upload). The pending chunk then stayed queued and a later blob finalize inserted it into the chunks table with no blob_chunks row, so a snapshot could complete holding a file whose chunk is in no blob and cannot be restored.

Errors from addChunkToPacker are now marked and abort the run regardless of --skip-errors; only open and read errors are skipped. The bookkeeping order is unchanged. Flag help and comments now say only unreadable files are skipped.

Model: opus-4-8
 2026-09-22 12:28:44 +02:00
clawbot 39aef1c47c Stop config set echoing secrets; reject credential-bearing storage URLs (closes #166)
check / check (push) Successful in 1m21s
Details
check / check (pull_request) Successful in 1m18s
Details 
config set now prints only the key name after a write, never the value: a value may be a secret such as s3.secret_access_key, and echoing it leaks into captured stdout and pasted terminals. The set logic moves into writeConfigSet so this is testable.

config set also tightens a pre-existing group- or world-readable config to 0600 after writing; the previous stat-and-preserve-mode block had no effect (os.WriteFile does not change an existing file mode) and is removed.

ParseStorageURL now rejects s3:// and rclone:// URLs that carry credentials in the userinfo or an unknown query parameter, naming s3.access_key_id and s3.secret_access_key as where credentials belong. On a url.Parse failure only the inner cause is wrapped, so the raw URL is not echoed. file:// is unchanged.

Model: opus-4-8
 2026-09-22 12:28:32 +02:00
clawbot 96ebcd40d7 Reconcile purge against remote by hashed key, not human ID (closes #160)
check / check (pull_request) Successful in 1m20s
Details
check / check (push) Successful in 2m51s
Details 
syncWithRemote compared human snapshot IDs against the hashed metadata/<key>/ directory names, which never match, so it deleted every local snapshot record; the purge that followed then found nothing to remove remotely. Reconcile via listAllRemoteSnapshotKeys and RemoteSnapshotKey(id), matching CleanupLocalSnapshots, so a row still backed by remote metadata is kept.

The purge tests only passed because their stubs used the human-ID layout production never writes; they now write metadata under the hashed remote key. New tests prove remotely-backed local rows survive the reconcile and that a purge removes the local row and remote metadata together.

Model: opus-4-8
 2026-09-22 12:11:49 +02:00
clawbot d9f0220f94 Restore files at 0600 and make the blob hash check unskippable (closes #163)
check / check (pull_request) Successful in 1m21s
Details
check / check (push) Successful in 2m42s
Details 
Regular files are now created with O_EXCL at mode 0600 and given their stored mode only after the content is written and closed, so a file whose stored mode is restrictive is never briefly readable by other local users mid-restore. A file whose write or close fails is removed rather than left partial, and a chmod failure is a user-visible warning instead of a debug line.

hashVerifyReader.Close now errors when closed before EOF, so a short read or early close can never obtain a blob whose hash was not verified; downloadBlobToCache drops the cache entry on any such failure.

verifyFile (--verify) now rejects a restored file with bytes past its last chunk. Tests cover each behaviour under umask 022.

Model: opus-4-8
 2026-09-22 11:45:52 +02:00
clawbot 4c83e82543 Reject a blob_size_limit below the largest possible chunk (closes #167)
check / check (push) Successful in 1m20s
Details
check / check (pull_request) Successful in 1m16s
Details 
Validate only rejected blob_size_limit below chunk_size, but the chunker can emit chunks up to chunk_size times the FastCDC size spread (four times), and the packer puts a single chunk of any size into an otherwise empty blob. A limit between one and four times chunk_size therefore let a blob reach four times the configured maximum, with most blobs holding a single chunk and so exposing individual chunk lengths to anyone who can list the destination.

Validate now rejects blob_size_limit below chunk_size times the spread, reusing the chunker's one constant (now exported as ChunkSizeSpread) instead of a second literal. The rule is stated in the error text, the Validate comment, the README config table, config.example.yml, and the generated config template.

Model: opus-4-8
 2026-09-22 11:45:41 +02:00
clawbot 86361c8b50 Fail closed on unreadable manifests instead of losing blobs (closes #157)
check / check (pull_request) Successful in 1m20s
Details
check / check (push) Successful in 2m42s
Details 
Prune learned which blobs are in use by reading every snapshot's manifest, but merely logged and skipped one it could not download or decode. Blobs referenced only by that snapshot then looked unreferenced and were deleted, with a zero exit -- and snapshot create --prune runs this unattended. collectReferencedBlobs now errors, naming the remote key, so prune deletes nothing and exits non-zero.

Manifest generation likewise skipped a blob whose lookup failed or was missing, yielding a manifest short of what the snapshot needs; it now fails. Deep verify only warned when the manifest omitted a database blob; it now fails on any divergence. Docs corrected.

Model: opus-4-8
 2026-09-22 11:45:30 +02:00
clawbot d77663d039 Default a scheme-less s3.* endpoint to TLS (closes #158)
check / check (pull_request) Successful in 1m19s
Details
check / check (push) Successful in 3m15s
Details 
With the s3.* config form and an endpoint written without a scheme, use_ssl being omitted built an http:// endpoint, while config.example.yml documented use_ssl as defaulting to true. Over plain HTTP a network observer sees manifests, object names, sizes and the access key id, and can alter responses.

use_ssl is now *bool: omitted (nil) means the default, TLS; only an explicit use_ssl: false forces plain HTTP. This matches the s3:// URL form, which already defaults to TLS. The config init template dropped its misleading use_ssl line from the s3:// block (that key is never read for URLs; ?ssl=false controls TLS there) and points at ?ssl=false instead.

Model: opus-4-8
 2026-09-22 11:11:34 +02:00
clawbot 3abe9cbd9e Scope the PID lock to mutating commands (closes #150)
check / check (push) Successful in 1m19s
Details
check / check (pull_request) Successful in 1m16s
Details 
RunWithApp took the process-wide PID lock for every fx-backed command, so read-only commands (info, snapshot list, snapshot verify, remote info) failed with "already running" while a backup held it.

AppOptions now carries a lockMode declared at each call site: only mutating commands (snapshot create, snapshot purge, snapshot remove, prune, remote nuke) acquire the lock; read-only ones run without it. snapshot restore is classified read-only -- it writes only to its target directory, not the local index or remote store. The decision moves to a small acquireLockIfMutating helper, with a test that a read-only command runs while the lock is held and two mutators still exclude. The README locking section is rewritten to match.

Model: opus-4-8
 2026-09-22 11:01:29 +02:00
clawbot 76a6917a35 Keep restore writes inside the target directory (closes #154)
check / check (pull_request) Successful in 1m21s
Details
check / check (push) Successful in 2m51s
Details 
restoreFile and verifyRestoredFiles joined the stored path onto the target with no containment check, so a ".." segment or an absolute path escaped the target, and a restored symlink could redirect a later child write anywhere on disk. Since age decryption proves a snapshot is readable but not honest, and restore usually runs as root, a forged snapshot became an arbitrary file write.

Both call sites now go through containedRestorePath: it rejects a stored path unless filepath.IsLocal accepts it with the leading separator removed (barring "..", absolute, and empty paths), then Lstats each existing ancestor below the target and refuses to descend through a symlink. The target directory itself may be a symlink, and honest symlinks pointing outside the tree are still written verbatim.

Model: opus-4-8
 2026-09-22 11:01:01 +02:00
clawbot 38ebfd843a Trust only uploaded blobs for deduplication (closes #148)
check / check (pull_request) Successful in 1m23s
Details
check / check (push) Successful in 3m10s
Details 
An interrupted blob upload left the blob's chunks, blob_chunks, and blobs rows committed before the upload was attempted, so a later run deduplicated against data that never reached storage and produced a snapshot that reported success but could not be restored.

Fix (issue option b): a chunk counts as known only when a blob holding it has uploaded_ts set, and each run drops un-uploaded blob rows and the chunks they orphan at startup, so the affected data is re-chunked and re-uploaded. A blob recorded with no remote backend is marked uploaded so the invariant holds uniformly.

The reproduction is the interrupted-upload test from #72: its t.Skip is removed and it passes against this fix, and this branch's earlier duplicate copy is dropped. The interrupted metadata-export case is split to #177.

Model: opus-4-8
 2026-09-22 10:29:21 +02:00
190 changed files with 14869 additions and 4018 deletions
Expand all files Collapse all files
.dockerignore
+79 -10
View File
@@ -1,10 +1,79 @@
.git
.gitea
*.md
LICENSE
vaultik
dist
.tool
coverage.out
coverage.html
.DS_Store
# .dockerignore does NOT use .gitignore semantics. Docker matches with
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
# `/` and an unprefixed pattern is anchored at the context root. Every
# depth-independent pattern therefore needs `**/`, or `config/.env` and
# `certs/server.key` still ship while this file reads as solved. Only
# genuinely root-anchored entries go unprefixed. Never transplant these
# into .gitignore, where `**/` is wrong.
#
# Matching is case-sensitive, so secrets use character ranges rather
# than an ALL-CAPS twin, which would still miss `Server.Key`.
#
# Extend with this repo's own host-built artifacts, written anchored:
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context.
# .git is sent without its config. Without a VERSION build argument the
# stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
# Each submodule keeps a config with the same exposure in its git directory
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root.
# KNOWN GAP: a repo running agents in subdirectories still ships
# `services/api/.claude/` and must add its own anchored entry.
.claude
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Re-include a committed template with a negation if the
# build needs one: `!docs/example.env`.
**/*.[eE][nN][vV]
**/.[eE][nN][vV].*
**/.[eE][nN][vV][rR][cC]
# Private keys and the bundles carrying them. Public certificates
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
**/*.[pP][eE][mM]
**/*.[kK][eE][yY]
**/*.[pP]12
**/*.[pP][fF][xX]
**/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# Dependencies: restored inside the image, never copied in.
**/node_modules
# OS metadata.
**/.DS_Store
**/Thumbs.db
# Editor state: never a build input, and it churns COPY.
**/*.swp
**/*.swo
**/*~
**/*.bak
**/.idea
**/.vscode
**/*.sublime-*
# This repo's own host-built artifacts.
/vaultik
/dist
/.tool
/coverage.out
/coverage.html
.editorconfig
+3
View File
@@ -10,3 +10,6 @@ insert_final_newline = true
[Makefile]
indent_style = tab
[*.go]
indent_style = tab
.gitea/workflows/check.yml
+7 -12
View File
@@ -1,14 +1,9 @@
name: check
on:
push:
branches: [main, next]
pull_request:
branches: [main, next]
on: [push]
jobs:
check:
runs-on: ubuntu-latest
steps:
# actions/checkout v4, 2024-09-16
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Build and check
run: script/cibuild
check:
runs-on: ubuntu-latest
steps:
# actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- run: script/cibuild
.gitea/workflows/release.yml
+6 -5
View File
@@ -16,11 +16,12 @@ jobs:
fetch-depth: 0
# goreleaser is not a compiler: it shells out to `go` for the
# `before:` hook and for every one of the four cross-compiles.
# Nothing else in this repo puts a Go toolchain on the runner --
# check.yml runs script/cibuild, which does all of its work inside
# the digest-pinned Dockerfile images -- so without this step the
# release either fails at the before-hook or, worse, ships binaries
# built by whatever Go the runner happens to carry.
# Without this step the release either fails at the before-hook or,
# worse, ships binaries built by whatever Go the runner happens to
# carry. check.yml's runner gets the same Go through
# script/bootstrap, which calls script/install-go, and uses it only
# for `go mod download` and gofmt; it compiles inside the
# digest-pinned Dockerfile images.
#
# actions/setup-go would pin the action by commit sha, but the Go
# tarball it downloads at runtime is verified against no value in
.gitignore
+57 -23
View File
@@ -1,28 +1,62 @@
# Binary
/vaultik
# goreleaser output
/dist/
# Locally installed pinned tools (script/install-goreleaser)
/.tool/
# Test artifacts
*.out
*.test
coverage.html
coverage.out
# IDE
.vscode/
.idea/
*.swp
*.swo
# OS
.DS_Store
Thumbs.db
# Local config for development
# Editors
*.swp
*.swo
*~
*.bak
.idea/
.vscode/
*.sublime-*
# Agent scratch (worktrees of this repo, created and destroyed by
# in-flight tooling). Unanchored: .gitignore patterns already match at
# every depth, so no prefix is wanted here. This is not a .dockerignore
# entry and must not be given a `**/` prefix on the way into one.
.claude/
# Node
node_modules/
# Secrets. Unanchored like every entry above, so each matches at every
# depth. Matching is case-sensitive on Linux, so names use character
# ranges rather than a lowercase form that misses `Server.Key`.
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation after these lines, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# Go build and test output.
*.log
*.out
*.test
coverage.html
# This repo's own host-built artifacts.
/vaultik
/dist/
/.tool/
# Local configs for development; they hold storage credentials.
local-config.yaml
dev-config.yaml
dev-config.yaml
.golangci.yml
+71 -2
View File
@@ -10,14 +10,21 @@ run:
linters:
default: all
enable:
# Successor to the deprecated gomodguard. Named explicitly, rather than
# left to `default: all`, because it carries the module policy below.
- gomodguard_v2
disable:
# Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields
- depguard # Dependency allow/block lists
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
- godot # Requires comments to end with periods
- wsl # Deprecated, replaced by wsl_v5
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
# Deprecated: the warning is attached to the old name, so it is
# silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
settings:
lll:
line-length: 88
@@ -28,6 +35,68 @@ linters:
max-complexity: 15
dupl:
threshold: 100
depguard:
# Test-support code must not be compiled into the shipped binary. A
# test-support package exists to hand a test privileges the program
# itself must never have, so a file that is not a test must not import
# one. Test files, and the files inside a package whose directory name
# ends in `test`, are where that code belongs, and are exempt.
#
# The deny list below is the one part of this file a repository is
# expected to extend, and the only part it may. depguard matches an
# import path against a list of prefixes, so it cannot be told "any path
# whose last segment ends in test"; a repository's own test-support
# packages have to be named here one at a time, by full import path,
# under a module path that differs from repository to repository. Add
# them; change nothing else.
rules:
test-support:
list-mode: lax
files:
- "$all"
- "!$test"
- "!**/*test/**"
deny:
- pkg: net/http/httptest
desc: >-
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
- pkg: sneak.berlin/go/vaultik/internal/storage/faultstore
desc: >-
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
# Only decisions already recorded in the Go package defaults are
# listed here. Every entry matches the module path exactly.
gomodguard_v2:
blocked:
- module: github.com/rs/zerolog
recommendations:
- log/slog
reason: "Structured logging is stdlib log/slog."
# One entry per pre-fork module path, because the later releases
# are separate paths. A prefix match would be shorter but would
# also reach github.com/go-redis/redismock, the test double for
# the successor these entries recommend.
- module: github.com/go-redis/redis
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v7
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v8
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/sergi/go-diff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "No unified diff output; use go-udiff."
- module: github.com/hexops/gotextdiff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "Unmaintained fork; use go-udiff."
issues:
max-issues-per-linter: 0
.goreleaser.yaml
+1 -3
View File
@@ -47,9 +47,7 @@ checksum:
# A snapshot is not a release and must not name itself like one. The
# previous `{{ incpatch .Version }}-next` derived a plausible-looking
# release number from the last tag -- and with no tags in the repo at
# all, from goreleaser's fabricated v0.0.0. This produces the same
# string script/version produces for an untagged build, so a snapshot
# binary and a `make vaultik` binary of the same clean commit agree.
# all, from goreleaser's fabricated v0.0.0.
snapshot:
version_template: "dev-{{ slice .FullCommit 0 12 }}"
AGENTS.md
+25 -10
View File
@@ -102,14 +102,29 @@ Version: 2025-06-08
build files are acceptable in the root, but source code and other files
should be organized in appropriate subdirectories.
13. Pre-1.0: NEVER write database migrations. There are no live databases
anywhere — every user's local index can be rebuilt from a fresh full
backup. To change the schema, edit `internal/database/schema/001.sql`
(and any code that touches the affected tables) directly; do not add new
numbered schema files. Those numbered files and the `schema_migrations`
table they populate only bootstrap a fresh database — they are not an
upgrade path. The local index is disposable until 1.0 ships and is
tagged; once 1.0 is tagged that clause expires and the question of
upgrading existing indexes returns. See [`docs/DATAMODEL.md`](docs/DATAMODEL.md)
for the full explanation.
13. Pre-1.0: NEVER add a database migration. Migrations are supported, but
nothing is installed anywhere yet, so there is nothing to migrate. To
change the schema, edit `internal/database/schema/001.sql` (and any
code that touches the affected tables) directly. After 1.0, each schema
change is a new numbered file in that directory and a released file is
never edited; an existing local database is then migrated when vaultik
is updated. Before 1.0, a local database left on an older schema is
deleted and re-created by a full backup. See
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations).
14. Never use `git add -A`. Stage only the files you intentionally
changed.
15. Commit messages carry no attribution or advertising trailers for the
tool that helped write the code, or for its vendor: the owner is the
sole author of code written with a tool.
16. Run the whole test suite with `make test` every time, and read its full
output. Never run `go test`, a single test or a single package, and
never grep the output.
17. Do not stop working on a task until the definition of done given in the
initial instruction is met: all of the work, not part or most of it.
18. For estimates: backing up over 2.5Gbit/s ethernet to an S3 server
backed by 2000MB/sec SSD takes about 4 seconds per gigabyte.
ARCHITECTURE.md
+26 -19
View File
@@ -54,10 +54,12 @@ The database tracks five primary entities and their relationships:
#### File (`database.File`)
Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration:
- Path, source_path (for restore path stripping), mtime
- Path, mtime
- Size, mode, ownership (uid, gid)
- Symlink target (if applicable)
It also stores `source_path`, the source directory the scan found it under, made absolute and with symlinks resolved. Restore does not read it.
#### Chunk (`database.Chunk`)
A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm:
- `ChunkHash`: SHA256 hash of chunk content (primary key)
@@ -74,17 +76,19 @@ Maps files to their constituent chunks:
#### Blob (`database.Blob`)
The final storage unit uploaded to S3. Contains many compressed and encrypted chunks:
- `ID`: UUID assigned at creation
- `Hash`: SHA256 of final compressed+encrypted content
- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (see [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs-directory-blobs))
- `UncompressedSize`: Total raw chunk data before compression
- `CompressedSize`: Size after zstd compression and age encryption
- `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps
Blob creation process:
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
2. Compressed with zstd
3. Encrypted with age (recipients configured in config)
4. SHA256 hash computed → becomes filename in S3
5. Uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}`
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted
A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp`.
#### BlobChunk (`database.BlobChunk`)
Maps chunks to their position within blobs:
@@ -284,9 +288,10 @@ Manages snapshot lifecycle and metadata export.
Key methods:
- `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record
- `CompleteSnapshot(ctx, snapshotID)` → Mark snapshot complete
- `PopulateSnapshotBlobs(ctx, snapshotID)` → Record every blob the snapshot references
- `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3
- `CleanupIncompleteSnapshots(ctx, hostname)` → Remove failed snapshots
- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a successful export
- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark complete (no export between)
### `internal/database`
SQLite database for local index. Single-writer mode for thread safety.
@@ -307,7 +312,7 @@ Repository interfaces:
```
CreateSnapshot(opts)
│
├─► CleanupIncompleteSnapshots() // Critical: avoid dedup errors
├─► PruneDatabase() // Critical: avoid dedup errors
│
├─► SnapshotManager.CreateSnapshot() // Create DB record
│
@@ -334,23 +339,25 @@ CreateSnapshot(opts)
│ │
│ └─► Accumulate statistics
│
├─► SnapshotManager.PopulateSnapshotBlobs() // record referenced blobs
│
├─► SnapshotManager.UpdateSnapshotStatsExtended()
│
├─► SnapshotManager.CompleteSnapshot()
├─► SnapshotManager.ExportSnapshotMetadata()
│ │
│ ├─► Copy database to temp file
│ ├─► Clean to only current snapshot data (VACUUM)
│ ├─► Compress binary SQLite with zstd
│ ├─► Encrypt with age
│ ├─► Upload db.zst.age to storage
│ └─► Upload manifest.json.zst to storage
│
└─► SnapshotManager.ExportSnapshotMetadata()
│
├─► Copy database to temp file
├─► Clean to only current snapshot data (VACUUM)
├─► Compress binary SQLite with zstd
├─► Encrypt with age
├─► Upload db.zst.age to storage
└─► Upload manifest.json.zst to storage
└─► SnapshotManager.MarkSnapshotComplete() // only after the export succeeds
```
## Deduplication Strategy
1. **File-level**: Files unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid)
1. **File-level**: Files unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded.
CLAUDE.md
-44
View File
@@ -1,44 +0,0 @@
# Rules
Read the rules in AGENTS.md and follow them.
# Memory
* Claude is an inanimate tool. The spam that Claude attempts to insert into
commit messages (which it erroneously refers to as "attribution") is not
attribution, as I am the sole author of code created using Claude. It is
corporate advertising for Anthropic and is therefore completely
unacceptable in commit messages.
* NEVER use `git add -A`. Always add only the files you intentionally
changed.
* Tests should always be run before committing code. No commits should be
made that do not pass tests.
* Code should always be formatted before committing. Do not commit
unformatted code.
* Code should always be linted before committing. Do not commit
unlinted code.
* The test suite is fast and local. When running tests, don't run
individual parts of the test suite, always run the whole thing by running
"make test".
* Do not stop working on a task until you have reached the definition of
done provided to you in the initial instruction. Don't do part or most of
the work, do all of the work until the criteria for done are met.
* We do not need to support migrations; schema upgrades can be handled by
deleting the local state file and doing a full backup to re-create it.
* When testing on a 2.5Gbit/s ethernet to an s3 server backed by 2000MB/sec SSD,
estimate about 4 seconds per gigabyte of backup time.
* When running tests, don't run individual tests, or grep the output. run
the entire test suite every time and read the full output.
* When running tests, don't run individual tests, or try to grep the output.
never run "go test". only ever run "make test" to run the full test
suite, and examine the full output.
Dockerfile
+66 -86
View File
@@ -1,96 +1,76 @@
# This file has no lint stage, deliberately.
#
# Linting lives in Dockerfile.lint, built by script/lint, and
# script/cibuild builds both. A lint stage here would have to either
# shell out to `make lint` -- which is now `docker build`, so
# docker-in-docker inside a BuildKit step with no daemon -- or call
# golangci-lint directly, which would mean a second, independently
# bumpable digest pin for the linter alongside the one in
# Dockerfile.lint. Two pins for one tool is the drift that
# https://git.eeqj.de/sneak/vaultik/issues/78 was filed over. See
# https://git.eeqj.de/sneak/vaultik/issues/113 for the ruling.
#
# Consequence, stated rather than left to be discovered: script/docker
# builds this file only and therefore does not lint. `make fmt-check`
# and `make test` still run here, so what a green build of this file
# means is "formatted, tested, and it compiles" -- the lint verdict
# comes from script/lint or script/cibuild.
# Build stage
# golang:1.26.1-alpine, 2026-03-17
FROM golang:1.26.1-alpine@sha256:2389ebfa5b7f43eeafbd6be0c3700cc46690ef842ad962f6c5bd6be49ed82039 AS builder
# Build tooling: make, plus a C toolchain because `go test -race` needs cgo.
# The sqlite driver is pure Go (modernc.org/sqlite), so no sqlite library or
# CLI is required.
RUN apk add --no-cache make build-base
# Lint phase. The linter is invoked directly rather than through `make
# lint` or `script/lint`, which are themselves a docker build and would
# recurse into a daemon that does not exist in a build step.
# golangci/golangci-lint:v2.14.0, 2026-10-05
FROM golangci/golangci-lint:v2.14.0@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
WORKDIR /src
# Copy go mod files first for better layer caching
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# `golangci-lint run` silently ignores an unknown top-level key in
# .golangci.yml, such as a misspelt `linters:`; `config verify` fails on it.
RUN golangci-lint config verify --config .golangci.yml
RUN golangci-lint run --config .golangci.yml ./...
# Copy source code
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not.
# golang:1.26.1 (Debian trixie), 2026-10-05
FROM golang:1.26.1@sha256:cd78d88e00afadbedd272f977d375a6247455f3a4b1178f8ae8bbcb201743a8a AS test
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# golang:1.26.1-alpine, 2026-03-17
FROM golang:1.26.1-alpine@sha256:2389ebfa5b7f43eeafbd6be0c3700cc46690ef842ad962f6c5bd6be49ed82039 AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# Run the format check and the tests.
#
# CHECK_EPOCH must stay immediately above these RUNs. These layers are
# keyed on its value, so they are cache-eligible only for a value
# already built against this same tree. script/cibuild and script/docker
# each pass a fresh value on every invocation, which is what makes their
# green mean the checks really executed.
#
# The value is expanded into each check command rather than left to a
# bare declaration, so the cache miss does not depend on BuildKit's
# unreferenced-ARG handling staying as it is. It also puts the epoch in
# the build log, where a reader can see the layer was keyed fresh.
#
# The guard is what makes a build that omits --build-arg fail instead of
# lie. An unset ARG is an empty string, and an empty string is a
# perfectly stable cache key: without the guard the first such build
# runs the checks and every one after it on an unchanged tree replays
# these layers from cache, executes nothing, and still exits 0. Failed
# steps are never cached, so the guard fails on EVERY invocation rather
# than once -- a bare `docker build .` is a loud error, not a quiet
# green. Do not give CHECK_EPOCH a default value; a default would
# satisfy the guard with a constant and restore the hole.
#
# Everything above this line (apk, go.mod, `go mod download`) is
# deliberately outside the busted range and keeps caching.
ARG CHECK_EPOCH
RUN [ -n "$CHECK_EPOCH" ] || exit 1
RUN echo "check epoch: ${CHECK_EPOCH}" && make fmt-check
RUN echo "check epoch: ${CHECK_EPOCH}" && make test
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. The
# commit and its date always come from that .git. With .git present, a
# version that is still empty, dev or unknown, or a commit or date that
# is unknown, fails the build: git is missing or could not read the
# checkout, as when .git is a file pointing outside the context. A
# context without .git, such as a source export, stamps "dev" and an
# "unknown" commit and date.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always || echo dev)}"; \
commit="$(git rev-parse HEAD || echo unknown)"; \
commit_date="$(git show -s --format=%cs HEAD || echo unknown)"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
if [ "$commit" = unknown ] || [ "$commit_date" = unknown ]; then \
echo "commit is '$commit' and its date '$commit_date'" \
"although .git is present" >&2; \
exit 1; \
fi; \
fi; \
globals=sneak.berlin/go/vaultik/internal/globals; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X ${globals}.Version=${VERSION} \
-X ${globals}.Commit=${commit} \
-X ${globals}.CommitDate=${commit_date}" \
-o /vaultik ./cmd/vaultik
# Version, commit and build date are computed on the host by
# script/docker and script/cibuild (where .git exists) and passed in as
# build args. The build context excludes .git (see .dockerignore), so
# the build cannot derive them itself: it used to try, with `git
# rev-parse` inside this stage, and always got "unknown". VERSION comes
# from script/version, the source of truth shared with the Makefile, so
# it carries the same tag / dev-<sha> / -dirty rules and a Docker image
# reports the same string a local build of the same tree would.
#
# The defaults are the fallback for a bare `docker build .` that passes
# none of them: an unset arg would otherwise stamp an empty string and
# produce an image that cannot report its own version, commit or date.
# They match what an out-of-git build reports elsewhere.
#
# These ARGs sit here, after the checks, rather than at the top of the
# stage: every commit changes their values, and a value change
# invalidates all layers below the ARG. Declared up top they would bust
# `go mod download`; here they only rekey this build layer, which the
# COPY of the sources above already rebuilds on any change anyway.
ARG VERSION=dev
ARG COMMIT=unknown
ARG COMMIT_DATE=unknown
# Build (pure Go, no CGO required since we use modernc.org/sqlite)
RUN CGO_ENABLED=0 go build -ldflags "-X 'sneak.berlin/go/vaultik/internal/globals.Version=${VERSION}' -X 'sneak.berlin/go/vaultik/internal/globals.Commit=${COMMIT}' -X 'sneak.berlin/go/vaultik/internal/globals.CommitDate=${COMMIT_DATE}'" -o /vaultik ./cmd/vaultik
# Runtime stage
# Runtime stage, and the last one: a plain `docker build .` builds this
# stage's chain and nothing else.
# alpine:3.21, 2026-02-25
FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709
Dockerfile.lint
-104
View File
@@ -1,104 +0,0 @@
# Lint image.
#
# Every lint run in this repo happens inside this image, invoked through
# script/lint, and linting is a BUILD STEP rather than a container
# command: a successful build of this file IS a clean lint. That shape
# also works where the docker daemon is remote and bind mounts are
# impossible, which `docker run` against a mounted worktree does not.
#
# This FROM line is the single source of truth for the linter version in
# this repo. Nothing else pins golangci-lint: the product Dockerfile has
# no lint stage, deliberately, so there is no second digest to bump and
# no pair of pins that can drift apart. Bump the tag AND the digest here
# and nowhere else.
#
# Note for readers coming from REPO_POLICIES.md: that document still
# describes the older pattern, a lint stage inside the product
# Dockerfile wired up with `COPY --from=lint /src/go.sum /dev/null`.
# That pattern is superseded here by the owner's ruling recorded in
# https://git.eeqj.de/sneak/vaultik/issues/113 -- lint runs in its own
# image, per run, with its own cache and its own lock, which is what
# makes concurrent runs on one host safe. The policy text is org-wide
# and is being amended separately; this file is what this repo does.
#
# golangci/golangci-lint:v2.12.2, 2026-08-10
FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240
WORKDIR /src
# Copy the dependency manifests first so the module download layer stays
# cached until they change. Everything above the ARG below is cacheable
# on purpose; a cold module download on every lint would make the inner
# loop unusable and buys nothing, because it is not what the gate is
# asserting.
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# Force the check layers to execute on every invocation.
#
# CHECK_EPOCH must stay immediately above the RUNs below. Those layers
# are keyed on its value, so they are cache-eligible only for a value
# already built against this same tree; script/lint and script/cibuild
# each pass a fresh value on every invocation, which is what makes their
# green mean the linter really ran. Without it, `docker build -f
# Dockerfile.lint .` on an unchanged tree exits 0 in well under a second
# having linted nothing.
#
# The value is expanded into each check command itself rather than left
# to a bare declaration, so the cache miss does not depend on BuildKit's
# unreferenced-ARG handling staying as it is. It also puts the epoch in
# the build log, where a reader can see the layer was keyed fresh.
#
# The guard is what makes a build that omits --build-arg fail instead of
# lie. An unset ARG is an empty string, and an empty string is a
# perfectly stable cache key: without the guard the first such build
# lints and every one after it on an unchanged tree replays this layer,
# executes nothing, and still exits 0. Failed steps are never cached, so
# the guard fails on EVERY invocation rather than once. Do not give
# CHECK_EPOCH a default value; a default would satisfy the guard with a
# constant and restore the hole.
ARG CHECK_EPOCH
RUN [ -n "$CHECK_EPOCH" ] || exit 1
# Validate .golangci.yml before linting with it.
#
# This is not belt-and-braces; it closes a hole that `golangci-lint run`
# leaves wide open. `run` rejects YAML it cannot PARSE, but it silently
# IGNORES an unknown top-level KEY. Renaming `linters:` to `linterz:` --
# one character -- discards `default: all`, the whole disable list and
# every threshold, leaves only golangci-lint's small default linter set
# running, and exits 0 reporting `0 issues.` on a tree the real config
# fails. Demonstrated on this repo at this pin, recorded on
# https://git.eeqj.de/sneak/vaultik/pulls/114: with a planted
# over-length line, `script/lint` exits 1 naming the `revive` finding
# with `linters:` and exits 0 with `linterz:`. A set-but-ineffective
# config quietly falling back to defaults is precisely the false-green
# class this gate exists to eliminate, so it must not sit in the gate's
# own configuration.
#
# `config verify` catches it, and it does so OFFLINE at this pinned
# version -- verified, not assumed. Under `docker run --network none`
# against the pinned digest it exits 0 on this repo's config and exits 3
# on the `linterz:` variant with `additional properties 'linterz' not
# allowed`. An earlier revision of this file asserted the opposite, that
# the schema is fetched over live HTTPS from an unpinned URL, and used
# that to justify omitting this line. That claim was false at v2.12.2;
# the schema is embedded. If a future bump reintroduces a network fetch
# the failure is loud and this comment is where to record it.
#
# It is keyed on CHECK_EPOCH, like the lint run below, so it executes on
# every invocation. Content-addressing alone would arguably be enough --
# .golangci.yml arrives through `COPY . .`, so a cache hit here implies
# a byte-identical config was validated when the layer really ran. That
# argument is exactly the one that would also excuse caching the lint
# layer, and this repo has ruled it insufficient: a cached check layer
# checks nothing, and the cost of being wrong is silent. Forcing it costs
# milliseconds and puts the epoch in the log, where a reader can see that
# this validation ran rather than being replayed.
RUN echo "check epoch: ${CHECK_EPOCH}" && \
golangci-lint config verify --config .golangci.yml
RUN echo "check epoch: ${CHECK_EPOCH}" && \
golangci-lint run --config .golangci.yml ./...
Makefile
+17 -13
View File
@@ -1,7 +1,10 @@
.PHONY: all bootstrap setup check test lint lint-fix fmt fmt-check build clean deps test-coverage local install release release-snapshot docker hooks
# Version number, derived from git by script/version -- the tag when
# HEAD is on one, otherwise dev-<sha>. This used to be a hardcoded
# Where script/bootstrap installs Go when the host has none.
export PATH := $(PATH):$(CURDIR)/.tool/go/bin
# Version number, derived from git by script/version (`git describe
# --tags --always --dirty`). This used to be a hardcoded
# constant, which meant every local build claimed to be a release that
# had never been tagged.
VERSION := $(shell script/version)
@@ -40,9 +43,10 @@ setup:
check:
@script/check
# Run tests only. This runs the ENTIRE suite -- there is no separate
# integration target and no build-tagged subset held back. In
# particular internal/vaultik/integration_test.go, which does full
# Run tests only, by building the test phase of the Dockerfile. This
# runs the ENTIRE suite -- there is no separate integration target and
# no build-tagged subset held back. In particular
# internal/vaultik/integration_test.go, which does full
# chunk -> pack -> encrypt -> upload -> restore round-trips, runs here.
# A `test-integration` target used to exist and was removed: no file in
# the repo carried a build tag, so `-tags=integration` selected nothing
@@ -87,17 +91,17 @@ clean:
go clean
# Install dependencies. The linter is deliberately not installed here:
# script/lint lints by building Dockerfile.lint, whose FROM line is the
# single source of truth for the linter version. A second, separately
# pinned copy on PATH could drift from it and make a local `make lint`
# disagree with CI.
# script/lint lints by building the lint phase of the Dockerfile, whose
# FROM line is the single source of truth for the linter version. A
# second, separately pinned copy on PATH could drift from it and make a
# local `make lint` disagree with CI.
deps:
go mod download
# Run tests with coverage. -count=1 for the same reason script/test
# uses it: without it an unchanged package is served from Go's test
# result cache, and a coverage profile assembled from cached results
# describes a run that did not happen.
# Run tests with coverage, on the host. -count=1 because without it an
# unchanged package is served from Go's test result cache, and a
# coverage profile assembled from cached results describes a run that
# did not happen.
test-coverage:
go test -v -count=1 -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
README.md
+229 -204
View File
@@ -38,9 +38,10 @@ vaultik snapshot list
Features:
* modern encryption ([age](https://age-encryption.org/), X25519 + XChaCha20-Poly1305)
* modern encryption ([age](https://age-encryption.org/), X25519 + ChaCha20-Poly1305)
* content-defined chunking with deduplication (FastCDC)
* incremental backups (only changed files are re-chunked)
* incremental backups (a file is re-chunked only when it changed or a
chunk it lists is held by no uploaded blob)
* multithreaded zstd compression at configurable levels
* content-addressed immutable storage
* local state tracking in SQLite (enables write-only incremental backups)
@@ -71,14 +72,21 @@ Requirements that no existing tool meets:
## daily use
```sh
# verify a snapshot (shallow: checks all blobs exist)
# verify a snapshot (shallow: checks all blobs are present with the listed size)
vaultik snapshot verify <snapshot-id>
# deep verify (downloads and cryptographically verifies every blob)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot verify --deep <snapshot-id>
# put the private key file in the environment (reading it from the file
# keeps the key out of your shell history); the whole age-keygen file,
# with one or more identities, is accepted
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
# restore (requires the private key)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot restore <snapshot-id> /tmp/restored
# deep verify (downloads every blob, decrypts it, and re-hashes it to
# detect corruption — this checks integrity, not who wrote the blob)
vaultik snapshot verify --deep <snapshot-id>
# restore (requires the private key). Restore into a new directory you own,
# writable only by you — not a shared location like /tmp
vaultik snapshot restore <snapshot-id> ~/vaultik-restore
# daily cron job: back up, keep a 4-week rolling window of snapshots
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
@@ -119,15 +127,18 @@ Use that remote key — the hex printed inside `<remote only:...>`, or the
full `remote_key` from `snapshot list --json` — to restore and verify:
```sh
# restore everything to /tmp/restored, then check every restored file's
# chunk hashes
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' \
vaultik snapshot restore --verify <remote-key> /tmp/restored
# put the private key file in the environment (reading it from the file
# keeps the key out of your shell history)
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
# optionally, deep-verify the snapshot against the store (downloads and
# cryptographically checks every blob)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' \
vaultik snapshot verify --deep <remote-key>
# restore everything to a new directory you own (writable only by you, not a
# shared location like /tmp), then check every restored file's chunk hashes
vaultik snapshot restore --verify <remote-key> ~/vaultik-restore
# optionally, deep-verify the snapshot against the store (downloads every
# blob, decrypts it, and re-hashes it to detect corruption — this checks
# integrity, not who wrote the blob)
vaultik snapshot verify --deep <remote-key>
```
`age_recipients` (the public key) is not needed to restore — only the
@@ -167,28 +178,32 @@ vaultik version
* `--verbose`, `-v`: Enable verbose output (on stderr — see below)
* `--debug`: Enable debug output (on stderr — see below)
* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner)
* `--skip-errors`: Continue past per-file errors instead of aborting (applies to `snapshot create` and `restore`)
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run.
### locking
Every command that opens the local index — `snapshot create`, `snapshot
list`, `snapshot verify`, `snapshot purge`, `snapshot remove`, `snapshot
restore`, `prune`, `info`, and `remote info`/`remote nuke` — takes a
Commands that write persistent state — `snapshot create`, `snapshot
purge`, `snapshot remove`, `prune`, and `remote nuke` — take a
process-wide lock at `$XDG_DATA_HOME/vaultik/vaultik.pid`
(`~/.local/share/vaultik/vaultik.pid` on Linux) for the whole run. Only
one such command runs at a time: a second one exits immediately with an
"already running" error rather than waiting. The lock is not scoped to
mutating commands, so read-only commands are affected too — `vaultik
snapshot list` fails while a backup is in progress; scoping it so
read-only commands run during a backup is tracked in
[issue #150](https://git.eeqj.de/sneak/vaultik/issues/150). `config`,
`database delete`, `completion`, and `version` do not take the lock.
one of them runs at a time: a second one exits immediately with an
"already running" error rather than waiting, so two writers can never
corrupt the local index or the destination store.
Read-only commands — `info`, `snapshot list`, `snapshot verify`, and
`remote info` — do not take the lock and are never blocked, so they run
even while a backup is in progress. `snapshot restore` does not take the
lock either: it writes only to the target directory you name, not the
local index or the destination store. `config`, `database delete`,
`completion`, and `version` do not take the lock.
### stdout and stderr
Log output — everything from `--verbose` and `--debug`, and every
warning and error the logger emits — goes to **stderr**. stdout carries
the output you asked for: tables, and the documents produced by `--json`.
warning and error the logger emits — goes to **stderr**, and so does the
startup banner. stdout carries the output you asked for: tables, the
documents produced by `--json`, `config get` values, and completion
scripts.
This means `vaultik snapshot list --verbose > out.txt` captures the
listing and leaves the diagnostics on your terminal. To capture both,
@@ -215,7 +230,7 @@ and `vaultik prune --json | jq .` both work as written.
### environment variables
* `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for `snapshot restore` and `snapshot verify --deep`)
* `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for `snapshot restore` and `snapshot verify --deep`). May hold the whole `age-keygen` file — comments and every identity in it are accepted. Set it from the file, e.g. `export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"`, so the key is not typed into your shell history.
* `VAULTIK_CONFIG`: Path to config file (overridden by `--config`)
* `VAULTIK_INDEX_PATH`: Override local SQLite index path
* `VAULTIK_CPUPROFILE`: Write a CPU profile to this path for the duration of the run (development/debugging)
@@ -310,7 +325,9 @@ local index alone, and still exits zero.
logger, so stdout stays a single parseable document.
**`snapshot verify`**: Verify snapshot integrity.
* Default (shallow): checks that all blobs referenced in the manifest exist in storage
* Default (shallow): checks that every blob the manifest lists is present in
storage with the size the manifest records, and that the encrypted database is
present. It does not read blob contents.
* `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the
encrypted metadata database
* Accepts the same identifiers as `snapshot restore`: a snapshot ID, or a
@@ -335,13 +352,15 @@ may hold snapshots this host doesn't know about), which is what
prune` invocation to run as a follow-up. Local row cleanup (files,
chunks, blobs the snapshot was the last referrer for) runs
automatically. If the destination store is unreachable, the local-DB
removal still completes and a warning is emitted; rerun `vaultik prune`
once the store is reachable to finish remote cleanup. To wipe everything
removal still completes and a warning is emitted; run `vaultik snapshot
remove <snapshot-id>` again once the store is reachable to remove the
snapshot's metadata from it (`vaultik prune` does not). To wipe everything
on the destination in one go, use `vaultik remote nuke --force`.
* `--local-only`: Skip remote cleanup; only touch the local index
* `--dry-run`: Show what would be deleted without deleting
* `--force`: Skip confirmation prompt
* `--json`: Output result as JSON
* `--json`: Output result as JSON. Also skips the confirmation prompt, as
`--force` does.
**`snapshot restore`**: Restore files from a backup snapshot.
* Requires `VAULTIK_AGE_SECRET_KEY` environment variable
@@ -365,14 +384,22 @@ manifests — network cost scales with the number of snapshots. `snapshot
create --prune` runs the same cleanup automatically; this is the
manual entry point for the same work.
* `--force`: Skip confirmation prompt
* `--json`: Output stats as JSON
* `--json`: Output stats as JSON. Also skips the confirmation prompt, as
`--force` does.
**`info`**: Display system configuration, storage settings, encryption
recipients, and local database statistics.
**`remote info`**: Show storage backend type and location plus detailed
remote storage inventory: per-snapshot metadata sizes, blob counts, and
orphaned blob detection.
orphaned blob detection. A name under `metadata/` that is not a remote
key is skipped with a warning and is not printed. If a listed
`manifest.json.zst` cannot be read, or sits under a skipped name, the
orphaned blob figures are reported as unknown; `--json` gives them as
`null`, lists the remote key of each unreadable manifest in
`unreadable_manifests` and counts the manifests under skipped names in
`skipped_manifest_count`. An unreadable manifest also leaves its
snapshot's blob count and blob size unknown, `null` in `--json`.
* `--json`: Output as JSON
**`remote nuke`**: Delete every snapshot's metadata and every blob from the
@@ -404,6 +431,16 @@ a local or mounted filesystem. Useful for testing or backing up to a NAS.
**Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud
providers. Requires rclone to be configured separately (`rclone config`).
An upload cut off part-way leaves nothing under the object's name on S3, which
shows an object only once its upload has completed, and on the local filesystem
backend, which writes a temporary file and renames it into place. Rclone
remotes with a server-side move (local and sftp among them) are written under a
temporary name ending in `.partial` and moved into place. Rclone remotes without
one are written in place: where such a remote shows a file while it is still
being written, a killed upload can leave a truncated object under its name,
which a later backup takes for complete. A leftover `.partial` file is ignored
and can be deleted.
Legacy S3 configuration via `s3.*` fields (endpoint, bucket, prefix, etc.) is
still supported for backward compatibility. `storage_url` takes precedence if
both are set.
@@ -438,14 +475,19 @@ Snapshot IDs follow the human-readable format
`<hostname>_<snapshot-name>_<RFC3339-timestamp>` (e.g.
`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the
destination store in plaintext. Each snapshot's metadata directory is named
with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a listing
of the store reveals no hostname or snapshot name. The backup time is not
hidden: manifest.json.zst carries a plaintext timestamp, and object
with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a plain
listing of the store shows no hostname or snapshot name. The hash uses no
secret, though, so an observer who guesses a candidate hostname and snapshot
name can hash it and confirm the snapshot is present; the remote key keeps
names out of a listing but does not hide them from a guess. The backup time is
not hidden either: manifest.json.zst carries a plaintext timestamp, and object
modification times are visible at the storage layer regardless. For example,
`server1_home_2025-06-01T12:00:00Z` is stored under
`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`.
See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation.
derivation, and [Security Considerations](docs/REPOSTRUCTURE.md#security-considerations)
(including [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks)) for what the
format does and does not protect.
### data flow
@@ -480,13 +522,14 @@ derivation.
* Content-defined chunking using the FastCDC algorithm
* Average chunk size: configurable (default 10MB)
* Deduplication at file level (unchanged files skipped) and chunk level
(identical chunks across files stored once)
* Deduplication at file level (unchanged files skipped, unless a chunk
the file lists is held by no uploaded blob) and chunk level (identical
chunks across files stored once)
* Multiple chunks packed into blobs to reduce object count
### encryption
* Asymmetric encryption using age (X25519 + XChaCha20-Poly1305)
* Asymmetric encryption using age (X25519 + ChaCha20-Poly1305)
* Only the public key is needed on the source host
* Each blob and each metadata database is encrypted independently
* Multiple recipients supported (encrypt to multiple keys)
@@ -509,14 +552,14 @@ complete annotated example also lives in
| Field | Default | Description |
|-------|---------|-------------|
| `age_recipients` | (required) | Age public keys for encryption |
| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty |
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
| `snapshots` | (required) | Named snapshot definitions with paths and excludes |
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
| `exclude` | | Global exclude patterns (applied to all snapshots) |
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
| `compression_level` | `3` | zstd compression level (1-19) |
| `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path |
@@ -540,13 +583,13 @@ complete annotated example also lives in
sequentially. Restore speed is bound by single-stream throughput.
* **Device nodes, named pipes, and sockets are silently skipped.** Only
regular files, directories, and symlinks are backed up.
* **No upgrade path between versions.** There is no supported way to carry
an existing local index across a schema change; if the local SQLite
schema changes between versions, delete the local database (`vaultik
database delete`) and run a full backup. Remote storage is unaffected.
(The binary does embed numbered schema files and a `schema_migrations`
table to bootstrap a fresh database — see [`docs/DATAMODEL.md`](docs/DATAMODEL.md)
— but that is not an upgrade path.)
* **Before 1.0, an update can make the local index unusable.** Vaultik
supports schema migrations, but none are added before 1.0 because
there is no installed base yet. If an update leaves your local index
unusable, run `vaultik database delete` and then a full backup; remote
storage is unaffected. After 1.0, the local index is migrated when
vaultik is updated. See
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations).
* **Files that change during backup may be inconsistent.** There is no
filesystem snapshot or freeze. If a file is modified between the scan
and chunk phases, the backed-up copy may reflect a partial write.
@@ -558,22 +601,18 @@ complete annotated example also lives in
## roadmap
Items still to do before / shortly after 1.0. Loosely ordered by
priority.
Work planned after 1.0. Loosely ordered by priority.
### correctness and operability
* **Security audit of the encryption implementation.** Pre-1.0
blocker if we're advertising "secure" at the top of this README.
age + zstd + content-defined chunking is mostly off-the-shelf
pieces, but the seams (key handling, recipient parsing, manifest
trust boundary, restore-time identity validation) need an outside
read.
* **Error-condition tests.** Today's coverage is the happy path
plus a few specific regressions. Need fault-injection coverage:
network failures mid-blob, disk-full during restore, corrupted /
truncated / missing blobs, partial uploads, kill -9 between
manifest and db.zst.age writes.
* **Outside security audit.** Before 1.0 the encryption and
blob-generation code was reviewed: every bug the review found was
fixed, and the risks it accepted are listed in
[Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside
audit has been done. age + zstd + content-defined chunking
is mostly off-the-shelf pieces, but the seams (key handling,
recipient parsing, manifest trust boundary, restore-time identity
validation) need an outside read.
* **Verify restored content end-to-end in CI.** The current
integration test does this for a small synthetic snapshot but
not at scale. A nightly job against a multi-GB representative
@@ -596,13 +635,15 @@ priority.
doesn't resume from where it stopped or skip already-present
files. A `--resume` mode that checks targets before fetching
blobs would matter for very large restores.
* **Daemon mode.** A long-running mode that watches for file
changes so frequent backups, such as hourly, skip the full scan.
It adds little for the usual runs from cron every 12 to 36 hours.
See [issue #204](https://git.eeqj.de/sneak/vaultik/issues/204).
### usability
* **Man pages and richer `--help` examples.** Cobra generates
basic help; man pages would be a separate target.
* **`--bwlimit` style human-readable size flags** across the
command surface where they're currently raw integers.
* **`vaultik snapshot diff <a> <b>`** — show which files changed
between two snapshots without restoring either.
* **Status reporting hook for `--cron`.** When a backup fails
@@ -612,12 +653,11 @@ priority.
### infrastructure
* **Cross-version schema upgrades.** There is no upgrade path between
released versions — pre-1.0 schema changes are handled by `vaultik
database delete` plus a full re-scan (see
[`docs/DATAMODEL.md`](docs/DATAMODEL.md)). Post-1.0 we'll need a
migration story to keep existing index databases usable across
upgrades.
* **Schema migrations after 1.0.** Migrations are supported, but none
are added before 1.0 because there is no installed base yet. After
1.0, each schema change is a new migration, so an existing local
index is migrated when vaultik is updated (see
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations)).
* **Storage backend coverage tests.** S3, file://, and rclone://
all share the Storer interface but the rclone path is the least
exercised in CI.
@@ -626,27 +666,38 @@ priority.
## output style
The operational narration of the long-running commands — the Begin,
Complete, Progress, and status lines of `snapshot create`, `prune`,
`snapshot restore`, and the like — goes through helpers in `internal/ui`
and conforms to the uniform style below. Some commands instead write
plain text straight to stdout (`version`, `info`, `config`, the
`database delete` prompt, and the `snapshot list` table); that output is
unstyled and does not honor `--quiet`. Routing it through `internal/ui`
is tracked in
[issue #149](https://git.eeqj.de/sneak/vaultik/issues/149). Color is
enabled when stdout is a TTY and the `NO_COLOR` environment variable is
unset (https://no-color.org/).
Every command's user-facing output is governed by `internal/ui`, in one
of two ways. Color is enabled when the stream written to is a TTY and
the `NO_COLOR` environment variable is unset (https://no-color.org/).
`internal/ui` writes to stdout; it is the output the user asked for.
Structured log records are a different thing and go through
`internal/log`, which writes to stderr (see "stdout and stderr" above).
* **Status, progress, warnings, and errors** go through the `internal/ui`
message methods below: marker-prefixed, colored on a TTY, and — except
warnings and errors — silenced by `--quiet`. This is the operational
narration of the long-running commands (`snapshot create`, `prune`,
`snapshot restore`, and the like) and the confirmations of
`config init`, `config set`, and `database delete`.
* **The data a command exists to produce** is written plain, with no
marker and no color, because a marker would corrupt a table or a
parsed document. This covers the `version`, `info`, `remote info` and
`snapshot verify` reports, the `snapshot list` table, `config get`
values, and every `--json` document. `--quiet` silences the human
reports and tables (`version`, `info`, `remote info`, `snapshot
verify`, `snapshot list`) but never the machine-consumed `config get`
value or the `--json` documents, which a script depends on. The
`database delete` confirmation prompt is also written this way and
always shown: it is an interactive exchange the operator must see.
`internal/ui` writes to stdout; it is the output the user asked for. The
exceptions are the startup banner and the error a failed command ends
with, which go to stderr. Structured log records are a different thing
and go through `internal/log`, which writes to stderr (see "stdout and
stderr" above).
Message classes:
| Class | Marker | Alignment | Use for |
|-------|--------|-----------|---------|
| Banner | none | column 0 | The startup line printed once per invocation |
| Banner | none | column 0 | The startup line printed once per invocation, on stderr |
| Begin | `》` (white) | column 0 | An operation is about to start (present-continuous verb) |
| Complete | `》` (green) | column 0 | An operation just finished (past-tense verb) |
| Info | `》` (white) | column 0 | Neutral status update |
@@ -701,12 +752,11 @@ regardless of color setting (emoji are not color).
## requirements
* Go 1.26 or later
* Docker, with a reachable daemon, to lint, check, or commit:
`script/lint` lints by building `Dockerfile.lint`, which runs the
digest-pinned `golangci-lint` image as a build step, and `make check`
and the pre-commit hook both run it. A `golangci-lint` installed on
`PATH` is not a substitute and is never used on a host, whatever its
version.
* Docker, with a reachable daemon, to test, lint, check, or commit:
`script/test` and `script/lint` build the `test` and `lint` phases of
the `Dockerfile`, and `make check` and the pre-commit hook both run
them. A `golangci-lint` installed on `PATH` is not a substitute and is
never used on a host, whatever its version.
* S3-compatible object storage (or local filesystem, or rclone remote)
## development workflow
@@ -737,16 +787,20 @@ standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call
them. We provide:
* `script/bootstrap` — install all development dependencies (go, Go
module download). It deliberately does not install `golangci-lint`;
see `script/lint` below.
* `script/bootstrap` — install all development dependencies (Go, Go
module download). A host without Go gets the `go.mod` version through
`script/install-go`, in `.tool/go`, which `script/bootstrap` itself,
the `Makefile`, `script/fmt`, `script/fmt-check`, `script/precommit`
and `script/release` add to their `PATH`. It
deliberately does not install `golangci-lint`; see `script/lint`
below.
* `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`
* `script/projectname` — print the project name (used for the Docker
image tag)
* `script/version` — print the version string to bake into the binary.
The `Makefile`'s `LDFLAGS` call this; it is the single source of truth
for the version. See [releasing](#releasing) for the rules.
The `Makefile`'s `LDFLAGS` call this. See [releasing](#releasing) for
the rules.
* `script/install-goreleaser` — install the pinned `goreleaser` into
`.tool/bin` from a sha256-verified release archive. Idempotent, and
called by `script/bootstrap`; the release workflow calls it directly
@@ -754,100 +808,60 @@ them. We provide:
`script/bootstrap` insists on.
* `script/install-go` — install the Go toolchain named by `go.mod`'s
`go` directive into `.tool/go` from a sha256-verified `go.dev`
archive, and put it on `PATH`. Idempotent. Called only by the release
workflow, which needs a host Go for `goreleaser` to shell out to;
nothing else on the release runner does. `actions/setup-go` is not
used because it verifies the downloaded toolchain against no value in
this repo. Bumping Go edits `go.mod`, the checksum in this script, and
the `Dockerfile` `golang` digest together.
archive, for Linux or macOS on amd64 or arm64. Idempotent. On a CI
runner it also puts `.tool/go/bin` on `PATH` for the steps that
follow. Called by the release workflow, which needs a host Go for
`goreleaser` to shell out to, and by `script/bootstrap` on a host
without Go. `actions/setup-go` is not used because it verifies the
downloaded toolchain against no value in this repo. Bumping Go edits
`go.mod`, the checksums in this script, and the two `golang` digests
in the `Dockerfile` together.
* `script/release` — cross-compile and publish the release artifacts
with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose
version is not the pinned one, on the same reasoning as `script/lint`.
version is not the pinned one, because a different version would build
a different release from the same tag.
* `script/release-snapshot` — the same build with no publishing and no
tagging, into `./dist`
* `script/test` — run the test suite (verbose rerun on failure). This
runs *everything*: there is no separate integration target and no
build-tagged subset held back, so the full round-trip tests in
`internal/vaultik/integration_test.go` run on every invocation. It
passes `-count=1`, which disables Go's test result cache. That is
deliberate and it is not free: on this repo's suite it costs about 11
seconds on every repeat run (measured, back to back: 0.4s cached
versus 11.6s with `-count=1`). That is the price of the run meaning
anything, because without it an unchanged package prints
`ok <pkg> (cached)`, which is indistinguishable from a package that
really ran, so the whole suite can report a full set of `ok` lines in
under half a second having executed nothing. The `-timeout` is a hang
backstop rather than a performance budget — it applies per test binary
to test execution only, not to compilation — and is set well above the
slowest package's measured runtime. Its 120s value deliberately
diverges from the 30s `REPO_POLICIES.md` mandates; the reasoning is in
the comment in the script, and issue #101 proposes amending the policy
text.
* `script/lint` — lint by building `Dockerfile.lint`, which runs
`golangci-lint run --config .golangci.yml ./...` as a build step
inside the digest-pinned `golangci-lint` image, so a successful build
*is* a clean lint. Nothing lints on the host, at any version, ever;
the script requires Docker and fails loudly rather than falling back
to a `golangci-lint` on `PATH`. That `FROM` line is the single source
of truth for the linter version — bump it there and nowhere else.
It takes no arguments, because a build step has no command line to
pass flags to, and it passes a fresh `--build-arg CHECK_EPOCH` on
every invocation so the lint layer cannot be replayed from cache (see
`script/cibuild` below for what that mechanism defends against). To
watch the linter execute, run it as
`BUILDKIT_PROGRESS=plain script/lint` and check that the lint layer
says `RUN … golangci-lint` rather than `CACHED`.
One container per run means one lint cache and one `golangci-lint`
lock per run, both private to it and discarded with it, so concurrent
runs on one host cannot contaminate or block each other.
* `script/test` — run the test suite by building the `test` phase of
the `Dockerfile` (verbose rerun on failure). This runs *everything*:
there is no separate integration target and no build-tagged subset
held back, so the full round-trip tests in
`internal/vaultik/integration_test.go` run on every invocation. The
90s `-timeout` is a hang backstop rather than a performance budget; it
applies per test binary, to test execution only.
* `script/lint` — lint by building the `lint` phase of the
`Dockerfile`, which runs `golangci-lint config verify` and then
`golangci-lint run --config .golangci.yml ./...` as build steps in
the digest-pinned `golangci-lint` image. Nothing lints on the host.
That `FROM` line is the only pin of the linter version, and it changes
in the same commit as a re-vendored `.golangci.yml`.
* `script/lint-fix` — apply the linter's autofixes (rewrites files),
using the same pinned image, parsed out of `Dockerfile.lint`. It
cannot be a build step, because fixes have to land in the worktree, so
it bind-mounts the tree into a `docker run` and therefore needs a
*local* daemon. It is a developer convenience and never a gate: no
gate reads its exit status. Run `make lint` afterwards to find out
whether the tree is clean.
using the same pinned image, parsed out of the `lint` phase's `FROM`
line. It cannot be a build step, because fixes have to land in the
worktree, so it bind-mounts the tree into a `docker run` and therefore
needs a *local* daemon. It is a developer convenience and never a
gate: no gate reads its exit status. Run `make lint` afterwards to
find out whether the tree is clean.
* `script/fmt` — format all code (writes)
* `script/fmt-check` — check formatting (read-only)
* `script/fmt-check` — check formatting (read-only). It runs `gofmt` on
the host, over every Go file outside `.tool`.
* `script/check` — run `script/test`, `script/lint`, and
`script/fmt-check`. This is authoritative *because* `script/lint`
builds `Dockerfile.lint`: a local `make check` and CI cannot disagree
about lint findings.
`script/fmt-check`.
* `script/docker` — build the Docker image tagged via
`script/projectname`. Passes a fresh `--build-arg CHECK_EPOCH` for the
same reason `script/cibuild` does, so a local image build cannot be
green on checks it replayed from cache. It builds the *product* image
only, and the product `Dockerfile` has no lint stage, so it does not
lint: a green here means formatted, tested, and it compiles.
* `script/cibuild` — CI entrypoint, and the full gate. Two builds, in
order: `Dockerfile.lint` (the linter, as a build step) and then
`Dockerfile` (`make fmt-check` and `make test` in its builder stage,
then the product image). Either failing fails the script. It runs the
checks in the same containers CI does, from a clean copy of the tree,
so it also catches anything that depends on host state.
`.gitea/workflows/check.yml` runs it on every push to `main` and
`next` and on every pull request against either.
`script/projectname`, stamped with the version
`git describe --tags --always --dirty` gives on the host, or `unknown`
outside a git checkout. The image's build stage depends on the `lint`
and `test` phases, so this lints and tests too.
* `script/cibuild` — CI entrypoint: runs `script/bootstrap`,
`script/check`, and then the same image build as `script/docker`.
`.gitea/workflows/check.yml` runs it on every push.
It passes a fresh `--build-arg CHECK_EPOCH` to each build, unique per
invocation, which both files declare immediately above their check
`RUN`s and expand into each check command. Those layers are keyed on
that value, so a new value re-runs them even on a byte-identical tree,
and a green from this script means the checks executed. Dependency and
module layers sit above the `ARG` and still cache, so a build is not
cold.
A build that supplies no `CHECK_EPOCH` — a bare `docker build .` or
`docker build -f Dockerfile.lint .` — fails rather than lying. An
unset `ARG` is an empty string and an empty string is a stable cache
key, so without a guard such a build would serve every check layer
from cache, execute nothing, and still exit 0. Each file therefore
asserts the value is non-empty before running anything, and because
failed steps are never cached that assertion fires on every
invocation rather than once. Use `script/lint`, `script/docker` or
`script/cibuild`, which pass the arg; a bare `docker build` is a loud
error.
Every `docker build` in these scripts passes `--no-cache`, because on
an unchanged tree a cached check layer is replayed without running and
the build still exits 0. A plain `docker build .` is therefore no
evidence that the checks ran. The cost is that `script/cibuild` runs
the `lint` and `test` phases twice: once in `script/check` and again
in the image build.
* `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must
not change files), then `script/check`
* `script/install-precommit` — install the git pre-commit hook that
@@ -858,24 +872,35 @@ them. We provide:
### version numbers
The version a binary reports comes from git, not from a constant in a
file. `script/version` decides it, and everything that stamps a binary
agrees with it:
file. It is `git describe --tags --always --dirty`, which
`script/version` runs for the `Makefile`, and `script/docker` and
`script/cibuild` run themselves:
* `HEAD` is exactly on a tag → that tag with a leading `v` stripped, so
the tag `v1.0.0` produces `vaultik 1.0.0`, matching the archive name
`vaultik_1.0.0_linux_amd64.tar.gz`. `goreleaser` strips the prefix the
same way.
* anything else → `dev-<12 chars of the commit sha>`.
* either, with uncommitted changes to tracked files → a `-dirty`
* `HEAD` is exactly on a tag → that tag, such as `v1.0.0`.
* a commit after a tag → `<tag>-<N>-g<short sha>`.
* no tag reachable → the short commit sha.
* any of these, with uncommitted changes to tracked files → a `-dirty`
suffix, because a modified checkout of a tag is not that tag.
A build that is not a release never names itself like one. `vaultik
version` says so in as many words on a development build, and
`goreleaser --snapshot` stamps the same `dev-<sha>` string rather than
inventing the next patch number. If `script/version` cannot be run at
all, `make` stops with an error instead of building an unversioned
binary, and a binary that somehow carries an empty version string still
reports itself as a development build.
A `docker build .` of a clone, with no build arguments, runs the same
`git describe` (without `--dirty`) on the `.git` in its build context,
so it stamps the same value for a clean commit; the build fails if the
context carries `.git` and no version, commit or commit date comes out.
A binary built without
git metadata reports `dev`, or `unknown` when `script/docker` or
`script/cibuild` built it outside a git checkout.
`goreleaser` stamps a release binary with the tag minus its leading
`v`, so the tag `v1.0.0` produces `vaultik 1.0.0`, matching the archive
name `vaultik_1.0.0_linux_amd64.tar.gz`. `goreleaser --snapshot` stamps
`dev-<12 chars of the commit sha>` rather than inventing the next patch
number. `vaultik version` calls a build a development build when its
version is `dev`, `unknown`, `dev-<sha>`, the short commit sha or
`<tag>-<N>-g<short sha>`, with or without `-dirty`; only a plain tag,
such as `v1.0.0` or `1.0.0`, is a release. If `script/version` cannot
be run at all, `make` stops with an error instead of building an
unversioned binary, and a binary that somehow carries an empty version
string still reports itself as a development build.
### cutting a release
@@ -904,14 +929,14 @@ It is passed to `goreleaser` as `GITEA_TOKEN`. The runner's automatic
token is deliberately not used: it is not guaranteed to carry release
write access.
The Go toolchain that compiles the released binaries comes from an
`actions/setup-go` step pinned by commit sha, reading its version from
`go.mod` (currently `1.26.1`, the same version the `Dockerfile` builder
stage pins by digest). `goreleaser` shells out to `go` for every
The Go toolchain that compiles the released binaries is installed by
`script/install-go`, which downloads the version named by `go.mod`
(currently `1.26.1`, the same version the `Dockerfile` builder stage
pins by digest) and refuses the archive unless its sha256 matches the
value committed in the script. `goreleaser` shells out to `go` for every
cross-compile, so without that step the release would either fail
outright or ship binaries built by whatever unpinned toolchain the
runner happened to carry — the one unpinned thing in an otherwise
hash-pinned release path.
runner happened to carry.
To rehearse the whole build without publishing or tagging anything:
REPO_POLICIES.md
+357 -86
View File
@@ -1,6 +1,6 @@
---
title: Repository Policies
last_modified: 2026-07-06
last_modified: 2026-10-04
---
This document covers repository structure, tooling, and workflow standards. Code
@@ -60,17 +60,28 @@ style conventions are in separate documents:
prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root and runs `docker build .`; the Gitea workflow calls it. Four further
scripts are our own extensions to the standard: `script/check` runs
`script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
what the git pre-commit hook runs, and it calls `script/check`;
`script/install-precommit` installs the git pre-commit hook (the `make hooks`
target shims to it); and `script/projectname` (literally that filename) simply
outputs the project's name. Scripts that need the name call
`script/projectname` — e.g. `script/docker` assembles its image tag from it —
so those scripts stay byte-identical across all repos. Repo-type-specific
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
`script/precommit`, not in the hook itself. Model scripts are at
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
with the version; the Gitea workflow calls it. **`script/cibuild` runs
`script/bootstrap` first**, because the workflow checks out the repo and runs
nothing else, while `script/fmt-check` runs the formatter on the host: on a
pristine checkout with nothing installed the run dies there, after the
containerised gates have passed. **The bootstrap alone is not enough**:
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
source nvm for the pinned node version before invoking it, exactly as
`script/bootstrap`'s own install step does. A runner carrying nothing but
docker and git then gets through `script/check`. Four further scripts are our
own extensions to the standard: `script/check` runs `script/test`,
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
installs the git pre-commit hook (the `make hooks` target shims to it); and
`script/projectname` (literally that filename) simply outputs the project's
name. Scripts that need the name call `script/projectname` — e.g.
`script/docker` assembles its image tag from it — so those scripts stay
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the
README requirements below).
@@ -89,87 +100,198 @@ style conventions are in separate documents:
contributor should be able to understand the entire development workflow by
reading the Makefile.
- Every repo should have a `Dockerfile`. All Dockerfiles must run `make check`
as a build step so the build fails if the branch is not green. For non-server
repos, the Dockerfile should bring up a development environment and run
`make check`. For server repos, `make check` should run as an early build
stage before the final image is assembled. Dockerfiles install development
prerequisites by running `script/bootstrap` rather than duplicating installs
inline; COPY `script/` and the dependency manifests (`package.json` +
`yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
layer stays cached until dependencies change.
- Every repo should have a `Dockerfile`, and it carries the repo's gates: a
`lint` phase and a `test` phase, with the final stage depending on both so the
image cannot be built unless they pass. For non-server repos the final stage
brings up a development environment; for server repos it is the runtime image.
The gate phases and the build stage start from their pinned base images and
install what those images lack either inline, as the canonical Go `Dockerfile`
below does for `git`, or by running `script/bootstrap`, as the `prompts`
repo's own `Dockerfile` does for its yarn packages. The development
environment stage installs development prerequisites by running
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
repos use a multistage build where linting runs in an independent stage based
on the `golangci/golangci-lint` image (pinned by hash). This stage runs
`make fmt-check` and `make lint` before the full build begins. The build stage
then declares an explicit dependency on the lint stage via
`COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
linting before proceeding to compilation and tests. This ensures lint failures
surface in seconds rather than minutes, without blocking on dependency
download or compilation in the build stage.
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
no separate lint file. `script/lint` and `script/test` each build one phase
and nothing else:
The standard pattern for a Go repo Dockerfile is:
```sh
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
docker build --no-cache --target test -t "$(script/projectname)-test" .
```
**A stage that is not the last one in the file is built only when the final
stage's chain depends on it, or when `--target` names it.** That is why the
two gates are always invoked by name here, and why the final stage carries a
`COPY --from=` of a harmless file from each of them: without that edge a
plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing.
**Every `docker build` in `script/` is tagged**, here and in
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
image behind on every invocation, on every developer host and every CI
runner; a tagged one replaces the previous image.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
themselves a `docker build` and would recurse into a daemon that does not
exist in a build step. Formatting is the exception and stays on the host:
`script/fmt` writes the working tree, and `script/fmt-check` is its
read-only twin.
**No lint verdict may come from a host invocation of the linter.** On a
shared host golangci-lint reads a result cache keyed on file content rather
than location, so a second checkout of the same content is served the first
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own cache, its own `TMPDIR` and
a digest-pinned binary, so neither is reachable.
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
a `COPY` layer only when the copied content changes, so on an unchanged tree
the check `RUN` is served from cache, nothing executes, and the build still
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
four, and there is no fifth — `script/check` runs the two gate phases and
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
not evidence that anything ran: a sub-second build reporting success is a
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
and friends destroy a build cache shared with every other build on the host.
When a check is added or changed, prove it works by planting a defect it must
catch and watching the run fail on it, then revert the defect. A green run
alone shows neither that the check ran nor that it covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Debian Go image. The canonical Go repo
`Dockerfile`:
```dockerfile
# Lint stage — fast feedback on formatting and lint issues
# Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make fmt-check
RUN make lint
RUN golangci-lint run --config .golangci.yml ./...
# Build stage
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test
WORKDIR /src
# Force BuildKit to run the lint stage before proceeding
COPY --from=lint /src/go.sum /dev/null
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make test
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
ARG VERSION=dev
RUN CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# Runtime stage
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage, and the last one
FROM alpine@sha256:...
COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"]
```
Key points:
- The lint stage uses the `golangci/golangci-lint` image directly (it
includes both Go and the linter), so there is no need to install the
linter separately.
- `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates
a stage dependency. BuildKit runs stages in parallel by default; without
this line, the build stage would not wait for lint to finish and a lint
failure might not fail the overall build.
- The lint phase uses the `golangci/golangci-lint` image directly (it has
both Go and the linter), so nothing needs installing.
- `COPY --from=<phase> /src/go.sum /dev/null` is a no-op copy whose only
purpose is the ordering edge. BuildKit runs stages in parallel by default,
and a stage nothing depends on is not built at all, so without these two
lines a red gate would not fail the build.
- Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain
and nothing else.
- If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint stage must
(e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
The lint stage should not depend on the actual build output — it exists to
fail fast.
- If the project requires CGO or system libraries for linting (e.g.
`vips-dev`), install them in the lint stage with `apk add`.
- The build stage runs `make test` after compilation setup. Tests run in the
build stage, not the lint stage, because they may require compiled
artifacts or heavier dependencies.
- If the project requires CGO or system libraries for linting, install them
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
has no `apk`, so install with `apt-get` under the Debian package name
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them:
```dockerfile
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` (which runs `docker build .`) on push. Since the
Dockerfile already runs `make check`, a successful build implies all checks
pass.
runs `script/cibuild` on push, and checks out the repo as its only other step.
That script bootstraps, runs the gate phases, and then builds the image, so a
successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. A separate
workflow limited to `main` by a `branches` list under `on: push` cannot be
checked by review: to try a change to it, add the feature branch to that list
and push, then remove the branch from the list again before merging. Keep any
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
from the feature branch publishes nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -189,14 +311,21 @@ style conventions are in separate documents:
module under test to verify it compiles/parses. There is no excuse for
`make test` to be a no-op.
- `make test` must complete in under 20 seconds. Add a 30-second timeout in the
Makefile.
- `make test` must complete in under 60 seconds. That is the hard cap, and a
suite that exceeds it fails. Under 20 seconds is the target. A suite between
20 and 60 seconds is still green, but the overage must be filed as an
improvement bug against that repo. Add a 90-second timeout to the test
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
hard cap so that it catches a genuinely hung test rather than a merely slow
one.
- **`make test` should use the conditional verbose rerun pattern.** Run tests
without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to
show full output. This keeps CI logs and `docker build` output clean on
success (just package/suite summaries) while providing full diagnostic detail
on failure (every test case, every assertion). The general shell pattern:
- **The test command should use the conditional verbose rerun pattern.** Run
tests without `-v` (verbose) first. If tests fail, automatically rerun with
`-v` to show full output. This keeps CI logs and `docker build` output clean
on success (just package/suite summaries) while providing full diagnostic
detail on failure (every test case, every assertion). The command lives in the
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the
Makefile form below is the same pattern for any repo-local invocation:
```makefile
test:
@@ -209,11 +338,26 @@ style conventions are in separate documents:
```makefile
test:
@go test -timeout 30s -race -cover ./... || \
@go test -count=1 -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 30s -race -v ./...; exit 1; }
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
```
`-count=1` is required on both invocations: it defeats Go's test _result_
cache, so neither run can report a stored pass in place of running the
tests. It leaves the build cache alone, so it costs the runtime of the suite
and no recompilation.
That cache is Go's own, separate from Docker's layer cache. Go stores a
passing result in its cache directory (`GOCACHE`), and when the same tests
run again on unchanged code it prints that result, marked `(cached)`,
without running them. That matters on a developer's machine, where this
target runs and the directory lasts from one run to the next. The `test`
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
result for this repo's tests and nothing before its `go test` step runs a
test, so there is nothing to replay. `--no-cache` (above) is what makes that
step run on an unchanged tree.
Python example:
```makefile
@@ -239,10 +383,84 @@ style conventions are in separate documents:
must be in `.gitignore`. No exceptions.
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), language build artifacts, and `node_modules/`.
Fetch the standard `.gitignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
a new repo.
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
setting up a new repo. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified.
- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns
across unmodified leaves secrets in the build context.** Docker matches with
`moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension, so
`*` does not cross `/` and a pattern without a leading `**/` is anchored at
the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key`
therefore excludes only the copies at the repository root, while `config/.env`
and `certs/server.key` still reach the context and can land in an image layer
— which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
secret names use character ranges — `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`,
and likewise for `.envrc` and the extensionless SSH keys. Where such a pattern
also catches something the build needs, re-include it with a negation
(`!docs/example.env`); deleting the pattern reopens the exposure for every
other file it covers. Fetch the standard `.dockerignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend
it with the repo's own artifacts.
- **In-repo agent scratch belongs in both files, written to each file's own
semantics.** `.claude/` holds one worktree per in-flight agent — an entire
additional checkout of the repo — so under `COPY . .` the build context
inflates by a multiple of the repo and another session's unreviewed work can
be copied into an image layer. In `.gitignore` the entry is `.claude/`,
unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/`
prefix, because the prefixed form would also delete any nested directory of
that name from the build. Anchoring carries a known gap that the canonical
`.dockerignore` states in its own comment, since consuming repos receive the
file and not the tracker: the directory is created in the agent's working
directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there.
- **A plain `docker build .` of a clone stamps the version that
`git describe --tags --always` gives**, derived from the `.git` in the build
context as the canonical `Dockerfile` above shows. Without its failure check,
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
and the build would still exit 0. `script/docker` and `script/cibuild` pass
the version they compute on the host; it takes precedence. They do this
byte-identically across repos:
```sh
# Own line: a failing command substitution inside an argument does not
# trip `set -e`, so the inline form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$(script/projectname)" .
```
`--always` makes an untagged repo yield an abbreviated commit hash rather
than failing, and the `[ -n "$version" ]` line is the single place the
fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so a repo that embeds a
tag-derived version must set `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build
a probe image that does `COPY . .`, and list what actually landed
(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context`
size is not a substitute: a nested secret is a few bytes, and BuildKit
transfers only the delta from the previous build.
- **No build artifacts in version control.** Code-derived data (compiled
bundles, minified output, generated assets) must never be committed to the
@@ -258,9 +476,56 @@ style conventions are in separate documents:
- Make all changes on a feature branch. You can do whatever you want on a
feature branch.
- `.golangci.yml` is standardized and must _NEVER_ be modified by an agent, only
manually by the user. Fetch from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`.
- `.golangci.yml` is standardized. The vendored copy in a consuming repo must
_NEVER_ be modified by an agent: fetch it from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it
byte-identical, so that no repo can quietly loosen its own linting. Linter
configuration changes are made to the canonical copy in the `prompts` repo and
reach consuming repos by re-vendoring; an agent may open a PR against
canonical, which only the user merges. One list is exempt from byte-identity,
because it cannot be written once for every repo: the `deny` list of the
`test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
image
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
directive must not name a newer Go minor version than the one golangci-lint
was built with, or golangci-lint refuses to lint it: this release lints
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
`PATH` only, so on an already-provisioned machine the pin is inert and a
version bump is a silent no-op — while the Dockerfile, installing into a clean
image, gets the pinned version, so a local `make check` and `make docker` can
disagree about what the tool even is. The canonical form:
- compares the installed version against the pin over the **whole** version
token; a parser that stops at the first `-` reports `2.12.2` for a host
running `2.12.2-rc1` and skips the install;
- treats absent, non-zero, empty or unrecognised `--version` output as a
mismatch, so the failure direction is a redundant install and never a
skipped one;
- after installing, re-resolves the binary the way callers do — `hash -r`,
then through `PATH`, not through the directory the installer wrote to —
and fails naming the resolved path, since an install that a shadowing
binary hides succeeds while changing nothing any caller sees;
- is actually called, and prints the version on both success paths: a
function defined and never invoked has the same exit status and the same
empty output as one that worked.
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD).
@@ -374,12 +639,14 @@ style conventions are in separate documents:
settings.
- Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
language-specific config). Everything else goes in a subdirectory. Canonical
subdirectory names:
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
and language-specific config). Everything else goes in a subdirectory.
Canonical subdirectory names:
- `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
body is a single call into `internal/` or `pkg/`, no project logic in
`cmd/`
- `configs/` — configuration templates and examples
- `deploy/` — deployment manifests (k8s, compose, terraform)
- `docs/` — documentation and markdown (README.md stays in root)
@@ -406,3 +673,7 @@ style conventions are in separate documents:
- Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
TODO.md
+414 -9
View File
@@ -14,17 +14,408 @@ pre-1.0
# Next Step
Define the remaining scope for the first tagged release under the 1.0.0
milestone, then cut that tag. The mechanism to cut it now exists and is
exercised; what is left is the scope decision, which is the owner's.
This step deliberately names one version number: it previously said
"cut v0.1.0" while the `Makefile` baked in `1.0.0-rc.1` and the issue
milestone said 1.0.0, and three different answers to "what is the next
release" is exactly the contradiction
[issue #65](https://git.eeqj.de/sneak/vaultik/issues/65) was filed over.
The 1.0 work is complete on `next`: the scope settled on
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) was the work
already planned for 1.0, and all of it has landed. The mechanism to cut
the tag exists and is exercised; what is left is merging `next` to
`main` and tagging, both the owner's.
# Completed Steps
- 2026-10-08: Made a backup cancelled under `--skip-errors` stop at once
([issue #286](https://git.eeqj.de/sneak/vaultik/issues/286)). After
Ctrl-C or SIGTERM, phase 2 treated the cancellation error from each
remaining file like an unreadable file: it opened the file, printed an
error line for it, counted it as failed and went on to the next one.
The run now stops at the first file after the cancellation, and no
file is counted as failed because of it.
- 2026-10-08: Made `remote nuke` delete the `.partial` files that
uploads cut off part-way leave on the destination store
([issue #281](https://git.eeqj.de/sneak/vaultik/issues/281)). The
file and rclone listings skip such a file, so the command left it in
place and still reported the store empty. It now removes them under
`metadata/` and `blobs/` as its last step.
- 2026-10-08: Made a local index error while recording a directory or
symlink stop a backup under `--skip-errors`
([issue #284](https://git.eeqj.de/sneak/vaultik/issues/284)). Phase 2
only records such an entry in the local index, since it has no data
to open or read, but an error doing so was skipped like an unreadable
file. The snapshot then completed without the entry, and a restore
did not recreate it. The error now stops the backup, as it does
without the flag.
- 2026-10-08: Counted a file that a backup could not store as failed
([issue #280](https://git.eeqj.de/sneak/vaultik/issues/280)). A file
that phase 1 counted and phase 2 could not open, because it was
unreadable under `--skip-errors` or removed in between, was reported
in the summary as unchanged with its bytes as backed up, and the
`snapshots` row's `file_count` and `total_size` included it. The
summary now counts it as failed, and its data total and the row leave
it out.
- 2026-10-08: Made `remote info` report a snapshot's blob count and
blob size as unknown when its manifest cannot be read
([issue #272](https://git.eeqj.de/sneak/vaultik/issues/272)). The
orphan figures were already unknown in that case, but the snapshot's
row still gave 0 blobs and 0 B, in the table and in `--json`. The row
now reads `unknown` and `--json` gives `null`. A directory with no
manifest still shows 0, since the orphan figures count its blobs as
orphaned.
- 2026-10-08: Kept the progress line of a snapshot with more than one
path within 100%
([issue #271](https://git.eeqj.de/sneak/vaultik/issues/271)). The
bytes and files processed counted every path, but the totals they
were divided by held only the current path's, so a second path
smaller than the first showed more than 100% and an ETA of `unknown`.
The totals now add up over the paths scanned so far. The rate behind
the ETA restarts when each path's scan phase ends, so that phase does
not lower the rate while the path is processed; until then the rate
is the previous path's.
- 2026-10-08: Made a second `snapshot create` of one name succeed when
it starts in the same second as the first
([issue #270](https://git.eeqj.de/sneak/vaultik/issues/270)). The
timestamp in a snapshot ID is in whole seconds, so the second run got
the first run's ID and failed with `UNIQUE constraint failed:
snapshots.id`. When the local index already has a snapshot with the
ID, the create now waits a second and takes a new timestamp.
- 2026-10-07: Documented that `--json` skips the confirmation prompt of
`snapshot remove` and `prune`
([issue #268](https://git.eeqj.de/sneak/vaultik/issues/268)). Both
commands delete without asking under `--json`, since a prompt on stdout
would break the JSON document, but the help and the README described
only `--force` as skipping it. The `--json` help of both commands and
their README entries now say so.
- 2026-10-07: Made a symlink whose target cannot be read stop the backup
([issue #269](https://git.eeqj.de/sneak/vaultik/issues/269)). It was
left out of the snapshot with only a debug log line, even without
`--skip-errors`. It now aborts the run, or with `--skip-errors` is
skipped with the `Failed to access` error line that any other entry
the scan cannot read gets.
- 2026-10-07: Stopped a killed rclone upload from leaving a truncated
object under its key
([issue #266](https://git.eeqj.de/sneak/vaultik/issues/266)). The
rclone backend wrote each object straight to its key, so killing an
upload to a local or sftp remote left a truncated object there; the
next backup found the key with `Stat`, skipped the upload and recorded
a snapshot that could not be restored. On a remote with a server-side
move, an object is now written under a temporary name ending in
`.partial` and moved into place, and listings skip such names. Remotes
without one are still written in place.
- 2026-10-07: Made an interrupted command exit 130 and say so
([issue #267](https://git.eeqj.de/sneak/vaultik/issues/267)). Ctrl-C
or SIGTERM during `snapshot create`, `snapshot restore` or `snapshot
verify` exited 0 with no error line (`snapshot verify --json` exited
1, also without one), so a `--cron` run that never finished looked
like a success. A command stopped by either signal now exits 130 and
prints `interrupted before the command finished` on stderr, under
`--cron` and `--json` too.
- 2026-10-07: Corrected documentation, help text and comments that were
false about the code
([issue #233](https://git.eeqj.de/sneak/vaultik/issues/233)). A blob
is not streamed to storage. The README, `ARCHITECTURE.md` and
`config.example.yml` now say a backup needs free temporary space,
because each blob is written whole to a temporary file before it is
uploaded (up to about `blob_size_limit`; an rclone destination that
cannot stream uploads needs about twice that) and the metadata export
writes copies of the local index. Temporary files go to `$TMPDIR`
(default `/tmp`); with `TMPDIR` unset, SQLite writes one of those
copies to `/var/tmp`. Also corrected: the snapshot ID format, what restore
reads and how incomplete snapshots are removed in `docs/DATAMODEL.md`,
what `source_path` holds, the `index_path` and config file defaults,
what `snapshot remove` cleans up, how the release gets its Go
toolchain, and the `script/release` and `script/fmt-check` comments.
- 2026-10-07: Cut the time the `internal/vaultik` and `internal/database`
tests take ([issue #235](https://git.eeqj.de/sneak/vaultik/issues/235)).
Most of the `internal/vaultik` time went to 24 tests that ran one at a
time only because they call `log.Initialize`; they now call it before
`t.Parallel()`, as the package's other tests do. `TestLargeDatasets`
committed each of its 1,500 inserts on its own and now makes them in
one transaction. `TestDedupOnlySnapshotRestores` gives its second
backup its own snapshot name instead of sleeping past the one-second
timestamp in the snapshot ID.
- 2026-10-07: Made two messages say only what is true
([issue #240](https://git.eeqj.de/sneak/vaultik/issues/240)). A config
file that others can read was warned about as containing S3
credentials even when it set none, as a `file://` config does. The
warning now says the file may contain S3 credentials only when
`s3.access_key_id` or `s3.secret_access_key` is set, since either may
come from a `${...}` reference rather than the file, and otherwise
says the file is readable by others. `snapshot purge` against a
destination store it could not list gave an error with
`listing remote snapshots:` in it twice; the prefix now appears once.
- 2026-10-07: Made `s3.part_size` set the multipart upload part size
([issue #232](https://git.eeqj.de/sneak/vaultik/issues/232)). It was
loaded and defaulted but never passed to the S3 client, whose uploader
used a fixed 10MiB part. It now reaches the uploader for `storage_url`
and for the `s3.*` fields, and a part size S3 refuses, below 5MiB or
above 5GiB, `0` included, fails at config load. A blob too large for
S3's limit of 10,000 parts at the configured size is uploaded in larger
parts. The docs gave the default as `5MB`, which the config file reads
as 5,000,000 bytes, below the minimum; they now say `5MiB`.
- 2026-10-07: Made per-name retention work when the hostname contains `_`
([issue #230](https://git.eeqj.de/sneak/vaultik/issues/230)). A
snapshot ID is `hostname_name_timestamp`, and the name was read as
everything between the first and the last `_`, so with
`hostname: my_host` the name `home` came out as `host_home`.
`snapshot purge --keep-latest --snapshot home` then printed "No
snapshots to delete", and `snapshot create --prune` purged nothing
without a message. The name is now read using the hostname the
`snapshots` table stores with each snapshot, cut at its first `.` as it
is in the ID.
- 2026-10-07: Made `remote info` stop reporting a snapshot's blobs as
orphaned when its manifest cannot be read, and stop printing raw
names from under `metadata/`
([issue #228](https://git.eeqj.de/sneak/vaultik/issues/228)). A
manifest it failed to read was skipped, so that snapshot's blobs were
counted as orphaned and the report advised running `vaultik prune`.
The orphan figures are now unknown in that case, with no prune
advice, and `--json` gives them as `null` with the unreadable remote
keys in `unreadable_manifests`. A name under `metadata/` that is not
64 lowercase hex characters is now skipped with a warning instead of
being printed, control characters included. A manifest under a
skipped name is then not read either, so it also leaves the orphan
figures unknown, and `--json` counts such manifests in
`skipped_manifest_count`. A directory with no manifest in it, as left
by an interrupted backup, leaves the figures known.
- 2026-10-07: Made `config set` keep a string that looks like a number
([issue #229](https://git.eeqj.de/sneak/vaultik/issues/229)). It wrote
every value unquoted, and `config.Load` reads the file through untyped
YAML, so an access key `00112233` loaded as `38043` and a hostname `007`
as `7`. A value for a string setting in `config.Config` is now tagged as
a YAML string, which the file quotes wherever YAML would read a number or
a boolean; other settings are still written unquoted.
- 2026-10-07: Made a backup notice a file rewritten with its size
unchanged and a new mtime in the same second as the one in the index
([issue #226](https://git.eeqj.de/sneak/vaultik/issues/226)). The
`files` table held mtime in whole seconds and the scanner compared
whole seconds, so every later snapshot kept the old content. A new
`mtime_nsec` column holds the nanoseconds within the second that
`mtime` holds, and the scanner compares the full mtime. A local index
created before the change lacks the column and is rebuilt with
`vaultik database delete` and a full backup.
- 2026-10-07: Made taking the process-wide lock atomic
([issue #227](https://git.eeqj.de/sneak/vaultik/issues/227)). The lock
read `vaultik.pid`, checked whether that PID was alive and then wrote
its own, so two writers started together could both pass the check and
both run. It is now an `flock` on `vaultik.pid`, held until the run
ends; the kernel drops it when the process exits, so a crash leaves no
lock behind. A clean exit now empties the file instead of deleting it,
because deleting it would let two later runs each lock a different
file.
- 2026-10-06: Made `snapshot remove --json` write only its document to
stdout when the destination store cannot be reached
([issue #251](https://git.eeqj.de/sneak/vaultik/issues/251)). Its
warning that the snapshot's metadata was left on the destination store
went to stdout ahead of the document, breaking `| jq` on a command that
exited 0. Under `--json` the warning now reaches stderr only, through
the logger. The warning, the README and the command's help said
`vaultik prune` would finish the cleanup, but `prune` never removes
snapshot metadata; they now say to run `vaultik snapshot remove` for the
snapshot again once the destination store is reachable.
- 2026-10-06: Made the backup summary and the `snapshots` row count each
file, byte and upload once
([issue #225](https://git.eeqj.de/sneak/vaultik/issues/225)). The
scanner added a file's bytes again for each new chunk and counted a
file as unchanged for each chunk already stored, so a first backup
reported twice its size and "backed up" could go negative. Upload
figures came from the progress reporter, which `--cron` turns off, and
`blob_count` counted earlier paths' blobs again for each later path.
The scanner now counts uploads itself; `blob_size`,
`blob_uncompressed_size` and `compression_ratio` describe the blobs
the snapshot references, and `docs/DATAMODEL.md` now says
`chunk_count` and `blob_count` count what the run added.
- 2026-10-06: Made command output follow the README's stdout and stderr
rules ([issue #224](https://git.eeqj.de/sneak/vaultik/issues/224)). The
startup banner went to stdout, so a `completion` script or a
`config get` value started with it; the banner now goes to stderr. A
failing `remote info`, `prune` or `snapshot remove` under `--json`
printed nothing on either stream, and now reports its error on stderr.
`snapshot verify --quiet` printed its whole report; it now prints
none, and a failure still reaches stderr with the same exit status.
- 2026-10-06: Made a backup without `--cron` of a snapshot with two or
more `paths` complete instead of panicking with `close of closed
channel` ([issue #253](https://git.eeqj.de/sneak/vaultik/issues/253)).
`Scan` runs once per path and started and stopped the progress
reporter each time, and a second stop panics. The reporter is now
started and stopped once per snapshot, around the scans of all its
paths.
- 2026-10-06: Made a restore path argument select only that path and
what is beneath it
([issue #223](https://git.eeqj.de/sneak/vaultik/issues/223)). The
lookup matched with SQL `LIKE`, so `/home/u/doc` also restored
`doc2`, `DOC` and `doc.txt.bak`, and a `_` or `%` in the path acted
as a wildcard. A backup used the same lookup to load the known files
of each configured path, so files of a longer sibling path were
counted as deleted. `FileRepository.ListUnderPath`, which replaces
`ListByPrefix`, returns the file at the path and every file whose
path starts with the path plus `/`, compared case-sensitively.
- 2026-10-06: Made restore return an error instead of panicking on a
malformed snapshot database
([issue #231](https://git.eeqj.de/sneak/vaultik/issues/231)). A chunk
hash shorter than 16 characters crashed the error message naming it,
and `--verify` dereferenced a missing `chunks` row and allocated
whatever chunk size the database gave. Those messages now go through
`shortHash`, a missing row is an error, and `--verify` rejects a
negative size and hashes each chunk as a stream.
- 2026-10-06: Made `s3://bucket/prefix` and `s3://bucket/prefix/` the same
destination ([issue #222](https://git.eeqj.de/sneak/vaultik/issues/222)).
The S3 client put the prefix directly in front of each key, so a prefix
without a trailing slash stored `prefixblobs/...`. A non-empty prefix is
now joined to every key with one `/`, giving the README's
`<bucket>/<prefix>/blobs/...` layout. The `s3.prefix` config setting goes
through the same client and gets the same join.
- 2026-10-06: Made restore apply owners, modes and times in an order
that keeps them
([issue #219](https://git.eeqj.de/sneak/vaultik/issues/219)). A
directory got its stored mode and mtime before its contents were
written, so a read-only directory came back without its files and a
non-empty one carried the time of the restore. Directories are now
created owner-only and get their stored owner, mode and mtime once the
restore loop is done, each before its parent. A file's mode is applied
after its chown, which on Linux clears setuid and setgid, and a
symlink gets its stored owner (as root) and mtime on the link itself.
- 2026-10-06: Made `go.mod` what `go mod tidy` writes, so the
pre-commit hook no longer stops every commit
([issue #246](https://git.eeqj.de/sneak/vaultik/issues/246)). A test
in `internal/cli` imports `github.com/spf13/pflag` directly, but
`go.mod` still marked it `// indirect`, and `script/precommit` fails
whenever the tidy changes `go.mod`. It is now in the direct `require`
block.
- 2026-10-06: Made the README's steps for restoring on another machine
work ([issue #221](https://git.eeqj.de/sneak/vaultik/issues/221)).
`config init` wrote a placeholder recipient that `config.Load`
rejects, so every command on the new machine failed before reaching
the store. The file now has an empty `age_recipients` list,
`config.Load` accepts an empty list, and `snapshot create` refuses to
run without a recipient.
- 2026-10-06: Made `snapshot restore --skip-errors` skip the files that
need a blob it cannot download
([issue #218](https://git.eeqj.de/sneak/vaultik/issues/218)). A missing
or damaged blob ended the restore even with the flag, after restoring
whichever files happened to come first. Every file that needs such a
blob is now reported as failed, the rest are restored, and the command
still exits non-zero. Without the flag the blob error still aborts.
- 2026-10-06: Re-vendored the canonical files from `sneak/prompts` at
`dd4027b` ([issue #213](https://git.eeqj.de/sneak/vaultik/issues/213)).
Linting and testing are now the `lint` and `test` phases of the
`Dockerfile`, and the build stage depends on both. `Dockerfile.lint`,
`CHECK_EPOCH` and the tests that checked them are gone; every
`docker build` in `script/` passes `--no-cache` instead. golangci-lint
is v2.14.0, with its new findings fixed in the code, and the rules in
`CLAUDE.md` now live in `AGENTS.md`.
- 2026-10-06: Made the local index actually run in WAL mode with a busy
timeout ([issue #217](https://git.eeqj.de/sneak/vaultik/issues/217)).
The connection settings were written in a form the SQLite driver
ignores, so the index ran without either and `snapshot list` or `info`
during a backup could make the backup's next write fail with
`database is locked`. They are now `_pragma=` parameters, and the
metadata export copies the open index with `VACUUM INTO`, because a
copy of the file alone misses rows still in the `-wal` file.
- 2026-10-06: Made a backup record the real uid and gid of files,
directories and symlinks
([issue #216](https://git.eeqj.de/sneak/vaultik/issues/216)). The
scanner asked the stat result for `Uid()` and `Gid()` methods, which
`*syscall.Stat_t` does not have, so every entry was stored as `0:0`
and a restore as root gave every file to root. It now reads the
`Uid` and `Gid` fields of `*syscall.Stat_t`.
- 2026-10-06: Made a `file://` destination whose directory is missing
count as one that cannot be listed
([issue #220](https://git.eeqj.de/sneak/vaultik/issues/220)). The file
backend listed a missing directory as an empty store, so with the
quickstart's USB stick unplugged `snapshot list` reported every local
snapshot as missing from the store, `snapshot remove` claimed to have
removed metadata it never reached, and `prune` dropped every local
snapshot record. Listing a missing destination directory is now an
error; a first backup still creates the directory.
- 2026-10-06: Made `--older-than` and `--keep-newer-than` reject a
duration with characters outside its number-and-unit parts
([issue #215](https://git.eeqj.de/sneak/vaultik/issues/215)). The
parser picked out the parts it recognised and skipped the rest, so
`1.0y` became zero and `--prune --keep-newer-than 1.0y` deleted every
snapshot of the backed-up names, the new one included. `1.0y`,
`1,5y`, `30 days`, `x7d` and a bare `0` are now errors; decimals still
work in Go units such as `1.5h`.
- 2026-10-06: Made a backup re-chunk a known file that lists a chunk no
uploaded blob holds
([issue #214](https://git.eeqj.de/sneak/vaultik/issues/214)). File
rows are shared by every snapshot and updated in place, so removing or
pruning the only snapshot that referenced a changed file's current
blob left an older snapshot keeping a row that matched the disk while
no blob held its chunks. The next backup skipped the file and
completed a snapshot that could not restore it.
- 2026-09-22: Routed the last direct-to-stdout command output through
`internal/ui`
([issue #149](https://git.eeqj.de/sneak/vaultik/issues/149)). The
`version`, `info`, `remote info`, `config`, and `database delete`
commands wrote plain text straight to stdout, so they were unstyled and
ignored `--quiet`. Output now falls in two buckets, both governed by
`internal/ui`: status lines and confirmations go through its message
methods (styled, and `--quiet` silences them), while the data a command
exists to produce — the reports, the `snapshot list` table, `config get`
values, and the `--json` documents — is written plain. `--quiet`
silences the human reports and tables but never the `config get` value
or the `--json` documents, which a script depends on, and the
`database delete` confirmation prompt is always shown. The README
output-style section now states this rule.
- 2026-09-22: Validated blob hashes, offsets and lengths read back from
the destination before using them
([issue #155](https://git.eeqj.de/sneak/vaultik/issues/155)). A blob
hash taken from the downloaded database or the store listing was
trusted unchecked, so a hostile remote could drive a decrypted blob to
be written outside the cache directory (a hash like `aa/../../etc`) or
crash a command with a short or negative value. `blobDiskCache.path`
now refuses any key containing a path separator, and `ReadAt` rejects a
negative offset or length and bounds with `length > size-offset` so a
sum cannot overflow past the check. A new `isBlobHash` helper (a plain
function, not a method — the packer stores `temp-placeholder-{uuid}` as
a hash) gates `FetchBlob`, shallow and deep verify; the `blobs/` and
`metadata/` listings skip a non-conforming name with a warning; and
short-hash prefixes in log and error text go through a `shortHash`
helper that cannot panic. `verify`'s chunk reader also rejects a
negative `blob_chunks` length and streams the chunk instead of
allocating a database-supplied size. `restore.go` and
`internal/database` were left untouched to avoid colliding with the
in-flight [issue #156](https://git.eeqj.de/sneak/vaultik/issues/156)
work; the cache-path and `FetchBlob` guards already stop the unsafe
write and fetch, so restore's own `buildBlobIndexes` early check is
deferred as fail-fast defense in depth.
- 2026-09-21: Stopped an interrupted blob upload from making a later
backup deduplicate against data that was never stored
([issue #148](https://git.eeqj.de/sneak/vaultik/issues/148)). The
@@ -38,6 +429,19 @@ release" is exactly the contradiction
recorded with no remote backend are marked uploaded so this invariant
holds uniformly.
- 2026-09-22: Made restore refuse any snapshot path that would write
outside the target directory
([issue #154](https://git.eeqj.de/sneak/vaultik/issues/154)).
`restoreFile` and `verifyRestoredFiles` joined the stored path onto the
target with no containment check, so a `..` segment or an absolute path
escaped the target and a restored symlink could redirect a later child
write anywhere on disk. Every stored path is now rejected unless
`filepath.IsLocal` accepts it with the leading separator removed, and
each existing ancestor directory below the target is `Lstat`ed to refuse
descending through a symlink; honest symlinks pointing outside the tree
are still written verbatim. age decryption proves a snapshot is
readable, not honest, and restore usually runs as root.
- 2026-09-21: Stopped `--json` from silencing stderr diagnostics
([issue #112](https://git.eeqj.de/sneak/vaultik/issues/112)). `--json`
used to be folded into `Quiet`, which pinned the log level to `WARN`,
@@ -642,4 +1046,5 @@ release" is exactly the contradiction
# Future Steps
None queued; the release-scoping item is now the Next Step.
Work planned after 1.0 is listed in the README
[roadmap](README.md#roadmap).
cmd/vaultik/dockerversion_test.go
+101 -54
View File
@@ -11,92 +11,139 @@ import (
// This file guards the version stamping of the product image (issue
// #75). The failure it protects against is silent: the image still
// builds and runs, but `vaultik version` inside it reports "commit:
// unknown", so an operator cannot tell which source produced a given
// backup. .dockerignore excludes .git, so the build cannot derive the
// commit itself; the values must be computed on the host and passed in.
// unknown" or a version of "dev", so an operator cannot tell which
// source produced a given backup. The build takes the version as a
// build arg, which script/docker computes on the host, and otherwise
// derives it from the .git in its context; the commit and its date
// always come from that .git.
//
// These are parses of the committed files, for the same reason the lint
// guards next door are: shelling out to docker would nest a build
// inside `make test`. That `vaultik version` in the built image really
// prints the host's version is verified by hand and recorded on the
// pull request.
// These are parses of the committed files, because shelling out to
// docker would nest a build inside `make test`. That `vaultik version`
// in the built image really prints the host's version is verified by
// hand.
// dockerScript is script/docker, relative to the repository root.
const dockerScript = "script/docker"
// The files under guard, relative to the repository root.
const (
productDockerfile = "Dockerfile"
dockerScript = "script/docker"
)
// versionArgs are the ldflag targets the build stamps and, matching
// them, the build args the host must supply. The names line up so the
// same list checks both files.
func versionArgs() []string {
return []string{"VERSION", "COMMIT", "COMMIT_DATE"}
}
// TestProductDockerfileTakesVersionAsBuildArgs fails unless the build
// declares each version arg and stamps it into the binary by ldflag
// reference, rather than computing it in the container.
func TestProductDockerfileTakesVersionAsBuildArgs(t *testing.T) {
// TestProductDockerfileTakesVersionAsBuildArg fails unless the build
// declares ARG VERSION, with no default, and stamps it into the binary
// whenever it is given, ahead of the value derived in the container.
func TestProductDockerfileTakesVersionAsBuildArg(t *testing.T) {
t.Parallel()
found := instructions(t, productDockerfile)
for _, arg := range versionArgs() {
require.GreaterOrEqual(t, indexOf(found, "ARG "+arg), 0,
"%s must declare `ARG %s` so the host can pass it in",
productDockerfile, arg)
require.Contains(t, found, "ARG VERSION",
"%s must declare `ARG VERSION`, with no default, so the host can"+
" pass it in", productDockerfile)
assertLdflagReferences(t, found, arg)
}
assertLdflagReferences(t, found, "VERSION")
}
// TestProductDockerfileDoesNotDeriveVersionItself is the anti-regression
// for the original defect: the container ran `git rev-parse`, but .git
// is not in the build context, so it always resolved to "unknown". No
// git command may reach into a build that cannot see the history.
func TestProductDockerfileDoesNotDeriveVersionItself(t *testing.T) {
// TestProductDockerfileDerivesVersionFromGit fails unless a build given
// no VERSION, such as a plain `docker build .` of a clone, takes it from
// `git describe` of the .git in its context, stamps "dev" when the
// context has no .git, and fails rather than stamp "dev" when that .git
// yields no version. The commit and its date come from the same .git,
// and the build fails rather than stamp them "unknown" when it is
// present.
func TestProductDockerfileDerivesVersionFromGit(t *testing.T) {
t.Parallel()
text := instructionText(readRepoFile(t, productDockerfile))
found := instructions(t, productDockerfile)
assert.NotContains(t, text, "git ",
"%s must not run git: .git is excluded from the build context, so"+
" any value it derives is wrong. Pass version, commit and date"+
" in as build args instead.", productDockerfile)
buildAt := indexContaining(found, "go build")
require.GreaterOrEqual(t, buildAt, 0, "%s must build", productDockerfile)
assert.Contains(t, found[buildAt], "git describe --tags --always || echo dev",
"%s must derive the version from git when no VERSION is given,"+
" and stamp dev when the context has no .git", productDockerfile)
assert.Contains(t, found[buildAt], "[ -e .git ]",
"%s must fail when the context carries .git but yields no version",
productDockerfile)
assert.Contains(t, found[buildAt], "git rev-parse HEAD",
"%s must stamp the commit from git", productDockerfile)
assert.Contains(t, found[buildAt], "git show -s --format=%cs HEAD",
"%s must stamp the commit date from git", productDockerfile)
assert.Contains(t, found[buildAt],
`[ "$commit" = unknown ] || [ "$commit_date" = unknown ]`,
"%s must fail when the context carries .git but yields no commit"+
" or date", productDockerfile)
}
// TestDockerScriptComputesVersionOnTheHost fails unless script/docker
// derives each value where .git exists and passes it as a build arg,
// with VERSION coming from script/version so a Docker build reports the
// same string a local build of the same tree would.
// passes the version it derives where .git exists as a build arg.
func TestDockerScriptComputesVersionOnTheHost(t *testing.T) {
t.Parallel()
script := readRepoFile(t, dockerScript)
for _, arg := range versionArgs() {
assert.Contains(t, script, "--build-arg "+arg+"=",
"%s must pass --build-arg %s to the build", dockerScript, arg)
}
assert.Contains(t, script, "/version",
"%s must take VERSION from script/version, the source of truth"+
" shared with the Makefile", dockerScript)
assert.Contains(t, script, "--build-arg VERSION=",
"%s must pass --build-arg VERSION to the build", dockerScript)
}
// assertLdflagReferences fails unless some build instruction stamps the
// named variable from the ARG (a ${arg} reference), not from a value
// computed inside the container.
// assertLdflagReferences fails unless the build instruction uses the
// named ARG whenever it is given (a ${arg:- reference), so a value
// passed in is not overridden by one derived inside the container.
func assertLdflagReferences(t *testing.T, found []string, arg string) {
t.Helper()
for _, instruction := range found {
if strings.HasPrefix(instruction, "RUN ") &&
strings.Contains(instruction, "go build") &&
strings.Contains(instruction, "${"+arg+"}") {
strings.Contains(instruction, "${"+arg+":-") {
return
}
}
assert.Fail(t, "version arg is declared but never stamped",
"the go build in %s must reference ${%s} in its ldflags, or the"+
" arg is passed and discarded", productDockerfile, arg)
"the go build in %s must use ${%s:-...}, or the arg is passed and"+
" discarded", productDockerfile, arg)
}
// instructions returns the Dockerfile's instructions, one per element,
// with comments and blank lines dropped and continuation lines joined,
// so a multi-line RUN is one string.
func instructions(t *testing.T, name string) []string {
t.Helper()
var (
out []string
continued string
isContinued bool
)
for line := range strings.SplitSeq(readRepoFile(t, name), "\n") {
trimmed := strings.TrimSpace(line)
if !isContinued && (trimmed == "" || strings.HasPrefix(trimmed, "#")) {
continue
}
isContinued = strings.HasSuffix(trimmed, `\`)
continued += strings.TrimSuffix(trimmed, `\`)
if isContinued {
continue
}
out = append(out, strings.Join(strings.Fields(continued), " "))
continued = ""
}
return out
}
// indexContaining returns the position of the first instruction
// containing want; -1 if there is none.
func indexContaining(found []string, want string) int {
for i, instruction := range found {
if strings.Contains(instruction, want) {
return i
}
}
return -1
}
cmd/vaultik/lintdocker_test.go
-367
View File
@@ -1,367 +0,0 @@
package main_test
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// This file guards the shape of the lint gate. Every property asserted
// here is one whose loss is SILENT: the build still exits 0, the gate
// still looks green, and nothing was linted or tested.
//
// The gate is a build step. script/lint builds Dockerfile.lint, which
// runs golangci-lint as a RUN instruction, so a successful build is a
// clean lint. BuildKit will happily replay that RUN from cache on an
// unchanged tree in well under a second, which is why the check layers
// are keyed on a CHECK_EPOCH build arg that the calling script
// regenerates per invocation, and why an empty value is a hard error
// rather than a stable cache key.
//
// These are parses rather than invocations. Shelling out to docker from
// the test suite would nest a build inside `make test`, which itself
// runs inside a build in CI. The one property a parse cannot establish
// -- that a real finding actually fails the build -- is verified by
// hand against a deliberately broken tree, recorded on the pull
// request.
//
// One property is deliberately NOT tested here: that no script runs the
// linter on the host. script/lint is the only lint entry point, and it
// runs golangci-lint only inside the container; keeping it that way is a
// review matter, not something a test in this file establishes.
// The files under guard, relative to the repository root.
const (
lintDockerfile = "Dockerfile.lint"
productDockerfile = "Dockerfile"
lintScript = "script/lint"
cibuildScript = "script/cibuild"
)
// linterBinary is the linter's command name, used to locate the
// config-verify and lint steps in Dockerfile.lint.
const linterBinary = "golangci-lint"
// checkEpochARG is the declaration, with no default value. A default
// would satisfy the non-empty guard with a constant, and a constant is
// a stable cache key: the checks would be replayed from cache forever
// after the first build.
const checkEpochARG = "ARG CHECK_EPOCH"
// checkEpochGuard is what turns a build that omits --build-arg into a
// loud failure instead of a quiet green. Failed steps are never cached,
// so it fires on every such invocation rather than once.
const checkEpochGuard = `RUN [ -n "$CHECK_EPOCH" ] || exit 1`
// freshEpoch is the epoch computation the calling scripts must use, as
// a bare assignment on its own line. Inline in an argument, a failing
// `date` would not abort under `set -eu`; CHECK_EPOCH would become the
// empty string, and the guard above would be the only thing standing
// between that and a permanently cached green. `$$` is required because
// `date +%s` is second-granular and busybox silently drops `%N`, so
// without the pid two concurrent runs in one second can collide.
const freshEpoch = `epoch="$(date +%s%N)$$"`
// TestLintDockerfilePinsTheLinterByDigest fails if the lint image stops
// being pinned. An unpinned tag makes the gate's verdict depend on
// whatever the registry currently serves under that name.
func TestLintDockerfilePinsTheLinterByDigest(t *testing.T) {
t.Parallel()
from := ""
for _, instruction := range instructions(t, lintDockerfile) {
if strings.HasPrefix(instruction, "FROM ") {
from = instruction
break
}
}
require.NotEmpty(t, from, "%s declares no FROM", lintDockerfile)
assert.Contains(t, from, "golangci/golangci-lint",
"the lint image must be the golangci-lint image")
assert.Contains(t, from, "@sha256:",
"the lint image must be pinned by digest, not by tag alone")
}
// TestLintDockerfileCannotBeCachedGreen pins the whole cache-busting
// mechanism in the file that lints: the declaration with no default,
// the non-empty guard, and the value expanded into the lint command
// itself rather than merely declared.
func TestLintDockerfileCannotBeCachedGreen(t *testing.T) {
t.Parallel()
found := instructions(t, lintDockerfile)
argAt := indexOf(found, checkEpochARG)
require.GreaterOrEqual(t, argAt, 0,
"%s must declare `%s` with no default value",
lintDockerfile, checkEpochARG)
assert.GreaterOrEqual(t, indexOf(found, checkEpochGuard), argAt,
"%s must guard against an empty CHECK_EPOCH with `%s`",
lintDockerfile, checkEpochGuard)
assertEpochExpandedInto(t, found[argAt:], "golangci-lint run")
// Dependency layers must stay above the ARG, or every lint run
// re-downloads the module cache and the inner loop becomes
// unusable.
download := indexOf(found, "RUN go mod download")
require.GreaterOrEqual(t, download, 0,
"%s must download modules in their own layer", lintDockerfile)
assert.Less(t, download, argAt,
"`%s` must come after `go mod download` so dependency layers"+
" still cache", checkEpochARG)
}
// TestLintDockerfileVerifiesTheLinterConfig guards the validation of
// .golangci.yml itself. `golangci-lint run` rejects a config it cannot
// parse but silently IGNORES an unknown top-level key, so renaming
// `linters:` to `linterz:` discards `default: all` and every threshold
// and still exits 0 reporting no issues. `config verify` is what turns
// that into a failure, and it has to run BEFORE the lint, or the lint
// spends a minute reporting a verdict from a config already known to be
// wrong.
func TestLintDockerfileVerifiesTheLinterConfig(t *testing.T) {
t.Parallel()
found := instructions(t, lintDockerfile)
verify := linterBinary + " config verify"
verifyAt := indexContaining(found, verify)
require.GreaterOrEqual(t, verifyAt, 0,
"%s must run `%s --config .golangci.yml`: without it a typo'd"+
" top-level key in .golangci.yml is silently ignored and the"+
" gate passes with only the default linter set", lintDockerfile,
verify)
runAt := indexContaining(found, linterBinary+" run")
require.GreaterOrEqual(t, runAt, 0, "%s must lint", lintDockerfile)
assert.Less(t, verifyAt, runAt,
"%s must verify the config before linting with it", lintDockerfile)
// Keyed on the epoch like every other check layer, so it executes
// per invocation rather than being replayed. A cached validation
// validates nothing.
assertEpochExpandedInto(t, found, verify)
}
// TestProductDockerfileCannotBeCachedGreen holds the same line for the
// checks that remain in the product image build.
func TestProductDockerfileCannotBeCachedGreen(t *testing.T) {
t.Parallel()
found := instructions(t, productDockerfile)
argAt := indexOf(found, checkEpochARG)
require.GreaterOrEqual(t, argAt, 0,
"%s must declare `%s` with no default value",
productDockerfile, checkEpochARG)
assert.GreaterOrEqual(t, indexOf(found, checkEpochGuard), argAt,
"%s must guard against an empty CHECK_EPOCH", productDockerfile)
assertEpochExpandedInto(t, found[argAt:], "make fmt-check")
assertEpochExpandedInto(t, found[argAt:], "make test")
}
// TestProductDockerfileDoesNotLint records the split deliberately: the
// linter lives in Dockerfile.lint and nowhere else, so there is exactly
// one digest pinning it. A lint stage reintroduced here would either be
// docker-in-docker (`make lint` is now `docker build`) or a second,
// independently bumpable pin.
func TestProductDockerfileDoesNotLint(t *testing.T) {
t.Parallel()
contents := readRepoFile(t, productDockerfile)
for _, forbidden := range []string{"golangci", "make lint"} {
assert.NotContains(t, instructionText(contents), forbidden,
"%s must not lint: the linter is pinned once, in %s",
productDockerfile, lintDockerfile)
}
}
// TestLintScriptBuildsTheLintDockerfileWithAFreshEpoch is the other
// half of the mechanism. The Dockerfile's guard only rejects an EMPTY
// epoch; a constant non-empty one would satisfy it and still be served
// from cache forever.
func TestLintScriptBuildsTheLintDockerfileWithAFreshEpoch(t *testing.T) {
t.Parallel()
script := readRepoFile(t, lintScript)
assertBareEpochAssignment(t, script, lintScript)
assert.Contains(t, script, `--build-arg CHECK_EPOCH="$epoch"`,
"%s must pass the fresh epoch to the build", lintScript)
assert.Contains(t, script, lintDockerfile,
"%s must build %s", lintScript, lintDockerfile)
}
// TestCibuildBuildsBothDockerfilesWithFreshEpochs guards the CI gate:
// dropping either build silently removes a whole class of check from
// CI while leaving it green.
func TestCibuildBuildsBothDockerfilesWithFreshEpochs(t *testing.T) {
t.Parallel()
script := readRepoFile(t, cibuildScript)
assertBareEpochAssignment(t, script, cibuildScript)
assert.Equal(t, 2, strings.Count(script, freshEpoch),
"%s must compute a fresh epoch for each of its two builds",
cibuildScript)
assert.Equal(t, 2,
strings.Count(script, `--build-arg CHECK_EPOCH="$epoch"`),
"%s must pass a fresh epoch to both builds", cibuildScript)
assert.Contains(t, script, "-f Dockerfile.lint",
"%s must build %s", cibuildScript, lintDockerfile)
}
// assertEpochExpandedInto fails unless some instruction runs the named
// command with the epoch expanded into it. Expansion, not mere
// declaration: an ARG that no instruction references is not guaranteed
// to key the layer, and the expansion also puts the value in the build
// log where a reader can see the layer was keyed fresh.
func assertEpochExpandedInto(t *testing.T, found []string, command string) {
t.Helper()
for _, instruction := range found {
if !strings.HasPrefix(instruction, "RUN ") {
continue
}
if strings.Contains(instruction, command) &&
strings.Contains(instruction, "${CHECK_EPOCH}") {
return
}
}
assert.Fail(t, "no epoch-keyed layer runs the command",
"`%s` must run in a layer that expands ${CHECK_EPOCH}, or it"+
" will be replayed from cache without executing", command)
}
// assertBareEpochAssignment fails unless the script computes the epoch
// as a bare assignment on its own line.
func assertBareEpochAssignment(t *testing.T, script, name string) {
t.Helper()
for line := range strings.SplitSeq(script, "\n") {
if strings.TrimSpace(line) == freshEpoch {
return
}
}
assert.Fail(t, "no bare epoch assignment",
"%s must compute `%s` as a bare assignment on its own line, so"+
" `set -e` catches a failing date instead of quietly"+
" building with an empty epoch", name, freshEpoch)
}
// instructions returns the Dockerfile's instructions, one per element,
// with comments and blank lines dropped and continuation lines joined,
// so a multi-line RUN is one string.
func instructions(t *testing.T, name string) []string {
t.Helper()
return strings.Split(instructionText(readRepoFile(t, name)), "\n")
}
// instructionText is instructions' parse, before splitting: it is also
// what a "must not contain" assertion should look at, so that a word
// appearing only in a comment is not mistaken for behaviour.
func instructionText(contents string) string {
var (
out []string
continued string
isContinued bool
)
for line := range strings.SplitSeq(contents, "\n") {
trimmed := strings.TrimSpace(line)
if !isContinued && (trimmed == "" || strings.HasPrefix(trimmed, "#")) {
continue
}
isContinued = strings.HasSuffix(trimmed, `\`)
continued += strings.TrimSuffix(trimmed, `\`)
if isContinued {
continue
}
out = append(out, strings.Join(strings.Fields(continued), " "))
continued = ""
}
return strings.Join(out, "\n")
}
// indexOf returns the position of the first instruction equal to, or
// beginning with, want; -1 if there is none. An `ARG NAME=default`
// counts as beginning with `ARG NAME`, so a declared arg is found
// whether or not it carries a default.
func indexOf(found []string, want string) int {
for i, instruction := range found {
if instruction == want ||
strings.HasPrefix(instruction, want+" ") ||
strings.HasPrefix(instruction, want+"=") {
return i
}
}
return -1
}
// indexContaining returns the position of the first instruction
// containing want; -1 if there is none.
func indexContaining(found []string, want string) int {
for i, instruction := range found {
if strings.Contains(instruction, want) {
return i
}
}
return -1
}
// readRepoFile reads a file by its path relative to the repository
// root.
func readRepoFile(t *testing.T, name string) string {
t.Helper()
//nolint:gosec // G304: the path is a constant relative to this repo
contents, err := os.ReadFile(filepath.Join(repoRoot(t), name))
require.NoError(t, err)
return string(contents)
}
// repoRoot returns the repository root. The test binary runs with its
// package directory as the working directory, so the root is found by
// walking up until the module file appears.
func repoRoot(t *testing.T) string {
t.Helper()
dir, err := os.Getwd()
require.NoError(t, err)
for {
_, err = os.Stat(filepath.Join(dir, "go.mod"))
if err == nil {
return dir
}
parent := filepath.Dir(dir)
require.NotEqual(t, dir, parent,
"walked to the filesystem root without finding a go.mod")
dir = parent
}
}
cmd/vaultik/makefile_test.go
+38 -2
View File
@@ -1,6 +1,8 @@
package main_test
import (
"os"
"path/filepath"
"regexp"
"slices"
"strings"
@@ -84,14 +86,48 @@ func TestBuildTargetBuildsTheBinary(t *testing.T) {
"`make build` must depend on the rule that builds the binary")
}
// readMakefile returns the contents of the repository's Makefile. The
// root is located by the shared walk in lintdocker_test.go.
// readMakefile returns the contents of the repository's Makefile.
func readMakefile(t *testing.T) string {
t.Helper()
return readRepoFile(t, "Makefile")
}
// readRepoFile reads a file by its path relative to the repository
// root.
func readRepoFile(t *testing.T, name string) string {
t.Helper()
//nolint:gosec // G304: the path is a constant relative to this repo
contents, err := os.ReadFile(filepath.Join(repoRoot(t), name))
require.NoError(t, err)
return string(contents)
}
// repoRoot returns the repository root. The test binary runs with its
// package directory as the working directory, so the root is found by
// walking up until the module file appears.
func repoRoot(t *testing.T) string {
t.Helper()
dir, err := os.Getwd()
require.NoError(t, err)
for {
_, err = os.Stat(filepath.Join(dir, "go.mod"))
if err == nil {
return dir
}
parent := filepath.Dir(dir)
require.NotEqual(t, dir, parent,
"walked to the filesystem root without finding a go.mod")
dir = parent
}
}
// phonyTargets returns every name declared phony, across all .PHONY
// lines.
func phonyTargets(makefile string) []string {
config.example.yml
+27 -12
View File
@@ -3,7 +3,8 @@
# Copy this file and uncomment/modify the values you need
# Age recipient public keys for encryption
# This is REQUIRED - backups are encrypted to these public keys
# Backups are encrypted to these public keys. snapshot create needs at least
# one; listing, verifying and restoring do not
# Generate with: age-keygen | grep "public key"
age_recipients:
- age1cj2k2addawy294f6k2gr2mf9gps9r3syplryxca3nvxj3daqm96qfp84tz
@@ -257,16 +258,16 @@ exclude:
# Storage URL - use either this OR the s3 section below
# Supports: s3://bucket/prefix, file:///path, rclone://remote/path
storage_url: "rclone://las1stor1//srv/pool.2024.04/backups/heraklion"
storage_url: "rclone://myremote/path/to/backups"
# S3-compatible storage configuration
#s3:
# # S3-compatible endpoint URL
# # Examples: https://s3.amazonaws.com, https://storage.googleapis.com
# endpoint: http://10.100.205.122:8333
# endpoint: https://s3.example.com
#
# # Bucket name where backups will be stored
# bucket: testbucket
# bucket: mybucket
#
# # Prefix (folder) within the bucket for this host's backups
# # Useful for organizing backups from multiple hosts
@@ -274,8 +275,8 @@ storage_url: "rclone://las1stor1//srv/pool.2024.04/backups/heraklion"
# #prefix: "hosts/myserver/"
#
# # S3 access credentials
# access_key_id: Z9GT22M9YFU08WRMC5D4
# secret_access_key: Pi0tPKjFbN4rZlRhcA4zBtEkib04yy2WcIzI+AXk
# access_key_id: YOUR_ACCESS_KEY
# secret_access_key: YOUR_SECRET_KEY
#
# # S3 region
# # Default: us-east-1
@@ -286,15 +287,18 @@ storage_url: "rclone://las1stor1//srv/pool.2024.04/backups/heraklion"
# #use_ssl: true
#
# # Part size for multipart uploads
# # Minimum 5MB, affects memory usage during upload
# # Supports: 5MB, 10M, 100MiB, etc.
# # Default: 5MB
# #part_size: 5MB
# # Minimum 5MiB, maximum 5GiB; affects memory usage during upload
# # A blob too large for 10,000 parts of this size gets larger parts
# # Supports: 10MB, 16MiB, 100MiB, etc. (5MB is below the minimum)
# # Default: 5MiB
# #part_size: 5MiB
# Path to local SQLite index database
# This database tracks file state for incremental backups
# Default: /var/lib/vaultik/index.sqlite
#index_path: /var/lib/vaultik/index.sqlite
# Default: the platform data directory, e.g.
# macOS: ~/Library/Application Support/vaultik/index.sqlite
# Linux: ~/.local/share/vaultik/index.sqlite
#index_path: /path/to/index.sqlite
# Average chunk size for content-defined chunking
# Smaller chunks = better deduplication but more metadata
@@ -304,6 +308,17 @@ storage_url: "rclone://las1stor1//srv/pool.2024.04/backups/heraklion"
# Maximum blob size
# Multiple chunks are packed into blobs up to this size
# Must be at least four times chunk_size (the largest chunk the chunker can
# emit); a smaller limit would let a single-chunk blob exceed it.
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a
# large limit a blob holds hundreds of chunks, so individual chunk lengths are
# not visible in its size; lowering the limit toward chunk_size exposes them.
# A backup needs free temporary space, because each blob is written whole to
# a temporary file before it is uploaded (up to about blob_size_limit; an
# rclone destination that cannot stream uploads needs about twice that) and
# the metadata export writes copies of the local index. Temporary files go to
# $TMPDIR (default /tmp); with TMPDIR unset, SQLite writes one of those copies
# to /var/tmp.
# Supports: 1GB, 10G, 500MB, 1GiB, etc.
# Default: 10GB
#blob_size_limit: 10GB
docs/DATAMODEL.md
+32 -36
View File
@@ -6,33 +6,28 @@ Vaultik uses a local SQLite database to track file metadata, chunk mappings, and
**Important Notes:**
This section is the authoritative explanation of the schema/migration story;
other documents (the README and `AGENTS.md`) link here.
- **No upgrade path between versions (pre-1.0)**: Vaultik has no supported way to
carry an existing local index across a schema change. The index is disposable
— if the on-disk schema changes between versions, delete the local SQLite
database (`vaultik database delete`) and run a full backup. Remote storage is
unaffected; the new index re-deduplicates against existing remote blobs. This
is the standing project policy, and it is separate from the schema bootstrap
described next.
- **Schema bootstrap**: a fresh database is populated from numbered SQL files
embedded in the binary under `internal/database/schema/`. `000.sql` creates the
`schema_migrations` table; `001.sql` creates the application tables. On opening
a database the code applies each numbered file that has not yet run and records
its version in `schema_migrations`. This bootstraps a new database; it does not
upgrade an existing one between released versions.
- **Changing the schema (pre-1.0)**: edit `internal/database/schema/001.sql` (and
the code that touches the affected tables) directly. Do not add new numbered
files — there is no installed base to migrate.
- **Disposability expires at 1.0**: the index is treated as disposable only until
1.0 ships and is tagged. Once 1.0 is tagged that clause expires and the
question of upgrading existing indexes returns. It is deliberately left open
here.
- **Version Compatibility**: In rare cases, you may need to use the same version
of Vaultik to restore a backup as was used to create it. This ensures
compatibility with the metadata format stored in S3.
## Schema Migrations
Vaultik supports schema migrations. They are the numbered SQL files in
`internal/database/schema/`, embedded in the binary: `000.sql` creates the
`schema_migrations` table, which records each migration that has run, and
`001.sql` creates the application tables. `database.New` opens a database and
applies, in order, every migration that database has not yet recorded.
**Before 1.0** no migrations are added, because nothing is installed anywhere
yet. A schema change edits `001.sql` (and the code that uses the affected
tables) directly. A local database created before the change has already
recorded `001.sql` as run, so it keeps the old schema and can become unusable;
`vaultik database delete` followed by a full backup rebuilds it.
**After 1.0** each schema change is a new numbered file, so an existing local
database is migrated the first time an updated vaultik opens it. A file that
has shipped in a release is never edited.
## Database Tables
### 1. `files`
@@ -41,7 +36,8 @@ Stores metadata about files in the filesystem being backed up.
**Columns:**
- `id` (TEXT PRIMARY KEY) - UUID for the file record
- `path` (TEXT NOT NULL UNIQUE) - Absolute file path
- `mtime` (INTEGER NOT NULL) - Modification time as Unix timestamp
- `mtime` (INTEGER NOT NULL) - Modification time, whole seconds since the Unix epoch
- `mtime_nsec` (INTEGER NOT NULL) - Nanoseconds within that second, 0 to 999999999
- `size` (INTEGER NOT NULL) - File size in bytes
- `mode` (INTEGER NOT NULL) - Unix file permissions and type
- `uid` (INTEGER NOT NULL) - User ID of file owner
@@ -90,7 +86,7 @@ Stores information about packed, compressed, and encrypted blob files.
**Columns:**
- `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts
- `blob_hash` (TEXT UNIQUE) - SHA256 hash of final blob (NULL until finalized)
- `blob_hash` (TEXT UNIQUE) - `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (NULL until finalized); see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs)
- `created_ts` (INTEGER NOT NULL) - Creation timestamp
- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression
@@ -115,17 +111,17 @@ Maps chunks to the blobs that contain them.
Tracks backup snapshots.
**Columns:**
- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: hostname-YYYYMMDD-HHMMSSZ)
- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: `hostname_name_timestamp`, e.g. `server1_home_2025-06-01T12:00:00Z`: the hostname up to its first `.`, the snapshot name, and an RFC 3339 UTC timestamp)
- `hostname` (TEXT) - Hostname where backup was created
- `vaultik_version` (TEXT) - Version of Vaultik used
- `vaultik_git_revision` (TEXT) - Git revision of Vaultik used
- `started_at` (INTEGER) - Start timestamp
- `completed_at` (INTEGER) - Completion timestamp (NULL if in progress)
- `file_count` (INTEGER) - Number of files in snapshot
- `chunk_count` (INTEGER) - Number of unique chunks
- `blob_count` (INTEGER) - Number of blobs referenced
- `chunk_count` (INTEGER) - Number of chunks this snapshot stored that were not stored before
- `blob_count` (INTEGER) - Number of blobs this snapshot created
- `total_size` (INTEGER) - Total size of all files
- `blob_size` (INTEGER) - Total size of all blobs (compressed)
- `blob_size` (INTEGER) - Total compressed size of all referenced blobs
- `blob_uncompressed_size` (INTEGER) - Total uncompressed size of all referenced blobs
- `compression_ratio` (REAL) - Compression ratio achieved
- `compression_level` (INTEGER) - Compression level used for this snapshot
@@ -200,6 +196,7 @@ Tracks blob upload metrics.
1. **Change Detection**
- `SELECT * FROM files WHERE path = ?` - Get previous file metadata
- Compare mtime, size, mode to detect changes
- Re-chunk a file that lists a chunk no uploaded blob holds, even when its metadata is unchanged
- Skip unchanged files but still add to `snapshot_files`
2. **Chunk Reuse**
@@ -216,13 +213,13 @@ After a snapshot is completed:
5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst`
The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation).
The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext. The hash uses no secret, so a guessed hostname and snapshot name can still be confirmed against a listing; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) and its [Accepted Risks](REPOSTRUCTURE.md#accepted-risks).
### 4. Restore Process
The restore process doesn't use the local database. Instead:
1. Downloads snapshot metadata from S3
2. Downloads required blobs based on manifest
1. Downloads and decrypts the snapshot's metadata database (`db.zst.age`) from S3
2. Downloads the blobs holding the chunks of the files being restored, found through that database's `blob_chunks` table; the manifest is not read
3. Reconstructs files from decrypted and decompressed chunks
### 5. Pruning
@@ -235,9 +232,8 @@ The restore process doesn't use the local database. Instead:
Before each backup:
1. Query incomplete snapshots (where `completed_at IS NULL`)
2. Check if metadata exists in S3
3. If no metadata, delete snapshot and all associations
4. Clean up orphaned files, chunks, and blobs
2. Delete each one and all its associations, without checking S3 for its metadata
3. Clean up orphaned files, chunks, and blobs
## Repository Pattern
@@ -284,7 +280,7 @@ This ensures consistency, especially important for operations like:
3. **Batch Operations**: Where possible, operations are batched within transactions
4. **Write-Ahead Logging**: SQLite WAL mode is enabled for better concurrency
4. **Write-Ahead Logging**: The local index runs in SQLite WAL mode with a 10-second busy timeout, so a read-only command such as `snapshot list` can read it while a backup writes to it. Committed rows can sit in the `-wal` file beside the index until a checkpoint, so the metadata export copies the index through SQLite (`VACUUM INTO`), not as a file
## Data Integrity
docs/REPOSTRUCTURE.md
+21 -8
View File
@@ -35,7 +35,7 @@ The metadata subdirectory is named with the **remote key**, a one-way hash of th
- **What it contains**: Packed collections of content-defined chunks from files
- **Format**: Zstandard compressed, then Age encrypted
- **Encryption**: Always encrypted with Age using the configured recipients
- **Naming**: Content-addressed using SHA256 hash of the encrypted blob
- **Naming**: Content-addressed. The blob's name is `hex(SHA256(SHA256(uncompressed blob contents)))` — the double SHA-256 of the concatenated chunk data, computed before compression and encryption, not a hash of the stored (compressed, encrypted) bytes. One consequence: only a holder of the age private key can check a stored blob's integrity, because matching a blob to its name means decrypting and decompressing it first — which is what `restore` and `verify --deep` do. Implemented in `internal/blobgen` (`DoubleSHA256`). This is the canonical description of blob naming; other documents and comments point here.
### Why Encrypted
Blobs contain the actual file data from backups and must be encrypted for security. The content-addressing ensures deduplication while the encryption ensures privacy.
@@ -59,14 +59,14 @@ This ID reveals the hostname, the configured snapshot name, and the backup time,
### Remote Key Derivation
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string with no structure a remote observer can reverse. Implemented in `internal/snapshot/remotekey.go`.
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string. The hash is not reversible, but it uses no secret: an observer who guesses a candidate hostname and snapshot name can hash it the same way and confirm whether that snapshot is present. The remote key keeps names out of a plain listing; it does not hide them from a guess. Implemented in `internal/snapshot/remotekey.go`.
Worked example:
- Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
- Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
Because the hash is one-way, a listing of the destination store reveals neither the hostname nor the snapshot name of any backup. The same remote key is stored in the manifest's `snapshot_id` field.
A plain listing of the destination store therefore shows only these hashes, not the hostname or snapshot name of any backup — but because the hash uses no secret, a guessed hostname and snapshot name can be hashed and confirmed against the listing. The same remote key is stored in the manifest's `snapshot_id` field.
### Files in Each Snapshot Directory
@@ -124,31 +124,44 @@ From the unencrypted data, an observer of the destination store can determine:
- **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear
- How many blobs each snapshot references, and the total compressed size
- The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns)
- **Whether a guessed hostname and snapshot name are present** — the remote key is an unkeyed hash, so an observer holding candidate names can hash each one and match it against the directory listing. The human ID is never published, so it cannot be read off directly, but it can be confirmed by guessing.
Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless.
An observer cannot determine:
- The hostname or snapshot name of any backup (the directory name and the manifest `snapshot_id` are one-way hashes of the human ID)
- The hostname or snapshot name of any backup by reading it off the store — the directory name and the manifest `snapshot_id` are unkeyed hashes of the human ID, so the text is never published (though a guessed name can be confirmed, as above)
- File names or paths
- File contents
- File permissions or ownership
- Directory structure
- Which chunks belong to which files
### Accepted Risks
These are known, deliberate properties of the format and the tooling, recorded so an operator can weigh them rather than discover them.
1. **No proof of authorship.** Restore and `verify --deep` prove that data decrypts with the age private key and matches its unkeyed content hashes. They do not prove who wrote it: anyone who knows a recipient public key and can replace objects on the destination can substitute a snapshot they built. The recipient string is not stored at the destination, but a compromised backed-up host has it. Defences live on the destination side — bucket versioning or object lock, credentials for the source host that cannot delete or overwrite existing versions, and pruning from a trusted host. Note that S3 `PutObject` overwrites an existing key, so PUT permission alone is not append-only.
2. **Compression reveals sizes.** Blobs and `db.zst.age` are zstd-compressed then age-encrypted; the manifest is compressed only. age does not pad, so an object's size is the exact compressed length of its contents. All new chunks packed into one blob share a single zstd stream (8 MiB window, 4 MiB at compression levels 1-2), and a blob is closed at `blob_size_limit` and at the end of each configured path. Because a stored chunk is never packed again, someone who can write into a backed-up file and watch blob sizes learns something only when their controlled data and a secret land in the same chunk of a file that keeps changing. Advice: back up any outsider-writable directory as its own snapshot.
3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of 2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a blob holds hundreds of chunks, so individual chunk lengths are not visible in the blob's size; lowering the limit toward the chunk size begins to expose them.
4. **Decrypted data on local disk.** Several commands stage plaintext under `$TMPDIR`: `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600` and removed on success, but a `kill -9` or a power loss leaves them behind — delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR` should be trusted to the same degree as the restore target.
5. **Store permissions differ per command.** The backed-up host needs only PUT to run `snapshot create`: it writes blobs and metadata and neither reads nor deletes them. Other commands need more — `snapshot verify`, `snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`, `snapshot remove`, and `remote nuke` also delete. The recommended cron line uses `--prune`, which runs `prune` on the backed-up host, so granting that host `--prune` gives it credentials that can delete its own backups. To keep the source host to PUT only, prune from a separate trusted host instead.
6. **Changing recipients does not re-encrypt existing data.** Deduplicated chunks and same-named blobs already on the destination stay encrypted to the recipients in force when they were written. A new snapshot that reuses them cannot be restored with a newly added recipient's key alone, because those reused objects were never encrypted to it. To make everything readable by a new key, run `vaultik database delete` and take a full backup to a fresh destination or prefix.
7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients. Long-lived ciphertext held by a third party (the destination operator) has no fallback if X25519 is ever broken: there is no second recipient type and no post-quantum option.
## Consistency Guarantees
1. **Blobs are immutable** - Once written, a blob is never modified
2. **Blobs are written before metadata** - A snapshot's metadata is only written after all its blobs are successfully uploaded
3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst are written as complete files
4. **Snapshots are marked complete in local DB only after metadata upload** - Ensures consistency between local and remote state
4. **A snapshot is marked complete in the local DB only after its metadata is uploaded** - `finalizeSnapshotMetadata` runs `ExportSnapshotMetadata` first and records completion (`MarkSnapshotComplete`) only once the export succeeds (see the backup data flow in [ARCHITECTURE.md](../ARCHITECTURE.md)). A crash during the export therefore leaves the snapshot incomplete, so the next backup's `PruneDatabase` drops it and re-backs-up its data, rather than leaving a completed-looking row in the local index with no matching metadata on the destination store. (A crash in the brief moment after the export succeeds but before completion is recorded leaves a fully-restorable snapshot on the destination that the local index drops as incomplete on the next run; `snapshot list` then reports it honestly as remote-only, which is the safe direction: the destination copy stays restorable.)
## Pruning Safety
The prune operation is safe because:
1. It only deletes blobs not referenced in any manifest
1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references
2. Manifests are unencrypted and can be read without keys
3. The operation compares the latest local DB snapshot with the latest S3 snapshot to ensure consistency
4. Pruning will fail if these don't match, preventing accidental deletion of needed blobs
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and exits with an error, rather than treating that snapshot's blobs as unreferenced
4. Prune requires exclusive access to the destination: running it during a concurrent backup can race a snapshot whose manifest is not yet written, so do not prune while a backup is in progress
## Restoration Requirements
go.mod
+2 -2
View File
@@ -20,9 +20,11 @@ require (
github.com/rclone/rclone v1.72.1
github.com/spf13/afero v1.15.0
github.com/spf13/cobra v1.10.1
github.com/spf13/pflag v1.0.10
github.com/stretchr/testify v1.11.1
go.uber.org/fx v1.24.0
golang.org/x/sync v0.18.0
golang.org/x/sys v0.38.0
golang.org/x/term v0.37.0
gopkg.in/yaml.v3 v3.0.1
modernc.org/sqlite v1.38.0
@@ -225,7 +227,6 @@ require (
github.com/smarty/assertions v1.16.0 // indirect
github.com/sony/gobreaker v1.0.0 // indirect
github.com/spacemonkeygo/monkit/v3 v3.0.25-0.20251022131615-eb24eb109368 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/t3rm1n4l/go-mega v0.0.0-20251031123324-a804aaa87491 // indirect
github.com/tidwall/gjson v1.18.0 // indirect
github.com/tidwall/match v1.1.1 // indirect
@@ -263,7 +264,6 @@ require (
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 // indirect
golang.org/x/net v0.47.0 // indirect
golang.org/x/oauth2 v0.33.0 // indirect
golang.org/x/sys v0.38.0 // indirect
golang.org/x/text v0.31.0 // indirect
golang.org/x/time v0.14.0 // indirect
golang.org/x/tools v0.38.0 // indirect
internal/blob/packer.go
+10 -8
View File
@@ -1,14 +1,16 @@
// Package blob handles the creation of blobs - the final storage units for Vaultik.
// A blob is a large file (up to 10GB) containing many compressed and encrypted chunks
// from multiple source files. Blobs are content-addressed, meaning their filename
// is derived from the SHA256 hash of their compressed and encrypted content.
// from multiple source files. Blobs are content-addressed: a blob's filename is
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the concatenated
// chunk data before compression and encryption, not from the stored bytes. See
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
//
// The blob creation process:
// 1. Chunks are accumulated from multiple files
// 2. The collection is compressed using zstd
// 3. The compressed data is encrypted using age
// 4. The encrypted blob is hashed to create its content-addressed name
// 5. The blob is uploaded to S3 using the hash as the filename
// 1. Chunks are accumulated from multiple files
// 2. Each chunk's uncompressed bytes are fed to a running SHA-256 and, in the same
// pass, compressed with zstd and encrypted with age into the temp file
// 3. On finalize, the name is the double SHA-256 of that uncompressed content
// 4. The blob is uploaded to S3 using the name as the filename
//
// This design optimizes storage efficiency by batching many small chunks into
// larger blobs, reducing the number of S3 operations and associated costs.
@@ -487,7 +489,7 @@ func (p *Packer) closeBlobWriter() (string, int64, error) {
return "", 0, fmt.Errorf("seeking to start: %w", err)
}
finalHash := p.currentBlob.writer.Sum256()
finalHash := p.currentBlob.writer.ContentID()
return hex.EncodeToString(finalHash), finalSize, nil
}
internal/blobgen/compress.go
-89
View File
@@ -1,89 +0,0 @@
// Package blobgen implements the blob data pipeline: streaming zstd
// compression, age encryption, and SHA256 content hashing for blob
// creation, plus the matching decrypt/decompress/verify reader.
package blobgen
import (
"bytes"
"encoding/hex"
"fmt"
"io"
)
// CompressResult contains the results of compression
type CompressResult struct {
Data []byte
UncompressedSize int64
CompressedSize int64
SHA256 string
}
// CompressData compresses and encrypts data, returning the result with hash
func CompressData(
data []byte, compressionLevel int, recipients []string,
) (*CompressResult, error) {
var buf bytes.Buffer
// Create writer
w, err := NewWriter(&buf, compressionLevel, recipients)
if err != nil {
return nil, fmt.Errorf("creating writer: %w", err)
}
// Write data
_, err = w.Write(data)
if err != nil {
_ = w.Close()
return nil, fmt.Errorf("writing data: %w", err)
}
// Close to flush
err = w.Close()
if err != nil {
return nil, fmt.Errorf("closing writer: %w", err)
}
return &CompressResult{
Data: buf.Bytes(),
UncompressedSize: int64(len(data)),
CompressedSize: int64(buf.Len()),
SHA256: hex.EncodeToString(w.Sum256()),
}, nil
}
// CompressStream compresses and encrypts from reader to writer, returning
// the number of uncompressed bytes written and the content hash.
func CompressStream(
dst io.Writer, src io.Reader, compressionLevel int, recipients []string,
) (int64, string, error) {
// Create writer
w, err := NewWriter(dst, compressionLevel, recipients)
if err != nil {
return 0, "", fmt.Errorf("creating writer: %w", err)
}
closed := false
defer func() {
if !closed {
_ = w.Close()
}
}()
// Copy data
_, err = io.Copy(w, src)
if err != nil {
return 0, "", fmt.Errorf("copying data: %w", err)
}
// Close to flush
err = w.Close()
if err != nil {
return 0, "", fmt.Errorf("closing writer: %w", err)
}
closed = true
return w.BytesWritten(), hex.EncodeToString(w.Sum256()), nil
}
internal/blobgen/compress_test.go
-80
View File
@@ -1,80 +0,0 @@
package blobgen_test
import (
"bytes"
"crypto/rand"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// testRecipient is a static age recipient for tests.
const testRecipient = "age1cplgrwj77ta54dnmydvvmzn64ltk83ankxl5sww04mrtmu62kv3s89gmvv"
// TestCompressStreamNoDoubleClose is a regression test for issue #28.
// It verifies that CompressStream does not panic or return an error due to
// double-closing the underlying blobgen.Writer. Before the fix in PR #33,
// the explicit Close() on the happy path combined with defer Close() would
// cause a double close.
func TestCompressStreamNoDoubleClose(t *testing.T) {
t.Parallel()
input := []byte("regression test data for issue #28 double-close fix")
var buf bytes.Buffer
written, hash, err := blobgen.CompressStream(
&buf, bytes.NewReader(input), 3, []string{testRecipient})
require.NoError(t, err, "CompressStream should not return an error")
assert.Positive(t, written, "expected bytes written > 0")
assert.NotEmpty(t, hash, "expected non-empty hash")
assert.Positive(t, buf.Len(), "expected non-empty output")
}
// TestCompressStreamLargeInput exercises CompressStream with a larger payload
// to ensure no double-close issues surface under heavier I/O.
func TestCompressStreamLargeInput(t *testing.T) {
t.Parallel()
data := make([]byte, 512*1024) // 512 KB
_, err := rand.Read(data)
require.NoError(t, err)
var buf bytes.Buffer
written, hash, err := blobgen.CompressStream(
&buf, bytes.NewReader(data), 3, []string{testRecipient})
require.NoError(t, err)
assert.Positive(t, written)
assert.NotEmpty(t, hash)
}
// TestCompressStreamEmptyInput verifies CompressStream handles empty input
// without double-close issues.
func TestCompressStreamEmptyInput(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
_, hash, err := blobgen.CompressStream(
&buf, strings.NewReader(""), 3, []string{testRecipient})
require.NoError(t, err)
assert.NotEmpty(t, hash)
}
// TestCompressDataNoDoubleClose mirrors the stream test for CompressData,
// ensuring the explicit Close + error-path Close pattern is also safe.
func TestCompressDataNoDoubleClose(t *testing.T) {
t.Parallel()
input := []byte("CompressData regression test for double-close")
result, err := blobgen.CompressData(input, 3, []string{testRecipient})
require.NoError(t, err)
assert.Positive(t, result.CompressedSize)
assert.Equal(t, result.UncompressedSize, int64(len(input)))
assert.NotEmpty(t, result.SHA256)
}
internal/blobgen/helpers_test.go
+119
View File
@@ -0,0 +1,119 @@
package blobgen_test
import (
"bytes"
"crypto/rand"
"errors"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// ageChunkSize is age's STREAM plaintext chunk size (64 KiB); each encrypted
// chunk adds a 16-byte ChaCha20-Poly1305 tag.
const (
ageChunkSize = 64 * 1024
ageChunkTagSize = 16
ageSegmentSize = ageChunkSize + ageChunkTagSize
ageNonceSize = 16
)
// makeIdentity returns a fresh X25519 identity and its recipient string.
func makeIdentity(t *testing.T) (*age.X25519Identity, string) {
t.Helper()
id, err := age.GenerateX25519Identity()
require.NoError(t, err)
return id, id.Recipient().String()
}
// randomBytes returns n cryptographically random bytes, which do not compress
// so the encrypted payload spans multiple age segments.
func randomBytes(t *testing.T, n int) []byte {
t.Helper()
b := make([]byte, n)
_, err := rand.Read(b)
require.NoError(t, err)
return b
}
// compressibleBytes returns n bytes of a repeating pattern, which zstd packs
// down to a small payload.
func compressibleBytes(n int) []byte {
pattern := bytes.Repeat([]byte("compressible-"), n/13+1)
return pattern[:n]
}
// encryptBlob compresses, encrypts and returns a blob for plaintext at
// compression level 1.
func encryptBlob(t *testing.T, plaintext []byte, recipients ...string) []byte {
t.Helper()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, 1, recipients)
require.NoError(t, err)
_, err = w.Write(plaintext)
require.NoError(t, err)
require.NoError(t, w.Close())
return buf.Bytes()
}
// ageHeaderLen returns the byte length of blob's age header, i.e. the offset
// of the 16-byte payload nonce that follows it. The header ends with a MAC
// line "--- <mac>\n"; the nonce begins right after that newline.
func ageHeaderLen(t *testing.T, blob []byte) int {
t.Helper()
i := bytes.Index(blob, []byte("\n--- "))
require.GreaterOrEqual(t, i, 0, "age MAC footer line not found")
nl := bytes.IndexByte(blob[i+1:], '\n')
require.GreaterOrEqual(t, nl, 0, "newline ending MAC line not found")
return i + 1 + nl + 1
}
// requireBlobUnreadable asserts that data never decrypts to a plaintext with a
// nil error: either NewReader fails, or reading it does.
func requireBlobUnreadable(t *testing.T, data []byte, id age.Identity) {
t.Helper()
r, err := blobgen.NewReader(bytes.NewReader(data), id)
if err != nil {
return
}
_, err = io.ReadAll(r)
_ = r.Close()
require.Error(t, err, "reading a damaged blob must fail")
}
// errFailWriter is returned by failAfterWriter once its byte limit is passed.
var errFailWriter = errors.New("destination write failed")
// failAfterWriter accepts writes until more than limit bytes have been sent,
// then fails every write. It models a destination that dies mid-blob.
type failAfterWriter struct {
limit int
written int
}
func (f *failAfterWriter) Write(p []byte) (int, error) {
f.written += len(p)
if f.written > f.limit {
return 0, errFailWriter
}
return len(p), nil
}
internal/blobgen/limit.go
+49
View File
@@ -0,0 +1,49 @@
package blobgen
import (
"errors"
"io"
)
// ErrOutputTooLarge is returned by a reader from LimitReader once it has
// been asked for more than its limit. It bounds how far an untrusted
// compressed stream may expand, so a small, highly compressible object
// from the store cannot decompress without limit.
var ErrOutputTooLarge = errors.New("output exceeds size limit")
// LimitReader returns a reader that yields at most limit bytes from r and
// then fails with ErrOutputTooLarge. Unlike io.LimitReader, which reports
// a silent io.EOF at the limit (indistinguishable from a stream that
// simply ended), this fails, so a caller decoding or copying the stream
// sees an error rather than a truncated value. A stream of exactly limit
// bytes reads back cleanly to EOF; the first byte beyond it is the error.
func LimitReader(r io.Reader, limit int64) io.Reader {
// remaining counts down from limit+1: the extra byte is the one that,
// if it ever arrives, proves the stream is longer than the limit.
return &limitReader{r: r, remaining: limit + 1}
}
type limitReader struct {
r io.Reader
remaining int64
}
func (l *limitReader) Read(p []byte) (int, error) {
if l.remaining <= 0 {
return 0, ErrOutputTooLarge
}
if int64(len(p)) > l.remaining {
p = p[:l.remaining]
}
n, err := l.r.Read(p)
l.remaining -= int64(n)
if l.remaining <= 0 {
// The (limit+1)th byte was just read: the stream is too long.
return n, ErrOutputTooLarge
}
return n, err
}
internal/blobgen/limit_test.go
+43
View File
@@ -0,0 +1,43 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestLimitReaderPassesExactSize checks that a stream of exactly the limit
// reads back cleanly to EOF: the bound must not reject a legitimate blob
// whose plaintext equals its recorded size.
func TestLimitReaderPassesExactSize(t *testing.T) {
t.Parallel()
const n = 1000
r := blobgen.LimitReader(bytes.NewReader(bytes.Repeat([]byte("a"), n)), n)
got, err := io.ReadAll(r)
require.NoError(t, err)
require.Len(t, got, n)
}
// TestLimitReaderFailsPastLimit feeds a large, highly compressible run of
// zeros — the decompressed output a zip bomb would produce — through a
// small limit and checks it fails within the bound rather than passing
// the whole stream through.
func TestLimitReaderFailsPastLimit(t *testing.T) {
t.Parallel()
const limit = 1000
r := blobgen.LimitReader(
bytes.NewReader(bytes.Repeat([]byte{0}, limit*1000)), limit)
n, err := io.Copy(io.Discard, r)
require.ErrorIs(t, err, blobgen.ErrOutputTooLarge)
require.LessOrEqual(t, n, int64(limit)+1,
"reader must stop within one byte of the limit")
}
internal/blobgen/negative_test.go
+190
View File
@@ -0,0 +1,190 @@
package blobgen_test
import (
"bytes"
"fmt"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestNewReaderWrongIdentity covers issue case 4: opening a blob with an
// identity other than the recipient reports no matching identity.
func TestNewReaderWrongIdentity(t *testing.T) {
t.Parallel()
_, recipient := makeIdentity(t)
other, _ := makeIdentity(t)
blob := encryptBlob(t, []byte("secret payload"), recipient)
_, err := blobgen.NewReader(bytes.NewReader(blob), other)
require.Error(t, err)
var noMatch *age.NoIdentityMatchError
assert.ErrorAs(t, err, &noMatch)
}
// TestNewReaderTruncated covers issue case 6: a multi-segment blob cut at
// several points must never read back as valid data. The point immediately
// after the header and nonce is intentionally excluded: it reads as a valid
// empty blob today and is the regression case for
// https://git.eeqj.de/sneak/vaultik/issues/152.
func TestNewReaderTruncated(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
blob := encryptBlob(t, randomBytes(t, 4*65536+123), recipient)
h := ageHeaderLen(t, blob)
require.Greater(t, len(blob), h+ageNonceSize+ageSegmentSize,
"test needs a blob of at least two age segments")
cases := []struct {
name string
size int
}{
{"inside header", h / 2},
{"inside nonce", h + 8},
{"inside first segment", h + ageNonceSize + 100},
{"end of first full segment", h + ageNonceSize + ageSegmentSize},
{"last byte removed", len(blob) - 1},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, blob[:tc.size], id)
})
}
}
// TestNewReaderCorrupted covers issue case 7: one flipped byte in each region
// of a multi-segment blob makes it unreadable.
func TestNewReaderCorrupted(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
blob := encryptBlob(t, randomBytes(t, 4*65536+123), recipient)
h := ageHeaderLen(t, blob)
firstNL := bytes.IndexByte(blob, '\n')
require.Positive(t, firstNL, "header must have a version line")
cases := []struct {
name string
pos int
}{
{"header stanza", firstNL + 5},
{"header MAC line", h - 2},
{"nonce", h + 4},
{"body segment", h + ageNonceSize + 50},
{"final tag", len(blob) - 1},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
corrupt := append([]byte(nil), blob...)
corrupt[tc.pos] ^= 0xff
requireBlobUnreadable(t, corrupt, id)
})
}
}
// TestNewReaderTrailingAndGarbage covers issue case 8: bytes appended after a
// valid blob, empty input, and random garbage each fail to read.
func TestNewReaderTrailingAndGarbage(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
valid := encryptBlob(t, []byte("small payload"), recipient)
appended := append(append([]byte(nil), valid...), []byte("trailing junk")...)
t.Run("appended bytes", func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, appended, id)
})
t.Run("empty input", func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, []byte{}, id)
})
t.Run("random garbage", func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, randomBytes(t, 512), id)
})
}
// TestNewWriterInvalidLevel covers the rejected end of issue case 9: an
// out-of-range compression level errors and writes nothing to the destination.
func TestNewWriterInvalidLevel(t *testing.T) {
t.Parallel()
_, recipient := makeIdentity(t)
for _, level := range []int{0, -1, 20} {
t.Run(fmt.Sprintf("level%d", level), func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, level, []string{recipient})
require.ErrorIs(t, err, blobgen.ErrInvalidCompressionLevel)
assert.Nil(t, w)
assert.Zero(t, buf.Len(), "nothing written on an invalid level")
})
}
}
// TestNewWriterInvalidRecipients covers issue case 10: nil and empty recipient
// lists and an unparsable recipient string each error.
func TestNewWriterInvalidRecipients(t *testing.T) {
t.Parallel()
cases := []struct {
name string
recipients []string
}{
{"nil list", nil},
{"empty list", []string{}},
{"invalid recipient string", []string{"not-a-recipient"}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, 1, tc.recipients)
require.Error(t, err)
assert.Nil(t, w)
})
}
}
// TestNewWriterFailingDestination covers issue case 11: a destination that
// fails mid-blob surfaces its error from Write or Close.
func TestNewWriterFailingDestination(t *testing.T) {
t.Parallel()
_, recipient := makeIdentity(t)
// The limit clears the age header and nonce so NewWriter succeeds, then
// trips once the compressed body starts flowing.
dst := &failAfterWriter{limit: 512}
w, err := blobgen.NewWriter(dst, 1, []string{recipient})
require.NoError(t, err)
_, writeErr := w.Write(randomBytes(t, 256*1024))
closeErr := w.Close()
assert.True(t, writeErr != nil || closeErr != nil,
"destination failure must surface from Write or Close")
}
internal/blobgen/reader.go
+25 -4
View File
@@ -2,6 +2,7 @@ package blobgen
import (
"crypto/sha256"
"errors"
"fmt"
"hash"
"io"
@@ -20,10 +21,12 @@ type Reader struct {
bytesRead int64
}
// NewReader creates a new Reader that decrypts, decompresses, and verifies data
func NewReader(r io.Reader, identity age.Identity) (*Reader, error) {
// NewReader creates a new Reader that decrypts, decompresses, and verifies
// data. Every supplied identity is offered to age.Decrypt, so a blob
// encrypted to any one of them can be read.
func NewReader(r io.Reader, identities ...age.Identity) (*Reader, error) {
// Create decryption reader
decReader, err := age.Decrypt(r, identity)
decReader, err := age.Decrypt(r, identities...)
if err != nil {
return nil, fmt.Errorf("creating decryption reader: %w", err)
}
@@ -54,6 +57,22 @@ func (r *Reader) Read(p []byte) (int, error) {
n, err := r.teeReader.Read(p)
r.bytesRead += int64(n)
// When the ciphertext is cut right after the age header plus its
// 16-byte nonce, the age reader's first read fails with
// io.ErrUnexpectedEOF, and the zstd decoder maps that to a clean
// io.EOF at frame start. That makes a truncated stream look like a
// valid empty one. Distinguish the two: on EOF, read once more from
// the age reader. A genuine end leaves it at (0, io.EOF); a truncated
// stream leaves its stored io.ErrUnexpectedEOF, which we surface.
if errors.Is(err, io.EOF) {
var probe [1]byte
m, ageErr := r.decryptor.Read(probe[:])
if m != 0 || !errors.Is(ageErr, io.EOF) {
return n, io.ErrUnexpectedEOF
}
}
return n, err
}
@@ -64,7 +83,9 @@ func (r *Reader) Close() error {
return nil
}
// Sum256 returns the SHA256 hash of all data read
// Sum256 returns the single SHA-256 of the plaintext read so far. This is the
// first hash only; the stored object name is its double hash, which callers
// obtain by passing this digest to DoubleSHA256.
func (r *Reader) Sum256() []byte {
return r.hasher.Sum(nil)
}
internal/blobgen/reader_test.go
+54
View File
@@ -0,0 +1,54 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestMultipleRecipients verifies that data written for several recipients can
// be read back by each recipient's identity. Moved from internal/crypto, which
// held the only multi-recipient test; blobgen is now the sole encryption path.
func TestMultipleRecipients(t *testing.T) {
t.Parallel()
identities := make([]*age.X25519Identity, 3)
recipients := make([]string, 3)
for i := range identities {
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
identities[i] = identity
recipients[i] = identity.Recipient().String()
}
plaintext := []byte("Secret message for multiple recipients")
var encrypted bytes.Buffer
writer, err := blobgen.NewWriter(&encrypted, 3, recipients)
require.NoError(t, err)
_, err = writer.Write(plaintext)
require.NoError(t, err)
require.NoError(t, writer.Close())
// Every recipient's identity must recover the original plaintext.
for i, identity := range identities {
reader, err := blobgen.NewReader(
bytes.NewReader(encrypted.Bytes()), identity)
require.NoError(t, err, "recipient %d should open the reader", i+1)
got, err := io.ReadAll(reader)
require.NoError(t, err, "recipient %d should read the plaintext", i+1)
require.NoError(t, reader.Close())
assert.Equal(t, plaintext, got,
"recipient %d should recover the original plaintext", i+1)
}
}
internal/blobgen/roundtrip_test.go
+132
View File
@@ -0,0 +1,132 @@
package blobgen_test
import (
"bytes"
"crypto/sha256"
"fmt"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// checkRoundTrip writes input through a Writer, reads it back through a Reader,
// and verifies the plaintext, the byte counts, and the content hashes.
func checkRoundTrip(
t *testing.T, id *age.X25519Identity, recipient string,
level int, input []byte,
) {
t.Helper()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, level, []string{recipient})
require.NoError(t, err)
n, err := w.Write(input)
require.NoError(t, err)
assert.Equal(t, len(input), n)
require.NoError(t, w.Close())
require.Equal(t, int64(len(input)), w.BytesWritten())
r, err := blobgen.NewReader(bytes.NewReader(buf.Bytes()), id)
require.NoError(t, err)
got, err := io.ReadAll(r)
require.NoError(t, err)
require.NoError(t, r.Close())
assert.Equal(t, input, got, "decrypted output must equal input")
require.Equal(t, int64(len(input)), r.BytesRead())
// The hash values are checked by decrypting: the reader's single SHA-256
// is the hash of the plaintext, and hashing it once more (DoubleSHA256)
// gives the writer's ContentID.
single := sha256.Sum256(got)
assert.Equal(t, single[:], r.Sum256())
assert.Equal(t, blobgen.DoubleSHA256(r.Sum256()), w.ContentID())
}
// TestWriterReaderRoundTrip covers issue cases 1 and 2: every size round trips
// for both random and compressible data, and the reader hash, its double hash
// and the byte counts all agree.
func TestWriterReaderRoundTrip(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
// Sizes exercise the age segment boundary (64 KiB) from just below to a
// few segments above it, plus the empty and single-byte edges.
sizes := []int{0, 1, 65535, 65536, 65537, 4*65536 + 123}
kinds := []struct {
name string
fill func(*testing.T, int) []byte
}{
{"random", randomBytes},
{"compressible", func(_ *testing.T, n int) []byte {
return compressibleBytes(n)
}},
}
for _, k := range kinds {
for _, size := range sizes {
name := fmt.Sprintf("%s/%d", k.name, size)
t.Run(name, func(t *testing.T) {
t.Parallel()
checkRoundTrip(t, id, recipient, 1, k.fill(t, size))
})
}
}
}
// TestZeroLengthNoWrite covers issue case 3: a Writer closed with no Write at
// all produces the double hash of the empty input, and the blob reads back as
// empty with no error.
func TestZeroLengthNoWrite(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, 1, []string{recipient})
require.NoError(t, err)
require.NoError(t, w.Close())
assert.Equal(t, int64(0), w.BytesWritten())
empty := sha256.Sum256(nil)
doubled := sha256.Sum256(empty[:])
assert.Equal(t, doubled[:], w.ContentID(),
"ContentID of empty input is SHA256(SHA256(\"\"))")
r, err := blobgen.NewReader(bytes.NewReader(buf.Bytes()), id)
require.NoError(t, err)
got, err := io.ReadAll(r)
require.NoError(t, err)
require.NoError(t, r.Close())
assert.Empty(t, got, "empty blob decrypts to empty output")
assert.Equal(t, int64(0), r.BytesRead())
assert.Equal(t, empty[:], r.Sum256())
}
// TestNewWriterValidLevelsRoundTrip covers the accepted end of issue case 9:
// the boundary compression levels 1 and 19 both round trip.
func TestNewWriterValidLevelsRoundTrip(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
input := randomBytes(t, 4096)
for _, level := range []int{1, 19} {
t.Run(fmt.Sprintf("level%d", level), func(t *testing.T) {
t.Parallel()
checkRoundTrip(t, id, recipient, level, input)
})
}
}
internal/blobgen/truncation_test.go
+70
View File
@@ -0,0 +1,70 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestReaderRejectsHeaderNonceTruncation guards against a stream cut right
// after the age header plus its 16-byte nonce. age.Decrypt still succeeds on
// such an object, and the zstd decoder maps the age reader's
// io.ErrUnexpectedEOF to a clean io.EOF at frame start, so without the extra
// check the truncated stream would read as a valid empty one. Reading it must
// now fail.
func TestReaderRejectsHeaderNonceTruncation(t *testing.T) {
t.Parallel()
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
// Encrypting empty plaintext yields header + nonce(16) + a single
// 16-byte final chunk tag. Dropping the trailing tag leaves exactly the
// age header plus its nonce — the truncation point that triggers the bug.
var full bytes.Buffer
w, err := age.Encrypt(&full, identity.Recipient())
require.NoError(t, err)
require.NoError(t, w.Close())
truncated := full.Bytes()[:full.Len()-16]
reader, err := blobgen.NewReader(bytes.NewReader(truncated), identity)
require.NoError(t, err)
defer func() { _ = reader.Close() }()
_, err = io.ReadAll(reader)
require.Error(t, err)
require.ErrorIs(t, err, io.ErrUnexpectedEOF)
}
// TestReaderReadsGenuinelyEmptyBlob confirms the truncation check does not
// reject a legitimately empty payload: a blob written with no data must round
// trip back to zero bytes with no error.
func TestReaderReadsGenuinelyEmptyBlob(t *testing.T) {
t.Parallel()
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
var encrypted bytes.Buffer
writer, err := blobgen.NewWriter(
&encrypted, 3, []string{identity.Recipient().String()})
require.NoError(t, err)
require.NoError(t, writer.Close())
reader, err := blobgen.NewReader(bytes.NewReader(encrypted.Bytes()), identity)
require.NoError(t, err)
defer func() { _ = reader.Close() }()
data, err := io.ReadAll(reader)
require.NoError(t, err)
require.Empty(t, data)
}
internal/blobgen/writer.go
+36 -13
View File
@@ -1,3 +1,6 @@
// Package blobgen implements the blob data pipeline: streaming zstd
// compression, age encryption, and SHA256 content hashing for blob
// creation, plus the matching decrypt/decompress/verify reader.
package blobgen
import (
@@ -12,6 +15,24 @@ import (
"github.com/klauspost/compress/zstd"
)
// DoubleSHA256 returns the double SHA-256 of content whose single SHA-256
// digest is sum: it hashes that digest once more. Stored objects — a blob, and
// the metadata database export — are named by this second hash.
//
// The second hash does not hide whether known content is stored: an attacker
// who can reproduce an object's entire plaintext computes the same name simply
// by hashing twice, exactly as this code does. What limits that is blob
// packing, not the double hash — a blob's name covers all of its concatenated
// chunk plaintext, so a name can be confirmed only by someone who can
// reproduce the whole blob (a snapshot made entirely of known content, or a
// known file large enough to fill blobs on its own). An ordinary file that
// shares a blob with other, unknown data cannot be confirmed this way.
func DoubleSHA256(sum []byte) []byte {
h := sha256.Sum256(sum)
return h[:]
}
// Zstd compression level bounds accepted by NewWriter.
const (
minCompressionLevel = 1
@@ -27,6 +48,11 @@ const reservedCompressionCPUs = 2
var ErrInvalidCompressionLevel = errors.New(
"invalid compression level: must be between 1 and 19")
// errInvalidRecipient is returned when a recipient string does not parse as
// an X25519 age1... public key. It omits the value, which can be sensitive.
var errInvalidRecipient = errors.New(
"not a valid X25519 age1... recipient")
// Writer wraps compression and encryption with SHA256 hashing.
// Data flows: input -> tee(hasher, compressor -> encryptor -> destination)
// The hash is computed on the uncompressed input for deterministic content-addressing.
@@ -57,10 +83,12 @@ func NewWriter(
// Parse recipients
var ageRecipients []age.Recipient
for _, recipient := range recipients {
for i, recipient := range recipients {
// The recipient string can be sensitive (e.g. a secret key pasted by
// mistake), so the error names its position, never its value.
r, err := age.ParseX25519Recipient(recipient)
if err != nil {
return nil, fmt.Errorf("parsing recipient %s: %w", recipient, err)
return nil, fmt.Errorf("%w: recipient %d", errInvalidRecipient, i)
}
ageRecipients = append(ageRecipients, r)
@@ -123,17 +151,12 @@ func (w *Writer) Close() error {
return nil
}
// Sum256 returns the double SHA256 hash of the uncompressed input data.
// Double hashing (SHA256(SHA256(data))) prevents information leakage about
// the plaintext - an attacker cannot confirm existence of known content
// by computing its hash and checking for a matching blob filename.
func (w *Writer) Sum256() []byte {
// First hash: SHA256(plaintext)
firstHash := w.hasher.Sum(nil)
// Second hash: SHA256(firstHash) - this is the blob ID
secondHash := sha256.Sum256(firstHash)
return secondHash[:]
// ContentID returns the double SHA-256 of the uncompressed input data: the
// name under which this content is stored. It is the second hash of the
// running SHA-256, via DoubleSHA256; see that function for what naming content
// this way does and does not hide.
func (w *Writer) ContentID() []byte {
return DoubleSHA256(w.hasher.Sum(nil))
}
// BytesWritten returns the number of uncompressed bytes written
internal/blobgen/writer_test.go
+27 -9
View File
@@ -12,9 +12,10 @@ import (
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestWriterHashIsDoubleHash verifies that Writer.Sum256() returns
// the double hash SHA256(SHA256(plaintext)) for security.
// Double hashing prevents attackers from confirming existence of known content.
// TestWriterHashIsDoubleHash verifies that Writer.ContentID() returns
// SHA256(SHA256(plaintext)). Stored objects are named by this second hash so a
// name is not the plaintext's own SHA-256; this does not stop someone who
// already holds the plaintext from confirming it.
func TestWriterHashIsDoubleHash(t *testing.T) {
t.Parallel()
@@ -43,7 +44,7 @@ func TestWriterHashIsDoubleHash(t *testing.T) {
require.NoError(t, err)
// Get the hash from the writer
writerHash := hex.EncodeToString(writer.Sum256())
writerHash := hex.EncodeToString(writer.ContentID())
// Calculate the expected double hash: SHA256(SHA256(plaintext))
firstHash := sha256.Sum256(testData)
@@ -60,11 +61,11 @@ func TestWriterHashIsDoubleHash(t *testing.T) {
// The writer hash should match the double hash
assert.Equal(t, expectedDoubleHash, writerHash,
"Writer.Sum256() should return SHA256(SHA256(plaintext)) for security")
"Writer.ContentID() must be SHA256(SHA256(plaintext))")
// Verify it's NOT the single hash (would leak information)
// It must be the second hash, not the plaintext's own SHA-256.
assert.NotEqual(t, singleHashStr, writerHash,
"Writer hash should not be single hash (would allow content confirmation attacks)")
"Writer hash must be the double hash, not the single SHA-256")
}
// TestWriterDeterministicHash verifies that the same input always produces
@@ -93,8 +94,8 @@ func TestWriterDeterministicHash(t *testing.T) {
require.NoError(t, err)
require.NoError(t, writer2.Close())
hash1 := hex.EncodeToString(writer1.Sum256())
hash2 := hex.EncodeToString(writer2.Sum256())
hash1 := hex.EncodeToString(writer1.ContentID())
hash2 := hex.EncodeToString(writer2.ContentID())
// Hashes should be identical (deterministic)
assert.Equal(t, hash1, hash2, "Same input should produce same hash")
@@ -108,3 +109,20 @@ func TestWriterDeterministicHash(t *testing.T) {
t.Logf("Encrypted size 1: %d bytes", buf1.Len())
t.Logf("Encrypted size 2: %d bytes", buf2.Len())
}
// TestNewWriterSecretKeyNotEchoed verifies that a secret key mistakenly passed
// as a recipient does not appear in the returned error. A recipient string can
// be sensitive, so the error must name only the position, not the value.
func TestNewWriterSecretKeyNotEchoed(t *testing.T) {
t.Parallel()
secretKey := "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GX" +
"VEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5"
var buf bytes.Buffer
_, err := blobgen.NewWriter(&buf, 3, []string{secretKey})
require.Error(t, err)
assert.NotContains(t, err.Error(), secretKey,
"error must not echo the recipient value")
}
internal/chunker/chunker.go
+5 -4
View File
@@ -33,9 +33,10 @@ type Chunker struct {
maxChunkSize int
}
// chunkSizeSpread is the FastCDC-recommended factor between the average
// ChunkSizeSpread is the FastCDC-recommended factor between the average
// chunk size and the minimum (avg/spread) and maximum (avg*spread) sizes.
const chunkSizeSpread = 4
// The largest chunk the chunker can emit is therefore avg*ChunkSizeSpread.
const ChunkSizeSpread = 4
// NewChunker creates a new chunker with the specified average chunk size.
// The actual chunk sizes will vary between avgChunkSize/4 and avgChunkSize*4
@@ -45,8 +46,8 @@ func NewChunker(avgChunkSize int64) *Chunker {
// FastCDC recommends min = avg/4 and max = avg*4
return &Chunker{
avgChunkSize: int(avgChunkSize),
minChunkSize: int(avgChunkSize / chunkSizeSpread),
maxChunkSize: int(avgChunkSize * chunkSizeSpread),
minChunkSize: int(avgChunkSize / ChunkSizeSpread),
maxChunkSize: int(avgChunkSize * ChunkSizeSpread),
}
}
internal/cli/app.go
+154 -104
View File
@@ -7,12 +7,9 @@ import (
"context"
"errors"
"fmt"
"os"
"os/signal"
"path/filepath"
"strings"
"sync"
"syscall"
"time"
"github.com/adrg/xdg"
@@ -33,14 +30,33 @@ import (
// may take before we give up.
const shutdownTimeout = 30 * time.Second
// AppOptions contains common options for creating the fx application.
// It includes the configuration file path, logging options, and additional
// fx modules and invocations that should be included in the application.
// lockMode says whether a command mutates persistent state — the local
// index database or the remote store — and so must hold the process-wide
// PID lock, or only reads that state and may run alongside a mutator.
type lockMode int
const (
// mutating commands (snapshot create, snapshot purge, snapshot remove,
// prune, remote nuke) write the local index or the remote store. They
// hold the PID lock so that at most one runs at a time.
mutating lockMode = iota
// readOnly commands (info, snapshot list, snapshot verify, remote info,
// snapshot restore) do not write the local index or the remote store,
// so they run without the lock and are never blocked by a running
// mutator. restore writes only to the target directory it is given.
readOnly
)
// AppOptions contains common options for creating and running the fx
// application: the configuration file path, logging options, additional fx
// modules and invocations, and whether the command mutates persistent
// state (which decides whether it takes the PID lock).
type AppOptions struct {
ConfigPath string
LogOptions log.Options
Modules []fx.Option
Invokes []fx.Option
Mode lockMode
}
// setupGlobals records the startup time and, when an output-suppression
@@ -142,73 +158,58 @@ func cleanStartupError(err error) error {
return &startupError{msg: msg}
}
// RunApp starts and stops the fx application within the given context.
// It handles graceful shutdown on interrupt signals (SIGINT, SIGTERM) and
// ensures the application stops cleanly. The function blocks until the
// application completes or is interrupted. Returns an error if startup fails.
// RunApp starts the fx application, blocks until it is asked to stop, and
// then stops it. The app is asked to stop either by an OS interrupt
// (SIGINT/SIGTERM — fx installs its own handler when app.Wait is called) or,
// on normal completion, by the finished operation calling
// Shutdowner.Shutdown(); both arrive on the app.Wait channel.
//
// Stopping runs the fx OnStop hooks, and RunApp does not return until Stop
// returns. On an interrupt the operation's OnStop hook cancels the running
// command and waits for it to unwind — removing its decrypted scratch files —
// so the process cannot proceed to exit mid-cleanup (issue #159). Waiting for
// Stop before returning is what makes that hook effective: routing the
// interrupt through app.Stop and not returning until it completes is required,
// because fx also fires the app.Wait channel on the signal, and an earlier
// version returned on that alone — unwinding to os.Exit while the concurrent
// cleanup still ran. The stop is bounded by shutdownTimeout. Returns an error
// if startup fails.
func RunApp(ctx context.Context, app *fx.App) error {
// Set up signal handling for graceful shutdown
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
// Create a context that will be cancelled on signal
ctx, cancel := context.WithCancel(ctx)
defer cancel()
// Start the app
err := app.Start(ctx)
if err != nil {
return cleanStartupError(err)
}
// Handle shutdown
shutdownComplete := make(chan struct{})
// Block until an interrupt or the finished operation's
// Shutdowner.Shutdown() arrives, then stop the app in this goroutine so we
// return only after its OnStop hooks — including the operation's cleanup
// wait — have run. Detach the stop from ctx's cancellation but keep its
// values, and bound it by shutdownTimeout.
<-app.Wait()
go func() {
defer close(shutdownComplete)
shutdownCtx, cancel := context.WithTimeout(
context.WithoutCancel(ctx), shutdownTimeout)
defer cancel()
<-sigChan
log.Notice("Received interrupt signal, shutting down gracefully...")
// Create a timeout context for shutdown. The parent ctx is being
// cancelled, so detach from its cancellation but keep its values.
shutdownCtx, shutdownCancel := context.WithTimeout(
context.WithoutCancel(ctx), shutdownTimeout)
defer shutdownCancel()
err := app.Stop(shutdownCtx)
if err != nil {
log.Error("Error during shutdown", "error", err)
}
}()
// Wait for the signal handler to complete shutdown or the app to
// request shutdown.
select {
case <-shutdownComplete:
// Shutdown completed via signal
return nil
case <-ctx.Done():
// Context cancelled (shouldn't happen in normal operation)
err := app.Stop(context.WithoutCancel(ctx))
if err != nil {
log.Error("Error stopping app", "error", err)
}
return ctx.Err()
case <-app.Done():
// App finished running (e.g., backup completed)
return nil
err = app.Stop(shutdownCtx)
if err != nil {
log.Error("Error during shutdown", "error", err)
}
return nil
}
// errReported marks a failure the operation has already shown the user
// (and deliberately withheld under --json). Entry turns it into a
// non-zero exit status without printing anything further, so the error
// line is not doubled. It flows up from RunOperation through cobra to
// Entry.
// (or, under `snapshot verify --json`, put in its document). Entry
// turns it into a non-zero exit status without printing anything
// further, so the error line is not doubled. It flows up from
// RunOperation through cobra to Entry.
var errReported = errors.New("operation failed")
// errInterrupted marks an operation that SIGINT or SIGTERM stopped
// before it finished. Entry shows it and returns exitCodeInterrupted.
var errInterrupted = errors.New("interrupted before the command finished")
// RunOperation runs op against the Vaultik instance inside the fx app
// and turns a failure into a returned error rather than an os.Exit from
// within the goroutine. An os.Exit there skipped main's deferred
@@ -219,31 +220,50 @@ var errReported = errors.New("operation failed")
//
// op runs in a goroutine so OnStart returns promptly and an interrupt
// can still cancel through OnStop; when it finishes, success or failure,
// it triggers shutdown, which is what lets RunWithApp return. report is
// called with a non-canceled failure so the caller can log it (and
// suppress it under --json) before it becomes errReported. A context
// cancellation is the interrupt path, not a failure: it is neither
// reported nor counted as one.
// it triggers shutdown, which is what lets RunWithApp return. On an
// interrupt OnStop cancels op and waits for the goroutine to return, so
// op's cleanup (removing decrypted scratch files) runs before the
// process exits; the wait is bounded by shutdownTimeout. report is
// called with a failure so the caller can show it to the user before
// it becomes errReported.
//
// The run counts as interrupted unless op returned, without an
// interrupt having cancelled it, before RunWithApp returned. An
// interrupted op is not reported, whatever it returned; RunOperation
// returns errInterrupted instead.
func RunOperation(
ctx context.Context, opts AppOptions,
op func(v *vaultik.Vaultik) error, report func(err error),
) error {
var (
mu sync.Mutex
failed bool
mu sync.Mutex
finished bool // op returned before any interrupt cancelled it
failed bool // op finished with an error
)
opts.Invokes = append(opts.Invokes,
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
var stop func(context.Context) bool
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
go func() {
stop = v.StartOperation(func() {
err := op(v)
if err != nil && !errors.Is(err, context.Canceled) {
report(err)
// Only stop, called from OnStop below, cancels the
// Vaultik context, so a live context means no
// interrupt cancelled op. Check the context, not
// err: an interrupted op need not return
// context.Canceled (`snapshot verify --json`
// returns a verification failure).
if v.Context().Err() == nil {
if err != nil {
report(err)
}
mu.Lock()
failed = true
finished = true
failed = err != nil
mu.Unlock()
}
@@ -251,12 +271,20 @@ func RunOperation(
if stopErr != nil {
log.Error("Failed to shutdown", "error", stopErr)
}
}()
})
return nil
},
OnStop: func(_ context.Context) error {
v.Cancel()
// On an interrupt, cancel the operation and wait for it to
// unwind so its cleanup defers (which remove decrypted
// scratch files from the temp directory) run before the
// process exits. The wait is bounded by ctx, the existing
// shutdownTimeout.
OnStop: func(ctx context.Context) error {
if !stop(ctx) {
log.Warn("Shutdown timed out before the operation " +
"finished; decrypted temporary files may remain")
}
return nil
},
@@ -268,27 +296,34 @@ func RunOperation(
return err
}
// The goroutine sets failed before triggering the shutdown that lets
// RunWithApp return, so the write is in place by the time we read it.
// RunWithApp returns only after the app was asked to stop, either by
// an interrupt or by the goroutine's Shutdown call. When op finished
// without being cancelled, the goroutine set finished before that
// call. So if finished is unset here, an interrupt stopped the app,
// and op either returned after it was cancelled or is still running
// because the shutdown timed out.
mu.Lock()
defer mu.Unlock()
if failed {
switch {
case !finished:
return errInterrupted
case failed:
return errReported
default:
return nil
}
return nil
}
// runVaultikApp runs the standard single-operation command lifecycle
// shared by the list/purge/verify/remove/remote-info subcommands:
// shared by the snapshot list/purge/remove and remote nuke subcommands:
// resolve the config, then run op against the Vaultik instance through
// RunOperation, reporting a failure prefixed with failMsg (suppressed
// while suppressErrors is true, e.g. under --json). jsonOutput marks a
// command whose stdout is a JSON document: it quiets the UI but, unlike
// Quiet, leaves the stderr log level alone.
// RunOperation, reporting a failure prefixed with failMsg on stderr. mode
// says whether the command takes the PID lock. jsonOutput marks a command
// whose stdout is a JSON document: it quiets the UI but, unlike Quiet,
// leaves the stderr log level alone.
func runVaultikApp(
cmd *cobra.Command, jsonOutput, suppressErrors bool,
cmd *cobra.Command, mode lockMode, jsonOutput bool,
failMsg string, op func(v *vaultik.Vaultik) error,
) error {
configPath, err := ResolveConfigPath()
@@ -306,11 +341,8 @@ func runVaultikApp(
Quiet: rootFlags.Quiet,
JSON: jsonOutput,
},
Mode: mode,
}, op, func(err error) {
if suppressErrors {
return
}
log.Error(failMsg, "error", err)
ReportErrorf("%s: %v", failMsg, err)
})
@@ -319,28 +351,46 @@ func runVaultikApp(
// RunWithApp is a helper that creates and runs an fx app with the given options.
// It combines NewApp and RunApp into a single convenient function. This is the
// preferred way to run CLI commands that need the full application context.
// It acquires a PID lock before starting to prevent concurrent instances.
// A mutating command takes the process-wide PID lock before starting so that
// only one runs at a time; a read-only command runs without it and is not
// blocked while a mutator holds the lock (opts.Mode).
func RunWithApp(ctx context.Context, opts AppOptions) error {
// Acquire PID lock to prevent concurrent instances
lockDir := filepath.Join(xdg.DataHome, "vaultik")
lock, err := pidlock.Acquire(lockDir)
release, err := acquireLockIfMutating(opts.Mode,
filepath.Join(xdg.DataHome, "vaultik"))
if err != nil {
if errors.Is(err, pidlock.ErrAlreadyRunning) {
return fmt.Errorf("cannot start: %w", err)
}
return fmt.Errorf("failed to acquire lock: %w", err)
return err
}
defer func() {
err := lock.Release()
if err != nil {
log.Warn("Failed to release PID lock", "error", err)
}
}()
defer release()
app := NewApp(opts)
return RunApp(ctx, app)
}
// acquireLockIfMutating takes the process-wide PID lock in lockDir for a
// mutating command and returns a function that releases it. A read-only
// command takes no lock, so it returns a no-op release and is never blocked
// while a mutator holds the lock. ErrAlreadyRunning (another mutator holds
// the lock) is surfaced as a "cannot start" error.
func acquireLockIfMutating(mode lockMode, lockDir string) (func(), error) {
if mode != mutating {
return func() {}, nil
}
lock, err := pidlock.Acquire(lockDir)
if err != nil {
if errors.Is(err, pidlock.ErrAlreadyRunning) {
return nil, fmt.Errorf("cannot start: %w", err)
}
return nil, fmt.Errorf("failed to acquire lock: %w", err)
}
return func() {
err := lock.Release()
if err != nil {
log.Warn("Failed to release PID lock", "error", err)
}
}, nil
}
internal/cli/app_test.go
+42
View File
@@ -2,7 +2,10 @@ package cli //nolint:testpackage // needs access to unexported cleanStartupError
import (
"errors"
"path/filepath"
"testing"
"sneak.berlin/go/vaultik/internal/pidlock"
)
func TestCleanStartupError(t *testing.T) {
@@ -53,3 +56,42 @@ func TestCleanStartupError(t *testing.T) {
})
}
}
// TestLockScopedToMutatingCommands proves the partition the PID lock now
// enforces: a read-only command runs while a mutator holds the lock, and
// two mutating commands still mutually exclude.
func TestLockScopedToMutatingCommands(t *testing.T) {
t.Parallel()
lockDir := filepath.Join(t.TempDir(), "vaultik")
// A mutating command takes the process-wide lock.
releaseMutator, err := acquireLockIfMutating(mutating, lockDir)
if err != nil {
t.Fatalf("mutating command could not acquire lock: %v", err)
}
// A read-only command runs to completion even while the lock is held.
releaseReader, err := acquireLockIfMutating(readOnly, lockDir)
if err != nil {
t.Fatalf("read-only command was blocked by held lock: %v", err)
}
releaseReader()
// A second mutating command is refused while the first holds the lock.
_, err = acquireLockIfMutating(mutating, lockDir)
if !errors.Is(err, pidlock.ErrAlreadyRunning) {
t.Fatalf("second mutating command was not excluded, got: %v", err)
}
// Once the first mutator releases, another mutating command may run.
releaseMutator()
release, err := acquireLockIfMutating(mutating, lockDir)
if err != nil {
t.Fatalf("mutating command could not acquire released lock: %v", err)
}
release()
}
internal/cli/config.go
+129 -44
View File
@@ -7,11 +7,15 @@ import (
"os"
"os/exec"
"path/filepath"
"reflect"
"strconv"
"strings"
"unicode/utf8"
"github.com/spf13/cobra"
"gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/ui"
)
// configFileMode is the permission set for freshly written config files;
@@ -30,6 +34,9 @@ const configDirMode = 0o755
// yaml.Marshal's 4-space default.
const configYAMLIndent = 2
// yamlStringTag is YAML's tag for a string scalar.
const yamlStringTag = "!!str"
var (
errConfigExists = errors.New("config file already exists")
errEmptyConfig = errors.New("empty config file")
@@ -44,13 +51,19 @@ const defaultConfigTemplate = `# vaultik configuration
# ─── REQUIRED ────────────────────────────────────────────────────────────────
# Age recipient public keys for encryption.
# Backups are encrypted to ALL listed recipients. Any one of the corresponding
# private keys can decrypt. Generate a keypair with:
# Age recipient public keys for encryption. snapshot create needs at least
# one; listing, verifying and restoring do not, so a machine that only
# restores can leave this empty.
# Backups are encrypted to ALL listed recipients; any one of the corresponding
# private keys can decrypt. Adding a recipient later does not re-encrypt data
# already stored: deduplicated chunks and existing blobs stay encrypted to the
# earlier recipients, so a newly added key cannot restore them on its own (see
# docs/REPOSTRUCTURE.md, Accepted Risks). Generate a keypair and add its
# public key with:
# age-keygen -o vaultik_backup_private_key.txt
# grep 'public key' vaultik_backup_private_key.txt
age_recipients:
- age1REPLACE_WITH_YOUR_PUBLIC_KEY
# vaultik config set age_recipients.0 age1...
age_recipients: []
# Named snapshots. Each snapshot backs up one or more paths and can have its
# own exclude patterns in addition to the global excludes below.
@@ -192,8 +205,8 @@ storage_url: ""
# access_key_id: YOUR_ACCESS_KEY
# secret_access_key: YOUR_SECRET_KEY
# # region: us-east-1 # Default: us-east-1
# # use_ssl: true # Default: true
# # part_size: 5MB # Multipart upload part size. Default: 5MB
# # part_size: 5MiB # Upload part size, 5MiB to 5GiB. Default: 5MiB
# # For the s3:// form, disable TLS with ?ssl=false in the URL, not use_ssl.
# ─── OPTIONAL ────────────────────────────────────────────────────────────────
@@ -212,6 +225,8 @@ storage_url: ""
# chunk_size: 10MB
# Maximum blob size before splitting into a new blob.
# Must be at least four times chunk_size (the largest chunk the chunker can
# emit); a smaller limit would let a single-chunk blob exceed it.
# Accepts: 1GB, 10G, 500MB, etc.
# Default: 10GB
# blob_size_limit: 10GB
@@ -259,7 +274,7 @@ The config is written to the path from --config, $VAULTIK_CONFIG, or
the platform default config directory (e.g. ~/Library/Application Support/
on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`,
Args: cobra.NoArgs,
RunE: func(_ *cobra.Command, _ []string) error {
RunE: func(cmd *cobra.Command, _ []string) error {
path := configPathForInit()
_, err := os.Stat(path)
@@ -279,8 +294,11 @@ on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`,
return fmt.Errorf("writing config file: %w", err)
}
_, _ = fmt.Fprintf(os.Stdout, "Config written to %s\n", path)
_, _ = fmt.Fprintln(os.Stdout,
// A written-confirmation, not scriptable output: route it
// through the UI so it is styled and --quiet silences it.
out := commandUI(cmd)
out.Infof("Config written to %s.", path)
out.Infof(
"Edit it to set your age_recipients, snapshots, and storage_url.")
return nil
@@ -322,7 +340,7 @@ func newConfigGetCommand() *cobra.Command {
Use: "get <key>",
Short: "Print a config value by dotted path (e.g. storage_url, compression_level)",
Args: cobra.ExactArgs(1),
RunE: func(_ *cobra.Command, args []string) error {
RunE: func(cmd *cobra.Command, args []string) error {
path, err := ResolveConfigPath()
if err != nil {
return err
@@ -338,8 +356,13 @@ func newConfigGetCommand() *cobra.Command {
return err
}
// The value is scriptable output: it must stay machine-plain
// (no marker, no color) and is never silenced by --quiet, so it
// is written straight to stdout rather than through the UI.
w := cmd.OutOrStdout()
if node.Kind == yaml.ScalarNode {
_, _ = fmt.Fprintln(os.Stdout, node.Value)
_, _ = fmt.Fprintln(w, node.Value)
return nil
}
@@ -349,7 +372,7 @@ func newConfigGetCommand() *cobra.Command {
return fmt.Errorf("marshaling value: %w", err)
}
_, _ = fmt.Fprint(os.Stdout, string(out))
_, _ = fmt.Fprint(w, string(out))
return nil
},
@@ -371,46 +394,59 @@ Examples:
vaultik config set compression_level 9
vaultik config set s3.bucket mybucket # legacy S3 fields still supported`,
Args: cobra.ExactArgs(configSetArgs),
RunE: func(_ *cobra.Command, args []string) error {
RunE: func(cmd *cobra.Command, args []string) error {
path, err := ResolveConfigPath()
if err != nil {
return err
}
root, err := loadYAMLFile(path)
if err != nil {
return err
}
err = yamlPathSet(root, strings.Split(args[0], "."), args[1])
if err != nil {
return err
}
out, err := marshalConfigYAML(root)
if err != nil {
return fmt.Errorf("marshaling config: %w", err)
}
mode := os.FileMode(configFileMode)
info, statErr := os.Stat(path)
if statErr == nil {
mode = info.Mode().Perm()
}
err = os.WriteFile(path, out, mode)
if err != nil {
return fmt.Errorf("writing config file: %w", err)
}
_, _ = fmt.Fprintf(os.Stdout, "%s = %s\n", args[0], args[1])
return nil
return writeConfigSet(commandUI(cmd), path, args[0], args[1])
},
}
}
// writeConfigSet applies key=value to the config at path, writes it back
// owner-only, and confirms the write by naming just the key through the
// UI writer (styled, and silenced by --quiet). The value is never
// echoed: it may be a secret such as s3.secret_access_key, and captured
// stdout or a pasted terminal would then leak it.
func writeConfigSet(out *ui.Writer, path, key, value string) error {
root, err := loadYAMLFile(path)
if err != nil {
return err
}
err = yamlPathSet(root, strings.Split(key, "."), value)
if err != nil {
return err
}
data, err := marshalConfigYAML(root)
if err != nil {
return fmt.Errorf("marshaling config: %w", err)
}
err = os.WriteFile(path, data, configFileMode)
if err != nil {
return fmt.Errorf("writing config file: %w", err)
}
// os.WriteFile does not change the mode of a file that already exists,
// so a config that was group- or world-readable stays that way. As it
// may hold S3 credentials, tighten it to owner-only after writing.
info, statErr := os.Stat(path)
if statErr == nil && info.Mode().Perm()&0o044 != 0 {
err = os.Chmod(path, configFileMode)
if err != nil {
return fmt.Errorf("tightening config file permissions: %w", err)
}
}
out.Infof("Set %s.", key)
return nil
}
// marshalConfigYAML renders a config document tree with 2-space indentation,
// matching defaultConfigTemplate. yaml.Marshal defaults to 4 spaces, which
// would reindent the whole file on the first `config set` despite the promise
@@ -553,9 +589,58 @@ func yamlPathSet(root *yaml.Node, keys []string, value string) error {
}
}
// config.Load reads the file through untyped YAML, which turns an
// unquoted 00112233 into the number 38043 and 1e5 into 100000. Tagging
// a string setting as a string makes the encoder quote such a value.
// Other settings stay unquoted, so compression_level 9 is a number.
// The encoder refuses to write a value that is not valid UTF-8 as a
// string. Left untagged, such a value is written as base64 !!binary and
// loads back unchanged.
if configKeyIsString(keys) && utf8.ValidString(value) {
node.Tag = yamlStringTag
}
return nil
}
// configKeyIsString reports whether the dotted key names a string in
// config.Config, following the fields' yaml tags, as s3.access_key_id and
// snapshots.home.exclude.0 do.
func configKeyIsString(keys []string) bool {
typ := reflect.TypeFor[config.Config]()
for _, key := range keys {
switch {
case typ.Kind() == reflect.Map || typ.Kind() == reflect.Slice:
// The key is a snapshot name or a list index.
typ = typ.Elem()
case typ.Kind() == reflect.Struct:
field, ok := yamlField(typ, key)
if !ok {
return false
}
typ = field.Type
default:
return false
}
}
return typ.Kind() == reflect.String
}
// yamlField returns the field of struct type typ whose yaml tag names key.
func yamlField(typ reflect.Type, key string) (reflect.StructField, bool) {
for field := range typ.Fields() {
name, _, _ := strings.Cut(field.Tag.Get("yaml"), ",")
if name == key {
return field, true
}
}
return reflect.StructField{}, false
}
// yamlSetInMapping resolves (creating if needed) the value node for key
// within a mapping node, setting it to value when it is the final path
// element, and returns the node to descend into.
internal/cli/config_test.go
+214 -2
View File
@@ -1,11 +1,16 @@
package cli //nolint:testpackage // exercises unexported yamlPathGet/yamlPathSet
import (
"bytes"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/ui"
)
// TestDefaultConfigTemplateParses ensures the init template is valid YAML
@@ -20,8 +25,10 @@ func TestDefaultConfigTemplateParses(t *testing.T) {
t.Fatalf("default config template is not valid YAML: %v", err)
}
if len(cfg.AgeRecipients) != 1 {
t.Errorf("expected 1 placeholder age recipient, got %d", len(cfg.AgeRecipients))
// A placeholder recipient would fail config.Load, so the template
// leaves the list empty.
if len(cfg.AgeRecipients) != 0 {
t.Errorf("expected no age recipients, got %d", len(cfg.AgeRecipients))
}
home, ok := cfg.Snapshots["home"]
@@ -51,6 +58,147 @@ func TestDefaultConfigTemplateParses(t *testing.T) {
}
}
// TestConfigSetRecipientOnFreshConfig follows the README quickstart: on the
// file `config init` writes, `config set age_recipients.0` and
// `config set storage_url` give a config that loads with that recipient.
func TestConfigSetRecipientOnFreshConfig(t *testing.T) {
t.Parallel()
const recipient = "age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj"
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(defaultConfigTemplate), configFileMode)
if err != nil {
t.Fatalf("write config: %v", err)
}
out := ui.NewWithColor(&bytes.Buffer{}, false)
err = writeConfigSet(out, path, "age_recipients.0", recipient)
if err != nil {
t.Fatalf("config set age_recipients.0: %v", err)
}
err = writeConfigSet(out, path, "storage_url", "file:///mnt/backups")
if err != nil {
t.Fatalf("config set storage_url: %v", err)
}
cfg, err := config.Load(path)
if err != nil {
t.Fatalf("config.Load: %v", err)
}
if len(cfg.AgeRecipients) != 1 || cfg.AgeRecipients[0] != recipient {
t.Errorf("age_recipients = %v, want [%s]", cfg.AgeRecipients, recipient)
}
}
// TestConfigSetStringLooksLikeNumber sets string settings to values that
// YAML reads as numbers or booleans when they are unquoted, and checks that
// config.Load returns each one unchanged.
func TestConfigSetStringLooksLikeNumber(t *testing.T) {
t.Parallel()
tests := []struct {
key string
value string
field func(cfg *config.Config) string
}{
{"s3.access_key_id", "00112233",
func(cfg *config.Config) string { return cfg.S3.AccessKeyID }},
{"s3.secret_access_key", "12345678901234567890123456789012",
func(cfg *config.Config) string { return cfg.S3.SecretAccessKey }},
{"hostname", "007",
func(cfg *config.Config) string { return cfg.Hostname }},
{"s3.prefix", "1e5",
func(cfg *config.Config) string { return cfg.S3.Prefix }},
{"s3.bucket", "true",
func(cfg *config.Config) string { return cfg.S3.Bucket }},
{"s3.region", "FALSE",
func(cfg *config.Config) string { return cfg.S3.Region }},
{"snapshots.home.exclude.0", "1.10",
func(cfg *config.Config) string { return cfg.Snapshots["home"].Exclude[0] }},
}
for _, tt := range tests {
t.Run(tt.key+"="+tt.value, func(t *testing.T) {
t.Parallel()
cfg := loadAfterConfigSet(t, tt.key, tt.value)
got := tt.field(cfg)
if got != tt.value {
t.Errorf("%s = %q after config set %q", tt.key, got, tt.value)
}
})
}
}
// TestConfigSetNonUTF8Path checks that config set still accepts a value that
// is not valid UTF-8, such as a path with a Latin-1 file name, and that
// config.Load returns it unchanged.
func TestConfigSetNonUTF8Path(t *testing.T) {
t.Parallel()
const dir = "/srv/caf\xe9"
cfg := loadAfterConfigSet(t, "snapshots.home.paths.0", dir)
got := cfg.Snapshots["home"].Paths[0]
if got != dir {
t.Errorf("snapshots.home.paths.0 = %q, want %q", got, dir)
}
}
// TestConfigSetNumberStaysNumber checks that a number set for an integer
// setting is still read as a number, not as a quoted string.
func TestConfigSetNumberStaysNumber(t *testing.T) {
t.Parallel()
const level = 9
cfg := loadAfterConfigSet(t, "compression_level", strconv.Itoa(level))
if cfg.CompressionLevel != level {
t.Errorf("compression_level = %d, want %d", cfg.CompressionLevel, level)
}
}
// loadAfterConfigSet writes the file `config init` writes, sets storage_url
// to a local directory so that the file passes validation, applies
// `config set key value` and returns what config.Load reads back.
func loadAfterConfigSet(t *testing.T, key, value string) *config.Config {
t.Helper()
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(defaultConfigTemplate), configFileMode)
if err != nil {
t.Fatalf("write config: %v", err)
}
out := ui.NewWithColor(&bytes.Buffer{}, false)
err = writeConfigSet(out, path, "storage_url", "file:///mnt/backups")
if err != nil {
t.Fatalf("config set storage_url: %v", err)
}
err = writeConfigSet(out, path, key, value)
if err != nil {
t.Fatalf("config set %s: %v", key, err)
}
cfg, err := config.Load(path)
if err != nil {
t.Fatalf("config.Load: %v", err)
}
return cfg
}
const testYAML = `# top comment
compression_level: 3
age_recipients:
@@ -229,6 +377,70 @@ func TestConfigSetPreservesFormatting(t *testing.T) {
}
}
// TestWriteConfigSetHidesSecret checks that setting a secret key prints
// only the key name, never the value, to the confirmation output.
func TestWriteConfigSetHidesSecret(t *testing.T) {
t.Parallel()
const secret = "SUPERSECRETVALUE"
path := filepath.Join(t.TempDir(), "config.yaml")
err := os.WriteFile(path, []byte("version: 1\n"), 0o600)
if err != nil {
t.Fatalf("seed config: %v", err)
}
var buf bytes.Buffer
err = writeConfigSet(ui.NewWithColor(&buf, false), path,
"s3.secret_access_key", secret)
if err != nil {
t.Fatalf("writeConfigSet: %v", err)
}
if strings.Contains(buf.String(), secret) {
t.Errorf("output echoed the secret value: %q", buf.String())
}
if !strings.Contains(buf.String(), "s3.secret_access_key") {
t.Errorf("output did not confirm the key name: %q", buf.String())
}
}
// TestWriteConfigSetTightensMode checks that a pre-existing group- or
// world-readable config is tightened to owner-only after a set, since
// os.WriteFile leaves an existing file's mode untouched.
func TestWriteConfigSetTightensMode(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "config.yaml")
// Seed a world-readable config; the loose mode is the condition under
// test, so gosec's G306 is expected here.
err := os.WriteFile(path, []byte("version: 1\n"), 0o644) //nolint:gosec // G306
if err != nil {
t.Fatalf("seed config: %v", err)
}
var buf bytes.Buffer
err = writeConfigSet(ui.NewWithColor(&buf, false), path,
"compression_level", "9")
if err != nil {
t.Fatalf("writeConfigSet: %v", err)
}
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat config: %v", err)
}
if info.Mode().Perm() != 0o600 {
t.Errorf("config mode = %04o, want 0600", info.Mode().Perm())
}
}
func splitPath(s string) []string {
return strings.Split(s, ".")
}
internal/cli/database.go
+12 -11
View File
@@ -48,7 +48,7 @@ storage destination on that run.
Use --force to skip the confirmation prompt.`,
Args: cobra.NoArgs,
RunE: func(_ *cobra.Command, _ []string) error {
RunE: func(cmd *cobra.Command, _ []string) error {
// Resolve config path
configPath, err := ResolveConfigPath()
if err != nil {
@@ -62,26 +62,31 @@ Use --force to skip the confirmation prompt.`,
}
dbPath := cfg.IndexPath
out := commandUI(cmd)
// Check if database exists
_, err = os.Stat(dbPath)
if os.IsNotExist(err) {
_, _ = fmt.Fprintf(os.Stdout, "Database does not exist: %s\n", dbPath)
out.Infof("Local state database does not exist: %s.", dbPath)
return nil
}
// Confirm unless --force
// Confirm unless --force. The prompt and its immediate result
// are an interactive exchange the operator must see, so they go
// straight to stdout rather than through the UI and --quiet does
// not silence them.
if !force {
_, _ = fmt.Fprintf(os.Stdout,
w := cmd.OutOrStdout()
_, _ = fmt.Fprintf(w,
"This will delete the local state database at:\n %s\n\n", dbPath)
_, _ = fmt.Fprint(os.Stdout, "Are you sure? Type 'yes' to confirm: ")
_, _ = fmt.Fprint(w, "Are you sure? Type 'yes' to confirm: ")
var confirm string
_, err = fmt.Scanln(&confirm)
if err != nil || confirm != "yes" {
_, _ = fmt.Fprintln(os.Stdout, "Aborted.")
_, _ = fmt.Fprintln(w, "Aborted.")
//nolint:nilerr // a failed/aborted confirmation is a clean abort
return nil
@@ -100,11 +105,7 @@ Use --force to skip the confirmation prompt.`,
_ = os.Remove(walPath) // Ignore errors - files may not exist
_ = os.Remove(shmPath)
rootFlags := GetRootFlags()
if !rootFlags.Quiet {
_, _ = fmt.Fprintf(os.Stdout, "Database deleted: %s\n", dbPath)
}
out.Infof("Local state database deleted: %s.", dbPath)
log.Info("Local state database deleted", "path", dbPath)
return nil
internal/cli/entry.go
+25 -13
View File
@@ -15,17 +15,25 @@ import (
// the startup banner.
const shortCommitLen = 12
// exitCodeInterrupted is the exit status of a command that SIGINT or
// SIGTERM stopped. It is 128 plus SIGINT's number, 2, which is what a
// shell reports for a command stopped by Ctrl-C.
const exitCodeInterrupted = 130
// Entry is the main entry point for the CLI application.
// It prints the startup banner to stdout (unless a banner-suppressing
// It prints the startup banner to stderr (unless a banner-suppressing
// flag is present in os.Args — see bannerSuppressedInArgs), executes the
// root cobra command, and routes any returned error through the
// ui.Writer so the user sees a properly formatted "🛑 ERROR:" line.
// The banner goes to stderr because stdout carries only the output the
// user asked for, such as a completion script or a `config get` value.
//
// It returns the process exit code (0 on success, 1 on error) rather
// than calling os.Exit, so that main's deferred profile writers run
// before the process ends. See run in cmd/vaultik/main.go.
// It returns the process exit code (0 on success, 130 when interrupted,
// 1 on any other error) rather than calling os.Exit, so that main's
// deferred profile writers run before the process ends. See run in
// cmd/vaultik/main.go.
func Entry() int {
emitStartupBanner(os.Args[1:], os.Stdout)
emitStartupBanner(os.Args[1:], os.Stderr)
rootCmd := NewRootCommand()
rootCmd.SilenceErrors = true
@@ -33,14 +41,20 @@ func Entry() int {
err := rootCmd.Execute()
if err != nil {
// An operation that ran inside the fx app has already reported
// its own failure (and suppressed it under --json); errReported
// says so. Printing it again here would double the error line.
// its own failure (`snapshot verify --json` puts it in the
// document instead); errReported says so. Printing it again
// here would double the error line.
// Every other error — bad arguments, a config that would not
// load — reaches Entry unreported, so it is shown here.
// load, an interrupt — reaches Entry unreported, so it is shown
// here.
if !errors.Is(err, errReported) {
ReportErrorf("%s", err.Error())
}
if errors.Is(err, errInterrupted) {
return exitCodeInterrupted
}
return 1
}
@@ -49,9 +63,8 @@ func Entry() int {
// emitStartupBanner writes the startup banner to w unless args (the
// argument vector with the program name already stripped) contains a
// flag that suppresses it. Split out of Entry so that the decision — the
// only thing standing between a --json invocation and a parseable
// stdout — is reachable from a test without running the whole CLI.
// flag that suppresses it. Split out of Entry so that the decision is
// reachable from a test without running the whole CLI.
func emitStartupBanner(args []string, w io.Writer) {
if bannerSuppressedInArgs(args) {
return
@@ -86,8 +99,7 @@ func ReportErrorf(format string, args ...any) {
// --json is a subcommand flag rather than a persistent one, but so is
// --cron (it exists only on `snapshot create`), so this adds no new
// class of imprecision. The only cost of a false positive is a missing
// decorative banner; the cost of a false negative is a corrupt document
// on stdout, so the scan errs deliberately in that direction.
// decorative banner.
func bannerSuppressedInArgs(args []string) bool {
for _, a := range args {
if a == "--" {
internal/cli/entry_banner_test.go
+25 -49
View File
@@ -36,23 +36,14 @@ const (
// strips it before scanning, so it has to be present.
programName = "vaultik"
// someSnapshotID is any snapshot identifier: these tests never run
// the command, so it only has to occupy the positional argument.
// someSnapshotID only fills the positional argument; no test needs
// the snapshot to exist.
someSnapshotID = "host_2026-01-01T00:00:00Z"
)
// placeholderJSONDocument stands in for whatever document a --json
// command writes to stdout. `snapshot list --json` with no snapshots
// prints exactly this; the other --json commands print an object rather
// than an array, but this test is not about their shape. It is about
// what is on stdout *before* them, which is the same for all of them
// because Entry prints the banner before cobra has parsed anything and
// therefore before it can know which command is running.
const placeholderJSONDocument = "[]\n"
// jsonArgumentVectors are the argument vectors of every --json
// invocation the CLI accepts, with the program name stripped exactly as
// Entry strips it. Each one must leave stdout untouched by the banner.
// Entry strips it. Each one must suppress the banner.
//
//nolint:gochecknoglobals // read-only test fixture shared by two tests
var jsonArgumentVectors = map[string][]string{
@@ -74,39 +65,23 @@ var jsonArgumentVectors = map[string][]string{
},
}
// TestJSONInvocationStdoutIsExactlyOneDocument is the CLI-layer
// regression guard for issue #106: `vaultik snapshot list --json | jq`
// must work with no other flags.
//
// internal/vaultik's TestListSnapshots_JSONStdoutIsOnlyTheDocument
// guards the same contract one layer down, but it calls the library
// function directly and so cannot see Entry, which is where the
// contamination was: the startup banner is written to stdout before
// cobra parses anything, and the suppression scan did not know about
// --json. The two banner lines and the blank line landed ahead of the
// document and `jq` refused the result.
//
// The document is a constant here because this test is about the
// argument vectors, one per --json command; the one that runs a real
// command end to end is TestEntryJSONStdoutIsExactlyOneDocument below.
func TestJSONInvocationStdoutIsExactlyOneDocument(t *testing.T) {
// TestJSONInvocationSuppressesBanner checks that every --json
// invocation suppresses the startup banner, as the README says --json
// does along with --quiet and --cron. The scan is over the raw argument
// vector, so each position and spelling of --json is listed.
func TestJSONInvocationSuppressesBanner(t *testing.T) {
t.Parallel()
for name, argv := range jsonArgumentVectors {
t.Run(name, func(t *testing.T) {
t.Parallel()
var stdout bytes.Buffer
var banner bytes.Buffer
emitStartupBanner(argv, &stdout)
emitStartupBanner(argv, &banner)
require.Empty(t, stdout.String(),
"nothing may reach stdout ahead of a --json document")
_, err := stdout.WriteString(placeholderJSONDocument)
require.NoError(t, err)
requireExactlyOneJSONDocument(t, stdout.String())
assert.Empty(t, banner.String(),
"--json suppresses the banner")
})
}
}
@@ -127,11 +102,11 @@ func TestBannerStillPrintedWithoutSuppressingFlag(t *testing.T) {
t.Run(name, func(t *testing.T) {
t.Parallel()
var stdout bytes.Buffer
var banner bytes.Buffer
emitStartupBanner(argv, &stdout)
emitStartupBanner(argv, &banner)
assert.Contains(t, stdout.String(), "starting up at",
assert.Contains(t, banner.String(), "starting up at",
"the banner belongs on invocations that did not opt out")
})
}
@@ -172,9 +147,9 @@ func TestBannerSuppressedInArgs(t *testing.T) {
// hermeticConfig is a complete, valid config that needs no network and
// no credentials: file:// storage is exempt from the S3 credential
// checks, and FileStorer over a directory that does not exist lists
// zero objects without erroring. Chunk, blob and compression settings
// are filled in by config.Load.
// checks. A test that lists the destination must create its directory
// first, because listing a directory that does not exist is an error.
// Chunk, blob and compression settings are filled in by config.Load.
const hermeticConfig = `age_recipients:
- age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj
snapshots:
@@ -196,22 +171,25 @@ hostname: test-host
// `snapshot list` is the command chosen because it is the only --json
// command that reaches its document without a populated destination
// store: it reads the local index, streams `metadata/` (empty here),
// and treats a barren destination as an empty list rather than a
// failure.
// and treats an empty destination directory as an empty list rather
// than a failure.
//
// Not parallel: it replaces os.Args, os.Stdout and the xdg globals.
func TestEntryJSONStdoutIsExactlyOneDocument(t *testing.T) {
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
storeDir := filepath.Join(dir, "store")
contents := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"),
filepath.Join(dir, "store"),
storeDir,
filepath.Join(dir, "index.sqlite"))
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
require.NoError(t, os.Mkdir(storeDir, 0o750))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
@@ -244,9 +222,7 @@ func TestEntryJSONStdoutIsExactlyOneDocument(t *testing.T) {
// captureProcessStdout redirects the process's own stdout to a pipe for
// the duration of fn and returns what was written to it. The redirection
// has to be at the file-descriptor level rather than through an injected
// writer, because the banner and the JSON encoder reach os.Stdout
// independently and the point of the test is that both land in the same
// place.
// writer, because the commands Entry runs reach os.Stdout directly.
//
// Not parallel-safe: os.Stdout is process-global.
func captureProcessStdout(t *testing.T, fn func()) string {
internal/cli/entry_interrupt_test.go
+203
View File
@@ -0,0 +1,203 @@
package cli //nolint:testpackage // shares runEntry and the argument constants
import (
"fmt"
"net/http"
"net/http/httptest"
"os"
"os/signal"
"path/filepath"
"strings"
"testing"
"time"
"github.com/adrg/xdg"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// stalledStoreConfig is hermeticConfig with an s3:// destination store
// in place of the file:// one. The server behind it accepts any
// credentials.
const stalledStoreConfig = `age_recipients:
- age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj
snapshots:
test:
paths:
- %s
storage_url: s3://bucket?endpoint=%s&ssl=false
s3:
access_key_id: key
secret_access_key: secret
index_path: %s
hostname: test-host
`
// interruptRepeat is how often interruptOnFirstRequest sends SIGINT.
const interruptRepeat = 50 * time.Millisecond
// TestEntryInterruptedRun sends SIGINT to the test process while a
// command waits on the destination store, and checks that Entry returns
// 130 and prints one line on stderr saying the run was interrupted. The
// store is a local HTTP server that holds every request open, so the
// command is always mid-operation when the signal arrives. The two
// cases cover --cron and --json, which silence other output.
//
// Not parallel: it signals the process and replaces os.Args, os.Stdout,
// os.Stderr and the xdg globals.
//
//nolint:paralleltest // signals the process and replaces process globals
func TestEntryInterruptedRun(t *testing.T) {
for _, testCase := range []struct {
name string
args []string
}{
{
name: "snapshot create --cron",
args: []string{cmdSnapshot, cmdCreate, "--cron"},
},
{
name: "snapshot verify --json",
args: []string{cmdSnapshot, cmdVerify, someSnapshotID, flagJSON},
},
} {
t.Run(testCase.name, func(t *testing.T) {
endpoint, requestArrived := startStalledStore(t)
configPath := writeStalledStoreConfig(t, endpoint)
interruptOnFirstRequest(t, requestArrived)
code, _, stderr := runEntry(t,
append([]string{flagConfig, configPath}, testCase.args...)...)
assert.Equal(t, 130, code)
assert.Equal(t, 1,
strings.Count(stderr, errInterrupted.Error()), stderr)
})
}
}
// interruptOnFirstRequest sends SIGINT to the test process every
// interruptRepeat, from the first request to the destination store until
// the test ends. One signal is not enough: the command can reach the
// store before fx has started catching signals. The test catches SIGINT
// too, so that a signal fx is not catching does not kill the test
// binary.
func interruptOnFirstRequest(t *testing.T, requestArrived <-chan struct{}) {
t.Helper()
self, err := os.FindProcess(os.Getpid())
require.NoError(t, err)
caught := make(chan os.Signal, 1)
signal.Notify(caught, os.Interrupt)
testEnded := make(chan struct{})
senderDone := make(chan struct{})
// Stop catching SIGINT only after the sender has returned. The sender
// waits for each SIGINT it sends to arrive on caught; one still on
// its way after signal.Stop would kill the test binary.
t.Cleanup(func() {
close(testEnded)
<-senderDone
signal.Stop(caught)
})
go func() {
defer close(senderDone)
select {
case <-requestArrived:
case <-testEnded:
return
}
ticker := time.NewTicker(interruptRepeat)
defer ticker.Stop()
for {
// Empty caught, so that the receive below waits for this
// SIGINT rather than an earlier one.
select {
case <-caught:
default:
}
sendErr := self.Signal(os.Interrupt)
if sendErr != nil {
t.Errorf("sending SIGINT: %v", sendErr)
return
}
<-caught
select {
case <-testEnded:
return
case <-ticker.C:
}
}
}()
}
// startStalledStore starts an HTTP server that never answers: each
// request is held until the client gives up on it or the test ends.
// It returns the server's host:port and a channel that receives a value
// when the first request arrives.
func startStalledStore(t *testing.T) (string, <-chan struct{}) {
t.Helper()
requestArrived := make(chan struct{}, 1)
release := make(chan struct{})
server := httptest.NewServer(http.HandlerFunc(
func(_ http.ResponseWriter, r *http.Request) {
select {
case requestArrived <- struct{}{}:
default:
}
select {
case <-r.Context().Done():
case <-release:
}
}))
// Cleanups run last-registered first, so release lets any held
// request return before Close waits for it.
t.Cleanup(server.Close)
t.Cleanup(func() { close(release) })
return server.Listener.Addr().String(), requestArrived
}
// writeStalledStoreConfig writes a config whose destination store is the
// server at endpoint and whose snapshot source holds one small file, so
// that `snapshot create` has a blob to upload. Returns the config path.
func writeStalledStoreConfig(t *testing.T, endpoint string) string {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
sourceDir := filepath.Join(dir, "source")
require.NoError(t, os.Mkdir(sourceDir, 0o750))
require.NoError(t, os.WriteFile(filepath.Join(sourceDir, "file.txt"),
[]byte("contents"), 0o600))
contents := fmt.Sprintf(stalledStoreConfig,
sourceDir, endpoint, filepath.Join(dir, "index.sqlite"))
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
t.Setenv("XDG_DATA_HOME", filepath.Join(dir, "data"))
xdg.Reload()
t.Cleanup(xdg.Reload)
return configPath
}
internal/cli/entry_prune_json_test.go
+10 -5
View File
@@ -98,25 +98,30 @@ func TestEntryPruneJSONStdoutIsExactlyOneDocument(t *testing.T) {
}
}
// writeHermeticPruneConfig builds a config over a temp directory and, if
// seedStale is set, creates the index database up front with one
// snapshot record that has no counterpart on the destination store.
// Returns the config path.
// writeHermeticPruneConfig builds a config over a temp directory with an
// empty destination directory and, if seedStale is set, creates the
// index database up front with one snapshot record that has no
// counterpart on the destination store. Returns the config path.
func writeHermeticPruneConfig(t *testing.T, seedStale bool) string {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
indexPath := filepath.Join(dir, "index.sqlite")
storeDir := filepath.Join(dir, "store")
contents := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"),
filepath.Join(dir, "store"),
storeDir,
indexPath)
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
// prune fails on a destination directory that does not exist, so the
// empty store is created here rather than left to a first backup.
require.NoError(t, os.Mkdir(storeDir, 0o750))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
internal/cli/entry_status_test.go
+3 -3
View File
@@ -15,8 +15,8 @@ import (
// run, so a failing command must come back with a non-zero code rather
// than ending the process here.
//
// Stdout is captured only to keep the banner and command output off the
// test log; the assertion is on the returned code.
// Stdout and stderr are captured only to keep the banner and command
// output off the test log; the assertion is on the returned code.
//
//nolint:paralleltest // replaces os.Args and rootFlags
func TestEntryReturnsStatusCode(t *testing.T) {
@@ -50,7 +50,7 @@ func TestEntryReturnsStatusCode(t *testing.T) {
var code int
_ = captureProcessStdout(t, func() { code = Entry() })
_, _ = captureProcessStdoutAndStderr(t, func() { code = Entry() })
assert.Equal(t, testCase.want, code)
})
internal/cli/entry_stdout_stderr_test.go
+200
View File
@@ -0,0 +1,200 @@
package cli //nolint:testpackage // shares hermeticConfig and the capture helpers
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
"github.com/adrg/xdg"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
)
// TestEntryCompletionStdoutIsTheScript runs `vaultik completion bash`,
// whose stdout the README tells the user to source. The script has to
// start on the first line.
//
//nolint:paralleltest // replaces os.Args, os.Stdout and os.Stderr
func TestEntryCompletionStdoutIsTheScript(t *testing.T) {
code, stdout, _ := runEntry(t, "completion", "bash")
require.Equal(t, 0, code)
firstLine, _, _ := strings.Cut(stdout, "\n")
assert.True(t, strings.HasPrefix(firstLine, "# bash completion"),
"the first line of stdout must be the script's, got %q", firstLine)
}
// TestEntryConfigGetStdoutIsTheValue runs `vaultik config get`, whose
// stdout a script reads as the value and nothing else.
//
//nolint:paralleltest // replaces os.Args, os.Stdout and os.Stderr
func TestEntryConfigGetStdoutIsTheValue(t *testing.T) {
configPath := filepath.Join(t.TempDir(), "config.yml")
require.NoError(t, os.WriteFile(configPath,
[]byte("hostname: test-host\n"), configFileMode))
code, stdout, _ := runEntry(t,
flagConfig, configPath, "config", "get", "hostname")
require.Equal(t, 0, code)
assert.Equal(t, "test-host\n", stdout)
}
// TestEntryJSONFailureIsReportedOnStderr runs each --json command that
// writes no document when it fails, against a destination it cannot
// use. The error must reach stderr, and stdout must stay empty.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntryJSONFailureIsReportedOnStderr(t *testing.T) {
for _, testCase := range []struct {
name string
args []string
wantOnStderr string
}{
{
name: "remote info",
args: []string{cmdRemote, cmdInfo, flagJSON},
wantOnStderr: "Failed to get remote info",
},
{
name: "prune",
args: []string{cmdPrune, flagJSON},
wantOnStderr: "Prune failed",
},
{
name: "snapshot remove",
args: []string{cmdSnapshot, cmdRemove, someSnapshotID, flagJSON},
wantOnStderr: "Failed to remove snapshot",
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath := writeUnusableDestinationConfig(t)
code, stdout, stderr := runEntry(t,
append([]string{flagConfig, configPath}, testCase.args...)...)
assert.Equal(t, 1, code)
assert.Empty(t, stdout,
"a failed --json command has no document to write")
assert.Contains(t, stderr, testCase.wantOnStderr,
"the failure must be reported on stderr")
})
}
}
// TestEntrySnapshotRemoveJSONWarningIsOnStderr runs `snapshot remove
// --json` on a snapshot in the local index, against a destination
// directory that does not exist. The command removes the snapshot from
// the local index and still exits 0. Its stdout must hold the document
// alone, with the warning about the destination store on stderr: the
// command to run again once it is reachable, and the snapshot's ID in
// the record's snapshot_id field.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotRemoveJSONWarningIsOnStderr(t *testing.T) {
configPath, indexPath := writeMissingDestinationConfig(t)
seedStaleSnapshotRecord(t, indexPath)
code, stdout, stderr := runEntry(t, flagConfig, configPath,
cmdSnapshot, cmdRemove, stalePruneSnapshotID, flagJSON)
require.Equal(t, 0, code)
requireExactlyOneJSONDocument(t, stdout)
// stderr is a pipe here, so the logger writes one JSON record a line.
var warning map[string]any
for line := range strings.Lines(stderr) {
if strings.Contains(line,
"Could not remove snapshot metadata from remote storage") {
require.NoError(t, json.Unmarshal([]byte(line), &warning))
}
}
require.NotNil(t, warning, "the warning must reach stderr")
assert.Contains(t, warning[slog.MessageKey],
"run 'vaultik snapshot remove' with the snapshot's ID again")
assert.Equal(t, stalePruneSnapshotID, warning["snapshot_id"])
}
// writeMissingDestinationConfig builds a config whose destination
// directory does not exist. Returns the config path and the path of
// its local index, which is not created here.
func writeMissingDestinationConfig(t *testing.T) (string, string) {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
indexPath := filepath.Join(dir, "index.sqlite")
contents := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"),
filepath.Join(dir, "missing-store"),
indexPath)
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
t.Setenv("XDG_DATA_HOME", filepath.Join(dir, "data"))
xdg.Reload()
t.Cleanup(xdg.Reload)
return configPath, indexPath
}
// writeUnusableDestinationConfig builds a config whose destination
// directory does not exist, which fails `remote info`, and whose local
// index is bound to another destination, which fails `prune` and
// `snapshot remove` (a missing destination alone only makes `snapshot
// remove` warn). Returns the config path.
func writeUnusableDestinationConfig(t *testing.T) string {
t.Helper()
configPath, indexPath := writeMissingDestinationConfig(t)
ctx := context.Background()
db, err := database.New(ctx, indexPath)
require.NoError(t, err)
defer func() { require.NoError(t, db.Close()) }()
require.NoError(t, database.NewRepositories(db).LocalMeta.Set(ctx,
database.LocalMetaKeyStorageURL, "file://"+t.TempDir()))
return configPath
}
// runEntry runs Entry with args after the program name and returns its
// exit code and what it wrote to stdout and stderr.
//
// Not parallel-safe: it replaces os.Args, os.Stdout and os.Stderr.
func runEntry(t *testing.T, args ...string) (int, string, string) {
t.Helper()
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
os.Args = append([]string{programName}, args...)
var code int
stdout, stderr := captureProcessStdoutAndStderr(t,
func() { code = Entry() })
return code, stdout, stderr
}
internal/cli/info.go
+1
View File
@@ -35,6 +35,7 @@ func NewInfoCommand() *cobra.Command {
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
},
Mode: readOnly,
}, func(v *vaultik.Vaultik) error {
return v.ShowInfo()
}, func(err error) {
internal/cli/json_help_test.go
+24
View File
@@ -0,0 +1,24 @@
package cli //nolint:testpackage // exercises the unexported command constructor
import (
"testing"
"github.com/spf13/cobra"
"github.com/stretchr/testify/assert"
)
// TestJSONHelpSaysConfirmationPromptIsSkipped checks that the --json help
// of `snapshot remove` and `prune` says the flag skips the confirmation
// prompt. Both delete without asking under --json, since a prompt on
// stdout would break the JSON document.
func TestJSONHelpSaysConfirmationPromptIsSkipped(t *testing.T) {
t.Parallel()
for _, cmd := range []*cobra.Command{
newSnapshotRemoveCommand(),
NewPruneCommand(),
} {
assert.Contains(t, cmd.Flags().Lookup("json").Usage,
"skips the confirmation prompt", cmd.Name())
}
}
internal/cli/prune.go
+8 -8
View File
@@ -22,9 +22,11 @@ scans every snapshot manifest in the destination store, builds the
set of still-referenced blob hashes, and deletes any blob not in that
set.
Snapshot create --prune and snapshot remove run the same cleanup
automatically; this command is the manual entry point for the same
work (e.g. after a crashed backup or to reclaim storage).`,
Snapshot create --prune runs the same cleanup automatically; this
command is the manual entry point for the same work (e.g. after a
crashed backup or to reclaim storage). Snapshot remove leaves blobs in
place; run this command afterwards to delete the ones no longer
referenced.`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
// Use unified config resolution
@@ -44,13 +46,10 @@ work (e.g. after a crashed backup or to reclaim storage).`,
Quiet: rootFlags.Quiet,
JSON: opts.JSON,
},
Mode: mutating,
}, func(v *vaultik.Vaultik) error {
return v.Prune(opts)
}, func(err error) {
if opts.JSON {
return
}
log.Error("Prune operation failed", "error", err)
ReportErrorf("Prune failed: %v", err)
})
@@ -58,7 +57,8 @@ work (e.g. after a crashed backup or to reclaim storage).`,
}
cmd.Flags().BoolVar(&opts.Force, "force", false, "Skip confirmation prompt")
cmd.Flags().BoolVar(&opts.JSON, "json", false, "Output pruning stats as JSON")
cmd.Flags().BoolVar(&opts.JSON, "json", false,
"Output pruning stats as JSON; skips the confirmation prompt, as --force does")
return cmd
}
internal/cli/quiet_test.go
+206
View File
@@ -0,0 +1,206 @@
package cli //nolint:testpackage // sets the unexported rootFlags directly
import (
"bytes"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/spf13/cobra"
)
// setRootFlags overrides the global rootFlags for the duration of one
// test and restores it afterward. These tests must not run in parallel:
// the flags are process-global, so the whole struct is saved and put
// back rather than left mutated for the next test.
func setRootFlags(t *testing.T, f RootFlags) {
t.Helper()
old := rootFlags
rootFlags = f
t.Cleanup(func() { rootFlags = old })
}
// seedFile writes content to a fresh file and returns its path.
func seedFile(t *testing.T, dir, name, content string) string {
t.Helper()
path := filepath.Join(dir, name)
err := os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("seeding %s: %v", name, err)
}
return path
}
// mustExecute runs a command with its output captured and fails the test
// if it errors, returning what the command printed.
func mustExecute(t *testing.T, cmd *cobra.Command, args ...string) string {
t.Helper()
var out bytes.Buffer
cmd.SetOut(&out)
cmd.SetArgs(args)
err := cmd.Execute()
if err != nil {
t.Fatalf("%s failed: %v", cmd.Name(), err)
}
return out.String()
}
// TestVersionQuietSuppressesReport checks that --quiet silences the whole
// version report: it is human-facing output, not a scriptable value.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestVersionQuietSuppressesReport(t *testing.T) {
setRootFlags(t, RootFlags{Quiet: true})
out := mustExecute(t, NewVersionCommand())
if out != "" {
t.Errorf("--quiet version printed %q, want nothing", out)
}
}
// TestConfigGetIgnoresQuiet checks that a config value is printed even
// under --quiet: it is scriptable output a caller depends on, so --quiet
// must not suppress it, and it stays machine-plain (no marker, no color).
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigGetIgnoresQuiet(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "storage_url: file:///mnt/x\n")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigGetCommand(), "storage_url")
if out != "file:///mnt/x\n" {
t.Errorf("config get --quiet = %q, want the plain value", out)
}
}
// TestConfigSetQuietSuppressesConfirmation checks that --quiet silences
// the confirmation line while still writing the value to the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigSetQuietSuppressesConfirmation(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "compression_level: 3\n")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigSetCommand(), "compression_level", "9")
if out != "" {
t.Errorf("--quiet config set printed %q, want nothing", out)
}
data, err := os.ReadFile(path) //nolint:gosec // G304: test-controlled path
if err != nil {
t.Fatalf("reading config back: %v", err)
}
if !strings.Contains(string(data), "compression_level: 9") {
t.Errorf("config set did not write the value under --quiet:\n%s", data)
}
}
// TestConfigSetConfirmsWhenNotQuiet checks that the confirmation names
// the key (styled) when --quiet is not set.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigSetConfirmsWhenNotQuiet(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "compression_level: 3\n")
setRootFlags(t, RootFlags{ConfigPath: path})
out := mustExecute(t, newConfigSetCommand(), "compression_level", "9")
if !strings.Contains(out, "compression_level") {
t.Errorf("config set did not confirm the key: %q", out)
}
}
// TestConfigInitQuietSuppressesConfirmation checks that --quiet silences
// the "config written" confirmation while still writing the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigInitQuietSuppressesConfirmation(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "new-config.yml")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigInitCommand())
if out != "" {
t.Errorf("--quiet config init printed %q, want nothing", out)
}
_, err := os.Stat(path)
if err != nil {
t.Errorf("config init did not write the file under --quiet: %v", err)
}
}
// seedDatabaseDeleteConfig writes a valid config whose index_path is a
// seeded database file, and returns both paths.
func seedDatabaseDeleteConfig(t *testing.T, dir string) (string, string) {
t.Helper()
dbPath := seedFile(t, dir, "index.sqlite", "not-a-real-db")
cfg := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"), filepath.Join(dir, "store"), dbPath)
cfgPath := seedFile(t, dir, "config.yml", cfg)
return dbPath, cfgPath
}
// TestDatabaseDeleteQuietSuppressesMessage checks that --quiet silences
// the "database deleted" line while still removing the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestDatabaseDeleteQuietSuppressesMessage(t *testing.T) {
dir := t.TempDir()
dbPath, cfgPath := seedDatabaseDeleteConfig(t, dir)
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: cfgPath})
out := mustExecute(t, newDatabaseDeleteCommand(), "--force")
if out != "" {
t.Errorf("--quiet database delete printed %q, want nothing", out)
}
_, err := os.Stat(dbPath)
if !os.IsNotExist(err) {
t.Errorf("database delete did not remove the file: stat err = %v", err)
}
}
// TestDatabaseDeleteReportsWhenNotQuiet checks that the deletion is
// reported when --quiet is not set.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestDatabaseDeleteReportsWhenNotQuiet(t *testing.T) {
dir := t.TempDir()
_, cfgPath := seedDatabaseDeleteConfig(t, dir)
setRootFlags(t, RootFlags{ConfigPath: cfgPath})
out := mustExecute(t, newDatabaseDeleteCommand(), "--force")
if !strings.Contains(out, "deleted") {
t.Errorf("database delete did not report the deletion: %q", out)
}
}
internal/cli/remote.go
+2 -5
View File
@@ -45,7 +45,7 @@ This is destructive and irreversible. Requires --force.`,
return errNukeNeedsForce
}
return runVaultikApp(cmd, false, false, "Remote nuke failed",
return runVaultikApp(cmd, mutating, false, "Remote nuke failed",
func(v *vaultik.Vaultik) error {
return v.NukeRemote(true)
})
@@ -88,13 +88,10 @@ func newRemoteInfoCommand() *cobra.Command {
Quiet: rootFlags.Quiet,
JSON: jsonOutput,
},
Mode: readOnly,
}, func(v *vaultik.Vaultik) error {
return v.RemoteInfo(jsonOutput)
}, func(err error) {
if jsonOutput {
return
}
log.Error("Failed to get remote info", "error", err)
ReportErrorf("Failed to get remote info: %v", err)
})
internal/cli/root.go
+19 -2
View File
@@ -9,6 +9,7 @@ import (
"github.com/adrg/xdg"
"github.com/spf13/cobra"
"sneak.berlin/go/vaultik/internal/ui"
)
// errConfigNotFound is wrapped by all config-resolution failures.
@@ -57,8 +58,10 @@ on the source system.`,
cmd.PersistentFlags().BoolVarP(&rootFlags.Quiet, "quiet", "q", false,
"Suppress non-error output")
cmd.PersistentFlags().BoolVar(&rootFlags.SkipErrors, "skip-errors", false,
"Continue past per-file errors instead of aborting "+
"(applies to snapshot create and restore)")
"Skip files that cannot be read when creating a snapshot, or "+
"that cannot be restored when restoring, instead of aborting "+
"(packing and storage errors while creating a snapshot still "+
"abort)")
// Add subcommands
cmd.AddCommand(
@@ -80,6 +83,20 @@ func GetRootFlags() RootFlags {
return rootFlags
}
// commandUI returns a UI writer for a command's stdout, in quiet mode
// when the global --quiet flag is set. This is how the pure-cli
// commands (version, config, database) reach internal/ui: color follows
// the writer (a TTY gets color, a captured test buffer does not), and
// --quiet silences the same message classes it silences everywhere else.
func commandUI(cmd *cobra.Command) *ui.Writer {
w := ui.New(cmd.OutOrStdout())
if GetRootFlags().Quiet {
w.SetQuiet(true)
}
return w
}
// ResolveConfigPath resolves the config file path from flags, environment, or default.
// Search order: --config flag, VAULTIK_CONFIG env, XDG config dir,
// /etc/vaultik/config.yml.
internal/cli/runapp_interrupt_test.go
+97
View File
@@ -0,0 +1,97 @@
package cli_test
import (
"context"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/cli"
)
// TestRunAppWaitsForOperationCleanupOnShutdown drives RunApp with an fx app
// wired the way RunOperation wires a command: a single lifecycle hook whose
// OnStart launches the operation in its own goroutine and whose OnStop cancels
// it and blocks until that goroutine returns. The operation stands in for a
// restore blocked mid-download — it holds a decrypted "scratch" file and only
// removes it as it unwinds on cancellation.
//
// The app is asked to stop once the operation is running (standing in for an
// OS interrupt; fx delivers a real signal and Shutdowner.Shutdown() on the
// same app.Wait channel, so both drive the identical shutdown path). RunApp
// must not return until app.Stop has run the OnStop hook, so the scratch file
// must be gone by the time RunApp returns. Before the fix RunApp returned as
// soon as the app.Wait/Done channel fired, without running app.Stop, so the
// cleanup never ran and this file would still be on disk (issue #159).
func TestRunAppWaitsForOperationCleanupOnShutdown(t *testing.T) {
t.Parallel()
scratch := filepath.Join(t.TempDir(), "decrypted-scratch")
require.NoError(t, os.WriteFile(scratch, []byte("secret"), 0o600))
// Cancel and reap the operation even if RunApp returns without doing so
// (the buggy path), so the goroutine cannot leak past the test.
opCtx, opCancel := context.WithCancel(context.Background())
t.Cleanup(opCancel)
var stop func(context.Context) bool
app := fx.New(
fx.NopLogger,
fx.Invoke(func(lc fx.Lifecycle, sh fx.Shutdowner) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
done := make(chan struct{})
go func() {
defer close(done)
// Blocked mid-operation until cancelled, then run the
// cleanup an interrupted restore would run.
<-opCtx.Done()
_ = os.Remove(scratch)
}()
stop = func(ctx context.Context) bool {
opCancel()
select {
case <-done:
return true
case <-ctx.Done():
return false
}
}
// Ask the app to stop now that the operation is running.
go func() { _ = sh.Shutdown() }()
return nil
},
OnStop: func(ctx context.Context) error {
stop(ctx)
return nil
},
})
}),
)
done := make(chan error, 1)
go func() { done <- cli.RunApp(context.Background(), app) }()
select {
case err := <-done:
require.NoError(t, err)
case <-time.After(30 * time.Second):
t.Fatal("RunApp did not return after shutdown was requested")
}
_, err := os.Stat(scratch)
require.True(t, os.IsNotExist(err),
"RunApp returned before the operation removed its decrypted scratch file")
}
internal/cli/snapshot.go
+19 -10
View File
@@ -66,8 +66,9 @@ func newSnapshotCreateCommand() *cobra.Command {
If snapshot names are provided, only those snapshots are created.
If no names are provided, all configured snapshots are created.
Config is located at /etc/vaultik/config.yml by default, but can be overridden by
specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
The config is read from the path given by --config or VAULTIK_CONFIG;
otherwise from the platform config directory (~/.config/vaultik/config.yml
on Linux), then /etc/vaultik/config.yml.`,
Args: cobra.ArbitraryArgs,
RunE: func(cmd *cobra.Command, args []string) error {
// Pass snapshot names from args
@@ -92,6 +93,7 @@ specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
Cron: opts.Cron,
Quiet: rootFlags.Quiet,
},
Mode: mutating,
}, func(v *vaultik.Vaultik) error {
return v.CreateSnapshot(opts)
}, func(err error) {
@@ -125,7 +127,7 @@ func newSnapshotListCommand() *cobra.Command {
Long: "Lists all snapshots with their ID, timestamp, and compressed size",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
return runVaultikApp(cmd, false, false,
return runVaultikApp(cmd, readOnly, false,
"Failed to list snapshots",
func(v *vaultik.Vaultik) error {
return v.ListSnapshots(jsonOutput)
@@ -161,7 +163,7 @@ restrict the operation to specific snapshot names.`,
return errPurgeCriteriaBoth
}
return runVaultikApp(cmd, false, false,
return runVaultikApp(cmd, mutating, false,
"Failed to purge snapshots",
func(v *vaultik.Vaultik) error {
return v.PurgeSnapshotsWithOptions(opts)
@@ -187,8 +189,12 @@ func newSnapshotVerifyCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "verify <snapshot-id>",
Short: "Verify snapshot integrity",
Long: "Verifies that all blobs referenced in a snapshot exist.\n\n" +
Short: "Check a snapshot's blobs are present with the listed size",
Long: "Checks that every blob the snapshot's manifest lists is present\n" +
"in storage with the size the manifest records, and that the\n" +
"snapshot's encrypted database is present. It does not read blob\n" +
"contents; use --deep to download, decrypt, and re-hash every blob\n" +
"to detect corruption -- integrity, not who wrote it.\n\n" +
"The snapshot may be named by its ID or, on a host with no local\n" +
"index, by the remote key that 'snapshot list' prints for a\n" +
"remote-only snapshot (an unambiguous leading part is enough).",
@@ -212,6 +218,7 @@ func newSnapshotVerifyCommand() *cobra.Command {
Quiet: rootFlags.Quiet,
JSON: opts.JSON,
},
Mode: readOnly,
}, func(v *vaultik.Vaultik) error {
return v.VerifySnapshotWithOptions(snapshotID, opts)
}, func(err error) {
@@ -252,14 +259,15 @@ Use --local-only to skip the remote half (e.g. when you want to forget a
snapshot locally without touching the destination store).
If the remote is unreachable, the local-database removal still completes
and a warning is emitted; rerun 'vaultik prune' once the destination store
is reachable to finish remote cleanup.
and a warning is emitted; run 'vaultik snapshot remove <snapshot-id>' again
once the destination store is reachable to remove the snapshot's metadata
from it ('vaultik prune' does not).
To wipe the entire destination store and start over, use 'vaultik remote
nuke --force' — it is the single supported entry point for that.`,
Args: requireSnapshotIDArg,
RunE: func(cmd *cobra.Command, args []string) error {
return runVaultikApp(cmd, opts.JSON, opts.JSON,
return runVaultikApp(cmd, mutating, opts.JSON,
"Failed to remove snapshot",
func(v *vaultik.Vaultik) error {
_, err := v.RemoveSnapshot(args[0], opts)
@@ -272,7 +280,8 @@ nuke --force' — it is the single supported entry point for that.`,
cmd.Flags().BoolVarP(&opts.Force, "force", "f", false, "Skip confirmation prompt")
cmd.Flags().BoolVar(&opts.DryRun, "dry-run", false,
"Show what would be removed without removing")
cmd.Flags().BoolVar(&opts.JSON, "json", false, "Output result as JSON")
cmd.Flags().BoolVar(&opts.JSON, "json", false,
"Output result as JSON; skips the confirmation prompt, as --force does")
cmd.Flags().BoolVar(&opts.LocalOnly, "local-only", false,
"Skip remote cleanup; only touch the local index")
internal/cli/snapshot_restore.go
+7 -2
View File
@@ -35,8 +35,12 @@ The snapshot may be named by its ID or, when restoring on a host with no
local index, by the remote key that 'snapshot list' prints for a
remote-only snapshot (an unambiguous leading part is enough).
Requires the VAULTIK_AGE_SECRET_KEY environment variable to be set with
the age private key.
Requires the age private key in the VAULTIK_AGE_SECRET_KEY environment
variable. The variable may hold the whole age-keygen file (comments and
all of its identities are accepted); read it from the file rather than
typing the key, so it does not land in your shell history:
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
Examples:
# Restore entire snapshot
@@ -88,6 +92,7 @@ func runRestore(cmd *cobra.Command, args []string, opts *RestoreOptions) error {
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
},
Mode: readOnly,
}, func(v *vaultik.Vaultik) error {
return v.Restore(&vaultik.RestoreOptions{
SnapshotID: snapshotID,
internal/cli/snapshot_restore_test.go
+37
View File
@@ -0,0 +1,37 @@
package cli //nolint:testpackage // exercises the unexported command constructor
import (
"strings"
"testing"
"github.com/spf13/pflag"
)
// TestRestoreCommandDoesNotTakeKeyAsArgument guards the fix for the age
// key being echoed on the command line: restore must take the key only
// from the environment, never as a flag value, and its help must show the
// file-based form rather than a literal key that would land in shell
// history.
func TestRestoreCommandDoesNotTakeKeyAsArgument(t *testing.T) {
t.Parallel()
cmd := newSnapshotRestoreCommand()
cmd.Flags().VisitAll(func(f *pflag.Flag) {
lower := strings.ToLower(f.Name)
for _, banned := range []string{"key", "secret", "age", "identity"} {
if strings.Contains(lower, banned) {
t.Errorf("restore must not accept the key as a flag; found --%s", f.Name)
}
}
})
help := cmd.Long
if strings.Contains(help, "AGE-SECRET-KEY-") {
t.Error("restore help must not show a literal age private key to type")
}
if !strings.Contains(help, "$(cat ") {
t.Error("restore help should read the key from a file, e.g. $(cat ...)")
}
}
internal/cli/version.go
+14 -5
View File
@@ -2,11 +2,11 @@ package cli
import (
"fmt"
"io"
"runtime"
"github.com/spf13/cobra"
"sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/ui"
)
// NewVersionCommand creates the version command
@@ -17,16 +17,25 @@ func NewVersionCommand() *cobra.Command {
Long: `Print version, git commit, and build information for vaultik.`,
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, _ []string) {
writeVersion(cmd.OutOrStdout())
writeVersion(commandUI(cmd))
},
}
return cmd
}
// writeVersion prints the version report. It takes a writer rather than
// using os.Stdout directly so the output can be asserted on in tests.
func writeVersion(w io.Writer) {
// writeVersion prints the version report through the UI writer. The
// report is the output this command exists to produce, so it is written
// plain (markers would corrupt the aligned report) via the writer's
// underlying stdout; --quiet silences it like any other non-error
// output.
func writeVersion(out *ui.Writer) {
if out.Quiet() {
return
}
w := out.Out()
_, _ = fmt.Fprintf(w, "vaultik %s\n", globals.Version)
_, _ = fmt.Fprintf(w, " commit: %s\n", globals.Commit)
_, _ = fmt.Fprintf(w, " build date: %s\n", globals.CommitDate)
internal/cli/version_test.go
+4 -4
View File
@@ -34,9 +34,9 @@ func runVersionCommand(t *testing.T) string {
// the report is the version the binary was actually built with. The
// test binary carries no -ldflags, so that is the "dev" default -- the
// same string an untagged `make vaultik` build stamps a prefix of.
//
//nolint:paralleltest // executes a command that reads the global rootFlags
func TestVersionCommandReportsBuildVersion(t *testing.T) {
t.Parallel()
out := runVersionCommand(t)
wantFirst := "vaultik " + globals.Version
@@ -55,9 +55,9 @@ func TestVersionCommandReportsBuildVersion(t *testing.T) {
// being exactly "dev", so once untagged builds started carrying their
// commit sha it would have gone silent and an unreleased binary would
// have looked like a release.
//
//nolint:paralleltest // executes a command that reads the global rootFlags
func TestVersionCommandFlagsDevelopmentBuild(t *testing.T) {
t.Parallel()
if !globals.IsDevVersion(globals.Version) {
t.Skipf("test binary was stamped with release version %q",
globals.Version)
internal/config/config.go
+131 -46
View File
@@ -16,34 +16,50 @@ import (
"github.com/adrg/xdg"
"go.uber.org/fx"
"gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/chunker"
"sneak.berlin/go/vaultik/internal/log"
)
const appName = "vaultik"
// secretKeyPrefix marks an age secret (private) key. It is compared
// case-insensitively so a recipient entry that is actually a private key is
// caught and never passed to age or echoed back.
//
//nolint:gosec // G101: marker for detecting a pasted secret key, not a credential
const secretKeyPrefix = "AGE-SECRET-KEY-"
// Defaults and validation bounds for tunable settings.
const (
defaultBlobSizeLimit = Size(10 * 1024 * 1024 * 1024) // 10GB
defaultChunkSize = Size(10 * 1024 * 1024) // 10MB
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MB
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MiB
defaultCompressionLevel = 3
minChunkSize = 1024 * 1024 // 1MB
minCompressionLevel = 1
maxCompressionLevel = 19
// S3 accepts a multipart upload part from 5MiB to 5GiB.
minS3PartSize = 5 * 1024 * 1024
maxS3PartSize = 5 * 1024 * 1024 * 1024
)
// Sentinel validation errors.
var (
errNoConfigPath = errors.New("config path not provided")
errNoAgeRecipients = errors.New(
"at least one age_recipient is required (generate with: age-keygen)")
errNoConfigPath = errors.New("config path not provided")
errRecipientIsSecretKey = errors.New(
"an age secret key was given where a public key (age1...) belongs")
errRecipientNotX25519 = errors.New(
"not a valid recipient; only X25519 age1... public keys are supported")
errNoSnapshots = errors.New(
"at least one snapshot must be configured (see config.example.yml)")
errSnapshotNoPaths = errors.New("snapshot must have at least one path")
errChunkSizeTooSmall = errors.New("chunk_size must be at least 1MB")
errBlobSizeTooSmall = errors.New("blob_size_limit must be at least chunk_size")
errBadCompression = errors.New("compression_level must be between 1 and 19")
errBadStorageScheme = errors.New(
errBlobSizeTooSmall = errors.New(
"blob_size_limit must be at least the largest chunk the chunker can " +
"emit (chunk_size times the FastCDC size spread)")
errBadCompression = errors.New("compression_level must be between 1 and 19")
errBadS3PartSize = errors.New("s3.part_size must be between 5MiB and 5GiB")
errBadStorageScheme = errors.New(
"storage_url must start with s3://, file://, or rclone://")
errStorageNotConfigured = errors.New(
"storage not configured; set storage_url or provide s3.endpoint + " +
@@ -121,6 +137,26 @@ func (c *Config) SnapshotNames() []string {
return names
}
// Names of the two places the age secret key can be configured, used by
// AgeSecretKeySourceName for error messages that must not echo the value.
//
//nolint:gosec // G101: these are the names of the config sources, not a key
const (
ageSecretKeySourceEnv = "VAULTIK_AGE_SECRET_KEY"
ageSecretKeySourceConfig = "age_secret_key"
)
// AgeSecretKeySourceName returns the human name of where AgeSecretKey was
// configured. A Config built directly (as in tests) has no recorded
// source, so it reports the config-file field name.
func (c *Config) AgeSecretKeySourceName() string {
if c.AgeSecretKeySource != "" {
return c.AgeSecretKeySource
}
return ageSecretKeySourceConfig
}
// Config represents the application configuration for Vaultik.
// It defines all settings for backup operations, including source directories,
// encryption recipients, storage configuration, and performance tuning parameters.
@@ -130,8 +166,13 @@ func (c *Config) SnapshotNames() []string {
type Config struct {
AgeRecipients []string `yaml:"age_recipients"`
AgeSecretKey string `yaml:"age_secret_key"`
BlobSizeLimit Size `yaml:"blob_size_limit"`
ChunkSize Size `yaml:"chunk_size"`
// AgeSecretKeySource names where AgeSecretKey was configured
// ("VAULTIK_AGE_SECRET_KEY" or "age_secret_key") so a later parse
// failure can name the source without echoing the secret value. It is
// set by Load and never read from or written to the config file.
AgeSecretKeySource string `yaml:"-"`
BlobSizeLimit Size `yaml:"blob_size_limit"`
ChunkSize Size `yaml:"chunk_size"`
// Exclude holds global excludes applied to all snapshots.
Exclude []string `yaml:"exclude"`
Hostname string `yaml:"hostname"`
@@ -162,8 +203,10 @@ type S3Config struct {
AccessKeyID string `yaml:"access_key_id"`
SecretAccessKey string `yaml:"secret_access_key"`
Region string `yaml:"region"`
UseSSL bool `yaml:"use_ssl"`
PartSize Size `yaml:"part_size"`
// UseSSL selects HTTPS for a scheme-less endpoint. Omitted (nil) means
// the default, TLS; set it to false only to force plain HTTP.
UseSSL *bool `yaml:"use_ssl"`
PartSize Size `yaml:"part_size"`
}
// Path wraps the config file path for fx dependency injection.
@@ -205,6 +248,7 @@ func Load(path string) (*Config, error) {
ChunkSize: defaultChunkSize,
IndexPath: filepath.Join(xdg.DataHome, appName, "index.sqlite"),
CompressionLevel: defaultCompressionLevel,
S3: S3Config{PartSize: defaultS3PartSize},
}
// Convert smartconfig data to YAML then unmarshal
@@ -238,10 +282,7 @@ func Load(path string) (*Config, error) {
cfg.IndexPath = expandTilde(envIndexPath)
}
// Check for environment variable override for AgeSecretKey
if envAgeSecretKey := os.Getenv("VAULTIK_AGE_SECRET_KEY"); envAgeSecretKey != "" {
cfg.AgeSecretKey = extractAgeSecretKey(envAgeSecretKey)
}
cfg.setAgeSecretKey()
// Get hostname if not set
if cfg.Hostname == "" {
@@ -258,17 +299,13 @@ func Load(path string) (*Config, error) {
cfg.S3.Region = "us-east-1"
}
if cfg.S3.PartSize == 0 {
cfg.S3.PartSize = defaultS3PartSize
}
// Check config file permissions (warn if world or group readable)
//nolint:gosec // G703: config path is operator-supplied by design
info, statErr := os.Stat(path)
if statErr == nil {
mode := info.Mode().Perm()
if mode&0044 != 0 { // group or world readable
log.Warn("Config file has insecure permissions (contains S3 credentials)",
log.Warn(cfg.readableByOthersWarning(),
"path", path,
"mode", fmt.Sprintf("%04o", mode),
"recommendation", "chmod 600 "+path)
@@ -285,16 +322,26 @@ func Load(path string) (*Config, error) {
// Validate checks if the configuration is valid and complete.
// It ensures all required fields are present and have valid values:
// - At least one age recipient must be specified
// - At least one snapshot must be configured with at least one path
// - Storage must be configured (either storage_url or s3.* fields)
// - Chunk size must be at least 1MB
// - Blob size limit must be at least the chunk size
// - Compression level must be between 1 and 19
// - Every age recipient must parse as an X25519 age1... public key (so a
// bad entry fails at load, not mid-backup); errors name the position,
// never the value. An empty list is accepted, because only snapshot
// create needs a recipient and it checks for one itself
// - At least one snapshot must be configured with at least one path
// - Storage must be configured (either storage_url or s3.* fields)
// - Chunk size must be at least 1MB
// - Blob size limit must be at least the largest chunk the chunker can emit
// (chunk_size times chunker.ChunkSizeSpread), so a single-chunk blob never
// exceeds the configured limit
// - Compression level must be between 1 and 19
// - S3 part size must be between 5MiB and 5GiB, the part sizes S3 accepts
//
// Returns an error describing the first validation failure encountered.
func (c *Config) Validate() error {
if len(c.AgeRecipients) == 0 {
return errNoAgeRecipients
for i, recipient := range c.AgeRecipients {
err := validateAgeRecipient(recipient)
if err != nil {
return fmt.Errorf("age_recipients[%d]: %w", i, err)
}
}
if len(c.Snapshots) == 0 {
@@ -317,8 +364,13 @@ func (c *Config) Validate() error {
return errChunkSizeTooSmall
}
if c.BlobSizeLimit.Int64() < c.ChunkSize.Int64() {
return errBlobSizeTooSmall
// The chunker can emit chunks up to chunk_size * ChunkSizeSpread, and the
// packer places a single such chunk into an otherwise empty blob. A limit
// below that bound would let a blob exceed it, so reject it.
largestChunk := c.ChunkSize.Int64() * chunker.ChunkSizeSpread
if c.BlobSizeLimit.Int64() < largestChunk {
return fmt.Errorf("%w: need at least %d bytes",
errBlobSizeTooSmall, largestChunk)
}
if c.CompressionLevel < minCompressionLevel ||
@@ -326,9 +378,58 @@ func (c *Config) Validate() error {
return errBadCompression
}
if c.S3.PartSize.Int64() < minS3PartSize ||
c.S3.PartSize.Int64() > maxS3PartSize {
return errBadS3PartSize
}
return nil
}
// validateAgeRecipient parses one age_recipients entry with the age library
// and returns a value-free error on failure. A recipient string can be
// sensitive (an operator may paste a secret key by mistake), so neither the
// entry nor age's own error (which quotes its input) is ever included.
func validateAgeRecipient(recipient string) error {
if strings.HasPrefix(strings.ToUpper(recipient), secretKeyPrefix) {
return errRecipientIsSecretKey
}
_, err := age.ParseX25519Recipient(recipient)
if err != nil {
return errRecipientNotX25519
}
return nil
}
// setAgeSecretKey records the age secret key and where it came from. The
// value is stored raw and parsed only where decryption happens
// (internal/vaultik), so backup, list and prune keep working whatever the
// field holds. The environment variable overrides the config-file field.
func (c *Config) setAgeSecretKey() {
if c.AgeSecretKey != "" {
c.AgeSecretKeySource = ageSecretKeySourceConfig
}
if env := os.Getenv("VAULTIK_AGE_SECRET_KEY"); env != "" {
c.AgeSecretKey = env
c.AgeSecretKeySource = ageSecretKeySourceEnv
}
}
// readableByOthersWarning is the warning Load logs when others can read
// the config file. It says "may contain" because the S3 credentials are
// seen only after smartconfig has replaced any ${...} reference in the
// file with its value, so a set credential need not be in the file.
func (c *Config) readableByOthersWarning() string {
if c.S3.AccessKeyID != "" || c.S3.SecretAccessKey != "" {
return "Config file is readable by others and may contain S3 credentials"
}
return "Config file is readable by others"
}
// validateStorage validates storage configuration.
// If StorageURL is set, it takes precedence. S3 URLs require credentials.
// File URLs don't require any S3 configuration.
@@ -385,22 +486,6 @@ func (c *Config) validateStorageURL() error {
}
}
// extractAgeSecretKey extracts the AGE-SECRET-KEY from the input using
// the age library's parser, which handles comments and whitespace.
func extractAgeSecretKey(input string) string {
identities, err := age.ParseIdentities(strings.NewReader(input))
if err != nil || len(identities) == 0 {
// Fall back to trimmed input if parsing fails
return strings.TrimSpace(input)
}
// Return the string representation of the first identity
if id, ok := identities[0].(*age.X25519Identity); ok {
return id.String()
}
return strings.TrimSpace(input)
}
// Module exports the config module for fx dependency injection.
// It provides the Config type to other modules in the application.
//
internal/config/config_test.go
+458 -39
View File
@@ -1,9 +1,14 @@
package config //nolint:testpackage // exercises unexported extractAgeSecretKey
package config //nolint:testpackage // exercises unexported source constants
import (
"errors"
"os"
"path/filepath"
"strings"
"testing"
"sneak.berlin/go/vaultik/internal/chunker"
"sneak.berlin/go/vaultik/internal/log"
)
const (
@@ -83,6 +88,48 @@ func TestConfigLoad(t *testing.T) {
}
}
// TestExampleConfigIsScrubbedAndLoads checks that the shipped
// config.example.yml carries only neutral placeholders (no real credentials,
// private addresses, or internal host names) and still parses.
func TestExampleConfigIsScrubbedAndLoads(t *testing.T) {
t.Parallel()
examplePath := filepath.Join("..", "..", "config.example.yml")
cfg, err := Load(examplePath)
if err != nil {
t.Fatalf("Failed to load config.example.yml: %v", err)
}
if cfg.StorageURL != "rclone://myremote/path/to/backups" {
t.Errorf("Expected neutral storage_url, got '%s'", cfg.StorageURL)
}
//nolint:gosec // G304: examplePath is a fixed in-repo path, not user input
raw, err := os.ReadFile(examplePath)
if err != nil {
t.Fatalf("Failed to read config.example.yml: %v", err)
}
text := string(raw)
wantSubstrings := []string{
"YOUR_ACCESS_KEY",
"YOUR_SECRET_KEY",
"endpoint: https://",
}
for _, want := range wantSubstrings {
if !strings.Contains(text, want) {
t.Errorf("Expected config.example.yml to contain %q", want)
}
}
// A raw "http://" scheme would mean a plaintext, likely private endpoint.
if strings.Contains(text, "http://") {
t.Error("config.example.yml should not contain an http:// endpoint")
}
}
// TestConfigFromEnv tests loading config path from environment variable
func TestConfigFromEnv(t *testing.T) {
t.Parallel()
@@ -101,53 +148,58 @@ func TestConfigFromEnv(t *testing.T) {
}
}
// TestExtractAgeSecretKey tests extraction of AGE-SECRET-KEY from various inputs
func TestExtractAgeSecretKey(t *testing.T) {
// TestValidateBlobSizeLimit checks the blob_size_limit boundary: it must be at
// least the largest chunk the chunker can emit (chunk_size times
// chunker.ChunkSizeSpread), because the packer places a single such chunk into
// an otherwise empty blob. A limit between chunk_size and that bound is rejected.
func TestValidateBlobSizeLimit(t *testing.T) {
t.Parallel()
const chunkSize = Size(10 * 1024 * 1024) // 10MB
largestChunk := chunkSize.Int64() * chunker.ChunkSizeSpread
newConfig := func(blobLimit Size) *Config {
return &Config{
AgeRecipients: []string{testSneakAgePublicKey},
Snapshots: map[string]SnapshotConfig{"test": {Paths: []string{"/tmp/src"}}},
StorageURL: "file:///tmp/vaultik-test-store",
ChunkSize: chunkSize,
BlobSizeLimit: blobLimit,
CompressionLevel: 3,
S3: S3Config{PartSize: defaultS3PartSize},
}
}
tests := []struct {
name string
input string
expected string
name string
blobLimit Size
wantErr bool
}{
{
name: "plain key",
input: testIntegrationAgePrivateKey,
expected: testIntegrationAgePrivateKey,
name: "at chunk_size but below largest chunk is rejected",
blobLimit: chunkSize,
wantErr: true,
},
{
name: "key with trailing newline",
input: testIntegrationAgePrivateKey + "\n",
expected: testIntegrationAgePrivateKey,
name: "between chunk_size and largest chunk is rejected",
blobLimit: Size(chunkSize.Int64() * 2),
wantErr: true,
},
{
name: "full age-keygen output",
input: "# created: 2025-01-14T12:00:00Z\n" +
"# public key: " + testIntegrationAgePublicKey + "\n" +
testIntegrationAgePrivateKey + "\n",
expected: testIntegrationAgePrivateKey,
name: "one byte below largest chunk is rejected",
blobLimit: Size(largestChunk - 1),
wantErr: true,
},
{
name: "age-keygen output with extra blank lines",
input: "# created: 2025-01-14T12:00:00Z\n" +
"# public key: " + testIntegrationAgePublicKey + "\n\n" +
testIntegrationAgePrivateKey + "\n\n",
expected: testIntegrationAgePrivateKey,
name: "exactly at largest chunk is accepted",
blobLimit: Size(largestChunk),
wantErr: false,
},
{
name: "key with leading whitespace",
input: " " + testIntegrationAgePrivateKey + " ",
expected: testIntegrationAgePrivateKey,
},
{
name: "empty input",
input: "",
expected: "",
},
{
name: "only comments",
input: "# this is a comment\n# another comment",
expected: "# this is a comment\n# another comment",
name: "above largest chunk is accepted",
blobLimit: Size(largestChunk * 100),
wantErr: false,
},
}
@@ -155,10 +207,377 @@ func TestExtractAgeSecretKey(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
result := extractAgeSecretKey(tt.input)
if result != tt.expected {
t.Errorf("extractAgeSecretKey(%q) = %q, want %q",
tt.input, result, tt.expected)
err := newConfig(tt.blobLimit).Validate()
if tt.wantErr {
if !errors.Is(err, errBlobSizeTooSmall) {
t.Fatalf("Validate() error = %v, want errBlobSizeTooSmall", err)
}
return
}
if err != nil {
t.Fatalf("Validate() unexpected error: %v", err)
}
})
}
}
// TestValidateS3PartSize checks that s3.part_size is held to the part sizes
// S3 accepts, 5MiB to 5GiB, by changing only the part size of the test
// config. "5MB" in the config file is 5,000,000 bytes, below the minimum.
func TestValidateS3PartSize(t *testing.T) {
t.Parallel()
base, err := Load(os.Getenv("VAULTIK_CONFIG"))
if err != nil {
t.Fatalf("Failed to load config: %v", err)
}
tests := []struct {
name string
partSize Size
wantErr bool
}{
{
name: "5MB is rejected",
partSize: 5_000_000,
wantErr: true,
},
{
name: "one byte below 5MiB is rejected",
partSize: minS3PartSize - 1,
wantErr: true,
},
{
name: "5MiB is accepted",
partSize: minS3PartSize,
wantErr: false,
},
{
name: "5GiB is accepted",
partSize: maxS3PartSize,
wantErr: false,
},
{
name: "one byte above 5GiB is rejected",
partSize: maxS3PartSize + 1,
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
cfg := *base
cfg.S3.PartSize = tt.partSize
err := cfg.Validate()
if tt.wantErr {
if !errors.Is(err, errBadS3PartSize) {
t.Fatalf("Validate() error = %v, want errBadS3PartSize", err)
}
return
}
if err != nil {
t.Fatalf("Validate() unexpected error: %v", err)
}
})
}
}
// TestLoadS3PartSize checks that a config file without s3.part_size loads
// with the 5MiB default, and that an explicit 0 fails at load like any other
// part size S3 refuses.
func TestLoadS3PartSize(t *testing.T) {
t.Parallel()
const withoutPartSize = "snapshots:\n" +
" test:\n" +
" paths: [/tmp/vaultik-test-source]\n" +
"storage_url: file:///tmp/vaultik-test-storage\n"
writeConfig := func(t *testing.T, text string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(text), 0o600)
if err != nil {
t.Fatalf("write config: %v", err)
}
return path
}
t.Run("absent loads as 5MiB", func(t *testing.T) {
t.Parallel()
cfg, err := Load(writeConfig(t, withoutPartSize))
if err != nil {
t.Fatalf("Load() unexpected error: %v", err)
}
if cfg.S3.PartSize != defaultS3PartSize {
t.Errorf("s3.part_size = %d, want %d",
cfg.S3.PartSize, defaultS3PartSize)
}
})
t.Run("0 is rejected", func(t *testing.T) {
t.Parallel()
_, err := Load(writeConfig(t, withoutPartSize+"s3:\n part_size: 0\n"))
if !errors.Is(err, errBadS3PartSize) {
t.Fatalf("Load() error = %v, want errBadS3PartSize", err)
}
})
}
// TestValidateAgeRecipients checks that recipients are parsed at config load
// (a bad entry fails immediately, not mid-backup) and that no invalid entry —
// least of all a pasted secret key — is echoed in the error. An empty list
// loads, because only snapshot create needs a recipient.
func TestValidateAgeRecipients(t *testing.T) {
t.Parallel()
baseConfig := func(recipients []string) *Config {
return &Config{
AgeRecipients: recipients,
Snapshots: map[string]SnapshotConfig{"test": {Paths: []string{"/tmp/src"}}},
StorageURL: "file:///tmp/vaultik-test-store",
ChunkSize: Size(10 * 1024 * 1024),
BlobSizeLimit: Size(10 * 1024 * 1024 * 1024),
CompressionLevel: 3,
S3: S3Config{PartSize: defaultS3PartSize},
}
}
tests := []struct {
name string
recipients []string
wantErr bool
}{
{
name: "no recipients is accepted",
recipients: nil,
wantErr: false,
},
{
name: "placeholder recipient is rejected",
recipients: []string{"age1REPLACE_WITH_YOUR_PUBLIC_KEY"},
wantErr: true,
},
{
name: "ssh-ed25519 recipient is rejected",
recipients: []string{"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIexamplekeydata"},
wantErr: true,
},
{
name: "truncated age1 string is rejected",
recipients: []string{"age1short"},
wantErr: true,
},
{
name: "secret key passed as recipient is rejected",
recipients: []string{testIntegrationAgePrivateKey},
wantErr: true,
},
{
name: "two valid recipients are accepted",
recipients: []string{testSneakAgePublicKey, testIntegrationAgePublicKey},
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
err := baseConfig(tt.recipients).Validate()
if !tt.wantErr {
if err != nil {
t.Fatalf("Validate() unexpected error: %v", err)
}
return
}
if err == nil {
t.Fatal("Validate() returned nil, want error")
}
// The entry itself must never appear in the error, since a
// recipient string can be a secret key.
for _, recipient := range tt.recipients {
if strings.Contains(err.Error(), recipient) {
t.Fatalf("Validate() error echoed the recipient value: %v", err)
}
}
})
}
}
// TestAgeSecretKeySourceName checks the name reported for the configured
// age secret key: the recorded source when Load set one, and the
// config-file field name for a Config built directly (as in tests).
func TestAgeSecretKeySourceName(t *testing.T) {
t.Parallel()
tests := []struct {
name string
source string
want string
}{
{
name: "unset defaults to config field",
source: "",
want: ageSecretKeySourceConfig,
},
{
name: "environment source",
source: ageSecretKeySourceEnv,
want: ageSecretKeySourceEnv,
},
{
name: "config-file source",
source: ageSecretKeySourceConfig,
want: ageSecretKeySourceConfig,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
cfg := &Config{AgeSecretKeySource: tt.source}
if got := cfg.AgeSecretKeySourceName(); got != tt.want {
t.Errorf("AgeSecretKeySourceName() = %q, want %q", got, tt.want)
}
})
}
}
// loadReadableConfig writes configYAML to a file that others can read,
// loads it, and returns what the logger wrote to stderr meanwhile. The
// logger writes to the os.Stderr it finds when it is initialized, so
// os.Stderr is pointed at a file first. Not parallel-safe: os.Stderr and
// the logger are process-global.
func loadReadableConfig(t *testing.T, configYAML string) string {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
stderrPath := filepath.Join(dir, "stderr")
err := os.WriteFile(configPath, []byte(configYAML), 0o600)
if err != nil {
t.Fatalf("writing config: %v", err)
}
//nolint:gosec // G302: the test needs a config file others can read
err = os.Chmod(configPath, 0o644)
if err != nil {
t.Fatalf("chmod config: %v", err)
}
stderrFile, err := os.Create(stderrPath) //nolint:gosec // G304: test temp path
if err != nil {
t.Fatalf("creating stderr file: %v", err)
}
previous := os.Stderr
os.Stderr = stderrFile
log.Initialize(log.Config{})
_, loadErr := Load(configPath)
os.Stderr = previous
log.Initialize(log.Config{})
_ = stderrFile.Close()
if loadErr != nil {
t.Fatalf("Load() error = %v", loadErr)
}
captured, err := os.ReadFile(stderrPath) //nolint:gosec // G304: test temp path
if err != nil {
t.Fatalf("reading stderr file: %v", err)
}
return string(captured)
}
// TestLoadWarnsReadableConfigWithoutS3Credentials checks that a config
// file others can read, holding no S3 credentials, is warned about
// without a claim that it holds them.
//
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
func TestLoadWarnsReadableConfigWithoutS3Credentials(t *testing.T) {
stderr := loadReadableConfig(t, `
storage_url: file:///var/backups/vaultik
snapshots:
home:
paths:
- /home
`)
if !strings.Contains(stderr, "Config file is readable by others") {
t.Errorf("expected a warning that the file is readable by others, got %q",
stderr)
}
if strings.Contains(stderr, "S3 credentials") {
t.Errorf("warning names S3 credentials the file does not set: %q", stderr)
}
}
// TestLoadWarnsReadableConfigWithS3Credentials checks that a config file
// others can read and that sets S3 credentials, as values or as ${ENV:...}
// references, is warned about as one that may contain them.
//
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
func TestLoadWarnsReadableConfigWithS3Credentials(t *testing.T) {
t.Setenv("VAULTIK_TEST_ACCESS_KEY_ID", "test-access-key")
t.Setenv("VAULTIK_TEST_SECRET_ACCESS_KEY", "test-secret-key")
configs := map[string]string{
"values": `
storage_url: s3://bucket/prefix?endpoint=s3.example.com
s3:
access_key_id: test-access-key
secret_access_key: test-secret-key
snapshots:
home:
paths:
- /home
`,
"references": `
storage_url: s3://bucket/prefix?endpoint=s3.example.com
s3:
access_key_id: ${ENV:VAULTIK_TEST_ACCESS_KEY_ID}
secret_access_key: ${ENV:VAULTIK_TEST_SECRET_ACCESS_KEY}
snapshots:
home:
paths:
- /home
`,
}
for name, configYAML := range configs {
t.Run(name, func(t *testing.T) {
stderr := loadReadableConfig(t, configYAML)
if !strings.Contains(stderr,
"Config file is readable by others and may contain S3 credentials") {
t.Errorf("expected a warning naming the S3 credentials, got %q",
stderr)
}
})
}
internal/config/size.go
-2
View File
@@ -16,8 +16,6 @@ var (
// Size represents a byte size that can be specified in configuration files.
// It can unmarshal from both numeric values (interpreted as bytes) and
// human-readable strings like "10MB", "2.5GB", or "1TB".
//
//nolint:recvcheck // UnmarshalYAML requires a pointer; String/Int64 are value reads
type Size int64
// UnmarshalYAML implements yaml.Unmarshaler for Size, allowing it to be
internal/crypto/encryption.go
-224
View File
@@ -1,224 +0,0 @@
// Package crypto provides thread-safe age encryption and decryption
// helpers used to protect blob and metadata content.
package crypto //nolint:revive,nolintlint // stdlib crypto unused; see #76
import (
"bytes"
"errors"
"fmt"
"io"
"sync"
"filippo.io/age"
"go.uber.org/fx"
)
// ErrNoRecipients is returned when an encryptor is created or updated
// without any recipient public keys.
var ErrNoRecipients = errors.New("at least one recipient is required")
// Encryptor provides thread-safe encryption using the age encryption library.
// It supports encrypting data for multiple recipients simultaneously, allowing
// any of the corresponding private keys to decrypt the data. This is useful
// for backup scenarios where multiple parties should be able to decrypt the data.
type Encryptor struct {
recipients []age.Recipient
mu sync.RWMutex
}
// NewEncryptor creates a new encryptor with the given age public keys.
// Each public key should be a valid age X25519 recipient string (e.g., "age1...")
// At least one recipient must be provided. Returns an error if any of the
// public keys are invalid or if no recipients are specified.
func NewEncryptor(publicKeys []string) (*Encryptor, error) {
if len(publicKeys) == 0 {
return nil, ErrNoRecipients
}
recipients := make([]age.Recipient, 0, len(publicKeys))
for _, key := range publicKeys {
recipient, err := age.ParseX25519Recipient(key)
if err != nil {
return nil, fmt.Errorf("parsing age recipient %s: %w", key, err)
}
recipients = append(recipients, recipient)
}
return &Encryptor{
recipients: recipients,
}, nil
}
// Encrypt encrypts data using age encryption for all configured recipients.
// The encrypted data can be decrypted by any of the corresponding private keys.
// This method is suitable for small to medium amounts of data that fit in memory.
// For large data streams, use EncryptStream or EncryptWriter instead.
func (e *Encryptor) Encrypt(data []byte) ([]byte, error) {
e.mu.RLock()
recipients := e.recipients
e.mu.RUnlock()
var buf bytes.Buffer
// Create encrypted writer for all recipients
w, err := age.Encrypt(&buf, recipients...)
if err != nil {
return nil, fmt.Errorf("creating encrypted writer: %w", err)
}
// Write data
_, err = w.Write(data)
if err != nil {
return nil, fmt.Errorf("writing encrypted data: %w", err)
}
// Close to flush
err = w.Close()
if err != nil {
return nil, fmt.Errorf("closing encrypted writer: %w", err)
}
return buf.Bytes(), nil
}
// EncryptStream encrypts data from reader to writer using age encryption.
// This method is suitable for encrypting large files or streams as it processes
// data in a streaming fashion without loading everything into memory.
// The encrypted data is written directly to the destination writer.
func (e *Encryptor) EncryptStream(dst io.Writer, src io.Reader) error {
e.mu.RLock()
recipients := e.recipients
e.mu.RUnlock()
// Create encrypted writer for all recipients
w, err := age.Encrypt(dst, recipients...)
if err != nil {
return fmt.Errorf("creating encrypted writer: %w", err)
}
// Copy data
_, err = io.Copy(w, src)
if err != nil {
return fmt.Errorf("copying encrypted data: %w", err)
}
// Close to flush
err = w.Close()
if err != nil {
return fmt.Errorf("closing encrypted writer: %w", err)
}
return nil
}
// EncryptWriter creates a writer that encrypts data written to it.
// All data written to the returned WriteCloser will be encrypted and written
// to the destination writer. The caller must call Close() on the returned
// writer to ensure all encrypted data is properly flushed and finalized.
// This is useful for integrating encryption into existing writer-based pipelines.
func (e *Encryptor) EncryptWriter(dst io.Writer) (io.WriteCloser, error) {
e.mu.RLock()
recipients := e.recipients
e.mu.RUnlock()
// Create encrypted writer for all recipients
w, err := age.Encrypt(dst, recipients...)
if err != nil {
return nil, fmt.Errorf("creating encrypted writer: %w", err)
}
return w, nil
}
// UpdateRecipients updates the recipients for future encryption operations.
// This method is thread-safe and can be called while other encryption operations
// are in progress. Existing encryption operations will continue with the old
// recipients. At least one recipient must be provided. Returns an error if any
// of the public keys are invalid or if no recipients are specified.
func (e *Encryptor) UpdateRecipients(publicKeys []string) error {
if len(publicKeys) == 0 {
return ErrNoRecipients
}
recipients := make([]age.Recipient, 0, len(publicKeys))
for _, key := range publicKeys {
recipient, err := age.ParseX25519Recipient(key)
if err != nil {
return fmt.Errorf("parsing age recipient %s: %w", key, err)
}
recipients = append(recipients, recipient)
}
e.mu.Lock()
e.recipients = recipients
e.mu.Unlock()
return nil
}
// Decryptor provides thread-safe decryption using the age encryption library.
// It uses a private key to decrypt data that was encrypted for the corresponding
// public key.
type Decryptor struct {
identity age.Identity
mu sync.RWMutex
}
// NewDecryptor creates a new decryptor with the given age private key.
// The private key should be a valid age X25519 identity string.
// Returns an error if the private key is invalid.
func NewDecryptor(privateKey string) (*Decryptor, error) {
identity, err := age.ParseX25519Identity(privateKey)
if err != nil {
return nil, fmt.Errorf("parsing age identity: %w", err)
}
return &Decryptor{
identity: identity,
}, nil
}
// Decrypt decrypts data using age decryption.
// This method is suitable for small to medium amounts of data that fit in memory.
// For large data streams, use DecryptStream instead.
func (d *Decryptor) Decrypt(data []byte) ([]byte, error) {
d.mu.RLock()
identity := d.identity
d.mu.RUnlock()
r, err := age.Decrypt(bytes.NewReader(data), identity)
if err != nil {
return nil, fmt.Errorf("creating decrypted reader: %w", err)
}
decrypted, err := io.ReadAll(r)
if err != nil {
return nil, fmt.Errorf("reading decrypted data: %w", err)
}
return decrypted, nil
}
// DecryptStream returns a reader that decrypts data from the provided reader.
// This method is suitable for decrypting large files or streams as it processes
// data in a streaming fashion without loading everything into memory.
// The caller should close the input reader when done.
func (d *Decryptor) DecryptStream(src io.Reader) (io.Reader, error) {
d.mu.RLock()
identity := d.identity
d.mu.RUnlock()
r, err := age.Decrypt(src, identity)
if err != nil {
return nil, fmt.Errorf("creating decrypted reader: %w", err)
}
return r, nil
}
// Module exports the crypto module for fx dependency injection.
//
//nolint:gochecknoglobals // fx module definitions are package globals
var Module = fx.Module("crypto")
internal/crypto/encryption_test.go
-178
View File
@@ -1,178 +0,0 @@
package crypto_test
import (
"bytes"
"testing"
"filippo.io/age"
"sneak.berlin/go/vaultik/internal/crypto"
)
func TestEncryptor(t *testing.T) {
t.Parallel()
// Generate a test key pair
identity, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity: %v", err)
}
publicKey := identity.Recipient().String()
// Create encryptor
enc, err := crypto.NewEncryptor([]string{publicKey})
if err != nil {
t.Fatalf("failed to create encryptor: %v", err)
}
// Test data
plaintext := []byte("Hello, World! This is a test message.")
// Encrypt
ciphertext, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// Verify it's actually encrypted (should be larger and different)
if bytes.Equal(plaintext, ciphertext) {
t.Error("ciphertext equals plaintext")
}
// Decrypt to verify
r, err := age.Decrypt(bytes.NewReader(ciphertext), identity)
if err != nil {
t.Fatalf("failed to decrypt: %v", err)
}
var decrypted bytes.Buffer
_, err = decrypted.ReadFrom(r)
if err != nil {
t.Fatalf("failed to read decrypted data: %v", err)
}
if !bytes.Equal(plaintext, decrypted.Bytes()) {
t.Error("decrypted data doesn't match original")
}
}
func TestEncryptorMultipleRecipients(t *testing.T) {
t.Parallel()
// Generate three test key pairs
identity1, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity1: %v", err)
}
identity2, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity2: %v", err)
}
identity3, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity3: %v", err)
}
publicKeys := []string{
identity1.Recipient().String(),
identity2.Recipient().String(),
identity3.Recipient().String(),
}
// Create encryptor with multiple recipients
enc, err := crypto.NewEncryptor(publicKeys)
if err != nil {
t.Fatalf("failed to create encryptor: %v", err)
}
// Test data
plaintext := []byte("Secret message for multiple recipients")
// Encrypt
ciphertext, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// Verify each recipient can decrypt
identities := []age.Identity{identity1, identity2, identity3}
for i, identity := range identities {
r, err := age.Decrypt(bytes.NewReader(ciphertext), identity)
if err != nil {
t.Fatalf("recipient %d failed to decrypt: %v", i+1, err)
}
var decrypted bytes.Buffer
_, err = decrypted.ReadFrom(r)
if err != nil {
t.Fatalf("recipient %d failed to read decrypted data: %v", i+1, err)
}
if !bytes.Equal(plaintext, decrypted.Bytes()) {
t.Errorf("recipient %d: decrypted data doesn't match original", i+1)
}
}
}
func TestEncryptorUpdateRecipients(t *testing.T) {
t.Parallel()
// Generate two identities
identity1, _ := age.GenerateX25519Identity()
identity2, _ := age.GenerateX25519Identity()
publicKey1 := identity1.Recipient().String()
publicKey2 := identity2.Recipient().String()
// Create encryptor with first key
enc, err := crypto.NewEncryptor([]string{publicKey1})
if err != nil {
t.Fatalf("failed to create encryptor: %v", err)
}
// Encrypt with first key
plaintext := []byte("test data")
ciphertext1, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// Update to second key
err = enc.UpdateRecipients([]string{publicKey2})
if err != nil {
t.Fatalf("failed to update recipients: %v", err)
}
// Encrypt with second key
ciphertext2, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// First ciphertext should only decrypt with first identity
_, err = age.Decrypt(bytes.NewReader(ciphertext1), identity1)
if err != nil {
t.Error("failed to decrypt with identity1")
}
_, err = age.Decrypt(bytes.NewReader(ciphertext1), identity2)
if err == nil {
t.Error("should not decrypt with identity2")
}
// Second ciphertext should only decrypt with second identity
_, err = age.Decrypt(bytes.NewReader(ciphertext2), identity2)
if err != nil {
t.Error("failed to decrypt with identity2")
}
_, err = age.Decrypt(bytes.NewReader(ciphertext2), identity1)
if err == nil {
t.Error("should not decrypt with identity1")
}
}
internal/database/chunk_files.go
+1 -1
View File
@@ -220,7 +220,7 @@ func (r *ChunkFileRepository) CreateBatch(
cf.ChunkHash.String(), cf.FileID.String(), cf.FileOffset, cf.Length)
}
query += querySb183.String() //nolint:gosec // G202: appends "?" placeholders only
query += querySb183.String()
query += " ON CONFLICT(chunk_hash, file_id) DO NOTHING"
internal/database/chunks.go
+1 -1
View File
@@ -98,7 +98,7 @@ func (r *ChunkRepository) GetByHashes(
args[i] = hash
}
query += querySb75.String() //nolint:gosec // G202: appends "?" placeholders only
query += querySb75.String()
query += ") ORDER BY chunk_hash"
internal/database/database.go
+173 -46
View File
@@ -3,8 +3,10 @@
//
// Blobs in Vaultik are the final storage units uploaded to S3. Each blob is a
// large (up to 10GB) file containing many compressed and encrypted chunks from
// multiple source files. Blobs are content-addressed, meaning their filename
// is derived from their SHA256 hash after compression and encryption.
// multiple source files. Blobs are content-addressed: the filename in S3 is
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the chunk data
// before compression and encryption (not from the stored bytes). See
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
//
// Schema is managed via numbered SQL migrations embedded in the schema/
// directory. Migration 000.sql bootstraps the schema_migrations tracking
@@ -17,6 +19,7 @@ import (
"embed"
"errors"
"fmt"
"net/url"
"os"
"path/filepath"
"sort"
@@ -39,6 +42,10 @@ var schemaFS embed.FS
// table itself. It is applied before the normal migration loop.
const bootstrapVersion = 0
// busyTimeoutMs is how long a connection to the index waits for another
// connection's lock before failing with "database is locked".
const busyTimeoutMs = 10000
// DB represents the Vaultik local index database connection.
// It uses SQLite to track file metadata, content-defined chunks, and blob associations.
// The database enables incremental backups by detecting changed files and
@@ -91,10 +98,27 @@ func ParseMigrationVersion(filename string) (int, error) {
return version, nil
}
// indexDSN returns the driver DSN that opens the database at path. The
// driver runs each _pragma parameter on every connection it opens and drops
// any parameter it does not know without an error, so a setting written in
// another form silently does nothing. In WAL mode one connection can write
// while others read; the busy timeout makes a connection wait for a lock
// instead of failing at once.
func indexDSN(path string) string {
return fmt.Sprintf(
"%s?_pragma=busy_timeout(%d)&_pragma=journal_mode(WAL)"+
"&_pragma=synchronous(NORMAL)&_pragma=foreign_keys(1)",
path, busyTimeoutMs)
}
// New creates a new database connection at the specified path.
// It creates the schema if needed and configures SQLite with WAL mode for
// better concurrency. SQLite handles crash recovery automatically when
// opening a database with journal/WAL files present.
// It creates the schema if needed. Every connection runs in WAL mode with
// a busy timeout and foreign keys on (see indexDSN), so a read-only command
// can read the index while a backup writes to it. Committed rows can sit in
// the -wal file beside the database until a checkpoint, so a copy of the
// database file alone may miss them.
// SQLite handles crash recovery automatically when opening a database with
// journal/WAL files present.
// The path parameter can be a file path for persistent storage or ":memory:"
// for an in-memory database (useful for testing).
func New(ctx context.Context, path string) (*DB, error) {
@@ -107,11 +131,7 @@ func New(ctx context.Context, path string) (*DB, error) {
// First attempt with standard WAL mode
log.Debug("Attempting to open database with WAL mode", "path", path)
conn, err := sql.Open(
"sqlite",
path+"?_journal_mode=WAL&_synchronous=NORMAL&_busy_timeout=10000"+
"&_locking_mode=NORMAL&_foreign_keys=ON",
)
conn, err := sql.Open("sqlite", indexDSN(path))
if err == nil {
configureConnPool(conn)
@@ -131,8 +151,8 @@ func New(ctx context.Context, path string) (*DB, error) {
_ = conn.Close()
}
// If first attempt failed, try with TRUNCATE mode to clear any locks
return openWithRecovery(ctx, path)
// If the first attempt failed, try once more
return retryOpen(ctx, path)
}
// configureConnPool serializes all database access through one connection.
@@ -144,18 +164,12 @@ func configureConnPool(conn *sql.DB) {
conn.SetMaxIdleConns(1)
}
// finishOpen enables foreign keys, wraps the connection, and applies any
// pending migrations. On migration failure the connection is closed.
// finishOpen wraps the connection and applies any pending migrations. On
// migration failure the connection is closed.
func finishOpen(ctx context.Context, conn *sql.DB, path string) (*DB, error) {
// Enable foreign keys explicitly
_, err := conn.ExecContext(ctx, "PRAGMA foreign_keys = ON")
if err != nil {
log.Warn("Failed to enable foreign keys", "path", path, "error", err)
}
db := &DB{conn: conn, path: path}
err = applyMigrations(ctx, conn)
err := applyMigrations(ctx, conn)
if err != nil {
_ = conn.Close()
@@ -165,21 +179,15 @@ func finishOpen(ctx context.Context, conn *sql.DB, path string) (*DB, error) {
return db, nil
}
// openWithRecovery retries opening the database in TRUNCATE journal mode to
// clear stale locks, then switches back to WAL mode.
func openWithRecovery(ctx context.Context, path string) (*DB, error) {
log.Info(
"Database appears locked, attempting recovery with TRUNCATE mode",
"path", path,
)
// retryOpen makes a second attempt to open the database, with the same
// settings, after the first attempt failed, for example because another
// process held a lock for longer than the busy timeout.
func retryOpen(ctx context.Context, path string) (*DB, error) {
log.Info("Database appears locked, retrying open", "path", path)
conn, err := sql.Open(
"sqlite",
path+"?_journal_mode=TRUNCATE&_synchronous=NORMAL&_busy_timeout=10000"+
"&_foreign_keys=ON",
)
conn, err := sql.Open("sqlite", indexDSN(path))
if err != nil {
return nil, fmt.Errorf("opening database in recovery mode: %w", err)
return nil, fmt.Errorf("opening database on retry: %w", err)
}
configureConnPool(conn)
@@ -187,28 +195,18 @@ func openWithRecovery(ctx context.Context, path string) (*DB, error) {
err = conn.PingContext(ctx)
if err != nil {
log.Debug(
"Failed to ping database in recovery mode, closing",
"Failed to ping database on retry, closing",
"path", path, "error", err,
)
_ = conn.Close()
return nil, fmt.Errorf(
"database still locked after recovery attempt: %w",
"database still locked on retry: %w",
err,
)
}
log.Debug("Database opened in TRUNCATE mode", "path", path)
// Switch back to WAL mode
log.Debug("Switching database back to WAL mode", "path", path)
_, err = conn.ExecContext(ctx, "PRAGMA journal_mode=WAL")
if err != nil {
log.Warn("Failed to switch back to WAL mode", "path", path, "error", err)
}
db, err := finishOpen(ctx, conn, path)
if err != nil {
return nil, err
@@ -219,6 +217,135 @@ func openWithRecovery(ctx context.Context, path string) (*DB, error) {
return db, nil
}
// errUntrustedSnapshotSchema is returned when a downloaded snapshot
// database carries schema objects the real schema never defines, or is
// missing a table the restore and deep-verify queries read.
var errUntrustedSnapshotSchema = errors.New(
"downloaded snapshot database has an untrusted schema")
// snapshotReadOnlyDSN builds the driver DSN that opens a materialized
// snapshot database file read-only. mode=ro opens the file read-only at
// the OS level, query_only rejects any write the engine is asked to make,
// and trusted_schema=OFF refuses to run application code named in the
// schema. The file: URI form is required for the driver to honour the
// mode parameter.
func snapshotReadOnlyDSN(path string) string {
u := url.URL{
Scheme: "file",
Path: path,
RawQuery: "mode=ro&_pragma=query_only(true)&_pragma=trusted_schema(false)",
}
return u.String()
}
// OpenReadOnly opens an already-materialized SQLite file for read-only
// querying of a snapshot database downloaded from the store, used by
// restore and deep verify. Unlike New it never applies schema migrations
// and never writes: the connection is opened read-only with query_only
// and trusted_schema=OFF. It refuses any file whose schema carries a
// trigger, view or virtual table, or lacks an expected table, so a forged
// file cannot redefine what the restore queries return. The caller owns
// the file and must remove it.
func OpenReadOnly(ctx context.Context, path string) (*DB, error) {
conn, err := sql.Open("sqlite", snapshotReadOnlyDSN(path))
if err != nil {
return nil, fmt.Errorf("opening read-only database: %w", err)
}
configureConnPool(conn)
err = conn.PingContext(ctx)
if err != nil {
_ = conn.Close()
return nil, fmt.Errorf("opening read-only database: %w", err)
}
err = verifySnapshotSchema(ctx, conn)
if err != nil {
_ = conn.Close()
return nil, err
}
return &DB{conn: conn, path: path}, nil
}
// verifySnapshotSchema rejects a downloaded database whose schema is not
// the plain table set the real schema defines. Any trigger, view or
// virtual table, or a missing expected table, fails the open.
func verifySnapshotSchema(ctx context.Context, conn *sql.DB) error {
// expectedSnapshotTables are the tables the restore and deep-verify
// queries read. A downloaded database missing any of them is not a
// genuine snapshot database and is refused.
expectedSnapshotTables := []string{
"blob_chunks",
"blobs",
"chunks",
"file_chunks",
"files",
}
rows, err := conn.QueryContext(
ctx, "SELECT type, name, sql FROM sqlite_master")
if err != nil {
return fmt.Errorf("reading snapshot schema: %w", err)
}
defer func() { _ = rows.Close() }()
present := make(map[string]struct{})
for rows.Next() {
var objType, name string
var objSQL sql.NullString
err = rows.Scan(&objType, &name, &objSQL)
if err != nil {
return fmt.Errorf("reading snapshot schema: %w", err)
}
switch objType {
case "trigger", "view":
return fmt.Errorf(
"%w: unexpected %s %q", errUntrustedSnapshotSchema, objType, name)
case "table":
if isVirtualTableSQL(objSQL.String) {
return fmt.Errorf(
"%w: unexpected virtual table %q",
errUntrustedSnapshotSchema, name)
}
present[name] = struct{}{}
}
}
err = rows.Err()
if err != nil {
return fmt.Errorf("reading snapshot schema: %w", err)
}
for _, table := range expectedSnapshotTables {
if _, ok := present[table]; !ok {
return fmt.Errorf(
"%w: missing table %q", errUntrustedSnapshotSchema, table)
}
}
return nil
}
// isVirtualTableSQL reports whether a sqlite_master row's SQL defines a
// virtual table. Virtual tables are recorded with type 'table' but a
// "CREATE VIRTUAL TABLE" definition and can run module code, so they are
// refused alongside triggers and views.
func isVirtualTableSQL(createSQL string) bool {
return strings.HasPrefix(
strings.ToUpper(strings.TrimSpace(createSQL)), "CREATE VIRTUAL TABLE")
}
// NewTestDB creates an in-memory SQLite database for testing purposes.
// The database is automatically initialized with the schema and is ready
// for use. Each call creates a new independent database instance.
internal/database/database_test.go
+83
View File
@@ -120,6 +120,89 @@ func TestDatabaseConcurrentAccess(t *testing.T) {
}
}
// TestNewSetsJournalModeAndBusyTimeout checks that the connection settings
// New passes reach SQLite. The driver drops a setting it does not recognise
// without an error, so only reading the value back shows it took effect.
func TestNewSetsJournalModeAndBusyTimeout(t *testing.T) {
t.Parallel()
ctx := context.Background()
db, err := New(ctx, filepath.Join(t.TempDir(), "index.db"))
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
var journalMode string
err = db.conn.QueryRowContext(ctx, "PRAGMA journal_mode").Scan(&journalMode)
if err != nil {
t.Fatalf("reading journal_mode: %v", err)
}
if journalMode != "wal" {
t.Errorf("journal_mode = %q, want %q", journalMode, "wal")
}
var busyTimeout int
err = db.conn.QueryRowContext(ctx, "PRAGMA busy_timeout").Scan(&busyTimeout)
if err != nil {
t.Fatalf("reading busy_timeout: %v", err)
}
if busyTimeout != busyTimeoutMs {
t.Errorf("busy_timeout = %d, want %d", busyTimeout, busyTimeoutMs)
}
}
// TestNewWriteSucceedsWhileAnotherHandleReads opens the same index twice,
// as a read-only command does while a backup runs, and checks that a write
// on one handle commits while the other is in the middle of a read.
func TestNewWriteSucceedsWhileAnotherHandleReads(t *testing.T) {
t.Parallel()
ctx := context.Background()
dbPath := filepath.Join(t.TempDir(), "index.db")
reader, err := New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to open reading handle: %v", err)
}
defer func() { _ = reader.Close() }()
writer, err := New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to open writing handle: %v", err)
}
defer func() { _ = writer.Close() }()
// The read lock taken by the SELECT is held until the transaction ends.
readTx, err := reader.BeginTx(ctx, nil)
if err != nil {
t.Fatalf("beginning read transaction: %v", err)
}
defer func() { _ = readTx.Rollback() }()
var count int
err = readTx.QueryRowContext(ctx, "SELECT COUNT(*) FROM chunks").Scan(&count)
if err != nil {
t.Fatalf("reading chunks: %v", err)
}
_, err = writer.ExecWithLog(ctx,
"INSERT INTO chunks (chunk_hash, size) VALUES (?, ?)", "hash", 1024)
if err != nil {
t.Fatalf("write while another handle reads: %v", err)
}
}
func TestParseMigrationVersion(t *testing.T) {
t.Parallel()
internal/database/file_chunks.go
+1 -1
View File
@@ -253,7 +253,7 @@ func (r *FileChunkRepository) CreateBatch(
args = append(args, fc.FileID.String(), fc.Idx, fc.ChunkHash.String())
}
query += querySb211.String() //nolint:gosec // G202: appends "?" placeholders only
query += querySb211.String()
query += " ON CONFLICT(file_id, idx) DO NOTHING"
internal/database/files.go
+90 -26
View File
@@ -33,11 +33,13 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
}
query := `
INSERT INTO files (id, path, source_path, mtime, size, mode, uid, gid, link_target)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
INSERT INTO files
(id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(path) DO UPDATE SET
source_path = excluded.source_path,
mtime = excluded.mtime,
mtime_nsec = excluded.mtime_nsec,
size = excluded.size,
mode = excluded.mode,
uid = excluded.uid,
@@ -54,16 +56,19 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
if tx != nil {
LogSQL("Execute", query,
file.ID.String(), file.Path.String(), file.SourcePath.String(),
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
file.MTime.Unix(), file.MTime.Nanosecond(),
file.Size, file.Mode, file.UID, file.GID,
file.LinkTarget.String())
err = tx.QueryRowContext(ctx, query,
file.ID.String(), file.Path.String(), file.SourcePath.String(),
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
file.MTime.Unix(), file.MTime.Nanosecond(),
file.Size, file.Mode, file.UID, file.GID,
file.LinkTarget.String()).Scan(&idStr)
} else {
err = r.db.QueryRowWithLog(ctx, query,
file.ID.String(), file.Path.String(), file.SourcePath.String(),
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
file.MTime.Unix(), file.MTime.Nanosecond(),
file.Size, file.Mode, file.UID, file.GID,
file.LinkTarget.String()).Scan(&idStr)
}
@@ -84,7 +89,7 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
// in the index.
func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, error) {
query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
FROM files
WHERE path = ?
`
@@ -104,7 +109,7 @@ func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, err
// GetByID retrieves a file by its UUID
func (r *FileRepository) GetByID(ctx context.Context, id types.FileID) (*File, error) {
query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
FROM files
WHERE id = ?
`
@@ -127,7 +132,7 @@ func (r *FileRepository) GetByPathTx(
ctx context.Context, tx *sql.Tx, path string,
) (*File, error) {
query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
FROM files
WHERE path = ?
`
@@ -158,13 +163,14 @@ func (r *FileRepository) ListModifiedSince(
ctx context.Context, since time.Time,
) ([]*File, error) {
query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
FROM files
WHERE mtime >= ?
WHERE (mtime, mtime_nsec) >= (?, ?)
ORDER BY path
`
rows, err := r.db.conn.QueryContext(ctx, query, since.Unix())
rows, err := r.db.conn.QueryContext(ctx, query,
since.Unix(), since.Nanosecond())
if err != nil {
return nil, fmt.Errorf("querying files: %w", err)
}
@@ -228,19 +234,24 @@ func (r *FileRepository) DeleteByID(
return nil
}
// ListByPrefix returns all files whose path starts with prefix, ordered by
// path.
func (r *FileRepository) ListByPrefix(
ctx context.Context, prefix string,
// ListUnderPath returns the file at path and every file beneath it,
// ordered by path. Paths are compared case-sensitively, and a trailing
// slash on path is ignored, so "/" lists every file.
func (r *FileRepository) ListUnderPath(
ctx context.Context, path string,
) ([]*File, error) {
path = strings.TrimRight(path, "/")
dirPrefix := path + "/"
// LIKE would ignore ASCII case and treat _ and % in path as wildcards.
query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
FROM files
WHERE path LIKE ? || '%'
WHERE path = ? OR substr(path, 1, length(?)) = ?
ORDER BY path
`
rows, err := r.db.conn.QueryContext(ctx, query, prefix)
rows, err := r.db.conn.QueryContext(ctx, query, path, dirPrefix, dirPrefix)
if err != nil {
return nil, fmt.Errorf("querying files: %w", err)
}
@@ -266,10 +277,60 @@ func (r *FileRepository) ListByPrefix(
return files, rows.Err()
}
// ListIDsWithChunksNotInUploadedBlobs returns the IDs of the files whose
// path starts with prefix and that list at least one chunk held by no
// blob whose upload has completed (uploaded_ts set). A new snapshot
// cannot reference such a chunk, so a backup must not treat the file as
// unchanged even when its metadata matches the file on disk.
func (r *FileRepository) ListIDsWithChunksNotInUploadedBlobs(
ctx context.Context, prefix string,
) ([]types.FileID, error) {
query := `
SELECT DISTINCT f.id
FROM files f
JOIN file_chunks fc ON fc.file_id = f.id
WHERE f.path LIKE ? || '%'
AND NOT EXISTS (
SELECT 1
FROM blob_chunks bc
JOIN blobs b ON bc.blob_id = b.id
WHERE bc.chunk_hash = fc.chunk_hash
AND b.uploaded_ts IS NOT NULL
)
`
rows, err := r.db.conn.QueryContext(ctx, query, prefix)
if err != nil {
return nil, fmt.Errorf("querying files: %w", err)
}
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var ids []types.FileID
for rows.Next() {
var id types.FileID
err := rows.Scan(&id)
if err != nil {
return nil, fmt.Errorf("scanning file ID: %w", err)
}
ids = append(ids, id)
}
return ids, rows.Err()
}
// ListAll returns all files in the database
func (r *FileRepository) ListAll(ctx context.Context) ([]*File, error) {
query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
FROM files
ORDER BY path
`
@@ -310,7 +371,7 @@ func (r *FileRepository) CreateBatch(
}
// Each files row binds this many SQL variables.
const fileCols = 9
const fileCols = 10
// Batch at 100 rows to be safe with SQLite's variable limit.
const batchSize = 100
@@ -321,7 +382,7 @@ func (r *FileRepository) CreateBatch(
batch := files[i:end]
query := `INSERT INTO files
(id, path, source_path, mtime, size, mode, uid, gid, link_target)
(id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target)
VALUES `
args := make([]any, 0, len(batch)*fileCols)
@@ -333,19 +394,21 @@ func (r *FileRepository) CreateBatch(
querySb325.WriteString(", ")
}
querySb325.WriteString("(?, ?, ?, ?, ?, ?, ?, ?, ?)")
querySb325.WriteString("(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
args = append(args,
f.ID.String(), f.Path.String(), f.SourcePath.String(),
f.MTime.Unix(), f.Size, f.Mode, f.UID, f.GID,
f.MTime.Unix(), f.MTime.Nanosecond(),
f.Size, f.Mode, f.UID, f.GID,
f.LinkTarget.String())
}
query += querySb325.String() //nolint:gosec // G202: appends "?" placeholders only
query += querySb325.String()
query += ` ON CONFLICT(path) DO UPDATE SET
source_path = excluded.source_path,
mtime = excluded.mtime,
mtime_nsec = excluded.mtime_nsec,
size = excluded.size,
mode = excluded.mode,
uid = excluded.uid,
@@ -405,7 +468,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
var (
file File
idStr, pathStr, sourcePathStr string
mtimeUnix int64
mtimeUnix, mtimeNsec int64
linkTarget sql.NullString
)
@@ -414,6 +477,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
&pathStr,
&sourcePathStr,
&mtimeUnix,
&mtimeNsec,
&file.Size,
&file.Mode,
&file.UID,
@@ -432,7 +496,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
file.Path = types.FilePath(pathStr)
file.SourcePath = types.SourcePath(sourcePathStr)
file.MTime = time.Unix(mtimeUnix, 0).UTC()
file.MTime = time.Unix(mtimeUnix, mtimeNsec).UTC()
if linkTarget.Valid {
file.LinkTarget = types.FilePath(linkTarget.String)
}
internal/database/files_test.go
+187
View File
@@ -5,10 +5,12 @@ import (
"database/sql"
"errors"
"os"
"slices"
"testing"
"time"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
)
// errTestRollback is the sentinel returned from transaction bodies to
@@ -134,6 +136,82 @@ func TestFileRepositoryListDelete(t *testing.T) {
}
}
func TestFileRepositoryListUnderPath(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewFileRepository(db)
const (
docDir = "/home/u/doc"
docFile = "/home/u/doc/a.txt"
)
// In path order, so the root case can expect all of them as listed.
paths := []string{
"/home/u/50%/x.txt",
"/home/u/50percent/y.txt",
"/home/u/DOC/c.txt",
"/home/u/a_b/x.txt",
"/home/u/axb/y.txt",
docDir,
"/home/u/doc.txt.bak",
docFile,
"/home/u/doc/sub/b.txt",
"/home/u/doc2/b.txt",
}
for _, path := range paths {
err := repo.Create(ctx, nil, &database.File{
Path: types.FilePath(path),
MTime: time.Now().Truncate(time.Second),
Mode: 0644,
})
if err != nil {
t.Fatalf("failed to create %s: %v", path, err)
}
}
docTree := []string{docDir, docFile, "/home/u/doc/sub/b.txt"}
tests := []struct {
name string
path string
want []string
}{
{"directory", docDir, docTree},
{"directory with trailing slash", docDir + "/", docTree},
{"directory differing only in case", "/home/u/DOC",
[]string{"/home/u/DOC/c.txt"}},
{"file", docFile, []string{docFile}},
{"underscore is literal", "/home/u/a_b",
[]string{"/home/u/a_b/x.txt"}},
{"percent is literal", "/home/u/50%",
[]string{"/home/u/50%/x.txt"}},
{"root", "/", paths},
}
for _, tt := range tests {
files, err := repo.ListUnderPath(ctx, tt.path)
if err != nil {
t.Fatalf("%s: failed to list files: %v", tt.name, err)
}
got := make([]string, 0, len(files))
for _, f := range files {
got = append(got, f.Path.String())
}
if !slices.Equal(got, tt.want) {
t.Errorf("%s: listing %q got %q, want %q",
tt.name, tt.path, got, tt.want)
}
}
}
func TestFileRepositorySymlink(t *testing.T) {
t.Parallel()
@@ -174,6 +252,115 @@ func TestFileRepositorySymlink(t *testing.T) {
}
}
// An mtime after 2262 or before 1678 does not fit in int64 nanoseconds
// since the epoch, and must still come back from the database unchanged.
func TestFileRepositoryMTimeOutsideInt64NanosecondRange(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewFileRepository(db)
mtimes := []time.Time{
time.Date(2300, time.January, 1, 0, 0, 0, 123456789, time.UTC),
time.Date(1601, time.January, 1, 0, 0, 0, 987654321, time.UTC),
}
for _, mtime := range mtimes {
created := &database.File{
Path: types.FilePath("/created-" + mtime.Format(time.RFC3339Nano)),
MTime: mtime,
}
err := repo.Create(ctx, nil, created)
if err != nil {
t.Fatalf("failed to create file: %v", err)
}
batched := &database.File{
ID: types.NewFileID(),
Path: types.FilePath("/batched-" + mtime.Format(time.RFC3339Nano)),
MTime: mtime,
}
err = repo.CreateBatch(ctx, nil, []*database.File{batched})
if err != nil {
t.Fatalf("failed to batch create file: %v", err)
}
for _, path := range []types.FilePath{created.Path, batched.Path} {
retrieved, err := repo.GetByPath(ctx, path.String())
if err != nil {
t.Fatalf("failed to get file: %v", err)
}
if !retrieved.MTime.Equal(mtime) {
t.Errorf("%s: mtime got %v, want %v",
path, retrieved.MTime, mtime)
}
}
}
}
// A file already in the index and rewritten within the same second must get
// its new nanoseconds stored, through both Create and CreateBatch.
func TestFileRepositoryUpsertMTimeInSameSecond(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewFileRepository(db)
indexed := time.Date(2026, time.October, 7, 12, 0, 0, 100000000, time.UTC)
rewritten := indexed.Add(800 * time.Millisecond)
tests := []struct {
name string
upsert func(file *database.File) error
}{
{"Create", func(file *database.File) error {
return repo.Create(ctx, nil, file)
}},
{"CreateBatch", func(file *database.File) error {
return repo.CreateBatch(ctx, nil, []*database.File{file})
}},
}
for _, tt := range tests {
file := &database.File{
ID: types.NewFileID(),
Path: types.FilePath("/" + tt.name),
MTime: indexed,
}
err := tt.upsert(file)
if err != nil {
t.Fatalf("%s: failed to create file: %v", tt.name, err)
}
file.MTime = rewritten
err = tt.upsert(file)
if err != nil {
t.Fatalf("%s: failed to update file: %v", tt.name, err)
}
retrieved, err := repo.GetByPath(ctx, file.Path.String())
if err != nil {
t.Fatalf("%s: failed to get file: %v", tt.name, err)
}
if !retrieved.MTime.Equal(rewritten) {
t.Errorf("%s: mtime got %v, want %v",
tt.name, retrieved.MTime, rewritten)
}
}
}
func TestFileRepositoryTransaction(t *testing.T) {
t.Parallel()
internal/database/models.go
+9 -10
View File
@@ -14,8 +14,7 @@ type File struct {
ID types.FileID // UUID primary key
Path types.FilePath // Absolute path of the file
// SourcePath is the source directory this file came from (used for
// restore path stripping).
// SourcePath is the source directory this file came from.
SourcePath types.SourcePath
MTime time.Time
Size int64
@@ -51,15 +50,15 @@ type Chunk struct {
// Blob represents a blob record in the database.
// A blob is Vaultik's final storage unit - a large file (up to 10GB) containing
// many compressed and encrypted chunks from multiple source files.
// Blobs are content-addressed, meaning their filename in S3 is derived from
// the SHA256 hash of their compressed and encrypted content.
// The blob creation process is: chunks are accumulated -> compressed with zstd
// -> encrypted with age -> hashed -> uploaded to S3 with the hash as filename.
// Blobs are content-addressed: the filename in S3 is
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the chunk data
// before compression and encryption (not from the stored bytes). See
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
type Blob struct {
ID types.BlobID // UUID assigned when blob creation starts
// Hash is the SHA256 of the final compressed+encrypted content
// (empty until finalized).
// Hash is hex(SHA256(SHA256(uncompressed blob contents)))
// (empty until finalized); see the type comment above.
Hash types.BlobHash
CreatedTS time.Time // When blob creation started
FinishedTS *time.Time // When blob was finalized (nil if still packing)
@@ -99,8 +98,8 @@ type Snapshot struct {
StartedAt time.Time
CompletedAt *time.Time // nil if still in progress
FileCount int64
ChunkCount int64
BlobCount int64
ChunkCount int64 // Chunks this snapshot stored that were not stored before
BlobCount int64 // Blobs this snapshot created
TotalSize int64 // Total size of all referenced files
// BlobSize is the total size of all referenced blobs (compressed and
internal/database/module.go
+44
View File
@@ -2,6 +2,7 @@ package database
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
@@ -15,6 +16,44 @@ import (
// the index describes the backed-up file tree and must stay private.
const indexDirPerm = 0o700
// indexFilePerm restricts the index file to the owning user; it lists every
// backed-up path and chunk hash and must stay private.
const indexFilePerm = 0o600
// ensureIndexFileMode makes the index file owner-only before the SQLite
// driver opens it: it creates the file 0600 if absent, or chmods an existing
// one to 0600. Doing this first matters because SQLite creates its -wal and
// -shm side files with the mode of the main database file, so a private main
// file yields private side files. The driver treats a zero-byte file as an
// empty database, so pre-creating it here is safe.
func ensureIndexFileMode(path string) error {
info, err := os.Stat(path)
switch {
case err == nil:
if info.Mode().Perm() == indexFilePerm {
return nil
}
err = os.Chmod(path, indexFilePerm)
if err != nil {
return fmt.Errorf("restricting index file permissions: %w", err)
}
return nil
case errors.Is(err, os.ErrNotExist):
//nolint:gosec // G304: the index path is operator-configured by design
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY, indexFilePerm)
if err != nil {
return fmt.Errorf("creating index file: %w", err)
}
return f.Close()
default:
return fmt.Errorf("checking index file: %w", err)
}
}
// Module provides database dependencies
//
//nolint:gochecknoglobals // fx module definitions are package globals by convention
@@ -34,6 +73,11 @@ func provideDatabase(lc fx.Lifecycle, cfg *config.Config) (*DB, error) {
return nil, fmt.Errorf("creating index directory: %w", err)
}
err = ensureIndexFileMode(cfg.IndexPath)
if err != nil {
return nil, err
}
db, err := New(context.Background(), cfg.IndexPath)
if err != nil {
return nil, fmt.Errorf("opening database: %w", err)
internal/database/module_internal_test.go
+100
View File
@@ -0,0 +1,100 @@
package database
import (
"os"
"path/filepath"
"syscall"
"testing"
"go.uber.org/fx/fxtest"
"sneak.berlin/go/vaultik/internal/config"
)
// TestProvideDatabaseFreshIndexMode verifies that provideDatabase creates a
// missing index file owner-only (0600), even under a lenient 022 umask that
// would otherwise leave a freshly created file world-readable.
//
//nolint:paralleltest // syscall.Umask is process-global; parallel tests would clash
func TestProvideDatabaseFreshIndexMode(t *testing.T) {
restore := syscall.Umask(0o022)
defer syscall.Umask(restore)
indexPath := filepath.Join(t.TempDir(), "index.sqlite")
openIndex(t, indexPath)
assertPerm(t, indexPath, 0o600)
}
// TestProvideDatabaseExistingIndexMode verifies that provideDatabase tightens
// an existing world-readable index (0644) in a group/other-readable directory
// down to owner-only (0600).
//
//nolint:paralleltest // syscall.Umask is process-global; parallel tests would clash
func TestProvideDatabaseExistingIndexMode(t *testing.T) {
restore := syscall.Umask(0o022)
defer syscall.Umask(restore)
dir := filepath.Join(t.TempDir(), "data")
//nolint:gosec // G301: the test intentionally uses a 0755 directory
err := os.MkdirAll(dir, 0o755)
if err != nil {
t.Fatalf("creating index directory: %v", err)
}
//nolint:gosec // G302: the test intentionally uses a 0755 directory
err = os.Chmod(dir, 0o755)
if err != nil {
t.Fatalf("relaxing index directory permissions: %v", err)
}
indexPath := filepath.Join(dir, "index.sqlite")
//nolint:gosec // G306: the test intentionally starts from a 0644 index
err = os.WriteFile(indexPath, nil, 0o644)
if err != nil {
t.Fatalf("creating pre-existing index: %v", err)
}
//nolint:gosec // G302: the test intentionally starts from a 0644 index
err = os.Chmod(indexPath, 0o644)
if err != nil {
t.Fatalf("relaxing pre-existing index permissions: %v", err)
}
openIndex(t, indexPath)
assertPerm(t, indexPath, 0o600)
}
// openIndex runs provideDatabase against indexPath and closes the resulting
// database before returning.
func openIndex(t *testing.T, indexPath string) {
t.Helper()
cfg := &config.Config{IndexPath: indexPath}
db, err := provideDatabase(fxtest.NewLifecycle(t), cfg)
if err != nil {
t.Fatalf("provideDatabase: %v", err)
}
err = db.Close()
if err != nil {
t.Fatalf("closing database: %v", err)
}
}
// assertPerm fails the test unless path has exactly the given permission bits.
func assertPerm(t *testing.T, path string, want os.FileMode) {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat %s: %v", path, err)
}
got := info.Mode().Perm()
if got != want {
t.Fatalf("permissions of %s = %#o, want %#o", path, got, want)
}
}
internal/database/read_only_test.go
+145
View File
@@ -0,0 +1,145 @@
//nolint:testpackage // exercises unexported read-only open internals
package database
import (
"context"
"database/sql"
"errors"
"path/filepath"
"testing"
)
// genuineSnapshotDB writes a real snapshot database (the full schema
// applied) to a fresh file and returns its path.
func genuineSnapshotDB(t *testing.T) string {
t.Helper()
path := filepath.Join(t.TempDir(), "snapshot.db")
db, err := New(context.Background(), path)
if err != nil {
t.Fatalf("creating snapshot database: %v", err)
}
err = db.Close()
if err != nil {
t.Fatalf("closing snapshot database: %v", err)
}
return path
}
// forgedDB creates an empty database file and runs the given statements
// against it read-write, so a test can plant schema objects the real
// schema never defines.
func forgedDB(t *testing.T, stmts ...string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "forged.db")
db, err := sql.Open("sqlite", path)
if err != nil {
t.Fatalf("opening forged database: %v", err)
}
for _, stmt := range stmts {
_, err = db.ExecContext(context.Background(), stmt)
if err != nil {
t.Fatalf("executing %q: %v", stmt, err)
}
}
err = db.Close()
if err != nil {
t.Fatalf("closing forged database: %v", err)
}
return path
}
func TestOpenReadOnlyAcceptsGenuineSnapshot(t *testing.T) {
t.Parallel()
db, err := OpenReadOnly(context.Background(), genuineSnapshotDB(t))
if err != nil {
t.Fatalf("OpenReadOnly refused a genuine snapshot database: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
}
func TestOpenReadOnlyRefusesWrites(t *testing.T) {
t.Parallel()
db, err := OpenReadOnly(context.Background(), genuineSnapshotDB(t))
if err != nil {
t.Fatalf("OpenReadOnly: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
// A schema write depends on no table columns, so the only reason it
// can fail is that the database is open read-only.
_, err = db.Conn().ExecContext(context.Background(),
"CREATE TABLE probe_readonly (x)")
if err == nil {
t.Fatal("expected a write to a read-only snapshot database to fail")
}
}
func TestOpenReadOnlyRejectsView(t *testing.T) {
t.Parallel()
path := forgedDB(t, "CREATE VIEW files AS SELECT 1 AS path")
_, err := OpenReadOnly(context.Background(), path)
if !errors.Is(err, errUntrustedSnapshotSchema) {
t.Fatalf("expected a view named files to be refused, got %v", err)
}
}
func TestOpenReadOnlyRejectsTrigger(t *testing.T) {
t.Parallel()
path := forgedDB(t,
"CREATE TABLE files (path TEXT)",
"CREATE TRIGGER t AFTER INSERT ON files BEGIN SELECT 1; END")
_, err := OpenReadOnly(context.Background(), path)
if !errors.Is(err, errUntrustedSnapshotSchema) {
t.Fatalf("expected a trigger to be refused, got %v", err)
}
}
func TestOpenReadOnlyRejectsMissingTable(t *testing.T) {
t.Parallel()
// Only one of the expected tables is present.
path := forgedDB(t, "CREATE TABLE files (path TEXT)")
_, err := OpenReadOnly(context.Background(), path)
if !errors.Is(err, errUntrustedSnapshotSchema) {
t.Fatalf("expected a missing expected table to be refused, got %v", err)
}
}
func TestIsVirtualTableSQL(t *testing.T) {
t.Parallel()
cases := []struct {
sql string
want bool
}{
{"CREATE VIRTUAL TABLE t USING fts5(x)", true},
{" create virtual table t using fts5(x)", true},
{"CREATE TABLE t (x)", false},
{"CREATE VIEW t AS SELECT 1", false},
{"", false},
}
for _, c := range cases {
if got := isVirtualTableSQL(c.sql); got != c.want {
t.Errorf("isVirtualTableSQL(%q) = %v, want %v", c.sql, got, c.want)
}
}
}
internal/database/repository_comprehensive_test.go
+2 -2
View File
@@ -824,7 +824,7 @@ func TestTransactionIsolation(t *testing.T) {
}
// Verify the file was not created (transaction rolled back)
files, err := repos.Files.ListByPrefix(ctx, "/tx-test")
files, err := repos.Files.ListUnderPath(ctx, "/tx-test.txt")
if err != nil {
t.Fatal(err)
}
@@ -916,7 +916,7 @@ func TestConcurrentOrphanedCleanup(t *testing.T) {
}
// Verify correct files were deleted
files, err := repos.Files.ListByPrefix(ctx, "/concurrent-")
files, err := repos.Files.ListAll(ctx)
if err != nil {
t.Fatal(err)
}
internal/database/repository_debug_test.go
+1 -1
View File
@@ -147,7 +147,7 @@ func TestOrphanedFileCleanupDebug(t *testing.T) {
t.Logf("Files count after cleanup: %d", count)
// List remaining files
files, err := repos.Files.ListByPrefix(ctx, "/")
files, err := repos.Files.ListUnderPath(ctx, "/")
if err != nil {
t.Fatal(err)
}
internal/database/repository_edge_cases_test.go
+34 -27
View File
@@ -3,6 +3,7 @@ package database
import (
"context"
"database/sql"
"fmt"
"strings"
"testing"
@@ -367,7 +368,7 @@ func verifyBlobNullUploadTS(
}
// createLargeDatasetFiles creates fileCount files and adds every other
// one to the snapshot.
// one to the snapshot, in one transaction as a backup writes them.
func createLargeDatasetFiles(
t *testing.T,
repos *Repositories,
@@ -376,31 +377,38 @@ func createLargeDatasetFiles(
) {
t.Helper()
ctx := context.Background()
start := time.Now()
for i := range fileCount {
file := &File{
Path: types.FilePath(fmt.Sprintf("/large/file%05d.txt", i)),
MTime: time.Now(),
Size: int64(i * 1024),
Mode: 0644,
UID: uint32(1000 + (i % 10)),
GID: uint32(1000 + (i % 10)),
}
err := repos.WithTx(context.Background(),
func(ctx context.Context, tx *sql.Tx) error {
for i := range fileCount {
file := &File{
Path: types.FilePath(fmt.Sprintf("/large/file%05d.txt", i)),
MTime: time.Now(),
Size: int64(i * 1024),
Mode: 0644,
UID: uint32(1000 + (i % 10)),
GID: uint32(1000 + (i % 10)),
}
err := repos.Files.Create(ctx, nil, file)
if err != nil {
t.Fatalf("failed to create file %d: %v", i, err)
}
err := repos.Files.Create(ctx, tx, file)
if err != nil {
return fmt.Errorf("creating file %d: %w", i, err)
}
// Add half to snapshot
if i%2 == 0 {
err = repos.Snapshots.AddFileByID(ctx, nil, snapshotID, file.ID)
if err != nil {
t.Fatal(err)
// Add half to snapshot
if i%2 == 0 {
err = repos.Snapshots.AddFileByID(ctx, tx, snapshotID, file.ID)
if err != nil {
return err
}
}
}
}
return nil
})
if err != nil {
t.Fatal(err)
}
t.Logf("Created %d files in %v", fileCount, time.Since(start))
@@ -442,12 +450,12 @@ func TestLargeDatasets(t *testing.T) {
createLargeDatasetFiles(t, repos, snapshot.ID.String(), fileCount)
})
// Test ListByPrefix performance
// Test ListUnderPath performance
//nolint:paralleltest // phases share one database and are order-dependent
t.Run("list by prefix performance", func(t *testing.T) {
t.Run("list under path performance", func(t *testing.T) {
start := time.Now()
files, err := repos.Files.ListByPrefix(ctx, "/large/")
files, err := repos.Files.ListUnderPath(ctx, "/large/")
if err != nil {
t.Fatal(err)
}
@@ -472,7 +480,7 @@ func TestLargeDatasets(t *testing.T) {
t.Logf("Cleaned up orphaned files in %v", time.Since(start))
// Verify correct number remain
files, err := repos.Files.ListByPrefix(ctx, "/large/")
files, err := repos.Files.ListUnderPath(ctx, "/large/")
if err != nil {
t.Fatal(err)
}
@@ -606,8 +614,7 @@ func TestTimezoneHandling(t *testing.T) {
t.Skip("timezone not available")
}
// Use Truncate to remove sub-second precision since we store as Unix timestamps
nyTime := time.Now().In(loc).Truncate(time.Second)
nyTime := time.Now().In(loc)
file := &File{
Path: "/timezone-test.txt",
MTime: nyTime,
internal/database/schema/001.sql
+3 -2
View File
@@ -5,8 +5,9 @@
CREATE TABLE IF NOT EXISTS files (
id TEXT PRIMARY KEY, -- UUID
path TEXT NOT NULL UNIQUE,
source_path TEXT NOT NULL DEFAULT '', -- The source directory this file came from (for restore path stripping)
mtime INTEGER NOT NULL,
source_path TEXT NOT NULL DEFAULT '', -- The source directory this file came from
mtime INTEGER NOT NULL, -- whole seconds since the Unix epoch
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
size INTEGER NOT NULL,
mode INTEGER NOT NULL,
uid INTEGER NOT NULL,
internal/database/snapshots.go
+83 -4
View File
@@ -11,6 +11,18 @@ import (
"sneak.berlin/go/vaultik/internal/types"
)
// Sentinel errors for the single-snapshot invariant that an exported
// per-snapshot metadata database must satisfy.
var (
// ErrNoSnapshotInDatabase means the metadata database has no snapshot
// row at all.
ErrNoSnapshotInDatabase = errors.New("database contains no snapshot")
// ErrMultipleSnapshotsInDatabase means the metadata database holds
// more than the single snapshot an export is supposed to contain.
ErrMultipleSnapshotsInDatabase = errors.New(
"database contains more than one snapshot")
)
// SnapshotRepository provides access to the snapshots table and its
// snapshot_files / snapshot_blobs association tables.
type SnapshotRepository struct {
@@ -115,6 +127,7 @@ func (r *SnapshotRepository) UpdateExtendedStats(
snapshotID string,
blobUncompressedSize int64,
compressionLevel int,
uploadBytes int64,
uploadDurationMs int64,
) error {
compressionRatio, err := r.extendedCompressionRatio(
@@ -129,7 +142,7 @@ func (r *SnapshotRepository) UpdateExtendedStats(
SET blob_uncompressed_size = ?,
compression_ratio = ?,
compression_level = ?,
upload_bytes = blob_size,
upload_bytes = ?,
upload_duration_ms = ?
WHERE id = ?
`
@@ -137,11 +150,11 @@ func (r *SnapshotRepository) UpdateExtendedStats(
if tx != nil {
_, err = tx.ExecContext(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel,
uploadDurationMs, snapshotID)
uploadBytes, uploadDurationMs, snapshotID)
} else {
_, err = r.db.ExecWithLog(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel,
uploadDurationMs, snapshotID)
uploadBytes, uploadDurationMs, snapshotID)
}
if err != nil {
@@ -206,6 +219,48 @@ func (r *SnapshotRepository) GetByID(
return &snapshot, nil
}
// GetOnlySnapshot returns the sole snapshot in an exported per-snapshot
// metadata database. The backup path writes each snapshot's database with
// exactly one snapshot row (see cleanSnapshotDB), so restore and deep
// verify expect exactly one. Zero rows return ErrNoSnapshotInDatabase and
// more than one returns ErrMultipleSnapshotsInDatabase; callers treat
// either as a failed identity check on the downloaded database.
func (r *SnapshotRepository) GetOnlySnapshot(ctx context.Context) (*Snapshot, error) {
query := `
SELECT id, hostname, vaultik_version, vaultik_git_revision,
started_at, completed_at, file_count, chunk_count, blob_count,
total_size, blob_size, compression_ratio
FROM snapshots
LIMIT 2
`
rows, err := r.db.conn.QueryContext(ctx, query)
if err != nil {
return nil, fmt.Errorf("querying snapshots: %w", err)
}
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
snapshots, err := r.scanSnapshotRows(rows)
if err != nil {
return nil, err
}
switch len(snapshots) {
case 1:
return snapshots[0], nil
case 0:
return nil, ErrNoSnapshotInDatabase
default:
return nil, ErrMultipleSnapshotsInDatabase
}
}
// ListRecent returns up to limit snapshots, most recently started first.
func (r *SnapshotRepository) ListRecent(
ctx context.Context, limit int,
@@ -341,7 +396,7 @@ func (r *SnapshotRepository) AddFilesByIDBatch(
args = append(args, snapshotID, fileID.String())
}
query += querySb312.String() //nolint:gosec // G202: appends "?" placeholders only
query += querySb312.String()
var err error
if tx != nil {
@@ -489,6 +544,30 @@ func (r *SnapshotRepository) GetSnapshotTotalCompressedSize(
return totalSize, nil
}
// GetSnapshotBlobSizes returns the total compressed and uncompressed sizes
// of all blobs referenced by a snapshot.
func (r *SnapshotRepository) GetSnapshotBlobSizes(
ctx context.Context, snapshotID string,
) (int64, int64, error) {
query := `
SELECT COALESCE(SUM(b.compressed_size), 0),
COALESCE(SUM(b.uncompressed_size), 0)
FROM snapshot_blobs sb
JOIN blobs b ON sb.blob_hash = b.blob_hash
WHERE sb.snapshot_id = ?
`
var compressed, uncompressed int64
err := r.db.conn.QueryRowContext(ctx, query, snapshotID).Scan(
&compressed, &uncompressed)
if err != nil {
return 0, 0, fmt.Errorf("querying snapshot blob sizes: %w", err)
}
return compressed, uncompressed, nil
}
// GetSnapshotUncompressedChunkSize returns the sum of plaintext sizes of all unique
// chunks referenced by a snapshot (via snapshot_files → file_chunks → chunks).
func (r *SnapshotRepository) GetSnapshotUncompressedChunkSize(
internal/database/snapshots_test.go
+59
View File
@@ -145,6 +145,65 @@ func TestSnapshotRepositoryUpdateCounts(t *testing.T) {
}
}
// GetSnapshotBlobSizes totals the blobs the snapshot references, and only
// those.
func TestSnapshotRepositoryGetSnapshotBlobSizes(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := database.NewRepositories(db)
snapshot := &database.Snapshot{
ID: "2024-01-03T12:00:00Z",
Hostname: testHostname,
VaultikVersion: testVersion,
StartedAt: time.Now().Truncate(time.Second),
}
err := repos.Snapshots.Create(ctx, nil, snapshot)
if err != nil {
t.Fatalf("failed to create snapshot: %v", err)
}
blobs := []*database.Blob{
{Hash: "referenced-1", CompressedSize: 10, UncompressedSize: 100},
{Hash: "referenced-2", CompressedSize: 20, UncompressedSize: 200},
{Hash: "unreferenced", CompressedSize: 40, UncompressedSize: 400},
}
for _, blob := range blobs {
blob.ID = types.NewBlobID()
blob.CreatedTS = time.Now().Truncate(time.Second)
err = repos.Blobs.Create(ctx, nil, blob)
if err != nil {
t.Fatalf("failed to create blob %s: %v", blob.Hash, err)
}
}
for _, blob := range blobs[:2] {
err = repos.Snapshots.AddBlob(ctx, nil, snapshot.ID.String(),
blob.ID, blob.Hash)
if err != nil {
t.Fatalf("failed to add blob %s to snapshot: %v", blob.Hash, err)
}
}
compressed, uncompressed, err := repos.Snapshots.GetSnapshotBlobSizes(
ctx, snapshot.ID.String())
if err != nil {
t.Fatalf("failed to get snapshot blob sizes: %v", err)
}
if compressed != 30 || uncompressed != 300 {
t.Errorf("blob sizes: got %d and %d, want 30 and 300",
compressed, uncompressed)
}
}
func TestSnapshotRepositoryListRecent(t *testing.T) {
t.Parallel()
internal/database/uploads.go
-16
View File
@@ -158,19 +158,3 @@ type UploadStats struct {
MinDurationMs int64
MaxDurationMs int64
}
// GetCountBySnapshot returns the count of uploads for a specific snapshot
func (r *UploadRepository) GetCountBySnapshot(
ctx context.Context, snapshotID string,
) (int64, error) {
query := `SELECT COUNT(*) FROM uploads WHERE snapshot_id = ?`
var count int64
err := r.conn.QueryRowContext(ctx, query, snapshotID).Scan(&count)
if err != nil {
return 0, err
}
return count, nil
}
internal/globals/globals.go
+28 -13
View File
@@ -3,6 +3,7 @@
package globals
import (
"regexp"
"strings"
"time"
)
@@ -10,22 +11,26 @@ import (
// Appname is the application name, populated from main().
var Appname = "vaultik" //nolint:gochecknoglobals // set via -ldflags at build time
// DevVersion is the version a binary reports when it was not built
// from a tagged commit. script/version emits either this exact string
// (outside a git checkout) or this string followed by "-" and the
// commit it was built from, and goreleaser's snapshot template matches
// that shape. It is deliberately not a number: a build that is not a
// release must not name itself like one.
// DevVersion is the version a binary reports when it was built without
// git metadata: script/version emits it outside a git checkout, and an
// unstamped `go build` keeps it. goreleaser's snapshot template stamps
// it followed by "-" and the commit it was built from. It is
// deliberately not a number.
const DevVersion = "dev"
// Unknown is what Commit and CommitDate hold when the build did not
// stamp them, and the version script/docker and script/cibuild stamp
// when the host has no git checkout.
const Unknown = "unknown"
// Version is the application version, populated from main().
var Version = DevVersion //nolint:gochecknoglobals // set via -ldflags at build time
// Commit is the git commit hash, populated from main().
var Commit = "unknown" //nolint:gochecknoglobals // set via -ldflags at build time
var Commit = Unknown //nolint:gochecknoglobals // set via -ldflags at build time
// CommitDate is the ISO-8601 date of the commit, populated from main().
var CommitDate = "unknown" //nolint:gochecknoglobals // set via -ldflags at build time
var CommitDate = Unknown //nolint:gochecknoglobals // set via -ldflags at build time
// Author identifies the upstream author of vaultik.
const Author = "Jeffrey Paul <sneak@sneak.berlin>"
@@ -60,18 +65,28 @@ func New() (*Globals, error) {
}
// IsDevVersion reports whether v names a development build rather than
// a release. Both "dev" and "dev-<sha>" (and its "-dirty" variant)
// count: a caller that compares against "dev" exactly would treat every
// commit-stamped development build as a release.
// a release. "dev" and goreleaser's snapshot "dev-<sha>" count, and so
// does what `git describe --tags --always --dirty` gives a make or
// docker build of an untagged commit: the bare short commit, or
// tag-N-gHASH on a commit after a tag. Any version ending in "-dirty"
// counts, a modified checkout of a tag ("v1.0.0-dirty") included.
// A plain tag such as "v1.0.0" or "1.0.0" is a release.
//
// The empty string counts too. Nothing that knows its version reports
// no version, so an empty Version means the stamping failed, and the
// safe reading of "we could not establish that this is a release" is
// that it is not one. The Makefile refuses to build at all in that
// case; this is the second line of defence, for a binary linked by
// something other than the Makefile.
// something other than the Makefile. Unknown counts for the same
// reason.
func IsDevVersion(v string) bool {
return v == "" || v == DevVersion || strings.HasPrefix(v, DevVersion+"-")
if v == "" || v == Unknown || v == DevVersion ||
strings.HasPrefix(v, DevVersion+"-") ||
strings.HasSuffix(v, "-dirty") {
return true
}
return regexp.MustCompile(`^(.+-[0-9]+-g)?[0-9a-f]+$`).MatchString(v)
}
// shortCommitLen is the number of commit-hash characters ShortCommit keeps.
internal/globals/globals_test.go
+19 -6
View File
@@ -34,10 +34,11 @@ func TestGlobalsNew(t *testing.T) {
}
// TestIsDevVersion covers the boundary that matters: everything
// script/version and goreleaser's snapshot template can emit for an
// untagged build must be recognised as a development build, and a real
// tag must not be. A plain equality check against "dev" used to decide
// this, which classified every commit-stamped dev build as a release.
// script/version, a plain docker build and goreleaser's snapshot
// template can emit for an untagged build must be recognised as a
// development build, and a real tag must not be. A plain equality check
// against "dev" used to decide this, which classified every
// commit-stamped dev build as a release.
func TestIsDevVersion(t *testing.T) {
t.Parallel()
@@ -49,8 +50,17 @@ func TestIsDevVersion(t *testing.T) {
{"dev", true},
{"dev-b6e4a218a39e", true},
{"dev-b6e4a218a39e-dirty", true},
// What a tagged build produces (script/version strips the
// leading "v", matching goreleaser's .Version).
// What `git describe --tags --always --dirty` produces with no
// tag reachable, and on a commit after a tag.
{"877eb2f", true},
{"877eb2f-dirty", true},
{"v1.0.0-3-g877eb2f", true},
{"v1.0.0-3-g877eb2f-dirty", true},
{"1.0.0-rc.1-12-g877eb2f", true},
// A tagged commit with uncommitted changes is not that tag.
{"v1.0.0-dirty", true},
// What a tagged build produces (goreleaser's .Version strips
// the leading "v"; script/version keeps it).
{"1.0.0", false},
{"0.1.0", false},
{"1.0.0-rc.1", false},
@@ -64,6 +74,9 @@ func TestIsDevVersion(t *testing.T) {
// as one. The Makefile refuses to build when script/version
// yields nothing; this covers a binary linked some other way.
{"", true},
// What script/docker and script/cibuild stamp when the host
// has no git checkout.
{"unknown", true},
}
for _, tc := range cases {
internal/log/tty_escape_test.go
+39
View File
@@ -0,0 +1,39 @@
package log_test
import (
"bytes"
"log/slog"
"strings"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
)
// TestTTYHandlerEscapesControlCharacters logs a message and an attribute
// value that each carry an ESC and a newline — the shape a crafted path or
// storage error from the destination would take — and checks neither raw
// byte reaches the output. The handler's own colour codes (ESC ... m) are
// stripped first; any ESC left after that came from the untrusted value.
func TestTTYHandlerEscapesControlCharacters(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
logger := slog.New(log.NewTTYHandler(&buf, debugHandlerOptions()))
logger.Info("start\x1b[31mZAP\nend", "target", "a\x1b[31mZAP\nb")
out := buf.String()
// The only newline is the line terminator; the injected ones were escaped.
require.Equal(t, 1, strings.Count(out, "\n"),
"a newline in the message or a value must be escaped, not emitted raw")
// After the handler's own colour codes are removed, no ESC survives.
stripped := ansiEscape.ReplaceAllString(out, "")
require.NotContains(t, stripped, "\x1b",
"a raw ESC from the message or a value must not reach the terminal")
// The escaped form is what appears instead.
require.Contains(t, out, `\x1b`)
}
internal/log/tty_handler.go
+29 -4
View File
@@ -5,9 +5,11 @@ import (
"fmt"
"io"
"log/slog"
"strconv"
"strings"
"sync"
"time"
"unicode"
)
// groupSeparator joins an open group path to an attribute key. This
@@ -116,11 +118,14 @@ func (h *TTYHandler) Handle(_ context.Context, r slog.Record) error {
levelColor = colorReset
}
// Print main message
// Print main message. The message is escaped before the colour codes
// are written around it: it can carry text from an untrusted source
// (a storage error, for one), and a raw control character would
// otherwise reach the terminal.
_, _ = fmt.Fprintf(h.out, "%s%s%s %s%s%s %s%s%s",
colorGray, timestamp, colorReset,
levelColor, level, colorReset,
colorBold, r.Message, colorReset)
colorBold, sanitize(r.Message), colorReset)
// Attributes carried by the handler come first, then the record's
// own. Handler attributes were qualified when they were added; the
@@ -260,9 +265,29 @@ func (h *TTYHandler) writeAttr(a slog.Attr) {
// Future kinds also use the plain string form.
}
// Escape the key and value before the colour codes are written around
// them. Both can carry text from an untrusted source — a manifest
// timestamp, a storage error, a path or symlink target read back from
// the snapshot database — so a control character in one of them must
// be rendered as an escape sequence rather than reaching the terminal,
// where it could move the cursor or inject its own colours.
_, _ = fmt.Fprintf(h.out, " %s%s%s=%s%s%s",
colorCyan, a.Key, colorReset,
colorBlue, value, colorReset)
colorCyan, sanitize(a.Key), colorReset,
colorBlue, sanitize(value), colorReset)
}
// sanitize returns s unchanged when every rune in it is printable, and a
// double-quoted, backslash-escaped form (\n, \x1b, …) otherwise. It is
// applied to untrusted text before any colour code is written, so a
// control character can never reach the terminal raw.
func sanitize(s string) string {
for _, r := range s {
if !unicode.IsPrint(r) {
return strconv.Quote(s)
}
}
return s
}
// formatDuration formats a duration in a human-readable way
internal/pidlock/pidlock.go
+67 -52
View File
@@ -10,15 +10,18 @@ import (
"path/filepath"
"strconv"
"strings"
"syscall"
"golang.org/x/sys/unix"
)
// ErrAlreadyRunning indicates another vaultik instance is running.
var ErrAlreadyRunning = errors.New("another vaultik instance is already running")
// Lock represents an acquired PID lock.
// Lock represents an acquired PID lock: an flock(2) on the PID file,
// held while the file stays open. The kernel drops it when the process
// exits, however it exits, so a crashed run never leaves the lock held.
type Lock struct {
path string
file *os.File
}
const (
@@ -29,10 +32,9 @@ const (
)
// Acquire attempts to acquire a PID lock in the specified directory.
// If the lock file exists and the process is still running, it returns
// ErrAlreadyRunning with details about the existing process.
// On success, it writes the current PID to the lock file and returns
// a Lock that must be released with Release().
// If another process holds the lock, it returns ErrAlreadyRunning with
// that process's PID. On success, it writes the current PID to the lock
// file and returns a Lock that must be released with Release().
func Acquire(lockDir string) (*Lock, error) {
// Ensure lock directory exists
err := os.MkdirAll(lockDir, lockDirPerm)
@@ -42,56 +44,82 @@ func Acquire(lockDir string) (*Lock, error) {
lockPath := filepath.Join(lockDir, "vaultik.pid")
// Check for existing lock
existingPID, err := readPIDFile(lockPath)
if err == nil {
// Lock file exists, check if process is running
if isProcessRunning(existingPID) {
return nil, fmt.Errorf("%w (PID %d)", ErrAlreadyRunning, existingPID)
}
// Process is not running, stale lock file - we can take over
}
// Write our PID
pid := os.Getpid()
err = os.WriteFile(lockPath, []byte(strconv.Itoa(pid)), pidFilePerm)
// No O_TRUNC: the file may hold the PID of the process that has the
// lock, which the error below reports.
file, err := os.OpenFile( //nolint:gosec // G304: path is our own lock file
lockPath, os.O_RDWR|os.O_CREATE, pidFilePerm)
if err != nil {
return nil, fmt.Errorf("writing PID file: %w", err)
return nil, fmt.Errorf("opening PID file: %w", err)
}
return &Lock{path: lockPath}, nil
err = unix.Flock(int(file.Fd()), unix.LOCK_EX|unix.LOCK_NB)
if err != nil {
_ = file.Close()
if errors.Is(err, unix.EWOULDBLOCK) {
return nil, alreadyRunningError(lockPath)
}
return nil, fmt.Errorf("locking PID file: %w", err)
}
err = writePID(file)
if err != nil {
_ = file.Close()
return nil, err
}
return &Lock{file: file}, nil
}
// Release removes the PID lock file.
// Release empties the PID file and closes it, which drops the lock.
// It is safe to call Release multiple times.
func (l *Lock) Release() error {
if l == nil || l.path == "" {
if l == nil || l.file == nil {
return nil
}
// Verify we still own the lock (our PID is in the file)
existingPID, err := readPIDFile(l.path)
file := l.file
l.file = nil
// Do not remove the file here. A process that opened it a moment
// earlier could then lock the removed file while another creates and
// locks a new one, and both would run.
truncateErr := file.Truncate(0)
closeErr := file.Close()
return errors.Join(truncateErr, closeErr)
}
// writePID replaces the contents of the locked PID file with the current
// PID.
func writePID(file *os.File) error {
err := file.Truncate(0)
if err != nil {
// File already gone or unreadable - that's fine
return nil //nolint:nilerr // unreadable lock file means nothing to release
return fmt.Errorf("truncating PID file: %w", err)
}
if existingPID != os.Getpid() {
// Someone else wrote to our lock file - don't remove it
return nil
_, err = file.WriteAt([]byte(strconv.Itoa(os.Getpid())), 0)
if err != nil {
return fmt.Errorf("writing PID file: %w", err)
}
err = os.Remove(l.path)
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("removing PID file: %w", err)
}
l.path = "" // Prevent double-release
return nil
}
// alreadyRunningError reports that another process holds the lock,
// naming its PID when the file holds one. The holder writes its PID just
// after it locks, so the file can briefly be empty.
func alreadyRunningError(lockPath string) error {
pid, err := readPIDFile(lockPath)
if err != nil {
return ErrAlreadyRunning
}
return fmt.Errorf("%w (PID %d)", ErrAlreadyRunning, pid)
}
// readPIDFile reads and parses the PID from a lock file.
func readPIDFile(path string) (int, error) {
data, err := os.ReadFile(path) //nolint:gosec // G304: path is our own lock file
@@ -106,16 +134,3 @@ func readPIDFile(path string) (int, error) {
return pid, nil
}
// isProcessRunning checks if a process with the given PID is running.
func isProcessRunning(pid int) bool {
process, err := os.FindProcess(pid)
if err != nil {
return false
}
// On Unix, FindProcess always succeeds. We need to send signal 0 to check.
err = process.Signal(syscall.Signal(0))
return err == nil
}
internal/pidlock/pidlock_test.go
+63 -3
View File
@@ -4,6 +4,7 @@ import (
"os"
"path/filepath"
"strconv"
"sync"
"testing"
"github.com/stretchr/testify/assert"
@@ -33,9 +34,10 @@ func TestAcquireAndRelease(t *testing.T) {
err = lock.Release()
require.NoError(t, err)
// Verify PID file is gone
_, err = os.Stat(pidPath)
assert.True(t, os.IsNotExist(err))
// Verify PID file is empty
data, err = os.ReadFile(pidPath) //nolint:gosec // G304: test's own temp file
require.NoError(t, err)
assert.Empty(t, data)
}
func TestAcquireBlocksSecondInstance(t *testing.T) {
@@ -55,6 +57,64 @@ func TestAcquireBlocksSecondInstance(t *testing.T) {
lock2, err := pidlock.Acquire(tmpDir)
require.ErrorIs(t, err, pidlock.ErrAlreadyRunning)
assert.Nil(t, lock2)
// Once the first lock is released, the next Acquire succeeds
require.NoError(t, lock1.Release())
lock3, err := pidlock.Acquire(tmpDir)
require.NoError(t, err)
require.NoError(t, lock3.Release())
}
// TestConcurrentAcquireAdmitsOne starts many Acquire calls at the same
// moment, as two cron entries firing together would, and checks that
// exactly one of them gets the lock.
func TestConcurrentAcquireAdmitsOne(t *testing.T) {
t.Parallel()
const callers = 50
tmpDir := t.TempDir()
start := make(chan struct{})
var (
mu sync.Mutex
acquired []*pidlock.Lock
failures []error
wg sync.WaitGroup
)
for range callers {
wg.Go(func() {
<-start
lock, err := pidlock.Acquire(tmpDir)
mu.Lock()
defer mu.Unlock()
if err != nil {
failures = append(failures, err)
return
}
acquired = append(acquired, lock)
})
}
close(start)
wg.Wait()
for _, lock := range acquired {
require.NoError(t, lock.Release())
}
assert.Len(t, acquired, 1, "exactly one caller should hold the lock")
for _, err := range failures {
require.ErrorIs(t, err, pidlock.ErrAlreadyRunning)
}
}
func TestAcquireWithStaleLock(t *testing.T) {
internal/s3/client.go
+34 -6
View File
@@ -6,6 +6,7 @@ import (
"context"
"errors"
"io"
"strings"
"sync/atomic"
"github.com/aws/aws-sdk-go-v2/aws"
@@ -26,10 +27,14 @@ type Client struct {
bucket string
prefix string
endpoint string
partSize int64
}
// Config contains S3 client configuration.
// All fields are required except Prefix, which defaults to an empty string.
// All fields are required except Prefix, which defaults to an empty string,
// and PartSize, where zero means the SDK default of 5 MiB.
// A non-empty Prefix is joined to every key with one "/", whether or not
// it ends with one.
// The Endpoint field should include the protocol (http:// or https://).
type Config struct {
Endpoint string
@@ -38,6 +43,9 @@ type Config struct {
AccessKeyID string
SecretAccessKey string
Region string
// PartSize is the size in bytes of each part of a multipart upload.
// An upload too large for S3's limit of 10,000 parts gets larger parts.
PartSize int64
}
// nopLogger is a logger that discards all output.
@@ -75,11 +83,19 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) {
s3Client := s3.NewFromConfig(awsCfg, s3Opts)
// Every method below builds a key as prefix + key, so the prefix
// must carry its own trailing "/".
prefix := strings.TrimRight(cfg.Prefix, "/")
if prefix != "" {
prefix += "/"
}
return &Client{
s3Client: s3Client,
bucket: cfg.Bucket,
prefix: cfg.Prefix,
prefix: prefix,
endpoint: cfg.Endpoint,
partSize: cfg.PartSize,
}, nil
}
@@ -113,12 +129,9 @@ func (c *Client) PutObjectWithProgress(
) error {
fullKey := c.prefix + key
// uploadPartSize is 10MB for better progress granularity.
const uploadPartSize = 10 * 1024 * 1024
// Create an uploader with the S3 client
uploader := manager.NewUploader(c.s3Client, func(u *manager.Uploader) {
u.PartSize = uploadPartSize
u.PartSize = uploadPartSize(c.partSize, size)
})
// Create a progress reader that tracks upload progress
@@ -139,6 +152,21 @@ func (c *Client) PutObjectWithProgress(
return err
}
// uploadPartSize returns the part size for an upload of size bytes: the
// configured part size (the SDK default when zero), raised where needed so
// the upload fits in S3's limit of 10,000 parts. The uploader cannot raise
// it itself, because it cannot seek the progress reader to learn its size.
func uploadPartSize(configured, size int64) int64 {
if configured == 0 {
configured = manager.DefaultUploadPartSize
}
maxParts := int64(manager.MaxUploadParts)
smallestThatFits := (size + maxParts - 1) / maxParts // rounded up
return max(configured, smallestThatFits)
}
// GetObject downloads an object from S3 with the specified key.
// The key is automatically prefixed with the configured prefix.
// Returns a ReadCloser containing the object data. The caller must
internal/s3/client_internal_test.go
+55
View File
@@ -0,0 +1,55 @@
package s3
import "testing"
// TestUploadPartSize checks that an upload too large for 10,000 parts of the
// configured size gets parts just large enough to fit in 10,000.
func TestUploadPartSize(t *testing.T) {
t.Parallel()
const mib = 1024 * 1024
tests := []struct {
name string
configured int64
size int64
want int64
}{
{
name: "an upload that fits keeps the configured size",
configured: 5 * mib,
size: 10 * 1024 * mib,
want: 5 * mib,
},
{
name: "exactly 10,000 parts keeps the configured size",
configured: 6 * mib,
size: 10_000 * 6 * mib,
want: 6 * mib,
},
{
name: "one byte more than 10,000 parts adds a byte to each",
configured: 6 * mib,
size: 10_000*6*mib + 1,
want: 6*mib + 1,
},
{
name: "zero means the SDK default of 5MiB",
configured: 0,
size: 1,
want: 5 * mib,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got := uploadPartSize(tt.configured, tt.size)
if got != tt.want {
t.Errorf("uploadPartSize(%d, %d) = %d, want %d",
tt.configured, tt.size, got, tt.want)
}
})
}
}
internal/s3/module.go
+1
View File
@@ -28,6 +28,7 @@ func provideClient(lc fx.Lifecycle, cfg *config.Config) (*Client, error) {
AccessKeyID: cfg.S3.AccessKeyID,
SecretAccessKey: cfg.S3.SecretAccessKey,
Region: cfg.S3.Region,
PartSize: cfg.S3.PartSize.Int64(),
})
if err != nil {
return nil, err
internal/snapshot/backup_test.go
+1 -1
View File
@@ -71,7 +71,7 @@ func verifyBackupFiles(
) {
t.Helper()
files, err := repos.Files.ListByPrefix(ctx, "")
files, err := repos.Files.ListAll(ctx)
if err != nil {
t.Fatalf("Failed to list files: %v", err)
}
internal/snapshot/export_perm_test.go
+57
View File
@@ -0,0 +1,57 @@
//nolint:testpackage // exercises the unexported copyDatabase helper
package snapshot
import (
"context"
"os"
"path/filepath"
"syscall"
"testing"
"github.com/spf13/afero"
"sneak.berlin/go/vaultik/internal/database"
)
// TestCopyDatabaseExportCopyMode verifies that the exported snapshot
// database copy is created owner-only (0600), even under a lenient 022
// umask that would otherwise leave a fresh file world-readable.
//
//nolint:paralleltest // syscall.Umask is process-global; parallel tests would clash
func TestCopyDatabaseExportCopyMode(t *testing.T) {
restore := syscall.Umask(0o022)
defer syscall.Umask(restore)
ctx := context.Background()
dir := t.TempDir()
src := filepath.Join(dir, "index.sqlite")
db, err := database.New(ctx, src)
if err != nil {
t.Fatalf("creating source index: %v", err)
}
err = db.Close()
if err != nil {
t.Fatalf("closing source index: %v", err)
}
dst := filepath.Join(dir, "snapshot.db")
sm := &SnapshotManager{fs: afero.NewOsFs()}
err = sm.copyDatabase(ctx, src, dst)
if err != nil {
t.Fatalf("copyDatabase: %v", err)
}
info, err := os.Stat(dst)
if err != nil {
t.Fatalf("stat export copy: %v", err)
}
got := info.Mode().Perm()
if got != 0o600 {
t.Fatalf("export copy permissions = %#o, want %#o", got, 0o600)
}
}
internal/snapshot/manifest.go
+30 -5
View File
@@ -7,6 +7,19 @@ import (
"io"
"github.com/klauspost/compress/zstd"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// Manifest size bounds. A manifest lists one small entry per blob, and
// blobs are large (the default target is 10 GB), so even a manifest for a
// petabyte-scale backup is a few megabytes. These caps are far above any
// manifest the writer can emit, yet stop a crafted, highly compressible
// manifest from expanding without limit when decoded: the manifest is
// fetched from the store, which is not trusted, and json.Decode buffers
// the whole value in memory.
const (
manifestMaxCompressed = 256 * 1024 * 1024 // 256 MiB
manifestMaxDecompressed = 1024 * 1024 * 1024 // 1 GiB
)
// Manifest represents the structure of a snapshot's blob manifest
@@ -28,19 +41,31 @@ type BlobInfo struct {
CompressedSize int64 `json:"compressed_size"`
}
// DecodeManifest decodes a manifest from a reader containing compressed JSON
// DecodeManifest decodes a manifest from a reader containing compressed
// JSON, reading through byte limits on both the compressed input and the
// decompressed output so an untrusted manifest cannot exhaust memory.
func DecodeManifest(r io.Reader) (*Manifest, error) {
// Decompress using zstd
zr, err := zstd.NewReader(r)
return decodeManifest(r, manifestMaxCompressed, manifestMaxDecompressed)
}
// decodeManifest is DecodeManifest with explicit limits, so tests can drive
// the bounds with small inputs instead of gigabyte-scale ones.
func decodeManifest(
r io.Reader, maxCompressed, maxDecompressed int64,
) (*Manifest, error) {
// Decompress using zstd, bounding how many compressed bytes are read.
zr, err := zstd.NewReader(blobgen.LimitReader(r, maxCompressed))
if err != nil {
return nil, fmt.Errorf("creating zstd reader: %w", err)
}
defer zr.Close()
// Decode JSON manifest
// Decode JSON manifest, bounding how far the compressed input may
// expand: json.Decode buffers the whole value, so without this a
// small, highly compressible manifest could expand to gigabytes.
var manifest Manifest
err = json.NewDecoder(zr).Decode(&manifest)
err = json.NewDecoder(blobgen.LimitReader(zr, maxDecompressed)).Decode(&manifest)
if err != nil {
return nil, fmt.Errorf("decoding manifest: %w", err)
}
internal/snapshot/manifest_bound_test.go
+79
View File
@@ -0,0 +1,79 @@
//nolint:testpackage // exercises the unexported decodeManifest bounds
package snapshot
import (
"bytes"
"strings"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// testSnapshotID is a stand-in snapshot ID reused across the bound cases.
const testSnapshotID = "host_home_2026-01-01T00:00:00Z"
// TestDecodeManifestRoundTrip is the baseline: with generous bounds a
// manifest the writer produced decodes back unchanged.
func TestDecodeManifestRoundTrip(t *testing.T) {
t.Parallel()
want := &Manifest{
SnapshotID: testSnapshotID,
Timestamp: "2026-01-01T00:00:00Z",
BlobCount: 2,
TotalCompressedSize: 42,
Blobs: []BlobInfo{
{Hash: "aa", CompressedSize: 21},
{Hash: "bb", CompressedSize: 21},
},
}
compressed, err := EncodeManifest(want, 3)
require.NoError(t, err)
got, err := decodeManifest(
bytes.NewReader(compressed), manifestMaxCompressed, manifestMaxDecompressed)
require.NoError(t, err)
require.Equal(t, want, got)
}
// TestDecodeManifestBoundsDecompressedOutput feeds a valid but highly
// compressible manifest — one whose timestamp is a megabyte of the same
// character — through a small decompressed bound. The compressed form is
// tiny, so only the decompressed bound stops it; decoding must fail within
// that bound rather than expanding the value in memory.
func TestDecodeManifestBoundsDecompressedOutput(t *testing.T) {
t.Parallel()
bomb := &Manifest{
SnapshotID: testSnapshotID,
Timestamp: strings.Repeat("a", 1<<20),
}
compressed, err := EncodeManifest(bomb, 3)
require.NoError(t, err)
require.Less(t, len(compressed), 4096,
"the compressible manifest must be small compressed")
_, err = decodeManifest(bytes.NewReader(compressed), 1<<20, 4096)
require.ErrorIs(t, err, blobgen.ErrOutputTooLarge)
}
// TestDecodeManifestBoundsCompressedInput checks the compressed-input
// bound fires independently: a valid manifest with a generous decompressed
// bound but a tiny compressed bound still fails.
func TestDecodeManifestBoundsCompressedInput(t *testing.T) {
t.Parallel()
manifest := &Manifest{
SnapshotID: testSnapshotID,
Timestamp: strings.Repeat("a", 4096),
}
compressed, err := EncodeManifest(manifest, 3)
require.NoError(t, err)
_, err = decodeManifest(bytes.NewReader(compressed), 8, manifestMaxDecompressed)
require.Error(t, err)
}
internal/snapshot/manifest_gen_test.go
+64
View File
@@ -0,0 +1,64 @@
//nolint:testpackage // exercises the unexported generateBlobManifest
package snapshot
import (
"context"
"path/filepath"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/types"
)
// TestGenerateBlobManifest_MissingBlobFails is the regression guard for
// issue #157: a blob the snapshot references but that is absent from the
// blobs table used to be logged and skipped, yielding a manifest with
// fewer blobs than the snapshot needs. Since prune trusts the manifest
// alone, that omitted blob would be deleted at the next prune. Manifest
// generation must fail instead.
func TestGenerateBlobManifest_MissingBlobFails(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
dbPath := filepath.Join(t.TempDir(), "snapshot.db")
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
repos := database.NewRepositories(db)
// A real blob row satisfies the snapshot_blobs foreign key on
// blob_id; the snapshot then references a different, absent hash.
presentBlob := &database.Blob{
ID: types.NewBlobID(),
Hash: types.BlobHash("present-blob-hash"),
CreatedTS: time.Now().Truncate(time.Second),
}
require.NoError(t, repos.Blobs.Create(ctx, nil, presentBlob))
snap := &database.Snapshot{
ID: "testhost_home_2026-05-01T00:00:00Z",
Hostname: "testhost",
}
require.NoError(t, repos.Snapshots.Create(ctx, nil, snap))
require.NoError(t, repos.Snapshots.AddBlob(ctx, nil,
snap.ID.String(), presentBlob.ID, types.BlobHash("absent-blob-hash")))
require.NoError(t, db.Close())
sm := &SnapshotManager{
config: &config.Config{CompressionLevel: 3},
fs: afero.NewOsFs(),
}
_, err = sm.generateBlobManifest(ctx, dbPath, snap.ID.String())
require.Error(t, err, "manifest generation must fail on a missing blob")
assert.Contains(t, err.Error(), "absent-blob-hash")
}
internal/snapshot/progress.go
+69 -57
View File
@@ -56,24 +56,27 @@ const (
// ProgressStats holds atomic counters for progress tracking
type ProgressStats struct {
FilesScanned atomic.Int64 // Total files seen during scan (includes skipped)
FilesProcessed atomic.Int64 // Files actually processed in phase 2
FilesSkipped atomic.Int64 // Files skipped due to no changes
BytesScanned atomic.Int64 // Bytes from new/changed files only
BytesSkipped atomic.Int64 // Bytes from unchanged files
BytesProcessed atomic.Int64 // Actual bytes processed (for ETA calculation)
ChunksCreated atomic.Int64
BlobsCreated atomic.Int64
BlobsUploaded atomic.Int64
BytesUploaded atomic.Int64
UploadDurationMs atomic.Int64 // Total milliseconds spent uploading
CurrentFile atomic.Value // stores string
TotalSize atomic.Int64 // Total size to process (set after scan phase)
TotalFiles atomic.Int64 // Total files to process in phase 2
ProcessStartTime atomic.Value // stores time.Time when processing starts
StartTime time.Time
mu sync.RWMutex
lastDetailTime time.Time
FilesScanned atomic.Int64 // Total files seen during scan (includes skipped)
FilesProcessed atomic.Int64 // Files actually processed in phase 2
FilesSkipped atomic.Int64 // Files skipped due to no changes
BytesScanned atomic.Int64 // Bytes from new/changed files only
BytesSkipped atomic.Int64 // Bytes from unchanged files
BytesProcessed atomic.Int64 // Actual bytes processed (for ETA calculation)
ChunksCreated atomic.Int64
BlobsCreated atomic.Int64
BlobsUploaded atomic.Int64
BytesUploaded atomic.Int64
CurrentFile atomic.Value // stores string
TotalSize atomic.Int64 // Size to process in the paths scanned so far
TotalFiles atomic.Int64 // Files to process in the paths scanned so far
StartTime time.Time
mu sync.RWMutex
lastDetailTime time.Time
// Guarded by mu: when the last scan phase ended, and
// BytesProcessed at that moment.
processStartTime time.Time
processStartBytes int64
// Upload tracking
CurrentUpload atomic.Value // stores *UploadInfo
@@ -149,10 +152,36 @@ func (pr *ProgressReporter) GetStats() *ProgressStats {
return pr.stats
}
// SetTotalSize sets the total size to process (after scan phase)
func (pr *ProgressReporter) SetTotalSize(size int64) {
pr.stats.TotalSize.Store(size)
pr.stats.ProcessStartTime.Store(time.Now().UTC())
// AddTotalSize adds the size one path of the snapshot has to process to
// the total, once that path's scan phase is done, and starts measuring
// the processing rate again. The processed counts run across every path,
// so the total does too. Restarting the rate keeps a path's scan phase,
// which processes nothing, out of the rate while that path is processed.
// During a later path's scan phase the rate is still the previous
// path's, and falls as that scan goes on.
func (pr *ProgressReporter) AddTotalSize(size int64) {
pr.stats.TotalSize.Add(size)
pr.stats.mu.Lock()
defer pr.stats.mu.Unlock()
pr.stats.processStartTime = time.Now().UTC()
pr.stats.processStartBytes = pr.stats.BytesProcessed.Load()
}
// processRate returns the bytes processed per second since the last scan
// phase ended, or 0 before the first path's scan phase is done.
func (s *ProgressStats) processRate() float64 {
s.mu.RLock()
defer s.mu.RUnlock()
if s.processStartTime.IsZero() {
return 0
}
processed := s.BytesProcessed.Load() - s.processStartBytes
return float64(processed) / time.Since(s.processStartTime).Seconds()
}
// Helper functions
@@ -231,9 +260,6 @@ func (pr *ProgressReporter) ReportUploadComplete(
// Clear current upload
pr.stats.CurrentUpload.Store((*UploadInfo)(nil))
// Add to total upload duration
pr.stats.UploadDurationMs.Add(duration.Milliseconds())
// Calculate speed
if duration < time.Millisecond {
duration = time.Millisecond
@@ -361,19 +387,12 @@ func (pr *ProgressReporter) printSummaryStatus() {
// Calculate ETA if we have total size and are processing
etaStr := ""
if totalSize > 0 && bytesProcessed > 0 {
processStart, ok := pr.stats.ProcessStartTime.Load().(time.Time)
if ok && !processStart.IsZero() {
processElapsed := time.Since(processStart)
rate := float64(bytesProcessed) / processElapsed.Seconds()
if rate > 0 {
remainingBytes := totalSize - bytesProcessed
remainingSeconds := float64(remainingBytes) / rate
eta := time.Duration(remainingSeconds * float64(time.Second))
etaStr = " | ETA: " + formatDuration(eta)
}
}
processRate := pr.stats.processRate()
if totalSize > 0 && processRate > 0 {
remainingBytes := totalSize - bytesProcessed
remainingSeconds := float64(remainingBytes) / processRate
eta := time.Duration(remainingSeconds * float64(time.Second))
etaStr = " | ETA: " + formatDuration(eta)
}
rate := float64(bytesScanned+bytesSkipped) / elapsed.Seconds()
@@ -425,25 +444,18 @@ func (pr *ProgressReporter) printDetailedStatus() {
log.Info("Elapsed time", "duration", formatDuration(elapsed))
// Calculate and show ETA if we have data
if totalSize > 0 && bytesProcessed > 0 {
processStart, ok := pr.stats.ProcessStartTime.Load().(time.Time)
if ok && !processStart.IsZero() {
processElapsed := time.Since(processStart)
processRate := float64(bytesProcessed) / processElapsed.Seconds()
if processRate > 0 {
remainingBytes := totalSize - bytesProcessed
remainingSeconds := float64(remainingBytes) / processRate
eta := time.Duration(remainingSeconds * float64(time.Second))
percentComplete := float64(bytesProcessed) / float64(totalSize) * percentScale
log.Info("Overall progress",
"percent", fmt.Sprintf("%.1f%%", percentComplete),
"processed", humanize.Bytes(safeUint64(bytesProcessed)),
"total", humanize.Bytes(safeUint64(totalSize)),
"rate", humanize.Bytes(uint64(processRate))+"/s",
"eta", formatDuration(eta))
}
}
processRate := pr.stats.processRate()
if totalSize > 0 && processRate > 0 {
remainingBytes := totalSize - bytesProcessed
remainingSeconds := float64(remainingBytes) / processRate
eta := time.Duration(remainingSeconds * float64(time.Second))
percentComplete := float64(bytesProcessed) / float64(totalSize) * percentScale
log.Info("Overall progress",
"percent", fmt.Sprintf("%.1f%%", percentComplete),
"processed", humanize.Bytes(safeUint64(bytesProcessed)),
"total", humanize.Bytes(safeUint64(totalSize)),
"rate", humanize.Bytes(uint64(processRate))+"/s",
"eta", formatDuration(eta))
}
log.Info("Files processed",
internal/snapshot/progress_rate_test.go
+54
View File
@@ -0,0 +1,54 @@
//nolint:testpackage // exercises the unexported processRate helper
package snapshot
import (
"testing"
"time"
)
// TestProcessRateNotLoweredByLaterScanPhase feeds the reporter two paths
// that each take 10 seconds to process, the second after a 30-second scan
// phase. That scan phase processes nothing, so the rate while the second
// path is processed must be that path's own.
func TestProcessRateNotLoweredByLaterScanPhase(t *testing.T) {
t.Parallel()
const (
pathSize = 1000
processingTime = 10 * time.Second
scanTime = 30 * time.Second
)
// Never started, but Stop releases its tickers and signal handler.
progress := NewProgressReporter()
defer progress.Stop()
stats := progress.GetStats()
// Moves the processing start time back instead of sleeping.
elapse := func(d time.Duration) {
stats.mu.Lock()
defer stats.mu.Unlock()
stats.processStartTime = stats.processStartTime.Add(-d)
}
progress.AddTotalSize(pathSize)
stats.BytesProcessed.Add(pathSize)
elapse(processingTime)
elapse(scanTime)
progress.AddTotalSize(pathSize)
stats.BytesProcessed.Add(pathSize)
elapse(processingTime)
want := pathSize / processingTime.Seconds()
got := stats.processRate()
// The test's own run time adds to the elapsed time, so got is a hair
// under want.
if got > want || got < want*0.99 {
t.Errorf("rate while the second path is processed is %.1f bytes/s, "+
"want %.1f", got, want)
}
}
internal/snapshot/progress_test.go
+94
View File
@@ -0,0 +1,94 @@
package snapshot_test
import (
"context"
"path/filepath"
"strings"
"testing"
"github.com/spf13/afero"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/snapshot"
)
// TestProgressPercentWithSecondPathSmaller backs up two paths with one
// scanner, as a snapshot with two paths does, the second path smaller
// than the first. The progress line divides the bytes processed by the
// total size, so both must count both paths to stay within 100%.
func TestProgressPercentWithSecondPathSmaller(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
files := map[string]string{
"/large/one.txt": strings.Repeat("1", 4000),
"/large/two.txt": strings.Repeat("2", 4000),
"/small/three.txt": strings.Repeat("3", 1000),
}
for path, content := range files {
err := fs.MkdirAll(filepath.Dir(path), 0755)
if err != nil {
t.Fatalf("mkdir: %v", err)
}
err = afero.WriteFile(fs, path, []byte(content), 0644)
if err != nil {
t.Fatalf("write %s: %v", path, err)
}
}
db, err := database.NewTestDB()
if err != nil {
t.Fatalf("create test db: %v", err)
}
t.Cleanup(func() {
cerr := db.Close()
if cerr != nil {
t.Errorf("close db: %v", cerr)
}
})
repos := database.NewRepositories(db)
scanner := snapshot.NewScanner(snapshot.ScannerConfig{
FS: fs,
ChunkSize: int64(1024 * 16),
Repositories: repos,
MaxBlobSize: int64(1024 * 1024),
CompressionLevel: 3,
AgeRecipients: []string{testAgePublicKey},
EnableProgress: true,
})
// Never started, but Stop releases its tickers and signal handler.
progress := scanner.GetProgress()
defer progress.Stop()
ctx := context.Background()
snapshotID := "test-snapshot-progress"
createTestSnapshotRecord(ctx, t, repos, snapshotID)
for _, path := range []string{"/large", "/small"} {
_, err := scanner.Scan(ctx, path, snapshotID)
if err != nil {
t.Fatalf("scanning %s: %v", path, err)
}
}
stats := progress.GetStats()
// Directories count toward the total size but produce no chunks, so
// the percentage ends just under 100%.
percent := float64(stats.BytesProcessed.Load()) /
float64(stats.TotalSize.Load()) * 100
if percent > 100 {
t.Errorf("progress after both paths is %.1f%%, want at most 100%%",
percent)
}
if stats.FilesProcessed.Load() != stats.TotalFiles.Load() {
t.Errorf("progress after both paths is %d of %d files, want all",
stats.FilesProcessed.Load(), stats.TotalFiles.Load())
}
}

Some files were not shown because too many files have changed in this diff Show More